From ab6393b7c88b5070d7328d6af8f00e441a648c34 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 17:33:48 -0800 Subject: [PATCH 01/10] burn down broken files with new mdx transform tool Co-authored-by: promptless[bot] --- about/faq.mdx | 74 +- about/intro.mdx | 95 +- about/roadmap.mdx | 88 +- about/vision.mdx | 110 +- about/why.mdx | 66 +- .../build-performance-breakdown.mdx | 248 +- .../performance/build-performance-metrics.mdx | 76 +- advanced/performance/iteration-speed.mdx | 84 +- advanced/performance/json-trace-profile.mdx | 102 +- advanced/performance/memory.mdx | 60 +- basics/artifact-based-builds.mdx | 277 +-- basics/build-systems.mdx | 136 +- basics/dependencies.mdx | 306 +-- basics/distributed-builds.mdx | 128 +- basics/hermeticity.mdx | 117 +- basics/index.mdx | 54 +- basics/task-based-builds.mdx | 238 +- brand/index.mdx | 84 +- build/share-variables.mdx | 31 +- build/style-guide.mdx | 279 +-- community/recommended-rules.mdx | 50 +- community/remote-execution-services.mdx | 31 +- community/sig.mdx | 146 +- community/users.mdx | 533 ++-- concepts/build-ref.mdx | 100 +- concepts/platforms.mdx | 283 +-- concepts/visibility.mdx | 331 +-- configure/attributes.mdx | 338 +-- configure/best-practices.mdx | 67 +- configure/coverage.mdx | 106 +- configure/windows.mdx | 240 +- contribute/breaking-changes.mdx | 112 +- contribute/codebase.mdx | 1881 ++++----------- contribute/design-documents.mdx | 238 +- contribute/docs.mdx | 32 +- contribute/index.mdx | 76 +- contribute/maintainers-guide.mdx | 254 +- contribute/naming.mdx | 62 +- contribute/patch-acceptance.mdx | 65 +- contribute/policy.mdx | 103 +- contribute/release-notes.mdx | 67 +- contribute/statemachine-guide.mdx | 568 ++--- contribute/windows-chocolatey-maintenance.mdx | 50 +- contribute/windows-scoop-maintenance.mdx | 28 +- copy-upstream-docs.sh | 17 +- docs/android-build-performance.mdx | 70 +- docs/android-instrumentation-test.mdx | 192 +- docs/android-ndk.mdx | 147 +- docs/bazel-and-android.mdx | 43 +- docs/bazel-and-apple.mdx | 87 +- docs/bazel-and-cpp.mdx | 125 +- docs/bazel-and-java.mdx | 219 +- docs/bazel-and-javascript.mdx | 30 +- docs/configurable-attributes.mdx | 337 +-- docs/mobile-install.mdx | 183 +- docs/sandboxing.mdx | 144 +- extending/aspects.mdx | 202 +- extending/auto-exec-groups.mdx | 97 +- extending/concepts.mdx | 109 +- extending/depsets.mdx | 130 +- extending/exec-groups.mdx | 106 +- extending/legacy-macros.mdx | 137 +- extending/macros.mdx | 270 +-- extending/platforms.mdx | 170 +- extending/rules.mdx | 879 ++----- extending/toolchains.mdx | 314 +-- external/extension.mdx | 202 +- external/faq.mdx | 327 +-- external/lockfile.mdx | 201 +- external/migration.mdx | 979 +++----- external/module.mdx | 222 +- external/overview.mdx | 261 +- external/repo.mdx | 141 +- external/vendor.mdx | 136 +- help.mdx | 52 +- install/bazelisk.mdx | 53 +- install/compile-source.mdx | 262 +- install/completion.mdx | 177 +- install/docker-container.mdx | 28 +- install/ide.mdx | 81 +- install/index.mdx | 32 +- install/os-x.mdx | 82 +- install/suse.mdx | 16 +- install/ubuntu.mdx | 82 +- install/windows.mdx | 185 +- migrate/index.mdx | 6 +- migrate/maven.mdx | 250 +- migrate/xcode.mdx | 229 +- query/aquery.mdx | 177 +- query/cquery.mdx | 350 +-- query/guide.mdx | 206 +- reference/glossary.mdx | 516 +--- reference/skyframe.mdx | 211 +- release/backward-compatibility.mdx | 70 +- release/index.mdx | 254 +- release/rolling.mdx | 11 +- release/rule-compatibility.mdx | 110 +- remote/bep-examples.mdx | 110 +- remote/bep-glossary.mdx | 148 +- remote/bep.mdx | 143 +- remote/cache-local.mdx | 68 +- remote/cache-remote.mdx | 152 +- remote/caching.mdx | 330 +-- remote/ci.mdx | 173 +- remote/creating.mdx | 195 +- remote/multiplex.mdx | 108 +- remote/output-directories.mdx | 133 +- remote/persistent.mdx | 219 +- remote/rbe.mdx | 28 +- remote/rules.mdx | 196 +- remote/sandbox.mdx | 210 +- remote/workspace.mdx | 122 +- rules/bzl-style.mdx | 221 +- rules/challenges.mdx | 230 +- rules/deploying.mdx | 143 +- rules/errors/read-only-variable.mdx | 9 +- rules/faq.mdx | 57 +- rules/index.mdx | 81 +- rules/language.mdx | 127 +- rules/legacy-macro-tutorial.mdx | 49 +- rules/macro-tutorial.mdx | 37 +- rules/performance.mdx | 147 +- rules/rules-tutorial.mdx | 126 +- rules/testing.mdx | 157 +- rules/verbs-tutorial.mdx | 118 +- run/bazelrc.mdx | 243 +- run/client-server.mdx | 54 +- run/scripts.mdx | 136 +- start/android-app.mdx | 261 +- start/cpp.mdx | 178 +- start/go.mdx | 224 +- start/ios-app.mdx | 3 +- start/java.mdx | 233 +- tools/mdx-transform/package-lock.json | 2142 +++++++++++++++++ tools/mdx-transform/package.json | 34 + tools/mdx-transform/transform.mjs | 357 +++ transform-docs.awk | 357 ++- tutorials/ccp-toolchain-config.mdx | 832 +++---- tutorials/cpp-dependency.mdx | 31 +- tutorials/cpp-labels.mdx | 20 +- tutorials/cpp-use-cases.mdx | 55 +- versions/index.mdx | 12 +- 142 files changed, 9845 insertions(+), 17495 deletions(-) create mode 100644 tools/mdx-transform/package-lock.json create mode 100644 tools/mdx-transform/package.json create mode 100644 tools/mdx-transform/transform.mjs diff --git a/about/faq.mdx b/about/faq.mdx index dd5be8a9..fc0c31d1 100644 --- a/about/faq.mdx +++ b/about/faq.mdx @@ -2,8 +2,6 @@ title: 'FAQ' --- - - If you have questions or need support, see [Getting Help](/help). ## What is Bazel? @@ -14,19 +12,19 @@ Bazel is a tool that automates software builds and tests. Supported build tasks Bazel was designed to fit the way software is developed at Google. It has the following features: -* Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. -* High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. -* Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. -* Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. -* Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about ~200ms. +- Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. +- High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. +- Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. +- Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. +- Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about \~200ms. ## Why doesn’t Google use...? -* Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. - * Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. -* Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. -* Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. -* Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. +- Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. + - Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. +- Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. +- Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. +- Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. ## Where did Bazel come from? @@ -48,10 +46,10 @@ Bazel runs build operations locally by default. However, Bazel can also connect For our server code base, we use the following development workflow: -* All our server code is in a single, gigantic version control system. -* Everybody builds their software with Bazel. -* Different teams own different parts of the source tree, and make their components available as `BUILD` targets. -* Branching is primarily used for managing releases, so everybody develops their software at the head revision. +- All our server code is in a single, gigantic version control system. +- Everybody builds their software with Bazel. +- Different teams own different parts of the source tree, and make their components available as `BUILD` targets. +- Branching is primarily used for managing releases, so everybody develops their software at the head revision. Bazel is a cornerstone of this philosophy: since Bazel requires all dependencies to be fully specified, we can predict which programs and tests are affected by a change, and vet them before submission. @@ -63,24 +61,22 @@ Building software should be fun and easy. Slow and unpredictable builds take the ## Why would I want to use Bazel? -* Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. -* Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. -* Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. +- Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. +- Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. +- Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. ## Can I see examples? -Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) -or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. - +Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. ## What is Bazel best at? Bazel shines at building and testing projects with the following properties: -* Projects with a large codebase -* Projects written in (multiple) compiled languages -* Projects that deploy on multiple platforms -* Projects that have extensive tests +- Projects with a large codebase +- Projects written in (multiple) compiled languages +- Projects that deploy on multiple platforms +- Projects that have extensive tests ## Where can I run Bazel? @@ -90,11 +86,13 @@ Porting to other UNIX platforms should be relatively easy, as long as a JDK is a ## What should I not use Bazel for? -* Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: - * A compilation step that fetches data from the internet. - * A test step that connects to the QA instance of your site. - * A deployment step that changes your site’s cloud configuration. -* If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. +- Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: + + - A compilation step that fetches data from the internet. + - A test step that connects to the QA instance of your site. + - A deployment step that changes your site’s cloud configuration. + +- If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. ## How stable is Bazel’s feature set? @@ -132,10 +130,10 @@ Yes, you can use our [Docker rules](https://github.com/bazelbuild/rules_docker) For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (for example, executing binaries through a shell script inside a rule), you will need to take some extra care: -* Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. -* Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. -* Avoid connecting to the network. Sandboxed execution can help here too. -* Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. +- Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. +- Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. +- Avoid connecting to the network. Sandboxed execution can help here too. +- Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. ## Do you have binary releases? @@ -179,8 +177,8 @@ We still have to refactor the interfaces between the public code in Bazel and ou Open sourcing Bazel is a work-in-progress. In particular, we’re still working on open sourcing: -* Many of our unit and integration tests (which should make contributing patches easier). -* Full IDE integration. +- Many of our unit and integration tests (which should make contributing patches easier). +- Full IDE integration. Beyond code, we’d like to eventually have all code reviews, bug tracking, and design decisions happen publicly, with the Bazel community involved. We are not there yet, so some changes will simply appear in the Bazel repository without clear explanation. Despite this lack of transparency, we want to support external developers and collaborate. Thus, we are opening up the code, even though some of the development is still happening internal to Google. Please let us know if anything seems unclear or unjustified as we transition to an open model. @@ -190,7 +188,7 @@ Yes, some of the code base either integrates with Google-specific technology or ## How do I contact the team? -We are reachable at bazel-discuss@googlegroups.com. +We are reachable at [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com). ## Where do I report bugs? diff --git a/about/intro.mdx b/about/intro.mdx index a531ac2a..a70f715b 100644 --- a/about/intro.mdx +++ b/about/intro.mdx @@ -2,110 +2,63 @@ title: 'Intro to Bazel' --- - - -Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. -It uses a human-readable, high-level build language. Bazel supports projects in -multiple languages and builds outputs for multiple platforms. Bazel supports -large codebases across multiple repositories, and large numbers of users. +Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users. ## Benefits Bazel offers the following advantages: -* **High-level build language.** Bazel uses an abstract, human-readable - language to describe the build properties of your project at a high - semantical level. Unlike other tools, Bazel operates on the *concepts* - of libraries, binaries, scripts, and data sets, shielding you from the - complexity of writing individual calls to tools such as compilers and - linkers. +- **High-level build language.** Bazel uses an abstract, human-readable language to describe the build properties of your project at a high semantical level. Unlike other tools, Bazel operates on the *concepts* of libraries, binaries, scripts, and data sets, shielding you from the complexity of writing individual calls to tools such as compilers and linkers. -* **Bazel is fast and reliable.** Bazel caches all previously done work and - tracks changes to both file content and build commands. This way, Bazel - knows when something needs to be rebuilt, and rebuilds only that. To further - speed up your builds, you can set up your project to build in a highly - parallel and incremental fashion. +- **Bazel is fast and reliable.** Bazel caches all previously done work and tracks changes to both file content and build commands. This way, Bazel knows when something needs to be rebuilt, and rebuilds only that. To further speed up your builds, you can set up your project to build in a highly parallel and incremental fashion. -* **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel - can build binaries and deployable packages for multiple platforms, including - desktop, server, and mobile, from the same project. +- **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel can build binaries and deployable packages for multiple platforms, including desktop, server, and mobile, from the same project. -* **Bazel scales.** Bazel maintains agility while handling builds with 100k+ - source files. It works with multiple repositories and user bases in the tens - of thousands. +- **Bazel scales.** Bazel maintains agility while handling builds with 100k+ source files. It works with multiple repositories and user bases in the tens of thousands. -* **Bazel is extensible.** Many [languages](/rules) are - supported, and you can extend Bazel to support any other language or - framework. +- **Bazel is extensible.** Many [languages](/rules) are supported, and you can extend Bazel to support any other language or framework. ## Using Bazel To build or test a project with Bazel, you typically do the following: -1. **Set up Bazel.** Download and [install Bazel](/install). +1. **Set up Bazel.** Download and [install Bazel](/install). -2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a - directory where Bazel looks for build inputs and `BUILD` files, and where it - stores build outputs. +2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a directory where Bazel looks for build inputs and `BUILD` files, and where it stores build outputs. -3. **Write a `BUILD` file**, which tells Bazel what to build and how to - build it. +3. **Write a `BUILD` file**, which tells Bazel what to build and how to build it. - You write your `BUILD` file by declaring build targets using - [Starlark](/rules/language), a domain-specific language. (See example - [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) + You write your `BUILD` file by declaring build targets using [Starlark](/rules/language), a domain-specific language. (See example [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) - A build target specifies a set of input artifacts that Bazel will build plus - their dependencies, the build rule Bazel will use to build it, and options - that configure the build rule. + A build target specifies a set of input artifacts that Bazel will build plus their dependencies, the build rule Bazel will use to build it, and options that configure the build rule. - A build rule specifies the build tools Bazel will use, such as compilers and - linkers, and their configurations. Bazel ships with a number of build rules - covering the most common artifact types in the supported languages on - supported platforms. + A build rule specifies the build tools Bazel will use, such as compilers and linkers, and their configurations. Bazel ships with a number of build rules covering the most common artifact types in the supported languages on supported platforms. -4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel - places your outputs within the workspace. +4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel places your outputs within the workspace. -In addition to building, you can also use Bazel to run -[tests](/reference/test-encyclopedia) and [query](/query/guide) the build -to trace dependencies in your code. +In addition to building, you can also use Bazel to run [tests](/reference/test-encyclopedia) and [query](/query/guide) the build to trace dependencies in your code. ## Bazel build process When running a build or a test, Bazel does the following: -1. **Loads** the `BUILD` files relevant to the target. +1. **Loads** the `BUILD` files relevant to the target. -2. **Analyzes** the inputs and their - [dependencies](/concepts/dependencies), applies the specified build - rules, and produces an [action](/extending/concepts#evaluation-model) - graph. +2. **Analyzes** the inputs and their [dependencies](/concepts/dependencies), applies the specified build rules, and produces an [action](/extending/concepts#evaluation-model) graph. -3. **Executes** the build actions on the inputs until the final build outputs - are produced. +3. **Executes** the build actions on the inputs until the final build outputs are produced. -Since all previous build work is cached, Bazel can identify and reuse cached -artifacts and only rebuild or retest what's changed. To further enforce -correctness, you can set up Bazel to run builds and tests -[hermetically](/basics/hermeticity) through sandboxing, minimizing skew -and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). +Since all previous build work is cached, Bazel can identify and reuse cached artifacts and only rebuild or retest what's changed. To further enforce correctness, you can set up Bazel to run builds and tests [hermetically](/basics/hermeticity) through sandboxing, minimizing skew and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). ### Action graph -The action graph represents the build artifacts, the relationships between them, -and the build actions that Bazel will perform. Thanks to this graph, Bazel can -[track](/run/build#build-consistency) changes to -file content as well as changes to actions, such as build or test commands, and -know what build work has previously been done. The graph also enables you to -easily [trace dependencies](/query/guide) in your code. +The action graph represents the build artifacts, the relationships between them, and the build actions that Bazel will perform. Thanks to this graph, Bazel can [track](/run/build#build-consistency) changes to file content as well as changes to actions, such as build or test commands, and know what build work has previously been done. The graph also enables you to easily [trace dependencies](/query/guide) in your code. ## Getting started tutorials -To get started with Bazel, see [Getting Started](/start/) or jump -directly to the Bazel tutorials: +To get started with Bazel, see [Getting Started](/start/) or jump directly to the Bazel tutorials: -* [Tutorial: Build a C++ Project](/start/cpp) -* [Tutorial: Build a Java Project](/start/java) -* [Tutorial: Build an Android Application](/start/android-app) -* [Tutorial: Build an iOS Application](/start/ios-app) +- [Tutorial: Build a C++ Project](/start/cpp) +- [Tutorial: Build a Java Project](/start/java) +- [Tutorial: Build an Android Application](/start/android-app) +- [Tutorial: Build an iOS Application](/start/ios-app) diff --git a/about/roadmap.mdx b/about/roadmap.mdx index 42e63e9b..c17485d2 100644 --- a/about/roadmap.mdx +++ b/about/roadmap.mdx @@ -2,98 +2,50 @@ title: 'Bazel roadmap' --- +As Bazel continues to evolve in response to your needs, we want to share our 2025 roadmap update. - -As Bazel continues to evolve in response to your needs, we want to share our -2025 roadmap update. - -We plan to bring Bazel 9.0 -[long term support (LTS)](https://bazel.build/release/versioning) to you in late -2025. +We plan to bring Bazel 9.0 [long term support (LTS)](https://bazel.build/release/versioning) to you in late 2025. ## Full transition to Bzlmod -[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external -dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. -As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) -hosts more than 650 modules. +[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) hosts more than 650 modules. -With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will -be the only way to introduce external dependencies in Bazel. To minimize the -migration cost for the community, we'll focus on further improving our migration -[guide](https://bazel.build/external/migration) and -[tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). +With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will be the only way to introduce external dependencies in Bazel. To minimize the migration cost for the community, we'll focus on further improving our migration [guide](https://bazel.build/external/migration) and [tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). -Additionally, we aim to implement an improved shared repository cache (see -[#12227](https://github.com/bazelbuild/bazel/issues/12227)) -with garbage collection, and may backport it to Bazel 8. The Bazel Central -Registry will also support verifying SLSA attestations. +Additionally, we aim to implement an improved shared repository cache (see [#12227](https://github.com/bazelbuild/bazel/issues/12227)) with garbage collection, and may backport it to Bazel 8. The Bazel Central Registry will also support verifying SLSA attestations. ## Migration of Android, C++, Java, Python, and Proto rules -With Bazel 8, we have migrated support for Android, Java, Python, and Proto -rules out of the Bazel codebase into Starlark rules in their corresponding -repositories. To ease the migration, we implemented the autoload features in -Bazel, which can be controlled with -[--incompatible_autoload_externally](https://github.com/bazelbuild/bazel/issues/23043) -and [--incompatible_disable_autoloads_in_main_repo](https://github.com/bazelbuild/bazel/issues/25755) -flags. +With Bazel 8, we have migrated support for Android, Java, Python, and Proto rules out of the Bazel codebase into Starlark rules in their corresponding repositories. To ease the migration, we implemented the autoload features in Bazel, which can be controlled with [--incompatible\_autoload\_externally](https://github.com/bazelbuild/bazel/issues/23043) and [--incompatible\_disable\_autoloads\_in\_main\_repo](https://github.com/bazelbuild/bazel/issues/25755) flags. -With Bazel 9, we aim to disable autoloads by default and require every project -to explicitly load required rules in BUILD files. +With Bazel 9, we aim to disable autoloads by default and require every project to explicitly load required rules in BUILD files. -We will rewrite most of C++ language support to Starlark, detach it from Bazel -binary and move it into the [/rules_cc](https://github.com/bazelbuild/rules_cc) -repository. This is the last remaining major language support that is still part -of Bazel. +We will rewrite most of C++ language support to Starlark, detach it from Bazel binary and move it into the [/rules\_cc](https://github.com/bazelbuild/rules_cc) repository. This is the last remaining major language support that is still part of Bazel. -We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving -them to repositories next to the implementation to increase velocity of rule -authors. +We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving them to repositories next to the implementation to increase velocity of rule authors. ## Starlark improvements -Bazel will have the ability to evaluate symbolic macros lazily. This means that -a symbolic macro won't run if the targets it declares are not requested, -improving performance for very large packages. +Bazel will have the ability to evaluate symbolic macros lazily. This means that a symbolic macro won't run if the targets it declares are not requested, improving performance for very large packages. -Starlark will have an experimental type system, similar to Python's type -annotations. We expect the type system to stabilize _after_ Bazel 9 is launched. +Starlark will have an experimental type system, similar to Python's type annotations. We expect the type system to stabilize *after* Bazel 9 is launched. ## Configurability Our main focus is reducing the cost and confusion of build flags. -We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a -new project configuration model that doesn't make users have to know which build -and test flags to set where. So `$ bazel test //foo` automatically sets the -right flags based on `foo`'s project's policy. This will likely remain -experimental in 9.0 but guiding feedback is welcome. - -[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip -out Starlark flags when they leave project boundaries, so they don't break -caching on transitive dependencies that don't need them. This makes builds that -use [transitions](https://bazel.build/extending/config#user-defined-transitions) -cheaper and faster. -[Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) -an example. We're extending the idea to control which flags propagate to -[exec configurations](https://bazel.build/extending/rules#configurations) and -are considering even more flexible support like custom Starlark to determine -which dependency edges should propagate flags. - -We're up-prioritizing effort to move built-in language flags out of Bazel and -into Starlark, where they can live with related rule definitions. +We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a new project configuration model that doesn't make users have to know which build and test flags to set where. So `$ bazel test //foo` automatically sets the right flags based on `foo`'s project's policy. This will likely remain experimental in 9.0 but guiding feedback is welcome. + +[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip out Starlark flags when they leave project boundaries, so they don't break caching on transitive dependencies that don't need them. This makes builds that use [transitions](https://bazel.build/extending/config#user-defined-transitions) cheaper and faster. [Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) an example. We're extending the idea to control which flags propagate to [exec configurations](https://bazel.build/extending/rules#configurations) and are considering even more flexible support like custom Starlark to determine which dependency edges should propagate flags. + +We're up-prioritizing effort to move built-in language flags out of Bazel and into Starlark, where they can live with related rule definitions. ## Remote execution improvements -We plan to add support for asynchronous execution, speeding up remote execution -by increasing parallelism. +We plan to add support for asynchronous execution, speeding up remote execution by increasing parallelism. ---- +*** -To follow updates to the roadmap and discuss planned features, join the -community Slack server at [slack.bazel.build](https://slack.bazel.build/). +To follow updates to the roadmap and discuss planned features, join the community Slack server at [slack.bazel.build](https://slack.bazel.build/). -*This roadmap is intended to help inform the community about the team's -intentions for Bazel 9.0. Priorities are subject to change in response to -developer and customer feedback, or to new market opportunities.* +*This roadmap is intended to help inform the community about the team's intentions for Bazel 9.0. Priorities are subject to change in response to developer and customer feedback, or to new market opportunities.* diff --git a/about/vision.mdx b/about/vision.mdx index da0ed02d..b80ca03a 100644 --- a/about/vision.mdx +++ b/about/vision.mdx @@ -2,96 +2,48 @@ title: 'Bazel Vision' --- +Any software developer can efficiently build, test, and package any project, of any size or complexity, with tooling that's easy to adopt and extend. +- **Engineers can take build fundamentals for granted.** Software developers focus on the creative process of authoring code because the mechanical process of build and test is solved. When customizing the build system to support new languages or unique organizational needs, users focus on the aspects of extensibility that are unique to their use case, without having to reinvent the basic plumbing. -Any software developer can efficiently build, test, and package -any project, of any size or complexity, with tooling that's easy to adopt and -extend. +- **Engineers can easily contribute to any project.** A developer who wants to start working on a new project can simply clone the project and run the build. There's no need for local configuration - it just works. With cross-platform remote execution, they can work on any machine anywhere and fully test their changes against all platforms the project targets. Engineers can quickly configure the build for a new project or incrementally migrate an existing build. -* **Engineers can take build fundamentals for granted.** Software developers - focus on the creative process of authoring code because the mechanical - process of build and test is solved. When customizing the build system to - support new languages or unique organizational needs, users focus on the - aspects of extensibility that are unique to their use case, without having - to reinvent the basic plumbing. - -* **Engineers can easily contribute to any project.** A developer who wants to - start working on a new project can simply clone the project and run the - build. There's no need for local configuration - it just works. With - cross-platform remote execution, they can work on any machine anywhere and - fully test their changes against all platforms the project targets. - Engineers can quickly configure the build for a new project or incrementally - migrate an existing build. - -* **Projects can scale to any size codebase, any size team.** Fast, - incremental testing allows teams to fully validate every change before it is - committed. This remains true even as repos grow, projects span multiple - repos, and multiple languages are introduced. Infrastructure does not force - developers to trade test coverage for build speed. +- **Projects can scale to any size codebase, any size team.** Fast, incremental testing allows teams to fully validate every change before it is committed. This remains true even as repos grow, projects span multiple repos, and multiple languages are introduced. Infrastructure does not force developers to trade test coverage for build speed. **We believe Bazel has the potential to fulfill this vision.** -Bazel was built from the ground up to enable builds that are reproducible (a -given set of inputs will always produce the same outputs) and portable (a build -can be run on any machine without affecting the output). +Bazel was built from the ground up to enable builds that are reproducible (a given set of inputs will always produce the same outputs) and portable (a build can be run on any machine without affecting the output). -These characteristics support safe incrementality (rebuilding only changed -inputs doesn't introduce the risk of corruption) and distributability (build -actions are isolated and can be offloaded). By minimizing the work needed to do -a correct build and parallelizing that work across multiple cores and remote -systems, Bazel can make any build fast. +These characteristics support safe incrementality (rebuilding only changed inputs doesn't introduce the risk of corruption) and distributability (build actions are isolated and can be offloaded). By minimizing the work needed to do a correct build and parallelizing that work across multiple cores and remote systems, Bazel can make any build fast. -Bazel's abstraction layer — instructions specific to languages, platforms, and -toolchains implemented in a simple extensibility language — allows it to be -easily applied to any context. +Bazel's abstraction layer — instructions specific to languages, platforms, and toolchains implemented in a simple extensibility language — allows it to be easily applied to any context. ## Bazel core competencies -1. Bazel supports **multi-language, multi-platform** builds and tests. You can - run a single command to build and test your entire source tree, no matter - which combination of languages and platforms you target. -1. Bazel builds are **fast and correct**. Every build and test run is - incremental, on your developers' machines and on CI. -1. Bazel provides a **uniform, extensible language** to define builds for any - language or platform. -1. Bazel allows your builds **to scale** by connecting to remote execution and - caching services. -1. Bazel works across **all major development platforms** (Linux, MacOS, and - Windows). -1. We accept that adopting Bazel requires effort, but **gradual adoption** is - possible. Bazel interfaces with de-facto standard tools for a given - language/platform. +1. Bazel supports **multi-language, multi-platform** builds and tests. You can run a single command to build and test your entire source tree, no matter which combination of languages and platforms you target. +2. Bazel builds are **fast and correct**. Every build and test run is incremental, on your developers' machines and on CI. +3. Bazel provides a **uniform, extensible language** to define builds for any language or platform. +4. Bazel allows your builds **to scale** by connecting to remote execution and caching services. +5. Bazel works across **all major development platforms** (Linux, MacOS, and Windows). +6. We accept that adopting Bazel requires effort, but **gradual adoption** is possible. Bazel interfaces with de-facto standard tools for a given language/platform. ## Serving language communities -Software engineering evolves in the context of language communities — typically, -self-organizing groups of people who use common tools and practices. - -To be of use to members of a language community, high-quality Bazel rules must be -available that integrate with the workflows and conventions of that community. - -Bazel is committed to be extensible and open, and to support good rulesets for -any language. - -### Requirements of a good ruleset - -1. The rules need to support efficient **building and testing** for the - language, including code coverage. -1. The rules need to **interface with a widely-used "package manager"** for the - language (such as Maven for Java), and support incremental migration paths - from other widely-used build systems. -1. The rules need to be **extensible and interoperable**, following - ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) - principles. -1. The rules need to be **remote-execution ready**. In practice, this means - **configurable using the [toolchains](/extending/toolchains) mechanism**. -1. The rules (and Bazel) need to interface with a **widely-used IDE** for the - language, if there is one. -1. The rules need to have **thorough, usable documentation,** with introductory - material for new users, comprehensive docs for expert users. - -Each of these items is essential and only together do they deliver on Bazel's -competencies for their particular ecosystem. - -They are also, by and large, sufficient - once all are fulfilled, Bazel fully -delivers its value to members of that language community. +Software engineering evolves in the context of language communities — typically, self-organizing groups of people who use common tools and practices. + +To be of use to members of a language community, high-quality Bazel rules must be available that integrate with the workflows and conventions of that community. + +Bazel is committed to be extensible and open, and to support good rulesets for any language. + +### Requirements of a good ruleset + +1. The rules need to support efficient **building and testing** for the language, including code coverage. +2. The rules need to **interface with a widely-used "package manager"** for the language (such as Maven for Java), and support incremental migration paths from other widely-used build systems. +3. The rules need to be **extensible and interoperable**, following ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) principles. +4. The rules need to be **remote-execution ready**. In practice, this means **configurable using the [toolchains](/extending/toolchains) mechanism**. +5. The rules (and Bazel) need to interface with a **widely-used IDE** for the language, if there is one. +6. The rules need to have **thorough, usable documentation,** with introductory material for new users, comprehensive docs for expert users. + +Each of these items is essential and only together do they deliver on Bazel's competencies for their particular ecosystem. + +They are also, by and large, sufficient - once all are fulfilled, Bazel fully delivers its value to members of that language community. diff --git a/about/why.mdx b/about/why.mdx index 6224abf6..dcdec18e 100644 --- a/about/why.mdx +++ b/about/why.mdx @@ -2,84 +2,44 @@ title: 'Why Bazel?' --- - - -Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) -build tool with [integrated testing](#integrated-testing) that supports multiple -[languages](#multi-language), [repositories](#multi-repository), and -[platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). +Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) build tool with [integrated testing](#integrated-testing) that supports multiple [languages](#multi-language), [repositories](#multi-repository), and [platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). ## Bazel is fast -Bazel knows exactly what input files each build command needs, avoiding -unnecessary work by re-running only when the set of input files have -changed between each build. -It runs build commands with as much parallelism as possible, either within the -same computer or on [remote build nodes](/remote/rbe). If the structure of build -allows for it, it can run thousands of build or test commands at the same time. +Bazel knows exactly what input files each build command needs, avoiding unnecessary work by re-running only when the set of input files have changed between each build. It runs build commands with as much parallelism as possible, either within the same computer or on [remote build nodes](/remote/rbe). If the structure of build allows for it, it can run thousands of build or test commands at the same time. -This is supported by multiple caching layers, in memory, on disk and on the -remote build farm, if available. At Google, we routinely achieve cache hit rates -north of 99%. +This is supported by multiple caching layers, in memory, on disk and on the remote build farm, if available. At Google, we routinely achieve cache hit rates north of 99%. ## Bazel is correct -Bazel ensures that your binaries are built *only* from your own -source code. Bazel actions run in individual sandboxes and Bazel tracks -every input file of the build, only and always re-running build -commands when it needs to. This keeps your binaries up-to-date so that the -[same source code always results in the same binary](/basics/hermeticity), bit -by bit. +Bazel ensures that your binaries are built *only* from your own source code. Bazel actions run in individual sandboxes and Bazel tracks every input file of the build, only and always re-running build commands when it needs to. This keeps your binaries up-to-date so that the [same source code always results in the same binary](/basics/hermeticity), bit by bit. -Say goodbye to endless `make clean` invocations and to chasing phantom bugs -that were in fact resolved in source code that never got built. +Say goodbye to endless `make clean` invocations and to chasing phantom bugs that were in fact resolved in source code that never got built. ## Bazel is extensible -Harness the full power of Bazel by writing your own rules and macros to -customize Bazel for your specific needs across a wide range of projects. +Harness the full power of Bazel by writing your own rules and macros to customize Bazel for your specific needs across a wide range of projects. -Bazel rules are written in [Starlark](/rules/language), our -in-house programming language that's a subset of Python. Starlark makes -rule-writing accessible to most developers, while also creating rules that can -be used across the ecosystem. +Bazel rules are written in [Starlark](/rules/language), our in-house programming language that's a subset of Python. Starlark makes rule-writing accessible to most developers, while also creating rules that can be used across the ecosystem. ## Integrated testing -Bazel's [integrated test runner](/docs/user-manual#running-tests) -knows and runs only those tests needing to be re-run, using remote execution -(if available) to run them in parallel. Detect flakes early by using remote -execution to quickly run a test thousands of times. +Bazel's [integrated test runner](/docs/user-manual#running-tests) knows and runs only those tests needing to be re-run, using remote execution (if available) to run them in parallel. Detect flakes early by using remote execution to quickly run a test thousands of times. -Bazel [provides facilities](/remote/bep) to upload test results to a central -location, thereby facilitating efficient communication of test outcomes, be it -on CI or by individual developers. +Bazel [provides facilities](/remote/bep) to upload test results to a central location, thereby facilitating efficient communication of test outcomes, be it on CI or by individual developers. ## Multi-language support -Bazel supports many common programming languages including C++, Java, -Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, -backend, web UI and mobile app) in the same Bazel invocation without being -constrained to one language's idiomatic build tool. +Bazel supports many common programming languages including C++, Java, Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, backend, web UI and mobile app) in the same Bazel invocation without being constrained to one language's idiomatic build tool. ## Multi-repository support -Bazel can [gather source code from multiple locations](/external/overview): you -don't need to vendor your dependencies (but you can!), you can instead point -Bazel to the location of your source code or prebuilt artifacts (e.g. a git -repository or Maven Central), and it takes care of the rest. +Bazel can [gather source code from multiple locations](/external/overview): you don't need to vendor your dependencies (but you can!), you can instead point Bazel to the location of your source code or prebuilt artifacts (e.g. a git repository or Maven Central), and it takes care of the rest. ## Multi-platform support -Bazel can simultaneously build projects for multiple platforms including Linux, -macOS, Windows, and Android. It also provides powerful -[cross-compilation capabilities](/extending/platforms) to build code for one -platform while running the build on another. +Bazel can simultaneously build projects for multiple platforms including Linux, macOS, Windows, and Android. It also provides powerful [cross-compilation capabilities](/extending/platforms) to build code for one platform while running the build on another. ## Wide ecosystem -[Industry leaders](/community/users) love Bazel, building a large -community of developers who use and contribute to Bazel. Find a tools, services -and documentation, including [consulting and SaaS offerings](/community/experts) -Bazel can use. Explore extensions like support for programming languages in -our [open source software repositories](/rules). +[Industry leaders](/community/users) love Bazel, building a large community of developers who use and contribute to Bazel. Find a tools, services and documentation, including [consulting and SaaS offerings](/community/experts) Bazel can use. Explore extensions like support for programming languages in our [open source software repositories](/rules). diff --git a/advanced/performance/build-performance-breakdown.mdx b/advanced/performance/build-performance-breakdown.mdx index 477e7578..8bebff8c 100644 --- a/advanced/performance/build-performance-breakdown.mdx +++ b/advanced/performance/build-performance-breakdown.mdx @@ -2,234 +2,120 @@ title: 'Breaking down build performance' --- +Bazel is complex and does a lot of different things over the course of a build, some of which can have an impact on build performance. This page attempts to map some of these Bazel concepts to their implications on build performance. While not extensive, we have included some examples of how to detect build performance issues through [extracting metrics](/configure/build-performance-metrics) and what you can do to fix them. With this, we hope you can apply these concepts when investigating build performance regressions. +### Clean vs Incremental builds -Bazel is complex and does a lot of different things over the course of a build, -some of which can have an impact on build performance. This page attempts to map -some of these Bazel concepts to their implications on build performance. While -not extensive, we have included some examples of how to detect build performance -issues through [extracting metrics](/configure/build-performance-metrics) -and what you can do to fix them. With this, we hope you can apply these concepts -when investigating build performance regressions. +A clean build is one that builds everything from scratch, while an incremental build reuses some already completed work. -### Clean vs Incremental builds +We suggest looking at clean and incremental builds separately, especially when you are collecting / aggregating metrics that are dependent on the state of Bazel’s caches (for example [build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) ). They also represent two different user experiences. As compared to starting a clean build from scratch (which takes longer due to a cold cache), incremental builds happen far more frequently as developers iterate on code (typically faster since the cache is usually already warm). -A clean build is one that builds everything from scratch, while an incremental -build reuses some already completed work. - -We suggest looking at clean and incremental builds separately, especially when -you are collecting / aggregating metrics that are dependent on the state of -Bazel’s caches (for example -[build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) -). They also represent two different user experiences. As compared to starting -a clean build from scratch (which takes longer due to a cold cache), incremental -builds happen far more frequently as developers iterate on code (typically -faster since the cache is usually already warm). - -You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify -builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly -categorize it to likely be an incremental build - the user could have switched -to different flags or different targets causing an effectively clean build. Any -more rigorous definition of incrementality will likely have to come in the form -of a heuristic, for example looking at the number of packages loaded -(`PackageMetrics.packages_loaded`). +You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly categorize it to likely be an incremental build - the user could have switched to different flags or different targets causing an effectively clean build. Any more rigorous definition of incrementality will likely have to come in the form of a heuristic, for example looking at the number of packages loaded (`PackageMetrics.packages_loaded`). ### Deterministic build metrics as a proxy for build performance -Measuring build performance can be difficult due to the non-deterministic nature -of certain metrics (for example Bazel’s CPU time or queue times on a remote -cluster). As such, it can be useful to use deterministic metrics as a proxy for -the amount of work done by Bazel, which in turn affects its performance. - -The size of a build request can have a significant implication on build -performance. A larger build could represent more work in analyzing and -constructing the build graphs. Organic growth of builds comes naturally with -development, as more dependencies are added/created, and thus grow in complexity -and become more expensive to build. - -We can slice this problem into the various build phases, and use the following -metrics as proxy metrics for work done at each phase: - -1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. - A regression here represents more work that needs to be done to read and parse - each additional BUILD file in the loading phase. - - This is often due to the addition of dependencies and having to load their - transitive closure. - - Use [query](/query/quickstart) / [cquery](/query/cquery) to find - where new dependencies might have been added. - -2. `TargetMetrics.targets_configured`: representing the number of targets and - aspects configured in the build. A regression represents more work in - constructing and traversing the configured target graph. - - This is often due to the addition of dependencies and having to construct - the graph of their transitive closure. - - Use [cquery](/query/cquery) to find where new - dependencies might have been added. - -3. `ActionSummary.actions_created`: represents the actions created in the build, - and a regression represents more work in constructing the action graph. Note - that this also includes unused actions that might not have been executed. - - Use [aquery](/query/aquery) for debugging regressions; - we suggest starting with - [`--output=summary`](/reference/command-line-reference#flag--output) - before further drilling down with - [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). - -4. `ActionSummary.actions_executed`: the number of actions executed, a - regression directly represents more work in executing these actions. - - The [BEP](/remote/bep) writes out the action statistics - `ActionData` that shows the most executed action types. By default, it - collects the top 20 action types, but you can pass in the - [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) - to collect this data for all action types that were executed. - - This should help you to figure out what kind of actions were executed - (additionally). - -5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by - the executed actions. - - If the number of actions executed did not increase, then it is likely that - a rule implementation was changed. - - -These metrics are all affected by the state of the local cache, hence you will -want to ensure that the builds you extract these metrics from are -**clean builds**. - -We have noted that a regression in any of these metrics can be accompanied by -regressions in wall time, cpu time and memory usage. +Measuring build performance can be difficult due to the non-deterministic nature of certain metrics (for example Bazel’s CPU time or queue times on a remote cluster). As such, it can be useful to use deterministic metrics as a proxy for the amount of work done by Bazel, which in turn affects its performance. + +The size of a build request can have a significant implication on build performance. A larger build could represent more work in analyzing and constructing the build graphs. Organic growth of builds comes naturally with development, as more dependencies are added/created, and thus grow in complexity and become more expensive to build. + +We can slice this problem into the various build phases, and use the following metrics as proxy metrics for work done at each phase: + +1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. A regression here represents more work that needs to be done to read and parse each additional BUILD file in the loading phase. + + - This is often due to the addition of dependencies and having to load their transitive closure. + - Use [query](/query/quickstart) / [cquery](/query/cquery) to find where new dependencies might have been added. + +2. `TargetMetrics.targets_configured`: representing the number of targets and aspects configured in the build. A regression represents more work in constructing and traversing the configured target graph. + + - This is often due to the addition of dependencies and having to construct the graph of their transitive closure. + - Use [cquery](/query/cquery) to find where new dependencies might have been added. + +3. `ActionSummary.actions_created`: represents the actions created in the build, and a regression represents more work in constructing the action graph. Note that this also includes unused actions that might not have been executed. + + - Use [aquery](/query/aquery) for debugging regressions; we suggest starting with [`--output=summary`](/reference/command-line-reference#flag--output) before further drilling down with [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). + +4. `ActionSummary.actions_executed`: the number of actions executed, a regression directly represents more work in executing these actions. + + - The [BEP](/remote/bep) writes out the action statistics `ActionData` that shows the most executed action types. By default, it collects the top 20 action types, but you can pass in the [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) to collect this data for all action types that were executed. + - This should help you to figure out what kind of actions were executed (additionally). + +5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by the executed actions. + + - If the number of actions executed did not increase, then it is likely that a rule implementation was changed. + +These metrics are all affected by the state of the local cache, hence you will want to ensure that the builds you extract these metrics from are **clean builds**. + +We have noted that a regression in any of these metrics can be accompanied by regressions in wall time, cpu time and memory usage. ### Usage of local resources -Bazel consumes a variety of resources on your local machine (both for analyzing -the build graph and driving the execution, and for running local actions), this -can affect the performance / availability of your machine in performing the -build, and also other tasks. +Bazel consumes a variety of resources on your local machine (both for analyzing the build graph and driving the execution, and for running local actions), this can affect the performance / availability of your machine in performing the build, and also other tasks. #### Time spent -Perhaps the metrics most susceptible to noise (and can vary greatly from build -to build) is time; in particular - wall time, cpu time and system time. You can -use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get -a benchmark for these metrics, and with a sufficient number of `--runs`, you can -increase the statistical significance of your measurement. +Perhaps the metrics most susceptible to noise (and can vary greatly from build to build) is time; in particular - wall time, cpu time and system time. You can use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get a benchmark for these metrics, and with a sufficient number of `--runs`, you can increase the statistical significance of your measurement. - **Wall time** is the real world time elapsed. - - If _only_ wall time regresses, we suggest collecting a - [JSON trace profile](/advanced/performance/json-trace-profile) and looking - for differences. Otherwise, it would likely be more efficient to - investigate other regressed metrics as they could have affected the wall - time. + + - If *only* wall time regresses, we suggest collecting a [JSON trace profile](/advanced/performance/json-trace-profile) and looking for differences. Otherwise, it would likely be more efficient to investigate other regressed metrics as they could have affected the wall time. - **CPU time** is the time spent by the CPU executing user code. - - If the CPU time regresses across two project commits, we suggest collecting - a Starlark CPU profile. You should probably also use `--nobuild` to - restrict the build to the analysis phase since that is where most of the - CPU heavy work is done. + + - If the CPU time regresses across two project commits, we suggest collecting a Starlark CPU profile. You should probably also use `--nobuild` to restrict the build to the analysis phase since that is where most of the CPU heavy work is done. - System time is the time spent by the CPU in the kernel. - - If system time regresses, it is mostly correlated with I/O when Bazel reads - files from your file system. + + - If system time regresses, it is mostly correlated with I/O when Bazel reads files from your file system. #### System-wide load profiling -Using the -[`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) -flag introduced in Bazel 6.0, the -[JSON trace profiler](/advanced/performance/json-trace-profile) collects the -system load average during the invocation. +Using the [`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) flag introduced in Bazel 6.0, the [JSON trace profiler](/advanced/performance/json-trace-profile) collects the system load average during the invocation. ![Profile that includes system load average](/docs/images/json-trace-profile-system-load-average.png "Profile that includes system load average") **Figure 1.** Profile that includes system load average. -A high load during a Bazel invocation can be an indication that Bazel schedules -too many local actions in parallel for your machine. You might want to look into -adjusting -[`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) -and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), -especially in container environments (at least until -[#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). - +A high load during a Bazel invocation can be an indication that Bazel schedules too many local actions in parallel for your machine. You might want to look into adjusting [`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), especially in container environments (at least until [#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). #### Monitoring Bazel memory usage -There are two main sources to get Bazel’s memory usage, Bazel `info` and the -[BEP](/remote/bep). - -- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after - a call to `System.gc()`. - - [Bazel bench](https://github.com/bazelbuild/bazel-bench) - provides benchmarks for this metric as well. - - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` - and `committed-heap-size` (see - [documentation](/docs/user-manual#configuration-independent-data)), but are - less relevant. - -- [BEP](/remote/bep)’s - `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in - bytes post GC (requires setting - [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) - that attempts to force a full GC). - -A regression in memory usage is usually a result of a regression in -[build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), -which are often due to addition of dependencies or a change in the rule -implementation. - -To analyze Bazel’s memory footprint on a more granular level, we recommend using -the [built-in memory profiler](/rules/performance#memory-profiling) -for rules. +There are two main sources to get Bazel’s memory usage, Bazel `info` and the [BEP](/remote/bep). + +- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after a call to `System.gc()`. + + - [Bazel bench](https://github.com/bazelbuild/bazel-bench) provides benchmarks for this metric as well. + - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` and `committed-heap-size` (see [documentation](/docs/user-manual#configuration-independent-data)), but are less relevant. + +- [BEP](/remote/bep)’s `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in bytes post GC (requires setting [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) that attempts to force a full GC). + +A regression in memory usage is usually a result of a regression in [build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), which are often due to addition of dependencies or a change in the rule implementation. + +To analyze Bazel’s memory footprint on a more granular level, we recommend using the [built-in memory profiler](/rules/performance#memory-profiling) for rules. #### Memory profiling of persistent workers -While [persistent workers](/remote/persistent) can help to speed up builds -significantly (especially for interpreted languages) their memory footprint can -be problematic. Bazel collects metrics on its workers, in particular, the -`WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory -workers use (by mnemonic). +While [persistent workers](/remote/persistent) can help to speed up builds significantly (especially for interpreted languages) their memory footprint can be problematic. Bazel collects metrics on its workers, in particular, the `WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory workers use (by mnemonic). -The [JSON trace profiler](/advanced/performance/json-trace-profile) also -collects persistent worker memory usage during the invocation by passing in the -[`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) -flag (new in Bazel 6.0). +The [JSON trace profiler](/advanced/performance/json-trace-profile) also collects persistent worker memory usage during the invocation by passing in the [`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) flag (new in Bazel 6.0). ![Profile that includes workers memory usage](/docs/images/json-trace-profile-workers-memory-usage.png "Profile that includes workers memory usage") **Figure 2.** Profile that includes workers memory usage. -Lowering the value of -[`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) -(default 4) might help to reduce -the amount of memory used by persistent workers. We are actively working on -making Bazel’s resource manager and scheduler smarter so that such fine tuning -will be required less often in the future. +Lowering the value of [`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) (default 4) might help to reduce the amount of memory used by persistent workers. We are actively working on making Bazel’s resource manager and scheduler smarter so that such fine tuning will be required less often in the future. ### Monitoring network traffic for remote builds -In remote execution, Bazel downloads artifacts that were built as a result of -executing actions. As such, your network bandwidth can affect the performance -of your build. +In remote execution, Bazel downloads artifacts that were built as a result of executing actions. As such, your network bandwidth can affect the performance of your build. -If you are using remote execution for your builds, you might want to consider -monitoring the network traffic during the invocation using the -`NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) -(requires passing `--experimental_collect_system_network_usage`). +If you are using remote execution for your builds, you might want to consider monitoring the network traffic during the invocation using the `NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) (requires passing `--experimental_collect_system_network_usage`). -Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) -allow you to view system-wide network usage throughout the course of the build -by passing the `--experimental_collect_system_network_usage` flag (new in Bazel -6.0). +Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) allow you to view system-wide network usage throughout the course of the build by passing the `--experimental_collect_system_network_usage` flag (new in Bazel 6.0). ![Profile that includes system-wide network usage](/docs/images/json-trace-profile-network-usage.png "Profile that includes system-wide network usage") **Figure 3.** Profile that includes system-wide network usage. -A high but rather flat network usage when using remote execution might indicate -that network is the bottleneck in your build; if you are not using it already, -consider turning on Build without the Bytes by passing -[`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). -This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. +A high but rather flat network usage when using remote execution might indicate that network is the bottleneck in your build; if you are not using it already, consider turning on Build without the Bytes by passing [`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. -Another option is to configure a local -[disk cache](/reference/command-line-reference#flag--disk_cache) to save on -download bandwidth. +Another option is to configure a local [disk cache](/reference/command-line-reference#flag--disk_cache) to save on download bandwidth. diff --git a/advanced/performance/build-performance-metrics.mdx b/advanced/performance/build-performance-metrics.mdx index 8391ea87..8f92e75a 100644 --- a/advanced/performance/build-performance-metrics.mdx +++ b/advanced/performance/build-performance-metrics.mdx @@ -2,96 +2,50 @@ title: 'Extracting build performance metrics' --- - - -Probably every Bazel user has experienced builds that were slow or slower than -anticipated. Improving the performance of individual builds has particular value -for targets with significant impact, such as: +Probably every Bazel user has experienced builds that were slow or slower than anticipated. Improving the performance of individual builds has particular value for targets with significant impact, such as: 1. Core developer targets that are frequently iterated on and (re)built. 2. Common libraries widely depended upon by other targets. -3. A representative target from a class of targets (e.g. custom rules), - diagnosing and fixing issues in one build might help to resolve issues at the - larger scale. +3. A representative target from a class of targets (e.g. custom rules), diagnosing and fixing issues in one build might help to resolve issues at the larger scale. -An important step to improving the performance of builds is to understand where -resources are spent. This page lists different metrics you can collect. -[Breaking down build performance](/configure/build-performance-breakdown) showcases -how you can use these metrics to detect and fix build performance issues. +An important step to improving the performance of builds is to understand where resources are spent. This page lists different metrics you can collect. [Breaking down build performance](/configure/build-performance-breakdown) showcases how you can use these metrics to detect and fix build performance issues. There are a few main ways to extract metrics from your Bazel builds, namely: ## Build Event Protocol (BEP) -Bazel outputs a variety of protocol buffers -[`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) -through the [Build Event Protocol (BEP)](/remote/bep), which -can be aggregated by a backend specified by you. Depending on your use cases, -you might decide to aggregate the metrics in various ways, but here we will go -over some concepts and proto fields that would be useful in general to consider. +Bazel outputs a variety of protocol buffers [`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) through the [Build Event Protocol (BEP)](/remote/bep), which can be aggregated by a backend specified by you. Depending on your use cases, you might decide to aggregate the metrics in various ways, but here we will go over some concepts and proto fields that would be useful in general to consider. ## Bazel’s query / cquery / aquery commands -Bazel provides 3 different query modes ([query](/query/quickstart), -[cquery](/query/cquery) and [aquery](/query/aquery)) that allow users -to query the target graph, configured target graph and action graph -respectively. The query language provides a -[suite of functions](/query/language#functions) usable across the different -query modes, that allows you to customize your queries according to your needs. +Bazel provides 3 different query modes ([query](/query/quickstart), [cquery](/query/cquery) and [aquery](/query/aquery)) that allow users to query the target graph, configured target graph and action graph respectively. The query language provides a [suite of functions](/query/language#functions) usable across the different query modes, that allows you to customize your queries according to your needs. ## JSON Trace Profiles -For every build-like Bazel invocation, Bazel writes a trace profile in JSON -format. The [JSON trace profile](/advanced/performance/json-trace-profile) can -be very useful to quickly understand what Bazel spent time on during the -invocation. +For every build-like Bazel invocation, Bazel writes a trace profile in JSON format. The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. ## Execution Log -The [execution log](/remote/cache-remote) can help you to troubleshoot and fix -missing remote cache hits due to machine and environment differences or -non-deterministic actions. If you pass the flag -[`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) -(available from Bazel 5.2) it will also contain detailed spawn metrics, both for -locally and remotely executed actions. You can use these metrics for example to -make comparisons between local and remote machine performance or to find out -which part of the spawn execution is consistently slower than expected (for -example due to queuing). +The [execution log](/remote/cache-remote) can help you to troubleshoot and fix missing remote cache hits due to machine and environment differences or non-deterministic actions. If you pass the flag [`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) (available from Bazel 5.2) it will also contain detailed spawn metrics, both for locally and remotely executed actions. You can use these metrics for example to make comparisons between local and remote machine performance or to find out which part of the spawn execution is consistently slower than expected (for example due to queuing). ## Execution Graph Log -While the JSON trace profile contains the critical path information, sometimes -you need additional information on the dependency graph of the executed actions. -Starting with Bazel 6.0, you can pass the flags -`--experimental_execution_graph_log` and -`--experimental_execution_graph_log_dep_type=all` to write out a log about the -executed actions and their inter-dependencies. +While the JSON trace profile contains the critical path information, sometimes you need additional information on the dependency graph of the executed actions. Starting with Bazel 6.0, you can pass the flags `--experimental_execution_graph_log` and `--experimental_execution_graph_log_dep_type=all` to write out a log about the executed actions and their inter-dependencies. -This information can be used to understand the drag that is added by a node on -the critical path. The drag is the amount of time that can potentially be saved -by removing a particular node from the execution graph. +This information can be used to understand the drag that is added by a node on the critical path. The drag is the amount of time that can potentially be saved by removing a particular node from the execution graph. -The data helps you predict the impact of changes to the build and action graph -before you actually do them. +The data helps you predict the impact of changes to the build and action graph before you actually do them. ## Benchmarking with bazel-bench -[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a -benchmarking tool for Git projects to benchmark build performance in the -following cases: +[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a benchmarking tool for Git projects to benchmark build performance in the following cases: -* **Project benchmark:** Benchmarking two git commits against each other at a - single Bazel version. Used to detect regressions in your build (often through - the addition of dependencies). +- **Project benchmark:** Benchmarking two git commits against each other at a single Bazel version. Used to detect regressions in your build (often through the addition of dependencies). -* **Bazel benchmark:** Benchmarking two versions of Bazel against each other at - a single git commit. Used to detect regressions within Bazel itself (if you - happen to maintain / fork Bazel). +- **Bazel benchmark:** Benchmarking two versions of Bazel against each other at a single git commit. Used to detect regressions within Bazel itself (if you happen to maintain / fork Bazel). -Benchmarks monitor wall time, CPU time and system time and Bazel’s retained -heap size. +Benchmarks monitor wall time, CPU time and system time and Bazel’s retained heap size. -It is also recommended to run Bazel bench on dedicated, physical machines that -are not running other processes so as to reduce sources of variability. +It is also recommended to run Bazel bench on dedicated, physical machines that are not running other processes so as to reduce sources of variability. diff --git a/advanced/performance/iteration-speed.mdx b/advanced/performance/iteration-speed.mdx index 2bbf8398..79240cd0 100644 --- a/advanced/performance/iteration-speed.mdx +++ b/advanced/performance/iteration-speed.mdx @@ -2,92 +2,44 @@ title: 'Optimize Iteration Speed' --- - - -This page describes how to optimize Bazel's build performance when running Bazel -repeatedly. +This page describes how to optimize Bazel's build performance when running Bazel repeatedly. ## Bazel's Runtime State A Bazel invocation involves several interacting parts. -* The `bazel` command line interface (CLI) is the user-facing front-end tool - and receives commands from the user. +- The `bazel` command line interface (CLI) is the user-facing front-end tool and receives commands from the user. -* The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) - for each distinct [output base](https://bazel.build/remote/output-directories). - The Bazel server is generally persistent, but will shut down after some idle - time so as to not waste resources. +- The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) for each distinct [output base](https://bazel.build/remote/output-directories). The Bazel server is generally persistent, but will shut down after some idle time so as to not waste resources. -* The Bazel server performs the loading and analysis steps for a given command - (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts - of the build graph in memory. The resulting data structures are retained in - the Bazel server as part of the *analysis cache*. +- The Bazel server performs the loading and analysis steps for a given command (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts of the build graph in memory. The resulting data structures are retained in the Bazel server as part of the *analysis cache*. -* The Bazel server can also perform the action execution, or it can send - actions off for remote execution if it is set up to do so. The results of - action executions are also cached, namely in the *action cache* (or - *execution cache*, which may be either local or remote, and it may be shared - among Bazel servers). +- The Bazel server can also perform the action execution, or it can send actions off for remote execution if it is set up to do so. The results of action executions are also cached, namely in the *action cache* (or *execution cache*, which may be either local or remote, and it may be shared among Bazel servers). -* The result of the Bazel invocation is made available in the output tree. +- The result of the Bazel invocation is made available in the output tree. ## Running Bazel Iteratively -In a typical developer workflow, it is common to build (or run) a piece of code -repeatedly, often at a very high frequency (e.g. to resolve some compilation -error or investigate a failing test). In this situation, it is important that -repeated invocations of `bazel` have as little overhead as possible relative to -the underlying, repeated action (e.g. invoking a compiler, or executing a test). +In a typical developer workflow, it is common to build (or run) a piece of code repeatedly, often at a very high frequency (e.g. to resolve some compilation error or investigate a failing test). In this situation, it is important that repeated invocations of `bazel` have as little overhead as possible relative to the underlying, repeated action (e.g. invoking a compiler, or executing a test). With this in mind, we take another look at Bazel's runtime state: -The analysis cache is a critical piece of data. A significant amount of time can -be spent just on the loading and analysis phases of a cold run (i.e. a run just -after the Bazel server was started or when the analysis cache was discarded). -For a single, successful cold build (e.g. for a production release) this cost is -bearable, but for repeatedly building the same target it is important that this -cost be amortized and not repeated on each invocation. - -The analysis cache is rather volatile. First off, it is part of the in-process -state of the Bazel server, so losing the server loses the cache. But the cache -is also *invalidated* very easily: for example, many `bazel` command line flags -cause the cache to be discarded. This is because many flags affect the build -graph (e.g. because of -[configurable attributes](https://bazel.build/configure/attributes)). Some flag -changes can also cause the Bazel server to be restarted (e.g. changing -[startup options](https://bazel.build/docs/user-manual#startup-options)). - -A good execution cache is also valuable for build performance. An execution -cache can be kept locally -[on disk](https://bazel.build/remote/caching#disk-cache), or -[remotely](https://bazel.build/remote/caching). The cache can be shared among -Bazel servers, and indeed among developers. +The analysis cache is a critical piece of data. A significant amount of time can be spent just on the loading and analysis phases of a cold run (i.e. a run just after the Bazel server was started or when the analysis cache was discarded). For a single, successful cold build (e.g. for a production release) this cost is bearable, but for repeatedly building the same target it is important that this cost be amortized and not repeated on each invocation. + +The analysis cache is rather volatile. First off, it is part of the in-process state of the Bazel server, so losing the server loses the cache. But the cache is also *invalidated* very easily: for example, many `bazel` command line flags cause the cache to be discarded. This is because many flags affect the build graph (e.g. because of [configurable attributes](https://bazel.build/configure/attributes)). Some flag changes can also cause the Bazel server to be restarted (e.g. changing [startup options](https://bazel.build/docs/user-manual#startup-options)). + +A good execution cache is also valuable for build performance. An execution cache can be kept locally [on disk](https://bazel.build/remote/caching#disk-cache), or [remotely](https://bazel.build/remote/caching). The cache can be shared among Bazel servers, and indeed among developers. ## Avoid discarding the analysis cache -Bazel will print a warning if either the analysis cache was discarded or the -server was restarted. Either of these should be avoided during iterative use: +Bazel will print a warning if either the analysis cache was discarded or the server was restarted. Either of these should be avoided during iterative use: -* Be mindful of changing `bazel` flags in the middle of an iterative - workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` - causes each command to discard the analysis cache of the other. In general, - try to use a fixed set of flags for the duration of a particular workflow. +- Be mindful of changing `bazel` flags in the middle of an iterative workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` causes each command to discard the analysis cache of the other. In general, try to use a fixed set of flags for the duration of a particular workflow. -* Losing the Bazel server loses the analysis cache. The Bazel server has a - [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle - time, after which it shuts down. You can configure this time via your - bazelrc file to suit your needs. The server also restarted when startup - flags change, so, again, avoid changing those flags if possible. +- Losing the Bazel server loses the analysis cache. The Bazel server has a [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle time, after which it shuts down. You can configure this time via your bazelrc file to suit your needs. The server also restarted when startup flags change, so, again, avoid changing those flags if possible. -* Beware that the Bazel server is killed if you press - Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time - by interrupting a running build that is no longer needed, but only press - Ctrl-C once to request a graceful end of the current invocation. +- [Beware]() that the Bazel server is killed if you press Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time by interrupting a running build that is no longer needed, but only press Ctrl-C once to request a graceful end of the current invocation. -* If you want to use multiple sets of flags from the same workspace, you can - use multiple, distinct output bases, switched with the `--output_base` - flag. Each output base gets its own Bazel server. +- If you want to use multiple sets of flags from the same workspace, you can use multiple, distinct output bases, switched with the `--output_base` flag. Each output base gets its own Bazel server. -To make this condition an error rather than a warning, you can use the -`--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) +To make this condition an error rather than a warning, you can use the `--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) diff --git a/advanced/performance/json-trace-profile.mdx b/advanced/performance/json-trace-profile.mdx index 80c698c0..c452c4b0 100644 --- a/advanced/performance/json-trace-profile.mdx +++ b/advanced/performance/json-trace-profile.mdx @@ -2,34 +2,17 @@ title: 'JSON Trace Profile' --- +The JSON trace profile can be very useful to quickly understand what Bazel spent time on during the invocation. - -The JSON trace profile can be very useful to quickly understand what Bazel spent -time on during the invocation. - -By default, for all build-like commands and query, Bazel writes a profile into -the output base named `command-$INVOCATION_ID.profile.gz`, where -`$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates -a symlink called `command.profile.gz` in the output base that points the profile -of the latest command. You can configure whether a profile is written with the -[`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) -flag, and the location it is written to with the -[`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are -compressed with GZIP. Bazel keeps the last 5 profiles, configurable by -[`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), -in the output base by default for post-build analysis. Explicitly passing a -profile path with `--profile` disables automatic garbage collection. +By default, for all build-like commands and query, Bazel writes a profile into the output base named `command-$INVOCATION_ID.profile.gz`, where `$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates a symlink called `command.profile.gz` in the output base that points the profile of the latest command. You can configure whether a profile is written with the [`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) flag, and the location it is written to with the [`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are compressed with GZIP. Bazel keeps the last 5 profiles, configurable by [`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), in the output base by default for post-build analysis. Explicitly passing a profile path with `--profile` disables automatic garbage collection. ## Tools -You can load this profile into `chrome://tracing` or analyze and -post-process it with other tools. +You can load this profile into `chrome://tracing` or analyze and post-process it with other tools. ### `chrome://tracing` -To visualize the profile, open `chrome://tracing` in a Chrome browser tab, -click "Load" and pick the (potentially compressed) profile file. For more -detailed results, click the boxes in the lower left corner. +To visualize the profile, open `chrome://tracing` in a Chrome browser tab, click "Load" and pick the (potentially compressed) profile file. For more detailed results, click the boxes in the lower left corner. Example profile: @@ -39,29 +22,19 @@ Example profile: You can use these keyboard controls to navigate: -* Press `1` for "select" mode. In this mode, you can select - particular boxes to inspect the event details (see lower left corner). - Select multiple events to get a summary and aggregated statistics. -* Press `2` for "pan" mode. Then drag the mouse to move the view. You - can also use `a`/`d` to move left/right. -* Press `3` for "zoom" mode. Then drag the mouse to zoom. You can - also use `w`/`s` to zoom in/out. -* Press `4` for "timing" mode where you can measure the distance - between two events. -* Press `?` to learn about all controls. +- Press `1` for "select" mode. In this mode, you can select particular boxes to inspect the event details (see lower left corner). Select multiple events to get a summary and aggregated statistics. +- Press `2` for "pan" mode. Then drag the mouse to move the view. You can also use `a`/`d` to move left/right. +- Press `3` for "zoom" mode. Then drag the mouse to zoom. You can also use `w`/`s` to zoom in/out. +- Press `4` for "timing" mode where you can measure the distance between two events. +- Press `?` to learn about all controls. ### Bazel Invocation Analyzer -The open-source -[Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) -consumes a profile format and prints suggestions on how to improve -the build’s performance. This analysis can be performed using its CLI or on -[https://analyzer.engflow.com](https://analyzer.engflow.com). +The open-source [Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) consumes a profile format and prints suggestions on how to improve the build’s performance. This analysis can be performed using its CLI or on [https://analyzer.engflow.com](https://analyzer.engflow.com). ### `jq` -`jq` is like `sed` for JSON data. An example usage of `jq` to extract all -durations of the sandbox creation step in local action execution: +`jq` is like `sed` for JSON data. An example usage of `jq` to extract all durations of the sandbox creation step in local action execution: ``` $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | jq '.traceEvents | .[] | select(.name == "sandbox.createFileSystem") | .dur' @@ -78,50 +51,29 @@ $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | j ## Profile information -The profile contains multiple rows. Usually the bulk of rows represent Bazel -threads and their corresponding events, but some special rows are also included. +The profile contains multiple rows. Usually the bulk of rows represent Bazel threads and their corresponding events, but some special rows are also included. -The special rows included depend on the version of Bazel invoked when the -profile was created, and may be customized by different flags. +The special rows included depend on the version of Bazel invoked when the profile was created, and may be customized by different flags. Figure 1 shows a profile created with Bazel v5.3.1 and includes these rows: -* `action count`: Displays how many concurrent actions were in flight. Click - on it to see the actual value. Should go up to the value of - [`--jobs`](/reference/command-line-reference#flag--jobs) in clean - builds. -* `CPU usage (Bazel)`: For each second of the build, displays the amount of - CPU that was used by Bazel (a value of 1 equals one core being 100% busy). -* `Critical Path`: Displays one block for each action on the critical path. -* `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of - what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", - and "runAnalysisPhase". -* `Garbage Collector`: Displays minor and major Garbage Collection (GC) - pauses. +- `action count`: Displays how many concurrent actions were in flight. Click on it to see the actual value. Should go up to the value of [`--jobs`](/reference/command-line-reference#flag--jobs) in clean builds. +- `CPU usage (Bazel)`: For each second of the build, displays the amount of CPU that was used by Bazel (a value of 1 equals one core being 100% busy). +- `Critical Path`: Displays one block for each action on the critical path. +- `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", and "runAnalysisPhase". +- `Garbage Collector`: Displays minor and major Garbage Collection (GC) pauses. ## Common performance issues When analyzing performance profiles, look for: -* Slower than expected analysis phase (`runAnalysisPhase`), especially on - incremental builds. This can be a sign of a poor rule implementation, for - example one that flattens depsets. Package loading can be slow by an - excessive amount of targets, complex macros or recursive globs. -* Individual slow actions, especially those on the critical path. It might be - possible to split large actions into multiple smaller actions or reduce the - set of (transitive) dependencies to speed them up. Also check for an unusual - high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). -* Bottlenecks, that is a small number of threads is busy while all others are - idling / waiting for the result (see around 22s and 29s in Figure 1). - Optimizing this will most likely require touching the rule implementations - or Bazel itself to introduce more parallelism. This can also happen when - there is an unusual amount of GC. +- Slower than expected analysis phase (`runAnalysisPhase`), especially on incremental builds. This can be a sign of a poor rule implementation, for example one that flattens depsets. Package loading can be slow by an excessive amount of targets, complex macros or recursive globs. +- Individual slow actions, especially those on the critical path. It might be possible to split large actions into multiple smaller actions or reduce the set of (transitive) dependencies to speed them up. Also check for an unusual high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). +- Bottlenecks, that is a small number of threads is busy while all others are idling / waiting for the result (see around 22s and 29s in Figure 1). Optimizing this will most likely require touching the rule implementations or Bazel itself to introduce more parallelism. This can also happen when there is an unusual amount of GC. ## Profile file format -The top-level object contains metadata (`otherData`) and the actual tracing data -(`traceEvents`). The metadata contains extra info, for example the invocation ID -and date of the Bazel invocation. +The top-level object contains metadata (`otherData`) and the actual tracing data (`traceEvents`). The metadata contains extra info, for example the invocation ID and date of the Bazel invocation. Example: @@ -148,12 +100,6 @@ Example: } ``` -Timestamps (`ts`) and durations (`dur`) in the trace events are given in -microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. -Note that some events are merged together if they are very short and close to -each other; pass -[`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) -if you would like to prevent event merging. +Timestamps (`ts`) and durations (`dur`) in the trace events are given in microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. Note that some events are merged together if they are very short and close to each other; pass [`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) if you would like to prevent event merging. -See also the -[Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). +See also the [Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). diff --git a/advanced/performance/memory.mdx b/advanced/performance/memory.mdx index 844e691b..8f6c8b5f 100644 --- a/advanced/performance/memory.mdx +++ b/advanced/performance/memory.mdx @@ -2,50 +2,27 @@ title: 'Optimize Memory' --- - - This page describes how to limit and reduce the memory Bazel uses. ## Running Bazel with Limited RAM -In certain situations, you may want Bazel to use minimal memory. You can set the -maximum heap via the startup flag -[`--host_jvm_args`](/docs/user-manual#host-jvm-args), -like `--host_jvm_args=-Xmx2g`. +In certain situations, you may want Bazel to use minimal memory. You can set the maximum heap via the startup flag [`--host_jvm_args`](/docs/user-manual#host-jvm-args), like `--host_jvm_args=-Xmx2g`. ### Trade incremental build speeds for memory -If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when -it doesn't have enough memory. You can make Bazel use less memory, at the cost -of slower incremental builds, by passing the following command flags: -[`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), -[`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), -and -[`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). +If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when it doesn't have enough memory. You can make Bazel use less memory, at the cost of slower incremental builds, by passing the following command flags: [`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), [`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), and [`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). -These flags will minimize the memory that Bazel uses in a build, at the cost of -making future builds slower than a standard incremental build would be. +These flags will minimize the memory that Bazel uses in a build, at the cost of making future builds slower than a standard incremental build would be. You can also pass any one of these flags individually: - * `--discard_analysis_cache` will reduce the memory used during execution (not -analysis). Incremental builds will not have to redo package loading, but will -have to redo analysis and execution (although the on-disk action cache can -prevent most re-execution). - * `--notrack_incremental_state` will not store any edges in Bazel's internal - dependency graph, so that it is unusable for incremental builds. The next build - will discard that data, but it is preserved until then, for internal debugging, - unless `--nokeep_state_after_build` is specified. - * `--nokeep_state_after_build` will discard all data after the build, so that - incremental builds have to build from scratch (except for the on-disk action - cache). Alone, it does not affect the high-water mark of the current build. +- `--discard_analysis_cache` will reduce the memory used during execution (not analysis). Incremental builds will not have to redo package loading, but will have to redo analysis and execution (although the on-disk action cache can prevent most re-execution). +- `--notrack_incremental_state` will not store any edges in Bazel's internal dependency graph, so that it is unusable for incremental builds. The next build will discard that data, but it is preserved until then, for internal debugging, unless `--nokeep_state_after_build` is specified. +- `--nokeep_state_after_build` will discard all data after the build, so that incremental builds have to build from scratch (except for the on-disk action cache). Alone, it does not affect the high-water mark of the current build. ### Trade build flexibility for memory with Skyfocus (Experimental) -If you want to make Bazel use less memory *and* retain incremental build speeds, -you can tell Bazel the working set of files that you will be modifying, and -Bazel will only keep state needed to correctly incrementally rebuild changes to -those files. This feature is called **Skyfocus**. +If you want to make Bazel use less memory *and* retain incremental build speeds, you can tell Bazel the working set of files that you will be modifying, and Bazel will only keep state needed to correctly incrementally rebuild changes to those files. This feature is called **Skyfocus**. To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: @@ -53,21 +30,16 @@ To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: bazel build //pkg:target --experimental_enable_skyfocus ``` -By default, the working set will be the set of files next to the target being -built. In the example, all files in `//pkg` will be kept in the working set, and -changes to files outside of the working set will be disallowed, until you issue -`bazel clean` or restart the Bazel server. +By default, the working set will be the set of files next to the target being built. In the example, all files in `//pkg` will be kept in the working set, and changes to files outside of the working set will be disallowed, until you issue `bazel clean` or restart the Bazel server. -If you want to specify an exact set of files or directories, use the -`--experimental_working_set` flag, like so: +If you want to specify an exact set of files or directories, use the `--experimental_working_set` flag, like so: ```sh bazel build //pkg:target --experimental_enable_skyfocus --experimental_working_set=path/to/another/dir,path/to/tests/dir ``` -You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the -memory reduction amount: +You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the memory reduction amount: Putting it altogether, you should see something like this: @@ -79,19 +51,13 @@ INFO: Analyzed 149 targets (4533 packages loaded, 169438 targets configured). INFO: Found 25 targets and 124 test targets... INFO: Updated working set successfully. INFO: Focusing on 334 roots, 3 leafs... (use --experimental_skyfocus_dump_keys to show them) -INFO: Heap: 1237MB -> 676MB (-45.31%) +INFO: Heap: 1237MB -> 676MB (-45.31%) INFO: Elapsed time: 192.670s ... INFO: Build completed successfully, 62303 total actions ``` -For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, -and incremental builds to handle changes to files under `dir1`, `dir2`, and -`dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot -rebuild changed files outside of these directories. +For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, and incremental builds to handle changes to files under `dir1`, `dir2`, and `dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot rebuild changed files outside of these directories. ## Memory Profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s -memory use. Read more about this process on the -[Memory Profiling section](/rules/performance#memory-profiling) of our -documentation on how to improve the performance of custom rules. +Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. Read more about this process on the [Memory Profiling section](/rules/performance#memory-profiling) of our documentation on how to improve the performance of custom rules. diff --git a/basics/artifact-based-builds.mdx b/basics/artifact-based-builds.mdx index 4176a288..5dfc5896 100644 --- a/basics/artifact-based-builds.mdx +++ b/basics/artifact-based-builds.mdx @@ -2,57 +2,19 @@ title: 'Artifact-Based Build Systems' --- +This page covers artifact-based build systems and the philosophy behind their creation. Bazel is an artifact-based build system. While task-based build systems are good step above build scripts, they give too much power to individual engineers by letting them define their own tasks. - -This page covers artifact-based build systems and the philosophy behind their -creation. Bazel is an artifact-based build system. While task-based build -systems are good step above build scripts, they give too much power to -individual engineers by letting them define their own tasks. - -Artifact-based build systems have a small number of tasks defined by the system -that engineers can configure in a limited way. Engineers still tell the system -**what** to build, but the build system determines **how** to build it. As with -task-based build systems, artifact-based build systems, such as Bazel, still -have buildfiles, but the contents of those buildfiles are very different. Rather -than being an imperative set of commands in a Turing-complete scripting language -describing how to produce an output, buildfiles in Bazel are a declarative -manifest describing a set of artifacts to build, their dependencies, and a -limited set of options that affect how they’re built. When engineers run `bazel` -on the command line, they specify a set of targets to build (the **what**), and -Bazel is responsible for configuring, running, and scheduling the compilation -steps (the **how**). Because the build system now has full control over what -tools to run when, it can make much stronger guarantees that allow it to be far -more efficient while still guaranteeing correctness. +Artifact-based build systems have a small number of tasks defined by the system that engineers can configure in a limited way. Engineers still tell the system **what** to build, but the build system determines **how** to build it. As with task-based build systems, artifact-based build systems, such as Bazel, still have buildfiles, but the contents of those buildfiles are very different. Rather than being an imperative set of commands in a Turing-complete scripting language describing how to produce an output, buildfiles in Bazel are a declarative manifest describing a set of artifacts to build, their dependencies, and a limited set of options that affect how they’re built. When engineers run `bazel` on the command line, they specify a set of targets to build (the **what**), and Bazel is responsible for configuring, running, and scheduling the compilation steps (the **how**). Because the build system now has full control over what tools to run when, it can make much stronger guarantees that allow it to be far more efficient while still guaranteeing correctness. ## A functional perspective -It’s easy to make an analogy between artifact-based build systems and functional -programming. Traditional imperative programming languages (such as, Java, C, and -Python) specify lists of statements to be executed one after another, in the -same way that task-based build systems let programmers define a series of steps -to execute. Functional programming languages (such as, Haskell and ML), in -contrast, are structured more like a series of mathematical equations. In -functional languages, the programmer describes a computation to perform, but -leaves the details of when and exactly how that computation is executed to the -compiler. - -This maps to the idea of declaring a manifest in an artifact-based build system -and letting the system figure out how to execute the build. Many problems can't -be easily expressed using functional programming, but the ones that do benefit -greatly from it: the language is often able to trivially parallelize such -programs and make strong guarantees about their correctness that would be -impossible in an imperative language. The easiest problems to express using -functional programming are the ones that simply involve transforming one piece -of data into another using a series of rules or functions. And that’s exactly -what a build system is: the whole system is effectively a mathematical function -that takes source files (and tools like the compiler) as inputs and produces -binaries as outputs. So, it’s not surprising that it works well to base a build -system around the tenets of functional programming. +It’s easy to make an analogy between artifact-based build systems and functional programming. Traditional imperative programming languages (such as, Java, C, and Python) specify lists of statements to be executed one after another, in the same way that task-based build systems let programmers define a series of steps to execute. Functional programming languages (such as, Haskell and ML), in contrast, are structured more like a series of mathematical equations. In functional languages, the programmer describes a computation to perform, but leaves the details of when and exactly how that computation is executed to the compiler. + +This maps to the idea of declaring a manifest in an artifact-based build system and letting the system figure out how to execute the build. Many problems can't be easily expressed using functional programming, but the ones that do benefit greatly from it: the language is often able to trivially parallelize such programs and make strong guarantees about their correctness that would be impossible in an imperative language. The easiest problems to express using functional programming are the ones that simply involve transforming one piece of data into another using a series of rules or functions. And that’s exactly what a build system is: the whole system is effectively a mathematical function that takes source files (and tools like the compiler) as inputs and produces binaries as outputs. So, it’s not surprising that it works well to base a build system around the tenets of functional programming. ## Understanding artifact-based build systems -Google's build system, Blaze, was the first artifact-based build system. Bazel -is the open-sourced version of Blaze. +Google's build system, Blaze, was the first artifact-based build system. Bazel is the open-sourced version of Blaze. Here’s what a buildfile (normally named `BUILD`) looks like in Bazel: @@ -75,205 +37,64 @@ java_library( ) ``` -In Bazel, `BUILD` files define targets—the two types of targets here are -`java_binary` and `java_library`. Every target corresponds to an artifact that -can be created by the system: binary targets produce binaries that can be -executed directly, and library targets produce libraries that can be used by -binaries or other libraries. Every target has: - -* `name`: how the target is referenced on the command line and by other - targets -* `srcs`: the source files to be compiled to create the artifact for the target -* `deps`: other targets that must be built before this target and linked into - it - -Dependencies can either be within the same package (such as `MyBinary`’s -dependency on `:mylib`) or on a different package in the same source hierarchy -(such as `mylib`’s dependency on `//java/com/example/common`). - -As with task-based build systems, you perform builds using Bazel’s command-line -tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After -entering that command for the first time in a clean repository, Bazel: - -1. Parses every `BUILD` file in the workspace to create a graph of dependencies - among artifacts. -1. Uses the graph to determine the transitive dependencies of `MyBinary`; that - is, every target that `MyBinary` depends on and every target that those - targets depend on, recursively. -1. Builds each of those dependencies, in order. Bazel starts by building each - target that has no other dependencies and keeps track of which dependencies - still need to be built for each target. As soon as all of a target’s - dependencies are built, Bazel starts building that target. This process - continues until every one of `MyBinary`’s transitive dependencies have been - built. -1. Builds `MyBinary` to produce a final executable binary that links in all of - the dependencies that were built in step 3. - -Fundamentally, it might not seem like what’s happening here is that much -different than what happened when using a task-based build system. Indeed, the -end result is the same binary, and the process for producing it involved -analyzing a bunch of steps to find dependencies among them, and then running -those steps in order. But there are critical differences. The first one appears -in step 3: because Bazel knows that each target only produces a Java library, it -knows that all it has to do is run the Java compiler rather than an arbitrary -user-defined script, so it knows that it’s safe to run these steps in parallel. -This can produce an order of magnitude performance improvement over building -targets one at a time on a multicore machine, and is only possible because the -artifact-based approach leaves the build system in charge of its own execution -strategy so that it can make stronger guarantees about parallelism. - -The benefits extend beyond parallelism, though. The next thing that this -approach gives us becomes apparent when the developer types `bazel -build :MyBinary` a second time without making any changes: Bazel exits in less -than a second with a message saying that the target is up to date. This is -possible due to the functional programming paradigm we talked about -earlier—Bazel knows that each target is the result only of running a Java -compiler, and it knows that the output from the Java compiler depends only on -its inputs, so as long as the inputs haven’t changed, the output can be reused. -And this analysis works at every level; if `MyBinary.java` changes, Bazel knows -to rebuild `MyBinary` but reuse `mylib`. If a source file for -`//java/com/example/common` changes, Bazel knows to rebuild that library, -`mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. -Because Bazel knows about the properties of the tools it runs at every step, -it’s able to rebuild only the minimum set of artifacts each time while -guaranteeing that it won’t produce stale builds. - -Reframing the build process in terms of artifacts rather than tasks is subtle -but powerful. By reducing the flexibility exposed to the programmer, the build -system can know more about what is being done at every step of the build. It can -use this knowledge to make the build far more efficient by parallelizing build -processes and reusing their outputs. But this is really just the first step, and -these building blocks of parallelism and reuse form the basis for a distributed -and highly scalable build system. +In Bazel, `BUILD` files define targets—the two types of targets here are `java_binary` and `java_library`. Every target corresponds to an artifact that can be created by the system: binary targets produce binaries that can be executed directly, and library targets produce libraries that can be used by binaries or other libraries. Every target has: + +- `name`: how the target is referenced on the command line and by other targets +- `srcs`: the source files to be compiled to create the artifact for the target +- `deps`: other targets that must be built before this target and linked into it + +Dependencies can either be within the same package (such as `MyBinary`’s dependency on `:mylib`) or on a different package in the same source hierarchy (such as `mylib`’s dependency on `//java/com/example/common`). + +As with task-based build systems, you perform builds using Bazel’s command-line tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After entering that command for the first time in a clean repository, Bazel: + +1. Parses every `BUILD` file in the workspace to create a graph of dependencies among artifacts. +2. Uses the graph to determine the transitive dependencies of `MyBinary`; that is, every target that `MyBinary` depends on and every target that those targets depend on, recursively. +3. Builds each of those dependencies, in order. Bazel starts by building each target that has no other dependencies and keeps track of which dependencies still need to be built for each target. As soon as all of a target’s dependencies are built, Bazel starts building that target. This process continues until every one of `MyBinary`’s transitive dependencies have been built. +4. Builds `MyBinary` to produce a final executable binary that links in all of the dependencies that were built in step 3. + +Fundamentally, it might not seem like what’s happening here is that much different than what happened when using a task-based build system. Indeed, the end result is the same binary, and the process for producing it involved analyzing a bunch of steps to find dependencies among them, and then running those steps in order. But there are critical differences. The first one appears in step 3: because Bazel knows that each target only produces a Java library, it knows that all it has to do is run the Java compiler rather than an arbitrary user-defined script, so it knows that it’s safe to run these steps in parallel. This can produce an order of magnitude performance improvement over building targets one at a time on a multicore machine, and is only possible because the artifact-based approach leaves the build system in charge of its own execution strategy so that it can make stronger guarantees about parallelism. + +The benefits extend beyond parallelism, though. The next thing that this approach gives us becomes apparent when the developer types `bazel build :MyBinary` a second time without making any changes: Bazel exits in less than a second with a message saying that the target is up to date. This is possible due to the functional programming paradigm we talked about earlier—Bazel knows that each target is the result only of running a Java compiler, and it knows that the output from the Java compiler depends only on its inputs, so as long as the inputs haven’t changed, the output can be reused. And this analysis works at every level; if `MyBinary.java` changes, Bazel knows to rebuild `MyBinary` but reuse `mylib`. If a source file for `//java/com/example/common` changes, Bazel knows to rebuild that library, `mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. Because Bazel knows about the properties of the tools it runs at every step, it’s able to rebuild only the minimum set of artifacts each time while guaranteeing that it won’t produce stale builds. + +Reframing the build process in terms of artifacts rather than tasks is subtle but powerful. By reducing the flexibility exposed to the programmer, the build system can know more about what is being done at every step of the build. It can use this knowledge to make the build far more efficient by parallelizing build processes and reusing their outputs. But this is really just the first step, and these building blocks of parallelism and reuse form the basis for a distributed and highly scalable build system. ## Other nifty Bazel tricks -Artifact-based build systems fundamentally solve the problems with parallelism -and reuse that are inherent in task-based build systems. But there are still a -few problems that came up earlier that we haven’t addressed. Bazel has clever -ways of solving each of these, and we should discuss them before moving on. +Artifact-based build systems fundamentally solve the problems with parallelism and reuse that are inherent in task-based build systems. But there are still a few problems that came up earlier that we haven’t addressed. Bazel has clever ways of solving each of these, and we should discuss them before moving on. ### Tools as dependencies -One problem we ran into earlier was that builds depended on the tools installed -on our machine, and reproducing builds across systems could be difficult due to -different tool versions or locations. The problem becomes even more difficult -when your project uses languages that require different tools based on which -platform they’re being built on or compiled for (such as, Windows versus Linux), -and each of those platforms requires a slightly different set of tools to do the -same job. +One problem we ran into earlier was that builds depended on the tools installed on our machine, and reproducing builds across systems could be difficult due to different tool versions or locations. The problem becomes even more difficult when your project uses languages that require different tools based on which platform they’re being built on or compiled for (such as, Windows versus Linux), and each of those platforms requires a slightly different set of tools to do the same job. -Bazel solves the first part of this problem by treating tools as dependencies to -each target. Every `java_library` in the workspace implicitly depends on a Java -compiler, which defaults to a well-known compiler. Whenever Bazel builds a -`java_library`, it checks to make sure that the specified compiler is available -at a known location. Just like any other dependency, if the Java compiler -changes, every artifact that depends on it is rebuilt. +Bazel solves the first part of this problem by treating tools as dependencies to each target. Every `java_library` in the workspace implicitly depends on a Java compiler, which defaults to a well-known compiler. Whenever Bazel builds a `java_library`, it checks to make sure that the specified compiler is available at a known location. Just like any other dependency, if the Java compiler changes, every artifact that depends on it is rebuilt. -Bazel solves the second part of the problem, platform independence, by setting -[build configurations](/run/build#build-config-cross-compilation). Rather than -targets depending directly on their tools, they depend on types of configurations: +Bazel solves the second part of the problem, platform independence, by setting [build configurations](/run/build#build-config-cross-compilation). Rather than targets depending directly on their tools, they depend on types of configurations: -* **Host configuration**: building tools that run during the build -* **Target configuration**: building the binary you ultimately requested +- **Host configuration**: building tools that run during the build +- **Target configuration**: building the binary you ultimately requested ### Extending the build system -Bazel comes with targets for several popular programming languages out of the -box, but engineers will always want to do more—part of the benefit of task-based -systems is their flexibility in supporting any kind of build process, and it -would be better not to give that up in an artifact-based build system. -Fortunately, Bazel allows its supported target types to be extended by -[adding custom rules](/extending/rules). - -To define a rule in Bazel, the rule author declares the inputs that the rule -requires (in the form of attributes passed in the `BUILD` file) and the fixed -set of outputs that the rule produces. The author also defines the actions that -will be generated by that rule. Each action declares its inputs and outputs, -runs a particular executable or writes a particular string to a file, and can be -connected to other actions via its inputs and outputs. This means that actions -are the lowest-level composable unit in the build system—an action can do -whatever it wants so long as it uses only its declared inputs and outputs, and -Bazel takes care of scheduling actions and caching their results as appropriate. - -The system isn’t foolproof given that there’s no way to stop an action developer -from doing something like introducing a nondeterministic process as part of -their action. But this doesn’t happen very often in practice, and pushing the -possibilities for abuse all the way down to the action level greatly decreases -opportunities for errors. Rules supporting many common languages and tools are -widely available online, and most projects will never need to define their own -rules. Even for those that do, rule definitions only need to be defined in one -central place in the repository, meaning most engineers will be able to use -those rules without ever having to worry about their implementation. +Bazel comes with targets for several popular programming languages out of the box, but engineers will always want to do more—part of the benefit of task-based systems is their flexibility in supporting any kind of build process, and it would be better not to give that up in an artifact-based build system. Fortunately, Bazel allows its supported target types to be extended by [adding custom rules](/extending/rules). + +To define a rule in Bazel, the rule author declares the inputs that the rule requires (in the form of attributes passed in the `BUILD` file) and the fixed set of outputs that the rule produces. The author also defines the actions that will be generated by that rule. Each action declares its inputs and outputs, runs a particular executable or writes a particular string to a file, and can be connected to other actions via its inputs and outputs. This means that actions are the lowest-level composable unit in the build system—an action can do whatever it wants so long as it uses only its declared inputs and outputs, and Bazel takes care of scheduling actions and caching their results as appropriate. + +The system isn’t foolproof given that there’s no way to stop an action developer from doing something like introducing a nondeterministic process as part of their action. But this doesn’t happen very often in practice, and pushing the possibilities for abuse all the way down to the action level greatly decreases opportunities for errors. Rules supporting many common languages and tools are widely available online, and most projects will never need to define their own rules. Even for those that do, rule definitions only need to be defined in one central place in the repository, meaning most engineers will be able to use those rules without ever having to worry about their implementation. ### Isolating the environment -Actions sound like they might run into the same problems as tasks in other -systems—isn’t it still possible to write actions that both write to the same -file and end up conflicting with one another? Actually, Bazel makes these -conflicts impossible by using _[sandboxing](/docs/sandboxing)_. On supported -systems, every action is isolated from every other action via a filesystem -sandbox. Effectively, each action can see only a restricted view of the -filesystem that includes the inputs it has declared and any outputs it has -produced. This is enforced by systems such as LXC on Linux, the same technology -behind Docker. This means that it’s impossible for actions to conflict with one -another because they are unable to read any files they don’t declare, and any -files that they write but don’t declare will be thrown away when the action -finishes. Bazel also uses sandboxes to restrict actions from communicating via -the network. +Actions sound like they might run into the same problems as tasks in other systems—isn’t it still possible to write actions that both write to the same file and end up conflicting with one another? Actually, Bazel makes these conflicts impossible by using *[sandboxing](/docs/sandboxing)*. On supported systems, every action is isolated from every other action via a filesystem sandbox. Effectively, each action can see only a restricted view of the filesystem that includes the inputs it has declared and any outputs it has produced. This is enforced by systems such as LXC on Linux, the same technology behind Docker. This means that it’s impossible for actions to conflict with one another because they are unable to read any files they don’t declare, and any files that they write but don’t declare will be thrown away when the action finishes. Bazel also uses sandboxes to restrict actions from communicating via the network. ### Making external dependencies deterministic -There’s still one problem remaining: build systems often need to download -dependencies (whether tools or libraries) from external sources rather than -directly building them. This can be seen in the example via the -`@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file -from Maven. - -Depending on files outside of the current workspace is risky. Those files could -change at any time, potentially requiring the build system to constantly check -whether they’re fresh. If a remote file changes without a corresponding change -in the workspace source code, it can also lead to unreproducible builds—a build -might work one day and fail the next for no obvious reason due to an unnoticed -dependency change. Finally, an external dependency can introduce a huge security -risk when it is owned by a third party: if an attacker is able to infiltrate -that third-party server, they can replace the dependency file with something of -their own design, potentially giving them full control over your build -environment and its output. - -The fundamental problem is that we want the build system to be aware of these -files without having to check them into source control. Updating a dependency -should be a conscious choice, but that choice should be made once in a central -place rather than managed by individual engineers or automatically by the -system. This is because even with a “Live at Head” model, we still want builds -to be deterministic, which implies that if you check out a commit from last -week, you should see your dependencies as they were then rather than as they are -now. - -Bazel and some other build systems address this problem by requiring a -workspacewide manifest file that lists a _cryptographic hash_ for every external -dependency in the workspace. The hash is a concise way to uniquely represent the -file without checking the entire file into source control. Whenever a new -external dependency is referenced from a workspace, that dependency’s hash is -added to the manifest, either manually or automatically. When Bazel runs a -build, it checks the actual hash of its cached dependency against the expected -hash defined in the manifest and redownloads the file only if the hash differs. - -If the artifact we download has a different hash than the one declared in the -manifest, the build will fail unless the hash in the manifest is updated. This -can be done automatically, but that change must be approved and checked into -source control before the build will accept the new dependency. This means that -there’s always a record of when a dependency was updated, and an external -dependency can’t change without a corresponding change in the workspace source. -It also means that, when checking out an older version of the source code, the -build is guaranteed to use the same dependencies that it was using at the point -when that version was checked in (or else it will fail if those dependencies are -no longer available). - -Of course, it can still be a problem if a remote server becomes unavailable or -starts serving corrupt data—this can cause all of your builds to begin failing -if you don’t have another copy of that dependency available. To avoid this -problem, we recommend that, for any nontrivial project, you mirror all of its -dependencies onto servers or services that you trust and control. Otherwise you -will always be at the mercy of a third party for your build system’s -availability, even if the checked-in hashes guarantee its security. +There’s still one problem remaining: build systems often need to download dependencies (whether tools or libraries) from external sources rather than directly building them. This can be seen in the example via the `@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file from Maven. + +Depending on files outside of the current workspace is risky. Those files could change at any time, potentially requiring the build system to constantly check whether they’re fresh. If a remote file changes without a corresponding change in the workspace source code, it can also lead to unreproducible builds—a build might work one day and fail the next for no obvious reason due to an unnoticed dependency change. Finally, an external dependency can introduce a huge security risk when it is owned by a third party: if an attacker is able to infiltrate that third-party server, they can replace the dependency file with something of their own design, potentially giving them full control over your build environment and its output. + +The fundamental problem is that we want the build system to be aware of these files without having to check them into source control. Updating a dependency should be a conscious choice, but that choice should be made once in a central place rather than managed by individual engineers or automatically by the system. This is because even with a “Live at Head” model, we still want builds to be deterministic, which implies that if you check out a commit from last week, you should see your dependencies as they were then rather than as they are now. + +Bazel and some other build systems address this problem by requiring a workspacewide manifest file that lists a *cryptographic hash* for every external dependency in the workspace. The hash is a concise way to uniquely represent the file without checking the entire file into source control. Whenever a new external dependency is referenced from a workspace, that dependency’s hash is added to the manifest, either manually or automatically. When Bazel runs a build, it checks the actual hash of its cached dependency against the expected hash defined in the manifest and redownloads the file only if the hash differs. + +If the artifact we download has a different hash than the one declared in the manifest, the build will fail unless the hash in the manifest is updated. This can be done automatically, but that change must be approved and checked into source control before the build will accept the new dependency. This means that there’s always a record of when a dependency was updated, and an external dependency can’t change without a corresponding change in the workspace source. It also means that, when checking out an older version of the source code, the build is guaranteed to use the same dependencies that it was using at the point when that version was checked in (or else it will fail if those dependencies are no longer available). + +Of course, it can still be a problem if a remote server becomes unavailable or starts serving corrupt data—this can cause all of your builds to begin failing if you don’t have another copy of that dependency available. To avoid this problem, we recommend that, for any nontrivial project, you mirror all of its dependencies onto servers or services that you trust and control. Otherwise you will always be at the mercy of a third party for your build system’s availability, even if the checked-in hashes guarantee its security. diff --git a/basics/build-systems.mdx b/basics/build-systems.mdx index b3c63389..e2f80cb6 100644 --- a/basics/build-systems.mdx +++ b/basics/build-systems.mdx @@ -2,126 +2,44 @@ title: 'Why a Build System?' --- - - -This page discusses what build systems are, what they do, why you should use a -build system, and why compilers and build scripts aren't the best choice as your -organization starts to scale. It's intended for developers who don't have much -experience with a build system. +This page discusses what build systems are, what they do, why you should use a build system, and why compilers and build scripts aren't the best choice as your organization starts to scale. It's intended for developers who don't have much experience with a build system. ## What is a build system? -Fundamentally, all build systems have a straightforward purpose: they transform -the source code written by engineers into executable binaries that can be read -by machines. Build systems aren't just for human-authored code; they also allow -machines to create builds automatically, whether for testing or for releases to -production. In an organization with thousands of engineers, it's common that -most builds are triggered automatically rather than directly by engineers. +Fundamentally, all build systems have a straightforward purpose: they transform the source code written by engineers into executable binaries that can be read by machines. Build systems aren't just for human-authored code; they also allow machines to create builds automatically, whether for testing or for releases to production. In an organization with thousands of engineers, it's common that most builds are triggered automatically rather than directly by engineers. ### Can't I just use a compiler? -The need for a build system might not be immediately obvious. Most engineers -don't use a build system while learning to code: most start by invoking tools -like `gcc` or `javac` directly from the command line, or the equivalent in an -integrated development environment (IDE). As long as all the source code is in -the same directory, a command like this works fine: +The need for a build system might not be immediately obvious. Most engineers don't use a build system while learning to code: most start by invoking tools like `gcc` or `javac` directly from the command line, or the equivalent in an integrated development environment (IDE). As long as all the source code is in the same directory, a command like this works fine: ```posix-terminal javac *.java ``` -This instructs the Java compiler to take every Java source file in the current -directory and turn it into a binary class file. In the simplest case, this is -all you need. - -However, as soon as code expands, the complications begin. `javac` is smart -enough to look in subdirectories of the current directory to find code to -import. But it has no way of finding code stored in _other parts_ of the -filesystem (perhaps a library shared by several projects). It also only knows -how to build Java code. Large systems often involve different pieces written in -a variety of programming languages with webs of dependencies among those pieces, -meaning no compiler for a single language can possibly build the entire system. - -Once you're dealing with code from multiple languages or multiple compilation -units, building code is no longer a one-step process. Now you must evaluate what -your code depends on and build those pieces in the proper order, possibly using -a different set of tools for each piece. If any dependencies change, you must -repeat this process to avoid depending on stale binaries. For a codebase of even -moderate size, this process quickly becomes tedious and error-prone. - -The compiler also doesn’t know anything about how to handle external -dependencies, such as third-party `JAR` files in Java. Without a build system, -you could manage this by downloading the dependency from the internet, sticking -it in a `lib` folder on the hard drive, and configuring the compiler to read -libraries from that directory. Over time, it's difficult to maintain the -updates, versions, and source of these external dependencies. +This instructs the Java compiler to take every Java source file in the current directory and turn it into a binary class file. In the simplest case, this is all you need. + +However, as soon as code expands, the complications begin. `javac` is smart enough to look in subdirectories of the current directory to find code to import. But it has no way of finding code stored in *other parts* of the filesystem (perhaps a library shared by several projects). It also only knows how to build Java code. Large systems often involve different pieces written in a variety of programming languages with webs of dependencies among those pieces, meaning no compiler for a single language can possibly build the entire system. + +Once you're dealing with code from multiple languages or multiple compilation units, building code is no longer a one-step process. Now you must evaluate what your code depends on and build those pieces in the proper order, possibly using a different set of tools for each piece. If any dependencies change, you must repeat this process to avoid depending on stale binaries. For a codebase of even moderate size, this process quickly becomes tedious and error-prone. + +The compiler also doesn’t know anything about how to handle external dependencies, such as third-party `JAR` files in Java. Without a build system, you could manage this by downloading the dependency from the internet, sticking it in a `lib` folder on the hard drive, and configuring the compiler to read libraries from that directory. Over time, it's difficult to maintain the updates, versions, and source of these external dependencies. ### What about shell scripts? -Suppose that your hobby project starts out simple enough that you can build it -using just a compiler, but you begin running into some of the problems described -previously. Maybe you still don’t think you need a build system and can automate -away the tedious parts using some simple shell scripts that take care of -building things in the correct order. This helps out for a while, but pretty -soon you start running into even more problems: - -* It becomes tedious. As your system grows more complex, you begin spending - almost as much time working on your build scripts as on real code. Debugging - shell scripts is painful, with more and more hacks being layered on top of - one another. - -* It’s slow. To make sure you weren’t accidentally relying on stale libraries, - you have your build script build every dependency in order every time you - run it. You think about adding some logic to detect which parts need to be - rebuilt, but that sounds awfully complex and error prone for a script. Or - you think about specifying which parts need to be rebuilt each time, but - then you’re back to square one. - -* Good news: it’s time for a release! Better go figure out all the arguments - you need to pass to the jar command to make your final build. And remember - how to upload it and push it out to the central repository. And build and - push the documentation updates, and send out a notification to users. Hmm, - maybe this calls for another script... - -* Disaster! Your hard drive crashes, and now you need to recreate your entire - system. You were smart enough to keep all of your source files in version - control, but what about those libraries you downloaded? Can you find them - all again and make sure they were the same version as when you first - downloaded them? Your scripts probably depended on particular tools being - installed in particular places—can you restore that same environment so that - the scripts work again? What about all those environment variables you set a - long time ago to get the compiler working just right and then forgot about? - -* Despite the problems, your project is successful enough that you’re able to - begin hiring more engineers. Now you realize that it doesn’t take a disaster - for the previous problems to arise—you need to go through the same painful - bootstrapping process every time a new developer joins your team. And - despite your best efforts, there are still small differences in each - person’s system. Frequently, what works on one person’s machine doesn’t work - on another’s, and each time it takes a few hours of debugging tool paths or - library versions to figure out where the difference is. - -* You decide that you need to automate your build system. In theory, this is - as simple as getting a new computer and setting it up to run your build - script every night using cron. You still need to go through the painful - setup process, but now you don’t have the benefit of a human brain being - able to detect and resolve minor problems. Now, every morning when you get - in, you see that last night’s build failed because yesterday a developer - made a change that worked on their system but didn’t work on the automated - build system. Each time it’s a simple fix, but it happens so often that you - end up spending a lot of time each day discovering and applying these simple - fixes. - -* Builds become slower and slower as the project grows. One day, while waiting - for a build to complete, you gaze mournfully at the idle desktop of your - coworker, who is on vacation, and wish there were a way to take advantage of - all that wasted computational power. - -You’ve run into a classic problem of scale. For a single developer working on at -most a couple hundred lines of code for at most a week or two (which might have -been the entire experience thus far of a junior developer who just graduated -university), a compiler is all you need. Scripts can maybe take you a little bit -farther. But as soon as you need to coordinate across multiple developers and -their machines, even a perfect build script isn’t enough because it becomes very -difficult to account for the minor differences in those machines. At this point, -this simple approach breaks down and it’s time to invest in a real build system. +Suppose that your hobby project starts out simple enough that you can build it using just a compiler, but you begin running into some of the problems described previously. Maybe you still don’t think you need a build system and can automate away the tedious parts using some simple shell scripts that take care of building things in the correct order. This helps out for a while, but pretty soon you start running into even more problems: + +- It becomes tedious. As your system grows more complex, you begin spending almost as much time working on your build scripts as on real code. Debugging shell scripts is painful, with more and more hacks being layered on top of one another. + +- It’s slow. To make sure you weren’t accidentally relying on stale libraries, you have your build script build every dependency in order every time you run it. You think about adding some logic to detect which parts need to be rebuilt, but that sounds awfully complex and error prone for a script. Or you think about specifying which parts need to be rebuilt each time, but then you’re back to square one. + +- Good news: it’s time for a release! Better go figure out all the arguments you need to pass to the jar command to make your final build. And remember how to upload it and push it out to the central repository. And build and push the documentation updates, and send out a notification to users. Hmm, maybe this calls for another script... + +- Disaster! Your hard drive crashes, and now you need to recreate your entire system. You were smart enough to keep all of your source files in version control, but what about those libraries you downloaded? Can you find them all again and make sure they were the same version as when you first downloaded them? Your scripts probably depended on particular tools being installed in particular places—can you restore that same environment so that the scripts work again? What about all those environment variables you set a long time ago to get the compiler working just right and then forgot about? + +- Despite the problems, your project is successful enough that you’re able to begin hiring more engineers. Now you realize that it doesn’t take a disaster for the previous problems to arise—you need to go through the same painful bootstrapping process every time a new developer joins your team. And despite your best efforts, there are still small differences in each person’s system. Frequently, what works on one person’s machine doesn’t work on another’s, and each time it takes a few hours of debugging tool paths or library versions to figure out where the difference is. + +- You decide that you need to automate your build system. In theory, this is as simple as getting a new computer and setting it up to run your build script every night using cron. You still need to go through the painful setup process, but now you don’t have the benefit of a human brain being able to detect and resolve minor problems. Now, every morning when you get in, you see that last night’s build failed because yesterday a developer made a change that worked on their system but didn’t work on the automated build system. Each time it’s a simple fix, but it happens so often that you end up spending a lot of time each day discovering and applying these simple fixes. + +- Builds become slower and slower as the project grows. One day, while waiting for a build to complete, you gaze mournfully at the idle desktop of your coworker, who is on vacation, and wish there were a way to take advantage of all that wasted computational power. + +You’ve run into a classic problem of scale. For a single developer working on at most a couple hundred lines of code for at most a week or two (which might have been the entire experience thus far of a junior developer who just graduated university), a compiler is all you need. Scripts can maybe take you a little bit farther. But as soon as you need to coordinate across multiple developers and their machines, even a perfect build script isn’t enough because it becomes very difficult to account for the minor differences in those machines. At this point, this simple approach breaks down and it’s time to invest in a real build system. diff --git a/basics/dependencies.mdx b/basics/dependencies.mdx index 1d3bf8f1..308b627e 100644 --- a/basics/dependencies.mdx +++ b/basics/dependencies.mdx @@ -2,300 +2,84 @@ title: 'Dependency Management' --- - - -In looking through the previous pages, one theme repeats over and over: managing -your own code is fairly straightforward, but managing its dependencies is much -more difficult. There are all sorts of dependencies: sometimes there’s a -dependency on a task (such as “push the documentation before I mark a release as -complete”), and sometimes there’s a dependency on an artifact (such as “I need -to have the latest version of the computer vision library to build my code”). -Sometimes, you have internal dependencies on another part of your codebase, and -sometimes you have external dependencies on code or data owned by another team -(either in your organization or a third party). But in any case, the idea of “I -need that before I can have this” is something that recurs repeatedly in the -design of build systems, and managing dependencies is perhaps the most -fundamental job of a build system. +In looking through the previous pages, one theme repeats over and over: managing your own code is fairly straightforward, but managing its dependencies is much more difficult. There are all sorts of dependencies: sometimes there’s a dependency on a task (such as “push the documentation before I mark a release as complete”), and sometimes there’s a dependency on an artifact (such as “I need to have the latest version of the computer vision library to build my code”). Sometimes, you have internal dependencies on another part of your codebase, and sometimes you have external dependencies on code or data owned by another team (either in your organization or a third party). But in any case, the idea of “I need that before I can have this” is something that recurs repeatedly in the design of build systems, and managing dependencies is perhaps the most fundamental job of a build system. ## Dealing with Modules and Dependencies -Projects that use artifact-based build systems like Bazel are broken into a set -of modules, with modules expressing dependencies on one another via `BUILD` -files. Proper organization of these modules and dependencies can have a huge -effect on both the performance of the build system and how much work it takes to -maintain. +Projects that use artifact-based build systems like Bazel are broken into a set of modules, with modules expressing dependencies on one another via `BUILD` files. Proper organization of these modules and dependencies can have a huge effect on both the performance of the build system and how much work it takes to maintain. ## Using Fine-Grained Modules and the 1:1:1 Rule -The first question that comes up when structuring an artifact-based build is -deciding how much functionality an individual module should encompass. In Bazel, -a _module_ is represented by a target specifying a buildable unit like a -`java_library` or a `go_binary`. At one extreme, the entire project could be -contained in a single module by putting one `BUILD` file at the root and -recursively globbing together all of that project’s source files. At the other -extreme, nearly every source file could be made into its own module, effectively -requiring each file to list in a `BUILD` file every other file it depends on. - -Most projects fall somewhere between these extremes, and the choice involves a -trade-off between performance and maintainability. Using a single module for the -entire project might mean that you never need to touch the `BUILD` file except -when adding an external dependency, but it means that the build system must -always build the entire project all at once. This means that it won’t be able to -parallelize or distribute parts of the build, nor will it be able to cache parts -that it’s already built. One-module-per-file is the opposite: the build system -has the maximum flexibility in caching and scheduling steps of the build, but -engineers need to expend more effort maintaining lists of dependencies whenever -they change which files reference which. - -Though the exact granularity varies by language (and often even within -language), Google tends to favor significantly smaller modules than one might -typically write in a task-based build system. A typical production binary at -Google often depends on tens of thousands of targets, and even a moderate-sized -team can own several hundred targets within its codebase. For languages like -Java that have a strong built-in notion of packaging, each directory usually -contains a single package, target, and `BUILD` file (Pants, another build system -based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging -conventions frequently define multiple targets per `BUILD` file. - -The benefits of smaller build targets really begin to show at scale because they -lead to faster distributed builds and a less frequent need to rebuild targets. -The advantages become even more compelling after testing enters the picture, as -finer-grained targets mean that the build system can be much smarter about -running only a limited subset of tests that could be affected by any given -change. Because Google believes in the systemic benefits of using smaller -targets, we’ve made some strides in mitigating the downside by investing in -tooling to automatically manage `BUILD` files to avoid burdening developers. - -Some of these tools, such as `buildifier` and `buildozer`, are available with -Bazel in the [`buildtools` -directory](https://github.com/bazelbuild/buildtools). +The first question that comes up when structuring an artifact-based build is deciding how much functionality an individual module should encompass. In Bazel, a *module* is represented by a target specifying a buildable unit like a `java_library` or a `go_binary`. At one extreme, the entire project could be contained in a single module by putting one `BUILD` file at the root and recursively globbing together all of that project’s source files. At the other extreme, nearly every source file could be made into its own module, effectively requiring each file to list in a `BUILD` file every other file it depends on. + +Most projects fall somewhere between these extremes, and the choice involves a trade-off between performance and maintainability. Using a single module for the entire project might mean that you never need to touch the `BUILD` file except when adding an external dependency, but it means that the build system must always build the entire project all at once. This means that it won’t be able to parallelize or distribute parts of the build, nor will it be able to cache parts that it’s already built. One-module-per-file is the opposite: the build system has the maximum flexibility in caching and scheduling steps of the build, but engineers need to expend more effort maintaining lists of dependencies whenever they change which files reference which. + +Though the exact granularity varies by language (and often even within language), Google tends to favor significantly smaller modules than one might typically write in a task-based build system. A typical production binary at Google often depends on tens of thousands of targets, and even a moderate-sized team can own several hundred targets within its codebase. For languages like Java that have a strong built-in notion of packaging, each directory usually contains a single package, target, and `BUILD` file (Pants, another build system based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging conventions frequently define multiple targets per `BUILD` file. + +The benefits of smaller build targets really begin to show at scale because they lead to faster distributed builds and a less frequent need to rebuild targets. The advantages become even more compelling after testing enters the picture, as finer-grained targets mean that the build system can be much smarter about running only a limited subset of tests that could be affected by any given change. Because Google believes in the systemic benefits of using smaller targets, we’ve made some strides in mitigating the downside by investing in tooling to automatically manage `BUILD` files to avoid burdening developers. + +Some of these tools, such as `buildifier` and `buildozer`, are available with Bazel in the [`buildtools` directory](https://github.com/bazelbuild/buildtools). ## Minimizing Module Visibility -Bazel and other build systems allow each target to specify a visibility — a -property that determines which other targets may depend on it. A private target -can only be referenced within its own `BUILD` file. A target may grant broader -visibility to the targets of an explicitly defined list of `BUILD` files, or, in -the case of public visibility, to every target in the workspace. - -As with most programming languages, it is usually best to minimize visibility as -much as possible. Generally, teams at Google will make targets public only if -those targets represent widely used libraries available to any team at Google. -Teams that require others to coordinate with them before using their code will -maintain an allowlist of customer targets as their target’s visibility. Each -team’s internal implementation targets will be restricted to only directories -owned by the team, and most `BUILD` files will have only one target that isn’t -private. +Bazel and other build systems allow each target to specify a visibility — a property that determines which other targets may depend on it. A private target can only be referenced within its own `BUILD` file. A target may grant broader visibility to the targets of an explicitly defined list of `BUILD` files, or, in the case of public visibility, to every target in the workspace. + +As with most programming languages, it is usually best to minimize visibility as much as possible. Generally, teams at Google will make targets public only if those targets represent widely used libraries available to any team at Google. Teams that require others to coordinate with them before using their code will maintain an allowlist of customer targets as their target’s visibility. Each team’s internal implementation targets will be restricted to only directories owned by the team, and most `BUILD` files will have only one target that isn’t private. ## Managing Dependencies -Modules need to be able to refer to one another. The downside of breaking a -codebase into fine-grained modules is that you need to manage the dependencies -among those modules (though tools can help automate this). Expressing these -dependencies usually ends up being the bulk of the content in a `BUILD` file. +Modules need to be able to refer to one another. The downside of breaking a codebase into fine-grained modules is that you need to manage the dependencies among those modules (though tools can help automate this). Expressing these dependencies usually ends up being the bulk of the content in a `BUILD` file. ### Internal dependencies -In a large project broken into fine-grained modules, most dependencies are -likely to be internal; that is, on another target defined and built in the same -source repository. Internal dependencies differ from external dependencies in -that they are built from source rather than downloaded as a prebuilt artifact -while running the build. This also means that there’s no notion of “version” for -internal dependencies—a target and all of its internal dependencies are always -built at the same commit/revision in the repository. One issue that should be -handled carefully with regard to internal dependencies is how to treat -transitive dependencies (Figure 1). Suppose target A depends on target B, which -depends on a common library target C. Should target A be able to use classes -defined in target C? - -[![Transitive -dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) +In a large project broken into fine-grained modules, most dependencies are likely to be internal; that is, on another target defined and built in the same source repository. Internal dependencies differ from external dependencies in that they are built from source rather than downloaded as a prebuilt artifact while running the build. This also means that there’s no notion of “version” for internal dependencies—a target and all of its internal dependencies are always built at the same commit/revision in the repository. One issue that should be handled carefully with regard to internal dependencies is how to treat transitive dependencies (Figure 1). Suppose target A depends on target B, which depends on a common library target C. Should target A be able to use classes defined in target C? + +[![Transitive dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) **Figure 1**. Transitive dependencies -As far as the underlying tools are concerned, there’s no problem with this; both -B and C will be linked into target A when it is built, so any symbols defined in -C are known to A. Bazel allowed this for many years, but as Google grew, we -began to see problems. Suppose that B was refactored such that it no longer -needed to depend on C. If B’s dependency on C was then removed, A and any other -target that used C via a dependency on B would break. Effectively, a target’s -dependencies became part of its public contract and could never be safely -changed. This meant that dependencies accumulated over time and builds at Google -started to slow down. - -Google eventually solved this issue by introducing a “strict transitive -dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to -reference a symbol without depending on it directly and, if so, fails with an -error and a shell command that can be used to automatically insert the -dependency. Rolling this change out across Google’s entire codebase and -refactoring every one of our millions of build targets to explicitly list their -dependencies was a multiyear effort, but it was well worth it. Our builds are -now much faster given that targets have fewer unnecessary dependencies, and -engineers are empowered to remove dependencies they don’t need without worrying -about breaking targets that depend on them. - -As usual, enforcing strict transitive dependencies involved a trade-off. It made -build files more verbose, as frequently used libraries now need to be listed -explicitly in many places rather than pulled in incidentally, and engineers -needed to spend more effort adding dependencies to `BUILD` files. We’ve since -developed tools that reduce this toil by automatically detecting many missing -dependencies and adding them to a `BUILD` files without any developer -intervention. But even without such tools, we’ve found the trade-off to be well -worth it as the codebase scales: explicitly adding a dependency to `BUILD` file -is a one-time cost, but dealing with implicit transitive dependencies can cause -ongoing problems as long as the build target exists. Bazel [enforces strict -transitive -dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) -on Java code by default. +As far as the underlying tools are concerned, there’s no problem with this; both B and C will be linked into target A when it is built, so any symbols defined in C are known to A. Bazel allowed this for many years, but as Google grew, we began to see problems. Suppose that B was refactored such that it no longer needed to depend on C. If B’s dependency on C was then removed, A and any other target that used C via a dependency on B would break. Effectively, a target’s dependencies became part of its public contract and could never be safely changed. This meant that dependencies accumulated over time and builds at Google started to slow down. + +Google eventually solved this issue by introducing a “strict transitive dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to reference a symbol without depending on it directly and, if so, fails with an error and a shell command that can be used to automatically insert the dependency. Rolling this change out across Google’s entire codebase and refactoring every one of our millions of build targets to explicitly list their dependencies was a multiyear effort, but it was well worth it. Our builds are now much faster given that targets have fewer unnecessary dependencies, and engineers are empowered to remove dependencies they don’t need without worrying about breaking targets that depend on them. + +As usual, enforcing strict transitive dependencies involved a trade-off. It made build files more verbose, as frequently used libraries now need to be listed explicitly in many places rather than pulled in incidentally, and engineers needed to spend more effort adding dependencies to `BUILD` files. We’ve since developed tools that reduce this toil by automatically detecting many missing dependencies and adding them to a `BUILD` files without any developer intervention. But even without such tools, we’ve found the trade-off to be well worth it as the codebase scales: explicitly adding a dependency to `BUILD` file is a one-time cost, but dealing with implicit transitive dependencies can cause ongoing problems as long as the build target exists. Bazel [enforces strict transitive dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) on Java code by default. ### External dependencies -If a dependency isn’t internal, it must be external. External dependencies are -those on artifacts that are built and stored outside of the build system. The -dependency is imported directly from an artifact repository (typically accessed -over the internet) and used as-is rather than being built from source. One of -the biggest differences between external and internal dependencies is that -external dependencies have versions, and those versions exist independently of -the project’s source code. +If a dependency isn’t internal, it must be external. External dependencies are those on artifacts that are built and stored outside of the build system. The dependency is imported directly from an artifact repository (typically accessed over the internet) and used as-is rather than being built from source. One of the biggest differences between external and internal dependencies is that external dependencies have versions, and those versions exist independently of the project’s source code. ### Automatic versus manual dependency management -Build systems can allow the versions of external dependencies to be managed -either manually or automatically. When managed manually, the buildfile -explicitly lists the version it wants to download from the artifact repository, -often using a [semantic version string](https://semver.org/) such -as `1.1.4`. When managed automatically, the source file specifies a range of -acceptable versions, and the build system always downloads the latest one. For -example, Gradle allows a dependency version to be declared as “1.+” to specify -that any minor or patch version of a dependency is acceptable so long as the -major version is 1. - -Automatically managed dependencies can be convenient for small projects, but -they’re usually a recipe for disaster on projects of nontrivial size or that are -being worked on by more than one engineer. The problem with automatically -managed dependencies is that you have no control over when the version is -updated. There’s no way to guarantee that external parties won’t make breaking -updates (even when they claim to use semantic versioning), so a build that -worked one day might be broken the next with no easy way to detect what changed -or to roll it back to a working state. Even if the build doesn’t break, there -can be subtle behavior or performance changes that are impossible to track down. - -In contrast, because manually managed dependencies require a change in source -control, they can be easily discovered and rolled back, and it’s possible to -check out an older version of the repository to build with older dependencies. -Bazel requires that versions of all dependencies be specified manually. At even -moderate scales, the overhead of manual version management is well worth it for -the stability it provides. +Build systems can allow the versions of external dependencies to be managed either manually or automatically. When managed manually, the buildfile explicitly lists the version it wants to download from the artifact repository, often using a [semantic version string](https://semver.org/) such as `1.1.4`. When managed automatically, the source file specifies a range of acceptable versions, and the build system always downloads the latest one. For example, Gradle allows a dependency version to be declared as “1.+” to specify that any minor or patch version of a dependency is acceptable so long as the major version is 1. + +Automatically managed dependencies can be convenient for small projects, but they’re usually a recipe for disaster on projects of nontrivial size or that are being worked on by more than one engineer. The problem with automatically managed dependencies is that you have no control over when the version is updated. There’s no way to guarantee that external parties won’t make breaking updates (even when they claim to use semantic versioning), so a build that worked one day might be broken the next with no easy way to detect what changed or to roll it back to a working state. Even if the build doesn’t break, there can be subtle behavior or performance changes that are impossible to track down. + +In contrast, because manually managed dependencies require a change in source control, they can be easily discovered and rolled back, and it’s possible to check out an older version of the repository to build with older dependencies. Bazel requires that versions of all dependencies be specified manually. At even moderate scales, the overhead of manual version management is well worth it for the stability it provides. ### The One-Version Rule -Different versions of a library are usually represented by different artifacts, -so in theory there’s no reason that different versions of the same external -dependency couldn’t both be declared in the build system under different names. -That way, each target could choose which version of the dependency it wanted to -use. This causes a lot of problems in practice, so Google enforces a strict -[One-Version -Rule](https://opensource.google/docs/thirdparty/oneversion/) for -all third-party dependencies in our codebase. - -The biggest problem with allowing multiple versions is the diamond dependency -issue. Suppose that target A depends on target B and on v1 of an external -library. If target B is later refactored to add a dependency on v2 of the same -external library, target A will break because it now depends implicitly on two -different versions of the same library. Effectively, it’s never safe to add a -new dependency from a target to any third-party library with multiple versions, -because any of that target’s users could already be depending on a different -version. Following the One-Version Rule makes this conflict impossible—if a -target adds a dependency on a third-party library, any existing dependencies -will already be on that same version, so they can happily coexist. +Different versions of a library are usually represented by different artifacts, so in theory there’s no reason that different versions of the same external dependency couldn’t both be declared in the build system under different names. That way, each target could choose which version of the dependency it wanted to use. This causes a lot of problems in practice, so Google enforces a strict [One-Version Rule](https://opensource.google/docs/thirdparty/oneversion/) for all third-party dependencies in our codebase. + +The biggest problem with allowing multiple versions is the diamond dependency issue. Suppose that target A depends on target B and on v1 of an external library. If target B is later refactored to add a dependency on v2 of the same external library, target A will break because it now depends implicitly on two different versions of the same library. Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, because any of that target’s users could already be depending on a different version. Following the One-Version Rule makes this conflict impossible—if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist. ### Transitive external dependencies -Dealing with the transitive dependencies of an external dependency can be -particularly difficult. Many artifact repositories such as Maven Central, allow -artifacts to specify dependencies on particular versions of other artifacts in -the repository. Build tools like Maven or Gradle often recursively download each -transitive dependency by default, meaning that adding a single dependency in -your project could potentially cause dozens of artifacts to be downloaded in -total. - -This is very convenient: when adding a dependency on a new library, it would be -a big pain to have to track down each of that library’s transitive dependencies -and add them all manually. But there’s also a huge downside: because different -libraries can depend on different versions of the same third-party library, this -strategy necessarily violates the One-Version Rule and leads to the diamond -dependency problem. If your target depends on two external libraries that use -different versions of the same dependency, there’s no telling which one you’ll -get. This also means that updating an external dependency could cause seemingly -unrelated failures throughout the codebase if the new version begins pulling in -conflicting versions of some of its dependencies. - -Bazel did not use to automatically download transitive dependencies. It used to -employ a `WORKSPACE` file that required all transitive dependencies to be -listed, which led to a lot of pain when managing external dependencies. Bazel -has since added support for automatic transitive external dependency management -in the form of the `MODULE.bazel` file. See [external dependency -overview](/external/overview) for more details. - -Yet again, the choice here is one between convenience and scalability. Small -projects might prefer not having to worry about managing transitive dependencies -themselves and might be able to get away with using automatic transitive -dependencies. This strategy becomes less and less appealing as the organization -and codebase grows, and conflicts and unexpected results become more and more -frequent. At larger scales, the cost of manually managing dependencies is much -less than the cost of dealing with issues caused by automatic dependency -management. +Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central, allow artifacts to specify dependencies on particular versions of other artifacts in the repository. Build tools like Maven or Gradle often recursively download each transitive dependency by default, meaning that adding a single dependency in your project could potentially cause dozens of artifacts to be downloaded in total. + +This is very convenient: when adding a dependency on a new library, it would be a big pain to have to track down each of that library’s transitive dependencies and add them all manually. But there’s also a huge downside: because different libraries can depend on different versions of the same third-party library, this strategy necessarily violates the One-Version Rule and leads to the diamond dependency problem. If your target depends on two external libraries that use different versions of the same dependency, there’s no telling which one you’ll get. This also means that updating an external dependency could cause seemingly unrelated failures throughout the codebase if the new version begins pulling in conflicting versions of some of its dependencies. + +Bazel did not use to automatically download transitive dependencies. It used to employ a `WORKSPACE` file that required all transitive dependencies to be listed, which led to a lot of pain when managing external dependencies. Bazel has since added support for automatic transitive external dependency management in the form of the `MODULE.bazel` file. See [external dependency overview](/external/overview) for more details. + +Yet again, the choice here is one between convenience and scalability. Small projects might prefer not having to worry about managing transitive dependencies themselves and might be able to get away with using automatic transitive dependencies. This strategy becomes less and less appealing as the organization and codebase grows, and conflicts and unexpected results become more and more frequent. At larger scales, the cost of manually managing dependencies is much less than the cost of dealing with issues caused by automatic dependency management. ### Caching build results using external dependencies -External dependencies are most often provided by third parties that release -stable versions of libraries, perhaps without providing source code. Some -organizations might also choose to make some of their own code available as -artifacts, allowing other pieces of code to depend on them as third-party rather -than internal dependencies. This can theoretically speed up builds if artifacts -are slow to build but quick to download. - -However, this also introduces a lot of overhead and complexity: someone needs to -be responsible for building each of those artifacts and uploading them to the -artifact repository, and clients need to ensure that they stay up to date with -the latest version. Debugging also becomes much more difficult because different -parts of the system will have been built from different points in the -repository, and there is no longer a consistent view of the source tree. - -A better way to solve the problem of artifacts taking a long time to build is to -use a build system that supports remote caching, as described earlier. Such a -build system saves the resulting artifacts from every build to a location that -is shared across engineers, so if a developer depends on an artifact that was -recently built by someone else, the build system automatically downloads it -instead of building it. This provides all of the performance benefits of -depending directly on artifacts while still ensuring that builds are as -consistent as if they were always built from the same source. This is the -strategy used internally by Google, and Bazel can be configured to use a remote -cache. +External dependencies are most often provided by third parties that release stable versions of libraries, perhaps without providing source code. Some organizations might also choose to make some of their own code available as artifacts, allowing other pieces of code to depend on them as third-party rather than internal dependencies. This can theoretically speed up builds if artifacts are slow to build but quick to download. + +However, this also introduces a lot of overhead and complexity: someone needs to be responsible for building each of those artifacts and uploading them to the artifact repository, and clients need to ensure that they stay up to date with the latest version. Debugging also becomes much more difficult because different parts of the system will have been built from different points in the repository, and there is no longer a consistent view of the source tree. + +A better way to solve the problem of artifacts taking a long time to build is to use a build system that supports remote caching, as described earlier. Such a build system saves the resulting artifacts from every build to a location that is shared across engineers, so if a developer depends on an artifact that was recently built by someone else, the build system automatically downloads it instead of building it. This provides all of the performance benefits of depending directly on artifacts while still ensuring that builds are as consistent as if they were always built from the same source. This is the strategy used internally by Google, and Bazel can be configured to use a remote cache. ### Security and reliability of external dependencies -Depending on artifacts from third-party sources is inherently risky. There’s an -availability risk if the third-party source (such as an artifact repository) -goes down, because your entire build might grind to a halt if it’s unable to -download an external dependency. There’s also a security risk: if the -third-party system is compromised by an attacker, the attacker could replace the -referenced artifact with one of their own design, allowing them to inject -arbitrary code into your build. Both problems can be mitigated by mirroring any -artifacts you depend on onto servers you control and blocking your build system -from accessing third-party artifact repositories like Maven Central. The -trade-off is that these mirrors take effort and resources to maintain, so the -choice of whether to use them often depends on the scale of the project. The -security issue can also be completely prevented with little overhead by -requiring the hash of each third-party artifact to be specified in the source -repository, causing the build to fail if the artifact is tampered with. Another -alternative that completely sidesteps the issue is to vendor your project’s -dependencies. When a project vendors its dependencies, it checks them into -source control alongside the project’s source code, either as source or as -binaries. This effectively means that all of the project’s external dependencies -are converted to internal dependencies. Google uses this approach internally, -checking every third-party library referenced throughout Google into a -`third_party` directory at the root of Google’s source tree. However, this works -at Google only because Google’s source control system is custom built to handle -an extremely large monorepo, so vendoring might not be an option for all -organizations. +Depending on artifacts from third-party sources is inherently risky. There’s an availability risk if the third-party source (such as an artifact repository) goes down, because your entire build might grind to a halt if it’s unable to download an external dependency. There’s also a security risk: if the third-party system is compromised by an attacker, the attacker could replace the referenced artifact with one of their own design, allowing them to inject arbitrary code into your build. Both problems can be mitigated by mirroring any artifacts you depend on onto servers you control and blocking your build system from accessing third-party artifact repositories like Maven Central. The trade-off is that these mirrors take effort and resources to maintain, so the choice of whether to use them often depends on the scale of the project. The security issue can also be completely prevented with little overhead by requiring the hash of each third-party artifact to be specified in the source repository, causing the build to fail if the artifact is tampered with. Another alternative that completely sidesteps the issue is to vendor your project’s dependencies. When a project vendors its dependencies, it checks them into source control alongside the project’s source code, either as source or as binaries. This effectively means that all of the project’s external dependencies are converted to internal dependencies. Google uses this approach internally, checking every third-party library referenced throughout Google into a `third_party` directory at the root of Google’s source tree. However, this works at Google only because Google’s source control system is custom built to handle an extremely large monorepo, so vendoring might not be an option for all organizations. diff --git a/basics/distributed-builds.mdx b/basics/distributed-builds.mdx index c32f44ff..b11e7c9b 100644 --- a/basics/distributed-builds.mdx +++ b/basics/distributed-builds.mdx @@ -2,136 +2,46 @@ title: 'Distributed Builds' --- - - -When you have a large codebase, chains of dependencies can become very deep. -Even simple binaries can often depend on tens of thousands of build targets. At -this scale, it’s simply impossible to complete a build in a reasonable amount -of time on a single machine: no build system can get around the fundamental -laws of physics imposed on a machine’s hardware. The only way to make this work -is with a build system that supports distributed builds wherein the units of -work being done by the system are spread across an arbitrary and scalable -number of machines. Assuming we’ve broken the system’s work into small enough -units (more on this later), this would allow us to complete any build of any -size as quickly as we’re willing to pay for. This scalability is the holy grail -we’ve been working toward by defining an artifact-based build system. +When you have a large codebase, chains of dependencies can become very deep. Even simple binaries can often depend on tens of thousands of build targets. At this scale, it’s simply impossible to complete a build in a reasonable amount of time on a single machine: no build system can get around the fundamental laws of physics imposed on a machine’s hardware. The only way to make this work is with a build system that supports distributed builds wherein the units of work being done by the system are spread across an arbitrary and scalable number of machines. Assuming we’ve broken the system’s work into small enough units (more on this later), this would allow us to complete any build of any size as quickly as we’re willing to pay for. This scalability is the holy grail we’ve been working toward by defining an artifact-based build system. ## Remote caching -The simplest type of distributed build is one that only leverages _remote -caching_, which is shown in Figure 1. +The simplest type of distributed build is one that only leverages *remote caching*, which is shown in Figure 1. [![Distributed build with remote caching](/images/distributed-build-remote-cache.png)](/images/distributed-build-remote-cache.png) **Figure 1**. A distributed build showing remote caching -Every system that performs builds, including both developer workstations and -continuous integration systems, shares a reference to a common remote cache -service. This service might be a fast and local short-term storage system like -Redis or a cloud service like Google Cloud Storage. Whenever a user needs to -build an artifact, whether directly or as a dependency, the system first checks -with the remote cache to see if that artifact already exists there. If so, it -can download the artifact instead of building it. If not, the system builds the -artifact itself and uploads the result back to the cache. This means that -low-level dependencies that don’t change very often can be built once and shared -across users rather than having to be rebuilt by each user. At Google, many -artifacts are served from a cache rather than built from scratch, vastly -reducing the cost of running our build system. - -For a remote caching system to work, the build system must guarantee that builds -are completely reproducible. That is, for any build target, it must be possible -to determine the set of inputs to that target such that the same set of inputs -will produce exactly the same output on any machine. This is the only way to -ensure that the results of downloading an artifact are the same as the results -of building it oneself. Note that this requires that each artifact in the cache -be keyed on both its target and a hash of its inputs—that way, different -engineers could make different modifications to the same target at the same -time, and the remote cache would store all of the resulting artifacts and serve -them appropriately without conflict. - -Of course, for there to be any benefit from a remote cache, downloading an -artifact needs to be faster than building it. This is not always the case, -especially if the cache server is far from the machine doing the build. Google’s -network and build system is carefully tuned to be able to quickly share build -results. +Every system that performs builds, including both developer workstations and continuous integration systems, shares a reference to a common remote cache service. This service might be a fast and local short-term storage system like Redis or a cloud service like Google Cloud Storage. Whenever a user needs to build an artifact, whether directly or as a dependency, the system first checks with the remote cache to see if that artifact already exists there. If so, it can download the artifact instead of building it. If not, the system builds the artifact itself and uploads the result back to the cache. This means that low-level dependencies that don’t change very often can be built once and shared across users rather than having to be rebuilt by each user. At Google, many artifacts are served from a cache rather than built from scratch, vastly reducing the cost of running our build system. + +For a remote caching system to work, the build system must guarantee that builds are completely reproducible. That is, for any build target, it must be possible to determine the set of inputs to that target such that the same set of inputs will produce exactly the same output on any machine. This is the only way to ensure that the results of downloading an artifact are the same as the results of building it oneself. Note that this requires that each artifact in the cache be keyed on both its target and a hash of its inputs—that way, different engineers could make different modifications to the same target at the same time, and the remote cache would store all of the resulting artifacts and serve them appropriately without conflict. + +Of course, for there to be any benefit from a remote cache, downloading an artifact needs to be faster than building it. This is not always the case, especially if the cache server is far from the machine doing the build. Google’s network and build system is carefully tuned to be able to quickly share build results. ## Remote execution -Remote caching isn’t a true distributed build. If the cache is lost or if you -make a low-level change that requires everything to be rebuilt, you still need -to perform the entire build locally on your machine. The true goal is to support -remote execution, in which the actual work of doing the build can be spread -across any number of workers. Figure 2 depicts a remote execution system. +Remote caching isn’t a true distributed build. If the cache is lost or if you make a low-level change that requires everything to be rebuilt, you still need to perform the entire build locally on your machine. The true goal is to support remote execution, in which the actual work of doing the build can be spread across any number of workers. Figure 2 depicts a remote execution system. [![Remote execution system](/images/remote-execution-system.png)](/images/remote-execution-system.png) **Figure 2**. A remote execution system -The build tool running on each user’s machine (where users are either human -engineers or automated build systems) sends requests to a central build master. -The build master breaks the requests into their component actions and schedules -the execution of those actions over a scalable pool of workers. Each worker -performs the actions asked of it with the inputs specified by the user and -writes out the resulting artifacts. These artifacts are shared across the other -machines executing actions that require them until the final output can be -produced and sent to the user. - -The trickiest part of implementing such a system is managing the communication -between the workers, the master, and the user’s local machine. Workers might -depend on intermediate artifacts produced by other workers, and the final output -needs to be sent back to the user’s local machine. To do this, we can build on -top of the distributed cache described previously by having each worker write -its results to and read its dependencies from the cache. The master blocks -workers from proceeding until everything they depend on has finished, in which -case they’ll be able to read their inputs from the cache. The final product is -also cached, allowing the local machine to download it. Note that we also need a -separate means of exporting the local changes in the user’s source tree so that -workers can apply those changes before building. - -For this to work, all of the parts of the artifact-based build systems described -earlier need to come together. Build environments must be completely -self-describing so that we can spin up workers without human intervention. Build -processes themselves must be completely self-contained because each step might -be executed on a different machine. Outputs must be completely deterministic so -that each worker can trust the results it receives from other workers. Such -guarantees are extremely difficult for a task-based system to provide, which -makes it nigh-impossible to build a reliable remote execution system on top of -one. +The build tool running on each user’s machine (where users are either human engineers or automated build systems) sends requests to a central build master. The build master breaks the requests into their component actions and schedules the execution of those actions over a scalable pool of workers. Each worker performs the actions asked of it with the inputs specified by the user and writes out the resulting artifacts. These artifacts are shared across the other machines executing actions that require them until the final output can be produced and sent to the user. + +The trickiest part of implementing such a system is managing the communication between the workers, the master, and the user’s local machine. Workers might depend on intermediate artifacts produced by other workers, and the final output needs to be sent back to the user’s local machine. To do this, we can build on top of the distributed cache described previously by having each worker write its results to and read its dependencies from the cache. The master blocks workers from proceeding until everything they depend on has finished, in which case they’ll be able to read their inputs from the cache. The final product is also cached, allowing the local machine to download it. Note that we also need a separate means of exporting the local changes in the user’s source tree so that workers can apply those changes before building. + +For this to work, all of the parts of the artifact-based build systems described earlier need to come together. Build environments must be completely self-describing so that we can spin up workers without human intervention. Build processes themselves must be completely self-contained because each step might be executed on a different machine. Outputs must be completely deterministic so that each worker can trust the results it receives from other workers. Such guarantees are extremely difficult for a task-based system to provide, which makes it nigh-impossible to build a reliable remote execution system on top of one. ## Distributed builds at Google -Since 2008, Google has been using a distributed build system that employs both -remote caching and remote execution, which is illustrated in Figure 3. +Since 2008, Google has been using a distributed build system that employs both remote caching and remote execution, which is illustrated in Figure 3. [![High-level build system](/images/high-level-build-system.png)](/images/high-level-build-system.png) **Figure 3**. Google’s distributed build system -Google’s remote cache is called ObjFS. It consists of a backend that stores -build outputs in Bigtables distributed throughout our fleet of production -machines and a frontend FUSE daemon named objfsd that runs on each developer’s -machine. The FUSE daemon allows engineers to browse build outputs as if they -were normal files stored on the workstation, but with the file content -downloaded on-demand only for the few files that are directly requested by the -user. Serving file contents on-demand greatly reduces both network and disk -usage, and the system is able to build twice as fast compared to when we stored -all build output on the developer’s local disk. - -Google’s remote execution system is called Forge. A Forge client in Blaze -(Bazel's internal equivalent) called -the Distributor sends requests for each action to a job running in our -datacenters called the Scheduler. The Scheduler maintains a cache of action -results, allowing it to return a response immediately if the action has already -been created by any other user of the system. If not, it places the action into -a queue. A large pool of Executor jobs continually read actions from this queue, -execute them, and store the results directly in the ObjFS Bigtables. These -results are available to the executors for future actions, or to be downloaded -by the end user via objfsd. - -The end result is a system that scales to efficiently support all builds -performed at Google. And the scale of Google’s builds is truly massive: Google -runs millions of builds executing millions of test cases and producing petabytes -of build outputs from billions of lines of source code every day. Not only does -such a system let our engineers build complex codebases quickly, it also allows -us to implement a huge number of automated tools and systems that rely on our -build. +Google’s remote cache is called ObjFS. It consists of a backend that stores build outputs in Bigtables distributed throughout our fleet of production machines and a frontend FUSE daemon named objfsd that runs on each developer’s machine. The FUSE daemon allows engineers to browse build outputs as if they were normal files stored on the workstation, but with the file content downloaded on-demand only for the few files that are directly requested by the user. Serving file contents on-demand greatly reduces both network and disk usage, and the system is able to build twice as fast compared to when we stored all build output on the developer’s local disk. + +Google’s remote execution system is called Forge. A Forge client in Blaze (Bazel's internal equivalent) called the Distributor sends requests for each action to a job running in our datacenters called the Scheduler. The Scheduler maintains a cache of action results, allowing it to return a response immediately if the action has already been created by any other user of the system. If not, it places the action into a queue. A large pool of Executor jobs continually read actions from this queue, execute them, and store the results directly in the ObjFS Bigtables. These results are available to the executors for future actions, or to be downloaded by the end user via objfsd. + +The end result is a system that scales to efficiently support all builds performed at Google. And the scale of Google’s builds is truly massive: Google runs millions of builds executing millions of test cases and producing petabytes of build outputs from billions of lines of source code every day. Not only does such a system let our engineers build complex codebases quickly, it also allows us to implement a huge number of automated tools and systems that rely on our build. diff --git a/basics/hermeticity.mdx b/basics/hermeticity.mdx index dcee3a9e..a9765af8 100644 --- a/basics/hermeticity.mdx +++ b/basics/hermeticity.mdx @@ -2,108 +2,59 @@ title: 'Hermeticity' --- - - -This page covers hermeticity, the benefits of using hermetic builds, and -strategies for identifying non-hermetic behavior in your builds. +This page covers hermeticity, the benefits of using hermetic builds, and strategies for identifying non-hermetic behavior in your builds. ## Overview -When given the same input source code and product configuration, a hermetic -build system always returns the same output by isolating the build from changes -to the host system. +When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system. -In order to isolate the build, hermetic builds are insensitive to libraries and -other software installed on the local or remote host machine. They depend on -specific versions of build tools, such as compilers, and dependencies, such as -libraries. This makes the build process self-contained as it doesn't rely on -services external to the build environment. +In order to isolate the build, hermetic builds are insensitive to libraries and other software installed on the local or remote host machine. They depend on specific versions of build tools, such as compilers, and dependencies, such as libraries. This makes the build process self-contained as it doesn't rely on services external to the build environment. The two important aspects of hermeticity are: -* **Isolation**: Hermetic build systems treat tools as source code. They - download copies of tools and manage their storage and use inside managed file - trees. This creates isolation between the host machine and local user, - including installed versions of languages. -* **Source identity**: Hermetic build systems try to ensure the sameness of - inputs. Code repositories, such as Git, identify sets of code mutations with a - unique hash code. Hermetic build systems use this hash to identify changes to - the build's input. +- **Isolation**: Hermetic build systems treat tools as source code. They download copies of tools and manage their storage and use inside managed file trees. This creates isolation between the host machine and local user, including installed versions of languages. +- **Source identity**: Hermetic build systems try to ensure the sameness of inputs. Code repositories, such as Git, identify sets of code mutations with a unique hash code. Hermetic build systems use this hash to identify changes to the build's input. ## Benefits The major benefits of hermetic builds are: -* **Speed**: The output of an action can be cached, and the action need not be - run again unless inputs change. -* **Parallel execution**: For given input and output, the build system can - construct a graph of all actions to calculate efficient and parallel - execution. The build system loads the rules and calculates an action graph - and hash inputs to look up in the cache. -* **Multiple builds**: You can build multiple hermetic builds on the same - machine, each build using different tools and versions. -* **Reproducibility**: Hermetic builds are good for troubleshooting because you - know the exact conditions that produced the build. +- **Speed**: The output of an action can be cached, and the action need not be run again unless inputs change. +- **Parallel execution**: For given input and output, the build system can construct a graph of all actions to calculate efficient and parallel execution. The build system loads the rules and calculates an action graph and hash inputs to look up in the cache. +- **Multiple builds**: You can build multiple hermetic builds on the same machine, each build using different tools and versions. +- **Reproducibility**: Hermetic builds are good for troubleshooting because you know the exact conditions that produced the build. ## Identifying non-hermeticity -If you are preparing to switch to Bazel, migration is easier if you improve -your existing builds' hermeticity in advance. Some common sources of -non-hermeticity in builds are: +If you are preparing to switch to Bazel, migration is easier if you improve your existing builds' hermeticity in advance. Some common sources of non-hermeticity in builds are: -* Arbitrary processing in `.mk` files -* Actions or tooling that create files non-deterministically, usually involving - build IDs or timestamps -* System binaries that differ across hosts (such as `/usr/bin` binaries, absolute - paths, system C++ compilers for native C++ rules autoconfiguration) -* Writing to the source tree during the build. This prevents the same source - tree from being used for another target. The first build writes to the source - tree, fixing the source tree for target A. Then trying to build target B may - fail. +- Arbitrary processing in `.mk` files +- Actions or tooling that create files non-deterministically, usually involving build IDs or timestamps +- System binaries that differ across hosts (such as `/usr/bin` binaries, absolute paths, system C++ compilers for native C++ rules autoconfiguration) +- Writing to the source tree during the build. This prevents the same source tree from being used for another target. The first build writes to the source tree, fixing the source tree for target A. Then trying to build target B may fail. ## Troubleshooting non-hermetic builds -Starting with local execution, issues that affect local cache hits reveal -non-hermetic actions. - -* Ensure null sequential builds: If you run `make` and get a successful build, - running the build again should not rebuild any targets. If you run each build - step twice or on different systems, compare a hash of the file contents and - get results that differ, the build is not reproducible. -* Run steps to - [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) - from a variety of potential client machines to ensure that you catch any - cases of client environment leaking into the actions. -* Execute a build within a docker container that contains nothing but the - checked-out source tree and explicit list of host tools. Build breakages and - error messages will catch implicit system dependencies. -* Discover and fix hermeticity problems using - [remote execution rules](/remote/rules#overview). -* Enable strict [sandboxing](/docs/sandboxing) - at the per-action level, since actions in a build can be stateful and affect - the build or the output. -* [Workspace rules](/remote/workspace) - allow developers to add dependencies to external workspaces, but they are - rich enough to allow arbitrary processing to happen in the process. You can - get a log of some potentially non-hermetic actions in Bazel workspace rules by - adding the flag - `--experimental_workspace_rules_log_file=PATH` to - your Bazel command. - -Note: Make your build fully hermetic when mixing remote and local execution, -using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote -Docker container will enable the build to execute the same in both environments. +Starting with local execution, issues that affect local cache hits reveal non-hermetic actions. + +- Ensure null sequential builds: If you run `make` and get a successful build, running the build again should not rebuild any targets. If you run each build step twice or on different systems, compare a hash of the file contents and get results that differ, the build is not reproducible. +- Run steps to [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) from a variety of potential client machines to ensure that you catch any cases of client environment leaking into the actions. +- Execute a build within a docker container that contains nothing but the checked-out source tree and explicit list of host tools. Build breakages and error messages will catch implicit system dependencies. +- Discover and fix hermeticity problems using [remote execution rules](/remote/rules#overview). +- Enable strict [sandboxing](/docs/sandboxing) at the per-action level, since actions in a build can be stateful and affect the build or the output. +- [Workspace rules](/remote/workspace) allow developers to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. You can get a log of some potentially non-hermetic actions in Bazel workspace rules by adding the flag `--experimental_workspace_rules_log_file=<var>PATH</var>` to your Bazel command. + +Note: Make your build fully hermetic when mixing remote and local execution, using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote Docker container will enable the build to execute the same in both environments. ## Hermeticity with Bazel -For more information about how other projects have had success using hermetic -builds with Bazel, see these BazelCon talks: - -* [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) -* [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) -* [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) -* [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) -* [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) -* [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) -* [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=4&t=0s) (BMW) -* [Building Self Driving Cars with Bazel + Q&A](https://www.youtube.com/watch?v=fjfFe98LTm8&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=29) (GM Cruise) +For more information about how other projects have had success using hermetic builds with Bazel, see these BazelCon talks: + +- [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) +- [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) +- [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) +- [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) +- [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) +- [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) +- [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=4\&t=0s) (BMW) +- [Building Self Driving Cars with Bazel + Q\&A](https://www.youtube.com/watch?v=fjfFe98LTm8\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=29) (GM Cruise) diff --git a/basics/index.mdx b/basics/index.mdx index f3c833fa..ecd09227 100644 --- a/basics/index.mdx +++ b/basics/index.mdx @@ -2,56 +2,28 @@ title: 'Build Basics' --- +A build system is one of the most important parts of an engineering organization because each developer interacts with it potentially dozens or hundreds of times per day. A fully featured build system is necessary to enable developer productivity as an organization scales. For individual developers, it's straightforward to just compile your code and so a build system might seem excessive. But at a larger scale, having a build system helps with managing shared dependencies, such as relying on another part of the code base, or an external resource, such as a library. Build systems help to make sure that you have everything you need to build your code before it starts building. Build systems also increase velocity when they're set up to help engineers share resources and results. +This section covers some history and basics of building and build systems, including design decisions that went into making Bazel. If you're familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you can skip this section, but it's a helpful overview to understand why artifact-based build systems are excellent at enabling scale. -A build system is one of the most important parts of an engineering organization -because each developer interacts with it potentially dozens or hundreds of times -per day. A fully featured build system is necessary to enable developer -productivity as an organization scales. For individual developers, it's -straightforward to just compile your code and so a build system might seem -excessive. But at a larger scale, having a build system helps with managing -shared dependencies, such as relying on another part of the code base, or an -external resource, such as a library. Build systems help to make sure that you -have everything you need to build your code before it starts building. Build -systems also increase velocity when they're set up to help engineers share -resources and results. +Note: Much of this section's content comes from the *Build Systems and Build Philosophy* chapter of the [*Software Engineering at Google* book](https://abseil.io/resources/swe-book/html/ch18.html). Thank you to the original author, Erik Kuefler, for allowing its reuse and modification here! -This section covers some history and basics of building and build systems, -including design decisions that went into making Bazel. If you're -familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you -can skip this section, but it's a helpful overview to understand why -artifact-based build systems are excellent at enabling scale. +- **[Why a Build System?](/basics/build-systems)** -Note: Much of this section's content comes from the _Build Systems and -Build Philosophy_ chapter of the -[_Software Engineering at Google_ book](https://abseil.io/resources/swe-book/html/ch18.html). -Thank you to the original author, Erik Kuefler, for allowing its reuse and -modification here! + If you haven't used a build system before, start here. This page covers why you should use a build system, and why compilers and build scripts aren't the best choice once your organization starts to scale beyond a few developers. -* **[Why a Build System?](/basics/build-systems)** +- **[Task-Based Build Systems](/basics/task-based-builds)** - If you haven't used a build system before, start here. This page covers why - you should use a build system, and why compilers and build scripts aren't - the best choice once your organization starts to scale beyond a few - developers. + This page discusses task-based build systems (such as Make, Maven, and Gradle) and some of their challenges. -* **[Task-Based Build Systems](/basics/task-based-builds)** +- **[Artifact-Based Build Systems](/basics/artifact-based-builds)** - This page discusses task-based build systems (such as Make, Maven, and - Gradle) and some of their challenges. + This page discusses artifact-based build systems in response to the pain points of task-based build systems. -* **[Artifact-Based Build Systems](/basics/artifact-based-builds)** +- **[Distributed Builds](/basics/distributed-builds)** - This page discusses artifact-based build systems in response to the pain - points of task-based build systems. + This page covers distributed builds, or builds that are executed outside of your local machine. This requires more robust infrastructure to share resources and build results (and is where the true wizardry happens!) -* **[Distributed Builds](/basics/distributed-builds)** +- **[Dependency Management](/basics/dependencies)** - This page covers distributed builds, or builds that are executed outside of - your local machine. This requires more robust infrastructure to share - resources and build results (and is where the true wizardry happens!) - -* **[Dependency Management](/basics/dependencies)** - - This page covers some complications of dependencies at a large scale and - strategies to counteract those complications. + This page covers some complications of dependencies at a large scale and strategies to counteract those complications. diff --git a/basics/task-based-builds.mdx b/basics/task-based-builds.mdx index 9dd3f8c0..daef39b3 100644 --- a/basics/task-based-builds.mdx +++ b/basics/task-based-builds.mdx @@ -2,94 +2,68 @@ title: 'Task-Based Build Systems' --- - - -This page covers task-based build systems, how they work and some of the -complications that can occur with task-based systems. After shell scripts, -task-based build systems are the next logical evolution of building. - +This page covers task-based build systems, how they work and some of the complications that can occur with task-based systems. After shell scripts, task-based build systems are the next logical evolution of building. ## Understanding task-based build systems -In a task-based build system, the fundamental unit of work is the task. Each -task is a script that can execute any sort of logic, and tasks specify other -tasks as dependencies that must run before them. Most major build systems in use -today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of -shell scripts, most modern build systems require engineers to create build files -that describe how to perform the build. +In a task-based build system, the fundamental unit of work is the task. Each task is a script that can execute any sort of logic, and tasks specify other tasks as dependencies that must run before them. Most major build systems in use today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of shell scripts, most modern build systems require engineers to create build files that describe how to perform the build. -Take this example from the -[Ant manual](https://ant.apache.org/manual/using.html): +Take this example from the [Ant manual](https://ant.apache.org/manual/using.html): ```xml - - +<project name="MyProject" default="dist" basedir="."> + <description> simple example build file - - - - - - - - - - - - - - - - - - - - - - - - - - - - + </description> + + <property name="src" location="src"/> + <property name="build" location="build"/> + <property name="dist" location="dist"/> + + <target name="init"> + + <tstamp/> + + <mkdir dir="${build}"/> + </target> + <target name="compile" depends="init" + description="compile the source"> + + <javac srcdir="${src}" destdir="${build}"/> + </target> + <target name="dist" depends="compile" + description="generate the distribution"> + + <mkdir dir="${dist}/lib"/> + + <jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar" basedir="${build}"/> + </target> + <target name="clean" + description="clean up"> + + <delete dir="${build}"/> + <delete dir="${dist}"/> + </target> +</project> ``` -The buildfile is written in XML and defines some simple metadata about the build -along with a list of tasks (the `` tags in the XML). (Ant uses the word -_target_ to represent a _task_, and it uses the word _task_ to refer to -_commands_.) Each task executes a list of possible commands defined by Ant, -which here include creating and deleting directories, running `javac`, and -creating a JAR file. This set of commands can be extended by user-provided -plug-ins to cover any sort of logic. Each task can also define the tasks it -depends on via the depends attribute. These dependencies form an acyclic graph, -as seen in Figure 1. +The buildfile is written in XML and defines some simple metadata about the build along with a list of tasks (the `<target>` tags in the XML). (Ant uses the word *target* to represent a *task*, and it uses the word *task* to refer to *commands*.) Each task executes a list of possible commands defined by Ant, which here include creating and deleting directories, running `javac`, and creating a JAR file. This set of commands can be extended by user-provided plug-ins to cover any sort of logic. Each task can also define the tasks it depends on via the depends attribute. These dependencies form an acyclic graph, as seen in Figure 1. [![Acrylic graph showing dependencies](/images/task-dependencies.png)](/images/task-dependencies.png) Figure 1. An acyclic graph showing dependencies -Users perform builds by providing tasks to Ant’s command-line tool. For example, -when a user types `ant dist`, Ant takes the following steps: - -1. Loads a file named `build.xml` in the current directory and parses it to - create the graph structure shown in Figure 1. -1. Looks for the task named `dist` that was provided on the command line and - discovers that it has a dependency on the task named `compile`. -1. Looks for the task named `compile` and discovers that it has a dependency on - the task named `init`. -1. Looks for the task named `init` and discovers that it has no dependencies. -1. Executes the commands defined in the `init` task. -1. Executes the commands defined in the `compile` task given that all of that - task’s dependencies have been run. -1. Executes the commands defined in the `dist` task given that all of that - task’s dependencies have been run. - -In the end, the code executed by Ant when running the `dist` task is equivalent -to the following shell script: +Users perform builds by providing tasks to Ant’s command-line tool. For example, when a user types `ant dist`, Ant takes the following steps: + +1. Loads a file named `build.xml` in the current directory and parses it to create the graph structure shown in Figure 1. +2. Looks for the task named `dist` that was provided on the command line and discovers that it has a dependency on the task named `compile`. +3. Looks for the task named `compile` and discovers that it has a dependency on the task named `init`. +4. Looks for the task named `init` and discovers that it has no dependencies. +5. Executes the commands defined in the `init` task. +6. Executes the commands defined in the `compile` task given that all of that task’s dependencies have been run. +7. Executes the commands defined in the `dist` task given that all of that task’s dependencies have been run. + +In the end, the code executed by Ant when running the `dist` task is equivalent to the following shell script: ```posix-terminal ./createTimestamp.sh @@ -103,114 +77,32 @@ mkdir -p dist/lib/ jar cf dist/lib/MyProject-$(date --iso-8601).jar build/* ``` -When the syntax is stripped away, the buildfile and the build script actually -aren’t too different. But we’ve already gained a lot by doing this. We can -create new buildfiles in other directories and link them together. We can easily -add new tasks that depend on existing tasks in arbitrary and complex ways. We -need only pass the name of a single task to the `ant` command-line tool, and it -determines everything that needs to be run. - -Ant is an old piece of software, originally released in 2000. Other tools like -Maven and Gradle have improved on Ant in the intervening years and essentially -replaced it by adding features like automatic management of external -dependencies and a cleaner syntax without any XML. But the nature of these newer -systems remains the same: they allow engineers to write build scripts in a -principled and modular way as tasks and provide tools for executing those tasks -and managing dependencies among them. +When the syntax is stripped away, the buildfile and the build script actually aren’t too different. But we’ve already gained a lot by doing this. We can create new buildfiles in other directories and link them together. We can easily add new tasks that depend on existing tasks in arbitrary and complex ways. We need only pass the name of a single task to the `ant` command-line tool, and it determines everything that needs to be run. + +Ant is an old piece of software, originally released in 2000. Other tools like Maven and Gradle have improved on Ant in the intervening years and essentially replaced it by adding features like automatic management of external dependencies and a cleaner syntax without any XML. But the nature of these newer systems remains the same: they allow engineers to write build scripts in a principled and modular way as tasks and provide tools for executing those tasks and managing dependencies among them. ## The dark side of task-based build systems -Because these tools essentially let engineers define any script as a task, they -are extremely powerful, allowing you to do pretty much anything you can imagine -with them. But that power comes with drawbacks, and task-based build systems can -become difficult to work with as their build scripts grow more complex. The -problem with such systems is that they actually end up giving _too much power to -engineers and not enough power to the system_. Because the system has no idea -what the scripts are doing, performance suffers, as it must be very conservative -in how it schedules and executes build steps. And there’s no way for the system -to confirm that each script is doing what it should, so scripts tend to grow in -complexity and end up being another thing that needs debugging. +Because these tools essentially let engineers define any script as a task, they are extremely powerful, allowing you to do pretty much anything you can imagine with them. But that power comes with drawbacks, and task-based build systems can become difficult to work with as their build scripts grow more complex. The problem with such systems is that they actually end up giving *too much power to engineers and not enough power to the system*. Because the system has no idea what the scripts are doing, performance suffers, as it must be very conservative in how it schedules and executes build steps. And there’s no way for the system to confirm that each script is doing what it should, so scripts tend to grow in complexity and end up being another thing that needs debugging. ### Difficulty of parallelizing build steps -Modern development workstations are quite powerful, with multiple cores that are -capable of executing several build steps in parallel. But task-based systems are -often unable to parallelize task execution even when it seems like they should -be able to. Suppose that task A depends on tasks B and C. Because tasks B and C -have no dependency on each other, is it safe to run them at the same time so -that the system can more quickly get to task A? Maybe, if they don’t touch any -of the same resources. But maybe not—perhaps both use the same file to track -their statuses and running them at the same time causes a conflict. There’s no -way in general for the system to know, so either it has to risk these conflicts -(leading to rare but very difficult-to-debug build problems), or it has to -restrict the entire build to running on a single thread in a single process. -This can be a huge waste of a powerful developer machine, and it completely -rules out the possibility of distributing the build across multiple machines. +Modern development workstations are quite powerful, with multiple cores that are capable of executing several build steps in parallel. But task-based systems are often unable to parallelize task execution even when it seems like they should be able to. Suppose that task A depends on tasks B and C. Because tasks B and C have no dependency on each other, is it safe to run them at the same time so that the system can more quickly get to task A? Maybe, if they don’t touch any of the same resources. But maybe not—perhaps both use the same file to track their statuses and running them at the same time causes a conflict. There’s no way in general for the system to know, so either it has to risk these conflicts (leading to rare but very difficult-to-debug build problems), or it has to restrict the entire build to running on a single thread in a single process. This can be a huge waste of a powerful developer machine, and it completely rules out the possibility of distributing the build across multiple machines. ### Difficulty performing incremental builds -A good build system allows engineers to perform reliable incremental builds such -that a small change doesn’t require the entire codebase to be rebuilt from -scratch. This is especially important if the build system is slow and unable to -parallelize build steps for the aforementioned reasons. But unfortunately, -task-based build systems struggle here, too. Because tasks can do anything, -there’s no way in general to check whether they’ve already been done. Many tasks -simply take a set of source files and run a compiler to create a set of -binaries; thus, they don’t need to be rerun if the underlying source files -haven’t changed. But without additional information, the system can’t say this -for sure—maybe the task downloads a file that could have changed, or maybe it -writes a timestamp that could be different on each run. To guarantee -correctness, the system typically must rerun every task during each build. Some -build systems try to enable incremental builds by letting engineers specify the -conditions under which a task needs to be rerun. Sometimes this is feasible, but -often it’s a much trickier problem than it appears. For example, in languages -like C++ that allow files to be included directly by other files, it’s -impossible to determine the entire set of files that must be watched for changes -without parsing the input sources. Engineers often end up taking shortcuts, and -these shortcuts can lead to rare and frustrating problems where a task result is -reused even when it shouldn’t be. When this happens frequently, engineers get -into the habit of running clean before every build to get a fresh state, -completely defeating the purpose of having an incremental build in the first -place. Figuring out when a task needs to be rerun is surprisingly subtle, and is -a job better handled by machines than humans. +A good build system allows engineers to perform reliable incremental builds such that a small change doesn’t require the entire codebase to be rebuilt from scratch. This is especially important if the build system is slow and unable to parallelize build steps for the aforementioned reasons. But unfortunately, task-based build systems struggle here, too. Because tasks can do anything, there’s no way in general to check whether they’ve already been done. Many tasks simply take a set of source files and run a compiler to create a set of binaries; thus, they don’t need to be rerun if the underlying source files haven’t changed. But without additional information, the system can’t say this for sure—maybe the task downloads a file that could have changed, or maybe it writes a timestamp that could be different on each run. To guarantee correctness, the system typically must rerun every task during each build. Some build systems try to enable incremental builds by letting engineers specify the conditions under which a task needs to be rerun. Sometimes this is feasible, but often it’s a much trickier problem than it appears. For example, in languages like C++ that allow files to be included directly by other files, it’s impossible to determine the entire set of files that must be watched for changes without parsing the input sources. Engineers often end up taking shortcuts, and these shortcuts can lead to rare and frustrating problems where a task result is reused even when it shouldn’t be. When this happens frequently, engineers get into the habit of running clean before every build to get a fresh state, completely defeating the purpose of having an incremental build in the first place. Figuring out when a task needs to be rerun is surprisingly subtle, and is a job better handled by machines than humans. ### Difficulty maintaining and debugging scripts -Finally, the build scripts imposed by task-based build systems are often just -difficult to work with. Though they often receive less scrutiny, build scripts -are code just like the system being built, and are easy places for bugs to hide. -Here are some examples of bugs that are very common when working with a -task-based build system: - -* Task A depends on task B to produce a particular file as output. The owner - of task B doesn’t realize that other tasks rely on it, so they change it to - produce output in a different location. This can’t be detected until someone - tries to run task A and finds that it fails. -* Task A depends on task B, which depends on task C, which is producing a - particular file as output that’s needed by task A. The owner of task B - decides that it doesn’t need to depend on task C any more, which causes task - A to fail even though task B doesn’t care about task C at all! -* The developer of a new task accidentally makes an assumption about the - machine running the task, such as the location of a tool or the value of - particular environment variables. The task works on their machine, but fails - whenever another developer tries it. -* A task contains a nondeterministic component, such as downloading a file - from the internet or adding a timestamp to a build. Now, people get - potentially different results each time they run the build, meaning that - engineers won’t always be able to reproduce and fix one another’s failures - or failures that occur on an automated build system. -* Tasks with multiple dependencies can create race conditions. If task A - depends on both task B and task C, and task B and C both modify the same - file, task A gets a different result depending on which one of tasks B and C - finishes first. - -There’s no general-purpose way to solve these performance, correctness, or -maintainability problems within the task-based framework laid out here. So long -as engineers can write arbitrary code that runs during the build, the system -can’t have enough information to always be able to run builds quickly and -correctly. To solve the problem, we need to take some power out of the hands of -engineers and put it back in the hands of the system and reconceptualize the -role of the system not as running tasks, but as producing artifacts. - -This approach led to the creation of artifact-based build systems, like Blaze -and Bazel. +Finally, the build scripts imposed by task-based build systems are often just difficult to work with. Though they often receive less scrutiny, build scripts are code just like the system being built, and are easy places for bugs to hide. Here are some examples of bugs that are very common when working with a task-based build system: + +- Task A depends on task B to produce a particular file as output. The owner of task B doesn’t realize that other tasks rely on it, so they change it to produce output in a different location. This can’t be detected until someone tries to run task A and finds that it fails. +- Task A depends on task B, which depends on task C, which is producing a particular file as output that’s needed by task A. The owner of task B decides that it doesn’t need to depend on task C any more, which causes task A to fail even though task B doesn’t care about task C at all! +- The developer of a new task accidentally makes an assumption about the machine running the task, such as the location of a tool or the value of particular environment variables. The task works on their machine, but fails whenever another developer tries it. +- A task contains a nondeterministic component, such as downloading a file from the internet or adding a timestamp to a build. Now, people get potentially different results each time they run the build, meaning that engineers won’t always be able to reproduce and fix one another’s failures or failures that occur on an automated build system. +- Tasks with multiple dependencies can create race conditions. If task A depends on both task B and task C, and task B and C both modify the same file, task A gets a different result depending on which one of tasks B and C finishes first. + +There’s no general-purpose way to solve these performance, correctness, or maintainability problems within the task-based framework laid out here. So long as engineers can write arbitrary code that runs during the build, the system can’t have enough information to always be able to run builds quickly and correctly. To solve the problem, we need to take some power out of the hands of engineers and put it back in the hands of the system and reconceptualize the role of the system not as running tasks, but as producing artifacts. + +This approach led to the creation of artifact-based build systems, like Blaze and Bazel. diff --git a/brand/index.mdx b/brand/index.mdx index 2a21cd43..f6b1bd31 100644 --- a/brand/index.mdx +++ b/brand/index.mdx @@ -2,87 +2,57 @@ title: 'Bazel Brand Guidelines' --- - - -The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and -are treated separately from the copyright or patent license grants contained in -the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel -Trademarks other than those permitted in these guidelines must be approved in -advance. +The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and are treated separately from the copyright or patent license grants contained in the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel Trademarks other than those permitted in these guidelines must be approved in advance. ## Purpose of the Brand Guidelines -These guidelines exist to ensure that the Bazel project can share its technology -under open source licenses while making sure that the "Bazel" brand is protected -as a meaningful source identifier in a way that's consistent with trademark law. -By adhering to these guidelines, you help to promote the freedom to use and -develop high-quality Bazel technology. +These guidelines exist to ensure that the Bazel project can share its technology under open source licenses while making sure that the "Bazel" brand is protected as a meaningful source identifier in a way that's consistent with trademark law. By adhering to these guidelines, you help to promote the freedom to use and develop high-quality Bazel technology. ## Acceptable Uses -Given the open nature of Bazel, you may use the Bazel trademark to refer to the -project without prior written permission. Examples of these approved references -include the following: +Given the open nature of Bazel, you may use the Bazel trademark to refer to the project without prior written permission. Examples of these approved references include the following: -* To refer to the Bazel Project itself; -* To link to bazel.build; -* To refer to unmodified source code or other files shared by the Bazel - repositories on GitHub; -* In blog posts, news articles, or educational materials about Bazel; -* To accurately identify that your design or implementation is based on, is - for use with, or is compatible with Bazel technology. +- To refer to the Bazel Project itself; +- To link to bazel.build; +- To refer to unmodified source code or other files shared by the Bazel repositories on GitHub; +- In blog posts, news articles, or educational materials about Bazel; +- To accurately identify that your design or implementation is based on, is for use with, or is compatible with Bazel technology. Examples: -* \[Your Product\] for Bazel -* \[Your Product\] is compatible with Bazel -* \[XYZ\] Conference for Bazel Users +- \[Your Product] for Bazel +- \[Your Product] is compatible with Bazel +- \[XYZ] Conference for Bazel Users ## General Guidelines -* The Bazel name may never be used or registered in a manner that would cause - confusion as to Google's sponsorship, affiliation, or endorsement. -* Don't use the Bazel name as part of your company name, product name, domain - name, or social media profile. -* Other than as permitted by these guidelines, the Bazel name should not be - combined with other trademarks, terms, or source identifiers. -* Don't remove, distort or alter any element of the Bazel Trademarks. That - includes modifying the Bazel Trademark, for example, through hyphenation, - combination or abbreviation. Do not shorten, abbreviate, or create acronyms - out of the Bazel Trademarks. -* Don't display the word Bazel using any different stylization, color, or font - from the surrounding text. -* Don't use the term Bazel as a verb or use it in possessive form. -* Don't use the Bazel logo on any website, product UI, or promotional - materials without prior written permission from - [product@bazel.build](mailto:product@bazel.build). +- The Bazel name may never be used or registered in a manner that would cause confusion as to Google's sponsorship, affiliation, or endorsement. +- Don't use the Bazel name as part of your company name, product name, domain name, or social media profile. +- Other than as permitted by these guidelines, the Bazel name should not be combined with other trademarks, terms, or source identifiers. +- Don't remove, distort or alter any element of the Bazel Trademarks. That includes modifying the Bazel Trademark, for example, through hyphenation, combination or abbreviation. Do not shorten, abbreviate, or create acronyms out of the Bazel Trademarks. +- Don't display the word Bazel using any different stylization, color, or font from the surrounding text. +- Don't use the term Bazel as a verb or use it in possessive form. +- Don't use the Bazel logo on any website, product UI, or promotional materials without prior written permission from [product@bazel.build](mailto:product@bazel.build). ## Usage for Events and Community Groups -The Bazel word mark may be used referentially in events, community groups, or -other gatherings related to the Bazel build system, but it may not be used in a -manner that implies official status or endorsement. +The Bazel word mark may be used referentially in events, community groups, or other gatherings related to the Bazel build system, but it may not be used in a manner that implies official status or endorsement. Examples of appropriate naming conventions are: -* \[XYZ\] Bazel User Group -* Bazel Community Day at \[XYZ\] -* \[XYZ\] Conference for Bazel Users +- \[XYZ] Bazel User Group +- Bazel Community Day at \[XYZ] +- \[XYZ] Conference for Bazel Users -where \[XYZ\] represents the location and optionally other wordings. +where \[XYZ] represents the location and optionally other wordings. -Any naming convention that may imply official status or endorsement requires -review for approval from [product@bazel.build](mailto:product@bazel.build). +Any naming convention that may imply official status or endorsement requires review for approval from [product@bazel.build](mailto:product@bazel.build). Examples of naming conventions that require prior written permission: -* BazelCon -* Bazel Conference +- BazelCon +- Bazel Conference ## Contact Us -Please do not hesitate to contact us at -[product@bazel.build](mailto:product@bazel.build) if you are unsure whether your -intended use of the Bazel Trademarks is in compliance with these guidelines, or -to ask for permission to use the Bazel Trademarks, clearly describing the -intended usage and duration. +Please do not hesitate to contact us at [product@bazel.build](mailto:product@bazel.build) if you are unsure whether your intended use of the Bazel Trademarks is in compliance with these guidelines, or to ask for permission to use the Bazel Trademarks, clearly describing the intended usage and duration. diff --git a/build/share-variables.mdx b/build/share-variables.mdx index b248034e..3bf6683b 100644 --- a/build/share-variables.mdx +++ b/build/share-variables.mdx @@ -2,13 +2,9 @@ title: 'Sharing Variables' --- +`BUILD` files are intended to be simple and declarative. They will typically consist of a series of target declarations. As your code base and your `BUILD` files get larger, you will probably notice some duplication, such as: - -`BUILD` files are intended to be simple and declarative. They will typically -consist of a series of target declarations. As your code base and your `BUILD` -files get larger, you will probably notice some duplication, such as: - -``` python +```python cc_library( name = "foo", copts = ["-DVERSION=5"], @@ -23,17 +19,11 @@ cc_library( ) ``` -Code duplication in `BUILD` files is usually fine. This can make the file more -readable: each declaration can be read and understood without any context. This -is important, not only for humans, but also for external tools. For example, a -tool might be able to read and update `BUILD` files to add missing dependencies. -Code refactoring and code reuse might prevent this kind of automated -modification. +Code duplication in `BUILD` files is usually fine. This can make the file more readable: each declaration can be read and understood without any context. This is important, not only for humans, but also for external tools. For example, a tool might be able to read and update `BUILD` files to add missing dependencies. Code refactoring and code reuse might prevent this kind of automated modification. -If it is useful to share values (for example, if values must be kept in sync), -you can introduce a variable: +If it is useful to share values (for example, if values must be kept in sync), you can introduce a variable: -``` python +```python COPTS = ["-DVERSION=5"] cc_library( @@ -50,24 +40,21 @@ cc_library( ) ``` -Multiple declarations now use the value `COPTS`. By convention, use uppercase -letters to name global constants. +Multiple declarations now use the value `COPTS`. By convention, use uppercase letters to name global constants. ## Sharing variables across multiple BUILD files -If you need to share a value across multiple `BUILD` files, you have to put it -in a `.bzl` file. `.bzl` files contain definitions (variables and functions) -that can be used in `BUILD` files. +If you need to share a value across multiple `BUILD` files, you have to put it in a `.bzl` file. `.bzl` files contain definitions (variables and functions) that can be used in `BUILD` files. In `path/to/variables.bzl`, write: -``` python +```python COPTS = ["-DVERSION=5"] ``` Then, you can update your `BUILD` files to access the variable: -``` python +```python load("//path/to:variables.bzl", "COPTS") cc_library( diff --git a/build/style-guide.mdx b/build/style-guide.mdx index cfc3cf86..d0a01fe0 100644 --- a/build/style-guide.mdx +++ b/build/style-guide.mdx @@ -2,31 +2,17 @@ title: 'BUILD Style Guide' --- - - ## Prefer DAMP BUILD files over DRY -The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by -introducing abstractions such as variables and functions to avoid redundancy in -code. +The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by introducing abstractions such as variables and functions to avoid redundancy in code. -In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — -encourages readability over uniqueness to make files easier to understand and -maintain. +In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — encourages readability over uniqueness to make files easier to understand and maintain. -`BUILD` files aren't code, they are configurations. They aren't tested like -code, but do need to be maintained by people and tools. That makes DAMP better -for them than DRY. +`BUILD` files aren't code, they are configurations. They aren't tested like code, but do need to be maintained by people and tools. That makes DAMP better for them than DRY. ## BUILD.bazel file formatting -`BUILD` file formatting follows the same approach as Go, where a standardized -tool takes care of most formatting issues. -[Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and -emits the source code in a standard style. Every `BUILD` file is therefore -formatted in the same automated way, which makes formatting a non-issue during -code reviews. It also makes it easier for tools to understand, edit, and -generate `BUILD` files. +`BUILD` file formatting follows the same approach as Go, where a standardized tool takes care of most formatting issues. [Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and emits the source code in a standard style. Every `BUILD` file is therefore formatted in the same automated way, which makes formatting a non-issue during code reviews. It also makes it easier for tools to understand, edit, and generate `BUILD` files. `BUILD` file formatting must match the output of `buildifier`. @@ -59,18 +45,15 @@ py_test( **Recommendation**: Use the following order (every element is optional): -* Package description (a comment) +- Package description (a comment) -* All `load()` statements +- All `load()` statements -* The `package()` function. +- The `package()` function. -* Calls to rules and macros +- Calls to rules and macros -Buildifier makes a distinction between a standalone comment and a comment -attached to an element. If a comment is not attached to a specific element, use -an empty line after it. The distinction is important when doing automated -changes (for example, to keep or remove a comment when deleting a rule). +Buildifier makes a distinction between a standalone comment and a comment attached to an element. If a comment is not attached to a specific element, use an empty line after it. The distinction is important when doing automated changes (for example, to keep or remove a comment when deleting a rule). ```python # Standalone comment (such as to make a section in a file) @@ -81,11 +64,7 @@ cc_library(name = "cc") ## References to targets in the current package -Files should be referred to by their paths relative to the package directory -(without ever using up-references, such as `..`). Generated files should be -prefixed with "`:`" to indicate that they are not sources. Source files -should not be prefixed with `:`. Rules should be prefixed with `:`. For -example, assuming `x.cc` is a source file: +Files should be referred to by their paths relative to the package directory (without ever using up-references, such as `..`). Generated files should be prefixed with "`:`" to indicate that they are not sources. Source files should not be prefixed with `:`. Rules should be prefixed with `:`. For example, assuming `x.cc` is a source file: ```python cc_library( @@ -98,100 +77,70 @@ genrule( name = "gen_header", srcs = [], outs = ["x.h"], - cmd = "echo 'int x();' > $@", + cmd = "echo 'int x();' > $@", ) ``` ## Target naming -Target names should be descriptive. If a target contains one source file, -the target should generally have a name derived from that source (for example, a -`cc_library` for `chat.cc` could be named `chat`, or a `java_library` for -`DirectMessage.java` could be named `direct_message`). - -The eponymous target for a package (the target with the same name as the -containing directory) should provide the functionality described by the -directory name. If there is no such target, do not create an eponymous -target. - -Prefer using the short name when referring to an eponymous target (`//x` -instead of `//x:x`). If you are in the same package, prefer the local -reference (`:x` instead of `//x`). - -Avoid using "reserved" target names which have special meaning. This includes -`all`, `__pkg__`, and `__subpackages__`, these names have special -semantics and can cause confusion and unexpected behaviors when they are used. - -In the absence of a prevailing team convention these are some non-binding -recommendations that are broadly used at Google: - -* In general, use ["snake_case"](https://en.wikipedia.org/wiki/Snake_case) - * For a `java_library` with one `src` this means using a name that is not - the same as the filename without the extension - * For Java `*_binary` and `*_test` rules, use - ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). - This allows for the target name to match one of the `src`s. For - `java_test`, this makes it possible for the `test_class` attribute to be - inferred from the name of the target. -* If there are multiple variants of a particular target then add a suffix to - disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) -* Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` -* Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to - avoid conflicts between a `_library` target and its corresponding `_binary`) -* For proto related targets: - * `proto_library` targets should have names ending in `_proto` - * Languages specific `*_proto_library` rules should match the underlying - proto but replace `_proto` with a language specific suffix such as: - * **`cc_proto_library`**: `_cc_proto` - * **`java_proto_library`**: `_java_proto` - * **`java_lite_proto_library`**: `_java_proto_lite` +Target names should be descriptive. If a target contains one source file, the target should generally have a name derived from that source (for example, a `cc_library` for `chat.cc` could be named `chat`, or a `java_library` for `DirectMessage.java` could be named `direct_message`). + +The eponymous target for a package (the target with the same name as the containing directory) should provide the functionality described by the directory name. If there is no such target, do not create an eponymous target. + +Prefer using the short name when referring to an eponymous target (`//x` instead of `//x:x`). If you are in the same package, prefer the local reference (`:x` instead of `//x`). + +Avoid using "reserved" target names which have special meaning. This includes `all`, `__pkg__`, and `__subpackages__`, these names have special semantics and can cause confusion and unexpected behaviors when they are used. + +In the absence of a prevailing team convention these are some non-binding recommendations that are broadly used at Google: + +- In general, use ["snake\_case"](https://en.wikipedia.org/wiki/Snake_case) + + - For a `java_library` with one `src` this means using a name that is not the same as the filename without the extension + - For Java `*_binary` and `*_test` rules, use ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). This allows for the target name to match one of the `src`s. For `java_test`, this makes it possible for the `test_class` attribute to be inferred from the name of the target. + +- If there are multiple variants of a particular target then add a suffix to disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) + +- Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` + +- Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to avoid conflicts between a `_library` target and its corresponding `_binary`) + +- For proto related targets: + + - `proto_library` targets should have names ending in `_proto` + + - Languages specific `*_proto_library` rules should match the underlying proto but replace `_proto` with a language specific suffix such as: + + - **`cc_proto_library`**: `_cc_proto` + - **`java_proto_library`**: `_java_proto` + - **`java_lite_proto_library`**: `_java_proto_lite` ## Visibility -Visibility should be scoped as tightly as possible, while still allowing access -by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as -appropriate. +Visibility should be scoped as tightly as possible, while still allowing access by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as appropriate. -Avoid setting package `default_visibility` to `//visibility:public`. -`//visibility:public` should be individually set only for targets in the -project's public API. These could be libraries that are designed to be depended -on by external projects or binaries that could be used by an external project's -build process. +Avoid setting package `default_visibility` to `//visibility:public`. `//visibility:public` should be individually set only for targets in the project's public API. These could be libraries that are designed to be depended on by external projects or binaries that could be used by an external project's build process. ## Dependencies -Dependencies should be restricted to direct dependencies (dependencies -needed by the sources listed in the rule). Do not list transitive dependencies. +Dependencies should be restricted to direct dependencies (dependencies needed by the sources listed in the rule). Do not list transitive dependencies. -Package-local dependencies should be listed first and referred to in a way -compatible with the -[References to targets in the current package](#targets-current-package) -section above (not by their absolute package name). +Package-local dependencies should be listed first and referred to in a way compatible with the [References to targets in the current package](#targets-current-package) section above (not by their absolute package name). -Prefer to list dependencies directly, as a single list. Putting the "common" -dependencies of several targets into a variable reduces maintainability, makes -it impossible for tools to change the dependencies of a target, and can lead to -unused dependencies. +Prefer to list dependencies directly, as a single list. Putting the "common" dependencies of several targets into a variable reduces maintainability, makes it impossible for tools to change the dependencies of a target, and can lead to unused dependencies. ## Globs -Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it -is more error-prone and less obvious than an empty list. +Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it is more error-prone and less obvious than an empty list. ### Recursive -Do not use recursive globs to match source files (for example, -`glob(["**/*.java"])`). +Do not use recursive globs to match source files (for example, `glob(["**/*.java"])`). -Recursive globs make `BUILD` files difficult to reason about because they skip -subdirectories containing `BUILD` files. +Recursive globs make `BUILD` files difficult to reason about because they skip subdirectories containing `BUILD` files. -Recursive globs are generally less efficient than having a `BUILD` file per -directory with a dependency graph defined between them as this enables better -remote caching and parallelism. +Recursive globs are generally less efficient than having a `BUILD` file per directory with a dependency graph defined between them as this enables better remote caching and parallelism. -It is good practice to author a `BUILD` file in each directory and define a -dependency graph between them. +It is good practice to author a `BUILD` file in each directory and define a dependency graph between them. ### Non-recursive @@ -199,21 +148,16 @@ Non-recursive globs are generally acceptable. ## Avoid list comprehensions -Avoid using list comprehensions at the top level of a `BUILD.bazel` file. -Automate repetitive calls by creating each named target with a separate -top-level rule or macro call. Give each a short `name` parameter for clarity. +Avoid using list comprehensions at the top level of a `BUILD.bazel` file. Automate repetitive calls by creating each named target with a separate top-level rule or macro call. Give each a short `name` parameter for clarity. List comprehension reduces the following: -* Maintainability. It's difficult or impossible for human maintainers and - large scale automated changes to update list comprehensions correctly. -* Discoverability. Since the pattern doesn't have `name` parameters, - it's hard to find the rule by name. +- Maintainability. It's difficult or impossible for human maintainers and large scale automated changes to update list comprehensions correctly. +- Discoverability. Since the pattern doesn't have `name` parameters, it's hard to find the rule by name. -A common application of the list comprehension pattern is to generate tests. For -example: +A common application of the list comprehension pattern is to generate tests. For example: -```build {.bad} +```build [[java_test( name = "test_%s_%s" % (backend, count), srcs = [ ... ], @@ -228,8 +172,7 @@ example: ]] ``` -We recommend using simpler alternatives. For example, define a macro that -generates one test and invoke it for each top-level `name`: +We recommend using simpler alternatives. For example, define a macro that generates one test and invoke it for each top-level `name`: ```build my_java_test(name = "test_fake_1", @@ -243,7 +186,7 @@ my_java_test(name = "test_fake_10", Don't use list variables to encapsulate common dependencies: -```build {.bad} +```build COMMON_DEPS = [ "//d:e", "//x/y:z", @@ -260,12 +203,11 @@ cc_library(name = "b", ) ``` -Similarly, don't use a library target with -[`exports`](/reference/be/java#java_library.exports) to group dependencies. +Similarly, don't use a library target with [`exports`](/reference/be/java#java_library.exports) to group dependencies. Instead, list the dependencies separately for each target: -```build {.good} +```build cc_library(name = "a", srcs = ["a.cc"], deps = [ @@ -285,38 +227,25 @@ cc_library(name = "b", ) ``` -Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools -maintain them. There will be repetition, but you won't have to think about how -to manage the dependencies. +Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools maintain them. There will be repetition, but you won't have to think about how to manage the dependencies. ## Prefer literal strings -Although Starlark provides string operators for concatenation (`+`) and -formatting (`%`), use them with caution. It is tempting to factor out common -string parts to make expressions more concise or break long lines. However, +Although Starlark provides string operators for concatenation (`+`) and formatting (`%`), use them with caution. It is tempting to factor out common string parts to make expressions more concise or break long lines. However, -* It is harder to read broken-up string values at a glance. +- It is harder to read broken-up string values at a glance. -* Automated tools such as - [buildozer][buildozer] and Code Search have trouble finding values and - updating them correctly when the values broken up. +- Automated tools such as [buildozer](https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md) and Code Search have trouble finding values and updating them correctly when the values broken up. -* In `BUILD` files, readability is more important than avoiding repetition - (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). +- In `BUILD` files, readability is more important than avoiding repetition (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). -* This Style Guide - [warns against splitting label-valued strings](#other-conventions) - and - [explicitly permits long lines](#differences-python-style-guide). +- This Style Guide [warns against splitting label-valued strings](#other-conventions) and [explicitly permits long lines](#differences-python-style-guide). -* Buildifier automatically fuses concatenated strings when it detects that - they are labels. +- Buildifier automatically fuses concatenated strings when it detects that they are labels. -Therefore, prefer explicit, literal strings over concatenated or formatted -strings, especially in label-type attributes such as `name` and `deps`. For -example, this `BUILD` fragment: +Therefore, prefer explicit, literal strings over concatenated or formatted strings, especially in label-type attributes such as `name` and `deps`. For example, this `BUILD` fragment: -```build {.bad} +```build NAME = "foo" PACKAGE = "//a/b" @@ -330,7 +259,7 @@ proto_library( would be better rewritten as -```build {.good} +```build proto_library( name = "foo_proto", deps = ["//a/b:other_proto"], @@ -338,66 +267,32 @@ proto_library( ) ``` -[buildozer]: https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md - ## Limit the symbols exported by each `.bzl` file -Minimize the number of symbols (rules, macros, constants, functions) exported by -each public `.bzl` (Starlark) file. We recommend that a file should export -multiple symbols only if they are certain to be used together. Otherwise, split -it into multiple `.bzl` files, each with its own [bzl_library][bzl_library]. +Minimize the number of symbols (rules, macros, constants, functions) exported by each public `.bzl` (Starlark) file. We recommend that a file should export multiple symbols only if they are certain to be used together. Otherwise, split it into multiple `.bzl` files, each with its own [bzl\_library](https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library). -Excessive symbols can cause `.bzl` files to grow into broad "libraries" of -symbols, causing changes to single files to force Bazel to rebuild many targets. - -[bzl_library]: https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library +Excessive symbols can cause `.bzl` files to grow into broad "libraries" of symbols, causing changes to single files to force Bazel to rebuild many targets. ## Other conventions - * Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), - use lowercase and underscores to declare variables (such as `my_variable`). +- Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), use lowercase and underscores to declare variables (such as `my_variable`). - * Labels should never be split, even if they are longer than 79 characters. - Labels should be string literals whenever possible. *Rationale*: It makes - find and replace easy. It also improves readability. +- Labels should never be split, even if they are longer than 79 characters. Labels should be string literals whenever possible. *Rationale*: It makes find and replace easy. It also improves readability. - * The value of the name attribute should be a literal constant string (except - in macros). *Rationale*: External tools use the name attribute to refer a - rule. They need to find rules without having to interpret code. +- The value of the name attribute should be a literal constant string (except in macros). *Rationale*: External tools use the name attribute to refer a rule. They need to find rules without having to interpret code. - * When setting boolean-type attributes, use boolean values, not integer values. - For legacy reasons, rules still convert integers to booleans as needed, - but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying - "deflake this target by rerunning it once". `flaky = True` unambiguously says - "this test is flaky". +- When setting boolean-type attributes, use boolean values, not integer values. For legacy reasons, rules still convert integers to booleans as needed, but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying "deflake this target by rerunning it once". `flaky = True` unambiguously says "this test is flaky". ## Differences with Python style guide -Although compatibility with -[Python style guide](https://www.python.org/dev/peps/pep-0008/) -is a goal, there are a few differences: - - * No strict line length limit. Long comments and long strings are often split - to 79 columns, but it is not required. It should not be enforced in code - reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this - limit. It is common for `BUILD` files to be generated or edited by tools, - which does not go well with a line length limit. - - * Implicit string concatenation is not supported. Use the `+` operator. - *Rationale*: `BUILD` files contain many string lists. It is easy to forget a - comma, which leads to a complete different result. This has created many bugs - in the past. [See also this discussion.](https://lwn.net/Articles/551438/) - - * Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: - Named arguments are much more frequent than in Python and are always on a - separate line. Spaces improve readability. This convention has been around - for a long time, and it is not worth modifying all existing `BUILD` files. - - * By default, use double quotation marks for strings. *Rationale*: This is not - specified in the Python style guide, but it recommends consistency. So we - decided to use only double-quoted strings. Many languages use double-quotes - for string literals. - - * Use a single blank line between two top-level definitions. *Rationale*: The - structure of a `BUILD` file is not like a typical Python file. It has only - top-level statements. Using a single-blank line makes `BUILD` files shorter. +Although compatibility with [Python style guide](https://www.python.org/dev/peps/pep-0008/) is a goal, there are a few differences: + +- No strict line length limit. Long comments and long strings are often split to 79 columns, but it is not required. It should not be enforced in code reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this limit. It is common for `BUILD` files to be generated or edited by tools, which does not go well with a line length limit. + +- Implicit string concatenation is not supported. Use the `+` operator. *Rationale*: `BUILD` files contain many string lists. It is easy to forget a comma, which leads to a complete different result. This has created many bugs in the past. [See also this discussion.](https://lwn.net/Articles/551438/) + +- Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: Named arguments are much more frequent than in Python and are always on a separate line. Spaces improve readability. This convention has been around for a long time, and it is not worth modifying all existing `BUILD` files. + +- By default, use double quotation marks for strings. *Rationale*: This is not specified in the Python style guide, but it recommends consistency. So we decided to use only double-quoted strings. Many languages use double-quotes for string literals. + +- Use a single blank line between two top-level definitions. *Rationale*: The structure of a `BUILD` file is not like a typical Python file. It has only top-level statements. Using a single-blank line makes `BUILD` files shorter. diff --git a/community/recommended-rules.mdx b/community/recommended-rules.mdx index 86daa056..426d8784 100644 --- a/community/recommended-rules.mdx +++ b/community/recommended-rules.mdx @@ -2,53 +2,33 @@ title: 'Recommended Rules' --- +In the documentation, we provide a list of [recommended rules](/rules). - -In the documentation, we provide a list of -[recommended rules](/rules). - -This is a set of high quality rules, which will provide a good experience to our -users. We make a distinction between the supported rules, and the hundreds of -rules you can find on the Internet. +This is a set of high quality rules, which will provide a good experience to our users. We make a distinction between the supported rules, and the hundreds of rules you can find on the Internet. ## Nomination -If a ruleset meets the requirements below, a rule maintainer can nominate it -to be part of the _recommended rules_ by filing a -[GitHub issue](https://github.com/bazelbuild/bazel/). +If a ruleset meets the requirements below, a rule maintainer can nominate it to be part of the *recommended rules* by filing a [GitHub issue](https://github.com/bazelbuild/bazel/). -After a review by the [Bazel core team](/contribute/policy), it -will be recommended on the Bazel website. +After a review by the [Bazel core team](/contribute/policy), it will be recommended on the Bazel website. ## Requirements for the rule maintainers -* The ruleset provides an important feature, useful to a large number of Bazel - users (for example, support for a widely popular language). -* The ruleset is well maintained. There must be at least two active maintainers. -* The ruleset is well documented, with examples, and easy to use. -* The ruleset follows the best practices and is performant (see - [the performance guide](/rules/performance)). -* The ruleset has sufficient test coverage. -* The ruleset is tested on - [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) - with the latest version of Bazel. Tests should always pass (when used as a - presubmit check). -* The ruleset is also tested with the upcoming incompatible changes. Breakages - should be fixed within two weeks. Migration issues should be reported to the - Bazel team quickly. +- The ruleset provides an important feature, useful to a large number of Bazel users (for example, support for a widely popular language). +- The ruleset is well maintained. There must be at least two active maintainers. +- The ruleset is well documented, with examples, and easy to use. +- The ruleset follows the best practices and is performant (see [the performance guide](/rules/performance)). +- The ruleset has sufficient test coverage. +- The ruleset is tested on [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) with the latest version of Bazel. Tests should always pass (when used as a presubmit check). +- The ruleset is also tested with the upcoming incompatible changes. Breakages should be fixed within two weeks. Migration issues should be reported to the Bazel team quickly. ## Requirements for Bazel developers -* Recommended rules are frequently tested with Bazel at head (at least once a - day). -* No change in Bazel may break a recommended rule (with the default set of - flags). If it happens, the change should be fixed or rolled back. +- Recommended rules are frequently tested with Bazel at head (at least once a day). +- No change in Bazel may break a recommended rule (with the default set of flags). If it happens, the change should be fixed or rolled back. ## Demotion -If there is a concern that a particular ruleset is no longer meeting the -requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be -filed. +If there is a concern that a particular ruleset is no longer meeting the requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be filed. -Rule maintainers will be contacted and need to respond in 2 weeks. Based on the -outcome, Bazel core team might make a decision to demote the rule set. +Rule maintainers will be contacted and need to respond in 2 weeks. Based on the outcome, Bazel core team might make a decision to demote the rule set. diff --git a/community/remote-execution-services.mdx b/community/remote-execution-services.mdx index 6dee80fa..7a971881 100644 --- a/community/remote-execution-services.mdx +++ b/community/remote-execution-services.mdx @@ -2,28 +2,23 @@ title: 'Remote Execution Services' --- - - Use the following services to run Bazel with remote execution: -* Manual +- Manual - * Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) - directly to create your own remote execution service. + - Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) directly to create your own remote execution service. -* Self-service +- Self-service - * [Buildbarn](https://github.com/buildbarn) - * [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - * [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - * [NativeLink](https://github.com/TraceMachina/nativelink) + - [Buildbarn](https://github.com/buildbarn) + - [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + - [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + - [NativeLink](https://github.com/TraceMachina/nativelink) -* Commercial +- Commercial - * [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. - * [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. - * [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, - caching, and results UI. - * [EngFlow Remote Execution](https://www.engflow.com) - Remote execution - and remote caching service with Build and Test UI. Can be self-hosted or hosted. - * [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. + - [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. + - [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. + - [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, caching, and results UI. + - [EngFlow Remote Execution](https://www.engflow.com) - Remote execution and remote caching service with Build and Test UI. Can be self-hosted or hosted. + - [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. diff --git a/community/sig.mdx b/community/sig.mdx index ae5f9189..33ef6b15 100644 --- a/community/sig.mdx +++ b/community/sig.mdx @@ -2,42 +2,23 @@ title: 'Bazel Special Interest Groups' --- +Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular areas and to support communication and coordination between [Bazel owners, maintainers, and contributors](/contribute/policy). This policy applies to [`bazelbuild`](http://github.com/bazelbuild). +SIGs do their work in public. The ideal scope for a SIG covers a well-defined domain, where the majority of participation is from the community. SIGs may focus on community maintained repositories in `bazelbuild` (such as language rules) or focus on areas of code in the Bazel repository (such as Remote Execution). -Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular -areas and to support communication and coordination between [Bazel owners, -maintainers, and contributors](/contribute/policy). This policy -applies to [`bazelbuild`](http://github.com/bazelbuild). +While not all SIGs will have the same level of energy, breadth of scope, or governance models, there should be sufficient evidence that there are community members willing to engage and contribute should the interest group be established. Before joining, review the group's work, and then get in touch with the SIG leader. Membership policies vary on a per-SIG basis. -SIGs do their work in public. The ideal scope for a SIG covers a well-defined -domain, where the majority of participation is from the community. SIGs may -focus on community maintained repositories in `bazelbuild` (such as language -rules) or focus on areas of code in the Bazel repository (such as Remote -Execution). - -While not all SIGs will have the same level of energy, breadth of scope, or -governance models, there should be sufficient evidence that there are community -members willing to engage and contribute should the interest group be -established. Before joining, review the group's work, and then get in touch -with the SIG leader. Membership policies vary on a per-SIG basis. - -See the complete list of -[Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). +See the complete list of [Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). ### Non-goals: What a SIG is not -SIGs are intended to facilitate collaboration on shared work. A SIG is -therefore: +SIGs are intended to facilitate collaboration on shared work. A SIG is therefore: -- *Not a support forum:* a mailing list and a SIG is not the same thing -- *Not immediately required:* early on in a project's life, you may not know - if you have shared work or collaborators -- *Not free labor:* energy is required to grow and coordinate the work - collaboratively +- *Not a support forum:* a mailing list and a SIG is not the same thing +- *Not immediately required:* early on in a project's life, you may not know if you have shared work or collaborators +- *Not free labor:* energy is required to grow and coordinate the work collaboratively -Bazel Owners take a conservative approach to SIG creation—thanks to the ease of -starting projects on GitHub, there are many avenues where collaboration can -happen without the need for a SIG. +Bazel Owners take a conservative approach to SIG creation—thanks to the ease of starting projects on GitHub, there are many avenues where collaboration can happen without the need for a SIG. ## SIG lifecycle @@ -45,114 +26,65 @@ This section covers how to create a SIG. ### Research and consultation -To propose a new SIG group, first gather evidence for approval, as specified -below. Some possible avenues to consider are: +To propose a new SIG group, first gather evidence for approval, as specified below. Some possible avenues to consider are: -- A well-defined problem or set of problems the group would solve -- Consultation with community members who would benefit, assessing both the - benefit and their willingness to commit -- For existing projects, evidence from issues and PRs that contributors care - about the topic -- Potential goals for the group to achieve -- Resource requirements of running the group +- A well-defined problem or set of problems the group would solve +- Consultation with community members who would benefit, assessing both the benefit and their willingness to commit +- For existing projects, evidence from issues and PRs that contributors care about the topic +- Potential goals for the group to achieve +- Resource requirements of running the group -Even if the need for a SIG seems self-evident, the research and consultation is -still important to the success of the group. +Even if the need for a SIG seems self-evident, the research and consultation is still important to the success of the group. ### Create the new group -The new group should follow the below process for chartering. In particular, it -must demonstrate: - -- A clear purpose and benefit to Bazel (either around a sub-project or - application area) -- Two or more contributors willing to act as group leads, existence of other - contributors, and evidence of demand for the group -- Each group needs to use at least one publicly accessible mailing list. A SIG - may reuse one of the public lists, such as - [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list - for @bazel.build, or create their own list -- Resources the SIG initially requires (usually, mailing list and regular - video call.) -- SIGs can serve documents and files from their directory in - [`bazelbuild/community`](https://github.com/bazelbuild/community) - or from their own repository in the - [`bazelbuild`](https://github.com/bazelbuild) GitHub - organization. SIGs may link to external resources if they choose to organize - their work outside of the `bazelbuild` GitHub organization -- Bazel Owners approve or reject SIG applications and consult other - stakeholders as necessary - -Before entering the formal parts of the process, you should consult with -the Bazel product team, at product@bazel.build. Most SIGs require conversation -and iteration before approval. - -The formal request for the new group is done by submitting a charter as a PR to -[`bazelbuild/community`](https://github.com/bazelbuild/community), -and including the request in the comments on the PR following the template -below. On approval, the PR for the group is merged and the required resources -created. +The new group should follow the below process for chartering. In particular, it must demonstrate: + +- A clear purpose and benefit to Bazel (either around a sub-project or application area) +- Two or more contributors willing to act as group leads, existence of other contributors, and evidence of demand for the group +- Each group needs to use at least one publicly accessible mailing list. A SIG may reuse one of the public lists, such as [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list for @bazel.build, or create their own list +- Resources the SIG initially requires (usually, mailing list and regular video call.) +- SIGs can serve documents and files from their directory in [`bazelbuild/community`](https://github.com/bazelbuild/community) or from their own repository in the [`bazelbuild`](https://github.com/bazelbuild) GitHub organization. SIGs may link to external resources if they choose to organize their work outside of the `bazelbuild` GitHub organization +- Bazel Owners approve or reject SIG applications and consult other stakeholders as necessary + +Before entering the formal parts of the process, you should consult with the Bazel product team, at [product@bazel.build](mailto:product@bazel.build). Most SIGs require conversation and iteration before approval. + +The formal request for the new group is done by submitting a charter as a PR to [`bazelbuild/community`](https://github.com/bazelbuild/community), and including the request in the comments on the PR following the template below. On approval, the PR for the group is merged and the required resources created. ### Template Request for New SIG -To request a new SIG, use the template in the community repo: -[SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). +To request a new SIG, use the template in the community repo: [SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). ### Chartering -To establish a group, you need a charter and must follow the Bazel -[code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). -Archives of the group will be public. Membership may either be open to all -without approval, or available on request, pending approval of the group -administrator. +To establish a group, you need a charter and must follow the Bazel [code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). Archives of the group will be public. Membership may either be open to all without approval, or available on request, pending approval of the group administrator. -The charter must nominate an administrator. As well as an administrator, the -group must include at least one person as lead (these may be the same person), -who serves as point of contact for coordination as required with the Bazel -product team. +The charter must nominate an administrator. As well as an administrator, the group must include at least one person as lead (these may be the same person), who serves as point of contact for coordination as required with the Bazel product team. -Group creators must post their charter to the group mailing list. The community -repository in the Bazel GitHub organization archives such documents and -policies. As groups evolve their practices and conventions, they should update -their charters within the relevant part of the community repository. +Group creators must post their charter to the group mailing list. The community repository in the Bazel GitHub organization archives such documents and policies. As groups evolve their practices and conventions, they should update their charters within the relevant part of the community repository. ### Collaboration and inclusion -While not mandated, the group should choose to make use of collaboration -via scheduled conference calls or chat channels to conduct meetings. Any such -meetings should be advertised on the mailing list, and notes posted to the -mailing list afterwards. Regular meetings help drive accountability and progress -in a SIG. +While not mandated, the group should choose to make use of collaboration via scheduled conference calls or chat channels to conduct meetings. Any such meetings should be advertised on the mailing list, and notes posted to the mailing list afterwards. Regular meetings help drive accountability and progress in a SIG. -Bazel product team members may proactively monitor and encourage the group to -discussion and action as appropriate. +Bazel product team members may proactively monitor and encourage the group to discussion and action as appropriate. ### Launch a SIG Required activities: -- Notify Bazel general discussion groups - ([bazel-discuss](https://groups.google.com/g/bazel-discuss), - [bazel-dev](https://groups.google.com/g/bazel-dev)). +- Notify Bazel general discussion groups ([bazel-discuss](https://groups.google.com/g/bazel-discuss), [bazel-dev](https://groups.google.com/g/bazel-dev)). Optional activities: -- Create a blog post for the Bazel blog +- Create a blog post for the Bazel blog ### Health and termination of SIGs -The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners -occasionally request the SIG lead to report on the SIG's work, to inform the -broader Bazel community of the group's activity. +The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners occasionally request the SIG lead to report on the SIG's work, to inform the broader Bazel community of the group's activity. -If a SIG no longer has a useful purpose or interested community, it may be -archived and cease operation. The Bazel product team reserves the right to -archive such inactive SIGs to maintain the overall health of the project, -though it is a less preferable outcome. A SIG may also opt to disband if -it recognizes it has reached the end of its useful life. +If a SIG no longer has a useful purpose or interested community, it may be archived and cease operation. The Bazel product team reserves the right to archive such inactive SIGs to maintain the overall health of the project, though it is a less preferable outcome. A SIG may also opt to disband if it recognizes it has reached the end of its useful life. ## Note -*This content has been adopted from Tensorflow’s -[SIG playbook](https://www.tensorflow.org/community/sig_playbook) -with modifications.* +*This content has been adopted from Tensorflow’s [SIG playbook](https://www.tensorflow.org/community/sig_playbook) with modifications.* diff --git a/community/users.mdx b/community/users.mdx index 91e26c47..81a174c5 100644 --- a/community/users.mdx +++ b/community/users.mdx @@ -2,276 +2,189 @@ title: 'Who''s Using Bazel' --- +Note: Using Bazel? You can add your company on [StackShare](https://stackshare.io/bazel). To add yourself to this page, contact [product@bazel.build](mailto:product@bazel.build). - -Note: Using Bazel? You can add your company on -[StackShare](https://stackshare.io/bazel). To add yourself to this page, -contact [product@bazel.build](mailto:product@bazel.build). - -This page lists companies and OSS projects that are known to use Bazel. -This does not constitute an endorsement. +This page lists companies and OSS projects that are known to use Bazel. This does not constitute an endorsement. ## Companies using Bazel ### [acqio](https://acqio.com.br) - +![](/community/images/acqio_logo.svg) -Acqio is a Fintech that provides payment products and services for small and -medium merchants. Acqio has a handful of monorepos and uses Bazel along with -Kubernetes to deliver fast and reliable microservices. +Acqio is a Fintech that provides payment products and services for small and medium merchants. Acqio has a handful of monorepos and uses Bazel along with Kubernetes to deliver fast and reliable microservices. ### [Adobe](https://www.adobe.com/) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Adobe_logo_and_wordmark_%282017%29.svg/440px-Adobe_logo_and_wordmark_%282017%29.svg.png) -Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for -continuous, GitOps driven Kubernetes deployments. +Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for continuous, GitOps driven Kubernetes deployments. ### [Asana](https://asana.com) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Asana_logo.svg/256px-Asana_logo.svg.png) -Asana is a web and mobile application designed to help teams track their work. -In their own words: +Asana is a web and mobile application designed to help teams track their work. In their own words: -> Bazel has increased reliability, stability, and speed for all of builds/tests -at Asana. We no longer need to clean because of incorrect caches. +> Bazel has increased reliability, stability, and speed for all of builds/tests at Asana. We no longer need to clean because of incorrect caches. ### [Ascend.io](https://ascend.io) -Ascend is a Palo Alto startup that offers solutions for large data sets -analysis. Their motto is _Big data is hard. We make it easy_. +Ascend is a Palo Alto startup that offers solutions for large data sets analysis. Their motto is *Big data is hard. We make it easy*. ### [ASML](https://asml.com) - +![](https://upload.wikimedia.org/wikipedia/en/6/6c/ASML_Holding_N.V._logo.svg) -ASML is an innovation leader in the semiconductor industry. We provide chipmakers -with everything they need – hardware, software and services – to mass produce -patterns on silicon through lithography. +ASML is an innovation leader in the semiconductor industry. We provide chipmakers with everything they need – hardware, software and services – to mass produce patterns on silicon through lithography. ### [Beeswax](https://www.beeswax.com/) -> Beeswax is a New York based startup that provides real time bidding as -service. Bazel powers their Jenkins based continuous integration and deployment -framework. Beeswax loves Bazel because it is blazingly fast, correct and well -supported across many languages and platforms. +> Beeswax is a New York based startup that provides real time bidding as service. Bazel powers their Jenkins based continuous integration and deployment framework. Beeswax loves Bazel because it is blazingly fast, correct and well supported across many languages and platforms. ### [Braintree](https://www.braintreepayments.com) - +![](https://upload.wikimedia.org/wikipedia/commons/0/00/Braintree-logo1.png) -Braintree, a PayPal subsidiary, develops payment solutions for websites and -applications. They use Bazel for parts of their internal build and Paul Gross -even posted a -[nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). +Braintree, a PayPal subsidiary, develops payment solutions for websites and applications. They use Bazel for parts of their internal build and Paul Gross even posted a [nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). ### [Canva](https://www.canva.com/) - -Canva leverages Bazel to manage its large polyglot codebase, which includes -Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered -significant developer and compute infrastructure efficiencies, for example 5-6x -decreases in average CI build times, and it continues to become the foundation -of fast, reproducible, and standardised software builds at the company. +![](https://upload.wikimedia.org/wikipedia/commons/b/bb/Canva_Logo.svg) + +Canva leverages Bazel to manage its large polyglot codebase, which includes Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered significant developer and compute infrastructure efficiencies, for example 5-6x decreases in average CI build times, and it continues to become the foundation of fast, reproducible, and standardised software builds at the company. ### [CarGurus](https://www.cargurus.com) - -CarGurus is on a mission to build the world's most trusted and transparent -automotive marketplace and uses Bazel to build their polyglot monorepo. +![](https://www.cargurus.com/gfx/reskin/logos/logo_CarGurus.svg) + +CarGurus is on a mission to build the world's most trusted and transparent automotive marketplace and uses Bazel to build their polyglot monorepo. ### [Compass](https://www.compass.com) -Compass is a tech-driven real estate platform. With an elite team of real -estate, technology and business professionals, we aim to be the best and most -trusted source for home seekers. +Compass is a tech-driven real estate platform. With an elite team of real estate, technology and business professionals, we aim to be the best and most trusted source for home seekers. ### [Databricks](https://databricks.com) - -Databricks provides cloud-based integrated workspaces based on Apache Spark™. +![](https://databricks.com/wp-content/uploads/2021/10/db-nav-logo.svg) Databricks provides cloud-based integrated workspaces based on Apache Spark™. -> The Databricks codebase is a Monorepo, containing the Scala code that powers -most of our services, Javascript for front-end UI, Python for scripting, -Jsonnet to configure our infrastructure, and much more [...] Even though our -monorepo contains a million lines of Scala, working with code within is fast -and snappy. -([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) +> The Databricks codebase is a Monorepo, containing the Scala code that powers most of our services, Javascript for front-end UI, Python for scripting, Jsonnet to configure our infrastructure, and much more \[...] Even though our monorepo contains a million lines of Scala, working with code within is fast and snappy. ([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) ### [Dataform](https://dataform.co) -Dataform provides scalable analytics for data teams. They maintain a handful of -NPM packages and a documentation site in one single monorepo and they do it all -with Bazel. +Dataform provides scalable analytics for data teams. They maintain a handful of NPM packages and a documentation site in one single monorepo and they do it all with Bazel. -After the migration to Bazel, they -[reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), -including: +After the migration to Bazel, they [reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), including: -> * Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). -> * Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes -> * Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop +> - Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). +> - Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes +> - Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in \< 30 minutes on a brand new, empty laptop ### [Deep Silver FISHLABS](https://www.dsfishlabs.com) -Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with -C++/Python/Go/C as a base for their internal build tooling and especially for -baking and deploying all their 3D Assets. + +Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with C++/Python/Go/C as a base for their internal build tooling and especially for baking and deploying all their 3D Assets. ### [Dropbox](https://www.dropbox.com/) - -At Dropbox, Bazel is a key component to our distributed build and test -environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable -production releases. + +![](/community/images/dropbox.png) At Dropbox, Bazel is a key component to our distributed build and test environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable production releases. ### [Engel & Völkers](https://www.engelvoelkers.com) -Engel & Völkers AG is a privately owned German company that, via a series of -franchised offices, provides services related to real estate transactions. +Engel & Völkers AG is a privately owned German company that, via a series of franchised offices, provides services related to real estate transactions. -> One of our internal project has seen a decrease of compilation time from 11 -minutes to roughly 1 minute, this was an impressive achievement and we are -currently working on bringing Bazel to more projects. -([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) +> One of our internal project has seen a decrease of compilation time from 11 minutes to roughly 1 minute, this was an impressive achievement and we are currently working on bringing Bazel to more projects. ([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) ### [Etsy](https://www.etsy.com/) - -Etsy is an e-commerce website focused on handmade or vintage items and supplies, -as well as unique factory-manufactured items. +![](https://upload.wikimedia.org/wikipedia/commons/a/aa/Etsy_logo_lg_rgb.png) -They use Bazel to build and test its Java-based search platform. Bazel produces -both packages for bare metal servers and repeatable Docker images. +Etsy is an e-commerce website focused on handmade or vintage items and supplies, as well as unique factory-manufactured items. + +They use Bazel to build and test its Java-based search platform. Bazel produces both packages for bare metal servers and repeatable Docker images. ### [Evertz.io](https://www.evertz.io/) -Evertz.io is a multi-tenant, serverless SaaS platform for offering cost -effective, multi-regional services worldwide to the Broadcast Media Industry, -created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). +Evertz.io is a multi-tenant, serverless SaaS platform for offering cost effective, multi-regional services worldwide to the Broadcast Media Industry, created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). -The website is fully built and deployed with an Angular and Bazel workflow -([source](https://twitter.com/MattMackay/status/1113947685508341762)). +The website is fully built and deployed with an Angular and Bazel workflow ([source](https://twitter.com/MattMackay/status/1113947685508341762)). ### [FINDMINE](http://www.findmine.com) - -FINDMINE is a automation technology for the retail industry that uses machine -learning to scale the currently manual and tedious process of product curation. -We use Bazel to mechanize our entire python package building, testing, and -deployment process. +![](https://www.findmine.com/static/assets/landpage/findmine-color-logo.png) + +FINDMINE is a automation technology for the retail industry that uses machine learning to scale the currently manual and tedious process of product curation. We use Bazel to mechanize our entire python package building, testing, and deployment process. ### [Flexport](https://www.flexport.com/) -Flexport is a tech-enabled global freight forwarder; our mission is to make -global trade easier for everyone. At Flexport, we use Bazel to build/test our -Java/JavaScript services and client libraries and to generate Java and Ruby -code from protobuf definitions. -[Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) +Flexport is a tech-enabled global freight forwarder; our mission is to make global trade easier for everyone. At Flexport, we use Bazel to build/test our Java/JavaScript services and client libraries and to generate Java and Ruby code from protobuf definitions. [Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) ### [Foursquare](https://foursquare.com) - -Foursquare's mission is to create technology that constructs meaningful -bridges between digital spaces and physical places. We manage millions of -lines of primarily Scala and Python code powering data-intensive -applications, including complex codegen and container build processes, with -Bazel. +![](https://upload.wikimedia.org/wikipedia/commons/9/99/FSQ_logo.png) + +Foursquare's mission is to create technology that constructs meaningful bridges between digital spaces and physical places. We manage millions of lines of primarily Scala and Python code powering data-intensive applications, including complex codegen and container build processes, with Bazel. ### [GermanTechJobs](https://germantechjobs.de) - -Bazel has simplified our workflows 10-fold and enabled shipping features at -scale. +![](https://upload.wikimedia.org/wikipedia/commons/9/98/GermanTechJobs_Logo.png) + +Bazel has simplified our workflows 10-fold and enabled shipping features at scale. ### [Google](https://google.com) - -Bazel was designed to be able to scale to Google's needs and meet Google's -requirements of reproducibility and platform/language support. All software at -Google is built using Bazel. Google uses Bazel and its rules for millions of -builds every day. +![](https://upload.wikimedia.org/wikipedia/commons/2/2f/Google_2015_logo.svg) + +Bazel was designed to be able to scale to Google's needs and meet Google's requirements of reproducibility and platform/language support. All software at Google is built using Bazel. Google uses Bazel and its rules for millions of builds every day. ### [Huawei](http://www.huawei.com/) -> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go -projects, except for Go projects, others originally were built by Maven. We -write a simple tool to translate a Maven-built project into Bazel-built one. -More and more projects will use Bazel in recent future. +> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go projects, except for Go projects, others originally were built by Maven. We write a simple tool to translate a Maven-built project into Bazel-built one. More and more projects will use Bazel in recent future. ### [IMC Trading](https://imc.com) - -> IMC is a global proprietary trading firm and market maker headquarted in -Amsterdam. We are using Bazel to continuously build and test our -Java/C++/Python/SystemVerilog projects. +![](https://upload.wikimedia.org/wikipedia/commons/1/17/IMC_Logo.svg) + +> IMC is a global proprietary trading firm and market maker headquarted in Amsterdam. We are using Bazel to continuously build and test our Java/C++/Python/SystemVerilog projects. ### [Improbable.io](https://improbable.io/) -Improbable.io develops SpatialOS, a distributed operating system that enables -creating huge simulations inhabited by millions of complex entities. +Improbable.io develops SpatialOS, a distributed operating system that enables creating huge simulations inhabited by millions of complex entities. ### [Interaxon](https://www.choosemuse.com/) -InteraXon is a thought-controlled computing firm that creates hardware and -software platforms to convert brainwaves into digital signals. +InteraXon is a thought-controlled computing firm that creates hardware and software platforms to convert brainwaves into digital signals. ### [Jupiter](https://jupiter.co/) -Jupiter is a company that provides delivery of groceries and household -essentials every week. +Jupiter is a company that provides delivery of groceries and household essentials every week. -They use Bazel in their backend code, specifically to compile protos and Kotlin -to JVM binaries, using remote caching. -([source](https://starship.jupiter.co/jupiter-stack/)) +They use Bazel in their backend code, specifically to compile protos and Kotlin to JVM binaries, using remote caching. ([source](https://starship.jupiter.co/jupiter-stack/)) ### [Just](https://gojust.com/) -Just is an enterprise financial technology company, headquartered in Norway, -creating software solutions to transform how global corporate treasurers manage -risk and liquidity. Their entire application stack is built with Bazel. +Just is an enterprise financial technology company, headquartered in Norway, creating software solutions to transform how global corporate treasurers manage risk and liquidity. Their entire application stack is built with Bazel. ### [Line](https://line.me/) -Line provides an app for instant communications, which is the most popular -messaging application in Japan. -They use Bazel on their codebase consisting of about 60% Swift and 40% -C/C++/Objective-C/Objective-C++ -([source](https://twitter.com/thi_dt/status/1253334262020886532)). +Line provides an app for instant communications, which is the most popular messaging application in Japan. They use Bazel on their codebase consisting of about 60% Swift and 40% C/C++/Objective-C/Objective-C++ ([source](https://twitter.com/thi_dt/status/1253334262020886532)). -> After switching to Bazel, we were able to achieve a huge improvement in the -build times. This brought a significant improvement in the turn-around time -during a QA period. Distributing a new build to our testers no longer means -another hour waiting for building and testing. -([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) +> After switching to Bazel, we were able to achieve a huge improvement in the build times. This brought a significant improvement in the turn-around time during a QA period. Distributing a new build to our testers no longer means another hour waiting for building and testing. ([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) ### [LingoChamp](https://www.liulishuo.com/en) - -LingoChamp provides professional solutions to English learners. We use Bazel -for our go, java and python projects. +![](/community/images/liulishuo.png) LingoChamp provides professional solutions to English learners. We use Bazel for our go, java and python projects. ### [LinkedIn](https://linkedin.com/) - -LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social -network. LinkedIn uses Bazel for building its iOS Apps. +![](/community/images/Linkedin-Logo.png) LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social network. LinkedIn uses Bazel for building its iOS Apps. ### [Lucid Software](https://lucid.co/) - +![](/community/images/Lucid_Software-logo.svg) -Lucid Software is a leader in visual collaboration, helping teams see and build the -future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), -[Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams -can align around a shared vision, clarify complexity, and collaborate visually, no -matter where they’re located. +Lucid Software is a leader in visual collaboration, helping teams see and build the future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), [Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams can align around a shared vision, clarify complexity, and collaborate visually, no matter where they’re located. -Lucid uses Bazel to build millions of lines of Scala and TypeScript. -Migrating to Bazel has tremendously sped up its builds, reduced external -dependencies on the build environment, and simplified developers' experience -with the build system. Bazel has improved developer productivity at Lucid and -unlocked further growth. +Lucid uses Bazel to build millions of lines of Scala and TypeScript. Migrating to Bazel has tremendously sped up its builds, reduced external dependencies on the build environment, and simplified developers' experience with the build system. Bazel has improved developer productivity at Lucid and unlocked further growth. ### [Lyft](https://www.lyft.com/) @@ -279,131 +192,81 @@ Lyft is using Bazel for their iOS ([source](https://twitter.com/SmileyKeith/stat ### [Meetup](http://www.meetup.com/) -Meetup is an online social networking portal that facilitates offline group -meetings. -The Meetup engineering team contributes to -[rules_scala](https://github.com/bazelbuild/rules_scala) and is the -maintainer of [rules_avro](https://github.com/meetup/rules_avro) -and [rules_openapi](https://github.com/meetup/rules_openapi). - +Meetup is an online social networking portal that facilitates offline group meetings. The Meetup engineering team contributes to [rules\_scala](https://github.com/bazelbuild/rules_scala) and is the maintainer of [rules\_avro](https://github.com/meetup/rules_avro) and [rules\_openapi](https://github.com/meetup/rules_openapi). ### [Nvidia](https://www.nvidia.com/) -> At Nvidia we have been using dazel(docker bazel) for python to work around -some of bazel's python short comings. Everything else runs in normal bazel -(Mostly Go / Scala/ C++/ Cuda) -([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) - +> At Nvidia we have been using dazel(docker bazel) for python to work around some of bazel's python short comings. Everything else runs in normal bazel (Mostly Go / Scala/ C++/ Cuda) ([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) ### [Peloton Technology](http://www.peloton-tech.com) -Peloton Technology is an automated vehicle technology company that tackles truck -accidents and fuel use. They use Bazel to _enable reliable builds for automotive -safety systems_. +Peloton Technology is an automated vehicle technology company that tackles truck accidents and fuel use. They use Bazel to *enable reliable builds for automotive safety systems*. ### [Pigweed](https://pigweed.dev) - +![](https://pigweed.dev/_static/pw_logo.svg) -Pigweed is an open-source solution for sustained, robust, and rapid embedded -product development for large teams. Pigweed has shipped in millions of -devices, including Google's suite of Pixel devices, Nest thermostats, -[satellites](https://www.spinlaunch.com/), and [autonomous aerial -drones](https://www.flyzipline.com/). +Pigweed is an open-source solution for sustained, robust, and rapid embedded product development for large teams. Pigweed has shipped in millions of devices, including Google's suite of Pixel devices, Nest thermostats, [satellites](https://www.spinlaunch.com/), and [autonomous aerial drones](https://www.flyzipline.com/). -Pigweed [uses Bazel as its primary build -system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for -Embedded][pw-bazel-great] blog post discusses why we think it's a great build -system for embedded projects! - -[pw-bazel-great]: https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded +Pigweed [uses Bazel as its primary build system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for Embedded](https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded) blog post discusses why we think it's a great build system for embedded projects! ### [Pinterest](https://www.pinterest.com/) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Pinterest_Logo.svg/200px-Pinterest_Logo.svg.png) -Pinterest is the world’s catalog of ideas. They use Bazel to build various -backend services (Java/C++) and the iOS application (Objective-C/C++). +Pinterest is the world’s catalog of ideas. They use Bazel to build various backend services (Java/C++) and the iOS application (Objective-C/C++). -> We identified Bazel was the best fit for our goals to build a foundation for -an order of magnitude improvement in performance, eliminate variability in -build environments and adopt incrementally. As a result, we’re now shipping all -our iOS releases using Bazel. -[Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) +> We identified Bazel was the best fit for our goals to build a foundation for an order of magnitude improvement in performance, eliminate variability in build environments and adopt incrementally. As a result, we’re now shipping all our iOS releases using Bazel. [Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) ### [PubRef](https://github.com/pubref) -PubRef is an emerging scientific publishing platform. They use Bazel with -[rules_closure](https://github.com/bazelbuild/rules_closure) to build the -frontend, native java rules to build the main backend, -[rules_go](https://github.com/bazelbuild/rules_go), -[rules_node](https://github.com/pubref/rules_node), and -[rules_kotlin](https://github.com/pubref/rules_kotlin) to build assorted -backend services. [rules_protobuf](https://github.com/pubref/rules_protobuf) is -used to assist with gRPC-based communication between backend services. -PubRef.org is based in Boulder, CO. +PubRef is an emerging scientific publishing platform. They use Bazel with [rules\_closure](https://github.com/bazelbuild/rules_closure) to build the frontend, native java rules to build the main backend, [rules\_go](https://github.com/bazelbuild/rules_go), [rules\_node](https://github.com/pubref/rules_node), and [rules\_kotlin](https://github.com/pubref/rules_kotlin) to build assorted backend services. [rules\_protobuf](https://github.com/pubref/rules_protobuf) is used to assist with gRPC-based communication between backend services. PubRef.org is based in Boulder, CO. ### [Redfin](https://redfin.com/) -Redfin is a next-generation real estate brokerage with full-service local -agents. They use Bazel to build and deploy the website and various backend -services. - -> With the conversion mostly behind us, things are greatly improved! Our CI -builds are faster (*way* faster: they used to take 40–90 minutes, and now dev -builds average 5–6 minutes). Reliability is far higher, too. This is harder to -quantify, but the shift from unexplained build failures being something that -“just happens” to being viewed as real problems to be solved has put us on a -virtuous cycle of ever-increasing reliability. -([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) + +Redfin is a next-generation real estate brokerage with full-service local agents. They use Bazel to build and deploy the website and various backend services. + +> With the conversion mostly behind us, things are greatly improved! Our CI builds are faster (*way* faster: they used to take 40–90 minutes, and now dev builds average 5–6 minutes). Reliability is far higher, too. This is harder to quantify, but the shift from unexplained build failures being something that “just happens” to being viewed as real problems to be solved has put us on a virtuous cycle of ever-increasing reliability. ([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) ### [Ritual](https://ritual.co) - -Ritual is a mobile pick up app, connecting restaurants with customers to offer -a simple, time-saving tool to get the food and beverages they want, without the -wait. Ritual uses Bazel for their backend services. +![](https://lh3.googleusercontent.com/7Ir6j25ROnsXhtQXveOzup33cizxLf-TiifSC1cI6op0bQVB-WePmPjJOfXUBQ0L3KpkheObAiS28e-TS8hZtDzxOIc) + +Ritual is a mobile pick up app, connecting restaurants with customers to offer a simple, time-saving tool to get the food and beverages they want, without the wait. Ritual uses Bazel for their backend services. ### [Snap](https://www.snap.com/en-US/) -Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel -in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more -details about their process, see their [engineering blog](https://eng.snap.com/blog/). +Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more details about their process, see their [engineering blog](https://eng.snap.com/blog/). ### [Stripe](https://stripe.com) - + +![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Stripe_Logo%2C_revised_2016.svg/320px-Stripe_Logo%2C_revised_2016.svg.png) Stripe provides mobile payment solutions. They use Bazel in their build and test pipelines, as detailed in their [engineering blog](https://stripe.com/blog/fast-secure-builds-choose-two). ### [Tinder](https://tinder.com) - -Tinder migrated its iOS app from CocoaPods to Bazel -in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). +![](https://policies.tinder.com/static/b0327365f4c0a31c4337157c10e9fadf/c1b63/tinder_full_color_watermark.png) + +Tinder migrated its iOS app from CocoaPods to Bazel in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). ### [Tink](https://tink.com/) - -Tink is a european fintech, building the best way to connect to banks across -Europe. +![](https://cdn.tink.se/tink-logos/LOW/Tink_Black.png) -They are using Bazel to build their backend services from a polyglot monorepo. -Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) -meetup group. +Tink is a european fintech, building the best way to connect to banks across Europe. + +They are using Bazel to build their backend services from a polyglot monorepo. Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) meetup group. ### [Tokopedia](https://www.tokopedia.com/) -Tokopedia is an Indonesian technology company specializing in e-commerce, with -over 90 million monthly active users and over 7 million merchants on the -platform. +Tokopedia is an Indonesian technology company specializing in e-commerce, with over 90 million monthly active users and over 7 million merchants on the platform. -They wrote the article -[How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), -where they explain how Bazel sped up their builds. The build duration went from -55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote -caching. +They wrote the article [How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), where they explain how Bazel sped up their builds. The build duration went from 55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote caching. ### [Trunk.io](https://trunk.io/merge/trunk-merge-and-bazel) - + +![](/community/images/trunk-logo-dark.svg) Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initialized Capital. Trunk offers a powerful pull request merge service with first-class support for the Bazel build system. By leveraging Bazel's understanding of dependencies within a codebase, Trunk's merge service intelligently creates parallel merge lanes, allowing independent changes to be tested and merged simultaneously. @@ -411,71 +274,49 @@ Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initial ### [Twitter](https://twitter.com/) -Twitter has made the decision to migrate from Pants to Bazel as their primary -build tool -([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). +Twitter has made the decision to migrate from Pants to Bazel as their primary build tool ([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). ### [Two Sigma](https://www.twosigma.com/) - -Two Sigma is a New York-headquartered technology company dedicated to finding -value in the world’s data. +![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Two_Sigma_logo.svg/2880px-Two_Sigma_logo.svg.png) + +Two Sigma is a New York-headquartered technology company dedicated to finding value in the world’s data. ### [TypeDB](https://typedb.com) -TypeDB Logo -TypeDB is a database technology that can be used to intuitively model -interconnected data. Through its type-theoretic and polymorphic query language, -TypeQL, the data can be accessed with simple, human-readable queries that run at -lightspeed. +![TypeDB Logo](/community/images/typedb.png) + +TypeDB is a database technology that can be used to intuitively model interconnected data. Through its type-theoretic and polymorphic query language, TypeQL, the data can be accessed with simple, human-readable queries that run at lightspeed. -Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution -pipeline that manages many repositories in a wide variety of languages, and -deploys to numerous platforms seamlessly. The TypeDB team has also released -Bazel rules for assembling and deploying software distributions. +Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution pipeline that manages many repositories in a wide variety of languages, and deploys to numerous platforms seamlessly. The TypeDB team has also released Bazel rules for assembling and deploying software distributions. ### [Uber](https://www.uber.com) -Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo -is likely one of the largest Go repositories using Bazel. See the article -[Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) -to learn more about their experience. +Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo is likely one of the largest Go repositories using Bazel. See the article [Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) to learn more about their experience. ### [Uber Advanced Technologies Group](https://www.uber.com/info/atg/) - -Uber Advanced Technologies Group is focused on autonomous vehicle efforts at -Uber, including trucking/freight and autonomous ride sharing. The organization -uses Bazel as its primary build system. +![](https://upload.wikimedia.org/wikipedia/commons/thumb/6/62/Uber_logo.svg/220px-Uber_logo.svg.png) + +Uber Advanced Technologies Group is focused on autonomous vehicle efforts at Uber, including trucking/freight and autonomous ride sharing. The organization uses Bazel as its primary build system. ### [Vistar Media](http://vistarmedia.com) -Vistar Media is an advertising platform that enables brands to reach consumers -based on their behavior in the physical world. Their engineering team is -primarily based out of Philadelphia and is using Bazel for builds, deploys, to -speed up testing, and to consolidate repositories written with a variety of -different technologies. + +Vistar Media is an advertising platform that enables brands to reach consumers based on their behavior in the physical world. Their engineering team is primarily based out of Philadelphia and is using Bazel for builds, deploys, to speed up testing, and to consolidate repositories written with a variety of different technologies. ### [VMware](https://www.vmware.com/) -VMware uses Bazel to produce deterministic, reliable builds while developing -innovative products for their customers. + +VMware uses Bazel to produce deterministic, reliable builds while developing innovative products for their customers. ### [Wix](https://www.wix.com/) -Wix is a cloud-based web development platform. Their backend uses Java and Scala -code. They use remote execution with Google Cloud Build. +Wix is a cloud-based web development platform. Their backend uses Java and Scala code. They use remote execution with Google Cloud Build. -> We have seen about 5 times faster clean builds when running with bazel remote -execution which utilizes bazel’s great build/test parallelism capabilities when -it dispatches build/test actions to a worker farm. Average build times are more -than 10 times faster due to the utilization of bazel’s aggressive caching -mechanism. -([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) +> We have seen about 5 times faster clean builds when running with bazel remote execution which utilizes bazel’s great build/test parallelism capabilities when it dispatches build/test actions to a worker farm. Average build times are more than 10 times faster due to the utilization of bazel’s aggressive caching mechanism. ([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) ### [Zenly](https://zen.ly/) -Zenly is a live map of your friends and family. It’s the most fun way to meet up -— or just see what’s up! — so you can feel together, even when you're apart. - +Zenly is a live map of your friends and family. It’s the most fun way to meet up — or just see what’s up! — so you can feel together, even when you're apart. *** @@ -483,44 +324,33 @@ Zenly is a live map of your friends and family. It’s the most fun way to meet ### [Abseil](https://abseil.io/) -Abseil is an open-source collection of C++ code (compliant to C++11) designed -to augment the C++ standard library. +Abseil is an open-source collection of C++ code (compliant to C++11) designed to augment the C++ standard library. ### [Angular](https://angular.io) - +![](https://upload.wikimedia.org/wikipedia/commons/c/cf/Angular_full_color_logo.svg) -Angular is a popular web framework. -Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). +Angular is a popular web framework. Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). ### [Apollo](https://github.com/ApolloAuto/apollo) -Apollo is a high performance, flexible architecture which accelerates the -development, testing, and deployment of Autonomous Vehicles. +Apollo is a high performance, flexible architecture which accelerates the development, testing, and deployment of Autonomous Vehicles. ### [brpc](https://github.com/brpc/brpc) -An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ -instances(not counting clients) and thousands kinds of services, called -"baidu-rpc" inside Baidu. +An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ instances(not counting clients) and thousands kinds of services, called "baidu-rpc" inside Baidu. ### [cert-manager](https://github.com/jetstack/cert-manager) -cert-manager is a Kubernetes add-on to automate the management and issuance of -TLS certificates from various issuing sources. It will ensure certificates are -valid and up to date periodically, and attempt to renew certificates at an -appropriate time before expiry. +cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry. ### [CallBuilder](https://github.com/google/CallBuilder) -A Java code generator that allows you to create a builder by writing one -function. +A Java code generator that allows you to create a builder by writing one function. ### [CPPItertools](https://github.com/ryanhaining/cppitertools) -C++ library providing range-based for loop add-ons inspired by the Python -builtins and itertools library. Like itertools and the Python3 builtins, this -library uses lazy evaluation wherever possible. +C++ library providing range-based for loop add-ons inspired by the Python builtins and itertools library. Like itertools and the Python3 builtins, this library uses lazy evaluation wherever possible. ### [Copybara](https://github.com/google/copybara) @@ -528,13 +358,11 @@ Copybara is a tool for transforming and moving code between repositories. ### [Dagger](https://google.github.io/dagger/) -Dagger is a fully static, compile-time dependency injection framework for both -Java and Android. +Dagger is a fully static, compile-time dependency injection framework for both Java and Android. ### [DAML](https://github.com/digital-asset/daml) -DAML is a smart contract language for building future-proof distributed -applications on a safe, privacy-aware runtime. +DAML is a smart contract language for building future-proof distributed applications on a safe, privacy-aware runtime. ### [DeepMind Lab](https://github.com/deepmind/lab) @@ -542,10 +370,7 @@ A customisable 3D platform for agent-based AI research. ### [Drake](https://github.com/RobotLocomotion/drake) -Drake is a C++ toolbox started at MIT and now led by the Toyota Research -Institute. It is a collection of tools for analyzing the dynamics of our robots -and building control systems for them, with a heavy emphasis on -optimization-based design/analysis. +Drake is a C++ toolbox started at MIT and now led by the Toyota Research Institute. It is a collection of tools for analyzing the dynamics of our robots and building control systems for them, with a heavy emphasis on optimization-based design/analysis. ### [Envoy](https://github.com/lyft/envoy) @@ -553,19 +378,15 @@ C++ L7 proxy and communication bus ### [Error Prone](https://github.com/google/error-prone) -Catches common Java mistakes as compile-time errors. (Migration to Bazel is in -progress.) +Catches common Java mistakes as compile-time errors. (Migration to Bazel is in progress.) ### [Extensible Service Proxy](https://github.com/cloudendpoints/esp) -Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management -capabilities for JSON/REST or gRPC API services. The current implementation is -based on an NGINX HTTP reverse proxy server. +Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management capabilities for JSON/REST or gRPC API services. The current implementation is based on an NGINX HTTP reverse proxy server. ### [FFruit](https://gitlab.com/perezd/ffruit/) -FFruit is a free & open source Android application to the popular service -[Falling Fruit](https://fallingfruit.org). +FFruit is a free & open source Android application to the popular service [Falling Fruit](https://fallingfruit.org). ### [Gerrit Code Review](https://gerritcodereview.com) @@ -577,61 +398,51 @@ Gitiles is a simple repository browser for Git repositories, built on JGit. ### [Grakn](https://github.com/graknlabs/grakn) -Grakn (https://grakn.ai/) is the knowledge graph engine to organise complex -networks of data and make it queryable. +Grakn ([https://grakn.ai/](https://grakn.ai/)) is the knowledge graph engine to organise complex networks of data and make it queryable. ### [GRPC](http://www.grpc.io) -A language-and-platform-neutral remote procedure call system. -(Bazel is a supported, although not primary, build system.) + +A language-and-platform-neutral remote procedure call system. (Bazel is a supported, although not primary, build system.) ### [gVisor](https://github.com/google/gvisor) + gVisor is a container runtime sandbox. ### [Guetzli](https://github.com/google/guetzli/) -Guetzli is a JPEG encoder that aims for excellent compression density at high -visual quality. +Guetzli is a JPEG encoder that aims for excellent compression density at high visual quality. ### [Gulava](http://www.github.com/google/gulava/) -A Java code generator that lets you write Prolog-style predicates and use them -seamlessly from normal Java code. +A Java code generator that lets you write Prolog-style predicates and use them seamlessly from normal Java code. ### [Heron](https://github.com/apache/incubator-heron) -Heron is a realtime, distributed, fault-tolerant stream processing engine from -Twitter. +Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter. ### [Internet Computer Protocol](https://internetcomputer.org/) - +![](https://internetcomputer.org/img/IC_logo_horizontal_white.svg) -The Internet Computer Protocol is a publicly available blockchain network that -enables replicated execution of general-purpose computation, serving hundreds -of thousands of applications and their users. +The Internet Computer Protocol is a publicly available blockchain network that enables replicated execution of general-purpose computation, serving hundreds of thousands of applications and their users. ### [Jazzer](https://github.com/CodeIntelligenceTesting/jazzer) - +![](https://www.code-intelligence.com/hubfs/Logos/CI%20Logos/Jazzer_einfach.png) Jazzer is a fuzzer for Java and other JVM-based languages that integrates with JUnit 5. ### [JGit](https://eclipse.org/jgit/) -JGit is a lightweight, pure Java library implementing the Git version control -system. +JGit is a lightweight, pure Java library implementing the Git version control system. ### [Jsonnet](https://jsonnet.org/) -An elegant, formally-specified config generation language for JSON. -(Bazel is a supported build system.) +An elegant, formally-specified config generation language for JSON. (Bazel is a supported build system.) ### [Kubernetes](https://github.com/kubernetes/kubernetes) - -Kubernetes is an open source system for managing containerized applications -across multiple hosts, providing basic mechanisms for deployment, maintenance, -and scaling of applications. +![](https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.png) Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. ### [Kythe](https://github.com/google/kythe) @@ -639,10 +450,9 @@ An ecosystem for building tools that work with code. ### [ls-lint](https://github.com/loeffel-io/ls-lint) - +![](https://raw.githubusercontent.com/loeffel-io/ls-lint/master/assets/logo/ls-lint.png) -An extremely fast directory and filename linter - Bring some structure to your -project file system. +An extremely fast directory and filename linter - Bring some structure to your project file system. ### [Nomulus](https://github.com/google/nomulus) @@ -650,19 +460,11 @@ Top-level domain name registry service on Google App Engine. ### [ONOS : Open Network Operating System](https://github.com/opennetworkinglab/onos) - -ONOS is the only SDN controller platform that supports the transition from -legacy “brown field” networks to SDN “green field” networks. This enables -exciting new capabilities, and disruptive deployment and operational cost points -for network operators. +![](https://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Logo_for_the_ONOS_open_source_project.png/175px-Logo_for_the_ONOS_open_source_project.png) ONOS is the only SDN controller platform that supports the transition from legacy “brown field” networks to SDN “green field” networks. This enables exciting new capabilities, and disruptive deployment and operational cost points for network operators. ### [PetitParser for Java](https://github.com/petitparser/java-petitparser) -Grammars for programming languages are traditionally specified statically. -They are hard to compose and reuse due to ambiguities that inevitably arise. -PetitParser combines ideas from scannnerless parsing, parser combinators, -parsing expression grammars and packrat parsers to model grammars and parsers -as objects that can be reconfigured dynamically. +Grammars for programming languages are traditionally specified statically. They are hard to compose and reuse due to ambiguities that inevitably arise. PetitParser combines ideas from scannnerless parsing, parser combinators, parsing expression grammars and packrat parsers to model grammars and parsers as objects that can be reconfigured dynamically. ### [PlaidML](https://github.com/plaidml/plaidml) @@ -670,14 +472,11 @@ PlaidML is a framework for making deep learning work everywhere. ### [Project V](https://www.v2ray.com/) - -Project V is a set of tools to help you build your own privacy network over -internet. +![](https://www.v2ray.com/resources/v2ray_1024.png) Project V is a set of tools to help you build your own privacy network over internet. ### [Prysmatic Labs Ethereum 2.0 Implementation](https://github.com/prysmaticlabs/prysm) -Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed -computing platform. +Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed computing platform. ### [Ray](https://github.com/ray-project/ray) @@ -685,8 +484,7 @@ Ray is a flexible, high-performance distributed execution framework. ### [Resty](https://github.com/go-resty/resty) -Resty is a Simple HTTP and REST client library for Go (inspired by Ruby -rest-client). +Resty is a Simple HTTP and REST client library for Go (inspired by Ruby rest-client). ### [Roughtime](https://roughtime.googlesource.com/roughtime) @@ -698,9 +496,7 @@ Selenium is a portable framework for testing web applications. ### [Semantic](https://github.com/github/semantic) -Semantic is a Haskell library and command line tool for parsing, analyzing, and -comparing source code. It is developed by GitHub (and used for example for the -code navigation). +Semantic is a Haskell library and command line tool for parsing, analyzing, and comparing source code. It is developed by GitHub (and used for example for the code navigation). ### [Served](https://github.com/meltwater/served) @@ -708,13 +504,11 @@ Served is a C++ library for building high performance RESTful web servers. ### [Sonnet](https://github.com/deepmind/sonnet) -Sonnet is a library built on top of TensorFlow for building complex neural -networks. +Sonnet is a library built on top of TensorFlow for building complex neural networks. ### [Sorbet](https://github.com/sorbet/sorbet) -Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to -codebases with millions of lines of code and can be adopted incrementally. +Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to codebases with millions of lines of code and can be adopted incrementally. ### [Spotify](https://spotify.com) @@ -722,12 +516,11 @@ Spotify is using Bazel to build their iOS and Android Apps ([source](https://twi ### [Tink](https://github.com/google/tink) -Tink is a multi-language, cross-platform, open source library that provides -cryptographic APIs that are secure, easy to use correctly, and hard(er) to -misuse. +Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. ### [TensorFlow](http://tensorflow.org) - + +![](https://upload.wikimedia.org/wikipedia/commons/a/a4/TensorFlowLogo.png) An open source software library for machine intelligence. @@ -741,10 +534,8 @@ Project Wycheproof tests crypto libraries against known attacks. ### [XIOSim](https://github.com/s-kanev/XIOSim) -XIOSim is a detailed user-mode microarchitectural simulator for the x86 -architecture. +XIOSim is a detailed user-mode microarchitectural simulator for the x86 architecture. ### [ZhihuDailyPurify](https://github.com/izzyleung/ZhihuDailyPurify) -ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese -question-and-answer webs. +ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese question-and-answer webs. diff --git a/concepts/build-ref.mdx b/concepts/build-ref.mdx index e8839d40..33c1e76c 100644 --- a/concepts/build-ref.mdx +++ b/concepts/build-ref.mdx @@ -2,51 +2,27 @@ title: 'Repositories, workspaces, packages, and targets' --- - - -Bazel builds software from source code organized in directory trees called -repositories. A defined set of repositories comprises the workspace. Source -files in repositories are organized in a nested hierarchy of packages, where -each package is a directory that contains a set of related source files and one -`BUILD` file. The `BUILD` file specifies what software outputs can be built from -the source. +Bazel builds software from source code organized in directory trees called repositories. A defined set of repositories comprises the workspace. Source files in repositories are organized in a nested hierarchy of packages, where each package is a directory that contains a set of related source files and one `BUILD` file. The `BUILD` file specifies what software outputs can be built from the source. ### Repositories -Source files used in a Bazel build are organized in _repositories_ (often -shortened to _repos_). A repo is a directory tree with a boundary marker file at -its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or -in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. +Source files used in a Bazel build are organized in *repositories* (often shortened to *repos*). A repo is a directory tree with a boundary marker file at its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. -The repo in which the current Bazel command is being run is called the _main -repo_. Other, (external) repos are defined by _repo rules_; see [external -dependencies overview](/external/overview) for more information. +The repo in which the current Bazel command is being run is called the *main repo*. Other, (external) repos are defined by *repo rules*; see [external dependencies overview](/external/overview) for more information. ## Workspace -A _workspace_ is the environment shared by all Bazel commands run from the same -main repo. It encompasses the main repo and the set of all defined external -repos. +A *workspace* is the environment shared by all Bazel commands run from the same main repo. It encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". ## Packages -The primary unit of code organization in a repository is the _package_. A -package is a collection of related files and a specification of how they can be -used to produce output artifacts. +The primary unit of code organization in a repository is the *package*. A package is a collection of related files and a specification of how they can be used to produce output artifacts. -A package is defined as a directory containing a -[`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A -package includes all files in its directory, plus all subdirectories beneath it, -except those which themselves contain a `BUILD` file. From this definition, no -file or directory may be a part of two different packages. +A package is defined as a directory containing a [`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A package includes all files in its directory, plus all subdirectories beneath it, except those which themselves contain a `BUILD` file. From this definition, no file or directory may be a part of two different packages. -For example, in the following directory tree there are two packages, `my/app`, -and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but -a directory belonging to package `my/app`. +For example, in the following directory tree there are two packages, `my/app`, and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but a directory belonging to package `my/app`. ``` src/my/app/BUILD @@ -58,48 +34,18 @@ src/my/app/tests/test.cc ## Targets -A package is a container of _targets_, which are defined in the package's -`BUILD` file. Most targets are one of two principal kinds, _files_ and _rules_. - -Files are further divided into two kinds. _Source files_ are usually written by -the efforts of people, and checked in to the repository. _Generated files_, -sometimes called derived files or output files, are not checked in, but are -generated from source files. - -The second kind of target is declared with a _rule_. Each rule instance -specifies the relationship between a set of input and a set of output files. The -inputs to a rule may be source files, but they also may be the outputs of other -rules. - -Whether the input to a rule is a source file or a generated file is in most -cases immaterial; what matters is only the contents of that file. This fact -makes it easy to replace a complex source file with a generated file produced by -a rule, such as happens when the burden of manually maintaining a highly -structured file becomes too tiresome, and someone writes a program to derive it. -No change is required to the consumers of that file. Conversely, a generated -file may easily be replaced by a source file with only local changes. - -The inputs to a rule may also include _other rules_. The precise meaning of such -relationships is often quite complex and language- or rule-dependent, but -intuitively it is simple: a C++ library rule A might have another C++ library -rule B for an input. The effect of this dependency is that B's header files are -available to A during compilation, B's symbols are available to A during -linking, and B's runtime data is available to A during execution. - -An invariant of all rules is that the files generated by a rule always belong to -the same package as the rule itself; it is not possible to generate files into -another package. It is not uncommon for a rule's inputs to come from another -package, though. - -Package groups are sets of packages whose purpose is to limit accessibility of -certain rules. Package groups are defined by the `package_group` function. They -have three properties: the list of packages they contain, their name, and other -package groups they include. The only allowed ways to refer to them are from the -`visibility` attribute of rules or from the `default_visibility` attribute of -the `package` function; they do not generate or consume files. For more -information, refer to the [`package_group` -documentation](/reference/be/functions#package_group). - - - Labels - +A package is a container of *targets*, which are defined in the package's `BUILD` file. Most targets are one of two principal kinds, *files* and *rules*. + +Files are further divided into two kinds. *Source files* are usually written by the efforts of people, and checked in to the repository. *Generated files*, sometimes called derived files or output files, are not checked in, but are generated from source files. + +The second kind of target is declared with a *rule*. Each rule instance specifies the relationship between a set of input and a set of output files. The inputs to a rule may be source files, but they also may be the outputs of other rules. + +Whether the input to a rule is a source file or a generated file is in most cases immaterial; what matters is only the contents of that file. This fact makes it easy to replace a complex source file with a generated file produced by a rule, such as happens when the burden of manually maintaining a highly structured file becomes too tiresome, and someone writes a program to derive it. No change is required to the consumers of that file. Conversely, a generated file may easily be replaced by a source file with only local changes. + +The inputs to a rule may also include *other rules*. The precise meaning of such relationships is often quite complex and language- or rule-dependent, but intuitively it is simple: a C++ library rule A might have another C++ library rule B for an input. The effect of this dependency is that B's header files are available to A during compilation, B's symbols are available to A during linking, and B's runtime data is available to A during execution. + +An invariant of all rules is that the files generated by a rule always belong to the same package as the rule itself; it is not possible to generate files into another package. It is not uncommon for a rule's inputs to come from another package, though. + +Package groups are sets of packages whose purpose is to limit accessibility of certain rules. Package groups are defined by the `package_group` function. They have three properties: the list of packages they contain, their name, and other package groups they include. The only allowed ways to refer to them are from the `visibility` attribute of rules or from the `default_visibility` attribute of the `package` function; they do not generate or consume files. For more information, refer to the [`package_group` documentation](/reference/be/functions#package_group). + +[Labels→](/concepts/labels) diff --git a/concepts/platforms.mdx b/concepts/platforms.mdx index e560ea4d..e2ecbde8 100644 --- a/concepts/platforms.mdx +++ b/concepts/platforms.mdx @@ -2,30 +2,23 @@ title: 'Migrating to Platforms' --- - - -Bazel has sophisticated [support](#background) for modeling -[platforms][Platforms] and [toolchains][Toolchains] for multi-architecture and -cross-compiled builds. +Bazel has sophisticated [support](#background) for modeling [platforms](/extending/platforms) and [toolchains](/extending/toolchains) for multi-architecture and cross-compiled builds. This page summarizes the state of this support. -Key Point: Bazel's platform and toolchain APIs are available today. Not all -languages support them. Use these APIs with your project if you can. Bazel is -migrating all major languages so eventually all builds will be platform-based. +Key Point: Bazel's platform and toolchain APIs are available today. Not all languages support them. Use these APIs with your project if you can. Bazel is migrating all major languages so eventually all builds will be platform-based. See also: -* [Platforms][Platforms] -* [Toolchains][Toolchains] -* [Background][Background] +- [Platforms](/extending/platforms) +- [Toolchains](/extending/toolchains) +- [Background](#background) ## Status ### C++ -C++ rules use platforms to select toolchains when -`--incompatible_enable_cc_toolchain_resolution` is set. +C++ rules use platforms to select toolchains when `--incompatible_enable_cc_toolchain_resolution` is set. This means you can configure a C++ project with: @@ -41,23 +34,19 @@ bazel build //:my_cpp_project` --cpu=... --crosstool_top=... --compiler=... This will be enabled by default in Bazel 7.0 ([#7260](https://github.com/bazelbuild/bazel/issues/7260)). -To test your C++ project with platforms, see -[Migrating Your Project](#migrating-your-project) and -[Configuring C++ toolchains]. +To test your C++ project with platforms, see [Migrating Your Project](#migrating-your-project) and [Configuring C++ toolchains](/tutorials/ccp-toolchain-config). ### Java Java rules use platforms to select toolchains. -This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, -`--javabase`, and `--host_javabase`. +This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, `--javabase`, and `--host_javabase`. See [Java and Bazel](/docs/bazel-and-java) for details. ### Android -Android rules use platforms to select toolchains when -`--incompatible_enable_android_toolchain_resolution` is set. +Android rules use platforms to select toolchains when `--incompatible_enable_android_toolchain_resolution` is set. This means you can configure an Android project with: @@ -65,63 +54,42 @@ This means you can configure an Android project with: bazel build //:my_android_project --android_platforms=//:my_android_platform ``` -instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, -and `--fat_apk_cpu`. +instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, and `--fat_apk_cpu`. This will be enabled by default in Bazel 7.0 ([#16285](https://github.com/bazelbuild/bazel/issues/16285)). -To test your Android project with platforms, see -[Migrating Your Project](#migrating-your-project). +To test your Android project with platforms, see [Migrating Your Project](#migrating-your-project). ### Apple -[Apple rules] do not support platforms and are not yet scheduled -for support. +[Apple rules](https://github.com/bazelbuild/rules_apple) do not support platforms and are not yet scheduled for support. -You can still use platform APIs with Apple builds (for example, when building -with a mixture of Apple rules and pure C++) with [platform -mappings](#platform-mappings). +You can still use platform APIs with Apple builds (for example, when building with a mixture of Apple rules and pure C++) with [platform mappings](#platform-mappings). ### Other languages -* [Go rules] fully support platforms -* [Rust rules] fully support platforms. +- [Go rules](https://github.com/bazelbuild/rules_go) fully support platforms +- [Rust rules](https://github.com/bazelbuild/rules_rust) fully support platforms. -If you own a language rule set, see [Migrating your rule set] for adding -support. +If you own a language rule set, see [Migrating your rule set](#migrating-your-rule-set) for adding support. ## Background -*Platforms* and *toolchains* were introduced to standardize how software -projects target different architectures and cross-compile. +*Platforms* and *toolchains* were introduced to standardize how software projects target different architectures and cross-compile. -This was -[inspired][Inspiration] -by the observation that language maintainers were already doing this in ad -hoc, incompatible ways. For example, C++ rules used `--cpu` and - `--crosstool_top` to declare a target CPU and toolchain. Neither of these -correctly models a "platform". This produced awkward and incorrect builds. +This was [inspired](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) by the observation that language maintainers were already doing this in ad hoc, incompatible ways. For example, C++ rules used `--cpu` and `--crosstool_top` to declare a target CPU and toolchain. Neither of these correctly models a "platform". This produced awkward and incorrect builds. -Java, Android, and other languages evolved their own flags for similar purposes, -none of which interoperated with each other. This made cross-language builds -confusing and complicated. +Java, Android, and other languages evolved their own flags for similar purposes, none of which interoperated with each other. This made cross-language builds confusing and complicated. -Bazel is intended for large, multi-language, multi-platform projects. This -demands more principled support for these concepts, including a clear -standard API. +Bazel is intended for large, multi-language, multi-platform projects. This demands more principled support for these concepts, including a clear standard API. ### Need for migration -Upgrading to the new API requires two efforts: releasing the API and upgrading -rule logic to use it. +Upgrading to the new API requires two efforts: releasing the API and upgrading rule logic to use it. -The first is done but the second is ongoing. This consists of ensuring -language-specific platforms and toolchains are defined, language logic reads -toolchains through the new API instead of old flags like `--crosstool_top`, and -`config_setting`s select on the new API instead of old flags. +The first is done but the second is ongoing. This consists of ensuring language-specific platforms and toolchains are defined, language logic reads toolchains through the new API instead of old flags like `--crosstool_top`, and `config_setting`s select on the new API instead of old flags. -This work is straightforward but requires a distinct effort for each language, -plus fair warning for project owners to test against upcoming changes. +This work is straightforward but requires a distinct effort for each language, plus fair warning for project owners to test against upcoming changes. This is why this is an ongoing migration. @@ -136,25 +104,18 @@ bazel build //:myproject --platforms=//:myplatform This implies: 1. Your project's rules choose the right toolchains for `//:myplatform`. -1. Your project's dependencies choose the right toolchains for `//:myplatform`. -1. `//:myplatform` references -[common declarations][Common Platform Declarations] -of `CPU`, `OS`, and other generic, language-independent properties -1. All relevant [`select()`s][select()] properly match `//:myplatform`. -1. `//:myplatform` is defined in a clear, accessible place: in your project's -repo if the platform is unique to your project, or some common place all -consuming projects can find it - -Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be -deprecated and removed as soon as it's safe to do so. +2. Your project's dependencies choose the right toolchains for `//:myplatform`. +3. `//:myplatform` references [common declarations](https://github.com/bazelbuild/platforms) of `CPU`, `OS`, and other generic, language-independent properties +4. All relevant [`select()`s](/docs/configurable-attributes) properly match `//:myplatform`. +5. `//:myplatform` is defined in a clear, accessible place: in your project's repo if the platform is unique to your project, or some common place all consuming projects can find it -Ultimately, this will be the *sole* way to configure architectures. +Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be deprecated and removed as soon as it's safe to do so. +Ultimately, this will be the *sole* way to configure architectures. ## Migrating your project -If you build with languages that support platforms, your build should already -work with an invocation like: +If you build with languages that support platforms, your build should already work with an invocation like: ```posix-terminal bazel build //:myproject --platforms=//:myplatform @@ -162,49 +123,29 @@ bazel build //:myproject --platforms=//:myplatform See [Status](#status) and your language's documentation for precise details. -If a language requires a flag to enable platform support, you also need to set -that flag. See [Status](#status) for details. +If a language requires a flag to enable platform support, you also need to set that flag. See [Status](#status) for details. For your project to build, you need to check the following: -1. `//:myplatform` must exist. It's generally the project owner's responsibility - to define platforms because different projects target different machines. - See [Default platforms](#default-platforms). +1. `//:myplatform` must exist. It's generally the project owner's responsibility to define platforms because different projects target different machines. See [Default platforms](#default-platforms). -1. The toolchains you want to use must exist. If using stock toolchains, the - language owners should include instructions for how to register them. If - writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your - `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). +2. The toolchains you want to use must exist. If using stock toolchains, the language owners should include instructions for how to register them. If writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). -1. `select()`s and [configuration transitions][Starlark transitions] must - resolve properly. See [select()](#select) and [Transitions](#transitions). +3. `select()`s and [configuration transitions](/extending/config#user-defined-transitions) must resolve properly. See [select()](#select) and [Transitions](#transitions). -1. If your build mixes languages that do and don't support platforms, you may - need platform mappings to help the legacy languages work with the new API. - See [Platform mappings](#platform-mappings) for details. +4. If your build mixes languages that do and don't support platforms, you may need platform mappings to help the legacy languages work with the new API. See [Platform mappings](#platform-mappings) for details. If you still have problems, [reach out](#questions) for support. ### Default platforms -Project owners should define explicit -[platforms][Defining Constraints and Platforms] to describe the architectures -they want to build for. These are then triggered with `--platforms`. +Project owners should define explicit [platforms](/extending/platforms#constraints-platforms) to describe the architectures they want to build for. These are then triggered with `--platforms`. -When `--platforms` isn't set, Bazel defaults to a `platform` representing the -local build machine. This is auto-generated at `@platforms//host` (aliased as -`@bazel_tools//tools:host_platform`) -so there's no need to explicitly define it. It maps the local machine's `OS` -and `CPU` with `constraint_value`s declared in -[`@platforms`](https://github.com/bazelbuild/platforms). +When `--platforms` isn't set, Bazel defaults to a `platform` representing the local build machine. This is auto-generated at `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`) so there's no need to explicitly define it. It maps the local machine's `OS` and `CPU` with `constraint_value`s declared in [`@platforms`](https://github.com/bazelbuild/platforms). ### `select()` -Projects can [`select()`][select()] on -[`constraint_value` targets][constraint_value Rule] but not complete -platforms. This is intentional so `select()` supports as wide a variety of -machines as possible. A library with `ARM`-specific sources should support *all* -`ARM`-powered machines unless there's reason to be more specific. +Projects can [`select()`](/docs/configurable-attributes) on [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value) but not complete platforms. This is intentional so `select()` supports as wide a variety of machines as possible. A library with `ARM`-specific sources should support *all* `ARM`-powered machines unless there's reason to be more specific. To select on one or more `constraint_value`s, use: @@ -228,75 +169,47 @@ config_setting( ) ``` -More details [here][select() Platforms]. +More details [here](/docs/configurable-attributes#platforms). -`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. -When migrating your project to platforms, you must either convert them to -`constraint_values` or use [platform mappings](#platform-mappings) to support -both styles during migration. +`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. When migrating your project to platforms, you must either convert them to `constraint_values` or use [platform mappings](#platform-mappings) to support both styles during migration. ### Transitions -[Starlark transitions][Starlark transitions] change -flags down parts of your build graph. If your project uses a transition that -sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read -`--platforms` won't see these changes. +[Starlark transitions](/extending/config#user-defined-transitions) change flags down parts of your build graph. If your project uses a transition that sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read `--platforms` won't see these changes. -When migrating your project to platforms, you must either convert changes like -`return { "//command_line_option:cpu": "arm" }` to `return { -"//command_line_option:platforms": "//:my_arm_platform" }` or use [platform -mappings](#platform-mappings) to support both styles during migration. -window. +When migrating your project to platforms, you must either convert changes like `return { "//command_line_option:cpu": "arm" }` to `return { "//command_line_option:platforms": "//:my_arm_platform" }` or use [platform mappings](#platform-mappings) to support both styles during migration. window. ## Migrating your rule set If you own a rule set and want to support platforms, you need to: -1. Have rule logic resolve toolchains with the toolchain API. See - [toolchain API][Toolchains] (`ctx.toolchains`). +1. Have rule logic resolve toolchains with the toolchain API. See [toolchain API](/extending/toolchains) (`ctx.toolchains`). -1. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so - rule logic alternately resolves toolchains through the new API or old flags - like `--crosstool_top` during migration testing. +2. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so rule logic alternately resolves toolchains through the new API or old flags like `--crosstool_top` during migration testing. -1. Define the relevant properties that make up platform components. See - [Common platform properties](#common-platform-properties) +3. Define the relevant properties that make up platform components. See [Common platform properties](#common-platform-properties) -1. Define standard toolchains and make them accessible to users through your - rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) +4. Define standard toolchains and make them accessible to users through your rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) -1. Ensure [`select()`s](#select) and - [configuration transitions](#transitions) support platforms. This is the - biggest challenge. It's particularly challenging for multi-language projects - (which may fail if *all* languages can't read `--platforms`). +5. Ensure [`select()`s](#select) and [configuration transitions](#transitions) support platforms. This is the biggest challenge. It's particularly challenging for multi-language projects (which may fail if *all* languages can't read `--platforms`). -If you need to mix with rules that don't support platforms, you may need -[platform mappings](#platform-mappings) to bridge the gap. +If you need to mix with rules that don't support platforms, you may need [platform mappings](#platform-mappings) to bridge the gap. ### Common platform properties -Common, cross-language platform properties like `OS` and `CPU` should be -declared in [`@platforms`](https://github.com/bazelbuild/platforms). -This encourages sharing, standardization, and cross-language compatibility. +Common, cross-language platform properties like `OS` and `CPU` should be declared in [`@platforms`](https://github.com/bazelbuild/platforms). This encourages sharing, standardization, and cross-language compatibility. -Properties unique to your rules should be declared in your rule's repo. This -lets you maintain clear ownership over the specific concepts your rules are -responsible for. +Properties unique to your rules should be declared in your rule's repo. This lets you maintain clear ownership over the specific concepts your rules are responsible for. -If your rules use custom-purpose OSes or CPUs, these should be declared in your -rule's repo vs. -[`@platforms`](https://github.com/bazelbuild/platforms). +If your rules use custom-purpose OSes or CPUs, these should be declared in your rule's repo vs. [`@platforms`](https://github.com/bazelbuild/platforms). ## Platform mappings -*Platform mappings* is a temporary API that lets platform-aware logic mix with -legacy logic in the same build. This is a blunt tool that's only intended to -smooth incompatibilities with different migration timeframes. +*Platform mappings* is a temporary API that lets platform-aware logic mix with legacy logic in the same build. This is a blunt tool that's only intended to smooth incompatibilities with different migration timeframes. -Caution: Only use this if necessary, and expect to eventually eliminate it. +Caution: Only use this if necessary, and expect to eventually eliminate it. -A platform mapping is a map of either a `platform()` to a -corresponding set of legacy flags or the reverse. For example: +A platform mapping is a map of either a `platform()` to a corresponding set of legacy flags or the reverse. For example: ```python platforms: @@ -317,20 +230,15 @@ flags: //platforms:macos ``` -Bazel uses this to guarantee all settings, both platform-based and -legacy, are consistently applied throughout the build, including through -[transitions](#transitions). +Bazel uses this to guarantee all settings, both platform-based and legacy, are consistently applied throughout the build, including through [transitions](#transitions). -By default Bazel reads mappings from the `platform_mappings` file in your -workspace root. You can also set -`--platform_mappings=//:my_custom_mapping`. +By default Bazel reads mappings from the `platform_mappings` file in your workspace root. You can also set `--platform_mappings=//:my_custom_mapping`. -See the [platform mappings design] for details. +See the [platform mappings design](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit) for details. ## API review -A [`platform`][platform Rule] is a collection of -[`constraint_value` targets][constraint_value Rule]: +A [`platform`](/reference/be/platforms-and-toolchains#platform) is a collection of [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value): ```python platform( @@ -342,9 +250,7 @@ platform( ) ``` -A [`constraint_value`][constraint_value Rule] is a machine -property. Values of the same "kind" are grouped under a common -[`constraint_setting`][constraint_setting Rule]: +A [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) is a machine property. Values of the same "kind" are grouped under a common [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting): ```python constraint_setting(name = "os") @@ -358,72 +264,27 @@ constraint_value( ) ``` -A [`toolchain`][Toolchains] is a [Starlark rule][Starlark rule]. Its -attributes declare a language's tools (like `compiler = -"//mytoolchain:custom_gcc"`). Its [providers][Starlark Provider] pass -this information to rules that need to build with these tools. +A [`toolchain`](/extending/toolchains) is a [Starlark rule](/extending/rules). Its attributes declare a language's tools (like `compiler = "//mytoolchain:custom_gcc"`). Its [providers](/extending/rules#providers) pass this information to rules that need to build with these tools. -Toolchains declare the `constraint_value`s of machines they can -[target][target_compatible_with Attribute] -(`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can -[run on][exec_compatible_with Attribute] -(`exec_compatible_with = ["@platforms//os:mac"]`). +Toolchains declare the `constraint_value`s of machines they can [target](/reference/be/platforms-and-toolchains#toolchain.target_compatible_with) (`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can [run on](/reference/be/platforms-and-toolchains#toolchain.exec_compatible_with) (`exec_compatible_with = ["@platforms//os:mac"]`). -When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel -automatically selects a toolchain that can run on the build machine and -build binaries for `//:myplatform`. This is known as *toolchain resolution*. +When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel automatically selects a toolchain that can run on the build machine and build binaries for `//:myplatform`. This is known as *toolchain resolution*. -The set of available toolchains can be registered in the `MODULE.bazel` file -with [`register_toolchains`][register_toolchains Function] or at the -command line with [`--extra_toolchains`][extra_toolchains Flag]. +The set of available toolchains can be registered in the `MODULE.bazel` file with [`register_toolchains`](/rules/lib/globals/module#register_toolchains) or at the command line with [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). -For more information see [here][Toolchains]. +For more information see [here](/extending/toolchains). ## Questions -For general support and questions about the migration timeline, contact -[bazel-discuss] or the owners of the appropriate rules. +For general support and questions about the migration timeline, contact [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) or the owners of the appropriate rules. -For discussions on the design and evolution of the platform/toolchain APIs, -contact [bazel-dev]. +For discussions on the design and evolution of the platform/toolchain APIs, contact [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev). ## See also -* [Configurable Builds - Part 1] -* [Platforms] -* [Toolchains] -* [Bazel Platforms Cookbook] -* [Platforms examples] -* [Example C++ toolchain] - -[Android Rules]: /docs/bazel-and-android -[Apple Rules]: https://github.com/bazelbuild/rules_apple -[Background]: #background -[Bazel platforms Cookbook]: https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/ -[bazel-dev]: https://groups.google.com/forum/#!forum/bazel-dev -[bazel-discuss]: https://groups.google.com/forum/#!forum/bazel-discuss -[Common Platform Declarations]: https://github.com/bazelbuild/platforms -[constraint_setting Rule]: /reference/be/platforms-and-toolchains#constraint_setting -[constraint_value Rule]: /reference/be/platforms-and-toolchains#constraint_value -[Configurable Builds - Part 1]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html -[Configuring C++ toolchains]: /tutorials/ccp-toolchain-config -[Defining Constraints and Platforms]: /extending/platforms#constraints-platforms -[Example C++ toolchain]: https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms -[exec_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.exec_compatible_with -[extra_toolchains Flag]: /reference/command-line-reference#flag--extra_toolchains -[Go Rules]: https://github.com/bazelbuild/rules_go -[Inspiration]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html -[Migrating your rule set]: #migrating-your-rule-set -[Platforms]: /extending/platforms -[Platforms examples]: https://github.com/hlopko/bazel_platforms_examples -[platform mappings design]: https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit -[platform Rule]: /reference/be/platforms-and-toolchains#platform -[register_toolchains Function]: /rules/lib/globals/module#register_toolchains -[Rust rules]: https://github.com/bazelbuild/rules_rust -[select()]: /docs/configurable-attributes -[select() Platforms]: /docs/configurable-attributes#platforms -[Starlark provider]: /extending/rules#providers -[Starlark rule]: /extending/rules -[Starlark transitions]: /extending/config#user-defined-transitions -[target_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.target_compatible_with -[Toolchains]: /extending/toolchains +- [Configurable Builds - Part 1](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) +- [Platforms](/extending/platforms) +- [Toolchains](/extending/toolchains) +- [Bazel Platforms Cookbook](https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/) +- [Platforms examples](https://github.com/hlopko/bazel_platforms_examples) +- [Example C++ toolchain](https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms) diff --git a/concepts/visibility.mdx b/concepts/visibility.mdx index 982f0a0e..10fd9eec 100644 --- a/concepts/visibility.mdx +++ b/concepts/visibility.mdx @@ -2,97 +2,49 @@ title: 'Visibility' --- +This page covers Bazel's two visibility systems: [target visibility](#target-visibility) and [load visibility](#load-visibility). - -This page covers Bazel's two visibility systems: -[target visibility](#target-visibility) and [load visibility](#load-visibility). - -Both types of visibility help other developers distinguish between your -library's public API and its implementation details, and help enforce structure -as your workspace grows. You can also use visibility when deprecating a public -API to allow current users while denying new ones. +Both types of visibility help other developers distinguish between your library's public API and its implementation details, and help enforce structure as your workspace grows. You can also use visibility when deprecating a public API to allow current users while denying new ones. ## Target visibility -**Target visibility** controls who may depend on your target — that is, who may -use your target's label inside an attribute such as `deps`. A target will fail -to build during the [analysis](/reference/glossary#analysis-phase) phase if it -violates the visibility of one of its dependencies. +**Target visibility** controls who may depend on your target — that is, who may use your target's label inside an attribute such as `deps`. A target will fail to build during the [analysis](/reference/glossary#analysis-phase) phase if it violates the visibility of one of its dependencies. -Generally, a target `A` is visible to a target `B` if they are in the same -location, or if `A` grants visibility to `B`'s location. In the absence of -[symbolic macros](/extending/macros), the term "location" can be simplified -to just "package"; see [below](#symbolic-macros) for more on symbolic macros. +Generally, a target `A` is visible to a target `B` if they are in the same location, or if `A` grants visibility to `B`'s location. In the absence of [symbolic macros](/extending/macros), the term "location" can be simplified to just "package"; see [below](#symbolic-macros) for more on symbolic macros. -Visibility is specified by listing allowed packages. Allowing a package does not -necessarily mean that its subpackages are also allowed. For more details on -packages and subpackages, see [Concepts and terminology](/concepts/build-ref). +Visibility is specified by listing allowed packages. Allowing a package does not necessarily mean that its subpackages are also allowed. For more details on packages and subpackages, see [Concepts and terminology](/concepts/build-ref). -For prototyping, you can disable target visibility enforcement by setting the -flag `--check_visibility=false`. This shouldn't be done for production usage in -submitted code. +For prototyping, you can disable target visibility enforcement by setting the flag `--check_visibility=false`. This shouldn't be done for production usage in submitted code. -The primary way to control visibility is with a rule's -[`visibility`](/reference/be/common-definitions#common.visibility) attribute. -The following subsections describe the attribute's format, how to apply it to -various kinds of targets, and the interaction between the visibility system and -symbolic macros. +The primary way to control visibility is with a rule's [`visibility`](/reference/be/common-definitions#common.visibility) attribute. The following subsections describe the attribute's format, how to apply it to various kinds of targets, and the interaction between the visibility system and symbolic macros. ### Visibility specifications -All rule targets have a `visibility` attribute that takes a list of labels. Each -label has one of the following forms. With the exception of the last form, these -are just syntactic placeholders that don't correspond to any actual target. +All rule targets have a `visibility` attribute that takes a list of labels. Each label has one of the following forms. With the exception of the last form, these are just syntactic placeholders that don't correspond to any actual target. -* `"//visibility:public"`: Grants access to all packages. +- `"//visibility:public"`: Grants access to all packages. -* `"//visibility:private"`: Does not grant any additional access; only targets - in this location's package can use this target. +- `"//visibility:private"`: Does not grant any additional access; only targets in this location's package can use this target. -* `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its - subpackages). +- `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its subpackages). -* `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its - direct and indirect subpackages. +- `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its direct and indirect subpackages. -* `"//some_pkg:my_package_group"`: Grants access to all of the packages that - are part of the given [`package_group`](/reference/be/functions#package_group). +- `"//some_pkg:my_package_group"`: Grants access to all of the packages that are part of the given [`package_group`](/reference/be/functions#package_group). - * Package groups use a - [different syntax](/reference/be/functions#package_group.packages) for - specifying packages. Within a package group, the forms - `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively - replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, - `"//visibility:public"` and `"//visibility:private"` are just `"public"` - and `"private"`. + - Package groups use a [different syntax](/reference/be/functions#package_group.packages) for specifying packages. Within a package group, the forms `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, `"//visibility:public"` and `"//visibility:private"` are just `"public"` and `"private"`. -For example, if `//some/package:mytarget` has its `visibility` set to -`[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target -that is part of the `//some/package/...` source tree, as well as targets -declared in `//tests/BUILD`, but not by targets defined in -`//tests/integration/BUILD`. +For example, if `//some/package:mytarget` has its `visibility` set to `[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target that is part of the `//some/package/...` source tree, as well as targets declared in `//tests/BUILD`, but not by targets defined in `//tests/integration/BUILD`. -**Best practice:** To make several targets visible to the same set -of packages, use a `package_group` instead of repeating the list in each -target's `visibility` attribute. This increases readability and prevents the -lists from getting out of sync. +**Best practice:** To make several targets visible to the same set of packages, use a `package_group` instead of repeating the list in each target's `visibility` attribute. This increases readability and prevents the lists from getting out of sync. -**Best practice:** When granting visibility to another team's project, prefer -`__subpackages__` over `__pkg__` to avoid needless visibility churn as that -project evolves and adds new subpackages. +**Best practice:** When granting visibility to another team's project, prefer `__subpackages__` over `__pkg__` to avoid needless visibility churn as that project evolves and adds new subpackages. -Note: The `visibility` attribute may not specify non-`package_group` targets. -Doing so triggers a "Label does not refer to a package group" or "Cycle in -dependency graph" error. +Note: The `visibility` attribute may not specify non-`package_group` targets. Doing so triggers a "Label does not refer to a package group" or "Cycle in dependency graph" error. ### Rule target visibility -A rule target's visibility is determined by taking its `visibility` attribute --- or a suitable default if not given -- and appending the location where the -target was declared. For targets not declared in a symbolic macro, if the -package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), -this default is used; for all other packages and for targets declared in a -symbolic macro, the default is just `["//visibility:private"]`. +A rule target's visibility is determined by taking its `visibility` attribute -- or a suitable default if not given -- and appending the location where the target was declared. For targets not declared in a symbolic macro, if the package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), this default is used; for all other packages and for targets declared in a symbolic macro, the default is just `["//visibility:private"]`. ```starlark # //mypkg/BUILD @@ -130,15 +82,11 @@ package_group( ) ``` -**Best practice:** Avoid setting `default_visibility` to public. It may be -convenient for prototyping or in small codebases, but the risk of inadvertently -creating public targets increases as the codebase grows. It's better to be -explicit about which targets are part of a package's public interface. +**Best practice:** Avoid setting `default_visibility` to public. It may be convenient for prototyping or in small codebases, but the risk of inadvertently creating public targets increases as the codebase grows. It's better to be explicit about which targets are part of a package's public interface. ### Generated file target visibility -A generated file target has the same visibility as the rule target that -generates it. +A generated file target has the same visibility as the rule target that generates it. ```starlark # //mypkg/BUILD @@ -168,36 +116,21 @@ some_rule( ### Source file target visibility -Source file targets can either be explicitly declared using -[`exports_files`](/reference/be/functions#exports_files), or implicitly created -by referring to their filename in a label attribute of a rule (outside of a -symbolic macro). As with rule targets, the location of the call to -`exports_files`, or the BUILD file that referred to the input file, is always -automatically appended to the file's visibility. +Source file targets can either be explicitly declared using [`exports_files`](/reference/be/functions#exports_files), or implicitly created by referring to their filename in a label attribute of a rule (outside of a symbolic macro). As with rule targets, the location of the call to `exports_files`, or the BUILD file that referred to the input file, is always automatically appended to the file's visibility. -Files declared by `exports_files` can have their visibility set by the -`visibility` parameter to that function. If this parameter is not given, the visibility is public. +Files declared by `exports_files` can have their visibility set by the `visibility` parameter to that function. If this parameter is not given, the visibility is public. -Note: `exports_files` may not be used to override the visibility of a generated -file. +Note: `exports_files` may not be used to override the visibility of a generated file. -For files that do not appear in a call to `exports_files`, the visibility -depends on the value of the flag -[`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): +For files that do not appear in a call to `exports_files`, the visibility depends on the value of the flag [`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): -* If the flag is true, the visibility is private. +- If the flag is true, the visibility is private. -* Else, the legacy behavior applies: The visibility is the same as the - `BUILD` file's `default_visibility`, or private if a default visibility is - not specified. +- Else, the legacy behavior applies: The visibility is the same as the `BUILD` file's `default_visibility`, or private if a default visibility is not specified. -Avoid relying on the legacy behavior. Always write an `exports_files` -declaration whenever a source file target needs non-private visibility. +Avoid relying on the legacy behavior. Always write an `exports_files` declaration whenever a source file target needs non-private visibility. -**Best practice:** When possible, prefer to expose a rule target rather than a -source file. For example, instead of calling `exports_files` on a `.java` file, -wrap the file in a non-private `java_library` target. Generally, rule targets -should only directly reference source files that live in the same package. +**Best practice:** When possible, prefer to expose a rule target rather than a source file. For example, instead of calling `exports_files` on a `.java` file, wrap the file in a non-private `java_library` target. Generally, rule targets should only directly reference source files that live in the same package. #### Example @@ -218,98 +151,45 @@ cc_binary( ### Config setting visibility -Historically, Bazel has not enforced visibility for -[`config_setting`](/reference/be/general#config_setting) targets that are -referenced in the keys of a [`select()`](/reference/be/functions#select). There -are two flags to remove this legacy behavior: +Historically, Bazel has not enforced visibility for [`config_setting`](/reference/be/general#config_setting) targets that are referenced in the keys of a [`select()`](/reference/be/functions#select). There are two flags to remove this legacy behavior: -* [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) - enables visibility checking for these targets. To assist with migration, it - also causes any `config_setting` that does not specify a `visibility` to be - considered public (regardless of package-level `default_visibility`). +- [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) enables visibility checking for these targets. To assist with migration, it also causes any `config_setting` that does not specify a `visibility` to be considered public (regardless of package-level `default_visibility`). -* [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) - causes `config_setting`s that do not specify a `visibility` to respect the - package's `default_visibility` and to fallback on private visibility, just - like any other rule target. It is a no-op if - `--incompatible_enforce_config_setting_visibility` is not set. +- [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) causes `config_setting`s that do not specify a `visibility` to respect the package's `default_visibility` and to fallback on private visibility, just like any other rule target. It is a no-op if `--incompatible_enforce_config_setting_visibility` is not set. -Avoid relying on the legacy behavior. Any `config_setting` that is intended to -be used outside the current package should have an explicit `visibility`, if the -package does not already specify a suitable `default_visibility`. +Avoid relying on the legacy behavior. Any `config_setting` that is intended to be used outside the current package should have an explicit `visibility`, if the package does not already specify a suitable `default_visibility`. ### Package group target visibility -`package_group` targets do not have a `visibility` attribute. They are always -publicly visible. +`package_group` targets do not have a `visibility` attribute. They are always publicly visible. ### Visibility of implicit dependencies -Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — -dependencies that are not spelled out in a `BUILD` file but are inherent to -every instance of that rule. For example, a `cc_library` rule might create an -implicit dependency from each of its rule targets to an executable target -representing a C++ compiler. +Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — dependencies that are not spelled out in a `BUILD` file but are inherent to every instance of that rule. For example, a `cc_library` rule might create an implicit dependency from each of its rule targets to an executable target representing a C++ compiler. -The visibility of such an implicit dependency is checked with respect to the -package containing the `.bzl` file in which the rule (or aspect) is defined. In -our example, the C++ compiler could be private so long as it lives in the same -package as the definition of the `cc_library` rule. As a fallback, if the -implicit dependency is not visible from the definition, it is checked with -respect to the `cc_library` target. +The visibility of such an implicit dependency is checked with respect to the package containing the `.bzl` file in which the rule (or aspect) is defined. In our example, the C++ compiler could be private so long as it lives in the same package as the definition of the `cc_library` rule. As a fallback, if the implicit dependency is not visible from the definition, it is checked with respect to the `cc_library` target. -If you want to restrict the usage of a rule to certain packages, use -[load visibility](#load-visibility) instead. +If you want to restrict the usage of a rule to certain packages, use [load visibility](#load-visibility) instead. ### Visibility and symbolic macros -This section describes how the visibility system interacts with -[symbolic macros](/extending/macros). +This section describes how the visibility system interacts with [symbolic macros](/extending/macros). #### Locations within symbolic macros -A key detail of the visibility system is how we determine the location of a -declaration. For targets that are not declared in a symbolic macro, the location -is just the package where the target lives -- the package of the `BUILD` file. -But for targets created in a symbolic macro, the location is the package -containing the `.bzl` file where the macro's definition (the -`my_macro = macro(...)` statement) appears. When a target is created inside -multiple nested targets, it is always the innermost symbolic macro's definition -that is used. - -The same system is used to determine what location to check against a given -dependency's visibility. If the consuming target was created inside a macro, we -look at the innermost macro's definition rather than the package the consuming -target lives in. - -This means that all macros whose code is defined in the same package are -automatically "friends" with one another. Any target directly created by a macro -defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, -regardless of what packages the macros are actually instantiated in. Likewise, -they can see, and can be seen by, targets declared directly in `//lib/BUILD` and -its legacy macros. Conversely, targets that live in the same package cannot -necessarily see one another if at least one of them is created by a symbolic -macro. - -Within a symbolic macro's implementation function, the `visibility` parameter -has the effective value of the macro's `visibility` attribute after appending -the location where the macro was called. The standard way for a macro to export -one of its targets to its caller is to forward this value along to the target's -declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit -this attribute won't be visible to the caller of the macro unless the caller -happens to be in the same package as the macro definition. This behavior -composes, in the sense that a chain of nested calls to submacros may each pass -`visibility = visibility`, re-exporting the inner macro's exported targets to -the caller at each level, without exposing any of the macros' implementation -details. +A key detail of the visibility system is how we determine the location of a declaration. For targets that are not declared in a symbolic macro, the location is just the package where the target lives -- the package of the `BUILD` file. But for targets created in a symbolic macro, the location is the package containing the `.bzl` file where the macro's definition (the `my_macro = macro(...)` statement) appears. When a target is created inside multiple nested targets, it is always the innermost symbolic macro's definition that is used. + +The same system is used to determine what location to check against a given dependency's visibility. If the consuming target was created inside a macro, we look at the innermost macro's definition rather than the package the consuming target lives in. + +This means that all macros whose code is defined in the same package are automatically "friends" with one another. Any target directly created by a macro defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, regardless of what packages the macros are actually instantiated in. Likewise, they can see, and can be seen by, targets declared directly in `//lib/BUILD` and its legacy macros. Conversely, targets that live in the same package cannot necessarily see one another if at least one of them is created by a symbolic macro. + +Within a symbolic macro's implementation function, the `visibility` parameter has the effective value of the macro's `visibility` attribute after appending the location where the macro was called. The standard way for a macro to export one of its targets to its caller is to forward this value along to the target's declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit this attribute won't be visible to the caller of the macro unless the caller happens to be in the same package as the macro definition. This behavior composes, in the sense that a chain of nested calls to submacros may each pass `visibility = visibility`, re-exporting the inner macro's exported targets to the caller at each level, without exposing any of the macros' implementation details. #### Delegating privileges to a submacro -The visibility model has a special feature to allow a macro to delegate its -permissions to a submacro. This is important for factoring and composing macros. +The visibility model has a special feature to allow a macro to delegate its permissions to a submacro. This is important for factoring and composing macros. -Suppose you have a macro `my_macro` that creates a dependency edge using a rule -`some_library` from another package: +Suppose you have a macro `my_macro` that creates a dependency edge using a rule `some_library` from another package: ```starlark # //macro/defs.bzl @@ -338,10 +218,7 @@ load("//macro:defs.bzl", "my_macro") my_macro(name = "foo", ...) ``` -The `//pkg:foo_dependency` target has no `visibility` specified, so it is only -visible within `//macro`, which works fine for the consuming target. Now, what -happens if the author of `//lib` refactors `some_library` to instead be -implemented using a macro? +The `//pkg:foo_dependency` target has no `visibility` specified, so it is only visible within `//macro`, which works fine for the consuming target. Now, what happens if the author of `//lib` refactors `some_library` to instead be implemented using a macro? ```starlark # //lib:defs.bzl @@ -357,88 +234,43 @@ def _impl(name, visibility, deps, ...): some_library = macro(implementation = _impl, ...) ``` -With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than -`//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's -visibility. The author of `my_macro` can't be expected to pass -`visibility = ["//lib"]` to the declaration of the dependency just to work -around this implementation detail. +With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than `//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's visibility. The author of `my_macro` can't be expected to pass `visibility = ["//lib"]` to the declaration of the dependency just to work around this implementation detail. -For this reason, when a dependency of a target is also an attribute value of the -macro that declared the target, we check the dependency's visibility against the -location of the macro instead of the location of the consuming target. +For this reason, when a dependency of a target is also an attribute value of the macro that declared the target, we check the dependency's visibility against the location of the macro instead of the location of the consuming target. -In this example, to validate whether `//pkg:foo_consumer` can see -`//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an -input to the call to `some_library` inside of `my_macro`, and instead check the -dependency's visibility against the location of this call, `//macro`. +In this example, to validate whether `//pkg:foo_consumer` can see `//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an input to the call to `some_library` inside of `my_macro`, and instead check the dependency's visibility against the location of this call, `//macro`. -This process can repeat recursively, as long as a target or macro declaration is -inside of another symbolic macro taking the dependency's label in one of its -label-typed attributes. +This process can repeat recursively, as long as a target or macro declaration is inside of another symbolic macro taking the dependency's label in one of its label-typed attributes. -Note: Visibility delegation does not work for labels that were not passed into -the macro, such as labels derived by string manipulation. +Note: Visibility delegation does not work for labels that were not passed into the macro, such as labels derived by string manipulation. #### Finalizers -Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), -in addition to seeing targets following the usual symbolic macro visibility -rules, can *also* see all targets which are visible to the finalizer target's -package. +Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. -In other words, if you migrate a `native.existing_rules()`-based legacy macro to -a finalizer, the targets declared by the finalizer will still be able to see -their old dependencies. +In other words, if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. -It is possible to define targets that a finalizer can introspect using -`native.existing_rules()`, but which it cannot use as dependencies under the -visibility system. For example, if a macro-defined target is not visible to its -own package or to the finalizer macro's definition, and is not delegated to the -finalizer, the finalizer cannot see such a target. Note, however, that a -`native.existing_rules()`-based legacy macro will also be unable to see such a -target. +It is possible to define targets that a finalizer can introspect using `native.existing_rules()`, but which it cannot use as dependencies under the visibility system. For example, if a macro-defined target is not visible to its own package or to the finalizer macro's definition, and is not delegated to the finalizer, the finalizer cannot see such a target. Note, however, that a `native.existing_rules()`-based legacy macro will also be unable to see such a target. ## Load visibility -**Load visibility** controls whether a `.bzl` file may be loaded from other -`BUILD` or `.bzl` files outside the current package. +**Load visibility** controls whether a `.bzl` file may be loaded from other `BUILD` or `.bzl` files outside the current package. -In the same way that target visibility protects source code that is encapsulated -by targets, load visibility protects build logic that is encapsulated by `.bzl` -files. For instance, a `BUILD` file author might wish to factor some repetitive -target declarations into a macro in a `.bzl` file. Without the protection of -load visibility, they might find their macro reused by other collaborators in -the same workspace, so that modifying the macro breaks other teams' builds. +In the same way that target visibility protects source code that is encapsulated by targets, load visibility protects build logic that is encapsulated by `.bzl` files. For instance, a `BUILD` file author might wish to factor some repetitive target declarations into a macro in a `.bzl` file. Without the protection of load visibility, they might find their macro reused by other collaborators in the same workspace, so that modifying the macro breaks other teams' builds. -Note that a `.bzl` file may or may not have a corresponding source file target. -If it does, there is no guarantee that the load visibility and the target -visibility coincide. That is, the same `BUILD` file might be able to load the -`.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), -or vice versa. This can sometimes cause problems for rules that wish to consume -`.bzl` files as source code, such as for documentation generation or testing. +Note that a `.bzl` file may or may not have a corresponding source file target. If it does, there is no guarantee that the load visibility and the target visibility coincide. That is, the same `BUILD` file might be able to load the `.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), or vice versa. This can sometimes cause problems for rules that wish to consume `.bzl` files as source code, such as for documentation generation or testing. -For prototyping, you may disable load visibility enforcement by setting -`--check_bzl_visibility=false`. As with `--check_visibility=false`, this should -not be done for submitted code. +For prototyping, you may disable load visibility enforcement by setting `--check_bzl_visibility=false`. As with `--check_visibility=false`, this should not be done for submitted code. Load visibility is available as of Bazel 6.0. ### Declaring load visibility -To set the load visibility of a `.bzl` file, call the -[`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. -The argument to `visibility()` is a list of package specifications, just like -the [`packages`](/reference/be/functions#package_group.packages) attribute of -`package_group`. However, `visibility()` does not accept negative package -specifications. +To set the load visibility of a `.bzl` file, call the [`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. The argument to `visibility()` is a list of package specifications, just like the [`packages`](/reference/be/functions#package_group.packages) attribute of `package_group`. However, `visibility()` does not accept negative package specifications. -The call to `visibility()` must only occur once per file, at the top level (not -inside a function), and ideally immediately following the `load()` statements. +The call to `visibility()` must only occur once per file, at the top level (not inside a function), and ideally immediately following the `load()` statements. -Unlike target visibility, the default load visibility is always public. Files -that do not call `visibility()` are always loadable from anywhere in the -workspace. It is a good idea to add `visibility("private")` to the top of any -new `.bzl` file that is not specifically intended for use outside the package. +Unlike target visibility, the default load visibility is always public. Files that do not call `visibility()` are always loadable from anywhere in the workspace. It is a good idea to add `visibility("private")` to the top of any new `.bzl` file that is not specifically intended for use outside the package. ### Example @@ -480,8 +312,7 @@ This section describes tips for managing load visibility declarations. #### Factoring visibilities -When multiple `.bzl` files should have the same visibility, it can be helpful to -factor their package specifications into a common list. For example: +When multiple `.bzl` files should have the same visibility, it can be helpful to factor their package specifications into a common list. For example: ```starlark # //mylib/internal_defs.bzl @@ -513,18 +344,13 @@ visibility(clients) ... ``` -This helps prevent accidental skew between the various `.bzl` files' -visibilities. It also is more readable when the `clients` list is large. +This helps prevent accidental skew between the various `.bzl` files' visibilities. It also is more readable when the `clients` list is large. #### Composing visibilities -Sometimes a `.bzl` file might need to be visible to an allowlist that is -composed of multiple smaller allowlists. This is analogous to how a -`package_group` can incorporate other `package_group`s via its -[`includes`](/reference/be/functions#package_group.includes) attribute. +Sometimes a `.bzl` file might need to be visible to an allowlist that is composed of multiple smaller allowlists. This is analogous to how a `package_group` can incorporate other `package_group`s via its [`includes`](/reference/be/functions#package_group.includes) attribute. -Suppose you are deprecating a widely used macro. You want it to be visible only -to existing users and to the packages owned by your own team. You might write: +Suppose you are deprecating a widely used macro. You want it to be visible only to existing users and to the packages owned by your own team. You might write: ```starlark # //mylib/macros.bzl @@ -538,12 +364,7 @@ visibility(our_packages + their_remaining_uses) #### Deduplicating with package groups -Unlike target visibility, you cannot define a load visibility in terms of a -`package_group`. If you want to reuse the same allowlist for both target -visibility and load visibility, it's best to move the list of package -specifications into a .bzl file, where both kinds of declarations may refer to -it. Building off the example in [Factoring visibilities](#factoring-visibilities) -above, you might write: +Unlike target visibility, you cannot define a load visibility in terms of a `package_group`. If you want to reuse the same allowlist for both target visibility and load visibility, it's best to move the list of package specifications into a .bzl file, where both kinds of declarations may refer to it. Building off the example in [Factoring visibilities](#factoring-visibilities) above, you might write: ```starlark # //mylib/BUILD @@ -556,17 +377,11 @@ package_group( ) ``` -This only works if the list does not contain any negative package -specifications. +This only works if the list does not contain any negative package specifications. #### Protecting individual symbols -Any Starlark symbol whose name begins with an underscore cannot be loaded from -another file. This makes it easy to create private symbols, but does not allow -you to share these symbols with a limited set of trusted files. On the other -hand, load visibility gives you control over what other packages may see your -`.bzl file`, but does not allow you to prevent any non-underscored symbol from -being loaded. +Any Starlark symbol whose name begins with an underscore cannot be loaded from another file. This makes it easy to create private symbols, but does not allow you to share these symbols with a limited set of trusted files. On the other hand, load visibility gives you control over what other packages may see your `.bzl file`, but does not allow you to prevent any non-underscored symbol from being loaded. Luckily, you can combine these two features to get fine-grained control. @@ -603,8 +418,4 @@ public_util = _public_util #### bzl-visibility Buildifier lint -There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) -that provides a warning if users load a file from a directory named `internal` -or `private`, when the user's file is not itself underneath the parent of that -directory. This lint predates the load visibility feature and is unnecessary in -workspaces where `.bzl` files declare visibilities. +There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) that provides a warning if users load a file from a directory named `internal` or `private`, when the user's file is not itself underneath the parent of that directory. This lint predates the load visibility feature and is unnecessary in workspaces where `.bzl` files declare visibilities. diff --git a/configure/attributes.mdx b/configure/attributes.mdx index 7bc3f41e..df76c4ba 100644 --- a/configure/attributes.mdx +++ b/configure/attributes.mdx @@ -2,15 +2,9 @@ title: 'Configurable Build Attributes' --- +***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. - -**_Configurable attributes_**, commonly known as [`select()`]( -/reference/be/functions#select), is a Bazel feature that lets users toggle the values -of build rule attributes at the command line. - -This can be used, for example, for a multiplatform library that automatically -chooses the appropriate implementation for the architecture, or for a -feature-configurable binary that can be customized at build time. +This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. ## Example @@ -41,59 +35,30 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the -command line. Specifically, `deps` becomes: - - - - - - - - - - - - - - - - - - - - - - -
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
- -`select()` serves as a placeholder for a value that will be chosen based on -*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) -targets. By using `select()` in a configurable attribute, the attribute -effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: + +| | | +| ----------------------------------------------- | ------------------ | +| Command | deps = | +| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | +| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | + +`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either -* They all resolve to the same value. For example, when running on linux x86, this is unambiguous - `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` - is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when -nothing else does. +- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. + +The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, -`resources`, `cmd`, and most other attributes. Only a small number of attributes -are *non-configurable*, and these are clearly annotated. For example, -`config_setting`'s own -[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies -under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of -the machine running Bazel (which, thanks to cross-compilation, may be different -than the CPU the target is built for). This is known as a -[configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). Given @@ -137,30 +102,19 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and -`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s -build parameters, which include `--cpu=arm`. The `tools` attribute changes -`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on -`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a -[`config_setting`](/reference/be/general#config_setting) or -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of -expected command line flag settings. By encapsulating these in a target, it's -easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands -them for all builds in all projects. These are specified with -[`config_setting`](/reference/be/general#config_setting)'s -[`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -173,31 +127,21 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` -is the expected value for that flag. `:meaningful_condition_name` matches if -*every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, -[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because -it only affects how Bazel reports progress to the user. Targets can't use that -flag to construct their results. The exact set of supported flags isn't -documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with -[Starlark build settings][BuildSettings]. Unlike built-in flags, these are -defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s -[`flag_values`](/reference/be/general#config_setting.flag_values) -attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: ```python config_setting( @@ -210,29 +154,17 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) -for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. -[`--define`](/reference/command-line-reference#flag--define) -is an alternative legacy syntax for custom flags (for example -`--define foo=bar`). This can be expressed either in the -[values](/reference/be/general#config_setting.values) attribute -(`values = {"define": "foo=bar"}`) or the -[define_values](/reference/be/general#config_setting.define_values) attribute -(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards -compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The -`config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition -matches. +The built-in condition `//conditions:default` matches when no other condition matches. -Because of the "exactly one match" rule, a configurable attribute with no match -and no default condition emits a `"no matching conditions"` error. This can -protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -258,16 +190,11 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s -[`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides -flexibility, it can also be burdensome to individually set each one every time -you want to build a target. - [Platforms](/extending/platforms) -let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -325,12 +252,9 @@ platform( ) ``` -The platform can be specified on the command line. It activates the -`config_setting`s that contain a subset of the platform's `constraint_values`, -allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, -you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -357,11 +281,9 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to -check against single values. +This saves the need for boilerplate `config_setting`s when you only need to check against single values. -Platforms are still under development. See the -[documentation](/concepts/platforms) for details. +Platforms are still under development. See the [documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -383,12 +305,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: - - Duplicate labels can appear in different paths of the same `select`. - - Duplicate labels can *not* appear within the same path of a `select`. - - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -`select` cannot appear inside another `select`. If you need to nest `selects` -and your attribute takes other targets as values, use an intermediate target: +- Duplicate labels can appear in different paths of the same `select`. +- Duplicate labels can *not* appear within the same path of a `select`. +- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) + +`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -409,8 +331,7 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND -chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). ## OR chaining @@ -429,9 +350,7 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and -maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple -times. +Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. One option is to predefine the value as a BUILD variable: @@ -450,18 +369,13 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary -duplication. +This makes it easier to manage the dependency. But it still causes unnecessary duplication. For more direct support, use one of the following: ### `selects.with_or` -The -[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing conditions directly inside a `select`: +The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -480,18 +394,12 @@ sh_binary( ### `selects.config_setting_group` - -The -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing multiple `config_setting`s: +The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` - ```python config_setting( name = "config1", @@ -515,17 +423,13 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across -different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. -It's an error for multiple conditions to match unless one is an unambiguous -"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the -[Skylib](https://github.com/bazelbuild/bazel-skylib) macro -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -550,13 +454,11 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed -inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to -fails with the error: +By default, when no condition matches, the target the `select()` is attached to fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -566,8 +468,7 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) -attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: ```python cc_library( @@ -590,8 +491,7 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable -attributes. For example, given: +Rule implementations receive the *resolved values* of configurable attributes. For example, given: ```python # myapp/BUILD @@ -611,9 +511,7 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native -rules. But *they cannot directly manipulate them*. For example, there's no way -for a macro to convert +Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert ```python select({"foo": "val"}, ...) @@ -627,32 +525,22 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* -because macros are evaluated in Bazel's [loading phase](/run/build#loading), -which occurs before flag values are known. -This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while -technically feasible, lack a coherent UI. Further design is necessary to change -this. +Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's -[loading phase](/reference/glossary#loading-phase). -This means it doesn't know what command line flags a target uses since those -flags aren't evaluated until later in the build (in the -[analysis phase](/reference/glossary#analysis-phase)). -So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has -all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` + ```python # myapp/BUILD @@ -701,14 +589,9 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for -details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. -The key issue this question usually means is that select() doesn't work in -*macros*. These are different than *rules*. See the -documentation on [rules](/extending/rules) and [macros](/extending/macros) -to understand the difference. -Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: Define a rule and macro: @@ -788,9 +671,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before -Bazel reads the build's command line flags. That means there isn't enough -information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -813,9 +694,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition -[can't evaluate `select()`s](#faq-select-macro), any attempt to do so -usually produces an error: +Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -828,8 +707,7 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly -vigilant with them: +Booleans are a special case that fail silently, so you should be particularly vigilant with them: ```sh $ cat myapp/defs.bzl @@ -851,21 +729,13 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. -So what they're really evaluting is the `select()` object itself. According to -[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design -standards, all objects aside from a very small number of exceptions -automatically return true. +This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before -Bazel knows what the build's command line parameters are. Can they at least read -the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but it isn't yet a Bazel feature. -What you *can* do today is prepare a straight dictionary, then feed it into a -`select()`: +Conceptually this is possible, but it isn't yet a Bazel feature. What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: ```sh $ cat myapp/defs.bzl @@ -877,7 +747,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -909,7 +779,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -917,17 +787,11 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo -rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in -the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't -actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in -the `actual` attribute, to perform this type of run-time determination. This -works correctly, since `alias()` is a BUILD rule, and is evaluated with a -specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. ```sh $ cat WORKSPACE @@ -953,37 +817,31 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target -that depends on either `//:ssl` or `//external:ssl` will see the alternative -located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, -use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo +$ bazel cquery //myapp:foo <desired build flags> //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on -//myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' +$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the -configuration that resolves `//myapp:foo`'s `select()`. You can inspect its -values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -1002,18 +860,13 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the -[cquery docs](/query/cquery) for guidance on using `somepath` to get the right -one. +`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the -same command line flags as the `bazel cquery`. The `config` command relies on -the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform -is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. For example: @@ -1036,15 +889,11 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the -`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the -`:x86_linux_platform` defined here? The author of the `BUILD` file and the user -who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with -these constraints: +Instead, define a `config_setting` that matches **any** platform with these constraints: ```py config_setting( @@ -1065,13 +914,11 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what -platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you -can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -1091,7 +938,4 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and -confuses users when the expected condition does not match. - -[BuildSettings]: /extending/config#user-defined-build-settings +The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. diff --git a/configure/best-practices.mdx b/configure/best-practices.mdx index deecf9dd..90eec9fc 100644 --- a/configure/best-practices.mdx +++ b/configure/best-practices.mdx @@ -2,10 +2,7 @@ title: 'Best Practices' --- - - -This page assumes you are familiar with Bazel and provides guidelines and -advice on structuring your projects to take full advantage of Bazel's features. +This page assumes you are familiar with Bazel and provides guidelines and advice on structuring your projects to take full advantage of Bazel's features. The overall goals are: @@ -14,81 +11,45 @@ The overall goals are: - To make code well-structured and testable. - To create a build configuration that is easy to understand and maintain. -These guidelines are not requirements: few projects will be able to adhere to -all of them. As the man page for lint says, "A special reward will be presented -to the first person to produce a real program that produces no errors with -strict checking." However, incorporating as many of these principles as possible -should make a project more readable, less error-prone, and faster to build. +These guidelines are not requirements: few projects will be able to adhere to all of them. As the man page for lint says, "A special reward will be presented to the first person to produce a real program that produces no errors with strict checking." However, incorporating as many of these principles as possible should make a project more readable, less error-prone, and faster to build. -This page uses the requirement levels described in -[this RFC](https://www.ietf.org/rfc/rfc2119.txt). +This page uses the requirement levels described in [this RFC](https://www.ietf.org/rfc/rfc2119.txt). ## Running builds and tests -A project should always be able to run `bazel build //...` and -`bazel test //...` successfully on its stable branch. Targets that are necessary -but do not build under certain circumstances (such as,require specific build -flags, don't build on a certain platform, require license agreements) should be -tagged as specifically as possible (for example, "`requires-osx`"). This -tagging allows targets to be filtered at a more fine-grained level than the -"manual" tag and allows someone inspecting the `BUILD` file to understand what -a target's restrictions are. +A project should always be able to run `bazel build //...` and `bazel test //...` successfully on its stable branch. Targets that are necessary but do not build under certain circumstances (such as,require specific build flags, don't build on a certain platform, require license agreements) should be tagged as specifically as possible (for example, "`requires-osx`"). This tagging allows targets to be filtered at a more fine-grained level than the "manual" tag and allows someone inspecting the `BUILD` file to understand what a target's restrictions are. ## Third-party dependencies You may declare third-party dependencies: -* Either declare them as remote repositories in the `MODULE.bazel` file. -* Or put them in a directory called `third_party/` under your workspace directory. +- Either declare them as remote repositories in the `MODULE.bazel` file. +- Or put them in a directory called `third_party/` under your workspace directory. ## Depending on binaries -Everything should be built from source whenever possible. Generally this means -that, instead of depending on a library `some-library.so`, you'd create a -`BUILD` file and build `some-library.so` from its sources, then depend on that -target. +Everything should be built from source whenever possible. Generally this means that, instead of depending on a library `some-library.so`, you'd create a `BUILD` file and build `some-library.so` from its sources, then depend on that target. -Always building from source ensures that a build is not using a library that -was built with incompatible flags or a different architecture. There are also -some features like coverage, static analysis, or dynamic analysis that only -work on the source. +Always building from source ensures that a build is not using a library that was built with incompatible flags or a different architecture. There are also some features like coverage, static analysis, or dynamic analysis that only work on the source. ## Versioning -Prefer building all code from head whenever possible. When versions must be -used, avoid including the version in the target name (for example, `//guava`, -not `//guava-20.0`). This naming makes the library easier to update (only one -target needs to be updated). It's also more resilient to diamond dependency -issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, -you could end up with a library that tries to depend on two different versions. -If you created a misleading alias to point both targets to one `guava` library, -then the `BUILD` files are misleading. +Prefer building all code from head whenever possible. When versions must be used, avoid including the version in the target name (for example, `//guava`, not `//guava-20.0`). This naming makes the library easier to update (only one target needs to be updated). It's also more resilient to diamond dependency issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, you could end up with a library that tries to depend on two different versions. If you created a misleading alias to point both targets to one `guava` library, then the `BUILD` files are misleading. ## Using the `.bazelrc` file -For project-specific options, use the configuration file your -`workspace/.bazelrc` (see [bazelrc format](/run/bazelrc)). +For project-specific options, use the configuration file your `<var>workspace</var>/.bazelrc` (see [bazelrc format](/run/bazelrc)). -If you want to support per-user options for your project that you **do not** -want to check into source control, include the line: +If you want to support per-user options for your project that you **do not** want to check into source control, include the line: ``` try-import %workspace%/user.bazelrc ``` -(or any other file name) in your `workspace/.bazelrc` -and add `user.bazelrc` to your `.gitignore`. -The open-source -[bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) +(or any other file name) in your `<var>workspace</var>/.bazelrc` and add `user.bazelrc` to your `.gitignore`. -generates a custom bazelrc file that matches your Bazel version and provides a -preset of recommended flags. +The open-source [bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) generates a custom bazelrc file that matches your Bazel version and provides a preset of recommended flags. ## Packages -Every directory that contains buildable files should be a package. If a `BUILD` -file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's -a sign that a `BUILD` file should be added to that subdirectory. The longer -this structure exists, the more likely circular dependencies will be -inadvertently created, a target's scope will creep, and an increasing number -of reverse dependencies will have to be updated. +Every directory that contains buildable files should be a package. If a `BUILD` file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's a sign that a `BUILD` file should be added to that subdirectory. The longer this structure exists, the more likely circular dependencies will be inadvertently created, a target's scope will creep, and an increasing number of reverse dependencies will have to be updated. diff --git a/configure/coverage.mdx b/configure/coverage.mdx index 03a8ab9b..72cde9c5 100644 --- a/configure/coverage.mdx +++ b/configure/coverage.mdx @@ -2,129 +2,67 @@ title: 'Code coverage with Bazel' --- +Bazel features a `coverage` sub-command to produce code coverage reports on repositories that can be tested with `bazel coverage`. Due to the idiosyncrasies of the various language ecosystems, it is not always trivial to make this work for a given project. +This page documents the general process for creating and viewing coverage reports, and also features some language-specific notes for languages whose configuration is well-known. It is best read by first reading [the general section](#creating-a-coverage-report), and then reading about the requirements for a specific language. Note also the [remote execution section](#remote-execution), which requires some additional considerations. -Bazel features a `coverage` sub-command to produce code coverage -reports on repositories that can be tested with `bazel coverage`. Due -to the idiosyncrasies of the various language ecosystems, it is not -always trivial to make this work for a given project. - -This page documents the general process for creating and viewing -coverage reports, and also features some language-specific notes for -languages whose configuration is well-known. It is best read by first -reading [the general section](#creating-a-coverage-report), and then -reading about the requirements for a specific language. Note also the -[remote execution section](#remote-execution), which requires some -additional considerations. - -While a lot of customization is possible, this document focuses on -producing and consuming [`lcov`][lcov] reports, which is currently the -most well-supported route. +While a lot of customization is possible, this document focuses on producing and consuming [`lcov`](https://github.com/linux-test-project/lcov) reports, which is currently the most well-supported route. ## Creating a coverage report ### Preparation -The basic workflow for creating coverage reports requires the -following: +The basic workflow for creating coverage reports requires the following: - A basic repository with test targets - A toolchain with the language-specific code coverage tools installed - A correct "instrumentation" configuration -The former two are language-specific and mostly straightforward, -however the latter can be more difficult for complex projects. +The former two are language-specific and mostly straightforward, however the latter can be more difficult for complex projects. -"Instrumentation" in this case refers to the coverage tools that are -used for a specific target. Bazel allows turning this on for a -specific subset of files using the -[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) -flag, which specifies a filter for targets that are tested with the -instrumentation enabled. To enable instrumentation for tests, the -[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) -flag is required. +"Instrumentation" in this case refers to the coverage tools that are used for a specific target. Bazel allows turning this on for a specific subset of files using the [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) flag, which specifies a filter for targets that are tested with the instrumentation enabled. To enable instrumentation for tests, the [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) flag is required. -By default, bazel tries to match the target package(s), and prints the -relevant filter as an `INFO` message. +By default, bazel tries to match the target package(s), and prints the relevant filter as an `INFO` message. ### Running coverage -To produce a coverage report, use [`bazel coverage ---combined_report=lcov -[target]`](/reference/command-line-reference#coverage). This runs the -tests for the target, generating coverage reports in the lcov format -for each file. +To produce a coverage report, use [`bazel coverage --combined_report=lcov [target]`](/reference/command-line-reference#coverage). This runs the tests for the target, generating coverage reports in the lcov format for each file. -Once finished, bazel runs an action that collects all the produced -coverage files, and merges them into one, which is then finally -created under `$(bazel info -output_path)/_coverage/_coverage_report.dat`. +Once finished, bazel runs an action that collects all the produced coverage files, and merges them into one, which is then finally created under `$(bazel info output_path)/_coverage/_coverage_report.dat`. -Coverage reports are also produced if tests fail, though note that -this does not extend to the failed tests - only passing tests are -reported. +Coverage reports are also produced if tests fail, though note that this does not extend to the failed tests - only passing tests are reported. ### Viewing coverage -The coverage report is only output in the non-human-readable `lcov` -format. From this, we can use the `genhtml` utility (part of [the lcov -project][lcov]) to produce a report that can be viewed in a web -browser: +The coverage report is only output in the non-human-readable `lcov` format. From this, we can use the `genhtml` utility (part of [the lcov project](https://github.com/linux-test-project/lcov)) to produce a report that can be viewed in a web browser: ```console genhtml --branch-coverage --output genhtml "$(bazel info output_path)/_coverage/_coverage_report.dat" ``` -Note that `genhtml` reads the source code as well, to annotate missing -coverage in these files. For this to work, it is expected that -`genhtml` is executed in the root of the bazel project. +Note that `genhtml` reads the source code as well, to annotate missing coverage in these files. For this to work, it is expected that `genhtml` is executed in the root of the bazel project. -To view the result, simply open the `index.html` file produced in the -`genhtml` directory in any web browser. +To view the result, simply open the `index.html` file produced in the `genhtml` directory in any web browser. -For further help and information around the `genhtml` tool, or the -`lcov` coverage format, see [the lcov project][lcov]. +For further help and information around the `genhtml` tool, or the `lcov` coverage format, see [the lcov project](https://github.com/linux-test-project/lcov). ## Remote execution Running with remote test execution currently has a few caveats: -- The report combination action cannot yet run remotely. This is - because Bazel does not consider the coverage output files as part of - its graph (see [this issue][remote_report_issue]), and can therefore - not correctly treat them as inputs to the combination action. To - work around this, use `--strategy=CoverageReport=local`. - - Note: It may be necessary to specify something like - `--strategy=CoverageReport=local,remote` instead, if Bazel is set - up to try `local,remote`, due to how Bazel resolves strategies. -- `--remote_download_minimal` and similar flags can also not be used - as a consequence of the former. -- Bazel will currently fail to create coverage information if tests - have been cached previously. To work around this, - `--nocache_test_results` can be set specifically for coverage runs, - although this of course incurs a heavy cost in terms of test times. -- `--experimental_split_coverage_postprocessing` and - `--experimental_fetch_all_coverage_outputs` - - Usually coverage is run as part of the test action, and so by - default, we don't get all coverage back as outputs of the remote - execution by default. These flags override the default and obtain - the coverage data. See [this issue][split_coverage_issue] for more - details. +- The report combination action cannot yet run remotely. This is because Bazel does not consider the coverage output files as part of its graph (see [this issue](https://github.com/bazelbuild/bazel/issues/4685)), and can therefore not correctly treat them as inputs to the combination action. To work around this, use `--strategy=CoverageReport=local`. + - Note: It may be necessary to specify something like `--strategy=CoverageReport=local,remote` instead, if Bazel is set up to try `local,remote`, due to how Bazel resolves strategies. +- `--remote_download_minimal` and similar flags can also not be used as a consequence of the former. +- Bazel will currently fail to create coverage information if tests have been cached previously. To work around this, `--nocache_test_results` can be set specifically for coverage runs, although this of course incurs a heavy cost in terms of test times. +- `--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs` + - Usually coverage is run as part of the test action, and so by default, we don't get all coverage back as outputs of the remote execution by default. These flags override the default and obtain the coverage data. See [this issue](https://github.com/bazelbuild/bazel/issues/4685) for more details. ## Language-specific configuration ### Java -Java should work out-of-the-box with the default configuration. The -[bazel toolchains][bazel_toolchains] contain everything necessary for -remote execution, as well, including JUnit. +Java should work out-of-the-box with the default configuration. The [bazel toolchains](https://github.com/bazelbuild/bazel-toolchains) contain everything necessary for remote execution, as well, including JUnit. ### Python -See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) -for additional steps needed to enable coverage support in Python. - -[lcov]: https://github.com/linux-test-project/lcov -[bazel_toolchains]: https://github.com/bazelbuild/bazel-toolchains -[remote_report_issue]: https://github.com/bazelbuild/bazel/issues/4685 -[split_coverage_issue]: https://github.com/bazelbuild/bazel/issues/4685 +See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) for additional steps needed to enable coverage support in Python. diff --git a/configure/windows.mdx b/configure/windows.mdx index 2dbb90ee..566c425d 100644 --- a/configure/windows.mdx +++ b/configure/windows.mdx @@ -2,24 +2,17 @@ title: 'Using Bazel on Windows' --- - - -This page covers Best Practices for using Bazel on Windows. For installation -instructions, see [Install Bazel on Windows](/install/windows). +This page covers Best Practices for using Bazel on Windows. For installation instructions, see [Install Bazel on Windows](/install/windows). ## Known issues -Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. -[GitHub-Windows]. - -[GitHub-Windows]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows +Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. [GitHub-Windows](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows). ## Best practices ### Avoid long path issues -Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. -To avoid hitting this issue, you can specify a short output directory for Bazel by the [\-\-output_user_root](/reference/command-line-reference#flag--output_user_root) flag. +Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. To avoid hitting this issue, you can specify a short output directory for Bazel by the [--output\_user\_root](/reference/command-line-reference#flag--output_user_root) flag. For example, add the following line to your bazelrc file: @@ -29,14 +22,10 @@ startup --output_user_root=C:/tmp ### Enable symlink support -Some features require Bazel to be able to create file symlinks on Windows, -either by enabling -[Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) -(on Windows 10 version 1703 or newer), or by running Bazel as an administrator. -This enables the following features: +Some features require Bazel to be able to create file symlinks on Windows, either by enabling [Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) (on Windows 10 version 1703 or newer), or by running Bazel as an administrator. This enables the following features: -* [\-\-windows_enable_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) -* [\-\-enable_runfiles](/reference/command-line-reference#flag--enable_runfiles) +- [--windows\_enable\_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) +- [--enable\_runfiles](/reference/command-line-reference#flag--enable_runfiles) To make it easier, add the following lines to your bazelrc file: @@ -48,23 +37,11 @@ build --enable_runfiles **Note**: Creating symlinks on Windows is an expensive operation. The `--enable_runfiles` flag can potentially create a large amount of file symlinks. Only enable this feature when you need it. -{/* TODO(pcloudy): https://github.com/bazelbuild/bazel/issues/6402 - Write a doc about runfiles library and add a link to it here */} - ### Running Bazel: MSYS2 shell vs. command prompt vs. PowerShell -**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from -PowerShell. +**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from PowerShell. -As of 2020-01-15, **do not** run Bazel from `bash` -- either -from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel -may work for most use cases, some things are broken, like -[interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). -Also, if you choose to run under MSYS2, you need to disable MSYS2's -automatic path conversion, otherwise MSYS will convert command line arguments -that _look like_ Unix paths (such as `//foo:bar`) into Windows paths. See -[this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) -for details. +As of 2020-01-15, **do not** run Bazel from `bash` -- either from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel may work for most use cases, some things are broken, like [interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). Also, if you choose to run under MSYS2, you need to disable MSYS2's automatic path conversion, otherwise MSYS will convert command line arguments that *look like* Unix paths (such as `//foo:bar`) into Windows paths. See [this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) for details. ### Using Bazel without Bash (MSYS2) @@ -78,13 +55,7 @@ Starting with Bazel 1.0, you can build any rule without Bash unless it is a: - `sh_binary` or `sh_test` rule, because these inherently need Bash - Starlark rule that uses `ctx.actions.run_shell()` or `ctx.resolve_command()` -However, `genrule` is often used for simple tasks like -[copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) -or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). -Instead of using `genrule` (and depending on Bash) you may find a suitable rule -in the -[bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). -When built on Windows, **these rules do not require Bash**. +However, `genrule` is often used for simple tasks like [copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). Instead of using `genrule` (and depending on Bash) you may find a suitable rule in the [bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). When built on Windows, **these rules do not require Bash**. #### Using bazel test without Bash @@ -104,24 +75,15 @@ Starting with Bazel 1.0, you can run any rule without Bash, except when: - you use `--run_under` or `--script_path` - the test rule itself requires Bash (because its executable is a shell script) -#### Using sh\_binary and sh\_* rules, and ctx.actions.run_shell() without Bash +#### Using sh\_binary and sh\_\* rules, and ctx.actions.run\_shell() without Bash -You need Bash to build and test `sh_*` rules, and to build and test Starlark -rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This -applies not only to rules in your project, but to rules in any of the external -repositories your project depends on (even transitively). +You need Bash to build and test `sh_*` rules, and to build and test Starlark rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This applies not only to rules in your project, but to rules in any of the external repositories your project depends on (even transitively). -In the future, there may be an option to use Windows Subsystem for -Linux (WSL) to build these rules, but currently it is not a priority for -the Bazel-on-Windows subteam. +In the future, there may be an option to use Windows Subsystem for Linux (WSL) to build these rules, but currently it is not a priority for the Bazel-on-Windows subteam. ### Setting environment variables -Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only -set in that command prompt session. If you start a new `cmd.exe`, you need to -set the variables again. To always set the variables when `cmd.exe` starts, you -can add them to the User variables or System variables in the `Control Panel > -System Properties > Advanced > Environment Variables...` dialog box. +Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only set in that command prompt session. If you start a new `cmd.exe`, you need to set the variables again. To always set the variables when `cmd.exe` starts, you can add them to the User variables or System variables in the `Control Panel > System Properties > Advanced > Environment Variables...` dialog box. ## Build on Windows @@ -129,71 +91,54 @@ System Properties > Advanced > Environment Variables...` dialog box. To build C++ targets with MSVC, you need: -* [The Visual C++ compiler](/install/windows#install-vc). +- [The Visual C++ compiler](/install/windows#install-vc). -* (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. +- (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. - Bazel automatically detects the Visual C++ compiler on your system. - To tell Bazel to use a specific VC installation, you can set the - following environment variables: + Bazel automatically detects the Visual C++ compiler on your system. To tell Bazel to use a specific VC installation, you can set the following environment variables: - For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. + For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. - * `BAZEL_VC` the Visual C++ Build Tools installation directory + - `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC + ``` - * `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version - number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools - version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel - will choose the latest version. + - `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel will choose the latest version. - ``` - set BAZEL_VC_FULL_VERSION=14.16.27023 - ``` + ``` + set BAZEL_VC_FULL_VERSION=14.16.27023 + ``` - For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) + For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) - * `BAZEL_VC` the Visual C++ Build Tools installation directory + - `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC + ``` -* The [Windows - SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). +- The [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). - The Windows SDK contains header files and libraries you need when building - Windows applications, including Bazel itself. By default, the latest Windows SDK installed will - be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You - can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 - SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified - Windows SDK installed. + The Windows SDK contains header files and libraries you need when building Windows applications, including Bazel itself. By default, the latest Windows SDK installed will be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified Windows SDK installed. - **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't - support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise - `BAZEL_WINSDK_FULL_VERSION` will be ignored. + **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise `BAZEL_WINSDK_FULL_VERSION` will be ignored. - ``` - set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 - ``` + ``` + set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 + ``` If everything is set up, you can build a C++ target now! -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` -bazel build //examples/cpp:hello-world -bazel-bin\examples\cpp\hello-world.exe +bazel build //examples/cpp:hello-world +bazel-bin\examples\cpp\hello-world.exe ``` -By default, the built binaries target x64 architecture. To build for ARM64 -architecture, use +By default, the built binaries target x64 architecture. To build for ARM64 architecture, use ```none --platforms=//:windows_arm64 --extra_toolchains=@local_config_cc//:cc-toolchain-arm64_windows @@ -207,111 +152,88 @@ cc_configure = use_extension("@rules_cc//cc:extensions.bzl", "cc_configure_exten use_repo(cc_configure, "local_config_cc") ``` -To build and use Dynamically Linked Libraries (DLL files), see [this -example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). +To build and use Dynamically Linked Libraries (DLL files), see [this example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). -**Command Line Length Limit**: To prevent the -[Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), -enable the compiler parameter file feature via `--features=compiler_param_file`. +**Command Line Length Limit**: To prevent the [Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), enable the compiler parameter file feature via `--features=compiler_param_file`. ### Build C++ with Clang From 0.29.0, Bazel supports building with LLVM's MSVC-compatible compiler driver (`clang-cl.exe`). -**Requirement**: To build with Clang, you have to install **both** -[LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, -because although you use `clang-cl.exe` as compiler, you still need to link to -Visual C++ libraries. +**Requirement**: To build with Clang, you have to install **both** [LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, because although you use `clang-cl.exe` as compiler, you still need to link to Visual C++ libraries. -Bazel can automatically detect LLVM installation on your system, or you can explicitly tell -Bazel where LLVM is installed by `BAZEL_LLVM`. +Bazel can automatically detect LLVM installation on your system, or you can explicitly tell Bazel where LLVM is installed by `BAZEL_LLVM`. -* `BAZEL_LLVM` the LLVM installation directory +- `BAZEL_LLVM` the LLVM installation directory - ```posix-terminal - set BAZEL_LLVM=C:\Program Files\LLVM - ``` + ```posix-terminal + set BAZEL_LLVM=C:\Program Files\LLVM + ``` To enable the Clang toolchain for building C++, there are several situations. -* In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the - top level `BUILD` file): +- In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the top level `BUILD` file): - ``` - platform( - name = "x64_windows-clang-cl", - constraint_values = [ - "@platforms//cpu:x86_64", - "@platforms//os:windows", - "@bazel_tools//tools/cpp:clang-cl", - ], - ) - ``` + ``` + platform( + name = "x64_windows-clang-cl", + constraint_values = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@bazel_tools//tools/cpp:clang-cl", + ], + ) + ``` - Then enable the Clang toolchain by specifying the following build flags: + Then enable the Clang toolchain by specifying the following build flags: - ``` - --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl - ``` + ``` + --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl + ``` -* In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by - a build flag `--compiler=clang-cl`. +- In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by a build flag `--compiler=clang-cl`. - If your build sets the flag - [\-\-incompatible_enable_cc_toolchain_resolution] - (https://github.com/bazelbuild/bazel/issues/7260) - to `true`, then use the approach for Bazel 7.0.0. + If your build sets the flag [--incompatible\_enable\_cc\_toolchain\_resolution](https://github.com/bazelbuild/bazel/issues/7260) to `true`, then use the approach for Bazel 7.0.0. -* In Bazel 0.28 and older: Clang is not supported. +- In Bazel 0.28 and older: Clang is not supported. ### Build Java To build Java targets, you need: -* [The Java SE Development Kit](/install/windows#install-jdk) +- [The Java SE Development Kit](/install/windows#install-jdk) On Windows, Bazel builds two output files for `java_binary` rules: -* a `.jar` file -* a `.exe` file that can set up the environment for the JVM and run the binary +- a `.jar` file +- a `.exe` file that can set up the environment for the JVM and run the binary -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world - bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe + bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world + bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe ``` ### Build Python To build Python targets, you need: -* The [Python interpreter](/install/windows#install-python) +- The [Python interpreter](/install/windows#install-python) On Windows, Bazel builds two output files for `py_binary` rules: -* a self-extracting zip file -* an executable file that can launch the Python interpreter with the - self-extracting zip file as the argument +- a self-extracting zip file +- an executable file that can launch the Python interpreter with the self-extracting zip file as the argument -You can either run the executable file (it has a `.exe` extension) or you can run -Python with the self-extracting zip file as the argument. +You can either run the executable file (it has a `.exe` extension) or you can run Python with the self-extracting zip file as the argument. -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/py_native:bin - bazel-bin\examples\py_native\bin.exe - python bazel-bin\examples\py_native\bin.zip + bazel build //examples/py_native:bin + bazel-bin\examples\py_native\bin.exe + python bazel-bin\examples\py_native\bin.zip ``` -If you are interested in details about how Bazel builds Python targets on -Windows, check out this [design -doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). +If you are interested in details about how Bazel builds Python targets on Windows, check out this [design doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). diff --git a/contribute/breaking-changes.mdx b/contribute/breaking-changes.mdx index 5dda1b9d..c30b163a 100644 --- a/contribute/breaking-changes.mdx +++ b/contribute/breaking-changes.mdx @@ -2,65 +2,41 @@ title: 'Guide for rolling out breaking changes' --- - - -It is inevitable that we will make breaking changes to Bazel. We will have to -change our designs and fix the things that do not quite work. However, we need -to make sure that community and Bazel ecosystem can follow along. To that end, -Bazel project has adopted a -[backward compatibility policy](/release/backward-compatibility). -This document describes the process for Bazel contributors to make a breaking -change in Bazel to adhere to this policy. +It is inevitable that we will make breaking changes to Bazel. We will have to change our designs and fix the things that do not quite work. However, we need to make sure that community and Bazel ecosystem can follow along. To that end, Bazel project has adopted a [backward compatibility policy](/release/backward-compatibility). This document describes the process for Bazel contributors to make a breaking change in Bazel to adhere to this policy. 1. Follow the [design document policy](/contribute/design-documents). -1. [File a GitHub issue.](#github-issue) +2. [File a GitHub issue.](#github-issue) -1. [Implement the change.](#implementation) +3. [Implement the change.](#implementation) -1. [Update labels.](#labels) +4. [Update labels.](#labels) -1. [Update repositories.](#update-repos) +5. [Update repositories.](#update-repos) -1. [Flip the incompatible flag.](#flip-flag) +6. [Flip the incompatible flag.](#flip-flag) ## GitHub issue -[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) -in the Bazel repository. -[See example.](https://github.com/bazelbuild/bazel/issues/6611) +[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) in the Bazel repository. [See example.](https://github.com/bazelbuild/bazel/issues/6611) We recommend that: -* The title starts with the name of the flag (the flag name will start with - `incompatible_`). +- The title starts with the name of the flag (the flag name will start with `incompatible_`). -* You add the label - [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). +- You add the label [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). -* The description contains a description of the change and a link to relevant - design documents. +- The description contains a description of the change and a link to relevant design documents. -* The description contains a migration recipe, to explain users how they should - update their code. Ideally, when the change is mechanical, include a link to a - migration tool. +- The description contains a migration recipe, to explain users how they should update their code. Ideally, when the change is mechanical, include a link to a migration tool. -* The description includes an example of the error message users will get if - they don't migrate. This will make the GitHub issue more discoverable from - search engines. Make sure that the error message is helpful and actionable. - When possible, the error message should include the name of the incompatible - flag. +- The description includes an example of the error message users will get if they don't migrate. This will make the GitHub issue more discoverable from search engines. Make sure that the error message is helpful and actionable. When possible, the error message should include the name of the incompatible flag. -For the migration tool, consider contributing to -[Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). -It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. -It may also report warnings. +For the migration tool, consider contributing to [Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. It may also report warnings. ## Implementation -Create a new flag in Bazel. The default value must be false. The help text -should contain the URL of the GitHub issue. As the flag name starts with -`incompatible_`, it needs metadata tags: +Create a new flag in Bazel. The default value must be false. The help text should contain the URL of the GitHub issue. As the flag name starts with `incompatible_`, it needs metadata tags: ```java metadataTags = { @@ -68,80 +44,60 @@ should contain the URL of the GitHub issue. As the flag name starts with }, ``` -In the commit description, add a brief summary of the flag. -Also add [`RELNOTES:`](release-notes.md) in the following form: -`RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` +In the commit description, add a brief summary of the flag. Also add [`RELNOTES:`](release-notes.md) in the following form: `RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` -The commit should also update the relevant documentation, so that there is no -window of commits in which the code is inconsistent with the docs. Since our -documentation is versioned, changes to the docs will not be inadvertently -released prematurely. +The commit should also update the relevant documentation, so that there is no window of commits in which the code is inconsistent with the docs. Since our documentation is versioned, changes to the docs will not be inadvertently released prematurely. ## Labels -Once the commit is merged and the incompatible change is ready to be adopted, add the label -[`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) -to the GitHub issue. +Once the commit is merged and the incompatible change is ready to be adopted, add the label [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) to the GitHub issue. -If a problem is found with the flag and users are not expected to migrate yet: -remove the flags `migration-ready`. +If a problem is found with the flag and users are not expected to migrate yet: remove the flags `migration-ready`. -If you plan to flip the flag in the next major release, add label `breaking-change-X.0" to the issue. +If you plan to flip the flag in the next major release, add label \`breaking-change-X.0" to the issue. ## Updating repositories -Bazel CI tests a list of important projects at -[Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often -dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). -Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). +Bazel CI tests a list of important projects at [Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). Our dev support team monitors the [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) label. Once you add this label to the GitHub issue, they will handle the following: 1. Create a comment in the GitHub issue to track the list of failures and downstream projects that need to be migrated ([see example](https://github.com/bazelbuild/bazel/issues/17032#issuecomment-1353077469)) -1. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) +2. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) -1. Follow up to make sure all issues are addressed before the target release date +3. Follow up to make sure all issues are addressed before the target release date Migrating projects in the downstream pipeline is NOT entirely the responsibility of the incompatible change author, but you can do the following to accelerate the migration and make life easier for both Bazel users and the Bazel Green Team. 1. Send PRs to fix downstream projects. -1. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). +2. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). ## Flipping the flag Before flipping the default value of the flag to true, please make sure that: -* Core repositories in the ecosystem are migrated. +- Core repositories in the ecosystem are migrated. - On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), - the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. + On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. -* All issues in the checklist are marked as fixed/closed. +- All issues in the checklist are marked as fixed/closed. -* User concerns and questions have been resolved. +- User concerns and questions have been resolved. When the flag is ready to flip in Bazel, but blocked on internal migration at Google, please consider setting the flag value to false in the internal `blazerc` file to unblock the flag flip. By doing this, we can ensure Bazel users depend on the new behaviour by default as early as possible. When changing the flag default to true, please: -* Use `RELNOTES[INC]` in the commit description, with the - following format: - `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for - details` - You can include additional information in the rest of the commit description. -* Use `Fixes #xyz` in the description, so that the GitHub issue gets closed - when the commit is merged. -* Review and update documentation if needed. -* File a new issue `#abc` to track the removal of the flag. +- Use `RELNOTES[INC]` in the commit description, with the following format: `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for details` You can include additional information in the rest of the commit description. +- Use `Fixes #xyz` in the description, so that the GitHub issue gets closed when the commit is merged. +- Review and update documentation if needed. +- File a new issue `#abc` to track the removal of the flag. ## Removing the flag -After the flag is flipped at HEAD, it should be removed from Bazel eventually. -When you plan to remove the incompatible flag: +After the flag is flipped at HEAD, it should be removed from Bazel eventually. When you plan to remove the incompatible flag: -* Consider leaving more time for users to migrate if it's a major incompatible change. - Ideally, the flag should be available in at least one major release. -* For the commit that removes the flag, use `Fixes #abc` in the description - so that the GitHub issue gets closed when the commit is merged. +- Consider leaving more time for users to migrate if it's a major incompatible change. Ideally, the flag should be available in at least one major release. +- For the commit that removes the flag, use `Fixes #abc` in the description so that the GitHub issue gets closed when the commit is merged. diff --git a/contribute/codebase.mdx b/contribute/codebase.mdx index 44e0150d..4d12a684 100644 --- a/contribute/codebase.mdx +++ b/contribute/codebase.mdx @@ -2,1652 +2,773 @@ title: 'The Bazel codebase' --- - - -This document is a description of the codebase and how Bazel is structured. It -is intended for people willing to contribute to Bazel, not for end-users. +This document is a description of the codebase and how Bazel is structured. It is intended for people willing to contribute to Bazel, not for end-users. ## Introduction -The codebase of Bazel is large (~350KLOC production code and ~260 KLOC test -code) and no one is familiar with the whole landscape: everyone knows their -particular valley very well, but few know what lies over the hills in every -direction. +The codebase of Bazel is large (\~350KLOC production code and \~260 KLOC test code) and no one is familiar with the whole landscape: everyone knows their particular valley very well, but few know what lies over the hills in every direction. -In order for people midway upon the journey not to find themselves within a -forest dark with the straightforward pathway being lost, this document tries to -give an overview of the codebase so that it's easier to get started with -working on it. +In order for people midway upon the journey not to find themselves within a forest dark with the straightforward pathway being lost, this document tries to give an overview of the codebase so that it's easier to get started with working on it. -The public version of the source code of Bazel lives on GitHub at -[github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not -the "source of truth"; it's derived from a Google-internal source tree that -contains additional functionality that is not useful outside Google. The -long-term goal is to make GitHub the source of truth. +The public version of the source code of Bazel lives on GitHub at [github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not the "source of truth"; it's derived from a Google-internal source tree that contains additional functionality that is not useful outside Google. The long-term goal is to make GitHub the source of truth. -Contributions are accepted through the regular GitHub pull request mechanism, -and manually imported by a Googler into the internal source tree, then -re-exported back out to GitHub. +Contributions are accepted through the regular GitHub pull request mechanism, and manually imported by a Googler into the internal source tree, then re-exported back out to GitHub. ## Client/server architecture -The bulk of Bazel resides in a server process that stays in RAM between builds. -This allows Bazel to maintain state between builds. +The bulk of Bazel resides in a server process that stays in RAM between builds. This allows Bazel to maintain state between builds. -This is why the Bazel command line has two kinds of options: startup and -command. In a command line like this: +This is why the Bazel command line has two kinds of options: startup and command. In a command line like this: ``` bazel --host_jvm_args=-Xmx8G build -c opt //foo:bar ``` -Some options (`--host_jvm_args=`) are before the name of the command to be run -and some are after (`-c opt`); the former kind is called a "startup option" and -affects the server process as a whole, whereas the latter kind, the "command -option", only affects a single command. - -Each server instance has a single associated workspace (collection of source -trees known as "repositories") and each workspace usually has a single active -server instance. This can be circumvented by specifying a custom output base -(see the "Directory layout" section for more information). - -Bazel is distributed as a single ELF executable that is also a valid .zip file. -When you type `bazel`, the above ELF executable implemented in C++ (the -"client") gets control. It sets up an appropriate server process using the -following steps: - -1. Checks whether it has already extracted itself. If not, it does that. This - is where the implementation of the server comes from. -2. Checks whether there is an active server instance that works: it is running, - it has the right startup options and uses the right workspace directory. It - finds the running server by looking at the directory `$OUTPUT_BASE/server` - where there is a lock file with the port the server is listening on. -3. If needed, kills the old server process -4. If needed, starts up a new server process - -After a suitable server process is ready, the command that needs to be run is -communicated to it over a gRPC interface, then the output of Bazel is piped back -to the terminal. Only one command can be running at the same time. This is -implemented using an elaborate locking mechanism with parts in C++ and parts in -Java. There is some infrastructure for running multiple commands in parallel, -since the inability to run `bazel version` in parallel with another command -is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s -and some state in `BlazeRuntime`. - -At the end of a command, the Bazel server transmits the exit code the client -should return. An interesting wrinkle is the implementation of `bazel run`: the -job of this command is to run something Bazel just built, but it can't do that -from the server process because it doesn't have a terminal. So instead it tells -the client what binary it should `exec()` and with what arguments. - -When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC -connection, which tries to terminate the command as soon as possible. After the -third Ctrl-C, the client sends a SIGKILL to the server instead. - -The source code of the client is under `src/main/cpp` and the protocol used to -communicate with the server is in `src/main/protobuf/command_server.proto` . - -The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls -from the client are handled by `GrpcServerImpl.run()`. +Some options (`--host_jvm_args=`) are before the name of the command to be run and some are after (`-c opt`); the former kind is called a "startup option" and affects the server process as a whole, whereas the latter kind, the "command option", only affects a single command. + +Each server instance has a single associated workspace (collection of source trees known as "repositories") and each workspace usually has a single active server instance. This can be circumvented by specifying a custom output base (see the "Directory layout" section for more information). + +Bazel is distributed as a single ELF executable that is also a valid .zip file. When you type `bazel`, the above ELF executable implemented in C++ (the "client") gets control. It sets up an appropriate server process using the following steps: + +1. Checks whether it has already extracted itself. If not, it does that. This is where the implementation of the server comes from. +2. Checks whether there is an active server instance that works: it is running, it has the right startup options and uses the right workspace directory. It finds the running server by looking at the directory `$OUTPUT_BASE/server` where there is a lock file with the port the server is listening on. +3. If needed, kills the old server process +4. If needed, starts up a new server process + +After a suitable server process is ready, the command that needs to be run is communicated to it over a gRPC interface, then the output of Bazel is piped back to the terminal. Only one command can be running at the same time. This is implemented using an elaborate locking mechanism with parts in C++ and parts in Java. There is some infrastructure for running multiple commands in parallel, since the inability to run `bazel version` in parallel with another command is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s and some state in `BlazeRuntime`. + +At the end of a command, the Bazel server transmits the exit code the client should return. An interesting wrinkle is the implementation of `bazel run`: the job of this command is to run something Bazel just built, but it can't do that from the server process because it doesn't have a terminal. So instead it tells the client what binary it should `exec()` and with what arguments. + +When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC connection, which tries to terminate the command as soon as possible. After the third Ctrl-C, the client sends a SIGKILL to the server instead. + +The source code of the client is under `src/main/cpp` and the protocol used to communicate with the server is in `src/main/protobuf/command_server.proto` . + +The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls from the client are handled by `GrpcServerImpl.run()`. ## Directory layout -Bazel creates a somewhat complicated set of directories during a build. A full -description is available in [Output directory layout](/remote/output-directories). +Bazel creates a somewhat complicated set of directories during a build. A full description is available in [Output directory layout](/remote/output-directories). -The "main repo" is the source tree Bazel is run in. It usually corresponds to -something you checked out from source control. The root of this directory is -known as the "workspace root". +The "main repo" is the source tree Bazel is run in. It usually corresponds to something you checked out from source control. The root of this directory is known as the "workspace root". -Bazel puts all of its data under the "output user root". This is usually -`$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the -`--output_user_root` startup option. +Bazel puts all of its data under the "output user root". This is usually `$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the `--output_user_root` startup option. -The "install base" is where Bazel is extracted to. This is done automatically -and each Bazel version gets a subdirectory based on its checksum under the -install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed -using the `--install_base` command line option. +The "install base" is where Bazel is extracted to. This is done automatically and each Bazel version gets a subdirectory based on its checksum under the install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed using the `--install_base` command line option. -The "output base" is the place where the Bazel instance attached to a specific -workspace writes to. Each output base has at most one Bazel server instance -running at any time. It's usually at `$OUTPUT_USER_ROOT/`. It can be changed using the `--output_base` startup option, -which is, among other things, useful for getting around the limitation that only -one Bazel instance can be running in any workspace at any given time. +The "output base" is the place where the Bazel instance attached to a specific workspace writes to. Each output base has at most one Bazel server instance running at any time. It's usually at `$OUTPUT_USER_ROOT/<checksum of the path to the workspace>`. It can be changed using the `--output_base` startup option, which is, among other things, useful for getting around the limitation that only one Bazel instance can be running in any workspace at any given time. The output directory contains, among other things: -* The fetched external repositories at `$OUTPUT_BASE/external`. -* The exec root, a directory that contains symlinks to all the source - code for the current build. It's located at `$OUTPUT_BASE/execroot`. During - the build, the working directory is `$EXECROOT/`. We are planning to change this to `$EXECROOT`, although it's a - long term plan because it's a very incompatible change. -* Files built during the build. +- The fetched external repositories at `$OUTPUT_BASE/external`. +- The exec root, a directory that contains symlinks to all the source code for the current build. It's located at `$OUTPUT_BASE/execroot`. During the build, the working directory is `$EXECROOT/<name of main repository>`. We are planning to change this to `$EXECROOT`, although it's a long term plan because it's a very incompatible change. +- Files built during the build. ## The process of executing a command -Once the Bazel server gets control and is informed about a command it needs to -execute, the following sequence of events happens: +Once the Bazel server gets control and is informed about a command it needs to execute, the following sequence of events happens: -1. `BlazeCommandDispatcher` is informed about the new request. It decides - whether the command needs a workspace to run in (almost every command except - for ones that don't have anything to do with source code, such as version or - help) and whether another command is running. +1. `BlazeCommandDispatcher` is informed about the new request. It decides whether the command needs a workspace to run in (almost every command except for ones that don't have anything to do with source code, such as version or help) and whether another command is running. -2. The right command is found. Each command must implement the interface - `BlazeCommand` and must have the `@Command` annotation (this is a bit of an - antipattern, it would be nice if all the metadata a command needs was - described by methods on `BlazeCommand`) +2. The right command is found. Each command must implement the interface `BlazeCommand` and must have the `@Command` annotation (this is a bit of an antipattern, it would be nice if all the metadata a command needs was described by methods on `BlazeCommand`) -3. The command line options are parsed. Each command has different command line - options, which are described in the `@Command` annotation. +3. The command line options are parsed. Each command has different command line options, which are described in the `@Command` annotation. -4. An event bus is created. The event bus is a stream for events that happen - during the build. Some of these are exported to outside of Bazel under the - aegis of the Build Event Protocol in order to tell the world how the build - goes. +4. An event bus is created. The event bus is a stream for events that happen during the build. Some of these are exported to outside of Bazel under the aegis of the Build Event Protocol in order to tell the world how the build goes. -5. The command gets control. The most interesting commands are those that run a - build: build, test, run, coverage and so on: this functionality is - implemented by `BuildTool`. +5. The command gets control. The most interesting commands are those that run a build: build, test, run, coverage and so on: this functionality is implemented by `BuildTool`. -6. The set of target patterns on the command line is parsed and wildcards like - `//pkg:all` and `//pkg/...` are resolved. This is implemented in - `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as - `TargetPatternPhaseValue`. +6. The set of target patterns on the command line is parsed and wildcards like `//pkg:all` and `//pkg/...` are resolved. This is implemented in `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as `TargetPatternPhaseValue`. -7. The loading/analysis phase is run to produce the action graph (a directed - acyclic graph of commands that need to be executed for the build). +7. The loading/analysis phase is run to produce the action graph (a directed acyclic graph of commands that need to be executed for the build). -8. The execution phase is run. This means running every action required to - build the top-level targets that are requested are run. +8. The execution phase is run. This means running every action required to build the top-level targets that are requested are run. ## Command line options -The command line options for a Bazel invocation are described in an -`OptionsParsingResult` object, which in turn contains a map from "option -classes" to the values of the options. An "option class" is a subclass of -`OptionsBase` and groups command line options together that are related to each -other. For example: - -1. Options related to a programming language (`CppOptions` or `JavaOptions`). - These should be a subclass of `FragmentOptions` and are eventually wrapped - into a `BuildOptions` object. -2. Options related to the way Bazel executes actions (`ExecutionOptions`) - -These options are designed to be consumed in the analysis phase and (either -through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). -Some of them (for example, whether to do C++ include scanning or not) are read -in the execution phase, but that always requires explicit plumbing since -`BuildConfiguration` is not available then. For more information, see the -section "Configurations". - -**WARNING:** We like to pretend that `OptionsBase` instances are immutable and -use them that way (such as a part of `SkyKeys`). This is not the case and -modifying them is a really good way to break Bazel in subtle ways that are hard -to debug. Unfortunately, making them actually immutable is a large endeavor. -(Modifying a `FragmentOptions` immediately after construction before anyone else -gets a chance to keep a reference to it and before `equals()` or `hashCode()` is -called on it is okay.) +The command line options for a Bazel invocation are described in an `OptionsParsingResult` object, which in turn contains a map from "option classes" to the values of the options. An "option class" is a subclass of `OptionsBase` and groups command line options together that are related to each other. For example: + +1. Options related to a programming language (`CppOptions` or `JavaOptions`). These should be a subclass of `FragmentOptions` and are eventually wrapped into a `BuildOptions` object. +2. Options related to the way Bazel executes actions (`ExecutionOptions`) + +These options are designed to be consumed in the analysis phase and (either through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). Some of them (for example, whether to do C++ include scanning or not) are read in the execution phase, but that always requires explicit plumbing since `BuildConfiguration` is not available then. For more information, see the section "Configurations". + +**WARNING:** We like to pretend that `OptionsBase` instances are immutable and use them that way (such as a part of `SkyKeys`). This is not the case and modifying them is a really good way to break Bazel in subtle ways that are hard to debug. Unfortunately, making them actually immutable is a large endeavor. (Modifying a `FragmentOptions` immediately after construction before anyone else gets a chance to keep a reference to it and before `equals()` or `hashCode()` is called on it is okay.) Bazel learns about option classes in the following ways: -1. Some are hard-wired into Bazel (`CommonCommandOptions`) -2. From the `@Command` annotation on each Bazel command -3. From `ConfiguredRuleClassProvider` (these are command line options related - to individual programming languages) -4. Starlark rules can also define their own options (see - [here](/extending/config)) +1. Some are hard-wired into Bazel (`CommonCommandOptions`) +2. From the `@Command` annotation on each Bazel command +3. From `ConfiguredRuleClassProvider` (these are command line options related to individual programming languages) +4. Starlark rules can also define their own options (see [here](/extending/config)) -Each option (excluding Starlark-defined options) is a member variable of a -`FragmentOptions` subclass that has the `@Option` annotation, which specifies -the name and the type of the command line option along with some help text. +Each option (excluding Starlark-defined options) is a member variable of a `FragmentOptions` subclass that has the `@Option` annotation, which specifies the name and the type of the command line option along with some help text. -The Java type of the value of a command line option is usually something simple -(a string, an integer, a Boolean, a label, etc.). However, we also support -options of more complicated types; in this case, the job of converting from the -command line string to the data type falls to an implementation of -`com.google.devtools.common.options.Converter`. +The Java type of the value of a command line option is usually something simple (a string, an integer, a Boolean, a label, etc.). However, we also support options of more complicated types; in this case, the job of converting from the command line string to the data type falls to an implementation of `com.google.devtools.common.options.Converter`. ## The source tree, as seen by Bazel -Bazel is in the business of building software, which happens by reading and -interpreting the source code. The totality of the source code Bazel operates on -is called "the workspace" and it is structured into repositories, packages and -rules. +Bazel is in the business of building software, which happens by reading and interpreting the source code. The totality of the source code Bazel operates on is called "the workspace" and it is structured into repositories, packages and rules. ### Repositories -A "repository" is a source tree on which a developer works; it usually -represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, -that is, a single source tree that contains all source code used to run the build. -Bazel, in contrast, supports projects whose source code spans multiple -repositories. The repository from which Bazel is invoked is called the "main -repository", the others are called "external repositories". +A "repository" is a source tree on which a developer works; it usually represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, that is, a single source tree that contains all source code used to run the build. Bazel, in contrast, supports projects whose source code spans multiple repositories. The repository from which Bazel is invoked is called the "main repository", the others are called "external repositories". -A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or -in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The -main repo is the source tree where you're invoking Bazel from. External repos -are defined in various ways; see [external dependencies -overview](/external/overview) for more information. +A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The main repo is the source tree where you're invoking Bazel from. External repos are defined in various ways; see [external dependencies overview](/external/overview) for more information. -Code of external repositories is symlinked or downloaded under -`$OUTPUT_BASE/external`. +Code of external repositories is symlinked or downloaded under `$OUTPUT_BASE/external`. -When running the build, the whole source tree needs to be pieced together; this -is done by `SymlinkForest`, which symlinks every package in the main repository -to `$EXECROOT` and every external repository to either `$EXECROOT/external` or -`$EXECROOT/..`. +When running the build, the whole source tree needs to be pieced together; this is done by `SymlinkForest`, which symlinks every package in the main repository to `$EXECROOT` and every external repository to either `$EXECROOT/external` or `$EXECROOT/..`. ### Packages -Every repository is composed of packages, a collection of related files and -a specification of the dependencies. These are specified by a file called -`BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason -why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this -file name. However, it turned out to be a commonly used path segment, especially -on Windows, where file names are case-insensitive. - -Packages are independent of each other: changes to the `BUILD` file of a package -cannot cause other packages to change. The addition or removal of `BUILD` files -_can _change other packages, since recursive globs stop at package boundaries -and thus the presence of a `BUILD` file stops the recursion. - -The evaluation of a `BUILD` file is called "package loading". It's implemented -in the class `PackageFactory`, works by calling the Starlark interpreter and -requires knowledge of the set of available rule classes. The result of package -loading is a `Package` object. It's mostly a map from a string (the name of a -target) to the target itself. - -A large chunk of complexity during package loading is globbing: Bazel does not -require every source file to be explicitly listed and instead can run globs -(such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that -descend into subdirectories (but not into subpackages). This requires access to -the file system and since that can be slow, we implement all sorts of tricks to -make it run in parallel and as efficiently as possible. +Every repository is composed of packages, a collection of related files and a specification of the dependencies. These are specified by a file called `BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this file name. However, it turned out to be a commonly used path segment, especially on Windows, where file names are case-insensitive. + +Packages are independent of each other: changes to the `BUILD` file of a package cannot cause other packages to change. The addition or removal of `BUILD` files \_can \_change other packages, since recursive globs stop at package boundaries and thus the presence of a `BUILD` file stops the recursion. + +The evaluation of a `BUILD` file is called "package loading". It's implemented in the class `PackageFactory`, works by calling the Starlark interpreter and requires knowledge of the set of available rule classes. The result of package loading is a `Package` object. It's mostly a map from a string (the name of a target) to the target itself. + +A large chunk of complexity during package loading is globbing: Bazel does not require every source file to be explicitly listed and instead can run globs (such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that descend into subdirectories (but not into subpackages). This requires access to the file system and since that can be slow, we implement all sorts of tricks to make it run in parallel and as efficiently as possible. Globbing is implemented in the following classes: -* `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber -* `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to - the legacy globber in order to avoid "Skyframe restarts" (described below) +- `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber +- `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to the legacy globber in order to avoid "Skyframe restarts" (described below) -The `Package` class itself contains some members that are exclusively used to -parse the "external" package (related to external dependencies) and which do not -make sense for real packages. This is -a design flaw because objects describing regular packages should not contain -fields that describe something else. These include: +The `Package` class itself contains some members that are exclusively used to parse the "external" package (related to external dependencies) and which do not make sense for real packages. This is a design flaw because objects describing regular packages should not contain fields that describe something else. These include: -* The repository mappings -* The registered toolchains -* The registered execution platforms +- The repository mappings +- The registered toolchains +- The registered execution platforms -Ideally, there would be more separation between parsing the "external" package -from parsing regular packages so that `Package` does not need to cater for the -needs of both. This is unfortunately difficult to do because the two are -intertwined quite deeply. +Ideally, there would be more separation between parsing the "external" package from parsing regular packages so that `Package` does not need to cater for the needs of both. This is unfortunately difficult to do because the two are intertwined quite deeply. ### Labels, Targets, and Rules Packages are composed of targets, which have the following types: -1. **Files:** things that are either the input or the output of the build. In - Bazel parlance, we call them _artifacts_ (discussed elsewhere). Not all - files created during the build are targets; it's common for an output of - Bazel not to have an associated label. -2. **Rules:** these describe steps to derive its outputs from its inputs. They - are generally associated with a programming language (such as `cc_library`, - `java_library` or `py_library`), but there are some language-agnostic ones - (such as `genrule` or `filegroup`) -3. **Package groups:** discussed in the [Visibility](#visibility) section. - -The name of a target is called a _Label_. The syntax of labels is -`@repo//pac/kage:name`, where `repo` is the name of the repository the Label is -in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of -the file (if the label refers to a source file) relative to the directory of the -package. When referring to a target on the command line, some parts of the label -can be omitted: - -1. If the repository is omitted, the label is taken to be in the main - repository. -2. If the package part is omitted (such as `name` or `:name`), the label is taken - to be in the package of the current working directory (relative paths - containing uplevel references (..) are not allowed) - -A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may -be implemented either in Starlark (the `rule()` function) or in Java (so called -"native rules", type `RuleClass`). In the long term, every language-specific -rule will be implemented in Starlark, but some legacy rule families (such as Java -or C++) are still in Java for the time being. - -Starlark rule classes need to be imported at the beginning of `BUILD` files -using the `load()` statement, whereas Java rule classes are "innately" known by -Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. +1. **Files:** things that are either the input or the output of the build. In Bazel parlance, we call them *artifacts* (discussed elsewhere). Not all files created during the build are targets; it's common for an output of Bazel not to have an associated label. +2. **Rules:** these describe steps to derive its outputs from its inputs. They are generally associated with a programming language (such as `cc_library`, `java_library` or `py_library`), but there are some language-agnostic ones (such as `genrule` or `filegroup`) +3. **Package groups:** discussed in the [Visibility](#visibility) section. + +The name of a target is called a *Label*. The syntax of labels is `@repo//pac/kage:name`, where `repo` is the name of the repository the Label is in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of the file (if the label refers to a source file) relative to the directory of the package. When referring to a target on the command line, some parts of the label can be omitted: + +1. If the repository is omitted, the label is taken to be in the main repository. +2. If the package part is omitted (such as `name` or `:name`), the label is taken to be in the package of the current working directory (relative paths containing uplevel references (..) are not allowed) + +A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may be implemented either in Starlark (the `rule()` function) or in Java (so called "native rules", type `RuleClass`). In the long term, every language-specific rule will be implemented in Starlark, but some legacy rule families (such as Java or C++) are still in Java for the time being. + +Starlark rule classes need to be imported at the beginning of `BUILD` files using the `load()` statement, whereas Java rule classes are "innately" known by Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. Rule classes contain information such as: -1. Its attributes (such as `srcs`, `deps`): their types, default values, - constraints, etc. -2. The configuration transitions and aspects attached to each attribute, if any -3. The implementation of the rule -4. The transitive info providers the rule "usually" creates +1. Its attributes (such as `srcs`, `deps`): their types, default values, constraints, etc. +2. The configuration transitions and aspects attached to each attribute, if any +3. The implementation of the rule +4. The transitive info providers the rule "usually" creates -**Terminology note:** In the codebase, we often use "Rule" to mean the target -created by a rule class. But in Starlark and in user-facing documentation, -"Rule" should be used exclusively to refer to the rule class itself; the target -is just a "target". Also note that despite `RuleClass` having "class" in its -name, there is no Java inheritance relationship between a rule class and targets -of that type. +**Terminology note:** In the codebase, we often use "Rule" to mean the target created by a rule class. But in Starlark and in user-facing documentation, "Rule" should be used exclusively to refer to the rule class itself; the target is just a "target". Also note that despite `RuleClass` having "class" in its name, there is no Java inheritance relationship between a rule class and targets of that type. ## Skyframe -The evaluation framework underlying Bazel is called Skyframe. Its model is that -everything that needs to be built during a build is organized into a directed -acyclic graph with edges pointing from any pieces of data to its dependencies, -that is, other pieces of data that need to be known to construct it. - -The nodes in the graph are called `SkyValue`s and their names are called -`SkyKey`s. Both are deeply immutable; only immutable objects should be -reachable from them. This invariant almost always holds, and in case it doesn't -(such as for the individual options classes `BuildOptions`, which is a member of -`BuildConfigurationValue` and its `SkyKey`) we try really hard not to change -them or to change them in only ways that are not observable from the outside. -From this it follows that everything that is computed within Skyframe (such as -configured targets) must also be immutable. - -The most convenient way to observe the Skyframe graph is to run `bazel dump ---skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best -to do it for tiny builds, since it can get pretty large. - -Skyframe lives in the `com.google.devtools.build.skyframe` package. The -similarly-named package `com.google.devtools.build.lib.skyframe` contains the -implementation of Bazel on top of Skyframe. More information about Skyframe is -available [here](/reference/skyframe). - -To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the -`SkyFunction` corresponding to the type of the key. During the function's -evaluation, it may request other dependencies from Skyframe by calling the -various overloads of `SkyFunction.Environment.getValue()`. This has the -side-effect of registering those dependencies into Skyframe's internal graph, so -that Skyframe will know to re-evaluate the function when any of its dependencies -change. In other words, Skyframe's caching and incremental computation work at -the granularity of `SkyFunction`s and `SkyValue`s. - -Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` -will return null. The function should then yield control back to Skyframe by -itself returning null. At some later point, Skyframe will evaluate the -unavailable dependency, then restart the function from the beginning — only this -time the `getValue()` call will succeed with a non-null result. - -A consequence of this is that any computation performed inside the `SkyFunction` -prior to the restart must be repeated. But this does not include work done to -evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work -around this issue by: - -1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to - limit the number of restarts. -2. Breaking up a `SkyValue` into separate pieces computed by different - `SkyFunction`s, so that they can be computed and cached independently. This - should be done strategically, since it has the potential to increases memory - usage. -3. Storing state between restarts, either using - `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache - "behind the back of Skyframe". With complex SkyFunctions, state management - between restarts can get tricky, so - [`StateMachine`s](/contribute/statemachine-guide) were introduced for a - structured approach to logical concurrency, including hooks to suspend and - resume hierarchical computations within a `SkyFunction`. Example: - [`DependencyResolver#computeDependencies`][statemachine_example] - uses a `StateMachine` with `getState()` to compute the potentially huge set - of direct dependencies of a configured target, which otherwise can result in - expensive restarts. - -[statemachine_example]: https://developers.google.com/devsite/reference/markdown/links#reference_links - -Fundamentally, Bazel need these types of workarounds because hundreds of -thousands of in-flight Skyframe nodes is common, and Java's support of -lightweight threads [does not outperform][virtual_threads] the -`StateMachine` implementation as of 2023. - -[virtual_threads]: /contribute/statemachine-guide#epilogue_eventually_removing_callbacks +The evaluation framework underlying Bazel is called Skyframe. Its model is that everything that needs to be built during a build is organized into a directed acyclic graph with edges pointing from any pieces of data to its dependencies, that is, other pieces of data that need to be known to construct it. + +The nodes in the graph are called `SkyValue`s and their names are called `SkyKey`s. Both are deeply immutable; only immutable objects should be reachable from them. This invariant almost always holds, and in case it doesn't (such as for the individual options classes `BuildOptions`, which is a member of `BuildConfigurationValue` and its `SkyKey`) we try really hard not to change them or to change them in only ways that are not observable from the outside. From this it follows that everything that is computed within Skyframe (such as configured targets) must also be immutable. + +The most convenient way to observe the Skyframe graph is to run `bazel dump --skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best to do it for tiny builds, since it can get pretty large. + +Skyframe lives in the `com.google.devtools.build.skyframe` package. The similarly-named package `com.google.devtools.build.lib.skyframe` contains the implementation of Bazel on top of Skyframe. More information about Skyframe is available [here](/reference/skyframe). + +To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the `SkyFunction` corresponding to the type of the key. During the function's evaluation, it may request other dependencies from Skyframe by calling the various overloads of `SkyFunction.Environment.getValue()`. This has the side-effect of registering those dependencies into Skyframe's internal graph, so that Skyframe will know to re-evaluate the function when any of its dependencies change. In other words, Skyframe's caching and incremental computation work at the granularity of `SkyFunction`s and `SkyValue`s. + +Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` will return null. The function should then yield control back to Skyframe by itself returning null. At some later point, Skyframe will evaluate the unavailable dependency, then restart the function from the beginning — only this time the `getValue()` call will succeed with a non-null result. + +A consequence of this is that any computation performed inside the `SkyFunction` prior to the restart must be repeated. But this does not include work done to evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work around this issue by: + +1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to limit the number of restarts. +2. Breaking up a `SkyValue` into separate pieces computed by different `SkyFunction`s, so that they can be computed and cached independently. This should be done strategically, since it has the potential to increases memory usage. +3. Storing state between restarts, either using `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache "behind the back of Skyframe". With complex SkyFunctions, state management between restarts can get tricky, so [`StateMachine`s](/contribute/statemachine-guide) were introduced for a structured approach to logical concurrency, including hooks to suspend and resume hierarchical computations within a `SkyFunction`. Example: [`DependencyResolver#computeDependencies`](https://developers.google.com/devsite/reference/markdown/links#reference_links) uses a `StateMachine` with `getState()` to compute the potentially huge set of direct dependencies of a configured target, which otherwise can result in expensive restarts. + +Fundamentally, Bazel need these types of workarounds because hundreds of thousands of in-flight Skyframe nodes is common, and Java's support of lightweight threads [does not outperform](/contribute/statemachine-guide#epilogue_eventually_removing_callbacks) the `StateMachine` implementation as of 2023. ## Starlark -Starlark is the domain-specific language people use to configure and extend -Bazel. It's conceived as a restricted subset of Python that has far fewer types, -more restrictions on control flow, and most importantly, strong immutability -guarantees to enable concurrent reads. It is not Turing-complete, which -discourages some (but not all) users from trying to accomplish general -programming tasks within the language. +Starlark is the domain-specific language people use to configure and extend Bazel. It's conceived as a restricted subset of Python that has far fewer types, more restrictions on control flow, and most importantly, strong immutability guarantees to enable concurrent reads. It is not Turing-complete, which discourages some (but not all) users from trying to accomplish general programming tasks within the language. -Starlark is implemented in the `net.starlark.java` package. -It also has an independent Go implementation -[here](https://github.com/google/starlark-go). The Java -implementation used in Bazel is currently an interpreter. +Starlark is implemented in the `net.starlark.java` package. It also has an independent Go implementation [here](https://github.com/google/starlark-go). The Java implementation used in Bazel is currently an interpreter. Starlark is used in several contexts, including: -1. **`BUILD` files.** This is where new build targets are defined. Starlark - code running in this context only has access to the contents of the `BUILD` - file itself and `.bzl` files loaded by it. -2. **The `MODULE.bazel` file.** This is where external dependencies are - defined. Starlark code running in this context only has very limited access - to a few predefined directives. -3. **`.bzl` files.** This is where new build rules, repo rules, module - extensions are defined. Starlark code here can define new functions and load - from other `.bzl` files. +1. **`BUILD` files.** This is where new build targets are defined. Starlark code running in this context only has access to the contents of the `BUILD` file itself and `.bzl` files loaded by it. +2. **The `MODULE.bazel` file.** This is where external dependencies are defined. Starlark code running in this context only has very limited access to a few predefined directives. +3. **`.bzl` files.** This is where new build rules, repo rules, module extensions are defined. Starlark code here can define new functions and load from other `.bzl` files. -The dialects available for `BUILD` and `.bzl` files are slightly different -because they express different things. A list of differences is available -[here](/rules/language#differences-between-build-and-bzl-files). +The dialects available for `BUILD` and `.bzl` files are slightly different because they express different things. A list of differences is available [here](/rules/language#differences-between-build-and-bzl-files). More information about Starlark is available [here](/rules/language). ## The loading/analysis phase -The loading/analysis phase is where Bazel determines what actions are needed to -build a particular rule. Its basic unit is a "configured target", which is, -quite sensibly, a (target, configuration) pair. - -It's called the "loading/analysis phase" because it can be split into two -distinct parts, which used to be serialized, but they can now overlap in time: - -1. Loading packages, that is, turning `BUILD` files into the `Package` objects - that represent them -2. Analyzing configured targets, that is, running the implementation of the - rules to produce the action graph - -Each configured target in the transitive closure of the configured targets -requested on the command line must be analyzed bottom-up; that is, leaf nodes -first, then up to the ones on the command line. The inputs to the analysis of -a single configured target are: - -1. **The configuration.** ("how" to build that rule; for example, the target - platform but also things like command line options the user wants to be - passed to the C++ compiler) -2. **The direct dependencies.** Their transitive info providers are available - to the rule being analyzed. They are called like that because they provide a - "roll-up" of the information in the transitive closure of the configured - target, such as all the .jar files on the classpath or all the .o files that - need to be linked into a C++ binary) -3. **The target itself**. This is the result of loading the package the target - is in. For rules, this includes its attributes, which is usually what - matters. -4. **The implementation of the configured target.** For rules, this can either - be in Starlark or in Java. All non-rule configured targets are implemented - in Java. +The loading/analysis phase is where Bazel determines what actions are needed to build a particular rule. Its basic unit is a "configured target", which is, quite sensibly, a (target, configuration) pair. + +It's called the "loading/analysis phase" because it can be split into two distinct parts, which used to be serialized, but they can now overlap in time: + +1. Loading packages, that is, turning `BUILD` files into the `Package` objects that represent them +2. Analyzing configured targets, that is, running the implementation of the rules to produce the action graph + +Each configured target in the transitive closure of the configured targets requested on the command line must be analyzed bottom-up; that is, leaf nodes first, then up to the ones on the command line. The inputs to the analysis of a single configured target are: + +1. **The configuration.** ("how" to build that rule; for example, the target platform but also things like command line options the user wants to be passed to the C++ compiler) +2. **The direct dependencies.** Their transitive info providers are available to the rule being analyzed. They are called like that because they provide a "roll-up" of the information in the transitive closure of the configured target, such as all the .jar files on the classpath or all the .o files that need to be linked into a C++ binary) +3. **The target itself**. This is the result of loading the package the target is in. For rules, this includes its attributes, which is usually what matters. +4. **The implementation of the configured target.** For rules, this can either be in Starlark or in Java. All non-rule configured targets are implemented in Java. The output of analyzing a configured target is: -1. The transitive info providers that configured targets that depend on it can - access -2. The artifacts it can create and the actions that produce them. +1. The transitive info providers that configured targets that depend on it can access +2. The artifacts it can create and the actions that produce them. -The API offered to Java rules is `RuleContext`, which is the equivalent of the -`ctx` argument of Starlark rules. Its API is more powerful, but at the same -time, it's easier to do Bad Things™, for example to write code whose time or -space complexity is quadratic (or worse), to make the Bazel server crash with a -Java exception or to violate invariants (such as by inadvertently modifying an -`Options` instance or by making a configured target mutable) +The API offered to Java rules is `RuleContext`, which is the equivalent of the `ctx` argument of Starlark rules. Its API is more powerful, but at the same time, it's easier to do Bad Things™, for example to write code whose time or space complexity is quadratic (or worse), to make the Bazel server crash with a Java exception or to violate invariants (such as by inadvertently modifying an `Options` instance or by making a configured target mutable) -The algorithm that determines the direct dependencies of a configured target -lives in `DependencyResolver.dependentNodeMap()`. +The algorithm that determines the direct dependencies of a configured target lives in `DependencyResolver.dependentNodeMap()`. ### Configurations -Configurations are the "how" of building a target: for what platform, with what -command line options, etc. - -The same target can be built for multiple configurations in the same build. This -is useful, for example, when the same code is used for a tool that's run during -the build and for the target code and we are cross-compiling or when we are -building a fat Android app (one that contains native code for multiple CPU -architectures) - -Conceptually, the configuration is a `BuildOptions` instance. However, in -practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides -additional sundry pieces of functionality. It propagates from the top of the -dependency graph to the bottom. If it changes, the build needs to be -re-analyzed. - -This results in anomalies like having to re-analyze the whole build if, for -example, the number of requested test runs changes, even though that only -affects test targets (we have plans to "trim" configurations so that this is -not the case, but it's not ready yet). - -When a rule implementation needs part of the configuration, it needs to declare -it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` -. This is both to avoid mistakes (such as Python rules using the Java fragment) and -to facilitate configuration trimming so that such as if Python options change, C++ -targets don't need to be re-analyzed. - -The configuration of a rule is not necessarily the same as that of its "parent" -rule. The process of changing the configuration in a dependency edge is called a -"configuration transition". It can happen in two places: - -1. On a dependency edge. These transitions are specified in - `Attribute.Builder.cfg()` and are functions from a `Rule` (where the - transition happens) and a `BuildOptions` (the original configuration) to one - or more `BuildOptions` (the output configuration). -2. On any incoming edge to a configured target. These are specified in - `RuleClass.Builder.cfg()`. +Configurations are the "how" of building a target: for what platform, with what command line options, etc. + +The same target can be built for multiple configurations in the same build. This is useful, for example, when the same code is used for a tool that's run during the build and for the target code and we are cross-compiling or when we are building a fat Android app (one that contains native code for multiple CPU architectures) + +Conceptually, the configuration is a `BuildOptions` instance. However, in practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides additional sundry pieces of functionality. It propagates from the top of the dependency graph to the bottom. If it changes, the build needs to be re-analyzed. + +This results in anomalies like having to re-analyze the whole build if, for example, the number of requested test runs changes, even though that only affects test targets (we have plans to "trim" configurations so that this is not the case, but it's not ready yet). + +When a rule implementation needs part of the configuration, it needs to declare it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` . This is both to avoid mistakes (such as Python rules using the Java fragment) and to facilitate configuration trimming so that such as if Python options change, C++ targets don't need to be re-analyzed. + +The configuration of a rule is not necessarily the same as that of its "parent" rule. The process of changing the configuration in a dependency edge is called a "configuration transition". It can happen in two places: + +1. On a dependency edge. These transitions are specified in `Attribute.Builder.cfg()` and are functions from a `Rule` (where the transition happens) and a `BuildOptions` (the original configuration) to one or more `BuildOptions` (the output configuration). +2. On any incoming edge to a configured target. These are specified in `RuleClass.Builder.cfg()`. The relevant classes are `TransitionFactory` and `ConfigurationTransition`. Configuration transitions are used, for example: -1. To declare that a particular dependency is used during the build and it - should thus be built in the execution architecture -2. To declare that a particular dependency must be built for multiple - architectures (such as for native code in fat Android APKs) +1. To declare that a particular dependency is used during the build and it should thus be built in the execution architecture +2. To declare that a particular dependency must be built for multiple architectures (such as for native code in fat Android APKs) -If a configuration transition results in multiple configurations, it's called a -_split transition._ +If a configuration transition results in multiple configurations, it's called a *split transition.* -Configuration transitions can also be implemented in Starlark (documentation -[here](/extending/config)) +Configuration transitions can also be implemented in Starlark (documentation [here](/extending/config)) ### Transitive info providers -Transitive info providers are a way (and the _only _way) for configured targets -to learn things about other configured targets that they depend on, and the only -way to tell things about themselves to other configured targets that depend on -them. The reason why "transitive" is in their name is that this is usually some -sort of roll-up of the transitive closure of a configured target. - -There is generally a 1:1 correspondence between Java transitive info providers -and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of -`FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was -deemed to be more Starlark-ish than a direct transliteration of the Java one). -Their key is one of the following things: - -1. A Java Class object. This is only available for providers that are not - accessible from Starlark. These providers are a subclass of - `TransitiveInfoProvider`. -2. A string. This is legacy and heavily discouraged since it's susceptible to - name clashes. Such transitive info providers are direct subclasses of - `build.lib.packages.Info` . -3. A provider symbol. This can be created from Starlark using the `provider()` - function and is the recommended way to create new providers. The symbol is - represented by a `Provider.Key` instance in Java. - -New providers implemented in Java should be implemented using `BuiltinProvider`. -`NativeProvider` is deprecated (we haven't had time to remove it yet) and -`TransitiveInfoProvider` subclasses cannot be accessed from Starlark. +Transitive info providers are a way (and the \_only \_way) for configured targets to learn things about other configured targets that they depend on, and the only way to tell things about themselves to other configured targets that depend on them. The reason why "transitive" is in their name is that this is usually some sort of roll-up of the transitive closure of a configured target. + +There is generally a 1:1 correspondence between Java transitive info providers and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of `FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was deemed to be more Starlark-ish than a direct transliteration of the Java one). Their key is one of the following things: + +1. A Java Class object. This is only available for providers that are not accessible from Starlark. These providers are a subclass of `TransitiveInfoProvider`. +2. A string. This is legacy and heavily discouraged since it's susceptible to name clashes. Such transitive info providers are direct subclasses of `build.lib.packages.Info` . +3. A provider symbol. This can be created from Starlark using the `provider()` function and is the recommended way to create new providers. The symbol is represented by a `Provider.Key` instance in Java. + +New providers implemented in Java should be implemented using `BuiltinProvider`. `NativeProvider` is deprecated (we haven't had time to remove it yet) and `TransitiveInfoProvider` subclasses cannot be accessed from Starlark. ### Configured targets -Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a -subclass for each rule class implemented in Java. Starlark configured targets -are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . +Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a subclass for each rule class implemented in Java. Starlark configured targets are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . -Configured target factories should use `RuleConfiguredTargetBuilder` to -construct their return value. It consists of the following things: +Configured target factories should use `RuleConfiguredTargetBuilder` to construct their return value. It consists of the following things: -1. Their `filesToBuild`, the hazy concept of "the set of files this rule - represents." These are the files that get built when the configured target - is on the command line or in the srcs of a genrule. -2. Their runfiles, regular and data. -3. Their output groups. These are various "other sets of files" the rule can - build. They can be accessed using the output\_group attribute of the - filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. +1. Their `filesToBuild`, the hazy concept of "the set of files this rule represents." These are the files that get built when the configured target is on the command line or in the srcs of a genrule. +2. Their runfiles, regular and data. +3. Their output groups. These are various "other sets of files" the rule can build. They can be accessed using the output\_group attribute of the filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. ### Runfiles -Some binaries need data files to run. A prominent example is tests that need -input files. This is represented in Bazel by the concept of "runfiles". A -"runfiles tree" is a directory tree of the data files for a particular binary. -It is created in the file system as a symlink tree with individual symlinks -pointing to the files in the source or output trees. - -A set of runfiles is represented as a `Runfiles` instance. It is conceptually a -map from the path of a file in the runfiles tree to the `Artifact` instance that -represents it. It's a little more complicated than a single `Map` for two -reasons: - -* Most of the time, the runfiles path of a file is the same as its execpath. - We use this to save some RAM. -* There are various legacy kinds of entries in runfiles trees, which also need - to be represented. - -Runfiles are collected using `RunfilesProvider`: an instance of this class -represents the runfiles a configured target (such as a library) and its transitive -closure needs and they are gathered like a nested set (in fact, they are -implemented using nested sets under the cover): each target unions the runfiles -of its dependencies, adds some of its own, then sends the resulting set upwards -in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` -instances, one for when the rule is depended on through the "data" attribute and -one for every other kind of incoming dependency. This is because a target -sometimes presents different runfiles when depended on through a data attribute -than otherwise. This is undesired legacy behavior that we haven't gotten around -removing yet. - -Runfiles of binaries are represented as an instance of `RunfilesSupport`. This -is different from `Runfiles` because `RunfilesSupport` has the capability of -actually being built (unlike `Runfiles`, which is just a mapping). This -necessitates the following additional components: - -* **The input runfiles manifest.** This is a serialized description of the - runfiles tree. It is used as a proxy for the contents of the runfiles tree - and Bazel assumes that the runfiles tree changes if and only if the contents - of the manifest change. -* **The output runfiles manifest.** This is used by runtime libraries that - handle runfiles trees, notably on Windows, which sometimes doesn't support - symbolic links. -* **Command line arguments** for running the binary whose runfiles the - `RunfilesSupport` object represents. +Some binaries need data files to run. A prominent example is tests that need input files. This is represented in Bazel by the concept of "runfiles". A "runfiles tree" is a directory tree of the data files for a particular binary. It is created in the file system as a symlink tree with individual symlinks pointing to the files in the source or output trees. + +A set of runfiles is represented as a `Runfiles` instance. It is conceptually a map from the path of a file in the runfiles tree to the `Artifact` instance that represents it. It's a little more complicated than a single `Map` for two reasons: + +- Most of the time, the runfiles path of a file is the same as its execpath. We use this to save some RAM. +- There are various legacy kinds of entries in runfiles trees, which also need to be represented. + +Runfiles are collected using `RunfilesProvider`: an instance of this class represents the runfiles a configured target (such as a library) and its transitive closure needs and they are gathered like a nested set (in fact, they are implemented using nested sets under the cover): each target unions the runfiles of its dependencies, adds some of its own, then sends the resulting set upwards in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` instances, one for when the rule is depended on through the "data" attribute and one for every other kind of incoming dependency. This is because a target sometimes presents different runfiles when depended on through a data attribute than otherwise. This is undesired legacy behavior that we haven't gotten around removing yet. + +Runfiles of binaries are represented as an instance of `RunfilesSupport`. This is different from `Runfiles` because `RunfilesSupport` has the capability of actually being built (unlike `Runfiles`, which is just a mapping). This necessitates the following additional components: + +- **The input runfiles manifest.** This is a serialized description of the runfiles tree. It is used as a proxy for the contents of the runfiles tree and Bazel assumes that the runfiles tree changes if and only if the contents of the manifest change. +- **The output runfiles manifest.** This is used by runtime libraries that handle runfiles trees, notably on Windows, which sometimes doesn't support symbolic links. +- **Command line arguments** for running the binary whose runfiles the `RunfilesSupport` object represents. ### Aspects -Aspects are a way to "propagate computation down the dependency graph". They are -described for users of Bazel -[here](/extending/aspects). A good -motivating example is protocol buffers: a `proto_library` rule should not know -about any particular language, but building the implementation of a protocol -buffer message (the "basic unit" of protocol buffers) in any programming -language should be coupled to the `proto_library` rule so that if two targets in -the same language depend on the same protocol buffer, it gets built only once. - -Just like configured targets, they are represented in Skyframe as a `SkyValue` -and the way they are constructed is very similar to how configured targets are -built: they have a factory class called `ConfiguredAspectFactory` that has -access to a `RuleContext`, but unlike configured target factories, it also knows -about the configured target it is attached to and its providers. - -The set of aspects propagated down the dependency graph is specified for each -attribute using the `Attribute.Builder.aspects()` function. There are a few -confusingly-named classes that participate in the process: - -1. `AspectClass` is the implementation of the aspect. It can be either in Java - (in which case it's a subclass) or in Starlark (in which case it's an - instance of `StarlarkAspectClass`). It's analogous to - `RuleConfiguredTargetFactory`. -2. `AspectDefinition` is the definition of the aspect; it includes the - providers it requires, the providers it provides and contains a reference to - its implementation, such as the appropriate `AspectClass` instance. It's - analogous to `RuleClass`. -3. `AspectParameters` is a way to parametrize an aspect that is propagated down - the dependency graph. It's currently a string to string map. A good example - of why it's useful is protocol buffers: if a language has multiple APIs, the - information as to which API the protocol buffers should be built for should - be propagated down the dependency graph. -4. `Aspect` represents all the data that's needed to compute an aspect that - propagates down the dependency graph. It consists of the aspect class, its - definition and its parameters. -5. `RuleAspect` is the function that determines which aspects a particular rule - should propagate. It's a `Rule` -> `Aspect` function. - -A somewhat unexpected complication is that aspects can attach to other aspects; -for example, an aspect collecting the classpath for a Java IDE will probably -want to know about all the .jar files on the classpath, but some of them are -protocol buffers. In that case, the IDE aspect will want to attach to the -(`proto_library` rule + Java proto aspect) pair. - -The complexity of aspects on aspects is captured in the class -`AspectCollection`. +Aspects are a way to "propagate computation down the dependency graph". They are described for users of Bazel [here](/extending/aspects). A good motivating example is protocol buffers: a `proto_library` rule should not know about any particular language, but building the implementation of a protocol buffer message (the "basic unit" of protocol buffers) in any programming language should be coupled to the `proto_library` rule so that if two targets in the same language depend on the same protocol buffer, it gets built only once. + +Just like configured targets, they are represented in Skyframe as a `SkyValue` and the way they are constructed is very similar to how configured targets are built: they have a factory class called `ConfiguredAspectFactory` that has access to a `RuleContext`, but unlike configured target factories, it also knows about the configured target it is attached to and its providers. + +The set of aspects propagated down the dependency graph is specified for each attribute using the `Attribute.Builder.aspects()` function. There are a few confusingly-named classes that participate in the process: + +1. `AspectClass` is the implementation of the aspect. It can be either in Java (in which case it's a subclass) or in Starlark (in which case it's an instance of `StarlarkAspectClass`). It's analogous to `RuleConfiguredTargetFactory`. +2. `AspectDefinition` is the definition of the aspect; it includes the providers it requires, the providers it provides and contains a reference to its implementation, such as the appropriate `AspectClass` instance. It's analogous to `RuleClass`. +3. `AspectParameters` is a way to parametrize an aspect that is propagated down the dependency graph. It's currently a string to string map. A good example of why it's useful is protocol buffers: if a language has multiple APIs, the information as to which API the protocol buffers should be built for should be propagated down the dependency graph. +4. `Aspect` represents all the data that's needed to compute an aspect that propagates down the dependency graph. It consists of the aspect class, its definition and its parameters. +5. `RuleAspect` is the function that determines which aspects a particular rule should propagate. It's a `Rule` -\> `Aspect` function. + +A somewhat unexpected complication is that aspects can attach to other aspects; for example, an aspect collecting the classpath for a Java IDE will probably want to know about all the .jar files on the classpath, but some of them are protocol buffers. In that case, the IDE aspect will want to attach to the (`proto_library` rule + Java proto aspect) pair. + +The complexity of aspects on aspects is captured in the class `AspectCollection`. ### Platforms and toolchains -Bazel supports multi-platform builds, that is, builds where there may be -multiple architectures where build actions run and multiple architectures for -which code is built. These architectures are referred to as _platforms_ in Bazel -parlance (full documentation -[here](/extending/platforms)) - -A platform is described by a key-value mapping from _constraint settings_ (such as -the concept of "CPU architecture") to _constraint values_ (such as a particular CPU -like x86\_64). We have a "dictionary" of the most commonly used constraint -settings and values in the `@platforms` repository. - -The concept of _toolchain_ comes from the fact that depending on what platforms -the build is running on and what platforms are targeted, one may need to use -different compilers; for example, a particular C++ toolchain may run on a -specific OS and be able to target some other OSes. Bazel must determine the C++ -compiler that is used based on the set execution and target platform -(documentation for toolchains -[here](/extending/toolchains)). - -In order to do this, toolchains are annotated with the set of execution and -target platform constraints they support. In order to do this, the definition of -a toolchain are split into two parts: - -1. A `toolchain()` rule that describes the set of execution and target - constraints a toolchain supports and tells what kind (such as C++ or Java) of - toolchain it is (the latter is represented by the `toolchain_type()` rule) -2. A language-specific rule that describes the actual toolchain (such as - `cc_toolchain()`) - -This is done in this way because we need to know the constraints for every -toolchain in order to do toolchain resolution and language-specific -`*_toolchain()` rules contain much more information than that, so they take more -time to load. +Bazel supports multi-platform builds, that is, builds where there may be multiple architectures where build actions run and multiple architectures for which code is built. These architectures are referred to as *platforms* in Bazel parlance (full documentation [here](/extending/platforms)) + +A platform is described by a key-value mapping from *constraint settings* (such as the concept of "CPU architecture") to *constraint values* (such as a particular CPU like x86\_64). We have a "dictionary" of the most commonly used constraint settings and values in the `@platforms` repository. + +The concept of *toolchain* comes from the fact that depending on what platforms the build is running on and what platforms are targeted, one may need to use different compilers; for example, a particular C++ toolchain may run on a specific OS and be able to target some other OSes. Bazel must determine the C++ compiler that is used based on the set execution and target platform (documentation for toolchains [here](/extending/toolchains)). + +In order to do this, toolchains are annotated with the set of execution and target platform constraints they support. In order to do this, the definition of a toolchain are split into two parts: + +1. A `toolchain()` rule that describes the set of execution and target constraints a toolchain supports and tells what kind (such as C++ or Java) of toolchain it is (the latter is represented by the `toolchain_type()` rule) +2. A language-specific rule that describes the actual toolchain (such as `cc_toolchain()`) + +This is done in this way because we need to know the constraints for every toolchain in order to do toolchain resolution and language-specific `*_toolchain()` rules contain much more information than that, so they take more time to load. Execution platforms are specified in one of the following ways: -1. In the MODULE.bazel file using the `register_execution_platforms()` function -2. On the command line using the --extra\_execution\_platforms command line - option - -The set of available execution platforms is computed in -`RegisteredExecutionPlatformsFunction` . - -The target platform for a configured target is determined by -`PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we -eventually want to support multiple target platforms, but it's not implemented -yet. - -The set of toolchains to be used for a configured target is determined by -`ToolchainResolutionFunction`. It is a function of: - -* The set of registered toolchains (in the MODULE.bazel file and the - configuration) -* The desired execution and target platforms (in the configuration) -* The set of toolchain types that are required by the configured target (in - `UnloadedToolchainContextKey)` -* The set of execution platform constraints of the configured target (the - `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` - -Its result is an `UnloadedToolchainContext`, which is essentially a map from -toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of -the selected toolchain. It's called "unloaded" because it does not contain the -toolchains themselves, only their labels. - -Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` -and used by the implementation of the configured target that requested them. - -We also have a legacy system that relies on there being one single "host" -configuration and target configurations being represented by various -configuration flags, such as `--cpu` . We are gradually transitioning to the above -system. In order to handle cases where people rely on the legacy configuration -values, we have implemented -[platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) -to translate between the legacy flags and the new-style platform constraints. -Their code is in `PlatformMappingFunction` and uses a non-Starlark "little -language". +1. In the MODULE.bazel file using the `register_execution_platforms()` function +2. On the command line using the --extra\_execution\_platforms command line option + +The set of available execution platforms is computed in `RegisteredExecutionPlatformsFunction` . + +The target platform for a configured target is determined by `PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we eventually want to support multiple target platforms, but it's not implemented yet. + +The set of toolchains to be used for a configured target is determined by `ToolchainResolutionFunction`. It is a function of: + +- The set of registered toolchains (in the MODULE.bazel file and the configuration) +- The desired execution and target platforms (in the configuration) +- The set of toolchain types that are required by the configured target (in `UnloadedToolchainContextKey)` +- The set of execution platform constraints of the configured target (the `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` + +Its result is an `UnloadedToolchainContext`, which is essentially a map from toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of the selected toolchain. It's called "unloaded" because it does not contain the toolchains themselves, only their labels. + +Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` and used by the implementation of the configured target that requested them. + +We also have a legacy system that relies on there being one single "host" configuration and target configurations being represented by various configuration flags, such as `--cpu` . We are gradually transitioning to the above system. In order to handle cases where people rely on the legacy configuration values, we have implemented [platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) to translate between the legacy flags and the new-style platform constraints. Their code is in `PlatformMappingFunction` and uses a non-Starlark "little language". ### Constraints -Sometimes one wants to designate a target as being compatible with only a few -platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: +Sometimes one wants to designate a target as being compatible with only a few platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: -* Rule-specific constraints -* `environment_group()` / `environment()` -* Platform constraints +- Rule-specific constraints +- `environment_group()` / `environment()` +- Platform constraints -Rule-specific constraints are mostly used within Google for Java rules; they are -on their way out and they are not available in Bazel, but the source code may -contain references to it. The attribute that governs this is called -`constraints=` . +Rule-specific constraints are mostly used within Google for Java rules; they are on their way out and they are not available in Bazel, but the source code may contain references to it. The attribute that governs this is called `constraints=` . -#### environment_group() and environment() +#### environment\_group() and environment() These rules are a legacy mechanism and are not widely used. -All build rules can declare which "environments" they can be built for, where an -"environment" is an instance of the `environment()` rule. +All build rules can declare which "environments" they can be built for, where an "environment" is an instance of the `environment()` rule. There are various ways supported environments can be specified for a rule: -1. Through the `restricted_to=` attribute. This is the most direct form of - specification; it declares the exact set of environments the rule supports. -2. Through the `compatible_with=` attribute. This declares environments a rule - supports in addition to "standard" environments that are supported by - default. -3. Through the package-level attributes `default_restricted_to=` and - `default_compatible_with=`. -4. Through default specifications in `environment_group()` rules. Every - environment belongs to a group of thematically related peers (such as "CPU - architectures", "JDK versions" or "mobile operating systems"). The - definition of an environment group includes which of these environments - should be supported by "default" if not otherwise specified by the - `restricted_to=` / `environment()` attributes. A rule with no such - attributes inherits all defaults. -5. Through a rule class default. This overrides global defaults for all - instances of the given rule class. This can be used, for example, to make - all `*_test` rules testable without each instance having to explicitly - declare this capability. - -`environment()` is implemented as a regular rule whereas `environment_group()` -is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a -function that is available by default from Starlark -(`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous -target. This is to avoid a cyclic dependency that would arise because each -environment needs to declare the environment group it belongs to and each -environment group needs to declare its default environments. - -A build can be restricted to a certain environment with the -`--target_environment` command line option. - -The implementation of the constraint check is in -`RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. +1. Through the `restricted_to=` attribute. This is the most direct form of specification; it declares the exact set of environments the rule supports. +2. Through the `compatible_with=` attribute. This declares environments a rule supports in addition to "standard" environments that are supported by default. +3. Through the package-level attributes `default_restricted_to=` and `default_compatible_with=`. +4. Through default specifications in `environment_group()` rules. Every environment belongs to a group of thematically related peers (such as "CPU architectures", "JDK versions" or "mobile operating systems"). The definition of an environment group includes which of these environments should be supported by "default" if not otherwise specified by the `restricted_to=` / `environment()` attributes. A rule with no such attributes inherits all defaults. +5. Through a rule class default. This overrides global defaults for all instances of the given rule class. This can be used, for example, to make all `*_test` rules testable without each instance having to explicitly declare this capability. + +`environment()` is implemented as a regular rule whereas `environment_group()` is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a function that is available by default from Starlark (`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous target. This is to avoid a cyclic dependency that would arise because each environment needs to declare the environment group it belongs to and each environment group needs to declare its default environments. + +A build can be restricted to a certain environment with the `--target_environment` command line option. + +The implementation of the constraint check is in `RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. #### Platform constraints -The current "official" way to describe what platforms a target is compatible -with is by using the same constraints used to describe toolchains and platforms. -It was implemented in pull request -[#10945](https://github.com/bazelbuild/bazel/pull/10945). +The current "official" way to describe what platforms a target is compatible with is by using the same constraints used to describe toolchains and platforms. It was implemented in pull request [#10945](https://github.com/bazelbuild/bazel/pull/10945). ### Visibility -If you work on a large codebase with a lot of developers (like at Google), you -want to take care to prevent everyone else from arbitrarily depending on your -code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), -people _will_ come to rely on behaviors that you considered to be implementation -details. +If you work on a large codebase with a lot of developers (like at Google), you want to take care to prevent everyone else from arbitrarily depending on your code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), people *will* come to rely on behaviors that you considered to be implementation details. -Bazel supports this by the mechanism called _visibility_: you can limit which -targets can depend on a particular target using the -[visibility](/reference/be/common-definitions#common-attributes) attribute. This -attribute is a little special because, although it holds a list of labels, these -labels may encode a pattern over package names rather than a pointer to any -particular target. (Yes, this is a design flaw.) +Bazel supports this by the mechanism called *visibility*: you can limit which targets can depend on a particular target using the [visibility](/reference/be/common-definitions#common-attributes) attribute. This attribute is a little special because, although it holds a list of labels, these labels may encode a pattern over package names rather than a pointer to any particular target. (Yes, this is a design flaw.) This is implemented in the following places: -* The `RuleVisibility` interface represents a visibility declaration. It can - be either a constant (fully public or fully private) or a list of labels. -* Labels can refer to either package groups (predefined list of packages), to - packages directly (`//pkg:__pkg__`) or subtrees of packages - (`//pkg:__subpackages__`). This is different from the command line syntax, - which uses `//pkg:*` or `//pkg/...`. -* Package groups are implemented as their own target (`PackageGroup`) and - configured target (`PackageGroupConfiguredTarget`). We could probably - replace these with simple rules if we wanted to. Their logic is implemented - with the help of: `PackageSpecification`, which corresponds to a - single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds - to a single `package_group`'s `packages` attribute; and - `PackageSpecificationProvider`, which aggregates over a `package_group` and - its transitive `includes`. -* The conversion from visibility label lists to dependencies is done in - `DependencyResolver.visitTargetVisibility` and a few other miscellaneous - places. -* The actual check is done in - `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` +- The `RuleVisibility` interface represents a visibility declaration. It can be either a constant (fully public or fully private) or a list of labels. +- Labels can refer to either package groups (predefined list of packages), to packages directly (`//pkg:__pkg__`) or subtrees of packages (`//pkg:__subpackages__`). This is different from the command line syntax, which uses `//pkg:*` or `//pkg/...`. +- Package groups are implemented as their own target (`PackageGroup`) and configured target (`PackageGroupConfiguredTarget`). We could probably replace these with simple rules if we wanted to. Their logic is implemented with the help of: `PackageSpecification`, which corresponds to a single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds to a single `package_group`'s `packages` attribute; and `PackageSpecificationProvider`, which aggregates over a `package_group` and its transitive `includes`. +- The conversion from visibility label lists to dependencies is done in `DependencyResolver.visitTargetVisibility` and a few other miscellaneous places. +- The actual check is done in `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` ### Nested sets -Oftentimes, a configured target aggregates a set of files from its dependencies, -adds its own, and wraps the aggregate set into a transitive info provider so -that configured targets that depend on it can do the same. Examples: +Oftentimes, a configured target aggregates a set of files from its dependencies, adds its own, and wraps the aggregate set into a transitive info provider so that configured targets that depend on it can do the same. Examples: -* The C++ header files used for a build -* The object files that represent the transitive closure of a `cc_library` -* The set of .jar files that need to be on the classpath for a Java rule to - compile or run -* The set of Python files in the transitive closure of a Python rule +- The C++ header files used for a build +- The object files that represent the transitive closure of a `cc_library` +- The set of .jar files that need to be on the classpath for a Java rule to compile or run +- The set of Python files in the transitive closure of a Python rule -If we did this the naive way by using, for example, `List` or `Set`, we'd end up with -quadratic memory usage: if there is a chain of N rules and each rule adds a -file, we'd have 1+2+...+N collection members. +If we did this the naive way by using, for example, `List` or `Set`, we'd end up with quadratic memory usage: if there is a chain of N rules and each rule adds a file, we'd have 1+2+...+N collection members. -In order to get around this problem, we came up with the concept of a -`NestedSet`. It's a data structure that is composed of other `NestedSet` -instances and some members of its own, thereby forming a directed acyclic graph -of sets. They are immutable and their members can be iterated over. We define -multiple iteration order (`NestedSet.Order`): preorder, postorder, topological -(a node always comes after its ancestors) and "don't care, but it should be the -same each time". +In order to get around this problem, we came up with the concept of a `NestedSet`. It's a data structure that is composed of other `NestedSet` instances and some members of its own, thereby forming a directed acyclic graph of sets. They are immutable and their members can be iterated over. We define multiple iteration order (`NestedSet.Order`): preorder, postorder, topological (a node always comes after its ancestors) and "don't care, but it should be the same each time". The same data structure is called `depset` in Starlark. ### Artifacts and Actions -The actual build consists of a set of commands that need to be run to produce -the output the user wants. The commands are represented as instances of the -class `Action` and the files are represented as instances of the class -`Artifact`. They are arranged in a bipartite, directed, acyclic graph called the -"action graph". - -Artifacts come in two kinds: source artifacts (ones that are available -before Bazel starts executing) and derived artifacts (ones that need to be -built). Derived artifacts can themselves be multiple kinds: - -1. **Regular artifacts.** These are checked for up-to-dateness by computing - their checksum, with mtime as a shortcut; we don't checksum the file if its - ctime hasn't changed. -2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by - calling readlink(). Unlike regular artifacts, these can be dangling - symlinks. Usually used in cases where one then packs up some files into an - archive of some sort. -3. **Tree artifacts.** These are not single files, but directory trees. They - are checked for up-to-dateness by checking the set of files in it and their - contents. They are represented as a `TreeArtifact`. -4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a - rebuild. This is used exclusively for build stamp information: we don't want - to do a rebuild just because the current time changed. - -There is no fundamental reason why source artifacts cannot be tree artifacts or -unresolved symlink artifacts; it's just that we haven't implemented it yet. At -the moment, source symlinks are always resolved, and while source directories -are supported, their contents are entirely opaque to build rules and thus don't -support the same kind of lazy command line expansion as tree artifacts do. - -Actions are best understood as a command that needs to be run, the environment -it needs and the set of outputs it produces. The following things are the main -components of the description of an action: - -* The command line that needs to be run -* The input artifacts it needs -* The environment variables that need to be set -* Annotations that describe the environment (such as platform) it needs to run in - \ - -There are also a few other special cases, like writing a file whose content is -known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are -a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be -separate classes), although Java and C++ have their own action types -(`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). - -We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is -pretty close, but C++ is a bit of a special-case due to .d file parsing and -include scanning. - -The action graph is mostly "embedded" into the Skyframe graph: conceptually, the -execution of an action is represented as an invocation of -`ActionExecutionFunction`. The mapping from an action graph dependency edge to a -Skyframe dependency edge is described in -`ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few -optimizations in order to keep the number of Skyframe edges low: - -* Derived artifacts do not have their own `SkyValue`s. Instead, - `Artifact.getGeneratingActionKey()` is used to find out the key for the - action that generates it -* Nested sets have their own Skyframe key. +The actual build consists of a set of commands that need to be run to produce the output the user wants. The commands are represented as instances of the class `Action` and the files are represented as instances of the class `Artifact`. They are arranged in a bipartite, directed, acyclic graph called the "action graph". + +Artifacts come in two kinds: source artifacts (ones that are available before Bazel starts executing) and derived artifacts (ones that need to be built). Derived artifacts can themselves be multiple kinds: + +1. **Regular artifacts.** These are checked for up-to-dateness by computing their checksum, with mtime as a shortcut; we don't checksum the file if its ctime hasn't changed. +2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by calling readlink(). Unlike regular artifacts, these can be dangling symlinks. Usually used in cases where one then packs up some files into an archive of some sort. +3. **Tree artifacts.** These are not single files, but directory trees. They are checked for up-to-dateness by checking the set of files in it and their contents. They are represented as a `TreeArtifact`. +4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a rebuild. This is used exclusively for build stamp information: we don't want to do a rebuild just because the current time changed. + +There is no fundamental reason why source artifacts cannot be tree artifacts or unresolved symlink artifacts; it's just that we haven't implemented it yet. At the moment, source symlinks are always resolved, and while source directories are supported, their contents are entirely opaque to build rules and thus don't support the same kind of lazy command line expansion as tree artifacts do. + +Actions are best understood as a command that needs to be run, the environment it needs and the set of outputs it produces. The following things are the main components of the description of an action: + +- The command line that needs to be run +- The input artifacts it needs +- The environment variables that need to be set +- Annotations that describe the environment (such as platform) it needs to run in \\ + +There are also a few other special cases, like writing a file whose content is known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be separate classes), although Java and C++ have their own action types (`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). + +We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is pretty close, but C++ is a bit of a special-case due to .d file parsing and include scanning. + +The action graph is mostly "embedded" into the Skyframe graph: conceptually, the execution of an action is represented as an invocation of `ActionExecutionFunction`. The mapping from an action graph dependency edge to a Skyframe dependency edge is described in `ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few optimizations in order to keep the number of Skyframe edges low: + +- Derived artifacts do not have their own `SkyValue`s. Instead, `Artifact.getGeneratingActionKey()` is used to find out the key for the action that generates it +- Nested sets have their own Skyframe key. ### Shared actions -Some actions are generated by multiple configured targets; Starlark rules are -more limited since they are only allowed to put their derived actions into a -directory determined by their configuration and their package (but even so, -rules in the same package can conflict), but rules implemented in Java can put -derived artifacts anywhere. - -This is considered to be a misfeature, but getting rid of it is really hard -because it produces significant savings in execution time when, for example, a -source file needs to be processed somehow and that file is referenced by -multiple rules (handwave-handwave). This comes at the cost of some RAM: each -instance of a shared action needs to be stored in memory separately. - -If two actions generate the same output file, they must be exactly the same: -have the same inputs, the same outputs and run the same command line. This -equivalence relation is implemented in `Actions.canBeShared()` and it is -verified between the analysis and execution phases by looking at every Action. -This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` -and is one of the few places in Bazel that requires a "global" view of the -build. +Some actions are generated by multiple configured targets; Starlark rules are more limited since they are only allowed to put their derived actions into a directory determined by their configuration and their package (but even so, rules in the same package can conflict), but rules implemented in Java can put derived artifacts anywhere. + +This is considered to be a misfeature, but getting rid of it is really hard because it produces significant savings in execution time when, for example, a source file needs to be processed somehow and that file is referenced by multiple rules (handwave-handwave). This comes at the cost of some RAM: each instance of a shared action needs to be stored in memory separately. + +If two actions generate the same output file, they must be exactly the same: have the same inputs, the same outputs and run the same command line. This equivalence relation is implemented in `Actions.canBeShared()` and it is verified between the analysis and execution phases by looking at every Action. This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` and is one of the few places in Bazel that requires a "global" view of the build. ## The execution phase -This is when Bazel actually starts running build actions, such as commands that -produce outputs. - -The first thing Bazel does after the analysis phase is to determine what -Artifacts need to be built. The logic for this is encoded in -`TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the -configured targets on the command line and the contents of a special output -group for the explicit purpose of expressing "if this target is on the command -line, build these artifacts". - -The next step is creating the execution root. Since Bazel has the option to read -source packages from different locations in the file system (`--package_path`), -it needs to provide locally executed actions with a full source tree. This is -handled by the class `SymlinkForest` and works by taking note of every target -used in the analysis phase and building up a single directory tree that symlinks -every package with a used target from its actual location. An alternative would -be to pass the correct paths to commands (taking `--package_path` into account). -This is undesirable because: - -* It changes action command lines when a package is moved from a package path - entry to another (used to be a common occurrence) -* It results in different command lines if an action is run remotely than if - it's run locally -* It requires a command line transformation specific to the tool in use - (consider the difference between such as Java classpaths and C++ include paths) -* Changing the command line of an action invalidates its action cache entry -* `--package_path` is slowly and steadily being deprecated - -Then, Bazel starts traversing the action graph (the bipartite, directed graph -composed of actions and their input and output artifacts) and running actions. -The execution of each action is represented by an instance of the `SkyValue` -class `ActionExecutionValue`. - -Since running an action is expensive, we have a few layers of caching that can -be hit behind Skyframe: - -* `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts - of `ActionExecutionFunction` cheap -* The local action cache contains data about the state of the file system -* Remote execution systems usually also contain their own cache +This is when Bazel actually starts running build actions, such as commands that produce outputs. + +The first thing Bazel does after the analysis phase is to determine what Artifacts need to be built. The logic for this is encoded in `TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the configured targets on the command line and the contents of a special output group for the explicit purpose of expressing "if this target is on the command line, build these artifacts". + +The next step is creating the execution root. Since Bazel has the option to read source packages from different locations in the file system (`--package_path`), it needs to provide locally executed actions with a full source tree. This is handled by the class `SymlinkForest` and works by taking note of every target used in the analysis phase and building up a single directory tree that symlinks every package with a used target from its actual location. An alternative would be to pass the correct paths to commands (taking `--package_path` into account). This is undesirable because: + +- It changes action command lines when a package is moved from a package path entry to another (used to be a common occurrence) +- It results in different command lines if an action is run remotely than if it's run locally +- It requires a command line transformation specific to the tool in use (consider the difference between such as Java classpaths and C++ include paths) +- Changing the command line of an action invalidates its action cache entry +- `--package_path` is slowly and steadily being deprecated + +Then, Bazel starts traversing the action graph (the bipartite, directed graph composed of actions and their input and output artifacts) and running actions. The execution of each action is represented by an instance of the `SkyValue` class `ActionExecutionValue`. + +Since running an action is expensive, we have a few layers of caching that can be hit behind Skyframe: + +- `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts of `ActionExecutionFunction` cheap +- The local action cache contains data about the state of the file system +- Remote execution systems usually also contain their own cache ### The local action cache -This cache is another layer that sits behind Skyframe; even if an action is -re-executed in Skyframe, it can still be a hit in the local action cache. It -represents the state of the local file system and it's serialized to disk which -means that when one starts up a new Bazel server, one can get local action cache -hits even though the Skyframe graph is empty. +This cache is another layer that sits behind Skyframe; even if an action is re-executed in Skyframe, it can still be a hit in the local action cache. It represents the state of the local file system and it's serialized to disk which means that when one starts up a new Bazel server, one can get local action cache hits even though the Skyframe graph is empty. -This cache is checked for hits using the method -`ActionCacheChecker.getTokenIfNeedToExecute()` . +This cache is checked for hits using the method `ActionCacheChecker.getTokenIfNeedToExecute()` . -Contrary to its name, it's a map from the path of a derived artifact to the -action that emitted it. The action is described as: +Contrary to its name, it's a map from the path of a derived artifact to the action that emitted it. The action is described as: -1. The set of its input and output files and their checksum -2. Its "action key", which is usually the command line that was executed, but - in general, represents everything that's not captured by the checksum of the - input files (such as for `FileWriteAction`, it's the checksum of the data - that's written) +1. The set of its input and output files and their checksum +2. Its "action key", which is usually the command line that was executed, but in general, represents everything that's not captured by the checksum of the input files (such as for `FileWriteAction`, it's the checksum of the data that's written) -There is also a highly experimental "top-down action cache" that is still under -development, which uses transitive hashes to avoid going to the cache as many -times. +There is also a highly experimental "top-down action cache" that is still under development, which uses transitive hashes to avoid going to the cache as many times. ### Input discovery and input pruning -Some actions are more complicated than just having a set of inputs. Changes to -the set of inputs of an action come in two forms: - -* An action may discover new inputs before its execution or decide that some - of its inputs are not actually necessary. The canonical example is C++, - where it's better to make an educated guess about what header files a C++ - file uses from its transitive closure so that we don't heed to send every - file to remote executors; therefore, we have an option not to register every - header file as an "input", but scan the source file for transitively - included headers and only mark those header files as inputs that are - mentioned in `#include` statements (we overestimate so that we don't need to - implement a full C preprocessor) This option is currently hard-wired to - "false" in Bazel and is only used at Google. -* An action may realize that some files were not used during its execution. In - C++, this is called ".d files": the compiler tells which header files were - used after the fact, and in order to avoid the embarrassment of having worse - incrementality than Make, Bazel makes use of this fact. This offers a better - estimate than the include scanner because it relies on the compiler. +Some actions are more complicated than just having a set of inputs. Changes to the set of inputs of an action come in two forms: + +- An action may discover new inputs before its execution or decide that some of its inputs are not actually necessary. The canonical example is C++, where it's better to make an educated guess about what header files a C++ file uses from its transitive closure so that we don't heed to send every file to remote executors; therefore, we have an option not to register every header file as an "input", but scan the source file for transitively included headers and only mark those header files as inputs that are mentioned in `#include` statements (we overestimate so that we don't need to implement a full C preprocessor) This option is currently hard-wired to "false" in Bazel and is only used at Google. +- An action may realize that some files were not used during its execution. In C++, this is called ".d files": the compiler tells which header files were used after the fact, and in order to avoid the embarrassment of having worse incrementality than Make, Bazel makes use of this fact. This offers a better estimate than the include scanner because it relies on the compiler. These are implemented using methods on Action: -1. `Action.discoverInputs()` is called. It should return a nested set of - Artifacts that are determined to be required. These must be source artifacts - so that there are no dependency edges in the action graph that don't have an - equivalent in the configured target graph. -2. The action is executed by calling `Action.execute()`. -3. At the end of `Action.execute()`, the action can call - `Action.updateInputs()` to tell Bazel that not all of its inputs were - needed. This can result in incorrect incremental builds if a used input is - reported as unused. +1. `Action.discoverInputs()` is called. It should return a nested set of Artifacts that are determined to be required. These must be source artifacts so that there are no dependency edges in the action graph that don't have an equivalent in the configured target graph. +2. The action is executed by calling `Action.execute()`. +3. At the end of `Action.execute()`, the action can call `Action.updateInputs()` to tell Bazel that not all of its inputs were needed. This can result in incorrect incremental builds if a used input is reported as unused. -When an action cache returns a hit on a fresh Action instance (such as created -after a server restart), Bazel calls `updateInputs()` itself so that the set of -inputs reflects the result of input discovery and pruning done before. +When an action cache returns a hit on a fresh Action instance (such as created after a server restart), Bazel calls `updateInputs()` itself so that the set of inputs reflects the result of input discovery and pruning done before. -Starlark actions can make use of the facility to declare some inputs as unused -using the `unused_inputs_list=` argument of -`ctx.actions.run()`. +Starlark actions can make use of the facility to declare some inputs as unused using the `unused_inputs_list=` argument of `ctx.actions.run()`. ### Various ways to run actions: Strategies/ActionContexts -Some actions can be run in different ways. For example, a command line can be -executed locally, locally but in various kinds of sandboxes, or remotely. The -concept that embodies this is called an `ActionContext` (or `Strategy`, since we -successfully went only halfway with a rename...) +Some actions can be run in different ways. For example, a command line can be executed locally, locally but in various kinds of sandboxes, or remotely. The concept that embodies this is called an `ActionContext` (or `Strategy`, since we successfully went only halfway with a rename...) The life cycle of an action context is as follows: -1. When the execution phase is started, `BlazeModule` instances are asked what - action contexts they have. This happens in the constructor of - `ExecutionTool`. Action context types are identified by a Java `Class` - instance that refers to a sub-interface of `ActionContext` and which - interface the action context must implement. -2. The appropriate action context is selected from the available ones and is - forwarded to `ActionExecutionContext` and `BlazeExecutor` . -3. Actions request contexts using `ActionExecutionContext.getContext()` and - `BlazeExecutor.getStrategy()` (there should really be only one way to do - it…) - -Strategies are free to call other strategies to do their jobs; this is used, for -example, in the dynamic strategy that starts actions both locally and remotely, -then uses whichever finishes first. - -One notable strategy is the one that implements persistent worker processes -(`WorkerSpawnStrategy`). The idea is that some tools have a long startup time -and should therefore be reused between actions instead of starting one anew for -every action (This does represent a potential correctness issue, since Bazel -relies on the promise of the worker process that it doesn't carry observable -state between individual requests) - -If the tool changes, the worker process needs to be restarted. Whether a worker -can be reused is determined by computing a checksum for the tool used using -`WorkerFilesHash`. It relies on knowing which inputs of the action represent -part of the tool and which represent inputs; this is determined by the creator -of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are -counted as parts of the tool. +1. When the execution phase is started, `BlazeModule` instances are asked what action contexts they have. This happens in the constructor of `ExecutionTool`. Action context types are identified by a Java `Class` instance that refers to a sub-interface of `ActionContext` and which interface the action context must implement. +2. The appropriate action context is selected from the available ones and is forwarded to `ActionExecutionContext` and `BlazeExecutor` . +3. Actions request contexts using `ActionExecutionContext.getContext()` and `BlazeExecutor.getStrategy()` (there should really be only one way to do it…) + +Strategies are free to call other strategies to do their jobs; this is used, for example, in the dynamic strategy that starts actions both locally and remotely, then uses whichever finishes first. + +One notable strategy is the one that implements persistent worker processes (`WorkerSpawnStrategy`). The idea is that some tools have a long startup time and should therefore be reused between actions instead of starting one anew for every action (This does represent a potential correctness issue, since Bazel relies on the promise of the worker process that it doesn't carry observable state between individual requests) + +If the tool changes, the worker process needs to be restarted. Whether a worker can be reused is determined by computing a checksum for the tool used using `WorkerFilesHash`. It relies on knowing which inputs of the action represent part of the tool and which represent inputs; this is determined by the creator of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are counted as parts of the tool. More information about strategies (or action contexts!): -* Information about various strategies for running actions is available - [here](https://jmmv.dev/2019/12/bazel-strategies.html). -* Information about the dynamic strategy, one where we run an action both - locally and remotely to see whichever finishes first is available - [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). -* Information about the intricacies of executing actions locally is available - [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). +- Information about various strategies for running actions is available [here](https://jmmv.dev/2019/12/bazel-strategies.html). +- Information about the dynamic strategy, one where we run an action both locally and remotely to see whichever finishes first is available [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). +- Information about the intricacies of executing actions locally is available [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). ### The local resource manager -Bazel _can_ run many actions in parallel. The number of local actions that -_should_ be run in parallel differs from action to action: the more resources an -action requires, the less instances should be running at the same time to avoid -overloading the local machine. +Bazel *can* run many actions in parallel. The number of local actions that *should* be run in parallel differs from action to action: the more resources an action requires, the less instances should be running at the same time to avoid overloading the local machine. -This is implemented in the class `ResourceManager`: each action has to be -annotated with an estimate of the local resources it requires in the form of a -`ResourceSet` instance (CPU and RAM). Then when action contexts do something -that requires local resources, they call `ResourceManager.acquireResources()` -and are blocked until the required resources are available. +This is implemented in the class `ResourceManager`: each action has to be annotated with an estimate of the local resources it requires in the form of a `ResourceSet` instance (CPU and RAM). Then when action contexts do something that requires local resources, they call `ResourceManager.acquireResources()` and are blocked until the required resources are available. -A more detailed description of local resource management is available -[here](https://jmmv.dev/2019/12/bazel-local-resources.html). +A more detailed description of local resource management is available [here](https://jmmv.dev/2019/12/bazel-local-resources.html). ### The structure of the output directory -Each action requires a separate place in the output directory where it places -its outputs. The location of derived artifacts is usually as follows: +Each action requires a separate place in the output directory where it places its outputs. The location of derived artifacts is usually as follows: ``` -$EXECROOT/bazel-out//bin// +$EXECROOT/bazel-out/<configuration>/bin/<package>/<artifact name> ``` -How is the name of the directory that is associated with a particular -configuration determined? There are two conflicting desirable properties: - -1. If two configurations can occur in the same build, they should have - different directories so that both can have their own version of the same - action; otherwise, if the two configurations disagree about such as the command - line of an action producing the same output file, Bazel doesn't know which - action to choose (an "action conflict") -2. If two configurations represent "roughly" the same thing, they should have - the same name so that actions executed in one can be reused for the other if - the command lines match: for example, changes to the command line options to - the Java compiler should not result in C++ compile actions being re-run. - -So far, we have not come up with a principled way of solving this problem, which -has similarities to the problem of configuration trimming. A longer discussion -of options is available -[here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). -The main problematic areas are Starlark rules (whose authors usually aren't -intimately familiar with Bazel) and aspects, which add another dimension to the -space of things that can produce the "same" output file. - -The current approach is that the path segment for the configuration is -`-` with various suffixes added so that configuration -transitions implemented in Java don't result in action conflicts. In addition, a -checksum of the set of Starlark configuration transitions is added so that users -can't cause action conflicts. It is far from perfect. This is implemented in -`OutputDirectories.buildMnemonic()` and relies on each configuration fragment -adding its own part to the name of the output directory. +How is the name of the directory that is associated with a particular configuration determined? There are two conflicting desirable properties: + +1. If two configurations can occur in the same build, they should have different directories so that both can have their own version of the same action; otherwise, if the two configurations disagree about such as the command line of an action producing the same output file, Bazel doesn't know which action to choose (an "action conflict") +2. If two configurations represent "roughly" the same thing, they should have the same name so that actions executed in one can be reused for the other if the command lines match: for example, changes to the command line options to the Java compiler should not result in C++ compile actions being re-run. + +So far, we have not come up with a principled way of solving this problem, which has similarities to the problem of configuration trimming. A longer discussion of options is available [here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). The main problematic areas are Starlark rules (whose authors usually aren't intimately familiar with Bazel) and aspects, which add another dimension to the space of things that can produce the "same" output file. + +The current approach is that the path segment for the configuration is `<CPU>-<compilation mode>` with various suffixes added so that configuration transitions implemented in Java don't result in action conflicts. In addition, a checksum of the set of Starlark configuration transitions is added so that users can't cause action conflicts. It is far from perfect. This is implemented in `OutputDirectories.buildMnemonic()` and relies on each configuration fragment adding its own part to the name of the output directory. ## Tests Bazel has rich support for running tests. It supports: -* Running tests remotely (if a remote execution backend is available) -* Running tests multiple times in parallel (for deflaking or gathering timing - data) -* Sharding tests (splitting test cases in same test over multiple processes - for speed) -* Re-running flaky tests -* Grouping tests into test suites +- Running tests remotely (if a remote execution backend is available) +- Running tests multiple times in parallel (for deflaking or gathering timing data) +- Sharding tests (splitting test cases in same test over multiple processes for speed) +- Re-running flaky tests +- Grouping tests into test suites -Tests are regular configured targets that have a TestProvider, which describes -how the test should be run: +Tests are regular configured targets that have a TestProvider, which describes how the test should be run: -* The artifacts whose building result in the test being run. This is a "cache - status" file that contains a serialized `TestResultData` message -* The number of times the test should be run -* The number of shards the test should be split into -* Some parameters about how the test should be run (such as the test timeout) +- The artifacts whose building result in the test being run. This is a "cache status" file that contains a serialized `TestResultData` message +- The number of times the test should be run +- The number of shards the test should be split into +- Some parameters about how the test should be run (such as the test timeout) ### Determining which tests to run Determining which tests are run is an elaborate process. -First, during target pattern parsing, test suites are recursively expanded. The -expansion is implemented in `TestsForTargetPatternFunction`. A somewhat -surprising wrinkle is that if a test suite declares no tests, it refers to -_every_ test in its package. This is implemented in `Package.beforeBuild()` by -adding an implicit attribute called `$implicit_tests` to test suite rules. - -Then, tests are filtered for size, tags, timeout and language according to the -command line options. This is implemented in `TestFilter` and is called from -`TargetPatternPhaseFunction.determineTests()` during target parsing and the -result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason -why rule attributes which can be filtered for are not configurable is that this -happens before the analysis phase, therefore, the configuration is not -available. - -This is then processed further in `BuildView.createResult()`: targets whose -analysis failed are filtered out and tests are split into exclusive and -non-exclusive tests. It's then put into `AnalysisResult`, which is how -`ExecutionTool` knows which tests to run. - -In order to lend some transparency to this elaborate process, the `tests()` -query operator (implemented in `TestsFunction`) is available to tell which tests -are run when a particular target is specified on the command line. It's -unfortunately a reimplementation, so it probably deviates from the above in -multiple subtle ways. +First, during target pattern parsing, test suites are recursively expanded. The expansion is implemented in `TestsForTargetPatternFunction`. A somewhat surprising wrinkle is that if a test suite declares no tests, it refers to *every* test in its package. This is implemented in `Package.beforeBuild()` by adding an implicit attribute called `$implicit_tests` to test suite rules. + +Then, tests are filtered for size, tags, timeout and language according to the command line options. This is implemented in `TestFilter` and is called from `TargetPatternPhaseFunction.determineTests()` during target parsing and the result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason why rule attributes which can be filtered for are not configurable is that this happens before the analysis phase, therefore, the configuration is not available. + +This is then processed further in `BuildView.createResult()`: targets whose analysis failed are filtered out and tests are split into exclusive and non-exclusive tests. It's then put into `AnalysisResult`, which is how `ExecutionTool` knows which tests to run. + +In order to lend some transparency to this elaborate process, the `tests()` query operator (implemented in `TestsFunction`) is available to tell which tests are run when a particular target is specified on the command line. It's unfortunately a reimplementation, so it probably deviates from the above in multiple subtle ways. ### Running tests -The way the tests are run is by requesting cache status artifacts. This then -results in the execution of a `TestRunnerAction`, which eventually calls the -`TestActionContext` chosen by the `--test_strategy` command line option that -runs the test in the requested way. - -Tests are run according to an elaborate protocol that uses environment variables -to tell tests what's expected from them. A detailed description of what Bazel -expects from tests and what tests can expect from Bazel is available -[here](/reference/test-encyclopedia). At the -simplest, an exit code of 0 means success, anything else means failure. - -In addition to the cache status file, each test process emits a number of other -files. They are put in the "test log directory" which is the subdirectory called -`testlogs` of the output directory of the target configuration: - -* `test.xml`, a JUnit-style XML file detailing the individual test cases in - the test shard -* `test.log`, the console output of the test. stdout and stderr are not - separated. -* `test.outputs`, the "undeclared outputs directory"; this is used by tests - that want to output files in addition to what they print to the terminal. - -There are two things that can happen during test execution that cannot during -building regular targets: exclusive test execution and output streaming. - -Some tests need to be executed in exclusive mode, for example not in parallel with -other tests. This can be elicited either by adding `tags=["exclusive"]` to the -test rule or running the test with `--test_strategy=exclusive` . Each exclusive -test is run by a separate Skyframe invocation requesting the execution of the -test after the "main" build. This is implemented in -`SkyframeExecutor.runExclusiveTest()`. - -Unlike regular actions, whose terminal output is dumped when the action -finishes, the user can request the output of tests to be streamed so that they -get informed about the progress of a long-running test. This is specified by the -`--test_output=streamed` command line option and implies exclusive test -execution so that outputs of different tests are not interspersed. - -This is implemented in the aptly-named `StreamedTestOutput` class and works by -polling changes to the `test.log` file of the test in question and dumping new -bytes to the terminal where Bazel rules. - -Results of the executed tests are available on the event bus by observing -various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). -They are dumped to the Build Event Protocol and they are emitted to the console -by `AggregatingTestListener`. +The way the tests are run is by requesting cache status artifacts. This then results in the execution of a `TestRunnerAction`, which eventually calls the `TestActionContext` chosen by the `--test_strategy` command line option that runs the test in the requested way. + +Tests are run according to an elaborate protocol that uses environment variables to tell tests what's expected from them. A detailed description of what Bazel expects from tests and what tests can expect from Bazel is available [here](/reference/test-encyclopedia). At the simplest, an exit code of 0 means success, anything else means failure. + +In addition to the cache status file, each test process emits a number of other files. They are put in the "test log directory" which is the subdirectory called `testlogs` of the output directory of the target configuration: + +- `test.xml`, a JUnit-style XML file detailing the individual test cases in the test shard +- `test.log`, the console output of the test. stdout and stderr are not separated. +- `test.outputs`, the "undeclared outputs directory"; this is used by tests that want to output files in addition to what they print to the terminal. + +There are two things that can happen during test execution that cannot during building regular targets: exclusive test execution and output streaming. + +Some tests need to be executed in exclusive mode, for example not in parallel with other tests. This can be elicited either by adding `tags=["exclusive"]` to the test rule or running the test with `--test_strategy=exclusive` . Each exclusive test is run by a separate Skyframe invocation requesting the execution of the test after the "main" build. This is implemented in `SkyframeExecutor.runExclusiveTest()`. + +Unlike regular actions, whose terminal output is dumped when the action finishes, the user can request the output of tests to be streamed so that they get informed about the progress of a long-running test. This is specified by the `--test_output=streamed` command line option and implies exclusive test execution so that outputs of different tests are not interspersed. + +This is implemented in the aptly-named `StreamedTestOutput` class and works by polling changes to the `test.log` file of the test in question and dumping new bytes to the terminal where Bazel rules. + +Results of the executed tests are available on the event bus by observing various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). They are dumped to the Build Event Protocol and they are emitted to the console by `AggregatingTestListener`. ### Coverage collection -Coverage is reported by the tests in LCOV format in the files -`bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . - -To collect coverage, each test execution is wrapped in a script called -`collect_coverage.sh` . - -This script sets up the environment of the test to enable coverage collection -and determine where the coverage files are written by the coverage runtime(s). -It then runs the test. A test may itself run multiple subprocesses and consist -of parts written in multiple different programming languages (with separate -coverage collection runtimes). The wrapper script is responsible for converting -the resulting files to LCOV format if necessary, and merges them into a single -file. - -The interposition of `collect_coverage.sh` is done by the test strategies and -requires `collect_coverage.sh` to be on the inputs of the test. This is -accomplished by the implicit attribute `:coverage_support` which is resolved to -the value of the configuration flag `--coverage_support` (see -`TestConfiguration.TestOptions.coverageSupport`) - -Some languages do offline instrumentation, meaning that the coverage -instrumentation is added at compile time (such as C++) and others do online -instrumentation, meaning that coverage instrumentation is added at execution -time. - -Another core concept is _baseline coverage_. This is the coverage of a library, -binary, or test if no code in it was run. The problem it solves is that if you -want to compute the test coverage for a binary, it is not enough to merge the -coverage of all of the tests because there may be code in the binary that is not -linked into any test. Therefore, what we do is to emit a coverage file for every -binary which contains only the files we collect coverage for with no covered -lines. The default baseline coverage file for a target is at -`bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are -encouraged to generate their own baseline coverage files with more meaningful -content than just the names of the source files. - -We track two groups of files for coverage collection for each rule: the set of -instrumented files and the set of instrumentation metadata files. - -The set of instrumented files is just that, a set of files to instrument. For -online coverage runtimes, this can be used at runtime to decide which files to -instrument. It is also used to implement baseline coverage. - -The set of instrumentation metadata files is the set of extra files a test needs -to generate the LCOV files Bazel requires from it. In practice, this consists of -runtime-specific files; for example, gcc emits .gcno files during compilation. -These are added to the set of inputs of test actions if coverage mode is -enabled. - -Whether or not coverage is being collected is stored in the -`BuildConfiguration`. This is handy because it is an easy way to change the test -action and the action graph depending on this bit, but it also means that if -this bit is flipped, all targets need to be re-analyzed (some languages, such as -C++ require different compiler options to emit code that can collect coverage, -which mitigates this issue somewhat, since then a re-analysis is needed anyway). - -The coverage support files are depended on through labels in an implicit -dependency so that they can be overridden by the invocation policy, which allows -them to differ between the different versions of Bazel. Ideally, these -differences would be removed, and we standardized on one of them. - -We also generate a "coverage report" which merges the coverage collected for -every test in a Bazel invocation. This is handled by -`CoverageReportActionFactory` and is called from `BuildView.createResult()` . It -gets access to the tools it needs by looking at the `:coverage_report_generator` -attribute of the first test that is executed. +Coverage is reported by the tests in LCOV format in the files `bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . + +To collect coverage, each test execution is wrapped in a script called `collect_coverage.sh` . + +This script sets up the environment of the test to enable coverage collection and determine where the coverage files are written by the coverage runtime(s). It then runs the test. A test may itself run multiple subprocesses and consist of parts written in multiple different programming languages (with separate coverage collection runtimes). The wrapper script is responsible for converting the resulting files to LCOV format if necessary, and merges them into a single file. + +The interposition of `collect_coverage.sh` is done by the test strategies and requires `collect_coverage.sh` to be on the inputs of the test. This is accomplished by the implicit attribute `:coverage_support` which is resolved to the value of the configuration flag `--coverage_support` (see `TestConfiguration.TestOptions.coverageSupport`) + +Some languages do offline instrumentation, meaning that the coverage instrumentation is added at compile time (such as C++) and others do online instrumentation, meaning that coverage instrumentation is added at execution time. + +Another core concept is *baseline coverage*. This is the coverage of a library, binary, or test if no code in it was run. The problem it solves is that if you want to compute the test coverage for a binary, it is not enough to merge the coverage of all of the tests because there may be code in the binary that is not linked into any test. Therefore, what we do is to emit a coverage file for every binary which contains only the files we collect coverage for with no covered lines. The default baseline coverage file for a target is at `bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are encouraged to generate their own baseline coverage files with more meaningful content than just the names of the source files. + +We track two groups of files for coverage collection for each rule: the set of instrumented files and the set of instrumentation metadata files. + +The set of instrumented files is just that, a set of files to instrument. For online coverage runtimes, this can be used at runtime to decide which files to instrument. It is also used to implement baseline coverage. + +The set of instrumentation metadata files is the set of extra files a test needs to generate the LCOV files Bazel requires from it. In practice, this consists of runtime-specific files; for example, gcc emits .gcno files during compilation. These are added to the set of inputs of test actions if coverage mode is enabled. + +Whether or not coverage is being collected is stored in the `BuildConfiguration`. This is handy because it is an easy way to change the test action and the action graph depending on this bit, but it also means that if this bit is flipped, all targets need to be re-analyzed (some languages, such as C++ require different compiler options to emit code that can collect coverage, which mitigates this issue somewhat, since then a re-analysis is needed anyway). + +The coverage support files are depended on through labels in an implicit dependency so that they can be overridden by the invocation policy, which allows them to differ between the different versions of Bazel. Ideally, these differences would be removed, and we standardized on one of them. + +We also generate a "coverage report" which merges the coverage collected for every test in a Bazel invocation. This is handled by `CoverageReportActionFactory` and is called from `BuildView.createResult()` . It gets access to the tools it needs by looking at the `:coverage_report_generator` attribute of the first test that is executed. ## The query engine -Bazel has a -[little language](/query/guide) -used to ask it various things about various graphs. The following query kinds -are provided: - -* `bazel query` is used to investigate the target graph -* `bazel cquery` is used to investigate the configured target graph -* `bazel aquery` is used to investigate the action graph - -Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. -Additional additional query functions can be done by subclassing `QueryFunction` -. In order to allow streaming query results, instead of collecting them to some -data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which -calls it for results it wants to return. - -The result of a query can be emitted in various ways: labels, labels and rule -classes, XML, protobuf and so on. These are implemented as subclasses of -`OutputFormatter`. - -A subtle requirement of some query output formats (proto, definitely) is that -Bazel needs to emit _all _the information that package loading provides so that -one can diff the output and determine whether a particular target has changed. -As a consequence, attribute values need to be serializable, which is why there -are only so few attribute types without any attributes having complex Starlark -values. The usual workaround is to use a label, and attach the complex -information to the rule with that label. It's not a very satisfying workaround -and it would be very nice to lift this requirement. +Bazel has a [little language](/query/guide) used to ask it various things about various graphs. The following query kinds are provided: + +- `bazel query` is used to investigate the target graph +- `bazel cquery` is used to investigate the configured target graph +- `bazel aquery` is used to investigate the action graph + +Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. Additional additional query functions can be done by subclassing `QueryFunction` . In order to allow streaming query results, instead of collecting them to some data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which calls it for results it wants to return. + +The result of a query can be emitted in various ways: labels, labels and rule classes, XML, protobuf and so on. These are implemented as subclasses of `OutputFormatter`. + +A subtle requirement of some query output formats (proto, definitely) is that Bazel needs to emit \_all \_the information that package loading provides so that one can diff the output and determine whether a particular target has changed. As a consequence, attribute values need to be serializable, which is why there are only so few attribute types without any attributes having complex Starlark values. The usual workaround is to use a label, and attach the complex information to the rule with that label. It's not a very satisfying workaround and it would be very nice to lift this requirement. ## The module system -Bazel can be extended by adding modules to it. Each module must subclass -`BlazeModule` (the name is a relic of the history of Bazel when it used to be -called Blaze) and gets information about various events during the execution of -a command. +Bazel can be extended by adding modules to it. Each module must subclass `BlazeModule` (the name is a relic of the history of Bazel when it used to be called Blaze) and gets information about various events during the execution of a command. -They are mostly used to implement various pieces of "non-core" functionality -that only some versions of Bazel (such as the one we use at Google) need: +They are mostly used to implement various pieces of "non-core" functionality that only some versions of Bazel (such as the one we use at Google) need: -* Interfaces to remote execution systems -* New commands +- Interfaces to remote execution systems +- New commands -The set of extension points `BlazeModule` offers is somewhat haphazard. Don't -use it as an example of good design principles. +The set of extension points `BlazeModule` offers is somewhat haphazard. Don't use it as an example of good design principles. ## The event bus -The main way BlazeModules communicate with the rest of Bazel is by an event bus -(`EventBus`): a new instance is created for every build, various parts of Bazel -can post events to it and modules can register listeners for the events they are -interested in. For example, the following things are represented as events: +The main way BlazeModules communicate with the rest of Bazel is by an event bus (`EventBus`): a new instance is created for every build, various parts of Bazel can post events to it and modules can register listeners for the events they are interested in. For example, the following things are represented as events: -* The list of build targets to be built has been determined - (`TargetParsingCompleteEvent`) -* The top-level configurations have been determined - (`BuildConfigurationEvent`) -* A target was built, successfully or not (`TargetCompleteEvent`) -* A test was run (`TestAttempt`, `TestSummary`) +- The list of build targets to be built has been determined (`TargetParsingCompleteEvent`) +- The top-level configurations have been determined (`BuildConfigurationEvent`) +- A target was built, successfully or not (`TargetCompleteEvent`) +- A test was run (`TestAttempt`, `TestSummary`) -Some of these events are represented outside of Bazel in the -[Build Event Protocol](/remote/bep) -(they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things -outside the Bazel process to observe the build. They are accessible either as a -file that contains protocol messages or Bazel can connect to a server (called -the Build Event Service) to stream events. +Some of these events are represented outside of Bazel in the [Build Event Protocol](/remote/bep) (they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things outside the Bazel process to observe the build. They are accessible either as a file that contains protocol messages or Bazel can connect to a server (called the Build Event Service) to stream events. -This is implemented in the `build.lib.buildeventservice` and -`build.lib.buildeventstream` Java packages. +This is implemented in the `build.lib.buildeventservice` and `build.lib.buildeventstream` Java packages. ## External repositories -Note: The information in this section is out of date, as code in this area has -undergone extensive change in the past couple of years. Please refer to -[external dependencies overview](/external/overview) for more up-to-date -information. +Note: The information in this section is out of date, as code in this area has undergone extensive change in the past couple of years. Please refer to [external dependencies overview](/external/overview) for more up-to-date information. -Whereas Bazel was originally designed to be used in a monorepo (a single source -tree containing everything one needs to build), Bazel lives in a world where -this is not necessarily true. "External repositories" are an abstraction used to -bridge these two worlds: they represent code that is necessary for the build but -is not in the main source tree. +Whereas Bazel was originally designed to be used in a monorepo (a single source tree containing everything one needs to build), Bazel lives in a world where this is not necessarily true. "External repositories" are an abstraction used to bridge these two worlds: they represent code that is necessary for the build but is not in the main source tree. ### The WORKSPACE file -The set of external repositories is determined by parsing the WORKSPACE file. -For example, a declaration like this: +The set of external repositories is determined by parsing the WORKSPACE file. For example, a declaration like this: ``` local_repository(name="foo", path="/foo/bar") ``` -Results in the repository called `@foo` being available. Where this gets -complicated is that one can define new repository rules in Starlark files, which -can then be used to load new Starlark code, which can be used to define new -repository rules and so on… +Results in the repository called `@foo` being available. Where this gets complicated is that one can define new repository rules in Starlark files, which can then be used to load new Starlark code, which can be used to define new repository rules and so on… -To handle this case, the parsing of the WORKSPACE file (in -`WorkspaceFileFunction`) is split up into chunks delineated by `load()` -statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and -computing `WorkspaceFileFunction` until index X means evaluating it until the -Xth `load()` statement. +To handle this case, the parsing of the WORKSPACE file (in `WorkspaceFileFunction`) is split up into chunks delineated by `load()` statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and computing `WorkspaceFileFunction` until index X means evaluating it until the Xth `load()` statement. ### Fetching repositories -Before the code of the repository is available to Bazel, it needs to be -_fetched_. This results in Bazel creating a directory under -`$OUTPUT_BASE/external/`. +Before the code of the repository is available to Bazel, it needs to be *fetched*. This results in Bazel creating a directory under `$OUTPUT_BASE/external/<repository name>`. Fetching the repository happens in the following steps: -1. `PackageLookupFunction` realizes that it needs a repository and creates a - `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` -2. `RepositoryLoaderFunction` forwards the request to - `RepositoryDelegatorFunction` for unclear reasons (the code says it's to - avoid re-downloading things in case of Skyframe restarts, but it's not a - very solid reasoning) -3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to - fetch by iterating over the chunks of the WORKSPACE file until the requested - repository is found -4. The appropriate `RepositoryFunction` is found that implements the repository - fetching; it's either the Starlark implementation of the repository or a - hard-coded map for repositories that are implemented in Java. - -There are various layers of caching since fetching a repository can be very -expensive: - -1. There is a cache for downloaded files that is keyed by their checksum - (`RepositoryCache`). This requires the checksum to be available in the - WORKSPACE file, but that's good for hermeticity anyway. This is shared by - every Bazel server instance on the same workstation, regardless of which - workspace or output base they are running in. -2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` - that contains a checksum of the rule that was used to fetch it. If the Bazel - server restarts but the checksum does not change, it's not re-fetched. This - is implemented in `RepositoryDelegatorFunction.DigestWriter` . -3. The `--distdir` command line option designates another cache that is used to - look up artifacts to be downloaded. This is useful in enterprise settings - where Bazel should not fetch random things from the Internet. This is - implemented by `DownloadManager` . - -Once a repository is downloaded, the artifacts in it are treated as source -artifacts. This poses a problem because Bazel usually checks for up-to-dateness -of source artifacts by calling stat() on them, and these artifacts are also -invalidated when the definition of the repository they are in changes. Thus, -`FileStateValue`s for an artifact in an external repository need to depend on -their external repository. This is handled by `ExternalFilesHelper`. +1. `PackageLookupFunction` realizes that it needs a repository and creates a `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` +2. `RepositoryLoaderFunction` forwards the request to `RepositoryDelegatorFunction` for unclear reasons (the code says it's to avoid re-downloading things in case of Skyframe restarts, but it's not a very solid reasoning) +3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to fetch by iterating over the chunks of the WORKSPACE file until the requested repository is found +4. The appropriate `RepositoryFunction` is found that implements the repository fetching; it's either the Starlark implementation of the repository or a hard-coded map for repositories that are implemented in Java. + +There are various layers of caching since fetching a repository can be very expensive: + +1. There is a cache for downloaded files that is keyed by their checksum (`RepositoryCache`). This requires the checksum to be available in the WORKSPACE file, but that's good for hermeticity anyway. This is shared by every Bazel server instance on the same workstation, regardless of which workspace or output base they are running in. +2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` that contains a checksum of the rule that was used to fetch it. If the Bazel server restarts but the checksum does not change, it's not re-fetched. This is implemented in `RepositoryDelegatorFunction.DigestWriter` . +3. The `--distdir` command line option designates another cache that is used to look up artifacts to be downloaded. This is useful in enterprise settings where Bazel should not fetch random things from the Internet. This is implemented by `DownloadManager` . + +Once a repository is downloaded, the artifacts in it are treated as source artifacts. This poses a problem because Bazel usually checks for up-to-dateness of source artifacts by calling stat() on them, and these artifacts are also invalidated when the definition of the repository they are in changes. Thus, `FileStateValue`s for an artifact in an external repository need to depend on their external repository. This is handled by `ExternalFilesHelper`. ### Repository mappings -It can happen that multiple repositories want to depend on the same repository, -but in different versions (this is an instance of the "diamond dependency -problem"). For example, if two binaries in separate repositories in the build -want to depend on Guava, they will presumably both refer to Guava with labels -starting `@guava//` and expect that to mean different versions of it. - -Therefore, Bazel allows one to re-map external repository labels so that the -string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the -repository of one binary and another Guava repository (such as `@guava2//`) the -repository of the other. - -Alternatively, this can also be used to **join** diamonds. If a repository -depends on `@guava1//`, and another depends on `@guava2//`, repository mapping -allows one to re-map both repositories to use a canonical `@guava//` repository. - -The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute -of individual repository definitions. It then appears in Skyframe as a member of -`WorkspaceFileValue`, where it is plumbed to: - -* `Package.Builder.repositoryMapping` which is used to transform label-valued - attributes of rules in the package by - `RuleClass.populateRuleAttributeValues()` -* `Package.repositoryMapping` which is used in the analysis phase (for - resolving things like `$(location)` which are not parsed in the loading - phase) -* `BzlLoadFunction` for resolving labels in load() statements +It can happen that multiple repositories want to depend on the same repository, but in different versions (this is an instance of the "diamond dependency problem"). For example, if two binaries in separate repositories in the build want to depend on Guava, they will presumably both refer to Guava with labels starting `@guava//` and expect that to mean different versions of it. + +Therefore, Bazel allows one to re-map external repository labels so that the string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the repository of one binary and another Guava repository (such as `@guava2//`) the repository of the other. + +Alternatively, this can also be used to **join** diamonds. If a repository depends on `@guava1//`, and another depends on `@guava2//`, repository mapping allows one to re-map both repositories to use a canonical `@guava//` repository. + +The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute of individual repository definitions. It then appears in Skyframe as a member of `WorkspaceFileValue`, where it is plumbed to: + +- `Package.Builder.repositoryMapping` which is used to transform label-valued attributes of rules in the package by `RuleClass.populateRuleAttributeValues()` +- `Package.repositoryMapping` which is used in the analysis phase (for resolving things like `$(location)` which are not parsed in the loading phase) +- `BzlLoadFunction` for resolving labels in load() statements ## JNI bits -The server of Bazel is _mostly_ written in Java. The exception is the parts that -Java cannot do by itself or couldn't do by itself when we implemented it. This -is mostly limited to interaction with the file system, process control and -various other low-level things. +The server of Bazel is *mostly* written in Java. The exception is the parts that Java cannot do by itself or couldn't do by itself when we implemented it. This is mostly limited to interaction with the file system, process control and various other low-level things. -The C++ code lives under src/main/native and the Java classes with native -methods are: +The C++ code lives under src/main/native and the Java classes with native methods are: -* `NativePosixFiles` and `NativePosixFileSystem` -* `ProcessUtils` -* `WindowsFileOperations` and `WindowsFileProcesses` -* `com.google.devtools.build.lib.platform` +- `NativePosixFiles` and `NativePosixFileSystem` +- `ProcessUtils` +- `WindowsFileOperations` and `WindowsFileProcesses` +- `com.google.devtools.build.lib.platform` ## Console output -Emitting console output seems like a simple thing, but the confluence of running -multiple processes (sometimes remotely), fine-grained caching, the desire to -have a nice and colorful terminal output and having a long-running server makes -it non-trivial. - -Right after the RPC call comes in from the client, two `RpcOutputStream` -instances are created (for stdout and stderr) that forward the data printed into -them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) -pair). Anything that needs to be printed on the console goes through these -streams. Then these streams are handed over to -`BlazeCommandDispatcher.execExclusively()`. - -Output is by default printed with ANSI escape sequences. When these are not -desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In -addition, `System.out` and `System.err` are redirected to these output streams. -This is so that debugging information can be printed using -`System.err.println()` and still end up in the terminal output of the client -(which is different from that of the server). Care is taken that if a process -produces binary output (such as `bazel query --output=proto`), no munging of stdout -takes place. - -Short messages (errors, warnings and the like) are expressed through the -`EventHandler` interface. Notably, these are different from what one posts to -the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, -warning, info, and a few others) and they may have a `Location` (the place in -the source code that caused the event to happen). - -Some `EventHandler` implementations store the events they received. This is used -to replay information to the UI caused by various kinds of cached processing, -for example, the warnings emitted by a cached configured target. - -Some `EventHandler`s also allow posting events that eventually find their way to -the event bus (regular `Event`s do _not _appear there). These are -implementations of `ExtendedEventHandler` and their main use is to replay cached -`EventBus` events. These `EventBus` events all implement `Postable`, but not -everything that is posted to `EventBus` necessarily implements this interface; -only those that are cached by an `ExtendedEventHandler` (it would be nice and -most of the things do; it's not enforced, though) - -Terminal output is _mostly_ emitted through `UiEventHandler`, which is -responsible for all the fancy output formatting and progress reporting Bazel -does. It has two inputs: - -* The event bus -* The event stream piped into it through Reporter - -The only direct connection the command execution machinery (for example the rest of -Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, -which allows direct access to these streams. It's only used when a command needs -to dump large amounts of possible binary data (such as `bazel query`). +Emitting console output seems like a simple thing, but the confluence of running multiple processes (sometimes remotely), fine-grained caching, the desire to have a nice and colorful terminal output and having a long-running server makes it non-trivial. + +Right after the RPC call comes in from the client, two `RpcOutputStream` instances are created (for stdout and stderr) that forward the data printed into them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) pair). Anything that needs to be printed on the console goes through these streams. Then these streams are handed over to `BlazeCommandDispatcher.execExclusively()`. + +Output is by default printed with ANSI escape sequences. When these are not desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In addition, `System.out` and `System.err` are redirected to these output streams. This is so that debugging information can be printed using `System.err.println()` and still end up in the terminal output of the client (which is different from that of the server). Care is taken that if a process produces binary output (such as `bazel query --output=proto`), no munging of stdout takes place. + +Short messages (errors, warnings and the like) are expressed through the `EventHandler` interface. Notably, these are different from what one posts to the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, warning, info, and a few others) and they may have a `Location` (the place in the source code that caused the event to happen). + +Some `EventHandler` implementations store the events they received. This is used to replay information to the UI caused by various kinds of cached processing, for example, the warnings emitted by a cached configured target. + +Some `EventHandler`s also allow posting events that eventually find their way to the event bus (regular `Event`s do \_not \_appear there). These are implementations of `ExtendedEventHandler` and their main use is to replay cached `EventBus` events. These `EventBus` events all implement `Postable`, but not everything that is posted to `EventBus` necessarily implements this interface; only those that are cached by an `ExtendedEventHandler` (it would be nice and most of the things do; it's not enforced, though) + +Terminal output is *mostly* emitted through `UiEventHandler`, which is responsible for all the fancy output formatting and progress reporting Bazel does. It has two inputs: + +- The event bus +- The event stream piped into it through Reporter + +The only direct connection the command execution machinery (for example the rest of Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, which allows direct access to these streams. It's only used when a command needs to dump large amounts of possible binary data (such as `bazel query`). ## Profiling Bazel -Bazel is fast. Bazel is also slow, because builds tend to grow until just the -edge of what's bearable. For this reason, Bazel includes a profiler which can be -used to profile builds and Bazel itself. It's implemented in a class that's -aptly named `Profiler`. It's turned on by default, although it records only -abridged data so that its overhead is tolerable; The command line -`--record_full_profiler_data` makes it record everything it can. - -It emits a profile in the Chrome profiler format; it's best viewed in Chrome. -It's data model is that of task stacks: one can start tasks and end tasks and -they are supposed to be neatly nested within each other. Each Java thread gets -its own task stack. **TODO:** How does this work with actions and -continuation-passing style? - -The profiler is started and stopped in `BlazeRuntime.initProfiler()` and -`BlazeRuntime.afterCommand()` respectively and attempts to be live for as long -as possible so that we can profile everything. To add something to the profile, -call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure -represents the end of the task. It's best used with try-with-resources -statements. - -We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on -and it mostly records maximum heap sizes and GC behavior. +Bazel is fast. Bazel is also slow, because builds tend to grow until just the edge of what's bearable. For this reason, Bazel includes a profiler which can be used to profile builds and Bazel itself. It's implemented in a class that's aptly named `Profiler`. It's turned on by default, although it records only abridged data so that its overhead is tolerable; The command line `--record_full_profiler_data` makes it record everything it can. + +It emits a profile in the Chrome profiler format; it's best viewed in Chrome. It's data model is that of task stacks: one can start tasks and end tasks and they are supposed to be neatly nested within each other. Each Java thread gets its own task stack. **TODO:** How does this work with actions and continuation-passing style? + +The profiler is started and stopped in `BlazeRuntime.initProfiler()` and `BlazeRuntime.afterCommand()` respectively and attempts to be live for as long as possible so that we can profile everything. To add something to the profile, call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure represents the end of the task. It's best used with try-with-resources statements. + +We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on and it mostly records maximum heap sizes and GC behavior. ## Testing Bazel -Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and -ones that only run the analysis phase. We call the former "integration tests" -and the latter "unit tests", although they are more like integration tests that -are, well, less integrated. We also have some actual unit tests, where they are -necessary. +Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and ones that only run the analysis phase. We call the former "integration tests" and the latter "unit tests", although they are more like integration tests that are, well, less integrated. We also have some actual unit tests, where they are necessary. Of integration tests, we have two kinds: -1. Ones implemented using a very elaborate bash test framework under - `src/test/shell` -2. Ones implemented in Java. These are implemented as subclasses of - `BuildIntegrationTestCase` - -`BuildIntegrationTestCase` is the preferred integration testing framework as it -is well-equipped for most testing scenarios. As it is a Java framework, it -provides debuggability and seamless integration with many common development -tools. There are many examples of `BuildIntegrationTestCase` classes in the -Bazel repository. - -Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a -scratch file system you can use to write `BUILD` files, then various helper -methods can request configured targets, change the configuration and assert -various things about the result of the analysis. +1. Ones implemented using a very elaborate bash test framework under `src/test/shell` +2. Ones implemented in Java. These are implemented as subclasses of `BuildIntegrationTestCase` + +`BuildIntegrationTestCase` is the preferred integration testing framework as it is well-equipped for most testing scenarios. As it is a Java framework, it provides debuggability and seamless integration with many common development tools. There are many examples of `BuildIntegrationTestCase` classes in the Bazel repository. + +Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a scratch file system you can use to write `BUILD` files, then various helper methods can request configured targets, change the configuration and assert various things about the result of the analysis. diff --git a/contribute/design-documents.mdx b/contribute/design-documents.mdx index 1fe70b9d..47405539 100644 --- a/contribute/design-documents.mdx +++ b/contribute/design-documents.mdx @@ -2,145 +2,94 @@ title: 'Design Documents' --- - - -If you're planning to add, change, or remove a user-facing feature, or make a -*significant architectural change* to Bazel, you **must** write a design -document and have it reviewed before you can submit the change. +If you're planning to add, change, or remove a user-facing feature, or make a *significant architectural change* to Bazel, you **must** write a design document and have it reviewed before you can submit the change. Here are some examples of significant changes: -* Addition or deletion of native build rules -* Breaking-changes to native rules -* Changes to a native build rule semantics that affect the behavior of more - than a single rule -* Changes to Bazel's rule definition API -* Changes to the APIs that Bazel uses to connect to other systems -* Changes to the Starlark language, semantics, or APIs -* Changes that could have a pervasive effect on Bazel performance or memory - usage (for better or for worse) -* Changes to widely used internal APIs -* Changes to flags and command-line interface. +- Addition or deletion of native build rules +- Breaking-changes to native rules +- Changes to a native build rule semantics that affect the behavior of more than a single rule +- Changes to Bazel's rule definition API +- Changes to the APIs that Bazel uses to connect to other systems +- Changes to the Starlark language, semantics, or APIs +- Changes that could have a pervasive effect on Bazel performance or memory usage (for better or for worse) +- Changes to widely used internal APIs +- Changes to flags and command-line interface. ## Reasons for design reviews -When you write a design document, you can coordinate with other Bazel developers -and seek guidance from Bazel's core team. For example, when a proposal adds, -removes, or modifies any function or object available in BUILD, MODULE.bazel, or -bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. -Design documents are reviewed before submission because: - -* Bazel is a very complex system; seemingly innocuous local changes can have - significant global consequences. -* The team gets many feature requests from users; such requests need to be - evaluated not only for technical feasibility but importance with regards to - other feature requests. -* Bazel features are frequently implemented by people outside the core team; - such contributors have widely varying levels of Bazel expertise. -* The Bazel team itself has varying levels of expertise; no single team member - has a complete understanding of every corner of Bazel. -* Changes to Bazel must account for backward compatibility and avoid breaking - changes. +When you write a design document, you can coordinate with other Bazel developers and seek guidance from Bazel's core team. For example, when a proposal adds, removes, or modifies any function or object available in BUILD, MODULE.bazel, or bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. Design documents are reviewed before submission because: + +- Bazel is a very complex system; seemingly innocuous local changes can have significant global consequences. +- The team gets many feature requests from users; such requests need to be evaluated not only for technical feasibility but importance with regards to other feature requests. +- Bazel features are frequently implemented by people outside the core team; such contributors have widely varying levels of Bazel expertise. +- The Bazel team itself has varying levels of expertise; no single team member has a complete understanding of every corner of Bazel. +- Changes to Bazel must account for backward compatibility and avoid breaking changes. Bazel's design review policy helps to maximize the likelihood that: -* all feature requests get a baseline level of scrutiny. -* the right people will weigh in on designs before we've invested in an - implementation that may not work. +- all feature requests get a baseline level of scrutiny. +- the right people will weigh in on designs before we've invested in an implementation that may not work. -To help you get started, take a look at the design documents in the -[Bazel Proposals Repository](https://github.com/bazelbuild/proposals). -Designs are works in progress, so implementation details can change over time -and with feedback. The published design documents capture the initial design, -and *not* the ongoing changes as designs are implemented. Always go to the -documentation for descriptions of current Bazel functionality. +To help you get started, take a look at the design documents in the [Bazel Proposals Repository](https://github.com/bazelbuild/proposals). Designs are works in progress, so implementation details can change over time and with feedback. The published design documents capture the initial design, and *not* the ongoing changes as designs are implemented. Always go to the documentation for descriptions of current Bazel functionality. ## Contributor Workflow -As a contributor, you can write a design document, send pull requests and -request reviewers for your proposal. +As a contributor, you can write a design document, send pull requests and request reviewers for your proposal. ### Write the design document All design documents must have a header that includes: -* author -* date of last major change -* list of reviewers, including one (and only one) - [lead reviewer](#lead-reviewer) -* current status (_draft_, _in review_, _approved_, _rejected_, - _being implemented_, _implemented_) -* link to discussion thread (_to be added after the announcement_) +- author +- date of last major change +- list of reviewers, including one (and only one) [lead reviewer](#lead-reviewer) +- current status (*draft*, *in review*, *approved*, *rejected*, *being implemented*, *implemented*) +- link to discussion thread (*to be added after the announcement*) -The document can be written either [as a world-readable Google Doc](#gdocs) -or [using Markdown](#markdown). Read below about for a -[Markdown / Google Docs comparison](#markdown-versus-gdocs). +The document can be written either [as a world-readable Google Doc](#gdocs) or [using Markdown](#markdown). Read below about for a [Markdown / Google Docs comparison](#markdown-versus-gdocs). -Proposals that have a user-visible impact must have a section documenting the -impact on backward compatibility (and a rollout plan if needed). +Proposals that have a user-visible impact must have a section documenting the impact on backward compatibility (and a rollout plan if needed). ### Create a Pull Request -Share your design doc by creating a pull request (PR) to add the document to -[the design index](https://github.com/bazelbuild/proposals). Add -your markdown file or a document link to your PR. +Share your design doc by creating a pull request (PR) to add the document to [the design index](https://github.com/bazelbuild/proposals). Add your markdown file or a document link to your PR. -When possible, [choose a lead reviewer](#lead-reviewer). -and cc other reviewers. If you don't choose a lead reviewer, a Bazel -maintainer will assign one to your PR. +When possible, [choose a lead reviewer](#lead-reviewer). and cc other reviewers. If you don't choose a lead reviewer, a Bazel maintainer will assign one to your PR. -After you create your PR, reviewers can make preliminary comments during the -code review. For example, the lead reviewer can suggest extra reviewers, or -point out missing information. The lead reviewer approves the PR when they -believe the review process can start. This doesn't mean the proposal is perfect -or will be approved; it means that the proposal contains enough information to -start the discussion. +After you create your PR, reviewers can make preliminary comments during the code review. For example, the lead reviewer can suggest extra reviewers, or point out missing information. The lead reviewer approves the PR when they believe the review process can start. This doesn't mean the proposal is perfect or will be approved; it means that the proposal contains enough information to start the discussion. ### Announce the new proposal -Send an announcement to -[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when -the PR is submitted. +Send an announcement to [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when the PR is submitted. -You may copy other groups (for example, -[bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), -to get feedback from Bazel end-users). +You may copy other groups (for example, [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), to get feedback from Bazel end-users). ### Iterate with reviewers -Anyone interested can comment on your proposal. Try to answer questions, -clarify the proposal, and address concerns. +Anyone interested can comment on your proposal. Try to answer questions, clarify the proposal, and address concerns. -Discussion should happen on the announcement thread. If the proposal is in a -Google Doc, comments may be used instead (Note that anonymous comments are -allowed). +Discussion should happen on the announcement thread. If the proposal is in a Google Doc, comments may be used instead (Note that anonymous comments are allowed). ### Update the status -Create a new PR to update the status of the proposal, when iteration is -complete. Send the PR to the same lead reviewer and cc the other reviewers. +Create a new PR to update the status of the proposal, when iteration is complete. Send the PR to the same lead reviewer and cc the other reviewers. -To officially accept the proposal, the lead reviewer approves the PR after -ensuring that the other reviewers agree with the decision. +To officially accept the proposal, the lead reviewer approves the PR after ensuring that the other reviewers agree with the decision. -There must be at least 1 week between the first announcement and the approval of -a proposal. This ensures that users had enough time to read the document and -share their concerns. +There must be at least 1 week between the first announcement and the approval of a proposal. This ensures that users had enough time to read the document and share their concerns. -Implementation can begin before the proposal is accepted, for example as a -proof-of-concept or an experimentation. However, you cannot submit the change -before the review is complete. +Implementation can begin before the proposal is accepted, for example as a proof-of-concept or an experimentation. However, you cannot submit the change before the review is complete. ### Choosing a lead reviewer A lead reviewer should be a domain expert who is: -* Knowledgeable of the relevant subsystems -* Objective and capable of providing constructive feedback -* Available for the entire review period to lead the process +- Knowledgeable of the relevant subsystems +- Objective and capable of providing constructive feedback +- Available for the entire review period to lead the process -Consider checking the contacts for various [team -labels](/contribute/maintainers-guide#team-labels). +Consider checking the contacts for various [team labels](/contribute/maintainers-guide#team-labels). ## Markdown vs Google Docs @@ -148,49 +97,34 @@ Decide what works best for you, since both are accepted. Benefits of using Google Docs: -* Effective for brainstorming, since it is easy to get started with. -* Collaborative editing. -* Quick iteration. -* Easy way to suggest edits. +- Effective for brainstorming, since it is easy to get started with. +- Collaborative editing. +- Quick iteration. +- Easy way to suggest edits. Benefits of using Markdown files: -* Clean URLs for linking. -* Explicit record of revisions. -* No forgetting to set up access rights before publicizing a link. -* Easily searchable with search engines. -* Future-proof: Plain text is not at the mercy of any specific tool - and doesn't require an Internet connection. -* It is possible to update them even if the author is not around anymore. -* They can be processed automatically (update/detect dead links, fetch - list of authors, etc.). +- Clean URLs for linking. +- Explicit record of revisions. +- No forgetting to set up access rights before publicizing a link. +- Easily searchable with search engines. +- Future-proof: Plain text is not at the mercy of any specific tool and doesn't require an Internet connection. +- It is possible to update them even if the author is not around anymore. +- They can be processed automatically (update/detect dead links, fetch list of authors, etc.). -You can choose to first iterate on a Google Doc, and then convert it to -Markdown for posterity. +You can choose to first iterate on a Google Doc, and then convert it to Markdown for posterity. ### Using Google Docs -For consistency, use the [Bazel design doc template]( -https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). -It includes the necessary header and creates visual -consistency with other Bazel related documents. To do that, click on **File** > -**Make a copy** or click this link to [make a copy of the design doc -template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). +For consistency, use the [Bazel design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). It includes the necessary header and creates visual consistency with other Bazel related documents. To do that, click on **File** \> **Make a copy** or click this link to [make a copy of the design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). -To make your document readable to the world, click on -**Share** > **Advanced** > **Change…**, and -choose "On - Anyone with the link". If you allow comments on the document, -anyone can comment anonymously, even without a Google account. +To make your document readable to the world, click on **Share** \> **Advanced** \> **Change…**, and choose "On - Anyone with the link". If you allow comments on the document, anyone can comment anonymously, even without a Google account. ### Using Markdown -Documents are stored on GitHub and use the -[GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) -([Specification](https://github.github.com/gfm/)). +Documents are stored on GitHub and use the [GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) ([Specification](https://github.github.com/gfm/)). -Create a PR to update an existing document. Significant changes should be -reviewed by the document reviewers. Trivial changes (such as typos, formatting) -can be approved by anyone. +Create a PR to update an existing document. Significant changes should be reviewed by the document reviewers. Trivial changes (such as typos, formatting) can be approved by anyone. ## Reviewer workflow @@ -198,57 +132,41 @@ A reviewer comments, reviews and approves design documents. ### General reviewer responsibilities -You're responsible for reviewing design documents, asking for additional -information if needed, and approving a design that passes the review process. +You're responsible for reviewing design documents, asking for additional information if needed, and approving a design that passes the review process. #### When you receive a new proposal -1. Take a quick look at the document. -1. Comment if critical information is missing, or if the design doesn't fit - with the goals of the project. -1. Suggest additional reviewers. -1. Approve the PR when it is ready for review. +1. Take a quick look at the document. +2. Comment if critical information is missing, or if the design doesn't fit with the goals of the project. +3. Suggest additional reviewers. +4. Approve the PR when it is ready for review. #### During the review process -1. Engage in a dialogue with the design author about issues that are problematic - or require clarification. -1. If appropriate, invite comments from non-reviewers who should be aware of - the design. -1. Decide which comments must be addressed by the author as a prerequisite to - approval. -1. Write "LGTM" (_Looks Good To Me_) in the discussion thread when you are - happy with the current state of the proposal. +1. Engage in a dialogue with the design author about issues that are problematic or require clarification. +2. If appropriate, invite comments from non-reviewers who should be aware of the design. +3. Decide which comments must be addressed by the author as a prerequisite to approval. +4. Write "LGTM" (*Looks Good To Me*) in the discussion thread when you are happy with the current state of the proposal. -Follow this process for all design review requests. Do not approve designs -affecting Bazel if they are not in the -[design index](https://github.com/bazelbuild/proposals). +Follow this process for all design review requests. Do not approve designs affecting Bazel if they are not in the [design index](https://github.com/bazelbuild/proposals). ### Lead reviewer responsibilities -You're responsible for making the go / no-go decision on implementation -of a pending design. If you're not able to do this, you should identify a -suitable delegate (reassign the PR to the delegate), or reassign the bug to a -Bazel manager for further disposition. +You're responsible for making the go / no-go decision on implementation of a pending design. If you're not able to do this, you should identify a suitable delegate (reassign the PR to the delegate), or reassign the bug to a Bazel manager for further disposition. #### During the review process -1. Ensure that the comment and design iteration process moves forward - constructively. -1. Prior to approval, ensure that concerns from other reviewers have been - resolved. +1. Ensure that the comment and design iteration process moves forward constructively. +2. Prior to approval, ensure that concerns from other reviewers have been resolved. #### After approval by all reviewers -1. Make sure there has been at least 1 week since the announcement on the - mailing list. -1. Make sure the PR updates the status. -1. Approve the PR sent by the proposal author. +1. Make sure there has been at least 1 week since the announcement on the mailing list. +2. Make sure the PR updates the status. +3. Approve the PR sent by the proposal author. #### Rejecting designs -1. Make sure the PR author sends a PR; or send them a PR. -1. The PR updates the status of the document. -1. Add a comment to the document explaining why the design can't be approved in - its current state, and outlining next steps, if any (such as "revisit invalid - assumptions and resubmit"). +1. Make sure the PR author sends a PR; or send them a PR. +2. The PR updates the status of the document. +3. Add a comment to the document explaining why the design can't be approved in its current state, and outlining next steps, if any (such as "revisit invalid assumptions and resubmit"). diff --git a/contribute/docs.mdx b/contribute/docs.mdx index cc240cc4..940b20e2 100644 --- a/contribute/docs.mdx +++ b/contribute/docs.mdx @@ -2,42 +2,26 @@ title: 'Contribute to Bazel documentation' --- - - -Thank you for contributing to Bazel's documentation! There are a few ways to -help create better docs for our community. +Thank you for contributing to Bazel's documentation! There are a few ways to help create better docs for our community. ## Documentation types This site includes a few types of content. - - *Narrative documentation*, which is written by technical writers and - engineers. Most of this site is narrative documentation that covers - conceptual and task-based guides. - - *Reference documentation*, which is generated documentation from code comments. - You can't make changes to the reference doc pages directly, but instead need - to change their source. +- *Narrative documentation*, which is written by technical writers and engineers. Most of this site is narrative documentation that covers conceptual and task-based guides. +- *Reference documentation*, which is generated documentation from code comments. You can't make changes to the reference doc pages directly, but instead need to change their source. ## Documentation infrastructure -Bazel documentation is served from Google and the source files are mirrored in -Bazel's GitHub repository. You can make changes to the source files in GitHub. -If approved, you can merge the changes and a Bazel maintainer will update the -website source to publish your updates. - +Bazel documentation is served from Google and the source files are mirrored in Bazel's GitHub repository. You can make changes to the source files in GitHub. If approved, you can merge the changes and a Bazel maintainer will update the website source to publish your updates. ## Small changes -You can approach small changes, such as fixing errors or typos, in a couple of -ways. +You can approach small changes, such as fixing errors or typos, in a couple of ways. - - **Pull request**. You can create a pull request in GitHub with the - [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. - - **Bug**. You can file a bug with details and suggested changes and the Bazel - documentation owners will make the update. +- **Pull request**. You can create a pull request in GitHub with the [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. +- **Bug**. You can file a bug with details and suggested changes and the Bazel documentation owners will make the update. ## Large changes -If you want to make substantial changes to existing documentation or propose -new documentation, you can either create a pull request or start with a Google -doc and contact the Bazel Owners to collaborate. +If you want to make substantial changes to existing documentation or propose new documentation, you can either create a pull request or start with a Google doc and contact the Bazel Owners to collaborate. diff --git a/contribute/index.mdx b/contribute/index.mdx index ee667729..18b12854 100644 --- a/contribute/index.mdx +++ b/contribute/index.mdx @@ -2,57 +2,41 @@ title: 'Contributing to Bazel' --- - - There are many ways to help the Bazel project and ecosystem. ## Provide feedback -As you use Bazel, you may find things that can be improved. -You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) -when: +As you use Bazel, you may find things that can be improved. You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) when: - - Bazel crashes or you encounter a bug that can [only be resolved using `bazel - clean`](/run/build#correct-incremental-rebuilds). - - The documentation is incomplete or unclear. You can also report issues - from the page you are viewing by using the "Create issue" - link at the top right corner of the page. - - An error message could be improved. +- Bazel crashes or you encounter a bug that can [only be resolved using `bazel clean`](/run/build#correct-incremental-rebuilds). +- The documentation is incomplete or unclear. You can also report issues from the page you are viewing by using the "Create issue" link at the top right corner of the page. +- An error message could be improved. ## Participate in the community You can engage with the Bazel community by: - - Answering questions [on Stack Overflow]( - https://stackoverflow.com/questions/tagged/bazel). - - Helping other users [on Slack](https://slack.bazel.build). - - Improving documentation or [contributing examples]( - https://github.com/bazelbuild/examples). - - Sharing your experience or your tips, for example, on a blog or social media. +- Answering questions [on Stack Overflow](https://stackoverflow.com/questions/tagged/bazel). +- Helping other users [on Slack](https://slack.bazel.build). +- Improving documentation or [contributing examples](https://github.com/bazelbuild/examples). +- Sharing your experience or your tips, for example, on a blog or social media. ## Contribute code -Bazel is a large project and making a change to the Bazel source code -can be difficult. +Bazel is a large project and making a change to the Bazel source code can be difficult. You can contribute to the Bazel ecosystem by: - - Helping rules maintainers by contributing pull requests. - - Creating new rules and open-sourcing them. - - Contributing to Bazel-related tools, for example, migration tools. - - Improving Bazel integration with other IDEs and tools. +- Helping rules maintainers by contributing pull requests. +- Creating new rules and open-sourcing them. +- Contributing to Bazel-related tools, for example, migration tools. +- Improving Bazel integration with other IDEs and tools. -Before making a change, [create a GitHub -issue](http://github.com/bazelbuild/bazel/issues) -or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). +Before making a change, [create a GitHub issue](http://github.com/bazelbuild/bazel/issues) or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). -The most helpful contributions fix bugs or add features (as opposed -to stylistic, refactoring, or "cleanup" changes). Your change should -include tests and documentation, keeping in mind backward-compatibility, -portability, and the impact on memory usage and performance. +The most helpful contributions fix bugs or add features (as opposed to stylistic, refactoring, or "cleanup" changes). Your change should include tests and documentation, keeping in mind backward-compatibility, portability, and the impact on memory usage and performance. -To learn about how to submit a change, see the -[patch acceptance process](/contribute/patch-acceptance). +To learn about how to submit a change, see the [patch acceptance process](/contribute/patch-acceptance). ## Bazel's code description @@ -60,23 +44,19 @@ Bazel has a large codebase with code in multiple locations. See the [codebase gu Bazel is organized as follows: -* Client code is in `src/main/cpp` and provides the command-line interface. -* Protocol buffers are in `src/main/protobuf`. -* Server code is in `src/main/java` and `src/test/java`. - * Core code which is mostly composed of [SkyFrame](/reference/skyframe) - and some utilities. - * Built-in rules are in `com.google.devtools.build.lib.rules` and in - `com.google.devtools.build.lib.bazel.rules`. You might want to read about - the [Challenges of Writing Rules](/rules/challenges) first. -* Java native interfaces are in `src/main/native`. -* Various tooling for language support are described in the list in the - [compiling Bazel](/install/compile-source) section. +- Client code is in `src/main/cpp` and provides the command-line interface. + +- Protocol buffers are in `src/main/protobuf`. + +- Server code is in `src/main/java` and `src/test/java`. + + - Core code which is mostly composed of [SkyFrame](/reference/skyframe) and some utilities. + - Built-in rules are in `com.google.devtools.build.lib.rules` and in `com.google.devtools.build.lib.bazel.rules`. You might want to read about the [Challenges of Writing Rules](/rules/challenges) first. + +- Java native interfaces are in `src/main/native`. +- Various tooling for language support are described in the list in the [compiling Bazel](/install/compile-source) section. ### Searching Bazel's source code -To quickly search through Bazel's source code, use -[Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's -repositories, branches, and files. You can also view history, diffs, and blame -information. To learn more, see the -[Bazel Code Search User Guide](/contribute/search). +To quickly search through Bazel's source code, use [Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's repositories, branches, and files. You can also view history, diffs, and blame information. To learn more, see the [Bazel Code Search User Guide](/contribute/search). diff --git a/contribute/maintainers-guide.mdx b/contribute/maintainers-guide.mdx index 9d745afb..b98a408e 100644 --- a/contribute/maintainers-guide.mdx +++ b/contribute/maintainers-guide.mdx @@ -2,205 +2,141 @@ title: 'Guide for Bazel Maintainers' --- - - This is a guide for the maintainers of the Bazel open source project. -If you are looking to contribute to Bazel, please read [Contributing to -Bazel](/contribute) instead. +If you are looking to contribute to Bazel, please read [Contributing to Bazel](/contribute) instead. The objectives of this page are to: -1. Serve as the maintainers' source of truth for the project’s contribution - process. -1. Set expectations between the community contributors and the project - maintainers. +1. Serve as the maintainers' source of truth for the project’s contribution process. +2. Set expectations between the community contributors and the project maintainers. -Bazel's [core group of contributors](/contribute/policy) has dedicated -subteams to manage aspects of the open source project. These are: +Bazel's [core group of contributors](/contribute/policy) has dedicated subteams to manage aspects of the open source project. These are: -* **Release Process**: Manage Bazel's release process. -* **Green Team**: Grow a healthy ecosystem of rules and tools. -* **Developer Experience Gardeners**: Encourage external contributions, review - issues and pull requests, and make our development workflow more open. +- **Release Process**: Manage Bazel's release process. +- **Green Team**: Grow a healthy ecosystem of rules and tools. +- **Developer Experience Gardeners**: Encourage external contributions, review issues and pull requests, and make our development workflow more open. ## Releases -* [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) -* [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) +- [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) +- [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) ## Continuous Integration -Read the Green team's guide to Bazel's CI infrastructure on the -[bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) -repository. +Read the Green team's guide to Bazel's CI infrastructure on the [bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) repository. ## Lifecycle of an Issue -1. A user creates an issue by choosing one of the -[issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) - and it enters the pool of [unreviewed open - issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). -1. A member on the Developer Experience (DevEx) subteam rotation reviews the - issue. - 1. If the issue is **not a bug** or a **feature request**, the DevEx member - will usually close the issue and redirect the user to - [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and - [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for - higher visibility on the question. - 1. If the issue belongs in one of the rules repositories owned by the - community, like [rules_apple](https://github.com.bazelbuild/rules_apple), - the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) - to the correct repository. - 1. If the issue is vague or has missing information, the DevEx member will - assign the issue back to the user to request for more information before - continuing. This usually occurs when the user does not choose the right - [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) - or provides incomplete information. -1. After reviewing the issue, the DevEx member decides if the issue requires - immediate attention. If it does, they will assign the **P0** - [priority](#priority) label and an owner from the list of team leads. -1. The DevEx member assigns the `untriaged` label and exactly one [team - label](#team-labels) for routing. -1. The DevEx member also assigns exactly one `type:` label, such as `type: bug` - or `type: feature request`, according to the type of the issue. -1. For platform-specific issues, the DevEx member assigns one `platform:` label, - such as `platform:apple` for Mac-specific issues. -1. If the issue is low priority and can be worked on by a new community - contributor, the DevEx member assigns the `good first issue` label. -At this stage, the issue enters the pool of [untriaged open -issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). - -Each Bazel subteam will triage all issues under labels they own, preferably on a -weekly basis. The subteam will review and evaluate the issue and provide a -resolution, if possible. If you are an owner of a team label, see [this section -](#label-own) for more information. +1. A user creates an issue by choosing one of the [issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) and it enters the pool of [unreviewed open issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93\&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). + +2. A member on the Developer Experience (DevEx) subteam rotation reviews the issue. + + 1. If the issue is **not a bug** or a **feature request**, the DevEx member will usually close the issue and redirect the user to [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for higher visibility on the question. + 2. If the issue belongs in one of the rules repositories owned by the community, like [rules\_apple](https://github.com.bazelbuild/rules_apple), the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) to the correct repository. + 3. If the issue is vague or has missing information, the DevEx member will assign the issue back to the user to request for more information before continuing. This usually occurs when the user does not choose the right [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) or provides incomplete information. + +3. After reviewing the issue, the DevEx member decides if the issue requires immediate attention. If it does, they will assign the **P0** [priority](#priority) label and an owner from the list of team leads. + +4. The DevEx member assigns the `untriaged` label and exactly one [team label](#team-labels) for routing. + +5. The DevEx member also assigns exactly one `type:` label, such as `type: bug` or `type: feature request`, according to the type of the issue. + +6. For platform-specific issues, the DevEx member assigns one `platform:` label, such as `platform:apple` for Mac-specific issues. + +7. If the issue is low priority and can be worked on by a new community contributor, the DevEx member assigns the `good first issue` label. At this stage, the issue enters the pool of [untriaged open issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). + +Each Bazel subteam will triage all issues under labels they own, preferably on a weekly basis. The subteam will review and evaluate the issue and provide a resolution, if possible. If you are an owner of a team label, see [this section ](#label-own)for more information. When an issue is resolved, it can be closed. ## Lifecycle of a Pull Request 1. A user creates a pull request. -1. If you a member of a Bazel team and sending a PR against your own area, - you are responsible for assigning your team label and finding the best - reviewer. -1. Otherwise, during daily triage, a DevEx member assigns one - [team label](#team-labels) and the team's technical lead (TL) for routing. +2. If you a member of a Bazel team and sending a PR against your own area, you are responsible for assigning your team label and finding the best reviewer. +3. Otherwise, during daily triage, a DevEx member assigns one [team label](#team-labels) and the team's technical lead (TL) for routing. 1. The TL may optionally assign someone else to review the PR. -1. The assigned reviewer reviews the PR and works with the author until it is - approved or dropped. -1. If approved, the reviewer **imports** the PR's commit(s) into Google's - internal version control system for further tests. As Bazel is the same build - system used internally at Google, we need to test all PR commits against the - internal test suite. This is the reason why we do not merge PRs directly. -1. If the imported commit passes all internal tests, the commit will be squashed - and exported back out to GitHub. -1. When the commit merges into master, GitHub automatically closes the PR. - +4. The assigned reviewer reviews the PR and works with the author until it is approved or dropped. +5. If approved, the reviewer **imports** the PR's commit(s) into Google's internal version control system for further tests. As Bazel is the same build system used internally at Google, we need to test all PR commits against the internal test suite. This is the reason why we do not merge PRs directly. +6. If the imported commit passes all internal tests, the commit will be squashed and exported back out to GitHub. +7. When the commit merges into master, GitHub automatically closes the PR. ## My team owns a label. What should I do? -Subteams need to triage all issues in the [labels they own](#team-labels), -preferably on a weekly basis. +Subteams need to triage all issues in the [labels they own](#team-labels), preferably on a weekly basis. ### Issues 1. Filter the list of issues by your team label **and** the `untriaged` label. -1. Review the issue. -1. Identify a [priority level](#priority) and assign the label. - 1. The issue may have already been prioritized by the DevEx subteam if it's a - P0. Re-prioritize if needed. - 1. Each issue needs to have exactly one [priority label](#priority). If an - issue is either P0 or P1 we assume that is actively worked on. -1. Remove the `untriaged` label. +2. Review the issue. +3. Identify a [priority level](#priority) and assign the label. +4. The issue may have already been prioritized by the DevEx subteam if it's a P0. Re-prioritize if needed. +5. Each issue needs to have exactly one [priority label](#priority). If an issue is either P0 or P1 we assume that is actively worked on. +6. Remove the `untriaged` label. -Note that you need to be in the [bazelbuild -organization](https://github.com/bazelbuild) to be able to add or remove labels. +Note that you need to be in the [bazelbuild organization](https://github.com/bazelbuild) to be able to add or remove labels. ### Pull Requests 1. Filter the list of pull requests by your team label. -1. Review open pull requests. - 1. **Optional**: If you are assigned for the review but is not the right fit - for it, re-assign the appropriate reviewer to perform a code review. -1. Work with the pull request creator to complete a code review. -1. Approve the PR. -1. Ensure that all tests pass. -1. Import the patch to the internal version control system and run the internal - presubmits. -1. Submit the internal patch. If the patch submits and exports successfully, the - PR will be closed automatically by GitHub. +2. Review open pull requests. +3. **Optional**: If you are assigned for the review but is not the right fit for it, re-assign the appropriate reviewer to perform a code review. +4. Work with the pull request creator to complete a code review. +5. Approve the PR. +6. Ensure that all tests pass. +7. Import the patch to the internal version control system and run the internal presubmits. +8. Submit the internal patch. If the patch submits and exports successfully, the PR will be closed automatically by GitHub. ## Priority -The following definitions for priority will be used by the maintainers to triage -issues. - -* [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken - functionality that causes a Bazel release (minus release candidates) to be - unusable, or a downed service that severely impacts development of the Bazel - project. This includes regressions introduced in a new release that blocks a - significant number of users, or an incompatible breaking change that was not - compliant to the [Breaking - Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) - policy. No practical workaround exists. -* [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or - feature which should be addressed in the next release, or a serious issue that - impacts many users (including the development of the Bazel project), but a - practical workaround exists. Typically does not require immediate action. In - high demand and planned in the current quarter's roadmap. -* [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature - that should be addressed but we don't currently work on. Moderate live issue - in a released Bazel version that is inconvenient for a user that needs to be - addressed in an future release and/or an easy workaround exists. -* [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug - fix or enhancement with small impact. Not prioritized into Bazel roadmaps or - any imminent release, however community contributions are encouraged. -* [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect - or feature request that is unlikely to get closed. Can also be kept open for a - potential re-prioritization if more users are impacted. +The following definitions for priority will be used by the maintainers to triage issues. + +- [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken functionality that causes a Bazel release (minus release candidates) to be unusable, or a downed service that severely impacts development of the Bazel project. This includes regressions introduced in a new release that blocks a significant number of users, or an incompatible breaking change that was not compliant to the [Breaking Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) policy. No practical workaround exists. +- [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or feature which should be addressed in the next release, or a serious issue that impacts many users (including the development of the Bazel project), but a practical workaround exists. Typically does not require immediate action. In high demand and planned in the current quarter's roadmap. +- [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature that should be addressed but we don't currently work on. Moderate live issue in a released Bazel version that is inconvenient for a user that needs to be addressed in an future release and/or an easy workaround exists. +- [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug fix or enhancement with small impact. Not prioritized into Bazel roadmaps or any imminent release, however community contributions are encouraged. +- [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect or feature request that is unlikely to get closed. Can also be kept open for a potential re-prioritization if more users are impacted. ## Team labels -* [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team - * Contact: [ahumesky](https://github.com/ahumesky) -* [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues - * Contact: [meisterT](https://github.com/meisterT) -* [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI - * Contact: [meisterT](https://github.com/meisterT) -* [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags - * Contact: [gregestren](https://github.com/gregestren) -* [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc - * Contact: [haxorz](https://github.com/haxorz) -* [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team -* [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file - * Contact: [meteorcloudy](https://github.com/meteorcloudy) -* [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob - * Contact: [brandjon](https://github.com/brandjon) -* [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team - * Contact: [meisterT](https://github.com/meisterT) -* [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure - * Contact: [meteorcloudy](https://github.com/meteorcloudy) -* [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team - * Contact: [meisterT](https://github.com/meisterT) -* [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team - * Contact: [coeuvre](https://github.com/coeuvre) -* [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts - * Contact: [comius](https://github.com/comius) -* [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic - * Contact: [pzembrod](https://github.com/pzembrod) -* [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules - * Contact: [hvadehra](https://github.com/hvadehra) -* [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules - * Contact: [rickeylev](https://github.com/rickeylev) -* [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel - * Contact: [comius](https://github.com/comius) -* [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. - * Contact: [brandjon](https://github.com/brandjon) -* [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. - * Contact: [brandjon](https://github.com/brandjon) - -For new issues, we deprecated the `category: *` labels in favor of the team -labels. +- [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team + - Contact: [ahumesky](https://github.com/ahumesky) +- [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues + - Contact: [meisterT](https://github.com/meisterT) +- [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI + - Contact: [meisterT](https://github.com/meisterT) +- [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags + - Contact: [gregestren](https://github.com/gregestren) +- [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc + - Contact: [haxorz](https://github.com/haxorz) +- [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team +- [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file + - Contact: [meteorcloudy](https://github.com/meteorcloudy) +- [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob + - Contact: [brandjon](https://github.com/brandjon) +- [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team + - Contact: [meisterT](https://github.com/meisterT) +- [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure + - Contact: [meteorcloudy](https://github.com/meteorcloudy) +- [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team + - Contact: [meisterT](https://github.com/meisterT) +- [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team + - Contact: [coeuvre](https://github.com/coeuvre) +- [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts + - Contact: [comius](https://github.com/comius) +- [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic + - Contact: [pzembrod](https://github.com/pzembrod) +- [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules + - Contact: [hvadehra](https://github.com/hvadehra) +- [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules + - Contact: [rickeylev](https://github.com/rickeylev) +- [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel + - Contact: [comius](https://github.com/comius) +- [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. + - Contact: [brandjon](https://github.com/brandjon) +- [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. + - Contact: [brandjon](https://github.com/brandjon) + +For new issues, we deprecated the `category: *` labels in favor of the team labels. See the full list of labels [here](https://github.com/bazelbuild/bazel/labels). diff --git a/contribute/naming.mdx b/contribute/naming.mdx index 144b08af..806f7cd2 100644 --- a/contribute/naming.mdx +++ b/contribute/naming.mdx @@ -2,72 +2,42 @@ title: 'Naming a Bazel related project' --- +First, thank you for contributing to the Bazel ecosystem! Please reach out to the Bazel community on the [bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss) to share your project and its suggested name. - -First, thank you for contributing to the Bazel ecosystem! Please reach out to -the Bazel community on the -[bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss -) to share your project and its suggested name. - -If you are building a Bazel related tool or sharing your Skylark rules, -we recommend following these guidelines for the name of your project: +If you are building a Bazel related tool or sharing your Skylark rules, we recommend following these guidelines for the name of your project: ## Naming Starlark rules -See [Deploying new Starlark rules](/rules/deploying) -in the docs. +See [Deploying new Starlark rules](/rules/deploying) in the docs. ## Naming other Bazel related tools -This section applies if you are building a tool to enrich the Bazel ecosystem. -For example, a new IDE plugin or a new build system migrator. +This section applies if you are building a tool to enrich the Bazel ecosystem. For example, a new IDE plugin or a new build system migrator. -Picking a good name for your tool can be hard. If we’re not careful and use too -many codenames, the Bazel ecosystem could become very difficult to understand -for newcomers. +Picking a good name for your tool can be hard. If we’re not careful and use too many codenames, the Bazel ecosystem could become very difficult to understand for newcomers. Follow these guidelines for naming Bazel tools: -1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand -for our users, we should avoid confusing them with too many new names. - -2. Prefer **using a name that includes "Bazel"**: This helps to express that it -is a Bazel related tool, it also helps people find it with a search engine. +1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand for our users, we should avoid confusing them with too many new names. -3. Prefer **using names that are descriptive about what the tool is doing**: -Ideally, the name should not need a subtitle for users to have a first good -guess at what the tool does. Using english words separated by spaces is a good -way to achieve this. +2. Prefer **using a name that includes "Bazel"**: This helps to express that it is a Bazel related tool, it also helps people find it with a search engine. -4. **It is not a requirement to use a floral or food theme**: Bazel evokes -[basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to -look for a name that is a plant, food or that relates to "basil." +3. Prefer **using names that are descriptive about what the tool is doing**: Ideally, the name should not need a subtitle for users to have a first good guess at what the tool does. Using english words separated by spaces is a good way to achieve this. -5. **If your tool relates to another third party brand, use it only as a -descriptor**: For example, use "Bazel migrator for Cmake" instead of -"Cmake Bazel migrator". +4. **It is not a requirement to use a floral or food theme**: Bazel evokes [basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to look for a name that is a plant, food or that relates to "basil." -These guidelines also apply to the GitHub repository URL. Reading the repository -URL should help people understand what the tool does. Of course, the repository -name can be shorter and must use dashes instead of spaces and lower case letters. +5. **If your tool relates to another third party brand, use it only as a descriptor**: For example, use "Bazel migrator for Cmake" instead of "Cmake Bazel migrator". +These guidelines also apply to the GitHub repository URL. Reading the repository URL should help people understand what the tool does. Of course, the repository name can be shorter and must use dashes instead of spaces and lower case letters. Examples of good names: -* *Bazel for Eclipse*: Users will understand that if they want to use Bazel - with Eclipse, this is where they should be looking. It uses a third party brand - as a descriptor. -* *Bazel buildfarm*: A "buildfarm" is a - [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users - will understand that this project relates to building on servers. +- *Bazel for Eclipse*: Users will understand that if they want to use Bazel with Eclipse, this is where they should be looking. It uses a third party brand as a descriptor. +- *Bazel buildfarm*: A "buildfarm" is a [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users will understand that this project relates to building on servers. Examples of names to avoid: -* *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) - does not relate enough to the Bazel project. -* *Bazelizer*: The tool behind this name could do a lot of things, this name is - not descriptive enough. +- *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) does not relate enough to the Bazel project. +- *Bazelizer*: The tool behind this name could do a lot of things, this name is not descriptive enough. -Note that these recommendations are aligned with the -[guidelines](https://opensource.google.com/docs/releasing/preparing/#name) -Google uses when open sourcing a project. +Note that these recommendations are aligned with the [guidelines](https://opensource.google.com/docs/releasing/preparing/#name) Google uses when open sourcing a project. diff --git a/contribute/patch-acceptance.mdx b/contribute/patch-acceptance.mdx index 87376afd..c877c554 100644 --- a/contribute/patch-acceptance.mdx +++ b/contribute/patch-acceptance.mdx @@ -2,51 +2,26 @@ title: 'Patch Acceptance Process' --- +This page outlines how contributors can propose and make changes to the Bazel code base. +1. Read the [Bazel Contribution policy](/contribute/policy). -This page outlines how contributors can propose and make changes to the Bazel -code base. +2. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to discuss your plan and design. Pull requests that change or add behavior need a corresponding issue for tracking. -1. Read the [Bazel Contribution policy](/contribute/policy). -1. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to - discuss your plan and design. Pull requests that change or add behavior - need a corresponding issue for tracking. -1. If you're proposing significant changes, write a - [design document](/contribute/design-documents). -1. Ensure you've signed a [Contributor License - Agreement](https://cla.developers.google.com). -1. Prepare a git commit that implements the feature. Don't forget to add tests - and update the documentation. If your change has user-visible effects, please - [add release notes](/contribute/release-notes). If it is an incompatible change, - read the [guide for rolling out breaking changes](/contribute/breaking-changes). -1. Create a pull request on - [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, - read [about pull - requests](https://help.github.com/articles/about-pull-requests/). Note that - we restrict permissions to create branches on the main Bazel repository, so - you will need to push your commit to [your own fork of the - repository](https://help.github.com/articles/working-with-forks/). -1. A Bazel maintainer should assign you a reviewer within two business days - (excluding holidays in the USA and Germany). If you aren't assigned a - reviewer in that time, you can request one by emailing - [bazel-discuss@googlegroups.com] - (mailto:bazel-discuss@googlegroups.com). -1. Work with the reviewer to complete a code review. For each change, create a - new commit and push it to make changes to your pull request. If the review - takes too long (for instance, if the reviewer is unresponsive), send an email to - [bazel-discuss@googlegroups.com] - (mailto:bazel-discuss@googlegroups.com). -1. After your review is complete, a Bazel maintainer applies your patch to - Google's internal version control system. - - This triggers internal presubmit checks - that may suggest more changes. If you haven't expressed a preference, the - maintainer submitting your change adds "trivial" changes (such as - [linting](https://en.wikipedia.org/wiki/Lint_(software))) that don't affect - design. If deeper changes are required or you'd prefer to apply - changes directly, you and the reviewer should communicate preferences - clearly in review comments. - - After internal submission, the patch is exported as a Git commit, - at which point the GitHub pull request is closed. All final changes - are attributed to you. +3. If you're proposing significant changes, write a [design document](/contribute/design-documents). + +4. Ensure you've signed a [Contributor License Agreement](https://cla.developers.google.com). + +5. Prepare a git commit that implements the feature. Don't forget to add tests and update the documentation. If your change has user-visible effects, please [add release notes](/contribute/release-notes). If it is an incompatible change, read the [guide for rolling out breaking changes](/contribute/breaking-changes). + +6. Create a pull request on [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, read [about pull requests](https://help.github.com/articles/about-pull-requests/). Note that we restrict permissions to create branches on the main Bazel repository, so you will need to push your commit to [your own fork of the repository](https://help.github.com/articles/working-with-forks/). + +7. A Bazel maintainer should assign you a reviewer within two business days (excluding holidays in the USA and Germany). If you aren't assigned a reviewer in that time, you can request one by emailing \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). + +8. Work with the reviewer to complete a code review. For each change, create a new commit and push it to make changes to your pull request. If the review takes too long (for instance, if the reviewer is unresponsive), send an email to \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). + +9. After your review is complete, a Bazel maintainer applies your patch to Google's internal version control system. + + This triggers internal presubmit checks that may suggest more changes. If you haven't expressed a preference, the maintainer submitting your change adds "trivial" changes (such as [linting](https://en.wikipedia.org/wiki/Lint_\(software\))) that don't affect design. If deeper changes are required or you'd prefer to apply changes directly, you and the reviewer should communicate preferences clearly in review comments. + + After internal submission, the patch is exported as a Git commit, at which point the GitHub pull request is closed. All final changes are attributed to you. diff --git a/contribute/policy.mdx b/contribute/policy.mdx index 1bf00290..407f4c68 100644 --- a/contribute/policy.mdx +++ b/contribute/policy.mdx @@ -1,78 +1,59 @@ -translation: human -page_type: lcat --- title: 'Contribution policy' --- - - This page covers Bazel's governance model and contribution policy. ## Governance model -The [Bazel project](https://github.com/bazelbuild) is led and managed by Google -and has a large community of contributors outside of Google. Some Bazel -components (such as specific rules repositories under the -[bazelbuild](https://github.com/bazelbuild) organization) are led, -maintained, and managed by members of the community. The Google Bazel team -reviews suggestions to add community-owned repositories (such as rules) to the -[bazelbuild](https://github.com/bazelbuild) GitHub organization. +The [Bazel project](https://github.com/bazelbuild) is led and managed by Google and has a large community of contributors outside of Google. Some Bazel components (such as specific rules repositories under the [bazelbuild](https://github.com/bazelbuild) organization) are led, maintained, and managed by members of the community. The Google Bazel team reviews suggestions to add community-owned repositories (such as rules) to the [bazelbuild](https://github.com/bazelbuild) GitHub organization. ### Contributor roles -Here are outlines of the roles in the Bazel project, including their -responsibilities: - -* **Owners**: The Google Bazel team. Owners are responsible for: - * Strategy, maintenance, and leadership of the Bazel project. - * Building and maintaining Bazel's core functionality. - * Appointing Maintainers and approving new repositories. -* **Maintainers**: The Google Bazel team and designated GitHub users. - Maintainers are responsible for: - * Building and maintaining the primary functionality of their repository. - * Reviewing and approving contributions to areas of the Bazel code base. - * Supporting users and contributors with timely and transparent issue - management, PR review, and documentation. - * Releasing, testing and collaborating with Bazel Owners. -* **Contributors**: All users who contribute code or documentation to the - Bazel project. - * Creating well-written PRs to contribute to Bazel's codebase and - documentation. - * Using standard channels, such as GitHub Issues, to propose changes and - report issues. +Here are outlines of the roles in the Bazel project, including their responsibilities: + +- **Owners**: The Google Bazel team. Owners are responsible for: + + - Strategy, maintenance, and leadership of the Bazel project. + - Building and maintaining Bazel's core functionality. + - Appointing Maintainers and approving new repositories. + +- **Maintainers**: The Google Bazel team and designated GitHub users. Maintainers are responsible for: + + - Building and maintaining the primary functionality of their repository. + - Reviewing and approving contributions to areas of the Bazel code base. + - Supporting users and contributors with timely and transparent issue management, PR review, and documentation. + - Releasing, testing and collaborating with Bazel Owners. + +- **Contributors**: All users who contribute code or documentation to the Bazel project. + + - Creating well-written PRs to contribute to Bazel's codebase and documentation. + - Using standard channels, such as GitHub Issues, to propose changes and report issues. ### Becoming a Maintainer -Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as -rule sets. Contributors with a record of consistent, responsible past -contributions who are planning major contributions in the future could be -considered to become qualified Maintainers. +Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as rule sets. Contributors with a record of consistent, responsible past contributions who are planning major contributions in the future could be considered to become qualified Maintainers. ## Contribution policy -The Bazel project accepts contributions from external contributors. Here are the -contribution policies for Google-managed and Community-managed areas of code. - -* **Licensing**. All Maintainers and Contributors must sign the - [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). -* **Contributions**. Owners and Maintainers should make every effort to accept - worthwhile contributions. All contributions must be: - * Well written and well tested - * Discussed and approved by the Maintainers of the relevant area of code. - Discussions and approvals happen on GitHub Issues and in GitHub PRs. - Larger contributions require a - [design review](/contribute/design-documents). - * Added to Bazel's Continuous Integration system if not already present. - * Supportable and aligned with Bazel product direction -* **Code review**. All changes in all `bazelbuild` repositories require - review: - * All PRs must be approved by an Owner or Maintainer. - * Only Owners and Maintainers can merge PRs. -* **Compatibility**. Owners may need to reject or request modifications to PRs - in the unlikely event that the change requires substantial modifications to - internal Google systems. -* **Documentation**. Where relevant, feature contributions should include - documentation updates. - -For more details on contributing to Bazel, see our -[contribution guidelines](/contribute/). +The Bazel project accepts contributions from external contributors. Here are the contribution policies for Google-managed and Community-managed areas of code. + +- **Licensing**. All Maintainers and Contributors must sign the [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). + +- **Contributions**. Owners and Maintainers should make every effort to accept worthwhile contributions. All contributions must be: + + - Well written and well tested + - Discussed and approved by the Maintainers of the relevant area of code. Discussions and approvals happen on GitHub Issues and in GitHub PRs. Larger contributions require a [design review](/contribute/design-documents). + - Added to Bazel's Continuous Integration system if not already present. + - Supportable and aligned with Bazel product direction + +- **Code review**. All changes in all `bazelbuild` repositories require review: + + - All PRs must be approved by an Owner or Maintainer. + - Only Owners and Maintainers can merge PRs. + +- **Compatibility**. Owners may need to reject or request modifications to PRs in the unlikely event that the change requires substantial modifications to internal Google systems. + +- **Documentation**. Where relevant, feature contributions should include documentation updates. + +For more details on contributing to Bazel, see our [contribution guidelines](/contribute/). diff --git a/contribute/release-notes.mdx b/contribute/release-notes.mdx index 83e1d75b..bd0d9467 100644 --- a/contribute/release-notes.mdx +++ b/contribute/release-notes.mdx @@ -2,77 +2,44 @@ title: 'Writing release notes' --- - - This document is targeted at Bazel contributors. -Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release -note. This is used by the Bazel team to track changes in each release and write -the release announcement. +Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release note. This is used by the Bazel team to track changes in each release and write the release announcement. ## Overview -* Is your change a bugfix? In that case, you don't need a release note. Please - include a reference to the GitHub issue. +- Is your change a bugfix? In that case, you don't need a release note. Please include a reference to the GitHub issue. -* If the change adds / removes / changes Bazel in a user-visible way, then it - may be advantageous to mention it. +- If the change adds / removes / changes Bazel in a user-visible way, then it may be advantageous to mention it. -If the change is significant, follow the [design document -policy](/contribute/design-documents) first. +If the change is significant, follow the [design document policy](/contribute/design-documents) first. ## Guidelines -The release notes will be read by our users, so it should be short (ideally one -sentence), avoid jargon (Bazel-internal terminology), should focus on what the -change is about. +The release notes will be read by our users, so it should be short (ideally one sentence), avoid jargon (Bazel-internal terminology), should focus on what the change is about. -* Include a link to the relevant documentation. Almost any release note should - contain a link. If the description mentions a flag, a feature, a command name, - users will probably want to know more about it. +- Include a link to the relevant documentation. Almost any release note should contain a link. If the description mentions a flag, a feature, a command name, users will probably want to know more about it. -* Use backquotes around code, symbols, flags, or any word containing an - underscore. +- Use backquotes around code, symbols, flags, or any word containing an underscore. -* Do not just copy and paste bug descriptions. They are often cryptic and only - make sense to us and leave the user scratching their head. Release notes are - meant to explain what has changed and why in user-understandable language. +- Do not just copy and paste bug descriptions. They are often cryptic and only make sense to us and leave the user scratching their head. Release notes are meant to explain what has changed and why in user-understandable language. -* Always use present tense and the format "Bazel now supports Y" or "X now does - Z." We don't want our release notes to sound like bug entries. All release - note entries should be informative and use a consistent style and language. +- Always use present tense and the format "Bazel now supports Y" or "X now does Z." We don't want our release notes to sound like bug entries. All release note entries should be informative and use a consistent style and language. -* If something has been deprecated or removed, use "X has been deprecated" or "X - has been removed." Not "is removed" or "was removed." +- If something has been deprecated or removed, use "X has been deprecated" or "X has been removed." Not "is removed" or "was removed." -* If Bazel now does something differently, use "X now $newBehavior instead of - $oldBehavior" in present tense. This lets the user know in detail what to - expect when they use the new release. +- If Bazel now does something differently, use "X now $newBehavior instead of $oldBehavior" in present tense. This lets the user know in detail what to expect when they use the new release. -* If Bazel now supports or no longer supports something, use "Bazel now supports - / no longer supports X". +- If Bazel now supports or no longer supports something, use "Bazel now supports / no longer supports X". -* Explain why something has been removed / deprecated / changed. One sentence is - enough but we want the user to be able to evaluate impact on their builds. +- Explain why something has been removed / deprecated / changed. One sentence is enough but we want the user to be able to evaluate impact on their builds. -* Do NOT make any promises about future functionality. Avoid "this flag will be - removed" or "this will be changed." It introduces uncertainty. The first thing - the user will wonder is "when?" and we don't want them to start worrying about - their current builds breaking at some unknown time. +- Do NOT make any promises about future functionality. Avoid "this flag will be removed" or "this will be changed." It introduces uncertainty. The first thing the user will wonder is "when?" and we don't want them to start worrying about their current builds breaking at some unknown time. ## Process -As part of the [release -process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), -we collect the `RELNOTES` tags of every commit. We copy everything in a [Google -Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) -where we review, edit, and organize the notes. +As part of the [release process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), we collect the `RELNOTES` tags of every commit. We copy everything in a [Google Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) where we review, edit, and organize the notes. -The release manager sends an email to the -[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. -Bazel contributors are invited to contribute to the document and make sure -their changes are correctly reflected in the announcement. +The release manager sends an email to the [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. Bazel contributors are invited to contribute to the document and make sure their changes are correctly reflected in the announcement. -Later, the announcement will be submitted to the [Bazel -blog](https://blog.bazel.build/), using the [bazel-blog -repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). +Later, the announcement will be submitted to the [Bazel blog](https://blog.bazel.build/), using the [bazel-blog repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). diff --git a/contribute/statemachine-guide.mdx b/contribute/statemachine-guide.mdx index e98a96e8..86f34c50 100644 --- a/contribute/statemachine-guide.mdx +++ b/contribute/statemachine-guide.mdx @@ -1,64 +1,28 @@ --- -title: 'A Guide to Skyframe `StateMachine`s' +title: 'A Guide to Skyframe StateMachines' --- - - ## Overview -A Skyframe `StateMachine` is a *deconstructed* function-object that resides on -the heap. It supports flexible and evaluation without redundancy[^1] when -required values are not immediately available but computed asynchronously. The -`StateMachine` cannot tie up a thread resource while waiting, but instead has to -be suspended and resumed. The deconstruction thus exposes explicit re-entry -points so that prior computations can be skipped. +A Skyframe `StateMachine` is a *deconstructed* function-object that resides on the heap. It supports flexible and evaluation without redundancy[1](#user-content-fn-1) when required values are not immediately available but computed asynchronously. The `StateMachine` cannot tie up a thread resource while waiting, but instead has to be suspended and resumed. The deconstruction thus exposes explicit re-entry points so that prior computations can be skipped. -`StateMachine`s can be used to express sequences, branching, structured logical -concurrency and are tailored specifically for Skyframe interaction. -`StateMachine`s can be composed into larger `StateMachine`s and share -sub-`StateMachine`s. Concurrency is always hierarchical by construction and -purely logical. Every concurrent subtask runs in the single shared parent -SkyFunction thread. +`StateMachine`s can be used to express sequences, branching, structured logical concurrency and are tailored specifically for Skyframe interaction. `StateMachine`s can be composed into larger `StateMachine`s and share sub-`StateMachine`s. Concurrency is always hierarchical by construction and purely logical. Every concurrent subtask runs in the single shared parent SkyFunction thread. ## Introduction -This section briefly motivates and introduces `StateMachine`s, found in the -[`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) -package. +This section briefly motivates and introduces `StateMachine`s, found in the [`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) package. ### A brief introduction to Skyframe restarts -Skyframe is a framework that performs parallel evaluation of dependency graphs. -Each node in the graph corresponds with the evaluation of a SkyFunction with a -SkyKey specifying its parameters and SkyValue specifying its result. The -computational model is such that a SkyFunction may lookup SkyValues by SkyKey, -triggering recursive, parallel evaluation of additional SkyFunctions. Instead of -blocking, which would tie up a thread, when a requested SkyValue is not yet -ready because some subgraph of computation is incomplete, the requesting -SkyFunction observes a `null` `getValue` response and should return `null` -instead of a SkyValue, signaling that it is incomplete due to missing inputs. -Skyframe *restarts* the SkyFunctions when all previously requested SkyValues -become available. - -Before the introduction of `SkyKeyComputeState`, the traditional way of handling -a restart was to fully rerun the computation. Although this has quadratic -complexity, functions written this way eventually complete because each rerun, -fewer lookups return `null`. With `SkyKeyComputeState` it is possible to -associate hand-specified check-point data with a SkyFunction, saving significant -recomputation. - -`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate -virtually all recomputation when a SkyFunction restarts (assuming that -`SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume -execution hooks. +Skyframe is a framework that performs parallel evaluation of dependency graphs. Each node in the graph corresponds with the evaluation of a SkyFunction with a SkyKey specifying its parameters and SkyValue specifying its result. The computational model is such that a SkyFunction may lookup SkyValues by SkyKey, triggering recursive, parallel evaluation of additional SkyFunctions. Instead of blocking, which would tie up a thread, when a requested SkyValue is not yet ready because some subgraph of computation is incomplete, the requesting SkyFunction observes a `null` `getValue` response and should return `null` instead of a SkyValue, signaling that it is incomplete due to missing inputs. Skyframe *restarts* the SkyFunctions when all previously requested SkyValues become available. + +Before the introduction of `SkyKeyComputeState`, the traditional way of handling a restart was to fully rerun the computation. Although this has quadratic complexity, functions written this way eventually complete because each rerun, fewer lookups return `null`. With `SkyKeyComputeState` it is possible to associate hand-specified check-point data with a SkyFunction, saving significant recomputation. + +`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate virtually all recomputation when a SkyFunction restarts (assuming that `SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume execution hooks. ### Stateful computations inside `SkyKeyComputeState` -From an object-oriented design standpoint, it makes sense to consider storing -computational objects inside `SkyKeyComputeState` instead of pure data values. -In *Java*, the bare minimum description of a behavior carrying object is a -*functional interface* and it turns out to be sufficient. A `StateMachine` has -the following, curiously recursive, definition[^2]. +From an object-oriented design standpoint, it makes sense to consider storing computational objects inside `SkyKeyComputeState` instead of pure data values. In *Java*, the bare minimum description of a behavior carrying object is a *functional interface* and it turns out to be sufficient. A `StateMachine` has the following, curiously recursive, definition[2](#user-content-fn-2). ``` @FunctionalInterface @@ -67,12 +31,9 @@ public interface StateMachine { } ``` -The `Tasks` interface is analogous to `SkyFunction.Environment` but it is -designed for asynchrony and adds support for logically concurrent subtasks[^3]. +The `Tasks` interface is analogous to `SkyFunction.Environment` but it is designed for asynchrony and adds support for logically concurrent subtasks[3](#user-content-fn-3). -The return value of `step` is another `StateMachine`, allowing the specification -of a sequence of steps, inductively. `step` returns `DONE` when the -`StateMachine` is done. For example: +The return value of `step` is another `StateMachine`, allowing the specification of a sequence of steps, inductively. `step` returns `DONE` when the `StateMachine` is done. For example: ``` class HelloWorld implements StateMachine { @@ -98,98 +59,66 @@ hello world ``` -Note that the method reference `this::step2` is also a `StateMachine` due to -`step2` satisfying `StateMachine`'s functional interface definition. Method -references are the most common way to specify the next state in a -`StateMachine`. +Note that the method reference `this::step2` is also a `StateMachine` due to `step2` satisfying `StateMachine`'s functional interface definition. Method references are the most common way to specify the next state in a `StateMachine`. ![Suspending and resuming](/contribute/images/suspend-resume.svg) -Intuitively, breaking a computation down into `StateMachine` steps, instead of a -monolithic function, provides the hooks needed to *suspend* and *resume* a -computation. When `StateMachine.step` returns, there is an explicit *suspension* -point. The continuation specified by the returned `StateMachine` value is an -explicit *resume* point. Recomputation can thus be avoided because the -computation can be picked up exactly where it left off. +Intuitively, breaking a computation down into `StateMachine` steps, instead of a monolithic function, provides the hooks needed to *suspend* and *resume* a computation. When `StateMachine.step` returns, there is an explicit *suspension* point. The continuation specified by the returned `StateMachine` value is an explicit *resume* point. Recomputation can thus be avoided because the computation can be picked up exactly where it left off. ### Callbacks, continuations and asynchronous computation -In technical terms, a `StateMachine` serves as a *continuation*, determining the -subsequent computation to be executed. Instead of blocking, a `StateMachine` can -voluntarily *suspend* by returning from the `step` function, which transfers -control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can -then switch to a ready `StateMachine` or relinquish control back to Skyframe. +In technical terms, a `StateMachine` serves as a *continuation*, determining the subsequent computation to be executed. Instead of blocking, a `StateMachine` can voluntarily *suspend* by returning from the `step` function, which transfers control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can then switch to a ready `StateMachine` or relinquish control back to Skyframe. -Traditionally, *callbacks* and *continuations* are conflated into one concept. -However, `StateMachine`s maintain a distinction between the two. +Traditionally, *callbacks* and *continuations* are conflated into one concept. However, `StateMachine`s maintain a distinction between the two. -* *Callback* - describes where to store the result of an asynchronous - computation. -* *Continuation* - specifies the next execution state. +- *Callback* - describes where to store the result of an asynchronous computation. +- *Continuation* - specifies the next execution state. -Callbacks are required when invoking an asynchronous operation, which means that -the actual operation doesn't occur immediately upon calling the method, as in -the case of a SkyValue lookup. Callbacks should be kept as simple as possible. +Callbacks are required when invoking an asynchronous operation, which means that the actual operation doesn't occur immediately upon calling the method, as in the case of a SkyValue lookup. Callbacks should be kept as simple as possible. -Caution: A common pitfall of callbacks is that the asynchronous computation must -ensure the callback is called by the end of every reachable path. It's possible -to overlook some branches and the compiler doesn't give warnings about this. +Caution: A common pitfall of callbacks is that the asynchronous computation must ensure the callback is called by the end of every reachable path. It's possible to overlook some branches and the compiler doesn't give warnings about this. -*Continuations* are the `StateMachine` return values of `StateMachine`s and -encapsulate the complex execution that follows once all asynchronous -computations resolve. This structured approach helps to keep the complexity of -callbacks manageable. +*Continuations* are the `StateMachine` return values of `StateMachine`s and encapsulate the complex execution that follows once all asynchronous computations resolve. This structured approach helps to keep the complexity of callbacks manageable. ## Tasks -The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues -by SkyKey and to schedule concurrent subtasks. +The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues by SkyKey and to schedule concurrent subtasks. ``` interface Tasks { void enqueue(StateMachine subtask); - void lookUp(SkyKey key, Consumer sink); + void lookUp(SkyKey key, Consumer<SkyValue> sink); - - void lookUp(SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); + <E extends Exception> + void lookUp(SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); // lookUp overloads for 2 and 3 exception types exist, but are elided here. } ``` -Tip: When any state uses the `Tasks` interface to perform lookups or create -subtasks, those lookups and subtasks will complete before the next state begins. +Tip: When any state uses the `Tasks` interface to perform lookups or create subtasks, those lookups and subtasks will complete before the next state begins. -Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create -subtasks, they all *transitively* complete before the next state begins. +Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create subtasks, they all *transitively* complete before the next state begins. ### SkyValue lookups -`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are -analogous to `SkyFunction.Environment.getValue` and -`SkyFunction.Environment.getValueOrThrow` and have similar exception handling -semantics. The implementation does not immediately perform the lookup, but -instead, batches[^4] as many lookups as possible before doing so. The value -might not be immediately available, for example, requiring a Skyframe restart, -so the caller specifies what to do with the resulting value using a callback. +`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are analogous to `SkyFunction.Environment.getValue` and `SkyFunction.Environment.getValueOrThrow` and have similar exception handling semantics. The implementation does not immediately perform the lookup, but instead, batches[4](#user-content-fn-4) as many lookups as possible before doing so. The value might not be immediately available, for example, requiring a Skyframe restart, so the caller specifies what to do with the resulting value using a callback. -The `StateMachine` processor ([`Driver`s and bridging to -SkyFrame](#drivers-and-bridging)) guarantees that the value is available before -the next state begins. An example follows. +The `StateMachine` processor ([`Driver`s and bridging to SkyFrame](#drivers-and-bridging)) guarantees that the value is available before the next state begins. An example follows. ``` -class DoesLookup implements StateMachine, Consumer { +class DoesLookup implements StateMachine, Consumer<SkyValue> { private Value value; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new Key(), (Consumer) this); + tasks.lookUp(new Key(), (Consumer<SkyValue>) this); return this::processValue; } // The `lookUp` call in `step` causes this to be called before `processValue`. - @Override // Implementation of Consumer. + @Override // Implementation of Consumer<SkyValue>. public void accept(SkyValue value) { this.value = (Value)value; } @@ -201,25 +130,15 @@ class DoesLookup implements StateMachine, Consumer { } ``` -In the above example, the first step does a lookup for `new Key()`, passing -`this` as the consumer. That is possible because `DoesLookup` implements -`Consumer`. +In the above example, the first step does a lookup for `new Key()`, passing `this` as the consumer. That is possible because `DoesLookup` implements `Consumer<SkyValue>`. -Tip: When passing `this` as a value sink, it's helpful to readers to upcast it -to the receiver type to narrow down the purpose of passing `this`. The example -passes `(Consumer) this`. +Tip: When passing `this` as a value sink, it's helpful to readers to upcast it to the receiver type to narrow down the purpose of passing `this`. The example passes `(Consumer<SkyValue>) this`. -By contract, before the next state `DoesLookup.processValue` begins, all the -lookups of `DoesLookup.step` are complete. Therefore `value` is available when -it is accessed in `processValue`. +By contract, before the next state `DoesLookup.processValue` begins, all the lookups of `DoesLookup.step` are complete. Therefore `value` is available when it is accessed in `processValue`. ### Subtasks -`Tasks.enqueue` requests the execution of logically concurrent subtasks. -Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s -can do, including recursively creating more subtasks or looking up SkyValues. -Much like `lookUp`, the state machine driver ensures that all subtasks are -complete before proceeding to the next step. An example follows. +`Tasks.enqueue` requests the execution of logically concurrent subtasks. Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s can do, including recursively creating more subtasks or looking up SkyValues. Much like `lookUp`, the state machine driver ensures that all subtasks are complete before proceeding to the next step. An example follows. ``` class Subtasks implements StateMachine { @@ -257,22 +176,15 @@ class Subtasks implements StateMachine { } ``` -Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a -single thread so the "concurrent" update of `i` does not need any -synchronization. +Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a single thread so the "concurrent" update of `i` does not need any synchronization. ### Structured concurrency -Since every `lookUp` and `enqueue` must resolve before advancing to the next -state, it means that concurrency is naturally limited to tree-structures. It's -possible to create hierarchical[^5] concurrency as shown in the following -example. +Since every `lookUp` and `enqueue` must resolve before advancing to the next state, it means that concurrency is naturally limited to tree-structures. It's possible to create hierarchical[5](#user-content-fn-5) concurrency as shown in the following example. ![Structured Concurrency](/contribute/images/structured-concurrency.svg) -It's hard to tell from the *UML* that the concurrency structure forms a tree. -There's an [alternate view](#concurrency-tree-diagram) that better shows the -tree structure. +It's hard to tell from the *UML* that the concurrency structure forms a tree. There's an [alternate view](#concurrency-tree-diagram) that better shows the tree structure. ![Unstructured Concurrency](/contribute/images/unstructured-concurrency.svg) @@ -280,19 +192,15 @@ Structured concurrency is much easier to reason about. ## Composition and control flow patterns -This section presents examples for how multiple `StateMachine`s can be composed -and solutions to certain control flow problems. +This section presents examples for how multiple `StateMachine`s can be composed and solutions to certain control flow problems. ### Sequential states -This is the most common and straightforward control flow pattern. An example of -this is shown in [Stateful computations inside -`SkyKeyComputeState`](#stateful-computations). +This is the most common and straightforward control flow pattern. An example of this is shown in [Stateful computations inside `SkyKeyComputeState`](#stateful-computations). ### Branching -Branching states in `StateMachine`s can be achieved by returning different -values using regular *Java* control flow, as shown in the following example. +Branching states in `StateMachine`s can be achieved by returning different values using regular *Java* control flow, as shown in the following example. ``` class Branch implements StateMachine { @@ -312,18 +220,11 @@ It’s very common for certain branches to return `DONE`, for early completion. ### Advanced sequential composition -Since the `StateMachine` control structure is memoryless, sharing `StateMachine` -definitions as subtasks can sometimes be awkward. Let *M1* and -*M2* be `StateMachine` instances that share a `StateMachine`, *S*, -with *M1* and *M2* being the sequences *<A, S, B>* and -*<X, S, Y>* respectively. The problem is that *S* doesn’t know whether to -continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a -call stack. This section reviews some techniques for achieving this. +Since the `StateMachine` control structure is memoryless, sharing `StateMachine` definitions as subtasks can sometimes be awkward. Let *M1* and *M2* be `StateMachine` instances that share a `StateMachine`, *S*, with *M1* and *M2* being the sequences *\<A, S, B\>* and *\<X, S, Y\>* respectively. The problem is that *S* doesn’t know whether to continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a call stack. This section reviews some techniques for achieving this. #### `StateMachine` as terminal sequence element -This doesn’t solve the initial problem posed. It only demonstrates sequential -composition when the shared `StateMachine` is terminal in the sequence. +This doesn’t solve the initial problem posed. It only demonstrates sequential composition when the shared `StateMachine` is terminal in the sequence. ``` // S is the shared state machine. @@ -350,8 +251,7 @@ This works even if *S* is itself a complex state machine. #### Subtask for sequential composition -Since enqueued subtasks are guaranteed to complete before the next state, it’s -sometimes possible to slightly abuse[^6] the subtask mechanism. +Since enqueued subtasks are guaranteed to complete before the next state, it’s sometimes possible to slightly abuse[6](#user-content-fn-6) the subtask mechanism. ``` class M1 implements StateMachine { @@ -359,7 +259,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // S starts after `step` returns and by contract must complete before `doB` - // begins. It is effectively sequential, inducing the sequence < A, S, B >. + // begins. It is effectively sequential, inducing the sequence < A, S, B >. tasks.enqueue(new S()); return this::doB; } @@ -374,7 +274,7 @@ class M2 implements StateMachine { @Override public StateMachine step(Tasks tasks) { performX(); - // Similarly, this induces the sequence < X, S, Y>. + // Similarly, this induces the sequence < X, S, Y>. tasks.enqueue(new S()); return this::doY; } @@ -388,10 +288,7 @@ class M2 implements StateMachine { #### `runAfter` injection -Sometimes, abusing `Tasks.enqueue` is impossible because there are other -parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* -executes. In this case, injecting a `runAfter` parameter into *S* can be used to -inform *S* of what to do next. +Sometimes, abusing `Tasks.enqueue` is impossible because there are other parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* executes. In this case, injecting a `runAfter` parameter into *S* can be used to inform *S* of what to do next. ``` class S implements StateMachine { @@ -418,7 +315,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // Passes `this::doB` as the `runAfter` parameter of S, resulting in the - // sequence < A, S, B >. + // sequence < A, S, B >. return new S(/* runAfter= */ this::doB); } @@ -433,7 +330,7 @@ class M2 implements StateMachine { public StateMachine step(Tasks tasks) { performX(); // Passes `this::doY` as the `runAfter` parameter of S, resulting in the - // sequence < X, S, Y >. + // sequence < X, S, Y >. return new S(/* runAfter= */ this::doY); } @@ -444,10 +341,7 @@ class M2 implements StateMachine { } ``` -This approach is cleaner than abusing subtasks. However, applying this too -liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is -the road to [Callback Hell](#callback-hell). It’s better to break up sequential -`runAfter`s with ordinary sequential states instead. +This approach is cleaner than abusing subtasks. However, applying this too liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is the road to [Callback Hell](#callback-hell). It’s better to break up sequential `runAfter`s with ordinary sequential states instead. ``` return new S(/* runAfter= */ new T(/* runAfter= */ this::nextStep)) @@ -466,37 +360,21 @@ can be replaced with the following. } ``` -Note: It's possible to pass `DONE` as the `runAfter` parameter when there's -nothing to run afterwards. +Note: It's possible to pass `DONE` as the `runAfter` parameter when there's nothing to run afterwards. -Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` -to let the reader know the meaning at the callsite. +Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` to let the reader know the meaning at the callsite. #### *Forbidden* alternative: `runAfterUnlessError` -In an earlier draft, we had considered a `runAfterUnlessError` that would abort -early on errors. This was motivated by the fact that errors often end up getting -checked twice, once by the `StateMachine` that has a `runAfter` reference and -once by the `runAfter` machine itself. +In an earlier draft, we had considered a `runAfterUnlessError` that would abort early on errors. This was motivated by the fact that errors often end up getting checked twice, once by the `StateMachine` that has a `runAfter` reference and once by the `runAfter` machine itself. -After some deliberation, we decided that uniformity of the code is more -important than deduplicating the error checking. It would be confusing if the -`runAfter` mechanism did not work in a consistent manner with the -`tasks.enqueue` mechanism, which always requires error checking. +After some deliberation, we decided that uniformity of the code is more important than deduplicating the error checking. It would be confusing if the `runAfter` mechanism did not work in a consistent manner with the `tasks.enqueue` mechanism, which always requires error checking. -Warning: When using `runAfter`, the machine that has the injected `runAfter` -should invoke it unconditionally at completion, even on error, for consistency. +Warning: When using `runAfter`, the machine that has the injected `runAfter` should invoke it unconditionally at completion, even on error, for consistency. ### Direct delegation -Each time there is a formal state transition, the main `Driver` loop advances. -As per contract, advancing states means that all previously enqueued SkyValue -lookups and subtasks resolve before the next state executes. Sometimes the logic -of a delegate `StateMachine` makes a phase advance unnecessary or -counterproductive. For example, if the first `step` of the delegate performs -SkyKey lookups that could be parallelized with lookups of the delegating state -then a phase advance would make them sequential. It could make more sense to -perform direct delegation, as shown in the example below. +Each time there is a formal state transition, the main `Driver` loop advances. As per contract, advancing states means that all previously enqueued SkyValue lookups and subtasks resolve before the next state executes. Sometimes the logic of a delegate `StateMachine` makes a phase advance unnecessary or counterproductive. For example, if the first `step` of the delegate performs SkyKey lookups that could be parallelized with lookups of the delegating state then a phase advance would make them sequential. It could make more sense to perform direct delegation, as shown in the example below. ``` class Parent implements StateMachine { @@ -542,49 +420,37 @@ class Delegate implements StateMachine { ## Data flow -The focus of the previous discussion has been on managing control flow. This -section describes the propagation of data values. +The focus of the previous discussion has been on managing control flow. This section describes the propagation of data values. ### Implementing `Tasks.lookUp` callbacks -There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue -lookups](#skyvalue-lookups). This section provides rationale and suggests -approaches for handling multiple SkyValues. +There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue lookups](#skyvalue-lookups). This section provides rationale and suggests approaches for handling multiple SkyValues. #### `Tasks.lookUp` callbacks The `Tasks.lookUp` method takes a callback, `sink`, as a parameter. ``` - void lookUp(SkyKey key, Consumer sink); + void lookUp(SkyKey key, Consumer<SkyValue> sink); ``` The idiomatic approach would be to use a *Java* lambda to implement this: ``` - tasks.lookUp(key, value -> myValue = (MyValueClass)value); + tasks.lookUp(key, value -> myValue = (MyValueClass)value); ``` -with `myValue` being a member variable of the `StateMachine` instance doing the -lookup. However, the lambda requires an extra memory allocation compared to -implementing the `Consumer` interface in the `StateMachine` -implementation. The lambda is still useful when there are multiple lookups that -would be ambiguous. +with `myValue` being a member variable of the `StateMachine` instance doing the lookup. However, the lambda requires an extra memory allocation compared to implementing the `Consumer<SkyValue>` interface in the `StateMachine` implementation. The lambda is still useful when there are multiple lookups that would be ambiguous. -Note: Bikeshed warning. There is a noticeable difference of approximately 1% -end-to-end CPU usage when implementing callbacks systematically in -`StateMachine` implementations compared to using lambdas, which makes this -recommendation debatable. To avoid unnecessary debates, it is advised to leave -the decision up to the individual implementing the solution. +Note: Bikeshed warning. There is a noticeable difference of approximately 1% end-to-end CPU usage when implementing callbacks systematically in `StateMachine` implementations compared to using lambdas, which makes this recommendation debatable. To avoid unnecessary debates, it is advised to leave the decision up to the individual implementing the solution. -There are also error handling overloads of `Tasks.lookUp`, that are analogous to -`SkyFunction.Environment.getValueOrThrow`. +There are also error handling overloads of `Tasks.lookUp`, that are analogous to `SkyFunction.Environment.getValueOrThrow`. ``` - void lookUp( - SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); + <E extends Exception> void lookUp( + SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); - interface ValueOrExceptionSink { + interface ValueOrExceptionSink<E extends Exception> { void acceptValueOrException(@Nullable SkyValue value, @Nullable E exception); } ``` @@ -592,13 +458,13 @@ There are also error handling overloads of `Tasks.lookUp`, that are analogous to An example implementation is shown below. ``` -class PerformLookupWithError extends StateMachine, ValueOrExceptionSink { +class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyException> { private MyValue value; private MyException error; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink) this); + tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink<MyException>) this); return this::processResult; } @@ -627,33 +493,29 @@ class PerformLookupWithError extends StateMachine, ValueOrExceptionSink) this); + tasks.lookUp(configurationKey, (Consumer<SkyValue>) this); } var packageId = configuredTarget.getLabel().getPackageIdentifier(); - tasks.lookUp(PackageValue.key(packageId), (Consumer) this); + tasks.lookUp(PackageValue.key(packageId), (Consumer<SkyValue>) this); return this::constructResult; } - @Override // Implementation of `Consumer`. + @Override // Implementation of `Consumer<SkyValue>`. public void accept(SkyValue value) { if (value instanceof BuildConfigurationValue) { this.configurationValue = (BuildConfigurationValue) value; @@ -667,18 +529,11 @@ been simplified from prototype production code. } ``` -The `Consumer` callback implementation can be shared unambiguously -because the value types are different. When that’s not the case, falling back to -lambda-based implementations or full inner-class instances that implement the -appropriate callbacks is viable. +The `Consumer<SkyValue>` callback implementation can be shared unambiguously because the value types are different. When that’s not the case, falling back to lambda-based implementations or full inner-class instances that implement the appropriate callbacks is viable. ### Propagating values between `StateMachine`s -So far, this document has only explained how to arrange work in a subtask, but -subtasks also need to report a values back to the caller. Since subtasks are -logically asynchronous, their results are communicated back to the caller using -a *callback*. To make this work, the subtask defines a sink interface that is -injected via its constructor. +So far, this document has only explained how to arrange work in a subtask, but subtasks also need to report a values back to the caller. Since subtasks are logically asynchronous, their results are communicated back to the caller using a *callback*. To make this work, the subtask defines a sink interface that is injected via its constructor. ``` class BarProducer implements StateMachine { @@ -709,16 +564,9 @@ class BarProducer implements StateMachine { } ``` -Tip: It would be tempting to use the more concise signature void `accept(Bar -value)` rather than the stuttery `void acceptBarValue(Bar value)` above. -However, `Consumer` is a common overload of `void accept(Bar value)`, -so doing this often leads to violations of the [Overloads: never -split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) -style-guide rule. +Tip: It would be tempting to use the more concise signature void `accept(Bar value)` rather than the stuttery `void acceptBarValue(Bar value)` above. However, `Consumer<SkyValue>` is a common overload of `void accept(Bar value)`, so doing this often leads to violations of the [Overloads: never split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) style-guide rule. -Tip: Using a custom `ResultSink` type instead of a generic one from -`java.util.function` makes it easy to find implementations in the code base, -improving readability. +Tip: Using a custom `ResultSink` type instead of a generic one from `java.util.function` makes it easy to find implementations in the code base, improving readability. A caller `StateMachine` would then look like the following. @@ -767,70 +615,37 @@ class Caller implements StateMachine, BarProducer.ResultSink { } ``` -The preceding example demonstrates a few things. `Caller` has to propagate its -results back and defines its own `Caller.ResultSink`. `Caller` implements the -`BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if -`value` is null to determine if an error occurred. This is a common behavior -pattern after accepting output from either a subtask or SkyValue lookup. +The preceding example demonstrates a few things. `Caller` has to propagate its results back and defines its own `Caller.ResultSink`. `Caller` implements the `BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if `value` is null to determine if an error occurred. This is a common behavior pattern after accepting output from either a subtask or SkyValue lookup. -Note that the implementation of `acceptBarError` eagerly forwards the result to -the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). +Note that the implementation of `acceptBarError` eagerly forwards the result to the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). -Alternatives for top-level `StateMachine`s are described in [`Driver`s and -bridging to SkyFunctions](#drivers-and-bridging). +Alternatives for top-level `StateMachine`s are described in [`Driver`s and bridging to SkyFunctions](#drivers-and-bridging). ### Error handling -There's a couple of examples of error handling already in [`Tasks.lookUp` -callbacks](#tasks-lookup-callbacks) and [Propagating values between -`StateMachines`](#propagating-values). Exceptions, other than -`InterruptedException` are not thrown, but instead passed around through -callbacks as values. Such callbacks often have exclusive-or semantics, with -exactly one of a value or error being passed. +There's a couple of examples of error handling already in [`Tasks.lookUp` callbacks](#tasks-lookup-callbacks) and [Propagating values between `StateMachines`](#propagating-values). Exceptions, other than `InterruptedException` are not thrown, but instead passed around through callbacks as values. Such callbacks often have exclusive-or semantics, with exactly one of a value or error being passed. -The next section describes a a subtle, but important interaction with Skyframe -error handling. +The next section describes a a subtle, but important interaction with Skyframe error handling. #### Error bubbling (--nokeep\_going) -Warning: Errors need to be eagerly propagated all the way back to the -SkyFunction for error bubbling to function correctly. +Warning: Errors need to be eagerly propagated all the way back to the SkyFunction for error bubbling to function correctly. -During error bubbling, a SkyFunction may be restarted even if not all requested -SkyValues are available. In such cases, the subsequent state will never be -reached due to the `Tasks` API contract. However, the `StateMachine` should -still propagate the exception. +During error bubbling, a SkyFunction may be restarted even if not all requested SkyValues are available. In such cases, the subsequent state will never be reached due to the `Tasks` API contract. However, the `StateMachine` should still propagate the exception. -Since propagation must occur regardless of whether the next state is reached, -the error handling callback must perform this task. For an inner `StateMachine`, -this is achieved by invoking the parent callback. +Since propagation must occur regardless of whether the next state is reached, the error handling callback must perform this task. For an inner `StateMachine`, this is achieved by invoking the parent callback. -At the top-level `StateMachine`, which interfaces with the SkyFunction, this can -be done by calling the `setException` method of `ValueOrExceptionProducer`. -`ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even -if there are missing SkyValues. +At the top-level `StateMachine`, which interfaces with the SkyFunction, this can be done by calling the `setException` method of `ValueOrExceptionProducer`. `ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even if there are missing SkyValues. -If a `Driver` is being utilized directly, it is essential to check for -propagated errors from the SkyFunction, even if the machine has not finished -processing. +If a `Driver` is being utilized directly, it is essential to check for propagated errors from the SkyFunction, even if the machine has not finished processing. ### Event Handling -For SkyFunctions that need to emit events, a `StoredEventHandler` is injected -into SkyKeyComputeState and further injected into `StateMachine`s that require -them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping -certain events unless they are replayed but this was subsequently fixed. -`StoredEventHandler` injection is preserved because it simplifies the -implementation of events emitted from error handling callbacks. +For SkyFunctions that need to emit events, a `StoredEventHandler` is injected into SkyKeyComputeState and further injected into `StateMachine`s that require them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping certain events unless they are replayed but this was subsequently fixed. `StoredEventHandler` injection is preserved because it simplifies the implementation of events emitted from error handling callbacks. ## `Driver`s and bridging to SkyFunctions -A `Driver` is responsible for managing the execution of `StateMachine`s, -beginning with a specified root `StateMachine`. As `StateMachine`s can -recursively enqueue subtask `StateMachine`s, a single `Driver` can manage -numerous subtasks. These subtasks create a tree structure, a result of -[Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue -lookups across subtasks for improved efficiency. +A `Driver` is responsible for managing the execution of `StateMachine`s, beginning with a specified root `StateMachine`. As `StateMachine`s can recursively enqueue subtask `StateMachine`s, a single `Driver` can manage numerous subtasks. These subtasks create a tree structure, a result of [Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue lookups across subtasks for improved efficiency. There are a number of classes built around the `Driver`, with the following API. @@ -841,24 +656,15 @@ public final class Driver { } ``` -`Driver` takes a single root `StateMachine` as a parameter. Calling -`Driver.drive` executes the `StateMachine` as far as it can go without a -Skyframe restart. It returns true when the `StateMachine` completes and false -otherwise, indicating that not all values were available. +`Driver` takes a single root `StateMachine` as a parameter. Calling `Driver.drive` executes the `StateMachine` as far as it can go without a Skyframe restart. It returns true when the `StateMachine` completes and false otherwise, indicating that not all values were available. -`Driver` maintains the concurrent state of the `StateMachine` and it is well -suited for embedding in `SkyKeyComputeState`. +`Driver` maintains the concurrent state of the `StateMachine` and it is well suited for embedding in `SkyKeyComputeState`. ### Directly instantiating `Driver` -`StateMachine` implementations conventionally communicate their results via -callbacks. It's possible to directly instantiate a `Driver` as shown in the -following example. +`StateMachine` implementations conventionally communicate their results via callbacks. It's possible to directly instantiate a `Driver` as shown in the following example. -The `Driver` is embedded in the `SkyKeyComputeState` implementation along with -an implementation of the corresponding `ResultSink` to be defined a bit further -down. At the top level, the `State` object is an appropriate receiver for the -result of the computation as it is guaranteed to outlive `Driver`. +The `Driver` is embedded in the `SkyKeyComputeState` implementation along with an implementation of the corresponding `ResultSink` to be defined a bit further down. At the top level, the `State` object is an appropriate receiver for the result of the computation as it is guaranteed to outlive `Driver`. ``` class State implements SkyKeyComputeState, ResultProducer.ResultSink { @@ -940,8 +746,7 @@ private Result computeResult(State state, Skyfunction.Environment env) ### Embedding `Driver` -If the `StateMachine` produces a value and raises no exceptions, embedding -`Driver` is another possible implementation, as shown in the following example. +If the `StateMachine` produces a value and raises no exceptions, embedding `Driver` is another possible implementation, as shown in the following example. ``` class ResultProducer implements StateMachine { @@ -970,8 +775,7 @@ class ResultProducer implements StateMachine { } ``` -The SkyFunction may have code that looks like the following (where `State` is -the function specific type of `SkyKeyComputeState`). +The SkyFunction may have code that looks like the following (where `State` is the function specific type of `SkyKeyComputeState`). ``` @Nullable // Null when a Skyframe restart is needed. @@ -992,19 +796,16 @@ Result computeResult(SkyFunction.Environment env, State state) } ``` -Embedding `Driver` in the `StateMachine` implementation is a better fit for -Skyframe's synchronous coding style. +Embedding `Driver` in the `StateMachine` implementation is a better fit for Skyframe's synchronous coding style. ### StateMachines that may produce exceptions -Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` -and `ValueOrException2Producer` classes that have synchronous APIs to match -synchronous SkyFunction code. +Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` and `ValueOrException2Producer` classes that have synchronous APIs to match synchronous SkyFunction code. The `ValueOrExceptionProducer` abstract class includes the following methods. ``` -public abstract class ValueOrExceptionProducer +public abstract class ValueOrExceptionProducer<V, E extends Exception> implements StateMachine { @Nullable public final V tryProduceValue(Environment env) @@ -1017,65 +818,34 @@ public abstract class ValueOrExceptionProducer } ``` -It includes an embedded `Driver` instance and closely resembles the -`ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces -with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, -implementations call `setValue` or `setException` when either of those occur. -When both occur, the exception takes priority. The `tryProduceValue` method -bridges the asynchronous callback code to synchronous code and throws an -exception when one is set. +It includes an embedded `Driver` instance and closely resembles the `ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, implementations call `setValue` or `setException` when either of those occur. When both occur, the exception takes priority. The `tryProduceValue` method bridges the asynchronous callback code to synchronous code and throws an exception when one is set. -As previously noted, during error bubbling, it's possible for an error to occur -even if the machine is not yet done because not all inputs are available. To -accommodate this, `tryProduceValue` throws any set exceptions, even before the -machine is done. +As previously noted, during error bubbling, it's possible for an error to occur even if the machine is not yet done because not all inputs are available. To accommodate this, `tryProduceValue` throws any set exceptions, even before the machine is done. ## Epilogue: Eventually removing callbacks -`StateMachine`s are a highly efficient, but boilerplate intensive way to perform -asynchronous computation. Continuations (particularly in the form of `Runnable`s -passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, -but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and -there are no efficient asynchronous APIs for disk I/O. Eventually, it would be -good to optimize away callbacks as they have a learning curve and impede -readability. - -One of the most promising alternatives is *Java* virtual threads. Instead of -having to write callbacks, everything is replaced with synchronous, blocking -calls. This is possible because tying up a virtual thread resource, unlike a -platform thread, is supposed to be cheap. However, even with virtual threads, -replacing simple synchronous operations with thread creation and synchronization -primitives is too expensive. We performed a migration from `StateMachine`s to -*Java* virtual threads and they were orders of magnitude slower, leading to -almost a 3x increase in end-to-end analysis latency. Since virtual threads are -still a preview feature, it's possible that this migration can be performed at a -later date when performance improves. - -Another approach to consider is waiting for *Loom* coroutines, if they ever -become available. The advantage here is that it might be possible to reduce -synchronization overhead by using cooperative multitasking. - -If all else fails, low-level bytecode rewriting could also be a viable -alternative. With enough optimization, it might be possible to achieve -performance that approaches hand-written callback code. +`StateMachine`s are a highly efficient, but boilerplate intensive way to perform asynchronous computation. Continuations (particularly in the form of `Runnable`s passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and there are no efficient asynchronous APIs for disk I/O. Eventually, it would be good to optimize away callbacks as they have a learning curve and impede readability. + +One of the most promising alternatives is *Java* virtual threads. Instead of having to write callbacks, everything is replaced with synchronous, blocking calls. This is possible because tying up a virtual thread resource, unlike a platform thread, is supposed to be cheap. However, even with virtual threads, replacing simple synchronous operations with thread creation and synchronization primitives is too expensive. We performed a migration from `StateMachine`s to *Java* virtual threads and they were orders of magnitude slower, leading to almost a 3x increase in end-to-end analysis latency. Since virtual threads are still a preview feature, it's possible that this migration can be performed at a later date when performance improves. + +Another approach to consider is waiting for *Loom* coroutines, if they ever become available. The advantage here is that it might be possible to reduce synchronization overhead by using cooperative multitasking. + +If all else fails, low-level bytecode rewriting could also be a viable alternative. With enough optimization, it might be possible to achieve performance that approaches hand-written callback code. ## Appendix ### Callback Hell -Callback hell is an infamous problem in asynchronous code that uses callbacks. -It stems from the fact that the continuation for a subsequent step is nested -within the previous step. If there are many steps, this nesting can be extremely -deep. If coupled with control flow the code becomes unmanageable. +Callback hell is an infamous problem in asynchronous code that uses callbacks. It stems from the fact that the continuation for a subsequent step is nested within the previous step. If there are many steps, this nesting can be extremely deep. If coupled with control flow the code becomes unmanageable. ``` class CallbackHell implements StateMachine { @Override public StateMachine step(Tasks task) { doA(); - return (t, l) -> { + return (t, l) -> { doB(); - return (t1, l2) -> { + return (t1, l2) -> { doC(); return DONE; }; @@ -1084,11 +854,7 @@ class CallbackHell implements StateMachine { } ``` -One of the advantages of nested implementations is that the stack frame of the -outer step can be preserved. In *Java*, captured lambda variables must be -effectively final so using such variables can be cumbersome. Deep nesting is -avoided by returning method references as continuations instead of lambdas as -shown as follows. +One of the advantages of nested implementations is that the stack frame of the outer step can be preserved. In *Java*, captured lambda variables must be effectively final so using such variables can be cumbersome. Deep nesting is avoided by returning method references as continuations instead of lambdas as shown as follows. ``` class CallbackHellAvoided implements StateMachine { @@ -1110,23 +876,18 @@ class CallbackHellAvoided implements StateMachine { } ``` -Callback hell may also occur if the [`runAfter` injection](#runafter-injection) -pattern is used too densely, but this can be avoided by interspersing injections -with sequential steps. +Callback hell may also occur if the [`runAfter` injection](#runafter-injection) pattern is used too densely, but this can be avoided by interspersing injections with sequential steps. #### Example: Chained SkyValue lookups -It is often the case that the application logic requires dependent chains of -SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. -Thinking about this naively, this would result in a complex, deeply nested -callback structure. +It is often the case that the application logic requires dependent chains of SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. Thinking about this naively, this would result in a complex, deeply nested callback structure. ``` private ValueType1 value1; private ValueType2 value2; private StateMachine step1(...) { - tasks.lookUp(key1, (Consumer) this); // key1 has type KeyType1. + tasks.lookUp(key1, (Consumer<SkyValue>) this); // key1 has type KeyType1. return this::step2; } @@ -1146,91 +907,48 @@ private void acceptValueType2(SkyValue value) { } ``` -However, since continuations are specified as method references, the code looks -procedural across state transitions: `step2` follows `step1`. Note that here, a -lambda is used to assign `value2`. This makes the ordering of the code match the -ordering of the computation from top-to-bottom. +However, since continuations are specified as method references, the code looks procedural across state transitions: `step2` follows `step1`. Note that here, a lambda is used to assign `value2`. This makes the ordering of the code match the ordering of the computation from top-to-bottom. ### Miscellaneous Tips #### Readability: Execution Ordering -To improve readability, strive to keep the `StateMachine.step` implementations -in execution order and callback implementations immediately following where they -are passed in the code. This isn't always possible where the control flow -branches. Additional comments might be helpful in such cases. +To improve readability, strive to keep the `StateMachine.step` implementations in execution order and callback implementations immediately following where they are passed in the code. This isn't always possible where the control flow branches. Additional comments might be helpful in such cases. -In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an -intermediate method reference is created to achieve this. This trades a small -amount of performance for readability, which is likely worthwhile here. +In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an intermediate method reference is created to achieve this. This trades a small amount of performance for readability, which is likely worthwhile here. #### Generational Hypothesis -Medium-lived *Java* objects break the generational hypothesis of the *Java* -garbage collector, which is designed to handle objects that live for a very -short time or objects that live forever. By definition, objects in -`SkyKeyComputeState` violate this hypothesis. Such objects, containing the -constructed tree of all still-running `StateMachine`s, rooted at `Driver` have -an intermediate lifespan as they suspend, waiting for asynchronous computations -to complete. - -It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes -possible to observe an increase in GC time, even with dramatic decreases in -actual garbage generated. Since `StateMachine`s have an intermediate lifespan -they could be promoted to old gen, causing it to fill up more quickly, thus -necessitating more expensive major or full GCs to clean up. - -The initial precaution is to minimize the use of `StateMachine` variables, but -it is not always feasible, for example, if a value is needed across multiple -states. Where it is possible, local stack `step` variables are young generation -variables and efficiently GC'd. - -For `StateMachine` variables, breaking things down into subtasks and following -the recommended pattern for [Propagating values between -`StateMachine`s](#propagating-values) is also helpful. Observe that when -following the pattern, only child `StateMachine`s have references to parent -`StateMachine`s and not vice versa. This means that as children complete and -update the parents using result callbacks, the children naturally fall out of -scope and become eligible for GC. - -Finally, in some cases, a `StateMachine` variable is needed in earlier states -but not in later states. It can be beneficial to null out references of large -objects once it is known that they are no longer needed. +Medium-lived *Java* objects break the generational hypothesis of the *Java* garbage collector, which is designed to handle objects that live for a very short time or objects that live forever. By definition, objects in `SkyKeyComputeState` violate this hypothesis. Such objects, containing the constructed tree of all still-running `StateMachine`s, rooted at `Driver` have an intermediate lifespan as they suspend, waiting for asynchronous computations to complete. + +It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes possible to observe an increase in GC time, even with dramatic decreases in actual garbage generated. Since `StateMachine`s have an intermediate lifespan they could be promoted to old gen, causing it to fill up more quickly, thus necessitating more expensive major or full GCs to clean up. + +The initial precaution is to minimize the use of `StateMachine` variables, but it is not always feasible, for example, if a value is needed across multiple states. Where it is possible, local stack `step` variables are young generation variables and efficiently GC'd. + +For `StateMachine` variables, breaking things down into subtasks and following the recommended pattern for [Propagating values between `StateMachine`s](#propagating-values) is also helpful. Observe that when following the pattern, only child `StateMachine`s have references to parent `StateMachine`s and not vice versa. This means that as children complete and update the parents using result callbacks, the children naturally fall out of scope and become eligible for GC. + +Finally, in some cases, a `StateMachine` variable is needed in earlier states but not in later states. It can be beneficial to null out references of large objects once it is known that they are no longer needed. #### Naming states -When naming a method, it's usually possible to name a method for the behavior -that happens within that method. It's less clear how to do this in -`StateMachine`s because there is no stack. For example, suppose method `foo` -calls a sub-method `bar`. In a `StateMachine`, this could be translated into the -state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior -`bar`. As a result, method names for states tend to be narrower in scope, -potentially reflecting local behavior. +When naming a method, it's usually possible to name a method for the behavior that happens within that method. It's less clear how to do this in `StateMachine`s because there is no stack. For example, suppose method `foo` calls a sub-method `bar`. In a `StateMachine`, this could be translated into the state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior `bar`. As a result, method names for states tend to be narrower in scope, potentially reflecting local behavior. ### Concurrency tree diagram -The following is an alternative view of the diagram in [Structured -concurrency](#structured-concurrency) that better depicts the tree structure. -The blocks form a small tree. +The following is an alternative view of the diagram in [Structured concurrency](#structured-concurrency) that better depicts the tree structure. The blocks form a small tree. ![Structured Concurrency 3D](/contribute/images/structured-concurrency-3d.svg) -[^1]: In contrast to Skyframe's convention of restarting from the beginning when - values are not available. -[^2]: Note that `step` is permitted to throw `InterruptedException`, but the - examples omit this. There are a few low methods in *Bazel* code that throw - this exception and it propagates up to the `Driver`, to be described later, - that runs the `StateMachine`. It's fine to not declare it to be thrown when - unneeded. -[^3]: Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which - performs *independent* work for each dependency. Instead of manipulating - complex data structures that process all the dependencies at once, - introducing inefficiencies, each dependency has its own independent - `StateMachine`. -[^4]: Multiple `tasks.lookUp` calls within a single step are batched together. - Additional batching can be created by lookups occurring within concurrent - subtasks. -[^5]: This is conceptually similar to Java’s structured concurrency - [jeps/428](https://openjdk.org/jeps/428). -[^6]: Doing this is similar to spawning a thread and joining it to achieve - sequential composition. +## Footnotes + +1. In contrast to Skyframe's convention of restarting from the beginning when values are not available. [↩](#user-content-fnref-1) + +2. Note that `step` is permitted to throw `InterruptedException`, but the examples omit this. There are a few low methods in *Bazel* code that throw this exception and it propagates up to the `Driver`, to be described later, that runs the `StateMachine`. It's fine to not declare it to be thrown when unneeded. [↩](#user-content-fnref-2) + +3. Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which performs *independent* work for each dependency. Instead of manipulating complex data structures that process all the dependencies at once, introducing inefficiencies, each dependency has its own independent `StateMachine`. [↩](#user-content-fnref-3) + +4. Multiple `tasks.lookUp` calls within a single step are batched together. Additional batching can be created by lookups occurring within concurrent subtasks. [↩](#user-content-fnref-4) + +5. This is conceptually similar to Java’s structured concurrency [jeps/428](https://openjdk.org/jeps/428). [↩](#user-content-fnref-5) + +6. Doing this is similar to spawning a thread and joining it to achieve sequential composition. [↩](#user-content-fnref-6) diff --git a/contribute/windows-chocolatey-maintenance.mdx b/contribute/windows-chocolatey-maintenance.mdx index c6aee8fb..a570547d 100644 --- a/contribute/windows-chocolatey-maintenance.mdx +++ b/contribute/windows-chocolatey-maintenance.mdx @@ -2,22 +2,16 @@ title: 'Maintaining Bazel Chocolatey package on Windows' --- - - -Note: The Chocolatey package is experimental; please provide feedback -(`@petemounce` in issue tracker). +Note: The Chocolatey package is experimental; please provide feedback (`@petemounce` in issue tracker). ## Prerequisites You need: -* [chocolatey package manager](https://chocolatey.org) installed -* (to publish) a chocolatey API key granting you permission to publish the - `bazel` package - * [@petemounce](https://github.com/petemounce) currently - maintains this unofficial package. -* (to publish) to have set up that API key for the chocolatey source locally - via `choco apikey -k -s https://chocolatey.org/` +- [chocolatey package manager](https://chocolatey.org) installed +- (to publish) a chocolatey API key granting you permission to publish the `bazel` package + - [@petemounce](https://github.com/petemounce) currently maintains this unofficial package. +- (to publish) to have set up that API key for the chocolatey source locally via `choco apikey -k <your key here> -s https://chocolatey.org/` ## Build @@ -29,44 +23,38 @@ pushd scripts/packages/chocolatey popd ``` -Should result in `scripts/packages/chocolatey/bazel..nupkg` being -created. +Should result in `scripts/packages/chocolatey/bazel.<version>.nupkg` being created. The `build.ps1` script supports `mode` values `local`, `rc` and `release`. ## Test -0. Build the package (with `-mode local`) +1. Build the package (with `-mode local`) - * run a webserver (`python -m SimpleHTTPServer` in - `scripts/packages/chocolatey` is convenient and starts one on - `http://localhost:8000`) + - run a webserver (`python -m SimpleHTTPServer` in `scripts/packages/chocolatey` is convenient and starts one on `http://localhost:8000`) -0. Test the install +2. Test the install - The `test.ps1` should install the package cleanly (and error if it did not - install cleanly), then tell you what to do next. + The `test.ps1` should install the package cleanly (and error if it did not install cleanly), then tell you what to do next. -0. Test the uninstall +3. Test the uninstall - ```sh - choco uninstall bazel - # should remove bazel from the system - ``` + ```sh + choco uninstall bazel + # should remove bazel from the system + ``` Chocolatey's moderation process automates checks here as well. ## Release -Modify `tools/parameters.json` for the new release's URI and checksum once the -release has been published to github releases. +Modify `tools/parameters.json` for the new release's URI and checksum once the release has been published to github releases. ```powershell -./build.ps1 -version -isRelease -./test.ps1 -version +./build.ps1 -version <version> -isRelease +./test.ps1 -version <version> # if the test.ps1 passes choco push bazel.x.y.z.nupkg --source https://chocolatey.org/ ``` -Chocolatey.org will then run automated checks and respond to the push via email -to the maintainers. +Chocolatey.org will then run automated checks and respond to the push via email to the maintainers. diff --git a/contribute/windows-scoop-maintenance.mdx b/contribute/windows-scoop-maintenance.mdx index 58e2a6c4..dc4aadd9 100644 --- a/contribute/windows-scoop-maintenance.mdx +++ b/contribute/windows-scoop-maintenance.mdx @@ -2,38 +2,26 @@ title: 'Maintaining Bazel Scoop package on Windows' --- - - -Note: The Scoop package is experimental. To provide feedback, go to -`@excitoon` in issue tracker. +Note: The Scoop package is experimental. To provide feedback, go to `@excitoon` in issue tracker. ## Prerequisites You need: -* [Scoop package manager](https://scoop.sh/) installed -* GitHub account in order to publish and create pull requests to - [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) - * [@excitoon](https://github.com/excitoon) currently maintains this - unofficial package. Feel free to ask questions by - [e-mail](mailto:vladimir.chebotarev@gmail.com) or - [Telegram](http://telegram.me/excitoon). +- [Scoop package manager](https://scoop.sh/) installed +- GitHub account in order to publish and create pull requests to [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) + - [@excitoon](https://github.com/excitoon) currently maintains this unofficial package. Feel free to ask questions by [e-mail](mailto:vladimir.chebotarev@gmail.com) or [Telegram](http://telegram.me/excitoon). ## Release process -Scoop packages are very easy to maintain. Once you have the URL of released -Bazel, you need to make appropriate changes in -[this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): +Scoop packages are very easy to maintain. Once you have the URL of released Bazel, you need to make appropriate changes in [this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): - update version - update dependencies if needed - update URL - update hash (`sha256` by default) -In your filesystem, `bazel.json` is located in the directory -`%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to -your clone of a Git repository -[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). +In your filesystem, `bazel.json` is located in the directory `%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to your clone of a Git repository [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). Test the result: @@ -44,9 +32,7 @@ bazel version bazel something_else ``` -The first time, make a fork of -[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and -specify it as your own remote for `%UserProfile%/scoop/buckets/main`: +The first time, make a fork of [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and specify it as your own remote for `%UserProfile%/scoop/buckets/main`: ``` git remote add mine FORK_URL diff --git a/copy-upstream-docs.sh b/copy-upstream-docs.sh index ea388466..06659e22 100755 --- a/copy-upstream-docs.sh +++ b/copy-upstream-docs.sh @@ -83,15 +83,14 @@ transform_docs() { mkdir -p "$target_dir" - # Check if this file is in the BROKEN_FILES list + # Highlight files that previously needed manual fixes if echo "$BROKEN_FILES" | grep -q "^$target_file$"; then - echo "Skipping broken file: $target_file" - continue + echo "Processing previously broken file: $target_file" fi # Transform and copy the file echo "Transforming and copying $source_file to $DEST_DIR/$target_file" - awk -f transform-docs.awk "$source_file" > "$DEST_DIR/$target_file" + node tools/mdx-transform/transform.mjs "$source_file" "$DEST_DIR/$target_file" done } @@ -99,9 +98,13 @@ transform_docs() { transform_docs "$UPSTREAM_SITE" transform_docs "$REFERENCE_DOCS" -echo "Converting community YAML files to MDX..." -./convert-community-to-mdx.sh "$DEST_DIR/community/experts" -./convert-community-to-mdx.sh "$DEST_DIR/community/partners" +if command -v yq >/dev/null 2>&1; then + echo "Converting community YAML files to MDX..." + ./convert-community-to-mdx.sh "$DEST_DIR/community/experts" + ./convert-community-to-mdx.sh "$DEST_DIR/community/partners" +else + echo "Skipping community YAML conversion because 'yq' was not found in PATH." +fi echo "Copying community images..." mkdir -p "$DEST_DIR/community/images" diff --git a/docs/android-build-performance.mdx b/docs/android-build-performance.mdx index 0d5edc77..a96a199d 100644 --- a/docs/android-build-performance.mdx +++ b/docs/android-build-performance.mdx @@ -2,53 +2,39 @@ title: 'Android Build Performance' --- - - -This page contains information on optimizing build performance for Android -apps specifically. For general build performance optimization with Bazel, see -[Optimizing Performance](/rules/performance). +This page contains information on optimizing build performance for Android apps specifically. For general build performance optimization with Bazel, see [Optimizing Performance](/rules/performance). ## Recommended flags -The flags are in the -[`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so -they can be pasted directly into a `bazelrc` file and invoked with -`--config=` on the command line. +The flags are in the [`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so they can be pasted directly into a `bazelrc` file and invoked with `--config=<configuration_name>` on the command line. **Profiling performance** -Bazel writes a JSON trace profile by default to a file called -`command.profile.gz` in Bazel's output base. -See the [JSON Profile documentation](/rules/performance#performance-profiling) for -how to read and interact with the profile. +Bazel writes a JSON trace profile by default to a file called `command.profile.gz` in Bazel's output base. See the [JSON Profile documentation](/rules/performance#performance-profiling) for how to read and interact with the profile. **Persistent workers for Android build actions**. -A subset of Android build actions has support for -[persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). +A subset of Android build actions has support for [persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). These actions' mnemonics are: -* DexBuilder -* Javac -* Desugar -* AaptPackage -* AndroidResourceParser -* AndroidResourceValidator -* AndroidResourceCompiler -* RClassGenerator -* AndroidResourceLink -* AndroidAapt2 -* AndroidAssetMerger -* AndroidResourceMerger -* AndroidCompiledResourceMerger - -Enabling workers can result in better build performance by saving on JVM -startup costs from invoking each of these tools, but at the cost of increased -memory usage on the system by persisting them. - -To enable workers for these actions, apply these flags with -`--config=android_workers` on the command line: +- DexBuilder +- Javac +- Desugar +- AaptPackage +- AndroidResourceParser +- AndroidResourceValidator +- AndroidResourceCompiler +- RClassGenerator +- AndroidResourceLink +- AndroidAapt2 +- AndroidAssetMerger +- AndroidResourceMerger +- AndroidCompiledResourceMerger + +Enabling workers can result in better build performance by saving on JVM startup costs from invoking each of these tools, but at the cost of increased memory usage on the system by persisting them. + +To enable workers for these actions, apply these flags with `--config=android_workers` on the command line: ``` build:android_workers --strategy=DexBuilder=worker @@ -68,11 +54,7 @@ build:android_workers --strategy=Desugar=worker build:android_workers --persistent_android_resource_processor ``` -The default number of persistent workers created per action is `4`. We have -[measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) -by capping the number of instances for each action to `1` or `2`, although this -may vary depending on the system Bazel is running on, and the project being -built. +The default number of persistent workers created per action is `4`. We have [measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) by capping the number of instances for each action to `1` or `2`, although this may vary depending on the system Bazel is running on, and the project being built. To cap the number of instances for an action, apply these flags: @@ -86,12 +68,8 @@ build:android_workers --worker_max_instances=AaptPackage=2 **Using AAPT2** -[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved -performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the -`--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on -`android_binary` and `android_local_test`. +[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the `--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on `android_binary` and `android_local_test`. **SSD optimizations** -The `--experimental_multi_threaded_digest` flag is useful for optimizing digest -computation on SSDs. +The `--experimental_multi_threaded_digest` flag is useful for optimizing digest computation on SSDs. diff --git a/docs/android-instrumentation-test.mdx b/docs/android-instrumentation-test.mdx index fca7b577..e5b23f88 100644 --- a/docs/android-instrumentation-test.mdx +++ b/docs/android-instrumentation-test.mdx @@ -2,34 +2,23 @@ title: 'Android Instrumentation Tests' --- - - -_If you're new to Bazel, start with the [Building Android with -Bazel](/start/android-app ) tutorial._ +*If you're new to Bazel, start with the [Building Android with Bazel](/start/android-app) tutorial.* ![Running Android instrumentation tests in parallel](/docs/images/android_test.gif "Android instrumentation test") **Figure 1.** Running parallel Android instrumentation tests. -[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) -allows developers to test their apps on Android emulators and devices. -It utilizes real Android framework APIs and the Android Test Library. +[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) allows developers to test their apps on Android emulators and devices. It utilizes real Android framework APIs and the Android Test Library. -For hermeticity and reproducibility, Bazel creates and launches Android -emulators in a sandbox, ensuring that tests always run from a clean state. Each -test gets an isolated emulator instance, allowing tests to run in parallel -without passing states between them. +For hermeticity and reproducibility, Bazel creates and launches Android emulators in a sandbox, ensuring that tests always run from a clean state. Each test gets an isolated emulator instance, allowing tests to run in parallel without passing states between them. -For more information on Android instrumentation tests, check out the [Android -developer -documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). +For more information on Android instrumentation tests, check out the [Android developer documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). Please file issues in the [GitHub issue tracker](https://github.com/bazelbuild/bazel/issues). ## How it works -When you run `bazel test` on an `android_instrumentation_test` target for the -first time, Bazel performs the following steps: +When you run `bazel test` on an `android_instrumentation_test` target for the first time, Bazel performs the following steps: 1. Builds the test APK, APK under test, and their transitive dependencies 2. Creates, boots, and caches clean emulator states @@ -39,9 +28,7 @@ first time, Bazel performs the following steps: 6. Shuts down the emulator 7. Reports the results -In subsequent test runs, Bazel boots the emulator from the clean, cached state -created in step 2, so there are no leftover states from previous runs. Caching -emulator state also speeds up test runs. +In subsequent test runs, Bazel boots the emulator from the clean, cached state created in step 2, so there are no leftover states from previous runs. Caching emulator state also speeds up test runs. ## Prerequisites @@ -54,17 +41,14 @@ Ensure your environment satisfies the following prerequisites: ```posix-terminal bazel info release ``` + This results in output similar to the following: -```none {:.devsite-disable-click-to-copy} +```none release 4.1.0 ``` -- **KVM**. Bazel requires emulators to have [hardware - acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) - with KVM on Linux. You can follow these - [installation instructions](https://help.ubuntu.com/community/KVM/Installation) - for Ubuntu. +- **KVM**. Bazel requires emulators to have [hardware acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) with KVM on Linux. You can follow these [installation instructions](https://help.ubuntu.com/community/KVM/Installation) for Ubuntu. To verify that KVM has the correct configuration, run: @@ -74,34 +58,32 @@ apt-get install cpu-checker && kvm-ok If it prints the following message, you have the correct configuration: -```none {:.devsite-disable-click-to-copy} +```none INFO: /dev/kvm exists KVM acceleration can be used ``` -- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires - the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). +- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). To install it, run: ```posix-terminal apt-get install xvfb ``` -Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` -by running: + +Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` by running: ```posix-terminal which Xvfb ``` + The output is the following: ```{:.devsite-disable-click-to-copy} /usr/bin/Xvfb ``` -- **32-bit Libraries**. Some of the binaries used by the test infrastructure are - 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For - Ubuntu, install these 32-bit libraries: +- **32-bit Libraries**. Some of the binaries used by the test infrastructure are 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For Ubuntu, install these 32-bit libraries: ```posix-terminal sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 libbz2-1.0:i386 @@ -115,7 +97,6 @@ Here is a typical target dependency graph of an `android_instrumentation_test`: **Figure 2.** Target dependency graph of an `android_instrumentation_test`. - ### BUILD file The graph translates into a `BUILD` file like this: @@ -169,59 +150,45 @@ android_library( The main attributes of the rule `android_instrumentation_test` are: -- `test_app`: An `android_binary` target. This target contains test code and - dependencies like Espresso and UIAutomator. The selected `android_binary` - target is required to specify an `instruments` attribute pointing to another - `android_binary`, which is the app under test. +- `test_app`: An `android_binary` target. This target contains test code and dependencies like Espresso and UIAutomator. The selected `android_binary` target is required to specify an `instruments` attribute pointing to another `android_binary`, which is the app under test. -- `target_device`: An `android_device` target. This target describes the - specifications of the Android emulator which Bazel uses to create, launch and - run the tests. See the [section on choosing an Android - device](#android-device-target) for more information. +- `target_device`: An `android_device` target. This target describes the specifications of the Android emulator which Bazel uses to create, launch and run the tests. See the [section on choosing an Android device](#android-device-target) for more information. -The test app's `AndroidManifest.xml` must include [an `` -tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). -This tag must specify the attributes for the **package of the target app** and -the **fully qualified class name of the instrumentation test runner**, -`androidx.test.runner.AndroidJUnitRunner`. +The test app's `AndroidManifest.xml` must include [an `<instrumentation>` tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). This tag must specify the attributes for the **package of the target app** and the **fully qualified class name of the instrumentation test runner**, `androidx.test.runner.AndroidJUnitRunner`. Here is an example `AndroidTestManifest.xml` for the test app: ```xml - - + android:versionName="1.0"> - + android:targetPackage="com.example.android.app" /> - + android:targetSdkVersion="27" /> - - - - + <application > + + </application> +</manifest> ``` ### WORKSPACE dependencies -In order to use this rule, your project needs to depend on these external -repositories: +In order to use this rule, your project needs to depend on these external repositories: - `@androidsdk`: The Android SDK. Download this through Android Studio. -- `@android_test_support`: Hosts the test runner, emulator launcher, and - `android_device` targets. You can find the [latest release - here](https://github.com/android/android-test/releases). +- `@android_test_support`: Hosts the test runner, emulator launcher, and `android_device` targets. You can find the [latest release here](https://github.com/android/android-test/releases). -Enable these dependencies by adding the following lines to your `WORKSPACE` -file: +Enable these dependencies by adding the following lines to your `WORKSPACE` file: ```python # Android SDK @@ -243,25 +210,20 @@ android_test_repositories() ## Maven dependencies -For managing dependencies on Maven artifacts from repositories, such as [Google -Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), -you should use a Maven resolver, such as -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). +For managing dependencies on Maven artifacts from repositories, such as [Google Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), you should use a Maven resolver, such as [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). -The rest of this page shows how to use `rules_jvm_external` to -resolve and fetch dependencies from Maven repositories. +The rest of this page shows how to use `rules_jvm_external` to resolve and fetch dependencies from Maven repositories. -## Choosing an android_device target +## Choosing an android\_device target -`android_instrumentation_test.target_device` specifies which Android device to -run the tests on. These `android_device` targets are defined in -[`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). +`android_instrumentation_test.target_device` specifies which Android device to run the tests on. These `android_device` targets are defined in [`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). For example, you can query for the sources for a particular target by running: ```posix-terminal bazel query --output=build @android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86 ``` + Which results in output that looks similar to: ```python @@ -287,29 +249,22 @@ android_device( The device target names use this template: ``` -@android_test_support//tools/android/emulated_devices/{{ "" }}device_type{{ "" }}:{{ "" }}system{{ "" }}_{{ "" }}api_level{{ "" }}_x86_qemu2 +@android_test_support//tools/android/emulated_devices/<var>device_type</var>:<var>system</var>_<var>api_level</var>_x86_qemu2 ``` -In order to launch an `android_device`, the `system_image` for the selected API -level is required. To download the system image, use Android SDK's -`tools/bin/sdkmanager`. For example, to download the system image for -`generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager -"system-images;android-23;default;x86"`. +In order to launch an `android_device`, the `system_image` for the selected API level is required. To download the system image, use Android SDK's `tools/bin/sdkmanager`. For example, to download the system image for `generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager "system-images;android-23;default;x86"`. -To see the full list of supported `android_device` targets in -`@android_test_support`, run the following command: +To see the full list of supported `android_device` targets in `@android_test_support`, run the following command: ```posix-terminal bazel query 'filter("x86_qemu2$", kind(android_device, @android_test_support//tools/android/emulated_devices/...:*))' ``` -Bazel currently supports x86-based emulators only. For better performance, use -`QEMU2` `android_device` targets instead of `QEMU` ones. +Bazel currently supports x86-based emulators only. For better performance, use `QEMU2` `android_device` targets instead of `QEMU` ones. ## Running tests -To run tests, add these lines to your project's -`project root:/.bazelrc` file. +To run tests, add these lines to your project's `<var>project root</var>:<var>/.bazelrc` file. ``` # Configurations for testing with Bazel @@ -339,13 +294,11 @@ Then, use one of the configurations to run tests: - `bazel test //my/test:target --config=headless` - `bazel test //my/test:target --config=local_device` -Use __only one configuration__ or tests will fail. +Use **only one configuration** or tests will fail. ### Headless testing -With `Xvfb`, it is possible to test with emulators without the graphical -interface, also known as headless testing. To disable the graphical interface -when running tests, pass the test argument `--enable_display=false` to Bazel: +With `Xvfb`, it is possible to test with emulators without the graphical interface, also known as headless testing. To disable the graphical interface when running tests, pass the test argument `--enable_display=false` to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=false @@ -353,9 +306,7 @@ bazel test //my/test:target --test_arg=--enable_display=false ### GUI testing -If the `$DISPLAY` environment variable is set, it's possible to enable the -graphical interface of the emulator while the test is running. To do this, pass -these test arguments to Bazel: +If the `$DISPLAY` environment variable is set, it's possible to enable the graphical interface of the emulator while the test is running. To do this, pass these test arguments to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY @@ -363,26 +314,15 @@ bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY ### Testing with a local emulator or device -Bazel also supports testing directly on a locally launched emulator or connected -device. Pass the flags -`--test_strategy=exclusive` and -`--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. -If there is more than one connected device, pass the flag -`--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of -the device/emulator listed in `adb devices`. +Bazel also supports testing directly on a locally launched emulator or connected device. Pass the flags `--test_strategy=exclusive` and `--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. If there is more than one connected device, pass the flag `--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of the device/emulator listed in `adb devices`. ## Sample projects -If you are looking for canonical project samples, see the [Android testing -samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) -for projects using Espresso and UIAutomator. +If you are looking for canonical project samples, see the [Android testing samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) for projects using Espresso and UIAutomator. ## Espresso setup -If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) -(`androidx.test.espresso`), you can use the following snippets to set up your -Bazel workspace with the list of commonly used Espresso artifacts and their -dependencies: +If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) (`androidx.test.espresso`), you can use the following snippets to set up your Bazel workspace with the list of commonly used Espresso artifacts and their dependencies: ``` androidx.test.espresso:espresso-core @@ -393,8 +333,7 @@ org.hamcrest:java-hamcrest junit:junit ``` -One way to organize these dependencies is to create a `//:test_deps` shared -library in your `project root/BUILD.bazel` file: +One way to organize these dependencies is to create a `//:test_deps` shared library in your `<var>project root</var>/BUILD.bazel` file: ```python java_library( @@ -411,7 +350,7 @@ java_library( ) ``` -Then, add the required dependencies in `project root/WORKSPACE`: +Then, add the required dependencies in `<var>project root</var>/WORKSPACE`: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -444,8 +383,7 @@ maven_install( ) ``` -Finally, in your test `android_binary` target, add the `//:test_deps` -dependency: +Finally, in your test `android_binary` target, add the `//:test_deps` dependency: ```python android_binary( @@ -463,17 +401,14 @@ android_binary( ### Reading test logs -Use `--test_output=errors` to print logs for failing tests, or -`--test_output=all` to print all test output. If you're looking for an -individual test log, go to -`$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. +Use `--test_output=errors` to print logs for failing tests, or `--test_output=all` to print all test output. If you're looking for an individual test log, go to `$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. -For example, the test logs for `BasicSample` canonical project are in -`bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: +For example, the test logs for `BasicSample` canonical project are in `bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: ```posix-terminal tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ``` + This results in the following output: ```none @@ -531,9 +466,7 @@ $ tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ### Reading emulator logs -The emulator logs for `android_device` targets are stored in the `/tmp/` -directory with the name `emulator_xxxxx.log`, where `xxxxx` is a -randomly-generated sequence of characters. +The emulator logs for `android_device` targets are stored in the `/tmp/` directory with the name `emulator_xxxxx.log`, where `xxxxx` is a randomly-generated sequence of characters. Use this command to find the latest emulator log: @@ -543,8 +476,7 @@ ls -1t /tmp/emulator_*.log | head -n 1 ### Testing against multiple API levels -If you would like to test against multiple API levels, you can use a list -comprehension to create test targets for each API level. For example: +If you would like to test against multiple API levels, you can use a list comprehension to create test targets for each API level. For example: ```python API_LEVELS = [ @@ -563,14 +495,10 @@ API_LEVELS = [ ## Known issues -- [Forked adb server processes are not terminated after - tests](https://github.com/bazelbuild/bazel/issues/4853) -- While APK building works on all platforms (Linux, macOS, Windows), testing - only works on Linux. -- Even with `--config=local_adb`, users still need to specify - `android_instrumentation_test.target_device`. -- If using a local device or emulator, Bazel does not uninstall the APKs after - the test. Clean the packages by running this command: +- [Forked adb server processes are not terminated after tests](https://github.com/bazelbuild/bazel/issues/4853) +- While APK building works on all platforms (Linux, macOS, Windows), testing only works on Linux. +- Even with `--config=local_adb`, users still need to specify `android_instrumentation_test.target_device`. +- If using a local device or emulator, Bazel does not uninstall the APKs after the test. Clean the packages by running this command: ```posix-terminal adb shell pm list diff --git a/docs/android-ndk.mdx b/docs/android-ndk.mdx index b434797d..292231dc 100644 --- a/docs/android-ndk.mdx +++ b/docs/android-ndk.mdx @@ -2,25 +2,13 @@ title: 'Using the Android Native Development Kit with Bazel' --- - - -_If you're new to Bazel, please start with the [Building Android with -Bazel](/start/android-app ) tutorial._ +*If you're new to Bazel, please start with the [Building Android with Bazel](/start/android-app) tutorial.* ## Overview -Bazel can run in many different build configurations, including several that use -the Android Native Development Kit (NDK) toolchain. This means that normal -`cc_library` and `cc_binary` rules can be compiled for Android directly within -Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository -rule and its related bzlmod extension. +Bazel can run in many different build configurations, including several that use the Android Native Development Kit (NDK) toolchain. This means that normal `cc_library` and `cc_binary` rules can be compiled for Android directly within Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository rule and its related bzlmod extension. -For general Android -compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). -This tutorial demonstrates how to integrate C++ library dependencies into -Android apps and uses -[`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK -toolchain discovery and registration. +For general Android compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). This tutorial demonstrates how to integrate C++ library dependencies into Android apps and uses [`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK toolchain discovery and registration. ## Prerequisites @@ -28,12 +16,7 @@ Please ensure that you have installed the Android SDK and NDK. ### NDK and SDK setup -External repository setup varies depending on whether you are using WORKSPACE -or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* -Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of -each other. -If you are using one dependency management solution, you don't need to add -the boilerplate for the other. +External repository setup varies depending on whether you are using WORKSPACE or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of each other. If you are using one dependency management solution, you don't need to add the boilerplate for the other. #### Bzlmod MODULE.bazel setup @@ -77,23 +60,18 @@ android_ndk_repository( Compatibility notes for WORKSPACE: -* Both `rules_android` and `rules_android_ndk` rules require extra - boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date - and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) - file of `rules_android_ndk`'s basic example app. +- Both `rules_android` and `rules_android_ndk` rules require extra boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) file of `rules_android_ndk`'s basic example app. -For more information about the `android_ndk_repository` rule, see its -[docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). +For more information about the `android_ndk_repository` rule, see its [docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). ## Quick start -To build C++ for Android, simply add `cc_library` dependencies to your -`android_binary` or `android_library` rules. +To build C++ for Android, simply add `cc_library` dependencies to your `android_binary` or `android_library` rules. For example, given the following `BUILD` file for an Android app: ```python -# In /app/src/main/BUILD.bazel +# In <project>/app/src/main/BUILD.bazel load("@rules_cc//cc:cc_library.bzl", "cc_library") load("@rules_android//rules:rules.bzl", "android_binary", "android_library") @@ -122,19 +100,17 @@ This `BUILD` file results in the following target graph: ![Example results](/docs/images/android_ndk.png "Build graph results") -**Figure 1.** Build graph of Android project with cc_library dependencies. +**Figure 1.** Build graph of Android project with cc\_library dependencies. To build the app, simply run: ```posix-terminal -bazel build //app/src/main:app --android_platforms= +bazel build //app/src/main:app --android_platforms=<your platform> ``` -Note that if you don't specify `--android_platforms`, your build will fail with -errors about missing JNI headers. +Note that if you don't specify `--android_platforms`, your build will fail with errors about missing JNI headers. -The `bazel build` command compiles the Java files, Android resource files, and -`cc_library` rules, and packages everything into an APK: +The `bazel build` command compiles the Java files, Android resource files, and `cc_library` rules, and packages everything into an APK: ```posix-terminal $ zipinfo -1 bazel-bin/app/src/main/app.apk @@ -150,27 +126,21 @@ META-INF/CERT.RSA META-INF/MANIFEST.MF ``` -Bazel compiles all of the cc_libraries into a single shared object (`.so`) file, -targeted the architectures specified by `--android_platforms`. -See the section on [configuring the target ABI](#configuring-target-abi) for -more details. +Bazel compiles all of the cc\_libraries into a single shared object (`.so`) file, targeted the architectures specified by `--android_platforms`. See the section on [configuring the target ABI](#configuring-target-abi) for more details. ## Example setup -This example is available in the [Bazel examples -repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). +This example is available in the [Bazel examples repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). -In the `BUILD.bazel` file, three targets are defined with the `android_binary`, -`android_library`, and `cc_library` rules. +In the `BUILD.bazel` file, three targets are defined with the `android_binary`, `android_library`, and `cc_library` rules. The `android_binary` top-level target builds the APK. -The `cc_library` target contains a single C++ source file with a JNI function -implementation: +The `cc_library` target contains a single C++ source file with a JNI function implementation: ```c++ -#include -#include +#include <jni.h> +#include <string> extern "C" JNIEXPORT jstring @@ -180,14 +150,11 @@ Java_com_example_android_bazel_MainActivity_stringFromJNI( JNIEnv *env, jobject /* this */) { std::string hello = "Hello from C++"; - return env->NewStringUTF(hello.c_str()); + return env->NewStringUTF(hello.c_str()); } ``` -The `android_library` target specifies the Java sources, resource files, and the -dependency on a `cc_library` target. For this example, `MainActivity.java` loads -the shared object file `libapp.so`, and defines the method signature for the JNI -function: +The `android_library` target specifies the Java sources, resource files, and the dependency on a `cc_library` target. For this example, `MainActivity.java` loads the shared object file `libapp.so`, and defines the method signature for the JNI function: ```java public class MainActivity extends AppCompatActivity { @@ -206,23 +173,19 @@ public class MainActivity extends AppCompatActivity { } ``` -Note: The name of the native library is derived from the name of the top -level `android_binary` target. In this example, it is `app`. +Note: The name of the native library is derived from the name of the top level `android_binary` target. In this example, it is `app`. ## Configuring the target ABI To configure the target ABI, use the `--android_platforms` flag as follows: ```posix-terminal -bazel build //:app --android_platforms={{ "" }}comma-separated list of platforms{{ "" }} +bazel build //:app --android_platforms=<var>comma-separated list of platforms</var> ``` -Just like the `--platforms` flag, the values passed to `--android_platforms` are -the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) -targets, using standard constraint values to describe your device. +Just like the `--platforms` flag, the values passed to `--android_platforms` are the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) targets, using standard constraint values to describe your device. -For example, for an Android device with a 64-bit ARM processor, you'd define -your platform like this: +For example, for an Android device with a 64-bit ARM processor, you'd define your platform like this: ```py platform( @@ -234,43 +197,36 @@ platform( ) ``` -Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) -OS constraint. To migrate the CPU constraint, check this chart: +Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) OS constraint. To migrate the CPU constraint, check this chart: -CPU Value | Platform -------------- | ------------------------------------------ -`armeabi-v7a` | `@platforms//cpu:armv7` -`arm64-v8a` | `@platforms//cpu:arm64` -`x86` | `@platforms//cpu:x86_32` -`x86_64` | `@platforms//cpu:x86_64` +| CPU Value | Platform | +| ------------- | ------------------------ | +| `armeabi-v7a` | `@platforms//cpu:armv7` | +| `arm64-v8a` | `@platforms//cpu:arm64` | +| `x86` | `@platforms//cpu:x86_32` | +| `x86_64` | `@platforms//cpu:x86_64` | -And, of course, for a multi-architecture APK, you pass multiple labels, for -example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in -your top-level `BUILD.bazel` file). +And, of course, for a multi-architecture APK, you pass multiple labels, for example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in your top-level `BUILD.bazel` file). -Bazel is unable to select a default Android platform, so one must be defined and -specified with `--android_platforms`. +Bazel is unable to select a default Android platform, so one must be defined and specified with `--android_platforms`. -Depending on the NDK revision and Android API level, the following ABIs are -available: +Depending on the NDK revision and Android API level, the following ABIs are available: | NDK revision | ABIs | -|--------------|-------------------------------------------------------------| +| ------------ | ----------------------------------------------------------- | | 16 and lower | armeabi, armeabi-v7a, arm64-v8a, mips, mips64, x86, x86\_64 | | 17 and above | armeabi-v7a, arm64-v8a, x86, x86\_64 | -See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) -for more information on these ABIs. +See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) for more information on these ABIs. -Multi-ABI Fat APKs are not recommended for release builds since they increase -the size of the APK, but can be useful for development and QA builds. +Multi-ABI Fat APKs are not recommended for release builds since they increase the size of the APK, but can be useful for development and QA builds. ## Selecting a C++ standard Use the following flags to build according to a C++ standard: | C++ Standard | Flag | -|--------------|-------------------------| +| ------------ | ----------------------- | | C++98 | Default, no flag needed | | C++11 | `--cxxopt=-std=c++11` | | C++14 | `--cxxopt=-std=c++14` | @@ -282,11 +238,9 @@ For example: bazel build //:app --cxxopt=-std=c++11 ``` -Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and -`--linkopt` in the [User Manual](/docs/user-manual#cxxopt). +Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and `--linkopt` in the [User Manual](/docs/user-manual#cxxopt). -Compiler and linker flags can also be specified as attributes in `cc_library` -using `copts` and `linkopts`. For example: +Compiler and linker flags can also be specified as attributes in `cc_library` using `copts` and `linkopts`. For example: ```python cc_library( @@ -299,11 +253,9 @@ cc_library( ## Building a `cc_library` for Android without using `android_binary` -To build a standalone `cc_binary` or `cc_library` for Android without using an -`android_binary`, use the `--platforms` flag. +To build a standalone `cc_binary` or `cc_library` for Android without using an `android_binary`, use the `--platforms` flag. -For example, assuming you have defined Android platforms in -`my/platforms/BUILD`: +For example, assuming you have defined Android platforms in `my/platforms/BUILD`: ```posix-terminal bazel build //my/cc/jni:target \ @@ -312,13 +264,9 @@ bazel build //my/cc/jni:target \ With this approach, the entire build tree is affected. -Note: All of the targets on the command line must be compatible with -building for Android when specifying these flags, which may make it difficult to -use [Bazel wild-cards](/run/build#specifying-build-targets) like -`/...` and `:all`. +Note: All of the targets on the command line must be compatible with building for Android when specifying these flags, which may make it difficult to use [Bazel wild-cards](/run/build#specifying-build-targets) like `/...` and `:all`. -These flags can be put into a `bazelrc` config (one for each ABI), in -`project/.bazelrc`: +These flags can be put into a `bazelrc` config (one for each ABI), in `<var>project</var>/.bazelrc`: ``` common:android_x86 --platforms=//my/platforms:x86 @@ -326,7 +274,7 @@ common:android_x86 --platforms=//my/platforms:x86 common:android_armeabi-v7a --platforms=//my/platforms:armeabi-v7a # In general -common:android_ --platforms=//my/platforms: +common:android_<abi> --platforms=//my/platforms:<abi> ``` Then, to build a `cc_library` for `x86` for example, run: @@ -335,7 +283,4 @@ Then, to build a `cc_library` for `x86` for example, run: bazel build //my/cc/jni:target --config=android_x86 ``` -In general, use this method for low-level targets (like `cc_library`) or when -you know exactly what you're building; rely on the automatic configuration -transitions from `android_binary` for high-level targets where you're expecting -to build a lot of targets you don't control. +In general, use this method for low-level targets (like `cc_library`) or when you know exactly what you're building; rely on the automatic configuration transitions from `android_binary` for high-level targets where you're expecting to build a lot of targets you don't control. diff --git a/docs/bazel-and-android.mdx b/docs/bazel-and-android.mdx index bf3625c9..b043a8bc 100644 --- a/docs/bazel-and-android.mdx +++ b/docs/bazel-and-android.mdx @@ -2,44 +2,27 @@ title: 'Android and Bazel' --- - - -This page contains resources that help you use Bazel with Android projects. It -links to a tutorial, build rules, and other information specific to building -Android projects with Bazel. +This page contains resources that help you use Bazel with Android projects. It links to a tutorial, build rules, and other information specific to building Android projects with Bazel. ## Getting started The following resources will help you work with Bazel on Android projects: -* [Tutorial: Building an Android app](/start/android-app ). This - tutorial is a good place to start learning about Bazel commands and concepts, - and how to build Android apps with Bazel. -* [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). - This codelab explains how to build Android apps with Bazel. +- [Tutorial: Building an Android app](/start/android-app). This tutorial is a good place to start learning about Bazel commands and concepts, and how to build Android apps with Bazel. +- [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). This codelab explains how to build Android apps with Bazel. ## Features -Bazel has Android rules for building and testing Android apps, integrating with -the SDK/NDK, and creating emulator images. There are also Bazel plugins for -Android Studio and IntelliJ. - -* [Android rules](/reference/be/android). The Build Encyclopedia describes the rules - for building and testing Android apps with Bazel. -* [Integration with Android Studio](/install/ide). Bazel is compatible with - Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) - plugin. -* [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` - feature provides automated build-and-deploy functionality for building and - testing Android apps directly on Android devices and emulators. -* [Android instrumentation testing](/docs/android-instrumentation-test) on - emulators and devices. -* [Android NDK integration](/docs/android-ndk). Bazel supports compiling to - native code through direct NDK integration and the C++ rules. -* [Android build performance](/docs/android-build-performance). This page - provides information on optimizing build performance for Android apps. +Bazel has Android rules for building and testing Android apps, integrating with the SDK/NDK, and creating emulator images. There are also Bazel plugins for Android Studio and IntelliJ. + +- [Android rules](/reference/be/android). The Build Encyclopedia describes the rules for building and testing Android apps with Bazel. +- [Integration with Android Studio](/install/ide). Bazel is compatible with Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) plugin. +- [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` feature provides automated build-and-deploy functionality for building and testing Android apps directly on Android devices and emulators. +- [Android instrumentation testing](/docs/android-instrumentation-test) on emulators and devices. +- [Android NDK integration](/docs/android-ndk). Bazel supports compiling to native code through direct NDK integration and the C++ rules. +- [Android build performance](/docs/android-build-performance). This page provides information on optimizing build performance for Android apps. ## Further reading -* Integrating with dependencies from Google Maven and Maven Central with [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external). -* Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). +- Integrating with dependencies from Google Maven and Maven Central with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external). +- Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). diff --git a/docs/bazel-and-apple.mdx b/docs/bazel-and-apple.mdx index 6e4a06fe..430c0f1f 100644 --- a/docs/bazel-and-apple.mdx +++ b/docs/bazel-and-apple.mdx @@ -2,85 +2,52 @@ title: 'Apple Apps and Bazel' --- - - -This page contains resources that help you use Bazel to build macOS and iOS -projects. It links to a tutorial, build rules, and other information specific to -using Bazel to build and test for those platforms. +This page contains resources that help you use Bazel to build macOS and iOS projects. It links to a tutorial, build rules, and other information specific to using Bazel to build and test for those platforms. ## Working with Bazel The following resources will help you work with Bazel on macOS and iOS projects: -* [Tutorial: Building an iOS app](/start/ios-app) -* [Objective-C build rules](/reference/be/objective-c) -* [General Apple rules](https://github.com/bazelbuild/rules_apple) -* [Integration with Xcode](/install/ide) +- [Tutorial: Building an iOS app](/start/ios-app) +- [Objective-C build rules](/reference/be/objective-c) +- [General Apple rules](https://github.com/bazelbuild/rules_apple) +- [Integration with Xcode](/install/ide) ## Migrating to Bazel -If you currently build your macOS and iOS projects with Xcode, follow the steps -in the migration guide to start building them with Bazel: +If you currently build your macOS and iOS projects with Xcode, follow the steps in the migration guide to start building them with Bazel: -* [Migrating from Xcode to Bazel](/migrate/xcode) +- [Migrating from Xcode to Bazel](/migrate/xcode) ## Apple apps and new rules -**Note**: Creating new rules is for advanced build and test scenarios. -You do not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. -The following modules, configuration fragments, and providers will help you -[extend Bazel's capabilities](/extending/concepts) -when building your macOS and iOS projects: +The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your macOS and iOS projects: -* Modules: +- Modules: - * [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) - * [`apple_common`](/rules/lib/toplevel/apple_common) - * [`apple_platform`](/rules/lib/builtins/apple_platform) - * [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) - * [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) + - [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) + - [`apple_common`](/rules/lib/toplevel/apple_common) + - [`apple_platform`](/rules/lib/builtins/apple_platform) + - [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) + - [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) -* Configuration fragments: +- Configuration fragments: - * [`apple`](/rules/lib/fragments/apple) + - [`apple`](/rules/lib/fragments/apple) -* Providers: +- Providers: - * [`ObjcProvider`](/rules/lib/providers/ObjcProvider) - * [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) + - [`ObjcProvider`](/rules/lib/providers/ObjcProvider) + - [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) ## Xcode selection -If your build requires Xcode, Bazel will select an appropriate version based on -the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes -the set of available Xcode versions and sets a default version if -`--xcode_version` is not passed. This default is overridden by the -`--xcode_version` flag, as long as it is set to an Xcode version that is -represented in the `--xcode_config` target. - -If you do not pass `--xcode_config`, Bazel will use the autogenerated -[`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the -Xcode versions available on your host machine. The default version is -the newest available Xcode version. This is appropriate for local execution. - -If you are performing remote builds, you should set `--xcode_config` to an -[`xcode_config`](/reference/be/objective-c#xcode_config) -target whose `versions` attribute is a list of remotely available -[`xcode_version`](/reference/be/objective-c#xcode_version) -targets, and whose `default` attribute is one of these -[`xcode_versions`](/reference/be/objective-c#xcode_version). - -If you are using dynamic execution, you should set `--xcode_config` to an -[`xcode_config`](/reference/be/objective-c#xcode_config) -target whose `remote_versions` attribute is an -[`available_xcodes`](/reference/be/workspace#available_xcodes) -target containing the remotely available Xcode versions, and whose -`local_versions` attribute is an -[`available_xcodes`](/reference/be/workspace#available_xcodes) -target containing the locally available Xcode versions. For `local_versions`, -you probably want to use the autogenerated -`@local_config_xcode//:host_available_xcodes`. The default Xcode version is the -newest mutually available version, if there is one, otherwise the default of the -`local_versions` target. If you prefer to use the `local_versions` default -as the default, you can pass `--experimental_prefer_mutual_default=false`. +If your build requires Xcode, Bazel will select an appropriate version based on the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes the set of available Xcode versions and sets a default version if `--xcode_version` is not passed. This default is overridden by the `--xcode_version` flag, as long as it is set to an Xcode version that is represented in the `--xcode_config` target. + +If you do not pass `--xcode_config`, Bazel will use the autogenerated [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the Xcode versions available on your host machine. The default version is the newest available Xcode version. This is appropriate for local execution. + +If you are performing remote builds, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `versions` attribute is a list of remotely available [`xcode_version`](/reference/be/objective-c#xcode_version) targets, and whose `default` attribute is one of these [`xcode_versions`](/reference/be/objective-c#xcode_version). + +If you are using dynamic execution, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `remote_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the remotely available Xcode versions, and whose `local_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the locally available Xcode versions. For `local_versions`, you probably want to use the autogenerated `@local_config_xcode//:host_available_xcodes`. The default Xcode version is the newest mutually available version, if there is one, otherwise the default of the `local_versions` target. If you prefer to use the `local_versions` default as the default, you can pass `--experimental_prefer_mutual_default=false`. diff --git a/docs/bazel-and-cpp.mdx b/docs/bazel-and-cpp.mdx index 73e502cb..17ecb7c8 100644 --- a/docs/bazel-and-cpp.mdx +++ b/docs/bazel-and-cpp.mdx @@ -2,101 +2,78 @@ title: 'C++ and Bazel' --- - - -This page contains resources that help you use Bazel with C++ projects. It links -to a tutorial, build rules, and other information specific to building C++ -projects with Bazel. +This page contains resources that help you use Bazel with C++ projects. It links to a tutorial, build rules, and other information specific to building C++ projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on C++ projects: -* [Tutorial: Building a C++ project](/start/cpp) -* [C++ common use cases](/tutorials/cpp-use-cases) -* [C/C++ rules](/reference/be/c-cpp) -* Essential Libraries - - [Abseil](https://abseil.io/docs/cpp/quickstart) - - [Boost](https://github.com/nelhage/rules_boost) - - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) -* [C++ toolchain configuration](/docs/cc-toolchain-config-reference) -* [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) -* [Integrating with C++ rules](/configure/integrate-cpp) +- [Tutorial: Building a C++ project](/start/cpp) + +- [C++ common use cases](/tutorials/cpp-use-cases) + +- [C/C++ rules](/reference/be/c-cpp) + +- Essential Libraries + + - [Abseil](https://abseil.io/docs/cpp/quickstart) + - [Boost](https://github.com/nelhage/rules_boost) + - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) + +- [C++ toolchain configuration](/docs/cc-toolchain-config-reference) + +- [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) + +- [Integrating with C++ rules](/configure/integrate-cpp) ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are -best practices specific to C++ projects. +In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to C++ projects. ### BUILD files Follow the guidelines below when creating your BUILD files: -* Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) - rule target per compilation unit in the directory. - -* You should granularize your C++ libraries as much as - possible to maximize incrementality and parallelize the build. - -* If there is a single source file in `srcs`, name the library the same as - that C++ file's name. This library should contain C++ file(s), any matching - header file(s), and the library's direct dependencies. For example: - - ```python - cc_library( - name = "mylib", - srcs = ["mylib.cc"], - hdrs = ["mylib.h"], - deps = [":lower-level-lib"] - ) - ``` - -* Use one `cc_test` rule target per `cc_library` target in the file. Name the - target `[library-name]_test` and the source file `[library-name]_test.cc`. - For example, a test target for the `mylib` library target shown above would - look like this: - - ```python - cc_test( - name = "mylib_test", - srcs = ["mylib_test.cc"], - deps = [":mylib"] - ) - ``` +- Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) rule target per compilation unit in the directory. + +- You should granularize your C++ libraries as much as possible to maximize incrementality and parallelize the build. + +- If there is a single source file in `srcs`, name the library the same as that C++ file's name. This library should contain C++ file(s), any matching header file(s), and the library's direct dependencies. For example: + + ```python + cc_library( + name = "mylib", + srcs = ["mylib.cc"], + hdrs = ["mylib.h"], + deps = [":lower-level-lib"] + ) + ``` + +- Use one `cc_test` rule target per `cc_library` target in the file. Name the target `[library-name]_test` and the source file `[library-name]_test.cc`. For example, a test target for the `mylib` library target shown above would look like this: + + ```python + cc_test( + name = "mylib_test", + srcs = ["mylib_test.cc"], + deps = [":mylib"] + ) + ``` ### Include paths Follow these guidelines for include paths: -* Make all include paths relative to the workspace directory. +- Make all include paths relative to the workspace directory. -* Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not - angle-brackets (`#include <foo/bar/baz.h>`). +- Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not angle-brackets (`#include <foo/bar/baz.h>`). -* Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` - (parent directory). +- Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` (parent directory). -* For legacy or `third_party` code that requires includes pointing outside the - project repository, such as external repository includes requiring a prefix, - use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and - [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) - arguments on the `cc_library` rule target. +- For legacy or `third_party` code that requires includes pointing outside the project repository, such as external repository includes requiring a prefix, use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) arguments on the `cc_library` rule target. ### Toolchain features -The following optional [features](/docs/cc-toolchain-config-reference#features) -can improve the hygiene of a C++ project. They can be enabled using the -`--features` command-line flag or the `features` attribute of -[`repo`](/external/overview#repo.bazel), -[`package`](/reference/be/functions#package) or `cc_*` rules: - -* The `parse_headers` feature makes it so that the C++ compiler is used to parse - (but not compile) all header files in the built targets and their dependencies - when using the - [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) - flag. This can help catch issues in header-only libraries and ensure that - headers are self-contained and independent of the order in which they are - included. -* The `layering_check` feature enforces that targets only include headers - provided by their direct dependencies. The default toolchain supports this - feature on Linux with `clang` as the compiler. +The following optional [features](/docs/cc-toolchain-config-reference#features) can improve the hygiene of a C++ project. They can be enabled using the `--features` command-line flag or the `features` attribute of [`repo`](/external/overview#repo.bazel), [`package`](/reference/be/functions#package) or `cc_*` rules: + +- The `parse_headers` feature makes it so that the C++ compiler is used to parse (but not compile) all header files in the built targets and their dependencies when using the [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) flag. This can help catch issues in header-only libraries and ensure that headers are self-contained and independent of the order in which they are included. +- The `layering_check` feature enforces that targets only include headers provided by their direct dependencies. The default toolchain supports this feature on Linux with `clang` as the compiler. diff --git a/docs/bazel-and-java.mdx b/docs/bazel-and-java.mdx index bd5af010..f8aa4d9a 100644 --- a/docs/bazel-and-java.mdx +++ b/docs/bazel-and-java.mdx @@ -2,164 +2,120 @@ title: 'Java and Bazel' --- - - -This page contains resources that help you use Bazel with Java projects. It -links to a tutorial, build rules, and other information specific to building -Java projects with Bazel. +This page contains resources that help you use Bazel with Java projects. It links to a tutorial, build rules, and other information specific to building Java projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on Java projects: -* [Tutorial: Building a Java Project](/start/java) -* [Java rules](/reference/be/java) +- [Tutorial: Building a Java Project](/start/java) +- [Java rules](/reference/be/java) ## Migrating to Bazel -If you currently build your Java projects with Maven, follow the steps in the -migration guide to start building your Maven projects with Bazel: +If you currently build your Java projects with Maven, follow the steps in the migration guide to start building your Maven projects with Bazel: -* [Migrating from Maven to Bazel](/migrate/maven) +- [Migrating from Maven to Bazel](/migrate/maven) ## Java versions There are two relevant versions of Java that are set with configuration flags: -* the version of the source files in the repository -* the version of the Java runtime that is used to execute the code and to test - it +- the version of the source files in the repository +- the version of the Java runtime that is used to execute the code and to test it ### Configuring the version of the source code in your repository -Without an additional configuration, Bazel assumes all Java source files in the -repository are written in a single Java version. To specify the version of the -sources in the repository add `build --java_language_version={ver}` to -`.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners -should set this flag so that Bazel and its users can reference the source code's -Java version number. For more details, see -[Java language version flag](/docs/user-manual#java-language-version). +Without an additional configuration, Bazel assumes all Java source files in the repository are written in a single Java version. To specify the version of the sources in the repository add `build --java_language_version={ver}` to `.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners should set this flag so that Bazel and its users can reference the source code's Java version number. For more details, see [Java language version flag](/docs/user-manual#java-language-version). ### Configuring the JVM used to execute and test the code Bazel uses one JDK for compilation and another JVM to execute and test the code. -By default Bazel compiles the code using a JDK it downloads and it executes and -tests the code with the JVM installed on the local machine. Bazel searches for -the JVM using `JAVA_HOME` or path. +By default Bazel compiles the code using a JDK it downloads and it executes and tests the code with the JVM installed on the local machine. Bazel searches for the JVM using `JAVA_HOME` or path. -The resulting binaries are compatible with locally installed JVM in system -libraries, which means the resulting binaries depend on what is installed on the -machine. +The resulting binaries are compatible with locally installed JVM in system libraries, which means the resulting binaries depend on what is installed on the machine. -To configure the JVM used for execution and testing use `--java_runtime_version` -flag. The default value is `local_jdk`. +To configure the JVM used for execution and testing use `--java_runtime_version` flag. The default value is `local_jdk`. ### Hermetic testing and compilation -To create a hermetic compile, you can use command line flag -`--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and -tested on the JVM downloaded from a remote repository. For more details, see -[Java runtime version flag](/docs/user-manual#java_runtime_version). +To create a hermetic compile, you can use command line flag `--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and tested on the JVM downloaded from a remote repository. For more details, see [Java runtime version flag](/docs/user-manual#java_runtime_version). ### Configuring compilation and execution of build tools in Java -There is a second pair of JDK and JVM used to build and execute tools, which are -used in the build process, but are not in the build results. That JDK and JVM -are controlled using `--tool_java_language_version` and -`--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, -respectively. +There is a second pair of JDK and JVM used to build and execute tools, which are used in the build process, but are not in the build results. That JDK and JVM are controlled using `--tool_java_language_version` and `--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, respectively. #### Compiling using locally installed JDK -Bazel by default compiles using remote JDK, because it is overriding JDK's -internals. The compilation toolchains using locally installed JDK are configured, -however not used. +Bazel by default compiles using remote JDK, because it is overriding JDK's internals. The compilation toolchains using locally installed JDK are configured, however not used. -To compile using locally installed JDK, that is use the compilation toolchains -for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, -however, mind that this may not work on JDK of arbitrary vendors. +To compile using locally installed JDK, that is use the compilation toolchains for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, however, mind that this may not work on JDK of arbitrary vendors. -For more details, see -[configuring Java toolchains](#config-java-toolchains). +For more details, see [configuring Java toolchains](#config-java-toolchains). ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are -best practices specific to Java projects. +In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to Java projects. ### Directory structure -Prefer Maven's standard directory layout (sources under `src/main/java`, tests -under `src/test/java`). +Prefer Maven's standard directory layout (sources under `src/main/java`, tests under `src/test/java`). ### BUILD files Follow these guidelines when creating your `BUILD` files: -* Use one `BUILD` file per directory containing Java sources, because this - improves build performance. +- Use one `BUILD` file per directory containing Java sources, because this improves build performance. -* Every `BUILD` file should contain one `java_library` rule that looks like - this: +- Every `BUILD` file should contain one `java_library` rule that looks like this: - ```python - java_library( - name = "directory-name", - srcs = glob(["*.java"]), - deps = [...], - ) - ``` + ```python + java_library( + name = "directory-name", + srcs = glob(["*.java"]), + deps = [...], + ) + ``` -* The name of the library should be the name of the directory containing the - `BUILD` file. This makes the label of the library shorter, that is use - `"//package"` instead of `"//package:package"`. +- The name of the library should be the name of the directory containing the `BUILD` file. This makes the label of the library shorter, that is use `"//package"` instead of `"//package:package"`. -* The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of - all Java files in the directory. +- The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of all Java files in the directory. -* Tests should be in a matching directory under `src/test` and depend on this - library. +- Tests should be in a matching directory under `src/test` and depend on this library. ## Creating new rules for advanced Java builds -**Note**: Creating new rules is for advanced build and test scenarios. You do -not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. + +The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your Java projects: + +- Main Java module: [`java_common`](/rules/lib/toplevel/java_common) + +- Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) -The following modules, configuration fragments, and providers will help you -[extend Bazel's capabilities](/extending/concepts) when building your Java -projects: +- Configuration fragment: [`java`](/rules/lib/fragments/java) -* Main Java module: [`java_common`](/rules/lib/toplevel/java_common) -* Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) -* Configuration fragment: [`java`](/rules/lib/fragments/java) -* Other modules: +- Other modules: - * [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) - * [`java_compilation_info`](/rules/lib/providers/java_compilation_info) - * [`java_output_jars`](/rules/lib/providers/java_output_jars) - * [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) - * [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) + - [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) + - [`java_compilation_info`](/rules/lib/providers/java_compilation_info) + - [`java_output_jars`](/rules/lib/providers/java_output_jars) + - [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) + - [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) ## Configuring the Java toolchains Bazel uses two types of Java toolchains: -* execution, used to execute and test Java binaries, controlled with the - `--java_runtime_version` flag -* compilation, used to compile Java sources, controlled with the - `--java_language_version` flag +- execution, used to execute and test Java binaries, controlled with the `--java_runtime_version` flag +- compilation, used to compile Java sources, controlled with the `--java_language_version` flag ### Configuring additional execution toolchains -Execution toolchain is the JVM, either local or from a repository, with some -additional information about its version, operating system, and CPU -architecture. +Execution toolchain is the JVM, either local or from a repository, with some additional information about its version, operating system, and CPU architecture. -Java execution toolchains may added using the `local_java_repository` or -`remote_java_repository` repo rules in a module extension. Adding the rule makes -the JVM available using a flag. When multiple definitions for the same operating -system and CPU architecture are given, the first one is used. +Java execution toolchains may added using the `local_java_repository` or `remote_java_repository` repo rules in a module extension. Adding the rule makes the JVM available using a flag. When multiple definitions for the same operating system and CPU architecture are given, the first one is used. Example configuration of local JVM in MODULE.bazel: @@ -199,51 +155,29 @@ register_toolchains("@openjdk_canary_linux_arm_toolchain_config_repo//:all") ### Configuring additional compilation toolchains -Compilation toolchain is composed of JDK and multiple tools that Bazel uses -during the compilation and that provides additional features, such as: Error -Prone, strict Java dependencies, header compilation, Android desugaring, -coverage instrumentation, and genclass handling for IDEs. +Compilation toolchain is composed of JDK and multiple tools that Bazel uses during the compilation and that provides additional features, such as: Error Prone, strict Java dependencies, header compilation, Android desugaring, coverage instrumentation, and genclass handling for IDEs. -JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the -aforementioned features. Actual compilation is executed using the internal -compiler by the JDK. The JDK used for compilation is specified by `java_runtime` -attribute of the toolchain. +JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the aforementioned features. Actual compilation is executed using the internal compiler by the JDK. The JDK used for compilation is specified by `java_runtime` attribute of the toolchain. -Bazel overrides some JDK internals. In case of JDK version > 9, -`java.compiler` and `jdk.compiler` modules are patched using JDK's flag -`--patch_module`. In case of JDK version 8, the Java compiler is patched using -`-Xbootclasspath` flag. +Bazel overrides some JDK internals. In case of JDK version \> 9, `java.compiler` and `jdk.compiler` modules are patched using JDK's flag `--patch_module`. In case of JDK version 8, the Java compiler is patched using `-Xbootclasspath` flag. -VanillaJavaBuilder is a second implementation of JavaBuilder, -which does not modify JDK's internal compiler and does not have any of the -additional features. VanillaJavaBuilder is not used by any of the built-in -toolchains. +VanillaJavaBuilder is a second implementation of JavaBuilder, which does not modify JDK's internal compiler and does not have any of the additional features. VanillaJavaBuilder is not used by any of the built-in toolchains. In addition to JavaBuilder, Bazel uses several other tools during compilation. -The `ijar` tool processes `jar` files to remove everything except call -signatures. Resulting jars are called header jars. They are used to improve the -compilation incrementality by only recompiling downstream dependents when the -body of a function changes. +The `ijar` tool processes `jar` files to remove everything except call signatures. Resulting jars are called header jars. They are used to improve the compilation incrementality by only recompiling downstream dependents when the body of a function changes. The `singlejar` tool packs together multiple `jar` files into a single one. -The `genclass` tool post-processes the output of a Java compilation, and produces -a `jar` containing only the class files for sources that were generated by -annotation processors. +The `genclass` tool post-processes the output of a Java compilation, and produces a `jar` containing only the class files for sources that were generated by annotation processors. -The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in -LCOV format. +The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in LCOV format. The `TestRunner` tool executes JUnit 4 tests in a controlled environment. -You can reconfigure the compilation by adding `default_java_toolchain` macro to -a `BUILD` file and registering it either by adding `register_toolchains` rule to -the `MODULE.bazel` file or by using -[`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. +You can reconfigure the compilation by adding `default_java_toolchain` macro to a `BUILD` file and registering it either by adding `register_toolchains` rule to the `MODULE.bazel` file or by using [`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. -The toolchain is only used when the `source_version` attribute matches the -value specified by `--java_language_version` flag. +The toolchain is only used when the `source_version` attribute matches the value specified by `--java_language_version` flag. Example toolchain configuration: @@ -264,37 +198,26 @@ default_java_toolchain( ) ``` -which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` -or by adding `register_toolchains("//:repository_default_toolchain_definition")` -to the workpace. +which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` or by adding `register_toolchains("//:repository_default_toolchain_definition")` to the workpace. Predefined configurations: -- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 -- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of - arbitrary vendors. -- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt - tools (`ijar`, `singlejar`) -- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are - built from sources (this may be useful on operating system with different - libc) +- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions \>= 9 +- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of arbitrary vendors. +- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt tools (`ijar`, `singlejar`) +- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are built from sources (this may be useful on operating system with different libc) #### Configuring JVM and Java compiler flags -You may configure JVM and javac flags either with flags or with - `default_java_toolchain` attributes. +You may configure JVM and javac flags either with flags or with `default_java_toolchain` attributes. -The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and -`--host_javacopt`. +The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and `--host_javacopt`. -The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, -`javabuilder_jvm_opts`, and `turbine_jvm_opts`. +The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, `javabuilder_jvm_opts`, and `turbine_jvm_opts`. #### Package specific Java compiler flags configuration -You can configure different Java compiler flags for specific source -files using `package_configuration` attribute of `default_java_toolchain`. -Please refer to the example below. +You can configure different Java compiler flags for specific source files using `package_configuration` attribute of `default_java_toolchain`. Please refer to the example below. ```python load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolchain") @@ -329,14 +252,11 @@ package_group( #### Multiple versions of Java source code in a single repository -Bazel only supports compiling a single version of Java sources in a build. -build. This means that when building a Java test or an application, all - dependencies are built against the same Java version. +Bazel only supports compiling a single version of Java sources in a build. build. This means that when building a Java test or an application, all dependencies are built against the same Java version. However, separate builds may be executed using different flags. -To make the task of using different flags easier, sets of flags for a specific -version may be grouped with `.bazelrc` configs": +To make the task of using different flags easier, sets of flags for a specific version may be grouped with `.bazelrc` configs": ```python build:java8 --java_language_version=8 @@ -345,5 +265,4 @@ build:java11 --java_language_version=11 build:java11 --java_runtime_version=remotejdk_11 ``` -These configs can be used with the `--config` flag, for example -`bazel test --config=java11 //:java11_test`. +These configs can be used with the `--config` flag, for example `bazel test --config=java11 //:java11_test`. diff --git a/docs/bazel-and-javascript.mdx b/docs/bazel-and-javascript.mdx index 63d80189..4a2c176a 100644 --- a/docs/bazel-and-javascript.mdx +++ b/docs/bazel-and-javascript.mdx @@ -2,23 +2,19 @@ title: 'JavaScript and Bazel' --- - - -This page contains resources that help you use Bazel with JavaScript projects. -It links to build rules and other information specific to building JavaScript -with Bazel. +This page contains resources that help you use Bazel with JavaScript projects. It links to build rules and other information specific to building JavaScript with Bazel. The following resources will help you work with Bazel on JavaScript projects: -* [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) -* [rules_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs -* [rules_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler -* [rules_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier -* [rules_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) -* [rules_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) -* [rules_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) -* [rules_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler -* [rules_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) -* [rules_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) -* [rules_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) -* [rules_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) +- [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) +- [rules\_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs +- [rules\_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler +- [rules\_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier +- [rules\_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) +- [rules\_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) +- [rules\_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) +- [rules\_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler +- [rules\_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) +- [rules\_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) +- [rules\_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) +- [rules\_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) diff --git a/docs/configurable-attributes.mdx b/docs/configurable-attributes.mdx index 958341f3..6ec62150 100644 --- a/docs/configurable-attributes.mdx +++ b/docs/configurable-attributes.mdx @@ -2,15 +2,9 @@ title: 'Configurable Build Attributes' --- +***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. - -**_Configurable attributes_**, commonly known as [`select()`]( -/reference/be/functions#select), is a Bazel feature that lets users toggle the values -of build rule attributes at the command line. - -This can be used, for example, for a multiplatform library that automatically -chooses the appropriate implementation for the architecture, or for a -feature-configurable binary that can be customized at build time. +This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. ## Example @@ -41,60 +35,30 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the -command line. Specifically, `deps` becomes: - - - - - - - - - - - - - - - - - - - - - - -
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
- -`select()` serves as a placeholder for a value that will be chosen based on -*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) -targets. By using `select()` in a configurable attribute, the attribute -effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: + +| | | +| ----------------------------------------------- | ------------------ | +| Command | deps = | +| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | +| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | + +`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either: -* They all resolve to the same value. For example, when running on linux x86, this is unambiguous - `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` - is an unambiguous specialization of `values = {"cpu": "x86"}`. +- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when -nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, -`resources`, `cmd`, and most other attributes. Only a small number of attributes -are *non-configurable*, and these are clearly annotated. For example, -`config_setting`'s own -[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies -under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of -the machine running Bazel (which, thanks to cross-compilation, may be different -than the CPU the target is built for). This is known as a -[configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). Given @@ -138,30 +102,19 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and -`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s -build parameters, which include `--cpu=arm`. The `tools` attribute changes -`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on -`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a -[`config_setting`](/reference/be/general#config_setting) or -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of -expected command line flag settings. By encapsulating these in a target, it's -easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands -them for all builds in all projects. These are specified with -[`config_setting`](/reference/be/general#config_setting)'s -[`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -174,31 +127,21 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` -is the expected value for that flag. `:meaningful_condition_name` matches if -*every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, -[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because -it only affects how Bazel reports progress to the user. Targets can't use that -flag to construct their results. The exact set of supported flags isn't -documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with -[Starlark build settings][BuildSettings]. Unlike built-in flags, these are -defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s -[`flag_values`](/reference/be/general#config_setting.flag_values) -attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: ```python config_setting( @@ -211,29 +154,17 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) -for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. -[`--define`](/reference/command-line-reference#flag--define) -is an alternative legacy syntax for custom flags (for example -`--define foo=bar`). This can be expressed either in the -[values](/reference/be/general#config_setting.values) attribute -(`values = {"define": "foo=bar"}`) or the -[define_values](/reference/be/general#config_setting.define_values) attribute -(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards -compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The -`config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition -matches. +The built-in condition `//conditions:default` matches when no other condition matches. -Because of the "exactly one match" rule, a configurable attribute with no match -and no default condition emits a `"no matching conditions"` error. This can -protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -259,16 +190,11 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s -[`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides -flexibility, it can also be burdensome to individually set each one every time -you want to build a target. - [Platforms](/extending/platforms) -let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -326,12 +252,9 @@ platform( ) ``` -The platform can be specified on the command line. It activates the -`config_setting`s that contain a subset of the platform's `constraint_values`, -allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, -you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -358,11 +281,9 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to -check against single values. +This saves the need for boilerplate `config_setting`s when you only need to check against single values. -Platforms are still under development. See the -[documentation](/concepts/platforms) for details. +Platforms are still under development. See the [documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -384,12 +305,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: - - Duplicate labels can appear in different paths of the same `select`. - - Duplicate labels can *not* appear within the same path of a `select`. - - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -`select` cannot appear inside another `select`. If you need to nest `selects` -and your attribute takes other targets as values, use an intermediate target: +- Duplicate labels can appear in different paths of the same `select`. +- Duplicate labels can *not* appear within the same path of a `select`. +- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) + +`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -410,8 +331,7 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND -chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). ## OR chaining @@ -430,9 +350,7 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and -maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple -times. +Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. One option is to predefine the value as a BUILD variable: @@ -451,18 +369,13 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary -duplication. +This makes it easier to manage the dependency. But it still causes unnecessary duplication. For more direct support, use one of the following: ### `selects.with_or` -The -[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing conditions directly inside a `select`: +The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -481,18 +394,12 @@ sh_binary( ### `selects.config_setting_group` - -The -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing multiple `config_setting`s: +The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` - ```python config_setting( name = "config1", @@ -516,17 +423,13 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across -different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. -It's an error for multiple conditions to match unless one is an unambiguous -"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the -[Skylib](https://github.com/bazelbuild/bazel-skylib) macro -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -551,13 +454,11 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed -inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to -fails with the error: +By default, when no condition matches, the target the `select()` is attached to fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -567,8 +468,7 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) -attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: ```python cc_library( @@ -591,8 +491,7 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable -attributes. For example, given: +Rule implementations receive the *resolved values* of configurable attributes. For example, given: ```python # myapp/BUILD @@ -612,9 +511,7 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native -rules. But *they cannot directly manipulate them*. For example, there's no way -for a macro to convert +Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert ```python select({"foo": "val"}, ...) @@ -628,32 +525,22 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* -because macros are evaluated in Bazel's [loading phase](/run/build#loading), -which occurs before flag values are known. -This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while -technically feasible, lack a coherent UI. Further design is necessary to change -this. +Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's -[loading phase](/reference/glossary#loading-phase). -This means it doesn't know what command line flags a target uses since those -flags aren't evaluated until later in the build (in the -[analysis phase](/reference/glossary#analysis-phase)). -So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has -all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` + ```python # myapp/BUILD @@ -702,14 +589,9 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for -details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. -The key issue this question usually means is that select() doesn't work in -*macros*. These are different than *rules*. See the -documentation on [rules](/extending/rules) and [macros](/extending/macros) -to understand the difference. -Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: Define a rule and macro: @@ -789,9 +671,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before -Bazel reads the build's command line flags. That means there isn't enough -information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -814,9 +694,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition -[can't evaluate `select()`s](#faq-select-macro), any attempt to do so -usually produces an error: +Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -829,8 +707,7 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly -vigilant with them: +Booleans are a special case that fail silently, so you should be particularly vigilant with them: ```sh $ cat myapp/defs.bzl @@ -852,21 +729,13 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. -So what they're really evaluting is the `select()` object itself. According to -[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design -standards, all objects aside from a very small number of exceptions -automatically return true. +This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before -Bazel knows what the build's command line parameters are. Can they at least read -the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). -What you *can* do today is prepare a straight dictionary, then feed it into a -`select()`: +Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: ```sh $ cat myapp/defs.bzl @@ -878,7 +747,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -910,7 +779,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -918,17 +787,11 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo -rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in -the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't -actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in -the `actual` attribute, to perform this type of run-time determination. This -works correctly, since `alias()` is a BUILD rule, and is evaluated with a -specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. You can even have a `bind()` target point to an `alias()`, if needed. @@ -956,37 +819,31 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target -that depends on either `//:ssl` or `//external:ssl` will see the alternative -located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, -use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo +$ bazel cquery //myapp:foo <desired build flags> //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on -//myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' +$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the -configuration that resolves `//myapp:foo`'s `select()`. You can inspect its -values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -1005,18 +862,13 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the -[cquery docs](/query/cquery) for guidance on using `somepath` to get the right -one. +`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the -same command line flags as the `bazel cquery`. The `config` command relies on -the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform -is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. For example: @@ -1039,15 +891,11 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the -`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the -`:x86_linux_platform` defined here? The author of the `BUILD` file and the user -who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with -these constraints: +Instead, define a `config_setting` that matches **any** platform with these constraints: ```py config_setting( @@ -1068,13 +916,11 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what -platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you -can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -1094,7 +940,4 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and -confuses users when the expected condition does not match. - -[BuildSettings]: /extending/config#user-defined-build-settings +The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. diff --git a/docs/mobile-install.mdx b/docs/mobile-install.mdx index 8bc4a679..1d08dcd0 100644 --- a/docs/mobile-install.mdx +++ b/docs/mobile-install.mdx @@ -2,203 +2,100 @@ title: 'bazel mobile-install' --- +Fast iterative development for Android - - -

Fast iterative development for Android

- -This page describes how `bazel mobile-install` makes iterative development -for Android much faster. It describes the benefits of this approach versus the -drawbacks of separate build and install steps. +This page describes how `bazel mobile-install` makes iterative development for Android much faster. It describes the benefits of this approach versus the drawbacks of separate build and install steps. ## Summary To install small changes to an Android app very quickly, do the following: - 1. Find the `android_binary` rule of the app you want to install. - 2. Connect your device to `adb`. - 3. Run `bazel mobile-install :your_target`. App startup will be a little - slower than usual. - 4. Edit the code or Android resources. - 5. Run `bazel mobile-install :your_target`. - 6. Enjoy a fast and minimal incremental installation! +1. Find the `android_binary` rule of the app you want to install. +2. Connect your device to `adb`. +3. Run `bazel mobile-install :your_target`. App startup will be a little slower than usual. +4. Edit the code or Android resources. +5. Run `bazel mobile-install :your_target`. +6. Enjoy a fast and minimal incremental installation! Some command line options to Bazel that may be useful: - - `--adb` tells Bazel which adb binary to use - - `--adb_arg` can be used to add extra arguments to the command line of `adb`. - One useful application of this is to select which device you want to install - to if you have multiple devices connected to your workstation: - `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=` +- `--adb` tells Bazel which adb binary to use +- `--adb_arg` can be used to add extra arguments to the command line of `adb`. One useful application of this is to select which device you want to install to if you have multiple devices connected to your workstation: `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=<SERIAL>` -When in doubt, look at the -[example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), -contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), -or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) +When in doubt, look at the [example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) ## Introduction -One of the most important attributes of a developer's toolchain is speed: there -is a world of difference between changing the code and seeing it run within a -second and having to wait minutes, sometimes hours, before you get any feedback -on whether your changes do what you expect them to. +One of the most important attributes of a developer's toolchain is speed: there is a world of difference between changing the code and seeing it run within a second and having to wait minutes, sometimes hours, before you get any feedback on whether your changes do what you expect them to. -Unfortunately, the traditional Android toolchain for building an .apk entails -many monolithic, sequential steps and all of these have to be done in order to -build an Android app. At Google, waiting five minutes to build a single-line -change was not unusual on larger projects like Google Maps. +Unfortunately, the traditional Android toolchain for building an .apk entails many monolithic, sequential steps and all of these have to be done in order to build an Android app. At Google, waiting five minutes to build a single-line change was not unusual on larger projects like Google Maps. -`bazel mobile-install` makes iterative development for Android much faster by -using a combination of change pruning, work sharding, and clever manipulation of -Android internals, all without changing any of your app's code. +`bazel mobile-install` makes iterative development for Android much faster by using a combination of change pruning, work sharding, and clever manipulation of Android internals, all without changing any of your app's code. ## Problems with traditional app installation Building an Android app has some issues, including: -- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) -is invoked exactly once in the build and it does not know how to reuse work from -previous builds: it dexes every method again, even though only one method was -changed. +- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) is invoked exactly once in the build and it does not know how to reuse work from previous builds: it dexes every method again, even though only one method was changed. -- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 -connection, and larger apps can take a lot of time to upload. The entire app is -uploaded, even if only small parts have changed, for example, a resource or a -single method, so this can be a major bottleneck. +- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 connection, and larger apps can take a lot of time to upload. The entire app is uploaded, even if only small parts have changed, for example, a resource or a single method, so this can be a major bottleneck. -- Compilation to native code. Android L introduced ART, a new Android runtime, -which compiles apps ahead-of-time rather than compiling them just-in-time like -Dalvik. This makes apps much faster at the cost of longer installation -time. This is a good tradeoff for users because they typically install an app -once and use it many times, but results in slower development where an app is -installed many times and each version is run at most a handful of times. +- Compilation to native code. Android L introduced ART, a new Android runtime, which compiles apps ahead-of-time rather than compiling them just-in-time like Dalvik. This makes apps much faster at the cost of longer installation time. This is a good tradeoff for users because they typically install an app once and use it many times, but results in slower development where an app is installed many times and each version is run at most a handful of times. ## The approach of `bazel mobile-install` `bazel mobile-install `makes the following improvements: - - Sharded desugaring and dexing. After building the app's Java code, Bazel - shards the class files into approximately equal-sized parts and invokes `d8` - separately on them. `d8` is not invoked on shards that did not change since - the last build. These shards are then compiled into separate sharded APKs. +- Sharded desugaring and dexing. After building the app's Java code, Bazel shards the class files into approximately equal-sized parts and invokes `d8` separately on them. `d8` is not invoked on shards that did not change since the last build. These shards are then compiled into separate sharded APKs. - - Incremental file transfer. Android resources, .dex files, and native - libraries are removed from the main .apk and are stored in under a separate - mobile-install directory. This makes it possible to update code and Android - resources independently without reinstalling the whole app. Thus, - transferring the files takes less time and only the .dex files that have - changed are recompiled on-device. +- Incremental file transfer. Android resources, .dex files, and native libraries are removed from the main .apk and are stored in under a separate mobile-install directory. This makes it possible to update code and Android resources independently without reinstalling the whole app. Thus, transferring the files takes less time and only the .dex files that have changed are recompiled on-device. - - Sharded installation. Mobile-install uses Android Studio's - [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) - tool to combine sharded APKs on the connected device and provide a cohesive - experience. +- Sharded installation. Mobile-install uses Android Studio's [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) tool to combine sharded APKs on the connected device and provide a cohesive experience. ### Sharded Dexing -Sharded dexing is reasonably straightforward: once the .jar files are built, a -[tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) -shards them into separate .jar files of approximately equal size, then invokes -`d8` on those that were changed since the previous build. The logic that -determines which shards to dex is not specific to Android: it just uses the -general change pruning algorithm of Bazel. - -The first version of the sharding algorithm simply ordered the .class files -alphabetically, then cut the list up into equal-sized parts, but this proved to -be suboptimal: if a class was added or removed (even a nested or an anonymous -one), it would cause all the classes alphabetically after it to shift by one, -resulting in dexing those shards again. Thus, it was decided to shard Java -packages rather than individual classes. Of course, this still results in -dexing many shards if a new package is added or removed, but that is much less -frequent than adding or removing a single class. - -The number of shards is controlled by command-line configuration, using the -`--define=num_dex_shards=N` flag. In an ideal world, Bazel would -automatically determine how many shards are best, but Bazel currently must know -the set of actions (for example, commands to be executed during the build) before -executing any of them, so it cannot determine the optimal number of shards -because it doesn't know how many Java classes there will eventually be in the -app. Generally speaking, the more shards, the faster the build and the -installation will be, but the slower app startup becomes, because the dynamic -linker has to do more work. The sweet spot is usually between 10 and 50 shards. +Sharded dexing is reasonably straightforward: once the .jar files are built, a [tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) shards them into separate .jar files of approximately equal size, then invokes `d8` on those that were changed since the previous build. The logic that determines which shards to dex is not specific to Android: it just uses the general change pruning algorithm of Bazel. + +The first version of the sharding algorithm simply ordered the .class files alphabetically, then cut the list up into equal-sized parts, but this proved to be suboptimal: if a class was added or removed (even a nested or an anonymous one), it would cause all the classes alphabetically after it to shift by one, resulting in dexing those shards again. Thus, it was decided to shard Java packages rather than individual classes. Of course, this still results in dexing many shards if a new package is added or removed, but that is much less frequent than adding or removing a single class. + +The number of shards is controlled by command-line configuration, using the `--define=num_dex_shards=N` flag. In an ideal world, Bazel would automatically determine how many shards are best, but Bazel currently must know the set of actions (for example, commands to be executed during the build) before executing any of them, so it cannot determine the optimal number of shards because it doesn't know how many Java classes there will eventually be in the app. Generally speaking, the more shards, the faster the build and the installation will be, but the slower app startup becomes, because the dynamic linker has to do more work. The sweet spot is usually between 10 and 50 shards. ### Incremental deployment -Incremental APK shard transfer and installation is now handled by the -`apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). -Whereas earlier (native) versions of mobile-install required manually tracking -first-time installations and selectively apply the `--incremental` -flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) -has been greatly simplified. The same mobile-install -invocation can be used regardless of how many times the app has been installed -or reinstalled. - -At a high level, the `apkdeployer` tool is a wrapper around various `adb` -sub-commands. The main entrypoint logic can be found in the -[`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) -class, with other utility classes colocated in the same package. -The `Deployer` class ingests, among other things, a list of paths to split -APKs and a protobuf with information about the installation, and leverages -deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) -in order to create an install session and incrementally deploy app splits. -See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) -and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) -classes for implementation details. +Incremental APK shard transfer and installation is now handled by the `apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). Whereas earlier (native) versions of mobile-install required manually tracking first-time installations and selectively apply the `--incremental` flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) has been greatly simplified. The same mobile-install invocation can be used regardless of how many times the app has been installed or reinstalled. + +At a high level, the `apkdeployer` tool is a wrapper around various `adb` sub-commands. The main entrypoint logic can be found in the [`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) class, with other utility classes colocated in the same package. The `Deployer` class ingests, among other things, a list of paths to split APKs and a protobuf with information about the installation, and leverages deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) in order to create an install session and incrementally deploy app splits. See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) classes for implementation details. ## Results ### Performance -In general, `bazel mobile-install` results in a 4x to 10x speedup of building -and installing large apps after a small change. +In general, `bazel mobile-install` results in a 4x to 10x speedup of building and installing large apps after a small change. The following numbers were computed for a few Google products: - +![](/docs/images/mobile-install-performance.svg) -This, of course, depends on the nature of the change: recompilation after -changing a base library takes more time. +This, of course, depends on the nature of the change: recompilation after changing a base library takes more time. ### Limitations -The tricks the stub application plays don't work in every case. -The following cases highlight where it does not work as expected: +The tricks the stub application plays don't work in every case. The following cases highlight where it does not work as expected: - - Mobile-install is only supported via the Starlark rules of `rules_android`. - See the ["brief history of mobile-install"](#mobile-install-history) for - more detail. +- Mobile-install is only supported via the Starlark rules of `rules_android`. See the ["brief history of mobile-install"](#mobile-install-history) for more detail. - - Only devices running ART are supported. Mobile-install uses API and runtime features - that only exist on devices running ART, not Dalvik. Any Android runtime more - recent than Android L (API 21+) should be compatible. +- Only devices running ART are supported. Mobile-install uses API and runtime features that only exist on devices running ART, not Dalvik. Any Android runtime more recent than Android L (API 21+) should be compatible. - - Bazel itself must be run with a tool Java runtime _and_ language version - of 17 or higher. +- Bazel itself must be run with a tool Java runtime *and* language version of 17 or higher. - - Bazel versions prior to 8.4.0 must specify some additional flags for - mobile-install. See [the Bazel Android tutorial](/start/android-app). These - flags inform Bazel where the Starlark mobile-install aspect is and which - rules are supported. +- Bazel versions prior to 8.4.0 must specify some additional flags for mobile-install. See [the Bazel Android tutorial](/start/android-app). These flags inform Bazel where the Starlark mobile-install aspect is and which rules are supported. ### A brief history of mobile-install -Earlier Bazel versions _natively_ included built-in build and test rules for -popular languages and ecosystems such as C++, Java, and Android. These rules -were therefore referred to as _native_ rules. Bazel 8 (released in 2024) removed -support for these rules because many of them had been migrated to the -[Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) -for more details. - -The legacy native Android rules also supported a legacy _native_ version of -mobile-install functionality. This is referred to as "mobile-install v1" or -"native mobile-install" now. This functionality was deleted in Bazel 8, along -with the built-in Android rules. -Now, all mobile-install functionality, as well as all Android build and test -rules, are implemented in Starlark and reside in the `rules_android` GitHub -repository. The latest version is known as "mobile-install v3" or "MIv3". +Earlier Bazel versions *natively* included built-in build and test rules for popular languages and ecosystems such as C++, Java, and Android. These rules were therefore referred to as *native* rules. Bazel 8 (released in 2024) removed support for these rules because many of them had been migrated to the [Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) for more details. -_Naming note_: There was a "mobile-install **v2**" available only internally -at Google at one point, but this was never published externally, and only v3 -continues to be used for both Google-internal and OSS rules_android deployment. +The legacy native Android rules also supported a legacy *native* version of mobile-install functionality. This is referred to as "mobile-install v1" or "native mobile-install" now. This functionality was deleted in Bazel 8, along with the built-in Android rules. +Now, all mobile-install functionality, as well as all Android build and test rules, are implemented in Starlark and reside in the `rules_android` GitHub repository. The latest version is known as "mobile-install v3" or "MIv3". +*Naming note*: There was a "mobile-install **v2**" available only internally at Google at one point, but this was never published externally, and only v3 continues to be used for both Google-internal and OSS rules\_android deployment. diff --git a/docs/sandboxing.mdx b/docs/sandboxing.mdx index 68697953..b12027ec 100644 --- a/docs/sandboxing.mdx +++ b/docs/sandboxing.mdx @@ -2,119 +2,49 @@ title: 'Sandboxing' --- +This article covers sandboxing in Bazel and debugging your sandboxing environment. +*Sandboxing* is a permission restricting strategy that isolates processes from each other or from resources in a system. For Bazel, this means restricting file system access. -This article covers sandboxing in Bazel and debugging your sandboxing -environment. +Bazel's file system sandbox runs processes in a working directory that only contains known inputs, such that compilers and other tools don't see source files they should not access, unless they know the absolute paths to them. -*Sandboxing* is a permission restricting strategy that isolates processes from -each other or from resources in a system. For Bazel, this means restricting file -system access. +Sandboxing doesn't hide the host environment in any way. Processes can freely access all files on the file system. However, on platforms that support user namespaces, processes can't modify any files outside their working directory. This ensures that the build graph doesn't have hidden dependencies that could affect the reproducibility of the build. -Bazel's file system sandbox runs processes in a working directory that only -contains known inputs, such that compilers and other tools don't see source -files they should not access, unless they know the absolute paths to them. - -Sandboxing doesn't hide the host environment in any way. Processes can freely -access all files on the file system. However, on platforms that support user -namespaces, processes can't modify any files outside their working directory. -This ensures that the build graph doesn't have hidden dependencies that could -affect the reproducibility of the build. - -More specifically, Bazel constructs an `execroot/` directory for each action, -which acts as the action's work directory at execution time. `execroot/` -contains all input files to the action and serves as the container for any -generated outputs. Bazel then uses an operating-system-provided technique, -containers on Linux and `sandbox-exec` on macOS, to constrain the action within -`execroot/`. +More specifically, Bazel constructs an `execroot/` directory for each action, which acts as the action's work directory at execution time. `execroot/` contains all input files to the action and serves as the container for any generated outputs. Bazel then uses an operating-system-provided technique, containers on Linux and `sandbox-exec` on macOS, to constrain the action within `execroot/`. ## Reasons for sandboxing -- Without action sandboxing, Bazel doesn't know if a tool uses undeclared - input files (files that are not explicitly listed in the dependencies of an - action). When one of the undeclared input files changes, Bazel still - believes that the build is up-to-date and won’t rebuild the action. This can - result in an incorrect incremental build. +- Without action sandboxing, Bazel doesn't know if a tool uses undeclared input files (files that are not explicitly listed in the dependencies of an action). When one of the undeclared input files changes, Bazel still believes that the build is up-to-date and won’t rebuild the action. This can result in an incorrect incremental build. -- Incorrect reuse of cache entries creates problems during remote caching. A - bad cache entry in a shared cache affects every developer on the project, - and wiping the entire remote cache is not a feasible solution. +- Incorrect reuse of cache entries creates problems during remote caching. A bad cache entry in a shared cache affects every developer on the project, and wiping the entire remote cache is not a feasible solution. -- Sandboxing mimics the behavior of remote execution — if a build works well - with sandboxing, it will likely also work with remote execution. By making - remote execution upload all necessary files (including local tools), you can - significantly reduce maintenance costs for compile clusters compared to - having to install the tools on every machine in the cluster every time you - want to try out a new compiler or make a change to an existing tool. +- Sandboxing mimics the behavior of remote execution — if a build works well with sandboxing, it will likely also work with remote execution. By making remote execution upload all necessary files (including local tools), you can significantly reduce maintenance costs for compile clusters compared to having to install the tools on every machine in the cluster every time you want to try out a new compiler or make a change to an existing tool. ## What sandbox strategy to use -You can choose which kind of sandboxing to use, if any, with the -[strategy flags](user-manual.html#strategy-options). Using the `sandboxed` -strategy makes Bazel pick one of the sandbox implementations listed below, -preferring an OS-specific sandbox to the less hermetic generic one. -[Persistent workers](/remote/persistent) run in a generic sandbox if you pass -the `--worker_sandboxing` flag. - -The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. -It simply executes the action's command line with the working directory set to -the execroot of your workspace. - -`processwrapper-sandbox` is a sandboxing strategy that does not require any -"advanced" features - it should work on any POSIX system out of the box. It -builds a sandbox directory consisting of symlinks that point to the original -source files, executes the action's command line with the working directory set -to this directory instead of the execroot, then moves the known output artifacts -out of the sandbox into the execroot and deletes the sandbox. This prevents the -action from accidentally using any input files that are not declared and from -littering the execroot with unknown output files. - -`linux-sandbox` goes one step further and builds on top of the -`processwrapper-sandbox`. Similar to what Docker does under the hood, it uses -Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the -action from the host. That is, it makes the entire filesystem read-only except -for the sandbox directory, so the action cannot accidentally modify anything on -the host filesystem. This prevents situations like a buggy test accidentally rm --rf'ing your $HOME directory. Optionally, you can also prevent the action from -accessing the network. `linux-sandbox` uses PID namespaces to prevent the action -from seeing any other processes and to reliably kill all processes (even daemons -spawned by the action) at the end. - -`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool -to achieve roughly the same as the Linux sandbox. - -Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" -scenario due to restrictions in the mechanisms provided by the operating -systems. Because Docker also uses Linux namespaces for its container magic, you -cannot easily run `linux-sandbox` inside a Docker container, unless you use -`docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a -process that's already being sandboxed. Thus, in these cases, Bazel -automatically falls back to using `processwrapper-sandbox`. - -If you would rather get a build error — such as to not accidentally build with a -less strict execution strategy — explicitly modify the list of execution -strategies that Bazel tries to use (for example, `bazel build ---spawn_strategy=worker,linux-sandbox`). - -Dynamic execution usually requires sandboxing for local execution. To opt out, -pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently -sandboxes [persistent workers](/remote/persistent). +You can choose which kind of sandboxing to use, if any, with the [strategy flags](user-manual.html#strategy-options). Using the `sandboxed` strategy makes Bazel pick one of the sandbox implementations listed below, preferring an OS-specific sandbox to the less hermetic generic one. [Persistent workers](/remote/persistent) run in a generic sandbox if you pass the `--worker_sandboxing` flag. + +The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. It simply executes the action's command line with the working directory set to the execroot of your workspace. + +`processwrapper-sandbox` is a sandboxing strategy that does not require any "advanced" features - it should work on any POSIX system out of the box. It builds a sandbox directory consisting of symlinks that point to the original source files, executes the action's command line with the working directory set to this directory instead of the execroot, then moves the known output artifacts out of the sandbox into the execroot and deletes the sandbox. This prevents the action from accidentally using any input files that are not declared and from littering the execroot with unknown output files. + +`linux-sandbox` goes one step further and builds on top of the `processwrapper-sandbox`. Similar to what Docker does under the hood, it uses Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the action from the host. That is, it makes the entire filesystem read-only except for the sandbox directory, so the action cannot accidentally modify anything on the host filesystem. This prevents situations like a buggy test accidentally rm -rf'ing your $HOME directory. Optionally, you can also prevent the action from accessing the network. `linux-sandbox` uses PID namespaces to prevent the action from seeing any other processes and to reliably kill all processes (even daemons spawned by the action) at the end. + +`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool to achieve roughly the same as the Linux sandbox. + +Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" scenario due to restrictions in the mechanisms provided by the operating systems. Because Docker also uses Linux namespaces for its container magic, you cannot easily run `linux-sandbox` inside a Docker container, unless you use `docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a process that's already being sandboxed. Thus, in these cases, Bazel automatically falls back to using `processwrapper-sandbox`. + +If you would rather get a build error — such as to not accidentally build with a less strict execution strategy — explicitly modify the list of execution strategies that Bazel tries to use (for example, `bazel build --spawn_strategy=worker,linux-sandbox`). + +Dynamic execution usually requires sandboxing for local execution. To opt out, pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently sandboxes [persistent workers](/remote/persistent). ## Downsides to sandboxing -- Sandboxing incurs extra setup and teardown cost. How big this cost is - depends on many factors, including the shape of the build and the - performance of the host OS. For Linux, sandboxed builds are rarely more than - a few percent slower. Setting `--reuse_sandbox_directories` can - mitigate the setup and teardown cost. +- Sandboxing incurs extra setup and teardown cost. How big this cost is depends on many factors, including the shape of the build and the performance of the host OS. For Linux, sandboxed builds are rarely more than a few percent slower. Setting `--reuse_sandbox_directories` can mitigate the setup and teardown cost. -- Sandboxing effectively disables any cache the tool may have. You can - mitigate this by using [persistent workers](/remote/persistent), at - the cost of weaker sandbox guarantees. +- Sandboxing effectively disables any cache the tool may have. You can mitigate this by using [persistent workers](/remote/persistent), at the cost of weaker sandbox guarantees. -- [Multiplex workers](/remote/multiplex) require explicit worker support - to be sandboxed. Workers that do not support multiplex sandboxing run as - singleplex workers under dynamic execution, which can cost extra memory. +- [Multiplex workers](/remote/multiplex) require explicit worker support to be sandboxed. Workers that do not support multiplex sandboxing run as singleplex workers under dynamic execution, which can cost extra memory. ## Debugging @@ -122,11 +52,7 @@ Follow the strategies below to debug issues with sandboxing. ### Deactivated namespaces -On some platforms, such as -[Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) -cluster nodes or Debian, user namespaces are deactivated by default due to -security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file -exists and contains a 0, you can activate user namespaces by running: +On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file exists and contains a 0, you can activate user namespaces by running: ```posix-terminal sudo sysctl kernel.unprivileged_userns_clone=1 @@ -134,16 +60,11 @@ exists and contains a 0, you can activate user namespaces by running: ### Rule execution failures -The sandbox may fail to execute rules because of the system setup. If you see a -message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or -directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for -genrules, and `--spawn_strategy=local` for other rules. +The sandbox may fail to execute rules because of the system setup. If you see a message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for genrules, and `--spawn_strategy=local` for other rules. ### Detailed debugging for build failures -If your build failed, use `--verbose_failures` and `--sandbox_debug` to make -Bazel show the exact command it ran when your build failed, including the part -that sets up the sandbox. +If your build failed, use `--verbose_failures` and `--sandbox_debug` to make Bazel show the exact command it ran when your build failed, including the part that sets up the sandbox. Example error message: @@ -166,9 +87,6 @@ namespace-sandbox failed: error executing command /some/path/to/your/some-compiler --some-params some-target) ``` -You can now inspect the generated sandbox directory and see which files Bazel -created and run the command again to see how it behaves. +You can now inspect the generated sandbox directory and see which files Bazel created and run the command again to see how it behaves. -Note that Bazel does not delete the sandbox directory when you use -`--sandbox_debug`. Unless you are actively debugging, you should disable -`--sandbox_debug` because it fills up your disk over time. +Note that Bazel does not delete the sandbox directory when you use `--sandbox_debug`. Unless you are actively debugging, you should disable `--sandbox_debug` because it fills up your disk over time. diff --git a/extending/aspects.mdx b/extending/aspects.mdx index ac9a0273..c956a8ef 100644 --- a/extending/aspects.mdx +++ b/extending/aspects.mdx @@ -2,32 +2,16 @@ title: 'Aspects' --- +This page explains the basics and benefits of using [aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced examples. +Aspects allow augmenting build dependency graphs with additional information and actions. Some typical scenarios when aspects can be useful: -This page explains the basics and benefits of using -[aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced -examples. - -Aspects allow augmenting build dependency graphs with additional information -and actions. Some typical scenarios when aspects can be useful: - -* IDEs that integrate Bazel can use aspects to collect information about the - project. -* Code generation tools can leverage aspects to execute on their inputs in - *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy - of [protobuf](https://developers.google.com/protocol-buffers/) library - definitions, and language-specific rules can use aspects to attach - actions generating protobuf support code for a particular language. +- IDEs that integrate Bazel can use aspects to collect information about the project. +- Code generation tools can leverage aspects to execute on their inputs in *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy of [protobuf](https://developers.google.com/protocol-buffers/) library definitions, and language-specific rules can use aspects to attach actions generating protobuf support code for a particular language. ## Aspect basics -`BUILD` files provide a description of a project’s source code: what source -files are part of the project, what artifacts (_targets_) should be built from -those files, what the dependencies between those files are, etc. Bazel uses -this information to perform a build, that is, it figures out the set of actions -needed to produce the artifacts (such as running compiler or linker) and -executes those actions. Bazel accomplishes this by constructing a _dependency -graph_ between targets and visiting this graph to collect those actions. +`BUILD` files provide a description of a project’s source code: what source files are part of the project, what artifacts (*targets*) should be built from those files, what the dependencies between those files are, etc. Bazel uses this information to perform a build, that is, it figures out the set of actions needed to produce the artifacts (such as running compiler or linker) and executes those actions. Bazel accomplishes this by constructing a *dependency graph* between targets and visiting this graph to collect those actions. Consider the following `BUILD` file: @@ -46,41 +30,21 @@ This `BUILD` file defines a dependency graph shown in the following figure: **Figure 1.** `BUILD` file dependency graph. -Bazel analyzes this dependency graph by calling an implementation function of -the corresponding [rule](/extending/rules) (in this case "java_library") for every -target in the above example. Rule implementation functions generate actions that -build artifacts, such as `.jar` files, and pass information, such as locations -and names of those artifacts, to the reverse dependencies of those targets in -[providers](/extending/rules#providers). - -Aspects are similar to rules in that they have an implementation function that -generates actions and returns providers. However, their power comes from -the way the dependency graph is built for them. An aspect has an implementation -and a list of all attributes it propagates along. Consider an aspect A that -propagates along attributes named "deps". This aspect can be applied to -a target X, yielding an aspect application node A(X). During its application, -aspect A is applied recursively to all targets that X refers to in its "deps" -attribute (all attributes in A's propagation list). - -Thus a single act of applying aspect A to a target X yields a "shadow graph" of -the original dependency graph of targets shown in the following figure: +Bazel analyzes this dependency graph by calling an implementation function of the corresponding [rule](/extending/rules) (in this case "java\_library") for every target in the above example. Rule implementation functions generate actions that build artifacts, such as `.jar` files, and pass information, such as locations and names of those artifacts, to the reverse dependencies of those targets in [providers](/extending/rules#providers). + +Aspects are similar to rules in that they have an implementation function that generates actions and returns providers. However, their power comes from the way the dependency graph is built for them. An aspect has an implementation and a list of all attributes it propagates along. Consider an aspect A that propagates along attributes named "deps". This aspect can be applied to a target X, yielding an aspect application node A(X). During its application, aspect A is applied recursively to all targets that X refers to in its "deps" attribute (all attributes in A's propagation list). + +Thus a single act of applying aspect A to a target X yields a "shadow graph" of the original dependency graph of targets shown in the following figure: ![Build Graph with Aspect](/rules/build-graph-aspects.png "Build graph with aspects") **Figure 2.** Build graph with aspects. -The only edges that are shadowed are the edges along the attributes in -the propagation set, thus the `runtime_deps` edge is not shadowed in this -example. An aspect implementation function is then invoked on all nodes in -the shadow graph similar to how rule implementations are invoked on the nodes -of the original graph. +The only edges that are shadowed are the edges along the attributes in the propagation set, thus the `runtime_deps` edge is not shadowed in this example. An aspect implementation function is then invoked on all nodes in the shadow graph similar to how rule implementations are invoked on the nodes of the original graph. ## Simple example -This example demonstrates how to recursively print the source files for a -rule and all of its dependencies that have a `deps` attribute. It shows -an aspect implementation, an aspect definition, and how to invoke the aspect -from the Bazel command line. +This example demonstrates how to recursively print the source files for a rule and all of its dependencies that have a `deps` attribute. It shows an aspect implementation, an aspect definition, and how to invoke the aspect from the Bazel command line. ```python def _print_aspect_impl(target, ctx): @@ -111,25 +75,16 @@ print_aspect = aspect( required_providers = [CcInfo], ) ``` -Aspect definitions are similar to rule definitions, and defined using -the [`aspect`](/rules/lib/globals/bzl#aspect) function. -Just like a rule, an aspect has an implementation function which in this case is -``_print_aspect_impl``. +Aspect definitions are similar to rule definitions, and defined using the [`aspect`](/rules/lib/globals/bzl#aspect) function. + +Just like a rule, an aspect has an implementation function which in this case is `_print_aspect_impl`. -``attr_aspects`` is a list of rule attributes along which the aspect propagates. -In this case, the aspect will propagate along the ``deps`` attribute of the -rules that it is applied to. +`attr_aspects` is a list of rule attributes along which the aspect propagates. In this case, the aspect will propagate along the `deps` attribute of the rules that it is applied to. -Another common argument for `attr_aspects` is `['*']` which would propagate the -aspect to all attributes of a rule. +Another common argument for `attr_aspects` is `['*']` which would propagate the aspect to all attributes of a rule. -``required_providers`` is a list of providers that allows the aspect to limit -its propagation to only the targets whose rules advertise its required -providers. For more details consult -[the documentation of the aspect function](/rules/lib/globals/bzl#aspect). -In this case, the aspect will only apply on targets that declare `CcInfo` -provider. +`required_providers` is a list of providers that allows the aspect to limit its propagation to only the targets whose rules advertise its required providers. For more details consult [the documentation of the aspect function](/rules/lib/globals/bzl#aspect). In this case, the aspect will only apply on targets that declare `CcInfo` provider. ### Aspect implementation @@ -145,48 +100,32 @@ def _print_aspect_impl(target, ctx): return [] ``` -Aspect implementation functions are similar to the rule implementation -functions. They return [providers](/extending/rules#providers), can generate -[actions](/extending/rules#actions), and take two arguments: +Aspect implementation functions are similar to the rule implementation functions. They return [providers](/extending/rules#providers), can generate [actions](/extending/rules#actions), and take two arguments: -* `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. -* `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes - and generate outputs and actions. +- `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. +- `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes and generate outputs and actions. -The implementation function can access the attributes of the target rule via -[`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are -provided by the target to which it is applied (via the `target` argument). +The implementation function can access the attributes of the target rule via [`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are provided by the target to which it is applied (via the `target` argument). -Aspects are required to return a list of providers. In this example, the aspect -does not provide anything, so it returns an empty list. +Aspects are required to return a list of providers. In this example, the aspect does not provide anything, so it returns an empty list. ### Invoking the aspect using the command line -The simplest way to apply an aspect is from the command line using the -[`--aspects`](/reference/command-line-reference#flag--aspects) -argument. Assuming the aspect above were defined in a file named `print.bzl` -this: +The simplest way to apply an aspect is from the command line using the [`--aspects`](/reference/command-line-reference#flag--aspects) argument. Assuming the aspect above were defined in a file named `print.bzl` this: ```bash bazel build //MyExample:example --aspects print.bzl%print_aspect ``` -would apply the `print_aspect` to the target `example` and all of the -target rules that are accessible recursively via the `deps` attribute. +would apply the `print_aspect` to the target `example` and all of the target rules that are accessible recursively via the `deps` attribute. -The `--aspects` flag takes one argument, which is a specification of the aspect -in the format `%`. +The `--aspects` flag takes one argument, which is a specification of the aspect in the format `<extension file label>%<aspect top-level name>`. ## Advanced example -The following example demonstrates using an aspect from a target rule -that counts files in targets, potentially filtering them by extension. -It shows how to use a provider to return values, how to use parameters to pass -an argument into an aspect implementation, and how to invoke an aspect from a rule. +The following example demonstrates using an aspect from a target rule that counts files in targets, potentially filtering them by extension. It shows how to use a provider to return values, how to use parameters to pass an argument into an aspect implementation, and how to invoke an aspect from a rule. -Note: Aspects added in rules' attributes are called *rule-propagated aspects* as -opposed to *command-line aspects* that are specified using the ``--aspects`` -flag. +Note: Aspects added in rules' attributes are called *rule-propagated aspects* as opposed to *command-line aspects* that are specified using the `--aspects` flag. `file_count.bzl` file: @@ -274,28 +213,15 @@ file_count_aspect = aspect( ) ``` -This example shows how the aspect propagates through the ``deps`` attribute. +This example shows how the aspect propagates through the `deps` attribute. -``attrs`` defines a set of attributes for an aspect. Public aspect attributes -define parameters and can only be of types ``bool``, ``int`` or ``string``. -For rule-propagated aspects, ``int`` and ``string`` parameters must have -``values`` specified on them. This example has a parameter called ``extension`` -that is allowed to have '``*``', '``h``', or '``cc``' as a value. +`attrs` defines a set of attributes for an aspect. Public aspect attributes define parameters and can only be of types `bool`, `int` or `string`. For rule-propagated aspects, `int` and `string` parameters must have `values` specified on them. This example has a parameter called `extension` that is allowed to have '`*`', '`h`', or '`cc`' as a value. -For rule-propagated aspects, parameter values are taken from the rule requesting -the aspect, using the attribute of the rule that has the same name and type. -(see the definition of ``file_count_rule``). +For rule-propagated aspects, parameter values are taken from the rule requesting the aspect, using the attribute of the rule that has the same name and type. (see the definition of `file_count_rule`). -For command-line aspects, the parameters values can be passed using -[``--aspects_parameters``](/reference/command-line-reference#flag--aspects_parameters) -flag. The ``values`` restriction of ``int`` and ``string`` parameters may be -omitted. +For command-line aspects, the parameters values can be passed using [`--aspects_parameters`](/reference/command-line-reference#flag--aspects_parameters) flag. The `values` restriction of `int` and `string` parameters may be omitted. -Aspects are also allowed to have private attributes of types ``label`` or -``label_list``. Private label attributes can be used to specify dependencies on -tools or libraries that are needed for actions generated by aspects. There is not -a private attribute defined in this example, but the following code snippet -demonstrates how you could pass in a tool to an aspect: +Aspects are also allowed to have private attributes of types `label` or `label_list`. Private label attributes can be used to specify dependencies on tools or libraries that are needed for actions generated by aspects. There is not a private attribute defined in this example, but the following code snippet demonstrates how you could pass in a tool to an aspect: ```python ... @@ -333,40 +259,17 @@ def _file_count_aspect_impl(target, ctx): return [FileCountInfo(count = count)] ``` -Just like a rule implementation function, an aspect implementation function -returns a struct of providers that are accessible to its dependencies. - -In this example, the ``FileCountInfo`` is defined as a provider that has one -field ``count``. It is best practice to explicitly define the fields of a -provider using the ``fields`` attribute. - -The set of providers for an aspect application A(X) is the union of providers -that come from the implementation of a rule for target X and from the -implementation of aspect A. The providers that a rule implementation propagates -are created and frozen before aspects are applied and cannot be modified from an -aspect. It is an error if a target and an aspect that is applied to it each -provide a provider with the same type, with the exceptions of -[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) -(which is merged, so long as the -rule and aspect specify different output groups) and -[`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) -(which is taken from the aspect). This means that aspect implementations may -never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). - -The parameters and private attributes are passed in the attributes of the -``ctx``. This example references the ``extension`` parameter and determines -what files to count. - -For returning providers, the values of attributes along which -the aspect is propagated (from the `attr_aspects` list) are replaced with -the results of an application of the aspect to them. For example, if target -X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be [A(Y), A(Z)]. -In this example, ``ctx.rule.attr.deps`` are Target objects that are the -results of applying the aspect to the 'deps' of the original target to which -the aspect has been applied. - -In the example, the aspect accesses the ``FileCountInfo`` provider from the -target's dependencies to accumulate the total transitive number of files. +Just like a rule implementation function, an aspect implementation function returns a struct of providers that are accessible to its dependencies. + +In this example, the `FileCountInfo` is defined as a provider that has one field `count`. It is best practice to explicitly define the fields of a provider using the `fields` attribute. + +The set of providers for an aspect application A(X) is the union of providers that come from the implementation of a rule for target X and from the implementation of aspect A. The providers that a rule implementation propagates are created and frozen before aspects are applied and cannot be modified from an aspect. It is an error if a target and an aspect that is applied to it each provide a provider with the same type, with the exceptions of [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) (which is merged, so long as the rule and aspect specify different output groups) and [`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) (which is taken from the aspect). This means that aspect implementations may never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). + +The parameters and private attributes are passed in the attributes of the `ctx`. This example references the `extension` parameter and determines what files to count. + +For returning providers, the values of attributes along which the aspect is propagated (from the `attr_aspects` list) are replaced with the results of an application of the aspect to them. For example, if target X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be \[A(Y), A(Z)]. In this example, `ctx.rule.attr.deps` are Target objects that are the results of applying the aspect to the 'deps' of the original target to which the aspect has been applied. + +In the example, the aspect accesses the `FileCountInfo` provider from the target's dependencies to accumulate the total transitive number of files. ### Invoking the aspect from a rule @@ -384,13 +287,9 @@ file_count_rule = rule( ) ``` -The rule implementation demonstrates how to access the ``FileCountInfo`` -via the ``ctx.attr.deps``. +The rule implementation demonstrates how to access the `FileCountInfo` via the `ctx.attr.deps`. -The rule definition demonstrates how to define a parameter (``extension``) -and give it a default value (``*``). Note that having a default value that -was not one of '``cc``', '``h``', or '``*``' would be an error due to the -restrictions placed on the parameter in the aspect definition. +The rule definition demonstrates how to define a parameter (`extension`) and give it a default value (`*`). Note that having a default value that was not one of '`cc`', '`h`', or '`*`' would be an error due to the restrictions placed on the parameter in the aspect definition. ### Invoking an aspect through a target rule @@ -409,13 +308,10 @@ file_count_rule( ) ``` -This demonstrates how to pass the ``extension`` parameter into the aspect -via the rule. Since the ``extension`` parameter has a default value in the -rule implementation, ``extension`` would be considered an optional parameter. +This demonstrates how to pass the `extension` parameter into the aspect via the rule. Since the `extension` parameter has a default value in the rule implementation, `extension` would be considered an optional parameter. -When the ``file_count`` target is built, our aspect will be evaluated for -itself, and all of the targets accessible recursively via ``deps``. +When the `file_count` target is built, our aspect will be evaluated for itself, and all of the targets accessible recursively via `deps`. ## References -* [`aspect` API reference](/rules/lib/globals/bzl#aspect) +- [`aspect` API reference](/rules/lib/globals/bzl#aspect) diff --git a/extending/auto-exec-groups.mdx b/extending/auto-exec-groups.mdx index 9fee9783..72af2c3a 100644 --- a/extending/auto-exec-groups.mdx +++ b/extending/auto-exec-groups.mdx @@ -2,18 +2,11 @@ title: 'Automatic Execution Groups (AEGs)' --- - - -Automatic execution groups select an [execution platform][exec_platform] -for each toolchain type. In other words, one target can have multiple -execution platforms without defining execution groups. +Automatic execution groups select an [execution platform](https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs.) for each toolchain type. In other words, one target can have multiple execution platforms without defining execution groups. ## Quick summary -Automatic execution groups are closely connected to toolchains. If you are using -toolchains, you need to set them on the affected actions (actions which use an -executable or a tool from a toolchain) by adding `toolchain` parameter. For -example: +Automatic execution groups are closely connected to toolchains. If you are using toolchains, you need to set them on the affected actions (actions which use an executable or a tool from a toolchain) by adding `toolchain` parameter. For example: ```python ctx.actions.run( @@ -23,15 +16,10 @@ ctx.actions.run( toolchain = '@bazel_tools//tools/jdk:toolchain_type', ) ``` -If the action does not use a tool or executable from a toolchain, and Blaze -doesn't detect that ([the error](#first-error-message) is raised), you can set -`toolchain = None`. -If you need to use multiple toolchains on a single execution platform (an action -uses executable or tools from two or more toolchains), you need to manually -define [exec_groups][exec_groups] (check -[When should I use a custom exec_group?][multiple_toolchains_exec_groups] -section). +If the action does not use a tool or executable from a toolchain, and Blaze doesn't detect that ([the error](#first-error-message) is raised), you can set `toolchain = None`. + +If you need to use multiple toolchains on a single execution platform (an action uses executable or tools from two or more toolchains), you need to manually define [exec\_groups](https://bazel.build/extending/exec-groups) (check [When should I use a custom exec\_group?](/extending/auto-exec-groups#when-should-use-exec-groups) section). ## History @@ -44,20 +32,11 @@ my_rule = rule( ) ``` -Rule `my_rule` registers two toolchain types. This means that the [Toolchain -Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used -to find an execution platform which supports both toolchain types. The selected -execution platform was used for each registered action inside the rule, unless -specified differently with [exec_groups][exec_groups]. -In other words, all actions inside the rule used to have a single execution -platform even if they used tools from different toolchains (execution platform -is selected for each target). This resulted in failures when there was no -execution platform supporting all toolchains. +Rule `my_rule` registers two toolchain types. This means that the [Toolchain Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used to find an execution platform which supports both toolchain types. The selected execution platform was used for each registered action inside the rule, unless specified differently with [exec\_groups](https://bazel.build/extending/exec-groups). In other words, all actions inside the rule used to have a single execution platform even if they used tools from different toolchains (execution platform is selected for each target). This resulted in failures when there was no execution platform supporting all toolchains. ## Current state -With AEGs, the execution platform is selected for each toolchain type. The -implementation function of the earlier example, `my_rule`, would look like: +With AEGs, the execution platform is selected for each toolchain type. The implementation function of the earlier example, `my_rule`, would look like: ```python def _impl(ctx): @@ -74,29 +53,17 @@ def _impl(ctx): ) ``` -This rule creates two actions, the `First action` which uses executable from a -`//tools:toolchain_type_1` and the `Second action` which uses executable from a -`//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed -on a single execution platform which supports both toolchain types. With AEGs, -by adding the `toolchain` parameter inside the actions, each action executes on -the execution platform that provides the toolchain. The actions may be executed -on different execution platforms. +This rule creates two actions, the `First action` which uses executable from a `//tools:toolchain_type_1` and the `Second action` which uses executable from a `//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed on a single execution platform which supports both toolchain types. With AEGs, by adding the `toolchain` parameter inside the actions, each action executes on the execution platform that provides the toolchain. The actions may be executed on different execution platforms. -The same is effective with [ctx.actions.run_shell][run_shell] where `toolchain` -parameter should be added when `tools` are from a toolchain. +The same is effective with [ctx.actions.run\_shell](https://bazel.build/rules/lib/builtins/actions#run_shell) where `toolchain` parameter should be added when `tools` are from a toolchain. ## Difference between custom exec groups and automatic exec groups -As the name suggests, AEGs are exec groups created automatically for each -toolchain type registered on a rule. There is no need to manually specify them, -unlike the "classic" exec groups. Moreover, name of AEG is automatically set to -its toolchain type (e.g. `//tools:toolchain_type_1`). +As the name suggests, AEGs are exec groups created automatically for each toolchain type registered on a rule. There is no need to manually specify them, unlike the "classic" exec groups. Moreover, name of AEG is automatically set to its toolchain type (e.g. `//tools:toolchain_type_1`). -### When should I use a custom exec_group? +### When should I use a custom exec\_group? -Custom exec_groups are needed only in case where multiple toolchains need to -execute on a single execution platform. In all other cases there's no need to -define custom exec_groups. For example: +Custom exec\_groups are needed only in case where multiple toolchains need to execute on a single execution platform. In all other cases there's no need to define custom exec\_groups. For example: ```python def _impl(ctx): @@ -121,9 +88,7 @@ my_rule = rule( ## Migration of AEGs -Internally in google3, Blaze is already using AEGs. -Externally for Bazel, migration is in the process. Some rules are already using -this feature (e.g. Java and C++ rules). +Internally in google3, Blaze is already using AEGs. Externally for Bazel, migration is in the process. Some rules are already using this feature (e.g. Java and C++ rules). ### Which Bazel versions support this migration? @@ -131,8 +96,7 @@ AEGs are fully supported from Bazel 7. ### How to enable AEGs? -Set `--incompatible_auto_exec_groups` to true. More information about the flag -on [the GitHub issue][github_flag]. +Set `--incompatible_auto_exec_groups` to true. More information about the flag on [the GitHub issue](https://github.com/bazelbuild/bazel/issues/17134). ### How to enable AEGs inside a particular rule? @@ -146,38 +110,23 @@ my_rule = rule( } ) ``` -This enables AEGs only in `my_rule` and its actions start using the new logic -when selecting the execution platform. Incompatible flag is overridden with this -attribute. + +This enables AEGs only in `my_rule` and its actions start using the new logic when selecting the execution platform. Incompatible flag is overridden with this attribute. ### How to disable AEGs in case of an error? -Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in -your project ([flag's GitHub issue][github_flag]), or disable a particular rule -by setting `_use_auto_exec_groups` attribute to `False` -([more details about the attribute](#how-enable-particular-rule)). +Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in your project ([flag's GitHub issue](https://github.com/bazelbuild/bazel/issues/17134)), or disable a particular rule by setting `_use_auto_exec_groups` attribute to `False` ([more details about the attribute](#how-enable-particular-rule)). ### Error messages while migrating to AEGs #### Couldn't identify if tools are from implicit dependencies or a toolchain. Please set the toolchain parameter. If you're not using a toolchain, set it to 'None'. - * In this case you get a stack of calls before the error happened and you can - clearly see which exact action needs the toolchain parameter. Check which - toolchain is used for the action and set it with the toolchain param. If no - toolchain is used inside the action for tools or executable, set it to - `None`. -#### Action declared for non-existent toolchain '[toolchain_type]'. - * This means that you've set the toolchain parameter on the action but didn't -register it on the rule. Register the toolchain or set `None` inside the action. +- In this case you get a stack of calls before the error happened and you can clearly see which exact action needs the toolchain parameter. Check which toolchain is used for the action and set it with the toolchain param. If no toolchain is used inside the action for tools or executable, set it to `None`. -## Additional material +#### Action declared for non-existent toolchain '\[toolchain\_type]'. + +- This means that you've set the toolchain parameter on the action but didn't register it on the rule. Register the toolchain or set `None` inside the action. -For more information, check design document: -[Automatic exec groups for toolchains][aegs_design_doc]. +## Additional material -[exec_platform]: https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs. -[exec_groups]: https://bazel.build/extending/exec-groups -[github_flag]: https://github.com/bazelbuild/bazel/issues/17134 -[aegs_design_doc]: https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch -[run_shell]: https://bazel.build/rules/lib/builtins/actions#run_shell -[multiple_toolchains_exec_groups]: /extending/auto-exec-groups#when-should-use-exec-groups +For more information, check design document: [Automatic exec groups for toolchains](https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch). diff --git a/extending/concepts.mdx b/extending/concepts.mdx index e634c611..da9fbd1d 100644 --- a/extending/concepts.mdx +++ b/extending/concepts.mdx @@ -2,111 +2,62 @@ title: 'Extension Overview' --- +This page describes how to extend the BUILD language using macros and rules. - -{/* [TOC] */} - -This page describes how to extend the BUILD language using macros -and rules. - -Bazel extensions are files ending in `.bzl`. Use a -[load statement](/concepts/build-files#load) to import a symbol from an extension. +Bazel extensions are files ending in `.bzl`. Use a [load statement](/concepts/build-files#load) to import a symbol from an extension. Before learning the more advanced concepts, first: -* Read about the [Starlark language](/rules/language), used in both the - `BUILD` and `.bzl` files. +- Read about the [Starlark language](/rules/language), used in both the `BUILD` and `.bzl` files. -* Learn how you can [share variables](/build/share-variables) - between two `BUILD` files. +- Learn how you can [share variables](/build/share-variables) between two `BUILD` files. ## Macros and rules -A macro is a function that instantiates rules. Macros come in two flavors: -[symbolic macros](/extending/macros) (new in Bazel 8) and [legacy -macros](/extending/legacy-macros). The two flavors of macros are defined -differently, but behave almost the same from the point of view of a user. A -macro is useful when a `BUILD` file is getting too repetitive or too complex, as -it lets you reuse some code. The function is evaluated as soon as the `BUILD` -file is read. After the evaluation of the `BUILD` file, Bazel has little -information about macros. If your macro generates a `genrule`, Bazel will -behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one -exception is that targets declared in a symbolic macro have [special visibility -semantics](/extending/macros#visibility): a symbolic macro can hide its internal -targets from the rest of the package.) - -A [rule](/extending/rules) is more powerful than a macro. It can access Bazel -internals and have full control over what is going on. It may for example pass -information to other rules. - -If you want to reuse simple logic, start with a macro; we recommend a symbolic -macro, unless you need to support older Bazel versions. If a macro becomes -complex, it is often a good idea to make it a rule. Support for a new language -is typically done with a rule. Rules are for advanced users, and most users will -never have to write one; they will only load and call existing rules. +A macro is a function that instantiates rules. Macros come in two flavors: [symbolic macros](/extending/macros) (new in Bazel 8) and [legacy macros](/extending/legacy-macros). The two flavors of macros are defined differently, but behave almost the same from the point of view of a user. A macro is useful when a `BUILD` file is getting too repetitive or too complex, as it lets you reuse some code. The function is evaluated as soon as the `BUILD` file is read. After the evaluation of the `BUILD` file, Bazel has little information about macros. If your macro generates a `genrule`, Bazel will behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one exception is that targets declared in a symbolic macro have [special visibility semantics](/extending/macros#visibility): a symbolic macro can hide its internal targets from the rest of the package.) + +A [rule](/extending/rules) is more powerful than a macro. It can access Bazel internals and have full control over what is going on. It may for example pass information to other rules. + +If you want to reuse simple logic, start with a macro; we recommend a symbolic macro, unless you need to support older Bazel versions. If a macro becomes complex, it is often a good idea to make it a rule. Support for a new language is typically done with a rule. Rules are for advanced users, and most users will never have to write one; they will only load and call existing rules. ## Evaluation model A build consists of three phases. -* **Loading phase**. First, load and evaluate all extensions and all `BUILD` - files that are needed for the build. The execution of the `BUILD` files simply - instantiates rules (each time a rule is called, it gets added to a graph). - This is where macros are evaluated. - -* **Analysis phase**. The code of the rules is executed (their `implementation` - function), and actions are instantiated. An action describes how to generate - a set of outputs from a set of inputs, such as "run gcc on hello.c and get - hello.o". You must list explicitly which files will be generated before - executing the actual commands. In other words, the analysis phase takes - the graph generated by the loading phase and generates an action graph. - -* **Execution phase**. Actions are executed, when at least one of their outputs is - required. If a file is missing or if a command fails to generate one output, - the build fails. Tests are also run during this phase. - -Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` -files. A file is read at most once per build and the result of the evaluation is -cached and reused. A file is evaluated only once all its dependencies (`load()` -statements) have been resolved. By design, loading a `.bzl` file has no visible -side-effect, it only defines values and functions. - -Bazel tries to be clever: it uses dependency analysis to know which files must -be loaded, which rules must be analyzed, and which actions must be executed. For -example, if a rule generates actions that you don't need for the current build, -they will not be executed. +- **Loading phase**. First, load and evaluate all extensions and all `BUILD` files that are needed for the build. The execution of the `BUILD` files simply instantiates rules (each time a rule is called, it gets added to a graph). This is where macros are evaluated. + +- **Analysis phase**. The code of the rules is executed (their `implementation` function), and actions are instantiated. An action describes how to generate a set of outputs from a set of inputs, such as "run gcc on hello.c and get hello.o". You must list explicitly which files will be generated before executing the actual commands. In other words, the analysis phase takes the graph generated by the loading phase and generates an action graph. + +- **Execution phase**. Actions are executed, when at least one of their outputs is required. If a file is missing or if a command fails to generate one output, the build fails. Tests are also run during this phase. + +Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` files. A file is read at most once per build and the result of the evaluation is cached and reused. A file is evaluated only once all its dependencies (`load()` statements) have been resolved. By design, loading a `.bzl` file has no visible side-effect, it only defines values and functions. + +Bazel tries to be clever: it uses dependency analysis to know which files must be loaded, which rules must be analyzed, and which actions must be executed. For example, if a rule generates actions that you don't need for the current build, they will not be executed. ## Creating extensions -* [Create your first macro](/rules/macro-tutorial) in order to reuse some code. - Then [learn more about macros](/extending/macros) and [using them to create - "custom verbs"](/rules/verbs-tutorial). +- [Create your first macro](/rules/macro-tutorial) in order to reuse some code. Then [learn more about macros](/extending/macros) and [using them to create "custom verbs"](/rules/verbs-tutorial). -* [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. - Next, you can read more about the [rules concepts](/extending/rules). +- [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. Next, you can read more about the [rules concepts](/extending/rules). -The two links below will be very useful when writing your own extensions. Keep -them within reach: +The two links below will be very useful when writing your own extensions. Keep them within reach: -* The [API reference](/rules/lib) +- The [API reference](/rules/lib) -* [Examples](https://github.com/bazelbuild/examples/tree/master/rules) +- [Examples](https://github.com/bazelbuild/examples/tree/master/rules) ## Going further -In addition to [macros](/extending/macros) and [rules](/extending/rules), you -may want to write [aspects](/extending/aspects) and [repository -rules](/external/repo). +In addition to [macros](/extending/macros) and [rules](/extending/rules), you may want to write [aspects](/extending/aspects) and [repository rules](/external/repo). -* Use [Buildifier](https://github.com/bazelbuild/buildtools) - consistently to format and lint your code. +- Use [Buildifier](https://github.com/bazelbuild/buildtools) consistently to format and lint your code. -* Follow the [`.bzl` style guide](/rules/bzl-style). +- Follow the [`.bzl` style guide](/rules/bzl-style). -* [Test](/rules/testing) your code. +- [Test](/rules/testing) your code. -* [Generate documentation](https://skydoc.bazel.build/) to help your users. +- [Generate documentation](https://skydoc.bazel.build/) to help your users. -* [Optimize the performance](/rules/performance) of your code. +- [Optimize the performance](/rules/performance) of your code. -* [Deploy](/rules/deploying) your extensions to other people. +- [Deploy](/rules/deploying) your extensions to other people. diff --git a/extending/depsets.mdx b/extending/depsets.mdx index 4e84d3f2..1f0d18c4 100644 --- a/extending/depsets.mdx +++ b/extending/depsets.mdx @@ -2,40 +2,21 @@ title: 'Depsets' --- +[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently collecting data across a target’s transitive dependencies. They are an essential element of rule processing. - -[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently -collecting data across a target’s transitive dependencies. They are an essential -element of rule processing. - -The defining feature of depset is its time- and space-efficient union operation. -The depset constructor accepts a list of elements ("direct") and a list of other -depsets ("transitive"), and returns a depset representing a set containing all the -direct elements and the union of all the transitive sets. Conceptually, the -constructor creates a new graph node that has the direct and transitive nodes -as its successors. Depsets have a well-defined ordering semantics, based on -traversal of this graph. +The defining feature of depset is its time- and space-efficient union operation. The depset constructor accepts a list of elements ("direct") and a list of other depsets ("transitive"), and returns a depset representing a set containing all the direct elements and the union of all the transitive sets. Conceptually, the constructor creates a new graph node that has the direct and transitive nodes as its successors. Depsets have a well-defined ordering semantics, based on traversal of this graph. Example uses of depsets include: -* Storing the paths of all object files for a program’s libraries, which can - then be passed to a linker action through a provider. +- Storing the paths of all object files for a program’s libraries, which can then be passed to a linker action through a provider. -* For an interpreted language, storing the transitive source files that are - included in an executable's runfiles. +- For an interpreted language, storing the transitive source files that are included in an executable's runfiles. ## Description and operations -Conceptually, a depset is a directed acyclic graph (DAG) that typically looks -similar to the target graph. It is constructed from the leaves up to the root. -Each target in a dependency chain can add its own contents on top of the -previous without having to read or copy them. +Conceptually, a depset is a directed acyclic graph (DAG) that typically looks similar to the target graph. It is constructed from the leaves up to the root. Each target in a dependency chain can add its own contents on top of the previous without having to read or copy them. -Each node in the DAG holds a list of direct elements and a list of child nodes. -The contents of the depset are the transitive elements, such as the direct elements -of all the nodes. A new depset can be created using the -[depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct -elements and another list of child nodes. +Each node in the DAG holds a list of direct elements and a list of child nodes. The contents of the depset are the transitive elements, such as the direct elements of all the nodes. A new depset can be created using the [depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct elements and another list of child nodes. ```python s = depset(["a", "b", "c"]) @@ -45,11 +26,7 @@ print(s) # depset(["a", "b", "c"]) print(t) # depset(["d", "e", "a", "b", "c"]) ``` -To retrieve the contents of a depset, use the -[to_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive -elements, not including duplicates. There is no way to directly inspect the -precise structure of the DAG, although this structure does affect the order in -which the elements are returned. +To retrieve the contents of a depset, use the [to\_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive elements, not including duplicates. There is no way to directly inspect the precise structure of the DAG, although this structure does affect the order in which the elements are returned. ```python s = depset(["a", "b", "c"]) @@ -58,11 +35,9 @@ print("c" in s.to_list()) # True print(s.to_list() == ["a", "b", "c"]) # True ``` -The allowed items in a depset are restricted, just as the allowed keys in -dictionaries are restricted. In particular, depset contents may not be mutable. +The allowed items in a depset are restricted, just as the allowed keys in dictionaries are restricted. In particular, depset contents may not be mutable. -Depsets use reference equality: a depset is equal to itself, but unequal to any -other depset, even if they have the same contents and same internal structure. +Depsets use reference equality: a depset is equal to itself, but unequal to any other depset, even if they have the same contents and same internal structure. ```python s = depset(["a", "b", "c"]) @@ -86,9 +61,7 @@ t = depset(["c", "b", "a"]) print(sorted(s.to_list()) == sorted(t.to_list())) # True ``` -There is no ability to remove elements from a depset. If this is needed, you -must read out the entire contents of the depset, filter the elements you want to -remove, and reconstruct a new depset. This is not particularly efficient. +There is no ability to remove elements from a depset. If this is needed, you must read out the entire contents of the depset, filter the elements you want to remove, and reconstruct a new depset. This is not particularly efficient. ```python s = depset(["a", "b", "c"]) @@ -105,24 +78,9 @@ print(s) # depset(["a"]) ### Order -The `to_list` operation performs a traversal over the DAG. The kind of traversal -depends on the *order* that was specified at the time the depset was -constructed. It is useful for Bazel to support multiple orders because sometimes -tools care about the order of their inputs. For example, a linker action may -need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the -linker’s command line. Other tools might have the opposite requirement. - -Three traversal orders are supported: `postorder`, `preorder`, and -`topological`. The first two work exactly like [tree -traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) -except that they operate on DAGs and skip already visited nodes. The third order -works as a topological sort from root to leaves, essentially the same as -preorder except that shared children are listed only after all of their parents. -Preorder and postorder operate as left-to-right traversals, but note that within -each node direct elements have no order relative to children. For topological -order, there is no left-to-right guarantee, and even the -all-parents-before-child guarantee does not apply in the case that there are -duplicate elements in different nodes of the DAG. +The `to_list` operation performs a traversal over the DAG. The kind of traversal depends on the *order* that was specified at the time the depset was constructed. It is useful for Bazel to support multiple orders because sometimes tools care about the order of their inputs. For example, a linker action may need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the linker’s command line. Other tools might have the opposite requirement. + +Three traversal orders are supported: `postorder`, `preorder`, and `topological`. The first two work exactly like [tree traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) except that they operate on DAGs and skip already visited nodes. The third order works as a topological sort from root to leaves, essentially the same as preorder except that shared children are listed only after all of their parents. Preorder and postorder operate as left-to-right traversals, but note that within each node direct elements have no order relative to children. For topological order, there is no left-to-right guarantee, and even the all-parents-before-child guarantee does not apply in the case that there are duplicate elements in different nodes of the DAG. ```python # This demonstrates different traversal orders. @@ -151,20 +109,13 @@ print(create("preorder").to_list()) # ["d", "b", "a", "c"] print(create("topological").to_list()) # ["d", "b", "c", "a"] ``` -Due to how traversals are implemented, the order must be specified at the time -the depset is created with the constructor’s `order` keyword argument. If this -argument is omitted, the depset has the special `default` order, in which case -there are no guarantees about the order of any of its elements (except that it -is deterministic). +Due to how traversals are implemented, the order must be specified at the time the depset is created with the constructor’s `order` keyword argument. If this argument is omitted, the depset has the special `default` order, in which case there are no guarantees about the order of any of its elements (except that it is deterministic). ## Full example -This example is available at -[https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). +This example is available at [https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). -Suppose there is a hypothetical interpreted language Foo. In order to build -each `foo_binary` you need to know all the `*.foo` files that it directly or -indirectly depends on. +Suppose there is a hypothetical interpreted language Foo. In order to build each `foo_binary` you need to know all the `*.foo` files that it directly or indirectly depends on. ```python # //depsets:BUILD @@ -208,21 +159,14 @@ foo_binary( import sys if __name__ == "__main__": - assert len(sys.argv) >= 1 + assert len(sys.argv) >= 1 output = open(sys.argv[1], "wt") for path in sys.argv[2:]: input = open(path, "rt") output.write(input.read()) ``` -Here, the transitive sources of the binary `d` are all of the `*.foo` files in -the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` -target to know about any file besides `d.foo`, the `foo_library` targets need to -pass them along in a provider. Each library receives the providers from its own -dependencies, adds its own immediate sources, and passes on a new provider with -the augmented contents. The `foo_binary` rule does the same, except that instead -of returning a provider, it uses the complete list of sources to construct a -command line for an action. +Here, the transitive sources of the binary `d` are all of the `*.foo` files in the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` target to know about any file besides `d.foo`, the `foo_library` targets need to pass them along in a provider. Each library receives the providers from its own dependencies, adds its own immediate sources, and passes on a new provider with the augmented contents. The `foo_binary` rule does the same, except that instead of returning a provider, it uses the complete list of sources to construct a command line for an action. Here’s a complete implementation of the `foo_library` and `foo_binary` rules. @@ -279,15 +223,11 @@ foo_binary = rule( ) ``` -You can test this by copying these files into a fresh package, renaming the -labels appropriately, creating the source `*.foo` files with dummy content, and -building the `d` target. - +You can test this by copying these files into a fresh package, renaming the labels appropriately, creating the source `*.foo` files with dummy content, and building the `d` target. ## Performance -To see the motivation for using depsets, consider what would happen if -`get_transitive_srcs()` collected its sources in a list. +To see the motivation for using depsets, consider what would happen if `get_transitive_srcs()` collected its sources in a list. ```python def get_transitive_srcs(srcs, deps): @@ -298,12 +238,9 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This does not take into account duplicates, so the source files for `a` -will appear twice on the command line and twice in the contents of the output -file. +This does not take into account duplicates, so the source files for `a` will appear twice on the command line and twice in the contents of the output file. -An alternative is using a general set, which can be simulated by a -dictionary where the keys are the elements and all the keys map to `True`. +An alternative is using a general set, which can be simulated by a dictionary where the keys are the elements and all the keys map to `True`. ```python def get_transitive_srcs(srcs, deps): @@ -316,31 +253,16 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This gets rid of the duplicates, but it makes the order of the command line -arguments (and therefore the contents of the files) unspecified, although still -deterministic. +This gets rid of the duplicates, but it makes the order of the command line arguments (and therefore the contents of the files) unspecified, although still deterministic. -Moreover, both approaches are asymptotically worse than the depset-based -approach. Consider the case where there is a long chain of dependencies on -Foo libraries. Processing every rule requires copying all of the transitive -sources that came before it into a new data structure. This means that the -time and space cost for analyzing an individual library or binary target -is proportional to its own height in the chain. For a chain of length n, -foolib_1 ← foolib_2 ← … ← foolib_n, the overall cost is effectively O(n^2). +Moreover, both approaches are asymptotically worse than the depset-based approach. Consider the case where there is a long chain of dependencies on Foo libraries. Processing every rule requires copying all of the transitive sources that came before it into a new data structure. This means that the time and space cost for analyzing an individual library or binary target is proportional to its own height in the chain. For a chain of length n, foolib\_1 ← foolib\_2 ← … ← foolib\_n, the overall cost is effectively O(n^2). -Generally speaking, depsets should be used whenever you are accumulating -information through your transitive dependencies. This helps ensure that -your build scales well as your target graph grows deeper. +Generally speaking, depsets should be used whenever you are accumulating information through your transitive dependencies. This helps ensure that your build scales well as your target graph grows deeper. -Finally, it’s important to not retrieve the contents of the depset -unnecessarily in rule implementations. One call to `to_list()` -at the end in a binary rule is fine, since the overall cost is just O(n). It’s -when many non-terminal targets try to call `to_list()` that quadratic behavior -occurs. +Finally, it’s important to not retrieve the contents of the depset unnecessarily in rule implementations. One call to `to_list()` at the end in a binary rule is fine, since the overall cost is just O(n). It’s when many non-terminal targets try to call `to_list()` that quadratic behavior occurs. For more information about using depsets efficiently, see the [performance](/rules/performance) page. ## API Reference Please see [here](/rules/lib/builtins/depset) for more details. - diff --git a/extending/exec-groups.mdx b/extending/exec-groups.mdx index a8b45b92..6f7e7198 100644 --- a/extending/exec-groups.mdx +++ b/extending/exec-groups.mdx @@ -2,40 +2,21 @@ title: 'Execution Groups' --- - - -Execution groups allow for multiple execution platforms within a single target. -Each execution group has its own [toolchain](/extending/toolchains) dependencies and -performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). +Execution groups allow for multiple execution platforms within a single target. Each execution group has its own [toolchain](/extending/toolchains) dependencies and performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). ## Current status -Execution groups for certain natively declared actions, like `CppLink`, can be -used inside `exec_properties` to set per-action, per-target execution -requirements. For more details, see the -[Default execution groups](#exec-groups-for-native-rules) section. +Execution groups for certain natively declared actions, like `CppLink`, can be used inside `exec_properties` to set per-action, per-target execution requirements. For more details, see the [Default execution groups](#exec-groups-for-native-rules) section. ## Background -Execution groups allow the rule author to define sets of actions, each with a -potentially different execution platform. Multiple execution platforms can allow -actions to execution differently, for example compiling an iOS app on a remote -(linux) worker and then linking/code signing on a local mac worker. +Execution groups allow the rule author to define sets of actions, each with a potentially different execution platform. Multiple execution platforms can allow actions to execution differently, for example compiling an iOS app on a remote (linux) worker and then linking/code signing on a local mac worker. -Being able to define groups of actions also helps alleviate the usage of action -mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be -unique and can only reference a single action. This is especially helpful in -allocating extra resources to specific memory and processing intensive actions -like linking in C++ builds without over-allocating to less demanding tasks. +Being able to define groups of actions also helps alleviate the usage of action mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be unique and can only reference a single action. This is especially helpful in allocating extra resources to specific memory and processing intensive actions like linking in C++ builds without over-allocating to less demanding tasks. ## Defining execution groups -During rule definition, rule authors can -[declare](/rules/lib/globals/bzl#exec_group) -a set of execution groups. On each execution group, the rule author can specify -everything needed to select an execution platform for that execution group, -namely any constraints via `exec_compatible_with` and toolchain types via -`toolchain`. +During rule definition, rule authors can [declare](/rules/lib/globals/bzl#exec_group) a set of execution groups. On each execution group, the rule author can specify everything needed to select an execution platform for that execution group, namely any constraints via `exec_compatible_with` and toolchain types via `toolchain`. ```python # foo.bzl @@ -56,25 +37,13 @@ my_rule = rule( ) ``` -In the code snippet above, you can see that tool dependencies can also specify -transition for an exec group using the -[`cfg`](/rules/lib/toplevel/attr#label) -attribute param and the -[`config`](/rules/lib/toplevel/config) -module. The module exposes an `exec` function which takes a single string -parameter which is the name of the exec group for which the dependency should be -built. +In the code snippet above, you can see that tool dependencies can also specify transition for an exec group using the [`cfg`](/rules/lib/toplevel/attr#label) attribute param and the [`config`](/rules/lib/toplevel/config) module. The module exposes an `exec` function which takes a single string parameter which is the name of the exec group for which the dependency should be built. -As on native rules, the `test` execution group is present by default on Starlark -test rules. +As on native rules, the `test` execution group is present by default on Starlark test rules. ## Accessing execution groups -In the rule implementation, you can declare that actions should be run on the -execution platform of an execution group. You can do this by using the `exec_group` -param of action generating methods, specifically [`ctx.actions.run`] -(/rules/lib/builtins/actions#run) and -[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). +In the rule implementation, you can declare that actions should be run on the execution platform of an execution group. You can do this by using the `exec_group` param of action generating methods, specifically \[`ctx.actions.run`] (/rules/lib/builtins/actions#run) and [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). ```python # foo.bzl @@ -86,9 +55,7 @@ def _impl(ctx): ) ``` -Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) -of execution groups, similarly to how you -can access the resolved toolchain of a target: +Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) of execution groups, similarly to how you can access the resolved toolchain of a target: ```python # foo.bzl @@ -101,28 +68,18 @@ def _impl(ctx): ) ``` -Note: If an action uses a toolchain from an execution group, but doesn't specify -that execution group in the action declaration, that may potentially cause -issues. A mismatch like this may not immediately cause failures, but is a latent -problem. +Note: If an action uses a toolchain from an execution group, but doesn't specify that execution group in the action declaration, that may potentially cause issues. A mismatch like this may not immediately cause failures, but is a latent problem. ### Default execution groups The following execution groups are predefined: -* `test`: Test runner actions (for more details, see - the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). -* `cpp_link`: C++ linking actions. +- `test`: Test runner actions (for more details, see the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). +- `cpp_link`: C++ linking actions. ## Using execution groups to set execution properties -Execution groups are integrated with the -[`exec_properties`](/reference/be/common-definitions#common-attributes) -attribute that exists on every rule and allows the target writer to specify a -string dict of properties that is then passed to the execution machinery. For -example, if you wanted to set some property, say memory, for the target and give -certain actions a higher memory allocation, you would write an `exec_properties` -entry with an execution-group-augmented key, such as: +Execution groups are integrated with the [`exec_properties`](/reference/be/common-definitions#common-attributes) attribute that exists on every rule and allows the target writer to specify a string dict of properties that is then passed to the execution machinery. For example, if you wanted to set some property, say memory, for the target and give certain actions a higher memory allocation, you would write an `exec_properties` entry with an execution-group-augmented key, such as: ```python # BUILD @@ -136,24 +93,13 @@ my_rule( ) ``` -All actions with `exec_group = "link"` would see the exec properties -dictionary as `{"mem": "16g"}`. As you see here, execution-group-level -settings override target-level settings. +All actions with `exec_group = "link"` would see the exec properties dictionary as `{"mem": "16g"}`. As you see here, execution-group-level settings override target-level settings. ## Using execution groups to set platform constraints -Execution groups are also integrated with the -[`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and -[`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) -attributes that exist on every rule and allow the target writer to specify -additional constraints that must be satisfied by the execution platforms -selected for the target's actions. +Execution groups are also integrated with the [`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and [`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) attributes that exist on every rule and allow the target writer to specify additional constraints that must be satisfied by the execution platforms selected for the target's actions. -For example, if the rule `my_test` defines the `link` execution group in -addition to the default and the `test` execution group, then the following -usage of these attributes would run actions in the default execution group on -a platform with a high number of CPUs, the test action on Linux, and the link -action on the default execution platform: +For example, if the rule `my_test` defines the `link` execution group in addition to the default and the `test` execution group, then the following usage of these attributes would run actions in the default execution group on a platform with a high number of CPUs, the test action on Linux, and the link action on the default execution platform: ```python # BUILD @@ -180,23 +126,16 @@ my_test( ### Execution groups for native rules -The following execution groups are available for actions defined by native -rules: +The following execution groups are available for actions defined by native rules: -* `test`: Test runner actions. -* `cpp_link`: C++ linking actions. +- `test`: Test runner actions. +- `cpp_link`: C++ linking actions. ### Execution groups and platform execution properties -It is possible to define `exec_properties` for arbitrary execution groups on -platform targets (unlike `exec_properties` set directly on a target, where -properties for unknown execution groups are rejected). Targets then inherit the -execution platform's `exec_properties` that affect the default execution group -and any other relevant execution groups. +It is possible to define `exec_properties` for arbitrary execution groups on platform targets (unlike `exec_properties` set directly on a target, where properties for unknown execution groups are rejected). Targets then inherit the execution platform's `exec_properties` that affect the default execution group and any other relevant execution groups. -For example, suppose running tests on the exec platform requires some resource -to be available, but it isn't required for compiling and linking; this can be -modelled as follows: +For example, suppose running tests on the exec platform requires some resource to be available, but it isn't required for compiling and linking; this can be modelled as follows: ```python constraint_setting(name = "resource") @@ -217,5 +156,4 @@ cc_test( ) ``` -`exec_properties` defined directly on targets take precedence over those that -are inherited from the execution platform. +`exec_properties` defined directly on targets take precedence over those that are inherited from the execution platform. diff --git a/extending/legacy-macros.mdx b/extending/legacy-macros.mdx index 9c2b8e28..910cc46a 100644 --- a/extending/legacy-macros.mdx +++ b/extending/legacy-macros.mdx @@ -2,12 +2,7 @@ title: 'Legacy Macros' --- - - -Legacy macros are unstructured functions called from `BUILD` files that can -create targets. By the end of the -[loading phase](/extending/concepts#evaluation-model), legacy macros don't exist -anymore, and Bazel sees only the concrete set of instantiated rules. +Legacy macros are unstructured functions called from `BUILD` files that can create targets. By the end of the [loading phase](/extending/concepts#evaluation-model), legacy macros don't exist anymore, and Bazel sees only the concrete set of instantiated rules. ## Why you shouldn't use legacy macros (and should use Symbolic macros instead) @@ -15,38 +10,30 @@ Where possible you should use [symbolic macros](macros.md#macros). Symbolic macros -* Prevent action at a distance -* Make it possible to hide implementation details through granular visibility -* Take typed attributes, which in turn means automatic label and select - conversion. -* Are more readable -* Will soon have [lazy evaluation](macros.md#laziness) +- Prevent action at a distance +- Make it possible to hide implementation details through granular visibility +- Take typed attributes, which in turn means automatic label and select conversion. +- Are more readable +- Will soon have [lazy evaluation](macros.md#laziness) ## Usage The typical use case for a macro is when you want to reuse a rule. -For example, genrule in a `BUILD` file generates a file using `//:generator` -with a `some_arg` argument hardcoded in the command: +For example, genrule in a `BUILD` file generates a file using `//:generator` with a `some_arg` argument hardcoded in the command: ```python genrule( name = "file", outs = ["file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", tools = ["//:generator"], ) ``` -Note: `$@` is a -[Make variable](/reference/be/make-variables#predefined_genrule_variables) that -refers to the execution-time locations of the files in the `outs` attribute -list. It is equivalent to `$(locations :file.txt)`. +Note: `$@` is a [Make variable](/reference/be/make-variables#predefined_genrule_variables) that refers to the execution-time locations of the files in the `outs` attribute list. It is equivalent to `$(locations :file.txt)`. -If you want to generate more files with different arguments, you may want to -extract this code to a macro function. To create a macro called -`file_generator`, which has `name` and `arg` parameters, we can replace the -genrule with the following: +If you want to generate more files with different arguments, you may want to extract this code to a macro function. To create a macro called `file_generator`, which has `name` and `arg` parameters, we can replace the genrule with the following: ```python load("//path:generator.bzl", "file_generator") @@ -67,27 +54,22 @@ file_generator( ) ``` -Here, you load the `file_generator` symbol from a `.bzl` file located in the -`//path` package. By putting macro function definitions in a separate `.bzl` -file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be -loaded from any package in the workspace. +Here, you load the `file_generator` symbol from a `.bzl` file located in the `//path` package. By putting macro function definitions in a separate `.bzl` file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be loaded from any package in the workspace. -Finally, in `path/generator.bzl`, write the definition of the macro to -encapsulate and parameterize the original genrule definition: +Finally, in `path/generator.bzl`, write the definition of the macro to encapsulate and parameterize the original genrule definition: ```python def file_generator(name, arg, visibility=None): native.genrule( name = name, outs = [name + ".txt"], - cmd = "$(location //:generator) %s > $@" % arg, + cmd = "$(location //:generator) %s > $@" % arg, tools = ["//:generator"], visibility = visibility, ) ``` -You can also use macros to chain rules together. This example shows chained -genrules, where a genrule uses the outputs of a previous genrule as inputs: +You can also use macros to chain rules together. This example shows chained genrules, where a genrule uses the outputs of a previous genrule as inputs: ```python def chained_genrules(name, visibility=None): @@ -103,23 +85,19 @@ def chained_genrules(name, visibility=None): name = name + "-two", srcs = [name + ".one"], outs = [name + ".two"], - cmd = "$(location :tool-two) $< $@", + cmd = "$(location :tool-two) $< $@", tools = [":tool-two"], visibility = visibility, ) ``` -The example only assigns a visibility value to the second genrule. This allows -macro authors to hide the outputs of intermediate rules from being depended upon -by other targets in the workspace. +The example only assigns a visibility value to the second genrule. This allows macro authors to hide the outputs of intermediate rules from being depended upon by other targets in the workspace. -Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the -`srcs` attribute list. +Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the `srcs` attribute list. ## Expanding macros -When you want to investigate what a macro does, use the `query` command with -`--output=build` to see the expanded form: +When you want to investigate what a macro does, use the `query` command with `--output=build` to see the expanded form: ```none $ bazel query --output=build :file @@ -128,14 +106,13 @@ genrule( name = "file", tools = ["//:generator"], outs = ["//test:file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", ) ``` ## Instantiating native rules -Native rules (rules that don't need a `load()` statement) can be instantiated -from the [native](/rules/lib/toplevel/native) module: +Native rules (rules that don't need a `load()` statement) can be instantiated from the [native](/rules/lib/toplevel/native) module: ```python def my_macro(name, visibility=None): @@ -146,23 +123,13 @@ def my_macro(name, visibility=None): ) ``` -If you need to know the package name (for example, which `BUILD` file is calling -the macro), use the function -[native.package_name()](/rules/lib/toplevel/native#package_name). Note that -`native` can only be used in `.bzl` files, and not in `BUILD` files. +If you need to know the package name (for example, which `BUILD` file is calling the macro), use the function [native.package\_name()](/rules/lib/toplevel/native#package_name). Note that `native` can only be used in `.bzl` files, and not in `BUILD` files. ## Label resolution in macros -Since legacy macros are evaluated in the -[loading phase](concepts.md#evaluation-model), label strings such as -`"//foo:bar"` that occur in a legacy macro are interpreted relative to the -`BUILD` file in which the macro is used rather than relative to the `.bzl` file -in which it is defined. This behavior is generally undesirable for macros that -are meant to be used in other repositories, such as because they are part of a -published Starlark ruleset. +Since legacy macros are evaluated in the [loading phase](concepts.md#evaluation-model), label strings such as `"//foo:bar"` that occur in a legacy macro are interpreted relative to the `BUILD` file in which the macro is used rather than relative to the `.bzl` file in which it is defined. This behavior is generally undesirable for macros that are meant to be used in other repositories, such as because they are part of a published Starlark ruleset. -To get the same behavior as for Starlark rules, wrap the label strings with the -[`Label`](/rules/lib/builtins/Label#Label) constructor: +To get the same behavior as for Starlark rules, wrap the label strings with the [`Label`](/rules/lib/builtins/Label#Label) constructor: ```python # @my_ruleset//rules:defs.bzl @@ -184,71 +151,39 @@ def my_cc_wrapper(name, deps = [], **kwargs): ) ``` -With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys -that are label strings will be automatically resolved to `Label` objects -relative to the package of the file that contains the `select` call. If this is -not chosen, wrap the label string with -[native.package_relative_label()](/rules/lib/toplevel/native#package_relative_label). +With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys that are label strings will be automatically resolved to `Label` objects relative to the package of the file that contains the `select` call. If this is not chosen, wrap the label string with [native.package\_relative\_label()](/rules/lib/toplevel/native#package_relative_label). ## Debugging -* `bazel query --output=build //my/path:all` will show you how the `BUILD` - file looks after evaluation. All legacy macros, globs, loops are expanded. - Known limitation: `select` expressions are not shown in the output. +- `bazel query --output=build //my/path:all` will show you how the `BUILD` file looks after evaluation. All legacy macros, globs, loops are expanded. Known limitation: `select` expressions are not shown in the output. -* You may filter the output based on `generator_function` (which function - generated the rules) or `generator_name` (the name attribute of the macro): - `bash $ bazel query --output=build 'attr(generator_function, my_macro, - //my/path:all)'` +- You may filter the output based on `generator_function` (which function generated the rules) or `generator_name` (the name attribute of the macro): `bash $ bazel query --output=build 'attr(generator_function, my_macro, //my/path:all)'` -* To find out where exactly the rule `foo` is generated in a `BUILD` file, you - can try the following trick. Insert this line near the top of the `BUILD` - file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when - the rule `foo` is created (due to a name conflict), which will show you the - full stack trace. +- To find out where exactly the rule `foo` is generated in a `BUILD` file, you can try the following trick. Insert this line near the top of the `BUILD` file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when the rule `foo` is created (due to a name conflict), which will show you the full stack trace. -* You can also use [print](/rules/lib/globals/all#print) for debugging. It - displays the message as a `DEBUG` log line during the loading phase. Except - in rare cases, either remove `print` calls, or make them conditional under a - `debugging` parameter that defaults to `False` before submitting the code to - the depot. +- You can also use [print](/rules/lib/globals/all#print) for debugging. It displays the message as a `DEBUG` log line during the loading phase. Except in rare cases, either remove `print` calls, or make them conditional under a `debugging` parameter that defaults to `False` before submitting the code to the depot. ## Errors -If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) -function. Explain clearly to the user what went wrong and how to fix their -`BUILD` file. It is not possible to catch an error. +If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) function. Explain clearly to the user what went wrong and how to fix their `BUILD` file. It is not possible to catch an error. ```python def my_macro(name, deps, visibility=None): - if len(deps) < 2: + if len(deps) < 2: fail("Expected at least two values in deps") # ... ``` ## Conventions -* All public functions (functions that don't start with underscore) that - instantiate rules must have a `name` argument. This argument should not be - optional (don't give a default value). +- All public functions (functions that don't start with underscore) that instantiate rules must have a `name` argument. This argument should not be optional (don't give a default value). -* Public functions should use a docstring following - [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). +- Public functions should use a docstring following [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). -* In `BUILD` files, the `name` argument of the macros must be a keyword - argument (not a positional argument). +- In `BUILD` files, the `name` argument of the macros must be a keyword argument (not a positional argument). -* The `name` attribute of rules generated by a macro should include the name - argument as a prefix. For example, `macro(name = "foo")` can generate a - `cc_library` `foo` and a genrule `foo_gen`. +- The `name` attribute of rules generated by a macro should include the name argument as a prefix. For example, `macro(name = "foo")` can generate a `cc_library` `foo` and a genrule `foo_gen`. -* In most cases, optional parameters should have a default value of `None`. - `None` can be passed directly to native rules, which treat it the same as if - you had not passed in any argument. Thus, there is no need to replace it - with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer - to the rules it creates, as their defaults may be complex or may change over - time. Additionally, a parameter that is explicitly set to its default value - looks different than one that is never set (or set to `None`) when accessed - through the query language or build-system internals. +- In most cases, optional parameters should have a default value of `None`. `None` can be passed directly to native rules, which treat it the same as if you had not passed in any argument. Thus, there is no need to replace it with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer to the rules it creates, as their defaults may be complex or may change over time. Additionally, a parameter that is explicitly set to its default value looks different than one that is never set (or set to `None`) when accessed through the query language or build-system internals. -* Macros should have an optional `visibility` argument. +- Macros should have an optional `visibility` argument. diff --git a/extending/macros.mdx b/extending/macros.mdx index 136665d0..1e4ac1e3 100644 --- a/extending/macros.mdx +++ b/extending/macros.mdx @@ -2,42 +2,23 @@ title: 'Macros' --- +This page covers the basics of using macros and includes typical use cases, debugging, and conventions. +A macro is a function called from the `BUILD` file that can instantiate rules. Macros are mainly used for encapsulation and code reuse of existing rules and other macros. -This page covers the basics of using macros and includes typical use cases, -debugging, and conventions. +Macros come in two flavors: symbolic macros, which are described on this page, and [legacy macros](legacy-macros.md). Where possible, we recommend using symbolic macros for code clarity. -A macro is a function called from the `BUILD` file that can instantiate rules. -Macros are mainly used for encapsulation and code reuse of existing rules and -other macros. +Symbolic macros offer typed arguments (string to label conversion, relative to where the macro was called) and the ability to restrict and specify the visibility of targets created. They are designed to be amenable to lazy evaluation (which will be added in a future Bazel release). Symbolic macros are available by default in Bazel 8. Where this document mentions `macros`, it's referring to **symbolic macros**. -Macros come in two flavors: symbolic macros, which are described on this page, -and [legacy macros](legacy-macros.md). Where possible, we recommend using -symbolic macros for code clarity. - -Symbolic macros offer typed arguments (string to label conversion, relative to -where the macro was called) and the ability to restrict and specify the -visibility of targets created. They are designed to be amenable to lazy -evaluation (which will be added in a future Bazel release). Symbolic macros are -available by default in Bazel 8. Where this document mentions `macros`, it's -referring to **symbolic macros**. - -An executable example of symbolic macros can be found in the -[examples repository](https://github.com/bazelbuild/examples/tree/main/macros). +An executable example of symbolic macros can be found in the [examples repository](https://github.com/bazelbuild/examples/tree/main/macros). ## Usage -Macros are defined in `.bzl` files by calling the -[`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with -two required parameters: `attrs` and `implementation`. +Macros are defined in `.bzl` files by calling the [`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with two required parameters: `attrs` and `implementation`. ### Attributes -`attrs` accepts a dictionary of attribute name to [attribute -types](https://bazel.build/rules/lib/toplevel/attr#members), which represents -the arguments to the macro. Two common attributes – `name` and `visibility` – -are implicitly added to all macros and are not included in the dictionary passed -to `attrs`. +`attrs` accepts a dictionary of attribute name to [attribute types](https://bazel.build/rules/lib/toplevel/attr#members), which represents the arguments to the macro. Two common attributes – `name` and `visibility` – are implicitly added to all macros and are not included in the dictionary passed to `attrs`. ```starlark # macro/macro.bzl @@ -50,31 +31,13 @@ my_macro = macro( ) ``` -Attribute type declarations accept the -[parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), -`mandatory`, `default`, and `doc`. Most attribute types also accept the -`configurable` parameter, which determines whether the attribute accepts -`select`s. If an attribute is `configurable`, it will parse non-`select` values -as an unconfigurable `select` – `"foo"` will become -`select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). +Attribute type declarations accept the [parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), `mandatory`, `default`, and `doc`. Most attribute types also accept the `configurable` parameter, which determines whether the attribute accepts `select`s. If an attribute is `configurable`, it will parse non-`select` values as an unconfigurable `select` – `"foo"` will become `select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). #### Attribute inheritance -Macros are often intended to wrap a rule (or another macro), and the macro's -author often wants to forward the bulk of the wrapped symbol's attributes -unchanged, using `**kwargs`, to the macro's main target (or main inner macro). - -To support this pattern, a macro can *inherit attributes* from a rule or another -macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or -[macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s -`inherit_attrs` argument. (You can also use the special string `"common"` -instead of a rule or macro symbol to inherit the [common attributes defined for -all Starlark build -rules](https://bazel.build/reference/be/common-definitions#common-attributes).) -Only public attributes get inherited, and the attributes in the macro's own -`attrs` dictionary override inherited attributes with the same name. You can -also *remove* inherited attributes by using `None` as a value in the `attrs` -dictionary: +Macros are often intended to wrap a rule (or another macro), and the macro's author often wants to forward the bulk of the wrapped symbol's attributes unchanged, using `**kwargs`, to the macro's main target (or main inner macro). + +To support this pattern, a macro can *inherit attributes* from a rule or another macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or [macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s `inherit_attrs` argument. (You can also use the special string `"common"` instead of a rule or macro symbol to inherit the [common attributes defined for all Starlark build rules](https://bazel.build/reference/be/common-definitions#common-attributes).) Only public attributes get inherited, and the attributes in the macro's own `attrs` dictionary override inherited attributes with the same name. You can also *remove* inherited attributes by using `None` as a value in the `attrs` dictionary: ```starlark # macro/macro.bzl @@ -90,11 +53,7 @@ my_macro = macro( ) ``` -The default value of non-mandatory inherited attributes is always overridden to -be `None`, regardless of the original attribute definition's default value. If -you need to examine or modify an inherited non-mandatory attribute – for -example, if you want to add a tag to an inherited `tags` attribute – you must -make sure to handle the `None` case in your macro's implementation function: +The default value of non-mandatory inherited attributes is always overridden to be `None`, regardless of the original attribute definition's default value. If you need to examine or modify an inherited non-mandatory attribute – for example, if you want to add a tag to an inherited `tags` attribute – you must make sure to handle the `None` case in your macro's implementation function: ```starlark # macro/macro.bzl @@ -112,15 +71,9 @@ def _my_macro_impl(name, visibility, tags, **kwargs): ### Implementation -`implementation` accepts a function which contains the logic of the macro. -Implementation functions often create targets by calling one or more rules, and -they are usually private (named with a leading underscore). Conventionally, -they are named the same as their macro, but prefixed with `_` and suffixed with -`_impl`. +`implementation` accepts a function which contains the logic of the macro. Implementation functions often create targets by calling one or more rules, and they are usually private (named with a leading underscore). Conventionally, they are named the same as their macro, but prefixed with `_` and suffixed with `_impl`. -Unlike rule implementation functions, which take a single argument (`ctx`) that -contains a reference to the attributes, macro implementation functions accept a -parameter for each argument. +Unlike rule implementation functions, which take a single argument (`ctx`) that contains a reference to the attributes, macro implementation functions accept a parameter for each argument. ```starlark # macro/macro.bzl @@ -138,11 +91,7 @@ def _my_macro_impl(name, visibility, deps, create_test): ) ``` -If a macro inherits attributes, its implementation function *must* have a -`**kwargs` residual keyword parameter, which can be forwarded to the call that -invokes the inherited rule or submacro. (This helps ensure that your macro won't -be broken if the rule or macro which from which you are inheriting adds a new -attribute.) +If a macro inherits attributes, its implementation function *must* have a `**kwargs` residual keyword parameter, which can be forwarded to the call that invokes the inherited rule or submacro. (This helps ensure that your macro won't be broken if the rule or macro which from which you are inheriting adds a new attribute.) ### Declaration @@ -164,12 +113,9 @@ my_macro( ) ``` -This would create targets -`//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. +This would create targets `//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. -Just like in rule calls, if an attribute value in a macro call is set to `None`, -that attribute is treated as if it was omitted by the macro's caller. For -example, the following two macro calls are equivalent: +Just like in rule calls, if an attribute value in a macro call is set to `None`, that attribute is treated as if it was omitted by the macro's caller. For example, the following two macro calls are equivalent: ```starlark # pkg/BUILD @@ -177,27 +123,17 @@ my_macro(name = "abc", srcs = ["src.cc"], deps = None) my_macro(name = "abc", srcs = ["src.cc"]) ``` -This is generally not useful in `BUILD` files, but is helpful when -programmatically wrapping a macro inside another macro. +This is generally not useful in `BUILD` files, but is helpful when programmatically wrapping a macro inside another macro. ## Details ### Naming conventions for targets created -The names of any targets or submacros created by a symbolic macro must -either match the macro's `name` parameter or must be prefixed by `name` followed -by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only -create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, -for example, `foo_bar`. +The names of any targets or submacros created by a symbolic macro must either match the macro's `name` parameter or must be prefixed by `name` followed by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, for example, `foo_bar`. -Targets or files that violate macro naming convention can be declared, but -cannot be built and cannot be used as dependencies. +Targets or files that violate macro naming convention can be declared, but cannot be built and cannot be used as dependencies. -Non-macro files and targets within the same package as a macro instance should -*not* have names that conflict with potential macro target names, though this -exclusivity is not enforced. We are in the progress of implementing -[lazy evaluation](#laziness) as a performance improvement for Symbolic macros, -which will be impaired in packages that violate the naming schema. +Non-macro files and targets within the same package as a macro instance should *not* have names that conflict with potential macro target names, though this exclusivity is not enforced. We are in the progress of implementing [lazy evaluation](#laziness) as a performance improvement for Symbolic macros, which will be impaired in packages that violate the naming schema. ### Restrictions @@ -205,54 +141,35 @@ Symbolic macros have some additional restrictions compared to legacy macros. Symbolic macros -* must take a `name` argument and a `visibility` argument -* must have an `implementation` function -* may not return values -* may not mutate their arguments -* may not call `native.existing_rules()` unless they are special `finalizer` - macros -* may not call `native.package()` -* may not call `glob()` -* may not call `native.environment_group()` -* must create targets whose names adhere to the [naming schema](#naming) -* can't refer to input files that weren't declared or passed in as an argument -* can't refer to private targets of their callers (see - [visibility and macros](#visibility) for more details). +- must take a `name` argument and a `visibility` argument +- must have an `implementation` function +- may not return values +- may not mutate their arguments +- may not call `native.existing_rules()` unless they are special `finalizer` macros +- may not call `native.package()` +- may not call `glob()` +- may not call `native.environment_group()` +- must create targets whose names adhere to the [naming schema](#naming) +- can't refer to input files that weren't declared or passed in as an argument +- can't refer to private targets of their callers (see [visibility and macros](#visibility) for more details). ### Visibility and macros -The [visibility](/concepts/visibility) system helps protect the implementation -details of both (symbolic) macros and their callers. +The [visibility](/concepts/visibility) system helps protect the implementation details of both (symbolic) macros and their callers. -By default, targets created in a symbolic macro are visible within the macro -itself, but not necessarily to the macro's caller. The macro can "export" a -target as a public API by forwarding the value of its own `visibility` -attribute, as in `some_rule(..., visibility = visibility)`. +By default, targets created in a symbolic macro are visible within the macro itself, but not necessarily to the macro's caller. The macro can "export" a target as a public API by forwarding the value of its own `visibility` attribute, as in `some_rule(..., visibility = visibility)`. The key ideas of macro visibility are: -1. Visibility is checked based on what macro declared the target, not what - package called the macro. +1. Visibility is checked based on what macro declared the target, not what package called the macro. - * In other words, being in the same package does not by itself make one - target visible to another. This protects the macro's internal targets - from becoming dependencies of other macros or top-level targets in the - package. + - In other words, being in the same package does not by itself make one target visible to another. This protects the macro's internal targets from becoming dependencies of other macros or top-level targets in the package. -1. All `visibility` attributes, on both rules and macros, automatically - include the place where the rule or macro was called. +2. All `visibility` attributes, on both rules and macros, automatically include the place where the rule or macro was called. - * Thus, a target is unconditionally visible to other targets declared in the - same macro (or the `BUILD` file, if not in a macro). + - Thus, a target is unconditionally visible to other targets declared in the same macro (or the `BUILD` file, if not in a macro). -In practice, this means that when a macro declares a target without setting its -`visibility`, the target defaults to being internal to the macro. (The package's -[default visibility](/reference/be/functions#package.default_visibility) does -not apply within a macro.) Exporting the target means that the target is visible -to whatever the macro's caller specified in the macro's `visibility` attribute, -plus the package of the macro's caller itself, as well as the macro's own code. -Another way of thinking of it is that the visibility of a macro determines who -(aside from the macro itself) can see the macro's exported targets. +In practice, this means that when a macro declares a target without setting its `visibility`, the target defaults to being internal to the macro. (The package's [default visibility](/reference/be/functions#package.default_visibility) does not apply within a macro.) Exporting the target means that the target is visible to whatever the macro's caller specified in the macro's `visibility` attribute, plus the package of the macro's caller itself, as well as the macro's own code. Another way of thinking of it is that the visibility of a macro determines who (aside from the macro itself) can see the macro's exported targets. ```starlark # tool/BUILD @@ -314,51 +231,25 @@ some_rule( ) ``` -If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if -the `//pkg` package had set its `default_visibility` to that value, then -`//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a -macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain -protected. - -A macro can declare that a target is visible to a friend package by passing -`visibility = ["//some_friend:__pkg__"]` (for an internal target) or -`visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). -Note that it is an antipattern for a macro to declare a target with public -visibility (`visibility = ["//visibility:public"]`). This is because it makes -the target unconditionally visible to every package, even if the caller -specified a more restricted visibility. - -All visibility checking is done with respect to the innermost currently running -symbolic macro. However, there is a visibility delegation mechanism: If a macro -passes a label as an attribute value to an inner macro, any usages of the label -in the inner macro are checked with respect to the outer macro. See the -[visibility page](/concepts/visibility#symbolic-macros) for more details. - -Remember that legacy macros are entirely transparent to the visibility system, -and behave as though their location is whatever BUILD file or symbolic macro -they were called from. +If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if the `//pkg` package had set its `default_visibility` to that value, then `//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain protected. + +A macro can declare that a target is visible to a friend package by passing `visibility = ["//some_friend:__pkg__"]` (for an internal target) or `visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). Note that it is an antipattern for a macro to declare a target with public visibility (`visibility = ["//visibility:public"]`). This is because it makes the target unconditionally visible to every package, even if the caller specified a more restricted visibility. + +All visibility checking is done with respect to the innermost currently running symbolic macro. However, there is a visibility delegation mechanism: If a macro passes a label as an attribute value to an inner macro, any usages of the label in the inner macro are checked with respect to the outer macro. See the [visibility page](/concepts/visibility#symbolic-macros) for more details. + +Remember that legacy macros are entirely transparent to the visibility system, and behave as though their location is whatever BUILD file or symbolic macro they were called from. #### Finalizers and visibility -Targets declared in a rule finalizer, in addition to seeing targets following -the usual symbolic macro visibility rules, can *also* see all targets which are -visible to the finalizer target's package. +Targets declared in a rule finalizer, in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. -This means that if you migrate a `native.existing_rules()`-based legacy macro to -a finalizer, the targets declared by the finalizer will still be able to see -their old dependencies. +This means that if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. -However, note that it's possible to declare a target in a symbolic macro such -that a finalizer's targets cannot see it under the visibility system – even -though the finalizer can *introspect* its attributes using -`native.existing_rules()`. +However, note that it's possible to declare a target in a symbolic macro such that a finalizer's targets cannot see it under the visibility system – even though the finalizer can *introspect* its attributes using `native.existing_rules()`. ### Selects -If an attribute is `configurable` (the default) and its value is not `None`, -then the macro implementation function will see the attribute value as wrapped -in a trivial `select`. This makes it easier for the macro author to catch bugs -where they did not anticipate that the attribute value could be a `select`. +If an attribute is `configurable` (the default) and its value is not `None`, then the macro implementation function will see the attribute value as wrapped in a trivial `select`. This makes it easier for the macro author to catch bugs where they did not anticipate that the attribute value could be a `select`. For example, consider the following macro: @@ -369,38 +260,15 @@ my_macro = macro( ) ``` -If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` -to be invoked with its `deps` parameter set to `select({"//conditions:default": -["//a"]})`. If this causes the implementation function to fail (say, because the -code tried to index into the value as in `deps[0]`, which is not allowed for -`select`s), the macro author can then make a choice: either they can rewrite -their macro to only use operations compatible with `select`, or they can mark -the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The -latter ensures that users are not permitted to pass a `select` value in. - -Rule targets reverse this transformation, and store trivial `select`s as their -unconditional values; in the above example, if `_my_macro_impl` declares a rule -target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as -`["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` -values to be stored in all targets instantiated by macros. - -If the value of a configurable attribute is `None`, it does not get wrapped in a -`select`. This ensures that tests like `my_attr == None` still work, and that -when the attribute is forwarded to a rule with a computed default, the rule -behaves properly (that is, as if the attribute were not passed in at all). It is -not always possible for an attribute to take on a `None` value, but it can -happen for the `attr.label()` type, and for any inherited non-mandatory -attribute. +If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` to be invoked with its `deps` parameter set to `select({"//conditions:default": ["//a"]})`. If this causes the implementation function to fail (say, because the code tried to index into the value as in `deps[0]`, which is not allowed for `select`s), the macro author can then make a choice: either they can rewrite their macro to only use operations compatible with `select`, or they can mark the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The latter ensures that users are not permitted to pass a `select` value in. + +Rule targets reverse this transformation, and store trivial `select`s as their unconditional values; in the above example, if `_my_macro_impl` declares a rule target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as `["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` values to be stored in all targets instantiated by macros. + +If the value of a configurable attribute is `None`, it does not get wrapped in a `select`. This ensures that tests like `my_attr == None` still work, and that when the attribute is forwarded to a rule with a computed default, the rule behaves properly (that is, as if the attribute were not passed in at all). It is not always possible for an attribute to take on a `None` value, but it can happen for the `attr.label()` type, and for any inherited non-mandatory attribute. ## Finalizers -A rule finalizer is a special symbolic macro which – regardless of its lexical -position in a BUILD file – is evaluated in the final stage of loading a package, -after all non-finalizer targets have been defined. Unlike ordinary symbolic -macros, a finalizer can call `native.existing_rules()`, where it behaves -slightly differently than in legacy macros: it only returns the set of -non-finalizer rule targets. The finalizer may assert on the state of that set or -define new targets. +A rule finalizer is a special symbolic macro which – regardless of its lexical position in a BUILD file – is evaluated in the final stage of loading a package, after all non-finalizer targets have been defined. Unlike ordinary symbolic macros, a finalizer can call `native.existing_rules()`, where it behaves slightly differently than in legacy macros: it only returns the set of non-finalizer rule targets. The finalizer may assert on the state of that set or define new targets. To declare a finalizer, call `macro()` with `finalizer = True`: @@ -426,24 +294,17 @@ my_finalizer = macro( ## Laziness -IMPORTANT: We are in the process of implementing lazy macro expansion and -evaluation. This feature is not available yet. +IMPORTANT: We are in the process of implementing lazy macro expansion and evaluation. This feature is not available yet. -Currently, all macros are evaluated as soon as the BUILD file is loaded, which -can negatively impact performance for targets in packages that also have costly -unrelated macros. In the future, non-finalizer symbolic macros will only be -evaluated if they're required for the build. The prefix naming schema helps -Bazel determine which macro to expand given a requested target. +Currently, all macros are evaluated as soon as the BUILD file is loaded, which can negatively impact performance for targets in packages that also have costly unrelated macros. In the future, non-finalizer symbolic macros will only be evaluated if they're required for the build. The prefix naming schema helps Bazel determine which macro to expand given a requested target. ## Migration troubleshooting Here are some common migration headaches and how to fix them. -* Legacy macro calls `glob()` +- Legacy macro calls `glob()` -Move the `glob()` call to your BUILD file (or to a legacy macro called from the -BUILD file), and pass the `glob()` value to the symbolic macro using a -label-list attribute: +Move the `glob()` call to your BUILD file (or to a legacy macro called from the BUILD file), and pass the `glob()` value to the symbolic macro using a label-list attribute: ```starlark # BUILD file @@ -453,13 +314,10 @@ my_macro( ) ``` -* Legacy macro has a parameter that isn't a valid starlark `attr` type. - -Pull as much logic as possible into a nested symbolic macro, but keep the -top level macro a legacy macro. +- Legacy macro has a parameter that isn't a valid starlark `attr` type. -* Legacy macro calls a rule that creates a target that breaks the naming schema +Pull as much logic as possible into a nested symbolic macro, but keep the top level macro a legacy macro. -That's okay, just don't depend on the "offending" target. The naming check will -be quietly ignored. +- Legacy macro calls a rule that creates a target that breaks the naming schema +That's okay, just don't depend on the "offending" target. The naming check will be quietly ignored. diff --git a/extending/platforms.mdx b/extending/platforms.mdx index 94e6290f..e6e349fd 100644 --- a/extending/platforms.mdx +++ b/extending/platforms.mdx @@ -2,60 +2,29 @@ title: 'Platforms' --- +Bazel can build and test code on a variety of hardware, operating systems, and system configurations, using many different versions of build tools such as linkers and compilers. To help manage this complexity, Bazel has a concept of *constraints* and *platforms*. A constraint is a dimension in which build or production environments may differ, such as CPU architecture, the presence or absence of a GPU, or the version of a system-installed compiler. A platform is a named collection of choices for these constraints, representing the particular resources that are available in some environment. - -Bazel can build and test code on a variety of hardware, operating systems, and -system configurations, using many different versions of build tools such as -linkers and compilers. To help manage this complexity, Bazel has a concept of -*constraints* and *platforms*. A constraint is a dimension in which build or -production environments may differ, such as CPU architecture, the presence or -absence of a GPU, or the version of a system-installed compiler. A platform is a -named collection of choices for these constraints, representing the particular -resources that are available in some environment. - -Modeling the environment as a platform helps Bazel to automatically select the -appropriate -[toolchains](/extending/toolchains) -for build actions. Platforms can also be used in combination with the -[config_setting](/reference/be/general#config_setting) -rule to write [configurable attributes](/docs/configurable-attributes). +Modeling the environment as a platform helps Bazel to automatically select the appropriate [toolchains](/extending/toolchains) for build actions. Platforms can also be used in combination with the [config\_setting](/reference/be/general#config_setting) rule to write [configurable attributes](/docs/configurable-attributes). Bazel recognizes three roles that a platform may serve: -* **Host** - the platform on which Bazel itself runs. -* **Execution** - a platform on which build tools execute build actions to - produce intermediate and final outputs. -* **Target** - a platform on which a final output resides and executes. +- **Host** - the platform on which Bazel itself runs. +- **Execution** - a platform on which build tools execute build actions to produce intermediate and final outputs. +- **Target** - a platform on which a final output resides and executes. Bazel supports the following build scenarios regarding platforms: -* **Single-platform builds** (default) - host, execution, and target platforms - are the same. For example, building a Linux executable on Ubuntu running on - an Intel x64 CPU. +- **Single-platform builds** (default) - host, execution, and target platforms are the same. For example, building a Linux executable on Ubuntu running on an Intel x64 CPU. -* **Cross-compilation builds** - host and execution platforms are the same, but - the target platform is different. For example, building an iOS app on macOS - running on a MacBook Pro. +- **Cross-compilation builds** - host and execution platforms are the same, but the target platform is different. For example, building an iOS app on macOS running on a MacBook Pro. -* **Multi-platform builds** - host, execution, and target platforms are all - different. +- **Multi-platform builds** - host, execution, and target platforms are all different. -Tip: for detailed instructions on migrating your project to platforms, see -[Migrating to Platforms](/concepts/platforms). +Tip: for detailed instructions on migrating your project to platforms, see [Migrating to Platforms](/concepts/platforms). ## Defining constraints and platforms -The space of possible choices for platforms is defined by using the -[`constraint_setting`][constraint_setting] and -[`constraint_value`][constraint_value] rules within `BUILD` files. -`constraint_setting` creates a new dimension, while -`constraint_value` creates a new value for a given dimension; together they -effectively define an enum and its possible values. For example, the following -snippet of a `BUILD` file introduces a constraint for the system's glibc version -with two possible values. - -[constraint_setting]: /reference/be/platforms-and-toolchains#constraint_setting -[constraint_value]: /reference/be/platforms-and-toolchains#constraint_value +The space of possible choices for platforms is defined by using the [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) and [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) rules within `BUILD` files. `constraint_setting` creates a new dimension, while `constraint_value` creates a new value for a given dimension; together they effectively define an enum and its possible values. For example, the following snippet of a `BUILD` file introduces a constraint for the system's glibc version with two possible values. ```python constraint_setting(name = "glibc_version") @@ -71,16 +40,9 @@ constraint_value( ) ``` -Constraints and their values may be defined across different packages in the -workspace. They are referenced by label and subject to the usual visibility -controls. If visibility allows, you can extend an existing constraint setting by -defining your own value for it. +Constraints and their values may be defined across different packages in the workspace. They are referenced by label and subject to the usual visibility controls. If visibility allows, you can extend an existing constraint setting by defining your own value for it. -The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with -certain choices of constraint values. The -following creates a platform named `linux_x86`, and says that it describes any -environment that runs a Linux operating system on an x86_64 architecture with a -glibc version of 2.25. (See below for more on Bazel's built-in constraints.) +The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with certain choices of constraint values. The following creates a platform named `linux_x86`, and says that it describes any environment that runs a Linux operating system on an x86\_64 architecture with a glibc version of 2.25. (See below for more on Bazel's built-in constraints.) ```python platform( @@ -93,52 +55,33 @@ platform( ) ``` -Note: It is an error for a platform to specify more than one value of the -same constraint setting, such as `@platforms//cpu:x86_64` and -`@platforms//cpu:arm` for `@platforms//cpu:cpu`. +Note: It is an error for a platform to specify more than one value of the same constraint setting, such as `@platforms//cpu:x86_64` and `@platforms//cpu:arm` for `@platforms//cpu:cpu`. ## Generally useful constraints and platforms -To keep the ecosystem consistent, Bazel team maintains a repository with -constraint definitions for the most popular CPU architectures and operating -systems. These are all located in -[https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). +To keep the ecosystem consistent, Bazel team maintains a repository with constraint definitions for the most popular CPU architectures and operating systems. These are all located in [https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). -Bazel ships with the following special platform definition: -`@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the -autodetected host platform value - -represents autodetected platform for the system Bazel is running on. +Bazel ships with the following special platform definition: `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the autodetected host platform value - represents autodetected platform for the system Bazel is running on. ## Specifying a platform for a build -You can specify the host and target platforms for a build using the following -command-line flags: - -* `--host_platform` - defaults to `@bazel_tools//tools:host_platform` - * This target is aliased to `@platforms//host`, which is backed by a repo - rule that detects the host OS and CPU and writes the platform target. - * There's also `@platforms//host:constraints.bzl`, which exposes - an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and - Starlark files. -* `--platforms` - defaults to the host platform - * This means that when no other flags are set, - `@platforms//host` is the target platform. - * If `--host_platform` is set and not `--platforms`, the value of - `--host_platform` is both the host and target platform. +You can specify the host and target platforms for a build using the following command-line flags: + +- `--host_platform` - defaults to `@bazel_tools//tools:host_platform` + + - This target is aliased to `@platforms//host`, which is backed by a repo rule that detects the host OS and CPU and writes the platform target. + - There's also `@platforms//host:constraints.bzl`, which exposes an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and Starlark files. + +- `--platforms` - defaults to the host platform + + - This means that when no other flags are set, `@platforms//host` is the target platform. + - If `--host_platform` is set and not `--platforms`, the value of `--host_platform` is both the host and target platform. ## Skipping incompatible targets -When building for a specific target platform it is often desirable to skip -targets that will never work on that platform. For example, your Windows device -driver is likely going to generate lots of compiler errors when building on a -Linux machine with `//...`. Use the -[`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) -attribute to tell Bazel what target platform constraints your code has. +When building for a specific target platform it is often desirable to skip targets that will never work on that platform. For example, your Windows device driver is likely going to generate lots of compiler errors when building on a Linux machine with `//...`. Use the [`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) attribute to tell Bazel what target platform constraints your code has. -The simplest use of this attribute restricts a target to a single platform. -The target will not be built for any platform that doesn't satisfy all of the -constraints. The following example restricts `win_driver_lib.cc` to 64-bit -Windows. +The simplest use of this attribute restricts a target to a single platform. The target will not be built for any platform that doesn't satisfy all of the constraints. The following example restricts `win_driver_lib.cc` to 64-bit Windows. ```python cc_library( @@ -151,16 +94,11 @@ cc_library( ) ``` -`:win_driver_lib` is *only* compatible for building with 64-bit Windows and -incompatible with all else. Incompatibility is transitive. Any targets -that transitively depend on an incompatible target are themselves considered -incompatible. +`:win_driver_lib` is *only* compatible for building with 64-bit Windows and incompatible with all else. Incompatibility is transitive. Any targets that transitively depend on an incompatible target are themselves considered incompatible. ### When are targets skipped? -Targets are skipped when they are considered incompatible and included in the -build as part of a target pattern expansion. For example, the following two -invocations skip any incompatible targets found in a target pattern expansion. +Targets are skipped when they are considered incompatible and included in the build as part of a target pattern expansion. For example, the following two invocations skip any incompatible targets found in a target pattern expansion. ```console $ bazel build --platforms=//:myplatform //... @@ -170,15 +108,9 @@ $ bazel build --platforms=//:myplatform //... $ bazel build --platforms=//:myplatform //:all ``` -Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are -similarly skipped if the `test_suite` is specified on the command line with -[`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). -In other words, `test_suite` targets on the command line behave like `:all` and -`...`. Using `--noexpand_test_suites` prevents expansion and causes -`test_suite` targets with incompatible tests to also be incompatible. +Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are similarly skipped if the `test_suite` is specified on the command line with [`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). In other words, `test_suite` targets on the command line behave like `:all` and `...`. Using `--noexpand_test_suites` prevents expansion and causes `test_suite` targets with incompatible tests to also be incompatible. -Explicitly specifying an incompatible target on the command line results in an -error message and a failed build. +Explicitly specifying an incompatible target on the command line results in an error message and a failed build. ```console $ bazel build --platforms=//:myplatform //:target_incompatible_with_myplatform @@ -188,20 +120,13 @@ ERROR: Target //:target_incompatible_with_myplatform is incompatible and cannot FAILED: Build did NOT complete successfully ``` -Incompatible explicit targets are silently skipped if -`--skip_incompatible_explicit_targets` is enabled. +Incompatible explicit targets are silently skipped if `--skip_incompatible_explicit_targets` is enabled. ### More expressive constraints -For more flexibility in expressing constraints, use the -`@platforms//:incompatible` -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) -that no platform satisfies. +For more flexibility in expressing constraints, use the `@platforms//:incompatible` [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) that no platform satisfies. -Use [`select()`](/reference/be/functions#select) in combination with -`@platforms//:incompatible` to express more complicated restrictions. For -example, use it to implement basic OR logic. The following marks a library -compatible with macOS and Linux, but no other platforms. +Use [`select()`](/reference/be/functions#select) in combination with `@platforms//:incompatible` to express more complicated restrictions. For example, use it to implement basic OR logic. The following marks a library compatible with macOS and Linux, but no other platforms. Note: An empty constraints list is equivalent to "compatible with everything". @@ -221,16 +146,11 @@ The above can be interpreted as follows: 1. When targeting macOS, the target has no constraints. 2. When targeting Linux, the target has no constraints. -3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because - `@platforms//:incompatible` is not part of any platform, the target is - deemed incompatible. +3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because `@platforms//:incompatible` is not part of any platform, the target is deemed incompatible. -To make your constraints more readable, use -[skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). +To make your constraints more readable, use [skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). -You can express inverse compatibility in a similar way. The following example -describes a library that is compatible with everything _except_ for ARM. +You can express inverse compatibility in a similar way. The following example describes a library that is compatible with everything *except* for ARM. ```python cc_library( @@ -245,15 +165,9 @@ cc_library( ### Detecting incompatible targets using `bazel cquery` -You can use the -[`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) -in `bazel cquery`'s [Starlark output -format](/query/cquery#output-format-definition) to distinguish -incompatible targets from compatible ones. +You can use the [`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) in `bazel cquery`'s [Starlark output format](/query/cquery#output-format-definition) to distinguish incompatible targets from compatible ones. -This can be used to filter out incompatible targets. The example below will -only print the labels for targets that are compatible. Incompatible targets are -not printed. +This can be used to filter out incompatible targets. The example below will only print the labels for targets that are compatible. Incompatible targets are not printed. ```console $ cat example.cquery @@ -263,11 +177,9 @@ def format(target): return target.label return "" - $ bazel cquery //... --output=starlark --starlark:file=example.cquery ``` ### Known Issues -Incompatible targets [ignore visibility -restrictions](https://github.com/bazelbuild/bazel/issues/16044). +Incompatible targets [ignore visibility restrictions](https://github.com/bazelbuild/bazel/issues/16044). diff --git a/extending/rules.mdx b/extending/rules.mdx index 248ee328..b7afa71c 100644 --- a/extending/rules.mdx +++ b/extending/rules.mdx @@ -2,57 +2,28 @@ title: 'Rules' --- +A **rule** defines a series of [**actions**](#actions) that Bazel performs on inputs to produce a set of outputs, which are referenced in [**providers**](#providers) returned by the rule's [**implementation function**](#implementation_function). For example, a C++ binary rule might: +1. Take a set of `.cpp` source files (inputs). +2. Run `g++` on the source files (action). +3. Return the `DefaultInfo` provider with the executable output and other files to make available at runtime. +4. Return the `CcInfo` provider with C++-specific information gathered from the target and its dependencies. -A **rule** defines a series of [**actions**](#actions) that Bazel performs on -inputs to produce a set of outputs, which are referenced in -[**providers**](#providers) returned by the rule's -[**implementation function**](#implementation_function). For example, a C++ -binary rule might: - -1. Take a set of `.cpp` source files (inputs). -2. Run `g++` on the source files (action). -3. Return the `DefaultInfo` provider with the executable output and other files - to make available at runtime. -4. Return the `CcInfo` provider with C++-specific information gathered from the - target and its dependencies. - -From Bazel's perspective, `g++` and the standard C++ libraries are also inputs -to this rule. As a rule writer, you must consider not only the user-provided -inputs to a rule, but also all of the tools and libraries required to execute -the actions. - -Before creating or modifying any rule, ensure you are familiar with Bazel's -[build phases](/extending/concepts). It is important to understand the three -phases of a build (loading, analysis, and execution). It is also useful to -learn about [macros](/extending/macros) to understand the difference between rules and -macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). -Then, use this page as a reference. - -A few rules are built into Bazel itself. These *native rules*, such as -`genrule` and `filegroup`, provide some core support. -By defining your own rules, you can add support for languages and tools -that Bazel doesn't support natively. - -Bazel provides an extensibility model for writing rules using the -[Starlark](/rules/language) language. These rules are written in `.bzl` files, which -can be loaded directly from `BUILD` files. - -When defining your own rule, you get to decide what attributes it supports and -how it generates its outputs. - -The rule's `implementation` function defines its exact behavior during the -[analysis phase](/extending/concepts#evaluation-model). This function doesn't run any -external commands. Rather, it registers [actions](#actions) that will be used -later during the execution phase to build the rule's outputs, if they are -needed. +From Bazel's perspective, `g++` and the standard C++ libraries are also inputs to this rule. As a rule writer, you must consider not only the user-provided inputs to a rule, but also all of the tools and libraries required to execute the actions. + +Before creating or modifying any rule, ensure you are familiar with Bazel's [build phases](/extending/concepts). It is important to understand the three phases of a build (loading, analysis, and execution). It is also useful to learn about [macros](/extending/macros) to understand the difference between rules and macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). Then, use this page as a reference. + +A few rules are built into Bazel itself. These *native rules*, such as `genrule` and `filegroup`, provide some core support. By defining your own rules, you can add support for languages and tools that Bazel doesn't support natively. + +Bazel provides an extensibility model for writing rules using the [Starlark](/rules/language) language. These rules are written in `.bzl` files, which can be loaded directly from `BUILD` files. + +When defining your own rule, you get to decide what attributes it supports and how it generates its outputs. + +The rule's `implementation` function defines its exact behavior during the [analysis phase](/extending/concepts#evaluation-model). This function doesn't run any external commands. Rather, it registers [actions](#actions) that will be used later during the execution phase to build the rule's outputs, if they are needed. ## Rule creation -In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new -rule, and store the result in a global variable. The call to `rule` specifies -[attributes](#attributes) and an -[implementation function](#implementation_function): +In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new rule, and store the result in a global variable. The call to `rule` specifies [attributes](#attributes) and an [implementation function](#implementation_function): ```python example_library = rule( @@ -66,10 +37,7 @@ example_library = rule( This defines a [rule kind](/query/language#kind) named `example_library`. -The call to `rule` also must specify if the rule creates an -[executable](#executable-rules) output (with `executable = True`), or specifically -a test executable (with `test = True`). If the latter, the rule is a *test rule*, -and the name of the rule must end in `_test`. +The call to `rule` also must specify if the rule creates an [executable](#executable-rules) output (with `executable = True`), or specifically a test executable (with `test = True`). If the latter, the rule is a *test rule*, and the name of the rule must end in `_test`. ## Target instantiation @@ -85,48 +53,23 @@ example_library( ) ``` -Each call to a build rule returns no value, but has the side effect of defining -a target. This is called *instantiating* the rule. This specifies a name for the -new target and values for the target's [attributes](#attributes). +Each call to a build rule returns no value, but has the side effect of defining a target. This is called *instantiating* the rule. This specifies a name for the new target and values for the target's [attributes](#attributes). -Rules can also be called from Starlark functions and loaded in `.bzl` files. -Starlark functions that call rules are called [Starlark macros](/extending/macros). -Starlark macros must ultimately be called from `BUILD` files, and can only be -called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` -files are evaluated to instantiate targets. +Rules can also be called from Starlark functions and loaded in `.bzl` files. Starlark functions that call rules are called [Starlark macros](/extending/macros). Starlark macros must ultimately be called from `BUILD` files, and can only be called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` files are evaluated to instantiate targets. ## Attributes -An *attribute* is a rule argument. Attributes can provide specific values to a -target's [implementation](#implementation_function), or they can refer to other -targets, creating a graph of dependencies. +An *attribute* is a rule argument. Attributes can provide specific values to a target's [implementation](#implementation_function), or they can refer to other targets, creating a graph of dependencies. -Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map -from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) -module) to the `attrs` parameter of `rule`. -[Common attributes](/reference/be/common-definitions#common-attributes), such as -`name` and `visibility`, are implicitly added to all rules. Additional -attributes are implicitly added to -[executable and test rules](#executable-rules) specifically. Attributes which -are implicitly added to a rule can't be included in the dictionary passed to -`attrs`. +Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) module) to the `attrs` parameter of `rule`. [Common attributes](/reference/be/common-definitions#common-attributes), such as `name` and `visibility`, are implicitly added to all rules. Additional attributes are implicitly added to [executable and test rules](#executable-rules) specifically. Attributes which are implicitly added to a rule can't be included in the dictionary passed to `attrs`. ### Dependency attributes -Rules that process source code usually define the following attributes to handle -various [types of dependencies](/concepts/dependencies#types_of_dependencies): - -* `srcs` specifies source files processed by a target's actions. Often, the - attribute schema specifies which file extensions are expected for the sort - of source file the rule processes. Rules for languages with header files - generally specify a separate `hdrs` attribute for headers processed by a - target and its consumers. -* `deps` specifies code dependencies for a target. The attribute schema should - specify which [providers](#providers) those dependencies must provide. (For - example, `cc_library` provides `CcInfo`.) -* `data` specifies files to be made available at runtime to any executable - which depends on a target. That should allow arbitrary files to be - specified. +Rules that process source code usually define the following attributes to handle various [types of dependencies](/concepts/dependencies#types_of_dependencies): + +- `srcs` specifies source files processed by a target's actions. Often, the attribute schema specifies which file extensions are expected for the sort of source file the rule processes. Rules for languages with header files generally specify a separate `hdrs` attribute for headers processed by a target and its consumers. +- `deps` specifies code dependencies for a target. The attribute schema should specify which [providers](#providers) those dependencies must provide. (For example, `cc_library` provides `CcInfo`.) +- `data` specifies files to be made available at runtime to any executable which depends on a target. That should allow arbitrary files to be specified. ```python example_library = rule( @@ -141,16 +84,7 @@ example_library = rule( ) ``` -These are examples of *dependency attributes*. Any attribute that specifies -an input label (those defined with -[`attr.label_list`](/rules/lib/toplevel/attr#label_list), -[`attr.label`](/rules/lib/toplevel/attr#label), or -[`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) -specifies dependencies of a certain type -between a target and the targets whose labels (or the corresponding -[`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target -is defined. The repository, and possibly the path, for these labels is resolved -relative to the defined target. +These are examples of *dependency attributes*. Any attribute that specifies an input label (those defined with [`attr.label_list`](/rules/lib/toplevel/attr#label_list), [`attr.label`](/rules/lib/toplevel/attr#label), or [`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) specifies dependencies of a certain type between a target and the targets whose labels (or the corresponding [`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target is defined. The repository, and possibly the path, for these labels is resolved relative to the defined target. ```python example_library( @@ -164,27 +98,15 @@ example_library( ) ``` -In this example, `other_target` is a dependency of `my_target`, and therefore -`other_target` is analyzed first. It is an error if there is a cycle in the -dependency graph of targets. +In this example, `other_target` is a dependency of `my_target`, and therefore `other_target` is analyzed first. It is an error if there is a cycle in the dependency graph of targets. - +[]() ### Private attributes and implicit dependencies -A dependency attribute with a default value creates an *implicit dependency*. It -is implicit because it's a part of the target graph that the user doesn't -specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a -relationship between a rule and a *tool* (a build-time dependency, such as a -compiler), since most of the time a user is not interested in specifying what -tool the rule uses. Inside the rule's implementation function, this is treated -the same as other dependencies. +A dependency attribute with a default value creates an *implicit dependency*. It is implicit because it's a part of the target graph that the user doesn't specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a relationship between a rule and a *tool* (a build-time dependency, such as a compiler), since most of the time a user is not interested in specifying what tool the rule uses. Inside the rule's implementation function, this is treated the same as other dependencies. -If you want to provide an implicit dependency without allowing the user to -override that value, you can make the attribute *private* by giving it a name -that begins with an underscore (`_`). Private attributes must have default -values. It generally only makes sense to use private attributes for implicit -dependencies. +If you want to provide an implicit dependency without allowing the user to override that value, you can make the attribute *private* by giving it a name that begins with an underscore (`_`). Private attributes must have default values. It generally only makes sense to use private attributes for implicit dependencies. ```python example_library = rule( @@ -201,67 +123,34 @@ example_library = rule( ) ``` -In this example, every target of type `example_library` has an implicit -dependency on the compiler `//tools:example_compiler`. This allows -`example_library`'s implementation function to generate actions that invoke the -compiler, even though the user did not pass its label as an input. Since -`_compiler` is a private attribute, it follows that `ctx.attr._compiler` -will always point to `//tools:example_compiler` in all targets of this rule -type. Alternatively, you can name the attribute `compiler` without the -underscore and keep the default value. This allows users to substitute a -different compiler if necessary, but it requires no awareness of the compiler's -label. - -Implicit dependencies are generally used for tools that reside in the same -repository as the rule implementation. If the tool comes from the -[execution platform](/extending/platforms) or a different repository instead, the -rule should obtain that tool from a [toolchain](/extending/toolchains). +In this example, every target of type `example_library` has an implicit dependency on the compiler `//tools:example_compiler`. This allows `example_library`'s implementation function to generate actions that invoke the compiler, even though the user did not pass its label as an input. Since `_compiler` is a private attribute, it follows that `ctx.attr._compiler` will always point to `//tools:example_compiler` in all targets of this rule type. Alternatively, you can name the attribute `compiler` without the underscore and keep the default value. This allows users to substitute a different compiler if necessary, but it requires no awareness of the compiler's label. + +Implicit dependencies are generally used for tools that reside in the same repository as the rule implementation. If the tool comes from the [execution platform](/extending/platforms) or a different repository instead, the rule should obtain that tool from a [toolchain](/extending/toolchains). ### Output attributes -*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and -[`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the -target generates. These differ from dependency attributes in two ways: +*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and [`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the target generates. These differ from dependency attributes in two ways: -* They define output file targets instead of referring to targets defined - elsewhere. -* The output file targets depend on the instantiated rule target, instead of - the other way around. +- They define output file targets instead of referring to targets defined elsewhere. +- The output file targets depend on the instantiated rule target, instead of the other way around. -Typically, output attributes are only used when a rule needs to create outputs -with user-defined names which can't be based on the target name. If a rule has -one output attribute, it is typically named `out` or `outs`. +Typically, output attributes are only used when a rule needs to create outputs with user-defined names which can't be based on the target name. If a rule has one output attribute, it is typically named `out` or `outs`. -Output attributes are the preferred way of creating *predeclared outputs*, which -can be specifically depended upon or -[requested at the command line](#requesting_output_files). +Output attributes are the preferred way of creating *predeclared outputs*, which can be specifically depended upon or [requested at the command line](#requesting_output_files). ## Implementation function -Every rule requires an `implementation` function. These functions are executed -strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the -graph of targets generated in the loading phase into a graph of -[actions](#actions) to be performed during the execution phase. As such, -implementation functions can't actually read or write files. +Every rule requires an `implementation` function. These functions are executed strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the graph of targets generated in the loading phase into a graph of [actions](#actions) to be performed during the execution phase. As such, implementation functions can't actually read or write files. -Rule implementation functions are usually private (named with a leading -underscore). Conventionally, they are named the same as their rule, but suffixed -with `_impl`. +Rule implementation functions are usually private (named with a leading underscore). Conventionally, they are named the same as their rule, but suffixed with `_impl`. -Implementation functions take exactly one parameter: a -[rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of -[providers](#providers). +Implementation functions take exactly one parameter: a [rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of [providers](#providers). ### Targets -Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) -objects. These objects contain the [providers](#providers) generated when the -target's implementation function was executed. +Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) objects. These objects contain the [providers](#providers) generated when the target's implementation function was executed. -[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each -dependency attribute, containing `Target` objects representing each direct -dependency using that attribute. For `label_list` attributes, this is a list of -`Targets`. For `label` attributes, this is a single `Target` or `None`. +[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each dependency attribute, containing `Target` objects representing each direct dependency using that attribute. For `label_list` attributes, this is a list of `Targets`. For `label` attributes, this is a single `Target` or `None`. A list of provider objects are returned by a target's implementation function: @@ -269,14 +158,9 @@ A list of provider objects are returned by a target's implementation function: return [ExampleInfo(headers = depset(...))] ``` -Those can be accessed using index notation (`[]`), with the type of provider as -a key. These can be [custom providers](#custom_providers) defined in Starlark or -[providers for native rules](/rules/lib/providers) available as Starlark -global variables. +Those can be accessed using index notation (`[]`), with the type of provider as a key. These can be [custom providers](#custom_providers) defined in Starlark or [providers for native rules](/rules/lib/providers) available as Starlark global variables. -For example, if a rule takes header files using a `hdrs` attribute and provides -them to the compilation actions of the target and its consumers, it could -collect them like so: +For example, if a rule takes header files using a `hdrs` attribute and provides them to the compilation actions of the target and its consumers, it could collect them like so: ```python def _example_library_impl(ctx): @@ -284,24 +168,15 @@ def _example_library_impl(ctx): transitive_headers = [hdr[ExampleInfo].headers for hdr in ctx.attr.hdrs] ``` -There's a legacy struct style, which is strongly discouraged and rules should be -[migrated away from it](#migrating_from_legacy_providers). +There's a legacy struct style, which is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). ### Files -Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't -perform file I/O during the analysis phase, these objects can't be used to -directly read or write file content. Rather, they are passed to action-emitting -functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the -action graph. +Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't perform file I/O during the analysis phase, these objects can't be used to directly read or write file content. Rather, they are passed to action-emitting functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the action graph. -A `File` can either be a source file or a generated file. Each generated file -must be an output of exactly one action. Source files can't be the output of -any action. +A `File` can either be a source file or a generated file. Each generated file must be an output of exactly one action. Source files can't be the output of any action. -For each dependency attribute, the corresponding field of -[`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all -dependencies using that attribute: +For each dependency attribute, the corresponding field of [`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all dependencies using that attribute: ```python def _example_library_impl(ctx): @@ -311,20 +186,11 @@ def _example_library_impl(ctx): ... ``` -[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for -dependency attributes whose specs set `allow_single_file = True`. -[`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only -contains fields for dependency attributes whose specs set `executable = True`. +[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for dependency attributes whose specs set `allow_single_file = True`. [`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only contains fields for dependency attributes whose specs set `executable = True`. ### Declaring outputs -During the analysis phase, a rule's implementation function can create outputs. -Since all labels have to be known during the loading phase, these additional -outputs have no labels. `File` objects for outputs can be created using -[`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and -[`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). -Often, the names of outputs are based on the target's name, -[`ctx.label.name`](/rules/lib/builtins/ctx#label): +During the analysis phase, a rule's implementation function can create outputs. Since all labels have to be known during the loading phase, these additional outputs have no labels. `File` objects for outputs can be created using [`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and [`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). Often, the names of outputs are based on the target's name, [`ctx.label.name`](/rules/lib/builtins/ctx#label): ```python def _example_library_impl(ctx): @@ -333,31 +199,20 @@ def _example_library_impl(ctx): ... ``` -For *predeclared outputs*, like those created for -[output attributes](#output_attributes), `File` objects instead can be retrieved -from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). +For *predeclared outputs*, like those created for [output attributes](#output_attributes), `File` objects instead can be retrieved from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). ### Actions -An action describes how to generate a set of outputs from a set of inputs, for -example "run gcc on hello.c and get hello.o". When an action is created, Bazel -doesn't run the command immediately. It registers it in a graph of dependencies, -because an action can depend on the output of another action. For example, in C, -the linker must be called after the compiler. +An action describes how to generate a set of outputs from a set of inputs, for example "run gcc on hello.c and get hello.o". When an action is created, Bazel doesn't run the command immediately. It registers it in a graph of dependencies, because an action can depend on the output of another action. For example, in C, the linker must be called after the compiler. -General-purpose functions that create actions are defined in -[`ctx.actions`](/rules/lib/builtins/actions): +General-purpose functions that create actions are defined in [`ctx.actions`](/rules/lib/builtins/actions): -* [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. -* [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell - command. -* [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. -* [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to - generate a file from a template. +- [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. +- [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell command. +- [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. +- [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to generate a file from a template. -[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently -accumulate the arguments for actions. It avoids flattening depsets until -execution time: +[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently accumulate the arguments for actions. It avoids flattening depsets until execution time: ```python def _example_library_impl(ctx): @@ -384,61 +239,31 @@ def _example_library_impl(ctx): ... ``` -Actions take a list or depset of input files and generate a (non-empty) list of -output files. The set of input and output files must be known during the -[analysis phase](/extending/concepts#evaluation-model). It might depend on the value of -attributes, including providers from dependencies, but it can't depend on the -result of the execution. For example, if your action runs the unzip command, you -must specify which files you expect to be inflated (before running unzip). -Actions which create a variable number of files internally can wrap those in a -single file (such as a zip, tar, or other archive format). +Actions take a list or depset of input files and generate a (non-empty) list of output files. The set of input and output files must be known during the [analysis phase](/extending/concepts#evaluation-model). It might depend on the value of attributes, including providers from dependencies, but it can't depend on the result of the execution. For example, if your action runs the unzip command, you must specify which files you expect to be inflated (before running unzip). Actions which create a variable number of files internally can wrap those in a single file (such as a zip, tar, or other archive format). -Actions must list all of their inputs. Listing inputs that are not used is -permitted, but inefficient. +Actions must list all of their inputs. Listing inputs that are not used is permitted, but inefficient. -Actions must create all of their outputs. They may write other files, but -anything not in outputs won't be available to consumers. All declared outputs -must be written by some action. +Actions must create all of their outputs. They may write other files, but anything not in outputs won't be available to consumers. All declared outputs must be written by some action. -Actions are comparable to pure functions: They should depend only on the -provided inputs, and avoid accessing computer information, username, clock, -network, or I/O devices (except for reading inputs and writing outputs). This is -important because the output will be cached and reused. +Actions are comparable to pure functions: They should depend only on the provided inputs, and avoid accessing computer information, username, clock, network, or I/O devices (except for reading inputs and writing outputs). This is important because the output will be cached and reused. -Dependencies are resolved by Bazel, which decides which actions to -execute. It is an error if there is a cycle in the dependency graph. Creating -an action doesn't guarantee that it will be executed, that depends on whether -its outputs are needed for the build. +Dependencies are resolved by Bazel, which decides which actions to execute. It is an error if there is a cycle in the dependency graph. Creating an action doesn't guarantee that it will be executed, that depends on whether its outputs are needed for the build. ### Providers -Providers are pieces of information that a rule exposes to other rules that -depend on it. This data can include output files, libraries, parameters to pass -on a tool's command line, or anything else a target's consumers should know -about. +Providers are pieces of information that a rule exposes to other rules that depend on it. This data can include output files, libraries, parameters to pass on a tool's command line, or anything else a target's consumers should know about. -Since a rule's implementation function can only read providers from the -instantiated target's immediate dependencies, rules need to forward any -information from a target's dependencies that needs to be known by a target's -consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). +Since a rule's implementation function can only read providers from the instantiated target's immediate dependencies, rules need to forward any information from a target's dependencies that needs to be known by a target's consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). -A target's providers are specified by a list of provider objects returned by -the implementation function. +A target's providers are specified by a list of provider objects returned by the implementation function. -Old implementation functions can also be written in a legacy style where the -implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of -provider objects. This style is strongly discouraged and rules should be -[migrated away from it](#migrating_from_legacy_providers). +Old implementation functions can also be written in a legacy style where the implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of provider objects. This style is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). #### Default outputs -A target's *default outputs* are the outputs that are requested by default when -the target is requested for build at the command line. For example, a -`java_library` target `//pkg:foo` has `foo.jar` as a default output, so that -will be built by the command `bazel build //pkg:foo`. +A target's *default outputs* are the outputs that are requested by default when the target is requested for build at the command line. For example, a `java_library` target `//pkg:foo` has `foo.jar` as a default output, so that will be built by the command `bazel build //pkg:foo`. -Default outputs are specified by the `files` parameter of -[`DefaultInfo`](/rules/lib/providers/DefaultInfo): +Default outputs are specified by the `files` parameter of [`DefaultInfo`](/rules/lib/providers/DefaultInfo): ```python def _example_library_impl(ctx): @@ -449,37 +274,17 @@ def _example_library_impl(ctx): ] ``` -If `DefaultInfo` is not returned by a rule implementation or the `files` -parameter is not specified, `DefaultInfo.files` defaults to all -*predeclared outputs* (generally, those created by [output -attributes](#output_attributes)). +If `DefaultInfo` is not returned by a rule implementation or the `files` parameter is not specified, `DefaultInfo.files` defaults to all *predeclared outputs* (generally, those created by [output attributes](#output_attributes)). -Rules that perform actions should provide default outputs, even if those outputs -are not expected to be directly used. Actions that are not in the graph of the -requested outputs are pruned. If an output is only used by a target's consumers, -those actions won't be performed when the target is built in isolation. This -makes debugging more difficult because rebuilding just the failing target won't -reproduce the failure. +Rules that perform actions should provide default outputs, even if those outputs are not expected to be directly used. Actions that are not in the graph of the requested outputs are pruned. If an output is only used by a target's consumers, those actions won't be performed when the target is built in isolation. This makes debugging more difficult because rebuilding just the failing target won't reproduce the failure. #### Runfiles -Runfiles are a set of files used by a target at runtime (as opposed to build -time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates -a directory tree containing symlinks pointing to the runfiles. This stages the -environment for the binary so it can access the runfiles during runtime. +Runfiles are a set of files used by a target at runtime (as opposed to build time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates a directory tree containing symlinks pointing to the runfiles. This stages the environment for the binary so it can access the runfiles during runtime. -Runfiles can be added manually during rule creation. -[`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method -on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the -`runfiles` parameter on `DefaultInfo`. The executable output of -[executable rules](#executable-rules) is implicitly added to the runfiles. +Runfiles can be added manually during rule creation. [`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the `runfiles` parameter on `DefaultInfo`. The executable output of [executable rules](#executable-rules) is implicitly added to the runfiles. -Some rules specify attributes, generally named -[`data`](/reference/be/common-definitions#common.data), whose outputs are added to -a targets' runfiles. Runfiles should also be merged in from `data`, as well as -from any attributes which might provide code for eventual execution, generally -`srcs` (which might contain `filegroup` targets with associated `data`) and -`deps`. +Some rules specify attributes, generally named [`data`](/reference/be/common-definitions#common.data), whose outputs are added to a targets' runfiles. Runfiles should also be merged in from `data`, as well as from any attributes which might provide code for eventual execution, generally `srcs` (which might contain `filegroup` targets with associated `data`) and `deps`. ```python def _example_library_impl(ctx): @@ -503,8 +308,7 @@ def _example_library_impl(ctx): #### Custom providers -Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) -function to convey rule-specific information: +Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) function to convey rule-specific information: ```python ExampleInfo = provider( @@ -537,23 +341,11 @@ def _example_library_impl(ctx): ##### Custom initialization of providers -It's possible to guard the instantiation of a provider with custom -preprocessing and validation logic. This can be used to ensure that all -provider instances satisfy certain invariants, or to give users a cleaner API for -obtaining an instance. +It's possible to guard the instantiation of a provider with custom preprocessing and validation logic. This can be used to ensure that all provider instances satisfy certain invariants, or to give users a cleaner API for obtaining an instance. -This is done by passing an `init` callback to the -[`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the -return type of `provider()` changes to be a tuple of two values: the provider -symbol that is the ordinary return value when `init` is not used, and a "raw -constructor". +This is done by passing an `init` callback to the [`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the return type of `provider()` changes to be a tuple of two values: the provider symbol that is the ordinary return value when `init` is not used, and a "raw constructor". -In this case, when the provider symbol is called, instead of directly returning -a new instance, it will forward the arguments along to the `init` callback. The -callback's return value must be a dict mapping field names (strings) to values; -this is used to initialize the fields of the new instance. Note that the -callback may have any signature, and if the arguments don't match the signature -an error is reported as if the callback were invoked directly. +In this case, when the provider symbol is called, instead of directly returning a new instance, it will forward the arguments along to the `init` callback. The callback's return value must be a dict mapping field names (strings) to values; this is used to initialize the fields of the new instance. Note that the callback may have any signature, and if the arguments don't match the signature an error is reported as if the callback were invoked directly. The raw constructor, by contrast, will bypass the `init` callback. @@ -586,9 +378,7 @@ ExampleInfo( ) ``` -The raw constructor can be used to define alternative public factory functions -that don't go through the `init` logic. For example, exampleinfo.bzl -could define: +The raw constructor can be used to define alternative public factory functions that don't go through the `init` logic. For example, exampleinfo.bzl could define: ```python def make_barebones_exampleinfo(headers): @@ -596,12 +386,9 @@ def make_barebones_exampleinfo(headers): return _new_exampleinfo(files_to_link = depset(), headers = all_headers) ``` -Typically, the raw constructor is bound to a variable whose name begins with an -underscore (`_new_exampleinfo` above), so that user code can't load it and -generate arbitrary provider instances. +Typically, the raw constructor is bound to a variable whose name begins with an underscore (`_new_exampleinfo` above), so that user code can't load it and generate arbitrary provider instances. -Another use for `init` is to prevent the user from calling the provider -symbol altogether, and force them to use a factory function instead: +Another use for `init` is to prevent the user from calling the provider symbol altogether, and force them to use a factory function instead: ```python def _exampleinfo_init_banned(*args, **kwargs): @@ -616,15 +403,11 @@ def make_exampleinfo(...): return _new_exampleinfo(...) ``` - +[]() ## Executable rules and test rules -Executable rules define targets that can be invoked by a `bazel run` command. -Test rules are a special kind of executable rule whose targets can also be -invoked by a `bazel test` command. Executable and test rules are created by -setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or -[`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: +Executable rules define targets that can be invoked by a `bazel run` command. Test rules are a special kind of executable rule whose targets can also be invoked by a `bazel test` command. Executable and test rules are created by setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or [`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: ```python example_binary = rule( @@ -640,17 +423,9 @@ example_test = rule( ) ``` -Test rules must have names that end in `_test`. (Test *target* names also often -end in `_test` by convention, but this is not required.) Non-test rules must not -have this suffix. +Test rules must have names that end in `_test`. (Test *target* names also often end in `_test` by convention, but this is not required.) Non-test rules must not have this suffix. -Both kinds of rules must produce an executable output file (which may or may not -be predeclared) that will be invoked by the `run` or `test` commands. To tell -Bazel which of a rule's outputs to use as this executable, pass it as the -`executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) -provider. That `executable` is added to the default outputs of the rule (so you -don't need to pass that to both `executable` and `files`). It's also implicitly -added to the [runfiles](#runfiles): +Both kinds of rules must produce an executable output file (which may or may not be predeclared) that will be invoked by the `run` or `test` commands. To tell Bazel which of a rule's outputs to use as this executable, pass it as the `executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) provider. That `executable` is added to the default outputs of the rule (so you don't need to pass that to both `executable` and `files`). It's also implicitly added to the [runfiles](#runfiles): ```python def _example_binary_impl(ctx): @@ -662,30 +437,13 @@ def _example_binary_impl(ctx): ] ``` -The action that generates this file must set the executable bit on the file. For -a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or -[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done -by the underlying tool that is invoked by the action. For a -[`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. - -As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a -special `ctx.outputs.executable` predeclared output. This file serves as the -default executable if you don't specify one using `DefaultInfo`; it must not be -used otherwise. This output mechanism is deprecated because it doesn't support -customizing the executable file's name at analysis time. - -See examples of an -[executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) -and a -[test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). - -[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and -[test rules](/reference/be/common-definitions#common-attributes-tests) have additional -attributes implicitly defined, in addition to those added for -[all rules](/reference/be/common-definitions#common-attributes). The defaults of -implicitly-added attributes can't be changed, though this can be worked around -by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the -default: +The action that generates this file must set the executable bit on the file. For a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done by the underlying tool that is invoked by the action. For a [`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. + +As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a special `ctx.outputs.executable` predeclared output. This file serves as the default executable if you don't specify one using `DefaultInfo`; it must not be used otherwise. This output mechanism is deprecated because it doesn't support customizing the executable file's name at analysis time. + +See examples of an [executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) and a [test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). + +[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and [test rules](/reference/be/common-definitions#common-attributes-tests) have additional attributes implicitly defined, in addition to those added for [all rules](/reference/be/common-definitions#common-attributes). The defaults of implicitly-added attributes can't be changed, though this can be worked around by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the default: ```python def example_test(size = "small", **kwargs): @@ -698,8 +456,7 @@ _example_test = rule( ### Runfiles location -When an executable target is run with `bazel run` (or `test`), the root of the -runfiles directory is adjacent to the executable. The paths relate as follows: +When an executable target is run with `bazel run` (or `test`), the root of the runfiles directory is adjacent to the executable. The paths relate as follows: ```python # Given launcher_path and runfile_file: @@ -710,50 +467,21 @@ execution_root_relative_path = "%s/%s/%s" % ( runfiles_root, workspace_name, runfile_path) ``` -The path to a `File` under the runfiles directory corresponds to -[`File.short_path`](/rules/lib/builtins/File#short_path). +The path to a `File` under the runfiles directory corresponds to [`File.short_path`](/rules/lib/builtins/File#short_path). -The binary executed directly by `bazel` is adjacent to the root of the -`runfiles` directory. However, binaries called *from* the runfiles can't make -the same assumption. To mitigate this, each binary should provide a way to -accept its runfiles root as a parameter using an environment, or command line -argument or flag. This allows binaries to pass the correct canonical runfiles root -to the binaries it calls. If that's not set, a binary can guess that it was the -first binary called and look for an adjacent runfiles directory. +The binary executed directly by `bazel` is adjacent to the root of the `runfiles` directory. However, binaries called *from* the runfiles can't make the same assumption. To mitigate this, each binary should provide a way to accept its runfiles root as a parameter using an environment, or command line argument or flag. This allows binaries to pass the correct canonical runfiles root to the binaries it calls. If that's not set, a binary can guess that it was the first binary called and look for an adjacent runfiles directory. ## Advanced topics ### Requesting output files -A single target can have several output files. When a `bazel build` command is -run, some of the outputs of the targets given to the command are considered to -be *requested*. Bazel only builds these requested files and the files that they -directly or indirectly depend on. (In terms of the action graph, Bazel only -executes the actions that are reachable as transitive dependencies of the -requested files.) - -In addition to [default outputs](#default_outputs), any *predeclared output* can -be explicitly requested on the command line. Rules can specify predeclared -outputs using [output attributes](#output_attributes). In that case, the user -explicitly chooses labels for outputs when they instantiate the rule. To obtain -[`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding -attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can -[implicitly define predeclared outputs](#deprecated_predeclared_outputs) based -on the target name as well, but this feature is deprecated. - -In addition to default outputs, there are *output groups*, which are collections -of output files that may be requested together. These can be requested with -[`--output_groups`](/reference/command-line-reference#flag--output_groups). For -example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` -output group, these files can be built by running `bazel build //pkg:mytarget ---output_groups=debug_files`. Since non-predeclared outputs don't have labels, -they can only be requested by appearing in the default outputs or an output -group. - -Output groups can be specified with the -[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many -built-in providers, `OutputGroupInfo` can take parameters with arbitrary names -to define output groups with that name: +A single target can have several output files. When a `bazel build` command is run, some of the outputs of the targets given to the command are considered to be *requested*. Bazel only builds these requested files and the files that they directly or indirectly depend on. (In terms of the action graph, Bazel only executes the actions that are reachable as transitive dependencies of the requested files.) + +In addition to [default outputs](#default_outputs), any *predeclared output* can be explicitly requested on the command line. Rules can specify predeclared outputs using [output attributes](#output_attributes). In that case, the user explicitly chooses labels for outputs when they instantiate the rule. To obtain [`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can [implicitly define predeclared outputs](#deprecated_predeclared_outputs) based on the target name as well, but this feature is deprecated. + +In addition to default outputs, there are *output groups*, which are collections of output files that may be requested together. These can be requested with [`--output_groups`](/reference/command-line-reference#flag--output_groups). For example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` output group, these files can be built by running `bazel build //pkg:mytarget --output_groups=debug_files`. Since non-predeclared outputs don't have labels, they can only be requested by appearing in the default outputs or an output group. + +Output groups can be specified with the [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many built-in providers, `OutputGroupInfo` can take parameters with arbitrary names to define output groups with that name: ```python def _example_library_impl(ctx): @@ -770,82 +498,37 @@ def _example_library_impl(ctx): ] ``` -Also unlike most providers, `OutputGroupInfo` can be returned by both an -[aspect](/extending/aspects) and the rule target to which that aspect is applied, as -long as they don't define the same output groups. In that case, the resulting -providers are merged. +Also unlike most providers, `OutputGroupInfo` can be returned by both an [aspect](/extending/aspects) and the rule target to which that aspect is applied, as long as they don't define the same output groups. In that case, the resulting providers are merged. -Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts -of files from a target to the actions of its consumers. Define -[rule-specific providers](#custom_providers) for that instead. +Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts of files from a target to the actions of its consumers. Define [rule-specific providers](#custom_providers) for that instead. ### Configurations -Imagine that you want to build a C++ binary for a different architecture. The -build can be complex and involve multiple steps. Some of the intermediate -binaries, like compilers and code generators, have to run on -[the execution platform](/extending/platforms#overview) (which could be your host, -or a remote executor). Some binaries like the final output must be built for the -target architecture. - -For this reason, Bazel has a concept of "configurations" and transitions. The -topmost targets (the ones requested on the command line) are built-in the -"target" configuration, while tools that should run on the execution platform -are built-in an "exec" configuration. Rules may generate different actions based -on the configuration, for instance to change the cpu architecture that is passed -to the compiler. In some cases, the same library may be needed for different -configurations. If this happens, it will be analyzed and potentially built -multiple times. - -By default, Bazel builds a target's dependencies in the same configuration as -the target itself, in other words without transitions. When a dependency is a -tool that's needed to help build the target, the corresponding attribute should -specify a transition to an exec configuration. This causes the tool and all its -dependencies to build for the execution platform. - -For each dependency attribute, you can use `cfg` to decide if dependencies -should build in the same configuration or transition to an exec configuration. -If a dependency attribute has the flag `executable = True`, `cfg` must be set -explicitly. This is to guard against accidentally building a tool for the wrong -configuration. -[See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) - -In general, sources, dependent libraries, and executables that will be needed at -runtime can use the same configuration. - -Tools that are executed as part of the build (such as compilers or code generators) -should be built for an exec configuration. In this case, specify `cfg = "exec"` in -the attribute. - -Otherwise, executables that are used at runtime (such as as part of a test) should -be built for the target configuration. In this case, specify `cfg = "target"` in -the attribute. - -`cfg = "target"` doesn't actually do anything: it's purely a convenience value to -help rule designers be explicit about their intentions. When `executable = False`, -which means `cfg` is optional, only set this when it truly helps readability. - -You can also use `cfg = my_transition` to use -[user-defined transitions](/extending/config#user-defined-transitions), which allow -rule authors a great deal of flexibility in changing configurations, with the -drawback of -[making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). - -**Note**: Historically, Bazel didn't have the concept of execution platforms, -and instead all build actions were considered to run on the host machine. Bazel -versions before 6.0 created a distinct "host" configuration to represent this. -If you see references to "host" in code or old documentation, that's what this -refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual -overhead. - - +Imagine that you want to build a C++ binary for a different architecture. The build can be complex and involve multiple steps. Some of the intermediate binaries, like compilers and code generators, have to run on [the execution platform](/extending/platforms#overview) (which could be your host, or a remote executor). Some binaries like the final output must be built for the target architecture. + +For this reason, Bazel has a concept of "configurations" and transitions. The topmost targets (the ones requested on the command line) are built-in the "target" configuration, while tools that should run on the execution platform are built-in an "exec" configuration. Rules may generate different actions based on the configuration, for instance to change the cpu architecture that is passed to the compiler. In some cases, the same library may be needed for different configurations. If this happens, it will be analyzed and potentially built multiple times. + +By default, Bazel builds a target's dependencies in the same configuration as the target itself, in other words without transitions. When a dependency is a tool that's needed to help build the target, the corresponding attribute should specify a transition to an exec configuration. This causes the tool and all its dependencies to build for the execution platform. + +For each dependency attribute, you can use `cfg` to decide if dependencies should build in the same configuration or transition to an exec configuration. If a dependency attribute has the flag `executable = True`, `cfg` must be set explicitly. This is to guard against accidentally building a tool for the wrong configuration. [See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) + +In general, sources, dependent libraries, and executables that will be needed at runtime can use the same configuration. + +Tools that are executed as part of the build (such as compilers or code generators) should be built for an exec configuration. In this case, specify `cfg = "exec"` in the attribute. + +Otherwise, executables that are used at runtime (such as as part of a test) should be built for the target configuration. In this case, specify `cfg = "target"` in the attribute. + +`cfg = "target"` doesn't actually do anything: it's purely a convenience value to help rule designers be explicit about their intentions. When `executable = False`, which means `cfg` is optional, only set this when it truly helps readability. + +You can also use `cfg = my_transition` to use [user-defined transitions](/extending/config#user-defined-transitions), which allow rule authors a great deal of flexibility in changing configurations, with the drawback of [making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). + +**Note**: Historically, Bazel didn't have the concept of execution platforms, and instead all build actions were considered to run on the host machine. Bazel versions before 6.0 created a distinct "host" configuration to represent this. If you see references to "host" in code or old documentation, that's what this refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual overhead. + +[]() ### Configuration fragments -Rules may access -[configuration fragments](/rules/lib/fragments) such as -`cpp` and `java`. However, all required fragments must be declared in -order to avoid access errors: +Rules may access [configuration fragments](/rules/lib/fragments) such as `cpp` and `java`. However, all required fragments must be declared in order to avoid access errors: ```python def _impl(ctx): @@ -862,14 +545,7 @@ my_rule = rule( ### Runfiles symlinks -Normally, the relative path of a file in the runfiles tree is the same as the -relative path of that file in the source tree or generated output tree. If these -need to be different for some reason, you can specify the `root_symlinks` or -`symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to -files, where the paths are relative to the root of the runfiles directory. The -`symlinks` dictionary is the same, but paths are implicitly prefixed with the -name of the main workspace (*not* the name of the repository containing the -current target). +Normally, the relative path of a file in the runfiles tree is the same as the relative path of that file in the source tree or generated output tree. If these need to be different for some reason, you can specify the `root_symlinks` or `symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to files, where the paths are relative to the root of the runfiles directory. The `symlinks` dictionary is the same, but paths are implicitly prefixed with the name of the main workspace (*not* the name of the repository containing the current target). ```python ... @@ -881,37 +557,20 @@ current target). # sometarget.runfiles/ # some/ # path/ - # here.foo -> some_data_file2 - # / + # here.foo -> some_data_file2 + # <workspace_name>/ # some/ # path/ - # here.bar -> some_data_file3 + # here.bar -> some_data_file3 ``` -If `symlinks` or `root_symlinks` is used, be careful not to map two different -files to the same path in the runfiles tree. This will cause the build to fail -with an error describing the conflict. To fix, you will need to modify your -`ctx.runfiles` arguments to remove the collision. This checking will be done for -any targets using your rule, as well as targets of any kind that depend on those -targets. This is especially risky if your tool is likely to be used transitively -by another tool; symlink names must be unique across the runfiles of a tool and -all of its dependencies. +If `symlinks` or `root_symlinks` is used, be careful not to map two different files to the same path in the runfiles tree. This will cause the build to fail with an error describing the conflict. To fix, you will need to modify your `ctx.runfiles` arguments to remove the collision. This checking will be done for any targets using your rule, as well as targets of any kind that depend on those targets. This is especially risky if your tool is likely to be used transitively by another tool; symlink names must be unique across the runfiles of a tool and all of its dependencies. ### Code coverage -When the [`coverage`](/reference/command-line-reference#coverage) command is run, -the build may need to add coverage instrumentation for certain targets. The -build also gathers the list of source files that are instrumented. The subset of -targets that are considered is controlled by the flag -[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). -Test targets are excluded, unless -[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) -is specified. +When the [`coverage`](/reference/command-line-reference#coverage) command is run, the build may need to add coverage instrumentation for certain targets. The build also gathers the list of source files that are instrumented. The subset of targets that are considered is controlled by the flag [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). Test targets are excluded, unless [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) is specified. -If a rule implementation adds coverage instrumentation at build time, it needs -to account for that in its implementation function. -[ctx.coverage_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns -`True` in coverage mode if a target's sources should be instrumented: +If a rule implementation adds coverage instrumentation at build time, it needs to account for that in its implementation function. [ctx.coverage\_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns `True` in coverage mode if a target's sources should be instrumented: ```python # Are this rule's sources instrumented? @@ -919,13 +578,9 @@ if ctx.coverage_instrumented(): # Do something to turn on coverage for this compile action ``` -Logic that always needs to be on in coverage mode (whether a target's sources -specifically are instrumented or not) can be conditioned on -[ctx.configuration.coverage_enabled](/rules/lib/builtins/configuration#coverage_enabled). +Logic that always needs to be on in coverage mode (whether a target's sources specifically are instrumented or not) can be conditioned on [ctx.configuration.coverage\_enabled](/rules/lib/builtins/configuration#coverage_enabled). -If the rule directly includes sources from its dependencies before compilation -(such as header files), it may also need to turn on compile-time instrumentation if -the dependencies' sources should be instrumented: +If the rule directly includes sources from its dependencies before compilation (such as header files), it may also need to turn on compile-time instrumentation if the dependencies' sources should be instrumented: ```python # Are this rule's sources or any of the sources for its direct dependencies @@ -934,13 +589,7 @@ if ctx.coverage_instrumented() or any([ctx.coverage_instrumented(dep) for dep in # Do something to turn on coverage for this compile action ``` -Rules also should provide information about which attributes are relevant for -coverage with the `InstrumentedFilesInfo` provider, constructed using -[`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). -The `dependency_attributes` parameter of `instrumented_files_info` should list -all runtime dependency attributes, including code dependencies like `deps` and -data dependencies like `data`. The `source_attributes` parameter should list the -rule's source files attributes if coverage instrumentation might be added: +Rules also should provide information about which attributes are relevant for coverage with the `InstrumentedFilesInfo` provider, constructed using [`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). The `dependency_attributes` parameter of `instrumented_files_info` should list all runtime dependency attributes, including code dependencies like `deps` and data dependencies like `data`. The `source_attributes` parameter should list the rule's source files attributes if coverage instrumentation might be added: ```python def _example_library_impl(ctx): @@ -957,18 +606,11 @@ def _example_library_impl(ctx): ] ``` -If `InstrumentedFilesInfo` is not returned, a default one is created with each -non-tool [dependency attribute](#dependency_attributes) that doesn't set -[`cfg`](#configuration) to `"exec"` in the attribute schema. in -`dependency_attributes`. (This isn't ideal behavior, since it puts attributes -like `srcs` in `dependency_attributes` instead of `source_attributes`, but it -avoids the need for explicit coverage configuration for all rules in the -dependency chain.) +If `InstrumentedFilesInfo` is not returned, a default one is created with each non-tool [dependency attribute](#dependency_attributes) that doesn't set [`cfg`](#configuration) to `"exec"` in the attribute schema. in `dependency_attributes`. (This isn't ideal behavior, since it puts attributes like `srcs` in `dependency_attributes` instead of `source_attributes`, but it avoids the need for explicit coverage configuration for all rules in the dependency chain.) #### Test rules -Test rules require additional setup to generate coverage reports. The rule -itself has to add the following implicit attributes: +Test rules require additional setup to generate coverage reports. The rule itself has to add the following implicit attributes: ```python my_test = rule( @@ -991,115 +633,59 @@ my_test = rule( ) ``` -By using `configuration_field`, the dependency on the Java LCOV merger tool can -be avoided as long as coverage is not requested. +By using `configuration_field`, the dependency on the Java LCOV merger tool can be avoided as long as coverage is not requested. -When the test is run, it should emit coverage information in the form of one or -more [LCOV files] -(https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT) -with unique names into the directory specified by the `COVERAGE_DIR` environment -variable. Bazel will then merge these files into a single LCOV file using the -`_lcov_merger` tool. If present, it will also collect C/C++ coverage using the -`_collect_cc_coverage` tool. +When the test is run, it should emit coverage information in the form of one or more \[LCOV files] ([https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT](https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT)) with unique names into the directory specified by the `COVERAGE_DIR` environment variable. Bazel will then merge these files into a single LCOV file using the `_lcov_merger` tool. If present, it will also collect C/C++ coverage using the `_collect_cc_coverage` tool. ### Baseline coverage -Since coverage is only collected for code that ends up in the dependency tree of -a test, coverage reports can be misleading as they don't necessarily cover all -the code matched by the `--instrumentation_filter` flag. +Since coverage is only collected for code that ends up in the dependency tree of a test, coverage reports can be misleading as they don't necessarily cover all the code matched by the `--instrumentation_filter` flag. -For this reason, Bazel allows rules to specify baseline coverage files using the -`baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These -files must be generated in LCOV format by a user-defined action and are supposed -to list all lines, branches, functions and/or blocks in the target's source -files (according to the `sources_attributes` and `extensions` parameters). For -source files in targets that are instrumented for coverage, Bazel merges their -baseline coverage into the combined coverage report generated with -`--combined_report` and thus ensures that untested files still show up as -uncovered. +For this reason, Bazel allows rules to specify baseline coverage files using the `baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These files must be generated in LCOV format by a user-defined action and are supposed to list all lines, branches, functions and/or blocks in the target's source files (according to the `sources_attributes` and `extensions` parameters). For source files in targets that are instrumented for coverage, Bazel merges their baseline coverage into the combined coverage report generated with `--combined_report` and thus ensures that untested files still show up as uncovered. -If a rule doesn't provide any baseline coverage files, Bazel generates synthetic -coverage information that only mentions the source file paths, but doesn't -contain any information about their contents. +If a rule doesn't provide any baseline coverage files, Bazel generates synthetic coverage information that only mentions the source file paths, but doesn't contain any information about their contents. ### Validation Actions -Sometimes you need to validate something about the build, and the -information required to do that validation is available only in artifacts -(source files or generated files). Because this information is in artifacts, -rules can't do this validation at analysis time because rules can't read -files. Instead, actions must do this validation at execution time. When -validation fails, the action will fail, and hence so will the build. - -Examples of validations that might be run are static analysis, linting, -dependency and consistency checks, and style checks. - -Validation actions can also help to improve build performance by moving parts -of actions that are not required for building artifacts into separate actions. -For example, if a single action that does compilation and linting can be -separated into a compilation action and a linting action, then the linting -action can be run as a validation action and run in parallel with other actions. - -These "validation actions" often don't produce anything that is used elsewhere -in the build, since they only need to assert things about their inputs. This -presents a problem though: If a validation action doesn't produce anything that -is used elsewhere in the build, how does a rule get the action to run? -Historically, the approach was to have the validation action output an empty -file, and artificially add that output to the inputs of some other important -action in the build: - - - -This works, because Bazel will always run the validation action when the compile -action is run, but this has significant drawbacks: - -1. The validation action is in the critical path of the build. Because Bazel -thinks the empty output is required to run the compile action, it will run the -validation action first, even though the compile action will ignore the input. -This reduces parallelism and slows down builds. - -2. If other actions in the build might run instead of the -compile action, then the empty outputs of validation actions need to be added to -those actions as well (`java_library`'s source jar output, for example). This is -also a problem if new actions that might run instead of the compile action are -added later, and the empty validation output is accidentally left off. +Sometimes you need to validate something about the build, and the information required to do that validation is available only in artifacts (source files or generated files). Because this information is in artifacts, rules can't do this validation at analysis time because rules can't read files. Instead, actions must do this validation at execution time. When validation fails, the action will fail, and hence so will the build. + +Examples of validations that might be run are static analysis, linting, dependency and consistency checks, and style checks. + +Validation actions can also help to improve build performance by moving parts of actions that are not required for building artifacts into separate actions. For example, if a single action that does compilation and linting can be separated into a compilation action and a linting action, then the linting action can be run as a validation action and run in parallel with other actions. + +These "validation actions" often don't produce anything that is used elsewhere in the build, since they only need to assert things about their inputs. This presents a problem though: If a validation action doesn't produce anything that is used elsewhere in the build, how does a rule get the action to run? Historically, the approach was to have the validation action output an empty file, and artificially add that output to the inputs of some other important action in the build: + +![](/rules/validation_action_historical.svg) + +This works, because Bazel will always run the validation action when the compile action is run, but this has significant drawbacks: + +1. The validation action is in the critical path of the build. Because Bazel thinks the empty output is required to run the compile action, it will run the validation action first, even though the compile action will ignore the input. This reduces parallelism and slows down builds. + +2. If other actions in the build might run instead of the compile action, then the empty outputs of validation actions need to be added to those actions as well (`java_library`'s source jar output, for example). This is also a problem if new actions that might run instead of the compile action are added later, and the empty validation output is accidentally left off. The solution to these problems is to use the Validations Output Group. #### Validations Output Group -The Validations Output Group is an output group designed to hold the otherwise -unused outputs of validation actions, so that they don't need to be artificially -added to the inputs of other actions. +The Validations Output Group is an output group designed to hold the otherwise unused outputs of validation actions, so that they don't need to be artificially added to the inputs of other actions. -This group is special in that its outputs are always requested, regardless of -the value of the `--output_groups` flag, and regardless of how the target is -depended upon (for example, on the command line, as a dependency, or through -implicit outputs of the target). Note that normal caching and incrementality -still apply: if the inputs to the validation action have not changed and the -validation action previously succeeded, then the validation action won't be -run. +This group is special in that its outputs are always requested, regardless of the value of the `--output_groups` flag, and regardless of how the target is depended upon (for example, on the command line, as a dependency, or through implicit outputs of the target). Note that normal caching and incrementality still apply: if the inputs to the validation action have not changed and the validation action previously succeeded, then the validation action won't be run. - +![](/rules/validation_action.svg) -Using this output group still requires that validation actions output some file, -even an empty one. This might require wrapping some tools that normally don't -create outputs so that a file is created. +Using this output group still requires that validation actions output some file, even an empty one. This might require wrapping some tools that normally don't create outputs so that a file is created. A target's validation actions are not run in three cases: -* When the target is depended upon as a tool -* When the target is depended upon as an implicit dependency (for example, an - attribute that starts with "_") -* When the target is built in the exec configuration. +- When the target is depended upon as a tool +- When the target is depended upon as an implicit dependency (for example, an attribute that starts with "\_") +- When the target is built in the exec configuration. -It is assumed that these targets have their own -separate builds and tests that would uncover any validation failures. +It is assumed that these targets have their own separate builds and tests that would uncover any validation failures. #### Using the Validations Output Group -The Validations Output Group is named `_validation` and is used like any other -output group: +The Validations Output Group is named `_validation` and is used like any other output group: ```python def _rule_with_validation_impl(ctx): @@ -1119,7 +705,6 @@ def _rule_with_validation_impl(ctx): OutputGroupInfo(_validation = depset([validation_output])), ] - rule_with_validation = rule( implementation = _rule_with_validation_impl, outputs = { @@ -1136,17 +721,9 @@ rule_with_validation = rule( ) ``` -Notice that the validation output file is not added to the `DefaultInfo` or the -inputs to any other action. The validation action for a target of this rule kind -will still run if the target is depended upon by label, or any of the target's -implicit outputs are directly or indirectly depended upon. +Notice that the validation output file is not added to the `DefaultInfo` or the inputs to any other action. The validation action for a target of this rule kind will still run if the target is depended upon by label, or any of the target's implicit outputs are directly or indirectly depended upon. -It is usually important that the outputs of validation actions only go into the -validation output group, and are not added to the inputs of other actions, as -this could defeat parallelism gains. Note however that Bazel doesn't -have any special checking to enforce this. Therefore, you should test -that validation action outputs are not added to the inputs of any actions in the -tests for Starlark rules. For example: +It is usually important that the outputs of validation actions only go into the validation output group, and are not added to the inputs of other actions, as this could defeat parallelism gains. Note however that Bazel doesn't have any special checking to enforce this. Therefore, you should test that validation action outputs are not added to the inputs of any actions in the tests for Starlark rules. For example: ```python load("@bazel_skylib//lib:unittest.bzl", "analysistest") @@ -1171,8 +748,7 @@ validation_outputs_test = analysistest.make(_validation_outputs_test_impl) #### Validation Actions Flag -Running validation actions is controlled by the `--run_validations` command line -flag, which defaults to true. +Running validation actions is controlled by the `--run_validations` command line flag, which defaults to true. ## Deprecated features @@ -1180,52 +756,23 @@ flag, which defaults to true. There are two **deprecated** ways of using predeclared outputs: -* The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies - a mapping between output attribute names and string templates for generating - predeclared output labels. Prefer using non-predeclared outputs and - explicitly adding outputs to `DefaultInfo.files`. Use the rule target's - label as input for rules which consume the output instead of a predeclared - output's label. +- The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies a mapping between output attribute names and string templates for generating predeclared output labels. Prefer using non-predeclared outputs and explicitly adding outputs to `DefaultInfo.files`. Use the rule target's label as input for rules which consume the output instead of a predeclared output's label. -* For [executable rules](#executable-rules), `ctx.outputs.executable` refers - to a predeclared executable output with the same name as the rule target. - Prefer declaring the output explicitly, for example with - `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that - generates the executable sets its permissions to allow execution. Explicitly - pass the executable output to the `executable` parameter of `DefaultInfo`. +- For [executable rules](#executable-rules), `ctx.outputs.executable` refers to a predeclared executable output with the same name as the rule target. Prefer declaring the output explicitly, for example with `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that generates the executable sets its permissions to allow execution. Explicitly pass the executable output to the `executable` parameter of `DefaultInfo`. ### Runfiles features to avoid -[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) -type have a complex set of features, many of which are kept for legacy reasons. -The following recommendations help reduce complexity: - -* **Avoid** use of the `collect_data` and `collect_default` modes of - [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect - runfiles across certain hardcoded dependency edges in confusing ways. - Instead, add files using the `files` or `transitive_files` parameters of - `ctx.runfiles`, or by merging in runfiles from dependencies with - `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. - -* **Avoid** use of the `data_runfiles` and `default_runfiles` of the - `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. - The distinction between "default" and "data" runfiles is maintained for - legacy reasons. For example, some rules put their default outputs in - `data_runfiles`, but not `default_runfiles`. Instead of using - `data_runfiles`, rules should *both* include default outputs and merge in - `default_runfiles` from attributes which provide runfiles (often - [`data`](/reference/be/common-definitions#common-attributes.data)). - -* When retrieving `runfiles` from `DefaultInfo` (generally only for merging - runfiles between the current rule and its dependencies), use - `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. +[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) type have a complex set of features, many of which are kept for legacy reasons. The following recommendations help reduce complexity: + +- **Avoid** use of the `collect_data` and `collect_default` modes of [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect runfiles across certain hardcoded dependency edges in confusing ways. Instead, add files using the `files` or `transitive_files` parameters of `ctx.runfiles`, or by merging in runfiles from dependencies with `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. + +- **Avoid** use of the `data_runfiles` and `default_runfiles` of the `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. The distinction between "default" and "data" runfiles is maintained for legacy reasons. For example, some rules put their default outputs in `data_runfiles`, but not `default_runfiles`. Instead of using `data_runfiles`, rules should *both* include default outputs and merge in `default_runfiles` from attributes which provide runfiles (often [`data`](/reference/be/common-definitions#common-attributes.data)). + +- When retrieving `runfiles` from `DefaultInfo` (generally only for merging runfiles between the current rule and its dependencies), use `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. ### Migrating from legacy providers -Historically, Bazel providers were simple fields on the `Target` object. They -were accessed using the dot operator, and they were created by putting the field -in a [`struct`](/rules/lib/builtins/struct) returned by the rule's -implementation function instead of a list of provider objects: +Historically, Bazel providers were simple fields on the `Target` object. They were accessed using the dot operator, and they were created by putting the field in a [`struct`](/rules/lib/builtins/struct) returned by the rule's implementation function instead of a list of provider objects: ```python return struct(example_info = struct(headers = depset(...))) @@ -1237,13 +784,9 @@ Such providers can be retrieved from the corresponding field of the `Target` obj transitive_headers = [hdr.example_info.headers for hdr in ctx.attr.hdrs] ``` -*This style is deprecated and should not be used in new code;* see following for -information that may help you migrate. The new provider mechanism avoids name -clashes. It also supports data hiding, by requiring any code accessing a -provider instance to retrieve it using the provider symbol. +*This style is deprecated and should not be used in new code;* see following for information that may help you migrate. The new provider mechanism avoids name clashes. It also supports data hiding, by requiring any code accessing a provider instance to retrieve it using the provider symbol. -For the moment, legacy providers are still supported. A rule can return both -legacy and modern providers as follows: +For the moment, legacy providers are still supported. A rule can return both legacy and modern providers as follows: ```python def _old_rule_impl(ctx): @@ -1260,40 +803,18 @@ def _old_rule_impl(ctx): providers = [modern_data, ...]) ``` -If `dep` is the resulting `Target` object for an instance of this rule, the -providers and their contents can be retrieved as `dep.legacy_info.x` and -`dep[MyInfo].y`. - -In addition to `providers`, the returned struct can also take several other -fields that have special meaning (and thus don't create a corresponding legacy -provider): - -* The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and - `executable` correspond to the same-named fields of - [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of - these fields while also returning a `DefaultInfo` provider. - -* The field `output_groups` takes a struct value and corresponds to an - [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). - -In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in -[`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency -attributes, legacy providers are passed in as strings and modern providers are -passed in by their `Info` symbol. Be sure to change from strings to symbols -when migrating. For complex or large rule sets where it is difficult to update -all rules atomically, you may have an easier time if you follow this sequence of -steps: - -1. Modify the rules that produce the legacy provider to produce both the legacy - and modern providers, using the preceding syntax. For rules that declare they - return the legacy provider, update that declaration to include both the - legacy and modern providers. - -2. Modify the rules that consume the legacy provider to instead consume the - modern provider. If any attribute declarations require the legacy provider, - also update them to instead require the modern provider. Optionally, you can - interleave this work with step 1 by having consumers accept or require either - provider: Test for the presence of the legacy provider using - `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. - -3. Fully remove the legacy provider from all rules. +If `dep` is the resulting `Target` object for an instance of this rule, the providers and their contents can be retrieved as `dep.legacy_info.x` and `dep[MyInfo].y`. + +In addition to `providers`, the returned struct can also take several other fields that have special meaning (and thus don't create a corresponding legacy provider): + +- The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and `executable` correspond to the same-named fields of [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of these fields while also returning a `DefaultInfo` provider. + +- The field `output_groups` takes a struct value and corresponds to an [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). + +In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in [`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency attributes, legacy providers are passed in as strings and modern providers are passed in by their `Info` symbol. Be sure to change from strings to symbols when migrating. For complex or large rule sets where it is difficult to update all rules atomically, you may have an easier time if you follow this sequence of steps: + +1. Modify the rules that produce the legacy provider to produce both the legacy and modern providers, using the preceding syntax. For rules that declare they return the legacy provider, update that declaration to include both the legacy and modern providers. + +2. Modify the rules that consume the legacy provider to instead consume the modern provider. If any attribute declarations require the legacy provider, also update them to instead require the modern provider. Optionally, you can interleave this work with step 1 by having consumers accept or require either provider: Test for the presence of the legacy provider using `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. + +3. Fully remove the legacy provider from all rules. diff --git a/extending/toolchains.mdx b/extending/toolchains.mdx index 0d92d6e1..ef07d948 100644 --- a/extending/toolchains.mdx +++ b/extending/toolchains.mdx @@ -2,23 +2,11 @@ title: 'Toolchains' --- - - -This page describes the toolchain framework, which is a way for rule authors to -decouple their rule logic from platform-based selection of tools. It is -recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) -pages before continuing. This page covers why toolchains are needed, how to -define and use them, and how Bazel selects an appropriate toolchain based on -platform constraints. +This page describes the toolchain framework, which is a way for rule authors to decouple their rule logic from platform-based selection of tools. It is recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) pages before continuing. This page covers why toolchains are needed, how to define and use them, and how Bazel selects an appropriate toolchain based on platform constraints. ## Motivation -Let's first look at the problem toolchains are designed to solve. Suppose you -are writing rules to support the "bar" programming language. Your `bar_binary` -rule would compile `*.bar` files using the `barc` compiler, a tool that itself -is built as another target in your workspace. Since users who write `bar_binary` -targets shouldn't have to specify a dependency on the compiler, you make it an -implicit dependency by adding it to the rule definition as a private attribute. +Let's first look at the problem toolchains are designed to solve. Suppose you are writing rules to support the "bar" programming language. Your `bar_binary` rule would compile `*.bar` files using the `barc` compiler, a tool that itself is built as another target in your workspace. Since users who write `bar_binary` targets shouldn't have to specify a dependency on the compiler, you make it an implicit dependency by adding it to the rule definition as a private attribute. ```python bar_binary = rule( @@ -34,9 +22,7 @@ bar_binary = rule( ) ``` -`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so -it'll be built before any `bar_binary` target. It can be accessed by the rule's -implementation function just like any other attribute: +`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so it'll be built before any `bar_binary` target. It can be accessed by the rule's implementation function just like any other attribute: ```python BarcInfo = provider( @@ -58,16 +44,9 @@ def _bar_binary_impl(ctx): ... ``` -The issue here is that the compiler's label is hardcoded into `bar_binary`, yet -different targets may need different compilers depending on what platform they -are being built for and what platform they are being built on -- called the -*target platform* and *execution platform*, respectively. Furthermore, the rule -author does not necessarily even know all the available tools and platforms, so -it is not feasible to hardcode them in the rule's definition. +The issue here is that the compiler's label is hardcoded into `bar_binary`, yet different targets may need different compilers depending on what platform they are being built for and what platform they are being built on -- called the *target platform* and *execution platform*, respectively. Furthermore, the rule author does not necessarily even know all the available tools and platforms, so it is not feasible to hardcode them in the rule's definition. -A less-than-ideal solution would be to shift the burden onto users, by making -the `_compiler` attribute non-private. Then individual targets could be -hardcoded to build for one platform or another. +A less-than-ideal solution would be to shift the burden onto users, by making the `_compiler` attribute non-private. Then individual targets could be hardcoded to build for one platform or another. ```python bar_binary( @@ -83,8 +62,7 @@ bar_binary( ) ``` -You can improve on this solution by using `select` to choose the `compiler` -[based on the platform](/docs/configurable-attributes): +You can improve on this solution by using `select` to choose the `compiler` [based on the platform](/docs/configurable-attributes): ```python config_setting( @@ -111,26 +89,13 @@ bar_binary( ) ``` -But this is tedious and a bit much to ask of every single `bar_binary` user. -If this style is not used consistently throughout the workspace, it leads to -builds that work fine on a single platform but fail when extended to -multi-platform scenarios. It also does not address the problem of adding support -for new platforms and compilers without modifying existing rules or targets. +But this is tedious and a bit much to ask of every single `bar_binary` user. If this style is not used consistently throughout the workspace, it leads to builds that work fine on a single platform but fail when extended to multi-platform scenarios. It also does not address the problem of adding support for new platforms and compilers without modifying existing rules or targets. -The toolchain framework solves this problem by adding an extra level of -indirection. Essentially, you declare that your rule has an abstract dependency -on *some* member of a family of targets (a toolchain type), and Bazel -automatically resolves this to a particular target (a toolchain) based on the -applicable platform constraints. Neither the rule author nor the target author -need know the complete set of available platforms and toolchains. +The toolchain framework solves this problem by adding an extra level of indirection. Essentially, you declare that your rule has an abstract dependency on *some* member of a family of targets (a toolchain type), and Bazel automatically resolves this to a particular target (a toolchain) based on the applicable platform constraints. Neither the rule author nor the target author need know the complete set of available platforms and toolchains. ## Writing rules that use toolchains -Under the toolchain framework, instead of having rules depend directly on tools, -they instead depend on *toolchain types*. A toolchain type is a simple target -that represents a class of tools that serve the same role for different -platforms. For instance, you can declare a type that represents the bar -compiler: +Under the toolchain framework, instead of having rules depend directly on tools, they instead depend on *toolchain types*. A toolchain type is a simple target that represents a class of tools that serve the same role for different platforms. For instance, you can declare a type that represents the bar compiler: ```python # By convention, toolchain_type targets are named "toolchain_type" and @@ -139,9 +104,7 @@ compiler: toolchain_type(name = "toolchain_type") ``` -The rule definition in the previous section is modified so that instead of -taking in the compiler as an attribute, it declares that it consumes a -`//bar_tools:toolchain_type` toolchain. +The rule definition in the previous section is modified so that instead of taking in the compiler as an attribute, it declares that it consumes a `//bar_tools:toolchain_type` toolchain. ```python bar_binary = rule( @@ -155,8 +118,7 @@ bar_binary = rule( ) ``` -The implementation function now accesses this dependency under `ctx.toolchains` -instead of `ctx.attr`, using the toolchain type as the key. +The implementation function now accesses this dependency under `ctx.toolchains` instead of `ctx.attr`, using the toolchain type as the key. ```python def _bar_binary_impl(ctx): @@ -171,28 +133,15 @@ def _bar_binary_impl(ctx): ... ``` -`ctx.toolchains["//bar_tools:toolchain_type"]` returns the -[`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) -of whatever target Bazel resolved the toolchain dependency to. The fields of the -`ToolchainInfo` object are set by the underlying tool's rule; in the next -section, this rule is defined such that there is a `barcinfo` field that wraps -a `BarcInfo` object. +`ctx.toolchains["//bar_tools:toolchain_type"]` returns the [`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) of whatever target Bazel resolved the toolchain dependency to. The fields of the `ToolchainInfo` object are set by the underlying tool's rule; in the next section, this rule is defined such that there is a `barcinfo` field that wraps a `BarcInfo` object. -Bazel's procedure for resolving toolchains to targets is described -[below](#toolchain-resolution). Only the resolved toolchain target is actually -made a dependency of the `bar_binary` target, not the whole space of candidate -toolchains. +Bazel's procedure for resolving toolchains to targets is described [below](#toolchain-resolution). Only the resolved toolchain target is actually made a dependency of the `bar_binary` target, not the whole space of candidate toolchains. ### Mandatory and Optional Toolchains -By default, when a rule expresses a toolchain type dependency using a bare label -(as shown above), the toolchain type is considered to be **mandatory**. If Bazel -is unable to find a matching toolchain (see -[Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain -type, this is an error and analysis halts. +By default, when a rule expresses a toolchain type dependency using a bare label (as shown above), the toolchain type is considered to be **mandatory**. If Bazel is unable to find a matching toolchain (see [Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain type, this is an error and analysis halts. -It is possible instead to declare an **optional** toolchain type dependency, as -follows: +It is possible instead to declare an **optional** toolchain type dependency, as follows: ```python bar_binary = rule( @@ -203,20 +152,20 @@ bar_binary = rule( ) ``` -When an optional toolchain type cannot be resolved, analysis continues, and the -result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. +When an optional toolchain type cannot be resolved, analysis continues, and the result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. -The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) -function defaults to mandatory. +The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) function defaults to mandatory. The following forms can be used: -- Mandatory toolchain types: - - `toolchains = ["//bar_tools:toolchain_type"]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` +- Mandatory toolchain types: + + - `toolchains = ["//bar_tools:toolchain_type"]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` + - Optional toolchain types: - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` ```python bar_binary = rule( @@ -228,15 +177,11 @@ bar_binary = rule( ) ``` -You can mix and match forms in the same rule, also. However, if the same -toolchain type is listed multiple times, it will take the most strict version, -where mandatory is more strict than optional. +You can mix and match forms in the same rule, also. However, if the same toolchain type is listed multiple times, it will take the most strict version, where mandatory is more strict than optional. ### Writing aspects that use toolchains -Aspects have access to the same toolchain API as rules: you can define required -toolchain types, access toolchains via the context, and use them to generate new -actions using the toolchain. +Aspects have access to the same toolchain API as rules: you can define required toolchain types, access toolchains via the context, and use them to generate new actions using the toolchain. ```py bar_aspect = aspect( @@ -255,28 +200,15 @@ def _bar_aspect_impl(target, ctx): To define some toolchains for a given toolchain type, you need three things: -1. A language-specific rule representing the kind of tool or tool suite. By - convention this rule's name is suffixed with "\_toolchain". +1. A language-specific rule representing the kind of tool or tool suite. By convention this rule's name is suffixed with "\_toolchain". - 1. **Note:** The `\_toolchain` rule cannot create any build actions. - Rather, it collects artifacts from other rules and forwards them to the - rule that uses the toolchain. That rule is responsible for creating all - build actions. + 1. **Note:** The `\_toolchain` rule cannot create any build actions. Rather, it collects artifacts from other rules and forwards them to the rule that uses the toolchain. That rule is responsible for creating all build actions. -2. Several targets of this rule type, representing versions of the tool or tool - suite for different platforms. +2. Several targets of this rule type, representing versions of the tool or tool suite for different platforms. -3. For each such target, an associated target of the generic - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - rule, to provide metadata used by the toolchain framework. This `toolchain` - target also refers to the `toolchain_type` associated with this toolchain. - This means that a given `_toolchain` rule could be associated with any - `toolchain_type`, and that only in a `toolchain` instance that uses - this `_toolchain` rule that the rule is associated with a `toolchain_type`. +3. For each such target, an associated target of the generic [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) rule, to provide metadata used by the toolchain framework. This `toolchain` target also refers to the `toolchain_type` associated with this toolchain. This means that a given `_toolchain` rule could be associated with any `toolchain_type`, and that only in a `toolchain` instance that uses this `_toolchain` rule that the rule is associated with a `toolchain_type`. -For our running example, here's a definition for a `bar_toolchain` rule. Our -example has only a compiler, but other tools such as a linker could also be -grouped underneath it. +For our running example, here's a definition for a `bar_toolchain` rule. Our example has only a compiler, but other tools such as a linker could also be grouped underneath it. ```python def _bar_toolchain_impl(ctx): @@ -299,13 +231,7 @@ bar_toolchain = rule( ) ``` -The rule must return a `ToolchainInfo` provider, which becomes the object that -the consuming rule retrieves using `ctx.toolchains` and the label of the -toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value -pairs. The specification of exactly what fields are added to the `ToolchainInfo` -should be clearly documented at the toolchain type. In this example, the values -return wrapped in a `BarcInfo` object to reuse the schema defined above; this -style may be useful for validation and code reuse. +The rule must return a `ToolchainInfo` provider, which becomes the object that the consuming rule retrieves using `ctx.toolchains` and the label of the toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value pairs. The specification of exactly what fields are added to the `ToolchainInfo` should be clearly documented at the toolchain type. In this example, the values return wrapped in a `BarcInfo` object to reuse the schema defined above; this style may be useful for validation and code reuse. Now you can define targets for specific `barc` compilers. @@ -331,10 +257,7 @@ bar_toolchain( ) ``` -Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. -These definitions link the language-specific targets to the toolchain type and -provide the constraint information that tells Bazel when the toolchain is -appropriate for a given platform. +Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. These definitions link the language-specific targets to the toolchain type and provide the constraint information that tells Bazel when the toolchain is appropriate for a given platform. ```python toolchain( @@ -366,21 +289,13 @@ toolchain( ) ``` -The use of relative path syntax above suggests these definitions are all in the -same package, but there's no reason the toolchain type, language-specific -toolchain targets, and `toolchain` definition targets can't all be in separate -packages. +The use of relative path syntax above suggests these definitions are all in the same package, but there's no reason the toolchain type, language-specific toolchain targets, and `toolchain` definition targets can't all be in separate packages. -See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) -for a real-world example. +See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) for a real-world example. ### Toolchains and configurations -An important question for rule authors is, when a `bar_toolchain` target is -analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions -should be used for dependencies? The example above uses string attributes, but -what would happen for a more complicated toolchain that depends on other targets -in the Bazel repository? +An important question for rule authors is, when a `bar_toolchain` target is analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions should be used for dependencies? The example above uses string attributes, but what would happen for a more complicated toolchain that depends on other targets in the Bazel repository? Let's see a more complex version of `bar_toolchain`: @@ -406,32 +321,13 @@ bar_toolchain = rule( ) ``` -The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, -but the meaning of the `cfg` parameter is slightly different. - -The dependency from a target (called the "parent") to a toolchain via toolchain -resolution uses a special configuration transition called the "toolchain -transition". The toolchain transition keeps the configuration the same, except -that it forces the execution platform to be the same for the toolchain as for -the parent (otherwise, toolchain resolution for the toolchain could pick any -execution platform, and wouldn't necessarily be the same as for parent). This -allows any `exec` dependencies of the toolchain to also be executable for the -parent's build actions. Any of the toolchain's dependencies which use `cfg = -"target"` (or which don't specify `cfg`, since "target" is the default) are -built for the same target platform as the parent. This allows toolchain rules to -contribute both libraries (the `system_lib` attribute above) and tools (the -`compiler` attribute) to the build rules which need them. The system libraries -are linked into the final artifact, and so need to be built for the same -platform, whereas the compiler is a tool invoked during the build, and needs to -be able to run on the execution platform. +The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, but the meaning of the `cfg` parameter is slightly different. + +The dependency from a target (called the "parent") to a toolchain via toolchain resolution uses a special configuration transition called the "toolchain transition". The toolchain transition keeps the configuration the same, except that it forces the execution platform to be the same for the toolchain as for the parent (otherwise, toolchain resolution for the toolchain could pick any execution platform, and wouldn't necessarily be the same as for parent). This allows any `exec` dependencies of the toolchain to also be executable for the parent's build actions. Any of the toolchain's dependencies which use `cfg = "target"` (or which don't specify `cfg`, since "target" is the default) are built for the same target platform as the parent. This allows toolchain rules to contribute both libraries (the `system_lib` attribute above) and tools (the `compiler` attribute) to the build rules which need them. The system libraries are linked into the final artifact, and so need to be built for the same platform, whereas the compiler is a tool invoked during the build, and needs to be able to run on the execution platform. ## Registering and building with toolchains -At this point all the building blocks are assembled, and you just need to make -the toolchains available to Bazel's resolution procedure. This is done by -registering the toolchain, either in a `MODULE.bazel` file using -`register_toolchains()`, or by passing the toolchains' labels on the command -line using the `--extra_toolchains` flag. +At this point all the building blocks are assembled, and you just need to make the toolchains available to Bazel's resolution procedure. This is done by registering the toolchain, either in a `MODULE.bazel` file using `register_toolchains()`, or by passing the toolchains' labels on the command line using the `--extra_toolchains` flag. ```python register_toolchains( @@ -444,16 +340,12 @@ register_toolchains( ) ``` -When using target patterns to register toolchains, the order in which the -individual toolchains are registered is determined by the following rules: +When using target patterns to register toolchains, the order in which the individual toolchains are registered is determined by the following rules: -* The toolchains defined in a subpackage of a package are registered before the - toolchains defined in the package itself. -* Within a package, toolchains are registered in the lexicographical order of - their names. +- The toolchains defined in a subpackage of a package are registered before the toolchains defined in the package itself. +- Within a package, toolchains are registered in the lexicographical order of their names. -Now when you build a target that depends on a toolchain type, an appropriate -toolchain will be selected based on the target and execution platforms. +Now when you build a target that depends on a toolchain type, an appropriate toolchain will be selected based on the target and execution platforms. ```python # my_pkg/BUILD @@ -475,109 +367,52 @@ bar_binary( bazel build //my_pkg:my_bar_binary --platforms=//my_pkg:my_target_platform ``` -Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that -has `@platforms//os:linux` and therefore resolve the -`//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. -This will end up building `//bar_tools:barc_linux` but not -`//bar_tools:barc_windows`. +Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that has `@platforms//os:linux` and therefore resolve the `//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. This will end up building `//bar_tools:barc_linux` but not `//bar_tools:barc_windows`. ## Toolchain resolution -Note: [Some Bazel rules](/concepts/platforms#status) do not yet support -toolchain resolution. - -For each target that uses toolchains, Bazel's toolchain resolution procedure -determines the target's concrete toolchain dependencies. The procedure takes as -input a set of required toolchain types, the target platform, the list of -available execution platforms, and the list of available toolchains. Its outputs -are a selected toolchain for each toolchain type as well as a selected execution -platform for the current target. - -The available execution platforms and toolchains are gathered from the -external dependency graph via -[`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) -and -[`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in -`MODULE.bazel` files. -Additional execution platforms and toolchains may also be specified on the -command line via -[`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) -and -[`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). -The host platform is automatically included as an available execution platform. -Available platforms and toolchains are tracked as ordered lists for determinism, -with preference given to earlier items in the list. - -The set of available toolchains, in priority order, is created from -`--extra_toolchains` and `register_toolchains`: - -1. Toolchains registered using `--extra_toolchains` are added first. (Within - these, the **last** toolchain has highest priority.) -2. Toolchains registered using `register_toolchains` in the transitive external - dependency graph, in the following order: (Within these, the **first** - mentioned toolchain has highest priority.) - 1. Toolchains registered by the root module (as in, the `MODULE.bazel` at the - workspace root); - 2. Toolchains registered in the user's `WORKSPACE` file, including in any - macros invoked from there; - 3. Toolchains registered by non-root modules (as in, dependencies specified by - the root module, and their dependencies, and so forth); - 4. Toolchains registered in the "WORKSPACE suffix"; this is only used by - certain native rules bundled with the Bazel installation. - -**NOTE:** [Pseudo-targets like `:all`, `:*`, and -`/...`](/run/build#specifying-build-targets) are ordered by Bazel's package -loading mechanism, which uses a lexicographic ordering. +Note: [Some Bazel rules](/concepts/platforms#status) do not yet support toolchain resolution. + +For each target that uses toolchains, Bazel's toolchain resolution procedure determines the target's concrete toolchain dependencies. The procedure takes as input a set of required toolchain types, the target platform, the list of available execution platforms, and the list of available toolchains. Its outputs are a selected toolchain for each toolchain type as well as a selected execution platform for the current target. + +The available execution platforms and toolchains are gathered from the external dependency graph via [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) and [`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in `MODULE.bazel` files. Additional execution platforms and toolchains may also be specified on the command line via [`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) and [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). The host platform is automatically included as an available execution platform. Available platforms and toolchains are tracked as ordered lists for determinism, with preference given to earlier items in the list. + +The set of available toolchains, in priority order, is created from `--extra_toolchains` and `register_toolchains`: + +1. Toolchains registered using `--extra_toolchains` are added first. (Within these, the **last** toolchain has highest priority.) +2. Toolchains registered using `register_toolchains` in the transitive external dependency graph, in the following order: (Within these, the **first** mentioned toolchain has highest priority.) +3. Toolchains registered by the root module (as in, the `MODULE.bazel` at the workspace root); +4. Toolchains registered in the user's `WORKSPACE` file, including in any macros invoked from there; +5. Toolchains registered by non-root modules (as in, dependencies specified by the root module, and their dependencies, and so forth); +6. Toolchains registered in the "WORKSPACE suffix"; this is only used by certain native rules bundled with the Bazel installation. + +**NOTE:** [Pseudo-targets like `:all`, `:*`, and `/...`](/run/build#specifying-build-targets) are ordered by Bazel's package loading mechanism, which uses a lexicographic ordering. The resolution steps are as follows. -1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a - platform if, for each `constraint_value` in its list, the platform also has - that `constraint_value` (either explicitly or as a default). +1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a platform if, for each `constraint_value` in its list, the platform also has that `constraint_value` (either explicitly or as a default). - If the platform has `constraint_value`s from `constraint_setting`s not - referenced by the clause, these do not affect matching. + If the platform has `constraint_value`s from `constraint_setting`s not referenced by the clause, these do not affect matching. -1. If the target being built specifies the - [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) - (or its rule definition specifies the - [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), - the list of available execution platforms is filtered to remove - any that do not match the execution constraints. +2. If the target being built specifies the [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) (or its rule definition specifies the [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), the list of available execution platforms is filtered to remove any that do not match the execution constraints. -1. The list of available toolchains is filtered to remove any toolchains - specifying `target_settings` that don't match the current configuration. +3. The list of available toolchains is filtered to remove any toolchains specifying `target_settings` that don't match the current configuration. -1. For each available execution platform, you associate each toolchain type with - the first available toolchain, if any, that is compatible with this execution - platform and the target platform. +4. For each available execution platform, you associate each toolchain type with the first available toolchain, if any, that is compatible with this execution platform and the target platform. -1. Any execution platform that failed to find a compatible mandatory toolchain - for one of its toolchain types is ruled out. Of the remaining platforms, the - first one becomes the current target's execution platform, and its associated - toolchains (if any) become dependencies of the target. +5. Any execution platform that failed to find a compatible mandatory toolchain for one of its toolchain types is ruled out. Of the remaining platforms, the first one becomes the current target's execution platform, and its associated toolchains (if any) become dependencies of the target. -The chosen execution platform is used to run all actions that the target -generates. +The chosen execution platform is used to run all actions that the target generates. -In cases where the same target can be built in multiple configurations (such as -for different CPUs) within the same build, the resolution procedure is applied -independently to each version of the target. +In cases where the same target can be built in multiple configurations (such as for different CPUs) within the same build, the resolution procedure is applied independently to each version of the target. -If the rule uses [execution groups](/extending/exec-groups), each execution -group performs toolchain resolution separately, and each has its own execution -platform and toolchains. +If the rule uses [execution groups](/extending/exec-groups), each execution group performs toolchain resolution separately, and each has its own execution platform and toolchains. ## Debugging toolchains -If you are adding toolchain support to an existing rule, use the -`--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag -provides verbose output for toolchain types or target names that match the regex variable. You -can use `.*` to output all information. Bazel will output names of toolchains it -checks and skips during the resolution process. +If you are adding toolchain support to an existing rule, use the `--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag provides verbose output for toolchain types or target names that match the regex variable. You can use `.*` to output all information. Bazel will output names of toolchains it checks and skips during the resolution process. -For example, to debug toolchain selection for all actions created directly by -`//my:target`: +For example, to debug toolchain selection for all actions created directly by `//my:target`: ```sh $ bazel build //my:all --toolchain_resolution_debug=//my:target @@ -589,8 +424,7 @@ To debug toolchain selection for all actions over all build targets: $ bazel build //my:all --toolchain_resolution_debug=.* ``` -If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain -resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: +If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: ``` # Find all direct dependencies of //cc:my_cc_lib. This includes explicitly @@ -609,5 +443,5 @@ $ bazel cquery 'deps(//cc:my_cc_lib, 1)' # Which of these are from toolchain resolution? $ bazel cquery 'deps(//cc:my_cc_lib, 1)' --transitions=lite | grep "toolchain dependency" - [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 + [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 ``` diff --git a/external/extension.mdx b/external/extension.mdx index ababc7fe..679096e2 100644 --- a/external/extension.mdx +++ b/external/extension.mdx @@ -2,42 +2,20 @@ title: 'Module extensions' --- +Module extensions allow users to extend the module system by reading input data from modules across the dependency graph, performing necessary logic to resolve dependencies, and finally creating repos by calling [repo rules](/external/repo). These extensions have capabilities similar to repo rules, which enables them to perform file I/O, send network requests, and so on. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. - -Module extensions allow users to extend the module system by reading input data -from modules across the dependency graph, performing necessary logic to resolve -dependencies, and finally creating repos by calling [repo -rules](/external/repo). These extensions have capabilities similar to repo -rules, which enables them to perform file I/O, send network requests, and so on. -Among other things, they allow Bazel to interact with other package management -systems while also respecting the dependency graph built out of Bazel modules. - -You can define module extensions in `.bzl` files, just like repo rules. They're -not invoked directly; rather, each module specifies pieces of data called *tags* -for extensions to read. Bazel runs module resolution before evaluating any -extensions. The extension reads all the tags belonging to it across the entire -dependency graph. +You can define module extensions in `.bzl` files, just like repo rules. They're not invoked directly; rather, each module specifies pieces of data called *tags* for extensions to read. Bazel runs module resolution before evaluating any extensions. The extension reads all the tags belonging to it across the entire dependency graph. ## Extension usage -Extensions are hosted in Bazel modules themselves. To use an extension in a -module, first add a `bazel_dep` on the module hosting the extension, and then -call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function -to bring it into scope. Consider the following example — a snippet from a -`MODULE.bazel` file to use the "maven" extension defined in the -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) -module: +Extensions are hosted in Bazel modules themselves. To use an extension in a module, first add a `bazel_dep` on the module hosting the extension, and then call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function to bring it into scope. Consider the following example — a snippet from a `MODULE.bazel` file to use the "maven" extension defined in the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) module: ```python bazel_dep(name = "rules_jvm_external", version = "4.5") maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -This binds the return value of `use_extension` to a variable, which allows the -user to use dot-syntax to specify tags for the extension. The tags must follow -the schema defined by the corresponding *tag classes* specified in the -[extension definition](#extension_definition). For an example specifying some -`maven.install` and `maven.artifact` tags: +This binds the return value of `use_extension` to a variable, which allows the user to use dot-syntax to specify tags for the extension. The tags must follow the schema defined by the corresponding *tag classes* specified in the [extension definition](#extension_definition). For an example specifying some `maven.install` and `maven.artifact` tags: ```python maven.install(artifacts = ["org.junit:junit:4.13.2"]) @@ -47,33 +25,19 @@ maven.artifact(group = "com.google.guava", exclusions = ["com.google.j2objc:j2objc-annotations"]) ``` -Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos -generated by the extension into the scope of the current module. +Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos generated by the extension into the scope of the current module. ```python use_repo(maven, "maven") ``` -Repos generated by an extension are part of its API. In this example, the -"maven" module extension promises to generate a repo called `maven`. With the -declaration above, the extension properly resolves labels such as -`@maven//:org_junit_junit` to point to the repo generated by the "maven" -extension. +Repos generated by an extension are part of its API. In this example, the "maven" module extension promises to generate a repo called `maven`. With the declaration above, the extension properly resolves labels such as `@maven//:org_junit_junit` to point to the repo generated by the "maven" extension. -Note: Module extensions are evaluated lazily. This means that an extension will -typically not be evaluated unless some module brings one of its repositories -into scope using `use_repo` and that repository is referenced in a build. While -testing a module extension, `bazel mod deps` can be useful as it -unconditionally evaluates all module extensions. +Note: Module extensions are evaluated lazily. This means that an extension will typically not be evaluated unless some module brings one of its repositories into scope using `use_repo` and that repository is referenced in a build. While testing a module extension, `bazel mod deps` can be useful as it unconditionally evaluates all module extensions. ## Extension definition -You can define module extensions similarly to [repo rules](/external/repo), -using the [`module_extension`](/rules/lib/globals/bzl#module_extension) -function. However, while repo rules have a number of attributes, module -extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which -has a number of attributes. The tag classes define schemas for tags used by this -extension. For example, the "maven" extension above might be defined like this: +You can define module extensions similarly to [repo rules](/external/repo), using the [`module_extension`](/rules/lib/globals/bzl#module_extension) function. However, while repo rules have a number of attributes, module extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which has a number of attributes. The tag classes define schemas for tags used by this extension. For example, the "maven" extension above might be defined like this: ```python # @rules_jvm_external//:extensions.bzl @@ -86,13 +50,9 @@ maven = module_extension( ) ``` -These declarations show that `maven.install` and `maven.artifact` tags can be -specified using the specified attribute schema. +These declarations show that `maven.install` and `maven.artifact` tags can be specified using the specified attribute schema. -The implementation function of module extensions are similar to those of repo -rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, -which grants access to all modules using the extension and all pertinent tags. -The implementation function then calls repo rules to generate repos. +The implementation function of module extensions are similar to those of repo rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, which grants access to all modules using the extension and all pertinent tags. The implementation function then calls repo rules to generate repos. ```python # @rules_jvm_external//:extensions.bzl @@ -120,65 +80,33 @@ def _maven_impl(ctx): ### Extension identity -Module extensions are identified by the name and the `.bzl` file that appears -in the call to `use_extension`. In the following example, the extension `maven` -is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the -name `maven`: +Module extensions are identified by the name and the `.bzl` file that appears in the call to `use_extension`. In the following example, the extension `maven` is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the name `maven`: ```python maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -Re-exporting an extension from a different `.bzl` file gives it a new identity -and if both versions of the extension are used in the transitive module graph, -then they will be evaluated separately and will only see the tags associated -with that particular identity. +Re-exporting an extension from a different `.bzl` file gives it a new identity and if both versions of the extension are used in the transitive module graph, then they will be evaluated separately and will only see the tags associated with that particular identity. -As an extension author you should make sure that users will only use your -module extension from one single `.bzl` file. +As an extension author you should make sure that users will only use your module extension from one single `.bzl` file. ## Repository names and visibility -Repos generated by extensions have canonical names in the form of `{{ "" -}}module_repo_canonical_name+extension_name{{ -"" }}+repo_name`. Note that the canonical name -format is not an API you should depend on — it's subject to change at any time. - -This naming policy means that each extension has its own "repo namespace"; two -distinct extensions can each define a repo with the same name without risking -any clashes. It also means that `repository_ctx.name` reports the canonical name -of the repo, which is *not* the same as the name specified in the repo rule -call. - -Taking repos generated by module extensions into consideration, there are -several repo visibility rules: - -* A Bazel module repo can see all repos introduced in its `MODULE.bazel` file - via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and - [`use_repo`](/rules/lib/globals/module#use_repo). -* A repo generated by a module extension can see all repos visible to the - module that hosts the extension, *plus* all other repos generated by the - same module extension (using the names specified in the repo rule calls as - their apparent names). - * This might result in a conflict. If the module repo can see a repo with - the apparent name `foo`, and the extension generates a repo with the - specified name `foo`, then for all repos generated by that extension - `foo` refers to the former. -* Similarly, in a module extension's implementation function, repos created - by the extension can refer to each other by their apparent names in - attributes, regardless of the order in which they are created. - * In case of a conflict with a repository visible to the module, labels - passed to repository rule attributes can be wrapped in a call to - [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to - the repo visible to the module instead of the extension-generated repo - of the same name. +Repos generated by extensions have canonical names in the form of `<var>module_repo_canonical_name</var>+<var>extension_name</var>+<var>repo_name</var>`. Note that the canonical name format is not an API you should depend on — it's subject to change at any time. + +This naming policy means that each extension has its own "repo namespace"; two distinct extensions can each define a repo with the same name without risking any clashes. It also means that `repository_ctx.name` reports the canonical name of the repo, which is *not* the same as the name specified in the repo rule call. + +Taking repos generated by module extensions into consideration, there are several repo visibility rules: + +- A Bazel module repo can see all repos introduced in its `MODULE.bazel` file via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_repo`](/rules/lib/globals/module#use_repo). +- A repo generated by a module extension can see all repos visible to the module that hosts the extension, *plus* all other repos generated by the same module extension (using the names specified in the repo rule calls as their apparent names). + - This might result in a conflict. If the module repo can see a repo with the apparent name `foo`, and the extension generates a repo with the specified name `foo`, then for all repos generated by that extension `foo` refers to the former. +- Similarly, in a module extension's implementation function, repos created by the extension can refer to each other by their apparent names in attributes, regardless of the order in which they are created. + - In case of a conflict with a repository visible to the module, labels passed to repository rule attributes can be wrapped in a call to [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to the repo visible to the module instead of the extension-generated repo of the same name. ### Overriding and injecting module extension repos -The root module can use -[`override_repo`](/rules/lib/globals/module#override_repo) and -[`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject -module extension repos. +The root module can use [`override_repo`](/rules/lib/globals/module#override_repo) and [`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject module extension repos. #### Example: Replacing `rules_java`'s `java_tools` with a vendored copy @@ -232,80 +160,32 @@ inject_repo(go_deps, "zlib") ## Best practices -This section describes best practices when writing extensions so they are -straightforward to use, maintainable, and adapt well to changes over time. +This section describes best practices when writing extensions so they are straightforward to use, maintainable, and adapt well to changes over time. ### Put each extension in a separate file -When extensions are in a different files, it allows one extension to load -repositories generated by another extension. Even if you don't use this -functionality, it's best to put them in separate files in case you need it -later. This is because the extension's identify is based on its file, so moving -the extension into another file later changes your public API and is a backwards -incompatible change for your users. +When extensions are in a different files, it allows one extension to load repositories generated by another extension. Even if you don't use this functionality, it's best to put them in separate files in case you need it later. This is because the extension's identify is based on its file, so moving the extension into another file later changes your public API and is a backwards incompatible change for your users. ### Specify reproducibility and use facts -If your extension always defines the same repositories given the same inputs -(extension tags, files it reads, etc.) and in particular doesn't rely on -any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by -a checksum, consider returning -[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with -`reproducible = True`. This allows Bazel to skip this extension when writing to -the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces -the chance of merge conflicts. Note that Bazel still caches the results of -reproducible extensions in a way that persists across server restarts, so even -a long-running extension can be marked as reproducible without a performance -penalty. - -If your extension relies on effectively immutable data obtained from outside -the build, most commonly from the network, but you don't have a checksum -available to guard the download, consider using the `facts` parameter of -[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to -persistently record such data and thus allow your extension to become -reproducible. `facts` is expected to be a dictionary with string keys and -arbitrary JSON-like Starlark values that is always persisted in the lockfile and -available to future evaluations of the extension via the -[`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. - -`facts` are not invalidated even when the code of your module extension changes, -so be prepared to handle the case where the structure of `facts` changes. -Bazel also assumes that two different `facts` dicts produced by two different -evaluations of the same extension can be shallowly merged (i.e., as if by using -the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` -not supporting enumeration of its entries, just lookups by key. - -An example of using `facts` would be to record a mapping from version numbers of -some SDK to the an object containing the download URL and checksum of that -version. The first time the extension is evaluated, it can fetch this mapping -from the network, but on later evaluations it can use the mapping from `facts` -to avoid the network requests. +If your extension always defines the same repositories given the same inputs (extension tags, files it reads, etc.) and in particular doesn't rely on any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by a checksum, consider returning [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with `reproducible = True`. This allows Bazel to skip this extension when writing to the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces the chance of merge conflicts. Note that Bazel still caches the results of reproducible extensions in a way that persists across server restarts, so even a long-running extension can be marked as reproducible without a performance penalty. + +If your extension relies on effectively immutable data obtained from outside the build, most commonly from the network, but you don't have a checksum available to guard the download, consider using the `facts` parameter of [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to persistently record such data and thus allow your extension to become reproducible. `facts` is expected to be a dictionary with string keys and arbitrary JSON-like Starlark values that is always persisted in the lockfile and available to future evaluations of the extension via the [`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. + +`facts` are not invalidated even when the code of your module extension changes, so be prepared to handle the case where the structure of `facts` changes. Bazel also assumes that two different `facts` dicts produced by two different evaluations of the same extension can be shallowly merged (i.e., as if by using the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` not supporting enumeration of its entries, just lookups by key. + +An example of using `facts` would be to record a mapping from version numbers of some SDK to the an object containing the download URL and checksum of that version. The first time the extension is evaluated, it can fetch this mapping from the network, but on later evaluations it can use the mapping from `facts` to avoid the network requests. ### Specify dependence on operating system and architecture -If your extension relies on the operating system or its architecture type, -ensure to indicate this in the extension definition using the `os_dependent` -and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the -need for re-evaluation if there are changes to either of them. +If your extension relies on the operating system or its architecture type, ensure to indicate this in the extension definition using the `os_dependent` and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the need for re-evaluation if there are changes to either of them. -Since this kind of dependence on the host makes it more difficult to maintain -the lockfile entry for this extension, consider -[marking the extension reproducible](#specify_reproducibility) if possible. +Since this kind of dependence on the host makes it more difficult to maintain the lockfile entry for this extension, consider [marking the extension reproducible](#specify_reproducibility) if possible. ### Only the root module should directly affect repository names -Remember that when an extension creates repositories, they are created within -the namespace of the extension. This means collisions can occur if different -modules use the same extension and end up creating a repository with the same -name. This often manifests as a module extension's `tag_class` having a `name` -argument that is passed as a repository rule's `name` value. - -For example, say the root module, `A`, depends on module `B`. Both modules -depend on module `mylang`. If both `A` and `B` call -`mylang.toolchain(name="foo")`, they will both try to create a repository named -`foo` within the `mylang` module and an error will occur. - -To avoid this, either remove the ability to set the repository name directly, -or only allow the root module to do so. It's OK to allow the root module this -ability because nothing will depend on it, so it doesn't have to worry about -another module creating a conflicting name. +Remember that when an extension creates repositories, they are created within the namespace of the extension. This means collisions can occur if different modules use the same extension and end up creating a repository with the same name. This often manifests as a module extension's `tag_class` having a `name` argument that is passed as a repository rule's `name` value. + +For example, say the root module, `A`, depends on module `B`. Both modules depend on module `mylang`. If both `A` and `B` call `mylang.toolchain(name="foo")`, they will both try to create a repository named `foo` within the `mylang` module and an error will occur. + +To avoid this, either remove the ability to set the repository name directly, or only allow the root module to do so. It's OK to allow the root module this ability because nothing will depend on it, so it doesn't have to worry about another module creating a conflicting name. diff --git a/external/faq.mdx b/external/faq.mdx index a0239cb5..58d6c521 100644 --- a/external/faq.mdx +++ b/external/faq.mdx @@ -2,145 +2,63 @@ title: 'Frequently asked questions' --- +This page answers some frequently asked questions about external dependencies in Bazel. +## MODULE.bazel +### How should I version a Bazel module? +Setting `version` with the [`module`](/rules/lib/globals/module#module) directive in the source archive `MODULE.bazel` can have several downsides and unintended side effects if not managed carefully: -This page answers some frequently asked questions about external dependencies in -Bazel. +- Duplication: releasing a new version of a module typically involves both incrementing the version in `MODULE.bazel` and tagging the release, two separate steps that can fall out of sync. While automation can reduce this risk, it's simpler and safer to avoid it altogether. -## MODULE.bazel +- Inconsistency: users overriding a module with a specific commit using a [non-registry override](module.md#non-registry_overrides) will see an incorrect version. for example, if the `MODULE.bazel` in the source archive sets `version = "0.3.0"` but additional commits have been made since that release, a user overriding with one of those commits would still see `0.3.0`. In reality, the version should reflect that it's ahead of the release, for example `0.3.1-rc1`. -### How should I version a Bazel module? +- Non-registry override issues: using placeholder values can cause issues when users override a module with a non-registry override. For example, `0.0.0` doesn't sort as the highest version, which is usually the expected behavior users want when doing a non-registry override. + +Thus, it's best to avoid setting the version in the source archive `MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry (e.g., the [Bazel Central Registry](https://registry.bazel.build/)), which is the actual source of truth for the module version during Bazel's external dependency resolution (see [Bazel registries](https://bazel.build/external/registry)). + +This is usually automated, for example the [`rules-template`](https://github.com/bazel-contrib/rules-template) example rule repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml) to publish the release to the BCR. The action [generates a patch for the source archive `MODULE.bazel`](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216) with the release version. This patch is stored in the registry and is applied when the module is fetched during Bazel's external dependency resolution. -Setting `version` with the [`module`] directive in the source archive -`MODULE.bazel` can have several downsides and unintended side effects if not -managed carefully: - -* Duplication: releasing a new version of a module typically involves both - incrementing the version in `MODULE.bazel` and tagging the release, two - separate steps that can fall out of sync. While automation can - reduce this risk, it's simpler and safer to avoid it altogether. - -* Inconsistency: users overriding a module with a specific commit using a - [non-registry override] will see an incorrect version. for example, if the - `MODULE.bazel` in the source archive sets `version = "0.3.0"` but - additional commits have been made since that release, a user overriding - with one of those commits would still see `0.3.0`. In reality, the version - should reflect that it's ahead of the release, for example `0.3.1-rc1`. - -* Non-registry override issues: using placeholder values can cause issues - when users override a module with a non-registry override. For example, - `0.0.0` doesn't sort as the highest version, which is usually the expected - behavior users want when doing a non-registry override. - -Thus, it's best to avoid setting the version in the source archive -`MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry -(e.g., the [Bazel Central Registry]), which is the actual source of truth for -the module version during Bazel's external dependency resolution (see [Bazel -registries]). - -This is usually automated, for example the [`rules-template`] example rule -repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action] to -publish the release to the BCR. The action [generates a patch for the source -archive `MODULE.bazel`] with the release version. This patch is stored in the -registry and is applied when the module is fetched during Bazel's external -dependency resolution. - -This way, the version in the releases in the registry will be correctly set to -the released version and thus, `bazel_dep`, `single_version_override` and -`multiple_version_override` will work as expected, while avoiding potential -issues when doing a non-registry override because the version in the source -archive will be the default value (`''`), which will always be handled -correctly (it's the default version value after all) and will behave as -expected when sorting (the empty string is treated as the highest version). - -[Bazel Central Registry]: https://registry.bazel.build/ -[Bazel registries]: https://bazel.build/external/registry -[bazel-contrib/publish-to-bcr publish.yaml GitHub Action]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml -[generates a patch for the source archive `MODULE.bazel`]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216 -[`module`]: /rules/lib/globals/module#module -[non-registry override]: module.md#non-registry_overrides -[`rules-template`]: https://github.com/bazel-contrib/rules-template +This way, the version in the releases in the registry will be correctly set to the released version and thus, `bazel_dep`, `single_version_override` and `multiple_version_override` will work as expected, while avoiding potential issues when doing a non-registry override because the version in the source archive will be the default value (`''`), which will always be handled correctly (it's the default version value after all) and will behave as expected when sorting (the empty string is treated as the highest version). ### When should I increment the compatibility level? -The [`compatibility_level`](module.md#compatibility_level) of a Bazel module -should be incremented _in the same commit_ that introduces a backwards -incompatible ("breaking") change. +The [`compatibility_level`](module.md#compatibility_level) of a Bazel module should be incremented *in the same commit* that introduces a backwards incompatible ("breaking") change. -However, Bazel can throw an error if it detects that versions of the _same -module_ with _different compatibility levels_ exist in the resolved dependency -graph. This can happen when for example' two modules depend on versions of a -third module with different compatibility levels. +However, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. This can happen when for example' two modules depend on versions of a third module with different compatibility levels. -Thus, incrementing `compatibility_level` too frequently can be very disruptive -and is discouraged. To avoid this situation, the `compatibility_level` should be -incremented _only_ when the breaking change affects most use cases and isn't -easy to migrate and/or work-around. +Thus, incrementing `compatibility_level` too frequently can be very disruptive and is discouraged. To avoid this situation, the `compatibility_level` should be incremented *only* when the breaking change affects most use cases and isn't easy to migrate and/or work-around. ### Why does MODULE.bazel not support `load`s? -During dependency resolution, the MODULE.bazel file of all referenced external -dependencies are fetched from registries. At this stage, the source archives of -the dependencies are not fetched yet; so if the MODULE.bazel file `load`s -another file, there is no way for Bazel to actually fetch that file without -fetching the entire source archive. Note the MODULE.bazel file itself is -special, as it's directly hosted on the registry. - -There are a few use cases that people asking for `load`s in MODULE.bazel are -generally interested in, and they can be solved without `load`s: - -* Ensuring that the version listed in MODULE.bazel is consistent with build - metadata stored elsewhere, for example in a .bzl file: This can be achieved - by using the - [`native.module_version`](/rules/lib/toplevel/native#module_version) method - in a .bzl file loaded from a BUILD file. -* Splitting up a very large MODULE.bazel file into manageable sections, - particularly for monorepos: The root module can use the - [`include`](/rules/lib/globals/module#include) directive to split its - MODULE.bazel file into multiple segments. For the same reason we don't allow - `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. -* Users of the old WORKSPACE system might remember declaring a repo, and then - immediately `load`ing from that repo to perform complex logic. This - capability has been replaced by [module extensions](extension). +During dependency resolution, the MODULE.bazel file of all referenced external dependencies are fetched from registries. At this stage, the source archives of the dependencies are not fetched yet; so if the MODULE.bazel file `load`s another file, there is no way for Bazel to actually fetch that file without fetching the entire source archive. Note the MODULE.bazel file itself is special, as it's directly hosted on the registry. + +There are a few use cases that people asking for `load`s in MODULE.bazel are generally interested in, and they can be solved without `load`s: + +- Ensuring that the version listed in MODULE.bazel is consistent with build metadata stored elsewhere, for example in a .bzl file: This can be achieved by using the [`native.module_version`](/rules/lib/toplevel/native#module_version) method in a .bzl file loaded from a BUILD file. +- Splitting up a very large MODULE.bazel file into manageable sections, particularly for monorepos: The root module can use the [`include`](/rules/lib/globals/module#include) directive to split its MODULE.bazel file into multiple segments. For the same reason we don't allow `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. +- Users of the old WORKSPACE system might remember declaring a repo, and then immediately `load`ing from that repo to perform complex logic. This capability has been replaced by [module extensions](extension). ### Can I specify a SemVer range for a `bazel_dep`? -No. Some other package managers like [npm][npm-semver] and [Cargo][cargo-semver] -support version ranges (implicitly or explicitly), and this often requires a -constraint solver (making the output harder to predict for users) and makes -version resolution nonreproducible without a lockfile. +No. Some other package managers like [npm](https://docs.npmjs.com/about-semantic-versioning) and [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax) support version ranges (implicitly or explicitly), and this often requires a constraint solver (making the output harder to predict for users) and makes version resolution nonreproducible without a lockfile. -Bazel instead uses [Minimal Version Selection](module#version-selection) like -Go, which in contrast makes the output easy to predict and guarantees -reproducibility. This is a tradeoff that matches Bazel's design goals. +Bazel instead uses [Minimal Version Selection](module#version-selection) like Go, which in contrast makes the output easy to predict and guarantees reproducibility. This is a tradeoff that matches Bazel's design goals. -Furthermore, Bazel module versions are [a superset of -SemVer](module#version-format), so what makes sense in a strict SemVer -environment doesn't always carry over to Bazel module versions. +Furthermore, Bazel module versions are [a superset of SemVer](module#version-format), so what makes sense in a strict SemVer environment doesn't always carry over to Bazel module versions. ### Can I automatically get the latest version for a `bazel_dep`? -Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", -version = "latest")` to automatically get the latest version of a dep. This is -similar to [the question about SemVer -ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also -no. +Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", version = "latest")` to automatically get the latest version of a dep. This is similar to [the question about SemVer ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also no. -The recommended solution here is to have automation take care of this. For -example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports -Bazel modules. +The recommended solution here is to have automation take care of this. For example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports Bazel modules. -Sometimes, users asking this question are really looking for a way to quickly -iterate during local development. This can be achieved by using a -[`local_path_override`](/rules/lib/globals/module#local_path_override). +Sometimes, users asking this question are really looking for a way to quickly iterate during local development. This can be achieved by using a [`local_path_override`](/rules/lib/globals/module#local_path_override). ### Why all these `use_repo`s? -Module extension usages in MODULE.bazel files sometimes come with a big -`use_repo` directive. For example, a typical usage of the -[`go_deps` extension][go_deps] from `gazelle` might look like: +Module extension usages in MODULE.bazel files sometimes come with a big `use_repo` directive. For example, a typical usage of the [`go_deps` extension](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies) from `gazelle` might look like: ```python go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") @@ -155,175 +73,82 @@ use_repo( ) ``` -The long `use_repo` directive may seem redundant, since the information is -arguably already in the referenced `go.mod` file. +The long `use_repo` directive may seem redundant, since the information is arguably already in the referenced `go.mod` file. -The reason Bazel needs this `use_repo` directive is that it runs module -extensions lazily. That is, a module extension is only run if its result is -observed. Since a module extension's "output" is repo definitions, this means -that we only run a module extension if a repo it defines is requested (for -instance, if the target `@org_golang_x_net//:foo` is built, in the example -above). However, we don't know which repos a module extension would define until -after we run it. This is where the `use_repo` directive comes in; the user can -tell Bazel which repos they expect the extension to generate, and Bazel would -then only run the extension when these specific repos are used. +The reason Bazel needs this `use_repo` directive is that it runs module extensions lazily. That is, a module extension is only run if its result is observed. Since a module extension's "output" is repo definitions, this means that we only run a module extension if a repo it defines is requested (for instance, if the target `@org_golang_x_net//:foo` is built, in the example above). However, we don't know which repos a module extension would define until after we run it. This is where the `use_repo` directive comes in; the user can tell Bazel which repos they expect the extension to generate, and Bazel would then only run the extension when these specific repos are used. -To help the maintain this `use_repo` directive, a module extension can return -an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) -object from its implementation function. The user can run the `bazel mod tidy` -command to update the `use_repo` directives for these module extensions. +To help the maintain this `use_repo` directive, a module extension can return an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) object from its implementation function. The user can run the `bazel mod tidy` command to update the `use_repo` directives for these module extensions. ## Bzlmod migration ### Which is evaluated first, MODULE.bazel or WORKSPACE? -When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to -wonder which system is consulted first. The short answer is that MODULE.bazel -(Bzlmod) is evaluated first. - -The long answer is that "which evaluates first" is not the right question to -ask; rather, the right question to ask is: in the context of the repo with -[canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent -repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what -is the repo mapping of `@@base`? - -Labels with apparent repo names (a single leading `@`) can refer to different -things based on the context they're resolved from. When you see a label -`@bar//:baz` and wonder what it actually points to, you need to first find out -what the context repo is: for example, if the label is in a BUILD file located -in the repo `@@foo`, then the context repo is `@@foo`. - -Then, depending on what the context repo is, the ["repository -visibility" table](migration#repository-visibility) in the migration guide can -be used to find out which repo an apparent name actually resolves to. - -* If the context repo is the main repo (`@@`): - 1. If `bar` is an apparent repo name introduced by the root module's - MODULE.bazel file (through any of - [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), - [`use_repo`](/rules/lib/globals/module#use_repo), - [`module`](/rules/lib/globals/module#module.repo_name), - [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` - resolves to what that MODULE.bazel file claims. - 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its - canonical name is `@@bar`), then `@bar` resolves to `@@bar`. - 3. Otherwise, `@bar` resolves to something like - `@@[unknown repo 'bar' requested from @@]`, and this will ultimately - result in an error. -* If the context repo is a Bzlmod-world repo (that is, it corresponds to a - non-root Bazel module, or is generated by a module extension), then it - will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. - * Notably, this includes any repos introduced in a `non_module_deps`-like - module extension in the root module, or `use_repo_rule` instantiations - in the root module. -* If the context repo is defined in WORKSPACE: - 1. First, check if the context repo definition has the magical - `repo_mapping` attribute. If so, go through the mapping first (so for a - repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking - at `@baz` below). - 2. If `bar` is an apparent repo name introduced by the root module's - MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file - claims. (This is the same as item 1 in the main repo case.) - 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a - repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel - will throw an error. +When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to wonder which system is consulted first. The short answer is that MODULE.bazel (Bzlmod) is evaluated first. + +The long answer is that "which evaluates first" is not the right question to ask; rather, the right question to ask is: in the context of the repo with [canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what is the repo mapping of `@@base`? + +Labels with apparent repo names (a single leading `@`) can refer to different things based on the context they're resolved from. When you see a label `@bar//:baz` and wonder what it actually points to, you need to first find out what the context repo is: for example, if the label is in a BUILD file located in the repo `@@foo`, then the context repo is `@@foo`. + +Then, depending on what the context repo is, the ["repository visibility" table](migration#repository-visibility) in the migration guide can be used to find out which repo an apparent name actually resolves to. + +- If the context repo is the main repo (`@@`): + + 1. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file (through any of [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), [`use_repo`](/rules/lib/globals/module#use_repo), [`module`](/rules/lib/globals/module#module.repo_name), [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` resolves to what that MODULE.bazel file claims. + 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its canonical name is `@@bar`), then `@bar` resolves to `@@bar`. + 3. Otherwise, `@bar` resolves to something like `@@[unknown repo 'bar' requested from @@]`, and this will ultimately result in an error. + +- If the context repo is a Bzlmod-world repo (that is, it corresponds to a non-root Bazel module, or is generated by a module extension), then it will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. + - Notably, this includes any repos introduced in a `non_module_deps`-like module extension in the root module, or `use_repo_rule` instantiations in the root module. + +- If the context repo is defined in WORKSPACE: + + 1. First, check if the context repo definition has the magical `repo_mapping` attribute. If so, go through the mapping first (so for a repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking at `@baz` below). + 2. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file claims. (This is the same as item 1 in the main repo case.) + 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel will throw an error. For a more succinct version: -* Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world - repos. -* WORKSPACE-world repos (including the main repo) will first see what the root - module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world - repos. +- Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world repos. +- WORKSPACE-world repos (including the main repo) will first see what the root module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world repos. -Of note, labels in the Bazel command line (including Starlark flags, label-typed -flag values, and build/test target patterns) are treated as having the main repo -as the context repo. +Of note, labels in the Bazel command line (including Starlark flags, label-typed flag values, and build/test target patterns) are treated as having the main repo as the context repo. ## Other ### How do I prepare and run an offline build? -Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag -(like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the -context of the main repo, see [question -above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target -pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of -`@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). +Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag (like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the context of the main repo, see [question above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of `@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). -The make sure no fetches happen during a build, use `--nofetch`. More precisely, -this makes any attempt to run a non-local repository rule fail. +The make sure no fetches happen during a build, use `--nofetch`. More precisely, this makes any attempt to run a non-local repository rule fail. -If you want to fetch repos _and_ modify them to test locally, consider using -the [`bazel vendor`](vendor) command. +If you want to fetch repos *and* modify them to test locally, consider using the [`bazel vendor`](vendor) command. ### How do I use HTTP proxies? -Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly -accepted by other programs, such as -[curl](https://everything.curl.dev/usingcurl/proxies/env.html). +Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly accepted by other programs, such as [curl](https://everything.curl.dev/usingcurl/proxies/env.html). ### How do I make Bazel prefer IPv6 in dual-stack IPv4/IPv6 setups? -On IPv6-only machines, Bazel can download dependencies with no changes. However, -on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, -preferring IPv4 if enabled. In some situations, for example when the IPv4 -network cannot resolve/reach external addresses, this can cause `Network -unreachable` exceptions and build failures. In these cases, you can override -Bazel's behavior to prefer IPv6 by using the -[`java.net.preferIPv6Addresses=true` system -property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). -Specifically: - -* Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup - option](/docs/user-manual#startup-options), for example by adding the - following line in your [`.bazelrc` file](/run/bazelrc): - - `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` - -* When running Java build targets that need to connect to the internet (such - as for integration tests), use the - `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool - flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` - file](/run/bazelrc): - - `build --jvmopt=-Djava.net.preferIPv6Addresses` - -* If you are using - [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for - dependency version resolution, also add - `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment - variable to [provide JVM options for - Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). +On IPv6-only machines, Bazel can download dependencies with no changes. However, on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, preferring IPv4 if enabled. In some situations, for example when the IPv4 network cannot resolve/reach external addresses, this can cause `Network unreachable` exceptions and build failures. In these cases, you can override Bazel's behavior to prefer IPv6 by using the [`java.net.preferIPv6Addresses=true` system property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). Specifically: -### Can repo rules be run remotely with remote execution? +- Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup option](/docs/user-manual#startup-options), for example by adding the following line in your [`.bazelrc` file](/run/bazelrc): + + `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` + +- When running Java build targets that need to connect to the internet (such as for integration tests), use the `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` file](/run/bazelrc): -No; or at least, not yet. Users employing remote execution services to speed up -their builds may notice that repo rules are still run locally. For example, an -`http_archive` would be first downloaded onto the local machine (using any local -download cache if applicable), extracted, and then each source file would be -uploaded to the remote execution service as an input file. It's natural to ask -why the remote execution service doesn't just download and extract that archive, -saving a useless roundtrip. + `build --jvmopt=-Djava.net.preferIPv6Addresses` -Part of the reason is that repo rules (and module extensions) are akin to -"scripts" that are run by Bazel itself. A remote executor doesn't necessarily -even have a Bazel installed. +- If you are using [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for dependency version resolution, also add `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment variable to [provide JVM options for Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). + +### Can repo rules be run remotely with remote execution? -Another reason is that Bazel often needs the BUILD files in the downloaded and -extracted archives to perform loading and analysis, which _are_ performed -locally. +No; or at least, not yet. Users employing remote execution services to speed up their builds may notice that repo rules are still run locally. For example, an `http_archive` would be first downloaded onto the local machine (using any local download cache if applicable), extracted, and then each source file would be uploaded to the remote execution service as an input file. It's natural to ask why the remote execution service doesn't just download and extract that archive, saving a useless roundtrip. -There are preliminary ideas to solve this problem by re-imagining repo rules as -build rules, which would naturally allow them to be run remotely, but conversely -raise new architectural concerns (for example, the `query` commands would -potentially need to run actions, complicating their design). +Part of the reason is that repo rules (and module extensions) are akin to "scripts" that are run by Bazel itself. A remote executor doesn't necessarily even have a Bazel installed. -For more previous discussion on this topic, see [A way to support repositories -that need Bazel for being -fetched](https://github.com/bazelbuild/bazel/discussions/20464). +Another reason is that Bazel often needs the BUILD files in the downloaded and extracted archives to perform loading and analysis, which *are* performed locally. -[npm-semver]: https://docs.npmjs.com/about-semantic-versioning -[cargo-semver]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax -[go_deps]: https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies +There are preliminary ideas to solve this problem by re-imagining repo rules as build rules, which would naturally allow them to be run remotely, but conversely raise new architectural concerns (for example, the `query` commands would potentially need to run actions, complicating their design). +For more previous discussion on this topic, see [A way to support repositories that need Bazel for being fetched](https://github.com/bazelbuild/bazel/discussions/20464). diff --git a/external/lockfile.mdx b/external/lockfile.mdx index 9f312615..a20423c3 100644 --- a/external/lockfile.mdx +++ b/external/lockfile.mdx @@ -1,98 +1,46 @@ -keywords: product:Bazel,lockfile,Bzlmod --- title: 'Bazel Lockfile' --- - - -The lockfile feature in Bazel enables the recording of specific versions or -dependencies of software libraries or packages required by a project. It -achieves this by storing the result of module resolution and extension -evaluation. The lockfile promotes reproducible builds, ensuring consistent -development environments. Additionally, it enhances build efficiency by allowing -Bazel to skip the parts of the resolution process that are unaffected by changes -in project dependencies. Furthermore, the lockfile improves stability by -preventing unexpected updates or breaking changes in external libraries, thereby -reducing the risk of introducing bugs. +The lockfile feature in Bazel enables the recording of specific versions or dependencies of software libraries or packages required by a project. It achieves this by storing the result of module resolution and extension evaluation. The lockfile promotes reproducible builds, ensuring consistent development environments. Additionally, it enhances build efficiency by allowing Bazel to skip the parts of the resolution process that are unaffected by changes in project dependencies. Furthermore, the lockfile improves stability by preventing unexpected updates or breaking changes in external libraries, thereby reducing the risk of introducing bugs. ## Lockfile Generation -The lockfile is generated under the workspace root with the name -`MODULE.bazel.lock`. It is created or updated during the build process, -specifically after module resolution and extension evaluation. Importantly, it -only includes dependencies that are included in the current invocation of the -build. +The lockfile is generated under the workspace root with the name `MODULE.bazel.lock`. It is created or updated during the build process, specifically after module resolution and extension evaluation. Importantly, it only includes dependencies that are included in the current invocation of the build. -When changes occur in the project that affect its dependencies, the lockfile is -automatically updated to reflect the new state. This ensures that the lockfile -remains focused on the specific set of dependencies required for the current -build, providing an accurate representation of the project's resolved -dependencies. +When changes occur in the project that affect its dependencies, the lockfile is automatically updated to reflect the new state. This ensures that the lockfile remains focused on the specific set of dependencies required for the current build, providing an accurate representation of the project's resolved dependencies. ## Lockfile Usage -The lockfile can be controlled by the flag -[`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to -customize the behavior of Bazel when the project state differs from the -lockfile. The available modes are: - -* `update` (Default): Use the information that is present in the lockfile to - skip downloads of known registry files and to avoid re-evaluating extensions - whose results are still up-to-date. If information is missing, it will - be added to the lockfile. In this mode, Bazel also avoids refreshing - mutable information, such as yanked versions, for dependencies that haven't - changed. -* `refresh`: Like `update`, but mutable information is always refreshed when - switching to this mode and roughly every hour while in this mode. -* `error`: Like `update`, but if any information is missing or out-of-date, - Bazel will fail with an error. This mode never changes the lockfile or - performs network requests during resolution. Module extensions that marked - themselves as `reproducible` may still perform network requests, but are - expected to always produce the same result. -* `off`: The lockfile is neither checked nor updated. +The lockfile can be controlled by the flag [`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to customize the behavior of Bazel when the project state differs from the lockfile. The available modes are: + +- `update` (Default): Use the information that is present in the lockfile to skip downloads of known registry files and to avoid re-evaluating extensions whose results are still up-to-date. If information is missing, it will be added to the lockfile. In this mode, Bazel also avoids refreshing mutable information, such as yanked versions, for dependencies that haven't changed. +- `refresh`: Like `update`, but mutable information is always refreshed when switching to this mode and roughly every hour while in this mode. +- `error`: Like `update`, but if any information is missing or out-of-date, Bazel will fail with an error. This mode never changes the lockfile or performs network requests during resolution. Module extensions that marked themselves as `reproducible` may still perform network requests, but are expected to always produce the same result. +- `off`: The lockfile is neither checked nor updated. ## Lockfile Benefits The lockfile offers several benefits and can be utilized in various ways: -- **Reproducible builds.** By capturing the specific versions or dependencies - of software libraries, the lockfile ensures that builds are reproducible - across different environments and over time. Developers can rely on - consistent and predictable results when building their projects. +- **Reproducible builds.** By capturing the specific versions or dependencies of software libraries, the lockfile ensures that builds are reproducible across different environments and over time. Developers can rely on consistent and predictable results when building their projects. -- **Fast incremental resolutions.** The lockfile enables Bazel to avoid - downloading registry files that were already used in a previous build. - This significantly improves build efficiency, especially in scenarios where - resolution can be time-consuming. +- **Fast incremental resolutions.** The lockfile enables Bazel to avoid downloading registry files that were already used in a previous build. This significantly improves build efficiency, especially in scenarios where resolution can be time-consuming. -- **Stability and risk reduction.** The lockfile helps maintain stability by - preventing unexpected updates or breaking changes in external libraries. By - locking the dependencies to specific versions, the risk of introducing bugs - due to incompatible or untested updates is reduced. +- **Stability and risk reduction.** The lockfile helps maintain stability by preventing unexpected updates or breaking changes in external libraries. By locking the dependencies to specific versions, the risk of introducing bugs due to incompatible or untested updates is reduced. ### Hidden lockfile -Bazel also maintains another lockfile at -`"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this -lockfile are explicitly unspecified. It is only used as a performance -optimization. While it can be deleted together with the output base via -`bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a -module extension. +Bazel also maintains another lockfile at `"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this lockfile are explicitly unspecified. It is only used as a performance optimization. While it can be deleted together with the output base via `bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a module extension. ## Lockfile Contents -The lockfile contains all the necessary information to determine whether the -project state has changed. It also includes the result of building the project -in the current state. The lockfile consists of two main parts: +The lockfile contains all the necessary information to determine whether the project state has changed. It also includes the result of building the project in the current state. The lockfile consists of two main parts: -1. Hashes of all remote files that are inputs to module resolution. -2. For each module extension, the lockfile includes inputs that affect it, - represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as - well as the output of running that extension, referred to as - `generatedRepoSpecs` +1. Hashes of all remote files that are inputs to module resolution. +2. For each module extension, the lockfile includes inputs that affect it, represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as well as the output of running that extension, referred to as `generatedRepoSpecs` -Here is an example that demonstrates the structure of the lockfile, along with -explanations for each section: +Here is an example that demonstrates the structure of the lockfile, along with explanations for each section: ```json { @@ -152,106 +100,53 @@ explanations for each section: ### Registry File Hashes -The `registryFileHashes` section contains the hashes of all files from -remote registries accessed during module resolution. Since the resolution -algorithm is fully deterministic when given the same inputs and all remote -inputs are hashed, this ensures a fully reproducible resolution result while -avoiding excessive duplication of remote information in the lockfile. Note that -this also requires recording when a particular registry didn't contain a certain -module, but a registry with lower precedence did (see the "not found" entry in -the example). This inherently mutable information can be updated via -`bazel mod deps --lockfile_mode=refresh`. +The `registryFileHashes` section contains the hashes of all files from remote registries accessed during module resolution. Since the resolution algorithm is fully deterministic when given the same inputs and all remote inputs are hashed, this ensures a fully reproducible resolution result while avoiding excessive duplication of remote information in the lockfile. Note that this also requires recording when a particular registry didn't contain a certain module, but a registry with lower precedence did (see the "not found" entry in the example). This inherently mutable information can be updated via `bazel mod deps --lockfile_mode=refresh`. -Bazel uses the hashes from the lockfile to look up registry files in the -repository cache before downloading them, which speeds up subsequent -resolutions. +Bazel uses the hashes from the lockfile to look up registry files in the repository cache before downloading them, which speeds up subsequent resolutions. ### Selected Yanked Versions -The `selectedYankedVersions` section contains the yanked versions of modules -that were selected by module resolution. Since this usually result in an error -when trying to build, this section is only non-empty when yanked versions are -explicitly allowed via `--allow_yanked_versions` or -`BZLMOD_ALLOW_YANKED_VERSIONS`. +The `selectedYankedVersions` section contains the yanked versions of modules that were selected by module resolution. Since this usually result in an error when trying to build, this section is only non-empty when yanked versions are explicitly allowed via `--allow_yanked_versions` or `BZLMOD_ALLOW_YANKED_VERSIONS`. -This field is needed since, compared to module files, yanked version information -is inherently mutable and thus can't be referenced by a hash. This information -can be updated via `bazel mod deps --lockfile_mode=refresh`. +This field is needed since, compared to module files, yanked version information is inherently mutable and thus can't be referenced by a hash. This information can be updated via `bazel mod deps --lockfile_mode=refresh`. ### Module Extensions -The `moduleExtensions` section is a map that includes only the extensions used -in the current invocation or previously invoked, while excluding any extensions -that are no longer utilized. In other words, if an extension is not being used -anymore across the dependency graph, it is removed from the `moduleExtensions` -map. - -If an extension is independent of the operating system or architecture type, -this section features only a single "general" entry. Otherwise, multiple -entries are included, named after the OS, architecture, or both, with each -corresponding to the result of evaluating the extension on those specifics. - -Each entry in the extension map corresponds to a used extension and is -identified by its containing file and name. The corresponding value for each -entry contains the relevant information associated with that extension: - -1. The `bzlTransitiveDigest` is the digest of the extension implementation - and the .bzl files transitively loaded by it. -2. The `usagesDigest` is the digest of the _usages_ of the extension in the - dependency graph, which includes all tags. -3. Further unspecified fields that track other inputs to the extension, - such as contents of files or directories it reads or environment - variables it uses. -4. The `generatedRepoSpecs` encode the repositories created by the - extension with the current input. -5. The optional `moduleExtensionMetadata` field contains metadata provided by - the extension such as whether certain repositories it created should be - imported via `use_repo` by the root module. This information powers the - `bazel mod tidy` command. - -Module extensions can opt out of being included in the lockfile by setting the -returning metadata with `reproducible = True`. By doing so, they promise that -they will always create the same repositories when given the same inputs. +The `moduleExtensions` section is a map that includes only the extensions used in the current invocation or previously invoked, while excluding any extensions that are no longer utilized. In other words, if an extension is not being used anymore across the dependency graph, it is removed from the `moduleExtensions` map. + +If an extension is independent of the operating system or architecture type, this section features only a single "general" entry. Otherwise, multiple entries are included, named after the OS, architecture, or both, with each corresponding to the result of evaluating the extension on those specifics. + +Each entry in the extension map corresponds to a used extension and is identified by its containing file and name. The corresponding value for each entry contains the relevant information associated with that extension: + +1. The `bzlTransitiveDigest` is the digest of the extension implementation and the .bzl files transitively loaded by it. +2. The `usagesDigest` is the digest of the *usages* of the extension in the dependency graph, which includes all tags. +3. Further unspecified fields that track other inputs to the extension, such as contents of files or directories it reads or environment variables it uses. +4. The `generatedRepoSpecs` encode the repositories created by the extension with the current input. +5. The optional `moduleExtensionMetadata` field contains metadata provided by the extension such as whether certain repositories it created should be imported via `use_repo` by the root module. This information powers the `bazel mod tidy` command. + +Module extensions can opt out of being included in the lockfile by setting the returning metadata with `reproducible = True`. By doing so, they promise that they will always create the same repositories when given the same inputs. ## Best Practices -To maximize the benefits of the lockfile feature, consider the following best -practices: +To maximize the benefits of the lockfile feature, consider the following best practices: -* Regularly update the lockfile to reflect changes in project dependencies or - configuration. This ensures that subsequent builds are based on the most - up-to-date and accurate set of dependencies. To lock down all extensions - at once, run `bazel mod deps --lockfile_mode=update`. +- Regularly update the lockfile to reflect changes in project dependencies or configuration. This ensures that subsequent builds are based on the most up-to-date and accurate set of dependencies. To lock down all extensions at once, run `bazel mod deps --lockfile_mode=update`. -* Include the lockfile in version control to facilitate collaboration and - ensure that all team members have access to the same lockfile, promoting - consistent development environments across the project. +- Include the lockfile in version control to facilitate collaboration and ensure that all team members have access to the same lockfile, promoting consistent development environments across the project. -* Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a - `.bazelversion` file in version control that specifies the Bazel version - corresponding to the lockfile. Because Bazel itself is a dependency of - your build, the lockfile is specific to the Bazel version, and will - change even between [backwards compatible](/release/backward-compatibility) - Bazel releases. Using `bazelisk` ensures that all developers are using - a Bazel version that matches the lockfile. +- Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a `.bazelversion` file in version control that specifies the Bazel version corresponding to the lockfile. Because Bazel itself is a dependency of your build, the lockfile is specific to the Bazel version, and will change even between [backwards compatible](/release/backward-compatibility) Bazel releases. Using `bazelisk` ensures that all developers are using a Bazel version that matches the lockfile. -By following these best practices, you can effectively utilize the lockfile -feature in Bazel, leading to more efficient, reliable, and collaborative -software development workflows. +By following these best practices, you can effectively utilize the lockfile feature in Bazel, leading to more efficient, reliable, and collaborative software development workflows. ## Merge Conflicts -The lockfile format is designed to minimize merge conflicts, but they can still -happen. +The lockfile format is designed to minimize merge conflicts, but they can still happen. ### Automatic Resolution -Bazel provides a custom -[git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) -to help resolve these conflicts automatically. +Bazel provides a custom [git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) to help resolve these conflicts automatically. -Set up the driver by adding this line to a `.gitattributes` file in the root of -your git repository: +Set up the driver by adding this line to a `.gitattributes` file in the root of your git repository: ```gitattributes # A custom merge driver for the Bazel lockfile. @@ -259,8 +154,7 @@ your git repository: MODULE.bazel.lock merge=bazel-lockfile-merge ``` -Then each developer who wants to use the driver has to register it once by -following these steps: +Then each developer who wants to use the driver has to register it once by following these steps: 1. Install [jq](https://jqlang.github.io/jq/download/) (1.5 or higher). 2. Run the following commands: @@ -269,18 +163,15 @@ following these steps: jq_script=$(curl https://raw.githubusercontent.com/bazelbuild/bazel/master/scripts/bazel-lockfile-merge.jq) printf '%s\n' "${jq_script}" | less # to optionally inspect the jq script git config --global merge.bazel-lockfile-merge.name "Merge driver for the Bazel lockfile (MODULE.bazel.lock)" -git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" +git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" ``` ### Manual Resolution -Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` -fields can be safely resolved by keeping all the entries from both sides of the -conflict. +Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` fields can be safely resolved by keeping all the entries from both sides of the conflict. Other types of merge conflicts should not be resolved manually. Instead: -1. Restore the previous state of the lockfile - via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. +1. Restore the previous state of the lockfile via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. 2. Resolve any conflicts in the `MODULE.bazel` file. 3. Run `bazel mod deps` to update the lockfile. diff --git a/external/migration.mdx b/external/migration.mdx index 8e70b1f3..49ba462f 100644 --- a/external/migration.mdx +++ b/external/migration.mdx @@ -1,267 +1,200 @@ -keywords: bzlmod --- title: 'Bzlmod Migration Guide' --- - - -Due to the [shortcomings of -WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the -legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late -2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate -your project to Bzlmod and drop WORKSPACE for managing external dependencies. +Due to the [shortcomings of WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate your project to Bzlmod and drop WORKSPACE for managing external dependencies. ## Why migrate to Bzlmod? -* There are many [advantages](overview#benefits) compared to the legacy - WORKSPACE system, which helps to ensure a healthy growth of the Bazel - ecosystem. +- There are many [advantages](overview#benefits) compared to the legacy WORKSPACE system, which helps to ensure a healthy growth of the Bazel ecosystem. -* If your project is a dependency of other projects, migrating to Bzlmod will - unblock their migration and make it easier for them to depend on your - project. +- If your project is a dependency of other projects, migrating to Bzlmod will unblock their migration and make it easier for them to depend on your project. -* Migration to Bzlmod is a necessary step in order to use future Bazel - versions (mandatory in Bazel 9). +- Migration to Bzlmod is a necessary step in order to use future Bazel versions (mandatory in Bazel 9). ## How to migrate to Bzlmod? Recommended migration process: -1. Use [migration tool](/external/migration_tool) as a helper tool to - streamline the migration process as much as possible. -2. If there are errors left after using the migration tool, resolve them - manually. For understanding the main differences between concepts inside - `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus - Bzlmod](#workspace-vs-bzlmod) section. +1. Use [migration tool](/external/migration_tool) as a helper tool to streamline the migration process as much as possible. +2. If there are errors left after using the migration tool, resolve them manually. For understanding the main differences between concepts inside `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus Bzlmod](#workspace-vs-bzlmod) section. ## WORKSPACE vs Bzlmod -Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This -section explains how to migrate from specific WORKSPACE functionalities to -Bzlmod. +Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This section explains how to migrate from specific WORKSPACE functionalities to Bzlmod. ### Define the root of a Bazel workspace -The WORKSPACE file marks the source root of a Bazel project, this responsibility -is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions -prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at -your workspace root, maybe with comments like: +The WORKSPACE file marks the source root of a Bazel project, this responsibility is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at your workspace root, maybe with comments like: -* **WORKSPACE** +- **WORKSPACE** - ```python - # This file marks the root of the Bazel workspace. - # See MODULE.bazel for external dependencies setup. - ``` + ```python + # This file marks the root of the Bazel workspace. + # See MODULE.bazel for external dependencies setup. + ``` ### Enable Bzlmod in your bazelrc -`.bazelrc` lets you set flags that apply every time your run Bazel. To enable -Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so -it applies to every command: +`.bazelrc` lets you set flags that apply every time your run Bazel. To enable Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so it applies to every command: -* **.bazelrc** +- **.bazelrc** - ``` - # Enable Bzlmod for every Bazel command - common --enable_bzlmod - ``` + ``` + # Enable Bzlmod for every Bazel command + common --enable_bzlmod + ``` ### Specify repository name for your workspace -* **WORKSPACE** +- **WORKSPACE** - The [`workspace`](/rules/lib/globals/workspace#workspace) function is used - to specify a repository name for your workspace. This allows a target - `//foo:bar` in the workspace to be referenced as `@//foo:bar`. If not specified, the default repository name for your - workspace is `__main__`. + The [`workspace`](/rules/lib/globals/workspace#workspace) function is used to specify a repository name for your workspace. This allows a target `//foo:bar` in the workspace to be referenced as `@<workspace name>//foo:bar`. If not specified, the default repository name for your workspace is `__main__`. - ```python - ## WORKSPACE - workspace(name = "com_foo_bar") - ``` + ```python + ## WORKSPACE + workspace(name = "com_foo_bar") + ``` -* **Bzlmod** +- **Bzlmod** - It's recommended to reference targets in the same workspace with the - `//foo:bar` syntax without `@`. But if you do need the old syntax - , you can use the module name specified by the - [`module`](/rules/lib/globals/module#module) function as the repository - name. If the module name is different from the needed repository name, you - can use `repo_name` attribute of the - [`module`](/rules/lib/globals/module#module) function to override the - repository name. + It's recommended to reference targets in the same workspace with the `//foo:bar` syntax without `@<repo name>`. But if you do need the old syntax , you can use the module name specified by the [`module`](/rules/lib/globals/module#module) function as the repository name. If the module name is different from the needed repository name, you can use `repo_name` attribute of the [`module`](/rules/lib/globals/module#module) function to override the repository name. - ```python - ## MODULE.bazel - module( - name = "bar", - repo_name = "com_foo_bar", - ) - ``` + ```python + ## MODULE.bazel + module( + name = "bar", + repo_name = "com_foo_bar", + ) + ``` ### Fetch external dependencies as Bazel modules -If your dependency is a Bazel project, you should be able to depend on it as a -Bazel module when it also adopts Bzlmod. +If your dependency is a Bazel project, you should be able to depend on it as a Bazel module when it also adopts Bzlmod. -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, it's common to use the - [`http_archive`](/rules/lib/repo/http#http_archive) or - [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to - download the sources of the Bazel project. + With WORKSPACE, it's common to use the [`http_archive`](/rules/lib/repo/http#http_archive) or [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to download the sources of the Bazel project. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "bazel_skylib", - urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], - sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", - ) - load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") - bazel_skylib_workspace() + http_archive( + name = "bazel_skylib", + urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], + sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", + ) + load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") + bazel_skylib_workspace() - http_archive( - name = "rules_java", - urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], - sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", - ) - load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") - rules_java_dependencies() - rules_java_toolchains() - ``` + http_archive( + name = "rules_java", + urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], + sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", + ) + load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") + rules_java_dependencies() + rules_java_toolchains() + ``` - As you can see, it's a common pattern that users need to load transitive - dependencies from a macro of the dependency. Assume both `bazel_skylib` and - `rules_java` depends on `platform`, the exact version of the `platform` - dependency is determined by the order of the macros. + As you can see, it's a common pattern that users need to load transitive dependencies from a macro of the dependency. Assume both `bazel_skylib` and `rules_java` depends on `platform`, the exact version of the `platform` dependency is determined by the order of the macros. -* **Bzlmod** +- **Bzlmod** - With Bzlmod, as long as your dependency is available in [Bazel Central - Registry](https://registry.bazel.build) or your custom [Bazel - registry](/external/registry), you can simply depend on it with a - [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. + With Bzlmod, as long as your dependency is available in [Bazel Central Registry](https://registry.bazel.build) or your custom [Bazel registry](/external/registry), you can simply depend on it with a [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. - ```python - ## MODULE.bazel - bazel_dep(name = "bazel_skylib", version = "1.4.2") - bazel_dep(name = "rules_java", version = "6.1.1") - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "bazel_skylib", version = "1.4.2") + bazel_dep(name = "rules_java", version = "6.1.1") + ``` - Bzlmod resolves Bazel module dependencies transitively using the - [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal - required version of `platform` is selected automatically. + Bzlmod resolves Bazel module dependencies transitively using the [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal required version of `platform` is selected automatically. ### Override a dependency as a Bazel module -As the root module, you can override Bazel module dependencies in different -ways. +As the root module, you can override Bazel module dependencies in different ways. -Please read the [overrides](/external/module#overrides) section for more -information. +Please read the [overrides](/external/module#overrides) section for more information. -You can find some example usages in the -[examples][override-examples] -repository. - -[override-examples]: https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module +You can find some example usages in the [examples](https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module) repository. ### Fetch external dependencies with module extensions -If your dependency is not a Bazel project or not yet available in any Bazel -registry, you can introduce it using -[`use_repo_rule`](/external/module#use_repo_rule) or [module -extensions](/external/extension). +If your dependency is not a Bazel project or not yet available in any Bazel registry, you can introduce it using [`use_repo_rule`](/external/module#use_repo_rule) or [module extensions](/external/extension). -* **WORKSPACE** +- **WORKSPACE** - Download a file using the [`http_file`](/rules/lib/repo/http#http_file) - repository rule. + Download a file using the [`http_file`](/rules/lib/repo/http#http_file) repository rule. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` -* **Bzlmod** +- **Bzlmod** - With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel - file to directly instantiate repos: + With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel file to directly instantiate repos: - ```python - ## MODULE.bazel - http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## MODULE.bazel + http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Under the hood, this is implemented using a module extension. If you need to - perform more complex logic than simply invoking a repo rule, you could also - implement a module extension yourself. You'll need to move the definition - into a `.bzl` file, which also lets you share the definition between - WORKSPACE and Bzlmod during the migration period. + Under the hood, this is implemented using a module extension. If you need to perform more complex logic than simply invoking a repo rule, you could also implement a module extension yourself. You'll need to move the definition into a `.bzl` file, which also lets you share the definition between WORKSPACE and Bzlmod during the migration period. - ```python - ## repositories.bzl - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - def my_data_dependency(): - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## repositories.bzl + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + def my_data_dependency(): + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Implement a module extension to load the dependencies macro. You can define - it in the same `.bzl` file of the macro, but to keep compatibility with - older Bazel versions, it's better to define it in a separate `.bzl` file. + Implement a module extension to load the dependencies macro. You can define it in the same `.bzl` file of the macro, but to keep compatibility with older Bazel versions, it's better to define it in a separate `.bzl` file. - ```python - ## extensions.bzl - load("//:repositories.bzl", "my_data_dependency") - def _non_module_dependencies_impl(_ctx): - my_data_dependency() + ```python + ## extensions.bzl + load("//:repositories.bzl", "my_data_dependency") + def _non_module_dependencies_impl(_ctx): + my_data_dependency() - non_module_dependencies = module_extension( - implementation = _non_module_dependencies_impl, - ) - ``` + non_module_dependencies = module_extension( + implementation = _non_module_dependencies_impl, + ) + ``` - To make the repository visible to the root project, you should declare the - usages of the module extension and the repository in the MODULE.bazel file. + To make the repository visible to the root project, you should declare the usages of the module extension and the repository in the MODULE.bazel file. - ```python - ## MODULE.bazel - non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") - use_repo(non_module_dependencies, "data_file") - ``` + ```python + ## MODULE.bazel + non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") + use_repo(non_module_dependencies, "data_file") + ``` ### Resolve conflict external dependencies with module extension -A project can provide a macro that introduces external repositories based on -inputs from its callers. But what if there are multiple callers in the -dependency graph and they cause a conflict? +A project can provide a macro that introduces external repositories based on inputs from its callers. But what if there are multiple callers in the dependency graph and they cause a conflict? -Assume the project `foo` provides the following macro which takes `version` as -an argument. +Assume the project `foo` provides the following macro which takes `version` as an argument. ```python -## repositories.bzl in foo {:#repositories.bzl-foo} +## repositories.bzl in foo load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") def data_deps(version = "1.0"): http_file( @@ -271,297 +204,239 @@ def data_deps(version = "1.0"): ) ``` -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, you can load the macro from `@foo` and specify the version - of the data dependency you need. Assume you have another dependency `@bar`, - which also depends on `@foo` but requires a different version of the data - dependency. + With WORKSPACE, you can load the macro from `@foo` and specify the version of the data dependency you need. Assume you have another dependency `@bar`, which also depends on `@foo` but requires a different version of the data dependency. - ```python - ## WORKSPACE + ```python + ## WORKSPACE - # Introduce @foo and @bar. - ... + # Introduce @foo and @bar. + ... - load("@foo//:repositories.bzl", "data_deps") - data_deps(version = "2.0") + load("@foo//:repositories.bzl", "data_deps") + data_deps(version = "2.0") - load("@bar//:repositories.bzl", "bar_deps") - bar_deps() # -> which calls data_deps(version = "3.0") - ``` + load("@bar//:repositories.bzl", "bar_deps") + bar_deps() # -> which calls data_deps(version = "3.0") + ``` - In this case, the end user must carefully adjust the order of macros in the - WORKSPACE to get the version they need. This is one of the biggest pain - points with WORKSPACE since it doesn't really provide a sensible way to - resolve dependencies. + In this case, the end user must carefully adjust the order of macros in the WORKSPACE to get the version they need. This is one of the biggest pain points with WORKSPACE since it doesn't really provide a sensible way to resolve dependencies. -* **Bzlmod** +- **Bzlmod** - With Bzlmod, the author of project `foo` can use module extension to resolve - conflicts. For example, let's assume it makes sense to always select the - maximal required version of the data dependency among all Bazel modules. + With Bzlmod, the author of project `foo` can use module extension to resolve conflicts. For example, let's assume it makes sense to always select the maximal required version of the data dependency among all Bazel modules. - ```python - ## extensions.bzl in foo - load("//:repositories.bzl", "data_deps") - - data = tag_class(attrs={"version": attr.string()}) - - def _data_deps_extension_impl(module_ctx): - # Select the maximal required version in the dependency graph. - version = "1.0" - for mod in module_ctx.modules: - for data in mod.tags.data: - version = max(version, data.version) - data_deps(version) - - data_deps_extension = module_extension( - implementation = _data_deps_extension_impl, - tag_classes = {"data": data}, - ) - ``` + ```python + ## extensions.bzl in foo + load("//:repositories.bzl", "data_deps") - ```python - ## MODULE.bazel in bar - bazel_dep(name = "foo", version = "1.0") + data = tag_class(attrs={"version": attr.string()}) - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "3.0") - use_repo(foo_data_deps, "data_file") - ``` + def _data_deps_extension_impl(module_ctx): + # Select the maximal required version in the dependency graph. + version = "1.0" + for mod in module_ctx.modules: + for data in mod.tags.data: + version = max(version, data.version) + data_deps(version) - ```python - ## MODULE.bazel in root module - bazel_dep(name = "foo", version = "1.0") - bazel_dep(name = "bar", version = "1.0") + data_deps_extension = module_extension( + implementation = _data_deps_extension_impl, + tag_classes = {"data": data}, + ) + ``` - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "2.0") - use_repo(foo_data_deps, "data_file") - ``` + ```python + ## MODULE.bazel in bar + bazel_dep(name = "foo", version = "1.0") - In this case, the root module requires data version `2.0`, while its - dependency `bar` requires `3.0`. The module extension in `foo` can correctly - resolve this conflict and automatically select version `3.0` for the data - dependency. + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "3.0") + use_repo(foo_data_deps, "data_file") + ``` -### Integrate third party package manager + ```python + ## MODULE.bazel in root module + bazel_dep(name = "foo", version = "1.0") + bazel_dep(name = "bar", version = "1.0") + + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "2.0") + use_repo(foo_data_deps, "data_file") + ``` + + In this case, the root module requires data version `2.0`, while its dependency `bar` requires `3.0`. The module extension in `foo` can correctly resolve this conflict and automatically select version `3.0` for the data dependency. -Following the last section, since module extension provides a way to collect -information from the dependency graph, perform custom logic to resolve -dependencies and call repository rules to introduce external repositories, this -provides a great way for rules authors to enhance the rulesets that integrate -package managers for specific languages. +### Integrate third party package manager -Please read the [module extensions](/external/extension) page to learn more -about how to use module extensions. +Following the last section, since module extension provides a way to collect information from the dependency graph, perform custom logic to resolve dependencies and call repository rules to introduce external repositories, this provides a great way for rules authors to enhance the rulesets that integrate package managers for specific languages. -Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies -from different package managers: +Please read the [module extensions](/external/extension) page to learn more about how to use module extensions. -- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) -- [rules_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) -- [rules_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) +Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies from different package managers: -A minimal example that integrates a pseudo package manager is available at the -[examples][pkg-mgr-example] -repository. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) +- [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) +- [rules\_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) -[pkg-mgr-example]: https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager +A minimal example that integrates a pseudo package manager is available at the [examples](https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager) repository. ### Detect toolchains on the host machine -When Bazel build rules need to detect what toolchains are available on your host -machine, they use repository rules to inspect the host machine and generate -toolchain info as external repositories. +When Bazel build rules need to detect what toolchains are available on your host machine, they use repository rules to inspect the host machine and generate toolchain info as external repositories. -* **WORKSPACE** +- **WORKSPACE** - Given the following repository rule to detect a shell toolchain. + Given the following repository rule to detect a shell toolchain. - ```python - ## local_config_sh.bzl - def _sh_config_rule_impl(repository_ctx): - sh_path = get_sh_path_from_env("SH_BIN_PATH") - - if not sh_path: - sh_path = detect_sh_from_path() - - if not sh_path: - sh_path = "/shell/binary/not/found" - - repository_ctx.file("BUILD", """ - load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") - sh_toolchain( - name = "local_sh", - path = "{sh_path}", - visibility = ["//visibility:public"], - ) - toolchain( - name = "local_sh_toolchain", - toolchain = ":local_sh", - toolchain_type = "@bazel_tools//tools/sh:toolchain_type", - ) - """.format(sh_path = sh_path)) + ```python + ## local_config_sh.bzl + def _sh_config_rule_impl(repository_ctx): + sh_path = get_sh_path_from_env("SH_BIN_PATH") - sh_config_rule = repository_rule( - environ = ["SH_BIN_PATH"], - local = True, - implementation = _sh_config_rule_impl, - ) - ``` + if not sh_path: + sh_path = detect_sh_from_path() - You can load the repository rule in WORKSPACE. + if not sh_path: + sh_path = "/shell/binary/not/found" - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - ``` + repository_ctx.file("BUILD", """ + load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") + sh_toolchain( + name = "local_sh", + path = "{sh_path}", + visibility = ["//visibility:public"], + ) + toolchain( + name = "local_sh_toolchain", + toolchain = ":local_sh", + toolchain_type = "@bazel_tools//tools/sh:toolchain_type", + ) + """.format(sh_path = sh_path)) -* **Bzlmod** + sh_config_rule = repository_rule( + environ = ["SH_BIN_PATH"], + local = True, + implementation = _sh_config_rule_impl, + ) + ``` - With Bzlmod, you can introduce the same repository using a module extension, - which is similar to introducing the `@data_file` repository in the last - section. + You can load the repository rule in WORKSPACE. - ``` - ## local_config_sh_extension.bzl - load("//:local_config_sh.bzl", "sh_config_rule") + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + ``` - sh_config_extension = module_extension( - implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), - ) - ``` +- **Bzlmod** - Then use the extension in the MODULE.bazel file. + With Bzlmod, you can introduce the same repository using a module extension, which is similar to introducing the `@data_file` repository in the last section. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - ``` + ``` + ## local_config_sh_extension.bzl + load("//:local_config_sh.bzl", "sh_config_rule") -### Register toolchains & execution platforms + sh_config_extension = module_extension( + implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), + ) + ``` -Following the last section, after introducing a repository hosting toolchain -information (e.g. `local_config_sh`), you probably want to register the -toolchain. + Then use the extension in the MODULE.bazel file. -* **WORKSPACE** + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + ``` - With WORKSPACE, you can register the toolchain in the following ways. +### Register toolchains & execution platforms - 1. You can register the toolchain the `.bzl` file and load the macro in the - WORKSPACE file. +Following the last section, after introducing a repository hosting toolchain information (e.g. `local_config_sh`), you probably want to register the toolchain. - ```python - ## local_config_sh.bzl - def sh_configure(): - sh_config_rule(name = "local_config_sh") - native.register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +- **WORKSPACE** - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_configure") - sh_configure() - ``` + With WORKSPACE, you can register the toolchain in the following ways. - 2. Or register the toolchain in the WORKSPACE file directly. + 1. You can register the toolchain the `.bzl` file and load the macro in the WORKSPACE file. - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## local_config_sh.bzl + def sh_configure(): + sh_config_rule(name = "local_config_sh") + native.register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` -* **Bzlmod** + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_configure") + sh_configure() + ``` - With Bzlmod, the - [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and - [`register_execution_platforms`][register_execution_platforms] - APIs are only available in the MODULE.bazel file. You cannot call - `native.register_toolchains` in a module extension. + 2. Or register the toolchain in the WORKSPACE file directly. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` + +- **Bzlmod** + + With Bzlmod, the [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) APIs are only available in the MODULE.bazel file. You cannot call `native.register_toolchains` in a module extension. -The toolchains and execution platforms registered in `WORKSPACE`, -`WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this -order of precedence during toolchain selection (from highest to lowest): + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` -1. toolchains and execution platforms registered in the root module's - `MODULE.bazel` file. -2. toolchains and execution platforms registered in the `WORKSPACE` or - `WORKSPACE.bzlmod` file. -3. toolchains and execution platforms registered by modules that are - (transitive) dependencies of the root module. -4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` - [suffix](/external/migration#builtin-default-deps). +The toolchains and execution platforms registered in `WORKSPACE`, `WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this order of precedence during toolchain selection (from highest to lowest): -[register_execution_platforms]: /rules/lib/globals/module#register_execution_platforms +1. toolchains and execution platforms registered in the root module's `MODULE.bazel` file. +2. toolchains and execution platforms registered in the `WORKSPACE` or `WORKSPACE.bzlmod` file. +3. toolchains and execution platforms registered by modules that are (transitive) dependencies of the root module. +4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` [suffix](/external/migration#builtin-default-deps). ### Introduce local repositories -You may need to introduce a dependency as a local repository when you need a -local version of the dependency for debugging or you want to incorporate a -directory in your workspace as external repository. +You may need to introduce a dependency as a local repository when you need a local version of the dependency for debugging or you want to incorporate a directory in your workspace as external repository. -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, this is achieved by two native repository rules, - [`local_repository`](/reference/be/workspace#local_repository) and - [`new_local_repository`](/reference/be/workspace#new_local_repository). + With WORKSPACE, this is achieved by two native repository rules, [`local_repository`](/reference/be/workspace#local_repository) and [`new_local_repository`](/reference/be/workspace#new_local_repository). - ```python - ## WORKSPACE - local_repository( - name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## WORKSPACE + local_repository( + name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` -* **Bzlmod** +- **Bzlmod** - With Bzlmod, you can use - [`local_path_override`](/rules/lib/globals/module#local_path_override) to - override a module with a local path. + With Bzlmod, you can use [`local_path_override`](/rules/lib/globals/module#local_path_override) to override a module with a local path. - ```python - ## MODULE.bazel - bazel_dep(name = "rules_java") - local_path_override( - module_name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "rules_java") + local_path_override( + module_name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` - Note: With `local_path_override`, you can only introduce a local directory - as a Bazel module, which means it should have a MODULE.bazel file and its - transitive dependencies are taken into consideration during dependency - resolution. In addition, all module override directives can only be used by - the root module. + Note: With `local_path_override`, you can only introduce a local directory as a Bazel module, which means it should have a MODULE.bazel file and its transitive dependencies are taken into consideration during dependency resolution. In addition, all module override directives can only be used by the root module. - It is also possible to introduce a local repository with module extension. - However, you cannot call `native.local_repository` in module extension, - there is ongoing effort on starlarkifying all native repository rules (check - [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). - Then you can call the corresponding starlark `local_repository` in a module - extension. It's also trivial to implement a custom version of - `local_repository` repository rule if this is a blocking issue for you. + It is also possible to introduce a local repository with module extension. However, you cannot call `native.local_repository` in module extension, there is ongoing effort on starlarkifying all native repository rules (check [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). Then you can call the corresponding starlark `local_repository` in a module extension. It's also trivial to implement a custom version of `local_repository` repository rule if this is a blocking issue for you. ### Bind targets -The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and -not supported in Bzlmod. It was introduced to give a target an alias in the -special `//external` package. All users depending on this should migrate away. +The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and not supported in Bzlmod. It was introduced to give a target an alias in the special `//external` package. All users depending on this should migrate away. For example, if you have @@ -573,290 +448,180 @@ bind( ) ``` -This allows other targets to depend on `//external:openssl`. You can migrate -away from this by: +This allows other targets to depend on `//external:openssl`. You can migrate away from this by: -* Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. - * Tip: Use `bazel query --output=build --noenable_bzlmod - --enable_workspace [target]` command to find relevant info - about the target. +- Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. -* Or use the [`alias`](/reference/be/general#alias) build rule - * Define the following target in a package (e.g. `//third_party`) + - Tip: Use `bazel query --output=build --noenable_bzlmod --enable_workspace [target]` command to find relevant info about the target. - ```python - ## third_party/BUILD - alias( - name = "openssl", - actual = "@my-ssl//src:openssl-lib", - ) - ``` +- Or use the [`alias`](/reference/be/general#alias) build rule + + - Define the following target in a package (e.g. `//third_party`) + + ```python + ## third_party/BUILD + alias( + name = "openssl", + actual = "@my-ssl//src:openssl-lib", + ) + ``` - * Replace all usages of `//external:openssl` with `//third_party:openssl`. + - Replace all usages of `//external:openssl` with `//third_party:openssl`. ### Fetch versus Sync -Fetch and sync commands are used to download external repos locally and keep -them updated. Sometimes also to allow building offline using the `--nofetch` -flag after fetching all repos needed for a build. +Fetch and sync commands are used to download external repos locally and keep them updated. Sometimes also to allow building offline using the `--nofetch` flag after fetching all repos needed for a build. -* **WORKSPACE** +- **WORKSPACE** - Sync performs a force fetch for all repositories, or for a specific - configured set of repos, while fetch is _only_ used to fetch for a specific - target. + Sync performs a force fetch for all repositories, or for a specific configured set of repos, while fetch is *only* used to fetch for a specific target. -* **Bzlmod** +- **Bzlmod** - The sync command is no longer applicable, but fetch offers - [various options](/reference/command-line-reference#fetch-options). - You can fetch a target, a repository, a set of configured repos or all - repositories involved in your dependency resolution and module extensions. - The fetch result is cached and to force a fetch you must include the - `--force` option during the fetch process. + The sync command is no longer applicable, but fetch offers [various options](/reference/command-line-reference#fetch-options). You can fetch a target, a repository, a set of configured repos or all repositories involved in your dependency resolution and module extensions. The fetch result is cached and to force a fetch you must include the `--force` option during the fetch process. ## Manual migration -This section provides useful information and guidance for your **manual** Bzlmod -migration process. For more automatized migration process, check [recommended -migration process](#how-migrate-to-bzlmod) section. +This section provides useful information and guidance for your **manual** Bzlmod migration process. For more automatized migration process, check [recommended migration process](#how-migrate-to-bzlmod) section. ### Know your dependencies in WORKSPACE -The first step of migration is to understand what dependencies you have. It -could be hard to figure out what exact dependencies are introduced in the -WORKSPACE file because transitive dependencies are often loaded with `*_deps` -macros. +The first step of migration is to understand what dependencies you have. It could be hard to figure out what exact dependencies are introduced in the WORKSPACE file because transitive dependencies are often loaded with `*_deps` macros. #### Inspect external dependency with workspace resolved file -Fortunately, the flag -[`--experimental_repository_resolved_file`][resolved_file_flag] -can help. This flag essentially generates a "lock file" of all fetched external -dependencies in your last Bazel command. You can find more details in this [blog -post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). - -[resolved_file_flag]: /reference/command-line-reference#flag--experimental_repository_resolved_file +Fortunately, the flag [`--experimental_repository_resolved_file`](/reference/command-line-reference#flag--experimental_repository_resolved_file) can help. This flag essentially generates a "lock file" of all fetched external dependencies in your last Bazel command. You can find more details in this [blog post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). It can be used in two ways: -1. To fetch info of external dependencies needed for building certain targets. +1. To fetch info of external dependencies needed for building certain targets. - ```shell - bazel clean --expunge - bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar - ``` + ```shell + bazel clean --expunge + bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar + ``` -2. To fetch info of all external dependencies defined in the WORKSPACE file. +2. To fetch info of all external dependencies defined in the WORKSPACE file. - ```shell - bazel clean --expunge - bazel sync --experimental_repository_resolved_file=resolved.bzl - ``` + ```shell + bazel clean --expunge + bazel sync --experimental_repository_resolved_file=resolved.bzl + ``` - With the `bazel sync` command, you can fetch all dependencies defined in the - WORKSPACE file, which include: + With the `bazel sync` command, you can fetch all dependencies defined in the WORKSPACE file, which include: - * `bind` usages - * `register_toolchains` & `register_execution_platforms` usages + - `bind` usages + - `register_toolchains` & `register_execution_platforms` usages - However, if your project is cross platforms, bazel sync may break on certain - platforms because some repository rules may only run correctly on supported - platforms. + However, if your project is cross platforms, bazel sync may break on certain platforms because some repository rules may only run correctly on supported platforms. -After running the command, you should have information of your external -dependencies in the `resolved.bzl` file. +After running the command, you should have information of your external dependencies in the `resolved.bzl` file. #### Inspect external dependency with `bazel query` You may also know `bazel query` can be used for inspecting repository rules with ```shell -bazel query --output=build //external: +bazel query --output=build //external:<repo name> ``` -While it is more convenient and much faster, [bazel query can lie about -external dependency version](https://github.com/bazelbuild/bazel/issues/12947), -so be careful using it! Querying and inspecting external -dependencies with Bzlmod is going to achieved by a [new -subcommand](https://github.com/bazelbuild/bazel/issues/15365). +While it is more convenient and much faster, [bazel query can lie about external dependency version](https://github.com/bazelbuild/bazel/issues/12947), so be careful using it! Querying and inspecting external dependencies with Bzlmod is going to achieved by a [new subcommand](https://github.com/bazelbuild/bazel/issues/15365). #### Built-in default dependencies -If you check the file generated by `--experimental_repository_resolved_file`, -you are going to find many dependencies that are not defined in your WORKSPACE. -This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE -file content to inject some default dependencies, which are usually required by -native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With -Bzlmod, those dependencies are introduced with a built-in module -[`bazel_tools`][bazel_tools] , which is a default dependency for every other -Bazel module. - -[bazel_tools]: https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools +If you check the file generated by `--experimental_repository_resolved_file`, you are going to find many dependencies that are not defined in your WORKSPACE. This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE file content to inject some default dependencies, which are usually required by native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With Bzlmod, those dependencies are introduced with a built-in module [`bazel_tools`](https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools) , which is a default dependency for every other Bazel module. ### Hybrid mode for gradual migration -Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies -from the WORKSPACE file to Bzlmod to be a gradual process. +Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies from the WORKSPACE file to Bzlmod to be a gradual process. -Note: In practice, loading "*_deps" macros in WORKSPACE often causes confusions -with Bzlmod dependencies, therefore we recommend starting with a -WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. +Note: In practice, loading "\*\_deps" macros in WORKSPACE often causes confusions with Bzlmod dependencies, therefore we recommend starting with a WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. #### WORKSPACE.bzlmod -During the migration, Bazel users may need to switch between builds with and -without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the -process smoother. +During the migration, Bazel users may need to switch between builds with and without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the process smoother. -WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, -if a WORKSPACE.bzlmod file also exists at the workspace root: +WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, if a WORKSPACE.bzlmod file also exists at the workspace root: -* `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. -* No [prefixes or suffixes](/external/migration#builtin-default-deps) are - added to the WORKSPACE.bzlmod file. +- `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. +- No [prefixes or suffixes](/external/migration#builtin-default-deps) are added to the WORKSPACE.bzlmod file. Using the WORKSPACE.bzlmod file can make the migration easier because: -* When Bzlmod is disabled, you fall back to fetching dependencies from the - original WORKSPACE file. -* When Bzlmod is enabled, you can better track what dependencies are left to - migrate with WORKSPACE.bzlmod. +- When Bzlmod is disabled, you fall back to fetching dependencies from the original WORKSPACE file. +- When Bzlmod is enabled, you can better track what dependencies are left to migrate with WORKSPACE.bzlmod. #### Repository visibility -Bzlmod is able to control which other repositories are visible from a given -repository, check [repository names and strict -deps](/external/module#repository_names_and_strict_deps) for more details. +Bzlmod is able to control which other repositories are visible from a given repository, check [repository names and strict deps](/external/module#repository_names_and_strict_deps) for more details. -Here is a summary of repository visibilities from different types of -repositories when also taking WORKSPACE into consideration. +Here is a summary of repository visibilities from different types of repositories when also taking WORKSPACE into consideration. -| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | -|----------------|--------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------|----------------------| -| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | -| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | -| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | -| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | +| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | +| ---------------------- | ------------------ | ----------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------ | +| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | +| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | +| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | +| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | -Note: For the root module, if a repository `@foo` is defined in WORKSPACE and -`@foo` is also used as an [apparent repository -name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` -refers to the one introduced in MODULE.bazel. +Note: For the root module, if a repository `@foo` is defined in WORKSPACE and `@foo` is also used as an [apparent repository name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` refers to the one introduced in MODULE.bazel. -Note: For a module extension generated repository `@bar`, if `@foo` is used as -an [apparent repository name](/external/overview#apparent-repo-name) of -another repository generated by the same module extension and direct -dependencies of the module hosting the module extension, then for repository -`@bar`, `@foo` refers to the latter. +Note: For a module extension generated repository `@bar`, if `@foo` is used as an [apparent repository name](/external/overview#apparent-repo-name) of another repository generated by the same module extension and direct dependencies of the module hosting the module extension, then for repository `@bar`, `@foo` refers to the latter. ### Manual migration process A typical Bzlmod migration process can look like this: -1. Understand what dependencies you have in WORKSPACE. -1. Add an empty MODULE.bazel file at your project root. -1. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. -1. Build your targets with Bzlmod enabled and check which repository is - missing. -1. Check the definition of the missing repository in the resolved dependency - file. -1. Introduce the missing dependency as a Bazel module, through a module - extension, or leave it in the WORKSPACE.bzlmod for later migration. -1. Go back to 4 and repeat until all dependencies are available. +1. Understand what dependencies you have in WORKSPACE. +2. Add an empty MODULE.bazel file at your project root. +3. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. +4. Build your targets with Bzlmod enabled and check which repository is missing. +5. Check the definition of the missing repository in the resolved dependency file. +6. Introduce the missing dependency as a Bazel module, through a module extension, or leave it in the WORKSPACE.bzlmod for later migration. +7. Go back to 4 and repeat until all dependencies are available. ## Publish Bazel modules -If your Bazel project is a dependency for other projects, you can publish your -project in the [Bazel Central Registry](https://registry.bazel.build/). - -To be able to check in your project in the BCR, you need a source archive URL of -the project. Take note of a few things when creating the source archive: +If your Bazel project is a dependency for other projects, you can publish your project in the [Bazel Central Registry](https://registry.bazel.build/). -* **Make sure the archive is pointing to a specific version.** +To be able to check in your project in the BCR, you need a source archive URL of the project. Take note of a few things when creating the source archive: - The BCR can only accept versioned source archives because Bzlmod needs to - conduct version comparison during dependency resolution. +- **Make sure the archive is pointing to a specific version.** -* **Make sure the archive URL is stable.** + The BCR can only accept versioned source archives because Bzlmod needs to conduct version comparison during dependency resolution. - Bazel verifies the content of the archive by a hash value, so you should - make sure the checksum of the downloaded file never changes. If the URL is - from GitHub, please create and upload a release archive in the release page. - GitHub isn't going to guarantee the checksum of source archives generated on - demand. In short, URLs in the form of - `https://github.com///releases/download/...` is considered stable - while `https://github.com///archive/...` is not. Check [GitHub - Archive Checksum - Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) - for more context. +- **Make sure the archive URL is stable.** -* **Make sure the source tree follows the layout of the original repository.** + Bazel verifies the content of the archive by a hash value, so you should make sure the checksum of the downloaded file never changes. If the URL is from GitHub, please create and upload a release archive in the release page. GitHub isn't going to guarantee the checksum of source archives generated on demand. In short, URLs in the form of `https://github.com/<org>/<repo>/releases/download/...` is considered stable while `https://github.com/<org>/<repo>/archive/...` is not. Check [GitHub Archive Checksum Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) for more context. - In case your repository is very large and you want to create a distribution - archive with reduced size by stripping out unnecessary sources, please make - sure the stripped source tree is a subset of the original source tree. This - makes it easier for end users to override the module to a non-release - version by [`archive_override`](/rules/lib/globals/module#archive_override) - and [`git_override`](/rules/lib/globals/module#git_override). +- **Make sure the source tree follows the layout of the original repository.** -* **Include a test module in a subdirectory that tests your most common - APIs.** + In case your repository is very large and you want to create a distribution archive with reduced size by stripping out unnecessary sources, please make sure the stripped source tree is a subset of the original source tree. This makes it easier for end users to override the module to a non-release version by [`archive_override`](/rules/lib/globals/module#archive_override) and [`git_override`](/rules/lib/globals/module#git_override). - A test module is a Bazel project with its own WORKSPACE and MODULE.bazel - file located in a subdirectory of the source archive which depends on the - actual module to be published. It should contain examples or some - integration tests that cover your most common APIs. Check - [test module][test_module] to learn how to set it up. +- **Include a test module in a subdirectory that tests your most common APIs.** -[test_module]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module + A test module is a Bazel project with its own WORKSPACE and MODULE.bazel file located in a subdirectory of the source archive which depends on the actual module to be published. It should contain examples or some integration tests that cover your most common APIs. Check [test module](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module) to learn how to set it up. -When you have your source archive URL ready, follow the [BCR contribution -guidelines][bcr_contrib_guide] to submit your module to the BCR with a GitHub -Pull Request. +When you have your source archive URL ready, follow the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module) to submit your module to the BCR with a GitHub Pull Request. -[bcr_contrib_guide]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module - -It is **highly recommended** to set up the [Publish to -BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your -repository to automate the process of submitting your module to the BCR. +It is **highly recommended** to set up the [Publish to BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your repository to automate the process of submitting your module to the BCR. ## Best practices -This section documents a few best practices you should follow for better -managing your external dependencies. +This section documents a few best practices you should follow for better managing your external dependencies. #### Split targets into different packages to avoid fetching unnecessary dependencies. -Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev -dependencies for tests are forced to be fetched unnecessarily for building -targets that don't need them. This is actually not Bzlmod specific, but -following this practices makes it easier to specify dev dependencies correctly. +Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev dependencies for tests are forced to be fetched unnecessarily for building targets that don't need them. This is actually not Bzlmod specific, but following this practices makes it easier to specify dev dependencies correctly. #### Specify dev dependencies -You can set the `dev_dependency` attribute to true for -[`bazel_dep`](/rules/lib/globals/module#bazel_dep) and -[`use_extension`](/rules/lib/globals/module#use_extension) directives so that -they don't propagate to dependent projects. As the root module, you can use the -[`--ignore_dev_dependency`][ignore_dev_dep_flag] flag to verify if your targets -still build without dev dependencies and overrides. - -[ignore_dev_dep_flag]: /reference/command-line-reference#flag--ignore_dev_dependency - - +You can set the `dev_dependency` attribute to true for [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_extension`](/rules/lib/globals/module#use_extension) directives so that they don't propagate to dependent projects. As the root module, you can use the [`--ignore_dev_dependency`](/reference/command-line-reference#flag--ignore_dev_dependency) flag to verify if your targets still build without dev dependencies and overrides. ## Community migration progress -You can check the [Bazel Central Registry](https://registry.bazel.build) to find -out if your dependencies are already available. Otherwise feel free to join this -[GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to -upvote or post the dependencies that are blocking your migration. +You can check the [Bazel Central Registry](https://registry.bazel.build) to find out if your dependencies are already available. Otherwise feel free to join this [GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to upvote or post the dependencies that are blocking your migration. ## Report issues -Please check the [Bazel GitHub issue list][bzlmod_github_issue] for known Bzlmod -issues. Feel free to file new issues or feature requests that can help unblock -your migration! - -[bzlmod_github_issue]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod +Please check the [Bazel GitHub issue list](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod) for known Bzlmod issues. Feel free to file new issues or feature requests that can help unblock your migration! diff --git a/external/module.mdx b/external/module.mdx index 0499c4cd..9ac802a4 100644 --- a/external/module.mdx +++ b/external/module.mdx @@ -2,16 +2,9 @@ title: 'Bazel modules' --- +A Bazel **module** is a Bazel project that can have multiple versions, each of which publishes metadata about other modules that it depends on. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. - -A Bazel **module** is a Bazel project that can have multiple versions, each of -which publishes metadata about other modules that it depends on. This is -analogous to familiar concepts in other dependency management systems, such as a -Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. - -A module must have a `MODULE.bazel` file at its repo root. This file is the -module's manifest, declaring its name, version, list of direct dependencies, and -other information. For a basic example: +A module must have a `MODULE.bazel` file at its repo root. This file is the module's manifest, declaring its name, version, list of direct dependencies, and other information. For a basic example: ```python module(name = "my-module", version = "1.0") @@ -20,50 +13,29 @@ bazel_dep(name = "rules_cc", version = "0.0.1") bazel_dep(name = "protobuf", version = "3.19.0") ``` -See the [full list](/rules/lib/globals/module) of directives available in -`MODULE.bazel` files. +See the [full list](/rules/lib/globals/module) of directives available in `MODULE.bazel` files. -To perform module resolution, Bazel starts by reading the root module's -`MODULE.bazel` file, and then repeatedly requests any dependency's -`MODULE.bazel` file from a [Bazel registry](/external/registry) until it -discovers the entire dependency graph. +To perform module resolution, Bazel starts by reading the root module's `MODULE.bazel` file, and then repeatedly requests any dependency's `MODULE.bazel` file from a [Bazel registry](/external/registry) until it discovers the entire dependency graph. -By default, Bazel then [selects](#version-selection) one version of each module -to use. Bazel represents each module with a repo, and consults the registry -again to learn how to define each of the repos. +By default, Bazel then [selects](#version-selection) one version of each module to use. Bazel represents each module with a repo, and consults the registry again to learn how to define each of the repos. ## Version format -Bazel has a diverse ecosystem and projects use various versioning schemes. The -most popular by far is [SemVer](https://semver.org), but there are -also prominent projects using different schemes such as -[Abseil](https://github.com/abseil/abseil-cpp/releases), whose -versions are date-based, for example `20210324.2`). +Bazel has a diverse ecosystem and projects use various versioning schemes. The most popular by far is [SemVer](https://semver.org), but there are also prominent projects using different schemes such as [Abseil](https://github.com/abseil/abseil-cpp/releases), whose versions are date-based, for example `20210324.2`). -For this reason, Bazel adopts a more relaxed version of the SemVer spec. The -differences include: +For this reason, Bazel adopts a more relaxed version of the SemVer spec. The differences include: -* SemVer prescribes that the "release" part of the version must consist of 3 - segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so - that any number of segments is allowed. -* In SemVer, each of the segments in the "release" part must be digits only. - In Bazel, this is loosened to allow letters too, and the comparison - semantics match the "identifiers" in the "prerelease" part. -* Additionally, the semantics of major, minor, and patch version increases are - not enforced. However, see [compatibility level](#compatibility_level) for - details on how we denote backwards compatibility. +- SemVer prescribes that the "release" part of the version must consist of 3 segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so that any number of segments is allowed. +- In SemVer, each of the segments in the "release" part must be digits only. In Bazel, this is loosened to allow letters too, and the comparison semantics match the "identifiers" in the "prerelease" part. +- Additionally, the semantics of major, minor, and patch version increases are not enforced. However, see [compatibility level](#compatibility_level) for details on how we denote backwards compatibility. -Any valid SemVer version is a valid Bazel module version. Additionally, two -SemVer versions `a` and `b` compare `a < b` if and only if the same holds when -they're compared as Bazel module versions. +Any valid SemVer version is a valid Bazel module version. Additionally, two SemVer versions `a` and `b` compare `a < b` if and only if the same holds when they're compared as Bazel module versions. -Finally, to learn more about module versioning, [see the `MODULE.bazel` -FAQ](faq#module-versioning-best-practices). +Finally, to learn more about module versioning, [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). ## Version selection -Consider the diamond dependency problem, a staple in the versioned dependency -management space. Suppose you have the dependency graph: +Consider the diamond dependency problem, a staple in the versioned dependency management space. Suppose you have the dependency graph: ``` A 1.0 @@ -73,164 +45,78 @@ management space. Suppose you have the dependency graph: D 1.0 D 1.1 ``` -Which version of `D` should be used? To resolve this question, Bazel uses the -[Minimal Version Selection](https://research.swtch.com/vgo-mvs) -(MVS) algorithm introduced in the Go module system. MVS assumes that all new -versions of a module are backwards compatible, and so picks the highest version -specified by any dependent (`D 1.1` in our example). It's called "minimal" -because `D 1.1` is the earliest version that could satisfy our requirements — -even if `D 1.2` or newer exists, we don't select them. Using MVS creates a -version selection process that is *high-fidelity* and *reproducible*. +Which version of `D` should be used? To resolve this question, Bazel uses the [Minimal Version Selection](https://research.swtch.com/vgo-mvs) (MVS) algorithm introduced in the Go module system. MVS assumes that all new versions of a module are backwards compatible, and so picks the highest version specified by any dependent (`D 1.1` in our example). It's called "minimal" because `D 1.1` is the earliest version that could satisfy our requirements — even if `D 1.2` or newer exists, we don't select them. Using MVS creates a version selection process that is *high-fidelity* and *reproducible*. ### Yanked versions -The registry can declare certain versions as *yanked* if they should be avoided -(such as for security vulnerabilities). Bazel throws an error when selecting a -yanked version of a module. To fix this error, either upgrade to a newer, -non-yanked version, or use the -[`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) -flag to explicitly allow the yanked version. +The registry can declare certain versions as *yanked* if they should be avoided (such as for security vulnerabilities). Bazel throws an error when selecting a yanked version of a module. To fix this error, either upgrade to a newer, non-yanked version, or use the [`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) flag to explicitly allow the yanked version. ## Compatibility level -In Go, MVS's assumption about backwards compatibility works because it treats -backwards incompatible versions of a module as a separate module. In terms of -SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can -coexist in the resolved dependency graph. This is, in turn, made possible by -encoding the major version in the package path in Go, so there aren't any -compile-time or linking-time conflicts. Bazel, however, cannot provide such -guarantees because it follows [a relaxed version of SemVer](#version-format). - -Thus, Bazel needs the equivalent of the SemVer major version number to detect -backwards incompatible ("breaking") versions. This number is called the -*compatibility level*, and is specified by each module version in its -[`module()`](/rule/lib/globals/module#module) directive. With this information, -Bazel can throw an error if it detects that versions of the _same module_ with -_different compatibility levels_ exist in the resolved dependency graph. - -Finally, incrementing the compatibility level can be disruptive to the users. -To learn more about when and how to increment it, [check the `MODULE.bazel` -FAQ](faq#incrementing-compatibility-level). +In Go, MVS's assumption about backwards compatibility works because it treats backwards incompatible versions of a module as a separate module. In terms of SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can coexist in the resolved dependency graph. This is, in turn, made possible by encoding the major version in the package path in Go, so there aren't any compile-time or linking-time conflicts. Bazel, however, cannot provide such guarantees because it follows [a relaxed version of SemVer](#version-format). + +Thus, Bazel needs the equivalent of the SemVer major version number to detect backwards incompatible ("breaking") versions. This number is called the *compatibility level*, and is specified by each module version in its [`module()`](/rule/lib/globals/module#module) directive. With this information, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. + +Finally, incrementing the compatibility level can be disruptive to the users. To learn more about when and how to increment it, [check the `MODULE.bazel` FAQ](faq#incrementing-compatibility-level). ## Overrides -Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel -module resolution. Only the root module's overrides take effect — if a module is -used as a dependency, its overrides are ignored. +Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel module resolution. Only the root module's overrides take effect — if a module is used as a dependency, its overrides are ignored. -Each override is specified for a certain module name, affecting all of its -versions in the dependency graph. Although only the root module's overrides take -effect, they can be for transitive dependencies that the root module does not -directly depend on. +Each override is specified for a certain module name, affecting all of its versions in the dependency graph. Although only the root module's overrides take effect, they can be for transitive dependencies that the root module does not directly depend on. ### Single-version override -The [`single_version_override`](/rules/lib/globals/module#single_version_override) -serves multiple purposes: +The [`single_version_override`](/rules/lib/globals/module#single_version_override) serves multiple purposes: -* With the `version` attribute, you can pin a dependency to a specific - version, regardless of which versions of the dependency are requested in the - dependency graph. -* With the `registry` attribute, you can force this dependency to come from a - specific registry, instead of following the normal [registry - selection](/external/registry#selecting_registries) process. -* With the `patch*` attributes, you can specify a set of patches to apply to - the downloaded module. +- With the `version` attribute, you can pin a dependency to a specific version, regardless of which versions of the dependency are requested in the dependency graph. +- With the `registry` attribute, you can force this dependency to come from a specific registry, instead of following the normal [registry selection](/external/registry#selecting_registries) process. +- With the `patch*` attributes, you can specify a set of patches to apply to the downloaded module. These attributes are all optional and can be mixed and matched with each other. ### Multiple-version override -A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) -can be specified to allow multiple versions of the same module to coexist in the -resolved dependency graph. - -You can specify an explicit list of allowed versions for the module, which must -all be present in the dependency graph before resolution — there must exist -*some* transitive dependency depending on each allowed version. After -resolution, only the allowed versions of the module remain, while Bazel upgrades -other versions of the module to the nearest higher allowed version at the same -compatibility level. If no higher allowed version at the same compatibility -level exists, Bazel throws an error. - -For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the -dependency graph before resolution and the major version is the compatibility -level: - -* A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in - `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other - versions remaining the same. -* A multiple-version override allowing `1.5` and `2.0` results in an error, as - `1.7` has no higher version at the same compatibility level to upgrade to. -* A multiple-version override allowing `1.9` and `2.0` results in an error, as - `1.9` is not present in the dependency graph before resolution. - -Additionally, users can also override the registry using the `registry` -attribute, similarly to single-version overrides. +A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) can be specified to allow multiple versions of the same module to coexist in the resolved dependency graph. + +You can specify an explicit list of allowed versions for the module, which must all be present in the dependency graph before resolution — there must exist *some* transitive dependency depending on each allowed version. After resolution, only the allowed versions of the module remain, while Bazel upgrades other versions of the module to the nearest higher allowed version at the same compatibility level. If no higher allowed version at the same compatibility level exists, Bazel throws an error. + +For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the dependency graph before resolution and the major version is the compatibility level: + +- A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other versions remaining the same. +- A multiple-version override allowing `1.5` and `2.0` results in an error, as `1.7` has no higher version at the same compatibility level to upgrade to. +- A multiple-version override allowing `1.9` and `2.0` results in an error, as `1.9` is not present in the dependency graph before resolution. + +Additionally, users can also override the registry using the `registry` attribute, similarly to single-version overrides. ### Non-registry overrides -Non-registry overrides completely remove a module from version resolution. Bazel -does not request these `MODULE.bazel` files from a registry, but instead from -the repo itself. +Non-registry overrides completely remove a module from version resolution. Bazel does not request these `MODULE.bazel` files from a registry, but instead from the repo itself. Bazel supports the following non-registry overrides: -* [`archive_override`](/rules/lib/globals/module#archive_override) -* [`git_override`](/rules/lib/globals/module#git_override) -* [`local_path_override`](/rules/lib/globals/module#local_path_override) +- [`archive_override`](/rules/lib/globals/module#archive_override) +- [`git_override`](/rules/lib/globals/module#git_override) +- [`local_path_override`](/rules/lib/globals/module#local_path_override) -Note that setting a version value in the source archive `MODULE.bazel` can have -downsides when the module is being overridden with a non-registry override. To -learn more about this [see the `MODULE.bazel` -FAQ](faq#module-versioning-best-practices). +Note that setting a version value in the source archive `MODULE.bazel` can have downsides when the module is being overridden with a non-registry override. To learn more about this [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). ## Define repos that don't represent Bazel modules -With `bazel_dep`, you can define repos that represent other Bazel modules. -Sometimes there is a need to define a repo that does _not_ represent a Bazel -module; for example, one that contains a plain JSON file to be read as data. +With `bazel_dep`, you can define repos that represent other Bazel modules. Sometimes there is a need to define a repo that does *not* represent a Bazel module; for example, one that contains a plain JSON file to be read as data. -In this case, you could use the [`use_repo_rule` -directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo -by invoking a repo rule. This repo will only be visible to the module it's -defined in. +In this case, you could use the [`use_repo_rule` directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo by invoking a repo rule. This repo will only be visible to the module it's defined in. -Under the hood, this is implemented using the same mechanism as [module -extensions](/external/extension), which lets you define repos with more -flexibility. +Under the hood, this is implemented using the same mechanism as [module extensions](/external/extension), which lets you define repos with more flexibility. ## Repository names and strict deps -The [apparent name](/external/overview#apparent-repo-name) of a repo backing a -module to its direct dependents defaults to its module name, unless the -`repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) -directive says otherwise. Note that this means a module can only find its direct -dependencies. This helps prevent accidental breakages due to changes in -transitive dependencies. - -The [canonical name](/external/overview#canonical-repo-name) of a repo backing a -module is either `module_name+version{{ -"" }}` (for example, `bazel_skylib+1.0.3`) or `module_name{{ -"" }}+` (for example, `bazel_features+`), depending on whether there are -multiple versions of the module in the entire dependency graph (see -[`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). -Note that **the canonical name format** is not an API you should depend on and -**is subject to change at any time**. Instead of hard-coding the canonical name, -use a supported way to get it directly from Bazel: - -* In BUILD and `.bzl` files, use - [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance - constructed from a label string given by the apparent name of the repo, e.g., - `Label("@bazel_skylib").repo_name`. -* When looking up runfiles, use - [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) - or one of the runfiles libraries in - `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, - in `@rules_foo//foo/runfiles`. -* When interacting with Bazel from an external tool such as an IDE or language - server, use the `bazel mod dump_repo_mapping` command to get the mapping from - apparent names to canonical names for a given set of repositories. - -[Module extensions](/external/extension) can also introduce additional repos -into the visible scope of a module. +The [apparent name](/external/overview#apparent-repo-name) of a repo backing a module to its direct dependents defaults to its module name, unless the `repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive says otherwise. Note that this means a module can only find its direct dependencies. This helps prevent accidental breakages due to changes in transitive dependencies. + +The [canonical name](/external/overview#canonical-repo-name) of a repo backing a module is either `<var>module_name</var>+<var>version</var>` (for example, `bazel_skylib+1.0.3`) or `<var>module_name</var>+` (for example, `bazel_features+`), depending on whether there are multiple versions of the module in the entire dependency graph (see [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). Note that **the canonical name format** is not an API you should depend on and **is subject to change at any time**. Instead of hard-coding the canonical name, use a supported way to get it directly from Bazel: + +- In BUILD and `.bzl` files, use [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance constructed from a label string given by the apparent name of the repo, e.g., `Label("@bazel_skylib").repo_name`. +- When looking up runfiles, use [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) or one of the runfiles libraries in `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, in `@rules_foo//foo/runfiles`. +- When interacting with Bazel from an external tool such as an IDE or language server, use the `bazel mod dump_repo_mapping` command to get the mapping from apparent names to canonical names for a given set of repositories. + +[Module extensions](/external/extension) can also introduce additional repos into the visible scope of a module. diff --git a/external/overview.mdx b/external/overview.mdx index 6e55b579..84593001 100644 --- a/external/overview.mdx +++ b/external/overview.mdx @@ -2,28 +2,15 @@ title: 'External dependencies overview' --- +Bazel supports *external dependencies*, source files (both text and binary) used in your build that are not from your workspace. For example, they could be a ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local machine outside your current workspace. - - -Bazel supports *external dependencies*, source files (both text and binary) used -in your build that are not from your workspace. For example, they could be a -ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local -machine outside your current workspace. - -This document gives an overview of the system before examining some of the -concepts in more detail. +This document gives an overview of the system before examining some of the concepts in more detail. ## Overview of the system -Bazel's external dependency system works on the basis of [*Bazel -modules*](#module), each of which is a versioned Bazel project, and -[*repositories*](#repository) (or repos), which are directory trees containing -source files. +Bazel's external dependency system works on the basis of [*Bazel modules*](#module), each of which is a versioned Bazel project, and [*repositories*](#repository) (or repos), which are directory trees containing source files. -Bazel starts from the root module -- that is, the project you're working on. -Like all modules, it needs to have a `MODULE.bazel` file at its directory root, -declaring its basic metadata and direct dependencies. The following is a basic -example: +Bazel starts from the root module -- that is, the project you're working on. Like all modules, it needs to have a `MODULE.bazel` file at its directory root, declaring its basic metadata and direct dependencies. The following is a basic example: ```python module(name = "my-module", version = "1.0") @@ -32,29 +19,13 @@ bazel_dep(name = "rules_cc", version = "0.1.1") bazel_dep(name = "platforms", version = "0.0.11") ``` -From there, Bazel looks up all transitive dependency modules in a -[*Bazel registry*](registry) — by default, the [Bazel Central -Registry](https://bcr.bazel.build/). The registry provides the -dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire -transitive dependency graph before performing version resolution. - -After version resolution, in which one version is selected for each module, -Bazel consults the registry again to learn how to define a repo for each module --- that is, how the sources for each dependency module should be fetched. Most -of the time, these are just archives downloaded from the internet and extracted. - -Modules can also specify customized pieces of data called *tags*, which are -consumed by [*module extensions*](extension) after module resolution -to define additional repos. These extensions can perform actions like file I/O -and sending network requests. Among other things, they allow Bazel to interact -with other package management systems while also respecting the dependency graph -built out of Bazel modules. - -The three kinds of repos -- the main repo (which is the source tree you're -working in), the repos representing transitive dependency modules, and the repos -created by module extensions -- form the [*workspace*](#workspace) together. -External repos (non-main repos) are fetched on demand, for example when they're -referred to by labels (like `@repo//pkg:target`) in BUILD files. +From there, Bazel looks up all transitive dependency modules in a [*Bazel registry*](registry) — by default, the [Bazel Central Registry](https://bcr.bazel.build/). The registry provides the dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire transitive dependency graph before performing version resolution. + +After version resolution, in which one version is selected for each module, Bazel consults the registry again to learn how to define a repo for each module -- that is, how the sources for each dependency module should be fetched. Most of the time, these are just archives downloaded from the internet and extracted. + +Modules can also specify customized pieces of data called *tags*, which are consumed by [*module extensions*](extension) after module resolution to define additional repos. These extensions can perform actions like file I/O and sending network requests. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. + +The three kinds of repos -- the main repo (which is the source tree you're working in), the repos representing transitive dependency modules, and the repos created by module extensions -- form the [*workspace*](#workspace) together. External repos (non-main repos) are fetched on demand, for example when they're referred to by labels (like `@repo//pkg:target`) in BUILD files. ## Benefits @@ -62,54 +33,30 @@ Bazel's external dependency system offers a wide range of benefits. ### Automatic Dependency Resolution -- **Deterministic Version Resolution**: Bazel adopts the deterministic - [MVS](module#version-selection) version resolution algorithm, - minimizing conflicts and addressing diamond dependency issues. -- **Simplified Dependency Management**: `MODULE.bazel` declares only direct - dependencies, while transitive dependencies are automatically resolved, - providing a clearer overview of the project's dependencies. -- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: - Only direct dependencies are visible, ensuring correctness and - predictability. +- **Deterministic Version Resolution**: Bazel adopts the deterministic [MVS](module#version-selection) version resolution algorithm, minimizing conflicts and addressing diamond dependency issues. +- **Simplified Dependency Management**: `MODULE.bazel` declares only direct dependencies, while transitive dependencies are automatically resolved, providing a clearer overview of the project's dependencies. +- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: Only direct dependencies are visible, ensuring correctness and predictability. ### Ecosystem Integration -- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized - repository for discovering and managing common dependencies as Bazel - modules. -- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ - library) is adapted for Bazel and made available in BCR, it streamlines its - integration for the whole community and eliminates duplicated effort and - conflicts of custom BUILD files. -- **Unified Integration with Language-Specific Package Managers**: Rulesets - streamline integration with external package managers for non-Bazel - dependencies, including: - * [rules_jvm_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) - for Maven, - * [rules_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) - for PyPi, - * [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) - for Go Modules, - * [rules_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) - for Cargo. +- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized repository for discovering and managing common dependencies as Bazel modules. + +- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ library) is adapted for Bazel and made available in BCR, it streamlines its integration for the whole community and eliminates duplicated effort and conflicts of custom BUILD files. + +- **Unified Integration with Language-Specific Package Managers**: Rulesets streamline integration with external package managers for non-Bazel dependencies, including: + + - [rules\_jvm\_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) for Maven, + - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) for PyPi, + - [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) for Go Modules, + - [rules\_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) for Cargo. ### Advanced Features -- **[Module Extensions](extension)**: The - [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module - extension features allow flexible use of custom repository rules and - resolution logic to introduce any non-Bazel dependencies. -- **[`bazel mod` Command](mod-command)**: The sub-command offers - powerful ways to inspect external dependencies. You know exactly how an - external dependency is defined and where it comes from. -- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you - need to facilitate offline builds. -- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and - accelerates dependency resolution. -- **(Upcoming) [BCR Provenance - Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: - Strengthen supply chain security by ensuring verified provenance of - dependencies. +- **[Module Extensions](extension)**: The [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module extension features allow flexible use of custom repository rules and resolution logic to introduce any non-Bazel dependencies. +- **[`bazel mod` Command](mod-command)**: The sub-command offers powerful ways to inspect external dependencies. You know exactly how an external dependency is defined and where it comes from. +- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you need to facilitate offline builds. +- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and accelerates dependency resolution. +- **(Upcoming) [BCR Provenance Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: Strengthen supply chain security by ensuring verified provenance of dependencies. ## Concepts @@ -117,8 +64,7 @@ This section gives more detail on concepts related to external dependencies. ### Module -A Bazel project that can have multiple versions, each of which can have -dependencies on other modules. +A Bazel project that can have multiple versions, each of which can have dependencies on other modules. In a local Bazel workspace, a module is represented by a repository. @@ -126,118 +72,71 @@ For more details, see [Bazel modules](module). ### Repository -A directory tree with a boundary marker file at its root, containing source -files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo -represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in -legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file -will signify the boundary of a repo; multiple such files can coexist in a -directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. ### Main repository The repository in which the current Bazel command is being run. -The root of the main repository is also known as the -**workspace root**. +The root of the main repository is also known as the [](). ### Workspace -The environment shared by all Bazel commands run in the same main repository. It -encompasses the main repo and the set of all defined external repos. +The environment shared by all Bazel commands run in the same main repository. It encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". ### Canonical repository name -The name by which a repository is always addressable. Within the context of a -workspace, each repository has a single canonical name. A target inside a repo -whose canonical name is `canonical_name` can be addressed by the label -`@@canonical_name//package:target` (note the double `@`). +The name by which a repository is always addressable. Within the context of a workspace, each repository has a single canonical name. A target inside a repo whose canonical name is `canonical_name` can be addressed by the label `@@canonical_name//package:target` (note the double `@`). The main repository always has the empty string as the canonical name. ### Apparent repository name -The name by which a repository is addressable in the context of a certain other -repo. This can be thought of as a repo's "nickname": The repo with the canonical -name `michael` might have the apparent name `mike` in the context of the repo -`alice`, but might have the apparent name `mickey` in the context of the repo -`bob`. In this case, a target inside `michael` can be addressed by the label -`@mike//package:target` in the context of `alice` (note the single `@`). +The name by which a repository is addressable in the context of a certain other repo. This can be thought of as a repo's "nickname": The repo with the canonical name `michael` might have the apparent name `mike` in the context of the repo `alice`, but might have the apparent name `mickey` in the context of the repo `bob`. In this case, a target inside `michael` can be addressed by the label `@mike//package:target` in the context of `alice` (note the single `@`). -Conversely, this can be understood as a **repository mapping**: each repo -maintains a mapping from "apparent repo name" to a "canonical repo name". +Conversely, this can be understood as a **repository mapping**: each repo maintains a mapping from "apparent repo name" to a "canonical repo name". ### Repository rule -A schema for repository definitions that tells Bazel how to materialize a -repository. For example, it could be "download a zip archive from a certain URL -and extract it", or "fetch a certain Maven artifact and make it available as a -`java_import` target", or simply "symlink a local directory". Every repo is -**defined** by calling a repo rule with an appropriate number of arguments. +A schema for repository definitions that tells Bazel how to materialize a repository. For example, it could be "download a zip archive from a certain URL and extract it", or "fetch a certain Maven artifact and make it available as a `java_import` target", or simply "symlink a local directory". Every repo is **defined** by calling a repo rule with an appropriate number of arguments. -See [Repository rules](repo) for more information about how to write -your own repository rules. +See [Repository rules](repo) for more information about how to write your own repository rules. -The most common repo rules by far are -[`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive -from a URL and extracts it, and -[`local_repository`](/reference/be/workspace#local_repository), which symlinks a -local directory that is already a Bazel repository. +The most common repo rules by far are [`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive from a URL and extracts it, and [`local_repository`](/reference/be/workspace#local_repository), which symlinks a local directory that is already a Bazel repository. ### Fetch a repository -The action of making a repo available on local disk by running its associated -repo rule. The repos defined in a workspace are not available on local disk -before they are fetched. +The action of making a repo available on local disk by running its associated repo rule. The repos defined in a workspace are not available on local disk before they are fetched. -Normally, Bazel only fetches a repo when it needs something from the repo, -and the repo hasn't already been fetched. If the repo has already been fetched -before, Bazel only re-fetches it if its definition has changed. +Normally, Bazel only fetches a repo when it needs something from the repo, and the repo hasn't already been fetched. If the repo has already been fetched before, Bazel only re-fetches it if its definition has changed. -The `fetch` command can be used to initiate a pre-fetch for a repository, -target, or all necessary repositories to perform any build. This capability -enables offline builds using the `--nofetch` option. +The `fetch` command can be used to initiate a pre-fetch for a repository, target, or all necessary repositories to perform any build. This capability enables offline builds using the `--nofetch` option. -The `--fetch` option serves to manage network access. Its default value is true. -However, when set to false (`--nofetch`), the command will utilize any cached -version of the dependency, and if none exists, the command will result in -failure. +The `--fetch` option serves to manage network access. Its default value is true. However, when set to false (`--nofetch`), the command will utilize any cached version of the dependency, and if none exists, the command will result in failure. -See [fetch options](/reference/command-line-reference#fetch-options) for more -information about controlling fetch. +See [fetch options](/reference/command-line-reference#fetch-options) for more information about controlling fetch. ### Directory layout -After being fetched, the repo can be found in the subdirectory `external` in the -[output base](/remote/output-directories), under its canonical name. +After being fetched, the repo can be found in the subdirectory `external` in the [output base](/remote/output-directories), under its canonical name. -You can run the following command to see the contents of the repo with the -canonical name `canonical_name`: +You can run the following command to see the contents of the repo with the canonical name `canonical_name`: ```posix-terminal -ls $(bazel info output_base)/external/{{ '' }} canonical_name {{ '' }} +ls $(bazel info output_base)/external/<var> canonical_name </var> ``` ### REPO.bazel file -The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost -boundary of the directory tree that constitutes a repo. It doesn't need to -contain anything to serve as a repo boundary file; however, it can also be used -to specify some common attributes for all build targets inside the repo. +The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost boundary of the directory tree that constitutes a repo. It doesn't need to contain anything to serve as a repo boundary file; however, it can also be used to specify some common attributes for all build targets inside the repo. -The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no -`load` statements are supported. The `repo()` function takes the same arguments as the [`package()` -function](/reference/be/functions#package) in `BUILD` files; whereas `package()` -specifies common attributes for all build targets inside the package, `repo()` -analogously does so for all build targets inside the repo. +The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no `load` statements are supported. The `repo()` function takes the same arguments as the [`package()` function](/reference/be/functions#package) in `BUILD` files; whereas `package()` specifies common attributes for all build targets inside the package, `repo()` analogously does so for all build targets inside the repo. -For example, you can specify a common license for all targets in your repo by -having the following `REPO.bazel` file: +For example, you can specify a common license for all targets in your repo by having the following `REPO.bazel` file: ```python repo( @@ -247,12 +146,9 @@ repo( ## The legacy WORKSPACE system -In older Bazel versions (before 9.0), external dependencies were introduced by -defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a -similar syntax to `BUILD` files, employing repo rules instead of build rules. +In older Bazel versions (before 9.0), external dependencies were introduced by defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a similar syntax to `BUILD` files, employing repo rules instead of build rules. -The following snippet is an example to use the `http_archive` repo rule in the -`WORKSPACE` file: +The following snippet is an example to use the `http_archive` repo rule in the `WORKSPACE` file: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -263,43 +159,26 @@ http_archive( ) ``` -The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` -system, by default, the canonical name of a repo is also its apparent name to -all other repos. +The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` system, by default, the canonical name of a repo is also its apparent name to all other repos. -See the [full list](/rules/lib/globals/workspace) of functions available in -`WORKSPACE` files. +See the [full list](/rules/lib/globals/workspace) of functions available in `WORKSPACE` files. ### Shortcomings of the `WORKSPACE` system -In the years after the `WORKSPACE` system was introduced, users reported many -pain points, including: - -* Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all - transitive dependencies must be defined in the `WORKSPACE` file of the main - repo, in addition to direct dependencies. -* To work around this, projects have adopted the "deps.bzl" pattern, in which - they define a macro which in turn defines multiple repos, and ask users to - call this macro in their `WORKSPACE` files. - * This has its own problems: macros cannot `load` other `.bzl` files, so - these projects have to define their transitive dependencies in this - "deps" macro, or work around this issue by having the user call multiple - layered "deps" macros. - * Bazel evaluates the `WORKSPACE` file sequentially. Additionally, - dependencies are specified using `http_archive` with URLs, without any - version information. This means that there is no reliable way to perform - version resolution in the case of diamond dependencies (`A` depends on - `B` and `C`; `B` and `C` both depend on different versions of `D`). - -Due to the shortcomings of WORKSPACE, the new module-based system (codenamed -"Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. -Read the [Bzlmod migration guide](migration) on how to migrate -to Bzlmod. +In the years after the `WORKSPACE` system was introduced, users reported many pain points, including: + +- Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all transitive dependencies must be defined in the `WORKSPACE` file of the main repo, in addition to direct dependencies. + +- To work around this, projects have adopted the "deps.bzl" pattern, in which they define a macro which in turn defines multiple repos, and ask users to call this macro in their `WORKSPACE` files. + + - This has its own problems: macros cannot `load` other `.bzl` files, so these projects have to define their transitive dependencies in this "deps" macro, or work around this issue by having the user call multiple layered "deps" macros. + - Bazel evaluates the `WORKSPACE` file sequentially. Additionally, dependencies are specified using `http_archive` with URLs, without any version information. This means that there is no reliable way to perform version resolution in the case of diamond dependencies (`A` depends on `B` and `C`; `B` and `C` both depend on different versions of `D`). + +Due to the shortcomings of WORKSPACE, the new module-based system (codenamed "Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. Read the [Bzlmod migration guide](migration) on how to migrate to Bzlmod. ### External links on Bzlmod -* [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) -* [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) - (original Bzlmod design doc) -* [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) -* [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) +- [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) +- [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) (original Bzlmod design doc) +- [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) +- [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) diff --git a/external/repo.mdx b/external/repo.mdx index b878f030..bbf37aee 100644 --- a/external/repo.mdx +++ b/external/repo.mdx @@ -2,39 +2,19 @@ title: 'Repository Rules' --- +This page covers how to define repository rules and provides examples for more details. - -This page covers how to define repository rules and provides examples for more -details. - -An [external repository](/external/overview#repository) is a directory tree, -containing source files usable in a Bazel build, which is generated on demand by -running its corresponding **repo rule**. Repos can be defined in a multitude of -ways, but ultimately, each repo is defined by invoking a repo rule, just as -build targets are defined by invoking build rules. They can be used to depend on -third-party libraries (such as Maven packaged libraries) but also to generate -`BUILD` files specific to the host Bazel is running on. +An [external repository](/external/overview#repository) is a directory tree, containing source files usable in a Bazel build, which is generated on demand by running its corresponding **repo rule**. Repos can be defined in a multitude of ways, but ultimately, each repo is defined by invoking a repo rule, just as build targets are defined by invoking build rules. They can be used to depend on third-party libraries (such as Maven packaged libraries) but also to generate `BUILD` files specific to the host Bazel is running on. ## Repository rule definition -In a `.bzl` file, use the -[repository_rule](/rules/lib/globals/bzl#repository_rule) function to define a -new repo rule and store it in a global variable. After a repo rule is defined, -it can be invoked as a function to define repos. This invocation is usually -performed from inside a [module extension](/external/extension) implementation -function. +In a `.bzl` file, use the [repository\_rule](/rules/lib/globals/bzl#repository_rule) function to define a new repo rule and store it in a global variable. After a repo rule is defined, it can be invoked as a function to define repos. This invocation is usually performed from inside a [module extension](/external/extension) implementation function. -The two major components of a repo rule definition are its attribute schema and -implementation function. The attribute schema determines the names and types of -attributes passed to a repo rule invocation, and the implementation function is -run when the repo needs to be fetched. +The two major components of a repo rule definition are its attribute schema and implementation function. The attribute schema determines the names and types of attributes passed to a repo rule invocation, and the implementation function is run when the repo needs to be fetched. ## Attributes -Attributes are arguments passed to the repo rule invocation. The schema of -attributes accepted by a repo rule is specified using the `attrs` argument when -the repo rule is defined with a call to `repository_rule`. An example defining -`url` and `sha256` attributes as strings: +Attributes are arguments passed to the repo rule invocation. The schema of attributes accepted by a repo rule is specified using the `attrs` argument when the repo rule is defined with a call to `repository_rule`. An example defining `url` and `sha256` attributes as strings: ```python http_archive = repository_rule( @@ -46,8 +26,7 @@ http_archive = repository_rule( ) ``` -To access an attribute within the implementation function, use -`repository_ctx.attr.`: +To access an attribute within the implementation function, use `repository_ctx.attr.<attribute_name>`: ```python def _impl(repository_ctx): @@ -55,29 +34,15 @@ def _impl(repository_ctx): checksum = repository_ctx.attr.sha256 ``` -All `repository_rule`s have the implicitly defined attribute `name`. This is a -string attribute that behaves somewhat magically: when specified as an input to -a repo rule invocation, it takes an apparent repo name; but when read from the -repo rule's implementation function using `repository_ctx.attr.name`, it returns -the canonical repo name. +All `repository_rule`s have the implicitly defined attribute `name`. This is a string attribute that behaves somewhat magically: when specified as an input to a repo rule invocation, it takes an apparent repo name; but when read from the repo rule's implementation function using `repository_ctx.attr.name`, it returns the canonical repo name. ## Implementation function -Every repo rule requires an `implementation` function. It contains the actual -logic of the rule and is executed strictly in the Loading Phase. +Every repo rule requires an `implementation` function. It contains the actual logic of the rule and is executed strictly in the Loading Phase. -The function has exactly one input parameter, `repository_ctx`. The function -returns either `None` to signify that the rule is reproducible given the -specified parameters, or a dict with a set of parameters for that rule that -would turn that rule into a reproducible one generating the same repo. For -example, for a rule tracking a git repository that would mean returning a -specific commit identifier instead of a floating branch that was originally -specified. +The function has exactly one input parameter, `repository_ctx`. The function returns either `None` to signify that the rule is reproducible given the specified parameters, or a dict with a set of parameters for that rule that would turn that rule into a reproducible one generating the same repo. For example, for a rule tracking a git repository that would mean returning a specific commit identifier instead of a floating branch that was originally specified. -The input parameter `repository_ctx` can be used to access attribute values, and -non-hermetic functions (finding a binary, executing a binary, creating a file in -the repository or downloading a file from the Internet). See [the API -docs](/rules/lib/builtins/repository_ctx) for more context. Example: +The input parameter `repository_ctx` can be used to access attribute values, and non-hermetic functions (finding a binary, executing a binary, creating a file in the repository or downloading a file from the Internet). See [the API docs](/rules/lib/builtins/repository_ctx) for more context. Example: ```python def _impl(repository_ctx): @@ -90,72 +55,38 @@ local_repository = repository_rule( ## When is the implementation function executed? -The implementation function of a repo rule is executed when Bazel needs a target -from that repository, for example when another target (in another repo) depends -on it or if it is mentioned on the command line. The implementation function is -then expected to create the repo in the file system. This is called "fetching" -the repo. - -In contrast to regular targets, repos are not necessarily re-fetched when -something changes that would cause the repo to be different. This is because -there are things that Bazel either cannot detect changes to or it would cause -too much overhead on every build (for example, things that are fetched from the -network). Therefore, repos are re-fetched only if one of the following things -changes: - -* The attributes passed to the repo rule invocation. -* The Starlark code comprising the implementation of the repo rule. -* The value of any environment variable passed to `repository_ctx`'s - `getenv()` method or declared with the `environ` attribute of the - [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of - these environment variables can be hard-wired on the command line with the - [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. -* The existence, contents, and type of any paths being - [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation - function of the repo rule. - * Certain other methods of `repository_ctx` with a `watch` parameter, such - as `read()`, `execute()`, and `extract()`, can also cause paths to be - watched. - * Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) - and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths - to be watched in other ways. -* When `bazel fetch --force` is executed. - -There are two parameters of `repository_rule` that control when the repositories -are re-fetched: - -* If the `configure` flag is set, the repository is re-fetched on `bazel - fetch --force --configure` (non-`configure` repositories are not - re-fetched). -* If the `local` flag is set, in addition to the above cases, the repo is also - re-fetched when the Bazel server restarts. +The implementation function of a repo rule is executed when Bazel needs a target from that repository, for example when another target (in another repo) depends on it or if it is mentioned on the command line. The implementation function is then expected to create the repo in the file system. This is called "fetching" the repo. + +In contrast to regular targets, repos are not necessarily re-fetched when something changes that would cause the repo to be different. This is because there are things that Bazel either cannot detect changes to or it would cause too much overhead on every build (for example, things that are fetched from the network). Therefore, repos are re-fetched only if one of the following things changes: + +- The attributes passed to the repo rule invocation. + +- The Starlark code comprising the implementation of the repo rule. + +- The value of any environment variable passed to `repository_ctx`'s `getenv()` method or declared with the `environ` attribute of the [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of these environment variables can be hard-wired on the command line with the [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. + +- The existence, contents, and type of any paths being [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation function of the repo rule. + + - Certain other methods of `repository_ctx` with a `watch` parameter, such as `read()`, `execute()`, and `extract()`, can also cause paths to be watched. + - Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths to be watched in other ways. + +- When `bazel fetch --force` is executed. + +There are two parameters of `repository_rule` that control when the repositories are re-fetched: + +- If the `configure` flag is set, the repository is re-fetched on `bazel fetch --force --configure` (non-`configure` repositories are not re-fetched). +- If the `local` flag is set, in addition to the above cases, the repo is also re-fetched when the Bazel server restarts. ## Forcing refetch of external repos -Sometimes, an external repo can become outdated without any change to its -definition or dependencies. For example, a repo fetching sources might follow a -particular branch of a third-party repository, and new commits are available on -that branch. In this case, you can ask bazel to refetch all external repos -unconditionally by calling `bazel fetch --force --all`. +Sometimes, an external repo can become outdated without any change to its definition or dependencies. For example, a repo fetching sources might follow a particular branch of a third-party repository, and new commits are available on that branch. In this case, you can ask bazel to refetch all external repos unconditionally by calling `bazel fetch --force --all`. -Moreover, some repo rules inspect the local machine and might become outdated if -the local machine was upgraded. Here you can ask Bazel to only refetch those -external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) -definition has the `configure` attribute set, use `bazel fetch --force ---configure`. +Moreover, some repo rules inspect the local machine and might become outdated if the local machine was upgraded. Here you can ask Bazel to only refetch those external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) definition has the `configure` attribute set, use `bazel fetch --force --configure`. ## Examples -- [C++ auto-configured - toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): - it uses a repo rule to automatically create the C++ configuration files for - Bazel by looking for the local C++ compiler, the environment and the flags - the C++ compiler supports. +- [C++ auto-configured toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): it uses a repo rule to automatically create the C++ configuration files for Bazel by looking for the local C++ compiler, the environment and the flags the C++ compiler supports. -- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) - uses several `repository_rule` to defines the list of dependencies needed to - use the Go rules. +- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) uses several `repository_rule` to defines the list of dependencies needed to use the Go rules. -- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) - creates an external repository called `@maven` by default that generates - build targets for every Maven artifact in the transitive dependency tree. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) creates an external repository called `@maven` by default that generates build targets for every Maven artifact in the transitive dependency tree. diff --git a/external/vendor.mdx b/external/vendor.mdx index 653c1942..a2e79e8d 100644 --- a/external/vendor.mdx +++ b/external/vendor.mdx @@ -1,13 +1,8 @@ -keywords: product:Bazel,Bzlmod,vendor --- title: 'Vendor Mode' --- - - -Vendor mode is a feature that lets you create a local copy of -external dependencies. This is useful for offline builds, or when you want to -control the source of an external dependency. +Vendor mode is a feature that lets you create a local copy of external dependencies. This is useful for offline builds, or when you want to control the source of an external dependency. ## Enable vendor mode @@ -16,19 +11,15 @@ You can enable vendor mode by specifying `--vendor_dir` flag. For example, by adding it to your `.bazelrc` file: ```none -# Enable vendor mode with vendor directory under /vendor_src +# Enable vendor mode with vendor directory under <workspace>/vendor_src common --vendor_dir=vendor_src ``` -The vendor directory can be either a relative path to your workspace root or an -absolute path. +The vendor directory can be either a relative path to your workspace root or an absolute path. ## Vendor a specific external repository -You can use the `vendor` command with the `--repo` flag to specify which repo -to vendor, it accepts both [canonical repo -name](/external/overview#canonical-repo-name) and [apparent repo -name](/external/overview#apparent-repo-name). +You can use the `vendor` command with the `--repo` flag to specify which repo to vendor, it accepts both [canonical repo name](/external/overview#canonical-repo-name) and [apparent repo name](/external/overview#apparent-repo-name). For example, running: @@ -42,13 +33,11 @@ or bazel vendor --vendor_dir=vendor_src --repo=@@rules_cc+ ``` -will both get rules_cc to be vendored under -`/vendor_src/rules_cc+`. +will both get rules\_cc to be vendored under `<workspace root>/vendor_src/rules_cc+`. ## Vendor external dependencies for given targets -To vendor all external dependencies required for building given target patterns, -you can run `bazel vendor `. +To vendor all external dependencies required for building given target patterns, you can run `bazel vendor <target patterns>`. For example @@ -56,12 +45,9 @@ For example bazel vendor --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -will vendor all repos required for building the `//src/main:hello-world` target -and all targets under `//src/test/...` with the current configuration. +will vendor all repos required for building the `//src/main:hello-world` target and all targets under `//src/test/...` with the current configuration. -Under the hood, it's doing a `bazel build --nobuild` command to analyze the -target patterns, therefore build flags could be applied to this command and -affect the result. +Under the hood, it's doing a `bazel build --nobuild` command to analyze the target patterns, therefore build flags could be applied to this command and affect the result. ### Build the target offline @@ -71,20 +57,15 @@ With the external dependencies vendored, you can build the target offline by bazel build --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -The build should work in a clean build environment without network access and -repository cache. +The build should work in a clean build environment without network access and repository cache. -Therefore, you should be able to check in the vendored source and build the same -targets offline on another machine. +Therefore, you should be able to check in the vendored source and build the same targets offline on another machine. -Note: If you make changes to the targets to build, the external dependencies, -the build configuration, or the Bazel version, you may need to re-vendor to make -sure offline build still works. +Note: If you make changes to the targets to build, the external dependencies, the build configuration, or the Bazel version, you may need to re-vendor to make sure offline build still works. ## Vendor all external dependencies -To vendor all repos in your transitive external dependencies graph, you can -run: +To vendor all repos in your transitive external dependencies graph, you can run: ```none bazel vendor --vendor_dir=vendor_src @@ -92,25 +73,20 @@ bazel vendor --vendor_dir=vendor_src Note that vendoring all dependencies has a few **disadvantages**: -- Fetching all repos, including those introduced transitively, can be time-consuming. -- The vendor directory can become very large. -- Some repos may fail to fetch if they are not compatible with the current platform or environment. +- Fetching all repos, including those introduced transitively, can be time-consuming. +- The vendor directory can become very large. +- Some repos may fail to fetch if they are not compatible with the current platform or environment. Therefore, consider vendoring for specific targets first. ## Configure vendor mode with VENDOR.bazel -You can control how given repos are handled with the VENDOR.bazel file located -under the vendor directory. +You can control how given repos are handled with the VENDOR.bazel file located under the vendor directory. -There are two directives available, both accepting a list of -[canonical repo names](/external/overview#canonical-repo-name) as arguments: +There are two directives available, both accepting a list of [canonical repo names](/external/overview#canonical-repo-name) as arguments: - `ignore()`: to completely ignore a repository from vendor mode. -- `pin()`: to pin a repository to its current vendored source as if there is a - `--override_repository` flag for this repo. Bazel will NOT update the vendored - source for this repo while running the vendor command unless it's unpinned. - The user can modify and maintain the vendored source for this repo manually. +- `pin()`: to pin a repository to its current vendored source as if there is a `--override_repository` flag for this repo. Bazel will NOT update the vendored source for this repo while running the vendor command unless it's unpinned. The user can modify and maintain the vendored source for this repo manually. For example @@ -121,93 +97,59 @@ pin("@@bazel_skylib+") With this configuration -- Both repos will be excluded from subsequent vendor commands. -- Repo `bazel_skylib` will be overridden to the source located under the - vendor directory. -- The user can safely modify the vendored source of `bazel_skylib`. -- To re-vendor `bazel_skylib`, the user has to disable the pin statement - first. +- Both repos will be excluded from subsequent vendor commands. +- Repo `bazel_skylib` will be overridden to the source located under the vendor directory. +- The user can safely modify the vendored source of `bazel_skylib`. +- To re-vendor `bazel_skylib`, the user has to disable the pin statement first. -Note: Repository rules with -[`local`](/rules/lib/globals/bzl#repository_rule.local) or -[`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are -always excluded from vendoring. +Note: Repository rules with [`local`](/rules/lib/globals/bzl#repository_rule.local) or [`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are always excluded from vendoring. ## Understand how vendor mode works -Bazel fetches external dependencies of a project under `$(bazel info -output_base)/external`. Vendoring external dependencies means moving out -relevant files and directories to the given vendor directory and use the -vendored source for later builds. +Bazel fetches external dependencies of a project under `$(bazel info output_base)/external`. Vendoring external dependencies means moving out relevant files and directories to the given vendor directory and use the vendored source for later builds. The content being vendored includes: -- The repo directory -- The repo marker file +- The repo directory +- The repo marker file -During a build, if the vendored marker file is up-to-date or the repo is -pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating -a symlink to it under `$(bazel info output_base)/external` instead of actually -running the repository rule. Otherwise, a warning is printed and Bazel will -fallback to fetching the latest version of the repo. +During a build, if the vendored marker file is up-to-date or the repo is pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating a symlink to it under `$(bazel info output_base)/external` instead of actually running the repository rule. Otherwise, a warning is printed and Bazel will fallback to fetching the latest version of the repo. -Note: Bazel assumes the vendored source is not changed by users unless the repo -is pinned in the VENDOR.bazel file. If a user does change the vendored source -without pinning the repo, the changed vendored source will be used, but it will -be overwritten if its existing marker file is -outdated and the repo is vendored again. +Note: Bazel assumes the vendored source is not changed by users unless the repo is pinned in the VENDOR.bazel file. If a user does change the vendored source without pinning the repo, the changed vendored source will be used, but it will be overwritten if its existing marker file is outdated and the repo is vendored again. ### Vendor registry files -Bazel has to perform the Bazel module resolution in order to fetch external -dependencies, which may require accessing registry files through internet. To -achieve offline build, Bazel vendors all registry files fetched from -network under the `/_registries` directory. +Bazel has to perform the Bazel module resolution in order to fetch external dependencies, which may require accessing registry files through internet. To achieve offline build, Bazel vendors all registry files fetched from network under the `<vendor_dir>/_registries` directory. ### Vendor symlinks -External repositories may contain symlinks pointing to other files or -directories. To make sure symlinks work correctly, Bazel uses the following -strategy to rewrite symlinks in the vendored source: +External repositories may contain symlinks pointing to other files or directories. To make sure symlinks work correctly, Bazel uses the following strategy to rewrite symlinks in the vendored source: -- Create a symlink `/bazel-external` that points to `$(bazel info - output_base)/external`. It is refreshed by every Bazel command - automatically. -- For the vendored source, rewrite all symlinks that originally point to a - path under `$(bazel info output_base)/external` to a relative path under - `/bazel-external`. +- Create a symlink `<vendor_dir>/bazel-external` that points to `$(bazel info output_base)/external`. It is refreshed by every Bazel command automatically. +- For the vendored source, rewrite all symlinks that originally point to a path under `$(bazel info output_base)/external` to a relative path under `<vendor_dir>/bazel-external`. For example, if the original symlink is ```none -/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file +<vendor_dir>/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file ``` It will be rewritten to ```none -/repo_foo+/link => ../../bazel-external/repo_bar+/file +<vendor_dir>/repo_foo+/link => ../../bazel-external/repo_bar+/file ``` where ```none -/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed +<vendor_dir>/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed ``` -Since `/bazel-external` is generated by Bazel automatically, it's -recommended to add it to `.gitignore` or equivalent to avoid checking it in. - -With this strategy, symlinks in the vendored source should work correctly even -after the vendored source is moved to another location or the bazel output base -is changed. +Since `<vendor_dir>/bazel-external` is generated by Bazel automatically, it's recommended to add it to `.gitignore` or equivalent to avoid checking it in. -Note: symlinks that point to an absolute path outside of $(bazel info -output_base)/external are not rewritten. Therefore, it could still break -cross-machine compatibility. +With this strategy, symlinks in the vendored source should work correctly even after the vendored source is moved to another location or the bazel output base is changed. -Note: On Windows, vendoring symlinks only works with -[`--windows_enable_symlinks`][windows_enable_symlinks] -flag enabled. +Note: symlinks that point to an absolute path outside of $(bazel info output\_base)/external are not rewritten. Therefore, it could still break cross-machine compatibility. -[windows_enable_symlinks]: /reference/command-line-reference#flag--windows_enable_symlinks +Note: On Windows, vendoring symlinks only works with [`--windows_enable_symlinks`](/reference/command-line-reference#flag--windows_enable_symlinks) flag enabled. diff --git a/help.mdx b/help.mdx index 754b4bc2..af11fb80 100644 --- a/help.mdx +++ b/help.mdx @@ -2,53 +2,49 @@ title: 'Getting Help' --- - - -This page lists Bazel resources beyond the documentation and covers how to get -support from the Bazel team and community. +This page lists Bazel resources beyond the documentation and covers how to get support from the Bazel team and community. ## Search existing material In addition to the documentation, you can find helpful information by searching: -* [Bazel user group](https://groups.google.com/g/bazel-discuss) -* [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) -* [Bazel blog](https://blog.bazel.build/) -* [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -* [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) +- [Bazel user group](https://groups.google.com/g/bazel-discuss) +- [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) +- [Bazel blog](https://blog.bazel.build/) +- [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +- [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) ## Watch videos There are recordings of Bazel talks at various conferences, such as: -* Bazel’s annual conference, BazelCon: - * [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) - * [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) - * [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) - * [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) - * [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) - * [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) - * [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) - * [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) -* Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) +- Bazel’s annual conference, BazelCon: + + - [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) + - [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) + - [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) + - [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) + - [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) + - [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) + - [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) + - [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +- Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) ## Ask the Bazel community If there are no existing answers, you can ask the community by: -* Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) -* Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) -* Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -* Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) -* Consulting a [Bazel community expert](/community/experts) +- Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) +- Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) +- Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +- Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) +- Consulting a [Bazel community expert](/community/experts) ## Understand Bazel's support level -Please read the [release page](/release) to understand Bazel's release model and -what level of support Bazel provides. +Please read the [release page](/release) to understand Bazel's release model and what level of support Bazel provides. ## File a bug -If you encounter a bug or want to request a feature, file a [GitHub -Issue](https://github.com/bazelbuild/bazel/issues). +If you encounter a bug or want to request a feature, file a [GitHub Issue](https://github.com/bazelbuild/bazel/issues). diff --git a/install/bazelisk.mdx b/install/bazelisk.mdx index a3189cb7..4bc9f3e5 100644 --- a/install/bazelisk.mdx +++ b/install/bazelisk.mdx @@ -2,58 +2,39 @@ title: 'Installing / Updating Bazel using Bazelisk' --- - - ## Installing Bazel -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the -recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically -downloads and installs the appropriate version of Bazel. Use Bazelisk if you -need to switch between different versions of Bazel depending on the current -working directory, or to always keep Bazel updated to the latest release. +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically downloads and installs the appropriate version of Bazel. Use Bazelisk if you need to switch between different versions of Bazel depending on the current working directory, or to always keep Bazel updated to the latest release. -For more details, see -[the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). +For more details, see [the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). ## Updating Bazel -Bazel has a [backward compatibility policy](/release/backward-compatibility) -(see [guidance for rolling out incompatible -changes](/contribute/breaking-changes) if you -are the author of one). That page summarizes best practices on how to test and -migrate your project with upcoming incompatible changes and how to provide -feedback to the incompatible change authors. +Bazel has a [backward compatibility policy](/release/backward-compatibility) (see [guidance for rolling out incompatible changes](/contribute/breaking-changes) if you are the author of one). That page summarizes best practices on how to test and migrate your project with upcoming incompatible changes and how to provide feedback to the incompatible change authors. ### Managing Bazel versions with Bazelisk -[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage -Bazel versions. +[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage Bazel versions. Bazelisk can: -* Auto-update Bazel to the latest LTS or rolling release. -* Build the project with a Bazel version specified in the .bazelversion - file. Check in that file into your version control to ensure reproducibility - of your builds. -* Help migrate your project for incompatible changes (see above) -* Easily try release candidates +- Auto-update Bazel to the latest LTS or rolling release. +- Build the project with a Bazel version specified in the .bazelversion file. Check in that file into your version control to ensure reproducibility of your builds. +- Help migrate your project for incompatible changes (see above) +- Easily try release candidates ### Recommended migration process -Within minor updates to any LTS release, any -project can be prepared for the next release without breaking -compatibility with the current release. However, there may be -backward-incompatible changes between major LTS versions. +Within minor updates to any LTS release, any project can be prepared for the next release without breaking compatibility with the current release. However, there may be backward-incompatible changes between major LTS versions. Follow this process to migrate from one major version to another: 1. Read the release notes to get advice on how to migrate to the next version. -1. Major incompatible changes should have an associated `--incompatible_*` flag - and a corresponding GitHub issue: - * Migration guidance is available in the associated GitHub issue. - * Tooling is available for some of incompatible changes migration. For - example, [buildifier](https://github.com/bazelbuild/buildtools/releases). - * Report migration problems by commenting on the associated GitHub issue. - -After migration, you can continue to build your projects without worrying about -backward-compatibility until the next major release. + +2. Major incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue: + + - Migration guidance is available in the associated GitHub issue. + - Tooling is available for some of incompatible changes migration. For example, [buildifier](https://github.com/bazelbuild/buildtools/releases). + - Report migration problems by commenting on the associated GitHub issue. + +After migration, you can continue to build your projects without worrying about backward-compatibility until the next major release. diff --git a/install/compile-source.mdx b/install/compile-source.mdx index 3b87883d..8ba134d8 100644 --- a/install/compile-source.mdx +++ b/install/compile-source.mdx @@ -2,91 +2,67 @@ title: 'Compiling Bazel from Source' --- - - -This page describes how to install Bazel from source and provides -troubleshooting tips for common issues. +This page describes how to install Bazel from source and provides troubleshooting tips for common issues. To build Bazel from source, you can do one of the following: -* Build it [using an existing Bazel binary](#build-bazel-using-bazel) +- Build it [using an existing Bazel binary](#build-bazel-using-bazel) -* Build it [without an existing Bazel binary](#bootstrap-bazel) which is known - as _bootstrapping_. +- Build it [without an existing Bazel binary](#bootstrap-bazel) which is known as *bootstrapping*. ## Build Bazel using Bazel ### Summary -1. Get the latest Bazel release from the - [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with - [Bazelisk](https://github.com/bazelbuild/bazelisk). +1. Get the latest Bazel release from the [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with [Bazelisk](https://github.com/bazelbuild/bazelisk). -2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) - and extract somewhere. - Alternatively you can git clone the source tree from https://github.com/bazelbuild/bazel +2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) and extract somewhere. Alternatively you can git clone the source tree from [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) -3. Install the same prerequisites as for bootstrapping (see - [for Unix-like systems](#bootstrap-unix-prereq) or - [for Windows](#bootstrap-windows-prereq)) +3. Install the same prerequisites as for bootstrapping (see [for Unix-like systems](#bootstrap-unix-prereq) or [for Windows](#bootstrap-windows-prereq)) -4. Build a development build of Bazel using Bazel: - `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on - Windows) +4. Build a development build of Bazel using Bazel: `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on Windows) -5. The resulting binary is at `bazel-bin/src/bazel-dev` - (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you - like and use immediately without further installation. +5. The resulting binary is at `bazel-bin/src/bazel-dev` (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you like and use immediately without further installation. Detailed instructions follow below. ### Step 1: Get the latest Bazel release -**Goal**: Install or download a release version of Bazel. Make sure you can run -it by typing `bazel` in a terminal. +**Goal**: Install or download a release version of Bazel. Make sure you can run it by typing `bazel` in a terminal. -**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing -Bazel binary. You can install one from a package manager or download one from -GitHub. See [Installing Bazel](/install). (Or you can [build from -scratch (bootstrap)](#bootstrap-bazel).) +**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing Bazel binary. You can install one from a package manager or download one from GitHub. See [Installing Bazel](/install). (Or you can [build from scratch (bootstrap)](#bootstrap-bazel).) **Troubleshooting**: -* If you cannot run Bazel by typing `bazel` in a terminal: +- If you cannot run Bazel by typing `bazel` in a terminal: - * Maybe your Bazel binary's directory is not on the PATH. + - Maybe your Bazel binary's directory is not on the PATH. - This is not a big problem. Instead of typing `bazel`, you will need to - type the full path. + This is not a big problem. Instead of typing `bazel`, you will need to type the full path. - * Maybe the Bazel binary itself is not called `bazel` (on Unixes) or - `bazel.exe` (on Windows). + - Maybe the Bazel binary itself is not called `bazel` (on Unixes) or `bazel.exe` (on Windows). - This is not a big problem. You can either rename the binary, or type the - binary's name instead of `bazel`. + This is not a big problem. You can either rename the binary, or type the binary's name instead of `bazel`. - * Maybe the binary is not executable (on Unixes). + - Maybe the binary is not executable (on Unixes). - You must make the binary executable by running `chmod +x /path/to/bazel`. + You must make the binary executable by running `chmod +x /path/to/bazel`. ### Step 2: Download Bazel's sources from GitHub -If you are familiar with Git, then just git clone https://github.com/bazelbuild/bazel +If you are familiar with Git, then just git clone [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) Otherwise: -1. Download the - [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). +1. Download the [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). -2. Extract the contents somewhere. +2. Extract the contents somewhere. - For example create a `bazel-src` directory under your home directory and - extract there. + For example create a `bazel-src` directory under your home directory and extract there. ### Step 3: Install prerequisites -Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ -compiler, MSYS2 (if you are building on Windows), etc. +Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ compiler, MSYS2 (if you are building on Windows), etc. ### Step 4a: Build Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -96,66 +72,63 @@ For instructions for Windows, see [Build Bazel on Windows](#build-bazel-on-windo **Instructions**: -1. Start a Bash terminal +1. Start a Bash terminal -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - cd ~/bazel-src + ``` + cd ~/bazel-src + ``` -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev + ``` + bazel build //src:bazel-dev + ``` - Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` - to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel - version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). +4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). ### Step 4b: Build Bazel on Windows -For instructions for Unix-like systems, see -[Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). +For instructions for Unix-like systems, see [Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). -**Goal**: Run Bazel to build a custom Bazel binary -(`bazel-bin\src\bazel-dev.exe`). +**Goal**: Run Bazel to build a custom Bazel binary (`bazel-bin\src\bazel-dev.exe`). **Instructions**: -1. Start Command Prompt (Start Menu > Run > "cmd.exe") +1. Start Command Prompt (Start Menu \> Run \> "cmd.exe") -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - cd %USERPROFILE%\bazel-src + ``` + cd %USERPROFILE%\bazel-src + ``` -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev.exe + bazel build //src:bazel-dev.exe - Alternatively you can run `bazel build //src:bazel.exe - --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel.exe --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel - version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin\src\bazel-dev.exe` (or - `bazel-bin\src\bazel.exe`). +4. The output will be at `bazel-bin\src\bazel-dev.exe` (or `bazel-bin\src\bazel.exe`). ### Step 5: Install the built binary Actually, there's nothing to install. -The output of the previous step is a self-contained Bazel binary. You can copy -it to any directory and use immediately. (It's useful if that directory is on -your PATH so that you can run "bazel" everywhere.) +The output of the previous step is a self-contained Bazel binary. You can copy it to any directory and use immediately. (It's useful if that directory is on your PATH so that you can run "bazel" everywhere.) ---- +*** ## Build Bazel from scratch (bootstrapping) @@ -165,24 +138,16 @@ You can also build Bazel from scratch, without using an existing Bazel binary. (This step is the same for all platforms.) -1. Download `bazel--dist.zip` from - [GitHub](https://github.com/bazelbuild/bazel/releases), for example - `bazel-0.28.1-dist.zip`. +1. Download `bazel-<version>-dist.zip` from [GitHub](https://github.com/bazelbuild/bazel/releases), for example `bazel-0.28.1-dist.zip`. - **Attention**: + **Attention**: - - There is a **single, architecture-independent** distribution archive. - There are no architecture-specific or OS-specific distribution archives. - - These sources are **not the same as the GitHub source tree**. You - have to use the distribution archive to bootstrap Bazel. You cannot - use a source tree cloned from GitHub. (The distribution archive contains - generated source files that are required for bootstrapping and are not part - of the normal Git source tree.) + - There is a **single, architecture-independent** distribution archive. There are no architecture-specific or OS-specific distribution archives. + - These sources are **not the same as the GitHub source tree**. You have to use the distribution archive to bootstrap Bazel. You cannot use a source tree cloned from GitHub. (The distribution archive contains generated source files that are required for bootstrapping and are not part of the normal Git source tree.) -2. Unpack the distribution archive somewhere on disk. +2. Unpack the distribution archive somewhere on disk. - You should verify the signature made by Bazel's - [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. + You should verify the signature made by Bazel's [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. ### Step 2a: Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -190,18 +155,17 @@ For instructions for Windows, see [Bootstrap Bazel on Windows](#bootstrap-window #### 2.1. Install the prerequisites -* **Bash** +- **Bash** -* **zip, unzip** +- **zip, unzip** -* **C++ build toolchain** +- **C++ build toolchain** -* **JDK.** Version 21 is required. +- **JDK.** Version 21 is required. -* **Python**. Version 3 is required. +- **Python**. Version 3 is required. -For example on Ubuntu Linux you can install these requirements using the -following command: +For example on Ubuntu Linux you can install these requirements using the following command: ```sh sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip @@ -209,90 +173,76 @@ sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip #### 2.2. Bootstrap Bazel on Unix -1. Open a shell or Terminal window. +1. Open a shell or Terminal window. -3. `cd` to the directory where you unpacked the distribution archive. +2. `cd` to the directory where you unpacked the distribution archive. -3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. +3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. -The compiled output is placed into `output/bazel`. This is a self-contained -Bazel binary, without an embedded JDK. You can copy it anywhere or use it -in-place. For convenience, copy this binary to a directory that's on your -`PATH` (such as `/usr/local/bin` on Linux). +The compiled output is placed into `output/bazel`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH` (such as `/usr/local/bin` on Linux). -To build the `bazel` binary in a reproducible way, also set -[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) -in the "Run the compilation script" step. +To build the `bazel` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. ### Step 2b: Bootstrap Bazel on Windows -For instructions for Unix-like systems, see -[Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). +For instructions for Unix-like systems, see [Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). #### 2.1. Install the prerequisites -* [MSYS2 shell](https://msys2.github.io/) +- [MSYS2 shell](https://msys2.github.io/) -* **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: +- **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: - ``` - pacman -S zip unzip patch - ``` + ``` + pacman -S zip unzip patch + ``` -* **The Visual C++ compiler.** Install the Visual C++ compiler either as part - of Visual Studio 2015 or newer, or by installing the latest [Build Tools - for Visual Studio 2017](https://aka.ms/BuildTools). +- **The Visual C++ compiler.** Install the Visual C++ compiler either as part of Visual Studio 2015 or newer, or by installing the latest [Build Tools for Visual Studio 2017](https://aka.ms/BuildTools). -* **JDK.** Version 21 is required. +- **JDK.** Version 21 is required. -* **Python**. Versions 2 and 3 are supported, installing one of them is - enough. You need the Windows-native version (downloadable from - [https://www.python.org](https://www.python.org)). Versions installed via - pacman in MSYS2 will not work. +- **Python**. Versions 2 and 3 are supported, installing one of them is enough. You need the Windows-native version (downloadable from [https://www.python.org](https://www.python.org)). Versions installed via pacman in MSYS2 will not work. #### 2.2. Bootstrap Bazel on Windows -1. Open the MSYS2 shell. +1. Open the MSYS2 shell. + +2. Set the following environment variables: + + - Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the path to the Visual Studio directory (BAZEL\_V**S**) or to the Visual C++ directory (BAZEL\_V**C**). Setting one of them is enough. + + - `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the examples below. + + Do not set this to `C:\Windows\System32\bash.exe`. (You have that file if you installed Windows Subsystem for Linux.) Bazel does not support this version of `bash.exe`. -2. Set the following environment variables: - * Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the - path to the Visual Studio directory (BAZEL\_VS) or to the Visual - C++ directory (BAZEL\_VC). Setting one of them is enough. - * `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the - examples below. + - `PATH`: Add the Python directory. - Do not set this to `C:\Windows\System32\bash.exe`. (You have that file - if you installed Windows Subsystem for Linux.) Bazel does not support - this version of `bash.exe`. - * `PATH`: Add the Python directory. - * `JAVA_HOME`: Set to the JDK directory. + - `JAVA_HOME`: Set to the JDK directory. - **Example** (using BAZEL\_VS): + **Example** (using BAZEL\_V**S**): - export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` + export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` - or (using BAZEL\_VC): + or (using BAZEL\_V**C**): - export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` + export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` -3. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` +4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` -The compiled output is placed into `output/bazel.exe`. This is a self-contained -Bazel binary, without an embedded JDK. You can copy it anywhere or use it -in-place. For convenience, copy this binary to a directory that's on -your `PATH`. +The compiled output is placed into `output/bazel.exe`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH`. -To build the `bazel.exe` binary in a reproducible way, also set -[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) -in the "Run the compilation script" step. +To build the `bazel.exe` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. -You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the -Command Prompt (`cmd.exe`) or PowerShell. +You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the Command Prompt (`cmd.exe`) or PowerShell. diff --git a/install/completion.mdx b/install/completion.mdx index 1d1d1b76..309ce3f8 100644 --- a/install/completion.mdx +++ b/install/completion.mdx @@ -2,11 +2,7 @@ title: 'Command-Line Completion' --- - - -You can enable command-line completion (also known as tab-completion) in Bash -and Zsh. This lets you tab-complete command names, flags names and flag values, -and target names. +You can enable command-line completion (also known as tab-completion) in Bash and Zsh. This lets you tab-complete command names, flags names and flag values, and target names. ## Bash @@ -14,55 +10,51 @@ Bazel comes with a Bash completion script. If you installed Bazel: -* From the APT repository, then you're done -- the Bash completion script is - already installed in `/etc/bash_completion.d`. +- From the APT repository, then you're done -- the Bash completion script is already installed in `/etc/bash_completion.d`. + +- From Homebrew, then you're done -- the Bash completion script is already installed in `$(brew --prefix)/etc/bash_completion.d`. + +- From the installer downloaded from GitHub, then: + + 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. + + 2. Do one of the following: + + - Either copy this file to your completion directory (if you have one). + + Example: on Ubuntu this is the `/etc/bash_completion.d` directory. + + - Or source the completion file from Bash's RC file. -* From Homebrew, then you're done -- the Bash completion script is - already installed in `$(brew --prefix)/etc/bash_completion.d`. + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: -* From the installer downloaded from GitHub, then: - 1. Locate the absolute path of the completion file. The installer copied it - to the `bin` directory. + ``` + source /path/to/bazel-complete.bash + ``` - Example: if you ran the installer with `--user`, this will be - `$HOME/.bazel/bin`. If you ran the installer as root, this will be - `/usr/local/lib/bazel/bin`. - 2. Do one of the following: - * Either copy this file to your completion directory (if you have - one). +- Via [bootstrapping](/install/compile-source), then: - Example: on Ubuntu this is the `/etc/bash_completion.d` directory. - * Or source the completion file from Bash's RC file. + 1. Emit the completion script into a file: - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) - or `~/.bash_profile` (on macOS), using the path to your completion - file's absolute path: + ``` + bazel help completion bash > bazel-complete.bash + ``` - ``` - source /path/to/bazel-complete.bash - ``` + 2. Do one of the following: -* Via [bootstrapping](/install/compile-source), then: - 1. Emit the completion script into a file: + - Copy this file to your completion directory, if you have one. - ``` - bazel help completion bash > bazel-complete.bash - ``` - 2. Do one of the following: - * Copy this file to your completion directory, if you have - one. + Example: on Ubuntu this is the `/etc/bash_completion.d` directory - Example: on Ubuntu this is the `/etc/bash_completion.d` directory - * Copy it somewhere on your local disk, such as to `$HOME`, and - source the completion file from Bash's RC file. + - Copy it somewhere on your local disk, such as to `$HOME`, and source the completion file from Bash's RC file. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) - or `~/.bash_profile` (on macOS), using the path to your completion - file's absolute path: + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: - ``` - source /path/to/bazel-complete.bash - ``` + ``` + source /path/to/bazel-complete.bash + ``` ## Zsh @@ -70,57 +62,48 @@ Bazel comes with a Zsh completion script. If you installed Bazel: -* From the APT repository, then you're done -- the Zsh completion script is - already installed in `/usr/share/zsh/vendor-completions`. - - > If you have a heavily customized `.zshrc` and the autocomplete - > does not function, try one of the following solutions: - > - > Add the following to your `.zshrc`: - > - > ``` - > zstyle :compinstall filename '/home/tradical/.zshrc' - > - > autoload -Uz compinit - > compinit - > ``` - > - > or - > - > Follow the instructions - > [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) - > - > If you are using `oh-my-zsh`, you may want to install and enable - > the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the - > solutions described above. - -* From Homebrew, then you're done -- the Zsh completion script is - already installed in `$(brew --prefix)/share/zsh/site-functions`. - -* From the installer downloaded from GitHub, then: - 1. Locate the absolute path of the completion file. The installer copied it - to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be - `$HOME/.bazel/bin`. If you ran the installer as root, this will be - `/usr/local/lib/bazel/bin`. - - 2. Add this script to a directory on your `$fpath`: - - ``` - fpath[1,0]=~/.zsh/completion/ - mkdir -p ~/.zsh/completion/ - cp /path/from/above/step/_bazel ~/.zsh/completion - ``` - - You may have to call `rm -f ~/.zcompdump; compinit` - the first time to make it work. - - 3. Optionally, add the following to your .zshrc. - - ``` - # This way the completion script does not have to parse Bazel's options - # repeatedly. The directory in cache-path must be created manually. - zstyle ':completion:*' use-cache on - zstyle ':completion:*' cache-path ~/.zsh/cache - ``` +- From the APT repository, then you're done -- the Zsh completion script is already installed in `/usr/share/zsh/vendor-completions`. + + > If you have a heavily customized `.zshrc` and the autocomplete does not function, try one of the following solutions: + > + > Add the following to your `.zshrc`: + > + > ``` + > zstyle :compinstall filename '/home/tradical/.zshrc' + > + > autoload -Uz compinit + > compinit + > ``` + > + > or + > + > Follow the instructions [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) + > + > If you are using `oh-my-zsh`, you may want to install and enable the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the solutions described above. + +- From Homebrew, then you're done -- the Zsh completion script is already installed in `$(brew --prefix)/share/zsh/site-functions`. + +- From the installer downloaded from GitHub, then: + + 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. + + 2. Add this script to a directory on your `$fpath`: + + ``` + fpath[1,0]=~/.zsh/completion/ + mkdir -p ~/.zsh/completion/ + cp /path/from/above/step/_bazel ~/.zsh/completion + ``` + + You may have to call `rm -f ~/.zcompdump; compinit` the first time to make it work. + + 3. Optionally, add the following to your .zshrc. + + ``` + # This way the completion script does not have to parse Bazel's options + # repeatedly. The directory in cache-path must be created manually. + zstyle ':completion:*' use-cache on + zstyle ':completion:*' cache-path ~/.zsh/cache + ``` diff --git a/install/docker-container.mdx b/install/docker-container.mdx index b1efb76c..2e97dc2b 100644 --- a/install/docker-container.mdx +++ b/install/docker-container.mdx @@ -2,19 +2,11 @@ title: 'Getting Started with Bazel Docker Container' --- - - -This page provides details on the contents of the Bazel container, how to build -the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel -inside the Bazel container, and how to build this project directly -from the host machine using the Bazel container with directory mounting. +This page provides details on the contents of the Bazel container, how to build the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel inside the Bazel container, and how to build this project directly from the host machine using the Bazel container with directory mounting. ## Build Abseil project from your host machine with directory mounting -The instructions in this section allow you to build using the Bazel container -with the sources checked out in your host environment. A container is started up -for each build command you execute. Build results are cached in your host -environment so they can be reused across builds. +The instructions in this section allow you to build using the Bazel container with the sources checked out in your host environment. A container is started up for each build command you execute. Build results are cached in your host environment so they can be reused across builds. Clone the project to a directory in your host machine. @@ -28,8 +20,7 @@ Create a folder that will have cached results to be shared across builds. mkdir -p /tmp/build_output/ ``` -Use the Bazel container to build the project and make the build -outputs available in the output folder in your host machine. +Use the Bazel container to build the project and make the build outputs available in the output folder in your host machine. ```posix-terminal docker run \ @@ -43,9 +34,7 @@ docker run \ build //absl/... ``` -Build the project with sanitizers by adding the `--config=asan|tsan|msan` build -flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or -MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. ```posix-terminal docker run \ @@ -61,10 +50,7 @@ docker run \ ## Build Abseil project from inside the container -The instructions in this section allow you to build using the Bazel container -with the sources inside the container. By starting a container at the beginning -of your development workflow and doing changes in the worskpace within the -container, build results will be cached. +The instructions in this section allow you to build using the Bazel container with the sources inside the container. By starting a container at the beginning of your development workflow and doing changes in the worskpace within the container, build results will be cached. Start a shell in the Bazel container: @@ -86,9 +72,7 @@ Do a regular build. ubuntu@5a99103747c6:~/abseil-cpp$ bazel build //absl/... ``` -Build the project with sanitizers by adding the `--config=asan|tsan|msan` -build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or -MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. ```posix-terminal ubuntu@5a99103747c6:~/abseil-cpp$ bazel build --config={asan | tsan | msan} -- //absl/... -//absl/types:variant_test diff --git a/install/ide.mdx b/install/ide.mdx index aa6210b3..5c6252da 100644 --- a/install/ide.mdx +++ b/install/ide.mdx @@ -2,56 +2,39 @@ title: 'Integrating Bazel with IDEs' --- +This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android Studio, and CLion (or build your own IDE plugin). It also includes links to installation and plugin details. +IDEs integrate with Bazel in a variety of ways, from features that allow Bazel executions from within the IDE, to awareness of Bazel structures such as syntax highlighting of the `BUILD` files. -This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android -Studio, and CLion (or build your own IDE plugin). It also includes links to -installation and plugin details. - -IDEs integrate with Bazel in a variety of ways, from features that allow Bazel -executions from within the IDE, to awareness of Bazel structures such as syntax -highlighting of the `BUILD` files. - -If you are interested in developing an editor or IDE plugin for Bazel, please -join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start -a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). +If you are interested in developing an editor or IDE plugin for Bazel, please join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). ## IDEs and editors ### IntelliJ, Android Studio, and CLion -[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and -CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). +[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). This is the open source version of the plugin used internally at Google. Features: -* Interop with language-specific plugins. Supported languages include Java, - Scala, and Python. -* Import `BUILD` files into the IDE with semantic awareness of Bazel targets. -* Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and - `.bzl`files -* Build, test, and execute binaries directly from the IDE -* Create configurations for debugging and running binaries. +- Interop with language-specific plugins. Supported languages include Java, Scala, and Python. +- Import `BUILD` files into the IDE with semantic awareness of Bazel targets. +- Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and `.bzl`files +- Build, test, and execute binaries directly from the IDE +- Create configurations for debugging and running binaries. To install, go to the IDE's plugin browser and search for `Bazel`. -To manually install older versions, download the zip files from JetBrains' -Plugin Repository and install the zip file from the IDE's plugin browser: +To manually install older versions, download the zip files from JetBrains' Plugin Repository and install the zip file from the IDE's plugin browser: -* [Android Studio - plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) -* [IntelliJ - plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) -* [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) +- [Android Studio plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) +- [IntelliJ plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) +- [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) ### Xcode -[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), -[Tulsi](https://tulsi.bazel.build), and -[XCHammer](https://github.com/pinterest/xchammer) generate Xcode -projects from Bazel `BUILD` files. +[rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), [Tulsi](https://tulsi.bazel.build), and [XCHammer](https://github.com/pinterest/xchammer) generate Xcode projects from Bazel `BUILD` files. ### Visual Studio Code @@ -59,21 +42,16 @@ Official plugin for VS Code. Features: -* Bazel Build Targets tree -* Starlark debugger for `.bzl` files during a build (set breakpoints, step - through code, inspect variables, and so on) +- Bazel Build Targets tree +- Starlark debugger for `.bzl` files during a build (set breakpoints, step through code, inspect variables, and so on) -Find [the plugin on the Visual Studio -marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) -or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). -The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). +Find [the plugin on the Visual Studio marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Atom -Find the [`language-bazel` package](https://atom.io/packages/language-bazel) -on the Atom package manager. +Find the [`language-bazel` package](https://atom.io/packages/language-bazel) on the Atom package manager. See also: [Autocomplete for Source Code](#autocomplete-for-source-code) @@ -85,31 +63,23 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Emacs -See [`bazelbuild/bazel-emacs-mode` on -GitHub](https://github.com/bazelbuild/emacs-bazel-mode) +See [`bazelbuild/bazel-emacs-mode` on GitHub](https://github.com/bazelbuild/emacs-bazel-mode) See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Visual Studio -[Lavender](https://github.com/tmandry/lavender) is an experimental project for -generating Visual Studio projects that use Bazel for building. +[Lavender](https://github.com/tmandry/lavender) is an experimental project for generating Visual Studio projects that use Bazel for building. ### Eclipse -[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) -is a set of plugins for importing Bazel packages into an Eclipse workspace as -Eclipse projects. +[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) is a set of plugins for importing Bazel packages into an Eclipse workspace as Eclipse projects. ## Autocomplete for Source Code ### C Language Family (C++, C, Objective-C, Objective-C++, and CUDA) -[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) -run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. -The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to -provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and -consumes the Protobuf output of Bazel to extract the compile commands. +[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and consumes the Protobuf output of Bazel to extract the compile commands. [`hedronvision/bazel-compile-commands-extractor`](https://github.com/hedronvision/bazel-compile-commands-extractor) enables autocomplete, smart navigation, quick fixes, and more in a wide variety of extensible editors, including VSCode, Vim, Emacs, Atom, and Sublime. It lets language servers, like clangd and ccls, and other types of tooling, draw upon Bazel's understanding of how `cc` and `objc` code will be compiled, including how it configures cross-compilation for other platforms. @@ -119,11 +89,8 @@ consumes the Protobuf output of Bazel to extract the compile commands. ## Automatically run build and test on file change -[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a -tool for building Bazel targets when source files change. +[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a tool for building Bazel targets when source files change. ## Building your own IDE plugin -Read the [**IDE support** blog -post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about -the Bazel APIs to use when building an IDE plugin. +Read the [**IDE support** blog post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about the Bazel APIs to use when building an IDE plugin. diff --git a/install/index.mdx b/install/index.mdx index 29b46972..898cd417 100644 --- a/install/index.mdx +++ b/install/index.mdx @@ -2,10 +2,7 @@ title: 'Installing Bazel' --- - - -This page describes the various platforms supported by Bazel and links -to the packages for more details. +This page describes the various platforms supported by Bazel and links to the packages for more details. [Bazelisk](/install/bazelisk) is the recommended way to install Bazel on [Ubuntu Linux](/install/ubuntu), [macOS](/install/os-x), and [Windows](/install/windows). @@ -13,24 +10,21 @@ You can find available Bazel releases on our [release page](/release). ## Community-supported packages -Bazel community members maintain these packages. The Bazel team doesn't -officially support them. Contact the package maintainers for support. +Bazel community members maintain these packages. The Bazel team doesn't officially support them. Contact the package maintainers for support. -* [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*&branch=edge&repo=&arch=&origin=&flagged=&maintainer=) -* [Arch Linux][arch] -* [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) -* [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) -* [FreeBSD](https://www.freshports.org/devel/bazel) -* [Homebrew](https://formulae.brew.sh/formula/bazel) -* [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) -* [openSUSE](/install/suse) -* [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) -* [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) +- [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*\&branch=edge\&repo=\&arch=\&origin=\&flagged=\&maintainer=) +- [Arch Linux](https://archlinux.org/packages/extra/x86_64/bazel/) +- [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) +- [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) +- [FreeBSD](https://www.freshports.org/devel/bazel) +- [Homebrew](https://formulae.brew.sh/formula/bazel) +- [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) +- [openSUSE](/install/suse) +- [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) +- [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) ## Community-supported architectures -* [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) +- [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) For other platforms, you can try to [compile from source](/install/compile-source). - -[arch]: https://archlinux.org/packages/extra/x86_64/bazel/ diff --git a/install/os-x.mdx b/install/os-x.mdx index b8d5bca2..980d2d81 100644 --- a/install/os-x.mdx +++ b/install/os-x.mdx @@ -2,23 +2,21 @@ title: 'Installing Bazel on macOS' --- - - This page describes how to install Bazel on macOS and set up your environment. You can install Bazel on macOS using one of the following methods: -* *Recommended*: [Use Bazelisk](/install/bazelisk) -* [Use Homebrew](#install-on-mac-os-x-homebrew) -* [Use the binary installer](#install-with-installer-mac-os-x) -* [Compile Bazel from source](/install/compile-source) +- *Recommended*: [Use Bazelisk](/install/bazelisk) +- [Use Homebrew](#install-on-mac-os-x-homebrew) +- [Use the binary installer](#install-with-installer-mac-os-x) +- [Compile Bazel from source](/install/compile-source) Bazel comes with two completion scripts. After installing Bazel, you can: -* Access the [bash completion script](/install/completion#bash) -* Install the [zsh completion script](/install/completion#zsh) +- Access the [bash completion script](/install/completion#bash) +- Install the [zsh completion script](/install/completion#zsh) -

Installing using Homebrew

+## Installing using Homebrew ### Step 1: Install Homebrew on macOS @@ -36,46 +34,37 @@ Install the Bazel package via Homebrew as follows: brew install bazel ``` -All set! You can confirm Bazel is installed successfully by running the -following command: +All set! You can confirm Bazel is installed successfully by running the following command: ```posix-terminal bazel --version ``` -Once installed, you can upgrade to a newer version of Bazel using the -following command: +Once installed, you can upgrade to a newer version of Bazel using the following command: ```posix-terminal brew upgrade bazel ``` -

Installing using the binary installer

+## Installing using the binary installer -The binary installers are on Bazel's -[GitHub releases page](https://github.com/bazelbuild/bazel/releases). +The binary installers are on Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary. Some additional libraries -must also be installed for Bazel to work. +The installer contains the Bazel binary. Some additional libraries must also be installed for Bazel to work. ### Step 1: Install Xcode command line tools -If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode -command line tools package by using `xcode-select`: +If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode command line tools package by using `xcode-select`: ```posix-terminal xcode-select --install ``` -Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS -SDK 8.1 installed on your system. +Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS SDK 8.1 installed on your system. -Download Xcode from the -[App Store](https://apps.apple.com/us/app/xcode/id497799835) or the -[Apple Developer site](https://developer.apple.com/download/more/?=xcode). +Download Xcode from the [App Store](https://apps.apple.com/us/app/xcode/id497799835) or the [Apple Developer site](https://developer.apple.com/download/more/?=xcode). -Once Xcode is installed, accept the license agreement for all users with the -following command: +Once Xcode is installed, accept the license agreement for all users with the following command: ```posix-terminal sudo xcodebuild -license accept @@ -83,59 +72,46 @@ sudo xcodebuild -license accept ### Step 2: Download the Bazel installer -Next, download the Bazel binary installer named -`bazel--installer-darwin-x86_64.sh` from the -[Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-<version>-installer-darwin-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). -**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, -you need to download the installer from the terminal using `curl`, replacing -the version variable with the Bazel version you want to download: +**On macOS Catalina or newer (macOS \>= 11)**, due to Apple's new app signing requirements, you need to download the installer from the terminal using `curl`, replacing the version variable with the Bazel version you want to download: ```posix-terminal export BAZEL_VERSION=5.2.0 -curl -fLO "https://github.com/bazelbuild/bazel/releases/download/{{ '' }}$BAZEL_VERSION{{ '' }}/bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" +curl -fLO "https://github.com/bazelbuild/bazel/releases/download/<var>$BAZEL_VERSION</var>/bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" ``` -This is a temporary workaround until the macOS release flow supports -signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). +This is a temporary workaround until the macOS release flow supports signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). ### Step 3: Run the installer Run the Bazel installer as follows: ```posix-terminal -chmod +x "bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" +chmod +x "bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" -./bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh --user +./bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and -sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see -additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. -If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that _**“bazel-real” cannot be -opened because the developer cannot be verified**_, you need to re-download -the installer from the terminal using `curl` as a workaround; see Step 2 above. +If you are **on macOS Catalina or newer (macOS \>= 11)** and get an error that ***“bazel-real” cannot be opened because the developer cannot be verified***, you need to re-download the installer from the terminal using `curl` as a workaround; see Step 2 above. ### Step 4: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel -executable is installed in your `HOME/bin` directory. -It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `<var>HOME</var>/bin` directory. It's a good idea to add this directory to your default paths, as follows: ```posix-terminal -export PATH="{{ '' }}PATH{{ '' }}:{{ '' }}HOME{{ '' }}/bin" +export PATH="<var>PATH</var>:<var>HOME</var>/bin" ``` -You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` -file. +You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` file. -All set! You can confirm Bazel is installed successfully by running the -following command: +All set! You can confirm Bazel is installed successfully by running the following command: ```posix-terminal bazel --version ``` -To update to a newer release of Bazel, download and install the desired version. +To update to a newer release of Bazel, download and install the desired version. diff --git a/install/suse.mdx b/install/suse.mdx index 741b108f..73debc99 100644 --- a/install/suse.mdx +++ b/install/suse.mdx @@ -2,23 +2,17 @@ title: 'Installing Bazel on openSUSE Tumbleweed & Leap' --- - - This page describes how to install Bazel on openSUSE Tumbleweed and Leap. -`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues -using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). +`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). -Packages are provided for openSUSE Tumbleweed and Leap. You can find all -available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93&baseproject=ALL&q=bazel). +Packages are provided for openSUSE Tumbleweed and Leap. You can find all available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93\&baseproject=ALL\&q=bazel). The commands below must be run either via `sudo` or while logged in as `root`. ## Installing Bazel on openSUSE -Run the following commands to install the package. If you need a specific -version, you can install it via the specific `bazelXXX` package, otherwise, -just `bazel` is enough: +Run the following commands to install the package. If you need a specific version, you can install it via the specific `bazelXXX` package, otherwise, just `bazel` is enough: To install the latest version of Bazel, run: @@ -26,9 +20,7 @@ To install the latest version of Bazel, run: zypper install bazel ``` -You can also install a specific version of Bazel by specifying the package -version with `bazelversion`. For example, to install -Bazel 4.2, run: +You can also install a specific version of Bazel by specifying the package version with `bazel<var>version</var>`. For example, to install Bazel 4.2, run: ```posix-terminal zypper install bazel4.2 diff --git a/install/ubuntu.mdx b/install/ubuntu.mdx index 73d42751..b498f42f 100644 --- a/install/ubuntu.mdx +++ b/install/ubuntu.mdx @@ -2,37 +2,30 @@ title: 'Installing Bazel on Ubuntu' --- - - -This page describes the options for installing Bazel on Ubuntu. -It also provides links to the Bazel completion scripts and the binary installer, -if needed as a backup option (for example, if you don't have admin access). +This page describes the options for installing Bazel on Ubuntu. It also provides links to the Bazel completion scripts and the binary installer, if needed as a backup option (for example, if you don't have admin access). Supported Ubuntu Linux platforms: -* 22.04 (LTS) -* 20.04 (LTS) -* 18.04 (LTS) +- 22.04 (LTS) +- 20.04 (LTS) +- 18.04 (LTS) -Bazel should be compatible with other Ubuntu releases and Debian -"stretch" and above, but is untested and not guaranteed to work. +Bazel should be compatible with other Ubuntu releases and Debian "stretch" and above, but is untested and not guaranteed to work. Install Bazel on Ubuntu using one of the following methods: -* *Recommended*: [Use Bazelisk](/install/bazelisk) -* [Use our custom APT repository](#install-on-ubuntu) -* [Use the binary installer](#binary-installer) -* [Use the Bazel Docker container](#docker-container) -* [Compile Bazel from source](/install/compile-source) +- *Recommended*: [Use Bazelisk](/install/bazelisk) +- [Use our custom APT repository](#install-on-ubuntu) +- [Use the binary installer](#binary-installer) +- [Use the Bazel Docker container](#docker-container) +- [Compile Bazel from source](/install/compile-source) -**Note:** For Arm-based systems, the APT repository does not contain an `arm64` -release, and there is no binary installer available. Either use Bazelisk or -compile from source. +**Note:** For Arm-based systems, the APT repository does not contain an `arm64` release, and there is no binary installer available. Either use Bazelisk or compile from source. Bazel comes with two completion scripts. After installing Bazel, you can: -* Access the [bash completion script](/install/completion#bash) -* Install the [zsh completion script](/install/completion#zsh) +- Access the [bash completion script](/install/completion#bash) +- Install the [zsh completion script](/install/completion#zsh) ## Using Bazel's apt repository @@ -43,16 +36,14 @@ Bazel comes with two completion scripts. After installing Bazel, you can: ```posix-terminal sudo apt install apt-transport-https curl gnupg -y -curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg +curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg sudo mv bazel-archive-keyring.gpg /usr/share/keyrings echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list ``` -The component name "jdk1.8" is kept only for legacy reasons and doesn't relate -to supported or included JDK versions. Bazel releases are Java-version agnostic. -Changing the "jdk1.8" component name would break existing users of the repo. +The component name "jdk1.8" is kept only for legacy reasons and doesn't relate to supported or included JDK versions. Bazel releases are Java-version agnostic. Changing the "jdk1.8" component name would break existing users of the repo. ### Step 2: Install and update Bazel @@ -66,18 +57,13 @@ Once installed, you can upgrade to a newer version of Bazel as part of your norm sudo apt update && sudo apt full-upgrade ``` -The `bazel` package always installs the latest stable version of Bazel. You -can install specific, older versions of Bazel in addition to the latest one, -such as this: +The `bazel` package always installs the latest stable version of Bazel. You can install specific, older versions of Bazel in addition to the latest one, such as this: ```posix-terminal sudo apt install bazel-1.0.0 ``` -This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This -can be useful if you need a specific version of Bazel to build a project, for -example because it uses a `.bazelversion` file to explicitly state with which -Bazel version it should be built. +This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This can be useful if you need a specific version of Bazel to build a project, for example because it uses a `.bazelversion` file to explicitly state with which Bazel version it should be built. Optionally, you can set `bazel` to a specific version by creating a symlink: @@ -89,8 +75,7 @@ bazel --version # 1.0.0 ### Step 3: Install a JDK (optional) -Bazel includes a private, bundled JRE as its runtime and doesn't require you to -install any specific version of Java. +Bazel includes a private, bundled JRE as its runtime and doesn't require you to install any specific version of Java. However, if you want to build Java code using Bazel, you have to install a JDK. @@ -100,14 +85,11 @@ sudo apt install default-jdk ## Using the binary installer -Generally, you should use the apt repository, but the binary installer -can be useful if you don't have admin permissions on your machine or -can't add custom repositories. +Generally, you should use the apt repository, but the binary installer can be useful if you don't have admin permissions on your machine or can't add custom repositories. The binary installers can be downloaded from Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary and extracts it into your `$HOME/bin` -folder. Some additional libraries must be installed manually for Bazel to work. +The installer contains the Bazel binary and extracts it into your `$HOME/bin` folder. Some additional libraries must be installed manually for Bazel to work. ### Step 1: Install required packages @@ -125,42 +107,34 @@ sudo apt-get install default-jdk ### Step 2: Run the installer -Next, download the Bazel binary installer named `bazel-version-installer-linux-x86_64.sh` -from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-<var>version</var>-installer-linux-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). Run it as follows: ```posix-terminal -chmod +x bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh +chmod +x bazel-<var>version</var>-installer-linux-x86_64.sh -./bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh --user +./bazel-<var>version</var>-installer-linux-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and -sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see -additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. ### Step 3: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel -executable is installed in your `$HOME/bin` directory. -It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `$HOME/bin` directory. It's a good idea to add this directory to your default paths, as follows: ```posix-terminal export PATH="$PATH:$HOME/bin" ``` -You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it -permanent. +You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it permanent. ## Using the Bazel Docker container -We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. -You can use the Docker container as follows: +We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. You can use the Docker container as follows: ``` -$ docker pull gcr.io/bazel-public/bazel: +$ docker pull gcr.io/bazel-public/bazel:<bazel version> ``` The Docker container is built by [these steps](https://github.com/bazelbuild/continuous-integration/tree/master/bazel/oci). - diff --git a/install/windows.mdx b/install/windows.mdx index b38c0986..2f808353 100644 --- a/install/windows.mdx +++ b/install/windows.mdx @@ -2,16 +2,11 @@ title: 'Installing Bazel on Windows' --- - - -This page describes the requirements and steps to install Bazel on Windows. -It also includes troubleshooting and other ways to install Bazel, such as -using Chocolatey or Scoop. +This page describes the requirements and steps to install Bazel on Windows. It also includes troubleshooting and other ways to install Bazel, such as using Chocolatey or Scoop. ## Installing Bazel -This section covers the prerequisites, environment setup, and detailed -steps during installation on Windows. +This section covers the prerequisites, environment setup, and detailed steps during installation on Windows. ### Check your system @@ -19,33 +14,31 @@ Recommended: 64 bit Windows 10, version 1703 (Creators Update) or newer To check your Windows version: -* Click the Start button. -* Type `winver` in the search box and press Enter. -* You should see the About Windows box with your Windows version information. +- Click the Start button. +- Type `winver` in the search box and press Enter. +- You should see the About Windows box with your Windows version information. ### Install the prerequisites -* [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) +- [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) ### Download Bazel *Recommended*: [Use Bazelisk](/install/bazelisk) - Alternatively you can: -* [Download the Bazel binary (`bazel-version-windows-x86_64.exe`) from - GitHub](https://github.com/bazelbuild/bazel/releases). -* [Install Bazel from Chocolatey](#chocolately) -* [Install Bazel from Scoop](#scoop) -* [Build Bazel from source](/install/compile-source) +- [Download the Bazel binary (`bazel-<var>version</var>-windows-x86_64.exe`) from GitHub](https://github.com/bazelbuild/bazel/releases). +- [Install Bazel from Chocolatey](#chocolately) +- [Install Bazel from Scoop](#scoop) +- [Build Bazel from source](/install/compile-source) ### Set up your environment To make Bazel easily accessible from command prompts or PowerShell by default, you can rename the Bazel binary to `bazel.exe` and add it to your default paths. ```posix-terminal -set PATH=%PATH%;{{ '' }}path to the Bazel binary{{ '' }} +set PATH=%PATH%;<var>path to the Bazel binary</var> ``` You can also change your system `PATH` environment variable to make it permanent. Check out how to [set environment variables](/configure/windows#set-environment-variables). @@ -57,64 +50,57 @@ You can also change your system `PATH` environment variable to make it permanent To check the installation is correct, try to run: ```posix-terminal -bazel {{ '' }}version{{ '' }} +bazel <var>version</var> ``` Next, you can check out more tips and guidance here: -* [Installing compilers and language runtimes](#install-compilers) -* [Troubleshooting](#troubleshooting) -* [Best practices on Windows](/configure/windows#best-practices) -* [Tutorials](/start/#tutorials) +- [Installing compilers and language runtimes](#install-compilers) +- [Troubleshooting](#troubleshooting) +- [Best practices on Windows](/configure/windows#best-practices) +- [Tutorials](/start/#tutorials) ## Installing compilers and language runtimes Depending on which languages you want to build, you will need: -* [MSYS2 x86_64](https://www.msys2.org/) +- [MSYS2 x86\_64](https://www.msys2.org/) - MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix - tools (like `grep`, `tar`, `git`). + MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix tools (like `grep`, `tar`, `git`). - You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are - `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an - error if a build target needs Bash but Bazel could not locate it. + You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an error if a build target needs Bash but Bazel could not locate it. -* Common MSYS2 packages +- Common MSYS2 packages - You will likely need these to build and run targets that depend on Bash. MSYS2 does not install - these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). + You will likely need these to build and run targets that depend on Bash. MSYS2 does not install these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). - Open the MSYS2 terminal and run this command: + Open the MSYS2 terminal and run this command: - ```posix-terminal - pacman -S zip unzip patch diffutils git - ``` + ```posix-terminal + pacman -S zip unzip patch diffutils git + ``` - Optional: If you want to use Bazel from CMD or Powershell and still be able - to use Bash tools, make sure to add - `MSYS2_INSTALL_PATH/usr/bin` to your - `PATH` environment variable. + Optional: If you want to use Bazel from CMD or Powershell and still be able to use Bash tools, make sure to add `<var>MSYS2_INSTALL_PATH</var>/usr/bin` to your `PATH` environment variable. -* [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) +- [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) - You will need this to build C++ code on Windows. + You will need this to build C++ code on Windows. - Also supported: + Also supported: - * Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK + - Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK -* [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) +- [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) - You will need this to build Java code on Windows. + You will need this to build Java code on Windows. - Also supported: Java 8, 9, and 10 + Also supported: Java 8, 9, and 10 -* [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) +- [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) - You will need this to build Python code on Windows. + You will need this to build Python code on Windows. - Also supported: Python 2.7 or newer for Windows x86-64 + Also supported: Python 2.7 or newer for Windows x86-64 ## Troubleshooting @@ -122,11 +108,11 @@ Depending on which languages you want to build, you will need: **Possible reasons**: -* you installed MSYS2 not under the default install path +- you installed MSYS2 not under the default install path -* you installed MSYS2 i686 instead of MSYS2 x86\_64 +- you installed MSYS2 i686 instead of MSYS2 x86\_64 -* you installed MSYS instead of MSYS2 +- you installed MSYS instead of MSYS2 **Solution**: @@ -134,104 +120,95 @@ Ensure you installed MSYS2 x86\_64. If that doesn't help: -1. Go to Start Menu > Settings. +1. Go to Start Menu \> Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for <username>"), and click the "New..." - button below it. +3. Look at the list on the top ("User variables for \<username\>"), and click the "New\..." button below it. -4. For "Variable name", enter `BAZEL_SH` +4. For "Variable name", enter `BAZEL_SH` -5. Click "Browse File..." +5. Click "Browse File..." -6. Navigate to the MSYS2 directory, then to `usr\bin` below it. +6. Navigate to the MSYS2 directory, then to `usr\bin` below it. - For example, this might be `C:\msys64\usr\bin` on your system. + For example, this might be `C:\msys64\usr\bin` on your system. -7. Select the `bash.exe` or `bash` file and click OK +7. Select the `bash.exe` or `bash` file and click OK -8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. +8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. ### Bazel does not find Visual Studio or Visual C++ **Possible reasons**: -* you installed multiple versions of Visual Studio +- you installed multiple versions of Visual Studio -* you installed and removed various versions of Visual Studio +- you installed and removed various versions of Visual Studio -* you installed various versions of the Windows SDK +- you installed various versions of the Windows SDK -* you installed Visual Studio not under the default install path +- you installed Visual Studio not under the default install path **Solution**: -1. Go to Start Menu > Settings. +1. Go to Start Menu \> Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for <username>"), and click the "New..." - button below it. +3. Look at the list on the top ("User variables for \<username\>"), and click the "New\..." button below it. -4. For "Variable name", enter `BAZEL_VC` +4. For "Variable name", enter `BAZEL_VC` -5. Click "Browse Directory..." +5. Click "Browse Directory..." -6. Navigate to the `VC` directory of Visual Studio. +6. Navigate to the `VC` directory of Visual Studio. - For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` - on your system. + For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` on your system. -7. Select the `VC` folder and click OK +7. Select the `VC` folder and click OK -8. The "Variable value" field now has the path to `VC`. Click OK to close the window. +8. The "Variable value" field now has the path to `VC`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. ## Other ways to install Bazel ### Using Chocolatey -1. Install the [Chocolatey](https://chocolatey.org) package manager +1. Install the [Chocolatey](https://chocolatey.org) package manager -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - choco install bazel - ``` + ```posix-terminal + choco install bazel + ``` - This command will install the latest available version of Bazel and - its dependencies, such as the MSYS2 shell. This will not install Visual C++ - though. + This command will install the latest available version of Bazel and its dependencies, such as the MSYS2 shell. This will not install Visual C++ though. -See [Chocolatey installation and package maintenance -guide](/contribute/windows-chocolatey-maintenance) for more -information about the Chocolatey package. +See [Chocolatey installation and package maintenance guide](/contribute/windows-chocolatey-maintenance) for more information about the Chocolatey package. ### Using Scoop -1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: +1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: - ```posix-terminal - iex (new-object net.webclient).downloadstring('https://get.scoop.sh') - ``` + ```posix-terminal + iex (new-object net.webclient).downloadstring('https://get.scoop.sh') + ``` -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - scoop install bazel - ``` + ```posix-terminal + scoop install bazel + ``` -See [Scoop installation and package maintenance -guide](/contribute/windows-scoop-maintenance) for more -information about the Scoop package. +See [Scoop installation and package maintenance guide](/contribute/windows-scoop-maintenance) for more information about the Scoop package. ### Build from source diff --git a/migrate/index.mdx b/migrate/index.mdx index 2a574067..8683aedb 100644 --- a/migrate/index.mdx +++ b/migrate/index.mdx @@ -2,9 +2,7 @@ title: 'Migrating to Bazel' --- - - This page links to migration guides for Bazel. -* [Maven](/migrate/maven) -* [Xcode](/migrate/xcode) +- [Maven](/migrate/maven) +- [Xcode](/migrate/xcode) diff --git a/migrate/maven.mdx b/migrate/maven.mdx index 38aaffc0..0673ef53 100644 --- a/migrate/maven.mdx +++ b/migrate/maven.mdx @@ -2,55 +2,34 @@ title: 'Migrating from Maven to Bazel' --- +This page describes how to migrate from Maven to Bazel, including the prerequisites and installation steps. It describes the differences between Maven and Bazel, and provides a migration example using the Guava project. +When migrating from any build tool to Bazel, it's best to have both build tools running in parallel until you have fully migrated your development team, CI system, and any other relevant systems. You can run Maven and Bazel in the same repository. -This page describes how to migrate from Maven to Bazel, including the -prerequisites and installation steps. It describes the differences between Maven -and Bazel, and provides a migration example using the Guava project. - -When migrating from any build tool to Bazel, it's best to have both build tools -running in parallel until you have fully migrated your development team, CI -system, and any other relevant systems. You can run Maven and Bazel in the same -repository. - -Note: While Bazel supports downloading and publishing Maven artifacts with -[rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) -, it does not directly support Maven-based plugins. Maven plugins can't be -directly run by Bazel since there's no Maven compatibility layer. +Note: While Bazel supports downloading and publishing Maven artifacts with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) , it does not directly support Maven-based plugins. Maven plugins can't be directly run by Bazel since there's no Maven compatibility layer. ## Before you begin -* [Install Bazel](/install) if it's not yet installed. -* If you're new to Bazel, go through the tutorial [Introduction to Bazel: - Build Java](/start/java) before you start migrating. The tutorial explains - Bazel's concepts, structure, and label syntax. +- [Install Bazel](/install) if it's not yet installed. +- If you're new to Bazel, go through the tutorial [Introduction to Bazel: Build Java](/start/java) before you start migrating. The tutorial explains Bazel's concepts, structure, and label syntax. ## Differences between Maven and Bazel -* Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files - and multiple targets per `BUILD` file, allowing for builds that are more - incremental than Maven's. -* Maven takes charge of steps for the deployment process. Bazel does not - automate deployment. -* Bazel enables you to express dependencies between languages. -* As you add new sections to the project, with Bazel you may need to add new - `BUILD` files. Best practice is to add a `BUILD` file to each new Java - package. +- Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files and multiple targets per `BUILD` file, allowing for builds that are more incremental than Maven's. +- Maven takes charge of steps for the deployment process. Bazel does not automate deployment. +- Bazel enables you to express dependencies between languages. +- As you add new sections to the project, with Bazel you may need to add new `BUILD` files. Best practice is to add a `BUILD` file to each new Java package. ## Migrate from Maven to Bazel The steps below describe how to migrate your project to Bazel: -1. [Create the MODULE.bazel file](#1-build) -2. [Create one BUILD file](#2-build) -3. [Create more BUILD files](#3-build) -4. [Build using Bazel](#4-build) +1. [Create the MODULE.bazel file](#1-build) +2. [Create one BUILD file](#2-build) +3. [Create more BUILD files](#3-build) +4. [Build using Bazel](#4-build) -Examples below come from a migration of the [Guava -project](https://github.com/google/guava) from Maven to Bazel. The -Guava project used is release `v31.1`. The examples using Guava do not walk -through each step in the migration, but they do show the files and contents that -are generated or added manually for the migration. +Examples below come from a migration of the [Guava project](https://github.com/google/guava) from Maven to Bazel. The Guava project used is release `v31.1`. The examples using Guava do not walk through each step in the migration, but they do show the files and contents that are generated or added manually for the migration. ``` $ git clone https://github.com/google/guava.git && cd guava @@ -59,22 +38,13 @@ $ git checkout v31.1 ### 1. Create the MODULE.bazel file -Create a file named `MODULE.bazel` at the root of your project. If your project -has no external dependencies, this file can be empty. +Create a file named `MODULE.bazel` at the root of your project. If your project has no external dependencies, this file can be empty. -If your project depends on files or packages that are not in one of the -project's directories, specify these external dependencies in the MODULE.bazel -file. You can use `rules_jvm_external` to manage dependencies from Maven. For -instructions about using this ruleset, see [the -README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) -. +If your project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the MODULE.bazel file. You can use `rules_jvm_external` to manage dependencies from Maven. For instructions about using this ruleset, see [the README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) . #### Guava project example: external dependencies -You can list the external dependencies of the [Guava -project](https://github.com/google/guava) with the -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) -ruleset. +You can list the external dependencies of the [Guava project](https://github.com/google/guava) with the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) ruleset. Add the following snippet to the `MODULE.bazel` file: @@ -98,89 +68,74 @@ use_repo(maven, "maven") ### 2. Create one BUILD file -Now that you have your workspace defined and external dependencies (if -applicable) listed, you need to create `BUILD` files to describe how your -project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use -many `BUILD` files to build a project. These files specify multiple build -targets, which allow Bazel to produce incremental builds. - -Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of -your project and using it to do an initial build using Bazel. Then, you refine -your build by adding more `BUILD` files with more granular targets. - -1. In the same directory as your `MODULE.bazel` file, create a text file and - name it `BUILD`. - -2. In this `BUILD` file, use the appropriate rule to create one target to build - your project. Here are some tips: - - * Use the appropriate rule: - * To build projects with a single Maven module, use the - `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - ) - ``` - - * To build projects with multiple Maven modules, use the - `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob([ - "Module1/src/main/java/**/*.java", - "Module2/src/main/java/**/*.java", - ... - ]), - resources = glob([ - "Module1/src/main/resources/**", - "Module2/src/main/resources/**", - ... - ]), - deps = ["//:all-external-targets"], - ) - ``` - - * To build binaries, use the `java_binary` rule: - - ```python - java_binary( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - main_class = "com.example.Main" - ) - ``` - - * Specify the attributes: - * `name`: Give the target a meaningful name. In the examples - above, the target is called "everything." - * `srcs`: Use globbing to list all .java files in your project. - * `resources`: Use globbing to list all resources in your project. - * `deps`: You need to determine which external dependencies your - project needs. - * Take a look at the [example below of this top-level BUILD - file](#guava-2) from the migration of the Guava project. - -3. Now that you have a `BUILD` file at the root of your project, build your - project to ensure that it works. On the command line, from your workspace - directory, use `bazel build //:everything` to build your project with Bazel. - - The project has now been successfully built with Bazel. You will need to add - more `BUILD` files to allow incremental builds of the project. +Now that you have your workspace defined and external dependencies (if applicable) listed, you need to create `BUILD` files to describe how your project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use many `BUILD` files to build a project. These files specify multiple build targets, which allow Bazel to produce incremental builds. + +Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of your project and using it to do an initial build using Bazel. Then, you refine your build by adding more `BUILD` files with more granular targets. + +1. In the same directory as your `MODULE.bazel` file, create a text file and name it `BUILD`. + +2. In this `BUILD` file, use the appropriate rule to create one target to build your project. Here are some tips: + + - Use the appropriate rule: + + - To build projects with a single Maven module, use the `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + ) + ``` + + - To build projects with multiple Maven modules, use the `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob([ + "Module1/src/main/java/**/*.java", + "Module2/src/main/java/**/*.java", + ... + ]), + resources = glob([ + "Module1/src/main/resources/**", + "Module2/src/main/resources/**", + ... + ]), + deps = ["//:all-external-targets"], + ) + ``` + + - To build binaries, use the `java_binary` rule: + + ```python + java_binary( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + main_class = "com.example.Main" + ) + ``` + + - Specify the attributes: + + - `name`: Give the target a meaningful name. In the examples above, the target is called "everything." + - `srcs`: Use globbing to list all .java files in your project. + - `resources`: Use globbing to list all resources in your project. + - `deps`: You need to determine which external dependencies your project needs. + + - Take a look at the [example below of this top-level BUILD file](#guava-2) from the migration of the Guava project. + +3. Now that you have a `BUILD` file at the root of your project, build your project to ensure that it works. On the command line, from your workspace directory, use `bazel build //:everything` to build your project with Bazel. + + The project has now been successfully built with Bazel. You will need to add more `BUILD` files to allow incremental builds of the project. #### Guava project example: start with one BUILD file -When migrating the Guava project to Bazel, initially one `BUILD` file is used to -build the entire project. Here are the contents of this initial `BUILD` file in -the workspace directory: +When migrating the Guava project to Bazel, initially one `BUILD` file is used to build the entire project. Here are the contents of this initial `BUILD` file in the workspace directory: ```python java_library( @@ -202,40 +157,25 @@ java_library( ### 3. Create more BUILD files (optional) -Bazel does work with just one `BUILD file`, as you saw after completing your -first build. You should still consider breaking the build into smaller chunks by -adding more `BUILD` files with granular targets. +Bazel does work with just one `BUILD file`, as you saw after completing your first build. You should still consider breaking the build into smaller chunks by adding more `BUILD` files with granular targets. -Multiple `BUILD` files with multiple targets will give the build increased -granularity, allowing: +Multiple `BUILD` files with multiple targets will give the build increased granularity, allowing: -* increased incremental builds of the project, -* increased parallel execution of the build, -* better maintainability of the build for future users, and -* control over visibility of targets between packages, which can prevent - issues such as libraries containing implementation details leaking into - public APIs. +- increased incremental builds of the project, +- increased parallel execution of the build, +- better maintainability of the build for future users, and +- control over visibility of targets between packages, which can prevent issues such as libraries containing implementation details leaking into public APIs. Tips for adding more `BUILD` files: -* You can start by adding a `BUILD` file to each Java package. Start with Java - packages that have the fewest dependencies and work you way up to packages - with the most dependencies. -* As you add `BUILD` files and specify targets, add these new targets to the - `deps` sections of targets that depend on them. Note that the `glob()` - function does not cross package boundaries, so as the number of packages - grows the files matched by `glob()` will shrink. -* Any time you add a `BUILD` file to a `main` directory, ensure that you add a - `BUILD` file to the corresponding `test` directory. -* Take care to limit visibility properly between packages. -* To simplify troubleshooting errors in your setup of `BUILD` files, ensure - that the project continues to build with Bazel as you add each build file. - Run `bazel build //...` to ensure all of your targets still build. +- You can start by adding a `BUILD` file to each Java package. Start with Java packages that have the fewest dependencies and work you way up to packages with the most dependencies. +- As you add `BUILD` files and specify targets, add these new targets to the `deps` sections of targets that depend on them. Note that the `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. +- Any time you add a `BUILD` file to a `main` directory, ensure that you add a `BUILD` file to the corresponding `test` directory. +- Take care to limit visibility properly between packages. +- To simplify troubleshooting errors in your setup of `BUILD` files, ensure that the project continues to build with Bazel as you add each build file. Run `bazel build //...` to ensure all of your targets still build. ### 4. Build using Bazel -You've been building using Bazel as you add `BUILD` files to validate the setup -of the build. +You've been building using Bazel as you add `BUILD` files to validate the setup of the build. -When you have `BUILD` files at the desired granularity, you can use Bazel to -produce all of your builds. +When you have `BUILD` files at the desired granularity, you can use Bazel to produce all of your builds. diff --git a/migrate/xcode.mdx b/migrate/xcode.mdx index 986cd115..4cd229c9 100644 --- a/migrate/xcode.mdx +++ b/migrate/xcode.mdx @@ -2,243 +2,163 @@ title: 'Migrating from Xcode to Bazel' --- - - -This page describes how to build or test an Xcode project with Bazel. It -describes the differences between Xcode and Bazel, and provides the steps for -converting an Xcode project to a Bazel project. It also provides troubleshooting -solutions to address common errors. +This page describes how to build or test an Xcode project with Bazel. It describes the differences between Xcode and Bazel, and provides the steps for converting an Xcode project to a Bazel project. It also provides troubleshooting solutions to address common errors. ## Differences between Xcode and Bazel -* Bazel requires you to explicitly specify every build target and its - dependencies, plus the corresponding build settings via build rules. +- Bazel requires you to explicitly specify every build target and its dependencies, plus the corresponding build settings via build rules. -* Bazel requires all files on which the project depends to be present within - the workspace directory or specified as dependencies in the `MODULE.bazel` - file. +- Bazel requires all files on which the project depends to be present within the workspace directory or specified as dependencies in the `MODULE.bazel` file. -* When building Xcode projects with Bazel, the `BUILD` file(s) become the - source of truth. If you work on the project in Xcode, you must generate a - new version of the Xcode project that matches the `BUILD` files using - [rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) - whenever you update the `BUILD` files. Certain changes to the `BUILD` files - such as adding dependencies to a target don't require regenerating the - project which can speed up development. If you're not using Xcode, the - `bazel build` and `bazel test` commands provide build and test capabilities - with certain limitations described later in this guide. +- When building Xcode projects with Bazel, the `BUILD` file(s) become the source of truth. If you work on the project in Xcode, you must generate a new version of the Xcode project that matches the `BUILD` files using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) whenever you update the `BUILD` files. Certain changes to the `BUILD` files such as adding dependencies to a target don't require regenerating the project which can speed up development. If you're not using Xcode, the `bazel build` and `bazel test` commands provide build and test capabilities with certain limitations described later in this guide. ## Before you begin Before you begin, do the following: -1. [Install Bazel](/install) if you have not already done so. +1. [Install Bazel](/install) if you have not already done so. -2. If you're not familiar with Bazel and its concepts, complete the [iOS app - tutorial](/start/ios-app)). You should understand the Bazel workspace, - including the `MODULE.bazel` and `BUILD` files, as well as the concepts of - targets, build rules, and Bazel packages. +2. If you're not familiar with Bazel and its concepts, complete the [iOS app tutorial](/start/ios-app)). You should understand the Bazel workspace, including the `MODULE.bazel` and `BUILD` files, as well as the concepts of targets, build rules, and Bazel packages. -3. Analyze and understand the project's dependencies. +3. Analyze and understand the project's dependencies. ### Analyze project dependencies -Unlike Xcode, Bazel requires you to explicitly declare all dependencies for -every target in the `BUILD` file. +Unlike Xcode, Bazel requires you to explicitly declare all dependencies for every target in the `BUILD` file. -For more information on external dependencies, see [Working with external -dependencies](/docs/external). +For more information on external dependencies, see [Working with external dependencies](/docs/external). ## Build or test an Xcode project with Bazel To build or test an Xcode project with Bazel, do the following: -1. [Create the `MODULE.bazel` file](#create-workspace) +1. [Create the `MODULE.bazel` file](#create-workspace) -2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) +2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) -3. [Create a `BUILD` file:](#create-build-file) +3. [Create a `BUILD` file:](#create-build-file) - a. [Add the application target](#add-app-target) + a. [Add the application target](#add-app-target) - b. [(Optional) Add the test target(s)](#add-test-target) + b. [(Optional) Add the test target(s)](#add-test-target) - c. [Add the library target(s)](#add-library-target) + c. [Add the library target(s)](#add-library-target) -4. [(Optional) Granularize the build](#granularize-build) +4. [(Optional) Granularize the build](#granularize-build) -5. [Run the build](#run-build) +5. [Run the build](#run-build) -6. [Generate the Xcode project with rules_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) +6. [Generate the Xcode project with rules\_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) ### Step 1: Create the `MODULE.bazel` file -Create a `MODULE.bazel` file in a new directory. This directory becomes the -Bazel workspace root. If the project uses no external dependencies, this file -can be empty. If the project depends on files or packages that are not in one of -the project's directories, specify these external dependencies in the -`MODULE.bazel` file. +Create a `MODULE.bazel` file in a new directory. This directory becomes the Bazel workspace root. If the project uses no external dependencies, this file can be empty. If the project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the `MODULE.bazel` file. -Note: Place the project source code within the directory tree containing the -`MODULE.bazel` file. +Note: Place the project source code within the directory tree containing the `MODULE.bazel` file. ### Step 2: (Experimental) Integrate SwiftPM dependencies -To integrate SwiftPM dependencies into the Bazel workspace with -[swift_bazel](https://github.com/cgrindel/swift_bazel), you must -convert them into Bazel packages as described in the [following -tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) -. +To integrate SwiftPM dependencies into the Bazel workspace with [swift\_bazel](https://github.com/cgrindel/swift_bazel), you must convert them into Bazel packages as described in the [following tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) . -Note: SwiftPM support is a manual process with many variables. SwiftPM -integration with Bazel has not been fully verified and is not officially -supported. +Note: SwiftPM support is a manual process with many variables. SwiftPM integration with Bazel has not been fully verified and is not officially supported. ### Step 3: Create a `BUILD` file -Once you have defined the workspace and external dependencies, you need to -create a `BUILD` file that tells Bazel how the project is structured. Create the -`BUILD` file at the root of the Bazel workspace and configure it to do an -initial build of the project as follows: +Once you have defined the workspace and external dependencies, you need to create a `BUILD` file that tells Bazel how the project is structured. Create the `BUILD` file at the root of the Bazel workspace and configure it to do an initial build of the project as follows: -* [Step 3a: Add the application target](#step-3a-add-the-application-target) -* [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) -* [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) +- [Step 3a: Add the application target](#step-3a-add-the-application-target) +- [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) +- [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) -**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, -packages, and targets](/concepts/build-ref). +**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, packages, and targets](/concepts/build-ref). #### Step 3a: Add the application target -Add a -[`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) -or an -[`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) -rule target. This target builds a macOS or iOS application bundle, respectively. -In the target, specify the following at the minimum: +Add a [`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) or an [`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) rule target. This target builds a macOS or iOS application bundle, respectively. In the target, specify the following at the minimum: -* `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the - binary. +- `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the binary. -* `provisioning_profile` - provisioning profile from your Apple Developer - account (if building for an iOS device device). +- `provisioning_profile` - provisioning profile from your Apple Developer account (if building for an iOS device device). -* `families` (iOS only) - whether to build the application for iPhone, iPad, - or both. +- `families` (iOS only) - whether to build the application for iPhone, iPad, or both. -* `infoplists` - list of .plist files to merge into the final Info.plist file. +- `infoplists` - list of .plist files to merge into the final Info.plist file. -* `minimum_os_version` - the minimum version of macOS or iOS that the - application supports. This ensures Bazel builds the application with the - correct API levels. +- `minimum_os_version` - the minimum version of macOS or iOS that the application supports. This ensures Bazel builds the application with the correct API levels. #### Step 3b: (Optional) Add the test target(s) -Bazel's [Apple build -rules](https://github.com/bazelbuild/rules_apple) support running -unit and UI tests on all Apple platforms. Add test targets as follows: +Bazel's [Apple build rules](https://github.com/bazelbuild/rules_apple) support running unit and UI tests on all Apple platforms. Add test targets as follows: -* [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) - to run library-based and application-based unit tests on a macOS. +- [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) to run library-based and application-based unit tests on a macOS. -* [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) - to build and run library-based unit tests on iOS. +- [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) to build and run library-based unit tests on iOS. -* [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) - to build and run user interface tests in the iOS simulator. +- [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) to build and run user interface tests in the iOS simulator. -* Similar test rules exist for - [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), - [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) - and - [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). +- Similar test rules exist for [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) and [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). -At the minimum, specify a value for the `minimum_os_version` attribute. While -other packaging attributes, such as `bundle_identifier` and `infoplists`, -default to most commonly used values, ensure that those defaults are compatible -with the project and adjust them as necessary. For tests that require the iOS -simulator, also specify the `ios_application` target name as the value of the -`test_host` attribute. +At the minimum, specify a value for the `minimum_os_version` attribute. While other packaging attributes, such as `bundle_identifier` and `infoplists`, default to most commonly used values, ensure that those defaults are compatible with the project and adjust them as necessary. For tests that require the iOS simulator, also specify the `ios_application` target name as the value of the `test_host` attribute. #### Step 3c: Add the library target(s) -Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each -Objective-C library and a -[`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) -target for each Swift library on which the application and/or tests depend. +Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each Objective-C library and a [`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) target for each Swift library on which the application and/or tests depend. Add the library targets as follows: -* Add the application library targets as dependencies to the application - targets. +- Add the application library targets as dependencies to the application targets. -* Add the test library targets as dependencies to the test targets. +- Add the test library targets as dependencies to the test targets. -* List the implementation sources in the `srcs` attribute. +- List the implementation sources in the `srcs` attribute. -* List the headers in the `hdrs` attribute. +- List the headers in the `hdrs` attribute. -Note: You can use the [`glob`](/reference/be/functions#glob) function to include -all sources and/or headers of a certain type. Use it carefully as it might -include files you do not want Bazel to build. +Note: You can use the [`glob`](/reference/be/functions#glob) function to include all sources and/or headers of a certain type. Use it carefully as it might include files you do not want Bazel to build. -You can browse existing examples for various types of applications directly in -the [rules_apple examples -directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For -example: +You can browse existing examples for various types of applications directly in the [rules\_apple examples directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For example: -* [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) +- [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) -* [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) +- [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) -* [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) +- [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) -For more information on build rules, see [Apple Rules for -Bazel](https://github.com/bazelbuild/rules_apple). +For more information on build rules, see [Apple Rules for Bazel](https://github.com/bazelbuild/rules_apple). At this point, it is a good idea to test the build: -`bazel build //:` +`bazel build //:<application_target>` ### Step 4: (Optional) Granularize the build -If the project is large, or as it grows, consider chunking it into multiple -Bazel packages. This increased granularity provides: +If the project is large, or as it grows, consider chunking it into multiple Bazel packages. This increased granularity provides: -* Increased incrementality of builds, +- Increased incrementality of builds, -* Increased parallelization of build tasks, +- Increased parallelization of build tasks, -* Better maintainability for future users, +- Better maintainability for future users, -* Better control over source code visibility across targets and packages. This - prevents issues such as libraries containing implementation details leaking - into public APIs. +- Better control over source code visibility across targets and packages. This prevents issues such as libraries containing implementation details leaking into public APIs. Tips for granularizing the project: -* Put each library in its own Bazel package. Start with those requiring the - fewest dependencies and work your way up the dependency tree. +- Put each library in its own Bazel package. Start with those requiring the fewest dependencies and work your way up the dependency tree. -* As you add `BUILD` files and specify targets, add these new targets to the - `deps` attributes of targets that depend on them. +- As you add `BUILD` files and specify targets, add these new targets to the `deps` attributes of targets that depend on them. -* The `glob()` function does not cross package boundaries, so as the number of - packages grows the files matched by `glob()` will shrink. +- The `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. -* When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to - the corresponding `test` directory. +- When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to the corresponding `test` directory. -* Enforce healthy visibility limits across packages. +- Enforce healthy visibility limits across packages. -* Build the project after each major change to the `BUILD` files and fix build - errors as you encounter them. +- Build the project after each major change to the `BUILD` files and fix build errors as you encounter them. ### Step 5: Run the build -Run the fully migrated build to ensure it completes with no errors or warnings. -Run every application and test target individually to more easily find sources -of any errors that occur. +Run the fully migrated build to ensure it completes with no errors or warnings. Run every application and test target individually to more easily find sources of any errors that occur. For example: @@ -246,25 +166,17 @@ For example: bazel build //:my-target ``` -### Step 6: Generate the Xcode project with rules_xcodeproj +### Step 6: Generate the Xcode project with rules\_xcodeproj -When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source -of truth about the build. To make Xcode aware of this, you must generate a -Bazel-compatible Xcode project using -[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) -. +When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source of truth about the build. To make Xcode aware of this, you must generate a Bazel-compatible Xcode project using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) . ### Troubleshooting -Bazel errors can arise when it gets out of sync with the selected Xcode version, -like when you apply an update. Here are some things to try if you're -experiencing errors with Xcode, for example "Xcode version must be specified to -use an Apple CROSSTOOL". +Bazel errors can arise when it gets out of sync with the selected Xcode version, like when you apply an update. Here are some things to try if you're experiencing errors with Xcode, for example "Xcode version must be specified to use an Apple CROSSTOOL". -* Manually run Xcode and accept any terms and conditions. +- Manually run Xcode and accept any terms and conditions. -* Use Xcode select to indicate the correct version, accept the license, and - clear Bazel's state. +- Use Xcode select to indicate the correct version, accept the license, and clear Bazel's state. ```posix-terminal sudo xcode-select -s /Applications/Xcode.app/Contents/Developer @@ -274,7 +186,6 @@ use an Apple CROSSTOOL". bazel sync --configure ``` -* If this does not work, you may also try running `bazel clean --expunge`. +- If this does not work, you may also try running `bazel clean --expunge`. -Note: If you've saved your Xcode to a different path, you can use `xcode-select --s` to point to that path. +Note: If you've saved your Xcode to a different path, you can use `xcode-select -s` to point to that path. diff --git a/query/aquery.mdx b/query/aquery.mdx index 151565b7..446855d1 100644 --- a/query/aquery.mdx +++ b/query/aquery.mdx @@ -2,23 +2,13 @@ title: 'Action Graph Query (aquery)' --- +The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis Configured Target Graph and exposes information about **Actions, Artifacts and their relationships.** +`aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs/outputs/mnemonics. -The `aquery` command allows you to query for actions in your build graph. -It operates on the post-analysis Configured Target Graph and exposes -information about **Actions, Artifacts and their relationships.** +The tool accepts several command-line [options](#command-options). Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. -`aquery` is useful when you are interested in the properties of the Actions/Artifacts -generated from the Configured Target Graph. For example, the actual commands run -and their inputs/outputs/mnemonics. - -The tool accepts several command-line [options](#command-options). -Notably, the aquery command runs on top of a regular Bazel build and inherits -the set of options available during a build. - -It supports the same set of functions that is also available to traditional -`query` but `siblings`, `buildfiles` and -`tests`. +It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. An example `aquery` output (without specific details): @@ -41,11 +31,9 @@ A simple example of the syntax for `aquery` is as follows: The query expression (in quotes) consists of the following: -* `aquery_function(...)`: functions specific to `aquery`. - More details [below](#using-aquery-functions). -* `function(...)`: the standard [functions](/query/language#functions) - as traditional `query`. -* `//target` is the label to the interested target. +- `aquery_function(...)`: functions specific to `aquery`. More details [below](#using-aquery-functions). +- `function(...)`: the standard [functions](/query/language#functions) as traditional `query`. +- `//target` is the label to the interested target. ``` # aquery examples: @@ -64,14 +52,13 @@ $ bazel aquery 'inputs(".*cpp", deps(//src/target_a))' There are three `aquery` functions: -* `inputs`: filter actions by inputs. -* `outputs`: filter actions by outputs -* `mnemonic`: filter actions by mnemonic +- `inputs`: filter actions by inputs. +- `outputs`: filter actions by outputs +- `mnemonic`: filter actions by mnemonic `expr ::= inputs(word, expr)` - The `inputs` operator returns the actions generated from building `expr`, - whose input filenames match the regex provided by `word`. +The `inputs` operator returns the actions generated from building `expr`, whose input filenames match the regex provided by `word`. `$ bazel aquery 'inputs(".*cpp", deps(//src/target_a))'` @@ -83,13 +70,9 @@ You can also combine functions to achieve the AND operation. For example: $ bazel aquery 'mnemonic("Cpp.*", (inputs(".*cpp", inputs("foo.*", //src/target_a))))' ``` - The above command would find all actions involved in building `//src/target_a`, - whose mnemonics match `"Cpp.*"` and inputs match the patterns - `".*cpp"` and `"foo.*"`. +The above command would find all actions involved in building `//src/target_a`, whose mnemonics match `"Cpp.*"` and inputs match the patterns `".*cpp"` and `"foo.*"`. -Important: aquery functions can't be nested inside non-aquery functions. -Conceptually, this makes sense since the output of aquery functions is Actions, -not Configured Targets. +Important: aquery functions can't be nested inside non-aquery functions. Conceptually, this makes sense since the output of aquery functions is Actions, not Configured Targets. An example of the syntax error produced: @@ -104,23 +87,17 @@ An example of the syntax error produced: ### Build options -`aquery` runs on top of a regular Bazel build and thus inherits the set of -[options](/reference/command-line-reference#build-options) -available during a build. +`aquery` runs on top of a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. ### Aquery options #### `--output=(text|summary|commands|proto|jsonproto|textproto), default=text` -The default output format (`text`) is human-readable, -use `proto`, `textproto`, or `jsonproto` for machine-readable format. -The proto message is `analysis.ActionGraphContainer`. +The default output format (`text`) is human-readable, use `proto`, `textproto`, or `jsonproto` for machine-readable format. The proto message is `analysis.ActionGraphContainer`. -The `commands` output format prints a list of build commands with -one command per line. +The `commands` output format prints a list of build commands with one command per line. -In general, do not depend on the order of output. For more information, -see the [core query ordering contract](/query/language#graph-order). +In general, do not depend on the order of output. For more information, see the [core query ordering contract](/query/language#graph-order). #### `--include_commandline, default=true` @@ -142,45 +119,35 @@ Warning: Enabling this flag will automatically enable the `--include_commandline #### `--include_file_write_contents, default=false` -Include file contents for the `actions.write()` action and the contents of the -manifest file for the `SourceSymlinkManifest` action The file contents is -returned in the `file_contents` field with `--output=`xxx`proto`. -With `--output=text`, the output has +Include file contents for the `actions.write()` action and the contents of the manifest file for the `SourceSymlinkManifest` action The file contents is returned in the `file_contents` field with `--output=`xxx`proto`. With `--output=text`, the output has + ``` -FileWriteContents: [] +FileWriteContents: [<base64-encoded file contents>] ``` + line #### `--skyframe_state, default=false` Without performing extra analysis, dump the Action Graph from Skyframe. -Note: Specifying a target with `--skyframe_state` is currently not supported. -This flag is only available with `--output=proto` or `--output=textproto`. +Note: Specifying a target with `--skyframe_state` is currently not supported. This flag is only available with `--output=proto` or `--output=textproto`. ## Other tools and features ### Querying against the state of Skyframe -[Skyframe](/reference/skyframe) is the evaluation and -incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph -constructed from the previous runs of the [Analysis phase](/run/build#analysis). +[Skyframe](/reference/skyframe) is the evaluation and incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph constructed from the previous runs of the [Analysis phase](/run/build#analysis). -In some cases, it is useful to query the Action Graph on Skyframe. -An example use case would be: +In some cases, it is useful to query the Action Graph on Skyframe. An example use case would be: -1. Run `bazel build //target_a` -2. Run `bazel build //target_b` -3. File `foo.out` was generated. +1. Run `bazel build //target_a` +2. Run `bazel build //target_b` +3. File `foo.out` was generated. -_As a Bazel user, I want to determine if `foo.out` was generated from building -`//target_a` or `//target_b`_. +*As a Bazel user, I want to determine if `foo.out` was generated from building `//target_a` or `//target_b`*. -One could run `bazel aquery 'outputs("foo.out", //target_a)'` and -`bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible -for creating `foo.out`, and in turn the target. However, the number of different -targets previously built can be larger than 2, which makes running multiple `aquery` -commands a hassle. +One could run `bazel aquery 'outputs("foo.out", //target_a)'` and `bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible for creating `foo.out`, and in turn the target. However, the number of different targets previously built can be larger than 2, which makes running multiple `aquery` commands a hassle. As an alternative, the `--skyframe_state` flag can be used: @@ -194,22 +161,17 @@ As an alternative, the `--skyframe_state` flag can be used: $ bazel aquery --output=proto --skyframe_state 'outputs("foo.out")' ``` -With `--skyframe_state` mode, `aquery` takes the content of the Action Graph -that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and -outputs the content, without re-running the analysis phase. +With `--skyframe_state` mode, `aquery` takes the content of the Action Graph that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and outputs the content, without re-running the analysis phase. #### Special considerations ##### Output format -`--skyframe_state` is currently only available for `--output=proto` -and `--output=textproto` +`--skyframe_state` is currently only available for `--output=proto` and `--output=textproto` ##### Non-inclusion of target labels in the query expression -Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, -regardless of the targets. Having the target label specified in the query together with -`--skyframe_state` is considered a syntax error: +Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, regardless of the targets. Having the target label specified in the query together with `--skyframe_state` is considered a syntax error: ``` # WRONG: Target Included @@ -227,12 +189,9 @@ regardless of the targets. Having the target label specified in the query togeth ### Comparing aquery outputs -You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. -For instance: when you make some changes to your rule definition and want to verify that the -command lines being run did not change. `aquery_differ` is the tool for that. +You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. For instance: when you make some changes to your rule definition and want to verify that the command lines being run did not change. `aquery_differ` is the tool for that. -The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. -To use it, clone the repository to your local machine. An example usage: +The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. To use it, clone the repository to your local machine. An example usage: ``` $ bazel run //tools/aquery_differ -- \ @@ -243,9 +202,7 @@ To use it, clone the repository to your local machine. An example usage: --attrs=inputs ``` -The above command returns the difference between the `before` and `after` aquery outputs: -which actions were present in one but not the other, which actions have different -command line/inputs in each aquery output, ...). The result of running the above command would be: +The above command returns the difference between the `before` and `after` aquery outputs: which actions were present in one but not the other, which actions have different command line/inputs in each aquery output, ...). The result of running the above command would be: ``` Aquery output 'after' change contains an action that generates the following outputs that aquery output 'before' change doesn't: @@ -268,36 +225,29 @@ command line/inputs in each aquery output, ...). The result of running the above `--before, --after`: The aquery output files to be compared -`--input_type=(proto|text_proto), default=proto`: the format of the input -files. Support is provided for `proto` and `textproto` aquery output. +`--input_type=(proto|text_proto), default=proto`: the format of the input files. Support is provided for `proto` and `textproto` aquery output. -`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions -to be compared. +`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions to be compared. ### Aspect-on-aspect -It is possible for [Aspects](/extending/aspects) -to be applied on top of each other. The aquery output of the action generated by -these Aspects would then include the _Aspect path_, which is the sequence of -Aspects applied to the target which generated the action. +It is possible for [Aspects](/extending/aspects) to be applied on top of each other. The aquery output of the action generated by these Aspects would then include the *Aspect path*, which is the sequence of Aspects applied to the target which generated the action. An example of Aspect-on-Aspect: ``` t0 ^ - | <- a1 + | <- a1 t1 ^ - | <- a2 + | <- a2 t2 ``` -Let ti be a target of rule ri, which applies an Aspect ai -to its dependencies. +Let ti be a target of rule ri, which applies an Aspect ai to its dependencies. -Assume that a2 generates an action X when applied to target t0. The text output of -`bazel aquery --include_aspects 'deps(//t2)'` for action X would be: +Assume that a2 generates an action X when applied to target t0. The text output of `bazel aquery --include_aspects 'deps(//t2)'` for action X would be: ``` action ... @@ -305,13 +255,11 @@ Assume that a2 generates an action X when applied to target t0. The text output Target: //my_pkg:t0 Configuration: ... AspectDescriptors: [//my_pkg:rule.bzl%**a2**(foo=...) - -> //my_pkg:rule.bzl%**a1**(bar=...)] + -> //my_pkg:rule.bzl%**a1**(bar=...)] ... ``` -This means that action `X` was generated by Aspect `a2` applied onto -`a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied -onto target `t0`. +This means that action `X` was generated by Aspect `a2` applied onto `a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied onto target `t0`. Each `AspectDescriptor` has the following format: @@ -319,49 +267,31 @@ Each `AspectDescriptor` has the following format: AspectClass([param=value,...]) ``` -`AspectClass` could be the name of the Aspect class (for native Aspects) or -`bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are -sorted in topological order of the -[dependency graph](/extending/aspects#aspect_basics). +`AspectClass` could be the name of the Aspect class (for native Aspects) or `bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are sorted in topological order of the [dependency graph](/extending/aspects#aspect_basics). ### Linking with the JSON profile -While aquery provides information about the actions being run in a build (why they're being run, -their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) -tells us the timing and duration of their execution. -It is possible to combine these 2 sets of information via a common denominator: an action's primary output. +While aquery provides information about the actions being run in a build (why they're being run, their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) tells us the timing and duration of their execution. It is possible to combine these 2 sets of information via a common denominator: an action's primary output. -To include actions' outputs in the JSON profile, generate the profile with -`--experimental_include_primary_output --noslim_profile`. -Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output -is included by default by aquery. +To include actions' outputs in the JSON profile, generate the profile with `--experimental_include_primary_output --noslim_profile`. Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output is included by default by aquery. -We don't currently provide a canonical tool to combine these 2 data sources, but you should be -able to build your own script with the above information. +We don't currently provide a canonical tool to combine these 2 data sources, but you should be able to build your own script with the above information. ## Known issues ### Handling shared actions -Sometimes actions are -[shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) -between configured targets. +Sometimes actions are [shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) between configured targets. -In the execution phase, those shared actions are -[simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. -However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these -like separate actions whose output Artifacts have the exact same `execPath`. As a result, -equivalent Artifacts appear duplicated. +In the execution phase, those shared actions are [simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these like separate actions whose output Artifacts have the exact same `execPath`. As a result, equivalent Artifacts appear duplicated. -The list of aquery issues/planned features can be found on -[GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). +The list of aquery issues/planned features can be found on [GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). ## FAQs ### The ActionKey remains the same even though the content of an input file changed. -In the context of aquery, the `ActionKey` refers to the `String` gotten from -[ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): +In the context of aquery, the `ActionKey` refers to the `String` gotten from [ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): ``` Returns a string encoding all of the significant behaviour of this Action that might affect the @@ -383,8 +313,7 @@ In the context of aquery, the `ActionKey` refers to the `String` gotten from input names change or else action validation may falsely validate. ``` -This excludes the changes to the content of the input files, and is not to be confused with -[RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). +This excludes the changes to the content of the input files, and is not to be confused with [RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). ## Updates diff --git a/query/cquery.mdx b/query/cquery.mdx index d35ea829..6a073058 100644 --- a/query/cquery.mdx +++ b/query/cquery.mdx @@ -2,21 +2,14 @@ title: 'Configurable Query (cquery)' --- +`cquery` is a variant of [`query`](/query/language) that correctly handles [`select()`](/docs/configurable-attributes) and build options' effects on the build graph. - -`cquery` is a variant of [`query`](/query/language) that correctly handles -[`select()`](/docs/configurable-attributes) and build options' effects on the -build graph. - -It achieves this by running over the results of Bazel's [analysis -phase](/extending/concepts#evaluation-model), -which integrates these effects. `query`, by contrast, runs over the results of -Bazel's loading phase, before options are evaluated. +It achieves this by running over the results of Bazel's [analysis phase](/extending/concepts#evaluation-model), which integrates these effects. `query`, by contrast, runs over the results of Bazel's loading phase, before options are evaluated. For example: ``` -$ cat > tree/BUILD <<EOF +$ cat > tree/BUILD <<EOF sh_library( name = "ash", deps = select({ @@ -30,11 +23,11 @@ sh_library(name = "white-ash") sh_library(name = "common-ash") config_setting( name = "excelsior", - values = \{"define": "species=excelsior"\}, + values = {"define": "species=excelsior"}, ) config_setting( name = "americana", - values = \{"define": "species=americana"\}, + values = {"define": "species=americana"}, ) EOF ``` @@ -59,13 +52,9 @@ $ bazel cquery "deps(//tree:ash)" --define species=excelsior --noimplicit_deps //tree:excelsior (9f87702) ``` -Each result includes a [unique identifier](#configurations) `(9f87702)` of -the [configuration](/reference/glossary#configuration) the -target is built with. +Each result includes a [unique identifier](#configurations) `(9f87702)` of the [configuration](/reference/glossary#configuration) the target is built with. -Since `cquery` runs over the configured target graph. it doesn't have insight -into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) -rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). +Since `cquery` runs over the configured target graph. it doesn't have insight into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). ## Basic syntax @@ -75,19 +64,10 @@ A simple `cquery` call looks like: The query expression `"function(//target)"` consists of the following: -* **`function(...)`** is the function to run on the target. `cquery` - supports most - of `query`'s [functions](/query/language#functions), plus a - few new ones. -* **`//target`** is the expression fed to the function. In this example, the - expression is a simple target. But the query language also allows nesting of functions. - See the [Query guide](/query/guide) for examples. - +- **`function(...)`** is the function to run on the target. `cquery` supports most of `query`'s [functions](/query/language#functions), plus a few new ones. +- **`//target`** is the expression fed to the function. In this example, the expression is a simple target. But the query language also allows nesting of functions. See the [Query guide](/query/guide) for examples. -`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) -phases. Unless otherwise specified, `cquery` parses the target(s) listed in the -query expression. See [`--universe_scope`](#universe-scope) -for querying dependencies of top-level build targets. +`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) phases. Unless otherwise specified, `cquery` parses the target(s) listed in the query expression. See [`--universe_scope`](#universe-scope) for querying dependencies of top-level build targets. ## Configurations @@ -97,9 +77,7 @@ The line: //tree:ash (9f87702) ``` -means `//tree:ash` was built in a configuration with ID `9f87702`. For most -targets, this is an opaque hash of the build option values defining the -configuration. +means `//tree:ash` was built in a configuration with ID `9f87702`. For most targets, this is an opaque hash of the build option values defining the configuration. To see the configuration's complete contents, run: @@ -107,25 +85,15 @@ To see the configuration's complete contents, run: $ bazel config 9f87702 ``` -`9f87702` is a prefix of the complete ID. This is because complete IDs are -SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid -prefix of a complete ID, similar to -[Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). - To see complete IDs, run `$ bazel config`. +`9f87702` is a prefix of the complete ID. This is because complete IDs are SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid prefix of a complete ID, similar to [Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). To see complete IDs, run `$ bazel config`. ## Target pattern evaluation -`//foo` has a different meaning for `cquery` than for `query`. This is because -`cquery` evaluates _configured_ targets and the build graph may have multiple -configured versions of `//foo`. +`//foo` has a different meaning for `cquery` than for `query`. This is because `cquery` evaluates *configured* targets and the build graph may have multiple configured versions of `//foo`. -For `cquery`, a target pattern in the query expression evaluates -to every configured target with a label that matches that pattern. Output is -deterministic, but `cquery` makes no ordering guarantee beyond the -[core query ordering contract](/query/language#graph-order). +For `cquery`, a target pattern in the query expression evaluates to every configured target with a label that matches that pattern. Output is deterministic, but `cquery` makes no ordering guarantee beyond the [core query ordering contract](/query/language#graph-order). -This produces subtler results for query expressions than with `query`. -For example, the following can produce multiple results: +This produces subtler results for query expressions than with `query`. For example, the following can produce multiple results: ``` # Analyzes //foo in the target configuration, but also analyzes @@ -137,22 +105,13 @@ $ bazel cquery //foo --universe_scope=//foo,//genrule_with_foo_as_tool //foo (exec) ``` -If you want to precisely declare which instance to query over, use -the [`config`](#config) function. +If you want to precisely declare which instance to query over, use the [`config`](#config) function. -See `query`'s [target pattern -documentation](/query/language#target-patterns) for more information on target -patterns. +See `query`'s [target pattern documentation](/query/language#target-patterns) for more information on target patterns. ## Functions -Of the [set of functions](/query/language#functions "list of query functions") -supported by `query`, `cquery` supports all but -[`allrdeps`](/query/language#allrdeps), -[`buildfiles`](/query/language#buildfiles), -[`rbuildfiles`](/query/language#rbuildfiles), -[`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and -[`visible`](/query/language#visible). +Of the [set of functions](/query/language#functions "list of query functions") supported by `query`, `cquery` supports all but [`allrdeps`](/query/language#allrdeps), [`buildfiles`](/query/language#buildfiles), [`rbuildfiles`](/query/language#rbuildfiles), [`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and [`visible`](/query/language#visible). `cquery` also introduces the following new functions: @@ -160,13 +119,9 @@ supported by `query`, `cquery` supports all but `expr ::= config(expr, word)` -The `config` operator attempts to find the configured target for -the label denoted by the first argument and configuration specified by the -second argument. +The `config` operator attempts to find the configured target for the label denoted by the first argument and configuration specified by the second argument. -Valid values for the second argument are `null` or a -[custom configuration hash](#configurations). Hashes can be retrieved from `$ -bazel config` or a previous `cquery`'s output. +Valid values for the second argument are `null` or a [custom configuration hash](#configurations). Hashes can be retrieved from `$ bazel config` or a previous `cquery`'s output. Examples: @@ -182,27 +137,19 @@ $ bazel cquery "deps(//foo)" $ bazel cquery "config(//baz, 3732cc8)" ``` -If not all results of the first argument can be found in the specified -configuration, only those that can be found are returned. If no results -can be found in the specified configuration, the query fails. +If not all results of the first argument can be found in the specified configuration, only those that can be found are returned. If no results can be found in the specified configuration, the query fails. ## Options ### Build options -`cquery` runs over a regular Bazel build and thus inherits the set of -[options](/reference/command-line-reference#build-options) available during a -build. +`cquery` runs over a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. -### Using cquery options +### Using cquery options #### `--universe_scope` (comma-separated list) -Often, the dependencies of configured targets go through -[transitions](/extending/rules#configurations), -which causes their configuration to differ from their dependent. This flag -allows you to query a target as if it were built as a dependency or a transitive -dependency of another target. For example: +Often, the dependencies of configured targets go through [transitions](/extending/rules#configurations), which causes their configuration to differ from their dependent. This flag allows you to query a target as if it were built as a dependency or a transitive dependency of another target. For example: ``` # x/BUILD @@ -210,7 +157,7 @@ genrule( name = "my_gen", srcs = ["x.in"], outs = ["x.cc"], - cmd = "$(locations :tool) $< >$@", + cmd = "$(locations :tool) $< >$@", tools = [":tool"], ) cc_binary( @@ -219,75 +166,34 @@ cc_binary( ) ``` -Genrules configure their tools in the -[exec configuration](/extending/rules#configurations) -so the following queries would produce the following outputs: - - - - - - - - - - - - - - - - - - - - - -
QueryTarget BuiltOutput
bazel cquery "//x:tool"//x:tool//x:tool(targetconfig)
bazel cquery "//x:tool" --universe_scope="//x:my_gen"//x:my_gen//x:tool(execconfig)
- -If this flag is set, its contents are built. _If it's not set, all targets -mentioned in the query expression are built_ instead. The transitive closure of -the built targets are used as the universe of the query. Either way, the targets -to be built must be buildable at the top level (that is, compatible with -top-level options). `cquery` returns results in the transitive closure of these -top-level targets. - -Even if it's possible to build all targets in a query expression at the top -level, it may be beneficial to not do so. For example, explicitly setting -`--universe_scope` could prevent building targets multiple times in -configurations you don't care about. It could also help specify which -configuration version of a target you're looking for. You should set this flag -if your query expression is more complex than `deps(//foo)`. +Genrules configure their tools in the [exec configuration](/extending/rules#configurations) so the following queries would produce the following outputs: + +| Query | Target Built | Output | +| ------------------------------------------------------- | ------------ | ---------------------- | +| bazel cquery "//x:tool" | //x:tool | //x:tool(targetconfig) | +| bazel cquery "//x:tool" --universe\_scope="//x:my\_gen" | //x:my\_gen | //x:tool(execconfig) | + +If this flag is set, its contents are built. *If it's not set, all targets mentioned in the query expression are built* instead. The transitive closure of the built targets are used as the universe of the query. Either way, the targets to be built must be buildable at the top level (that is, compatible with top-level options). `cquery` returns results in the transitive closure of these top-level targets. + +Even if it's possible to build all targets in a query expression at the top level, it may be beneficial to not do so. For example, explicitly setting `--universe_scope` could prevent building targets multiple times in configurations you don't care about. It could also help specify which configuration version of a target you're looking for. You should set this flag if your query expression is more complex than `deps(//foo)`. #### `--implicit_deps` (boolean, default=True) -Setting this flag to false filters out all results that aren't explicitly set in -the BUILD file and instead set elsewhere by Bazel. This includes filtering -resolved toolchains. +Setting this flag to false filters out all results that aren't explicitly set in the BUILD file and instead set elsewhere by Bazel. This includes filtering resolved toolchains. #### `--tool_deps` (boolean, default=True) -Setting this flag to false filters out all configured targets for which the -path from the queried target to them crosses a transition between the target -configuration and the -[non-target configurations](/extending/rules#configurations). If the queried -target is in the target configuration, setting `--notool_deps` will only return -targets that also are in the target configuration. If the queried target is in a -non-target configuration, setting `--notool_deps` will only return targets also -in non-target configurations. This setting generally does not affect filtering -of resolved toolchains. +Setting this flag to false filters out all configured targets for which the path from the queried target to them crosses a transition between the target configuration and the [non-target configurations](/extending/rules#configurations). If the queried target is in the target configuration, setting `--notool_deps` will only return targets that also are in the target configuration. If the queried target is in a non-target configuration, setting `--notool_deps` will only return targets also in non-target configurations. This setting generally does not affect filtering of resolved toolchains. #### `--include_aspects` (boolean, default=True) Include dependencies added by [aspects](/extending/aspects). -If this flag is disabled, `cquery somepath(X, Y)` and -`cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. +If this flag is disabled, `cquery somepath(X, Y)` and `cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. ## Output formats -By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. -There are other options for exposing the results as well. +By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. There are other options for exposing the results as well. ### Transitions @@ -296,22 +202,11 @@ There are other options for exposing the results as well. --transitions=full ``` -Configuration [transitions](/extending/rules#configurations) -are used to build targets underneath the top level targets in different -configurations than the top level targets. +Configuration [transitions](/extending/rules#configurations) are used to build targets underneath the top level targets in different configurations than the top level targets. -For example, a target might impose a transition to the exec configuration on all -dependencies in its `tools` attribute. These are known as attribute -transitions. Rules can also impose transitions on their own configurations, -known as rule class transitions. This output format outputs information about -these transitions such as what type they are and the effect they have on build -options. +For example, a target might impose a transition to the exec configuration on all dependencies in its `tools` attribute. These are known as attribute transitions. Rules can also impose transitions on their own configurations, known as rule class transitions. This output format outputs information about these transitions such as what type they are and the effect they have on build options. -This output format is triggered by the `--transitions` flag which by default is -set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs -information about rule class transitions and attribute transitions including a -detailed diff of the options before and after the transition. `LITE` mode -outputs the same information without the options diff. +This output format is triggered by the `--transitions` flag which by default is set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs information about rule class transitions and attribute transitions including a detailed diff of the options before and after the transition. `LITE` mode outputs the same information without the options diff. ### Protocol message output @@ -319,27 +214,17 @@ outputs the same information without the options diff. --output=proto ``` -This option causes the resulting targets to be printed in a binary protocol -buffer form. The definition of the protocol buffer can be found at -[src/main/protobuf/analysis_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). +This option causes the resulting targets to be printed in a binary protocol buffer form. The definition of the protocol buffer can be found at [src/main/protobuf/analysis\_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). -`CqueryResult` is the top level message containing the results of the cquery. It -has a list of `ConfiguredTarget` messages and a list of `Configuration` -messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal -to that of the `id` field from the corresponding `Configuration` message. +`CqueryResult` is the top level message containing the results of the cquery. It has a list of `ConfiguredTarget` messages and a list of `Configuration` messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal to that of the `id` field from the corresponding `Configuration` message. -#### --[no]proto:include_configurations +#### --\[no]proto:include\_configurations -By default, cquery results return configuration information as part of each -configured target. If you'd like to omit this information and get proto output -that is formatted exactly like query's proto output, set this flag to false. +By default, cquery results return configuration information as part of each configured target. If you'd like to omit this information and get proto output that is formatted exactly like query's proto output, set this flag to false. -See [query's proto output documentation](/query/language#output-formats) -for more proto output-related options. +See [query's proto output documentation](/query/language#output-formats) for more proto output-related options. -Note: While selects are resolved both at the top level of returned -targets and within attributes, all possible inputs for selects are still -included as `rule_input` fields. +Note: While selects are resolved both at the top level of returned targets and within attributes, all possible inputs for selects are still included as `rule_input` fields. ### Graph output @@ -347,10 +232,7 @@ included as `rule_input` fields. --output=graph ``` -This option generates output as a Graphviz-compatible .dot file. See `query`'s -[graph output documentation](/query/language#display-result-graph) for details. `cquery` -also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and -[`--graph:factored`](/query/language#graph-factored). +This option generates output as a Graphviz-compatible .dot file. See `query`'s [graph output documentation](/query/language#display-result-graph) for details. `cquery` also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and [`--graph:factored`](/query/language#graph-factored). ### Files output @@ -358,23 +240,11 @@ also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and --output=files ``` -This option prints a list of the output files produced by each target matched -by the query similar to the list printed at the end of a `bazel build` -invocation. The output contains only the files advertised in the requested -output groups as determined by the -[`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. -It does include source files. +This option prints a list of the output files produced by each target matched by the query similar to the list printed at the end of a `bazel build` invocation. The output contains only the files advertised in the requested output groups as determined by the [`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. It does include source files. -All paths emitted by this output format are relative to the -[execroot](https://bazel.build/remote/output-directories), which can be obtained -via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, -paths to files in the main repository also resolve relative to the workspace -directory. +All paths emitted by this output format are relative to the [execroot](https://bazel.build/remote/output-directories), which can be obtained via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, paths to files in the main repository also resolve relative to the workspace directory. -Note: The output of `bazel cquery --output=files //pkg:foo` contains the output -files of `//pkg:foo` in *all* configurations that occur in the build (also see -the [section on target pattern evaluation](#target-pattern-evaluation)). If that -is not desired, wrap you query in [`config(..., target)`](#config). +Note: The output of `bazel cquery --output=files //pkg:foo` contains the output files of `//pkg:foo` in *all* configurations that occur in the build (also see the [section on target pattern evaluation](#target-pattern-evaluation)). If that is not desired, wrap you query in [`config(..., target)`](#config). ### Defining the output format using Starlark @@ -382,39 +252,21 @@ is not desired, wrap you query in [`config(..., target)`](#config). --output=starlark ``` -This output format calls a [Starlark](/rules/language) -function for each configured target in the query result, and prints the value -returned by the call. The `--starlark:file` flag specifies the location of a -Starlark file that defines a function named `format` with a single parameter, -`target`. This function is called for each [Target](/rules/lib/builtins/Target) -in the query result. Alternatively, for convenience, you may specify just the -body of a function declared as `def format(target): return expr` by using the -`--starlark:expr` flag. +This output format calls a [Starlark](/rules/language) function for each configured target in the query result, and prints the value returned by the call. The `--starlark:file` flag specifies the location of a Starlark file that defines a function named `format` with a single parameter, `target`. This function is called for each [Target](/rules/lib/builtins/Target) in the query result. Alternatively, for convenience, you may specify just the body of a function declared as `def format(target): return expr` by using the `--starlark:expr` flag. #### 'cquery' Starlark dialect -The cquery Starlark environment differs from a BUILD or .bzl file. It includes -all core Starlark -[built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), -plus a few cquery-specific ones described below, but not (for example) `glob`, -`native`, or `rule`, and it does not support load statements. +The cquery Starlark environment differs from a BUILD or .bzl file. It includes all core Starlark [built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), plus a few cquery-specific ones described below, but not (for example) `glob`, `native`, or `rule`, and it does not support load statements. -##### build_options(target) +##### build\_options(target) -`build_options(target)` returns a map whose keys are build option identifiers -(see [Configurations](/extending/config)) and whose values are their Starlark -values. Build options whose values are not legal Starlark values are omitted -from this map. +`build_options(target)` returns a map whose keys are build option identifiers (see [Configurations](/extending/config)) and whose values are their Starlark values. Build options whose values are not legal Starlark values are omitted from this map. -If the target is an input file, `build_options(target)` returns None, as input -file targets have a null configuration. +If the target is an input file, `build_options(target)` returns None, as input file targets have a null configuration. ##### providers(target) -`providers(target)` returns a map whose keys are names of -[providers](/extending/rules#providers) -(for example, `"DefaultInfo"`) and whose values are their Starlark values. -Providers whose values are not legal Starlark values are omitted from this map. +`providers(target)` returns a map whose keys are names of [providers](/extending/rules#providers) (for example, `"DefaultInfo"`) and whose values are their Starlark values. Providers whose values are not legal Starlark values are omitted from this map. #### Examples @@ -425,8 +277,7 @@ Print a space-separated list of the base names of all files produced by `//foo`: --starlark:expr="' '.join([f.basename for f in providers(target)['DefaultInfo'].files.to_list()])" ``` -Print a space-separated list of the paths of all files produced by **rule** targets in -`//bar` and its subpackages: +Print a space-separated list of the paths of all files produced by **rule** targets in `//bar` and its subpackages: ``` bazel cquery 'kind(rule, //bar/...)' --output=starlark \ @@ -454,8 +305,7 @@ Print the value of the command line option `--javacopt` when building `//foo`. --starlark:expr="build_options(target)['//command_line_option:javacopt']" ``` -Print the label of each target with exactly one output. This example uses -Starlark functions defined in a file. +Print the label of each target with exactly one output. This example uses Starlark functions defined in a file. ``` $ cat example.cquery @@ -472,8 +322,7 @@ Starlark functions defined in a file. $ bazel cquery //baz --output=starlark --starlark:file=example.cquery ``` -Print the label of each target which is strictly Python 3. This example uses -Starlark functions defined in a file. +Print the label of each target which is strictly Python 3. This example uses Starlark functions defined in a file. ``` $ cat example.cquery @@ -494,7 +343,7 @@ Extract a value from a user defined Provider. ``` $ cat some_package/my_rule.bzl - MyRuleInfo = provider(fields=\{"color": "the name of a color"\}) + MyRuleInfo = provider(fields={"color": "the name of a color"}) def _my_rule_impl(ctx): ... @@ -519,96 +368,51 @@ Extract a value from a user defined Provider. ## cquery vs. query -`cquery` and `query` complement each other and excel in -different niches. Consider the following to decide which is right for you: - -* `cquery` follows specific `select()` branches to - model the exact graph you build. `query` doesn't know which - branch the build chooses, so overapproximates by including all branches. -* `cquery`'s precision requires building more of the graph than - `query` does. Specifically, `cquery` - evaluates _configured targets_ while `query` only - evaluates _targets_. This takes more time and uses more memory. -* `cquery`'s interpretation of - the [query language](/query/language) introduces ambiguity - that `query` avoids. For example, - if `"//foo"` exists in two configurations, which one - should `cquery "deps(//foo)"` use? - The [`config`](#config) function can help with this. +`cquery` and `query` complement each other and excel in different niches. Consider the following to decide which is right for you: + +- `cquery` follows specific `select()` branches to model the exact graph you build. `query` doesn't know which branch the build chooses, so overapproximates by including all branches. +- `cquery`'s precision requires building more of the graph than `query` does. Specifically, `cquery` evaluates *configured targets* while `query` only evaluates *targets*. This takes more time and uses more memory. +- `cquery`'s interpretation of the [query language](/query/language) introduces ambiguity that `query` avoids. For example, if `"//foo"` exists in two configurations, which one should `cquery "deps(//foo)"` use? The [`config`](#config) function can help with this. ## Non-deterministic output -`cquery` does not automatically wipe the build graph from previous commands. -It's therefore prone to picking up results from past queries. +`cquery` does not automatically wipe the build graph from previous commands. It's therefore prone to picking up results from past queries. -For example, `genrule` exerts an exec transition on its `tools` attribute - -that is, it configures its tools in the [exec configuration] -(https://bazel.build/rules/rules#configurations). +For example, `genrule` exerts an exec transition on its `tools` attribute - that is, it configures its tools in the \[exec configuration] ([https://bazel.build/rules/rules#configurations](https://bazel.build/rules/rules#configurations)). You can see the lingering effects of that transition below. ``` -$ cat > foo/BUILD << /tmp/deps.svg +$ bazel query "allpaths(//foo, third_party/...)" --notool_deps --output graph | dot -Tsvg > /tmp/deps.svg ``` -Note: `dot` supports other image formats, just replace `svg` with the -format identifier, for example, `png`. +Note: `dot` supports other image formats, just replace `svg` with the format identifier, for example, `png`. When a dependency graph is big and complicated, it can be helpful start with a single path: @@ -62,8 +46,7 @@ $ bazel query "somepath(//foo:foo, third_party/zlib:zlibonly)" //third_party/zlib:zlibonly ``` -If you do not specify `--output graph` with `allpaths`, -you will get a flattened list of the dependency graph. +If you do not specify `--output graph` with `allpaths`, you will get a flattened list of the dependency graph. ``` $ bazel query "allpaths(//foo, third_party/...)" @@ -93,27 +76,15 @@ $ bazel query "allpaths(//foo, third_party/...)" ### Aside: implicit dependencies -The BUILD file for `//foo` never references -`//translations/tools:aggregator`. So, where's the direct dependency? +The BUILD file for `//foo` never references `//translations/tools:aggregator`. So, where's the direct dependency? -Certain rules include implicit dependencies on additional libraries or tools. -For example, to build a `genproto` rule, you need first to build the Protocol -Compiler, so every `genproto` rule carries an implicit dependency on the -protocol compiler. These dependencies are not mentioned in the build file, -but added in by the build tool. The full set of implicit dependencies is - currently undocumented. Using `--noimplicit_deps` allows you to filter out - these deps from your query results. For cquery, this will include resolved toolchains. +Certain rules include implicit dependencies on additional libraries or tools. For example, to build a `genproto` rule, you need first to build the Protocol Compiler, so every `genproto` rule carries an implicit dependency on the protocol compiler. These dependencies are not mentioned in the build file, but added in by the build tool. The full set of implicit dependencies is currently undocumented. Using `--noimplicit_deps` allows you to filter out these deps from your query results. For cquery, this will include resolved toolchains. ## Reverse dependencies -You might want to know the set of targets that depends on some target. For instance, -if you're going to change some code, you might want to know what other code -you're about to break. You can use `rdeps(u, x)` to find the reverse -dependencies of the targets in `x` within the transitive closure of `u`. +You might want to know the set of targets that depends on some target. For instance, if you're going to change some code, you might want to know what other code you're about to break. You can use `rdeps(u, x)` to find the reverse dependencies of the targets in `x` within the transitive closure of `u`. -Bazel's [Sky Query](/query/language#sky-query) -supports the `allrdeps` function which allows you to query reverse dependencies -in a universe you specify. +Bazel's [Sky Query](/query/language#sky-query) supports the `allrdeps` function which allows you to query reverse dependencies in a universe you specify. ## Miscellaneous uses @@ -123,33 +94,47 @@ You can use `bazel query` to analyze many dependency relationships. #### What packages exist beneath `foo`? -```bazel query 'foo/...' --output package``` +``` +bazel query 'foo/...' --output package +``` #### What rules are defined in the `foo` package? -```bazel query 'kind(rule, foo:*)' --output label_kind``` +``` +bazel query 'kind(rule, foo:*)' --output label_kind +``` #### What files are generated by rules in the `foo` package? -```bazel query 'kind("generated file", //foo:*)'``` +``` +bazel query 'kind("generated file", //foo:*)' +``` #### What targets are generated by starlark macro `foo`? -```bazel query 'attr(generator_function, foo, //path/to/search/...)'``` +``` +bazel query 'attr(generator_function, foo, //path/to/search/...)' +``` #### What's the set of BUILD files needed to build `//foo`? -```bazel query 'buildfiles(deps(//foo))' | cut -f1 -d:``` +``` +bazel query 'buildfiles(deps(//foo))' | cut -f1 -d: +``` #### What are the individual tests that a `test_suite` expands to? -```bazel query 'tests(//foo:smoke_tests)'``` +``` +bazel query 'tests(//foo:smoke_tests)' +``` #### Which of those are C++ tests? -```bazel query 'kind(cc_.*, tests(//foo:smoke_tests))'``` +``` +bazel query 'kind(cc_.*, tests(//foo:smoke_tests))' +``` -#### Which of those are small? Medium? Large? +#### Which of those are small? Medium? Large? ``` bazel query 'attr(size, small, tests(//foo:smoke_tests))' @@ -161,19 +146,27 @@ bazel query 'attr(size, large, tests(//foo:smoke_tests))' #### What are the tests beneath `foo` that match a pattern? -```bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))'``` +``` +bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))' +``` The pattern is a regex and is applied to the full name of the rule. It's similar to doing -```bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t'``` +``` +bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t' +``` #### What package contains file `path/to/file/bar.java`? -``` bazel query path/to/file/bar.java --output=package``` +``` + bazel query path/to/file/bar.java --output=package +``` #### What is the build label for `path/to/file/bar.java?` -```bazel query path/to/file/bar.java``` +``` +bazel query path/to/file/bar.java +``` #### What rule target(s) contain file `path/to/file/bar.java` as a source? @@ -186,28 +179,36 @@ bazel query "attr('srcs', $fullname, ${fullname//:*/}:*)" #### What packages does `foo` depend on? (What do I need to check out to build `foo`) -```bazel query 'buildfiles(deps(//foo:foo))' --output package``` +``` +bazel query 'buildfiles(deps(//foo:foo))' --output package +``` -Note: `buildfiles` is required in order to correctly obtain all files -referenced by `subinclude`; see the reference manual for details. +Note: `buildfiles` is required in order to correctly obtain all files referenced by `subinclude`; see the reference manual for details. #### What packages does the `foo` tree depend on, excluding `foo/contrib`? -```bazel query 'deps(foo/... except foo/contrib/...)' --output package``` +``` +bazel query 'deps(foo/... except foo/contrib/...)' --output package +``` ### What rule dependencies exist ... #### What genproto rules does bar depend upon? -```bazel query 'kind(genproto, deps(bar/...))'``` +``` +bazel query 'kind(genproto, deps(bar/...))' +``` #### Find the definition of some JNI (C++) library that is transitively depended upon by a Java binary rule in the servlet tree. -```bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location``` +``` +bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location +``` ##### ...Now find the definitions of all the Java binaries that depend on them -```bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in +``` +bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in let cls = kind(cc_.*library, deps($jbs)) in $jbs intersect allpaths($jbs, $cls)' ``` @@ -218,54 +219,67 @@ referenced by `subinclude`; see the reference manual for details. Source files: -```bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$``` +``` +bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$ +``` Generated files: -```bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$``` +``` +bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$ +``` -#### What is the complete set of Java source files required to build QUX's tests? +#### What is the complete set of Java source files required to build QUX's tests? Source files: -```bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` +``` +bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ +``` Generated files: -```bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` +``` +bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ +``` ### What differences in dependencies between X and Y exist ... #### What targets does `//foo` depend on that `//foo:foolib` does not? -```bazel query 'deps(//foo) except deps(//foo:foolib)'``` +``` +bazel query 'deps(//foo) except deps(//foo:foolib)' +``` -#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does _not_ depend on? +#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does *not* depend on? -```bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))'``` +``` +bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))' +``` ### Why does this dependency exist ... #### Why does `bar` depend on `groups2`? -```bazel query 'somepath(bar/...,groups2/...:*)'``` +``` +bazel query 'somepath(bar/...,groups2/...:*)' +``` -Once you have the results of this query, you will often find that a single -target stands out as being an unexpected or egregious and undesirable -dependency of `bar`. The query can then be further refined to: +Once you have the results of this query, you will often find that a single target stands out as being an unexpected or egregious and undesirable dependency of `bar`. The query can then be further refined to: #### Show me a path from `docker/updater:updater_systest` (a `py_test`) to some `cc_library` that it depends upon: -```bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in - somepath(docker/updater:updater_systest, $cc)'``` +``` +bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in + somepath(docker/updater:updater_systest, $cc)' +``` #### Why does library `//photos/frontend:lib` depend on two variants of the same library `//third_party/jpeglib` and `//third_party/jpeg`? -This query boils down to: "show me the subgraph of `//photos/frontend:lib` that -depends on both libraries". When shown in topological order, the last element -of the result is the most likely culprit. +This query boils down to: "show me the subgraph of `//photos/frontend:lib` that depends on both libraries". When shown in topological order, the last element of the result is the most likely culprit. -```bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) +``` +bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) intersect allpaths(//photos/frontend:lib, //third_party/jpeg)' //photos/frontend:lib @@ -277,41 +291,41 @@ of the result is the most likely culprit. //third_party/jpeg/img:renderer ``` -### What depends on ... +### What depends on ... #### What rules under bar depend on Y? -```bazel query 'bar/... intersect allpaths(bar/..., Y)'``` +``` +bazel query 'bar/... intersect allpaths(bar/..., Y)' +``` -Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X -depend on Y?" If expression X is non-trivial, it may be convenient to bind a -name to it using `let` to avoid duplication. +Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X depend on Y?" If expression X is non-trivial, it may be convenient to bind a name to it using `let` to avoid duplication. #### What targets directly depend on T, in T's package? -```bazel query 'same_pkg_direct_rdeps(T)'``` +``` +bazel query 'same_pkg_direct_rdeps(T)' +``` ### How do I break a dependency ... -{/* TODO find a convincing value of X to plug in here */} - #### What dependency paths do I have to break to make `bar` no longer depend on X? To output the graph to a `svg` file: -```bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg``` +``` +bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg +``` ### Misc #### How many sequential steps are there in the `//foo-tests` build? -Unfortunately, the query language can't currently give you the longest path -from x to y, but it can find the (or rather _a_) most distant node from the -starting point, or show you the _lengths_ of the longest path from x to every -y that it depends on. Use `maxrank`: +Unfortunately, the query language can't currently give you the longest path from x to y, but it can find the (or rather *a*) most distant node from the starting point, or show you the *lengths* of the longest path from x to every y that it depends on. Use `maxrank`: -```bazel query 'deps(//foo-tests)' --output maxrank | tail -1 -85 //third_party/zlib:zutil.c``` +``` +bazel query 'deps(//foo-tests)' --output maxrank | tail -1 +85 //third_party/zlib:zutil.c +``` -The result indicates that there exist paths of length 85 that must occur in -order in this build. +The result indicates that there exist paths of length 85 that must occur in order in this build. diff --git a/reference/glossary.mdx b/reference/glossary.mdx index da7478c3..f045f9e6 100644 --- a/reference/glossary.mdx +++ b/reference/glossary.mdx @@ -2,289 +2,155 @@ title: 'Bazel Glossary' --- - - ### Action -A command to run during the build, for example, a call to a compiler that takes -[artifacts](#artifact) as inputs and produces other artifacts as outputs. -Includes metadata like the command line arguments, action key, environment -variables, and declared input/output artifacts. +A command to run during the build, for example, a call to a compiler that takes [artifacts](#artifact) as inputs and produces other artifacts as outputs. Includes metadata like the command line arguments, action key, environment variables, and declared input/output artifacts. **See also:** [Rules documentation](/extending/rules#actions) ### Action cache -An on-disk cache that stores a mapping of executed [actions](#action) to the -outputs they created. The cache key is known as the [action key](#action-key). A -core component for Bazel's incrementality model. The cache is stored in the -output base directory and thus survives Bazel server restarts. +An on-disk cache that stores a mapping of executed [actions](#action) to the outputs they created. The cache key is known as the [action key](#action-key). A core component for Bazel's incrementality model. The cache is stored in the output base directory and thus survives Bazel server restarts. ### Action graph -An in-memory graph of [actions](#action) and the [artifacts](#artifact) that -these actions read and generate. The graph might include artifacts that exist as -source files (for example, in the file system) as well as generated -intermediate/final artifacts that are not mentioned in `BUILD` files. Produced -during the [analysis phase](#analysis-phase) and used during the [execution -phase](#execution-phase). +An in-memory graph of [actions](#action) and the [artifacts](#artifact) that these actions read and generate. The graph might include artifacts that exist as source files (for example, in the file system) as well as generated intermediate/final artifacts that are not mentioned in `BUILD` files. Produced during the [analysis phase](#analysis-phase) and used during the [execution phase](#execution-phase). ### Action graph query (aquery) -A [query](#query-concept) tool that can query over build [actions](#action). -This provides the ability to analyze how [build rules](#rule) translate into the -actual work builds do. +A [query](#query-concept) tool that can query over build [actions](#action). This provides the ability to analyze how [build rules](#rule) translate into the actual work builds do. ### Action key -The cache key of an [action](#action). Computed based on action metadata, which -might include the command to be executed in the action, compiler flags, library -locations, or system headers, depending on the action. Enables Bazel to cache or -invalidate individual actions deterministically. +The cache key of an [action](#action). Computed based on action metadata, which might include the command to be executed in the action, compiler flags, library locations, or system headers, depending on the action. Enables Bazel to cache or invalidate individual actions deterministically. ### Analysis phase -The second phase of a build. Processes the [target graph](#target-graph) -specified in [`BUILD` files](#build-file) to produce an in-memory [action -graph](#action-graph) that determines the order of actions to run during the -[execution phase](#execution-phase). This is the phase in which rule -implementations are evaluated. +The second phase of a build. Processes the [target graph](#target-graph) specified in [`BUILD` files](#build-file) to produce an in-memory [action graph](#action-graph) that determines the order of actions to run during the [execution phase](#execution-phase). This is the phase in which rule implementations are evaluated. ### Artifact -A source file or a generated file. Can also be a directory of files, known as -[tree artifacts](#tree-artifact). +A source file or a generated file. Can also be a directory of files, known as [tree artifacts](#tree-artifact). -An artifact may be an input to multiple actions, but must only be generated by -at most one action. +An artifact may be an input to multiple actions, but must only be generated by at most one action. -An artifact that corresponds to a [file target](#target) can be addressed by a -label. +An artifact that corresponds to a [file target](#target) can be addressed by a label. ### Aspect -A mechanism for rules to create additional [actions](#action) in their -dependencies. For example, if target A depends on B, one can apply an aspect on -A that traverses *up* a dependency edge to B, and runs additional actions in B -to generate and collect additional output files. These additional actions are -cached and reused between targets requiring the same aspect. Created with the -`aspect()` Starlark Build API function. Can be used, for example, to generate -metadata for IDEs, and create actions for linting. +A mechanism for rules to create additional [actions](#action) in their dependencies. For example, if target A depends on B, one can apply an aspect on A that traverses *up* a dependency edge to B, and runs additional actions in B to generate and collect additional output files. These additional actions are cached and reused between targets requiring the same aspect. Created with the `aspect()` Starlark Build API function. Can be used, for example, to generate metadata for IDEs, and create actions for linting. **See also:** [Aspects documentation](/extending/aspects) ### Aspect-on-aspect -A composition mechanism whereby aspects can be applied to the results -of other aspects. For example, an aspect that generates information for use by -IDEs can be applied on top of an aspect that generates `.java` files from a -proto. +A composition mechanism whereby aspects can be applied to the results of other aspects. For example, an aspect that generates information for use by IDEs can be applied on top of an aspect that generates `.java` files from a proto. -For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that -`B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute -must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) -attribute. +For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that `B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) attribute. ### Attribute -A parameter to a [rule](#rule), used to express per-target build information. -Examples include `srcs`, `deps`, and `copts`, which respectively declare a -target's source files, dependencies, and custom compiler options. The particular -attributes available for a given target depend on its rule type. +A parameter to a [rule](#rule), used to express per-target build information. Examples include `srcs`, `deps`, and `copts`, which respectively declare a target's source files, dependencies, and custom compiler options. The particular attributes available for a given target depend on its rule type. ### .bazelrc -Bazel’s configuration file used to change the default values for [startup -flags](#startup-flags) and [command flags](#command-flags), and to define common -groups of options that can then be set together on the Bazel command line using -a `--config` flag. Bazel can combine settings from multiple bazelrc files -(systemwide, per-workspace, per-user, or from a custom location), and a -`bazelrc` file may also import settings from other `bazelrc` files. +Bazel’s configuration file used to change the default values for [startup flags](#startup-flags) and [command flags](#command-flags), and to define common groups of options that can then be set together on the Bazel command line using a `--config` flag. Bazel can combine settings from multiple bazelrc files (systemwide, per-workspace, per-user, or from a custom location), and a `bazelrc` file may also import settings from other `bazelrc` files. ### Blaze -The Google-internal version of Bazel. Google’s main build system for its -mono-repository. +The Google-internal version of Bazel. Google’s main build system for its mono-repository. ### BUILD File -A `BUILD` file is the main configuration file that tells Bazel what software -outputs to build, what their dependencies are, and how to build them. Bazel -takes a `BUILD` file as input and uses the file to create a graph of dependencies -and to derive the actions that must be completed to build intermediate and final -software outputs. A `BUILD` file marks a directory and any sub-directories not -containing a `BUILD` file as a [package](#package), and can contain -[targets](#target) created by [rules](#rule). The file can also be named -`BUILD.bazel`. +A `BUILD` file is the main configuration file that tells Bazel what software outputs to build, what their dependencies are, and how to build them. Bazel takes a `BUILD` file as input and uses the file to create a graph of dependencies and to derive the actions that must be completed to build intermediate and final software outputs. A `BUILD` file marks a directory and any sub-directories not containing a `BUILD` file as a [package](#package), and can contain [targets](#target) created by [rules](#rule). The file can also be named `BUILD.bazel`. ### BUILD.bazel File -See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same -directory. +See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same directory. ### .bzl File -A file that defines rules, [macros](#macro), and constants written in -[Starlark](#starlark). These can then be imported into [`BUILD` -files](#build-file) using the `load()` function. - -{/* TODO: ### Build event protocol */} - -{/* TODO: ### Build flag */} +A file that defines rules, [macros](#macro), and constants written in [Starlark](#starlark). These can then be imported into [`BUILD` files](#build-file) using the `load()` function. ### Build graph -The dependency graph that Bazel constructs and traverses to perform a build. -Includes nodes like [targets](#target), [configured -targets](#configured-target), [actions](#action), and [artifacts](#artifact). A -build is considered complete when all [artifacts](#artifact) on which a set of -requested targets depend are verified as up-to-date. +The dependency graph that Bazel constructs and traverses to perform a build. Includes nodes like [targets](#target), [configured targets](#configured-target), [actions](#action), and [artifacts](#artifact). A build is considered complete when all [artifacts](#artifact) on which a set of requested targets depend are verified as up-to-date. ### Build setting -A Starlark-defined piece of [configuration](#configuration). -[Transitions](#transition) can set build settings to change a subgraph's -configuration. If exposed to the user as a [command-line flag](#command-flags), -also known as a build flag. +A Starlark-defined piece of [configuration](#configuration). [Transitions](#transition) can set build settings to change a subgraph's configuration. If exposed to the user as a [command-line flag](#command-flags), also known as a build flag. ### Clean build -A build that doesn't use the results of earlier builds. This is generally slower -than an [incremental build](#incremental-build) but commonly considered to be -more [correct](#correctness). Bazel guarantees both clean and incremental builds -are always correct. +A build that doesn't use the results of earlier builds. This is generally slower than an [incremental build](#incremental-build) but commonly considered to be more [correct](#correctness). Bazel guarantees both clean and incremental builds are always correct. ### Client-server model -The `bazel` command-line client automatically starts a background server on the -local machine to execute Bazel [commands](#command). The server persists across -commands but automatically stops after a period of inactivity (or explicitly via -bazel shutdown). Splitting Bazel into a server and client helps amortize JVM -startup time and supports faster [incremental builds](#incremental-build) -because the [action graph](#action-graph) remains in memory across commands. +The `bazel` command-line client automatically starts a background server on the local machine to execute Bazel [commands](#command). The server persists across commands but automatically stops after a period of inactivity (or explicitly via bazel shutdown). Splitting Bazel into a server and client helps amortize JVM startup time and supports faster [incremental builds](#incremental-build) because the [action graph](#action-graph) remains in memory across commands. ### Command -Used on the command line to invoke different Bazel functions, like `bazel -build`, `bazel test`, `bazel run`, and `bazel query`. +Used on the command line to invoke different Bazel functions, like `bazel build`, `bazel test`, `bazel run`, and `bazel query`. ### Command flags -A set of flags specific to a [command](#command). Command flags are specified -*after* the command (`bazel build `). Flags can be applicable to -one or more commands. For example, `--configure` is a flag exclusively for the -`bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, -`test` and more. Flags are often used for [configuration](#configuration) -purposes, so changes in flag values can cause Bazel to invalidate in-memory -graphs and restart the [analysis phase](#analysis-phase). +A set of flags specific to a [command](#command). Command flags are specified *after* the command (`bazel build <command flags>`). Flags can be applicable to one or more commands. For example, `--configure` is a flag exclusively for the `bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, `test` and more. Flags are often used for [configuration](#configuration) purposes, so changes in flag values can cause Bazel to invalidate in-memory graphs and restart the [analysis phase](#analysis-phase). ### Configuration -Information outside of [rule](#rule) definitions that impacts how rules generate -[actions](#action). Every build has at least one configuration specifying the -target platform, action environment variables, and command-line [build -flags](#command-flags). [Transitions](#transition) may create additional -configurations, such as for host tools or cross-compilation. +Information outside of [rule](#rule) definitions that impacts how rules generate [actions](#action). Every build has at least one configuration specifying the target platform, action environment variables, and command-line [build flags](#command-flags). [Transitions](#transition) may create additional configurations, such as for host tools or cross-compilation. **See also:** [Configurations](/extending/rules#configurations) -{/* TODO: ### Configuration fragment */} - ### Configuration trimming -The process of only including the pieces of [configuration](#configuration) a -target actually needs. For example, if you build Java binary `//:j` with C++ -dependency `//:c`, it's wasteful to include the value of `--javacopt` in the -configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ -build cacheability. +The process of only including the pieces of [configuration](#configuration) a target actually needs. For example, if you build Java binary `//:j` with C++ dependency `//:c`, it's wasteful to include the value of `--javacopt` in the configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ build cacheability. ### Configured query (cquery) -A [query](#query-concept) tool that queries over [configured -targets](#configured-target) (after the [analysis phase](#analysis-phase) -completes). This means `select()` and [build flags](#command-flags) (such as -`--platforms`) are accurately reflected in the results. +A [query](#query-concept) tool that queries over [configured targets](#configured-target) (after the [analysis phase](#analysis-phase) completes). This means `select()` and [build flags](#command-flags) (such as `--platforms`) are accurately reflected in the results. **See also:** [cquery documentation](/query/cquery) ### Configured target -The result of evaluating a [target](#target) with a -[configuration](#configuration). The [analysis phase](#analysis-phase) produces -this by combining the build's options with the targets that need to be built. -For example, if `//:foo` builds for two different architectures in the same -build, it has two configured targets: `` and ``. +The result of evaluating a [target](#target) with a [configuration](#configuration). The [analysis phase](#analysis-phase) produces this by combining the build's options with the targets that need to be built. For example, if `//:foo` builds for two different architectures in the same build, it has two configured targets: `<//:foo, x86>` and `<//:foo, arm>`. ### Correctness -A build is correct when its output faithfully reflects the state of its -transitive inputs. To achieve correct builds, Bazel strives to be -[hermetic](#hermeticity), reproducible, and making [build -analysis](#analysis-phase) and [action execution](#execution-phase) -deterministic. +A build is correct when its output faithfully reflects the state of its transitive inputs. To achieve correct builds, Bazel strives to be [hermetic](#hermeticity), reproducible, and making [build analysis](#analysis-phase) and [action execution](#execution-phase) deterministic. ### Dependency -A directed edge between two [targets](#target). A target `//:foo` has a *target -dependency* on target `//:bar` if `//:foo`'s attribute values contain a -reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an -action in `//:foo` depends on an input [artifact](#artifact) created by an -action in `//:bar`. +A directed edge between two [targets](#target). A target `//:foo` has a *target dependency* on target `//:bar` if `//:foo`'s attribute values contain a reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an action in `//:foo` depends on an input [artifact](#artifact) created by an action in `//:bar`. -In certain contexts, it could also refer to an _external dependency_; see -[modules](#module). +In certain contexts, it could also refer to an *external dependency*; see [modules](#module). ### Depset -A data structure for collecting data on transitive dependencies. Optimized so -that merging depsets is time and space efficient, because it’s common to have -very large depsets (hundreds of thousands of files). Implemented to -recursively refer to other depsets for space efficiency reasons. [Rule](#rule) -implementations should not "flatten" depsets by converting them to lists unless -the rule is at the top level of the build graph. Flattening large depsets incurs -huge memory consumption. Also known as *nested sets* in Bazel's internal -implementation. +A data structure for collecting data on transitive dependencies. Optimized so that merging depsets is time and space efficient, because it’s common to have very large depsets (hundreds of thousands of files). Implemented to recursively refer to other depsets for space efficiency reasons. [Rule](#rule) implementations should not "flatten" depsets by converting them to lists unless the rule is at the top level of the build graph. Flattening large depsets incurs huge memory consumption. Also known as *nested sets* in Bazel's internal implementation. **See also:** [Depset documentation](/extending/depsets) ### Disk cache -A local on-disk blob store for the remote caching feature. Can be used in -conjunction with an actual remote blob store. +A local on-disk blob store for the remote caching feature. Can be used in conjunction with an actual remote blob store. ### Distdir -A read-only directory containing files that Bazel would otherwise fetch from the -internet using repository rules. Enables builds to run fully offline. +A read-only directory containing files that Bazel would otherwise fetch from the internet using repository rules. Enables builds to run fully offline. ### Dynamic execution -An execution strategy that selects between local and remote execution based on -various heuristics, and uses the execution results of the faster successful -method. Certain [actions](#action) are executed faster locally (for example, -linking) and others are faster remotely (for example, highly parallelizable -compilation). A dynamic execution strategy can provide the best possible -incremental and clean build times. +An execution strategy that selects between local and remote execution based on various heuristics, and uses the execution results of the faster successful method. Certain [actions](#action) are executed faster locally (for example, linking) and others are faster remotely (for example, highly parallelizable compilation). A dynamic execution strategy can provide the best possible incremental and clean build times. ### Execution phase -The third phase of a build. Executes the [actions](#action) in the [action -graph](#action-graph) created during the [analysis phase](#analysis-phase). -These actions invoke executables (compilers, scripts) to read and write -[artifacts](#artifact). *Spawn strategies* control how these actions are -executed: locally, remotely, dynamically, sandboxed, docker, and so on. +The third phase of a build. Executes the [actions](#action) in the [action graph](#action-graph) created during the [analysis phase](#analysis-phase). These actions invoke executables (compilers, scripts) to read and write [artifacts](#artifact). *Spawn strategies* control how these actions are executed: locally, remotely, dynamically, sandboxed, docker, and so on. ### Execution root -A directory in the [workspace](#workspace)’s [output base](#output-base) -directory where local [actions](#action) are executed in -non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks -of input [artifacts](#artifact) from the workspace. The execution root also -contains symlinks to external repositories as other inputs and the `bazel-out` -directory to store outputs. Prepared during the [loading phase](#loading-phase) -by creating a *symlink forest* of the directories that represent the transitive -closure of packages on which a build depends. Accessible with `bazel info -execution_root` on the command line. +A directory in the [workspace](#workspace)’s [output base](#output-base) directory where local [actions](#action) are executed in non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks of input [artifacts](#artifact) from the workspace. The execution root also contains symlinks to external repositories as other inputs and the `bazel-out` directory to store outputs. Prepared during the [loading phase](#loading-phase) by creating a *symlink forest* of the directories that represent the transitive closure of packages on which a build depends. Accessible with `bazel info execution_root` on the command line. ### File @@ -292,52 +158,27 @@ See [Artifact](#artifact). ### Hermeticity -A build is hermetic if there are no external influences on its build and test -operations, which helps to make sure that results are deterministic and -[correct](#correctness). For example, hermetic builds typically disallow network -access to actions, restrict access to declared inputs, use fixed timestamps and -timezones, restrict access to environment variables, and use fixed seeds for -random number generators +A build is hermetic if there are no external influences on its build and test operations, which helps to make sure that results are deterministic and [correct](#correctness). For example, hermetic builds typically disallow network access to actions, restrict access to declared inputs, use fixed timestamps and timezones, restrict access to environment variables, and use fixed seeds for random number generators ### Incremental build -An incremental build reuses the results of earlier builds to reduce build time -and resource usage. Dependency checking and caching aim to produce correct -results for this type of build. An incremental build is the opposite of a clean -build. - -{/* TODO: ### Install base */} +An incremental build reuses the results of earlier builds to reduce build time and resource usage. Dependency checking and caching aim to produce correct results for this type of build. An incremental build is the opposite of a clean build. ### Label -An identifier for a [target](#target). Generally has the form -`@repo//path/to/package:target`, where `repo` is the (apparent) name of the -[repository](#repository) containing the target, `path/to/package` is the path -to the directory that contains the [`BUILD` file](#build-file) declaring the -target (this directory is also known as the [package](#package)), and `target` -is the name of the target itself. Depending on the situation, parts of this -syntax may be omitted. +An identifier for a [target](#target). Generally has the form `@repo//path/to/package:target`, where `repo` is the (apparent) name of the [repository](#repository) containing the target, `path/to/package` is the path to the directory that contains the [`BUILD` file](#build-file) declaring the target (this directory is also known as the [package](#package)), and `target` is the name of the target itself. Depending on the situation, parts of this syntax may be omitted. **See also**: [Labels](/concepts/labels) ### Loading phase -The first phase of a build where Bazel executes [`BUILD` files](#build-file) to -create [packages](#package). [Macros](#macro) and certain functions like -`glob()` are evaluated in this phase. Interleaved with the second phase of the -build, the [analysis phase](#analysis-phase), to build up a [target -graph](#target-graph). +The first phase of a build where Bazel executes [`BUILD` files](#build-file) to create [packages](#package). [Macros](#macro) and certain functions like `glob()` are evaluated in this phase. Interleaved with the second phase of the build, the [analysis phase](#analysis-phase), to build up a [target graph](#target-graph). ### Legacy macro -A flavor of [macro](#macro) which is declared as an ordinary -[Starlark](#starlark) function, and which runs as a side effect of executing a -`BUILD` file. +A flavor of [macro](#macro) which is declared as an ordinary [Starlark](#starlark) function, and which runs as a side effect of executing a `BUILD` file. -Legacy macros can do anything a function can. This means they can be convenient, -but they can also be harder to read, write, and use. A legacy macro might -unexpectedly mutate its arguments or fail when given a `select()` or ill-typed -argument. +Legacy macros can do anything a function can. This means they can be convenient, but they can also be harder to read, write, and use. A legacy macro might unexpectedly mutate its arguments or fail when given a `select()` or ill-typed argument. Contrast with [symbolic macros](#symbolic-macro). @@ -345,34 +186,19 @@ Contrast with [symbolic macros](#symbolic-macro). ### Macro -A mechanism to compose multiple [rule](#rule) target declarations together under -a single [Starlark](#starlark) callable. Enables reusing common rule declaration -patterns across `BUILD` files. Expanded to the underlying rule target -declarations during the [loading phase](#loading-phase). +A mechanism to compose multiple [rule](#rule) target declarations together under a single [Starlark](#starlark) callable. Enables reusing common rule declaration patterns across `BUILD` files. Expanded to the underlying rule target declarations during the [loading phase](#loading-phase). -Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and -[legacy macros](#legacy-macro). +Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and [legacy macros](#legacy-macro). ### Mnemonic -A short, human-readable string selected by a rule author to quickly understand -what an [action](#action) in the rule is doing. Mnemonics can be used as -identifiers for *spawn strategy* selections. Some examples of action mnemonics -are `Javac` from Java rules, `CppCompile` from C++ rules, and -`AndroidManifestMerger` from Android rules. +A short, human-readable string selected by a rule author to quickly understand what an [action](#action) in the rule is doing. Mnemonics can be used as identifiers for *spawn strategy* selections. Some examples of action mnemonics are `Javac` from Java rules, `CppCompile` from C++ rules, and `AndroidManifestMerger` from Android rules. ### Module -A Bazel project that can have multiple versions, each of which can have -dependencies on other modules. This is analogous to familiar concepts in other -dependency management systems, such as a Maven _artifact_, an npm _package_, a -Go _module_, or a Cargo _crate_. Modules form the backbone of Bazel's external -dependency management system. +A Bazel project that can have multiple versions, each of which can have dependencies on other modules. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. Modules form the backbone of Bazel's external dependency management system. -Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its -root. This file contains metadata about the module itself (such as its name and -version), its direct dependencies, and various other data including toolchain -registrations and [module extension](#module-extension) input. +Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its root. This file contains metadata about the module itself (such as its name and version), its direct dependencies, and various other data including toolchain registrations and [module extension](#module-extension) input. Module metadata is hosted in Bazel registries. @@ -380,336 +206,180 @@ Module metadata is hosted in Bazel registries. ### Module Extension -A piece of logic that can be run to generate [repos](#repository) by reading -inputs from across the [module](#module) dependency graph and invoking [repo -rules](#repository-rule). Module extensions have capabilities similar to repo -rules, allowing them to access the internet, perform file I/O, and so on. +A piece of logic that can be run to generate [repos](#repository) by reading inputs from across the [module](#module) dependency graph and invoking [repo rules](#repository-rule). Module extensions have capabilities similar to repo rules, allowing them to access the internet, perform file I/O, and so on. **See also:** [Module extensions](/external/extension) ### Native rules -[Rules](#rule) that are built into Bazel and implemented in Java. Such rules -appear in [`.bzl` files](#bzl-file) as functions in the native module (for -example, `native.cc_library` or `native.java_library`). User-defined rules -(non-native) are created using [Starlark](#starlark). +[Rules](#rule) that are built into Bazel and implemented in Java. Such rules appear in [`.bzl` files](#bzl-file) as functions in the native module (for example, `native.cc_library` or `native.java_library`). User-defined rules (non-native) are created using [Starlark](#starlark). ### Output base -A [workspace](#workspace)-specific directory to store Bazel output files. Used -to separate outputs from the *workspace*'s source tree (the [main -repo](#repository)). Located in the [output user root](#output-user-root). +A [workspace](#workspace)-specific directory to store Bazel output files. Used to separate outputs from the *workspace*'s source tree (the [main repo](#repository)). Located in the [output user root](#output-user-root). ### Output groups -A group of files that is expected to be built when Bazel finishes building a -target. [Rules](#rule) put their usual outputs in the "default output group" -(e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` -targets). The default output group is the output group whose -[artifacts](#artifact) are built when a target is requested on the command line. -Rules can define more named output groups that can be explicitly specified in -[`BUILD` files](#build-file) (`filegroup` rule) or the command line -(`--output_groups` flag). +A group of files that is expected to be built when Bazel finishes building a target. [Rules](#rule) put their usual outputs in the "default output group" (e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` targets). The default output group is the output group whose [artifacts](#artifact) are built when a target is requested on the command line. Rules can define more named output groups that can be explicitly specified in [`BUILD` files](#build-file) (`filegroup` rule) or the command line (`--output_groups` flag). ### Output user root -A user-specific directory to store Bazel's outputs. The directory name is -derived from the user's system username. Prevents output file collisions if -multiple users are building the same project on the system at the same time. -Contains subdirectories corresponding to build outputs of individual workspaces, -also known as [output bases](#output-base). +A user-specific directory to store Bazel's outputs. The directory name is derived from the user's system username. Prevents output file collisions if multiple users are building the same project on the system at the same time. Contains subdirectories corresponding to build outputs of individual workspaces, also known as [output bases](#output-base). ### Package -The set of [targets](#target) defined by a [`BUILD` file](#build-file). A -package's name is the `BUILD` file's path relative to the [repo](#repository) -root. A package can contain subpackages, or subdirectories containing `BUILD` -files, thus forming a package hierarchy. +The set of [targets](#target) defined by a [`BUILD` file](#build-file). A package's name is the `BUILD` file's path relative to the [repo](#repository) root. A package can contain subpackages, or subdirectories containing `BUILD` files, thus forming a package hierarchy. ### Package group -A [target](#target) representing a set of packages. Often used in `visibility` -attribute values. +A [target](#target) representing a set of packages. Often used in `visibility` attribute values. ### Platform -A "machine type" involved in a build. This includes the machine Bazel runs on -(the "host" platform), the machines build tools execute on ("exec" platforms), -and the machines targets are built for ("target platforms"). +A "machine type" involved in a build. This includes the machine Bazel runs on (the "host" platform), the machines build tools execute on ("exec" platforms), and the machines targets are built for ("target platforms"). ### Provider -A schema describing a unit of information to pass between -[rule targets](#rule-target) along dependency relationships. Typically this -contains information like compiler options, transitive source or output files, -and build metadata. Frequently used in conjunction with [depsets](#depset) to -efficiently store accumulated transitive data. An example of a built-in provider -is `DefaultInfo`. +A schema describing a unit of information to pass between [rule targets](#rule-target) along dependency relationships. Typically this contains information like compiler options, transitive source or output files, and build metadata. Frequently used in conjunction with [depsets](#depset) to efficiently store accumulated transitive data. An example of a built-in provider is `DefaultInfo`. -Note: The object holding specific data for a given rule target is -referred to as a "provider instance", although sometimes this is conflated with -"provider". +Note: The object holding specific data for a given rule target is referred to as a "provider instance", although sometimes this is conflated with "provider". **See also:** [Provider documentation](/extending/rules#providers) ### Query (concept) -The process of analyzing a [build graph](#build-graph) to understand -[target](#target) properties and dependency structures. Bazel supports three -query variants: [query](#query-command), [cquery](#configured-query), and -[aquery](#action-graph-query). +The process of analyzing a [build graph](#build-graph) to understand [target](#target) properties and dependency structures. Bazel supports three query variants: [query](#query-command), [cquery](#configured-query), and [aquery](#action-graph-query). ### query (command) -A [query](#query-concept) tool that operates over the build's post-[loading -phase](#loading-phase) [target graph](#target-graph). This is relatively fast, -but can't analyze the effects of `select()`, [build flags](#command-flags), -[artifacts](#artifact), or build [actions](#action). +A [query](#query-concept) tool that operates over the build's post-[loading phase](#loading-phase) [target graph](#target-graph). This is relatively fast, but can't analyze the effects of `select()`, [build flags](#command-flags), [artifacts](#artifact), or build [actions](#action). **See also:** [Query how-to](/query/guide), [Query reference](/query/language) ### Repository -A directory tree with a boundary marker file at its root, containing source -files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo -represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or -`WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a -repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. The *main repo* is the repo in which the current Bazel command is being run. -*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` -files, or invoking [repo rules](#repository-rule) in [module -extensions](#module-extension). They can be fetched on demand to a predetermined -"magical" location on disk. +*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` files, or invoking [repo rules](#repository-rule) in [module extensions](#module-extension). They can be fetched on demand to a predetermined "magical" location on disk. -Each repo has a unique, constant *canonical* name, and potentially different -*apparent* names when viewed from other repos. +Each repo has a unique, constant *canonical* name, and potentially different *apparent* names when viewed from other repos. **See also**: [External dependencies overview](/external/overview) ### Repository cache -A shared content-addressable cache of files downloaded by Bazel for builds, -shareable across [workspaces](#workspace). Enables offline builds after the -initial download. Commonly used to cache files downloaded through [repository -rules](#repository-rule) like `http_archive` and repository rule APIs like -`repository_ctx.download`. Files are cached only if their SHA-256 checksums are -specified for the download. +A shared content-addressable cache of files downloaded by Bazel for builds, shareable across [workspaces](#workspace). Enables offline builds after the initial download. Commonly used to cache files downloaded through [repository rules](#repository-rule) like `http_archive` and repository rule APIs like `repository_ctx.download`. Files are cached only if their SHA-256 checksums are specified for the download. ### Repository rule -A schema for repository definitions that tells Bazel how to materialize (or -"fetch") a [repository](#repository). Often shortened to just **repo rule**. -Repo rules are invoked by Bazel internally to define repos backed by -[modules](#module), or can be invoked by [module extensions](#module-extension). -Repo rules can access the internet or perform file I/O; the most common repo -rule is `http_archive` to download an archive containing source files from the -internet. +A schema for repository definitions that tells Bazel how to materialize (or "fetch") a [repository](#repository). Often shortened to just **repo rule**. Repo rules are invoked by Bazel internally to define repos backed by [modules](#module), or can be invoked by [module extensions](#module-extension). Repo rules can access the internet or perform file I/O; the most common repo rule is `http_archive` to download an archive containing source files from the internet. **See also:** [Repo rule documentation](/external/repo) ### Reproducibility -The property of a build or test that a set of inputs to the build or test will -always produce the same set of outputs every time, regardless of time, method, -or environment. Note that this does not necessarily imply that the outputs are -[correct](#correctness) or the desired outputs. +The property of a build or test that a set of inputs to the build or test will always produce the same set of outputs every time, regardless of time, method, or environment. Note that this does not necessarily imply that the outputs are [correct](#correctness) or the desired outputs. ### Rule -A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as -`cc_library`. From the perspective of a `BUILD` file author, a rule consists of -a set of [attributes](#attributes) and black box logic. The logic tells the -rule target how to produce output [artifacts](#artifact) and pass information to -other rule targets. From the perspective of `.bzl` authors, rules are the -primary way to extend Bazel to support new programming languages and -environments. - -Rules are instantiated to produce rule targets in the -[loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule -targets communicate information to their downstream dependencies in the form of -[providers](#provider), and register [actions](#action) describing how to -generate their output artifacts. These actions are run in the [execution -phase](#execution-phase). - -Note: Historically the term "rule" has been used to refer to a rule target. -This usage was inherited from tools like Make, but causes confusion and should -be avoided for Bazel. +A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as `cc_library`. From the perspective of a `BUILD` file author, a rule consists of a set of [attributes](#attributes) and black box logic. The logic tells the rule target how to produce output [artifacts](#artifact) and pass information to other rule targets. From the perspective of `.bzl` authors, rules are the primary way to extend Bazel to support new programming languages and environments. + +Rules are instantiated to produce rule targets in the [loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule targets communicate information to their downstream dependencies in the form of [providers](#provider), and register [actions](#action) describing how to generate their output artifacts. These actions are run in the [execution phase](#execution-phase). + +Note: Historically the term "rule" has been used to refer to a rule target. This usage was inherited from tools like Make, but causes confusion and should be avoided for Bazel. **See also:** [Rules documentation](/extending/rules) ### Rule target -A [target](#target) that is an instance of a rule. Contrasts with file targets -and package groups. Not to be confused with [rule](#rule). +A [target](#target) that is an instance of a rule. Contrasts with file targets and package groups. Not to be confused with [rule](#rule). ### Runfiles -The runtime dependencies of an executable [target](#target). Most commonly, the -executable is the executable output of a test rule, and the runfiles are runtime -data dependencies of the test. Before the invocation of the executable (during -bazel test), Bazel prepares the tree of runfiles alongside the test executable -according to their source directory structure. +The runtime dependencies of an executable [target](#target). Most commonly, the executable is the executable output of a test rule, and the runfiles are runtime data dependencies of the test. Before the invocation of the executable (during bazel test), Bazel prepares the tree of runfiles alongside the test executable according to their source directory structure. **See also:** [Runfiles documentation](/extending/rules#runfiles) ### Sandboxing -A technique to isolate a running [action](#action) inside a restricted and -temporary [execution root](#execution-root), helping to ensure that it doesn’t -read undeclared inputs or write undeclared outputs. Sandboxing greatly improves -[hermeticity](#hermeticity), but usually has a performance cost, and requires -support from the operating system. The performance cost depends on the platform. -On Linux, it's not significant, but on macOS it can make sandboxing unusable. +A technique to isolate a running [action](#action) inside a restricted and temporary [execution root](#execution-root), helping to ensure that it doesn’t read undeclared inputs or write undeclared outputs. Sandboxing greatly improves [hermeticity](#hermeticity), but usually has a performance cost, and requires support from the operating system. The performance cost depends on the platform. On Linux, it's not significant, but on macOS it can make sandboxing unusable. ### Skyframe [Skyframe](/reference/skyframe) is the core parallel, functional, and incremental evaluation framework of Bazel. -{/* TODO: ### Spawn strategy */} - ### Stamping -A feature to embed additional information into Bazel-built -[artifacts](#artifact). For example, this can be used for source control, build -time and other workspace or environment-related information for release builds. -Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that -support the stamp attribute. +A feature to embed additional information into Bazel-built [artifacts](#artifact). For example, this can be used for source control, build time and other workspace or environment-related information for release builds. Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that support the stamp attribute. ### Starlark -The extension language for writing [rules](/extending/rules) and [macros](#macro). A -restricted subset of Python (syntactically and grammatically) aimed for the -purpose of configuration, and for better performance. Uses the [`.bzl` -file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more -restricted version of Starlark (such as no `def` function definitions), formerly -known as Skylark. +The extension language for writing [rules](/extending/rules) and [macros](#macro). A restricted subset of Python (syntactically and grammatically) aimed for the purpose of configuration, and for better performance. Uses the [`.bzl` file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more restricted version of Starlark (such as no `def` function definitions), formerly known as Skylark. **See also:** [Starlark language documentation](/rules/language) -{/* TODO: ### Starlark rules */} - -{/* TODO: ### Starlark rule sandwich */} - ### Startup flags -The set of flags specified between `bazel` and the [command](#query-command), -for example, bazel `--host_jvm_debug` build. These flags modify the -[configuration](#configuration) of the Bazel server, so any modification to -startup flags causes a server restart. Startup flags are not specific to any -command. +The set of flags specified between `bazel` and the [command](#query-command), for example, bazel `--host_jvm_debug` build. These flags modify the [configuration](#configuration) of the Bazel server, so any modification to startup flags causes a server restart. Startup flags are not specific to any command. ### Symbolic macro -A flavor of [macro](#macro) which is declared with a [rule](#rule)-like -[attribute](#attribute) schema, allows hiding internal declared -[targets](#target) from their own package, and enforces a predictable naming -pattern on the targets that the macro declares. Designed to avoid some of the -problems seen in large [legacy macro](#legacy-macro) codebases. +A flavor of [macro](#macro) which is declared with a [rule](#rule)-like [attribute](#attribute) schema, allows hiding internal declared [targets](#target) from their own package, and enforces a predictable naming pattern on the targets that the macro declares. Designed to avoid some of the problems seen in large [legacy macro](#legacy-macro) codebases. **See also:** [Symbolic macro documentation](/extending/macros) ### Target -An object that is defined in a [`BUILD` file](#build-file) and identified by a -[label](#label). Targets represent the buildable units of a workspace from -the perspective of the end user. +An object that is defined in a [`BUILD` file](#build-file) and identified by a [label](#label). Targets represent the buildable units of a workspace from the perspective of the end user. -A target that is declared by instantiating a [rule](#rule) is called a [rule -target](#rule-target). Depending on the rule, these may be runnable (like -`cc_binary`) or testable (like `cc_test`). Rule targets typically depend on -other targets via their [attributes](#attribute) (such as `deps`); these -dependencies form the basis of the [target graph](#target-graph). +A target that is declared by instantiating a [rule](#rule) is called a [rule target](#rule-target). Depending on the rule, these may be runnable (like `cc_binary`) or testable (like `cc_test`). Rule targets typically depend on other targets via their [attributes](#attribute) (such as `deps`); these dependencies form the basis of the [target graph](#target-graph). -Aside from rule targets, there are also file targets and [package group](#package-group) -targets. File targets correspond to [artifacts](#artifact) that are referenced -within a `BUILD` file. As a special case, the `BUILD` file of any package is -always considered a source file target in that package. +Aside from rule targets, there are also file targets and [package group](#package-group) targets. File targets correspond to [artifacts](#artifact) that are referenced within a `BUILD` file. As a special case, the `BUILD` file of any package is always considered a source file target in that package. -Targets are discovered during the [loading phase](#loading-phase). During the -[analysis phase](#analysis-phase), targets are associated with [build -configurations](#configuration) to form [configured -targets](#configured-target). +Targets are discovered during the [loading phase](#loading-phase). During the [analysis phase](#analysis-phase), targets are associated with [build configurations](#configuration) to form [configured targets](#configured-target). ### Target graph -An in-memory graph of [targets](#target) and their dependencies. Produced during -the [loading phase](#loading-phase) and used as an input to the [analysis -phase](#analysis-phase). +An in-memory graph of [targets](#target) and their dependencies. Produced during the [loading phase](#loading-phase) and used as an input to the [analysis phase](#analysis-phase). ### Target pattern -A way to specify a group of [targets](#target) on the command line. Commonly -used patterns are `:all` (all rule targets), `:*` (all rule + file targets), -`...` (current [package](#package) and all subpackages recursively). Can be used -in combination, for example, `//...:*` means all rule and file targets in all -packages recursively from the root of the [workspace](#workspace). +A way to specify a group of [targets](#target) on the command line. Commonly used patterns are `:all` (all rule targets), `:*` (all rule + file targets), `...` (current [package](#package) and all subpackages recursively). Can be used in combination, for example, `//...:*` means all rule and file targets in all packages recursively from the root of the [workspace](#workspace). ### Tests -Rule [targets](#target) instantiated from test rules, and therefore contains a -test executable. A return code of zero from the completion of the executable -indicates test success. The exact contract between Bazel and tests (such as test -environment variables, test result collection methods) is specified in the [Test -Encyclopedia](/reference/test-encyclopedia). +Rule [targets](#target) instantiated from test rules, and therefore contains a test executable. A return code of zero from the completion of the executable indicates test success. The exact contract between Bazel and tests (such as test environment variables, test result collection methods) is specified in the [Test Encyclopedia](/reference/test-encyclopedia). ### Toolchain -A set of tools to build outputs for a language. Typically, a toolchain includes -compilers, linkers, interpreters or/and linters. A toolchain can also vary by -platform, that is, a Unix compiler toolchain's components may differ for the -Windows variant, even though the toolchain is for the same language. Selecting -the right toolchain for the platform is known as toolchain resolution. +A set of tools to build outputs for a language. Typically, a toolchain includes compilers, linkers, interpreters or/and linters. A toolchain can also vary by platform, that is, a Unix compiler toolchain's components may differ for the Windows variant, even though the toolchain is for the same language. Selecting the right toolchain for the platform is known as toolchain resolution. ### Top-level target -A build [target](#target) is top-level if it’s requested on the Bazel command -line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is -called, then for this build, `//:foo` is top-level, and `//:bar` isn’t -top-level, although both targets will need to be built. An important difference -between top-level and non-top-level targets is that [command -flags](#command-flags) set on the Bazel command line (or via -[.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level -targets, but might be modified by a [transition](#transition) for non-top-level -targets. +A build [target](#target) is top-level if it’s requested on the Bazel command line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is called, then for this build, `//:foo` is top-level, and `//:bar` isn’t top-level, although both targets will need to be built. An important difference between top-level and non-top-level targets is that [command flags](#command-flags) set on the Bazel command line (or via [.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level targets, but might be modified by a [transition](#transition) for non-top-level targets. ### Transition -A mapping of [configuration](#configuration) state from one value to another. -Enables [targets](#target) in the [build graph](#build-graph) to have different -configurations, even if they were instantiated from the same [rule](#rule). A -common usage of transitions is with *split* transitions, where certain parts of -the [target graph](#target-graph) is forked with distinct configurations for -each fork. For example, one can build an Android APK with native binaries -compiled for ARM and x86 using split transitions in a single build. +A mapping of [configuration](#configuration) state from one value to another. Enables [targets](#target) in the [build graph](#build-graph) to have different configurations, even if they were instantiated from the same [rule](#rule). A common usage of transitions is with *split* transitions, where certain parts of the [target graph](#target-graph) is forked with distinct configurations for each fork. For example, one can build an Android APK with native binaries compiled for ARM and x86 using split transitions in a single build. **See also:** [User-defined transitions](/extending/config#user-defined-transitions) ### Tree artifact -An [artifact](#artifact) that represents a collection of files. Since these -files are not themselves artifacts, an [action](#action) operating on them must -instead register the tree artifact as its input or output. +An [artifact](#artifact) that represents a collection of files. Since these files are not themselves artifacts, an [action](#action) operating on them must instead register the tree artifact as its input or output. ### Visibility -One of two mechanisms for preventing unwanted dependencies in the build system: -*target visibility* for controlling whether a [target](#target) can be depended -upon by other targets; and *load visibility* for controlling whether a `BUILD` -or `.bzl` file may load a given `.bzl` file. Without context, usually -"visibility" refers to target visibility. +One of two mechanisms for preventing unwanted dependencies in the build system: *target visibility* for controlling whether a [target](#target) can be depended upon by other targets; and *load visibility* for controlling whether a `BUILD` or `.bzl` file may load a given `.bzl` file. Without context, usually "visibility" refers to target visibility. **See also:** [Visibility documentation](/concepts/visibility) ### Workspace -The environment shared by all Bazel commands run from the same [main -repository](#repository). +The environment shared by all Bazel commands run from the same [main repository](#repository). -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". Such usage -should be avoided for clarity. +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". Such usage should be avoided for clarity. diff --git a/reference/skyframe.mdx b/reference/skyframe.mdx index ba9149fa..7f99346a 100644 --- a/reference/skyframe.mdx +++ b/reference/skyframe.mdx @@ -2,197 +2,86 @@ title: 'Skyframe' --- - - The parallel evaluation and incrementality model of Bazel. ## Data model The data model consists of the following items: -* `SkyValue`. Also called nodes. `SkyValues` are immutable objects that - contain all the data built over the course of the build and the inputs of - the build. Examples are: input files, output files, targets and configured - targets. -* `SkyKey`. A short immutable name to reference a `SkyValue`, for example, - `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. -* `SkyFunction`. Builds nodes based on their keys and dependent nodes. -* Node graph. A data structure containing the dependency relationship between - nodes. -* `Skyframe`. Code name for the incremental evaluation framework Bazel is - based on. +- `SkyValue`. Also called nodes. `SkyValues` are immutable objects that contain all the data built over the course of the build and the inputs of the build. Examples are: input files, output files, targets and configured targets. +- `SkyKey`. A short immutable name to reference a `SkyValue`, for example, `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. +- `SkyFunction`. Builds nodes based on their keys and dependent nodes. +- Node graph. A data structure containing the dependency relationship between nodes. +- `Skyframe`. Code name for the incremental evaluation framework Bazel is based on. ## Evaluation A build is achieved by evaluating the node that represents the build request. -First, Bazel finds the `SkyFunction` corresponding to the key of the top-level -`SkyKey`. The function then requests the evaluation of the nodes it needs to -evaluate the top-level node, which in turn result in other `SkyFunction` calls, -until the leaf nodes are reached. Leaf nodes are usually ones that represent -input files in the file system. Finally, Bazel ends up with the value of the -top-level `SkyValue`, some side effects (such as output files in the file -system) and a directed acyclic graph of the dependencies between the nodes -involved in the build. - -A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in -advance all of the nodes it needs to do its job. A simple example is evaluating -an input file node that turns out to be a symlink: the function tries to read -the file, realizes that it is a symlink, and thus fetches the file system node -representing the target of the symlink. But that itself can be a symlink, in -which case the original function will need to fetch its target, too. - -The functions are represented in the code by the interface `SkyFunction` and the -services provided to it by an interface called `SkyFunction.Environment`. These -are the things functions can do: - -* Request the evaluation of another node by way of calling `env.getValue`. If - the node is available, its value is returned, otherwise, `null` is returned - and the function itself is expected to return `null`. In the latter case, - the dependent node is evaluated, and then the original node builder is - invoked again, but this time the same `env.getValue` call will return a - non-`null` value. -* Request the evaluation of multiple other nodes by calling `env.getValues()`. - This does essentially the same, except that the dependent nodes are - evaluated in parallel. -* Do computation during their invocation -* Have side effects, for example, writing files to the file system. Care needs - to be taken that two different functions avoid stepping on each other's - toes. In general, write side effects (where data flows outwards from Bazel) - are okay, read side effects (where data flows inwards into Bazel without a - registered dependency) are not, because they are an unregistered dependency - and as such, can cause incorrect incremental builds. - -Well-behaved `SkyFunction` implementations avoid accessing data in any other way -than requesting dependencies (such as by directly reading the file system), -because that results in Bazel not registering the data dependency on the file -that was read, thus resulting in incorrect incremental builds. - -Once a function has enough data to do its job, it should return a non-`null` -value indicating completion. +First, Bazel finds the `SkyFunction` corresponding to the key of the top-level `SkyKey`. The function then requests the evaluation of the nodes it needs to evaluate the top-level node, which in turn result in other `SkyFunction` calls, until the leaf nodes are reached. Leaf nodes are usually ones that represent input files in the file system. Finally, Bazel ends up with the value of the top-level `SkyValue`, some side effects (such as output files in the file system) and a directed acyclic graph of the dependencies between the nodes involved in the build. + +A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in advance all of the nodes it needs to do its job. A simple example is evaluating an input file node that turns out to be a symlink: the function tries to read the file, realizes that it is a symlink, and thus fetches the file system node representing the target of the symlink. But that itself can be a symlink, in which case the original function will need to fetch its target, too. + +The functions are represented in the code by the interface `SkyFunction` and the services provided to it by an interface called `SkyFunction.Environment`. These are the things functions can do: + +- Request the evaluation of another node by way of calling `env.getValue`. If the node is available, its value is returned, otherwise, `null` is returned and the function itself is expected to return `null`. In the latter case, the dependent node is evaluated, and then the original node builder is invoked again, but this time the same `env.getValue` call will return a non-`null` value. +- Request the evaluation of multiple other nodes by calling `env.getValues()`. This does essentially the same, except that the dependent nodes are evaluated in parallel. +- Do computation during their invocation +- Have side effects, for example, writing files to the file system. Care needs to be taken that two different functions avoid stepping on each other's toes. In general, write side effects (where data flows outwards from Bazel) are okay, read side effects (where data flows inwards into Bazel without a registered dependency) are not, because they are an unregistered dependency and as such, can cause incorrect incremental builds. + +Well-behaved `SkyFunction` implementations avoid accessing data in any other way than requesting dependencies (such as by directly reading the file system), because that results in Bazel not registering the data dependency on the file that was read, thus resulting in incorrect incremental builds. + +Once a function has enough data to do its job, it should return a non-`null` value indicating completion. This evaluation strategy has a number of benefits: -* Hermeticity. If functions only request input data by way of depending on - other nodes, Bazel can guarantee that if the input state is the same, the - same data is returned. If all sky functions are deterministic, this means - that the whole build will also be deterministic. -* Correct and perfect incrementality. If all the input data of all functions - is recorded, Bazel can invalidate only the exact set of nodes that need to - be invalidated when the input data changes. -* Parallelism. Since functions can only interact with each other by way of - requesting dependencies, functions that don't depend on each other can be - run in parallel and Bazel can guarantee that the result is the same as if - they were run sequentially. +- Hermeticity. If functions only request input data by way of depending on other nodes, Bazel can guarantee that if the input state is the same, the same data is returned. If all sky functions are deterministic, this means that the whole build will also be deterministic. +- Correct and perfect incrementality. If all the input data of all functions is recorded, Bazel can invalidate only the exact set of nodes that need to be invalidated when the input data changes. +- Parallelism. Since functions can only interact with each other by way of requesting dependencies, functions that don't depend on each other can be run in parallel and Bazel can guarantee that the result is the same as if they were run sequentially. ## Incrementality -Since functions can only access input data by depending on other nodes, Bazel -can build up a complete data flow graph from the input files to the output -files, and use this information to only rebuild those nodes that actually need -to be rebuilt: the reverse transitive closure of the set of changed input files. - -In particular, two possible incrementality strategies exist: the bottom-up one -and the top-down one. Which one is optimal depends on how the dependency graph -looks like. - -* During bottom-up invalidation, after a graph is built and the set of changed - inputs is known, all the nodes are invalidated that transitively depend on - changed files. This is optimal if the same top-level node will be built - again. Note that bottom-up invalidation requires running `stat()` on all - input files of the previous build to determine if they were changed. This - can be improved by using `inotify` or a similar mechanism to learn about - changed files. - -* During top-down invalidation, the transitive closure of the top-level node - is checked and only those nodes are kept whose transitive closure is clean. - This is better if the node graph is large, but the next build only needs a - small subset of it: bottom-up invalidation would invalidate the larger graph - of the first build, unlike top-down invalidation, which just walks the small - graph of second build. +Since functions can only access input data by depending on other nodes, Bazel can build up a complete data flow graph from the input files to the output files, and use this information to only rebuild those nodes that actually need to be rebuilt: the reverse transitive closure of the set of changed input files. + +In particular, two possible incrementality strategies exist: the bottom-up one and the top-down one. Which one is optimal depends on how the dependency graph looks like. + +- During bottom-up invalidation, after a graph is built and the set of changed inputs is known, all the nodes are invalidated that transitively depend on changed files. This is optimal if the same top-level node will be built again. Note that bottom-up invalidation requires running `stat()` on all input files of the previous build to determine if they were changed. This can be improved by using `inotify` or a similar mechanism to learn about changed files. + +- During top-down invalidation, the transitive closure of the top-level node is checked and only those nodes are kept whose transitive closure is clean. This is better if the node graph is large, but the next build only needs a small subset of it: bottom-up invalidation would invalidate the larger graph of the first build, unlike top-down invalidation, which just walks the small graph of second build. Bazel only does bottom-up invalidation. -To get further incrementality, Bazel uses _change pruning_: if a node is -invalidated, but upon rebuild, it is discovered that its new value is the same -as its old value, the nodes that were invalidated due to a change in this node -are "resurrected". +To get further incrementality, Bazel uses *change pruning*: if a node is invalidated, but upon rebuild, it is discovered that its new value is the same as its old value, the nodes that were invalidated due to a change in this node are "resurrected". -This is useful, for example, if one changes a comment in a C++ file: then the -`.o` file generated from it will be the same, thus, it is unnecessary to call -the linker again. +This is useful, for example, if one changes a comment in a C++ file: then the `.o` file generated from it will be the same, thus, it is unnecessary to call the linker again. ## Incremental Linking / Compilation -The main limitation of this model is that the invalidation of a node is an -all-or-nothing affair: when a dependency changes, the dependent node is always -rebuilt from scratch, even if a better algorithm would exist that would mutate -the old value of the node based on the changes. A few examples where this would -be useful: +The main limitation of this model is that the invalidation of a node is an all-or-nothing affair: when a dependency changes, the dependent node is always rebuilt from scratch, even if a better algorithm would exist that would mutate the old value of the node based on the changes. A few examples where this would be useful: -* Incremental linking -* When a single class file changes in a JAR file, it is possible - modify the JAR file in-place instead of building it from scratch again. +- Incremental linking +- When a single class file changes in a JAR file, it is possible modify the JAR file in-place instead of building it from scratch again. -The reason why Bazel does not support these things in a principled way -is twofold: +The reason why Bazel does not support these things in a principled way is twofold: -* There were limited performance gains. -* Difficulty to validate that the result of the mutation is the same as that - of a clean rebuild would be, and Google values builds that are bit-for-bit - repeatable. +- There were limited performance gains. +- Difficulty to validate that the result of the mutation is the same as that of a clean rebuild would be, and Google values builds that are bit-for-bit repeatable. -Until now, it was possible to achieve good enough performance by decomposing an -expensive build step and achieving partial re-evaluation that way. For example, -in an Android app, you can split all the classes into multiple groups and dex -them separately. This way, if classes in a group are unchanged, the dexing does -not have to be redone. +Until now, it was possible to achieve good enough performance by decomposing an expensive build step and achieving partial re-evaluation that way. For example, in an Android app, you can split all the classes into multiple groups and dex them separately. This way, if classes in a group are unchanged, the dexing does not have to be redone. ## Mapping to Bazel concepts -This is high level summary of the key `SkyFunction` and `SkyValue` -implementations Bazel uses to perform a build: - -* **FileStateValue**. The result of an `lstat()`. For existent files, the - function also computes additional information in order to detect changes to - the file. This is the lowest level node in the Skyframe graph and has no - dependencies. -* **FileValue**. Used by anything that cares about the actual contents or - resolved path of a file. Depends on the corresponding `FileStateValue` and - any symlinks that need to be resolved (such as the `FileValue` for `a/b` - needs the resolved path of `a` and the resolved path of `a/b`). The - distinction between `FileValue` and `FileStateValue` is important because - the latter can be used in cases where the contents of the file are not - actually needed. For example, the file contents are irrelevant when - evaluating file system globs (such as `srcs=glob(["*/*.java"])`). -* **DirectoryListingStateValue**. The result of `readdir()`. Like - `FileStateValue`, this is the lowest level node and has no dependencies. -* **DirectoryListingValue**. Used by anything that cares about the entries of - a directory. Depends on the corresponding `DirectoryListingStateValue`, as - well as the associated `FileValue` of the directory. -* **PackageValue**. Represents the parsed version of a BUILD file. Depends on - the `FileValue` of the associated `BUILD` file, and also transitively on any - `DirectoryListingValue` that is used to resolve the globs in the package - (the data structure representing the contents of a `BUILD` file internally). -* **ConfiguredTargetValue**. Represents a configured target, which is a tuple - of the set of actions generated during the analysis of a target and - information provided to dependent configured targets. Depends on the - `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` - of direct dependencies, and a special node representing the build - configuration. -* **ArtifactValue**. Represents a file in the build, be it a source or an - output artifact. Artifacts are almost equivalent to files, and are used to - refer to files during the actual execution of build steps. Source files - depends on the `FileValue` of the associated node, and output artifacts - depend on the `ActionExecutionValue` of whatever action generates the - artifact. -* **ActionExecutionValue**. Represents the execution of an action. Depends on - the `ArtifactValues` of its input files. The action it executes is contained - within its SkyKey, which is contrary to the concept that SkyKeys should be - small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if - the execution phase does not run. - -As a visual aid, this diagram shows the relationships between -SkyFunction implementations after a build of Bazel itself: +This is high level summary of the key `SkyFunction` and `SkyValue` implementations Bazel uses to perform a build: + +- **FileStateValue**. The result of an `lstat()`. For existent files, the function also computes additional information in order to detect changes to the file. This is the lowest level node in the Skyframe graph and has no dependencies. +- **FileValue**. Used by anything that cares about the actual contents or resolved path of a file. Depends on the corresponding `FileStateValue` and any symlinks that need to be resolved (such as the `FileValue` for `a/b` needs the resolved path of `a` and the resolved path of `a/b`). The distinction between `FileValue` and `FileStateValue` is important because the latter can be used in cases where the contents of the file are not actually needed. For example, the file contents are irrelevant when evaluating file system globs (such as `srcs=glob(["*/*.java"])`). +- **DirectoryListingStateValue**. The result of `readdir()`. Like `FileStateValue`, this is the lowest level node and has no dependencies. +- **DirectoryListingValue**. Used by anything that cares about the entries of a directory. Depends on the corresponding `DirectoryListingStateValue`, as well as the associated `FileValue` of the directory. +- **PackageValue**. Represents the parsed version of a BUILD file. Depends on the `FileValue` of the associated `BUILD` file, and also transitively on any `DirectoryListingValue` that is used to resolve the globs in the package (the data structure representing the contents of a `BUILD` file internally). +- **ConfiguredTargetValue**. Represents a configured target, which is a tuple of the set of actions generated during the analysis of a target and information provided to dependent configured targets. Depends on the `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` of direct dependencies, and a special node representing the build configuration. +- **ArtifactValue**. Represents a file in the build, be it a source or an output artifact. Artifacts are almost equivalent to files, and are used to refer to files during the actual execution of build steps. Source files depends on the `FileValue` of the associated node, and output artifacts depend on the `ActionExecutionValue` of whatever action generates the artifact. +- **ActionExecutionValue**. Represents the execution of an action. Depends on the `ArtifactValues` of its input files. The action it executes is contained within its SkyKey, which is contrary to the concept that SkyKeys should be small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if the execution phase does not run. + +As a visual aid, this diagram shows the relationships between SkyFunction implementations after a build of Bazel itself: ![A graph of SkyFunction implementation relationships](/reference/skyframe.png) diff --git a/release/backward-compatibility.mdx b/release/backward-compatibility.mdx index af653ccd..42f40956 100644 --- a/release/backward-compatibility.mdx +++ b/release/backward-compatibility.mdx @@ -2,75 +2,51 @@ title: 'Backward Compatibility' --- +This page provides information about how to handle backward compatibility, including migrating from one release to another and how to communicate incompatible changes. - -This page provides information about how to handle backward compatibility, -including migrating from one release to another and how to communicate -incompatible changes. - -Bazel is evolving. Minor versions released as part of an [LTS major -version](/release#bazel-versioning) are fully backward-compatible. New major LTS -releases may contain incompatible changes that require some migration effort. -For more information about Bazel's release model, please check out the [Release -Model](/release) page. +Bazel is evolving. Minor versions released as part of an [LTS major version](/release#bazel-versioning) are fully backward-compatible. New major LTS releases may contain incompatible changes that require some migration effort. For more information about Bazel's release model, please check out the [Release Model](/release) page. ## Summary -1. It is recommended to use `--incompatible_*` flags for breaking changes. -1. For every `--incompatible_*` flag, a GitHub issue explains the change in - behavior and aims to provide a migration recipe. -1. Incompatible flags are recommended to be back-ported to the latest LTS - release without enabling the flag by default. -1. APIs and behavior guarded by an `--experimental_*` flag can change at any - time. -1. Never run production builds with `--experimental_*` or `--incompatible_*` - flags. +1. It is recommended to use `--incompatible_*` flags for breaking changes. +2. For every `--incompatible_*` flag, a GitHub issue explains the change in behavior and aims to provide a migration recipe. +3. Incompatible flags are recommended to be back-ported to the latest LTS release without enabling the flag by default. +4. APIs and behavior guarded by an `--experimental_*` flag can change at any time. +5. Never run production builds with `--experimental_*` or `--incompatible_*` flags. ## How to follow this policy -* [For Bazel users - how to update Bazel](/install/bazelisk) -* [For contributors - best practices for incompatible changes](/contribute/breaking-changes) -* [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) +- [For Bazel users - how to update Bazel](/install/bazelisk) +- [For contributors - best practices for incompatible changes](/contribute/breaking-changes) +- [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) ## What is stable functionality? -In general, APIs or behaviors without `--experimental_...` flags are considered -stable, supported features in Bazel. +In general, APIs or behaviors without `--experimental_...` flags are considered stable, supported features in Bazel. This includes: -* Starlark language and APIs -* Rules bundled with Bazel -* Bazel APIs such as Remote Execution APIs or Build Event Protocol -* Flags and their semantics +- Starlark language and APIs +- Rules bundled with Bazel +- Bazel APIs such as Remote Execution APIs or Build Event Protocol +- Flags and their semantics ## Incompatible changes and migration recipes -For every incompatible change in a new release, the Bazel team aims to provide a -_migration recipe_ that helps you update your code (`BUILD` and `.bzl` files, as -well as any Bazel usage in scripts, usage of Bazel API, and so on). +For every incompatible change in a new release, the Bazel team aims to provide a *migration recipe* that helps you update your code (`BUILD` and `.bzl` files, as well as any Bazel usage in scripts, usage of Bazel API, and so on). -Incompatible changes should have an associated `--incompatible_*` flag and a -corresponding GitHub issue. +Incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue. -The incompatible flag and relevant changes are recommended to be back-ported to -the latest LTS release without enabling the flag by default. This allows users -to migrate for the incompatible changes before the next LTS release is -available. +The incompatible flag and relevant changes are recommended to be back-ported to the latest LTS release without enabling the flag by default. This allows users to migrate for the incompatible changes before the next LTS release is available. ## Communicating incompatible changes -The primary source of information about incompatible changes are GitHub issues -marked with an ["incompatible-change" -label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). +The primary source of information about incompatible changes are GitHub issues marked with an ["incompatible-change" label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). For every incompatible change, the issue specifies the following: -* Name of the flag controlling the incompatible change -* Description of the changed functionality -* Migration recipe +- Name of the flag controlling the incompatible change +- Description of the changed functionality +- Migration recipe -When an incompatible change is ready for migration with Bazel at HEAD -(therefore, also with the next Bazel rolling release), it should be marked with -the `migration-ready` label. The incompatible change issue is closed when the -incompatible flag is flipped at HEAD. +When an incompatible change is ready for migration with Bazel at HEAD (therefore, also with the next Bazel rolling release), it should be marked with the `migration-ready` label. The incompatible change issue is closed when the incompatible flag is flipped at HEAD. diff --git a/release/index.mdx b/release/index.mdx index f58e8f5f..0d2629e7 100644 --- a/release/index.mdx +++ b/release/index.mdx @@ -2,66 +2,48 @@ title: 'Release Model' --- - - -As announced in [the original blog -post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel -4.0 and higher versions provides support for two release tracks: rolling -releases and long term support (LTS) releases. This page covers the latest -information about Bazel's release model. +As announced in [the original blog post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel 4.0 and higher versions provides support for two release tracks: rolling releases and long term support (LTS) releases. This page covers the latest information about Bazel's release model. ## Support matrix -| LTS release | Support stage | Latest version | End of support | -| ----------- | ------------- | -------------- | -------------- | -| Bazel 9 | Rolling| [Check rolling release page](/release/rolling) | N/A | -| Bazel 8 | Active| [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | -| Bazel 7 | Maintenance| [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | -| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | -| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | -| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | +| LTS release | Support stage | Latest version | End of support | +| ----------- | ------------- | --------------------------------------------------------------- | -------------- | +| Bazel 9 | Rolling | [Check rolling release page](/release/rolling) | N/A | +| Bazel 8 | Active | [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | +| Bazel 7 | Maintenance | [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | +| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | +| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | +| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | -All Bazel LTS releases can be found on the [release -page](https://github.com/bazelbuild/bazel/releases) on GitHub. +All Bazel LTS releases can be found on the [release page](https://github.com/bazelbuild/bazel/releases) on GitHub. -Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are -recommended to upgrade to the latest LTS release or use rolling releases if you -want to keep up with the latest changes at HEAD. +Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are recommended to upgrade to the latest LTS release or use rolling releases if you want to keep up with the latest changes at HEAD. ## Release versioning -Bazel uses a _major.minor.patch_ [Semantic -Versioning](https://semver.org/) scheme. +Bazel uses a *major.minor.patch* [Semantic Versioning](https://semver.org/) scheme. -* A _major release_ contains features that are not backward compatible with - the previous release. Each major Bazel version is an LTS release. -* A _minor release_ contains backward-compatible bug fixes and features - back-ported from the main branch. -* A _patch release_ contains critical bug fixes. +- A *major release* contains features that are not backward compatible with the previous release. Each major Bazel version is an LTS release. +- A *minor release* contains backward-compatible bug fixes and features back-ported from the main branch. +- A *patch release* contains critical bug fixes. -Additionally, pre-release versions are indicated by appending a hyphen and a -date suffix to the next major version number. +Additionally, pre-release versions are indicated by appending a hyphen and a date suffix to the next major version number. For example, a new release of each type would result in these version numbers: -* Major: 6.0.0 -* Minor: 6.1.0 -* Patch: 6.1.2 -* Pre-release: 7.0.0-pre.20230502.1 +- Major: 6.0.0 +- Minor: 6.1.0 +- Patch: 6.1.2 +- Pre-release: 7.0.0-pre.20230502.1 ## Support stages For each major Bazel version, there are four support stages: -* **Rolling**: This major version is still in pre-release, the Bazel team - publishes rolling releases from HEAD. -* **Active**: This major version is the current active LTS release. The Bazel - team backports important features and bug fixes into its minor releases. -* **Maintenance**: This major version is an old LTS release in maintenance - mode. The Bazel team only promises to backport critical bug fixes for - security issues and OS-compatibility issues into this LTS release. -* **Deprecated**: The Bazel team no longer provides support for this major - version, all users should migrate to newer Bazel LTS releases. +- **Rolling**: This major version is still in pre-release, the Bazel team publishes rolling releases from HEAD. +- **Active**: This major version is the current active LTS release. The Bazel team backports important features and bug fixes into its minor releases. +- **Maintenance**: This major version is an old LTS release in maintenance mode. The Bazel team only promises to backport critical bug fixes for security issues and OS-compatibility issues into this LTS release. +- **Deprecated**: The Bazel team no longer provides support for this major version, all users should migrate to newer Bazel LTS releases. ## Release cadence @@ -69,148 +51,90 @@ Bazel regularly publish releases for two release tracks. ### Rolling releases -* Rolling releases are coordinated with Google Blaze release and are released - from HEAD around every two weeks. It is a preview of the next Bazel LTS - release. -* Rolling releases can ship incompatible changes. Incompatible flags are - recommended for major breaking changes, rolling out incompatible changes - should follow our [backward compatibility - policy](/release/backward-compatibility). +- Rolling releases are coordinated with Google Blaze release and are released from HEAD around every two weeks. It is a preview of the next Bazel LTS release. +- Rolling releases can ship incompatible changes. Incompatible flags are recommended for major breaking changes, rolling out incompatible changes should follow our [backward compatibility policy](/release/backward-compatibility). ### LTS releases -* _Major release_: A new LTS release is expected to be cut from HEAD roughly - every - 12 months. Once a new LTS release is out, it immediately enters the Active - stage, and the previous LTS release enters the Maintenance stage. -* _Minor release_: New minor verions on the Active LTS track are expected to - be released once every 2 months. -* _Patch release_: New patch versions for LTS releases in Active and - Maintenance stages are expected to be released on demand for critical bug - fixes. -* A Bazel LTS release enters the Deprecated stage after being in ​​the - Maintenance stage for 2 years. - -For planned releases, please check our [release -issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) -on Github. +- *Major release*: A new LTS release is expected to be cut from HEAD roughly every 12 months. Once a new LTS release is out, it immediately enters the Active stage, and the previous LTS release enters the Maintenance stage. +- *Minor release*: New minor verions on the Active LTS track are expected to be released once every 2 months. +- *Patch release*: New patch versions for LTS releases in Active and Maintenance stages are expected to be released on demand for critical bug fixes. +- A Bazel LTS release enters the Deprecated stage after being in ​​the Maintenance stage for 2 years. + +For planned releases, please check our [release issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) on Github. ## Release procedure & policies -For rolling releases, the process is straightforward: about every two weeks, a -new release is created, aligning with the same baseline as the Google internal -Blaze release. Due to the rapid release schedule, we don't backport any changes -to rolling releases. +For rolling releases, the process is straightforward: about every two weeks, a new release is created, aligning with the same baseline as the Google internal Blaze release. Due to the rapid release schedule, we don't backport any changes to rolling releases. For LTS releases, the procedure and policies below are followed: -1. Determine a baseline commit for the release. - * For a new major LTS release, the baseline commit is the HEAD of the main - branch. - * For a minor or patch release, the baseline commit is the HEAD of the - current latest version of the same LTS release. -1. Create a release branch in the name of `release-` from the baseline - commit. -1. Backport changes via PRs to the release branch. - * The community can suggest certain commits to be back-ported by replying - "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential - release blockers, the Bazel team triages them and decide whether to - back-port the commits. - * Only backward-compatible commits on the main branch can be back-ported, - additional minor changes to resolve merge conflicts are acceptable. -1. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. - * Bazel maintainers can request to cherry-pick specific commit(s) - to a release branch. This process is initiated by creating a - cherry-pick request on GitHub. Here's how to do it. - 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=&labels=&projects=&template=cherry_pick_request.yml) - 2. Fill in the request details - * Title: Provide a concise and descriptive title for the request. - * Commit ID(s): Enter the ID(s) of the commit(s) you want to - cherry-pick. If there are multiple commits, then separate - them with commas. - * Category: Specify the category of the request. - * Reviewer(s): For multiple reviewers, separate their GitHub - ID's with commas. - 3. Set the milestone - * Find the "Milestone" section and click the setting. - * Select the appropriate X.Y.Z release blockers. This action - triggers the cherry-pick bot to process your request - for the "release-X.Y.Z" branch. - 4. Submit the Issue - * Once all details are filled in and the miestone is set, - submit the issue. - - * The cherry-pick bot will process the request and notify - if the commit(s) are eligible for cherry-picking. If - the commits are cherry-pickable, which means there's no - merge conflict while cherry-picking the commit, then - the bot will create a new pull request. When the pull - request is approved by a member of the Bazel team, the - commits are cherry-picked and merged to the release branch. - For a visual example of a completed cherry-pick request, - refer to this - [example](https://github.com/bazelbuild/bazel/issues/20230) - . - -1. Identify release blockers and fix issues found on the release branch. - * The release branch is tested with the same test suite in - [postsubmit](https://buildkite.com/bazel/bazel-bazel) and - [downstream test pipeline] - (https://buildkite.com/bazel/bazel-at-head-plus-downstream) - on Bazel CI. The Bazel team monitors testing results of the release - branch and fixes any regressions found. -1. Create a new release candidate from the release branch when all known - release blockers are resolved. - * The release candidate is announced on - [bazel-discuss](https://groups.google.com/g/bazel-discuss), - the Bazel team monitors community bug reports for the candidate. - * If new release blockers are identified, go back to the last step and - create a new release candidate after resolving all the issues. - * New features are not allowed to be added to the release branch after the - first release candidate is created; cherry-picks are limited to critical - fixes only. If a cherry-pick is needed, the requester must answer the - following questions: Why is this change critical, and what benefits does - it provide? What is the likelihood of this change introducing a - regression? -1. Push the release candidate as the official release if no further release - blockers are found - * For patch releases, push the release at least two business days after - the last release candidate is out. - * For major and minor releases, push the release two business days after - the last release candidate is out, but not earlier than one week after - the first release candidate is out. - * The release is only pushed on a day where the next day is a business - day. - * The release is announced on - [bazel-discuss](https://groups.google.com/g/bazel-discuss), - the Bazel team monitors and addresses community bug reports for the new - release. +1. Determine a baseline commit for the release. + + - For a new major LTS release, the baseline commit is the HEAD of the main branch. + - For a minor or patch release, the baseline commit is the HEAD of the current latest version of the same LTS release. + +2. Create a release branch in the name of `release-<version>` from the baseline commit. + +3. Backport changes via PRs to the release branch. + + - The community can suggest certain commits to be back-ported by replying "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential release blockers, the Bazel team triages them and decide whether to back-port the commits. + - Only backward-compatible commits on the main branch can be back-ported, additional minor changes to resolve merge conflicts are acceptable. + +4. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. + + - Bazel maintainers can request to cherry-pick specific commit(s) to a release branch. This process is initiated by creating a cherry-pick request on GitHub. Here's how to do it. + + 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=\&labels=\&projects=\&template=cherry_pick_request.yml) + + 2. Fill in the request details + + - Title: Provide a concise and descriptive title for the request. + - Commit ID(s): Enter the ID(s) of the commit(s) you want to cherry-pick. If there are multiple commits, then separate them with commas. + - Category: Specify the category of the request. + - Reviewer(s): For multiple reviewers, separate their GitHub ID's with commas. + + 3. Set the milestone + + - Find the "Milestone" section and click the setting. + - Select the appropriate X.Y.Z release blockers. This action triggers the cherry-pick bot to process your request for the "release-X.Y.Z" branch. + + 4. Submit the Issue + - Once all details are filled in and the miestone is set, submit the issue. + + - The cherry-pick bot will process the request and notify if the commit(s) are eligible for cherry-picking. If the commits are cherry-pickable, which means there's no merge conflict while cherry-picking the commit, then the bot will create a new pull request. When the pull request is approved by a member of the Bazel team, the commits are cherry-picked and merged to the release branch. For a visual example of a completed cherry-pick request, refer to this [example](https://github.com/bazelbuild/bazel/issues/20230) . + +5. Identify release blockers and fix issues found on the release branch. + + - The release branch is tested with the same test suite in [postsubmit](https://buildkite.com/bazel/bazel-bazel) and [downstream test pipeline](https://buildkite.com/bazel/bazel-at-head-plus-downstream) on Bazel CI. The Bazel team monitors testing results of the release branch and fixes any regressions found. + +6. Create a new release candidate from the release branch when all known release blockers are resolved. + + - The release candidate is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors community bug reports for the candidate. + - If new release blockers are identified, go back to the last step and create a new release candidate after resolving all the issues. + - New features are not allowed to be added to the release branch after the first release candidate is created; cherry-picks are limited to critical fixes only. If a cherry-pick is needed, the requester must answer the following questions: Why is this change critical, and what benefits does it provide? What is the likelihood of this change introducing a regression? + +7. Push the release candidate as the official release if no further release blockers are found + + - For patch releases, push the release at least two business days after the last release candidate is out. + - For major and minor releases, push the release two business days after the last release candidate is out, but not earlier than one week after the first release candidate is out. + - The release is only pushed on a day where the next day is a business day. + - The release is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors and addresses community bug reports for the new release. ## Report regressions -If a user finds a regression in a new Bazel release, release candidate or even -Bazel at HEAD, please file a bug on -[GitHub](https://github.com/bazelbuild/bazel/issues). You can use -Bazelisk to bisect the culprit commit and include this information in the bug -report. +If a user finds a regression in a new Bazel release, release candidate or even Bazel at HEAD, please file a bug on [GitHub](https://github.com/bazelbuild/bazel/issues). You can use Bazelisk to bisect the culprit commit and include this information in the bug report. -For example, if your build succeeds with Bazel 6.1.0 but fails with the second -release candidate of 6.2.0, you can do bisect via +For example, if your build succeeds with Bazel 6.1.0 but fails with the second release candidate of 6.2.0, you can do bisect via ```bash bazelisk --bisect=6.1.0..release-6.2.0rc2 build //foo:bar ``` -You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run -corresponding bazel commands to reset the build state if it's needed to -reproduce the issue. For more details, check out documentation about Bazelisk -[bisect feature] (https://github.com/bazelbuild/bazelisk#--bisect). +You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run corresponding bazel commands to reset the build state if it's needed to reproduce the issue. For more details, check out documentation about Bazelisk [bisect feature](https://github.com/bazelbuild/bazelisk#--bisect). -Remember to upgrade Bazelisk to the latest version to use the bisect -feature. +Remember to upgrade Bazelisk to the latest version to use the bisect feature. ## Rule compatibility -If you are a rule authors and want to maintain compatibility with different -Bazel versions, please check out the [Rule -Compatibility](/release/rule-compatibility) page. +If you are a rule authors and want to maintain compatibility with different Bazel versions, please check out the [Rule Compatibility](/release/rule-compatibility) page. diff --git a/release/rolling.mdx b/release/rolling.mdx index 8e79d6f5..4cb413ad 100644 --- a/release/rolling.mdx +++ b/release/rolling.mdx @@ -2,13 +2,6 @@ title: 'Rolling Releases' --- +This page contains an overview of all rolling releases, as per our [release policy](https://bazel.build/release#rolling-releases). [Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use these releases. - -This page contains an overview of all rolling releases, as per our -[release policy](https://bazel.build/release#rolling-releases). -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use -these releases. - -## Index - - +## Index diff --git a/release/rule-compatibility.mdx b/release/rule-compatibility.mdx index 05a8a95e..93027953 100644 --- a/release/rule-compatibility.mdx +++ b/release/rule-compatibility.mdx @@ -2,89 +2,55 @@ title: 'Rule Compatibility' --- +Bazel Starlark rules can break compatibility with Bazel LTS releases in the following two scenarios: +1. The rule breaks compatibility with future LTS releases because a feature it depends on is removed from Bazel at HEAD. +2. The rule breaks compatibility with the current or older LTS releases because a feature it depends on is only available in newer Bazel LTS releases. -Bazel Starlark rules can break compatibility with Bazel LTS releases in the -following two scenarios: - -1. The rule breaks compatibility with future LTS releases because a feature it - depends on is removed from Bazel at HEAD. -1. The rule breaks compatibility with the current or older LTS releases because - a feature it depends on is only available in newer Bazel LTS releases. - -Meanwhile, the rule itself can ship incompatible changes for their users as -well. When combined with breaking changes in Bazel, upgrading the rule version -and Bazel version can often be a source of frustration for Bazel users. This -page covers how rules authors should maintain rule compatibility with Bazel to -make it easier for users to upgrade Bazel and rules. +Meanwhile, the rule itself can ship incompatible changes for their users as well. When combined with breaking changes in Bazel, upgrading the rule version and Bazel version can often be a source of frustration for Bazel users. This page covers how rules authors should maintain rule compatibility with Bazel to make it easier for users to upgrade Bazel and rules. ## Manageable migration process -While it's obviously not feasible to guarantee compatibility between every -version of Bazel and every version of the rule, our aim is to ensure that the -migration process remains manageable for Bazel users. A manageable migration -process is defined as a process where **users are not forced to upgrade the -rule's major version and Bazel's major version simultaneously**, thereby -allowing users to handle incompatible changes from one source at a time. +While it's obviously not feasible to guarantee compatibility between every version of Bazel and every version of the rule, our aim is to ensure that the migration process remains manageable for Bazel users. A manageable migration process is defined as a process where **users are not forced to upgrade the rule's major version and Bazel's major version simultaneously**, thereby allowing users to handle incompatible changes from one source at a time. For example, with the following compatibility matrix: -* Migrating from rules_foo 1.x + Bazel 4.x to rules_foo 2.x + Bazel 5.x is not - considered manageable, as the users need to upgrade the major version of - rules_foo and Bazel at the same time. -* Migrating from rules_foo 2.x + Bazel 5.x to rules_foo 3.x + Bazel 6.x is - considered manageable, as the users can first upgrade rules_foo from 2.x to - 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to - 6.x. +- Migrating from rules\_foo 1.x + Bazel 4.x to rules\_foo 2.x + Bazel 5.x is not considered manageable, as the users need to upgrade the major version of rules\_foo and Bazel at the same time. +- Migrating from rules\_foo 2.x + Bazel 5.x to rules\_foo 3.x + Bazel 6.x is considered manageable, as the users can first upgrade rules\_foo from 2.x to 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to 6.x. -| | rules_foo 1.x | rules_foo 2.x | rules_foo 3.x | HEAD | -| --- | --- | --- | --- | --- | -| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | -| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | -| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | -| HEAD | ❌ | ❌ | ❌ | ✅ | +| | rules\_foo 1.x | rules\_foo 2.x | rules\_foo 3.x | HEAD | +| --------- | -------------- | -------------- | -------------- | ---- | +| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | +| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | +| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | +| HEAD | ❌ | ❌ | ❌ | ✅ | -❌: No version of the major rule version is compatible with the Bazel LTS -release. +❌: No version of the major rule version is compatible with the Bazel LTS release. -✅: At least one version of the rule is compatible with the latest version of the -Bazel LTS release. +✅: At least one version of the rule is compatible with the latest version of the Bazel LTS release. ## Best practices -As Bazel rules authors, you can ensure a manageable migration process for users -by following these best practices: - -1. The rule should follow [Semantic - Versioning](https://semver.org/): minor versions of the same - major version are backward compatible. -1. The rule at HEAD should be compatible with the latest Bazel LTS release. -1. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, - you can - * Set up your own CI testing with Bazel at HEAD - * Add your project to [Bazel downstream - testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); - the Bazel team files issues to your project if breaking changes in Bazel - affect your project, and you must follow our [downstream project - policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) - to address issues timely. -1. The latest major version of the rule must be compatible with the latest - Bazel LTS release. -1. A new major version of the rule should be compatible with the last Bazel LTS - release supported by the previous major version of the rule. - -Achieving 2. and 3. is the most important task since it allows achieving 4. and -5. naturally. - -To make it easier to keep compatibility with both Bazel at HEAD and the latest -Bazel LTS release, rules authors can: - -* Request backward-compatible features to be back-ported to the latest LTS - release, check out [release process](/release#release-procedure-policies) - for more details. -* Use [bazel_features](https://github.com/bazel-contrib/bazel_features) - to do Bazel feature detection. - -In general, with the recommended approaches, rules should be able to migrate for -Bazel incompatible changes and make use of new Bazel features at HEAD without -dropping compatibility with the latest Bazel LTS release. +As Bazel rules authors, you can ensure a manageable migration process for users by following these best practices: + +1. The rule should follow [Semantic Versioning](https://semver.org/): minor versions of the same major version are backward compatible. + +2. The rule at HEAD should be compatible with the latest Bazel LTS release. + +3. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, you can + + - Set up your own CI testing with Bazel at HEAD + - Add your project to [Bazel downstream testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); the Bazel team files issues to your project if breaking changes in Bazel affect your project, and you must follow our [downstream project policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) to address issues timely. + +4. The latest major version of the rule must be compatible with the latest Bazel LTS release. + +5. A new major version of the rule should be compatible with the last Bazel LTS release supported by the previous major version of the rule. + +Achieving 2. and 3. is the most important task since it allows achieving 4. and 5. naturally. + +To make it easier to keep compatibility with both Bazel at HEAD and the latest Bazel LTS release, rules authors can: + +- Request backward-compatible features to be back-ported to the latest LTS release, check out [release process](/release#release-procedure-policies) for more details. +- Use [bazel\_features](https://github.com/bazel-contrib/bazel_features) to do Bazel feature detection. + +In general, with the recommended approaches, rules should be able to migrate for Bazel incompatible changes and make use of new Bazel features at HEAD without dropping compatibility with the latest Bazel LTS release. diff --git a/remote/bep-examples.mdx b/remote/bep-examples.mdx index faf11bf9..68e4d42c 100644 --- a/remote/bep-examples.mdx +++ b/remote/bep-examples.mdx @@ -2,14 +2,9 @@ title: 'Build Event Protocol Examples' --- +The full specification of the Build Event Protocol can be found in its protocol buffer definition. However, it might be helpful to build up some intuition before looking at the specification. - -The full specification of the Build Event Protocol can be found in its protocol -buffer definition. However, it might be helpful to build up some intuition -before looking at the specification. - -Consider a simple Bazel workspace that consists of two empty shell scripts -`foo.sh` and `foo_test.sh` and the following `BUILD` file: +Consider a simple Bazel workspace that consists of two empty shell scripts `foo.sh` and `foo_test.sh` and the following `BUILD` file: ```bash sh_library( @@ -24,58 +19,32 @@ sh_test( ) ``` -When running `bazel test ...` on this project the build graph of the generated -build events will resemble the graph below. The arrows indicate the -aforementioned parent and child relationship. Note that some build events and -most fields have been omitted for brevity. +When running `bazel test ...` on this project the build graph of the generated build events will resemble the graph below. The arrows indicate the aforementioned parent and child relationship. Note that some build events and most fields have been omitted for brevity. ![bep-graph](/docs/images/bep-graph.png "BEP graph") **Figure 1.** BEP graph. -Initially, a `BuildStarted` event is published. The event informs us that the -build was invoked through the `bazel test` command and announces child events: +Initially, a `BuildStarted` event is published. The event informs us that the build was invoked through the `bazel test` command and announces child events: -* `OptionsParsed` -* `WorkspaceStatus` -* `CommandLine` -* `UnstructuredCommandLine` -* `BuildMetadata` -* `BuildFinished` -* `PatternExpanded` -* `Progress` +- `OptionsParsed` +- `WorkspaceStatus` +- `CommandLine` +- `UnstructuredCommandLine` +- `BuildMetadata` +- `BuildFinished` +- `PatternExpanded` +- `Progress` The first three events provide information about how Bazel was invoked. -The `PatternExpanded` build event provides insight -into which specific targets the `...` pattern expanded to: -`//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two -`TargetConfigured` events as children. Note that the `TargetConfigured` event -declares the `Configuration` event as a child event, even though `Configuration` -has been posted before the `TargetConfigured` event. - -Besides the parent and child relationship, events may also refer to each other -using their build event identifiers. For example, in the above graph the -`TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` -field. - -Build events that refer to files don’t usually embed the file -names and paths in the event. Instead, they contain the build event identifier -of a `NamedSetOfFiles` event, which will then contain the actual file names and -paths. The `NamedSetOfFiles` event allows a set of files to be reported once and -referred to by many targets. This structure is necessary because otherwise in -some cases the Build Event Protocol output size would grow quadratically with -the number of files. A `NamedSetOfFiles` event may also not have all its files -embedded, but instead refer to other `NamedSetOfFiles` events through their -build event identifiers. - -Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` -target from the above graph, printed in protocol buffer’s JSON representation. -The build event identifier contains the target as an opaque string and refers to -the `Configuration` event using its build event identifier. The event does not -announce any child events. The payload contains information about whether the -target was built successfully, the set of output files, and the kind of target -built. +The `PatternExpanded` build event provides insight into which specific targets the `...` pattern expanded to: `//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two `TargetConfigured` events as children. Note that the `TargetConfigured` event declares the `Configuration` event as a child event, even though `Configuration` has been posted before the `TargetConfigured` event. + +Besides the parent and child relationship, events may also refer to each other using their build event identifiers. For example, in the above graph the `TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` field. + +Build events that refer to files don’t usually embed the file names and paths in the event. Instead, they contain the build event identifier of a `NamedSetOfFiles` event, which will then contain the actual file names and paths. The `NamedSetOfFiles` event allows a set of files to be reported once and referred to by many targets. This structure is necessary because otherwise in some cases the Build Event Protocol output size would grow quadratically with the number of files. A `NamedSetOfFiles` event may also not have all its files embedded, but instead refer to other `NamedSetOfFiles` events through their build event identifiers. + +Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` target from the above graph, printed in protocol buffer’s JSON representation. The build event identifier contains the target as an opaque string and refers to the `Configuration` event using its build event identifier. The event does not announce any child events. The payload contains information about whether the target was built successfully, the set of output files, and the kind of target built. ```json { @@ -102,18 +71,9 @@ built. ## Aspect Results in BEP -Ordinary builds evaluate actions associated with `(target, configuration)` -pairs. When building with [aspects](/extending/aspects) enabled, Bazel -additionally evaluates targets associated with `(target, configuration, -aspect)` triples, for each target affected by a given enabled aspect. +Ordinary builds evaluate actions associated with `(target, configuration)` pairs. When building with [aspects](/extending/aspects) enabled, Bazel additionally evaluates targets associated with `(target, configuration, aspect)` triples, for each target affected by a given enabled aspect. -Evaluation results for aspects are available in BEP despite the absence of -aspect-specific event types. For each `(target, configuration)` pair with an -applicable aspect, Bazel publishes an additional `TargetConfigured` and -`TargetComplete` event bearing the result from applying the aspect to the -target. For example, if `//:foo_lib` is built with -`--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in -the BEP: +Evaluation results for aspects are available in BEP despite the absence of aspect-specific event types. For each `(target, configuration)` pair with an applicable aspect, Bazel publishes an additional `TargetConfigured` and `TargetComplete` event bearing the result from applying the aspect to the target. For example, if `//:foo_lib` is built with `--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in the BEP: ```json { @@ -138,37 +98,21 @@ the BEP: } ``` -Note: The only difference between the IDs is the presence of the `aspect` -field. A tool that does not check the `aspect` ID field and accumulates output -files by target may conflate target outputs with aspect outputs. +Note: The only difference between the IDs is the presence of the `aspect` field. A tool that does not check the `aspect` ID field and accumulates output files by target may conflate target outputs with aspect outputs. ## Consuming `NamedSetOfFiles` -Determining the artifacts produced by a given target (or aspect) is a common -BEP use-case that can be done efficiently with some preparation. This section -discusses the recursive, shared structure offered by the `NamedSetOfFiles` -event, which matches the structure of a Starlark [Depset](/extending/depsets). +Determining the artifacts produced by a given target (or aspect) is a common BEP use-case that can be done efficiently with some preparation. This section discusses the recursive, shared structure offered by the `NamedSetOfFiles` event, which matches the structure of a Starlark [Depset](/extending/depsets). -Consumers must take care to avoid quadratic algorithms when processing -`NamedSetOfFiles` events because large builds can contain tens of thousands of -such events, requiring hundreds of millions operations in a traversal with -quadratic complexity. +Consumers must take care to avoid quadratic algorithms when processing `NamedSetOfFiles` events because large builds can contain tens of thousands of such events, requiring hundreds of millions operations in a traversal with quadratic complexity. ![namedsetoffiles-bep-graph](/docs/images/namedsetoffiles-bep-graph.png "NamedSetOfFiles BEP graph") **Figure 2.** `NamedSetOfFiles` BEP graph. -A `NamedSetOfFiles` event always appears in the BEP stream *before* a -`TargetComplete` or `NamedSetOfFiles` event that references it. This is the -inverse of the "parent-child" event relationship, where all but the first event -appears after at least one event announcing it. A `NamedSetOfFiles` event is -announced by a `Progress` event with no semantics. - -Given these ordering and sharing constraints, a typical consumer must buffer all -`NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON -event stream and Python code demonstrate how to populate a map from -target/aspect to built artifacts in the "default" output group, and how to -process the outputs for a subset of built targets/aspects: +A `NamedSetOfFiles` event always appears in the BEP stream *before* a `TargetComplete` or `NamedSetOfFiles` event that references it. This is the inverse of the "parent-child" event relationship, where all but the first event appears after at least one event announcing it. A `NamedSetOfFiles` event is announced by a `Progress` event with no semantics. + +Given these ordering and sharing constraints, a typical consumer must buffer all `NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON event stream and Python code demonstrate how to populate a map from target/aspect to built artifacts in the "default" output group, and how to process the outputs for a subset of built targets/aspects: ```python named_sets = {} # type: dict[str, NamedSetOfFiles] diff --git a/remote/bep-glossary.mdx b/remote/bep-glossary.mdx index 3bd11eea..66f77ea5 100644 --- a/remote/bep-glossary.mdx +++ b/remote/bep-glossary.mdx @@ -2,22 +2,13 @@ title: 'Build Event Protocol Glossary' --- - - -Each BEP event type has its own semantics, minimally documented in -[build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). -The following glossary describes each event type. +Each BEP event type has its own semantics, minimally documented in [build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). The following glossary describes each event type. ## Aborted -Unlike other events, `Aborted` does not have a corresponding ID type, because -the `Aborted` event *replaces* events of other types. This event indicates that -the build terminated early and the event ID it appears under was not produced -normally. `Aborted` contains an enum and human-friendly description to explain -why the build did not complete. +Unlike other events, `Aborted` does not have a corresponding ID type, because the `Aborted` event *replaces* events of other types. This event indicates that the build terminated early and the event ID it appears under was not produced normally. `Aborted` contains an enum and human-friendly description to explain why the build did not complete. -For example, if a build is evaluating a target when the user interrupts Bazel, -BEP contains an event like the following: +For example, if a build is evaluating a target when the user interrupts Bazel, BEP contains an event like the following: ```json { @@ -37,35 +28,21 @@ BEP contains an event like the following: ## ActionExecuted -Provides details about the execution of a specific -[Action](/rules/lib/actions) in a build. By default, this event is -included in the BEP only for failed actions, to support identifying the root cause -of build failures. Users may set the `--build_event_publish_all_actions` flag -to include all `ActionExecuted` events. +Provides details about the execution of a specific [Action](/rules/lib/actions) in a build. By default, this event is included in the BEP only for failed actions, to support identifying the root cause of build failures. Users may set the `--build_event_publish_all_actions` flag to include all `ActionExecuted` events. ## BuildFinished -A single `BuildFinished` event is sent after the command is complete and -includes the exit code for the command. This event provides authoritative -success/failure information. +A single `BuildFinished` event is sent after the command is complete and includes the exit code for the command. This event provides authoritative success/failure information. ## BuildMetadata -Contains the parsed contents of the `--build_metadata` flag. This event exists -to support Bazel integration with other tooling by plumbing external data (such as -identifiers). +Contains the parsed contents of the `--build_metadata` flag. This event exists to support Bazel integration with other tooling by plumbing external data (such as identifiers). ## BuildMetrics -A single `BuildMetrics` event is sent at the end of every command and includes -counters/gauges useful for quantifying the build tool's behavior during the -command. These metrics indicate work actually done and does not count cached -work that is reused. +A single `BuildMetrics` event is sent at the end of every command and includes counters/gauges useful for quantifying the build tool's behavior during the command. These metrics indicate work actually done and does not count cached work that is reused. -Note that `memory_metrics` may not be populated if there was no Java garbage -collection during the command's execution. Users may set the -`--memory_profile=/dev/null` option which forces the garbage -collector to run at the end of the command to populate `memory_metrics`. +Note that `memory_metrics` may not be populated if there was no Java garbage collection during the command's execution. Users may set the `--memory_profile=/dev/null` option which forces the garbage collector to run at the end of the command to populate `memory_metrics`. ```json { @@ -94,14 +71,11 @@ collector to run at the end of the command to populate `memory_metrics`. ## BuildStarted -The first event in a BEP stream, `BuildStarted` includes metadata describing the -command before any meaningful work begins. +The first event in a BEP stream, `BuildStarted` includes metadata describing the command before any meaningful work begins. ## BuildToolLogs -A single `BuildToolLogs` event is sent at the end of a command, including URIs -of files generated by the build tool that may aid in understanding or debugging -build tool behavior. Some information may be included inline. +A single `BuildToolLogs` event is sent at the end of a command, including URIs of files generated by the build tool that may aid in understanding or debugging build tool behavior. Some information may be included inline. ```json { @@ -130,28 +104,15 @@ build tool behavior. Some information may be included inline. ## CommandLine -The BEP contains multiple `CommandLine` events containing representations of all -command-line arguments (including options and uninterpreted arguments). -Each `CommandLine` event has a label in its `StructuredCommandLineId` that -indicates which representation it conveys; three such events appear in the BEP: - -* `"original"`: Reconstructed commandline as Bazel received it from the Bazel - client, without startup options sourced from .rc files. -* `"canonical"`: The effective commandline with .rc files expanded and - invocation policy applied. -* `"tool"`: Populated from the `--experimental_tool_command_line` option. This - is useful to convey the command-line of a tool wrapping Bazel through the BEP. - This could be a base64-encoded `CommandLine` binary protocol buffer message - which is used directly, or a string which is parsed but not interpreted (as - the tool's options may differ from Bazel's). +The BEP contains multiple `CommandLine` events containing representations of all command-line arguments (including options and uninterpreted arguments). Each `CommandLine` event has a label in its `StructuredCommandLineId` that indicates which representation it conveys; three such events appear in the BEP: + +- `"original"`: Reconstructed commandline as Bazel received it from the Bazel client, without startup options sourced from .rc files. +- `"canonical"`: The effective commandline with .rc files expanded and invocation policy applied. +- `"tool"`: Populated from the `--experimental_tool_command_line` option. This is useful to convey the command-line of a tool wrapping Bazel through the BEP. This could be a base64-encoded `CommandLine` binary protocol buffer message which is used directly, or a string which is parsed but not interpreted (as the tool's options may differ from Bazel's). ## Configuration -A `Configuration` event is sent for every [`configuration`](/extending/config) -used in the top-level targets in a build. At least one configuration event is -always be present. The `id` is reused by the `TargetConfigured` and -`TargetComplete` event IDs and is necessary to disambiguate those events in -multi-configuration builds. +A `Configuration` event is sent for every [`configuration`](/extending/config) used in the top-level targets in a build. At least one configuration event is always be present. The `id` is reused by the `TargetConfigured` and `TargetComplete` event IDs and is necessary to disambiguate those events in multi-configuration builds. ```json { @@ -176,11 +137,7 @@ multi-configuration builds. ## ConvenienceSymlinksIdentified -**Experimental.** If the `--experimental_convenience_symlinks_bep_event` -option is set, a single `ConvenienceSymlinksIdentified` event is produced by -`build` commands to indicate how symlinks in the workspace should be managed. -This enables building tools that invoke Bazel remotely then arrange the local -workspace as if Bazel had been run locally. +**Experimental.** If the `--experimental_convenience_symlinks_bep_event` option is set, a single `ConvenienceSymlinksIdentified` event is produced by `build` commands to indicate how symlinks in the workspace should be managed. This enables building tools that invoke Bazel remotely then arrange the local workspace as if Bazel had been run locally. ```json { @@ -211,24 +168,17 @@ workspace as if Bazel had been run locally. ## Fetch -Indicates that a Fetch operation occurred as a part of the command execution. -Unlike other events, if a cached fetch result is re-used, this event does not -appear in the BEP stream. +Indicates that a Fetch operation occurred as a part of the command execution. Unlike other events, if a cached fetch result is re-used, this event does not appear in the BEP stream. ## NamedSetOfFiles -`NamedSetOfFiles` events report a structure matching a -[`depset`](/extending/depsets) of files produced during command evaluation. -Transitively included depsets are identified by `NamedSetOfFilesId`. +`NamedSetOfFiles` events report a structure matching a [`depset`](/extending/depsets) of files produced during command evaluation. Transitively included depsets are identified by `NamedSetOfFilesId`. -For more information on interpreting a stream's `NamedSetOfFiles` events, see the -[BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). +For more information on interpreting a stream's `NamedSetOfFiles` events, see the [BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). ## OptionsParsed -A single `OptionsParsed` event lists all options applied to the command, -separating startup options from command options. It also includes the -[InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. +A single `OptionsParsed` event lists all options applied to the command, separating startup options from command options. It also includes the [InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. ```json { @@ -264,13 +214,7 @@ separating startup options from command options. It also includes the ## PatternExpanded -`PatternExpanded` events indicate the set of all targets that match the patterns -supplied on the commandline. For successful commands, a single event is present -with all patterns in the `PatternExpandedId` and all targets in the -`PatternExpanded` event's *children*. If the pattern expands to any -`test_suite`s the set of test targets included by the `test_suite`. For each -pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) -event with a `PatternExpandedId` identifying the pattern. +`PatternExpanded` events indicate the set of all targets that match the patterns supplied on the commandline. For successful commands, a single event is present with all patterns in the `PatternExpandedId` and all targets in the `PatternExpanded` event's *children*. If the pattern expands to any `test_suite`s the set of test targets included by the `test_suite`. For each pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) event with a `PatternExpandedId` identifying the pattern. ```json { @@ -294,16 +238,11 @@ event with a `PatternExpandedId` identifying the pattern. ## Progress -Progress events contain the standard output and standard error produced by Bazel -during command execution. These events are also auto-generated as needed to -announce events that have not been announced by a logical "parent" event (in -particular, [NamedSetOfFiles](#namedsetoffiles).) +Progress events contain the standard output and standard error produced by Bazel during command execution. These events are also auto-generated as needed to announce events that have not been announced by a logical "parent" event (in particular, [NamedSetOfFiles](#namedsetoffiles).) ## TargetComplete -For each `(target, configuration, aspect)` combination that completes the -execution phase, a `TargetComplete` event is included in BEP. The event contains -the target's success/failure and the target's requested output groups. +For each `(target, configuration, aspect)` combination that completes the execution phase, a `TargetComplete` event is included in BEP. The event contains the target's success/failure and the target's requested output groups. ```json { @@ -333,14 +272,9 @@ the target's success/failure and the target's requested output groups. ## TargetConfigured -For each Target that completes the analysis phase, a `TargetConfigured` event is -included in BEP. This is the authoritative source for a target's "rule kind" -attribute. The configuration(s) applied to the target appear in the announced -*children* of the event. +For each Target that completes the analysis phase, a `TargetConfigured` event is included in BEP. This is the authoritative source for a target's "rule kind" attribute. The configuration(s) applied to the target appear in the announced *children* of the event. -For example, building with the `--experimental_multi_cpu` options may produce -the following `TargetConfigured` event for a single target with two -configurations: +For example, building with the `--experimental_multi_cpu` options may produce the following `TargetConfigured` event for a single target with two configurations: ```json { @@ -375,42 +309,26 @@ configurations: ## TargetSummary -For each `(target, configuration)` pair that is executed, a `TargetSummary` -event is included with an aggregate success result encompassing the configured -target's execution and all aspects applied to that configured target. +For each `(target, configuration)` pair that is executed, a `TargetSummary` event is included with an aggregate success result encompassing the configured target's execution and all aspects applied to that configured target. ## TestResult -If testing is requested, a `TestResult` event is sent for each test attempt, -shard, and run per test. This allows BEP consumers to identify precisely which -test actions failed their tests and identify the test outputs (such as logs, -test.xml files) for each test action. +If testing is requested, a `TestResult` event is sent for each test attempt, shard, and run per test. This allows BEP consumers to identify precisely which test actions failed their tests and identify the test outputs (such as logs, test.xml files) for each test action. ## TestSummary -If testing is requested, a `TestSummary` event is sent for each test `(target, -configuration)`, containing information necessary to interpret the test's -results. The number of attempts, shards and runs per test are included to enable -BEP consumers to differentiate artifacts across these dimensions. The attempts -and runs per test are considered while producing the aggregate `TestStatus` to -differentiate `FLAKY` tests from `FAILED` tests. +If testing is requested, a `TestSummary` event is sent for each test `(target, configuration)`, containing information necessary to interpret the test's results. The number of attempts, shards and runs per test are included to enable BEP consumers to differentiate artifacts across these dimensions. The attempts and runs per test are considered while producing the aggregate `TestStatus` to differentiate `FLAKY` tests from `FAILED` tests. ## UnstructuredCommandLine -Unlike [CommandLine](#commandline), this event carries the unparsed commandline -flags in string form as encountered by the build tool after expanding all -[`.bazelrc`](/run/bazelrc) files and -considering the `--config` flag. +Unlike [CommandLine](#commandline), this event carries the unparsed commandline flags in string form as encountered by the build tool after expanding all [`.bazelrc`](/run/bazelrc) files and considering the `--config` flag. -The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a -given command execution. +The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a given command execution. ## WorkspaceConfig -A single `WorkspaceConfig` event contains configuration information regarding the -workspace, such as the execution root. +A single `WorkspaceConfig` event contains configuration information regarding the workspace, such as the execution root. ## WorkspaceStatus -A single `WorkspaceStatus` event contains the result of the [workspace status -command](/docs/user-manual#workspace-status). +A single `WorkspaceStatus` event contains the result of the [workspace status command](/docs/user-manual#workspace-status). diff --git a/remote/bep.mdx b/remote/bep.mdx index bafdaa9e..4589f8b0 100644 --- a/remote/bep.mdx +++ b/remote/bep.mdx @@ -2,67 +2,28 @@ title: 'Build Event Protocol' --- +The [Build Event Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) (BEP) allows third-party programs to gain insight into a Bazel invocation. For example, you could use the BEP to gather information for an IDE plugin or a dashboard that displays build results. +The protocol is a set of [protocol buffer](https://developers.google.com/protocol-buffers/) messages with some semantics defined on top of it. It includes information about build and test results, build progress, the build configuration and much more. The BEP is intended to be consumed programmatically and makes parsing Bazel’s command line output a thing of the past. -The [Build Event -Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) -(BEP) allows third-party programs to gain insight into a Bazel invocation. For -example, you could use the BEP to gather information for an IDE -plugin or a dashboard that displays build results. - -The protocol is a set of [protocol -buffer](https://developers.google.com/protocol-buffers/) messages with some -semantics defined on top of it. It includes information about build and test -results, build progress, the build configuration and much more. The BEP is -intended to be consumed programmatically and makes parsing Bazel’s -command line output a thing of the past. - -The Build Event Protocol represents information about a build as events. A -build event is a protocol buffer message consisting of a build event identifier, -a set of child event identifiers, and a payload. - -* __Build Event Identifier:__ Depending on the kind of build event, it might be -an [opaque -string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) -or [structured -information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) -revealing more about the build event. A build event identifier is unique within -a build. - -* __Children:__ A build event may announce other build events, by including -their build event identifiers in its [children -field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). -For example, the `PatternExpanded` build event announces the targets it expands -to as children. The protocol guarantees that all events, except for the first -event, are announced by a previous event. - -* __Payload:__ The payload contains structured information about a build event, -encoded as a protocol buffer message specific to that event. Note that the -payload might not be the expected type, but could be an `Aborted` message -if the build aborted prematurely. +The Build Event Protocol represents information about a build as events. A build event is a protocol buffer message consisting of a build event identifier, a set of child event identifiers, and a payload. + +- **Build Event Identifier:** Depending on the kind of build event, it might be an [opaque string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) or [structured information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) revealing more about the build event. A build event identifier is unique within a build. + +- **Children:** A build event may announce other build events, by including their build event identifiers in its [children field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). For example, the `PatternExpanded` build event announces the targets it expands to as children. The protocol guarantees that all events, except for the first event, are announced by a previous event. + +- **Payload:** The payload contains structured information about a build event, encoded as a protocol buffer message specific to that event. Note that the payload might not be the expected type, but could be an `Aborted` message if the build aborted prematurely. ### Build event graph -All build events form a directed acyclic graph through their parent and child -relationship. Every build event except for the initial build event has one or -more parent events. Please note that not all parent events of a child event must -necessarily be posted before it. When a build is complete (succeeded or failed) -all announced events will have been posted. In case of a Bazel crash or a failed -network transport, some announced build events may never be posted. - -The event graph's structure reflects the lifecycle of a command. Every BEP -graph has the following characteristic shape: - -1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) - event. All other events are its descendants. -1. Immediate children of the BuildStarted event contain metadata about the - command. -1. Events containing data produced by the command, such as files built and test - results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) - event. -1. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed - by events containing summary information about the build (for example, metric - or profiling data). +All build events form a directed acyclic graph through their parent and child relationship. Every build event except for the initial build event has one or more parent events. Please note that not all parent events of a child event must necessarily be posted before it. When a build is complete (succeeded or failed) all announced events will have been posted. In case of a Bazel crash or a failed network transport, some announced build events may never be posted. + +The event graph's structure reflects the lifecycle of a command. Every BEP graph has the following characteristic shape: + +1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) event. All other events are its descendants. +2. Immediate children of the BuildStarted event contain metadata about the command. +3. Events containing data produced by the command, such as files built and test results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) event. +4. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed by events containing summary information about the build (for example, metric or profiling data). ## Consuming Build Event Protocol @@ -70,21 +31,13 @@ graph has the following characteristic shape: To consume the BEP in a binary format: -1. Have Bazel serialize the protocol buffer messages to a file by specifying the - option `--build_event_binary_file=/path/to/file`. The file will contain - serialized protocol buffer messages with each message being length delimited. - Each message is prefixed with its length encoded as a variable length integer. - This format can be read using the protocol buffer library’s - [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) - method. +1. Have Bazel serialize the protocol buffer messages to a file by specifying the option `--build_event_binary_file=/path/to/file`. The file will contain serialized protocol buffer messages with each message being length delimited. Each message is prefixed with its length encoded as a variable length integer. This format can be read using the protocol buffer library’s [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) method. -2. Then, write a program that extracts the relevant information from the - serialized protocol buffer message. +2. Then, write a program that extracts the relevant information from the serialized protocol buffer message. ### Consume in text or JSON formats -The following Bazel command line flags will output the BEP in -human-readable formats, such as text and JSON: +The following Bazel command line flags will output the BEP in human-readable formats, such as text and JSON: ``` --build_event_text_file @@ -93,56 +46,34 @@ human-readable formats, such as text and JSON: ## Build Event Service -The [Build Event -Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) -Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event -Service protocol is independent of the BEP and treats BEP events as opaque bytes. -Bazel ships with a gRPC client implementation of the Build Event Service protocol that -publishes Build Event Protocol events. One can specify the endpoint to send the -events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, -you must prefix the address with the appropriate scheme: `grpc://` for plaintext -gRPC and `grpcs://` for gRPC with TLS enabled. +The [Build Event Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event Service protocol is independent of the BEP and treats BEP events as opaque bytes. Bazel ships with a gRPC client implementation of the Build Event Service protocol that publishes Build Event Protocol events. One can specify the endpoint to send the events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, you must prefix the address with the appropriate scheme: `grpc://` for plaintext gRPC and `grpcs://` for gRPC with TLS enabled. ### Build Event Service flags Bazel has several flags related to the Build Event Service protocol, including: -* `--bes_backend` -* `--[no]bes_lifecycle_events` -* `--bes_results_url` -* `--bes_timeout` -* `--bes_instance_name` +- `--bes_backend` +- `--[no]bes_lifecycle_events` +- `--bes_results_url` +- `--bes_timeout` +- `--bes_instance_name` -For a description of each of these flags, see the -[Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). ### Authentication and security -Bazel’s Build Event Service implementation also supports authentication and TLS. -These settings can be controlled using the below flags. Please note that these -flags are also used for Bazel’s Remote Execution. This implies that the Build -Event Service and Remote Execution Endpoints need to share the same -authentication and TLS infrastructure. +Bazel’s Build Event Service implementation also supports authentication and TLS. These settings can be controlled using the below flags. Please note that these flags are also used for Bazel’s Remote Execution. This implies that the Build Event Service and Remote Execution Endpoints need to share the same authentication and TLS infrastructure. -* `--[no]google_default_credentials` -* `--google_credentials` -* `--google_auth_scopes` -* `--tls_certificate` -* `--[no]tls_enabled` +- `--[no]google_default_credentials` +- `--google_credentials` +- `--google_auth_scopes` +- `--tls_certificate` +- `--[no]tls_enabled` -For a description of each of these flags, see the -[Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). ### Build Event Service and remote caching -The BEP typically contains many references to log files (test.log, test.xml, -etc. ) stored on the machine where Bazel is running. A remote BES server -typically can't access these files as they are on different machines. A way to -work around this issue is to use Bazel with [remote -caching](/remote/caching). -Bazel will upload all output files to the remote cache (including files -referenced in the BEP) and the BES server can then fetch the referenced files -from the cache. - -See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for -more details. +The BEP typically contains many references to log files (test.log, test.xml, etc. ) stored on the machine where Bazel is running. A remote BES server typically can't access these files as they are on different machines. A way to work around this issue is to use Bazel with [remote caching](/remote/caching). Bazel will upload all output files to the remote cache (including files referenced in the BEP) and the BES server can then fetch the referenced files from the cache. + +See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for more details. diff --git a/remote/cache-local.mdx b/remote/cache-local.mdx index a3415a64..49b8d33b 100644 --- a/remote/cache-local.mdx +++ b/remote/cache-local.mdx @@ -2,39 +2,23 @@ title: 'Debugging Remote Cache Hits for Local Execution' --- +This page describes how to investigate cache misses in the context of local execution. +This page assumes that you have a build and/or test that successfully builds locally and is set up to utilize remote caching, and that you want to ensure that the remote cache is being effectively utilized. -This page describes how to investigate cache misses in the context of local -execution. - -This page assumes that you have a build and/or test that successfully builds -locally and is set up to utilize remote caching, and that you want to ensure -that the remote cache is being effectively utilized. - -For tips on how to check your cache hit rate and how to compare the execution -logs between two Bazel invocations, see -[Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). -Everything presented in that guide also applies to remote caching with local -execution. However, local execution presents some additional challenges. +For tips on how to check your cache hit rate and how to compare the execution logs between two Bazel invocations, see [Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). Everything presented in that guide also applies to remote caching with local execution. However, local execution presents some additional challenges. ## Checking your cache hit rate -Successful remote cache hits will show up in the status line, similar to -[Cache Hits rate with Remote -Execution](/remote/cache-remote#check-cache-hits). +Successful remote cache hits will show up in the status line, similar to [Cache Hits rate with Remote Execution](/remote/cache-remote#check-cache-hits). -In the standard output of your Bazel run, you will see something like the -following: +In the standard output of your Bazel run, you will see something like the following: -```none {:.devsite-disable-click-to-copy} +```none INFO: 7 processes: 3 remote cache hit, 4 linux-sandbox. ``` -This means that out of 7 attempted actions, 3 got a remote cache hit and 4 -actions did not have cache hits and were executed locally using `linux-sandbox` -strategy. Local cache hits are not included in this summary. If you are getting -0 processes (or a number lower than expected), run `bazel clean` followed by -your build/test command. +This means that out of 7 attempted actions, 3 got a remote cache hit and 4 actions did not have cache hits and were executed locally using `linux-sandbox` strategy. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. ## Troubleshooting cache hits @@ -42,50 +26,30 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure successful communication with the remote endpoint -To ensure your build is successfully communicating with the remote cache, follow -the steps in this section. +To ensure your build is successfully communicating with the remote cache, follow the steps in this section. 1. Check your output for warnings - With remote execution, a failure to talk to the remote endpoint would fail - your build. On the other hand, a cacheable local build would not fail if it - cannot cache. Check the output of your Bazel invocation for warnings, such - as: + With remote execution, a failure to talk to the remote endpoint would fail your build. On the other hand, a cacheable local build would not fail if it cannot cache. Check the output of your Bazel invocation for warnings, such as: - ```none + ```none WARNING: Error reading from the remote cache: ``` - or - ```none + ```none WARNING: Error writing to the remote cache: ``` + Such warnings will be followed by the error message detailing the connection problem that should help you debug: for example, mistyped endpoint name or incorrectly set credentials. Find and address any such errors. If the error message you see does not give you enough information, try adding `--verbose_failures`. - Such warnings will be followed by the error message detailing the connection - problem that should help you debug: for example, mistyped endpoint name or - incorrectly set credentials. Find and address any such errors. If the error - message you see does not give you enough information, try adding - `--verbose_failures`. - -2. Follow the steps from [Troubleshooting cache hits for remote - execution](/remote/cache-remote#troubleshooting_cache_hits) to - ensure that your cache-writing Bazel invocations are able to get cache hits - on the same machine and across machines. +2. Follow the steps from [Troubleshooting cache hits for remote execution](/remote/cache-remote#troubleshooting_cache_hits) to ensure that your cache-writing Bazel invocations are able to get cache hits on the same machine and across machines. 3. Ensure your cache-reading Bazel invocations can get cache hits. - a. Since cache-reading Bazel invocations will have a different command-line set - up, take additional care to ensure that they are properly set up to - communicate with the remote cache. Ensure the `--remote_cache` flag is set - and there are no warnings in the output. + a. Since cache-reading Bazel invocations will have a different command-line set up, take additional care to ensure that they are properly set up to communicate with the remote cache. Ensure the `--remote_cache` flag is set and there are no warnings in the output. - b. Ensure your cache-reading Bazel invocations build the same targets as the - cache-writing Bazel invocations. + b. Ensure your cache-reading Bazel invocations build the same targets as the cache-writing Bazel invocations. - c. Follow the same steps as to [ensure caching across - machines](/remote/cache-remote#caching-across-machines), - to ensure caching from your cache-writing Bazel invocation to your - cache-reading Bazel invocation. + c. Follow the same steps as to [ensure caching across machines](/remote/cache-remote#caching-across-machines), to ensure caching from your cache-writing Bazel invocation to your cache-reading Bazel invocation. diff --git a/remote/cache-remote.mdx b/remote/cache-remote.mdx index 896f1e58..3b96276a 100644 --- a/remote/cache-remote.mdx +++ b/remote/cache-remote.mdx @@ -2,36 +2,21 @@ title: 'Debugging Remote Cache Hits for Remote Execution' --- +This page describes how to check your cache hit rate and how to investigate cache misses in the context of remote execution. - -This page describes how to check your cache hit rate and how to investigate -cache misses in the context of remote execution. - -This page assumes that you have a build and/or test that successfully -utilizes remote execution, and you want to ensure that you are effectively -utilizing remote cache. +This page assumes that you have a build and/or test that successfully utilizes remote execution, and you want to ensure that you are effectively utilizing remote cache. ## Checking your cache hit rate -In the standard output of your Bazel run, look at the `INFO` line that lists -processes, which roughly correspond to Bazel actions. That line details -where the action was run. Look for the `remote` label, which indicates an action -executed remotely, `linux-sandbox` for actions executed in a local sandbox, -and other values for other execution strategies. An action whose result came -from a remote cache is displayed as `remote cache hit`. +In the standard output of your Bazel run, look at the `INFO` line that lists processes, which roughly correspond to Bazel actions. That line details where the action was run. Look for the `remote` label, which indicates an action executed remotely, `linux-sandbox` for actions executed in a local sandbox, and other values for other execution strategies. An action whose result came from a remote cache is displayed as `remote cache hit`. For example: -```none {:.devsite-disable-click-to-copy} +```none INFO: 11 processes: 6 remote cache hit, 3 internal, 2 remote. ``` -In this example there were 6 remote cache hits, and 2 actions did not have -cache hits and were executed remotely. The 3 internal part can be ignored. -It is typically tiny internal actions, such as creating symbolic links. Local -cache hits are not included in this summary. If you are getting 0 processes -(or a number lower than expected), run `bazel clean` followed by your build/test -command. +In this example there were 6 remote cache hits, and 2 actions did not have cache hits and were executed remotely. The 3 internal part can be ignored. It is typically tiny internal actions, such as creating symbolic links. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. ## Troubleshooting cache hits @@ -39,80 +24,47 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure re-running the same build/test command produces cache hits -1. Run the build(s) and/or test(s) that you expect to populate the cache. The - first time a new build is run on a particular stack, you can expect no remote - cache hits. As part of remote execution, action results are stored in the - cache and a subsequent run should pick them up. +1. Run the build(s) and/or test(s) that you expect to populate the cache. The first time a new build is run on a particular stack, you can expect no remote cache hits. As part of remote execution, action results are stored in the cache and a subsequent run should pick them up. -2. Run `bazel clean`. This command cleans your local cache, which allows - you to investigate remote cache hits without the results being masked by - local cache hits. +2. Run `bazel clean`. This command cleans your local cache, which allows you to investigate remote cache hits without the results being masked by local cache hits. -3. Run the build(s) and test(s) that you are investigating again (on the same - machine). +3. Run the build(s) and test(s) that you are investigating again (on the same machine). -4. Check the `INFO` line for cache hit rate. If you see no processes except - `remote cache hit` and `internal`, then your cache is being correctly populated and - accessed. In that case, skip to the next section. +4. Check the `INFO` line for cache hit rate. If you see no processes except `remote cache hit` and `internal`, then your cache is being correctly populated and accessed. In that case, skip to the next section. -5. A likely source of discrepancy is something non-hermetic in the build causing - the actions to receive different action keys across the two runs. To find - those actions, do the following: +5. A likely source of discrepancy is something non-hermetic in the build causing the actions to receive different action keys across the two runs. To find those actions, do the following: a. Re-run the build(s) or test(s) in question to obtain execution logs: - ```posix-terminal - bazel clean + ```posix-terminal + bazel clean - bazel --optional-flags build //your:target --execution_log_compact_file=/tmp/exec1.log - ``` + bazel <var>--optional-flags</var> build //<var>your:target</var> --execution_log_compact_file=/tmp/exec1.log + ``` - b. [Compare the execution logs](#compare-logs) between the - two runs. Ensure that the actions are identical across the two log files. - Discrepancies provide a clue about the changes that occurred between the - runs. Update your build to eliminate those discrepancies. + b. [Compare the execution logs](#compare-logs) between the two runs. Ensure that the actions are identical across the two log files. Discrepancies provide a clue about the changes that occurred between the runs. Update your build to eliminate those discrepancies. - If you are able to resolve the caching problems and now the repeated run - produces all cache hits, skip to the next section. + If you are able to resolve the caching problems and now the repeated run produces all cache hits, skip to the next section. - If your action IDs are identical but there are no cache hits, then something - in your configuration is preventing caching. Continue with this section to - check for common problems. + If your action IDs are identical but there are no cache hits, then something in your configuration is preventing caching. Continue with this section to check for common problems. -5. Check that all actions in the execution log have `cacheable` set to true. If - `cacheable` does not appear in the execution log for a give action, that - means that the corresponding rule may have a `no-cache` tag in its - definition in the `BUILD` file. Look at the `mnemonic` and `target_label` - fields in the execution log to help determine where the action is coming - from. +6. Check that all actions in the execution log have `cacheable` set to true. If `cacheable` does not appear in the execution log for a give action, that means that the corresponding rule may have a `no-cache` tag in its definition in the `BUILD` file. Look at the `mnemonic` and `target_label` fields in the execution log to help determine where the action is coming from. -6. If the actions are identical and `cacheable` but there are no cache hits, it - is possible that your command line includes `--noremote_accept_cached` which - would disable cache lookups for a build. +7. If the actions are identical and `cacheable` but there are no cache hits, it is possible that your command line includes `--noremote_accept_cached` which would disable cache lookups for a build. - If figuring out the actual command line is difficult, use the canonical - command line from the - [Build Event Protocol](/remote/bep) - as follows: + If figuring out the actual command line is difficult, use the canonical command line from the [Build Event Protocol](/remote/bep) as follows: - a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get - the text version of the log. + a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get the text version of the log. - b. Open the text version of the log and search for the - `structured_command_line` message with `command_line_label: "canonical"`. - It will list all the options after expansion. + b. Open the text version of the log and search for the `structured_command_line` message with `command_line_label: "canonical"`. It will list all the options after expansion. c. Search for `remote_accept_cached` and check whether it's set to `false`. - d. If `remote_accept_cached` is `false`, determine where it is being - set to `false`: either at the command line or in a - [bazelrc](/run/bazelrc#bazelrc-file-locations) file. + d. If `remote_accept_cached` is `false`, determine where it is being set to `false`: either at the command line or in a [bazelrc](/run/bazelrc#bazelrc-file-locations) file. ### Ensure caching across machines -After cache hits are happening as expected on the same machine, run the -same build(s)/test(s) on a different machine. If you suspect that caching is -not happening across machines, do the following: +After cache hits are happening as expected on the same machine, run the same build(s)/test(s) on a different machine. If you suspect that caching is not happening across machines, do the following: 1. Make a small modification to your build to avoid hitting existing caches. @@ -124,8 +76,7 @@ not happening across machines, do the following: bazel ... build ... --execution_log_compact_file=/tmp/exec1.log ``` -3. Run the build on the second machine, ensuring the modification from step 1 - is included: +3. Run the build on the second machine, ensuring the modification from step 1 is included: ```posix-terminal bazel clean @@ -133,47 +84,34 @@ not happening across machines, do the following: bazel ... build ... --execution_log_compact_file=/tmp/exec2.log ``` -4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two - runs. If the logs are not identical, investigate your build configurations - for discrepancies as well as properties from the host environment leaking - into either of the builds. +4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two runs. If the logs are not identical, investigate your build configurations for discrepancies as well as properties from the host environment leaking into either of the builds. ## Comparing the execution logs -The execution log contains records of actions executed during the build. -Each record describes both the inputs (not only files, but also command line -arguments, environment variables, etc) and the outputs of the action. Thus, -examination of the log can reveal why an action was reexecuted. +The execution log contains records of actions executed during the build. Each record describes both the inputs (not only files, but also command line arguments, environment variables, etc) and the outputs of the action. Thus, examination of the log can reveal why an action was reexecuted. -The execution log can be produced in one of three formats: -compact (`--execution_log_compact_file`), -binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). -The compact format is recommended, as it produces much smaller files with very -little runtime overhead. The following instructions work for any format. You -can also convert between them using the `//src/tools/execlog:converter` tool. +The execution log can be produced in one of three formats: compact (`--execution_log_compact_file`), binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). The compact format is recommended, as it produces much smaller files with very little runtime overhead. The following instructions work for any format. You can also convert between them using the `//src/tools/execlog:converter` tool. -To compare logs for two builds that are not sharing cache hits as expected, -do the following: +To compare logs for two builds that are not sharing cache hits as expected, do the following: -1. Get the execution logs from each build and store them as `/tmp/exec1.log` and - `/tmp/exec2.log`. +1. Get the execution logs from each build and store them as `/tmp/exec1.log` and `/tmp/exec2.log`. -2. Download the Bazel source code and build the `//src/tools/execlog:parser` - tool: +2. Download the Bazel source code and build the `//src/tools/execlog:parser` tool: - git clone https://github.com/bazelbuild/bazel.git - cd bazel - bazel build //src/tools/execlog:parser + ``` + git clone https://github.com/bazelbuild/bazel.git + cd bazel + bazel build //src/tools/execlog:parser + ``` -3. Use the `//src/tools/execlog:parser` tool to convert the logs into a - human-readable text format. In this format, the actions in the second log are - sorted to match the order in the first log, making a comparison easier. +3. Use the `//src/tools/execlog:parser` tool to convert the logs into a human-readable text format. In this format, the actions in the second log are sorted to match the order in the first log, making a comparison easier. - bazel-bin/src/tools/execlog/parser \ - --log_path=/tmp/exec1.log \ - --log_path=/tmp/exec2.log \ - --output_path=/tmp/exec1.log.txt \ - --output_path=/tmp/exec2.log.txt + ``` + bazel-bin/src/tools/execlog/parser \ + --log_path=/tmp/exec1.log \ + --log_path=/tmp/exec2.log \ + --output_path=/tmp/exec1.log.txt \ + --output_path=/tmp/exec2.log.txt + ``` -4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and - `/tmp/exec2.log.txt`. +4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and `/tmp/exec2.log.txt`. diff --git a/remote/caching.mdx b/remote/caching.mdx index f85f2c8e..1729ee02 100644 --- a/remote/caching.mdx +++ b/remote/caching.mdx @@ -2,94 +2,65 @@ title: 'Remote Caching' --- +This page covers remote caching, setting up a server to host the cache, and running builds using the remote cache. - -This page covers remote caching, setting up a server to host the cache, and -running builds using the remote cache. - -A remote cache is used by a team of developers and/or a continuous integration -(CI) system to share build outputs. If your build is reproducible, the -outputs from one machine can be safely reused on another machine, which can -make builds significantly faster. +A remote cache is used by a team of developers and/or a continuous integration (CI) system to share build outputs. If your build is reproducible, the outputs from one machine can be safely reused on another machine, which can make builds significantly faster. ## Overview -Bazel breaks a build into discrete steps, which are called actions. Each action -has inputs, output names, a command line, and environment variables. Required -inputs and expected outputs are declared explicitly for each action. +Bazel breaks a build into discrete steps, which are called actions. Each action has inputs, output names, a command line, and environment variables. Required inputs and expected outputs are declared explicitly for each action. -You can set up a server to be a remote cache for build outputs, which are these -action outputs. These outputs consist of a list of output file names and the -hashes of their contents. With a remote cache, you can reuse build outputs -from another user's build rather than building each new output locally. +You can set up a server to be a remote cache for build outputs, which are these action outputs. These outputs consist of a list of output file names and the hashes of their contents. With a remote cache, you can reuse build outputs from another user's build rather than building each new output locally. To use remote caching: -* Set up a server as the cache's backend -* Configure the Bazel build to use the remote cache -* Use Bazel version 0.10.0 or later +- Set up a server as the cache's backend +- Configure the Bazel build to use the remote cache +- Use Bazel version 0.10.0 or later The remote cache stores two types of data: -* The action cache, which is a map of action hashes to action result metadata. -* A content-addressable store (CAS) of output files. +- The action cache, which is a map of action hashes to action result metadata. +- A content-addressable store (CAS) of output files. -Note that the remote cache additionally stores the stdout and stderr for every -action. Inspecting the stdout/stderr of Bazel thus is not a good signal for -[estimating cache hits](/remote/cache-local). +Note that the remote cache additionally stores the stdout and stderr for every action. Inspecting the stdout/stderr of Bazel thus is not a good signal for [estimating cache hits](/remote/cache-local). ### How a build uses remote caching -Once a server is set up as the remote cache, you use the cache in multiple -ways: - -* Read and write to the remote cache -* Read and/or write to the remote cache except for specific targets -* Only read from the remote cache -* Not use the remote cache at all - -When you run a Bazel build that can read and write to the remote cache, -the build follows these steps: - -1. Bazel creates the graph of targets that need to be built, and then creates -a list of required actions. Each of these actions has declared inputs -and output filenames. -2. Bazel checks your local machine for existing build outputs and reuses any -that it finds. -3. Bazel checks the cache for existing build outputs. If the output is found, -Bazel retrieves the output. This is a cache hit. -4. For required actions where the outputs were not found, Bazel executes the -actions locally and creates the required build outputs. +Once a server is set up as the remote cache, you use the cache in multiple ways: + +- Read and write to the remote cache +- Read and/or write to the remote cache except for specific targets +- Only read from the remote cache +- Not use the remote cache at all + +When you run a Bazel build that can read and write to the remote cache, the build follows these steps: + +1. Bazel creates the graph of targets that need to be built, and then creates a list of required actions. Each of these actions has declared inputs and output filenames. +2. Bazel checks your local machine for existing build outputs and reuses any that it finds. +3. Bazel checks the cache for existing build outputs. If the output is found, Bazel retrieves the output. This is a cache hit. +4. For required actions where the outputs were not found, Bazel executes the actions locally and creates the required build outputs. 5. New build outputs are uploaded to the remote cache. ## Setting up a server as the cache's backend -You need to set up a server to act as the cache's backend. A HTTP/1.1 -server can treat Bazel's data as opaque bytes and so many existing servers -can be used as a remote caching backend. Bazel's -[HTTP Caching Protocol](#http-caching) is what supports remote -caching. +You need to set up a server to act as the cache's backend. A HTTP/1.1 server can treat Bazel's data as opaque bytes and so many existing servers can be used as a remote caching backend. Bazel's [HTTP Caching Protocol](#http-caching) is what supports remote caching. -You are responsible for choosing, setting up, and maintaining the backend -server that will store the cached outputs. When choosing a server, consider: +You are responsible for choosing, setting up, and maintaining the backend server that will store the cached outputs. When choosing a server, consider: -* Networking speed. For example, if your team is in the same office, you may -want to run your own local server. -* Security. The remote cache will have your binaries and so needs to be secure. -* Ease of management. For example, Google Cloud Storage is a fully managed service. +- Networking speed. For example, if your team is in the same office, you may want to run your own local server. +- Security. The remote cache will have your binaries and so needs to be secure. +- Ease of management. For example, Google Cloud Storage is a fully managed service. -There are many backends that can be used for a remote cache. Some options -include: +There are many backends that can be used for a remote cache. Some options include: -* [nginx](#nginx) -* [bazel-remote](#bazel-remote) -* [Google Cloud Storage](#cloud-storage) +- [nginx](#nginx) +- [bazel-remote](#bazel-remote) +- [Google Cloud Storage](#cloud-storage) ### nginx -nginx is an open source web server. With its [WebDAV module], it can be -used as a remote cache for Bazel. On Debian and Ubuntu you can install the -`nginx-extras` package. On macOS nginx is available via Homebrew: +nginx is an open source web server. With its \[WebDAV module], it can be used as a remote cache for Bazel. On Debian and Ubuntu you can install the `nginx-extras` package. On macOS nginx is available via Homebrew: ```posix-terminal brew tap denji/nginx @@ -97,12 +68,7 @@ brew tap denji/nginx brew install nginx-full --with-webdav ``` -Below is an example configuration for nginx. Note that you will need to -change `/path/to/cache/dir` to a valid directory where nginx has permission -to write and read. You may need to change `client_max_body_size` option to a -larger value if you have larger output files. The server will require other -configuration such as authentication. - +Below is an example configuration for nginx. Note that you will need to change `/path/to/cache/dir` to a valid directory where nginx has permission to write and read. You may need to change `client_max_body_size` option to a larger value if you have larger output files. The server will require other configuration such as authentication. Example configuration for `server` section in `nginx.conf`: @@ -122,72 +88,44 @@ location /cache/ { ### bazel-remote -bazel-remote is an open source remote build cache that you can use on -your infrastructure. It has been successfully used in production at -several companies since early 2018. Note that the Bazel project does -not provide technical support for bazel-remote. +bazel-remote is an open source remote build cache that you can use on your infrastructure. It has been successfully used in production at several companies since early 2018. Note that the Bazel project does not provide technical support for bazel-remote. -This cache stores contents on disk and also provides garbage collection -to enforce an upper storage limit and clean unused artifacts. The cache is -available as a [docker image] and its code is available on -[GitHub](https://github.com/buchgr/bazel-remote/). -Both the REST and gRPC remote cache APIs are supported. +This cache stores contents on disk and also provides garbage collection to enforce an upper storage limit and clean unused artifacts. The cache is available as a \[docker image] and its code is available on [GitHub](https://github.com/buchgr/bazel-remote/). Both the REST and gRPC remote cache APIs are supported. -Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) -page for instructions on how to use it. +Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) page for instructions on how to use it. ### Google Cloud Storage -[Google Cloud Storage] is a fully managed object store which provides an -HTTP API that is compatible with Bazel's remote caching protocol. It requires -that you have a Google Cloud account with billing enabled. +\[Google Cloud Storage] is a fully managed object store which provides an HTTP API that is compatible with Bazel's remote caching protocol. It requires that you have a Google Cloud account with billing enabled. To use Cloud Storage as the cache: -1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). -Ensure that you select a bucket location that's closest to you, as network bandwidth -is important for the remote cache. +1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). Ensure that you select a bucket location that's closest to you, as network bandwidth is important for the remote cache. -2. Create a service account for Bazel to authenticate to Cloud Storage. See -[Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). +2. Create a service account for Bazel to authenticate to Cloud Storage. See [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). -3. Generate a secret JSON key and then pass it to Bazel for authentication. Store -the key securely, as anyone with the key can read and write arbitrary data -to/from your GCS bucket. +3. Generate a secret JSON key and then pass it to Bazel for authentication. Store the key securely, as anyone with the key can read and write arbitrary data to/from your GCS bucket. 4. Connect to Cloud Storage by adding the following flags to your Bazel command: - * Pass the following URL to Bazel by using the flag: - `--remote_cache=https://storage.googleapis.com/bucket-name` where `bucket-name` is the name of your storage bucket. - * Pass the authentication key using the flag: `--google_credentials=/path/to/your/secret-key.json`, or - `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). -5. You can configure Cloud Storage to automatically delete old files. To do so, see -[Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). + - Pass the following URL to Bazel by using the flag: `--remote_cache=https://storage.googleapis.com<var>/bucket-name</var>` where `bucket-name` is the name of your storage bucket. + - Pass the authentication key using the flag: `--google_credentials=<var>/path/to/your/secret-key</var>.json`, or `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). + +5. You can configure Cloud Storage to automatically delete old files. To do so, see [Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). ### Other servers -You can set up any HTTP/1.1 server that supports PUT and GET as the cache's -backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), -[Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). +You can set up any HTTP/1.1 server that supports PUT and GET as the cache's backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), [Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). ## Authentication -As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. -You can pass a username and password to Bazel via the remote cache URL. The -syntax is `https://username:password@hostname.com:port/path`. Note that -HTTP Basic Authentication transmits username and password in plaintext over the -network and it's thus critical to always use it with HTTPS. +As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. You can pass a username and password to Bazel via the remote cache URL. The syntax is `https://username:password@hostname.com:port/path`. Note that HTTP Basic Authentication transmits username and password in plaintext over the network and it's thus critical to always use it with HTTPS. ## HTTP caching protocol -Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: -Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. -Action result metadata is stored under the path `/ac/` and output files are stored -under the path `/cas/`. +Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. Action result metadata is stored under the path `/ac/` and output files are stored under the path `/cas/`. -For example, consider a remote cache running under `http://localhost:8080/cache`. -A Bazel request to download action result metadata for an action with the SHA256 -hash `01ba4719...` will look as follows: +For example, consider a remote cache running under `http://localhost:8080/cache`. A Bazel request to download action result metadata for an action with the SHA256 hash `01ba4719...` will look as follows: ```http GET /cache/ac/01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b HTTP/1.1 @@ -196,8 +134,7 @@ Accept: */* Connection: Keep-Alive ``` -A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to -the CAS will look as follows: +A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to the CAS will look as follows: ```http PUT /cache/cas/15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225 HTTP/1.1 @@ -211,36 +148,29 @@ Connection: Keep-Alive ## Run Bazel using the remote cache -Once a server is set up as the remote cache, to use the remote cache you -need to add flags to your Bazel command. See list of configurations and -their flags below. +Once a server is set up as the remote cache, to use the remote cache you need to add flags to your Bazel command. See list of configurations and their flags below. -You may also need configure authentication, which is specific to your -chosen server. +You may also need configure authentication, which is specific to your chosen server. -You may want to add these flags in a `.bazelrc` file so that you don't -need to specify them every time you run Bazel. Depending on your project and -team dynamics, you can add flags to a `.bazelrc` file that is: +You may want to add these flags in a `.bazelrc` file so that you don't need to specify them every time you run Bazel. Depending on your project and team dynamics, you can add flags to a `.bazelrc` file that is: -* On your local machine -* In your project's workspace, shared with the team -* On the CI system +- On your local machine +- In your project's workspace, shared with the team +- On the CI system ### Read from and write to the remote cache -Take care in who has the ability to write to the remote cache. You may want -only your CI system to be able to write to the remote cache. +Take care in who has the ability to write to the remote cache. You may want only your CI system to be able to write to the remote cache. Use the following flag to read from and write to the remote cache: ```posix-terminal -build --remote_cache=http://{{ '' }}your.host:port{{ '' }} +build --remote_cache=http://<var>your.host:port</var> ``` Besides `HTTP`, the following protocols are also supported: `HTTPS`, `grpc`, `grpcs`. -Use the following flag in addition to the one above to only read from the -remote cache: +Use the following flag in addition to the one above to only read from the remote cache: ```posix-terminal build --remote_upload_local_results=false @@ -248,8 +178,7 @@ build --remote_upload_local_results=false ### Exclude specific targets from using the remote cache -To exclude specific targets from using the remote cache, tag the target with -`no-remote-cache`. For example: +To exclude specific targets from using the remote cache, tag the target with `no-remote-cache`. For example: ```starlark java_library( @@ -260,121 +189,100 @@ java_library( ### Delete content from the remote cache -Deleting content from the remote cache is part of managing your server. -How you delete content from the remote cache depends on the server you have -set up as the cache. When deleting outputs, either delete the entire cache, -or delete old outputs. +Deleting content from the remote cache is part of managing your server. How you delete content from the remote cache depends on the server you have set up as the cache. When deleting outputs, either delete the entire cache, or delete old outputs. -The cached outputs are stored as a set of names and hashes. When deleting -content, there's no way to distinguish which output belongs to a specific -build. +The cached outputs are stored as a set of names and hashes. When deleting content, there's no way to distinguish which output belongs to a specific build. You may want to delete content from the cache to: -* Create a clean cache after a cache was poisoned -* Reduce the amount of storage used by deleting old outputs +- Create a clean cache after a cache was poisoned +- Reduce the amount of storage used by deleting old outputs ### Unix sockets -The remote HTTP cache supports connecting over unix domain sockets. The behavior -is similar to curl's `--unix-socket` flag. Use the following to configure unix -domain socket: +The remote HTTP cache supports connecting over unix domain sockets. The behavior is similar to curl's `--unix-socket` flag. Use the following to configure unix domain socket: ```posix-terminal - build --remote_cache=http://{{ '' }}your.host:port{{ '' }} - build --remote_proxy=unix:/{{ '' }}path/to/socket{{ '' }} + build --remote_cache=http://<var>your.host:port</var> + build --remote_proxy=unix:/<var>path/to/socket</var> ``` This feature is unsupported on Windows. ## Disk cache -Bazel can use a directory on the file system as a remote cache. This is -useful for sharing build artifacts when switching branches and/or working -on multiple workspaces of the same project, such as multiple checkouts. -Enable the disk cache as follows: +Bazel can use a directory on the file system as a remote cache. This is useful for sharing build artifacts when switching branches and/or working on multiple workspaces of the same project, such as multiple checkouts. Enable the disk cache as follows: ```posix-terminal -build --disk_cache={{ '' }}path/to/build/cache{{ '' }} +build --disk_cache=<var>path/to/build/cache</var> ``` -You can pass a user-specific path to the `--disk_cache` flag using the `~` alias -(Bazel will substitute the current user's home directory). This comes in handy -when enabling the disk cache for all developers of a project via the project's -checked in `.bazelrc` file. +You can pass a user-specific path to the `--disk_cache` flag using the `~` alias (Bazel will substitute the current user's home directory). This comes in handy when enabling the disk cache for all developers of a project via the project's checked in `.bazelrc` file. ### Garbage collection -Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and -`--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache -or for the age of individual cache entries. Bazel will automatically garbage -collect the disk cache while idling between builds; the idle timer can be set -with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). +Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and `--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache or for the age of individual cache entries. Bazel will automatically garbage collect the disk cache while idling between builds; the idle timer can be set with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). -As an alternative to automatic garbage collection, we also provide a [tool]( -https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a -garbage collection on demand. +As an alternative to automatic garbage collection, we also provide a [tool](https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a garbage collection on demand. ## Known issues **Input file modification during a build** -When an input file is modified during a build, Bazel might upload invalid -results to the remote cache. You can enable a change detection with -the `--experimental_guard_against_concurrent_changes` flag. There -are no known issues and it will be enabled by default in a future release. -See [issue #3360] for updates. Generally, avoid modifying source files during a -build. +When an input file is modified during a build, Bazel might upload invalid results to the remote cache. You can enable a change detection with the `--experimental_guard_against_concurrent_changes` flag. There are no known issues and it will be enabled by default in a future release. See \[issue #3360] for updates. Generally, avoid modifying source files during a build. **Environment variables leaking into an action** -An action definition contains environment variables. This can be a problem for -sharing remote cache hits across machines. For example, environments with -different `$PATH` variables won't share cache hits. Only environment variables -explicitly whitelisted via `--action_env` are included in an action -definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` -with a whitelist of environment variables including `$PATH`. If you are getting -fewer cache hits than expected, check that your environment doesn't have an old -`/etc/bazel.bazelrc` file. +An action definition contains environment variables. This can be a problem for sharing remote cache hits across machines. For example, environments with different `$PATH` variables won't share cache hits. Only environment variables explicitly whitelisted via `--action_env` are included in an action definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` with a whitelist of environment variables including `$PATH`. If you are getting fewer cache hits than expected, check that your environment doesn't have an old `/etc/bazel.bazelrc` file. **Bazel does not track tools outside a workspace** -Bazel currently does not track tools outside a workspace. This can be a -problem if, for example, an action uses a compiler from `/usr/bin/`. Then, -two users with different compilers installed will wrongly share cache hits -because the outputs are different but they have the same action hash. See -[issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. +Bazel currently does not track tools outside a workspace. This can be a problem if, for example, an action uses a compiler from `/usr/bin/`. Then, two users with different compilers installed will wrongly share cache hits because the outputs are different but they have the same action hash. See [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. -**Incremental in-memory state is lost when running builds inside docker containers** -Bazel uses server/client architecture even when running in single docker container. -On the server side, Bazel maintains an in-memory state which speeds up builds. -When running builds inside docker containers such as in CI, the in-memory state is lost -and Bazel must rebuild it before using the remote cache. +**Incremental in-memory state is lost when running builds inside docker containers** Bazel uses server/client architecture even when running in single docker container. On the server side, Bazel maintains an in-memory state which speeds up builds. When running builds inside docker containers such as in CI, the in-memory state is lost and Bazel must rebuild it before using the remote cache. ## External links -* **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. - -* **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) -in which he benchmarks remote caching in Bazel. - -* [Adapting Rules for Remote Execution](/remote/rules) -* [Troubleshooting Remote Execution](/remote/sandbox) -* [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) -* [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) -* [bazel-remote](https://github.com/buchgr/bazel-remote/) -* [Google Cloud Storage](https://cloud.google.com/storage) -* [Google Cloud Console](https://cloud.google.com/console) -* [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) -* [Hazelcast](https://hazelcast.com) -* [Apache httpd](http://httpd.apache.org) -* [AWS S3](https://aws.amazon.com/s3) -* [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) -* [gRPC](https://grpc.io/) -* [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) -* [Buildbarn](https://github.com/buildbarn) -* [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) -* [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) -* [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) -* [Application Authentication](https://cloud.google.com/docs/authentication/production) -* [NativeLink](https://github.com/TraceMachina/nativelink) +- **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. + +- **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) in which he benchmarks remote caching in Bazel. + +- [Adapting Rules for Remote Execution](/remote/rules) + +- [Troubleshooting Remote Execution](/remote/sandbox) + +- [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) + +- [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) + +- [bazel-remote](https://github.com/buchgr/bazel-remote/) + +- [Google Cloud Storage](https://cloud.google.com/storage) + +- [Google Cloud Console](https://cloud.google.com/console) + +- [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) + +- [Hazelcast](https://hazelcast.com) + +- [Apache httpd](http://httpd.apache.org) + +- [AWS S3](https://aws.amazon.com/s3) + +- [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) + +- [gRPC](https://grpc.io/) + +- [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) + +- [Buildbarn](https://github.com/buildbarn) + +- [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + +- [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + +- [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) + +- [Application Authentication](https://cloud.google.com/docs/authentication/production) + +- [NativeLink](https://github.com/TraceMachina/nativelink) diff --git a/remote/ci.mdx b/remote/ci.mdx index 0a4e4488..9e5d786a 100644 --- a/remote/ci.mdx +++ b/remote/ci.mdx @@ -2,37 +2,24 @@ title: 'Configuring Bazel CI to Test Rules for Remote Execution' --- - - -This page is for owners and maintainers of Bazel rule repositories. It -describes how to configure the Bazel Continuous Integration (CI) system for -your repository to test your rules for compatibility against a remote execution -scenario. The instructions on this page apply to projects stored in -GitHub repositories. +This page is for owners and maintainers of Bazel rule repositories. It describes how to configure the Bazel Continuous Integration (CI) system for your repository to test your rules for compatibility against a remote execution scenario. The instructions on this page apply to projects stored in GitHub repositories. ## Prerequisites Before completing the steps on this page, ensure the following: -* Your GitHub repository is part of the - [Bazel GitHub organization](https://github.com/bazelbuild). -* You have configured Buildkite for your repository as described in - [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). +- Your GitHub repository is part of the [Bazel GitHub organization](https://github.com/bazelbuild). +- You have configured Buildkite for your repository as described in [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). ## Setting up the Bazel CI for testing -1. In your `.bazelci/presubmit.yml` file, do the following: +1. In your `.bazelci/presubmit.yml` file, do the following: - a. Add a config named `rbe_ubuntu1604`. + a. Add a config named `rbe_ubuntu1604`. - b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. + b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. -2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) - GitHub repository to your `WORKSPACE` file, pinned to the - [latest release](https://releases.bazel.build/bazel-toolchains.html). Also - add an `rbe_autoconfig` target with name `buildkite_config`. This example - creates toolchain configuration for remote execution with BuildKite CI - for `rbe_ubuntu1604`. +2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) GitHub repository to your `WORKSPACE` file, pinned to the [latest release](https://releases.bazel.build/bazel-toolchains.html). Also add an `rbe_autoconfig` target with name `buildkite_config`. This example creates toolchain configuration for remote execution with BuildKite CI for `rbe_ubuntu1604`. ```posix-terminal load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") @@ -40,43 +27,25 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") rbe_autoconfig(name = "buildkite_config") ``` -3. Send a pull request with your changes to the `presubmit.yml` file. (See - [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) +3. Send a pull request with your changes to the `presubmit.yml` file. (See [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) -4. To view build results, click **Details** for the RBE (Ubuntu - 16.04) pull request check in GitHub, as shown in the figure below. This link - becomes available after the pull request has been merged and the CI tests - have run. (See - [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) +4. To view build results, click **Details** for the RBE (Ubuntu 16.04) pull request check in GitHub, as shown in the figure below. This link becomes available after the pull request has been merged and the CI tests have run. (See [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) - ![Example results](/docs/images/rbe-ci-1.png "Example results") + ![Example results](/docs/images/rbe-ci-1.png "Example results") -5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test - required to pass before merging in your branch protection rule. The setting - is located in GitHub in **Settings > Branches > Branch protection rules**, - as shown in the following figure. +5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test required to pass before merging in your branch protection rule. The setting is located in GitHub in **Settings \> Branches \> Branch protection rules**, as shown in the following figure. - ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") + ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") ## Troubleshooting failed builds and tests If your build or tests fail, it's likely due to the following: -* **Required build or test tools are not installed in the default container.** - Builds using the `rbe_ubuntu1604` config run by default inside an - [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) - container, which includes tools common to many Bazel builds. However, if - your rules require tools not present in the default container, you must - create a custom container based on the - [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) - container and include those tools as described later. +- **Required build or test tools are not installed in the default container.** Builds using the `rbe_ubuntu1604` config run by default inside an [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container, which includes tools common to many Bazel builds. However, if your rules require tools not present in the default container, you must create a custom container based on the [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container and include those tools as described later. -* **Build or test targets are using rules that are incompatible with remote - execution.** See - [Adapting Bazel Rules for Remote Execution](/remote/rules) for - details about compatibility with remote execution. +- **Build or test targets are using rules that are incompatible with remote execution.** See [Adapting Bazel Rules for Remote Execution](/remote/rules) for details about compatibility with remote execution. -## Using a custom container in the rbe_ubuntu1604 CI config +## Using a custom container in the rbe\_ubuntu1604 CI config The `rbe-ubuntu16-04` container is publicly available at the following URL: @@ -84,121 +53,99 @@ The `rbe-ubuntu16-04` container is publicly available at the following URL: http://gcr.io/cloud-marketplace/google/rbe-ubuntu16-04 ``` -You can pull it directly from Container Registry or build it from source. The -next sections describe both options. +You can pull it directly from Container Registry or build it from source. The next sections describe both options. -Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. -If you are building the container from source, you must also install the latest -version of Bazel. +Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. If you are building the container from source, you must also install the latest version of Bazel. ### Pulling the rbe-ubuntu16-04 from Container Registry -To pull the `rbe-ubuntu16-04` container from Container Registry, run the -following command: +To pull the `rbe-ubuntu16-04` container from Container Registry, run the following command: ```posix-terminal -gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} +gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> ``` -Replace sha256-checksum with the SHA256 checksum value for -[the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). +Replace \{\{ '`' }}sha256-checksum{{ '`' \}\} with the SHA256 checksum value for [the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). ### Building the rbe-ubuntu16-04 container from source To build the `rbe-ubuntu16-04` container from source, do the following: -1. Clone the `bazel-toolchains` repository: +1. Clone the `bazel-toolchains` repository: - ```posix-terminal - git clone https://github.com/bazelbuild/bazel-toolchains - ``` + ```posix-terminal + git clone https://github.com/bazelbuild/bazel-toolchains + ``` -2. Set up toolchain container targets and build the container as explained in - [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). +2. Set up toolchain container targets and build the container as explained in [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). -3. Pull the freshly built container: +3. Pull the freshly built container: - ```posix-terminal -gcloud docker -- pull gcr.io/project-id/custom-container-namesha256-checksum - ``` + ```posix-terminal + ``` + +gcloud docker -- pull gcr.io/\{\{ '`' }}project-id{{ '`' \}\}/\{\{ '`' }}custom-container-name{{ '`' \}\}\{\{ '`' }}sha256-checksum{{ '`' \}\} \`\`\` ### Running the custom container To run the custom container, do one of the following: -* If you pulled the container from Container Registry, run the following - command: +- If you pulled the container from Container Registry, run the following command: - ```posix-terminal - docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:sha256-checksum/bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var>/bin/bash + ``` - Replace `sha256-checksum` with the SHA256 checksum value for the - [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). + Replace `sha256-checksum` with the SHA256 checksum value for the [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). -* If you built the container from source, run the following command: +- If you built the container from source, run the following command: - ```posix-terminal - docker run -it gcr.io/project-id/custom-container-name@sha256:sha256sum /bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/<var>project-id</var>/<var>custom-container-name</var>@sha256:<var>sha256sum</var> /bin/bash + ``` ### Adding resources to the custom container -Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or -[`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or -alternate versions of the original resources to the `rbe-ubuntu16-04` container. -If you are new to Docker, read the following: +Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or [`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or alternate versions of the original resources to the `rbe-ubuntu16-04` container. If you are new to Docker, read the following: -* [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) -* [Docker Samples](https://docs.docker.com/samples/) +- [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) +- [Docker Samples](https://docs.docker.com/samples/) -For example, the following `Dockerfile` snippet installs `my_tool_package`: +For example, the following `Dockerfile` snippet installs `<var>my_tool_package</var>`: ``` -FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} -RUN apt-get update && yes | apt-get install -y {{ '' }}my_tool_package{{ '' }} +FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> +RUN apt-get update && yes | apt-get install -y <var>my_tool_package</var> ``` ### Pushing the custom container to Container Registry -Once you have customized the container, build the container image and push it to -Container Registry as follows: +Once you have customized the container, build the container image and push it to Container Registry as follows: 1. Build the container image: - ```posix-terminal - docker build -t custom-container-name. - - docker tag custom-container-name gcr.io/project-id/custom-container-name - ``` + ```posix-terminal + docker build -t <var>custom-container-name</var>. -2. Push the container image to Container Registry: + docker tag <var>custom-container-name</var> gcr.io/<var>project-id</var>/<var>custom-container-name</var> + ``` - ```posix-terminal - gcloud docker -- push gcr.io/project-id/custom-container-name - ``` +2. Push the container image to Container Registry: -3. Navigate to the following URL to verify the container has been pushed: + ```posix-terminal + gcloud docker -- push gcr.io/<var>project-id</var>/<var>custom-container-name</var> + ``` - https://console.cloud.google.com/gcr/images/project-id/GLOBAL/custom-container-name +3. Navigate to the following URL to verify the container has been pushed: -4. Take note of the SHA256 checksum of your custom container. You will need to - provide it in your build platform definition later. + [https://console.cloud.google.com/gcr/images/\{\{](https://console.cloud.google.com/gcr/images/%7B%7B) '`' }}project-id{{ '`' \}\}/GLOBAL/\{\{ '`' }}custom-container-name{{ '`' \}\} -5. Configure the container for public access as described in publicly - accessible as explained in - [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). +4. Take note of the SHA256 checksum of your custom container. You will need to provide it in your build platform definition later. - For more information, see - [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). +5. Configure the container for public access as described in publicly accessible as explained in [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). + For more information, see [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). ### Specifying the build platform definition -You must include a [Bazel platform](/extending/platforms) configuration in your -custom toolchain configuration, which allows Bazel to select a toolchain -appropriate to the desired hardware/software platform. To generate -automatically a valid platform, you can add to your `WORKSPACE` an -`rbe_autoconfig` target with name `buildkite_config` which includes additional -attrs to select your custom container. For details on this setup, read -the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). +You must include a [Bazel platform](/extending/platforms) configuration in your custom toolchain configuration, which allows Bazel to select a toolchain appropriate to the desired hardware/software platform. To generate automatically a valid platform, you can add to your `WORKSPACE` an `rbe_autoconfig` target with name `buildkite_config` which includes additional attrs to select your custom container. For details on this setup, read the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). diff --git a/remote/creating.mdx b/remote/creating.mdx index 0e46a07a..4b2cf226 100644 --- a/remote/creating.mdx +++ b/remote/creating.mdx @@ -2,49 +2,30 @@ title: 'Creating Persistent Workers' --- +[Persistent workers](/remote/persistent) can make your build faster. If you have repeated actions in your build that have a high startup cost or would benefit from cross-action caching, you may want to implement your own persistent worker to perform these actions. - -[Persistent workers](/remote/persistent) can make your build faster. If -you have repeated actions in your build that have a high startup cost or would -benefit from cross-action caching, you may want to implement your own persistent -worker to perform these actions. - -The Bazel server communicates with the worker using `stdin`/`stdout`. It -supports the use of protocol buffers or JSON strings. +The Bazel server communicates with the worker using `stdin`/`stdout`. It supports the use of protocol buffers or JSON strings. The worker implementation has two parts: -* The [worker](#making-worker). -* The [rule that uses the worker](#rule-uses-worker). +- The [worker](#making-worker). +- The [rule that uses the worker](#rule-uses-worker). ## Making the worker A persistent worker upholds a few requirements: -* It reads - [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) - from its `stdin`. -* It writes - [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) - (and only `WorkResponse`s) to its `stdout`. -* It accepts the `--persistent_worker` flag. The wrapper must recognize the - `--persistent_worker` command-line flag and only make itself persistent if - that flag is passed, otherwise it must do a one-shot compilation and exit. +- It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) from its `stdin`. +- It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) (and only `WorkResponse`s) to its `stdout`. +- It accepts the `--persistent_worker` flag. The wrapper must recognize the `--persistent_worker` command-line flag and only make itself persistent if that flag is passed, otherwise it must do a one-shot compilation and exit. -If your program upholds these requirements, it can be used as a persistent -worker! +If your program upholds these requirements, it can be used as a persistent worker! ### Work requests -A `WorkRequest` contains a list of arguments to the worker, a list of -path-digest pairs representing the inputs the worker can access (this isn’t -enforced, but you can use this info for caching), and a request id, which is 0 -for singleplex workers. +A `WorkRequest` contains a list of arguments to the worker, a list of path-digest pairs representing the inputs the worker can access (this isn’t enforced, but you can use this info for caching), and a request id, which is 0 for singleplex workers. -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), -the JSON protocol uses "camel case" (`requestId`). This document uses camel case -in the JSON examples, but snake case when talking about the field regardless of -protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). This document uses camel case in the JSON examples, but snake case when talking about the field regardless of protocol. ```json { @@ -57,24 +38,13 @@ protocol. } ``` -The optional `verbosity` field can be used to request extra debugging output -from the worker. It is entirely up to the worker what and how to output. Higher -values indicate more verbose output. Passing the `--worker_verbose` flag to -Bazel sets the `verbosity` field to 10, but smaller or larger values can be used -manually for different amounts of output. +The optional `verbosity` field can be used to request extra debugging output from the worker. It is entirely up to the worker what and how to output. Higher values indicate more verbose output. Passing the `--worker_verbose` flag to Bazel sets the `verbosity` field to 10, but smaller or larger values can be used manually for different amounts of output. -The optional `sandbox_dir` field is used only by workers that support -[multiplex sandboxing](/remote/multiplex). +The optional `sandbox_dir` field is used only by workers that support [multiplex sandboxing](/remote/multiplex). ### Work responses -A `WorkResponse` contains a request id, a zero or nonzero exit code, and an -output message describing any errors encountered in processing or executing -the request. A worker should capture the `stdout` and `stderr` of any tool it -calls and report them through the `WorkResponse`. Writing it to the `stdout` of -the worker process is unsafe, as it will interfere with the worker protocol. -Writing it to the `stderr` of the worker process is safe, but the result is -collected in a per-worker log file instead of ascribed to individual actions. +A `WorkResponse` contains a request id, a zero or nonzero exit code, and an output message describing any errors encountered in processing or executing the request. A worker should capture the `stdout` and `stderr` of any tool it calls and report them through the `WorkResponse`. Writing it to the `stdout` of the worker process is unsafe, as it will interfere with the worker protocol. Writing it to the `stderr` of the worker process is safe, but the result is collected in a per-worker log file instead of ascribed to individual actions. ```json { @@ -85,10 +55,7 @@ collected in a per-worker log file instead of ascribed to individual actions. } ``` -As per the norm for protobufs, all fields are optional. However, Bazel requires -the `WorkRequest` and the corresponding `WorkResponse`, to have the same request -id, so the request id must be specified if it is nonzero. This is a valid -`WorkResponse`. +As per the norm for protobufs, all fields are optional. However, Bazel requires the `WorkRequest` and the corresponding `WorkResponse`, to have the same request id, so the request id must be specified if it is nonzero. This is a valid `WorkResponse`. ```json { @@ -96,69 +63,35 @@ id, so the request id must be specified if it is nonzero. This is a valid } ``` -A `request_id` of 0 indicates a "singleplex" request, used when this request -cannot be processed in parallel with other requests. The server guarantees that -a given worker receives requests with either only `request_id` 0 or only -`request_id` greater than zero. Singleplex requests are sent in serial, for -example if the server doesn't send another request until it has received a -response (except for cancel requests, see below). +A `request_id` of 0 indicates a "singleplex" request, used when this request cannot be processed in parallel with other requests. The server guarantees that a given worker receives requests with either only `request_id` 0 or only `request_id` greater than zero. Singleplex requests are sent in serial, for example if the server doesn't send another request until it has received a response (except for cancel requests, see below). **Notes** -* Each protocol buffer is preceded by its length in `varint` format (see - [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). -* JSON requests and responses are not preceded by a size indicator. -* JSON requests uphold the same structure as the protobuf, but use standard - JSON and use camel case for all field names. -* In order to maintain the same backward and forward compatibility properties - as protobuf, JSON workers must tolerate unknown fields in these messages, - and use the protobuf defaults for missing values. -* Bazel stores requests as protobufs and converts them to JSON using - [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) +- Each protocol buffer is preceded by its length in `varint` format (see [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). +- JSON requests and responses are not preceded by a size indicator. +- JSON requests uphold the same structure as the protobuf, but use standard JSON and use camel case for all field names. +- In order to maintain the same backward and forward compatibility properties as protobuf, JSON workers must tolerate unknown fields in these messages, and use the protobuf defaults for missing values. +- Bazel stores requests as protobufs and converts them to JSON using [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) ### Cancellation -Workers can optionally allow work requests to be cancelled before they finish. -This is particularly useful in connection with dynamic execution, where local -execution can regularly be interrupted by a faster remote execution. To allow -cancellation, add `supports-worker-cancellation: 1` to the -`execution-requirements` field (see below) and set the -`--experimental_worker_cancellation` flag. - -A **cancel request** is a `WorkRequest` with the `cancel` field set (and -similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` -field set). The only other field that must be in a cancel request or cancel -response is `request_id`, indicating which request to cancel. The `request_id` -field will be 0 for singleplex workers or the non-0 `request_id` of a previously -sent `WorkRequest` for multiplex workers. The server may send cancel requests -for requests that the worker has already responded to, in which case the cancel -request must be ignored. - -Each non-cancel `WorkRequest` message must be answered exactly once, whether or -not it was cancelled. Once the server has sent a cancel request, the worker may -respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` -field set to true. Sending a regular `WorkResponse` is also accepted, but the -`output` and `exit_code` fields will be ignored. - -Once a response has been sent for a `WorkRequest`, the worker must not touch the -files in its working directory. The server is free to clean up the files, -including temporary files. +Workers can optionally allow work requests to be cancelled before they finish. This is particularly useful in connection with dynamic execution, where local execution can regularly be interrupted by a faster remote execution. To allow cancellation, add `supports-worker-cancellation: 1` to the `execution-requirements` field (see below) and set the `--experimental_worker_cancellation` flag. + +A **cancel request** is a `WorkRequest` with the `cancel` field set (and similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` field set). The only other field that must be in a cancel request or cancel response is `request_id`, indicating which request to cancel. The `request_id` field will be 0 for singleplex workers or the non-0 `request_id` of a previously sent `WorkRequest` for multiplex workers. The server may send cancel requests for requests that the worker has already responded to, in which case the cancel request must be ignored. + +Each non-cancel `WorkRequest` message must be answered exactly once, whether or not it was cancelled. Once the server has sent a cancel request, the worker may respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` field set to true. Sending a regular `WorkResponse` is also accepted, but the `output` and `exit_code` fields will be ignored. + +Once a response has been sent for a `WorkRequest`, the worker must not touch the files in its working directory. The server is free to clean up the files, including temporary files. ## Making the rule that uses the worker -You'll also need to create a rule that generates actions to be performed by the -worker. Making a Starlark rule that uses a worker is just like -[creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). +You'll also need to create a rule that generates actions to be performed by the worker. Making a Starlark rule that uses a worker is just like [creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). -In addition, the rule needs to contain a reference to the worker itself, and -there are some requirements for the actions it produces. +In addition, the rule needs to contain a reference to the worker itself, and there are some requirements for the actions it produces. ### Referring to the worker -The rule that uses the worker needs to contain a field that refers to the worker -itself, so you'll need to create an instance of a `\*\_binary` rule to define -your worker. If your worker is called `MyWorker.Java`, this might be the -associated rule: +The rule that uses the worker needs to contain a field that refers to the worker itself, so you'll need to create an instance of a `\*\_binary` rule to define your worker. If your worker is called `MyWorker.Java`, this might be the associated rule: ```python java_binary( @@ -167,12 +100,9 @@ java_binary( ) ``` -This creates the "worker" label, which refers to the worker binary. You'll then -define a rule that *uses* the worker. This rule should define an attribute that -refers to the worker binary. +This creates the "worker" label, which refers to the worker binary. You'll then define a rule that *uses* the worker. This rule should define an attribute that refers to the worker binary. -If the worker binary you built is in a package named "work", which is at the top -level of the build, this might be the attribute definition: +If the worker binary you built is in a package named "work", which is at the top level of the build, this might be the attribute definition: ```python "worker": attr.label( @@ -182,46 +112,25 @@ level of the build, this might be the attribute definition: ) ``` -`cfg = "exec"` indicates that the worker should be built to run on your -execution platform rather than on the target platform (i.e., the worker is used -as tool during the build). +`cfg = "exec"` indicates that the worker should be built to run on your execution platform rather than on the target platform (i.e., the worker is used as tool during the build). ### Work action requirements -The rule that uses the worker creates actions for the worker to perform. These -actions have a couple of requirements. +The rule that uses the worker creates actions for the worker to perform. These actions have a couple of requirements. -* The *"arguments"* field. This takes a list of strings, all but the last of - which are arguments passed to the worker upon startup. The last element in - the "arguments" list is a `flag-file` (@-preceded) argument. Workers read - the arguments from the specified flagfile on a per-WorkRequest basis. Your - rule can write non-startup arguments for the worker to this flagfile. +- The *"arguments"* field. This takes a list of strings, all but the last of which are arguments passed to the worker upon startup. The last element in the "arguments" list is a `flag-file` (@-preceded) argument. Workers read the arguments from the specified flagfile on a per-WorkRequest basis. Your rule can write non-startup arguments for the worker to this flagfile. -* The *"execution-requirements"* field, which takes a dictionary containing - `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. +- The *"execution-requirements"* field, which takes a dictionary containing `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. - The "arguments" and "execution-requirements" fields are required for all - actions sent to workers. Additionally, actions that should be executed by - JSON workers need to include `"requires-worker-protocol" : "json"` in the - execution requirements field. `"requires-worker-protocol" : "proto"` is also - a valid execution requirement, though it’s not required for proto workers, - since they are the default. + The "arguments" and "execution-requirements" fields are required for all actions sent to workers. Additionally, actions that should be executed by JSON workers need to include `"requires-worker-protocol" : "json"` in the execution requirements field. `"requires-worker-protocol" : "proto"` is also a valid execution requirement, though it’s not required for proto workers, since they are the default. - You can also set a `worker-key-mnemonic` in the execution requirements. This - may be useful if you're reusing the executable for multiple action types and - want to distinguish actions by this worker. + You can also set a `worker-key-mnemonic` in the execution requirements. This may be useful if you're reusing the executable for multiple action types and want to distinguish actions by this worker. -* Temporary files generated in the course of the action should be saved to the - worker's directory. This enables sandboxing. +- Temporary files generated in the course of the action should be saved to the worker's directory. This enables sandboxing. -Note: To pass an argument starting with a literal `@`, start the argument with -`@@` instead. If an argument is also an external repository label, it will not -be considered a flagfile argument. +Note: To pass an argument starting with a literal `@`, start the argument with `@@` instead. If an argument is also an external repository label, it will not be considered a flagfile argument. -Assuming a rule definition with "worker" attribute described above, in addition -to a "srcs" attribute representing the inputs, an "output" attribute -representing the outputs, and an "args" attribute representing the worker -startup args, the call to `ctx.actions.run` might be: +Assuming a rule definition with "worker" attribute described above, in addition to a "srcs" attribute representing the inputs, an "output" attribute representing the outputs, and an "args" attribute representing the worker startup args, the call to `ctx.actions.run` might be: ```python ctx.actions.run( @@ -236,26 +145,14 @@ ctx.actions.run( ) ``` -For another example, see -[Implementing persistent workers](/remote/persistent#implementation). +For another example, see [Implementing persistent workers](/remote/persistent#implementation). ## Examples -The Bazel code base uses -[Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), -in addition to an -[example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) -that is used in our integration tests. +The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), in addition to an [example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) that is used in our integration tests. -You can use their -[scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) -to make any Java-based tool into a worker by passing in the correct callback. +You can use their [scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) to make any Java-based tool into a worker by passing in the correct callback. -For an example of a rule that uses a worker, take a look at Bazel's -[worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). +For an example of a rule that uses a worker, take a look at Bazel's [worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). -External contributors have implemented workers in a variety of languages; take a -look at -[Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). -You can -[find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)! +External contributors have implemented workers in a variety of languages; take a look at [Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). You can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest\&type=Code)! diff --git a/remote/multiplex.mdx b/remote/multiplex.mdx index b4b0a0d4..d9075a8a 100644 --- a/remote/multiplex.mdx +++ b/remote/multiplex.mdx @@ -2,112 +2,38 @@ title: 'Multiplex Workers (Experimental Feature)' --- - - -This page describes multiplex workers, how to write multiplex-compatible -rules, and workarounds for certain limitations. +This page describes multiplex workers, how to write multiplex-compatible rules, and workarounds for certain limitations. Caution: Experimental features are subject to change at any time. -_Multiplex workers_ allow Bazel to handle multiple requests with a single worker -process. For multi-threaded workers, Bazel can use fewer resources to -achieve the same, or better performance. For example, instead of having one -worker process per worker, Bazel can have four multiplexed workers talking to -the same worker process, which can then handle requests in parallel. For -languages like Java and Scala, this saves JVM warm-up time and JIT compilation -time, and in general it allows using one shared cache between all workers of -the same type. +*Multiplex workers* allow Bazel to handle multiple requests with a single worker process. For multi-threaded workers, Bazel can use fewer resources to achieve the same, or better performance. For example, instead of having one worker process per worker, Bazel can have four multiplexed workers talking to the same worker process, which can then handle requests in parallel. For languages like Java and Scala, this saves JVM warm-up time and JIT compilation time, and in general it allows using one shared cache between all workers of the same type. ## Overview -There are two layers between the Bazel server and the worker process. For certain -mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from -the worker pool. The `WorkerProxy` forwards requests to the worker process -sequentially along with a `request_id`, the worker process processes the request -and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` -receives a response, it parses the `request_id` and then forwards the responses -back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all -communication is done over standard in/out, but the tool cannot just use -`stderr` for user-visible output ([see below](#output)). - -Each worker has a key. Bazel uses the key's hash code (composed of environment -variables, the execution root, and the mnemonic) to determine which -`WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same -`WorkerMultiplexer` if they have the same hash code. Therefore, assuming -environment variables and the execution root are the same in a single Bazel -invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one -worker process. The total number of workers, including regular workers and -`WorkerProxy`s, is still limited by `--worker_max_instances`. +There are two layers between the Bazel server and the worker process. For certain mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from the worker pool. The `WorkerProxy` forwards requests to the worker process sequentially along with a `request_id`, the worker process processes the request and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` receives a response, it parses the `request_id` and then forwards the responses back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all communication is done over standard in/out, but the tool cannot just use `stderr` for user-visible output ([see below](#output)). + +Each worker has a key. Bazel uses the key's hash code (composed of environment variables, the execution root, and the mnemonic) to determine which `WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same `WorkerMultiplexer` if they have the same hash code. Therefore, assuming environment variables and the execution root are the same in a single Bazel invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one worker process. The total number of workers, including regular workers and `WorkerProxy`s, is still limited by `--worker_max_instances`. ## Writing multiplex-compatible rules -The rule's worker process should be multi-threaded to take advantage of -multiplex workers. Protobuf allows a ruleset to parse a single request even -though there might be multiple requests piling up in the stream. Whenever the -worker process parses a request from the stream, it should handle the request in -a new thread. Because different thread could complete and write to the stream at -the same time, the worker process needs to make sure the responses are written -atomically (messages don't overlap). Responses must contain the -`request_id` of the request they're handling. +The rule's worker process should be multi-threaded to take advantage of multiplex workers. Protobuf allows a ruleset to parse a single request even though there might be multiple requests piling up in the stream. Whenever the worker process parses a request from the stream, it should handle the request in a new thread. Because different thread could complete and write to the stream at the same time, the worker process needs to make sure the responses are written atomically (messages don't overlap). Responses must contain the `request_id` of the request they're handling. ### Handling multiplex output -Multiplex workers need to be more careful about handling their output than -singleplex workers. Anything sent to `stderr` will go into a single log file -shared among all `WorkerProxy`s of the same type, -randomly interleaved between concurrent requests. While redirecting `stdout` -into `stderr` is a good idea, do not collect that output into the `output` -field of `WorkResponse`, as that could show the user mangled pieces of output. -If your tool only sends user-oriented output to `stdout` or `stderr`, you will -need to change that behaviour before you can enable multiplex workers. +Multiplex workers need to be more careful about handling their output than singleplex workers. Anything sent to `stderr` will go into a single log file shared among all `WorkerProxy`s of the same type, randomly interleaved between concurrent requests. While redirecting `stdout` into `stderr` is a good idea, do not collect that output into the `output` field of `WorkResponse`, as that could show the user mangled pieces of output. If your tool only sends user-oriented output to `stdout` or `stderr`, you will need to change that behaviour before you can enable multiplex workers. ## Enabling multiplex workers -Multiplex workers are not enabled by default. A ruleset can turn on multiplex -workers by using the `supports-multiplex-workers` tag in the -`execution_requirements` of an action (just like the `supports-workers` tag -enables regular workers). As is the case when using regular workers, a worker -strategy needs to be specified, either at the ruleset level (for example, -`--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for -example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are -necessary, and `supports-multiplex-workers` takes precedence over -`supports-workers`, if both are set. You can turn off multiplex workers -globally by passing `--noworker_multiplex`. - -A ruleset is encouraged to use multiplex workers if possible, to reduce memory -pressure and improve performance. However, multiplex workers are not currently -compatible with [dynamic execution](/remote/dynamic) unless they -implement multiplex sandboxing. Attempting to run non-sandboxed multiplex -workers with dynamic execution will silently use sandboxed -singleplex workers instead. +Multiplex workers are not enabled by default. A ruleset can turn on multiplex workers by using the `supports-multiplex-workers` tag in the `execution_requirements` of an action (just like the `supports-workers` tag enables regular workers). As is the case when using regular workers, a worker strategy needs to be specified, either at the ruleset level (for example, `--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are necessary, and `supports-multiplex-workers` takes precedence over `supports-workers`, if both are set. You can turn off multiplex workers globally by passing `--noworker_multiplex`. + +A ruleset is encouraged to use multiplex workers if possible, to reduce memory pressure and improve performance. However, multiplex workers are not currently compatible with [dynamic execution](/remote/dynamic) unless they implement multiplex sandboxing. Attempting to run non-sandboxed multiplex workers with dynamic execution will silently use sandboxed singleplex workers instead. ## Multiplex sandboxing -Multiplex workers can be sandboxed by adding explicit support for it in the -worker implementations. While singleplex worker sandboxing can be done by -running each worker process in its own sandbox, multiplex workers share the -process working directory between multiple parallel requests. To allow -sandboxing of multiplex workers, the worker must support reading from and -writing to a subdirectory specified in each request, instead of directly in -its working directory. - -To support multiplex sandboxing, the worker must use the `sandbox_dir` field -from the `WorkRequest` and use that as a prefix for all file reads and writes. -While the `arguments` and `inputs` fields remain unchanged from an unsandboxed -request, the actual inputs are relative to the `sandbox_dir`. The worker must -translate file paths found in `arguments` and `inputs` to read from this -modified path, and must also write all outputs relative to the `sandbox_dir`. -This includes paths such as '.', as well as paths found in files specified -in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). - -Once a worker supports multiplex sandboxing, the ruleset can declare this -support by adding `supports-multiplex-sandboxing` to the -`execution_requirements` of an action. Bazel will then use multiplex sandboxing -if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if -the worker is used with dynamic execution. - -The worker files of a sandboxed multiplex worker are still relative to the -working directory of the worker process. Thus, if a file is -used both for running the worker and as an input, it must be specified both as -an input in the flagfile argument as well as in `tools`, `executable`, or -`runfiles`. +Multiplex workers can be sandboxed by adding explicit support for it in the worker implementations. While singleplex worker sandboxing can be done by running each worker process in its own sandbox, multiplex workers share the process working directory between multiple parallel requests. To allow sandboxing of multiplex workers, the worker must support reading from and writing to a subdirectory specified in each request, instead of directly in its working directory. + +To support multiplex sandboxing, the worker must use the `sandbox_dir` field from the `WorkRequest` and use that as a prefix for all file reads and writes. While the `arguments` and `inputs` fields remain unchanged from an unsandboxed request, the actual inputs are relative to the `sandbox_dir`. The worker must translate file paths found in `arguments` and `inputs` to read from this modified path, and must also write all outputs relative to the `sandbox_dir`. This includes paths such as '.', as well as paths found in files specified in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). + +Once a worker supports multiplex sandboxing, the ruleset can declare this support by adding `supports-multiplex-sandboxing` to the `execution_requirements` of an action. Bazel will then use multiplex sandboxing if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if the worker is used with dynamic execution. + +The worker files of a sandboxed multiplex worker are still relative to the working directory of the worker process. Thus, if a file is used both for running the worker and as an input, it must be specified both as an input in the flagfile argument as well as in `tools`, `executable`, or `runfiles`. diff --git a/remote/output-directories.mdx b/remote/output-directories.mdx index 9c1ba329..190892d8 100644 --- a/remote/output-directories.mdx +++ b/remote/output-directories.mdx @@ -2,141 +2,108 @@ title: 'Output Directory Layout' --- - - This page covers requirements and layout for output directories. ## Requirements Requirements for an output directory layout: -* Doesn't collide if multiple users are building on the same box. -* Supports building in multiple workspaces at the same time. -* Supports building for multiple target configurations in the same workspace. -* Doesn't collide with any other tools. -* Is easy to access. -* Is easy to clean, even selectively. -* Is unambiguous, even if the user relies on symbolic links when changing into - their client directory. -* All the build state per user should be underneath one directory ("I'd like to - clean all the .o files from all my clients.") +- Doesn't collide if multiple users are building on the same box. +- Supports building in multiple workspaces at the same time. +- Supports building for multiple target configurations in the same workspace. +- Doesn't collide with any other tools. +- Is easy to access. +- Is easy to clean, even selectively. +- Is unambiguous, even if the user relies on symbolic links when changing into their client directory. +- All the build state per user should be underneath one directory ("I'd like to clean all the .o files from all my clients.") ## Current layout The solution that's currently implemented: -* Bazel must be invoked from a directory containing a repo boundary file, or a - subdirectory thereof. In other words, Bazel must be invoked from inside a - [repository](../external/overview#repository). Otherwise, an error is - reported. -* The _outputRoot_ directory defaults to `~/.cache/bazel` on Linux, - `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if - set, else `%USERPROFILE%` if set, else the result of calling - `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the - environment variable `$XDG_CACHE_HOME` is set on either Linux or - macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. - If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel - itself, then that value overrides any defaults. -* The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. - This is called the _outputUserRoot_ directory. -* Beneath the `outputUserRoot` directory there is an `install` directory, and in - it is an `installBase` directory whose name is the MD5 hash of the Bazel - installation manifest. -* Beneath the `outputUserRoot` directory, an `outputBase` directory - is also created whose name is the MD5 hash of the path name of the workspace - root. So, for example, if Bazel is running in the workspace root - `/home/user/src/my-project` (or in a directory symlinked to that one), then - an output base directory is created called: - `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You - can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. -* You can use Bazel's `--output_base` startup option to override the default - output base directory. For example, - `bazel --output_base=/tmp/bazel/output build x/y:z`. -* You can also use Bazel's `--output_user_root` startup option to override the - default install base and output base directories. For example: - `bazel --output_user_root=/tmp/bazel build x/y:z`. - -The symlinks for "bazel-<workspace-name>", "bazel-out", "bazel-testlogs", -and "bazel-bin" are put in the workspace directory; these symlinks point to some -directories inside a target-specific directory inside the output directory. -These symlinks are only for the user's convenience, as Bazel itself does not -use them. Also, this is done only if the workspace root is writable. +- Bazel must be invoked from a directory containing a repo boundary file, or a subdirectory thereof. In other words, Bazel must be invoked from inside a [repository](../external/overview#repository). Otherwise, an error is reported. +- The *outputRoot* directory defaults to `~/.cache/bazel` on Linux, `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if set, else `%USERPROFILE%` if set, else the result of calling `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the environment variable `$XDG_CACHE_HOME` is set on either Linux or macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel itself, then that value overrides any defaults. +- The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. This is called the *outputUserRoot* directory. +- Beneath the `outputUserRoot` directory there is an `install` directory, and in it is an `installBase` directory whose name is the MD5 hash of the Bazel installation manifest. +- Beneath the `outputUserRoot` directory, an `outputBase` directory is also created whose name is the MD5 hash of the path name of the workspace root. So, for example, if Bazel is running in the workspace root `/home/user/src/my-project` (or in a directory symlinked to that one), then an output base directory is created called: `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. +- You can use Bazel's `--output_base` startup option to override the default output base directory. For example, `bazel --output_base=/tmp/bazel/output build x/y:z`. +- You can also use Bazel's `--output_user_root` startup option to override the default install base and output base directories. For example: `bazel --output_user_root=/tmp/bazel build x/y:z`. + +The symlinks for "bazel-\<workspace-name\>", "bazel-out", "bazel-testlogs", and "bazel-bin" are put in the workspace directory; these symlinks point to some directories inside a target-specific directory inside the output directory. These symlinks are only for the user's convenience, as Bazel itself does not use them. Also, this is done only if the workspace root is writable. ## Layout diagram The directories are laid out as follows: ``` -<workspace-name>/ <== The workspace root - bazel-my-project => <..._main> <== Symlink to execRoot - bazel-out => <...bazel-out> <== Convenience symlink to outputPath - bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) - bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory - -/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot - _bazel_$USER/ <== Top level directory for a given user depends on the user name: +<workspace-name>/ <== The workspace root + bazel-my-project => <..._main> <== Symlink to execRoot + bazel-out => <...bazel-out> <== Convenience symlink to outputPath + bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) + bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory + +/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot + _bazel_$USER/ <== Top level directory for a given user depends on the user name: outputUserRoot install/ - fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase - _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of + fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase + _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of the bazel executable on first run (such as helper scripts and the main Java file BazelServer_deploy.jar) - 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as + 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as /home/user/src/my-project): outputBase - action_cache/ <== Action cache directory hierarchy + action_cache/ <== Action cache directory hierarchy This contains the persistent record of the file metadata (timestamps, and perhaps eventually also MD5 sums) used by the FilesystemValueChecker. - command.log <== A copy of the stdout/stderr output from the most + command.log <== A copy of the stdout/stderr output from the most recent bazel command. - external/ <== The directory that remote repositories are + external/ <== The directory that remote repositories are downloaded/symlinked into. - server/ <== The Bazel server puts all server-related files (such + server/ <== The Bazel server puts all server-related files (such as socket file, logs, etc) here. - jvm.out <== The debugging output for the server. - execroot/ <== The working directory for all actions. For special + jvm.out <== The debugging output for the server. + execroot/ <== The working directory for all actions. For special cases such as sandboxing and remote execution, the actions run in a directory that mimics execroot. Implementation details, such as where the directories are created, are intentionally hidden from the action. Every action can access its inputs and outputs relative to the execroot directory. - _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot - _bin/ <== Helper tools are linked from or copied to here. + _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot + _bin/ <== Helper tools are linked from or copied to here. - bazel-out/ <== All actual output of the build is under here: outputPath - _tmp/actions/ <== Action output directory. This contains a file with the + bazel-out/ <== All actual output of the build is under here: outputPath + _tmp/actions/ <== Action output directory. This contains a file with the stdout/stderr for every action from the most recent bazel run that produced output. - local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; + local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; this is currently encoded - bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) - foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz - foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc - other_package/other.o <== Object files from source //other_package:other.cc - foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named + bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) + foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz + foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc + other_package/other.o <== Object files from source //other_package:other.cc + foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named //foo/bar:baz - foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. + foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. MANIFEST _main/ ... - genfiles/ <== Bazel puts generated source for the target configuration here: + genfiles/ <== Bazel puts generated source for the target configuration here: $(GENDIR) foo/bar.h such as foo/bar.h might be a headerfile generated by //foo:bargen - testlogs/ <== Bazel internal test runner puts test log files here + testlogs/ <== Bazel internal test runner puts test log files here foo/bartest.log such as foo/bar.log might be an output of the //foo:bartest test with foo/bartest.status foo/bartest.status containing exit status of the test (such as PASSED or FAILED (Exit 1), etc) - host/ <== BuildConfiguration for build host (user's workstation), for + host/ <== BuildConfiguration for build host (user's workstation), for building prerequisite tools, that will be used in later stages of the build (ex: Protocol Compiler) - <packages>/ <== Packages referenced in the build appear as if under a regular workspace + <packages>/ <== Packages referenced in the build appear as if under a regular workspace ``` The layout of the \*.runfiles directories is documented in more detail in the places pointed to by RunfilesSupport. ## `bazel clean` -`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` -directory. It also removes the workspace symlinks. The `--expunge` option -will clean the entire outputBase. +`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` directory. It also removes the workspace symlinks. The `--expunge` option will clean the entire outputBase. diff --git a/remote/persistent.mdx b/remote/persistent.mdx index 1a56946a..bd9029f5 100644 --- a/remote/persistent.mdx +++ b/remote/persistent.mdx @@ -2,158 +2,65 @@ title: 'Persistent Workers' --- +This page covers how to use persistent workers, the benefits, requirements, and how workers affect sandboxing. +A persistent worker is a long-running process started by the Bazel server, which functions as a *wrapper* around the actual *tool* (typically a compiler), or is the *tool* itself. In order to benefit from persistent workers, the tool must support doing a sequence of compilations, and the wrapper needs to translate between the tool's API and the request/response format described below. The same worker might be called with and without the `--persistent_worker` flag in the same build, and is responsible for appropriately starting and talking to the tool, as well as shutting down workers on exit. Each worker instance is assigned (but not chrooted to) a separate working directory under `<outputBase>/bazel-workers`. -This page covers how to use persistent workers, the benefits, requirements, and -how workers affect sandboxing. +Using persistent workers is an [execution strategy](/docs/user-manual#execution-strategy) that decreases start-up overhead, allows more JIT compilation, and enables caching of for example the abstract syntax trees in the action execution. This strategy achieves these improvements by sending multiple requests to a long-running process. -A persistent worker is a long-running process started by the Bazel server, which -functions as a *wrapper* around the actual *tool* (typically a compiler), or is -the *tool* itself. In order to benefit from persistent workers, the tool must -support doing a sequence of compilations, and the wrapper needs to translate -between the tool's API and the request/response format described below. The same -worker might be called with and without the `--persistent_worker` flag in the -same build, and is responsible for appropriately starting and talking to the -tool, as well as shutting down workers on exit. Each worker instance is assigned -(but not chrooted to) a separate working directory under -`/bazel-workers`. +Persistent workers are implemented for multiple languages, including Java, [Scala](https://github.com/bazelbuild/rules_scala), [Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. -Using persistent workers is an -[execution strategy](/docs/user-manual#execution-strategy) that decreases -start-up overhead, allows more JIT compilation, and enables caching of for -example the abstract syntax trees in the action execution. This strategy -achieves these improvements by sending multiple requests to a long-running -process. - -Persistent workers are implemented for multiple languages, including Java, -[Scala](https://github.com/bazelbuild/rules_scala), -[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. - -Programs using a NodeJS runtime can use the -[@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to -implement the worker protocol. +Programs using a NodeJS runtime can use the [@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to implement the worker protocol. ## Using persistent workers -[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) -uses persistent workers by default when executing builds, though remote -execution takes precedence. For actions that do not support persistent workers, -Bazel falls back to starting a tool instance for each action. You can explicitly -set your build to use persistent workers by setting the `worker` -[strategy](/docs/user-manual#execution-strategy) for the applicable tool -mnemonics. As a best practice, this example includes specifying `local` as a -fallback to the `worker` strategy: +[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) uses persistent workers by default when executing builds, though remote execution takes precedence. For actions that do not support persistent workers, Bazel falls back to starting a tool instance for each action. You can explicitly set your build to use persistent workers by setting the `worker` [strategy](/docs/user-manual#execution-strategy) for the applicable tool mnemonics. As a best practice, this example includes specifying `local` as a fallback to the `worker` strategy: ```posix-terminal -bazel build //{{ '' }}my:target{{ '' }} --strategy=Javac=worker,local +bazel build //<var>my:target</var> --strategy=Javac=worker,local ``` -Using the workers strategy instead of the local strategy can boost compilation -speed significantly, depending on implementation. For Java, builds can be 2–4 -times faster, sometimes more for incremental compilation. Compiling Bazel is -about 2.5 times as fast with workers. For more details, see the -"[Choosing number of workers](#number-of-workers)" section. - -If you also have a remote build environment that matches your local build -environment, you can use the experimental -[*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), -which races a remote execution and a worker execution. To enable the dynamic -strategy, pass the -[--experimental_spawn_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) -flag. This strategy automatically enables workers, so there is no need to -specify the `worker` strategy, but you can still use `local` or `sandboxed` as -fallbacks. +Using the workers strategy instead of the local strategy can boost compilation speed significantly, depending on implementation. For Java, builds can be 2–4 times faster, sometimes more for incremental compilation. Compiling Bazel is about 2.5 times as fast with workers. For more details, see the "[Choosing number of workers](#number-of-workers)" section. + +If you also have a remote build environment that matches your local build environment, you can use the experimental [*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), which races a remote execution and a worker execution. To enable the dynamic strategy, pass the [--experimental\_spawn\_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) flag. This strategy automatically enables workers, so there is no need to specify the `worker` strategy, but you can still use `local` or `sandboxed` as fallbacks. ## Choosing number of workers -The default number of worker instances per mnemonic is 4, but can be adjusted -with the -[`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) -flag. There is a trade-off between making good use of the available CPUs and the -amount of JIT compilation and cache hits you get. With more workers, more -targets will pay start-up costs of running non-JITted code and hitting cold -caches. If you have a small number of targets to build, a single worker may give -the best trade-off between compilation speed and resource usage (for example, -see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). -The `worker_max_instances` flag sets the maximum number of worker instances per -mnemonic and flag set (see below), so in a mixed system you could end up using -quite a lot of memory if you keep the default value. For incremental builds the -benefit of multiple worker instances is even smaller. - -This graph shows the from-scratch compilation times for Bazel (target -`//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation -with 64 GB of RAM. For each worker configuration, five clean builds are run and -the average of the last four are taken. +The default number of worker instances per mnemonic is 4, but can be adjusted with the [`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) flag. There is a trade-off between making good use of the available CPUs and the amount of JIT compilation and cache hits you get. With more workers, more targets will pay start-up costs of running non-JITted code and hitting cold caches. If you have a small number of targets to build, a single worker may give the best trade-off between compilation speed and resource usage (for example, see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). The `worker_max_instances` flag sets the maximum number of worker instances per mnemonic and flag set (see below), so in a mixed system you could end up using quite a lot of memory if you keep the default value. For incremental builds the benefit of multiple worker instances is even smaller. + +This graph shows the from-scratch compilation times for Bazel (target `//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation with 64 GB of RAM. For each worker configuration, five clean builds are run and the average of the last four are taken. ![Graph of performance improvements of clean builds](/docs/images/workers-clean-chart.png "Performance improvements of clean builds") **Figure 1.** Graph of performance improvements of clean builds. -For this configuration, two workers give the fastest compile, though at only 14% -improvement compared to one worker. One worker is a good option if you want to -use less memory. +For this configuration, two workers give the fastest compile, though at only 14% improvement compared to one worker. One worker is a good option if you want to use less memory. -Incremental compilation typically benefits even more. Clean builds are -relatively rare, but changing a single file between compiles is common, in -particular in test-driven development. The above example also has some non-Java -packaging actions to it that can overshadow the incremental compile time. +Incremental compilation typically benefits even more. Clean builds are relatively rare, but changing a single file between compiles is common, in particular in test-driven development. The above example also has some non-Java packaging actions to it that can overshadow the incremental compile time. -Recompiling the Java sources only -(`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) -after changing an internal string constant in -[AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) -gives a 3x speed-up (average of 20 incremental builds with one warmup build -discarded): +Recompiling the Java sources only (`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) after changing an internal string constant in [AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) gives a 3x speed-up (average of 20 incremental builds with one warmup build discarded): ![Graph of performance improvements of incremental builds](/docs/images/workers-incremental-chart.png "Performance improvements of incremental builds") **Figure 2.** Graph of performance improvements of incremental builds. -The speed-up depends on the change being made. A speed-up of a factor 6 is -measured in the above situation when a commonly used constant is changed. +The speed-up depends on the change being made. A speed-up of a factor 6 is measured in the above situation when a commonly used constant is changed. ## Modifying persistent workers -You can pass the -[`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) -flag to specify start-up flags to workers, keyed by mnemonic. For instance, -passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. -Only one worker flag can be set per use of this flag, and only for one mnemonic. -Workers are not just created separately for each mnemonic, but also for -variations in their start-up flags. Each combination of mnemonic and start-up -flags is combined into a `WorkerKey`, and for each `WorkerKey` up to -`worker_max_instances` workers may be created. See the next section for how the -action configuration can also specify set-up flags. - -Passing the -[`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) -flag makes each worker request use a separate sandbox directory for all its -inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, -especially on macOS, but gives a better correctness guarantee. - -The -[`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) -flag is mainly useful for debugging and profiling. This flag forces all workers -to quit once a build is done. You can also pass -[`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to -get more output about what the workers are doing. This flag is reflected in the -`verbosity` field in `WorkRequest`, allowing worker implementations to also be -more verbose. - -Workers store their logs in the `/bazel-workers` directory, for -example -`/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. -The file name includes the worker id and the mnemonic. Since there can be more -than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` -log files for a given mnemonic. - -For Android builds, see details at the -[Android Build Performance page](/docs/android-build-performance). +You can pass the [`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) flag to specify start-up flags to workers, keyed by mnemonic. For instance, passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. Only one worker flag can be set per use of this flag, and only for one mnemonic. Workers are not just created separately for each mnemonic, but also for variations in their start-up flags. Each combination of mnemonic and start-up flags is combined into a `WorkerKey`, and for each `WorkerKey` up to `worker_max_instances` workers may be created. See the next section for how the action configuration can also specify set-up flags. + +Passing the [`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) flag makes each worker request use a separate sandbox directory for all its inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, especially on macOS, but gives a better correctness guarantee. + +The [`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) flag is mainly useful for debugging and profiling. This flag forces all workers to quit once a build is done. You can also pass [`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to get more output about what the workers are doing. This flag is reflected in the `verbosity` field in `WorkRequest`, allowing worker implementations to also be more verbose. + +Workers store their logs in the `<outputBase>/bazel-workers` directory, for example `/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. The file name includes the worker id and the mnemonic. Since there can be more than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` log files for a given mnemonic. + +For Android builds, see details at the [Android Build Performance page](/docs/android-build-performance). ## Implementing persistent workers -See the [creating persistent workers](/remote/creating) page for more -information on how to make a worker. +See the [creating persistent workers](/remote/creating) page for more information on how to make a worker. This example shows a Starlark configuration for a worker that uses JSON: @@ -174,14 +81,9 @@ ctx.actions.run( ) ``` -With this definition, the first use of this action would start with executing -the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request -to compile `Foo.java` would then look like: +With this definition, the first use of this action would start with executing the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request to compile `Foo.java` would then look like: -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), -the JSON protocol uses "camel case" (`requestId`). In this document, we will use -camel case in the JSON examples, but snake case when talking about the field -regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). In this document, we will use camel case in the JSON examples, but snake case when talking about the field regardless of protocol. ```json { @@ -193,12 +95,7 @@ regardless of protocol. } ``` -The worker receives this on `stdin` in newline-delimited JSON format (because -`requires-worker-protocol` is set to JSON). The worker then performs the action, -and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then -parses this response and manually converts it to a `WorkResponse` proto. To -communicate with the associated worker using binary-encoded protobuf instead of -JSON, `requires-worker-protocol` would be set to `proto`, like this: +The worker receives this on `stdin` in newline-delimited JSON format (because `requires-worker-protocol` is set to JSON). The worker then performs the action, and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then parses this response and manually converts it to a `WorkResponse` proto. To communicate with the associated worker using binary-encoded protobuf instead of JSON, `requires-worker-protocol` would be set to `proto`, like this: ``` execution_requirements = { @@ -207,59 +104,31 @@ JSON, `requires-worker-protocol` would be set to `proto`, like this: } ``` -If you do not include `requires-worker-protocol` in the execution requirements, -Bazel will default the worker communication to use protobuf. +If you do not include `requires-worker-protocol` in the execution requirements, Bazel will default the worker communication to use protobuf. -Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this -configuration allowed changing the `max_mem` parameter, a separate worker would -be spawned for each value used. This can lead to excessive memory consumption if -too many variations are used. +Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this configuration allowed changing the `max_mem` parameter, a separate worker would be spawned for each value used. This can lead to excessive memory consumption if too many variations are used. -Each worker can currently only process one request at a time. The experimental -[multiplex workers](/remote/multiplex) feature allows using multiple -threads, if the underlying tool is multithreaded and the wrapper is set up to -understand this. +Each worker can currently only process one request at a time. The experimental [multiplex workers](/remote/multiplex) feature allows using multiple threads, if the underlying tool is multithreaded and the wrapper is set up to understand this. -In -[this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), -you can see example worker wrappers written in Java as well as in Python. If you -are working in JavaScript or TypeScript, the -[@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) -and -[nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) -might be helpful. +In [this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), you can see example worker wrappers written in Java as well as in Python. If you are working in JavaScript or TypeScript, the [@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) and [nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) might be helpful. ## How do workers affect sandboxing? -Using the `worker` strategy by default does not run the action in a -[sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the -`--worker_sandboxing` flag to run all workers inside sandboxes, making sure each -execution of the tool only sees the input files it's supposed to have. The tool -may still leak information between requests internally, for instance through a -cache. Using `dynamic` strategy -[requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). +Using the `worker` strategy by default does not run the action in a [sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the `--worker_sandboxing` flag to run all workers inside sandboxes, making sure each execution of the tool only sees the input files it's supposed to have. The tool may still leak information between requests internally, for instance through a cache. Using `dynamic` strategy [requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). -To allow correct use of compiler caches with workers, a digest is passed along -with each input file. Thus the compiler or the wrapper can check if the input is -still valid without having to read the file. +To allow correct use of compiler caches with workers, a digest is passed along with each input file. Thus the compiler or the wrapper can check if the input is still valid without having to read the file. -Even when using the input digests to guard against unwanted caching, sandboxed -workers offer less strict sandboxing than a pure sandbox, because the tool may -keep other internal state that has been affected by previous requests. +Even when using the input digests to guard against unwanted caching, sandboxed workers offer less strict sandboxing than a pure sandbox, because the tool may keep other internal state that has been affected by previous requests. -Multiplex workers can only be sandboxed if the worker implementation support it, -and this sandboxing must be separately enabled with the -`--experimental_worker_multiplex_sandboxing` flag. See more details in -[the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). +Multiplex workers can only be sandboxed if the worker implementation support it, and this sandboxing must be separately enabled with the `--experimental_worker_multiplex_sandboxing` flag. See more details in [the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). ## Further reading For more information on persistent workers, see: -* [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) -* [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) -* [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) -* [Front End Development with Bazel: Angular/TypeScript and Persistent Workers - w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) -* [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) -* [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) +- [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) +- [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) +- [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) +- [Front End Development with Bazel: Angular/TypeScript and Persistent Workers w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) +- [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) +- [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) diff --git a/remote/rbe.mdx b/remote/rbe.mdx index 75d4a155..0d8bf255 100644 --- a/remote/rbe.mdx +++ b/remote/rbe.mdx @@ -2,32 +2,20 @@ title: 'Remote Execution Overview' --- +This page covers the benefits, requirements, and options for running Bazel with remote execution. - -This page covers the benefits, requirements, and options for running Bazel -with remote execution. - -By default, Bazel executes builds and tests on your local machine. Remote -execution of a Bazel build allows you to distribute build and test actions -across multiple machines, such as a datacenter. +By default, Bazel executes builds and tests on your local machine. Remote execution of a Bazel build allows you to distribute build and test actions across multiple machines, such as a datacenter. Remote execution provides the following benefits: -* Faster build and test execution through scaling of nodes available - for parallel actions -* A consistent execution environment for a development team -* Reuse of build outputs across a development team +- Faster build and test execution through scaling of nodes available for parallel actions +- A consistent execution environment for a development team +- Reuse of build outputs across a development team -Bazel uses an open-source -[gRPC protocol](https://github.com/bazelbuild/remote-apis) -to allow for remote execution and remote caching. +Bazel uses an open-source [gRPC protocol](https://github.com/bazelbuild/remote-apis) to allow for remote execution and remote caching. -For a list of commercially supported remote execution services as well as -self-service tools, see -[Remote Execution Services](https://www.bazel.build/remote-execution-services.html) +For a list of commercially supported remote execution services as well as self-service tools, see [Remote Execution Services](https://www.bazel.build/remote-execution-services.html) ## Requirements -Remote execution of Bazel builds imposes a set of mandatory configuration -constraints on the build. For more information, see -[Adapting Bazel Rules for Remote Execution](/remote/rules). +Remote execution of Bazel builds imposes a set of mandatory configuration constraints on the build. For more information, see [Adapting Bazel Rules for Remote Execution](/remote/rules). diff --git a/remote/rules.mdx b/remote/rules.mdx index 340ab02c..4ea4c9c3 100644 --- a/remote/rules.mdx +++ b/remote/rules.mdx @@ -2,179 +2,79 @@ title: 'Adapting Bazel Rules for Remote Execution' --- +This page is intended for Bazel users writing custom build and test rules who want to understand the requirements for Bazel rules in the context of remote execution. +Remote execution allows Bazel to execute actions on a separate platform, such as a datacenter. Bazel uses a [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) for its remote execution. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source project that aims to provide a distributed remote execution platform. -This page is intended for Bazel users writing custom build and test rules -who want to understand the requirements for Bazel rules in the context of -remote execution. +This page uses the following terminology when referring to different environment types or *platforms*: -Remote execution allows Bazel to execute actions on a separate platform, such as -a datacenter. Bazel uses a -[gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) -for its remote execution. You can try remote execution with -[bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), -an open-source project that aims to provide a distributed remote execution -platform. - -This page uses the following terminology when referring to different -environment types or *platforms*: - -* **Host platform** - where Bazel runs. -* **Execution platform** - where Bazel actions run. -* **Target platform** - where the build outputs (and some actions) run. +- **Host platform** - where Bazel runs. +- **Execution platform** - where Bazel actions run. +- **Target platform** - where the build outputs (and some actions) run. ## Overview -When configuring a Bazel build for remote execution, you must follow the -guidelines described in this page to ensure the build executes remotely -error-free. This is due to the nature of remote execution, namely: +When configuring a Bazel build for remote execution, you must follow the guidelines described in this page to ensure the build executes remotely error-free. This is due to the nature of remote execution, namely: -* **Isolated build actions.** Build tools do not retain state and dependencies - cannot leak between them. +- **Isolated build actions.** Build tools do not retain state and dependencies cannot leak between them. -* **Diverse execution environments.** Local build configuration is not always - suitable for remote execution environments. +- **Diverse execution environments.** Local build configuration is not always suitable for remote execution environments. -This page describes the issues that can arise when implementing custom Bazel -build and test rules for remote execution and how to avoid them. It covers the -following topics: +This page describes the issues that can arise when implementing custom Bazel build and test rules for remote execution and how to avoid them. It covers the following topics: -* [Invoking build tools through toolchain rules](#toolchain-rules) -* [Managing implicit dependencies](#manage-dependencies) -* [Managing platform-dependent binaries](#manage-binaries) -* [Managing configure-style WORKSPACE rules](#manage-workspace-rules) +- [Invoking build tools through toolchain rules](#toolchain-rules) +- [Managing implicit dependencies](#manage-dependencies) +- [Managing platform-dependent binaries](#manage-binaries) +- [Managing configure-style WORKSPACE rules](#manage-workspace-rules) ## Invoking build tools through toolchain rules -A Bazel toolchain rule is a configuration provider that tells a build rule what -build tools, such as compilers and linkers, to use and how to configure them -using parameters defined by the rule's creator. A toolchain rule allows build -and test rules to invoke build tools in a predictable, preconfigured manner -that's compatible with remote execution. For example, use a toolchain rule -instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local -variables that may not be set to equivalent values (or at all) in the remote -execution environment. - -Toolchain rules currently exist for Bazel build and test rules for -[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch -ain.bzl), -[Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), -and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), -and new toolchain rules are under way for other languages and tools such as -[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). -If a toolchain rule does not exist for the tool your rule uses, consider -[creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). +A Bazel toolchain rule is a configuration provider that tells a build rule what build tools, such as compilers and linkers, to use and how to configure them using parameters defined by the rule's creator. A toolchain rule allows build and test rules to invoke build tools in a predictable, preconfigured manner that's compatible with remote execution. For example, use a toolchain rule instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local variables that may not be set to equivalent values (or at all) in the remote execution environment. + +Toolchain rules currently exist for Bazel build and test rules for \[Scala]\([https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch) ain.bzl), [Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), and new toolchain rules are under way for other languages and tools such as [bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). If a toolchain rule does not exist for the tool your rule uses, consider [creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). ## Managing implicit dependencies -If a build tool can access dependencies across build actions, those actions will -fail when remotely executed because each remote build action is executed -separately from others. Some build tools retain state across build actions and -access dependencies that have not been explicitly included in the tool -invocation, which will cause remotely executed build actions to fail. - -For example, when Bazel instructs a stateful compiler to locally build _foo_, -the compiler retains references to foo's build outputs. When Bazel then -instructs the compiler to build _bar_, which depends on _foo_, without -explicitly stating that dependency in the BUILD file for inclusion in the -compiler invocation, the action executes successfully as long as the same -compiler instance executes for both actions (as is typical for local execution). -However, since in a remote execution scenario each build action executes a -separate compiler instance, compiler state and _bar_'s implicit dependency on -_foo_ will be lost and the build will fail. - -To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the -local Docker sandbox, which has the same restrictions for dependencies as remote -execution. Use the sandbox to prepare your build for remote execution by -identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) -for more information. +If a build tool can access dependencies across build actions, those actions will fail when remotely executed because each remote build action is executed separately from others. Some build tools retain state across build actions and access dependencies that have not been explicitly included in the tool invocation, which will cause remotely executed build actions to fail. + +For example, when Bazel instructs a stateful compiler to locally build *foo*, the compiler retains references to foo's build outputs. When Bazel then instructs the compiler to build *bar*, which depends on *foo*, without explicitly stating that dependency in the BUILD file for inclusion in the compiler invocation, the action executes successfully as long as the same compiler instance executes for both actions (as is typical for local execution). However, since in a remote execution scenario each build action executes a separate compiler instance, compiler state and *bar*'s implicit dependency on *foo* will be lost and the build will fail. + +To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the local Docker sandbox, which has the same restrictions for dependencies as remote execution. Use the sandbox to prepare your build for remote execution by identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) for more information. ## Managing platform-dependent binaries -Typically, a binary built on the host platform cannot safely execute on an -arbitrary remote execution platform due to potentially mismatched dependencies. -For example, the SingleJar binary supplied with Bazel targets the host platform. -However, for remote execution, SingleJar must be compiled as part of the process -of building your code so that it targets the remote execution platform. (See the -[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) +Typically, a binary built on the host platform cannot safely execute on an arbitrary remote execution platform due to potentially mismatched dependencies. For example, the SingleJar binary supplied with Bazel targets the host platform. However, for remote execution, SingleJar must be compiled as part of the process of building your code so that it targets the remote execution platform. (See the [target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) -Do not ship binaries of build tools required by your build with your source code -unless you are sure they will safely run in your execution platform. Instead, do -one of the following: +Do not ship binaries of build tools required by your build with your source code unless you are sure they will safely run in your execution platform. Instead, do one of the following: -* Ship or externally reference the source code for the tool so that it can be - built for the remote execution platform. +- Ship or externally reference the source code for the tool so that it can be built for the remote execution platform. -* Pre-install the tool into the remote execution environment (for example, a - toolchain container) if it's stable enough and use toolchain rules to run it - in your build. +- Pre-install the tool into the remote execution environment (for example, a toolchain container) if it's stable enough and use toolchain rules to run it in your build. ## Managing configure-style WORKSPACE rules -Bazel's `WORKSPACE` rules can be used for probing the host platform for tools -and libraries required by the build, which, for local builds, is also Bazel's -execution platform. If the build explicitly depends on local build tools and -artifacts, it will fail during remote execution if the remote execution platform -is not identical to the host platform. - -The following actions performed by `WORKSPACE` rules are not compatible with -remote execution: - -* **Building binaries.** Executing compilation actions in `WORKSPACE` rules - results in binaries that are incompatible with the remote execution platform - if different from the host platform. - -* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` - rules require that their dependencies be pre-installed on the host platform. - Such packages, built specifically for the host platform, will be - incompatible with the remote execution platform if different from the host - platform. - -* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries - installed on the host platform created via `WORKSPACE` rules will cause the - build to fail on the remote execution platform as Bazel will not be able to - locate them. Instead, create symlinks using standard build actions so that - the symlinked tools and libraries are accessible from Bazel's `runfiles` - tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) - to symlink target files outside of the external repo directory. - -* **Mutating the host platform.** Avoid creating files outside of the Bazel - `runfiles` tree, creating environment variables, and similar actions, as - they may behave unexpectedly on the remote execution platform. +Bazel's `WORKSPACE` rules can be used for probing the host platform for tools and libraries required by the build, which, for local builds, is also Bazel's execution platform. If the build explicitly depends on local build tools and artifacts, it will fail during remote execution if the remote execution platform is not identical to the host platform. + +The following actions performed by `WORKSPACE` rules are not compatible with remote execution: + +- **Building binaries.** Executing compilation actions in `WORKSPACE` rules results in binaries that are incompatible with the remote execution platform if different from the host platform. + +- **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` rules require that their dependencies be pre-installed on the host platform. Such packages, built specifically for the host platform, will be incompatible with the remote execution platform if different from the host platform. + +- **Symlinking to local tools or artifacts.** Symlinks to tools or libraries installed on the host platform created via `WORKSPACE` rules will cause the build to fail on the remote execution platform as Bazel will not be able to locate them. Instead, create symlinks using standard build actions so that the symlinked tools and libraries are accessible from Bazel's `runfiles` tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) to symlink target files outside of the external repo directory. + +- **Mutating the host platform.** Avoid creating files outside of the Bazel `runfiles` tree, creating environment variables, and similar actions, as they may behave unexpectedly on the remote execution platform. To help find potential non-hermetic behavior you can use [Workspace rules log](/remote/workspace). -If an external dependency executes specific operations dependent on the host -platform, you should split those operations between `WORKSPACE` and build -rules as follows: - -* **Platform inspection and dependency enumeration.** These operations are - safe to execute locally via `WORKSPACE` rules, which can check which - libraries are installed, download packages that must be built, and prepare - required artifacts for compilation. For remote execution, these rules must - also support using pre-checked artifacts to provide the information that - would normally be obtained during host platform inspection. Pre-checked - artifacts allow Bazel to describe dependencies as if they were local. Use - conditional statements or the `--override_repository` flag for this. - -* **Generating or compiling target-specific artifacts and platform mutation**. - These operations must be executed via regular build rules. Actions that - produce target-specific artifacts for external dependencies must execute - during the build. - -To more easily generate pre-checked artifacts for remote execution, you can use -`WORKSPACE` rules to emit generated files. You can run those rules on each new -execution environment, such as inside each toolchain container, and check the -outputs of your remote execution build in to your source repo to reference. - -For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) -and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), -the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). -For local execution, files produced by checking the host environment are used. -For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) -on an environment variable allows the rule to use files that are checked into -the repo. - -The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) -that can run both locally and remotely, and perform the necessary processing -that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). +If an external dependency executes specific operations dependent on the host platform, you should split those operations between `WORKSPACE` and build rules as follows: + +- **Platform inspection and dependency enumeration.** These operations are safe to execute locally via `WORKSPACE` rules, which can check which libraries are installed, download packages that must be built, and prepare required artifacts for compilation. For remote execution, these rules must also support using pre-checked artifacts to provide the information that would normally be obtained during host platform inspection. Pre-checked artifacts allow Bazel to describe dependencies as if they were local. Use conditional statements or the `--override_repository` flag for this. + +- **Generating or compiling target-specific artifacts and platform mutation**. These operations must be executed via regular build rules. Actions that produce target-specific artifacts for external dependencies must execute during the build. + +To more easily generate pre-checked artifacts for remote execution, you can use `WORKSPACE` rules to emit generated files. You can run those rules on each new execution environment, such as inside each toolchain container, and check the outputs of your remote execution build in to your source repo to reference. + +For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). For local execution, files produced by checking the host environment are used. For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) on an environment variable allows the rule to use files that are checked into the repo. + +The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) that can run both locally and remotely, and perform the necessary processing that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/remote/sandbox.mdx b/remote/sandbox.mdx index 8c3f9565..98485efc 100644 --- a/remote/sandbox.mdx +++ b/remote/sandbox.mdx @@ -2,223 +2,143 @@ title: 'Troubleshooting Bazel Remote Execution with Docker Sandbox' --- +Bazel builds that succeed locally may fail when executed remotely due to restrictions and requirements that do not affect local builds. The most common causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). +This page describes how to identify and resolve the most common issues that arise with remote execution using the Docker sandbox feature, which imposes restrictions upon the build equal to those of remote execution. This allows you to troubleshoot your build without the need for a remote execution service. -Bazel builds that succeed locally may fail when executed remotely due to -restrictions and requirements that do not affect local builds. The most common -causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). +The Docker sandbox feature mimics the restrictions of remote execution as follows: -This page describes how to identify and resolve the most common issues that -arise with remote execution using the Docker sandbox feature, which imposes -restrictions upon the build equal to those of remote execution. This allows you -to troubleshoot your build without the need for a remote execution service. +- **Build actions execute in toolchain containers.** You can use the same toolchain containers to run your build locally and remotely via a service supporting containerized remote execution. -The Docker sandbox feature mimics the restrictions of remote execution as -follows: +- **No extraneous data crosses the container boundary.** Only explicitly declared inputs and outputs enter and leave the container, and only after the associated build action successfully completes. -* **Build actions execute in toolchain containers.** You can use the same - toolchain containers to run your build locally and remotely via a service - supporting containerized remote execution. +- **Each action executes in a fresh container.** A new, unique container is created for each spawned build action. -* **No extraneous data crosses the container boundary.** Only explicitly - declared inputs and outputs enter and leave the container, and only after - the associated build action successfully completes. - -* **Each action executes in a fresh container.** A new, unique container is - created for each spawned build action. - -Note: Builds take noticeably more time to complete when the Docker sandbox -feature is enabled. This is normal. +Note: Builds take noticeably more time to complete when the Docker sandbox feature is enabled. This is normal. You can troubleshoot these issues using one of the following methods: -* **[Troubleshooting natively.](#troubleshooting-natively)** With this method, - Bazel and its build actions run natively on your local machine. The Docker - sandbox feature imposes restrictions upon the build equal to those of remote - execution. However, this method will not detect local tools, states, and - data leaking into your build, which will cause problems with remote execution. +- **[Troubleshooting natively.](#troubleshooting-natively)** With this method, Bazel and its build actions run natively on your local machine. The Docker sandbox feature imposes restrictions upon the build equal to those of remote execution. However, this method will not detect local tools, states, and data leaking into your build, which will cause problems with remote execution. -* **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** - With this method, Bazel and its build actions run inside a Docker container, - which allows you to detect tools, states, and data leaking from the local - machine into the build in addition to imposing restrictions - equal to those of remote execution. This method provides insight into your - build even if portions of the build are failing. This method is experimental - and not officially supported. +- **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** With this method, Bazel and its build actions run inside a Docker container, which allows you to detect tools, states, and data leaking from the local machine into the build in addition to imposing restrictions equal to those of remote execution. This method provides insight into your build even if portions of the build are failing. This method is experimental and not officially supported. ## Prerequisites Before you begin troubleshooting, do the following if you have not already done so: -* Install Docker and configure the permissions required to run it. -* Install Bazel 0.14.1 or later. Earlier versions do not support the Docker - sandbox feature. -* Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) - repo, pinned to the latest release version, to your build's `WORKSPACE` file - as described [here](https://releases.bazel.build/bazel-toolchains.html). -* Add flags to your `.bazelrc` file to enable the feature. Create the file in - the root directory of your Bazel project if it does not exist. Flags below - are a reference sample. Please see the latest - [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) - file in the bazel-toolchains repo and copy the values of the flags defined - there for config `docker-sandbox`. +- Install Docker and configure the permissions required to run it. +- Install Bazel 0.14.1 or later. Earlier versions do not support the Docker sandbox feature. +- Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) repo, pinned to the latest release version, to your build's `WORKSPACE` file as described [here](https://releases.bazel.build/bazel-toolchains.html). +- Add flags to your `.bazelrc` file to enable the feature. Create the file in the root directory of your Bazel project if it does not exist. Flags below are a reference sample. Please see the latest [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) file in the bazel-toolchains repo and copy the values of the flags defined there for config `docker-sandbox`. ``` # Docker Sandbox Mode -build:docker-sandbox --host_javabase=<...> -build:docker-sandbox --javabase=<...> -build:docker-sandbox --crosstool_top=<...> -build:docker-sandbox --experimental_docker_image=<...> +build:docker-sandbox --host_javabase=<...> +build:docker-sandbox --javabase=<...> +build:docker-sandbox --crosstool_top=<...> +build:docker-sandbox --experimental_docker_image=<...> build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker build:docker-sandbox --experimental_docker_verbose build:docker-sandbox --experimental_enable_docker_sandbox ``` -Note: The flags referenced in the `.bazelrc` file shown above are configured -to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) -container. +Note: The flags referenced in the `.bazelrc` file shown above are configured to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) container. If your rules require additional tools, do the following: -1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) - and [building](https://docs.docker.com/engine/reference/commandline/build/) - the image locally. - -2. Replace the value of the `--experimental_docker_image` flag above with the - name of your custom container image. +1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) and [building](https://docs.docker.com/engine/reference/commandline/build/) the image locally. +2. Replace the value of the `--experimental_docker_image` flag above with the name of your custom container image. ## Troubleshooting natively -This method executes Bazel and all of its build actions directly on the local -machine and is a reliable way to confirm whether your build will succeed when -executed remotely. +This method executes Bazel and all of its build actions directly on the local machine and is a reliable way to confirm whether your build will succeed when executed remotely. -However, with this method, locally installed tools, binaries, and data may leak -into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). -Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) -in addition to troubleshooting natively. +However, with this method, locally installed tools, binaries, and data may leak into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) in addition to troubleshooting natively. ### Step 1: Run the build -1. Add the `--config=docker-sandbox` flag to the Bazel command that executes - your build. For example: +1. Add the `--config=docker-sandbox` flag to the Bazel command that executes your build. For example: - ```posix-terminal - bazel --bazelrc=.bazelrc build --config=docker-sandbox target - ``` + ```posix-terminal + bazel --bazelrc=.bazelrc build --config=docker-sandbox <var>target</var> + ``` -2. Run the build and wait for it to complete. The build will run up to four - times slower than normal due to the Docker sandbox feature. +2. Run the build and wait for it to complete. The build will run up to four times slower than normal due to the Docker sandbox feature. You may encounter the following error: -```none {:.devsite-disable-click-to-copy} +```none ERROR: 'docker' is an invalid value for docker spawn strategy. ``` -If you do, run the build again with the `--experimental_docker_verbose` flag. -This flag enables verbose error messages. This error is typically caused by a -faulty Docker installation or lack of permissions to execute it under the -current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) -for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). +If you do, run the build again with the `--experimental_docker_verbose` flag. This flag enables verbose error messages. This error is typically caused by a faulty Docker installation or lack of permissions to execute it under the current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). ### Step 2: Resolve detected issues The following are the most commonly encountered issues and their workarounds. -* **A file, tool, binary, or resource referenced by the Bazel runfiles tree is - missing.**. Confirm that all dependencies of the affected targets have been - [explicitly declared](/concepts/dependencies). See - [Managing implicit dependencies](/remote/rules#manage-dependencies) - for more information. +- **A file, tool, binary, or resource referenced by the Bazel runfiles tree is missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/concepts/dependencies). See [Managing implicit dependencies](/remote/rules#manage-dependencies) for more information. -* **A file, tool, binary, or resource referenced by an absolute path or the `PATH` - variable is missing.** Confirm that all required tools are installed within - the toolchain container and use [toolchain rules](/extending/toolchains) to properly - declare dependencies pointing to the missing resource. See - [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) - for more information. +- **A file, tool, binary, or resource referenced by an absolute path or the `PATH` variable is missing.** Confirm that all required tools are installed within the toolchain container and use [toolchain rules](/extending/toolchains) to properly declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) for more information. -* **A binary execution fails.** One of the build rules is referencing a binary - incompatible with the execution environment (the Docker container). See - [Managing platform-dependent binaries](/remote/rules#manage-binaries) - for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) - for help. +- **A binary execution fails.** One of the build rules is referencing a binary incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote/rules#manage-binaries) for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. -* **A file from `@local-jdk` is missing or causing errors.** The Java binaries - on your local machine are leaking into the build while being incompatible with - it. Use [`java_toolchain`](/reference/be/java#java_toolchain) - in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. +- **A file from `@local-jdk` is missing or causing errors.** The Java binaries on your local machine are leaking into the build while being incompatible with it. Use [`java_toolchain`](/reference/be/java#java_toolchain) in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. -* **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +- **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. ## Troubleshooting in a Docker container -With this method, Bazel runs inside a host Docker container, and Bazel's build -actions execute inside individual toolchain containers spawned by the Docker -sandbox feature. The sandbox spawns a brand new toolchain container for each -build action and only one action executes in each toolchain container. +With this method, Bazel runs inside a host Docker container, and Bazel's build actions execute inside individual toolchain containers spawned by the Docker sandbox feature. The sandbox spawns a brand new toolchain container for each build action and only one action executes in each toolchain container. -This method provides more granular control of tools installed in the host -environment. By separating the execution of the build from the execution of its -build actions and keeping the installed tooling to a minimum, you can verify -whether your build has any dependencies on the local execution environment. +This method provides more granular control of tools installed in the host environment. By separating the execution of the build from the execution of its build actions and keeping the installed tooling to a minimum, you can verify whether your build has any dependencies on the local execution environment. ### Step 1: Build the container -Note: The commands below are tailored specifically for a `debian:stretch` base. -For other bases, modify them as necessary. +Note: The commands below are tailored specifically for a `debian:stretch` base. For other bases, modify them as necessary. -1. Create a `Dockerfile` that creates the Docker container and installs Bazel - with a minimal set of build tools: +1. Create a `Dockerfile` that creates the Docker container and installs Bazel with a minimal set of build tools: - ``` - FROM debian:stretch + ``` + FROM debian:stretch - RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim + RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim - RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - + RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - - RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" - RUN apt-get update && apt-get install -y docker-ce + RUN apt-get update && apt-get install -y docker-ce - RUN wget https://releases.bazel.build//release/bazel--installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh + RUN wget https://releases.bazel.build/<latest Bazel version>/release/bazel-<latest Bazel version>-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh - RUN ./bazel-installer.sh - ``` + RUN ./bazel-installer.sh + ``` -2. Build the container as `bazel_container`: +2. Build the container as `bazel_container`: - ```posix-terminal - docker build -t bazel_container - < Dockerfile - ``` + ```posix-terminal + docker build -t bazel_container - < Dockerfile + ``` ### Step 2: Start the container -Start the Docker container using the command shown below. In the command, -substitute the path to the source code on your host that you want to build. +Start the Docker container using the command shown below. In the command, substitute the path to the source code on your host that you want to build. ```posix-terminal docker run -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp:/tmp \ - -v {{ '' }}your source code directory{{ '' }}:/src \ + -v <var>your source code directory</var>:/src \ -w /src \ bazel_container \ /bin/bash ``` -This command runs the container as root, mapping the docker socket, and mounting -the `/tmp` directory. This allows Bazel to spawn other Docker containers and to -use directories under `/tmp` to share files with those containers. Your source -code is available at `/src` inside the container. +This command runs the container as root, mapping the docker socket, and mounting the `/tmp` directory. This allows Bazel to spawn other Docker containers and to use directories under `/tmp` to share files with those containers. Your source code is available at `/src` inside the container. -The command intentionally starts from a `debian:stretch` base container that -includes binaries incompatible with the `rbe-ubuntu16-04` container used as a -toolchain container. If binaries from the local environment are leaking into the -toolchain container, they will cause build errors. +The command intentionally starts from a `debian:stretch` base container that includes binaries incompatible with the `rbe-ubuntu16-04` container used as a toolchain container. If binaries from the local environment are leaking into the toolchain container, they will cause build errors. ### Step 3: Test the container @@ -232,28 +152,18 @@ bazel version ### Step 4: Run the build -Run the build as shown below. The output user is root so that it corresponds to -a directory that is accessible with the same absolute path from inside the host -container in which Bazel runs, from the toolchain containers spawned by the Docker -sandbox feature in which Bazel's build actions are running, and from the local -machine on which the host and action containers run. +Run the build as shown below. The output user is root so that it corresponds to a directory that is accessible with the same absolute path from inside the host container in which Bazel runs, from the toolchain containers spawned by the Docker sandbox feature in which Bazel's build actions are running, and from the local machine on which the host and action containers run. ```posix-terminal -bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox {{ '' }}target{{ '' }} +bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <var>target</var> ``` ### Step 5: Resolve detected issues You can resolve build failures as follows: -* If the build fails with an "out of disk space" error, you can increase this - limit by starting the host container with the flag `--memory=XX` where `XX` - is the allocated disk space in gigabytes. This is experimental and may - result in unpredictable behavior. +- If the build fails with an "out of disk space" error, you can increase this limit by starting the host container with the flag `--memory=XX` where `XX` is the allocated disk space in gigabytes. This is experimental and may result in unpredictable behavior. -* If the build fails during the analysis or loading phases, one or more of - your build rules declared in the WORKSPACE file are not compatible with - remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) - for possible causes and workarounds. +- If the build fails during the analysis or loading phases, one or more of your build rules declared in the WORKSPACE file are not compatible with remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) for possible causes and workarounds. -* If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). +- If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). diff --git a/remote/workspace.mdx b/remote/workspace.mdx index ae0aea50..726f8bd3 100644 --- a/remote/workspace.mdx +++ b/remote/workspace.mdx @@ -2,126 +2,78 @@ title: 'Finding Non-Hermetic Behavior in WORKSPACE Rules' --- - - In the following, a host machine is the machine where Bazel runs. -When using remote execution, the actual build and/or test steps are not -happening on the host machine, but are instead sent off to the remote execution -system. However, the steps involved in resolving workspace rules are happening -on the host machine. If your workspace rules access information about the -host machine for use during execution, your build is likely to break due to -incompatibilities between the environments. - -As part of [adapting Bazel rules for remote -execution](/remote/rules), you need to find such workspace rules -and fix them. This page describes how to find potentially problematic workspace -rules using the workspace log. +When using remote execution, the actual build and/or test steps are not happening on the host machine, but are instead sent off to the remote execution system. However, the steps involved in resolving workspace rules are happening on the host machine. If your workspace rules access information about the host machine for use during execution, your build is likely to break due to incompatibilities between the environments. +As part of [adapting Bazel rules for remote execution](/remote/rules), you need to find such workspace rules and fix them. This page describes how to find potentially problematic workspace rules using the workspace log. ## Finding non-hermetic rules -[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to -external workspaces, but they are rich enough to allow arbitrary processing to -happen in the process. All related commands are happening locally and can be a -potential source of non-hermeticity. Usually non-hermetic behavior is -introduced through -[`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting -with the host machine. +[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. All related commands are happening locally and can be a potential source of non-hermeticity. Usually non-hermetic behavior is introduced through [`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting with the host machine. -Starting with Bazel 0.18, you can get a log of some potentially non-hermetic -actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to -your Bazel command. Here `[PATH]` is a filename under which the log will be -created. +Starting with Bazel 0.18, you can get a log of some potentially non-hermetic actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to your Bazel command. Here `[PATH]` is a filename under which the log will be created. Things to note: -* the log captures the events as they are executed. If some steps are - cached, they will not show up in the log, so to get a full result, don't - forget to run `bazel clean --expunge` beforehand. +- the log captures the events as they are executed. If some steps are cached, they will not show up in the log, so to get a full result, don't forget to run `bazel clean --expunge` beforehand. -* Sometimes functions might be re-executed, in which case the related - events will show up in the log multiple times. +- Sometimes functions might be re-executed, in which case the related events will show up in the log multiple times. -* Workspace rules currently only log Starlark events. +- Workspace rules currently only log Starlark events. - Note: These particular rules do not cause hermiticity concerns as long - as a hash is specified. + Note: These particular rules do not cause hermiticity concerns as long as a hash is specified. To find what was executed during workspace initialization: -1. Run `bazel clean --expunge`. This command will clean your local cache and - any cached repositories, ensuring that all initialization will be re-run. +1. Run `bazel clean --expunge`. This command will clean your local cache and any cached repositories, ensuring that all initialization will be re-run. -2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your - Bazel command and run the build. +2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your Bazel command and run the build. - This produces a binary proto file listing messages of type - [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) + This produces a binary proto file listing messages of type [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -3. Download the Bazel source code and navigate to the Bazel folder by using - the command below. You need the source code to be able to parse the - workspace log with the - [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). +3. Download the Bazel source code and navigate to the Bazel folder by using the command below. You need the source code to be able to parse the workspace log with the [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). - ```posix-terminal - git clone https://github.com/bazelbuild/bazel.git + ```posix-terminal + git clone https://github.com/bazelbuild/bazel.git - cd bazel - ``` + cd bazel + ``` -4. In the Bazel source code repo, convert the whole workspace log to text. +4. In the Bazel source code repo, convert the whole workspace log to text. - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt + ``` -5. The output may be quite verbose and include output from built in Bazel - rules. +5. The output may be quite verbose and include output from built in Bazel rules. - To exclude specific rules from the output, use `--exclude_rule` option. - For example: + To exclude specific rules from the output, use `--exclude_rule` option. For example: - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ - --exclude_rule "//external:local_config_cc" \ - --exclude_rule "//external:dep" > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ + --exclude_rule "//external:local_config_cc" \ + --exclude_rule "//external:dep" > /tmp/workspacelog.txt + ``` -5. Open `/tmp/workspacelog.txt` and check for unsafe operations. +6. Open `/tmp/workspacelog.txt` and check for unsafe operations. -The log consists of -[WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -messages outlining certain potentially non-hermetic actions performed on a -[`repository_ctx`](/rules/lib/builtins/repository_ctx). +The log consists of [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) messages outlining certain potentially non-hermetic actions performed on a [`repository_ctx`](/rules/lib/builtins/repository_ctx). The actions that have been highlighted as potentially non-hermetic are as follows: -* `execute`: executes an arbitrary command on the host environment. Check if - these may introduce any dependencies on the host environment. +- `execute`: executes an arbitrary command on the host environment. Check if these may introduce any dependencies on the host environment. -* `download`, `download_and_extract`: to ensure hermetic builds, make sure - that sha256 is specified +- `download`, `download_and_extract`: to ensure hermetic builds, make sure that sha256 is specified -* `file`, `template`: this is not non-hermetic in itself, but may be a mechanism - for introducing dependencies on the host environment into the repository. - Ensure that you understand where the input comes from, and that it does not - depend on the host environment. +- `file`, `template`: this is not non-hermetic in itself, but may be a mechanism for introducing dependencies on the host environment into the repository. Ensure that you understand where the input comes from, and that it does not depend on the host environment. -* `os`: this is not non-hermetic in itself, but an easy way to get dependencies - on the host environment. A hermetic build would generally not call this. - In evaluating whether your usage is hermetic, keep in mind that this is - running on the host and not on the workers. Getting environment specifics - from the host is generally not a good idea for remote builds. +- `os`: this is not non-hermetic in itself, but an easy way to get dependencies on the host environment. A hermetic build would generally not call this. In evaluating whether your usage is hermetic, keep in mind that this is running on the host and not on the workers. Getting environment specifics from the host is generally not a good idea for remote builds. -* `symlink`: this is normally safe, but look for red flags. Any symlinks to - outside the repository or to an absolute path would cause problems on the - remote worker. If the symlink is created based on host machine properties - it would probably be problematic as well. +- `symlink`: this is normally safe, but look for red flags. Any symlinks to outside the repository or to an absolute path would cause problems on the remote worker. If the symlink is created based on host machine properties it would probably be problematic as well. -* `which`: checking for programs installed on the host is usually problematic - since the workers may have different configurations. +- `which`: checking for programs installed on the host is usually problematic since the workers may have different configurations. diff --git a/rules/bzl-style.mdx b/rules/bzl-style.mdx index 65f589f4..8d85a7b5 100644 --- a/rules/bzl-style.mdx +++ b/rules/bzl-style.mdx @@ -2,92 +2,49 @@ title: '.bzl style guide' --- +This page covers basic style guidelines for Starlark and also includes information on macros and rules. +[Starlark](/rules/language) is a language that defines how software is built, and as such it is both a programming and a configuration language. -This page covers basic style guidelines for Starlark and also includes -information on macros and rules. +You will use Starlark to write `BUILD` files, macros, and build rules. Macros and rules are essentially meta-languages - they define how `BUILD` files are written. `BUILD` files are intended to be simple and repetitive. -[Starlark](/rules/language) is a -language that defines how software is built, and as such it is both a -programming and a configuration language. +All software is read more often than it is written. This is especially true for Starlark, as engineers read `BUILD` files to understand dependencies of their targets and details of their builds. This reading will often happen in passing, in a hurry, or in parallel to accomplishing some other task. Consequently, simplicity and readability are very important so that users can parse and comprehend `BUILD` files quickly. -You will use Starlark to write `BUILD` files, macros, and build rules. Macros and -rules are essentially meta-languages - they define how `BUILD` files are written. -`BUILD` files are intended to be simple and repetitive. +When a user opens a `BUILD` file, they quickly want to know the list of targets in the file; or review the list of sources of that C++ library; or remove a dependency from that Java binary. Each time you add a layer of abstraction, you make it harder for a user to do these tasks. -All software is read more often than it is written. This is especially true for -Starlark, as engineers read `BUILD` files to understand dependencies of their -targets and details of their builds. This reading will often happen in passing, -in a hurry, or in parallel to accomplishing some other task. Consequently, -simplicity and readability are very important so that users can parse and -comprehend `BUILD` files quickly. +`BUILD` files are also analyzed and updated by many different tools. Tools may not be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` files simple will allow you to get better tooling. As a code base grows, it becomes more and more frequent to do changes across many `BUILD` files in order to update a library or do a cleanup. -When a user opens a `BUILD` file, they quickly want to know the list of targets in -the file; or review the list of sources of that C++ library; or remove a -dependency from that Java binary. Each time you add a layer of abstraction, you -make it harder for a user to do these tasks. - -`BUILD` files are also analyzed and updated by many different tools. Tools may not -be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` -files simple will allow you to get better tooling. As a code base grows, it -becomes more and more frequent to do changes across many `BUILD` files in order to -update a library or do a cleanup. - -Important: Do not create a variable or macro just to avoid some amount of -repetition in `BUILD` files. Your `BUILD` file should be easily readable both by -developers and tools. The -[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't -really apply here. +Important: Do not create a variable or macro just to avoid some amount of repetition in `BUILD` files. Your `BUILD` file should be easily readable both by developers and tools. The [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't really apply here. ## General advice -* Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) - as a formatter and linter. -* Follow [testing guidelines](/rules/testing). +- Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) as a formatter and linter. +- Follow [testing guidelines](/rules/testing). ## Style ### Python style -When in doubt, follow the -[PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. -In particular, use four rather than two spaces for indentation to follow the -Python convention. - -Since -[Starlark is not Python](/rules/language#differences-with-python), -some aspects of Python style do not apply. For example, PEP 8 advises that -comparisons to singletons be done with `is`, which is not an operator in -Starlark. +When in doubt, follow the [PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. In particular, use four rather than two spaces for indentation to follow the Python convention. +Since [Starlark is not Python](/rules/language#differences-with-python), some aspects of Python style do not apply. For example, PEP 8 advises that comparisons to singletons be done with `is`, which is not an operator in Starlark. ### Docstring -Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). -Use a docstring at the top of each `.bzl` file, and a docstring for each public -function. +Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Use a docstring at the top of each `.bzl` file, and a docstring for each public function. ### Document rules and aspects -Rules and aspects, along with their attributes, as well as providers and their -fields, should be documented using the `doc` argument. +Rules and aspects, along with their attributes, as well as providers and their fields, should be documented using the `doc` argument. ### Naming convention -* Variables and function names use lowercase with words separated by - underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. -* Top-level private values start with one underscore. Bazel enforces that - private values cannot be used from other files. Local variables should not - use the underscore prefix. +- Variables and function names use lowercase with words separated by underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. +- Top-level private values start with one underscore. Bazel enforces that private values cannot be used from other files. Local variables should not use the underscore prefix. ### Line length -As in `BUILD` files, there is no strict line length limit as labels can be long. -When possible, try to use at most 79 characters per line (following Python's -style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline -should not be enforced strictly: editors should display more than 80 columns, -automated changes will frequently introduce longer lines, and humans shouldn't -spend time splitting lines that are already readable. +As in `BUILD` files, there is no strict line length limit as labels can be long. When possible, try to use at most 79 characters per line (following Python's style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline should not be enforced strictly: editors should display more than 80 columns, automated changes will frequently introduce longer lines, and humans shouldn't spend time splitting lines that are already readable. ### Keyword arguments @@ -105,67 +62,41 @@ def fct(name, srcs): ### Boolean values -Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values -(such as when using a boolean attribute in a rule). +Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values (such as when using a boolean attribute in a rule). ### Use print only for debugging -Do not use the `print()` function in production code; it is only intended for -debugging, and will spam all direct and indirect users of your `.bzl` file. The -only exception is that you may submit code that uses `print()` if it is disabled -by default and can only be enabled by editing the source -- for example, if all -uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to -`False`. Be mindful of whether these statements are useful enough to justify -their impact on readability. +Do not use the `print()` function in production code; it is only intended for debugging, and will spam all direct and indirect users of your `.bzl` file. The only exception is that you may submit code that uses `print()` if it is disabled by default and can only be enabled by editing the source -- for example, if all uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to `False`. Be mindful of whether these statements are useful enough to justify their impact on readability. ## Macros -A macro is a function which instantiates one or more rules during the loading -phase. In general, use rules whenever possible instead of macros. The build -graph seen by the user is not the same as the one used by Bazel during the -build - macros are expanded *before Bazel does any build graph analysis.* - -Because of this, when something goes wrong, the user will need to understand -your macro's implementation to troubleshoot build problems. Additionally, `bazel -query` results can be hard to interpret because targets shown in the results -come from macro expansion. Finally, aspects are not aware of macros, so tooling -depending on aspects (IDEs and others) might fail. - -A safe use for macros is for defining additional targets intended to be -referenced directly at the Bazel CLI or in BUILD files: In that case, only the -*end users* of those targets need to know about them, and any build problems -introduced by macros are never far from their usage. - -For macros that define generated targets (implementation details of the macro -which are not supposed to be referred to at the CLI or depended on by targets -not instantiated by that macro), follow these best practices: - -* A macro should take a `name` argument and define a target with that name. - That target becomes that macro's *main target*. -* Generated targets, that is all other targets defined by a macro, should: - * Have their names prefixed by ``. For example, using - `name = '%s_bar' % (name)`. - * Have restricted visibility (`//visibility:private`), and - * Have a `manual` tag to avoid expansion in wildcard targets (`:all`, - `...`, `:*`, etc). -* The `name` should only be used to derive names of targets defined by the - macro, and not for anything else. For example, don't use the name to derive - a dependency or input file that is not generated by the macro itself. -* All the targets created in the macro should be coupled in some way to the - main target. -* Conventionally, `name` should be the first argument when defining a macro. -* Keep the parameter names in the macro consistent. If a parameter is passed - as an attribute value to the main target, keep its name the same. If a macro - parameter serves the same purpose as a common rule attribute, such as - `deps`, name as you would the attribute (see below). -* When calling a macro, use only keyword arguments. This is consistent with - rules, and greatly improves readability. - -Engineers often write macros when the Starlark API of relevant rules is -insufficient for their specific use case, regardless of whether the rule is -defined within Bazel in native code, or in Starlark. If you're facing this -problem, ask the rule author if they can extend the API to accomplish your -goals. +A macro is a function which instantiates one or more rules during the loading phase. In general, use rules whenever possible instead of macros. The build graph seen by the user is not the same as the one used by Bazel during the build - macros are expanded *before Bazel does any build graph analysis.* + +Because of this, when something goes wrong, the user will need to understand your macro's implementation to troubleshoot build problems. Additionally, `bazel query` results can be hard to interpret because targets shown in the results come from macro expansion. Finally, aspects are not aware of macros, so tooling depending on aspects (IDEs and others) might fail. + +A safe use for macros is for defining additional targets intended to be referenced directly at the Bazel CLI or in BUILD files: In that case, only the *end users* of those targets need to know about them, and any build problems introduced by macros are never far from their usage. + +For macros that define generated targets (implementation details of the macro which are not supposed to be referred to at the CLI or depended on by targets not instantiated by that macro), follow these best practices: + +- A macro should take a `name` argument and define a target with that name. That target becomes that macro's *main target*. + +- Generated targets, that is all other targets defined by a macro, should: + + - Have their names prefixed by `<name>`. For example, using `name = '%s_bar' % (name)`. + - Have restricted visibility (`//visibility:private`), and + - Have a `manual` tag to avoid expansion in wildcard targets (`:all`, `...`, `:*`, etc). + +- The `name` should only be used to derive names of targets defined by the macro, and not for anything else. For example, don't use the name to derive a dependency or input file that is not generated by the macro itself. + +- All the targets created in the macro should be coupled in some way to the main target. + +- Conventionally, `name` should be the first argument when defining a macro. + +- Keep the parameter names in the macro consistent. If a parameter is passed as an attribute value to the main target, keep its name the same. If a macro parameter serves the same purpose as a common rule attribute, such as `deps`, name as you would the attribute (see below). + +- When calling a macro, use only keyword arguments. This is consistent with rules, and greatly improves readability. + +Engineers often write macros when the Starlark API of relevant rules is insufficient for their specific use case, regardless of whether the rule is defined within Bazel in native code, or in Starlark. If you're facing this problem, ask the rule author if they can extend the API to accomplish your goals. As a rule of thumb, the more macros resemble the rules, the better. @@ -173,40 +104,28 @@ See also [macros](/extending/macros#conventions). ## Rules -* Rules, aspects, and their attributes should use lower_case names ("snake - case"). -* Rule names are nouns that describe the main kind of artifact produced by the - rule, from the point of view of its dependencies (or for leaf rules, the - user). This is not necessarily a file suffix. For instance, a rule that - produces C++ artifacts meant to be used as Python extensions might be called - `py_extension`. For most languages, typical rules include: - * `*_library` - a compilation unit or "module". - * `*_binary` - a target producing an executable or a deployment unit. - * `*_test` - a test target. This can include multiple tests. Expect all - tests in a `*_test` target to be variations on the same theme, for - example, testing a single library. - * `*_import`: a target encapsulating a pre-compiled artifact, such as a - `.jar`, or a `.dll` that is used during compilation. -* Use consistent names and types for attributes. Some generally applicable - attributes include: - * `srcs`: `label_list`, allowing files: source files, typically - human-authored. - * `deps`: `label_list`, typically *not* allowing files: compilation - dependencies. - * `data`: `label_list`, allowing files: data files, such as test data etc. - * `runtime_deps`: `label_list`: runtime dependencies that are not needed - for compilation. -* For any attributes with non-obvious behavior (for example, string templates - with special substitutions, or tools that are invoked with specific - requirements), provide documentation using the `doc` keyword argument to the - attribute's declaration (`attr.label_list()` or similar). -* Rule implementation functions should almost always be private functions - (named with a leading underscore). A common style is to give the - implementation function for `myrule` the name `_myrule_impl`. -* Pass information between your rules using a well-defined - [provider](/extending/rules#providers) interface. Declare and document provider - fields. -* Design your rule with extensibility in mind. Consider that other rules might - want to interact with your rule, access your providers, and reuse the - actions you create. -* Follow [performance guidelines](/rules/performance) in your rules. +- Rules, aspects, and their attributes should use lower\_case names ("snake case"). + +- Rule names are nouns that describe the main kind of artifact produced by the rule, from the point of view of its dependencies (or for leaf rules, the user). This is not necessarily a file suffix. For instance, a rule that produces C++ artifacts meant to be used as Python extensions might be called `py_extension`. For most languages, typical rules include: + + - `*_library` - a compilation unit or "module". + - `*_binary` - a target producing an executable or a deployment unit. + - `*_test` - a test target. This can include multiple tests. Expect all tests in a `*_test` target to be variations on the same theme, for example, testing a single library. + - `*_import`: a target encapsulating a pre-compiled artifact, such as a `.jar`, or a `.dll` that is used during compilation. + +- Use consistent names and types for attributes. Some generally applicable attributes include: + + - `srcs`: `label_list`, allowing files: source files, typically human-authored. + - `deps`: `label_list`, typically *not* allowing files: compilation dependencies. + - `data`: `label_list`, allowing files: data files, such as test data etc. + - `runtime_deps`: `label_list`: runtime dependencies that are not needed for compilation. + +- For any attributes with non-obvious behavior (for example, string templates with special substitutions, or tools that are invoked with specific requirements), provide documentation using the `doc` keyword argument to the attribute's declaration (`attr.label_list()` or similar). + +- Rule implementation functions should almost always be private functions (named with a leading underscore). A common style is to give the implementation function for `myrule` the name `_myrule_impl`. + +- Pass information between your rules using a well-defined [provider](/extending/rules#providers) interface. Declare and document provider fields. + +- Design your rule with extensibility in mind. Consider that other rules might want to interact with your rule, access your providers, and reuse the actions you create. + +- Follow [performance guidelines](/rules/performance) in your rules. diff --git a/rules/challenges.mdx b/rules/challenges.mdx index 10ff7371..e31c5d45 100644 --- a/rules/challenges.mdx +++ b/rules/challenges.mdx @@ -2,222 +2,88 @@ title: 'Challenges of Writing Rules' --- - - -This page gives a high-level overview of the specific issues and challenges -of writing efficient Bazel rules. +This page gives a high-level overview of the specific issues and challenges of writing efficient Bazel rules. ## Summary Requirements -* Assumption: Aim for Correctness, Throughput, Ease of Use & Latency -* Assumption: Large Scale Repositories -* Assumption: BUILD-like Description Language -* Historic: Hard Separation between Loading, Analysis, and Execution is - Outdated, but still affects the API -* Intrinsic: Remote Execution and Caching are Hard -* Intrinsic: Using Change Information for Correct and Fast Incremental Builds - requires Unusual Coding Patterns -* Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard +- Assumption: Aim for Correctness, Throughput, Ease of Use & Latency +- Assumption: Large Scale Repositories +- Assumption: BUILD-like Description Language +- Historic: Hard Separation between Loading, Analysis, and Execution is Outdated, but still affects the API +- Intrinsic: Remote Execution and Caching are Hard +- Intrinsic: Using Change Information for Correct and Fast Incremental Builds requires Unusual Coding Patterns +- Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard ## Assumptions -Here are some assumptions made about the build system, such as need for -correctness, ease of use, throughput, and large scale repositories. The -following sections address these assumptions and offer guidelines to ensure -rules are written in an effective manner. +Here are some assumptions made about the build system, such as need for correctness, ease of use, throughput, and large scale repositories. The following sections address these assumptions and offer guidelines to ensure rules are written in an effective manner. ### Aim for correctness, throughput, ease of use & latency -We assume that the build system needs to be first and foremost correct with -respect to incremental builds. For a given source tree, the output of the -same build should always be the same, regardless of what the output tree looks -like. In the first approximation, this means Bazel needs to know every single -input that goes into a given build step, such that it can rerun that step if any -of the inputs change. There are limits to how correct Bazel can get, as it leaks -some information such as date / time of the build, and ignores certain types of -changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) -helps ensure correctness by preventing reads to undeclared input files. Besides -the intrinsic limits of the system, there are a few known correctness issues, -most of which are related to Fileset or the C++ rules, which are both hard -problems. We have long-term efforts to fix these. - -The second goal of the build system is to have high throughput; we are -permanently pushing the boundaries of what can be done within the current -machine allocation for a remote execution service. If the remote execution -service gets overloaded, nobody can get work done. - -Ease of use comes next. Of multiple correct approaches with the same (or -similar) footprint of the remote execution service, we choose the one that is -easier to use. - -Latency denotes the time it takes from starting a build to getting the intended -result, whether that is a test log from a passing or failing test, or an error -message that a `BUILD` file has a typo. - -Note that these goals often overlap; latency is as much a function of throughput -of the remote execution service as is correctness relevant for ease of use. +We assume that the build system needs to be first and foremost correct with respect to incremental builds. For a given source tree, the output of the same build should always be the same, regardless of what the output tree looks like. In the first approximation, this means Bazel needs to know every single input that goes into a given build step, such that it can rerun that step if any of the inputs change. There are limits to how correct Bazel can get, as it leaks some information such as date / time of the build, and ignores certain types of changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) helps ensure correctness by preventing reads to undeclared input files. Besides the intrinsic limits of the system, there are a few known correctness issues, most of which are related to Fileset or the C++ rules, which are both hard problems. We have long-term efforts to fix these. + +The second goal of the build system is to have high throughput; we are permanently pushing the boundaries of what can be done within the current machine allocation for a remote execution service. If the remote execution service gets overloaded, nobody can get work done. + +Ease of use comes next. Of multiple correct approaches with the same (or similar) footprint of the remote execution service, we choose the one that is easier to use. + +Latency denotes the time it takes from starting a build to getting the intended result, whether that is a test log from a passing or failing test, or an error message that a `BUILD` file has a typo. + +Note that these goals often overlap; latency is as much a function of throughput of the remote execution service as is correctness relevant for ease of use. ### Large scale repositories -The build system needs to operate at the scale of large repositories where large -scale means that it does not fit on a single hard drive, so it is impossible to -do a full checkout on virtually all developer machines. A medium-sized build -will need to read and parse tens of thousands of `BUILD` files, and evaluate -hundreds of thousands of globs. While it is theoretically possible to read all -`BUILD` files on a single machine, we have not yet been able to do so within a -reasonable amount of time and memory. As such, it is critical that `BUILD` files -can be loaded and parsed independently. +The build system needs to operate at the scale of large repositories where large scale means that it does not fit on a single hard drive, so it is impossible to do a full checkout on virtually all developer machines. A medium-sized build will need to read and parse tens of thousands of `BUILD` files, and evaluate hundreds of thousands of globs. While it is theoretically possible to read all `BUILD` files on a single machine, we have not yet been able to do so within a reasonable amount of time and memory. As such, it is critical that `BUILD` files can be loaded and parsed independently. ### BUILD-like description language -In this context, we assume a configuration language that is -roughly similar to `BUILD` files in declaration of library and binary rules -and their interdependencies. `BUILD` files can be read and parsed independently, -and we avoid even looking at source files whenever we can (except for -existence). +In this context, we assume a configuration language that is roughly similar to `BUILD` files in declaration of library and binary rules and their interdependencies. `BUILD` files can be read and parsed independently, and we avoid even looking at source files whenever we can (except for existence). ## Historic -There are differences between Bazel versions that cause challenges and some -of these are outlined in the following sections. +There are differences between Bazel versions that cause challenges and some of these are outlined in the following sections. ### Hard separation between loading, analysis, and execution is outdated but still affects the API -Technically, it is sufficient for a rule to know the input and output files of -an action just before the action is sent to remote execution. However, the -original Bazel code base had a strict separation of loading packages, then -analyzing rules using a configuration (command-line flags, essentially), and -only then running any actions. This distinction is still part of the rules API -today, even though the core of Bazel no longer requires it (more details below). - -That means that the rules API requires a declarative description of the rule -interface (what attributes it has, types of attributes). There are some -exceptions where the API allows custom code to run during the loading phase to -compute implicit names of output files and implicit values of attributes. For -example, a java_library rule named 'foo' implicitly generates an output named -'libfoo.jar', which can be referenced from other rules in the build graph. - -Furthermore, the analysis of a rule cannot read any source files or inspect the -output of an action; instead, it needs to generate a partial directed bipartite -graph of build steps and output file names that is only determined from the rule -itself and its dependencies. +Technically, it is sufficient for a rule to know the input and output files of an action just before the action is sent to remote execution. However, the original Bazel code base had a strict separation of loading packages, then analyzing rules using a configuration (command-line flags, essentially), and only then running any actions. This distinction is still part of the rules API today, even though the core of Bazel no longer requires it (more details below). + +That means that the rules API requires a declarative description of the rule interface (what attributes it has, types of attributes). There are some exceptions where the API allows custom code to run during the loading phase to compute implicit names of output files and implicit values of attributes. For example, a java\_library rule named 'foo' implicitly generates an output named 'libfoo.jar', which can be referenced from other rules in the build graph. + +Furthermore, the analysis of a rule cannot read any source files or inspect the output of an action; instead, it needs to generate a partial directed bipartite graph of build steps and output file names that is only determined from the rule itself and its dependencies. ## Intrinsic -There are some intrinsic properties that make writing rules challenging and -some of the most common ones are described in the following sections. +There are some intrinsic properties that make writing rules challenging and some of the most common ones are described in the following sections. ### Remote execution and caching are hard -Remote execution and caching improve build times in large repositories by -roughly two orders of magnitude compared to running the build on a single -machine. However, the scale at which it needs to perform is staggering: Google's -remote execution service is designed to handle a huge number of requests per -second, and the protocol carefully avoids unnecessary roundtrips as well as -unnecessary work on the service side. +Remote execution and caching improve build times in large repositories by roughly two orders of magnitude compared to running the build on a single machine. However, the scale at which it needs to perform is staggering: Google's remote execution service is designed to handle a huge number of requests per second, and the protocol carefully avoids unnecessary roundtrips as well as unnecessary work on the service side. -At this time, the protocol requires that the build system knows all inputs to a -given action ahead of time; the build system then computes a unique action -fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, -the scheduler replies with the digests of the output files; the files itself are -addressed by digest later on. However, this imposes restrictions on the Bazel -rules, which need to declare all input files ahead of time. +At this time, the protocol requires that the build system knows all inputs to a given action ahead of time; the build system then computes a unique action fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, the scheduler replies with the digests of the output files; the files itself are addressed by digest later on. However, this imposes restrictions on the Bazel rules, which need to declare all input files ahead of time. ### Using change information for correct and fast incremental builds requires unusual coding patterns -Above, we argued that in order to be correct, Bazel needs to know all the input -files that go into a build step in order to detect whether that build step is -still up-to-date. The same is true for package loading and rule analysis, and we -have designed [Skyframe](/reference/skyframe) to handle this -in general. Skyframe is a graph library and evaluation framework that takes a -goal node (such as 'build //foo with these options'), and breaks it down into -its constituent parts, which are then evaluated and combined to yield this -result. As part of this process, Skyframe reads packages, analyzes rules, and -executes actions. - -At each node, Skyframe tracks exactly which nodes any given node used to compute -its own output, all the way from the goal node down to the input files (which -are also Skyframe nodes). Having this graph explicitly represented in memory -allows the build system to identify exactly which nodes are affected by a given -change to an input file (including creation or deletion of an input file), doing -the minimal amount of work to restore the output tree to its intended state. - -As part of this, each node performs a dependency discovery process. Each -node can declare dependencies, and then use the contents of those dependencies -to declare even further dependencies. In principle, this maps well to a -thread-per-node model. However, medium-sized builds contain hundreds of -thousands of Skyframe nodes, which isn't easily possible with current Java -technology (and for historical reasons, we're currently tied to using Java, so -no lightweight threads and no continuations). - -Instead, Bazel uses a fixed-size thread pool. However, that means that if a node -declares a dependency that isn't available yet, we may have to abort that -evaluation and restart it (possibly in another thread), when the dependency is -available. This, in turn, means that nodes should not do this excessively; a -node that declares N dependencies serially can potentially be restarted N times, -costing O(N^2) time. Instead, we aim for up-front bulk declaration of -dependencies, which sometimes requires reorganizing the code, or even splitting -a node into multiple nodes to limit the number of restarts. - -Note that this technology isn't currently available in the rules API; instead, -the rules API is still defined using the legacy concepts of loading, analysis, -and execution phases. However, a fundamental restriction is that all accesses to -other nodes have to go through the framework so that it can track the -corresponding dependencies. Regardless of the language in which the build system -is implemented or in which the rules are written (they don't have to be the -same), rule authors must not use standard libraries or patterns that bypass -Skyframe. For Java, that means avoiding java.io.File as well as any form of -reflection, and any library that does either. Libraries that support dependency -injection of these low-level interfaces still need to be setup correctly for -Skyframe. - -This strongly suggests to avoid exposing rule authors to a full language runtime -in the first place. The danger of accidental use of such APIs is just too big - -several Bazel bugs in the past were caused by rules using unsafe APIs, even -though the rules were written by the Bazel team or other domain experts. +Above, we argued that in order to be correct, Bazel needs to know all the input files that go into a build step in order to detect whether that build step is still up-to-date. The same is true for package loading and rule analysis, and we have designed [Skyframe](/reference/skyframe) to handle this in general. Skyframe is a graph library and evaluation framework that takes a goal node (such as 'build //foo with these options'), and breaks it down into its constituent parts, which are then evaluated and combined to yield this result. As part of this process, Skyframe reads packages, analyzes rules, and executes actions. + +At each node, Skyframe tracks exactly which nodes any given node used to compute its own output, all the way from the goal node down to the input files (which are also Skyframe nodes). Having this graph explicitly represented in memory allows the build system to identify exactly which nodes are affected by a given change to an input file (including creation or deletion of an input file), doing the minimal amount of work to restore the output tree to its intended state. + +As part of this, each node performs a dependency discovery process. Each node can declare dependencies, and then use the contents of those dependencies to declare even further dependencies. In principle, this maps well to a thread-per-node model. However, medium-sized builds contain hundreds of thousands of Skyframe nodes, which isn't easily possible with current Java technology (and for historical reasons, we're currently tied to using Java, so no lightweight threads and no continuations). + +Instead, Bazel uses a fixed-size thread pool. However, that means that if a node declares a dependency that isn't available yet, we may have to abort that evaluation and restart it (possibly in another thread), when the dependency is available. This, in turn, means that nodes should not do this excessively; a node that declares N dependencies serially can potentially be restarted N times, costing O(N^2) time. Instead, we aim for up-front bulk declaration of dependencies, which sometimes requires reorganizing the code, or even splitting a node into multiple nodes to limit the number of restarts. + +Note that this technology isn't currently available in the rules API; instead, the rules API is still defined using the legacy concepts of loading, analysis, and execution phases. However, a fundamental restriction is that all accesses to other nodes have to go through the framework so that it can track the corresponding dependencies. Regardless of the language in which the build system is implemented or in which the rules are written (they don't have to be the same), rule authors must not use standard libraries or patterns that bypass Skyframe. For Java, that means avoiding java.io.File as well as any form of reflection, and any library that does either. Libraries that support dependency injection of these low-level interfaces still need to be setup correctly for Skyframe. + +This strongly suggests to avoid exposing rule authors to a full language runtime in the first place. The danger of accidental use of such APIs is just too big - several Bazel bugs in the past were caused by rules using unsafe APIs, even though the rules were written by the Bazel team or other domain experts. ### Avoiding quadratic time and memory consumption is hard -To make matters worse, apart from the requirements imposed by Skyframe, the -historical constraints of using Java, and the outdatedness of the rules API, -accidentally introducing quadratic time or memory consumption is a fundamental -problem in any build system based on library and binary rules. There are two -very common patterns that introduce quadratic memory consumption (and therefore -quadratic time consumption). - -1. Chains of Library Rules - -Consider the case of a chain of library rules A depends on B, depends on C, and -so on. Then, we want to compute some property over the transitive closure of -these rules, such as the Java runtime classpath, or the C++ linker command for -each library. Naively, we might take a standard list implementation; however, -this already introduces quadratic memory consumption: the first library -contains one entry on the classpath, the second two, the third three, and so -on, for a total of 1+2+3+...+N = O(N^2) entries. - -2. Binary Rules Depending on the Same Library Rules - -Consider the case where a set of binaries that depend on the same library -rules — such as if you have a number of test rules that test the same -library code. Let's say out of N rules, half the rules are binary rules, and -the other half library rules. Now consider that each binary makes a copy of -some property computed over the transitive closure of library rules, such as -the Java runtime classpath, or the C++ linker command line. For example, it -could expand the command line string representation of the C++ link action. N/2 -copies of N/2 elements is O(N^2) memory. +To make matters worse, apart from the requirements imposed by Skyframe, the historical constraints of using Java, and the outdatedness of the rules API, accidentally introducing quadratic time or memory consumption is a fundamental problem in any build system based on library and binary rules. There are two very common patterns that introduce quadratic memory consumption (and therefore quadratic time consumption). + +1. Chains of Library Rules - Consider the case of a chain of library rules A depends on B, depends on C, and so on. Then, we want to compute some property over the transitive closure of these rules, such as the Java runtime classpath, or the C++ linker command for each library. Naively, we might take a standard list implementation; however, this already introduces quadratic memory consumption: the first library contains one entry on the classpath, the second two, the third three, and so on, for a total of 1+2+3+...+N = O(N^2) entries. + +2. Binary Rules Depending on the Same Library Rules - Consider the case where a set of binaries that depend on the same library rules — such as if you have a number of test rules that test the same library code. Let's say out of N rules, half the rules are binary rules, and the other half library rules. Now consider that each binary makes a copy of some property computed over the transitive closure of library rules, such as the Java runtime classpath, or the C++ linker command line. For example, it could expand the command line string representation of the C++ link action. N/2 copies of N/2 elements is O(N^2) memory. #### Custom collections classes to avoid quadratic complexity -Bazel is heavily affected by both of these scenarios, so we introduced a set of -custom collection classes that effectively compress the information in memory by -avoiding the copy at each step. Almost all of these data structures have set -semantics, so we called it -[depset](/rules/lib/depset) -(also known as `NestedSet` in the internal implementation). The majority of -changes to reduce Bazel's memory consumption over the past several years were -changes to use depsets instead of whatever was previously used. - -Unfortunately, usage of depsets does not automatically solve all the issues; -in particular, even just iterating over a depset in each rule re-introduces -quadratic time consumption. Internally, NestedSets also has some helper methods -to facilitate interoperability with normal collections classes; unfortunately, -accidentally passing a NestedSet to one of these methods leads to copying -behavior, and reintroduces quadratic memory consumption. +Bazel is heavily affected by both of these scenarios, so we introduced a set of custom collection classes that effectively compress the information in memory by avoiding the copy at each step. Almost all of these data structures have set semantics, so we called it [depset](/rules/lib/depset) (also known as `NestedSet` in the internal implementation). The majority of changes to reduce Bazel's memory consumption over the past several years were changes to use depsets instead of whatever was previously used. + +Unfortunately, usage of depsets does not automatically solve all the issues; in particular, even just iterating over a depset in each rule re-introduces quadratic time consumption. Internally, NestedSets also has some helper methods to facilitate interoperability with normal collections classes; unfortunately, accidentally passing a NestedSet to one of these methods leads to copying behavior, and reintroduces quadratic memory consumption. diff --git a/rules/deploying.mdx b/rules/deploying.mdx index 3fe2c869..4395dd28 100644 --- a/rules/deploying.mdx +++ b/rules/deploying.mdx @@ -2,48 +2,30 @@ title: 'Deploying Rules' --- +This page is for rule writers who are planning to make their rules available to others. - -This page is for rule writers who are planning to make their rules available -to others. - -We recommend you start a new ruleset from the template repository: -https://github.com/bazel-contrib/rules-template -That template follows the recommendations below, and includes API documentation generation -and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. +We recommend you start a new ruleset from the template repository: [https://github.com/bazel-contrib/rules-template](https://github.com/bazel-contrib/rules-template) That template follows the recommendations below, and includes API documentation generation and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. ## Hosting and naming rules -New rules should go into their own GitHub repository under your organization. -Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) -if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) -organization. +New rules should go into their own GitHub repository under your organization. Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) organization. -Repository names for Bazel rules are standardized on the following format: -`$ORGANIZATION/rules_$NAME`. -See [examples on GitHub](https://github.com/search?q=rules+bazel&type=Repositories). -For consistency, you should follow this same format when publishing your Bazel rules. +Repository names for Bazel rules are standardized on the following format: `$ORGANIZATION/rules_$NAME`. See [examples on GitHub](https://github.com/search?q=rules+bazel\&type=Repositories). For consistency, you should follow this same format when publishing your Bazel rules. -Make sure to use a descriptive GitHub repository description and `README.md` -title, example: +Make sure to use a descriptive GitHub repository description and `README.md` title, example: -* Repository name: `bazelbuild/rules_go` -* Repository description: *Go rules for Bazel* -* Repository tags: `golang`, `bazel` -* `README.md` header: *Go rules for [Bazel](https://bazel.build)* -(note the link to https://bazel.build which will guide users who are unfamiliar -with Bazel to the right place) +- Repository name: `bazelbuild/rules_go` +- Repository description: *Go rules for Bazel* +- Repository tags: `golang`, `bazel` +- `README.md` header: *Go rules for [Bazel](https://bazel.build)* (note the link to [https://bazel.build](https://bazel.build) which will guide users who are unfamiliar with Bazel to the right place) -Rules can be grouped either by language (such as Scala), runtime platform -(such as Android), or framework (such as Spring). +Rules can be grouped either by language (such as Scala), runtime platform (such as Android), or framework (such as Spring). ## Repository content -Every rule repository should have a certain layout so that users can quickly -understand new rules. +Every rule repository should have a certain layout so that users can quickly understand new rules. -For example, when writing new rules for the (make-believe) -`mockascript` language, the rule repository would have the following structure: +For example, when writing new rules for the (make-believe) `mockascript` language, the rule repository would have the following structure: ``` / @@ -71,17 +53,9 @@ For example, when writing new rules for the (make-believe) ### MODULE.bazel -In the project's `MODULE.bazel`, you should define the name that users will use -to reference your rules. If your rules belong to the -[bazelbuild](https://github.com/bazelbuild) organization, you must use -`rules_` (such as `rules_mockascript`). Otherwise, you should name your -repository `_rules_` (such as `build_stack_rules_proto`). Please -start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) -if you feel like your rules should follow the convention for rules in the -[bazelbuild](https://github.com/bazelbuild) organization. +In the project's `MODULE.bazel`, you should define the name that users will use to reference your rules. If your rules belong to the [bazelbuild](https://github.com/bazelbuild) organization, you must use `rules_<lang>` (such as `rules_mockascript`). Otherwise, you should name your repository `<org>_rules_<lang>` (such as `build_stack_rules_proto`). Please start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules should follow the convention for rules in the [bazelbuild](https://github.com/bazelbuild) organization. -In the following sections, assume the repository belongs to the -[bazelbuild](https://github.com/bazelbuild) organization. +In the following sections, assume the repository belongs to the [bazelbuild](https://github.com/bazelbuild) organization. ``` module(name = "rules_mockascript") @@ -89,16 +63,11 @@ module(name = "rules_mockascript") ### README -At the top level, there should be a `README` that contains a brief description -of your ruleset, and the API users should expect. +At the top level, there should be a `README` that contains a brief description of your ruleset, and the API users should expect. ### Rules -Often times there will be multiple rules provided by your repository. Create a -directory named by the language and provide an entry point - `defs.bzl` file -exporting all rules (also include a `BUILD` file so the directory is a package). -For `rules_mockascript` that means there will be a directory named -`mockascript`, and a `BUILD` file and a `defs.bzl` file inside: +Often times there will be multiple rules provided by your repository. Create a directory named by the language and provide an entry point - `defs.bzl` file exporting all rules (also include a `BUILD` file so the directory is a package). For `rules_mockascript` that means there will be a directory named `mockascript`, and a `BUILD` file and a `defs.bzl` file inside: ``` / @@ -109,11 +78,7 @@ For `rules_mockascript` that means there will be a directory named ### Constraints -If your rule defines -[toolchain](/extending/toolchains) rules, -it's possible that you'll need to define custom `constraint_setting`s and/or -`constraint_value`s. Put these into a `///constraints` package. Your -directory structure will look like this: +If your rule defines [toolchain](/extending/toolchains) rules, it's possible that you'll need to define custom `constraint_setting`s and/or `constraint_value`s. Put these into a `//<LANG>/constraints` package. Your directory structure will look like this: ``` / @@ -124,100 +89,58 @@ directory structure will look like this: defs.bzl ``` -Please read -[github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) -for best practices, and to see what constraints are already present, and -consider contributing your constraints there if they are language independent. -Be mindful of introducing custom constraints, all users of your rules will -use them to perform platform specific logic in their `BUILD` files (for example, -using [selects](/reference/be/functions#select)). -With custom constraints, you define a language that the whole Bazel ecosystem -will speak. +Please read [github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) for best practices, and to see what constraints are already present, and consider contributing your constraints there if they are language independent. Be mindful of introducing custom constraints, all users of your rules will use them to perform platform specific logic in their `BUILD` files (for example, using [selects](/reference/be/functions#select)). With custom constraints, you define a language that the whole Bazel ecosystem will speak. ### Runfiles library -If your rule provides a standard library for accessing runfiles, it should be -in the form of a library target located at `///runfiles` (an abbreviation -of `///runfiles:runfiles`). User targets that need to access their data -dependencies will typically add this target to their `deps` attribute. +If your rule provides a standard library for accessing runfiles, it should be in the form of a library target located at `//<LANG>/runfiles` (an abbreviation of `//<LANG>/runfiles:runfiles`). User targets that need to access their data dependencies will typically add this target to their `deps` attribute. ### Repository rules #### Dependencies -Your rules might have external dependencies, which you'll need to specify in -your MODULE.bazel file. +Your rules might have external dependencies, which you'll need to specify in your MODULE.bazel file. #### Registering toolchains -Your rules might also register toolchains, which you can also specify in the -MODULE.bazel file. - -Note that in order to resolve toolchains in the analysis phase Bazel needs to -analyze all `toolchain` targets that are registered. Bazel will not need to -analyze all targets referenced by `toolchain.toolchain` attribute. If in order -to register toolchains you need to perform complex computation in the -repository, consider splitting the repository with `toolchain` targets from the -repository with `_toolchain` targets. Former will be always fetched, and -the latter will only be fetched when user actually needs to build `` code. +Your rules might also register toolchains, which you can also specify in the MODULE.bazel file. +Note that in order to resolve toolchains in the analysis phase Bazel needs to analyze all `toolchain` targets that are registered. Bazel will not need to analyze all targets referenced by `toolchain.toolchain` attribute. If in order to register toolchains you need to perform complex computation in the repository, consider splitting the repository with `toolchain` targets from the repository with `<LANG>_toolchain` targets. Former will be always fetched, and the latter will only be fetched when user actually needs to build `<LANG>` code. #### Release snippet -In your release announcement provide a snippet that your users can copy-paste -into their `MODULE.bazel` file. This snippet in general will look as follows: +In your release announcement provide a snippet that your users can copy-paste into their `MODULE.bazel` file. This snippet in general will look as follows: ``` -bazel_dep(name = "rules_", version = "") +bazel_dep(name = "rules_<LANG>", version = "<VERSION>") ``` - ### Tests -There should be tests that verify that the rules are working as expected. This -can either be in the standard location for the language the rules are for or a -`tests/` directory at the top level. +There should be tests that verify that the rules are working as expected. This can either be in the standard location for the language the rules are for or a `tests/` directory at the top level. ### Examples (optional) -It is useful to users to have an `examples/` directory that shows users a couple -of basic ways that the rules can be used. +It is useful to users to have an `examples/` directory that shows users a couple of basic ways that the rules can be used. ## CI/CD -Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib -org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. -See comments in the rules-template repo for more information. +Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. See comments in the rules-template repo for more information. -If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), -you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md&title=Request+to+add+new+project+%5BPROJECT_NAME%5D&labels=new-project) -it to [ci.bazel.build](http://ci.bazel.build). +If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md\&title=Request+to+add+new+project+%5BPROJECT_NAME%5D\&labels=new-project) it to [ci.bazel.build](http://ci.bazel.build). ## Documentation -See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for -instructions on how to comment your rules so that documentation can be generated -automatically. +See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for instructions on how to comment your rules so that documentation can be generated automatically. -The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) -shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date -as Starlark files are updated. +The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date as Starlark files are updated. ## FAQs ### Why can't we add our rule to the main Bazel GitHub repository? -We want to decouple rules from Bazel releases as much as possible. It's clearer -who owns individual rules, reducing the load on Bazel developers. For our users, -decoupling makes it easier to modify, upgrade, downgrade, and replace rules. -Contributing to rules can be lighter weight than contributing to Bazel - -depending on the rules -, including full submit access to the corresponding -GitHub repository. Getting submit access to Bazel itself is a much more involved -process. +We want to decouple rules from Bazel releases as much as possible. It's clearer who owns individual rules, reducing the load on Bazel developers. For our users, decoupling makes it easier to modify, upgrade, downgrade, and replace rules. Contributing to rules can be lighter weight than contributing to Bazel - depending on the rules -, including full submit access to the corresponding GitHub repository. Getting submit access to Bazel itself is a much more involved process. -The downside is a more complicated one-time installation process for our users: -they have to add a dependency on your ruleset in their `MODULE.bazel` file. +The downside is a more complicated one-time installation process for our users: they have to add a dependency on your ruleset in their `MODULE.bazel` file. -We used to have all of the rules in the Bazel repository (under -`//tools/build_rules` or `//tools/build_defs`). We still have a couple rules -there, but we are working on moving the remaining rules out. +We used to have all of the rules in the Bazel repository (under `//tools/build_rules` or `//tools/build_defs`). We still have a couple rules there, but we are working on moving the remaining rules out. diff --git a/rules/errors/read-only-variable.mdx b/rules/errors/read-only-variable.mdx index 2bfde654..ab4e4619 100644 --- a/rules/errors/read-only-variable.mdx +++ b/rules/errors/read-only-variable.mdx @@ -2,11 +2,7 @@ title: 'Error: Variable x is read only' --- - - -A global variable cannot be reassigned. It will always point to the same object. -However, its content might change, if the value is mutable (for example, the -content of a list). Local variables don't have this restriction. +A global variable cannot be reassigned. It will always point to the same object. However, its content might change, if the value is mutable (for example, the content of a list). Local variables don't have this restriction. ```python a = [1, 2] @@ -20,8 +16,7 @@ b = 4 # forbidden `ERROR: /path/ext.bzl:7:1: Variable b is read only` -You will get a similar error if you try to redefine a function (function -overloading is not supported), for example: +You will get a similar error if you try to redefine a function (function overloading is not supported), for example: ```python def foo(x): return x + 1 diff --git a/rules/faq.mdx b/rules/faq.mdx index 5321f0b7..89481fa7 100644 --- a/rules/faq.mdx +++ b/rules/faq.mdx @@ -2,79 +2,48 @@ title: 'Frequently Asked Questions' --- - - These are some common issues and questions with writing extensions. ## Why is my file not produced / my action never executed? Bazel only executes the actions needed to produce the *requested* output files. -* If the file you want has a label, you can request it directly: - `bazel build //pkg:myfile.txt` +- If the file you want has a label, you can request it directly: `bazel build //pkg:myfile.txt` -* If the file is in an output group of the target, you may need to specify that - output group on the command line: - `bazel build //pkg:mytarget --output_groups=foo` +- If the file is in an output group of the target, you may need to specify that output group on the command line: `bazel build //pkg:mytarget --output_groups=foo` -* If you want the file to be built automatically whenever your target is - mentioned on the command line, add it to your rule's default outputs by - returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. +- If you want the file to be built automatically whenever your target is mentioned on the command line, add it to your rule's default outputs by returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. See the [Rules page](/extending/rules#requesting-output-files) for more information. ## Why is my implementation function not executed? -Bazel analyzes only the targets that are requested for the build. You should -either name the target on the command line, or something that depends on the -target. +Bazel analyzes only the targets that are requested for the build. You should either name the target on the command line, or something that depends on the target. ## A file is missing when my action or binary is executed -Make sure that 1) the file has been registered as an input to the action or -binary, and 2) the script or tool being executed is accessing the file using the -correct path. +Make sure that 1) the file has been registered as an input to the action or binary, and 2) the script or tool being executed is accessing the file using the correct path. -For actions, you declare inputs by passing them to the `ctx.actions.*` function -that creates the action. The proper path for the file can be obtained using -[`File.path`](lib/File#path). +For actions, you declare inputs by passing them to the `ctx.actions.*` function that creates the action. The proper path for the file can be obtained using [`File.path`](lib/File#path). -For binaries (the executable outputs run by a `bazel run` or `bazel test` -command), you declare inputs by including them in the -[runfiles](/extending/rules#runfiles). Instead of using the `path` field, use -[`File.short_path`](lib/File#short_path), which is file's path relative to -the runfiles directory in which the binary executes. +For binaries (the executable outputs run by a `bazel run` or `bazel test` command), you declare inputs by including them in the [runfiles](/extending/rules#runfiles). Instead of using the `path` field, use [`File.short_path`](lib/File#short_path), which is file's path relative to the runfiles directory in which the binary executes. ## How can I control which files are built by `bazel build //pkg:mytarget`? -Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to -[set the default outputs](/extending/rules#requesting-output-files). +Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to [set the default outputs](/extending/rules#requesting-output-files). ## How can I run a program or do file I/O as part of my build? -A tool can be declared as a target, just like any other part of your build, and -run during the execution phase to help build other targets. To create an action -that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the -tool as the `executable` parameter. +A tool can be declared as a target, just like any other part of your build, and run during the execution phase to help build other targets. To create an action that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the tool as the `executable` parameter. -During the loading and analysis phases, a tool *cannot* run, nor can you perform -file I/O. This means that tools and file contents (except the contents of BUILD -and .bzl files) cannot affect how the target and action graphs get created. +During the loading and analysis phases, a tool *cannot* run, nor can you perform file I/O. This means that tools and file contents (except the contents of BUILD and .bzl files) cannot affect how the target and action graphs get created. ## What if I need to access the same structured data both before and during the execution phase? -You can format the structured data as a .bzl file. You can `load()` the file to -access it during the loading and analysis phases. You can pass it as an input or -runfile to actions and executables that need it during the execution phase. +You can format the structured data as a .bzl file. You can `load()` the file to access it during the loading and analysis phases. You can pass it as an input or runfile to actions and executables that need it during the execution phase. ## How should I document Starlark code? -For rules and rule attributes, you can pass a docstring literal (possibly -triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper -functions and macros, use a triple-quoted docstring literal following the format -given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). -Rule implementation functions generally do not need their own docstring. +For rules and rule attributes, you can pass a docstring literal (possibly triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper functions and macros, use a triple-quoted docstring literal following the format given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Rule implementation functions generally do not need their own docstring. -Using string literals in the expected places makes it easier for automated -tooling to extract documentation. Feel free to use standard non-string comments -wherever it may help the reader of your code. +Using string literals in the expected places makes it easier for automated tooling to extract documentation. Feel free to use standard non-string comments wherever it may help the reader of your code. diff --git a/rules/index.mdx b/rules/index.mdx index 2a6c3eb7..44590f11 100644 --- a/rules/index.mdx +++ b/rules/index.mdx @@ -2,11 +2,7 @@ title: 'Rules' --- - - -The Bazel ecosystem has a growing and evolving set of rules to support popular -languages and packages. Much of Bazel's strength comes from the ability to -[define new rules](/extending/concepts) that can be used by others. +The Bazel ecosystem has a growing and evolving set of rules to support popular languages and packages. Much of Bazel's strength comes from the ability to [define new rules](/extending/concepts) that can be used by others. This page describes the recommended, native, and non-native Bazel rules. @@ -14,58 +10,57 @@ This page describes the recommended, native, and non-native Bazel rules. Here is a selection of recommended rules: -* [Android](/docs/bazel-and-android) -* [C / C++](/docs/bazel-and-cpp) -* [Docker/OCI](https://github.com/bazel-contrib/rules_oci) -* [Go](https://github.com/bazelbuild/rules_go) -* [Haskell](https://github.com/tweag/rules_haskell) -* [Java](/docs/bazel-and-java) -* [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) -* [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) -* [Objective-C](/docs/bazel-and-apple) -* [Package building](https://github.com/bazelbuild/rules_pkg) -* [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) -* [Python](https://github.com/bazelbuild/rules_python) -* [Rust](https://github.com/bazelbuild/rules_rust) -* [Scala](https://github.com/bazelbuild/rules_scala) -* [Shell](/reference/be/shell) -* [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) - -The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains -additional functions that can be useful when writing new rules and new -macros. - -The rules above were reviewed and follow our -[requirements for recommended rules](/community/recommended-rules). -Contact the respective rule set's maintainers regarding issues and feature -requests. - -To find more Bazel rules, use a search engine, take a look on -[awesomebazel.com](https://awesomebazel.com/), or search on -[GitHub](https://github.com/search?o=desc&q=bazel+rules&s=stars&type=Repositories). +- [Android](/docs/bazel-and-android) +- [C / C++](/docs/bazel-and-cpp) +- [Docker/OCI](https://github.com/bazel-contrib/rules_oci) +- [Go](https://github.com/bazelbuild/rules_go) +- [Haskell](https://github.com/tweag/rules_haskell) +- [Java](/docs/bazel-and-java) +- [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) +- [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) +- [Objective-C](/docs/bazel-and-apple) +- [Package building](https://github.com/bazelbuild/rules_pkg) +- [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) +- [Python](https://github.com/bazelbuild/rules_python) +- [Rust](https://github.com/bazelbuild/rules_rust) +- [Scala](https://github.com/bazelbuild/rules_scala) +- [Shell](/reference/be/shell) +- [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) + +The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains additional functions that can be useful when writing new rules and new macros. + +The rules above were reviewed and follow our [requirements for recommended rules](/community/recommended-rules). Contact the respective rule set's maintainers regarding issues and feature requests. + +To find more Bazel rules, use a search engine, take a look on [awesomebazel.com](https://awesomebazel.com/), or search on [GitHub](https://github.com/search?o=desc\&q=bazel+rules\&s=stars\&type=Repositories). ## Native rules that do not apply to a specific programming language -Native rules are shipped with the Bazel binary, they are always available in -BUILD files without a `load` statement. +Native rules are shipped with the Bazel binary, they are always available in BUILD files without a `load` statement. + +- Extra actions -* Extra actions - [`extra_action`](/reference/be/extra-actions#extra_action) - [`action_listener`](/reference/be/extra-actions#action_listener) -* General + +- General + - [`filegroup`](/reference/be/general#filegroup) - [`genquery`](/reference/be/general#genquery) - [`test_suite`](/reference/be/general#test_suite) - [`alias`](/reference/be/general#alias) - [`config_setting`](/reference/be/general#config_setting) - [`genrule`](/reference/be/general#genrule) -* Platform + +- Platform + - [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) - [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) - [`platform`](/reference/be/platforms-and-toolchains#platform) - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - [`toolchain_type`](/reference/be/platforms-and-toolchains#toolchain_type) -* Workspace + +- Workspace + - [`bind`](/reference/be/workspace#bind) - [`local_repository`](/reference/be/workspace#local_repository) - [`new_local_repository`](/reference/be/workspace#new_local_repository) @@ -74,10 +69,10 @@ BUILD files without a `load` statement. ## Embedded non-native rules -Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from -the `@bazel_tools` built-in external repository. +Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from the `@bazel_tools` built-in external repository. + +- Repository rules -* Repository rules - [`git_repository`](/rules/lib/repo/git#git_repository) - [`http_archive`](/rules/lib/repo/http#http_archive) - [`http_file`](/rules/lib/repo/http#http_archive) diff --git a/rules/language.mdx b/rules/language.mdx index 13e33a4b..9fad7248 100644 --- a/rules/language.mdx +++ b/rules/language.mdx @@ -2,18 +2,11 @@ title: 'Starlark Language' --- - - -{/* [TOC] */} - -This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), -formerly known as Skylark, the language used in Bazel. For a complete list of -functions and types, see the [Bazel API reference](/rules/lib/overview). +This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), formerly known as Skylark, the language used in Bazel. For a complete list of functions and types, see the [Bazel API reference](/rules/lib/overview). For more information about the language, see [Starlark's GitHub repo](https://github.com/bazelbuild/starlark/). -For the authoritative specification of the Starlark syntax and -behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). +For the authoritative specification of the Starlark syntax and behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). ## Syntax @@ -33,44 +26,32 @@ def fizz_buzz(n): fizz_buzz(20) ``` -Starlark's semantics can differ from Python, but behavioral differences are -rare, except for cases where Starlark raises an error. The following Python -types are supported: +Starlark's semantics can differ from Python, but behavioral differences are rare, except for cases where Starlark raises an error. The following Python types are supported: -* [None](lib/globals#None) -* [bool](lib/bool) -* [dict](lib/dict) -* [tuple](lib/tuple) -* [function](lib/function) -* [int](lib/int) -* [list](lib/list) -* [string](lib/string) +- [None](lib/globals#None) +- [bool](lib/bool) +- [dict](lib/dict) +- [tuple](lib/tuple) +- [function](lib/function) +- [int](lib/int) +- [list](lib/list) +- [string](lib/string) ## Type annotations -**Experimental**. Type annotations are an experimental feature and may change -at any time. Don't depend on it. It may be enabled in Bazel at HEAD -by using the `--experimental_starlark_types` flag. +**Experimental**. Type annotations are an experimental feature and may change at any time. Don't depend on it. It may be enabled in Bazel at HEAD by using the `--experimental_starlark_types` flag. -Starlark in Bazel at HEAD is incrementally adding support for type annotations -with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). +Starlark in Bazel at HEAD is incrementally adding support for type annotations with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). -- Starlark type annotations are under active development. The progress is - tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). +- Starlark type annotations are under active development. The progress is tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). - The specification is incrementally extended: [starlark-with-types/spec.md](https://github.com/bazelbuild/starlark/blob/starlark-with-types/spec.md) - Initial proposal: [SEP-001 Bootstrapping Starlark types](https://docs.google.com/document/d/1Sid7EAbBd_w_T7D94Li_f_bK3zMTztFbzIMvcpzo1wY/edit?tab=t.0#heading=h.5mcn15i0e1ch) ## Mutability -Starlark favors immutability. Two mutable data structures are available: -[lists](lib/list) and [dicts](lib/dict). Changes to mutable -data-structures, such as appending a value to a list or deleting an entry in a -dictionary are valid only for objects created in the current context. After a -context finishes, its values become immutable. +Starlark favors immutability. Two mutable data structures are available: [lists](lib/list) and [dicts](lib/dict). Changes to mutable data-structures, such as appending a value to a list or deleting an entry in a dictionary are valid only for objects created in the current context. After a context finishes, its values become immutable. -This is because Bazel builds use parallel execution. During a build, each `.bzl` -file and each `BUILD` file get their own execution context. Each rule is also -analyzed in its own context. +This is because Bazel builds use parallel execution. During a build, each `.bzl` file and each `BUILD` file get their own execution context. Each rule is also analyzed in its own context. Let's go through an example with the file `foo.bzl`: @@ -84,13 +65,9 @@ def fct(): # declare a function fct() # execute the fct function ``` -Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s -context. When `fct()` runs, it does so within the context of `foo.bzl`. After -evaluation for `foo.bzl` completes, the environment contains an immutable entry, -`var`, with the value `[5]`. +Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s context. When `fct()` runs, it does so within the context of `foo.bzl`. After evaluation for `foo.bzl` completes, the environment contains an immutable entry, `var`, with the value `[5]`. -When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain -immutable. For this reason, the following code in `bar.bzl` is illegal: +When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain immutable. For this reason, the following code in `bar.bzl` is illegal: ```python # `bar.bzl` @@ -101,66 +78,52 @@ var.append(6) # runtime error, the list stored in var is frozen fct() # runtime error, fct() attempts to modify a frozen list ``` -Global variables defined in `bzl` files cannot be changed outside of the -`bzl` file that defined them. Just like the above example using `bzl` files, -values returned by rules are immutable. +Global variables defined in `bzl` files cannot be changed outside of the `bzl` file that defined them. Just like the above example using `bzl` files, values returned by rules are immutable. ## Differences between BUILD and .bzl files -`BUILD` files register targets via making calls to rules. `.bzl` files provide -definitions for constants, rules, macros, and functions. +`BUILD` files register targets via making calls to rules. `.bzl` files provide definitions for constants, rules, macros, and functions. -[Native functions](/reference/be/functions) and [native rules]( -/reference/be/overview#language-specific-native-rules) are global symbols in -`BUILD` files. `bzl` files need to load them using the [`native` module]( -/rules/lib/toplevel/native). +[Native functions](/reference/be/functions) and [native rules](/reference/be/overview#language-specific-native-rules) are global symbols in `BUILD` files. `bzl` files need to load them using the [`native` module](/rules/lib/toplevel/native). -There are two syntactic restrictions in `BUILD` files: 1) declaring functions is -illegal, and 2) `*args` and `**kwargs` arguments are not allowed. +There are two syntactic restrictions in `BUILD` files: 1) declaring functions is illegal, and 2) `*args` and `**kwargs` arguments are not allowed. ## Differences with Python -* Global variables are immutable. +- Global variables are immutable. -* `for` statements are not allowed at the top-level. Use them within functions - instead. In `BUILD` files, you may use list comprehensions. +- `for` statements are not allowed at the top-level. Use them within functions instead. In `BUILD` files, you may use list comprehensions. -* `if` statements are not allowed at the top-level. However, `if` expressions - can be used: `first = data[0] if len(data) > 0 else None`. +- `if` statements are not allowed at the top-level. However, `if` expressions can be used: `first = data[0] if len(data) > 0 else None`. -* Deterministic order for iterating through Dictionaries. +- Deterministic order for iterating through Dictionaries. -* Recursion is not allowed. +- Recursion is not allowed. -* Int type is limited to 32-bit signed integers. Overflows will throw an error. +- Int type is limited to 32-bit signed integers. Overflows will throw an error. -* Modifying a collection during iteration is an error. +- Modifying a collection during iteration is an error. -* Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are -not defined across value types. In short: `5 < 'foo'` will throw an error and -`5 == "5"` will return false. +- Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are not defined across value types. In short: `5 < 'foo'` will throw an error and `5 == "5"` will return false. -* In tuples, a trailing comma is valid only when the tuple is between - parentheses — when you write `(1,)` instead of `1,`. +- In tuples, a trailing comma is valid only when the tuple is between parentheses — when you write `(1,)` instead of `1,`. -* Dictionary literals cannot have duplicated keys. For example, this is an - error: `{"a": 4, "b": 7, "a": 1}`. +- Dictionary literals cannot have duplicated keys. For example, this is an error: `{"a": 4, "b": 7, "a": 1}`. -* Strings are represented with double-quotes (such as when you call - [repr](lib/globals#repr)). +- Strings are represented with double-quotes (such as when you call [repr](lib/globals#repr)). -* Strings aren't iterable. +- Strings aren't iterable. The following Python features are not supported: -* implicit string concatenation (use explicit `+` operator). -* Chained comparisons (such as `1 < x < 5`). -* `class` (see [`struct`](lib/struct#struct) function). -* `import` (see [`load`](/extending/concepts#loading-an-extension) statement). -* `while`, `yield`. -* float and set types. -* generators and generator expressions. -* `is` (use `==` instead). -* `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). -* `global`, `nonlocal`. -* most builtin functions, most methods. +- implicit string concatenation (use explicit `+` operator). +- Chained comparisons (such as `1 < x < 5`). +- `class` (see [`struct`](lib/struct#struct) function). +- `import` (see [`load`](/extending/concepts#loading-an-extension) statement). +- `while`, `yield`. +- float and set types. +- generators and generator expressions. +- `is` (use `==` instead). +- `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). +- `global`, `nonlocal`. +- most builtin functions, most methods. diff --git a/rules/legacy-macro-tutorial.mdx b/rules/legacy-macro-tutorial.mdx index 28b0fca8..8322ec42 100644 --- a/rules/legacy-macro-tutorial.mdx +++ b/rules/legacy-macro-tutorial.mdx @@ -2,20 +2,11 @@ title: 'Creating a Legacy Macro' --- +IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If you only need to support Bazel 8 or newer, we recommend using [symbolic macros](/extending/macros) instead; take a look at [Creating a Symbolic Macro](macro-tutorial). +Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a legacy macro that resizes an image. -IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If -you only need to support Bazel 8 or newer, we recommend using [symbolic -macros](/extending/macros) instead; take a look at [Creating a Symbolic -Macro](macro-tutorial). - -Imagine that you need to run a tool as part of your build. For example, you -may want to generate or preprocess a source file, or compress a binary. In this -tutorial, you are going to create a legacy macro that resizes an image. - -Macros are suitable for simple tasks. If you want to do anything more -complicated, for example add support for a new programming language, consider -creating a [rule](/extending/rules). Rules give you more control and flexibility. +Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -24,7 +15,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -34,8 +25,7 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, -define a function in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, define a function in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark def miniature(name, src, size = "100x100", **kwargs): @@ -48,25 +38,20 @@ def miniature(name, src, size = "100x100", **kwargs): srcs = [src], # Note that the line below will fail if `src` is not a filename string outs = ["small_" + src], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs ) ``` A few remarks: - * By convention, legacy macros have a `name` argument, just like rules. +- By convention, legacy macros have a `name` argument, just like rules. - * To document the behavior of a legacy macro, use - [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. +- To document the behavior of a legacy macro, use [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. - * To call a `genrule`, or any other native rule, prefix with `native.`. +- To call a `genrule`, or any other native rule, prefix with `native.`. - * Use `**kwargs` to forward the extra arguments to the underlying `genrule` - (it works just like in - [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). - This is useful, so that a user can use standard attributes like - `visibility`, or `tags`. +- Use `**kwargs` to forward the extra arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful, so that a user can use standard attributes like `visibility`, or `tags`. Now, use the macro from the `BUILD` file: @@ -85,14 +70,6 @@ cc_binary( ) ``` -And finally, a **warning note**: the macro assumes that `src` is a filename -string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` -works; but what happens if the `BUILD` file instead used `src = -"//other/package:image.png"`, or even `src = select(...)`? - -You should make sure to declare such assumptions in your macro's documentation. -Unfortunately, legacy macros, especially large ones, tend to be fragile because -it can be hard to notice and document all such assumptions in your code – and, -of course, some users of the macro won't read the documentation. We recommend, -if possible, instead using [symbolic macros](/extending/macros), which have -built\-in checks on attribute types. +And finally, a **warning note**: the macro assumes that `src` is a filename string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` works; but what happens if the `BUILD` file instead used `src = "//other/package:image.png"`, or even `src = select(...)`? + +You should make sure to declare such assumptions in your macro's documentation. Unfortunately, legacy macros, especially large ones, tend to be fragile because it can be hard to notice and document all such assumptions in your code – and, of course, some users of the macro won't read the documentation. We recommend, if possible, instead using [symbolic macros](/extending/macros), which have built-in checks on attribute types. diff --git a/rules/macro-tutorial.mdx b/rules/macro-tutorial.mdx index 2b3d8e4a..4cf91617 100644 --- a/rules/macro-tutorial.mdx +++ b/rules/macro-tutorial.mdx @@ -2,20 +2,11 @@ title: 'Creating a Symbolic Macro' --- +IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new macro system introduced in Bazel 8. If you need to support older Bazel versions, you will want to write a [legacy macro](/extending/legacy-macros) instead; take a look at [Creating a Legacy Macro](legacy-macro-tutorial). +Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a symbolic macro that resizes an image. -IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new -macro system introduced in Bazel 8. If you need to support older Bazel versions, -you will want to write a [legacy macro](/extending/legacy-macros) instead; take -a look at [Creating a Legacy Macro](legacy-macro-tutorial). - -Imagine that you need to run a tool as part of your build. For example, you -may want to generate or preprocess a source file, or compress a binary. In this -tutorial, you are going to create a symbolic macro that resizes an image. - -Macros are suitable for simple tasks. If you want to do anything more -complicated, for example add support for a new programming language, consider -creating a [rule](/extending/rules). Rules give you more control and flexibility. +Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -24,7 +15,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -34,9 +25,7 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, -define an *implementation function* and a *macro declaration* in a separate -`.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, define an *implementation function* and a *macro declaration* in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark # Implementation function @@ -46,7 +35,7 @@ def _miniature_impl(name, visibility, src, size, **kwargs): visibility = visibility, srcs = [src], outs = [name + "_small_" + src.name], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs, ) @@ -84,19 +73,13 @@ miniature = macro( A few remarks: - * Symbolic macro implementation functions must have `name` and `visibility` - parameters. They should used for the macro's main target. +- Symbolic macro implementation functions must have `name` and `visibility` parameters. They should used for the macro's main target. - * To document the behavior of a symbolic macro, use `doc` parameters for - `macro()` and its attributes. +- To document the behavior of a symbolic macro, use `doc` parameters for `macro()` and its attributes. - * To call a `genrule`, or any other native rule, use `native.`. +- To call a `genrule`, or any other native rule, use `native.`. - * Use `**kwargs` to forward the extra inherited arguments to the underlying - `genrule` (it works just like in - [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). - This is useful so that a user can set standard attributes like `tags` or - `testonly`. +- Use `**kwargs` to forward the extra inherited arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful so that a user can set standard attributes like `tags` or `testonly`. Now, use the macro from the `BUILD` file: diff --git a/rules/performance.mdx b/rules/performance.mdx index c415cf14..edc0a2fd 100644 --- a/rules/performance.mdx +++ b/rules/performance.mdx @@ -2,30 +2,20 @@ title: 'Optimizing Performance' --- +When writing rules, the most common performance pitfall is to traverse or copy data that is accumulated from dependencies. When aggregated over the whole build, these operations can easily take O(N^2) time or space. To avoid this, it is crucial to understand how to use depsets effectively. - -When writing rules, the most common performance pitfall is to traverse or copy -data that is accumulated from dependencies. When aggregated over the whole -build, these operations can easily take O(N^2) time or space. To avoid this, it -is crucial to understand how to use depsets effectively. - -This can be hard to get right, so Bazel also provides a memory profiler that -assists you in finding spots where you might have made a mistake. Be warned: -The cost of writing an inefficient rule may not be evident until it is in -widespread use. +This can be hard to get right, so Bazel also provides a memory profiler that assists you in finding spots where you might have made a mistake. Be warned: The cost of writing an inefficient rule may not be evident until it is in widespread use. ## Use depsets -Whenever you are rolling up information from rule dependencies you should use -[depsets](lib/depset). Only use plain lists or dicts to publish information -local to the current rule. +Whenever you are rolling up information from rule dependencies you should use [depsets](lib/depset). Only use plain lists or dicts to publish information local to the current rule. A depset represents information as a nested graph which enables sharing. Consider the following graph: ``` -C -> B -> A +C -> B -> A D ---^ ``` @@ -47,12 +37,9 @@ c = ['c', 'b', 'a'] d = ['d', 'b', 'a'] ``` -Note that in this case `'a'` is mentioned four times! With larger graphs this -problem will only get worse. +Note that in this case `'a'` is mentioned four times! With larger graphs this problem will only get worse. -Here is an example of a rule implementation that uses depsets correctly to -publish transitive information. Note that it is OK to publish rule-local -information using lists if you want since this is not O(N^2). +Here is an example of a rule implementation that uses depsets correctly to publish transitive information. Note that it is OK to publish rule-local information using lists if you want since this is not O(N^2). ``` MyProvider = provider() @@ -74,21 +61,13 @@ See the [depset overview](/extending/depsets) page for more information. ### Avoid calling `depset.to_list()` -You can coerce a depset to a flat list using -[`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) -cost. If at all possible, avoid any flattening of depsets except for debugging -purposes. +You can coerce a depset to a flat list using [`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) cost. If at all possible, avoid any flattening of depsets except for debugging purposes. -A common misconception is that you can freely flatten depsets if you only do it -at top-level targets, such as an `_binary` rule, since then the cost is not -accumulated over each level of the build graph. But this is *still* O(N^2) when -you build a set of targets with overlapping dependencies. This happens when -building your tests `//foo/tests/...`, or when importing an IDE project. +A common misconception is that you can freely flatten depsets if you only do it at top-level targets, such as an `<xx>_binary` rule, since then the cost is not accumulated over each level of the build graph. But this is *still* O(N^2) when you build a set of targets with overlapping dependencies. This happens when building your tests `//foo/tests/...`, or when importing an IDE project. ### Reduce the number of calls to `depset` -Calling `depset` inside a loop is often a mistake. It can lead to depsets with -very deep nesting, which perform poorly. For example: +Calling `depset` inside a loop is often a mistake. It can lead to depsets with very deep nesting, which perform poorly. For example: ```python x = depset() @@ -97,8 +76,7 @@ for i in inputs: x = depset(transitive = [x, i.deps]) ``` -This code can be replaced easily. First, collect the transitive depsets and -merge them all at once: +This code can be replaced easily. First, collect the transitive depsets and merge them all at once: ```python transitive = [] @@ -117,33 +95,19 @@ x = depset(transitive = [i.deps for i in inputs]) ## Use ctx.actions.args() for command lines -When building command lines you should use [ctx.actions.args()](lib/Args). -This defers expansion of any depsets to the execution phase. +When building command lines you should use [ctx.actions.args()](lib/Args). This defers expansion of any depsets to the execution phase. -Apart from being strictly faster, this will reduce the memory consumption of -your rules -- sometimes by 90% or more. +Apart from being strictly faster, this will reduce the memory consumption of your rules -- sometimes by 90% or more. Here are some tricks: -* Pass depsets and lists directly as arguments, instead of flattening them -yourself. They will get expanded by `ctx.actions.args()` for you. -If you need any transformations on the depset contents, look at -[ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. +- Pass depsets and lists directly as arguments, instead of flattening them yourself. They will get expanded by `ctx.actions.args()` for you. If you need any transformations on the depset contents, look at [ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. -* Are you passing `File#path` as arguments? No need. Any -[File](lib/File) is automatically turned into its -[path](lib/File#path), deferred to expansion time. +- Are you passing `File#path` as arguments? No need. Any [File](lib/File) is automatically turned into its [path](lib/File#path), deferred to expansion time. -* Avoid constructing strings by concatenating them together. -The best string argument is a constant as its memory will be shared between -all instances of your rule. +- Avoid constructing strings by concatenating them together. The best string argument is a constant as its memory will be shared between all instances of your rule. -* If the args are too long for the command line an `ctx.actions.args()` object -can be conditionally or unconditionally written to a param file using -[`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is -done behind the scenes when the action is executed. If you need to explicitly -control the params file you can write it manually using -[`ctx.actions.write`](lib/actions#write). +- If the args are too long for the command line an `ctx.actions.args()` object can be conditionally or unconditionally written to a param file using [`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is done behind the scenes when the action is executed. If you need to explicitly control the params file you can write it manually using [`ctx.actions.write`](lib/actions#write). Example: @@ -154,15 +118,15 @@ def _impl(ctx): file = ctx.declare_file(...) files = depset(...) - # Bad, constructs a full string "--foo=" for each rule instance + # Bad, constructs a full string "--foo=<file path>" for each rule instance args.add("--foo=" + file.path) # Good, shares "--foo" among all rule instances, and defers file.path to later - # It will however pass ["--foo", ] to the action command line, - # instead of ["--foo="] + # It will however pass ["--foo", <file path>] to the action command line, + # instead of ["--foo=<file_path>"] args.add("--foo", file) - # Use format if you prefer ["--foo="] to ["--foo", ] + # Use format if you prefer ["--foo=<file path>"] to ["--foo", <file path>] args.add(file, format="--foo=%s") # Bad, makes a giant string of a whole depset @@ -178,9 +142,7 @@ def _to_short_path(f): ## Transitive action inputs should be depsets -When building an action using [ctx.actions.run](lib/actions?#run), do not -forget that the `inputs` field accepts a depset. Use this whenever inputs are -collected from dependencies transitively. +When building an action using [ctx.actions.run](lib/actions?#run), do not forget that the `inputs` field accepts a depset. Use this whenever inputs are collected from dependencies transitively. ``` inputs = depset(...) @@ -192,54 +154,39 @@ ctx.actions.run( ## Hanging -If Bazel appears to be hung, you can hit Ctrl-\ or send -Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread -dump in the file `$(bazel info output_base)/server/jvm.out`. +If Bazel appears to be hung, you can hit `Ctrl-\` or send Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread dump in the file `$(bazel info output_base)/server/jvm.out`. -Since you may not be able to run `bazel info` if bazel is hung, the -`output_base` directory is usually the parent of the `bazel-` -symlink in your workspace directory. +Since you may not be able to run `bazel info` if bazel is hung, the `output_base` directory is usually the parent of the `bazel-<workspace>` symlink in your workspace directory. ## Performance profiling -The [JSON trace profile](/advanced/performance/json-trace-profile) can be very -useful to quickly understand what Bazel spent time on during the invocation. +The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. -The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) -flag may be used to capture Java Flight Recorder profiles of various kinds -(cpu time, wall time, memory allocations and lock contention). +The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) flag may be used to capture Java Flight Recorder profiles of various kinds (cpu time, wall time, memory allocations and lock contention). -The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) -flag may be used to write a pprof profile of CPU usage by all Starlark threads. +The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) flag may be used to write a pprof profile of CPU usage by all Starlark threads. ## Memory profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s -memory use. If there is a problem you can dump the heap to find the -exact line of code that is causing the problem. +Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. If there is a problem you can dump the heap to find the exact line of code that is causing the problem. ### Enabling memory tracking You must pass these two startup flags to *every* Bazel invocation: - ``` - STARTUP_FLAGS=\ - --host_jvm_args=-javaagent: \

- --host_jvm_args=-DRULE_MEMORY_TRACKER=1 - ``` -Note: You can download the allocation instrumenter jar file from [Maven Central -Repository][allocation-instrumenter-link]. +``` +STARTUP_FLAGS=\ +--host_jvm_args=-javaagent:<path to java-allocation-instrumenter-3.3.4.jar> \ +--host_jvm_args=-DRULE_MEMORY_TRACKER=1 +``` -[allocation-instrumenter-link]: https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4 +Note: You can download the allocation instrumenter jar file from [Maven Central Repository](https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4). -These start the server in memory tracking mode. If you forget these for even -one Bazel invocation the server will restart and you will have to start over. +These start the server in memory tracking mode. If you forget these for even one Bazel invocation the server will restart and you will have to start over. ### Using the Memory Tracker -As an example, look at the target `foo` and see what it does. To only -run the analysis and not run the build execution phase, add the -`--nobuild` flag. +As an example, look at the target `foo` and see what it does. To only run the analysis and not run the build execution phase, add the `--nobuild` flag. ``` $ bazel $(STARTUP_FLAGS) build --nobuild //foo:foo @@ -249,14 +196,14 @@ Next, see how much memory the whole Bazel instance consumes: ``` $ bazel $(STARTUP_FLAGS) info used-heap-size-after-gc -> 2594MB +> 2594MB ``` Break it down by rule class by using `bazel dump --rules`: ``` $ bazel $(STARTUP_FLAGS) dump --rules -> +> RULE COUNT ACTIONS BYTES EACH genrule 33,762 33,801 291,538,824 8,635 @@ -271,16 +218,14 @@ _check_proto_library_deps 719 668 1,835,288 2,5 ... (more output) ``` -Look at where the memory is going by producing a `pprof` file -using `bazel dump --skylark_memory`: +Look at where the memory is going by producing a `pprof` file using `bazel dump --skylark_memory`: ``` $ bazel $(STARTUP_FLAGS) dump --skylark_memory=$HOME/prof.gz -> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz +> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz ``` -Use the `pprof` tool to investigate the heap. A good starting point is -getting a flame graph by using `pprof -flame $HOME/prof.gz`. +Use the `pprof` tool to investigate the heap. A good starting point is getting a flame graph by using `pprof -flame $HOME/prof.gz`. Get `pprof` from [https://github.com/google/pprof](https://github.com/google/pprof). @@ -288,13 +233,13 @@ Get a text dump of the hottest call sites annotated with lines: ``` $ pprof -text -lines $HOME/prof.gz -> +> flat flat% sum% cum cum% - 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library :-1 - 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule :-1 - 74.11MB 9.96% 44.80% 74.11MB 9.96% glob :-1 - 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup :-1 - 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test :-1 + 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library <native>:-1 + 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule <native>:-1 + 74.11MB 9.96% 44.80% 74.11MB 9.96% glob <native>:-1 + 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup <native>:-1 + 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test <native>:-1 26.55MB 3.57% 63.07% 26.55MB 3.57% _generate_foo_files /foo/tc/tc.bzl:491 26.01MB 3.50% 66.57% 26.01MB 3.50% _build_foo_impl /foo/build_test.bzl:78 22.01MB 2.96% 69.53% 22.01MB 2.96% _build_foo_impl /foo/build_test.bzl:73 diff --git a/rules/rules-tutorial.mdx b/rules/rules-tutorial.mdx index 7700f20c..30ad839d 100644 --- a/rules/rules-tutorial.mdx +++ b/rules/rules-tutorial.mdx @@ -2,24 +2,9 @@ title: 'Rules Tutorial' --- +[Starlark](https://github.com/bazelbuild/starlark) is a Python-like configuration language originally developed for use in Bazel and since adopted by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of Starlark properly known as the "Build Language", though it is often simply referred to as "Starlark", especially when emphasizing that a feature is expressed in the Build Language as opposed to being a built-in or "native" part of Bazel. Bazel augments the core language with numerous build-related functions such as `glob`, `genrule`, `java_binary`, and so on. - -{/* [TOC] */} - -[Starlark](https://github.com/bazelbuild/starlark) is a Python-like -configuration language originally developed for use in Bazel and since adopted -by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of -Starlark properly known as the "Build Language", though it is often simply -referred to as "Starlark", especially when emphasizing that a feature is -expressed in the Build Language as opposed to being a built-in or "native" part -of Bazel. Bazel augments the core language with numerous build-related functions -such as `glob`, `genrule`, `java_binary`, and so on. - -See the -[Bazel](/start/) and [Starlark](/extending/concepts) documentation for -more details, and the -[Rules SIG template](https://github.com/bazel-contrib/rules-template) as a -starting point for new rulesets. +See the [Bazel](/start/) and [Starlark](/extending/concepts) documentation for more details, and the [Rules SIG template](https://github.com/bazel-contrib/rules-template) as a starting point for new rulesets. ## The empty rule @@ -34,10 +19,7 @@ foo_binary = rule( ) ``` -When you call the [`rule`](lib/globals#rule) function, you -must define a callback function. The logic will go there, but you -can leave the function empty for now. The [`ctx`](lib/ctx) argument -provides information about the target. +When you call the [`rule`](lib/globals#rule) function, you must define a callback function. The logic will go there, but you can leave the function empty for now. The [`ctx`](lib/ctx) argument provides information about the target. You can load the rule and use it from a `BUILD` file. @@ -58,9 +40,7 @@ INFO: Found 1 target... Target //:bin up-to-date (nothing to build) ``` -Even though the rule does nothing, it already behaves like other rules: it has a -mandatory name, it supports common attributes like `visibility`, `testonly`, and -`tags`. +Even though the rule does nothing, it already behaves like other rules: it has a mandatory name, it supports common attributes like `visibility`, `testonly`, and `tags`. ## Evaluation model @@ -89,10 +69,7 @@ foo_binary(name = "bin1") foo_binary(name = "bin2") ``` -[`ctx.label`](lib/ctx#label) -corresponds to the label of the target being analyzed. The `ctx` object has -many useful fields and methods; you can find an exhaustive list in the -[API reference](lib/ctx). +[`ctx.label`](lib/ctx#label) corresponds to the label of the target being analyzed. The `ctx` object has many useful fields and methods; you can find an exhaustive list in the [API reference](lib/ctx). Query the code: @@ -106,15 +83,10 @@ DEBUG: /usr/home/bazel-codelab/BUILD:2:1: BUILD file Make a few observations: -* "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, - Bazel evaluates all the files it loads. If multiple `BUILD` files are loading - foo.bzl, you would see only one occurrence of "bzl file evaluation" because - Bazel caches the result of the evaluation. -* The callback function `_foo_binary_impl` is not called. Bazel query loads - `BUILD` files, but doesn't analyze targets. +- "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, Bazel evaluates all the files it loads. If multiple `BUILD` files are loading foo.bzl, you would see only one occurrence of "bzl file evaluation" because Bazel caches the result of the evaluation. +- The callback function `_foo_binary_impl` is not called. Bazel query loads `BUILD` files, but doesn't analyze targets. -To analyze the targets, use the [`cquery`](/query/cquery) ("configured -query") or the `build` command: +To analyze the targets, use the [`cquery`](/query/cquery) ("configured query") or the `build` command: ``` $ bazel build :all @@ -126,15 +98,11 @@ INFO: Found 2 targets... As you can see, `_foo_binary_impl` is now called twice - once for each target. -Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, -because the evaluation of `foo.bzl` is cached after the call to `bazel query`. -Bazel only emits `print` statements when they are actually executed. +Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, because the evaluation of `foo.bzl` is cached after the call to `bazel query`. Bazel only emits `print` statements when they are actually executed. ## Creating a file -To make your rule more useful, update it to generate a file. First, declare the -file and give it a name. In this example, create a file with the same name as -the target: +To make your rule more useful, update it to generate a file. First, declare the file and give it a name. In this example, create a file with the same name as the target: ```python ctx.actions.declare_file(ctx.label.name) @@ -147,9 +115,7 @@ The following files have no generating action: bin2 ``` -Whenever you declare a file, you have to tell Bazel how to generate it by -creating an action. Use [`ctx.actions.write`](lib/actions#write), -to create a file with the given content. +Whenever you declare a file, you have to tell Bazel how to generate it by creating an action. Use [`ctx.actions.write`](lib/actions#write), to create a file with the given content. ```python def _foo_binary_impl(ctx): @@ -167,11 +133,7 @@ $ bazel build bin1 Target //:bin1 up-to-date (nothing to build) ``` -The `ctx.actions.write` function registered an action, which taught Bazel -how to generate the file. But Bazel won't create the file until it is -actually requested. So the last thing to do is tell Bazel that the file -is an output of the rule, and not a temporary file used within the rule -implementation. +The `ctx.actions.write` function registered an action, which taught Bazel how to generate the file. But Bazel won't create the file until it is actually requested. So the last thing to do is tell Bazel that the file is an output of the rule, and not a temporary file used within the rule implementation. ```python def _foo_binary_impl(ctx): @@ -183,8 +145,7 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Look at the `DefaultInfo` and `depset` functions later. For now, -assume that the last line is the way to choose the outputs of a rule. +Look at the `DefaultInfo` and `depset` functions later. For now, assume that the last line is the way to choose the outputs of a rule. Now, run Bazel: @@ -202,8 +163,7 @@ You have successfully generated a file! ## Attributes -To make the rule more useful, add new attributes using -[the `attr` module](lib/attr) and update the rule definition. +To make the rule more useful, add new attributes using [the `attr` module](lib/attr) and update the rule definition. Add a string attribute called `username`: @@ -225,8 +185,7 @@ foo_binary( ) ``` -To access the value in the callback function, use `ctx.attr.username`. For -example: +To access the value in the callback function, use `ctx.attr.username`. For example: ```python def _foo_binary_impl(ctx): @@ -238,38 +197,23 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Note that you can make the attribute mandatory or set a default value. Look at -the documentation of [`attr.string`](lib/attr#string). -You may also use other types of attributes, such as [boolean](lib/attr#bool) -or [list of integers](lib/attr#int_list). +Note that you can make the attribute mandatory or set a default value. Look at the documentation of [`attr.string`](lib/attr#string). You may also use other types of attributes, such as [boolean](lib/attr#bool) or [list of integers](lib/attr#int_list). ## Dependencies -Dependency attributes, such as [`attr.label`](lib/attr#label) -and [`attr.label_list`](lib/attr#label_list), -declare a dependency from the target that owns the attribute to the target whose -label appears in the attribute's value. This kind of attribute forms the basis -of the target graph. +Dependency attributes, such as [`attr.label`](lib/attr#label) and [`attr.label_list`](lib/attr#label_list), declare a dependency from the target that owns the attribute to the target whose label appears in the attribute's value. This kind of attribute forms the basis of the target graph. -In the `BUILD` file, the target label appears as a string object, such as -`//pkg:name`. In the implementation function, the target will be accessible as a -[`Target`](lib/Target) object. For example, view the files returned -by the target using [`Target.files`](lib/Target#modules.Target.files). +In the `BUILD` file, the target label appears as a string object, such as `//pkg:name`. In the implementation function, the target will be accessible as a [`Target`](lib/Target) object. For example, view the files returned by the target using [`Target.files`](lib/Target#modules.Target.files). ### Multiple files -By default, only targets created by rules may appear as dependencies (such as a -`foo_library()` target). If you want the attribute to accept targets that are -input files (such as source files in the repository), you can do it with -`allow_files` and specify the list of accepted file extensions (or `True` to -allow any file extension): +By default, only targets created by rules may appear as dependencies (such as a `foo_library()` target). If you want the attribute to accept targets that are input files (such as source files in the repository), you can do it with `allow_files` and specify the list of accepted file extensions (or `True` to allow any file extension): ```python "srcs": attr.label_list(allow_files = [".java"]), ``` -The list of files can be accessed with `ctx.files.`. For -example, the list of files in the `srcs` attribute can be accessed through +The list of files can be accessed with `ctx.files.<attribute name>`. For example, the list of files in the `srcs` attribute can be accessed through ```python ctx.files.srcs @@ -283,7 +227,7 @@ If you need only one file, use `allow_single_file`: "src": attr.label(allow_single_file = [".java"]) ``` -This file is then accessible under `ctx.file.`: +This file is then accessible under `ctx.file.<attribute name>`: ```python ctx.file.src @@ -291,17 +235,9 @@ ctx.file.src ## Create a file with a template -You can create a rule that generates a .cc file based on a template. Also, you -can use `ctx.actions.write` to output a string constructed in the rule -implementation function, but this has two problems. First, as the template gets -bigger, it becomes more memory efficient to put it in a separate file and avoid -constructing large strings during the analysis phase. Second, using a separate -file is more convenient for the user. Instead, use -[`ctx.actions.expand_template`](lib/actions#expand_template), -which performs substitutions on a template file. +You can create a rule that generates a .cc file based on a template. Also, you can use `ctx.actions.write` to output a string constructed in the rule implementation function, but this has two problems. First, as the template gets bigger, it becomes more memory efficient to put it in a separate file and avoid constructing large strings during the analysis phase. Second, using a separate file is more convenient for the user. Instead, use [`ctx.actions.expand_template`](lib/actions#expand_template), which performs substitutions on a template file. -Create a `template` attribute to declare a dependency on the template -file: +Create a `template` attribute to declare a dependency on the template file: ```python def _hello_world_impl(ctx): @@ -340,8 +276,7 @@ cc_binary( ) ``` -If you don't want to expose the template to the end-user and always use the -same one, you can set a default value and make the attribute private: +If you don't want to expose the template to the end-user and always use the same one, you can set a default value and make the attribute private: ```python "_template": attr.label( @@ -350,11 +285,7 @@ same one, you can set a default value and make the attribute private: ), ``` -Attributes that start with an underscore are private and cannot be set in a -`BUILD` file. The template is now an _implicit dependency_: Every `hello_world` -target has a dependency on this file. Don't forget to make this file visible -to other packages by updating the `BUILD` file and using -[`exports_files`](/reference/be/functions#exports_files): +Attributes that start with an underscore are private and cannot be set in a `BUILD` file. The template is now an *implicit dependency*: Every `hello_world` target has a dependency on this file. Don't forget to make this file visible to other packages by updating the `BUILD` file and using [`exports_files`](/reference/be/functions#exports_files): ```python exports_files(["file.cc.tpl"]) @@ -362,7 +293,6 @@ exports_files(["file.cc.tpl"]) ## Going further -* Take a look at the [reference documentation for rules](/extending/rules#contents). -* Get familiar with [depsets](/extending/depsets). -* Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) - which includes additional examples of rules. +- Take a look at the [reference documentation for rules](/extending/rules#contents). +- Get familiar with [depsets](/extending/depsets). +- Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) which includes additional examples of rules. diff --git a/rules/testing.mdx b/rules/testing.mdx index 2996e08c..4a191c15 100644 --- a/rules/testing.mdx +++ b/rules/testing.mdx @@ -2,47 +2,23 @@ title: 'Testing' --- - - -There are several different approaches to testing Starlark code in Bazel. This -page gathers the current best practices and frameworks by use case. +There are several different approaches to testing Starlark code in Bazel. This page gathers the current best practices and frameworks by use case. ## Testing rules -[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called -[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) -for checking the analysis-time behavior of rules, such as their actions and -providers. Such tests are called "analysis tests" and are currently the best -option for testing the inner workings of rules. +[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) for checking the analysis-time behavior of rules, such as their actions and providers. Such tests are called "analysis tests" and are currently the best option for testing the inner workings of rules. Some caveats: -* Test assertions occur within the build, not a separate test runner process. - Targets that are created by the test must be named such that they do not - collide with targets from other tests or from the build. An error that - occurs during the test is seen by Bazel as a build breakage rather than a - test failure. - -* It requires a fair amount of boilerplate to set up the rules under test and - the rules containing test assertions. This boilerplate may seem daunting at - first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros - are evaluated and targets generated during the loading phase, while rule - implementation functions don't run until later, during the analysis phase. - -* Analysis tests are intended to be fairly small and lightweight. Certain - features of the analysis testing framework are restricted to verifying - targets with a maximum number of transitive dependencies (currently 500). - This is due to performance implications of using these features with larger - tests. - -The basic principle is to define a testing rule that depends on the -rule-under-test. This gives the testing rule access to the rule-under-test's -providers. - -The testing rule's implementation function carries out assertions. If there are -any failures, these are not raised immediately by calling `fail()` (which would -trigger an analysis-time build error), but rather by storing the errors in a -generated script that fails at test execution time. +- Test assertions occur within the build, not a separate test runner process. Targets that are created by the test must be named such that they do not collide with targets from other tests or from the build. An error that occurs during the test is seen by Bazel as a build breakage rather than a test failure. + +- It requires a fair amount of boilerplate to set up the rules under test and the rules containing test assertions. This boilerplate may seem daunting at first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros are evaluated and targets generated during the loading phase, while rule implementation functions don't run until later, during the analysis phase. + +- Analysis tests are intended to be fairly small and lightweight. Certain features of the analysis testing framework are restricted to verifying targets with a maximum number of transitive dependencies (currently 500). This is due to performance implications of using these features with larger tests. + +The basic principle is to define a testing rule that depends on the rule-under-test. This gives the testing rule access to the rule-under-test's providers. + +The testing rule's implementation function carries out assertions. If there are any failures, these are not raised immediately by calling `fail()` (which would trigger an analysis-time build error), but rather by storing the errors in a generated script that fails at test execution time. See below for a minimal toy example, followed by an example that checks actions. @@ -69,7 +45,6 @@ myrule = rule( `//mypkg/myrules_test.bzl`: - ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest") load(":myrules.bzl", "myrule", "MyInfo") @@ -139,47 +114,31 @@ myrules_test_suite(name = "myrules_test") The test can be run with `bazel test //mypkg:myrules_test`. -Aside from the initial `load()` statements, there are two main parts to the -file: +Aside from the initial `load()` statements, there are two main parts to the file: -* The tests themselves, each of which consists of 1) an analysis-time - implementation function for the testing rule, 2) a declaration of the - testing rule via `analysistest.make()`, and 3) a loading-time function - (macro) for declaring the rule-under-test (and its dependencies) and testing - rule. If the assertions do not change between test cases, 1) and 2) may be - shared by multiple test cases. +- The tests themselves, each of which consists of 1) an analysis-time implementation function for the testing rule, 2) a declaration of the testing rule via `analysistest.make()`, and 3) a loading-time function (macro) for declaring the rule-under-test (and its dependencies) and testing rule. If the assertions do not change between test cases, 1) and 2) may be shared by multiple test cases. -* The test suite function, which calls the loading-time functions for each - test, and declares a `test_suite` target bundling all tests together. +- The test suite function, which calls the loading-time functions for each test, and declares a `test_suite` target bundling all tests together. -For consistency, follow the recommended naming convention: Let `foo` stand for -the part of the test name that describes what the test is checking -(`provider_contents` in the above example). For example, a JUnit test method -would be named `testFoo`. +For consistency, follow the recommended naming convention: Let `foo` stand for the part of the test name that describes what the test is checking (`provider_contents` in the above example). For example, a JUnit test method would be named `testFoo`. Then: -* the macro which generates the test and target under test should should be - named `_test_foo` (`_test_provider_contents`) +- the macro which generates the test and target under test should should be named `_test_foo` (`_test_provider_contents`) -* its test rule type should be named `foo_test` (`provider_contents_test`) +- its test rule type should be named `foo_test` (`provider_contents_test`) -* the label of the target of this rule type should be `foo_test` - (`provider_contents_test`) +- the label of the target of this rule type should be `foo_test` (`provider_contents_test`) -* the implementation function for the testing rule should be named - `_foo_test_impl` (`_provider_contents_test_impl`) +- the implementation function for the testing rule should be named `_foo_test_impl` (`_provider_contents_test_impl`) -* the labels of the targets of the rules under test and their dependencies - should be prefixed with `foo_` (`provider_contents_`) +- the labels of the targets of the rules under test and their dependencies should be prefixed with `foo_` (`provider_contents_`) -Note that the labels of all targets can conflict with other labels in the same -BUILD package, so it's helpful to use a unique name for the test. +Note that the labels of all targets can conflict with other labels in the same BUILD package, so it's helpful to use a unique name for the test. ### Failure testing -It may be useful to verify that a rule fails given certain inputs or in certain -state. This can be done using the analysis test framework: +It may be useful to verify that a rule fails given certain inputs or in certain state. This can be done using the analysis test framework: The test rule created with `analysistest.make` should specify `expect_failure`: @@ -190,8 +149,7 @@ failure_testing_test = analysistest.make( ) ``` -The test rule implementation should make assertions on the nature of the failure -that took place (specifically, the failure message): +The test rule implementation should make assertions on the nature of the failure that took place (specifically, the failure message): ```python def _failure_testing_test_impl(ctx): @@ -200,11 +158,7 @@ def _failure_testing_test_impl(ctx): return analysistest.end(env) ``` -Also make sure that your target under test is specifically tagged 'manual'. -Without this, building all targets in your package using `:all` will result in a -build of the intentionally-failing target and will exhibit a build failure. With -'manual', your target under test will build only if explicitly specified, or as -a dependency of a non-manual target (such as your test rule): +Also make sure that your target under test is specifically tagged 'manual'. Without this, building all targets in your package using `:all` will result in a build of the intentionally-failing target and will exhibit a build failure. With 'manual', your target under test will build only if explicitly specified, or as a dependency of a non-manual target (such as your test rule): ```python def _test_failure(): @@ -219,9 +173,7 @@ def _test_failure(): ### Verifying registered actions -You may want to write tests which make assertions about the actions that your -rule registers, for example, using `ctx.actions.run()`. This can be done in your -analysis test rule implementation function. An example: +You may want to write tests which make assertions about the actions that your rule registers, for example, using `ctx.actions.run()`. This can be done in your analysis test rule implementation function. An example: ```python def _inspect_actions_test_impl(ctx): @@ -236,14 +188,11 @@ def _inspect_actions_test_impl(ctx): return analysistest.end(env) ``` -Note that `analysistest.target_actions(env)` returns a list of -[`Action`](lib/Action) objects which represent actions registered by the -target under test. +Note that `analysistest.target_actions(env)` returns a list of [`Action`](lib/Action) objects which represent actions registered by the target under test. ### Verifying rule behavior under different flags -You may want to verify your real rule behaves a certain way given certain build -flags. For example, your rule may behave differently if a user specifies: +You may want to verify your real rule behaves a certain way given certain build flags. For example, your rule may behave differently if a user specifies: ```shell bazel build //mypkg:real_target -c opt @@ -255,20 +204,15 @@ versus bazel build //mypkg:real_target -c dbg ``` -At first glance, this could be done by testing the target under test using the -desired build flags: +At first glance, this could be done by testing the target under test using the desired build flags: ```shell bazel test //mypkg:myrules_test -c opt ``` -But then it becomes impossible for your test suite to simultaneously contain a -test which verifies the rule behavior under `-c opt` and another test which -verifies the rule behavior under `-c dbg`. Both tests would not be able to run -in the same build! +But then it becomes impossible for your test suite to simultaneously contain a test which verifies the rule behavior under `-c opt` and another test which verifies the rule behavior under `-c dbg`. Both tests would not be able to run in the same build! -This can be solved by specifying the desired build flags when defining the test -rule: +This can be solved by specifying the desired build flags when defining the test rule: ```python myrule_c_opt_test = analysistest.make( @@ -279,33 +223,21 @@ myrule_c_opt_test = analysistest.make( ) ``` -Normally, a target under test is analyzed given the current build flags. -Specifying `config_settings` overrides the values of the specified command line -options. (Any unspecified options will retain their values from the actual -command line). - -In the specified `config_settings` dictionary, command line flags must be -prefixed with a special placeholder value `//command_line_option:`, as is shown -above. +Normally, a target under test is analyzed given the current build flags. Specifying `config_settings` overrides the values of the specified command line options. (Any unspecified options will retain their values from the actual command line). +In the specified `config_settings` dictionary, command line flags must be prefixed with a special placeholder value `//command_line_option:`, as is shown above. ## Validating artifacts The main ways to check that your generated files are correct are: -* You can write a test script in shell, Python, or another language, and - create a target of the appropriate `*_test` rule type. +- You can write a test script in shell, Python, or another language, and create a target of the appropriate `*_test` rule type. -* You can use a specialized rule for the kind of test you want to perform. +- You can use a specialized rule for the kind of test you want to perform. ### Using a test target -The most straightforward way to validate an artifact is to write a script and -add a `*_test` target to your BUILD file. The specific artifacts you want to -check should be data dependencies of this target. If your validation logic is -reusable for multiple tests, it should be a script that takes command line -arguments that are controlled by the test target's `args` attribute. Here's an -example that validates that the output of `myrule` from above is `"abc"`. +The most straightforward way to validate an artifact is to write a script and add a `*_test` target to your BUILD file. The specific artifacts you want to check should be data dependencies of this target. If your validation logic is reusable for multiple tests, it should be a script that takes command line arguments that are controlled by the test target's `args` attribute. Here's an example that validates that the output of `myrule` from above is `"abc"`. `//mypkg/myrule_validator.sh`: @@ -341,12 +273,7 @@ sh_test( ### Using a custom rule -A more complicated alternative is to write the shell script as a template that -gets instantiated by a new rule. This involves more indirection and Starlark -logic, but leads to cleaner BUILD files. As a side-benefit, any argument -preprocessing can be done in Starlark instead of the script, and the script is -slightly more self-documenting since it uses symbolic placeholders (for -substitutions) instead of numeric ones (for arguments). +A more complicated alternative is to write the shell script as a template that gets instantiated by a new rule. This involves more indirection and Starlark logic, but leads to cleaner BUILD files. As a side-benefit, any argument preprocessing can be done in Starlark instead of the script, and the script is slightly more self-documenting since it uses symbolic placeholders (for substitutions) instead of numeric ones (for arguments). `//mypkg/myrule_validator.sh.template`: @@ -417,18 +344,11 @@ myrule_validation_test( ) ``` -Alternatively, instead of using a template expansion action, you could have -inlined the template into the .bzl file as a string and expanded it during the -analysis phase using the `str.format` method or `%`-formatting. +Alternatively, instead of using a template expansion action, you could have inlined the template into the .bzl file as a string and expanded it during the analysis phase using the `str.format` method or `%`-formatting. ## Testing Starlark utilities -[Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) -framework can be used to test utility functions (that is, functions that are -neither macros nor rule implementations). Instead of using `unittest.bzl`'s -`analysistest` library, `unittest` may be used. For such test suites, the -convenience function `unittest.suite()` can be used to reduce boilerplate. +[Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) framework can be used to test utility functions (that is, functions that are neither macros nor rule implementations). Instead of using `unittest.bzl`'s `analysistest` library, `unittest` may be used. For such test suites, the convenience function `unittest.suite()` can be used to reduce boilerplate. `//mypkg/myhelpers.bzl`: @@ -439,7 +359,6 @@ def myhelper(): `//mypkg/myhelpers_test.bzl`: - ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":myhelpers.bzl", "myhelper") diff --git a/rules/verbs-tutorial.mdx b/rules/verbs-tutorial.mdx index db7757e1..4cd6c990 100644 --- a/rules/verbs-tutorial.mdx +++ b/rules/verbs-tutorial.mdx @@ -2,29 +2,17 @@ title: 'Using Macros to Create Custom Verbs' --- - - -Day-to-day interaction with Bazel happens primarily through a few commands: -`build`, `test`, and `run`. At times, though, these can feel limited: you may -want to push packages to a repository, publish documentation for end-users, or -deploy an application with Kubernetes. But Bazel doesn't have a `publish` or -`deploy` command – where do these actions fit in? +Day-to-day interaction with Bazel happens primarily through a few commands: `build`, `test`, and `run`. At times, though, these can feel limited: you may want to push packages to a repository, publish documentation for end-users, or deploy an application with Kubernetes. But Bazel doesn't have a `publish` or `deploy` command – where do these actions fit in? ## The bazel run command -Bazel's focus on hermeticity, reproducibility, and incrementality means the -`build` and `test` commands aren't helpful for the above tasks. These actions -may run in a sandbox, with limited network access, and aren't guaranteed to be -re-run with every `bazel build`. +Bazel's focus on hermeticity, reproducibility, and incrementality means the `build` and `test` commands aren't helpful for the above tasks. These actions may run in a sandbox, with limited network access, and aren't guaranteed to be re-run with every `bazel build`. + +Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have side effects. Bazel users are accustomed to rules that create executables, and rule authors can follow a common set of patterns to extend this to "custom verbs". -Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have -side effects. Bazel users are accustomed to rules that create executables, and -rule authors can follow a common set of patterns to extend this to -"custom verbs". +### In the wild: rules\_k8s -### In the wild: rules_k8s -For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), -the Kubernetes rules for Bazel. Suppose you have the following target: +For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), the Kubernetes rules for Bazel. Suppose you have the following target: ```python # BUILD file in //application/k8s @@ -36,51 +24,21 @@ k8s_object( ) ``` -The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a -standard Kubernetes YAML file when `bazel build` is used on the `staging` -target. However, the additional targets are also created by the `k8s_object` -macro with names like `staging.apply` and `:staging.delete`. These build -scripts to perform those actions, and when executed with `bazel run -staging.apply`, these behave like our own `bazel k8s-apply` or `bazel -k8s-delete` commands. - -### Another example: ts_api_guardian_test - -This pattern can also be seen in the Angular project. The -[`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) -produces two targets. The first is a standard `nodejs_test` target which compares -some generated output against a "golden" file (that is, a file containing the -expected output). This can be built and run with a normal `bazel -test` invocation. In `angular-cli`, you can run [one such -target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) -with `bazel test //etc/api:angular_devkit_core_api`. - -Over time, this golden file may need to be updated for legitimate reasons. -Updating this manually is tedious and error-prone, so this macro also provides -a `nodejs_binary` target that updates the golden file, instead of comparing -against it. Effectively, the same test script can be written to run in "verify" -or "accept" mode, based on how it's invoked. This follows the same pattern -you've learned already: there is no native `bazel test-accept` command, but the -same effect can be achieved with -`bazel run //etc/api:angular_devkit_core_api.accept`. - -This pattern can be quite powerful, and turns out to be quite common once you -learn to recognize it. +The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a standard Kubernetes YAML file when `bazel build` is used on the `staging` target. However, the additional targets are also created by the `k8s_object` macro with names like `staging.apply` and `:staging.delete`. These build scripts to perform those actions, and when executed with `bazel run staging.apply`, these behave like our own `bazel k8s-apply` or `bazel k8s-delete` commands. + +### Another example: ts\_api\_guardian\_test + +This pattern can also be seen in the Angular project. The [`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) produces two targets. The first is a standard `nodejs_test` target which compares some generated output against a "golden" file (that is, a file containing the expected output). This can be built and run with a normal `bazel test` invocation. In `angular-cli`, you can run [one such target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) with `bazel test //etc/api:angular_devkit_core_api`. + +Over time, this golden file may need to be updated for legitimate reasons. Updating this manually is tedious and error-prone, so this macro also provides a `nodejs_binary` target that updates the golden file, instead of comparing against it. Effectively, the same test script can be written to run in "verify" or "accept" mode, based on how it's invoked. This follows the same pattern you've learned already: there is no native `bazel test-accept` command, but the same effect can be achieved with `bazel run //etc/api:angular_devkit_core_api.accept`. + +This pattern can be quite powerful, and turns out to be quite common once you learn to recognize it. ## Adapting your own rules -[Macros](/extending/macros) are the heart of this pattern. Macros are used like -rules, but they can create several targets. Typically, they will create a -target with the specified name which performs the primary build action: perhaps -it builds a normal binary, a Docker image, or an archive of source code. In -this pattern, additional targets are created to produce scripts performing side -effects based on the output of the primary target, like publishing the -resulting binary or updating the expected test output. +[Macros](/extending/macros) are the heart of this pattern. Macros are used like rules, but they can create several targets. Typically, they will create a target with the specified name which performs the primary build action: perhaps it builds a normal binary, a Docker image, or an archive of source code. In this pattern, additional targets are created to produce scripts performing side effects based on the output of the primary target, like publishing the resulting binary or updating the expected test output. -To illustrate this, wrap an imaginary rule that generates a website with -[Sphinx](https://www.sphinx-doc.org) with a macro to create an additional -target that allows the user to publish it when ready. Consider the following -existing rule for generating a website with Sphinx: +To illustrate this, wrap an imaginary rule that generates a website with [Sphinx](https://www.sphinx-doc.org) with a macro to create an additional target that allows the user to publish it when ready. Consider the following existing rule for generating a website with Sphinx: ```python _sphinx_site = rule( @@ -89,8 +47,7 @@ _sphinx_site = rule( ) ``` -Next, consider a rule like the following, which builds a script that, when run, -publishes the generated pages: +Next, consider a rule like the following, which builds a script that, when run, publishes the generated pages: ```python _sphinx_publisher = rule( @@ -106,8 +63,7 @@ _sphinx_publisher = rule( ) ``` -Finally, define the following symbolic macro (available in Bazel 8 or newer) to -create targets for both of the above rules together: +Finally, define the following symbolic macro (available in Bazel 8 or newer) to create targets for both of the above rules together: ```starlark def _sphinx_site_impl(name, visibility, srcs, **kwargs): @@ -128,8 +84,7 @@ sphinx_site = macro( ) ``` -Or, if you need to support Bazel releases older than Bazel 8, you would instead -define a legacy macro: +Or, if you need to support Bazel releases older than Bazel 8, you would instead define a legacy macro: ```starlark def sphinx_site(name, srcs = [], **kwargs): @@ -140,8 +95,7 @@ def sphinx_site(name, srcs = [], **kwargs): _sphinx_publisher(name = "%s.publish" % name, site = name, **kwargs) ``` -In the `BUILD` files, use the macro as though it just creates the primary -target: +In the `BUILD` files, use the macro as though it just creates the primary target: ```python sphinx_site( @@ -150,28 +104,8 @@ sphinx_site( ) ``` -In this example, a "docs" target is created, just as though the macro were a -standard, single Bazel rule. When built, the rule generates some configuration -and runs Sphinx to produce an HTML site, ready for manual inspection. However, -an additional "docs.publish" target is also created, which builds a script for -publishing the site. Once you check the output of the primary target, you can -use `bazel run :docs.publish` to publish it for public consumption, just like -an imaginary `bazel publish` command. - -It's not immediately obvious what the implementation of the `_sphinx_publisher` -rule might look like. Often, actions like this write a _launcher_ shell script. -This method typically involves using -[`ctx.actions.expand_template`](lib/actions#expand_template) -to write a very simple shell script, in this case invoking the publisher binary -with a path to the output of the primary target. This way, the publisher -implementation can remain generic, the `_sphinx_site` rule can just produce -HTML, and this small script is all that's necessary to combine the two -together. - -In `rules_k8s`, this is indeed what `.apply` does: -[`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) -writes a very simple Bash script, based on -[`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), -which runs `kubectl` with the output of the primary target. This script can -then be build and run with `bazel run :staging.apply`, effectively providing a -`k8s-apply` command for `k8s_object` targets. +In this example, a "docs" target is created, just as though the macro were a standard, single Bazel rule. When built, the rule generates some configuration and runs Sphinx to produce an HTML site, ready for manual inspection. However, an additional "docs.publish" target is also created, which builds a script for publishing the site. Once you check the output of the primary target, you can use `bazel run :docs.publish` to publish it for public consumption, just like an imaginary `bazel publish` command. + +It's not immediately obvious what the implementation of the `_sphinx_publisher` rule might look like. Often, actions like this write a *launcher* shell script. This method typically involves using [`ctx.actions.expand_template`](lib/actions#expand_template) to write a very simple shell script, in this case invoking the publisher binary with a path to the output of the primary target. This way, the publisher implementation can remain generic, the `_sphinx_site` rule can just produce HTML, and this small script is all that's necessary to combine the two together. + +In `rules_k8s`, this is indeed what `.apply` does: [`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) writes a very simple Bash script, based on [`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), which runs `kubectl` with the output of the primary target. This script can then be build and run with `bazel run :staging.apply`, effectively providing a `k8s-apply` command for `k8s_object` targets. diff --git a/run/bazelrc.mdx b/run/bazelrc.mdx index 90d4464a..2ca427d7 100644 --- a/run/bazelrc.mdx +++ b/run/bazelrc.mdx @@ -2,129 +2,85 @@ title: 'Write bazelrc configuration files' --- - - -Bazel accepts many options. Some options are varied frequently (for example, -`--subcommands`) while others stay the same across several builds (such as -`--package_path`). To avoid specifying these unchanged options for every build -(and other commands), you can specify options in a configuration file, called -`.bazelrc`. +Bazel accepts many options. Some options are varied frequently (for example, `--subcommands`) while others stay the same across several builds (such as `--package_path`). To avoid specifying these unchanged options for every build (and other commands), you can specify options in a configuration file, called `.bazelrc`. ### Where are the `.bazelrc` files? -Bazel looks for optional configuration files in the following locations, -in the order shown below. The options are interpreted in this order, so -options in later files can override a value from an earlier file if a -conflict arises. All options that control which of these files are loaded are -startup options, which means they must occur after `bazel` and -before the command (`build`, `test`, etc). - -1. **The system RC file**, unless `--nosystem_rc` is present. +Bazel looks for optional configuration files in the following locations, in the order shown below. The options are interpreted in this order, so options in later files can override a value from an earlier file if a conflict arises. All options that control which of these files are loaded are startup options, which means they must occur after `bazel` and before the command (`build`, `test`, etc). - Path: +1. **The system RC file**, unless `--nosystem_rc` is present. - - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` - - On Windows: `%ProgramData%\bazel.bazelrc` + Path: - It is not an error if this file does not exist. + - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` + - On Windows: `%ProgramData%\bazel.bazelrc` - If another system-specified location is required, you must build a custom - Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in - [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). - The system-specified location may contain environment variable references, - such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. + It is not an error if this file does not exist. -2. **The workspace RC file**, unless `--noworkspace_rc` is present. + If another system-specified location is required, you must build a custom Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). The system-specified location may contain environment variable references, such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. - Path: `.bazelrc` in your workspace directory (next to the main - `MODULE.bazel` file). +2. **The workspace RC file**, unless `--noworkspace_rc` is present. - It is not an error if this file does not exist. + Path: `.bazelrc` in your workspace directory (next to the main `MODULE.bazel` file). -3. **The home RC file**, unless `--nohome_rc` is present. + It is not an error if this file does not exist. - Path: +3. **The home RC file**, unless `--nohome_rc` is present. - - On Linux/macOS/Unixes: `$HOME/.bazelrc` - - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` + Path: - It is not an error if this file does not exist. + - On Linux/macOS/Unixes: `$HOME/.bazelrc` + - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` -4. **The environment variable RC file**, if its path is set with the `BAZELRC` - environment variable. + It is not an error if this file does not exist. - The environment variable can include multiple comma-separated paths. +4. **The environment variable RC file**, if its path is set with the `BAZELRC` environment variable. -5. **The user-specified RC file**, if specified with - --bazelrc=file + The environment variable can include multiple comma-separated paths. - This flag is optional but can also be specified multiple times. +5. **The user-specified RC file**, if specified with `--bazelrc=file` - `/dev/null` indicates that all further `--bazelrc`s will be ignored, which - is useful to disable the search for a user rc file, such as in release - builds. + This flag is optional but can also be specified multiple times. - For example: + `/dev/null` indicates that all further `--bazelrc`s will be ignored, which is useful to disable the search for a user rc file, such as in release builds. - ``` - --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc - ``` + For example: - - `x.rc` and `y.rc` are read. - - `z.rc` is ignored due to the prior `/dev/null`. + ``` + --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc + ``` -In addition to this optional configuration file, Bazel looks for a global rc -file. For more details, see the [global bazelrc section](#global-bazelrc). + - `x.rc` and `y.rc` are read. + - `z.rc` is ignored due to the prior `/dev/null`. +In addition to this optional configuration file, Bazel looks for a global rc file. For more details, see the [global bazelrc section](#global-bazelrc). ### `.bazelrc` syntax and semantics -Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based -grammar. Empty lines and lines starting with `#` (comments) are ignored. Each -line contains a sequence of words, which are tokenized according to the same -rules as the Bourne shell. +Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based grammar. Empty lines and lines starting with `#` (comments) are ignored. Each line contains a sequence of words, which are tokenized according to the same rules as the Bourne shell. #### Imports -Lines that start with `import` or `try-import` are special: use these to load -other "rc" files. To specify a path that is relative to the workspace root, -write `import %workspace%/path/to/bazelrc`. +Lines that start with `import` or `try-import` are special: use these to load other "rc" files. To specify a path that is relative to the workspace root, write `import %workspace%/path/to/bazelrc`. -The difference between `import` and `try-import` is that Bazel fails if the -`import`'ed file is missing (or can't be read), but not so for a `try-import`'ed -file. +The difference between `import` and `try-import` is that Bazel fails if the `import`'ed file is missing (or can't be read), but not so for a `try-import`'ed file. Import precedence: -- Options in the imported file take precedence over options specified before - the import statement. -- Options specified after the import statement take precedence over the - options in the imported file. -- Options in files imported later take precedence over files imported earlier. +- Options in the imported file take precedence over options specified before the import statement. +- Options specified after the import statement take precedence over the options in the imported file. +- Options in files imported later take precedence over files imported earlier. #### Option defaults -Most lines of a bazelrc define default option values. The first word on each -line specifies when these defaults are applied: - -- `startup`: startup options, which go before the command, and are described - in `bazel help startup_options`. -- `common`: options that should be applied to all Bazel commands that support - them. If a command does not support an option specified in this way, the - option is ignored so long as it is valid for *some* other Bazel command. - Note that this only applies to option names: If the current command accepts - an option with the specified name, but doesn't support the specified value, - it will fail. -- `always`: options that apply to all Bazel commands. If a command does not - support an option specified in this way, it will fail. -- _`command`_: Bazel command, such as `build` or `query` to which the options - apply. These options also apply to all commands that inherit from the - specified command. (For example, `test` inherits from `build`.) - -Each of these lines may be used more than once and the arguments that follow the -first word are combined as if they had appeared on a single line. (Users of CVS, -another tool with a "Swiss army knife" command-line interface, will find the -syntax similar to that of `.cvsrc`.) For example, the lines: +Most lines of a bazelrc define default option values. The first word on each line specifies when these defaults are applied: + +- `startup`: startup options, which go before the command, and are described in `bazel help startup_options`. +- `common`: options that should be applied to all Bazel commands that support them. If a command does not support an option specified in this way, the option is ignored so long as it is valid for *some* other Bazel command. Note that this only applies to option names: If the current command accepts an option with the specified name, but doesn't support the specified value, it will fail. +- `always`: options that apply to all Bazel commands. If a command does not support an option specified in this way, it will fail. +- *`command`*: Bazel command, such as `build` or `query` to which the options apply. These options also apply to all commands that inherit from the specified command. (For example, `test` inherits from `build`.) + +Each of these lines may be used more than once and the arguments that follow the first word are combined as if they had appeared on a single line. (Users of CVS, another tool with a "Swiss army knife" command-line interface, will find the syntax similar to that of `.cvsrc`.) For example, the lines: ```posix-terminal build --test_tmpdir=/tmp/foo --verbose_failures @@ -142,89 +98,49 @@ so the effective flags are `--verbose_failures` and `--test_tmpdir=/tmp/bar`. Option precedence: -- Options on the command line always take precedence over those in rc files. - For example, if a rc file says `build -c opt` but the command line flag is - `-c dbg`, the command line flag takes precedence. -- Within the rc file, precedence is governed by specificity: lines for a more - specific command take precedence over lines for a less specific command. +- Options on the command line always take precedence over those in rc files. For example, if a rc file says `build -c opt` but the command line flag is `-c dbg`, the command line flag takes precedence. - Specificity is defined by inheritance. Some commands inherit options from - other commands, making the inheriting command more specific than the base - command. For example `test` inherits from the `build` command, so all `bazel - build` flags are valid for `bazel test`, and all `build` lines apply also to - `bazel test` unless there's a `test` line for the same option. If the rc - file says: +- Within the rc file, precedence is governed by specificity: lines for a more specific command take precedence over lines for a less specific command. - ```posix-terminal - test -c dbg --test_env=PATH + Specificity is defined by inheritance. Some commands inherit options from other commands, making the inheriting command more specific than the base command. For example `test` inherits from the `build` command, so all `bazel build` flags are valid for `bazel test`, and all `build` lines apply also to `bazel test` unless there's a `test` line for the same option. If the rc file says: - build -c opt --verbose_failures - ``` + ```posix-terminal + test -c dbg --test_env=PATH - then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel - test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. + build -c opt --verbose_failures + ``` - The inheritance (specificity) graph is: + then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. - * Every command inherits from `common` - * The following commands inherit from (and are more specific than) - `build`: `test`, `run`, `clean`, `mobile-install`, `info`, - `print_action`, `config`, `cquery`, and `aquery` - * `coverage`, `fetch`, and `vendor` inherit from `test` + The inheritance (specificity) graph is: -- Two lines specifying options for the same command at equal specificity are - parsed in the order in which they appear within the file. + - Every command inherits from `common` + - The following commands inherit from (and are more specific than) `build`: `test`, `run`, `clean`, `mobile-install`, `info`, `print_action`, `config`, `cquery`, and `aquery` + - `coverage`, `fetch`, and `vendor` inherit from `test` -- Because this precedence rule does not match the file order, it helps - readability if you follow the precedence order within rc files: start with - `common` options at the top, and end with the most-specific commands at the - bottom of the file. This way, the order in which the options are read is the - same as the order in which they are applied, which is more intuitive. +- Two lines specifying options for the same command at equal specificity are parsed in the order in which they appear within the file. -The arguments specified on a line of an rc file may include arguments that are -not options, such as the names of build targets, and so on. These, like the -options specified in the same files, have lower precedence than their siblings -on the command line, and are always prepended to the explicit list of non- -option arguments. +- Because this precedence rule does not match the file order, it helps readability if you follow the precedence order within rc files: start with `common` options at the top, and end with the most-specific commands at the bottom of the file. This way, the order in which the options are read is the same as the order in which they are applied, which is more intuitive. + +The arguments specified on a line of an rc file may include arguments that are not options, such as the names of build targets, and so on. These, like the options specified in the same files, have lower precedence than their siblings on the command line, and are always prepended to the explicit list of non- option arguments. #### `--config` -In addition to setting option defaults, the rc file can be used to group options -and provide a shorthand for common groupings. This is done by adding a `:name` -suffix to the command. These options are ignored by default, but will be -included when the option --config=name is present, -either on the command line or in a `.bazelrc` file, recursively, even inside of -another config definition. The options specified by `command:name` will only be -expanded for applicable commands, in the precedence order described above. - -Note: Configs can be defined in any `.bazelrc` file, and that all lines of -the form `command:name` (for applicable commands) will be expanded, across the -different rc files. In order to avoid name conflicts, we suggest that configs -defined in personal rc files start with an underscore (`_`) to avoid -unintentional name sharing. - -`--config=foo` expands to the options defined in -[the rc files](#bazelrc-file-locations) "in-place" so that the options -specified for the config have the same precedence that the `--config=foo` option -had. - -This syntax does not extend to the use of `startup` to set -[startup options](#option-defaults). Setting -`startup:config-name --some_startup_option` in the .bazelrc will be ignored. +In addition to setting option defaults, the rc file can be used to group options and provide a shorthand for common groupings. This is done by adding a `:name` suffix to the command. These options are ignored by default, but will be included when the option `--config=name` is present, either on the command line or in a `.bazelrc` file, recursively, even inside of another config definition. The options specified by `command:name` will only be expanded for applicable commands, in the precedence order described above. + +Note: Configs can be defined in any `.bazelrc` file, and that all lines of the form `command:name` (for applicable commands) will be expanded, across the different rc files. In order to avoid name conflicts, we suggest that configs defined in personal rc files start with an underscore (`_`) to avoid unintentional name sharing. + +`--config=foo` expands to the options defined in [the rc files](#bazelrc-file-locations) "in-place" so that the options specified for the config have the same precedence that the `--config=foo` option had. + +This syntax does not extend to the use of `startup` to set [startup options](#option-defaults). Setting `startup:config-name --some_startup_option` in the .bazelrc will be ignored. #### `--enable_platform_specific_config` -In the `.bazelrc` you can use platform specific configs that will be -automatically enabled based on the host OS. For example, if the host OS -is Linux and the `build` command is run, the `build:linux` configuration -will be automatically enabled. Supported OS identifiers are `linux`, -`macos`, `windows`, `freebsd`, and `openbsd`. +In the `.bazelrc` you can use platform specific configs that will be automatically enabled based on the host OS. For example, if the host OS is Linux and the `build` command is run, the `build:linux` configuration will be automatically enabled. Supported OS identifiers are `linux`, `macos`, `windows`, `freebsd`, and `openbsd`. -This is equivalent to using `--config=linux` on Linux, -`--config=windows` on Windows, and so on. This can be disabled with -`--enable_platform_specific_config=false`. +This is equivalent to using `--config=linux` on Linux, `--config=windows` on Windows, and so on. This can be disabled with `--enable_platform_specific_config=false`. -See [--enable_platform_specific_config](/reference/command-line-reference#flag--enable_platform_specific_config). +See [--enable\_platform\_specific\_config](/reference/command-line-reference#flag--enable_platform_specific_config). #### Example @@ -247,27 +163,16 @@ build:memcheck --strip=never --test_timeout=3600 #### `.bazelignore` -You can specify directories within the workspace -that you want Bazel to ignore, such as related projects -that use other build systems. Place a file called -`.bazelignore` at the root of the workspace -and add the directories you want Bazel to ignore, one per -line. Entries are relative to the workspace root. +You can specify directories within the workspace that you want Bazel to ignore, such as related projects that use other build systems. Place a file called `.bazelignore` at the root of the workspace and add the directories you want Bazel to ignore, one per line. Entries are relative to the workspace root. -The `.bazelignore` file does not permit glob semantics. -Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. -It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. -See [#24203](https://github.com/bazelbuild/bazel/pull/24203). +The `.bazelignore` file does not permit glob semantics. Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. See [#24203](https://github.com/bazelbuild/bazel/pull/24203). ### The global bazelrc file Bazel reads optional bazelrc files in this order: -1. System rc-file located at `/etc/bazel.bazelrc`. -2. Workspace rc-file located at `$workspace/tools/bazel.rc`. -3. Home rc-file located at `$HOME/.bazelrc` +1. System rc-file located at `/etc/bazel.bazelrc`. +2. Workspace rc-file located at `$workspace/tools/bazel.rc`. +3. Home rc-file located at `$HOME/.bazelrc` -Each bazelrc file listed here has a corresponding flag which can be used to -disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can -also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` -startup option. +Each bazelrc file listed here has a corresponding flag which can be used to disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` startup option. diff --git a/run/client-server.mdx b/run/client-server.mdx index 1868635f..b335b0ca 100644 --- a/run/client-server.mdx +++ b/run/client-server.mdx @@ -2,57 +2,21 @@ title: 'Client/server implementation' --- +The Bazel system is implemented as a long-lived server process. This allows it to perform many optimizations not possible with a batch-oriented implementation, such as caching of BUILD files, dependency graphs, and other metadata from one build to the next. This improves the speed of incremental builds, and allows different commands, such as `build` and `query` to share the same cache of loaded packages, making queries very fast. Each server can handle at most one invocation at a time; further concurrent invocations will either block or fail-fast (see `--block_for_lock`). +When you run `bazel`, you're running the client. The client finds the server based on the [output base](/run/scripts#output-base-option), which by default is determined by the path of the base workspace directory and your userid, so if you build in multiple workspaces, you'll have multiple output bases and thus multiple Bazel server processes. Multiple users on the same workstation can build concurrently in the same workspace because their output bases will differ (different userids). -The Bazel system is implemented as a long-lived server process. This allows it -to perform many optimizations not possible with a batch-oriented implementation, -such as caching of BUILD files, dependency graphs, and other metadata from one -build to the next. This improves the speed of incremental builds, and allows -different commands, such as `build` and `query` to share the same cache of -loaded packages, making queries very fast. Each server can handle at most one -invocation at a time; further concurrent invocations will either block or -fail-fast (see `--block_for_lock`). - -When you run `bazel`, you're running the client. The client finds the server -based on the [output base](/run/scripts#output-base-option), which by default is -determined by the path of the base workspace directory and your userid, so if -you build in multiple workspaces, you'll have multiple output bases and thus -multiple Bazel server processes. Multiple users on the same workstation can -build concurrently in the same workspace because their output bases will differ -(different userids). - -If the client cannot find a running server instance, it starts a new one. It -does this by checking if the output base already exists, implying the blaze -archive has already been unpacked. Otherwise if the output base doesn't exist, -the client unzips the archive's files and sets their `mtime`s to a date 9 years -in the future. Once installed, the client confirms that the `mtime`s of the -unzipped files are equal to the far off date to ensure no installation tampering -has occurred. - -The server process will stop after a period of inactivity (3 hours, by default, -which can be modified using the startup option `--max_idle_secs`). For the most -part, the fact that there is a server running is invisible to the user, but -sometimes it helps to bear this in mind. For example, if you're running scripts -that perform a lot of automated builds in different directories, it's important -to ensure that you don't accumulate a lot of idle servers; you can do this by -explicitly shutting them down when you're finished with them, or by specifying -a short timeout period. - -The name of a Bazel server process appears in the output of `ps x` or `ps -e f` -as bazel(dirname), where _dirname_ is the basename of the -directory enclosing the root of your workspace directory. For example: +If the client cannot find a running server instance, it starts a new one. It does this by checking if the output base already exists, implying the blaze archive has already been unpacked. Otherwise if the output base doesn't exist, the client unzips the archive's files and sets their `mtime`s to a date 9 years in the future. Once installed, the client confirms that the `mtime`s of the unzipped files are equal to the far off date to ensure no installation tampering has occurred. + +The server process will stop after a period of inactivity (3 hours, by default, which can be modified using the startup option `--max_idle_secs`). For the most part, the fact that there is a server running is invisible to the user, but sometimes it helps to bear this in mind. For example, if you're running scripts that perform a lot of automated builds in different directories, it's important to ensure that you don't accumulate a lot of idle servers; you can do this by explicitly shutting them down when you're finished with them, or by specifying a short timeout period. + +The name of a Bazel server process appears in the output of `ps x` or `ps -e f` as `bazel(dirname)`, where *dirname* is the basename of the directory enclosing the root of your workspace directory. For example: ```posix-terminal ps -e f 16143 ? Sl 3:00 bazel(src-johndoe2) -server -Djava.library.path=... ``` -This makes it easier to find out which server process belongs to a given -workspace. (Beware that with certain other options to `ps`, Bazel server -processes may be named just `java`.) Bazel servers can be stopped using the -[shutdown](/docs/user-manual#shutdown) command. +This makes it easier to find out which server process belongs to a given workspace. (Beware that with certain other options to `ps`, Bazel server processes may be named just `java`.) Bazel servers can be stopped using the [shutdown](/docs/user-manual#shutdown) command. -When running `bazel`, the client first checks that the server is the appropriate -version; if not, the server is stopped and a new one started. This ensures that -the use of a long-running server process doesn't interfere with proper -versioning. +When running `bazel`, the client first checks that the server is the appropriate version; if not, the server is stopped and a new one started. This ensures that the use of a long-running server process doesn't interfere with proper versioning. diff --git a/run/scripts.mdx b/run/scripts.mdx index f267c903..25cb1f65 100644 --- a/run/scripts.mdx +++ b/run/scripts.mdx @@ -2,129 +2,93 @@ title: 'Calling Bazel from scripts' --- +You can call Bazel from scripts to perform a build, run tests, or query the dependency graph. Bazel has been designed to enable effective scripting, but this section lists some details to bear in mind to make your scripts more robust. +### Choosing the output base -You can call Bazel from scripts to perform a build, run tests, or query -the dependency graph. Bazel has been designed to enable effective scripting, but -this section lists some details to bear in mind to make your scripts more -robust. +The `--output_base` option controls where the Bazel process should write the outputs of a build to, as well as various working files used internally by Bazel, one of which is a lock that guards against concurrent mutation of the output base by multiple Bazel processes. -### Choosing the output base +Choosing the correct output base directory for your script depends on several factors. If you need to put the build outputs in a specific location, this will dictate the output base you need to use. If you are making a "read only" call to Bazel (such as `bazel query`), the locking factors will be more important. In particular, if you need to run multiple instances of your script concurrently, you should be mindful that each Blaze server process can handle at most one invocation [at a time](/run/client-server#clientserver-implementation). Depending on your situation it may make sense for each instance of your script to wait its turn, or it may make sense to use `--output_base` to run multiple Blaze servers and use those. -The `--output_base` option controls where the Bazel process should write the -outputs of a build to, as well as various working files used internally by -Bazel, one of which is a lock that guards against concurrent mutation of the -output base by multiple Bazel processes. - -Choosing the correct output base directory for your script depends on several -factors. If you need to put the build outputs in a specific location, this will -dictate the output base you need to use. If you are making a "read only" call to -Bazel (such as `bazel query`), the locking factors will be more important. In -particular, if you need to run multiple instances of your script concurrently, -you should be mindful that each Blaze server process can handle at most one -invocation [at a time](/run/client-server#clientserver-implementation). -Depending on your situation it may make sense for each instance of your script -to wait its turn, or it may make sense to use `--output_base` to run multiple -Blaze servers and use those. - -If you use the default output base value, you will be contending for the same -lock used by the user's interactive Bazel commands. If the user issues -long-running commands such as builds, your script will have to wait for those -commands to complete before it can continue. +If you use the default output base value, you will be contending for the same lock used by the user's interactive Bazel commands. If the user issues long-running commands such as builds, your script will have to wait for those commands to complete before it can continue. ### Notes about server mode -By default, Bazel uses a long-running [server process](/run/client-server) as an -optimization. When running Bazel in a script, don't forget to call `shutdown` -when you're finished with the server, or, specify `--max_idle_secs=5` so that -idle servers shut themselves down promptly. +By default, Bazel uses a long-running [server process](/run/client-server) as an optimization. When running Bazel in a script, don't forget to call `shutdown` when you're finished with the server, or, specify `--max_idle_secs=5` so that idle servers shut themselves down promptly. ### What exit code will I get? -Bazel attempts to differentiate failures due to the source code under -consideration from external errors that prevent Bazel from executing properly. -Bazel execution can result in following exit codes: +Bazel attempts to differentiate failures due to the source code under consideration from external errors that prevent Bazel from executing properly. Bazel execution can result in following exit codes: **Exit Codes common to all commands:** -- `0` - Success -- `2` - Command Line Problem, Bad or Illegal flags or command combination, or - Bad Environment Variables. Your command line must be modified. -- `8` - Build Interrupted but we terminated with an orderly shutdown. -- `9` - The server lock is held and `--noblock_for_lock` was passed. -- `32` - External Environment Failure not on this machine. - -- `33` - Bazel ran out of memory and crashed. You need to modify your command line. -- `34` - Reserved for Google-internal use. -- `35` - Reserved for Google-internal use. -- `36` - Local Environmental Issue, suspected permanent. -- `37` - Unhandled Exception / Internal Bazel Error. -- `38` - Transient error publishing results to the Build Event Service. -- `39` - Blobs required by Bazel are evicted from Remote Cache. -- `41-44` - Reserved for Google-internal use. -- `45` - Persistent error publishing results to the Build Event Service. -- `47` - Reserved for Google-internal use. -- `49` - Reserved for Google-internal use. +- `0` - Success -**Return codes for commands `bazel build`, `bazel test`:** +- `2` - Command Line Problem, Bad or Illegal flags or command combination, or Bad Environment Variables. Your command line must be modified. + +- `8` - Build Interrupted but we terminated with an orderly shutdown. + +- `9` - The server lock is held and `--noblock_for_lock` was passed. + +- `32` - External Environment Failure not on this machine. + +- `33` - Bazel ran out of memory and crashed. You need to modify your command line. + +- `34` - Reserved for Google-internal use. + +- `35` - Reserved for Google-internal use. + +- `36` - Local Environmental Issue, suspected permanent. -- `1` - Build failed. -- `3` - Build OK, but some tests failed or timed out. -- `4` - Build successful but no tests were found even though testing was - requested. +- `37` - Unhandled Exception / Internal Bazel Error. +- `38` - Transient error publishing results to the Build Event Service. + +- `39` - Blobs required by Bazel are evicted from Remote Cache. + +- `41-44` - Reserved for Google-internal use. + +- `45` - Persistent error publishing results to the Build Event Service. + +- `47` - Reserved for Google-internal use. + +- `49` - Reserved for Google-internal use. + +**Return codes for commands `bazel build`, `bazel test`:** + +- `1` - Build failed. +- `3` - Build OK, but some tests failed or timed out. +- `4` - Build successful but no tests were found even though testing was requested. **For `bazel run`:** -- `1` - Build failed. -- If the build succeeds but the executed subprocess returns a non-zero exit - code it will be the exit code of the command as well. +- `1` - Build failed. +- If the build succeeds but the executed subprocess returns a non-zero exit code it will be the exit code of the command as well. **For `bazel query`:** -- `3` - Partial success, but the query encountered 1 or more errors in the - input BUILD file set and therefore the results of the operation are not 100% - reliable. This is likely due to a `--keep_going` option on the command line. -- `7` - Command failure. - -Future Bazel versions may add additional exit codes, replacing generic failure -exit code `1` with a different non-zero value with a particular meaning. -However, all non-zero exit values will always constitute an error. +- `3` - Partial success, but the query encountered 1 or more errors in the input BUILD file set and therefore the results of the operation are not 100% reliable. This is likely due to a `--keep_going` option on the command line. +- `7` - Command failure. +Future Bazel versions may add additional exit codes, replacing generic failure exit code `1` with a different non-zero value with a particular meaning. However, all non-zero exit values will always constitute an error. ### Reading the .bazelrc file -By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base -workspace directory or the user's home directory. Whether or not this is -desirable is a choice for your script; if your script needs to be perfectly -hermetic (such as when doing release builds), you should disable reading the -.bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform -a build using the user's preferred settings, the default behavior is better. +By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base workspace directory or the user's home directory. Whether or not this is desirable is a choice for your script; if your script needs to be perfectly hermetic (such as when doing release builds), you should disable reading the .bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform a build using the user's preferred settings, the default behavior is better. ### Command log -The Bazel output is also available in a command log file which you can find with -the following command: +The Bazel output is also available in a command log file which you can find with the following command: ```posix-terminal bazel info command_log ``` -The command log file contains the interleaved stdout and stderr streams of the -most recent Bazel command. Note that running `bazel info` will overwrite the -contents of this file, since it then becomes the most recent Bazel command. -However, the location of the command log file will not change unless you change -the setting of the `--output_base` or `--output_user_root` options. +The command log file contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. ### Parsing output -The Bazel output is quite easy to parse for many purposes. Two options that may -be helpful for your script are `--noshow_progress` which suppresses progress -messages, and --show_result n, which controls whether or -not "build up-to-date" messages are printed; these messages may be parsed to -discover which targets were successfully built, and the location of the output -files they created. Be sure to specify a very large value of _n_ if you rely on -these messages. +The Bazel output is quite easy to parse for many purposes. Two options that may be helpful for your script are `--noshow_progress` which suppresses progress messages, and `--show_result n`, which controls whether or not "build up-to-date" messages are printed; these messages may be parsed to discover which targets were successfully built, and the location of the output files they created. Be sure to specify a very large value of *n* if you rely on these messages. ## Troubleshooting performance by profiling diff --git a/start/android-app.mdx b/start/android-app.mdx index 35da5828..3ae739fd 100644 --- a/start/android-app.mdx +++ b/start/android-app.mdx @@ -2,36 +2,22 @@ title: 'Bazel Tutorial: Build an Android App' --- - -**Note:** There are known limitations on using Bazel for building Android apps. -Visit the [rules_android issues page](https://github.com/bazelbuild/rules_android/issues) -to see the list of known issues. While the Bazel team and Open Source Software -(OSS) contributors work actively to address known issues, users should be aware -that Android Studio does not officially support Bazel projects. - This tutorial covers how to build a simple Android app using Bazel. -Bazel supports building Android apps using the -[Android rules](/reference/be/android). +Bazel supports building Android apps using the [Android rules](/reference/be/android). -This tutorial is intended for Windows, macOS and Linux users and does not -require experience with Bazel or Android app development. You do not need to -write any Android code in this tutorial. +This tutorial is intended for Windows, macOS and Linux users and does not require experience with Bazel or Android app development. You do not need to write any Android code in this tutorial. ## What you'll learn In this tutorial you learn how to: -* Set up your environment by installing Bazel and Android Studio, and - downloading the sample project. -* Set up a Bazel workspace that contains the source code - for the app and a `MODULE.bazel` file that identifies the top level of the - workspace directory. -* Update the `MODULE.bazel` file to contain references to the required - external dependencies, like the Android SDK. -* Create a `BUILD` file. -* Build the app with Bazel. -* Deploy and run the app on an Android emulator or physical device. +- Set up your environment by installing Bazel and Android Studio, and downloading the sample project. +- Set up a Bazel workspace that contains the source code for the app and a `MODULE.bazel` file that identifies the top level of the workspace directory. +- Update the `MODULE.bazel` file to contain references to the required external dependencies, like the Android SDK. +- Create a `BUILD` file. +- Build the app with Bazel. +- Deploy and run the app on an Android emulator or physical device. ## Before you begin @@ -39,16 +25,13 @@ In this tutorial you learn how to: Before you begin the tutorial, install the following software: -* **Bazel.** To install, follow the [installation instructions](/install). -* **Android Studio.** To install, follow the steps to [download Android - Studio](https://developer.android.com/sdk/index.html). - Execute the setup wizard to download the SDK and configure your environment. -* (Optional) **Git.** Use `git` to download the Android app project. +- **Bazel.** To install, follow the [installation instructions](/install). +- **Android Studio.** To install, follow the steps to [download Android Studio](https://developer.android.com/sdk/index.html). Execute the setup wizard to download the SDK and configure your environment. +- (Optional) **Git.** Use `git` to download the Android app project. ### Get the sample project -For the sample project, use the tutorial Android app project in -[Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). +For the sample project, use the tutorial Android app project in [Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). This app has a single button that prints a greeting when clicked: @@ -56,15 +39,13 @@ This app has a single button that prints a greeting when clicked: **Figure 1.** Android app button greeting. -Clone the repository with `git` (or [download the ZIP file -directly](https://github.com/bazelbuild/examples/archive/master.zip)): +Clone the repository with `git` (or [download the ZIP file directly](https://github.com/bazelbuild/examples/archive/master.zip)): ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in `examples/android/tutorial`. For -the rest of the tutorial, you will be executing commands in this directory. +The sample project for this tutorial is in `examples/android/tutorial`. For the rest of the tutorial, you will be executing commands in this directory. ### Review the source files @@ -99,24 +80,20 @@ The key files and directories are: | Android source files | `src/main/java/com/example/bazel/MainActivity.java` and `Greeter.java` | | Resource file directory | `src/main/java/com/example/bazel/res/` | - ## Build with Bazel ### Set up the workspace -A [workspace](/concepts/build-ref#workspace) is a directory that contains the -source files for one or more software projects, and has a `MODULE.bazel` file at -its root. +A [workspace](/concepts/build-ref#workspace) is a directory that contains the source files for one or more software projects, and has a `MODULE.bazel` file at its root. -The `MODULE.bazel` file may be empty or may contain references to [external -dependencies](/external/overview) required to build your project. +The `MODULE.bazel` file may be empty or may contain references to [external dependencies](/external/overview) required to build your project. First, run the following command to create an empty `MODULE.bazel` file: -| OS | Command | -| ------------------------ | ----------------------------------- | +| OS | Command | +| ------------------------ | -------------------------------------- | | Linux, macOS | `touch MODULE.bazel` | -| Windows (Command Prompt) | `type nul > MODULE.bazel` | +| Windows (Command Prompt) | `type nul > MODULE.bazel` | | Windows (PowerShell) | `New-Item MODULE.bazel -ItemType file` | ### Running Bazel @@ -127,8 +104,7 @@ You can now check if Bazel is running correctly with the command: bazel info workspace ``` -If Bazel prints the path of the current directory, you're good to go! If the -`MODULE.bazel` file does not exist, you may see an error message like: +If Bazel prints the path of the current directory, you're good to go! If the `MODULE.bazel` file does not exist, you may see an error message like: ``` ERROR: The 'info' command is only supported from within a workspace. @@ -136,10 +112,7 @@ ERROR: The 'info' command is only supported from within a workspace. ### Integrate with the Android SDK -Bazel needs to run the Android SDK -[build tools](https://developer.android.com/tools/revisions/build-tools.html) -to build the app. This means that you need to add some information to your -`MODULE.bazel` file so that Bazel knows where to find them. +Bazel needs to run the Android SDK [build tools](https://developer.android.com/tools/revisions/build-tools.html) to build the app. This means that you need to add some information to your `MODULE.bazel` file so that Bazel knows where to find them. Add the following line to your `MODULE.bazel` file: @@ -155,94 +128,53 @@ android_sdk_repository_extension = use_extension("@rules_android//rules/android_ use_repo(android_sdk_repository_extension, "androidsdk") ``` -This will use the Android SDK at the path referenced by the `ANDROID_HOME` -environment variable, and automatically detect the highest API level and the -latest version of build tools installed within that location. +This will use the Android SDK at the path referenced by the `ANDROID_HOME` environment variable, and automatically detect the highest API level and the latest version of build tools installed within that location. -You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find -the path to the installed SDK using Android Studio's [SDK -Manager](https://developer.android.com/studio/intro/update#sdk-manager). -Assuming the SDK is installed to default locations, you can use the following -commands to set the `ANDROID_HOME` variable: +You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find the path to the installed SDK using Android Studio's [SDK Manager](https://developer.android.com/studio/intro/update#sdk-manager). Assuming the SDK is installed to default locations, you can use the following commands to set the `ANDROID_HOME` variable: -| OS | Command | +| OS | Command | | ------------------------ | --------------------------------------------------- | | Linux | `export ANDROID_HOME=$HOME/Android/Sdk/` | | macOS | `export ANDROID_HOME=$HOME/Library/Android/sdk` | | Windows (Command Prompt) | `set ANDROID_HOME=%LOCALAPPDATA%\Android\Sdk` | | Windows (PowerShell) | `$env:ANDROID_HOME="$env:LOCALAPPDATA\Android\Sdk"` | -The above commands set the variable only for the current shell session. To make -them permanent, run the following commands: +The above commands set the variable only for the current shell session. To make them permanent, run the following commands: -| OS | Command | -| ------------------------ | --------------------------------------------------- | -| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | -| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | +| OS | Command | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | +| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | +| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | | Windows (Command Prompt) | `setx ANDROID_HOME "%LOCALAPPDATA%\Android\Sdk"` | | Windows (PowerShell) | `[System.Environment]::SetEnvironmentVariable('ANDROID_HOME', "$env:LOCALAPPDATA\Android\Sdk", [System.EnvironmentVariableTarget]::User)` | - -**Optional:** If you want to compile native code into your Android app, you -also need to download the [Android -NDK](https://developer.android.com/ndk/downloads/index.html) -and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: +**Optional:** If you want to compile native code into your Android app, you also need to download the [Android NDK](https://developer.android.com/ndk/downloads/index.html) and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: ```python bazel_dep(name = "rules_android_ndk", version = "0.1.3") ``` +For more information, read [Using the Android Native Development Kit with Bazel](/docs/android-ndk). -For more information, read [Using the Android Native Development Kit with -Bazel](/docs/android-ndk). - -It's not necessary to set the API levels to the same value for the SDK and NDK. -[This page](https://developer.android.com/ndk/guides/stable_apis.html) -contains a map from Android releases to NDK-supported API levels. +It's not necessary to set the API levels to the same value for the SDK and NDK. [This page](https://developer.android.com/ndk/guides/stable_apis.html) contains a map from Android releases to NDK-supported API levels. ### Create a BUILD file -A [`BUILD` file](/concepts/build-files) describes the relationship -between a set of build outputs, like compiled Android resources from `aapt` or -class files from `javac`, and their dependencies. These dependencies may be -source files (Java, C++) in your workspace or other build outputs. `BUILD` files -are written in a language called **Starlark**. - -`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. -The package hierarchy is a logical structure that overlays the directory -structure in your workspace. Each [package](/concepts/build-ref#packages) is a -directory (and its subdirectories) that contains a related set of source files -and a `BUILD` file. The package also includes any subdirectories, excluding -those that contain their own `BUILD` file. The *package name* is the path to the -`BUILD` file relative to the `MODULE.bazel` file. - -Note that Bazel's package hierarchy is conceptually different from the Java -package hierarchy of your Android App directory where the `BUILD` file is -located, although the directories may be organized identically. - -For the simple Android app in this tutorial, the source files in `src/main/` -comprise a single Bazel package. A more complex project may have many nested -packages. - -#### Add an android_library rule - -A `BUILD` file contains several different types of declarations for Bazel. The -most important type is the -[build rule](/concepts/build-files#types-of-build-rules), which tells -Bazel how to build an intermediate or final software output from a set of source -files or other dependencies. Bazel provides two build rules, -[`android_library`](/reference/be/android#android_library) and -[`android_binary`](/reference/be/android#android_binary), that you can use to -build an Android app. - -For this tutorial, you'll first use the -`android_library` rule to tell Bazel to build an [Android library -module](http://developer.android.com/tools/projects/index.html#LibraryProjects) -from the app source code and resource files. You'll then use the -`android_binary` rule to tell Bazel how to build the Android application package. - -Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, -and declare a new `android_library` target: +A [`BUILD` file](/concepts/build-files) describes the relationship between a set of build outputs, like compiled Android resources from `aapt` or class files from `javac`, and their dependencies. These dependencies may be source files (Java, C++) in your workspace or other build outputs. `BUILD` files are written in a language called **Starlark**. + +`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. The package hierarchy is a logical structure that overlays the directory structure in your workspace. Each [package](/concepts/build-ref#packages) is a directory (and its subdirectories) that contains a related set of source files and a `BUILD` file. The package also includes any subdirectories, excluding those that contain their own `BUILD` file. The *package name* is the path to the `BUILD` file relative to the `MODULE.bazel` file. + +Note that Bazel's package hierarchy is conceptually different from the Java package hierarchy of your Android App directory where the `BUILD` file is located, although the directories may be organized identically. + +For the simple Android app in this tutorial, the source files in `src/main/` comprise a single Bazel package. A more complex project may have many nested packages. + +#### Add an android\_library rule + +A `BUILD` file contains several different types of declarations for Bazel. The most important type is the [build rule](/concepts/build-files#types-of-build-rules), which tells Bazel how to build an intermediate or final software output from a set of source files or other dependencies. Bazel provides two build rules, [`android_library`](/reference/be/android#android_library) and [`android_binary`](/reference/be/android#android_binary), that you can use to build an Android app. + +For this tutorial, you'll first use the `android_library` rule to tell Bazel to build an [Android library module](http://developer.android.com/tools/projects/index.html#LibraryProjects) from the app source code and resource files. You'll then use the `android_binary` rule to tell Bazel how to build the Android application package. + +Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, and declare a new `android_library` target: `src/main/java/com/example/bazel/BUILD`: @@ -264,18 +196,13 @@ android_library( ) ``` -The `android_library` build rule contains a set of attributes that specify the -information that Bazel needs to build a library module from the source files. -Note also that the name of the rule is `greeter_activity`. You'll reference the -rule using this name as a dependency in the `android_binary` rule. +The `android_library` build rule contains a set of attributes that specify the information that Bazel needs to build a library module from the source files. Note also that the name of the rule is `greeter_activity`. You'll reference the rule using this name as a dependency in the `android_binary` rule. -#### Add an android_binary rule +#### Add an android\_binary rule -The [`android_binary`](/reference/be/android#android_binary) rule builds -the Android application package (`.apk` file) for your app. +The [`android_binary`](/reference/be/android#android_binary) rule builds the Android application package (`.apk` file) for your app. -Create a new `BUILD` file in the `src/main/` directory, -and declare a new `android_binary` target: +Create a new `BUILD` file in the `src/main/` directory, and declare a new `android_binary` target: `src/main/BUILD`: @@ -289,35 +216,23 @@ android_binary( ) ``` -Here, the `deps` attribute references the output of the `greeter_activity` rule -you added to the `BUILD` file above. This means that when Bazel builds the -output of this rule it checks first to see if the output of the -`greeter_activity` library rule has been built and is up-to-date. If not, Bazel -builds it and then uses that output to build the application package file. +Here, the `deps` attribute references the output of the `greeter_activity` rule you added to the `BUILD` file above. This means that when Bazel builds the output of this rule it checks first to see if the output of the `greeter_activity` library rule has been built and is up-to-date. If not, Bazel builds it and then uses that output to build the application package file. Now, save and close the file. ### Build the app -Try building the app! Run the following command to build the -`android_binary` target: +Try building the app! Run the following command to build the `android_binary` target: ```posix-terminal bazel build //src/main:app ``` -The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the -target that follows. The target is specified as the name of a build rule inside -a `BUILD` file, with along with the package path relative to your workspace -directory. For this example, the target is `app` and the package path is -`//src/main/`. +The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the target that follows. The target is specified as the name of a build rule inside a `BUILD` file, with along with the package path relative to your workspace directory. For this example, the target is `app` and the package path is `//src/main/`. -Note that you can sometimes omit the package path or target name, depending on -your current working directory at the command line and the name of the target. -For more details about target labels and paths, see [Labels](/concepts/labels). +Note that you can sometimes omit the package path or target name, depending on your current working directory at the command line and the name of the target. For more details about target labels and paths, see [Labels](/concepts/labels). -Bazel will start to build the sample app. During the build process, its output -will appear similar to the following: +Bazel will start to build the sample app. During the build process, its output will appear similar to the following: ```bash INFO: Analysed target //src/main:app (0 packages loaded, 0 targets configured). @@ -330,40 +245,25 @@ Target //src/main:app up-to-date: #### Locate the build outputs -Bazel puts the outputs of both intermediate and final build operations in a set -of per-user, per-workspace output directories. These directories are symlinked -from the following locations at the top-level of the project directory, where -the `MODULE.bazel` file is: +Bazel puts the outputs of both intermediate and final build operations in a set of per-user, per-workspace output directories. These directories are symlinked from the following locations at the top-level of the project directory, where the `MODULE.bazel` file is: -* `bazel-bin` stores binary executables and other runnable build outputs -* `bazel-genfiles` stores intermediary source files that are generated by - Bazel rules -* `bazel-out` stores other types of build outputs +- `bazel-bin` stores binary executables and other runnable build outputs +- `bazel-genfiles` stores intermediary source files that are generated by Bazel rules +- `bazel-out` stores other types of build outputs -Bazel stores the Android `.apk` file generated using the `android_binary` rule -in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is -derived from the name of the Bazel package. +Bazel stores the Android `.apk` file generated using the `android_binary` rule in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is derived from the name of the Bazel package. -At a command prompt, list the contents of this directory and find the `app.apk` -file: +At a command prompt, list the contents of this directory and find the `app.apk` file: -| OS | Command | +| OS | Command | | ------------------------ | ------------------------ | | Linux, macOS | `ls bazel-bin/src/main` | | Windows (Command Prompt) | `dir bazel-bin\src\main` | | Windows (PowerShell) | `ls bazel-bin\src\main` | - ### Run the app -You can now deploy the app to a connected Android device or emulator from the -command line using `bazel mobile-install`. -This command uses the Android Debug Bridge (`adb`) to communicate with the -device. You must set up your device to use `adb` following the instructions in -[Android Debug Bridge](http://developer.android.com/tools/help/adb.html) -before deployment. You can also choose to install the app on the Android emulator -included in Android Studio. Make sure the emulator is running before executing -the command below. +You can now deploy the app to a connected Android device or emulator from the command line using `bazel mobile-install`. This command uses the Android Debug Bridge (`adb`) to communicate with the device. You must set up your device to use `adb` following the instructions in [Android Debug Bridge](http://developer.android.com/tools/help/adb.html) before deployment. You can also choose to install the app on the Android emulator included in Android Studio. Make sure the emulator is running before executing the command below. Enter the following: @@ -378,12 +278,7 @@ bazel mobile-install //src/main:app \ --tool_java_language_version=17 ``` -Note that the extra flags required for mobile-install can be added to your -project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags -(`--mode`, `--mobile_install*`) will no longer be required starting from -Bazel 8.4.0 and onwards. The various Java flags for language and runtime version -may be required depending on your workspace's Java configuration. -_Mobile-install sub-tools require a language and runtime level of 17 or higher._ +Note that the extra flags required for mobile-install can be added to your project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags (`--mode`, `--mobile_install*`) will no longer be required starting from Bazel 8.4.0 and onwards. The various Java flags for language and runtime version may be required depending on your workspace's Java configuration. *Mobile-install sub-tools require a language and runtime level of 17 or higher.* Now the "Bazel Tutorial App" should install and launch automatically: @@ -397,16 +292,20 @@ Now the "Bazel Tutorial App" should install and launch automatically: For more details, see these pages: -* Open issues on [rules_android GitHub](https://github.com/bazelbuild/rules_android/issues) -* More information on [mobile-install](/docs/mobile-install) -* Integrate external dependencies like AppCompat, Guava and JUnit from Maven - repositories using [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) -* Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) - integration. -* Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) -* See more Bazel example projects of: - * [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) - * [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) - * [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) +- Open issues on [rules\_android GitHub](https://github.com/bazelbuild/rules_android/issues) + +- More information on [mobile-install](/docs/mobile-install) + +- Integrate external dependencies like AppCompat, Guava and JUnit from Maven repositories using [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) + +- Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) integration. + +- Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) + +- See more Bazel example projects of: + + - [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) + - [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) + - [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) Happy building! diff --git a/start/cpp.mdx b/start/cpp.mdx index adb7c71a..df9ec54e 100644 --- a/start/cpp.mdx +++ b/start/cpp.mdx @@ -2,37 +2,25 @@ title: 'Bazel Tutorial: Build a C++ Project' --- - - ## Introduction -New to Bazel? You're in the right place. Follow this First Build tutorial for a -simplified introduction to using Bazel. This tutorial defines key terms as they -are used in Bazel's context and walks you through the basics of the Bazel -workflow. Starting with the tools you need, you will build and run three -projects with increasing complexity and learn how and why they get more complex. +New to Bazel? You're in the right place. Follow this First Build tutorial for a simplified introduction to using Bazel. This tutorial defines key terms as they are used in Bazel's context and walks you through the basics of the Bazel workflow. Starting with the tools you need, you will build and run three projects with increasing complexity and learn how and why they get more complex. -While Bazel is a [build system](https://bazel.build/basics/build-systems) that -supports multi-language builds, this tutorial uses a C++ project as an example -and provides the general guidelines and flow that apply to most languages. +While Bazel is a [build system](https://bazel.build/basics/build-systems) that supports multi-language builds, this tutorial uses a C++ project as an example and provides the general guidelines and flow that apply to most languages. Estimated completion time: 30 minutes. ### Prerequisites -Start by [installing Bazel](https://bazel.build/install), if you haven't -already. This tutorial uses Git for source control, so for best results [install -Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. +Start by [installing Bazel](https://bazel.build/install), if you haven't already. This tutorial uses Git for source control, so for best results [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. -Next, retrieve the sample project from Bazel's GitHub repository by running the -following in your command-line tool of choice: +Next, retrieve the sample project from Bazel's GitHub repository by running the following in your command-line tool of choice: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/cpp-tutorial` -directory. +The sample project for this tutorial is in the `examples/cpp-tutorial` directory. Take a look at how it's structured: @@ -64,52 +52,24 @@ examples └── MODULE.bazel ``` -There are three sets of files, each set representing a stage in this tutorial. -In the first stage, you will build a single [target] -(https://bazel.build/reference/glossary#target) residing in a single [package] -(https://bazel.build/reference/glossary#package). In the second stage, you will -build both a binary and a library from a single package. In the third and final -stage, you will build a project with multiple packages and build it with -multiple targets. +There are three sets of files, each set representing a stage in this tutorial. In the first stage, you will build a single \[target] ([https://bazel.build/reference/glossary#target](https://bazel.build/reference/glossary#target)) residing in a single \[package] ([https://bazel.build/reference/glossary#package](https://bazel.build/reference/glossary#package)). In the second stage, you will build both a binary and a library from a single package. In the third and final stage, you will build a project with multiple packages and build it with multiple targets. ### Summary: Introduction -By installing Bazel (and Git) and cloning the repository for this tutorial, you -have laid the foundation for your first build with Bazel. Continue to the next -section to define some terms and set up your -[workspace](https://bazel.build/reference/glossary#workspace). +By installing Bazel (and Git) and cloning the repository for this tutorial, you have laid the foundation for your first build with Bazel. Continue to the next section to define some terms and set up your [workspace](https://bazel.build/reference/glossary#workspace). ## Getting started -Before you can build a project, you need to set up its workspace. A workspace -is a directory that holds your project's source files and Bazel's build outputs. -It also contains these significant files: +Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains these significant files: -* The `MODULE.bazel` file, which identifies the directory and its contents as - a Bazel workspace and lives at the root of the project's directory - structure. It's also where you specify your external dependencies. -* One or more [`BUILD` - files](https://bazel.build/reference/glossary#build-file), which tell Bazel - how to build different parts of the project. A directory within the - workspace that contains a `BUILD` file is a - [package](https://bazel.build/reference/glossary#package). (More on packages - later in this tutorial.) +- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure. It's also where you specify your external dependencies. +- One or more [`BUILD` files](https://bazel.build/reference/glossary#build-file), which tell Bazel how to build different parts of the project. A directory within the workspace that contains a `BUILD` file is a [package](https://bazel.build/reference/glossary#package). (More on packages later in this tutorial.) -In future projects, to designate a directory as a Bazel workspace, create an -empty file named `MODULE.bazel` in that directory. For the purposes of this -tutorial, a `MODULE.bazel` file is already present in each stage. +In future projects, to designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. For the purposes of this tutorial, a `MODULE.bazel` file is already present in each stage. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. Each -`BUILD` file requires at least one -[rule](https://bazel.build/reference/glossary#rule) as a set of instructions, -which tells Bazel how to build the outputs you want, such as executable binaries -or libraries. Each instance of a build rule in the `BUILD` file is called a -[target](https://bazel.build/reference/glossary#target) and points to a specific -set of source files and -[dependencies](https://bazel.build/reference/glossary#dependency). A target can -also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. Each `BUILD` file requires at least one [rule](https://bazel.build/reference/glossary#rule) as a set of instructions, which tells Bazel how to build the outputs you want, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a [target](https://bazel.build/reference/glossary#target) and points to a specific set of source files and [dependencies](https://bazel.build/reference/glossary#dependency). A target can also point to other targets. Take a look at the `BUILD` file in the `cpp-tutorial/stage1/main` directory: @@ -120,21 +80,15 @@ cc_binary( ) ``` -In our example, the `hello-world` target instantiates Bazel's built-in -[`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule -tells Bazel to build a self-contained executable binary from the -`hello-world.cc`> source file with no dependencies. +In our example, the `hello-world` target instantiates Bazel's built-in [`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule tells Bazel to build a self-contained executable binary from the `hello-world.cc`\> source file with no dependencies. ### Summary: getting started -Now you are familiar with some key terms, and what they mean in the context of -this project and Bazel in general. In the next section, you will build and test -Stage 1 of the project. +Now you are familiar with some key terms, and what they mean in the context of this project and Bazel in general. In the next section, you will build and test Stage 1 of the project. ## Stage 1: single target, single package -It's time to build the first part of the project. For a visual reference, the -structure of the Stage 1 section of the project is: +It's time to build the first part of the project. For a visual reference, the structure of the Stage 1 section of the project is: ```none examples @@ -158,9 +112,7 @@ Next, run: bazel build //main:hello-world ``` -In the target label, the `//main:` part is the location of the `BUILD` file -relative to the root of the workspace, and `hello-world` is the target name in -the `BUILD` file. +In the target label, the `//main:` part is the location of the `BUILD` file relative to the root of the workspace, and `hello-world` is the target name in the `BUILD` file. Bazel produces something that looks like this: @@ -171,8 +123,7 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.267s, Critical Path: 0.25s ``` -You just built your first Bazel target. Bazel places build outputs in the -`bazel-bin` directory at the root of the workspace. +You just built your first Bazel target. Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Now test your freshly built binary, which is: @@ -184,23 +135,15 @@ This results in a printed "`Hello world`" message. Here's the dependency graph of Stage 1: -![Dependency graph for hello-world displays a single target with a single source -file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world -displays a single target with a single source file.") +![Dependency graph for hello-world displays a single target with a single source file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world displays a single target with a single source file.") ### Summary: stage 1 -Now that you have completed your first build, you have a basic idea of how a -build is structured. In the next stage, you will add complexity by adding -another target. +Now that you have completed your first build, you have a basic idea of how a build is structured. In the next stage, you will add complexity by adding another target. ## Stage 2: multiple build targets -While a single target is sufficient for small projects, you may want to split -larger projects into multiple targets and packages. This allows for fast -incremental builds – that is, Bazel only rebuilds what's changed – and speeds up -your builds by building multiple parts of a project at once. This stage of the -tutorial adds a target, and the next adds a package. +While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages. This allows for fast incremental builds – that is, Bazel only rebuilds what's changed – and speeds up your builds by building multiple parts of a project at once. This stage of the tutorial adds a target, and the next adds a package. This is the directory you are working with for Stage 2: @@ -232,15 +175,9 @@ cc_binary( ) ``` -With this `BUILD` file, Bazel first builds the `hello-greet` library (using -Bazel's built-in [`cc_library` -rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the -`hello-world` binary. The `deps` attribute in the `hello-world` target tells -Bazel that the `hello-greet` library is required to build the `hello-world` -binary. +With this `BUILD` file, Bazel first builds the `hello-greet` library (using Bazel's built-in [`cc_library` rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the `hello-world` binary. The `deps` attribute in the `hello-world` target tells Bazel that the `hello-greet` library is required to build the `hello-world` binary. -Before you can build this new version of the project, you need to change -directories, switching to the `cpp-tutorial/stage2` directory by running: +Before you can build this new version of the project, you need to change directories, switching to the `cpp-tutorial/stage2` directory by running: ```posix-terminal cd ../stage2 @@ -261,36 +198,25 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.399s, Critical Path: 0.30s ``` -Now you can test your freshly built binary, which returns another "`Hello -world`": +Now you can test your freshly built binary, which returns another "`Hello world`": ```posix-terminal bazel-bin/main/hello-world ``` -If you now modify `hello-greet.cc` and rebuild the project, Bazel only -recompiles that file. +If you now modify `hello-greet.cc` and rebuild the project, Bazel only recompiles that file. -Looking at the dependency graph, you can see that `hello-world` depends on an -extra input named `hello-greet`: +Looking at the dependency graph, you can see that `hello-world` depends on an extra input named `hello-greet`: -![Dependency graph for `hello-world` displays dependency changes after -modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency -graph for `hello-world` displays dependency changes after modification to the -file.") +![Dependency graph for hello-world displays dependency changes after modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency graph for `hello-world` displays dependency changes after modification to the file.") ### Summary: stage 2 -You've now built the project with two targets. The `hello-world` target builds -one source file and depends on one other target (`//main:hello-greet`), which -builds two additional source files. In the next section, take it a step further -and add another package. +You've now built the project with two targets. The `hello-world` target builds one source file and depends on one other target (`//main:hello-greet`), which builds two additional source files. In the next section, take it a step further and add another package. ## Stage 3: multiple packages -This next stage adds another layer of complication and builds a project with -multiple packages. Take a look at the structure and contents of the -`cpp-tutorial/stage3` directory: +This next stage adds another layer of complication and builds a project with multiple packages. Take a look at the structure and contents of the `cpp-tutorial/stage3` directory: ```none └──stage3 @@ -306,9 +232,7 @@ multiple packages. Take a look at the structure and contents of the └── MODULE.bazel ``` -You can see that now there are two sub-directories, and each contains a `BUILD` -file. Therefore, to Bazel, the workspace now contains two packages: `lib` and -`main`. +You can see that now there are two sub-directories, and each contains a `BUILD` file. Therefore, to Bazel, the workspace now contains two packages: `lib` and `main`. Take a look at the `lib/BUILD` file: @@ -340,25 +264,13 @@ cc_binary( ) ``` -The `hello-world` target in the main package depends on the` hello-time` target -in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows -this through the `deps` attribute. You can see this reflected in the dependency -graph: +The `hello-world` target in the main package depends on the` hello-time` target in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows this through the `deps` attribute. You can see this reflected in the dependency graph: -![Dependency graph for `hello-world` displays how the target in the main package -depends on the target in the `lib` -package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for -`hello-world` displays how the target in the main package depends on the target -in the `lib` package.") +![Dependency graph for hello-world displays how the target in the main package depends on the target in the lib package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for `hello-world` displays how the target in the main package depends on the target in the `lib` package.") -For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` -explicitly visible to targets in `main/BUILD` using the visibility attribute. -This is because by default targets are only visible to other targets in the same -`BUILD` file. Bazel uses target visibility to prevent issues such as libraries -containing implementation details leaking into public APIs. +For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` explicitly visible to targets in `main/BUILD` using the visibility attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs. -Now build this final version of the project. Switch to the `cpp-tutorial/stage3` -directory by running: +Now build this final version of the project. Switch to the `cpp-tutorial/stage3` directory by running: ```posix-terminal cd ../stage3 @@ -387,25 +299,15 @@ bazel-bin/main/hello-world ### Summary: stage 3 -You've now built the project as two packages with three targets and understand -the dependencies between them, which equips you to go forth and build future -projects with Bazel. In the next section, take a look at how to continue your -Bazel journey. +You've now built the project as two packages with three targets and understand the dependencies between them, which equips you to go forth and build future projects with Bazel. In the next section, take a look at how to continue your Bazel journey. ## Next steps -You've now completed your first basic build with Bazel, but this is just the -start. Here are some more resources to continue learning with Bazel: - -* To keep focusing on C++, read about common [C++ build use - cases](https://bazel.build/tutorials/cpp-use-cases). -* To get started with building other applications with Bazel, see the - tutorials for [Java](https://bazel.build/start/java), [Android - application](https://bazel.build/start/android-app), or [iOS - application](https://bazel.build/start/ios-app). -* To learn more about working with local and remote repositories, read about - [external dependencies](https://bazel.build/docs/external). -* To learn more about Bazel's other rules, see this [reference - guide](https://bazel.build/rules). +You've now completed your first basic build with Bazel, but this is just the start. Here are some more resources to continue learning with Bazel: + +- To keep focusing on C++, read about common [C++ build use cases](https://bazel.build/tutorials/cpp-use-cases). +- To get started with building other applications with Bazel, see the tutorials for [Java](https://bazel.build/start/java), [Android application](https://bazel.build/start/android-app), or [iOS application](https://bazel.build/start/ios-app). +- To learn more about working with local and remote repositories, read about [external dependencies](https://bazel.build/docs/external). +- To learn more about Bazel's other rules, see this [reference guide](https://bazel.build/rules). Happy building! diff --git a/start/go.mdx b/start/go.mdx index 3a095681..d5742314 100644 --- a/start/go.mdx +++ b/start/go.mdx @@ -2,12 +2,7 @@ title: 'Bazel Tutorial: Build a Go Project' --- - - -This tutorial introduces you to the basics of Bazel by showing you how to build -a Go (Golang) project. You'll learn how to set up your workspace, build a small -program, import a library, and run its test. Along the way, you'll learn key -Bazel concepts, such as targets and `BUILD` files. +This tutorial introduces you to the basics of Bazel by showing you how to build a Go (Golang) project. You'll learn how to set up your workspace, build a small program, import a library, and run its test. Along the way, you'll learn key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes @@ -15,35 +10,27 @@ Estimated completion time: 30 minutes ### Install Bazel -Before you get started, first [install bazel](/install) if you haven't done so -already. +Before you get started, first [install bazel](/install) if you haven't done so already. You can check if Bazel is installed by running `bazel version` in any directory. ### Install Go (optional) -You don't need to [install Go](https://go.dev/doc/install) to build Go projects -with Bazel. The Bazel Go rule set automatically downloads and uses a Go -toolchain instead of using the toolchain installed on your machine. This ensures -all developers on a project build with same version of Go. +You don't need to [install Go](https://go.dev/doc/install) to build Go projects with Bazel. The Bazel Go rule set automatically downloads and uses a Go toolchain instead of using the toolchain installed on your machine. This ensures all developers on a project build with same version of Go. -However, you may still want to install a Go toolchain to run commands like `go -get` and `go mod tidy`. +However, you may still want to install a Go toolchain to run commands like `go get` and `go mod tidy`. You can check if Go is installed by running `go version` in any directory. ### Get the sample project -The Bazel examples are stored in a Git repository, so you'll need to [install -Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you -haven't already. To download the examples repository, run this command: +The Bazel examples are stored in a Git repository, so you'll need to [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you haven't already. To download the examples repository, run this command: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/go-tutorial` directory. -See what it contains: +The sample project for this tutorial is in the `examples/go-tutorial` directory. See what it contains: ```none go-tutorial/ @@ -52,13 +39,11 @@ go-tutorial/ └── stage3 ``` -There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a -different section of this tutorial. Each stage builds on the previous one. +There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a different section of this tutorial. Each stage builds on the previous one. ## Build with Bazel -Start in the `stage1` directory, where we'll find a program. We can -build it with `bazel build`, then run it: +Start in the `stage1` directory, where we'll find a program. We can build it with `bazel build`, then run it: ```posix-shell $ cd go-tutorial/stage1/ @@ -107,9 +92,7 @@ func main() { } ``` -`BUILD` contains some instructions for Bazel, telling it what we want to build. -You'll typically write a file like this in each directory. For this project, we -have a single `go_binary` target that builds our program from `hello.go`. +`BUILD` contains some instructions for Bazel, telling it what we want to build. You'll typically write a file like this in each directory. For this project, we have a single `go_binary` target that builds our program from `hello.go`. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -120,17 +103,9 @@ go_binary( ) ``` -`MODULE.bazel` tracks your project's dependencies. It also marks your project's -root directory, so you'll only write one `MODULE.bazel` file per project. It -serves a similar purpose to Go's `go.mod` file. You don't actually need a -`go.mod` file in a Bazel project, but it may still be useful to have one so that -you can continue using `go get` and `go mod tidy` for dependency management. The -Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in -another tutorial. +`MODULE.bazel` tracks your project's dependencies. It also marks your project's root directory, so you'll only write one `MODULE.bazel` file per project. It serves a similar purpose to Go's `go.mod` file. You don't actually need a `go.mod` file in a Bazel project, but it may still be useful to have one so that you can continue using `go get` and `go mod tidy` for dependency management. The Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in another tutorial. -Our `MODULE.bazel` file contains a single dependency on -[rules_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need -this dependency because Bazel doesn't have built-in support for Go. +Our `MODULE.bazel` file contains a single dependency on [rules\_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need this dependency because Bazel doesn't have built-in support for Go. ```bazel bazel_dep( @@ -139,50 +114,25 @@ bazel_dep( ) ``` -Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes -and other metadata about our dependencies. It includes implicit dependencies -added by Bazel itself, so it's quite long, and we won't show it here. Just like -`go.sum`, you should commit your `MODULE.bazel.lock` file to source control to -ensure everyone on your project gets the same version of each dependency. You -shouldn't need to edit `MODULE.bazel.lock` manually. +Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes and other metadata about our dependencies. It includes implicit dependencies added by Bazel itself, so it's quite long, and we won't show it here. Just like `go.sum`, you should commit your `MODULE.bazel.lock` file to source control to ensure everyone on your project gets the same version of each dependency. You shouldn't need to edit `MODULE.bazel.lock` manually. ### Understand the BUILD file -Most of your interaction with Bazel will be through `BUILD` files (or -equivalently, `BUILD.bazel` files), so it's important to understand what they -do. +Most of your interaction with Bazel will be through `BUILD` files (or equivalently, `BUILD.bazel` files), so it's important to understand what they do. -`BUILD` files are written in a scripting language called -[Starlark](https://bazel.build/rules/language), a limited subset of Python. +`BUILD` files are written in a scripting language called [Starlark](https://bazel.build/rules/language), a limited subset of Python. -A `BUILD` file contains a list of -[targets](https://bazel.build/reference/glossary#target). A target is something -Bazel can build, like a binary, library, or test. +A `BUILD` file contains a list of [targets](https://bazel.build/reference/glossary#target). A target is something Bazel can build, like a binary, library, or test. -A target calls a rule function with a list of -[attributes](https://bazel.build/reference/glossary#attribute) to describe what -should be built. Our example has two attributes: `name` identifies the target on -the command line, and `srcs` is a list of source file paths (slash-separated, -relative to the directory containing the `BUILD` file). +A target calls a rule function with a list of [attributes](https://bazel.build/reference/glossary#attribute) to describe what should be built. Our example has two attributes: `name` identifies the target on the command line, and `srcs` is a list of source file paths (slash-separated, relative to the directory containing the `BUILD` file). -A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a -target. In our example, we used the -[`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) -rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) -(commands) that generate a set of output files. For example, `go_binary` defines -Go compile and link actions that produce an executable output file. +A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a target. In our example, we used the [`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) (commands) that generate a set of output files. For example, `go_binary` defines Go compile and link actions that produce an executable output file. -Bazel has built-in rules for a few languages like Java and C++. You can find -their [documentation in the Build -Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find -rule sets for many other languages and tools on the [Bazel Central Registry -(BCR)](https://registry.bazel.build/). +Bazel has built-in rules for a few languages like Java and C++. You can find their [documentation in the Build Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find rule sets for many other languages and tools on the [Bazel Central Registry (BCR)](https://registry.bazel.build/). ## Add a library -Move onto the `stage2` directory, where we'll build a new program that -prints your fortune. This program uses a separate Go package as a library that -selects a fortune from a predefined list of messages. +Move onto the `stage2` directory, where we'll build a new program that prints your fortune. This program uses a separate Go package as a library that selects a fortune from a predefined list of messages. ```none go-tutorial/stage2 @@ -195,11 +145,7 @@ go-tutorial/stage2 └── print_fortune.go ``` -`fortune.go` is the source file for the library. The `fortune` library is a -separate Go package, so its source files are in a separate directory. Bazel -doesn't require you to keep Go packages in separate directories, but it's a -strong convention in the Go ecosystem, and following it will help you stay -compatible with other Go tools. +`fortune.go` is the source file for the library. The `fortune` library is a separate Go package, so its source files are in a separate directory. Bazel doesn't require you to keep Go packages in separate directories, but it's a strong convention in the Go ecosystem, and following it will help you stay compatible with other Go tools. ```go package fortune @@ -217,19 +163,11 @@ func Get() string { } ``` -The `fortune` directory has its own `BUILD` file that tells Bazel how to build -this package. We use `go_library` here instead of `go_binary`. +The `fortune` directory has its own `BUILD` file that tells Bazel how to build this package. We use `go_library` here instead of `go_binary`. -We also need to set the `importpath` attribute to a string with which the -library can be imported into other Go source files. This name should be the -repository path (or module path) concatenated with the directory within the -repository. +We also need to set the `importpath` attribute to a string with which the library can be imported into other Go source files. This name should be the repository path (or module path) concatenated with the directory within the repository. -Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. -[`visibility`](https://bazel.build/concepts/visibility) may be set on any -target. It determines which Bazel packages may depend on this target. In our -case, we want any target to be able to depend on this library, so we use the -special value `//visibility:public`. +Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. [`visibility`](https://bazel.build/concepts/visibility) may be set on any target. It determines which Bazel packages may depend on this target. In our case, we want any target to be able to depend on this library, so we use the special value `//visibility:public`. ```bazel load("@rules_go//go:def.bzl", "go_library") @@ -264,11 +202,9 @@ func main() { } ``` -`print_fortune.go` imports the package using the same string declared in the -`importpath` attribute of the `fortune` library. +`print_fortune.go` imports the package using the same string declared in the `importpath` attribute of the `fortune` library. -We also need to declare this dependency to Bazel. Here's the `BUILD` file in the -`stage2` directory. +We also need to declare this dependency to Bazel. Here's the `BUILD` file in the `stage2` directory. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -286,57 +222,25 @@ You can run this with the command below. bazel run //:print_fortune ``` -The `print_fortune` target has a `deps` attribute, a list of other targets that -it depends on. It contains `"//fortune"`, a label string referring to the target -in the `fortune` directory named `fortune`. +The `print_fortune` target has a `deps` attribute, a list of other targets that it depends on. It contains `"//fortune"`, a label string referring to the target in the `fortune` directory named `fortune`. -Bazel requires that all targets declare their dependencies explicitly with -attributes like `deps`. This may seem cumbersome since dependencies are *also* -specified in source files, but Bazel's explictness gives it an advantage. Bazel -builds an [action graph](https://bazel.build/reference/glossary#action-graph) -containing all commands, inputs, and outputs before running any commands, -without reading any source files. Bazel can then cache action results or send -actions for [remote execution](https://bazel.build/remote/rbe) without built-in -language-specific logic. +Bazel requires that all targets declare their dependencies explicitly with attributes like `deps`. This may seem cumbersome since dependencies are *also* specified in source files, but Bazel's explictness gives it an advantage. Bazel builds an [action graph](https://bazel.build/reference/glossary#action-graph) containing all commands, inputs, and outputs before running any commands, without reading any source files. Bazel can then cache action results or send actions for [remote execution](https://bazel.build/remote/rbe) without built-in language-specific logic. ### Understanding labels -A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses -to identify a target or a file. Labels are used in command line arguments and in -`BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, -`//:print-fortune`, and `@rules_go//go:def.bzl`. - -A label has three parts: a repository name, a package name, and a target (or -file) name. - -The repository name is written between `@` and `//` and is used to refer to a -target from a different Bazel module (for historical reasons, *module* and -*repository* are sometimes used synonymously). In the label, -`@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name -can be omitted when referring to targets in the same repository. - -The package name is written between `//` and `:` and is used to refer to a -target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, -the package name is `go`. A Bazel -[package](https://bazel.build/reference/glossary#package) is a set of files and -targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. -Its package name is a slash-separated path from the module root directory -(containing `MODULE.bazel`) to the directory containing the `BUILD` file. A -package may include subdirectories, but only if they don't also contain `BUILD` -files defining their own packages. - -Most Go projects have one `BUILD` file per directory and one Go package per -`BUILD` file. The package name in a label may be omitted when referring to -targets in the same directory. - -The target name is written after `:` and refers to a target within a package. -The target name may be omitted if it's the same as the last component of the -package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is -the same as `//fortune`). - -On the command-line, you can use `...` as a wildcard to refer to all the targets -within a package. This is useful for building or testing all the targets in a -repository. +A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses to identify a target or a file. Labels are used in command line arguments and in `BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, `//:print-fortune`, and `@rules_go//go:def.bzl`. + +A label has three parts: a repository name, a package name, and a target (or file) name. + +The repository name is written between `@` and `//` and is used to refer to a target from a different Bazel module (for historical reasons, *module* and *repository* are sometimes used synonymously). In the label, `@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name can be omitted when referring to targets in the same repository. + +The package name is written between `//` and `:` and is used to refer to a target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, the package name is `go`. A Bazel [package](https://bazel.build/reference/glossary#package) is a set of files and targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. Its package name is a slash-separated path from the module root directory (containing `MODULE.bazel`) to the directory containing the `BUILD` file. A package may include subdirectories, but only if they don't also contain `BUILD` files defining their own packages. + +Most Go projects have one `BUILD` file per directory and one Go package per `BUILD` file. The package name in a label may be omitted when referring to targets in the same directory. + +The target name is written after `:` and refers to a target within a package. The target name may be omitted if it's the same as the last component of the package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is the same as `//fortune`). + +On the command-line, you can use `...` as a wildcard to refer to all the targets within a package. This is useful for building or testing all the targets in a repository. ```posix-shell # Build everything @@ -372,15 +276,13 @@ import ( // TestGet checks that Get returns one of the strings from fortunes. func TestGet(t *testing.T) { msg := Get() - if i := slices.Index(fortunes, msg); i < 0 { + if i := slices.Index(fortunes, msg); i < 0 { t.Errorf("Get returned %q, not one the expected messages", msg) } } ``` -This file uses the unexported `fortunes` variable, so it needs to be compiled -into the same Go package as `fortune.go`. Look at the `BUILD` file to see -how that works: +This file uses the unexported `fortunes` variable, so it needs to be compiled into the same Go package as `fortune.go`. Look at the `BUILD` file to see how that works: ```bazel load("@rules_go//go:def.bzl", "go_library", "go_test") @@ -399,19 +301,9 @@ go_test( ) ``` -We have a new `fortune_test` target that uses the `go_test` rule to compile and -link a test executable. `go_test` needs to compile `fortune.go` and -`fortune_test.go` together with the same command, so we use the `embed` -attribute here to incorporate the attributes of the `fortune` target into -`fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, -but it also works with `go_library`, which is sometimes useful for generated -code. +We have a new `fortune_test` target that uses the `go_test` rule to compile and link a test executable. `go_test` needs to compile `fortune.go` and `fortune_test.go` together with the same command, so we use the `embed` attribute here to incorporate the attributes of the `fortune` target into `fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, but it also works with `go_library`, which is sometimes useful for generated code. -You may be wondering if the `embed` attribute is related to Go's -[`embed`](https://pkg.go.dev/embed) package, which is used to access data files -copied into an executable. This is an unfortunate name collision: rules_go's -`embed` attribute was introduced before Go's `embed` package. Instead, rules_go -uses the `embedsrcs` to list files that can be loaded with the `embed` package. +You may be wondering if the `embed` attribute is related to Go's [`embed`](https://pkg.go.dev/embed) package, which is used to access data files copied into an executable. This is an unfortunate name collision: rules\_go's `embed` attribute was introduced before Go's `embed` package. Instead, rules\_go uses the `embedsrcs` to list files that can be loaded with the `embed` package. Try running our test with `bazel test`: @@ -430,9 +322,7 @@ Executed 0 out of 1 test: 1 test passes. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. ``` -You can use the `...` wildcard to run all tests. Bazel will also build targets -that aren't tests, so this can catch compile errors even in packages that don't -have tests. +You can use the `...` wildcard to run all tests. Bazel will also build targets that aren't tests, so this can catch compile errors even in packages that don't have tests. ```posix-shell $ bazel test //... @@ -440,21 +330,9 @@ $ bazel test //... ## Conclusion and further reading -In this tutorial, we built and tested a small Go project with Bazel, and we -learned some core Bazel concepts along the way. - -- To get started building other applications with Bazel, see the tutorials for - [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and - [iOS](/start/ios-app). -- You can also check the list of [recommended rules](/rules) for other - languages. -- For more information on Go, see the - [rules_go](https://github.com/bazel-contrib/rules_go) module, especially the - [Core Go - rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) - documentation. -- To learn more about working with Bazel modules outside your project, see - [external dependencies](/docs/external). In particular, for information on - how to depend on Go modules and toolchains through Bazel's module system, - see [Go with - bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). +In this tutorial, we built and tested a small Go project with Bazel, and we learned some core Bazel concepts along the way. + +- To get started building other applications with Bazel, see the tutorials for [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and [iOS](/start/ios-app). +- You can also check the list of [recommended rules](/rules) for other languages. +- For more information on Go, see the [rules\_go](https://github.com/bazel-contrib/rules_go) module, especially the [Core Go rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) documentation. +- To learn more about working with Bazel modules outside your project, see [external dependencies](/docs/external). In particular, for information on how to depend on Go modules and toolchains through Bazel's module system, see [Go with bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). diff --git a/start/ios-app.mdx b/start/ios-app.mdx index 0b860ab6..5004bbd2 100644 --- a/start/ios-app.mdx +++ b/start/ios-app.mdx @@ -2,5 +2,4 @@ title: 'Bazel Tutorial: Build an iOS App' --- - -This tutorial has been moved into the [bazelbuild/rules_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. +This tutorial has been moved into the [bazelbuild/rules\_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. diff --git a/start/java.mdx b/start/java.mdx index b892917d..edb9bc0d 100644 --- a/start/java.mdx +++ b/start/java.mdx @@ -2,11 +2,7 @@ title: 'Bazel Tutorial: Build a Java Project' --- - - -This tutorial covers the basics of building Java applications with -Bazel. You will set up your workspace and build a simple Java project that -illustrates key Bazel concepts, such as targets and `BUILD` files. +This tutorial covers the basics of building Java applications with Bazel. You will set up your workspace and build a simple Java project that illustrates key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes. @@ -14,36 +10,40 @@ Estimated completion time: 30 minutes. In this tutorial you learn how to: -* Build a target -* Visualize the project's dependencies -* Split the project into multiple targets and packages -* Control target visibility across packages -* Reference targets through labels -* Deploy a target +- Build a target +- Visualize the project's dependencies +- Split the project into multiple targets and packages +- Control target visibility across packages +- Reference targets through labels +- Deploy a target ## Before you begin ### Install Bazel -To prepare for the tutorial, first [Install Bazel](/install) if -you don't have it installed already. +To prepare for the tutorial, first [Install Bazel](/install) if you don't have it installed already. ### Install the JDK -1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). +1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). + +2. Set the JAVA\_HOME environment variable to point to the JDK. + + - On Linux/macOS: + + ``` + export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" + ``` -2. Set the JAVA\_HOME environment variable to point to the JDK. - * On Linux/macOS: + - On Windows: - export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" - * On Windows: - 1. Open Control Panel. - 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . - 3. Under the "User variables" list (the one on the top), click "New...". - 4. In the "Variable name" field, enter `JAVA_HOME`. - 5. Click "Browse Directory...". - 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). - 7. Click "OK" on all dialog windows. + 1. Open Control Panel. + 2. Go to "System and Security" \> "System" \> "Advanced System Settings" \> "Advanced" tab \> "Environment Variables..." . + 3. Under the "User variables" list (the one on the top), click "New\...". + 4. In the "Variable name" field, enter `JAVA_HOME`. + 5. Click "Browse Directory...". + 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). + 7. Click "OK" on all dialog windows. ### Get the sample project @@ -53,8 +53,7 @@ Retrieve the sample project from Bazel's GitHub repository: git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/java-tutorial` -directory and is structured as follows: +The sample project for this tutorial is in the `examples/java-tutorial` directory and is structured as follows: ``` java-tutorial @@ -76,32 +75,19 @@ java-tutorial ### Set up the workspace -Before you can build a project, you need to set up its workspace. A workspace is -a directory that holds your project's source files and Bazel's build outputs. It -also contains files that Bazel recognizes as special: +Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains files that Bazel recognizes as special: -* The `MODULE.bazel` file, which identifies the directory and its contents as a - Bazel workspace and lives at the root of the project's directory structure, +- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure, -* One or more `BUILD` files, which tell Bazel how to build different parts of - the project. (A directory within the workspace that contains a `BUILD` file - is a *package*. You will learn about packages later in this tutorial.) +- One or more `BUILD` files, which tell Bazel how to build different parts of the project. (A directory within the workspace that contains a `BUILD` file is a *package*. You will learn about packages later in this tutorial.) -To designate a directory as a Bazel workspace, create an empty file named -`MODULE.bazel` in that directory. +To designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. -When Bazel builds the project, all inputs and dependencies must be in the same -workspace. Files residing in different workspaces are independent of one -another unless linked, which is beyond the scope of this tutorial. +When Bazel builds the project, all inputs and dependencies must be in the same workspace. Files residing in different workspaces are independent of one another unless linked, which is beyond the scope of this tutorial. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. -The most important type is the *build rule*, which tells Bazel how to build the -desired outputs, such as executable binaries or libraries. Each instance -of a build rule in the `BUILD` file is called a *target* and points to a -specific set of source files and dependencies. A target can also point to other -targets. +A `BUILD` file contains several different types of instructions for Bazel. The most important type is the *build rule*, which tells Bazel how to build the desired outputs, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a *target* and points to a specific set of source files and dependencies. A target can also point to other targets. Take a look at the `java-tutorial/BUILD` file: @@ -112,30 +98,19 @@ java_binary( ) ``` -In our example, the `ProjectRunner` target instantiates Bazel's built-in -[`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to -build a `.jar` file and a wrapper shell script (both named after the target). +In our example, the `ProjectRunner` target instantiates Bazel's built-in [`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to build a `.jar` file and a wrapper shell script (both named after the target). -The attributes in the target explicitly state its dependencies and options. -While the `name` attribute is mandatory, many are optional. For example, in the -`ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies -the source files that Bazel uses to build the target, and `main_class` specifies -the class that contains the main method. (You may have noticed that our example -uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel -instead of listing them one by one.) +The attributes in the target explicitly state its dependencies and options. While the `name` attribute is mandatory, many are optional. For example, in the `ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies the source files that Bazel uses to build the target, and `main_class` specifies the class that contains the main method. (You may have noticed that our example uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel instead of listing them one by one.) ### Build the project -To build your sample project, navigate to the `java-tutorial` directory -and run: +To build your sample project, navigate to the `java-tutorial` directory and run: ```posix-terminal bazel build //:ProjectRunner ``` -In the target label, the `//` part is the location of the `BUILD` file -relative to the root of the workspace (in this case, the root itself), -and `ProjectRunner` is the target name in the `BUILD` file. (You will -learn about target labels in more detail at the end of this tutorial.) + +In the target label, the `//` part is the location of the `BUILD` file relative to the root of the workspace (in this case, the root itself), and `ProjectRunner` is the target name in the `BUILD` file. (You will learn about target labels in more detail at the end of this tutorial.) Bazel produces output similar to the following: @@ -147,9 +122,7 @@ Bazel produces output similar to the following: INFO: Elapsed time: 1.021s, Critical Path: 0.83s ``` -Congratulations, you just built your first Bazel target! Bazel places build -outputs in the `bazel-bin` directory at the root of the workspace. Browse -through its contents to get an idea for Bazel's output structure. +Congratulations, you just built your first Bazel target! Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Browse through its contents to get an idea for Bazel's output structure. Now test your freshly built binary: @@ -159,43 +132,31 @@ bazel-bin/ProjectRunner ### Review the dependency graph -Bazel requires build dependencies to be explicitly declared in BUILD files. -Bazel uses those statements to create the project's dependency graph, which -enables accurate incremental builds. +Bazel requires build dependencies to be explicitly declared in BUILD files. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text -representation of the dependency graph by running this command at the -workspace root: +To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: ```posix-terminal bazel query --notool_deps --noimplicit_deps "deps(//:ProjectRunner)" --output graph ``` -The above command tells Bazel to look for all dependencies for the target -`//:ProjectRunner` (excluding host and implicit dependencies) and format the -output as a graph. +The above command tells Bazel to look for all dependencies for the target `//:ProjectRunner` (excluding host and implicit dependencies) and format the output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -As you can see, the project has a single target that build two source files with -no additional dependencies: +As you can see, the project has a single target that build two source files with no additional dependencies: ![Dependency graph of the target 'ProjectRunner'](/docs/images/tutorial_java_01.svg) -After you set up your workspace, build your project, and examine its -dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. ## Refine your Bazel build -While a single target is sufficient for small projects, you may want to split -larger projects into multiple targets and packages to allow for fast incremental -builds (that is, only rebuild what's changed) and to speed up your builds by -building multiple parts of a project at once. +While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages to allow for fast incremental builds (that is, only rebuild what's changed) and to speed up your builds by building multiple parts of a project at once. ### Specify multiple build targets -You can split the sample project build into two targets. Replace the contents of -the `java-tutorial/BUILD` file with the following: +You can split the sample project build into two targets. Replace the contents of the `java-tutorial/BUILD` file with the following: ```python java_binary( @@ -211,9 +172,7 @@ java_library( ) ``` -With this configuration, Bazel first builds the `greeter` library, then the -`ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that -the `greeter` library is required to build the `ProjectRunner` binary. +With this configuration, Bazel first builds the `greeter` library, then the `ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that the `greeter` library is required to build the `ProjectRunner` binary. To build this new version of the project, run the following command: @@ -237,26 +196,17 @@ Now test your freshly built binary: bazel-bin/ProjectRunner ``` -If you now modify `ProjectRunner.java` and rebuild the project, Bazel only -recompiles that file. +If you now modify `ProjectRunner.java` and rebuild the project, Bazel only recompiles that file. -Looking at the dependency graph, you can see that `ProjectRunner` depends on the -same inputs as it did before, but the structure of the build is different: +Looking at the dependency graph, you can see that `ProjectRunner` depends on the same inputs as it did before, but the structure of the build is different: -![Dependency graph of the target 'ProjectRunner' after adding a dependency]( -/docs/images/tutorial_java_02.svg) +![Dependency graph of the target 'ProjectRunner' after adding a dependency](/docs/images/tutorial_java_02.svg) -You've now built the project with two targets. The `ProjectRunner` target builds -one source files and depends on one other target (`:greeter`), which builds -one additional source file. +You've now built the project with two targets. The `ProjectRunner` target builds one source files and depends on one other target (`:greeter`), which builds one additional source file. ### Use multiple packages -Let’s now split the project into multiple packages. If you take a look at the -`src/main/java/com/example/cmdline` directory, you can see that it also contains -a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now -contains two packages, `//src/main/java/com/example/cmdline` and `//` (since -there is a `BUILD` file at the root of the workspace). +Let’s now split the project into multiple packages. If you take a look at the `src/main/java/com/example/cmdline` directory, you can see that it also contains a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now contains two packages, `//src/main/java/com/example/cmdline` and `//` (since there is a `BUILD` file at the root of the workspace). Take a look at the `src/main/java/com/example/cmdline/BUILD` file: @@ -269,21 +219,13 @@ java_binary( ) ``` -The `runner` target depends on the `greeter` target in the `//` package (hence -the target label `//:greeter`) - Bazel knows this through the `deps` attribute. -Take a look at the dependency graph: +The `runner` target depends on the `greeter` target in the `//` package (hence the target label `//:greeter`) - Bazel knows this through the `deps` attribute. Take a look at the dependency graph: ![Dependency graph of the target 'runner'](/docs/images/tutorial_java_03.svg) -However, for the build to succeed, you must explicitly give the `runner` target -in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in -`//BUILD` using the `visibility` attribute. This is because by default targets -are only visible to other targets in the same `BUILD` file. (Bazel uses target -visibility to prevent issues such as libraries containing implementation details -leaking into public APIs.) +However, for the build to succeed, you must explicitly give the `runner` target in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in `//BUILD` using the `visibility` attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. (Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs.) -To do this, add the `visibility` attribute to the `greeter` target in -`java-tutorial/BUILD` as shown below: +To do this, add the `visibility` attribute to the `greeter` target in `java-tutorial/BUILD` as shown below: ```python java_library( @@ -293,8 +235,7 @@ java_library( ) ``` -Now you can build the new package by running the following command at the root -of the workspace: +Now you can build the new package by running the following command at the root of the workspace: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner @@ -316,48 +257,29 @@ Now test your freshly built binary: ./bazel-bin/src/main/java/com/example/cmdline/runner ``` -You've now modified the project to build as two packages, each containing one -target, and understand the dependencies between them. - +You've now modified the project to build as two packages, each containing one target, and understand the dependencies between them. ## Use labels to reference targets -In `BUILD` files and at the command line, Bazel uses target labels to reference -targets - for example, `//:ProjectRunner` or -`//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: +In `BUILD` files and at the command line, Bazel uses target labels to reference targets - for example, `//:ProjectRunner` or `//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path to the -directory containing the `BUILD` file, and `target-name` is what you named the -target in the `BUILD` file (the `name` attribute). If the target is a file -target, then `path/to/package` is the path to the root of the package, and -`target-name` is the name of the target file, including its full path. +If the target is a rule target, then `path/to/package` is the path to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path. -When referencing targets at the repository root, the package path is empty, -just use `//:target-name`. When referencing targets within the same `BUILD` -file, you can even skip the `//` workspace root identifier and just use -`:target-name`. +When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. -For example, for targets in the `java-tutorial/BUILD` file, you did not have to -specify a package path, since the workspace root is itself a package (`//`), and -your two target labels were simply `//:ProjectRunner` and `//:greeter`. +For example, for targets in the `java-tutorial/BUILD` file, you did not have to specify a package path, since the workspace root is itself a package (`//`), and your two target labels were simply `//:ProjectRunner` and `//:greeter`. -However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you -had to specify the full package path of `//src/main/java/com/example/cmdline` -and your target label was `//src/main/java/com/example/cmdline:runner`. +However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you had to specify the full package path of `//src/main/java/com/example/cmdline` and your target label was `//src/main/java/com/example/cmdline:runner`. ## Package a Java target for deployment -Let’s now package a Java target for deployment by building the binary with all -of its runtime dependencies. This lets you run the binary outside of your -development environment. +Let’s now package a Java target for deployment by building the binary with all of its runtime dependencies. This lets you run the binary outside of your development environment. -As you remember, the [java_binary](/reference/be/java#java_binary) build rule -produces a `.jar` and a wrapper shell script. Take a look at the contents of -`runner.jar` using this command: +As you remember, the [java\_binary](/reference/be/java#java_binary) build rule produces a `.jar` and a wrapper shell script. Take a look at the contents of `runner.jar` using this command: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner.jar @@ -373,12 +295,8 @@ com/example/ com/example/cmdline/ com/example/cmdline/Runner.class ``` -As you can see, `runner.jar` contains `Runner.class`, but not its dependency, -`Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` -to the classpath, so if you leave it like this, it will run locally, but it -won't run standalone on another machine. Fortunately, the `java_binary` rule -allows you to build a self-contained, deployable binary. To build it, append -`_deploy.jar` to the target name: + +As you can see, `runner.jar` contains `Runner.class`, but not its dependency, `Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` to the classpath, so if you leave it like this, it will run locally, but it won't run standalone on another machine. Fortunately, the `java_binary` rule allows you to build a self-contained, deployable binary. To build it, append `_deploy.jar` to the target name: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner_deploy.jar @@ -392,10 +310,8 @@ Target //src/main/java/com/example/cmdline:runner_deploy.jar up-to-date: bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar INFO: Elapsed time: 1.700s, Critical Path: 0.23s ``` -You have just built `runner_deploy.jar`, which you can run standalone away from -your development environment since it contains the required runtime -dependencies. Take a look at the contents of this standalone JAR using the -same command as before: + +You have just built `runner_deploy.jar`, which you can run standalone away from your development environment since it contains the required runtime dependencies. Take a look at the contents of this standalone JAR using the same command as before: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar @@ -418,19 +334,14 @@ com/example/Greeting.class For more details, see: -* [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) for - rules to manage transitive Maven dependencies. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) for rules to manage transitive Maven dependencies. -* [External Dependencies](/docs/external) to learn more about working with - local and remote repositories. +- [External Dependencies](/docs/external) to learn more about working with local and remote repositories. -* The [other rules](/rules) to learn more about Bazel. +- The [other rules](/rules) to learn more about Bazel. -* The [C++ build tutorial](/start/cpp) to get started with building - C++ projects with Bazel. +- The [C++ build tutorial](/start/cpp) to get started with building C++ projects with Bazel. -* The [Android application tutorial](/start/android-app ) and - [iOS application tutorial](/start/ios-app)) to get started with - building mobile applications for Android and iOS with Bazel. +- The [Android application tutorial](/start/android-app) and [iOS application tutorial](/start/ios-app)) to get started with building mobile applications for Android and iOS with Bazel. Happy building! diff --git a/tools/mdx-transform/package-lock.json b/tools/mdx-transform/package-lock.json new file mode 100644 index 00000000..f9b006a3 --- /dev/null +++ b/tools/mdx-transform/package-lock.json @@ -0,0 +1,2142 @@ +{ + "name": "mdx-transform", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "mdx-transform", + "version": "0.1.0", + "license": "ISC", + "dependencies": { + "hast-util-to-mdast": "^10.1.2", + "hast-util-to-text": "^4.0.2", + "mdast-util-to-hast": "^13.2.0", + "rehype-raw": "^7.0.0", + "rehype-stringify": "^10.0.1", + "remark-frontmatter": "^5.0.0", + "remark-gfm": "^4.0.1", + "remark-mdx": "^3.1.1", + "remark-parse": "^11.0.0", + "remark-rehype": "^11.1.2", + "remark-stringify": "^11.0.0", + "unified": "^11.0.5", + "unist-util-is": "^6.0.1", + "unist-util-remove": "^4.0.0", + "unist-util-visit": "^5.0.0" + }, + "bin": { + "mdx-transform": "transform.mjs" + } + }, + "node_modules/@types/debug": { + "version": "4.1.12", + "resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.12.tgz", + "integrity": "sha512-vIChWdVG3LG1SMxEvI/AK+FWJthlrqlTu7fbrlywTkkaONwk/UAGaULXRlf8vkzFBLVm0zkMdCquhL5aOjhXPQ==", + "license": "MIT", + "dependencies": { + "@types/ms": "*" + } + }, + "node_modules/@types/estree": { + "version": "1.0.8", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", + "integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==", + "license": "MIT" + }, + "node_modules/@types/estree-jsx": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/@types/estree-jsx/-/estree-jsx-1.0.5.tgz", + "integrity": "sha512-52CcUVNFyfb1A2ALocQw/Dd1BQFNmSdkuC3BkZ6iqhdMfQz7JWOFRuJFloOzjk+6WijU56m9oKXFAXc7o3Towg==", + "license": "MIT", + "dependencies": { + "@types/estree": "*" + } + }, + "node_modules/@types/hast": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.4.tgz", + "integrity": "sha512-WPs+bbQw5aCj+x6laNGWLH3wviHtoCv/P3+otBhbOhJgG8qtpdAMlTCxLtsTWA7LH1Oh/bFCHsBn0TPS5m30EQ==", + "license": "MIT", + "dependencies": { + "@types/unist": "*" + } + }, + "node_modules/@types/mdast": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@types/mdast/-/mdast-4.0.4.tgz", + "integrity": "sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==", + "license": "MIT", + "dependencies": { + "@types/unist": "*" + } + }, + "node_modules/@types/ms": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/@types/ms/-/ms-2.1.0.tgz", + "integrity": "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==", + "license": "MIT" + }, + "node_modules/@types/unist": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/@types/unist/-/unist-3.0.3.tgz", + "integrity": "sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==", + "license": "MIT" + }, + "node_modules/@ungap/structured-clone": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/@ungap/structured-clone/-/structured-clone-1.3.0.tgz", + "integrity": "sha512-WmoN8qaIAo7WTYWbAZuG8PYEhn5fkz7dZrqTBZ7dtt//lL2Gwms1IcnQ5yHqjDfX8Ft5j4YzDM23f87zBfDe9g==", + "license": "ISC" + }, + "node_modules/acorn": { + "version": "8.15.0", + "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", + "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", + "license": "MIT", + "bin": { + "acorn": "bin/acorn" + }, + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/acorn-jsx": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/acorn-jsx/-/acorn-jsx-5.3.2.tgz", + "integrity": "sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ==", + "license": "MIT", + "peerDependencies": { + "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" + } + }, + "node_modules/bail": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/bail/-/bail-2.0.2.tgz", + "integrity": "sha512-0xO6mYd7JB2YesxDKplafRpsiOzPt9V02ddPCLbY1xYGPOX24NTyN50qnUxgCPcSoYMhKpAuBTjQoRZCAkUDRw==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/ccount": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz", + "integrity": "sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-entities": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/character-entities/-/character-entities-2.0.2.tgz", + "integrity": "sha512-shx7oQ0Awen/BRIdkjkvz54PnEEI/EjwXDSIZp86/KKdbafHh1Df/RYGBhn4hbe2+uKC9FnT5UCEdyPz3ai9hQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-entities-html4": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/character-entities-html4/-/character-entities-html4-2.1.0.tgz", + "integrity": "sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-entities-legacy": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz", + "integrity": "sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/character-reference-invalid": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/character-reference-invalid/-/character-reference-invalid-2.0.1.tgz", + "integrity": "sha512-iBZ4F4wRbyORVsu0jPV7gXkOsGYjGHPmAyv+HiHG8gi5PtC9KI2j1+v8/tlibRvjoWX027ypmG/n0HtO5t7unw==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/comma-separated-tokens": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", + "integrity": "sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/decode-named-character-reference": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/decode-named-character-reference/-/decode-named-character-reference-1.2.0.tgz", + "integrity": "sha512-c6fcElNV6ShtZXmsgNgFFV5tVX2PaV4g+MOAkb8eXHvn6sryJBrZa9r0zV6+dtTyoCKxtDy5tyQ5ZwQuidtd+Q==", + "license": "MIT", + "dependencies": { + "character-entities": "^2.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/dequal": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", + "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/devlop": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/devlop/-/devlop-1.1.0.tgz", + "integrity": "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==", + "license": "MIT", + "dependencies": { + "dequal": "^2.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/entities": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz", + "integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=0.12" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/escape-string-regexp": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-5.0.0.tgz", + "integrity": "sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw==", + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/estree-util-is-identifier-name": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/estree-util-is-identifier-name/-/estree-util-is-identifier-name-3.0.0.tgz", + "integrity": "sha512-hFtqIDZTIUZ9BXLb8y4pYGyk6+wekIivNVTcmvk8NoOh+VeRn5y6cEHzbURrWbfp1fIqdVipilzj+lfaadNZmg==", + "license": "MIT", + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/estree-util-visit": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/estree-util-visit/-/estree-util-visit-2.0.0.tgz", + "integrity": "sha512-m5KgiH85xAhhW8Wta0vShLcUvOsh3LLPI2YVwcbio1l7E09NTLL1EyMZFM1OyWowoH0skScNbhOPl4kcBgzTww==", + "license": "MIT", + "dependencies": { + "@types/estree-jsx": "^1.0.0", + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/extend": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", + "integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==", + "license": "MIT" + }, + "node_modules/fault": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/fault/-/fault-2.0.1.tgz", + "integrity": "sha512-WtySTkS4OKev5JtpHXnib4Gxiurzh5NCGvWrFaZ34m6JehfTUhKZvn9njTfw48t6JumVQOmrKqpmGcdwxnhqBQ==", + "license": "MIT", + "dependencies": { + "format": "^0.2.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/format": { + "version": "0.2.2", + "resolved": "https://registry.npmjs.org/format/-/format-0.2.2.tgz", + "integrity": "sha512-wzsgA6WOq+09wrU1tsJ09udeR/YZRaeArL9e1wPbFg3GG2yDnC2ldKpxs4xunpFF9DgqCqOIra3bc1HWrJ37Ww==", + "engines": { + "node": ">=0.4.x" + } + }, + "node_modules/hast-util-embedded": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/hast-util-embedded/-/hast-util-embedded-3.0.0.tgz", + "integrity": "sha512-naH8sld4Pe2ep03qqULEtvYr7EjrLK2QHY8KJR6RJkTUjPGObe1vnx585uzem2hGra+s1q08DZZpfgDVYRbaXA==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-is-element": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-from-parse5": { + "version": "8.0.3", + "resolved": "https://registry.npmjs.org/hast-util-from-parse5/-/hast-util-from-parse5-8.0.3.tgz", + "integrity": "sha512-3kxEVkEKt0zvcZ3hCRYI8rqrgwtlIOFMWkbclACvjlDw8Li9S2hk/d51OI0nr/gIpdMHNepwgOKqZ/sy0Clpyg==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/unist": "^3.0.0", + "devlop": "^1.0.0", + "hastscript": "^9.0.0", + "property-information": "^7.0.0", + "vfile": "^6.0.0", + "vfile-location": "^5.0.0", + "web-namespaces": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-has-property": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/hast-util-has-property/-/hast-util-has-property-3.0.0.tgz", + "integrity": "sha512-MNilsvEKLFpV604hwfhVStK0usFY/QmM5zX16bo7EjnAEGofr5YyI37kzopBlZJkHD4t887i+q/C8/tr5Q94cA==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-is-body-ok-link": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/hast-util-is-body-ok-link/-/hast-util-is-body-ok-link-3.0.1.tgz", + "integrity": "sha512-0qpnzOBLztXHbHQenVB8uNuxTnm/QBFUOmdOSsEn7GnBtyY07+ENTWVFBAnXd/zEgd9/SUG3lRY7hSIBWRgGpQ==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-is-element": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/hast-util-is-element/-/hast-util-is-element-3.0.0.tgz", + "integrity": "sha512-Val9mnv2IWpLbNPqc/pUem+a7Ipj2aHacCwgNfTiK0vJKl0LF+4Ba4+v1oPHFpf3bLYmreq0/l3Gud9S5OH42g==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-minify-whitespace": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/hast-util-minify-whitespace/-/hast-util-minify-whitespace-1.0.1.tgz", + "integrity": "sha512-L96fPOVpnclQE0xzdWb/D12VT5FabA7SnZOUMtL1DbXmYiHJMXZvFkIZfiMmTCNJHUeO2K9UYNXoVyfz+QHuOw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-embedded": "^3.0.0", + "hast-util-is-element": "^3.0.0", + "hast-util-whitespace": "^3.0.0", + "unist-util-is": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-parse-selector": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/hast-util-parse-selector/-/hast-util-parse-selector-4.0.0.tgz", + "integrity": "sha512-wkQCkSYoOGCRKERFWcxMVMOcYE2K1AaNLU8DXS9arxnLOUEWbOXKXiJUNzEpqZ3JOKpnha3jkFrumEjVliDe7A==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-phrasing": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/hast-util-phrasing/-/hast-util-phrasing-3.0.1.tgz", + "integrity": "sha512-6h60VfI3uBQUxHqTyMymMZnEbNl1XmEGtOxxKYL7stY2o601COo62AWAYBQR9lZbYXYSBoxag8UpPRXK+9fqSQ==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-embedded": "^3.0.0", + "hast-util-has-property": "^3.0.0", + "hast-util-is-body-ok-link": "^3.0.0", + "hast-util-is-element": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-raw": { + "version": "9.1.0", + "resolved": "https://registry.npmjs.org/hast-util-raw/-/hast-util-raw-9.1.0.tgz", + "integrity": "sha512-Y8/SBAHkZGoNkpzqqfCldijcuUKh7/su31kEBp67cFY09Wy0mTRgtsLYsiIxMJxlu0f6AA5SUTbDR8K0rxnbUw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/unist": "^3.0.0", + "@ungap/structured-clone": "^1.0.0", + "hast-util-from-parse5": "^8.0.0", + "hast-util-to-parse5": "^8.0.0", + "html-void-elements": "^3.0.0", + "mdast-util-to-hast": "^13.0.0", + "parse5": "^7.0.0", + "unist-util-position": "^5.0.0", + "unist-util-visit": "^5.0.0", + "vfile": "^6.0.0", + "web-namespaces": "^2.0.0", + "zwitch": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-to-html": { + "version": "9.0.5", + "resolved": "https://registry.npmjs.org/hast-util-to-html/-/hast-util-to-html-9.0.5.tgz", + "integrity": "sha512-OguPdidb+fbHQSU4Q4ZiLKnzWo8Wwsf5bZfbvu7//a9oTYoqD/fWpe96NuHkoS9h0ccGOTe0C4NGXdtS0iObOw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/unist": "^3.0.0", + "ccount": "^2.0.0", + "comma-separated-tokens": "^2.0.0", + "hast-util-whitespace": "^3.0.0", + "html-void-elements": "^3.0.0", + "mdast-util-to-hast": "^13.0.0", + "property-information": "^7.0.0", + "space-separated-tokens": "^2.0.0", + "stringify-entities": "^4.0.0", + "zwitch": "^2.0.4" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-to-mdast": { + "version": "10.1.2", + "resolved": "https://registry.npmjs.org/hast-util-to-mdast/-/hast-util-to-mdast-10.1.2.tgz", + "integrity": "sha512-FiCRI7NmOvM4y+f5w32jPRzcxDIz+PUqDwEqn1A+1q2cdp3B8Gx7aVrXORdOKjMNDQsD1ogOr896+0jJHW1EFQ==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "@ungap/structured-clone": "^1.0.0", + "hast-util-phrasing": "^3.0.0", + "hast-util-to-html": "^9.0.0", + "hast-util-to-text": "^4.0.0", + "hast-util-whitespace": "^3.0.0", + "mdast-util-phrasing": "^4.0.0", + "mdast-util-to-hast": "^13.0.0", + "mdast-util-to-string": "^4.0.0", + "rehype-minify-whitespace": "^6.0.0", + "trim-trailing-lines": "^2.0.0", + "unist-util-position": "^5.0.0", + "unist-util-visit": "^5.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-to-parse5": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/hast-util-to-parse5/-/hast-util-to-parse5-8.0.0.tgz", + "integrity": "sha512-3KKrV5ZVI8if87DVSi1vDeByYrkGzg4mEfeu4alwgmmIeARiBLKCZS2uw5Gb6nU9x9Yufyj3iudm6i7nl52PFw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "comma-separated-tokens": "^2.0.0", + "devlop": "^1.0.0", + "property-information": "^6.0.0", + "space-separated-tokens": "^2.0.0", + "web-namespaces": "^2.0.0", + "zwitch": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-to-parse5/node_modules/property-information": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/property-information/-/property-information-6.5.0.tgz", + "integrity": "sha512-PgTgs/BlvHxOu8QuEN7wi5A0OmXaBcHpmCSTehcs6Uuu9IkDIEo13Hy7n898RHfrQ49vKCoGeWZSaAK01nwVig==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/hast-util-to-text": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/hast-util-to-text/-/hast-util-to-text-4.0.2.tgz", + "integrity": "sha512-KK6y/BN8lbaq654j7JgBydev7wuNMcID54lkRav1P0CaE1e47P72AWWPiGKXTJU271ooYzcvTAn/Zt0REnvc7A==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/unist": "^3.0.0", + "hast-util-is-element": "^3.0.0", + "unist-util-find-after": "^5.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hast-util-whitespace": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/hast-util-whitespace/-/hast-util-whitespace-3.0.0.tgz", + "integrity": "sha512-88JUN06ipLwsnv+dVn+OIYOvAuvBMy/Qoi6O7mQHxdPXpjy+Cd6xRkWwux7DKO+4sYILtLBRIKgsdpS2gQc7qw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/hastscript": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/hastscript/-/hastscript-9.0.1.tgz", + "integrity": "sha512-g7df9rMFX/SPi34tyGCyUBREQoKkapwdY/T04Qn9TDWfHhAYt4/I0gMVirzK5wEzeUqIjEB+LXC/ypb7Aqno5w==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "comma-separated-tokens": "^2.0.0", + "hast-util-parse-selector": "^4.0.0", + "property-information": "^7.0.0", + "space-separated-tokens": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/html-void-elements": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/html-void-elements/-/html-void-elements-3.0.0.tgz", + "integrity": "sha512-bEqo66MRXsUGxWHV5IP0PUiAWwoEjba4VCzg0LjFJBpchPaTfyfCKTG6bc5F8ucKec3q5y6qOdGyYTSBEvhCrg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/is-alphabetical": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-alphabetical/-/is-alphabetical-2.0.1.tgz", + "integrity": "sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/is-alphanumerical": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-alphanumerical/-/is-alphanumerical-2.0.1.tgz", + "integrity": "sha512-hmbYhX/9MUMF5uh7tOXyK/n0ZvWpad5caBA17GsC6vyuCqaWliRG5K1qS9inmUhEMaOBIW7/whAnSwveW/LtZw==", + "license": "MIT", + "dependencies": { + "is-alphabetical": "^2.0.0", + "is-decimal": "^2.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/is-decimal": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-decimal/-/is-decimal-2.0.1.tgz", + "integrity": "sha512-AAB9hiomQs5DXWcRB1rqsxGUstbRroFOPPVAomNk/3XHR5JyEZChOyTWe2oayKnsSsr/kcGqF+z6yuH6HHpN0A==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/is-hexadecimal": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-hexadecimal/-/is-hexadecimal-2.0.1.tgz", + "integrity": "sha512-DgZQp241c8oO6cA1SbTEWiXeoxV42vlcJxgH+B3hi1AiqqKruZR3ZGF8In3fj4+/y/7rHvlOZLZtgJ/4ttYGZg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/is-plain-obj": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-4.1.0.tgz", + "integrity": "sha512-+Pgi+vMuUNkJyExiMBt5IlFoMyKnr5zhJ4Uspz58WOhBF5QoIZkFyNHIbBAtHwzVAgk5RtndVNsDRN61/mmDqg==", + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/longest-streak": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/longest-streak/-/longest-streak-3.1.0.tgz", + "integrity": "sha512-9Ri+o0JYgehTaVBBDoMqIl8GXtbWg711O3srftcHhZ0dqnETqLaoIK0x17fUw9rFSlK/0NlsKe0Ahhyl5pXE2g==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/markdown-table": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/markdown-table/-/markdown-table-3.0.4.tgz", + "integrity": "sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/mdast-util-find-and-replace": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/mdast-util-find-and-replace/-/mdast-util-find-and-replace-3.0.2.tgz", + "integrity": "sha512-Tmd1Vg/m3Xz43afeNxDIhWRtFZgM2VLyaf4vSTYwudTyeuTneoL3qtWMA5jeLyz/O1vDJmmV4QuScFCA2tBPwg==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "escape-string-regexp": "^5.0.0", + "unist-util-is": "^6.0.0", + "unist-util-visit-parents": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-from-markdown": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/mdast-util-from-markdown/-/mdast-util-from-markdown-2.0.2.tgz", + "integrity": "sha512-uZhTV/8NBuw0WHkPTrCqDOl0zVe1BIng5ZtHoDk49ME1qqcjYmmLmOf0gELgcRMxN4w2iuIeVso5/6QymSrgmA==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "@types/unist": "^3.0.0", + "decode-named-character-reference": "^1.0.0", + "devlop": "^1.0.0", + "mdast-util-to-string": "^4.0.0", + "micromark": "^4.0.0", + "micromark-util-decode-numeric-character-reference": "^2.0.0", + "micromark-util-decode-string": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0", + "unist-util-stringify-position": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-frontmatter": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-frontmatter/-/mdast-util-frontmatter-2.0.1.tgz", + "integrity": "sha512-LRqI9+wdgC25P0URIJY9vwocIzCcksduHQ9OF2joxQoyTNVduwLAFUzjoopuRJbJAReaKrNQKAZKL3uCMugWJA==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "escape-string-regexp": "^5.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0", + "micromark-extension-frontmatter": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm/-/mdast-util-gfm-3.1.0.tgz", + "integrity": "sha512-0ulfdQOM3ysHhCJ1p06l0b0VKlhU0wuQs3thxZQagjcjPrlFRqY215uZGHHJan9GEAXd9MbfPjFJz+qMkVR6zQ==", + "license": "MIT", + "dependencies": { + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-gfm-autolink-literal": "^2.0.0", + "mdast-util-gfm-footnote": "^2.0.0", + "mdast-util-gfm-strikethrough": "^2.0.0", + "mdast-util-gfm-table": "^2.0.0", + "mdast-util-gfm-task-list-item": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-autolink-literal": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-autolink-literal/-/mdast-util-gfm-autolink-literal-2.0.1.tgz", + "integrity": "sha512-5HVP2MKaP6L+G6YaxPNjuL0BPrq9orG3TsrZ9YXbA3vDw/ACI4MEsnoDpn6ZNm7GnZgtAcONJyPhOP8tNJQavQ==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "ccount": "^2.0.0", + "devlop": "^1.0.0", + "mdast-util-find-and-replace": "^3.0.0", + "micromark-util-character": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-footnote": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-footnote/-/mdast-util-gfm-footnote-2.1.0.tgz", + "integrity": "sha512-sqpDWlsHn7Ac9GNZQMeUzPQSMzR6Wv0WKRNvQRg0KqHh02fpTz69Qc1QSseNX29bhz1ROIyNyxExfawVKTm1GQ==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.1.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-strikethrough": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-strikethrough/-/mdast-util-gfm-strikethrough-2.0.0.tgz", + "integrity": "sha512-mKKb915TF+OC5ptj5bJ7WFRPdYtuHv0yTRxK2tJvi+BDqbkiG7h7u/9SI89nRAYcmap2xHQL9D+QG/6wSrTtXg==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-table": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-table/-/mdast-util-gfm-table-2.0.0.tgz", + "integrity": "sha512-78UEvebzz/rJIxLvE7ZtDd/vIQ0RHv+3Mh5DR96p7cS7HsBhYIICDBCu8csTNWNO6tBWfqXPWekRuj2FNOGOZg==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "markdown-table": "^3.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-task-list-item": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-task-list-item/-/mdast-util-gfm-task-list-item-2.0.0.tgz", + "integrity": "sha512-IrtvNvjxC1o06taBAVJznEnkiHxLFTzgonUdy8hzFVeDun0uTjxxrRGVaNFqkU1wJR3RBPEfsxmU6jDWPofrTQ==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-mdx": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-mdx/-/mdast-util-mdx-3.0.0.tgz", + "integrity": "sha512-JfbYLAW7XnYTTbUsmpu0kdBUVe+yKVJZBItEjwyYJiDJuZ9w4eeaqks4HQO+R7objWgS2ymV60GYpI14Ug554w==", + "license": "MIT", + "dependencies": { + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-mdx-expression": "^2.0.0", + "mdast-util-mdx-jsx": "^3.0.0", + "mdast-util-mdxjs-esm": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-mdx-expression": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-mdx-expression/-/mdast-util-mdx-expression-2.0.1.tgz", + "integrity": "sha512-J6f+9hUp+ldTZqKRSg7Vw5V6MqjATc+3E4gf3CFNcuZNWD8XdyI6zQ8GqH7f8169MM6P7hMBRDVGnn7oHB9kXQ==", + "license": "MIT", + "dependencies": { + "@types/estree-jsx": "^1.0.0", + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-mdx-jsx": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/mdast-util-mdx-jsx/-/mdast-util-mdx-jsx-3.2.0.tgz", + "integrity": "sha512-lj/z8v0r6ZtsN/cGNNtemmmfoLAFZnjMbNyLzBafjzikOM+glrjNHPlf6lQDOTccj9n5b0PPihEBbhneMyGs1Q==", + "license": "MIT", + "dependencies": { + "@types/estree-jsx": "^1.0.0", + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "@types/unist": "^3.0.0", + "ccount": "^2.0.0", + "devlop": "^1.1.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0", + "parse-entities": "^4.0.0", + "stringify-entities": "^4.0.0", + "unist-util-stringify-position": "^4.0.0", + "vfile-message": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-mdxjs-esm": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-mdxjs-esm/-/mdast-util-mdxjs-esm-2.0.1.tgz", + "integrity": "sha512-EcmOpxsZ96CvlP03NghtH1EsLtr0n9Tm4lPUJUBccV9RwUOneqSycg19n5HGzCf+10LozMRSObtVr3ee1WoHtg==", + "license": "MIT", + "dependencies": { + "@types/estree-jsx": "^1.0.0", + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-phrasing": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/mdast-util-phrasing/-/mdast-util-phrasing-4.1.0.tgz", + "integrity": "sha512-TqICwyvJJpBwvGAMZjj4J2n0X8QWp21b9l0o7eXyVJ25YNWYbJDVIyD1bZXE6WtV6RmKJVYmQAKWa0zWOABz2w==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "unist-util-is": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-to-hast": { + "version": "13.2.0", + "resolved": "https://registry.npmjs.org/mdast-util-to-hast/-/mdast-util-to-hast-13.2.0.tgz", + "integrity": "sha512-QGYKEuUsYT9ykKBCMOEDLsU5JRObWQusAolFMeko/tYPufNkRffBAQjIE+99jbA87xv6FgmjLtwjh9wBWajwAA==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "@ungap/structured-clone": "^1.0.0", + "devlop": "^1.0.0", + "micromark-util-sanitize-uri": "^2.0.0", + "trim-lines": "^3.0.0", + "unist-util-position": "^5.0.0", + "unist-util-visit": "^5.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-to-markdown": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/mdast-util-to-markdown/-/mdast-util-to-markdown-2.1.2.tgz", + "integrity": "sha512-xj68wMTvGXVOKonmog6LwyJKrYXZPvlwabaryTjLh9LuvovB/KAH+kvi8Gjj+7rJjsFi23nkUxRQv1KqSroMqA==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "@types/unist": "^3.0.0", + "longest-streak": "^3.0.0", + "mdast-util-phrasing": "^4.0.0", + "mdast-util-to-string": "^4.0.0", + "micromark-util-classify-character": "^2.0.0", + "micromark-util-decode-string": "^2.0.0", + "unist-util-visit": "^5.0.0", + "zwitch": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-to-string": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-to-string/-/mdast-util-to-string-4.0.0.tgz", + "integrity": "sha512-0H44vDimn51F0YwvxSJSm0eCDOJTRlmN0R1yBh4HLj9wiV1Dn0QoXGbvFAWj2hSItVTlCmBF1hqKlIyUBVFLPg==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/micromark/-/micromark-4.0.2.tgz", + "integrity": "sha512-zpe98Q6kvavpCr1NPVSCMebCKfD7CA2NqZ+rykeNhONIJBpc1tFKt9hucLGwha3jNTNI8lHpctWJWoimVF4PfA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "@types/debug": "^4.0.0", + "debug": "^4.0.0", + "decode-named-character-reference": "^1.0.0", + "devlop": "^1.0.0", + "micromark-core-commonmark": "^2.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-chunked": "^2.0.0", + "micromark-util-combine-extensions": "^2.0.0", + "micromark-util-decode-numeric-character-reference": "^2.0.0", + "micromark-util-encode": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0", + "micromark-util-resolve-all": "^2.0.0", + "micromark-util-sanitize-uri": "^2.0.0", + "micromark-util-subtokenize": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-core-commonmark": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/micromark-core-commonmark/-/micromark-core-commonmark-2.0.3.tgz", + "integrity": "sha512-RDBrHEMSxVFLg6xvnXmb1Ayr2WzLAWjeSATAoxwKYJV94TeNavgoIdA0a9ytzDSVzBy2YKFK+emCPOEibLeCrg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "decode-named-character-reference": "^1.0.0", + "devlop": "^1.0.0", + "micromark-factory-destination": "^2.0.0", + "micromark-factory-label": "^2.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-factory-title": "^2.0.0", + "micromark-factory-whitespace": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-chunked": "^2.0.0", + "micromark-util-classify-character": "^2.0.0", + "micromark-util-html-tag-name": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0", + "micromark-util-resolve-all": "^2.0.0", + "micromark-util-subtokenize": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-extension-frontmatter": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-frontmatter/-/micromark-extension-frontmatter-2.0.0.tgz", + "integrity": "sha512-C4AkuM3dA58cgZha7zVnuVxBhDsbttIMiytjgsM2XbHAB2faRVaHRle40558FBN+DJcrLNCoqG5mlrpdU4cRtg==", + "license": "MIT", + "dependencies": { + "fault": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm/-/micromark-extension-gfm-3.0.0.tgz", + "integrity": "sha512-vsKArQsicm7t0z2GugkCKtZehqUm31oeGBV/KVSorWSy8ZlNAv7ytjFhvaryUiCUJYqs+NoE6AFhpQvBTM6Q4w==", + "license": "MIT", + "dependencies": { + "micromark-extension-gfm-autolink-literal": "^2.0.0", + "micromark-extension-gfm-footnote": "^2.0.0", + "micromark-extension-gfm-strikethrough": "^2.0.0", + "micromark-extension-gfm-table": "^2.0.0", + "micromark-extension-gfm-tagfilter": "^2.0.0", + "micromark-extension-gfm-task-list-item": "^2.0.0", + "micromark-util-combine-extensions": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-autolink-literal": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-autolink-literal/-/micromark-extension-gfm-autolink-literal-2.1.0.tgz", + "integrity": "sha512-oOg7knzhicgQ3t4QCjCWgTmfNhvQbDDnJeVu9v81r7NltNCVmhPy1fJRX27pISafdjL+SVc4d3l48Gb6pbRypw==", + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-sanitize-uri": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-footnote": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-footnote/-/micromark-extension-gfm-footnote-2.1.0.tgz", + "integrity": "sha512-/yPhxI1ntnDNsiHtzLKYnE3vf9JZ6cAisqVDauhp4CEHxlb4uoOTxOCJ+9s51bIB8U1N1FJ1RXOKTIlD5B/gqw==", + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-core-commonmark": "^2.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0", + "micromark-util-sanitize-uri": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-strikethrough": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-strikethrough/-/micromark-extension-gfm-strikethrough-2.1.0.tgz", + "integrity": "sha512-ADVjpOOkjz1hhkZLlBiYA9cR2Anf8F4HqZUO6e5eDcPQd0Txw5fxLzzxnEkSkfnD0wziSGiv7sYhk/ktvbf1uw==", + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-util-chunked": "^2.0.0", + "micromark-util-classify-character": "^2.0.0", + "micromark-util-resolve-all": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-table": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-table/-/micromark-extension-gfm-table-2.1.1.tgz", + "integrity": "sha512-t2OU/dXXioARrC6yWfJ4hqB7rct14e8f7m0cbI5hUmDyyIlwv5vEtooptH8INkbLzOatzKuVbQmAYcbWoyz6Dg==", + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-tagfilter": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-tagfilter/-/micromark-extension-gfm-tagfilter-2.0.0.tgz", + "integrity": "sha512-xHlTOmuCSotIA8TW1mDIM6X2O1SiX5P9IuDtqGonFhEK0qgRI4yeC6vMxEV2dgyr2TiD+2PQ10o+cOhdVAcwfg==", + "license": "MIT", + "dependencies": { + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-gfm-task-list-item": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/micromark-extension-gfm-task-list-item/-/micromark-extension-gfm-task-list-item-2.1.0.tgz", + "integrity": "sha512-qIBZhqxqI6fjLDYFTBIa4eivDMnP+OZqsNwmQ3xNLE4Cxwc+zfQEfbs6tzAo2Hjq+bh6q5F+Z8/cksrLFYWQQw==", + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-mdx-expression": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/micromark-extension-mdx-expression/-/micromark-extension-mdx-expression-3.0.1.tgz", + "integrity": "sha512-dD/ADLJ1AeMvSAKBwO22zG22N4ybhe7kFIZ3LsDI0GlsNr2A3KYxb0LdC1u5rj4Nw+CHKY0RVdnHX8vj8ejm4Q==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0", + "devlop": "^1.0.0", + "micromark-factory-mdx-expression": "^2.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-events-to-acorn": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-extension-mdx-jsx": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/micromark-extension-mdx-jsx/-/micromark-extension-mdx-jsx-3.0.2.tgz", + "integrity": "sha512-e5+q1DjMh62LZAJOnDraSSbDMvGJ8x3cbjygy2qFEi7HCeUT4BDKCvMozPozcD6WmOt6sVvYDNBKhFSz3kjOVQ==", + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0", + "devlop": "^1.0.0", + "estree-util-is-identifier-name": "^3.0.0", + "micromark-factory-mdx-expression": "^2.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-events-to-acorn": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0", + "vfile-message": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-mdx-md": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-mdx-md/-/micromark-extension-mdx-md-2.0.0.tgz", + "integrity": "sha512-EpAiszsB3blw4Rpba7xTOUptcFeBFi+6PY8VnJ2hhimH+vCQDirWgsMpz7w1XcZE7LVrSAUGb9VJpG9ghlYvYQ==", + "license": "MIT", + "dependencies": { + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-mdxjs": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-mdxjs/-/micromark-extension-mdxjs-3.0.0.tgz", + "integrity": "sha512-A873fJfhnJ2siZyUrJ31l34Uqwy4xIFmvPY1oj+Ean5PHcPBYzEsvqvWGaWcfEIr11O5Dlw3p2y0tZWpKHDejQ==", + "license": "MIT", + "dependencies": { + "acorn": "^8.0.0", + "acorn-jsx": "^5.0.0", + "micromark-extension-mdx-expression": "^3.0.0", + "micromark-extension-mdx-jsx": "^3.0.0", + "micromark-extension-mdx-md": "^2.0.0", + "micromark-extension-mdxjs-esm": "^3.0.0", + "micromark-util-combine-extensions": "^2.0.0", + "micromark-util-types": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-extension-mdxjs-esm": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/micromark-extension-mdxjs-esm/-/micromark-extension-mdxjs-esm-3.0.0.tgz", + "integrity": "sha512-DJFl4ZqkErRpq/dAPyeWp15tGrcrrJho1hKK5uBS70BCtfrIFg81sqcTVu3Ta+KD1Tk5vAtBNElWxtAa+m8K9A==", + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0", + "devlop": "^1.0.0", + "micromark-core-commonmark": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-events-to-acorn": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0", + "unist-util-position-from-estree": "^2.0.0", + "vfile-message": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/micromark-factory-destination": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-factory-destination/-/micromark-factory-destination-2.0.1.tgz", + "integrity": "sha512-Xe6rDdJlkmbFRExpTOmRj9N3MaWmbAgdpSrBQvCFqhezUn4AHqJHbaEnfbVYYiexVSs//tqOdY/DxhjdCiJnIA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-factory-label": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-factory-label/-/micromark-factory-label-2.0.1.tgz", + "integrity": "sha512-VFMekyQExqIW7xIChcXn4ok29YE3rnuyveW3wZQWWqF4Nv9Wk5rgJ99KzPvHjkmPXF93FXIbBp6YdW3t71/7Vg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-factory-mdx-expression": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/micromark-factory-mdx-expression/-/micromark-factory-mdx-expression-2.0.3.tgz", + "integrity": "sha512-kQnEtA3vzucU2BkrIa8/VaSAsP+EJ3CKOvhMuJgOEGg9KDC6OAY6nSnNDVRiVNRqj7Y4SlSzcStaH/5jge8JdQ==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0", + "devlop": "^1.0.0", + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-events-to-acorn": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0", + "unist-util-position-from-estree": "^2.0.0", + "vfile-message": "^4.0.0" + } + }, + "node_modules/micromark-factory-space": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-factory-space/-/micromark-factory-space-2.0.1.tgz", + "integrity": "sha512-zRkxjtBxxLd2Sc0d+fbnEunsTj46SWXgXciZmHq0kDYGnck/ZSGj9/wULTV95uoeYiK5hRXP2mJ98Uo4cq/LQg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-factory-title": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-factory-title/-/micromark-factory-title-2.0.1.tgz", + "integrity": "sha512-5bZ+3CjhAd9eChYTHsjy6TGxpOFSKgKKJPJxr293jTbfry2KDoWkhBb6TcPVB4NmzaPhMs1Frm9AZH7OD4Cjzw==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-factory-whitespace": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-factory-whitespace/-/micromark-factory-whitespace-2.0.1.tgz", + "integrity": "sha512-Ob0nuZ3PKt/n0hORHyvoD9uZhr+Za8sFoP+OnMcnWK5lngSzALgQYKMr9RJVOWLqQYuyn6ulqGWSXdwf6F80lQ==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-factory-space": "^2.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-character": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/micromark-util-character/-/micromark-util-character-2.1.1.tgz", + "integrity": "sha512-wv8tdUTJ3thSFFFJKtpYKOYiGP2+v96Hvk4Tu8KpCAsTMs6yi+nVmGh1syvSCsaxz45J6Jbw+9DD6g97+NV67Q==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-chunked": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-chunked/-/micromark-util-chunked-2.0.1.tgz", + "integrity": "sha512-QUNFEOPELfmvv+4xiNg2sRYeS/P84pTW0TCgP5zc9FpXetHY0ab7SxKyAQCNCc1eK0459uoLI1y5oO5Vc1dbhA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-classify-character": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-classify-character/-/micromark-util-classify-character-2.0.1.tgz", + "integrity": "sha512-K0kHzM6afW/MbeWYWLjoHQv1sgg2Q9EccHEDzSkxiP/EaagNzCm7T/WMKZ3rjMbvIpvBiZgwR3dKMygtA4mG1Q==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-combine-extensions": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-combine-extensions/-/micromark-util-combine-extensions-2.0.1.tgz", + "integrity": "sha512-OnAnH8Ujmy59JcyZw8JSbK9cGpdVY44NKgSM7E9Eh7DiLS2E9RNQf0dONaGDzEG9yjEl5hcqeIsj4hfRkLH/Bg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-chunked": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-decode-numeric-character-reference": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/micromark-util-decode-numeric-character-reference/-/micromark-util-decode-numeric-character-reference-2.0.2.tgz", + "integrity": "sha512-ccUbYk6CwVdkmCQMyr64dXz42EfHGkPQlBj5p7YVGzq8I7CtjXZJrubAYezf7Rp+bjPseiROqe7G6foFd+lEuw==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-decode-string": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-decode-string/-/micromark-util-decode-string-2.0.1.tgz", + "integrity": "sha512-nDV/77Fj6eH1ynwscYTOsbK7rR//Uj0bZXBwJZRfaLEJ1iGBR6kIfNmlNqaqJf649EP0F3NWNdeJi03elllNUQ==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "decode-named-character-reference": "^1.0.0", + "micromark-util-character": "^2.0.0", + "micromark-util-decode-numeric-character-reference": "^2.0.0", + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-encode": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-encode/-/micromark-util-encode-2.0.1.tgz", + "integrity": "sha512-c3cVx2y4KqUnwopcO9b/SCdo2O67LwJJ/UyqGfbigahfegL9myoEFoDYZgkT7f36T0bLrM9hZTAaAyH+PCAXjw==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/micromark-util-events-to-acorn": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/micromark-util-events-to-acorn/-/micromark-util-events-to-acorn-2.0.3.tgz", + "integrity": "sha512-jmsiEIiZ1n7X1Rr5k8wVExBQCg5jy4UXVADItHmNk1zkwEVhBuIUKRu3fqv+hs4nxLISi2DQGlqIOGiFxgbfHg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0", + "@types/unist": "^3.0.0", + "devlop": "^1.0.0", + "estree-util-visit": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0", + "vfile-message": "^4.0.0" + } + }, + "node_modules/micromark-util-html-tag-name": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-html-tag-name/-/micromark-util-html-tag-name-2.0.1.tgz", + "integrity": "sha512-2cNEiYDhCWKI+Gs9T0Tiysk136SnR13hhO8yW6BGNyhOC4qYFnwF1nKfD3HFAIXA5c45RrIG1ub11GiXeYd1xA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/micromark-util-normalize-identifier": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-normalize-identifier/-/micromark-util-normalize-identifier-2.0.1.tgz", + "integrity": "sha512-sxPqmo70LyARJs0w2UclACPUUEqltCkJ6PhKdMIDuJ3gSf/Q+/GIe3WKl0Ijb/GyH9lOpUkRAO2wp0GVkLvS9Q==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-resolve-all": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-resolve-all/-/micromark-util-resolve-all-2.0.1.tgz", + "integrity": "sha512-VdQyxFWFT2/FGJgwQnJYbe1jjQoNTS4RjglmSjTUlpUMa95Htx9NHeYW4rGDJzbjvCsl9eLjMQwGeElsqmzcHg==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-sanitize-uri": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-sanitize-uri/-/micromark-util-sanitize-uri-2.0.1.tgz", + "integrity": "sha512-9N9IomZ/YuGGZZmQec1MbgxtlgougxTodVwDzzEouPKo3qFWvymFHWcnDi2vzV1ff6kas9ucW+o3yzJK9YB1AQ==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "micromark-util-character": "^2.0.0", + "micromark-util-encode": "^2.0.0", + "micromark-util-symbol": "^2.0.0" + } + }, + "node_modules/micromark-util-subtokenize": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/micromark-util-subtokenize/-/micromark-util-subtokenize-2.1.0.tgz", + "integrity": "sha512-XQLu552iSctvnEcgXw6+Sx75GflAPNED1qx7eBJ+wydBb2KCbRZe+NwvIEEMM83uml1+2WSXpBAcp9IUCgCYWA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT", + "dependencies": { + "devlop": "^1.0.0", + "micromark-util-chunked": "^2.0.0", + "micromark-util-symbol": "^2.0.0", + "micromark-util-types": "^2.0.0" + } + }, + "node_modules/micromark-util-symbol": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/micromark-util-symbol/-/micromark-util-symbol-2.0.1.tgz", + "integrity": "sha512-vs5t8Apaud9N28kgCrRUdEed4UJ+wWNvicHLPxCa9ENlYuAY31M0ETy5y1vA33YoNPDFTghEbnh6efaE8h4x0Q==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/micromark-util-types": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/micromark-util-types/-/micromark-util-types-2.0.2.tgz", + "integrity": "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA==", + "funding": [ + { + "type": "GitHub Sponsors", + "url": "https://github.com/sponsors/unifiedjs" + }, + { + "type": "OpenCollective", + "url": "https://opencollective.com/unified" + } + ], + "license": "MIT" + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/parse-entities": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/parse-entities/-/parse-entities-4.0.2.tgz", + "integrity": "sha512-GG2AQYWoLgL877gQIKeRPGO1xF9+eG1ujIb5soS5gPvLQ1y2o8FL90w2QWNdf9I361Mpp7726c+lj3U0qK1uGw==", + "license": "MIT", + "dependencies": { + "@types/unist": "^2.0.0", + "character-entities-legacy": "^3.0.0", + "character-reference-invalid": "^2.0.0", + "decode-named-character-reference": "^1.0.0", + "is-alphanumerical": "^2.0.0", + "is-decimal": "^2.0.0", + "is-hexadecimal": "^2.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/parse-entities/node_modules/@types/unist": { + "version": "2.0.11", + "resolved": "https://registry.npmjs.org/@types/unist/-/unist-2.0.11.tgz", + "integrity": "sha512-CmBKiL6NNo/OqgmMn95Fk9Whlp2mtvIv+KNpQKN2F4SjvrEesubTRWGYSg+BnWZOnlCaSTU1sMpsBOzgbYhnsA==", + "license": "MIT" + }, + "node_modules/parse5": { + "version": "7.3.0", + "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz", + "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==", + "license": "MIT", + "dependencies": { + "entities": "^6.0.0" + }, + "funding": { + "url": "https://github.com/inikulin/parse5?sponsor=1" + } + }, + "node_modules/property-information": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/property-information/-/property-information-7.1.0.tgz", + "integrity": "sha512-TwEZ+X+yCJmYfL7TPUOcvBZ4QfoT5YenQiJuX//0th53DE6w0xxLEtfK3iyryQFddXuvkIk51EEgrJQ0WJkOmQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/rehype-minify-whitespace": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/rehype-minify-whitespace/-/rehype-minify-whitespace-6.0.2.tgz", + "integrity": "sha512-Zk0pyQ06A3Lyxhe9vGtOtzz3Z0+qZ5+7icZ/PL/2x1SHPbKao5oB/g/rlc6BCTajqBb33JcOe71Ye1oFsuYbnw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-minify-whitespace": "^1.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/rehype-raw": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/rehype-raw/-/rehype-raw-7.0.0.tgz", + "integrity": "sha512-/aE8hCfKlQeA8LmyeyQvQF3eBiLRGNlfBJEvWH7ivp9sBqs7TNqBL5X3v157rM4IFETqDnIOO+z5M/biZbo9Ww==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-raw": "^9.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/rehype-stringify": { + "version": "10.0.1", + "resolved": "https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-10.0.1.tgz", + "integrity": "sha512-k9ecfXHmIPuFVI61B9DeLPN0qFHfawM6RsuX48hoqlaKSF61RskNjSm1lI8PhBEM0MRdLxVVm4WmTqJQccH9mA==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-to-html": "^9.0.0", + "unified": "^11.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-frontmatter": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/remark-frontmatter/-/remark-frontmatter-5.0.0.tgz", + "integrity": "sha512-XTFYvNASMe5iPN0719nPrdItC9aU0ssC4v14mH1BCi1u0n1gAocqcujWUrByftZTbLhRtiKRyjYTSIOcr69UVQ==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-frontmatter": "^2.0.0", + "micromark-extension-frontmatter": "^2.0.0", + "unified": "^11.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-gfm": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/remark-gfm/-/remark-gfm-4.0.1.tgz", + "integrity": "sha512-1quofZ2RQ9EWdeN34S79+KExV1764+wCUGop5CPL1WGdD0ocPpu91lzPGbwWMECpEpd42kJGQwzRfyov9j4yNg==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-gfm": "^3.0.0", + "micromark-extension-gfm": "^3.0.0", + "remark-parse": "^11.0.0", + "remark-stringify": "^11.0.0", + "unified": "^11.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-mdx": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/remark-mdx/-/remark-mdx-3.1.1.tgz", + "integrity": "sha512-Pjj2IYlUY3+D8x00UJsIOg5BEvfMyeI+2uLPn9VO9Wg4MEtN/VTIq2NEJQfde9PnX15KgtHyl9S0BcTnWrIuWg==", + "license": "MIT", + "dependencies": { + "mdast-util-mdx": "^3.0.0", + "micromark-extension-mdxjs": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-parse": { + "version": "11.0.0", + "resolved": "https://registry.npmjs.org/remark-parse/-/remark-parse-11.0.0.tgz", + "integrity": "sha512-FCxlKLNGknS5ba/1lmpYijMUzX2esxW5xQqjWxw2eHFfS2MSdaHVINFmhjo+qN1WhZhNimq0dZATN9pH0IDrpA==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-from-markdown": "^2.0.0", + "micromark-util-types": "^2.0.0", + "unified": "^11.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-rehype": { + "version": "11.1.2", + "resolved": "https://registry.npmjs.org/remark-rehype/-/remark-rehype-11.1.2.tgz", + "integrity": "sha512-Dh7l57ianaEoIpzbp0PC9UKAdCSVklD8E5Rpw7ETfbTl3FqcOOgq5q2LVDhgGCkaBv7p24JXikPdvhhmHvKMsw==", + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "mdast-util-to-hast": "^13.0.0", + "unified": "^11.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/remark-stringify": { + "version": "11.0.0", + "resolved": "https://registry.npmjs.org/remark-stringify/-/remark-stringify-11.0.0.tgz", + "integrity": "sha512-1OSmLd3awB/t8qdoEOMazZkNsfVTeY4fTsgzcQFdXNq8ToTN4ZGwrMnlda4K6smTFKD+GRV6O48i6Z4iKgPPpw==", + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-to-markdown": "^2.0.0", + "unified": "^11.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/space-separated-tokens": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/space-separated-tokens/-/space-separated-tokens-2.0.2.tgz", + "integrity": "sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/stringify-entities": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz", + "integrity": "sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg==", + "license": "MIT", + "dependencies": { + "character-entities-html4": "^2.0.0", + "character-entities-legacy": "^3.0.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/trim-lines": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz", + "integrity": "sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/trim-trailing-lines": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/trim-trailing-lines/-/trim-trailing-lines-2.1.0.tgz", + "integrity": "sha512-5UR5Biq4VlVOtzqkm2AZlgvSlDJtME46uV0br0gENbwN4l5+mMKT4b9gJKqWtuL2zAIqajGJGuvbCbcAJUZqBg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/trough": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/trough/-/trough-2.2.0.tgz", + "integrity": "sha512-tmMpK00BjZiUyVyvrBK7knerNgmgvcV/KLVyuma/SC+TQN167GrMRciANTz09+k3zW8L8t60jWO1GpfkZdjTaw==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/unified": { + "version": "11.0.5", + "resolved": "https://registry.npmjs.org/unified/-/unified-11.0.5.tgz", + "integrity": "sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "bail": "^2.0.0", + "devlop": "^1.0.0", + "extend": "^3.0.0", + "is-plain-obj": "^4.0.0", + "trough": "^2.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-find-after": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/unist-util-find-after/-/unist-util-find-after-5.0.0.tgz", + "integrity": "sha512-amQa0Ep2m6hE2g72AugUItjbuM8X8cGQnFoHk0pGfrFeT9GZhzN5SW8nRsiGKK7Aif4CrACPENkA6P/Lw6fHGQ==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-is": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/unist-util-is/-/unist-util-is-6.0.1.tgz", + "integrity": "sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-position": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/unist-util-position/-/unist-util-position-5.0.0.tgz", + "integrity": "sha512-fucsC7HjXvkB5R3kTCO7kUjRdrS0BJt3M/FPxmHMBOm8JQi2BsHAHFsy27E0EolP8rp0NzXsJ+jNPyDWvOJZPA==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-position-from-estree": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/unist-util-position-from-estree/-/unist-util-position-from-estree-2.0.0.tgz", + "integrity": "sha512-KaFVRjoqLyF6YXCbVLNad/eS4+OfPQQn2yOd7zF/h5T/CSL2v8NpN6a5TPvtbXthAGw5nG+PuTtq+DdIZr+cRQ==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-remove": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/unist-util-remove/-/unist-util-remove-4.0.0.tgz", + "integrity": "sha512-b4gokeGId57UVRX/eVKej5gXqGlc9+trkORhFJpu9raqZkZhU0zm8Doi05+HaiBsMEIJowL+2WtQ5ItjsngPXg==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0", + "unist-util-visit-parents": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-stringify-position": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-4.0.0.tgz", + "integrity": "sha512-0ASV06AAoKCDkS2+xw5RXJywruurpbC4JZSm7nr7MOt1ojAzvyyaO+UxZf18j8FCF6kmzCZKcAgN/yu2gm2XgQ==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-visit": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/unist-util-visit/-/unist-util-visit-5.0.0.tgz", + "integrity": "sha512-MR04uvD+07cwl/yhVuVWAtw+3GOR/knlL55Nd/wAdblk27GCVt3lqpTivy/tkJcZoNPzTwS1Y+KMojlLDhoTzg==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0", + "unist-util-visit-parents": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/unist-util-visit-parents": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-6.0.2.tgz", + "integrity": "sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-is": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vfile": { + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/vfile/-/vfile-6.0.3.tgz", + "integrity": "sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "vfile-message": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vfile-location": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/vfile-location/-/vfile-location-5.0.3.tgz", + "integrity": "sha512-5yXvWDEgqeiYiBe1lbxYF7UMAIm/IcopxMHrMQDq3nvKcjPKIhZklUKL+AE7J7uApI4kwe2snsK+eI6UTj9EHg==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "vfile": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/vfile-message": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-4.0.3.tgz", + "integrity": "sha512-QTHzsGd1EhbZs4AsQ20JX1rC3cOlt/IWJruk893DfLRr57lcnOeMaWG4K0JrRta4mIJZKth2Au3mM3u03/JWKw==", + "license": "MIT", + "dependencies": { + "@types/unist": "^3.0.0", + "unist-util-stringify-position": "^4.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/web-namespaces": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/web-namespaces/-/web-namespaces-2.0.1.tgz", + "integrity": "sha512-bKr1DkiNa2krS7qxNtdrtHAmzuYGFQLiQ13TsorsdT6ULTkPLKuu5+GsFpDlg6JFjUTwX2DyhMPG2be8uPrqsQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, + "node_modules/zwitch": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/zwitch/-/zwitch-2.0.4.tgz", + "integrity": "sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + } + } +} diff --git a/tools/mdx-transform/package.json b/tools/mdx-transform/package.json new file mode 100644 index 00000000..9e4a0528 --- /dev/null +++ b/tools/mdx-transform/package.json @@ -0,0 +1,34 @@ +{ + "name": "mdx-transform", + "version": "0.1.0", + "type": "module", + "private": true, + "bin": { + "mdx-transform": "transform.mjs" + }, + "description": "", + "main": "index.js", + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + }, + "keywords": [], + "author": "", + "license": "ISC", + "dependencies": { + "hast-util-to-mdast": "^10.1.2", + "hast-util-to-text": "^4.0.2", + "mdast-util-to-hast": "^13.2.0", + "rehype-raw": "^7.0.0", + "rehype-stringify": "^10.0.1", + "remark-frontmatter": "^5.0.0", + "remark-gfm": "^4.0.1", + "remark-mdx": "^3.1.1", + "remark-parse": "^11.0.0", + "remark-rehype": "^11.1.2", + "remark-stringify": "^11.0.0", + "unified": "^11.0.5", + "unist-util-is": "^6.0.1", + "unist-util-remove": "^4.0.0", + "unist-util-visit": "^5.0.0" + } +} diff --git a/tools/mdx-transform/transform.mjs b/tools/mdx-transform/transform.mjs new file mode 100644 index 00000000..afa0861c --- /dev/null +++ b/tools/mdx-transform/transform.mjs @@ -0,0 +1,357 @@ +#!/usr/bin/env node + +import fs from 'fs'; +import path from 'path'; +import {fileURLToPath, pathToFileURL} from 'url'; +import {unified} from 'unified'; +import remarkParse from 'remark-parse'; +import remarkFrontmatter from 'remark-frontmatter'; +import remarkGfm from 'remark-gfm'; +import remarkStringify from 'remark-stringify'; +import {toHast} from 'mdast-util-to-hast'; +import {toMdast} from 'hast-util-to-mdast'; +import rehypeRaw from 'rehype-raw'; +import {visit} from 'unist-util-visit'; +import {toString} from 'mdast-util-to-string'; +import {VFile} from 'vfile'; +import {toText} from 'hast-util-to-text'; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); + +const remarkBase = unified() + .use(remarkParse) + .use(remarkFrontmatter, ['yaml']) + .use(remarkGfm); + +function escapeYaml(str) { + return str.replace(/'/g, "''"); +} + +function normalizeLinkContinuations(content) { + return content.replace(/\[([^\]]+)]\s*\((https?:\/\/[^\s)]+)\)\s*\{:[^}]*\}/g, '[$1]($2)'); +} + +function preprocessMdast(tree, file) { + const metadata = []; + let title = null; + const retained = []; + let encounteredHeading = false; + + for (const node of tree.children) { + if (node.type === 'paragraph') { + const text = toString(node).trim(); + if (!text) { + continue; + } + if (text === '{% include "_buttons.html" %}' || text.startsWith('{% ') || text.includes('{% dynamic setvar')) { + continue; + } + } + + if (node.type === 'html') { + const value = node.value.trim(); + if (value === '{% include "_buttons.html" %}' || value.startsWith('{% ') || value.includes('{% dynamic setvar')) { + continue; + } + } + + if (!encounteredHeading) { + if (node.type === 'heading' && node.depth === 1 && !title) { + title = toString(node).trim(); + encounteredHeading = true; + continue; + } + if (node.type === 'paragraph') { + const text = toString(node).trim(); + if (/^[A-Za-z0-9_-]+: /.test(text)) { + if (!text.includes('_project.yaml') && !text.includes('_book.yaml')) { + metadata.push(text); + } + continue; + } + } + } + retained.push(node); + } + + tree.children = retained; + const frontmatter = []; + if (title) { + frontmatter.push(`title: '${escapeYaml(title)}'`); + } + for (const line of metadata) { + frontmatter.push(line); + } + if (frontmatter.length > 0) { + file.data.frontmatter = frontmatter.join('\n'); + } +} + +function stripLiquidExpressions(value) { + if (typeof value !== 'string' || value.indexOf('{{') === -1) { + return value; + } + return value.replace(/\{\{\s*['"]?([^{}'"]+?)['"]?\s*\}\}/g, (_, inner) => inner.trim()); +} + +function sanitizeMdast(tree) { + visit(tree, (node) => { + if (node.type === 'raw') { + const value = typeof node.value === 'string' ? node.value : ''; + const cleaned = stripLiquidExpressions( + value.replace(/\{:[^}]*\}/g, ''), + ).replace(//g, ''); + node.type = 'html'; + node.value = cleaned; + } + if (node.type === 'text' || node.type === 'html' || node.type === 'code' || node.type === 'inlineCode') { + let value = node.value; + if (typeof value === 'string') { + value = stripLiquidExpressions(value); + value = value + .replace(/\{:[^}]*\}/g, '') + .replace(/\{#[^#]*?#\}/g, '') + .replace(//g, '') + .replace(/<([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,})>/g, '$1') + .replace(/\s]+>/g, (match) => { + const url = match.slice(1, -1); + return `[${url}](${url})`; + }); + if (node.type === 'code' || node.type === 'inlineCode') { + value = value.replace(//g, '>'); + } + if (node.type === 'text') { + value = value + .replace(//g, '>') + .replace(/\{/g, '{') + .replace(/\}/g, '}'); + } + node.value = value; + } + } + }); +} + +function classList(value) { + if (!value) return []; + if (Array.isArray(value)) { + return value; + } + if (typeof value === 'string') { + return value.split(/\s+/).filter(Boolean); + } + return []; +} + +function convertNavTable(node) { + const anchors = []; + visit(node, 'element', (el) => { + if (el.tagName === 'a' && el.properties && el.properties.href) { + const label = toText(el).replace(/arrow_(back|forward(_ios)?)?/gi, '').trim(); + anchors.push({href: el.properties.href, label}); + } + }); + if (anchors.length === 0) { + return null; + } + const children = []; + anchors.slice(0, 2).forEach((anchor, index) => { + const label = index === 0 && !anchor.label.startsWith('←') + ? `← ${anchor.label}` + : index === 1 && !anchor.label.endsWith('→') + ? `${anchor.label} →` + : anchor.label; + if (index > 0) { + children.push({type: 'text', value: ' · '}); + } + children.push({ + type: 'element', + tagName: 'a', + properties: {href: anchor.href}, + children: [{type: 'text', value: label}], + }); + }); + return { + type: 'element', + tagName: 'p', + properties: {}, + children, + }; +} + +export function rehypeCustom() { + return (tree) => { + visit(tree, 'element', (node, index, parent) => { + if (!parent) return; + + if (node.properties) { + for (const key of Object.keys(node.properties)) { + const value = node.properties[key]; + if (key === 'style') { + delete node.properties[key]; + continue; + } + if (typeof value === 'string' && /[\r\n]/.test(value)) { + const collapsed = value.replace(/\s+/g, ' ').trim(); + node.properties[key] = collapsed; + } else if (Array.isArray(value)) { + node.properties[key] = value.map((item) => + typeof item === 'string' && /[\r\n]/.test(item) + ? item.replace(/\s+/g, ' ').trim() + : item, + ); + } + } + } + + // Normalize span icons → text arrows + if (node.tagName === 'span') { + const classNames = classList(node.properties?.className); + const aria = node.properties?.ariaHidden; + if (classNames.includes('material-icons') || aria === 'true') { + const text = toText(node); + if (/arrow_back/.test(text)) { + parent.children[index] = {type: 'text', value: '←'}; + return; + } + if (/arrow_forward(_ios)?/.test(text)) { + parent.children[index] = {type: 'text', value: '→'}; + return; + } + } + if (node.properties && node.properties.id) { + parent.children[index] = { + type: 'element', + tagName: 'a', + properties: {id: node.properties.id}, + children: [], + }; + return; + } + } + + // Navigation tables + if (node.tagName === 'table') { + const classes = classList(node.properties?.className); + if (classes.includes('columns')) { + const replacement = convertNavTable(node); + if (replacement) { + parent.children.splice(index, 1, replacement); + } else { + parent.children.splice(index, 1); + } + return [visit.SKIP, index]; + } + } + + // Compare callouts + if (node.tagName === 'p' && node.children?.length) { + const first = node.children[0]; + if (first?.type === 'element' && first.tagName === 'span') { + const classes = classList(first.properties?.className); + const compare = classes.find((cls) => cls.startsWith('compare-')); + if (compare) { + const type = compare === 'compare-better' ? 'success' : 'warning'; + const title = toText(first).replace(/[–—-]\s*$/,'').trim() || (type === 'success' ? 'Yes' : 'No'); + const bodyNodes = node.children.slice(1); + const bodyText = toText({type: 'element', tagName: 'div', children: bodyNodes}).trim(); + const callout = `${bodyText ? '\n' + bodyText + '\n' : ''}`; + parent.children[index] = {type: 'raw', value: callout}; + return [visit.SKIP, index]; + } + } + } + + // Compare list items + if (node.tagName === 'li' && node.children?.length) { + const first = node.children[0]; + if (first?.type === 'element' && first.tagName === 'span') { + const classes = classList(first.properties?.className); + const compare = classes.find((cls) => cls.startsWith('compare-')); + if (compare) { + const icon = compare === 'compare-better' ? '✅' : '⚠️'; + const label = toText(first).trim(); + const rest = node.children.slice(1); + const restText = toText({type: 'element', tagName: 'div', children: rest}).trim(); + node.children = [{type: 'text', value: `${icon} ${label}:${restText ? ' ' + restText : ''}`}]; + } + } + } + + // Normalize align attr + if (node.properties?.align && !node.properties.style) { + node.properties.align = node.properties.align.toString(); + } + + if (node.tagName === 'img') { + node.properties = node.properties || {}; + if (node.properties.align) { + node.properties.align = node.properties.align.toString(); + } + } + }); + }; +} + +export function transformContent(content) { + const normalizedContent = normalizeLinkContinuations(content); + const file = new VFile({value: normalizedContent}); + const mdast = remarkBase.parse(file); + preprocessMdast(mdast, file); + sanitizeMdast(mdast); + + let hast = toHast(mdast, {allowDangerousHtml: true}); + hast = unified().use(rehypeRaw).runSync(hast, file); + hast = unified().use(rehypeCustom).runSync(hast, file); + + const mdastResult = toMdast(hast, {}); + sanitizeMdast(mdastResult); + + let output = unified() + .use(remarkGfm) + .use(remarkStringify, { + bullet: '-', + fences: true, + entities: {useNamedReferences: true}, + listItemIndent: 'one', + allowDangerousHtml: true, + }) + .stringify(mdastResult); + output = output.replace(/\n{3,}/g, '\n\n'); + if (file.data.frontmatter) { + output = `---\n${file.data.frontmatter}\n---\n\n${output}`; + } + output = stripLiquidExpressions(output); + output = output.replace(/<([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,})>/g, '[$1](mailto:$1)'); + output = output + .replace(/\\(<\/??Callout)/g, '$1') + .replace(/\s]+>/g, (match) => { + const url = match.slice(1, -1); + return `[${url}](${url})`; + }) + .replace(/https\\:\/\//g, 'https://') + .replace(/http\\:\/\//g, 'http://'); + return output; +} + +function runCli() { + const [, , inputPath, outputPath] = process.argv; + if (!inputPath || !outputPath) { + console.error('Usage: mdx-transform '); + process.exit(1); + } + const absoluteInput = path.resolve(inputPath); + const absoluteOutput = path.resolve(outputPath); + const content = fs.readFileSync(absoluteInput, 'utf8'); + const transformed = transformContent(content); + fs.mkdirSync(path.dirname(absoluteOutput), {recursive: true}); + fs.writeFileSync(absoluteOutput, transformed, 'utf8'); +} + +if (import.meta.url === pathToFileURL(process.argv[1]).href) { + runCli(); +} + +export {preprocessMdast, sanitizeMdast}; diff --git a/transform-docs.awk b/transform-docs.awk index e5f230ff..d55e9c67 100755 --- a/transform-docs.awk +++ b/transform-docs.awk @@ -2,18 +2,162 @@ # Transform script for converting Bazel docs to Mintlify MDX format # Usage: awk -f transform-docs.awk input.md > output.mdx +# Trim leading/trailing whitespace +function trim(str, s) { + s = str + gsub(/^[ \t\r\n]+/, "", s) + gsub(/[ \t\r\n]+$/, "", s) + return s +} + +# Decode common HTML entities +function html_decode(text, t) { + t = text + gsub(/&/, "&", t) + gsub(/</, "<", t) + gsub(/>/, ">", t) + gsub(/"/, "\"", t) + gsub(/'/, "'", t) + gsub(/ /, " ", t) + return t +} + +# Convert inline HTML tags to Markdown/MDX-friendly syntax +function inline_to_md(text, tmp) { + tmp = text + gsub(//, "`", tmp) + gsub(/<\/code>/, "`", tmp) + gsub(//, "**", tmp) + gsub(/<\/strong>/, "**", tmp) + gsub(//, "*", tmp) + gsub(/<\/em>/, "*", tmp) + gsub(/

/, "", tmp) + gsub(/<\/p>/, "", tmp) + tmp = html_decode(tmp) + return trim(tmp) +} + +# Map PrettyPrint language labels to code fence languages +function map_lang(lang) { + lang = tolower(lang) + if (lang == "py" || lang == "python") return "python" + if (lang == "shell" || lang == "sh" || lang == "bash") return "bash" + if (lang == "java" || lang == "lang-java") return "java" + if (lang == "lang") return "" + return lang +} + +# Emit a Mintlify Callout component +function emit_callout(type, title, body) { + print "" + print body + print "" + print "" +} + +# Convert navigation tables into simple Markdown links +function emit_navigation(text, sanitized, count, hrefs, labels, match_str, href_val, label, prev_label, next_label, output_line, i) { + sanitized = text + gsub(/\n/, " ", sanitized) + gsub(/]*>[^<]*<\/span>/, "", sanitized) + gsub(/<\/?(tr|td|table)>/, "", sanitized) + gsub(/class="[^"]*"/, "", sanitized) + + count = 0 + while (match(sanitized, /]*href="[^"]*"[^>]*>[^<]*<\/a>/)) { + match_str = substr(sanitized, RSTART, RLENGTH) + sanitized = substr(sanitized, RSTART + RLENGTH) + + href_val = "" + if (match(match_str, /href="[^"]*"/)) { + href_val = substr(match_str, RSTART + 6, RLENGTH - 7) + } + + label = match_str + sub(/^[^>]*>/, "", label) + sub(/<\/a>.*$/, "", label) + gsub(/[[:space:]]+/, " ", label) + label = trim(label) + gsub(/arrow_forward/, "", label) + gsub(/arrow_back/, "", label) + + count++ + hrefs[count] = href_val + labels[count] = label + } + + if (count == 0) { + return + } + + prev_label = trim(labels[1]) + if (prev_label !~ /←/) { + prev_label = "← " prev_label + } + + output_line = "[" prev_label "](" hrefs[1] ")" + + if (count > 1 && hrefs[2] != "") { + next_label = trim(labels[2]) + if (next_label !~ /→/) { + next_label = next_label " →" + } + output_line = output_line " · [" next_label "](" hrefs[2] ")" + } + + print "" + print output_line + print "" + + for (i in hrefs) { delete hrefs[i] } + for (i in labels) { delete labels[i] } +} +# Convert compare paragraphs into Mintlify Callouts +function handle_compare(text, type, title_segment, title, body_segment, content) { + type = (index(text, "compare-better") > 0) ? "success" : "warning" + + title_segment = text + sub(/^

/, "", title_segment) + sub(/<\/span>.*/, "", title_segment) + title = inline_to_md(title_segment) + + body_segment = text + sub(/^

[^<]*<\/span>/, "", body_segment) + sub(/<\/p>[[:space:]]*$/, "", body_segment) + gsub(/\n[ \t]*/, " ", body_segment) + sub(/^[[:space:]]*(—|--|-)?[[:space:]]*/, "", body_segment) + content = inline_to_md(body_segment) + + emit_callout(type, title, content) +} + BEGIN { in_frontmatter = 0 first_h1_found = 0 frontmatter_printed = 0 before_first_h1 = 1 in_code_block = 0 + in_pre_block = 0 + meta_index = 0 + capture_compare = 0 + compare_buffer = "" + capture_nav_table = 0 + nav_buffer = "" } # Skip Jekyll front-matter lines /^Project: \/_project\.yaml$/ { next } /^Book: \/_book\.yaml$/ { next } +# Skip Starlark lint directives embedded as comments +/^\{#.*#\}$/ { next } + +# Stash metadata lines before the first H1 so we can emit them as frontmatter +before_first_h1 && /^[A-Za-z0-9_-]+: / { + meta_lines[meta_index++] = $0 + next +} + # Remove lines that contain only '{% include "_buttons.html" %}' /^{% include "_buttons\.html" %}$/ { next } @@ -23,6 +167,51 @@ BEGIN { # Remove any lines that start with '{%' /^{%/ { next } +# Convert

 blocks (with optional classes) into fenced code blocks
+/^[ \t]*
]*>.*<\/pre>/) {
+        content = line
+        sub(/^[ \t]*]*>/, "", content)
+        sub(/<\/pre>[ \t]*$/, "", content)
+        gsub(/<\/?span[^>]*>/, "", content)
+        gsub(/<\/?div[^>]*>/, "", content)
+        gsub(/<\/?code[^>]*>/, "", content)
+        content = html_decode(content)
+        print "```" lang
+        print content
+        print "```"
+        next
+    }
+    print "```" lang
+    in_pre_block = 1
+    next
+}
+
+in_pre_block && /<\/pre>/ {
+    print "```"
+    in_pre_block = 0
+    next
+}
+
+in_pre_block {
+    line = $0
+    gsub(/<\/?span[^>]*>/, "", line)
+    gsub(/<\/?div[^>]*>/, "", line)
+    gsub(/<\/?code[^>]*>/, "", line)
+    line = html_decode(line)
+    print line
+    next
+}
+
 # Track code blocks to avoid processing their content
 /^```/ {
     in_code_block = !in_code_block
@@ -60,32 +249,132 @@ in_code_block {
     next
 }
 
-# Convert 
 tags to markdown code blocks
-/^
/ {
-    gsub(/^
/, "```")
-    gsub(/<\/pre>$/, "```")
-    print
+# Convert navigation tables into inline Markdown
+capture_nav_table == 1 {
+    nav_buffer = nav_buffer "\n" $0
+    if ($0 ~ /<\/table>/) {
+        emit_navigation(nav_buffer)
+        nav_buffer = ""
+        capture_nav_table = 0
+    }
     next
 }
 
-# Fix 
 tags that don't close properly
-/]*>/ {
-    # If it has content after the tag and ends with ```, it's malformed
-    if (/]*>[^<]*```$/) {
-        # Replace content``` with just content (already has ```)
-        gsub(/]*>/, "", $0)
-        print
-        next
+/^/ {
+    capture_nav_table = 1
+    nav_buffer = $0
+    next
+}
+
+# Convert compare callouts (Wrong/Correct) to Mintlify Callout components
+capture_compare == 1 {
+    compare_buffer = compare_buffer "\n" $0
+    if ($0 ~ /<\/p>/) {
+        handle_compare(compare_buffer)
+        compare_buffer = ""
+        capture_compare = 0
     }
-    # If it has content after the tag, it's likely malformed
-    if (/]*>[^<]*$/) {
-        gsub(/]*>/, "```", $0)
+    next
+}
+
+/^

/ { + compare_buffer = $0 + if ($0 ~ /<\/p>/) { + handle_compare(compare_buffer) + compare_buffer = "" + } else { + capture_compare = 1 } + next } -# Remove tags that appear at the end of lines -{ - gsub(/<\/pre>$/, "```", $0) +# Handle inline compare badges inside lists +/^[ \t]*[-*][ \t]*/ { + line = $0 + icon = (index(line, "compare-better") > 0) ? "✅" : "⚠️" + label_segment = line + sub(/^[ \t]*[-*][ \t]*/, "", label_segment) + sub(/<\/span>.*/, "", label_segment) + label = inline_to_md(label_segment) + rest_segment = line + sub(/^.*<\/span>:[[:space:]]*/, "", rest_segment) + rest = inline_to_md(rest_segment) + print "- " icon " **" label "**: " rest + next +} + +# Promote **Note:** style callouts to Mintlify Callout components +/^\*\*Note\*\*:/ { + line = $0 + sub(/^\*\*Note\*\*:[ \t]*/, "", line) + body = inline_to_md(line) + emit_callout("info", "Note", body) + next +} + +/^[ \t]*

[ \t]*$/ { next } + +# Convert styled horizontal rules to Markdown +/^
and
-/^[^<]*]*>[^<]*$/ { +/^[^<]*]*>[[:space:]]*[^<[:space:]]+[[:space:]]*$/ { if ($0 !~ /<\/th>$/) { $0 = $0 "" } @@ -281,6 +570,9 @@ in_code_block { gsub(/'/, "''", title) print "---" print "title: '" title "'" + for (i = 0; i < meta_index; i++) { + print meta_lines[i] + } print "---" print "" first_h1_found = 1 @@ -288,7 +580,26 @@ in_code_block { next } -# Print all other lines { + gsub(/ class="/, " className=\"", $0) + gsub(/ style="[^"]*"/, "", $0) + gsub(/<([A-Za-z0-9]+) /, "<\\1 ", $0) + gsub(/ (href|target|rel|aria|data)/, " \\1", $0) + gsub(/[ \t]+>/, ">", $0) +} + +{ + gsub(/]*>arrow_back<\/span>/, "←", $0) + gsub(/]*>arrow_forward<\/span>/, "→", $0) + gsub(/]*>arrow_forward_ios<\/span>/, "→", $0) +} + +{ + gsub(/ className="[^"]*"/, "", $0) + gsub(/<\/?span[^>]*>/, "", $0) +} + +{ + $0 = html_decode($0) print -} \ No newline at end of file +} diff --git a/tutorials/ccp-toolchain-config.mdx b/tutorials/ccp-toolchain-config.mdx index 0b14cb55..3595614f 100644 --- a/tutorials/ccp-toolchain-config.mdx +++ b/tutorials/ccp-toolchain-config.mdx @@ -2,473 +2,407 @@ title: 'Bazel Tutorial: Configure C++ Toolchains' --- +This tutorial uses an example scenario to describe how to configure C++ toolchains for a project. - -This tutorial uses an example scenario to describe how to configure C++ -toolchains for a project. - -## What you'll learn +## What you'll learn In this tutorial you learn how to: -* Set up the build environment -* Use `--toolchain_resolution_debug` to debug toolchain resolution -* Configure the C++ toolchain -* Create a Starlark rule that provides additional configuration for the - `cc_toolchain` so that Bazel can build the application with `clang` -* Build the C++ binary by running `bazel build //main:hello-world` on a - Linux machine -* Cross-compile the binary for android by running `bazel build - //main:hello-world --platforms=//:android_x86_64` +- Set up the build environment +- Use `--toolchain_resolution_debug` to debug toolchain resolution +- Configure the C++ toolchain +- Create a Starlark rule that provides additional configuration for the `cc_toolchain` so that Bazel can build the application with `clang` +- Build the C++ binary by running `bazel build //main:hello-world` on a Linux machine +- Cross-compile the binary for android by running `bazel build //main:hello-world --platforms=//:android_x86_64` -## Before you begin +## Before you begin -This tutorial assumes you are on Linux and have successfully built C++ -applications and installed the appropriate tooling and libraries. The tutorial -uses `clang version 19`, which you can install on your system. +This tutorial assumes you are on Linux and have successfully built C++ applications and installed the appropriate tooling and libraries. The tutorial uses `clang version 19`, which you can install on your system. -### Set up the build environment +### Set up the build environment Set up your build environment as follows: -1. If you have not already done so, [download and install Bazel - 7.0.2](https://bazel.build/install) or later. - -2. Add an empty `MODULE.bazel` file at the root folder. +1. If you have not already done so, [download and install Bazel 7.0.2](https://bazel.build/install) or later. + +2. Add an empty `MODULE.bazel` file at the root folder. + +3. Add the following `cc_binary` target to the `main/BUILD` file: + + ```python + cc_binary( + name = "hello-world", + srcs = ["hello-world.cc"], + ) + ``` + + Because Bazel uses many internal tools written in C++ during the build, such as `process-wrapper`, the pre-existing default C++ toolchain is specified for the host platform. This enables these internal tools to build using that toolchain of the one created in this tutorial. Hence, the `cc_binary` target is also built with the default toolchain. + +4. Run the build with the following command: + + ```bash + bazel build //main:hello-world + ``` -3. Add the following `cc_binary` target to the `main/BUILD` file: + The build succeeds without any toolchain registered in `MODULE.bazel`. - ```python - cc_binary( - name = "hello-world", - srcs = ["hello-world.cc"], - ) - ``` + To further see what's under the hood, run: - Because Bazel uses many internal tools written in C++ during the build, such - as `process-wrapper`, the pre-existing default C++ toolchain is specified - for the host platform. This enables these internal tools to build using that - toolchain of the one created in this tutorial. Hence, the `cc_binary` target - is also built with the default toolchain. + ```bash + bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' -4. Run the build with the following command: + INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 + ``` - ```bash - bazel build //main:hello-world - ``` + Without specifying `--platforms`, Bazel builds the target for `@platforms//host` using `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. - The build succeeds without any toolchain registered in `MODULE.bazel`. +## Configure the C++ toolchain - To further see what's under the hood, run: +To configure the C++ toolchain, repeatedly build the application and eliminate each error one by one as described as following. - ```bash - bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' - - INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 - ``` - - Without specifying `--platforms`, Bazel builds the target for - `@platforms//host` using - `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. - -## Configure the C++ toolchain - -To configure the C++ toolchain, repeatedly build the application and eliminate -each error one by one as described as following. - -Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using -an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` -flag to enable C++ toolchain resolution. - -It also assumes `clang version 9.0.1`, although the details should only change -slightly between different versions of clang. - -1. Add `toolchain/BUILD` with - - ```python - filegroup(name = "empty") - - cc_toolchain( - name = "linux_x86_64_toolchain", - toolchain_identifier = "linux_x86_64-toolchain", - toolchain_config = ":linux_x86_64_toolchain_config", - all_files = ":empty", - compiler_files = ":empty", - dwp_files = ":empty", - linker_files = ":empty", - objcopy_files = ":empty", - strip_files = ":empty", - supports_param_files = 0, - ) - - toolchain( - name = "cc_toolchain_for_linux_x86_64", - toolchain = ":linux_x86_64_toolchain", - toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", - exec_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - target_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - ) - ``` - - Then add appropriate dependencies and register the toolchain with - `MODULE.bazel` with - - ```python - bazel_dep(name = "platforms", version = "0.0.10") - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64" - ) - ``` - - This step defines a `cc_toolchain` and binds it to a `toolchain` target for - the host configuration. - -2. Run the build again. Because the `toolchain` package doesn't yet define the - `linux_x86_64_toolchain_config` target, Bazel throws the following error: - - ```bash - ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. - ``` - -3. In the `toolchain/BUILD` file, define an empty filegroup as follows: - - ```python - package(default_visibility = ["//visibility:public"]) - - filegroup(name = "linux_x86_64_toolchain_config") - ``` - -4. Run the build again. Bazel throws the following error: - - ```bash - '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. - ``` - - `CcToolchainConfigInfo` is a provider that you use to configure your C++ - toolchains. To fix this error, create a Starlark rule that provides - `CcToolchainConfigInfo` to Bazel by making a - `toolchain/cc_toolchain_config.bzl` file with the following content: - - ```python - def _impl(ctx): - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "k8-toolchain", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - `cc_common.create_cc_toolchain_config_info()` creates the needed provider - `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load - statement to `toolchain/BUILD` right below the package statement: - - ```python - load(":cc_toolchain_config.bzl", "cc_toolchain_config") - ``` - - And replace the "linux_x86_64_toolchain_config" filegroup with a declaration - of a `cc_toolchain_config` rule: - - ```python - cc_toolchain_config(name = "linux_x86_64_toolchain_config") - ``` - -5. Run the build again. Bazel throws the following error: - - ```bash - .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) - src/main/tools/linux-sandbox-pid1.cc:421: - "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory - Target //:hello-world failed to build` - ``` - - At this point, Bazel has enough information to attempt building the code but - it still does not know what tools to use to complete the required build - actions. You will modify the Starlark rule implementation to tell Bazel what - tools to use. For that, you need the `tool_path()` constructor from - [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): - - ```python - # toolchain/cc_toolchain_config.bzl: - # NEW - load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") - - def _impl(ctx): - tool_paths = [ # NEW - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/usr/bin/ar", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, # NEW - ) - ``` - - Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for - your system. Note that the compiler is referenced by the name "gcc" for - historic reasons. - -6. Run the build again. Bazel throws the following error: - - ```bash - ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': - the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): - '/usr/include/c++/13/ctime' - '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' - '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' - ... - ``` - - Bazel needs to know where to search for included headers. There are multiple - ways to solve this, such as using the `includes` attribute of `cc_binary`, - but here this is solved at the toolchain level with the - [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) - parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you - are using a different version of `clang`, the include path will be - different. These paths may also be different depending on the distribution. - - Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like - this: - - ```python - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - cxx_builtin_include_directories = [ # NEW - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - ``` - -7. Run the build command again, you will see an error like: - - ```bash - /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': - hello-world.cc:(.text+0x68): undefined reference to `std::cout' - ``` - - The reason for this is because the linker is missing the C++ standard - library and it can't find its symbols. There are many ways to solve this, - such as using the `linkopts` attribute of `cc_binary`. Here it is solved by - making sure that any target using the toolchain doesn't have to specify this - flag. - - Copy the following code to `toolchain/cc_toolchain_config.bzl`: - - ```python - # NEW - load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") - # NEW - load( - "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", - "feature", # NEW - "flag_group", # NEW - "flag_set", # NEW - "tool_path", - ) - - all_link_actions = [ # NEW - ACTION_NAMES.cpp_link_executable, - ACTION_NAMES.cpp_link_dynamic_library, - ACTION_NAMES.cpp_link_nodeps_dynamic_library, - ] - - def _impl(ctx): - tool_paths = [ - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/bin/false", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - features = [ # NEW - feature( - name = "default_linker_flags", - enabled = True, - flag_sets = [ - flag_set( - actions = all_link_actions, - flag_groups = ([ - flag_group( - flags = [ - "-lstdc++", - ], - ), - ]), - ), - ], - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - features = features, # NEW - cxx_builtin_include_directories = [ - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - Note that this code uses the GNU C++ library libstdc++. If you want to use - the LLVM C++ library, use "-lc++" instead of "-lstdc++". - -8. Running `bazel build //main:hello-world`, it should finally build the binary - successfully for host. - -9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and - `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in - target names. - - In `MODULE.bazel`, register the toolchain for android - - ```python - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64", - "//toolchain:cc_toolchain_for_android_x86_64" - ) - ``` - -10. Run `bazel build //main:hello-world - --android_platforms=//toolchain:android_x86_64` to build the binary for - Android. - -In practice, Linux and Android should have different C++ toolchain configs. You -can either modify the existing `cc_toolchain_config` for the differences or -create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate -platforms. - -## Review your work - -In this tutorial you learned how to configure a basic C++ toolchain, but -toolchains are more powerful than this example. +Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` flag to enable C++ toolchain resolution. + +It also assumes `clang version 9.0.1`, although the details should only change slightly between different versions of clang. + +1. Add `toolchain/BUILD` with + + ```python + filegroup(name = "empty") + + cc_toolchain( + name = "linux_x86_64_toolchain", + toolchain_identifier = "linux_x86_64-toolchain", + toolchain_config = ":linux_x86_64_toolchain_config", + all_files = ":empty", + compiler_files = ":empty", + dwp_files = ":empty", + linker_files = ":empty", + objcopy_files = ":empty", + strip_files = ":empty", + supports_param_files = 0, + ) + + toolchain( + name = "cc_toolchain_for_linux_x86_64", + toolchain = ":linux_x86_64_toolchain", + toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", + exec_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + ) + ``` + + Then add appropriate dependencies and register the toolchain with `MODULE.bazel` with + + ```python + bazel_dep(name = "platforms", version = "0.0.10") + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64" + ) + ``` + + This step defines a `cc_toolchain` and binds it to a `toolchain` target for the host configuration. + +2. Run the build again. Because the `toolchain` package doesn't yet define the `linux_x86_64_toolchain_config` target, Bazel throws the following error: + + ```bash + ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. + ``` + +3. In the `toolchain/BUILD` file, define an empty filegroup as follows: + + ```python + package(default_visibility = ["//visibility:public"]) + + filegroup(name = "linux_x86_64_toolchain_config") + ``` + +4. Run the build again. Bazel throws the following error: + + ```bash + '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. + ``` + + `CcToolchainConfigInfo` is a provider that you use to configure your C++ toolchains. To fix this error, create a Starlark rule that provides `CcToolchainConfigInfo` to Bazel by making a `toolchain/cc_toolchain_config.bzl` file with the following content: + + ```python + def _impl(ctx): + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "k8-toolchain", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + `cc_common.create_cc_toolchain_config_info()` creates the needed provider `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load statement to `toolchain/BUILD` right below the package statement: + + ```python + load(":cc_toolchain_config.bzl", "cc_toolchain_config") + ``` + + And replace the "linux\_x86\_64\_toolchain\_config" filegroup with a declaration of a `cc_toolchain_config` rule: + + ```python + cc_toolchain_config(name = "linux_x86_64_toolchain_config") + ``` + +5. Run the build again. Bazel throws the following error: + + ```bash + .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) + src/main/tools/linux-sandbox-pid1.cc:421: + "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory + Target //:hello-world failed to build` + ``` + + At this point, Bazel has enough information to attempt building the code but it still does not know what tools to use to complete the required build actions. You will modify the Starlark rule implementation to tell Bazel what tools to use. For that, you need the `tool_path()` constructor from [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): + + ```python + # toolchain/cc_toolchain_config.bzl: + # NEW + load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") + + def _impl(ctx): + tool_paths = [ # NEW + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/usr/bin/ar", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, # NEW + ) + ``` + + Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for your system. Note that the compiler is referenced by the name "gcc" for historic reasons. + +6. Run the build again. Bazel throws the following error: + + ```bash + ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': + the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): + '/usr/include/c++/13/ctime' + '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' + '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' + ... + ``` + + Bazel needs to know where to search for included headers. There are multiple ways to solve this, such as using the `includes` attribute of `cc_binary`, but here this is solved at the toolchain level with the [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you are using a different version of `clang`, the include path will be different. These paths may also be different depending on the distribution. + + Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like this: + + ```python + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + cxx_builtin_include_directories = [ # NEW + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + ``` + +7. Run the build command again, you will see an error like: + + ```bash + /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': + hello-world.cc:(.text+0x68): undefined reference to `std::cout' + ``` + + The reason for this is because the linker is missing the C++ standard library and it can't find its symbols. There are many ways to solve this, such as using the `linkopts` attribute of `cc_binary`. Here it is solved by making sure that any target using the toolchain doesn't have to specify this flag. + + Copy the following code to `toolchain/cc_toolchain_config.bzl`: + + ```python + # NEW + load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") + # NEW + load( + "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", + "feature", # NEW + "flag_group", # NEW + "flag_set", # NEW + "tool_path", + ) + + all_link_actions = [ # NEW + ACTION_NAMES.cpp_link_executable, + ACTION_NAMES.cpp_link_dynamic_library, + ACTION_NAMES.cpp_link_nodeps_dynamic_library, + ] + + def _impl(ctx): + tool_paths = [ + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/bin/false", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + features = [ # NEW + feature( + name = "default_linker_flags", + enabled = True, + flag_sets = [ + flag_set( + actions = all_link_actions, + flag_groups = ([ + flag_group( + flags = [ + "-lstdc++", + ], + ), + ]), + ), + ], + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + features = features, # NEW + cxx_builtin_include_directories = [ + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + Note that this code uses the GNU C++ library libstdc++. If you want to use the LLVM C++ library, use "-lc++" instead of "-lstdc++". + +8. Running `bazel build //main:hello-world`, it should finally build the binary successfully for host. + +9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in target names. + + In `MODULE.bazel`, register the toolchain for android + + ```python + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64", + "//toolchain:cc_toolchain_for_android_x86_64" + ) + ``` + +10. Run `bazel build //main:hello-world --android_platforms=//toolchain:android_x86_64` to build the binary for Android. + +In practice, Linux and Android should have different C++ toolchain configs. You can either modify the existing `cc_toolchain_config` for the differences or create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate platforms. + +## Review your work + +In this tutorial you learned how to configure a basic C++ toolchain, but toolchains are more powerful than this example. The key takeaways are: -- You need to specify a matching `platforms` flag in the command line for - Bazel to resolve to the toolchain for the same constraint values on the - platform. The documentation holds more [information about language specific - configuration flags](/concepts/platforms). -- You have to let the toolchain know where the tools live. In this tutorial - there is a simplified version where you access the tools from the system. If - you are interested in a more self-contained approach, you can read about - [external dependencies](/external/overview). Your tools could come from a - different module and you would have to make their files available to the - `cc_toolchain` with target dependencies on attributes, such as - `compiler_files`. The `tool_paths` would need to be changed as well. -- You can create features to customize which flags should be passed to - different actions, be it linking or any other type of action. - -## Further reading - -For more details, see [C++ toolchain -configuration](/docs/cc-toolchain-config-reference) +- You need to specify a matching `platforms` flag in the command line for Bazel to resolve to the toolchain for the same constraint values on the platform. The documentation holds more [information about language specific configuration flags](/concepts/platforms). +- You have to let the toolchain know where the tools live. In this tutorial there is a simplified version where you access the tools from the system. If you are interested in a more self-contained approach, you can read about [external dependencies](/external/overview). Your tools could come from a different module and you would have to make their files available to the `cc_toolchain` with target dependencies on attributes, such as `compiler_files`. The `tool_paths` would need to be changed as well. +- You can create features to customize which flags should be passed to different actions, be it linking or any other type of action. + +## Further reading + +For more details, see [C++ toolchain configuration](/docs/cc-toolchain-config-reference) diff --git a/tutorials/cpp-dependency.mdx b/tutorials/cpp-dependency.mdx index 194cc73c..24872bb8 100644 --- a/tutorials/cpp-dependency.mdx +++ b/tutorials/cpp-dependency.mdx @@ -2,49 +2,36 @@ title: 'Review the dependency graph' --- +A successful build has all of its dependencies explicitly stated in the `BUILD` file. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. - -A successful build has all of its dependencies explicitly stated in the `BUILD` -file. Bazel uses those statements to create the project's dependency graph, -which enables accurate incremental builds. - -To visualize the sample project's dependencies, you can generate a text -representation of the dependency graph by running this command at the -workspace root: +To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: ``` bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph ``` -The above command tells Bazel to look for all dependencies for the target -`//main:hello-world` (excluding host and implicit dependencies) and format the -output as a graph. +The above command tells Bazel to look for all dependencies for the target `//main:hello-world` (excluding host and implicit dependencies) and format the output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -On Ubuntu, you can view the graph locally by installing GraphViz and the xdot -Dot Viewer: +On Ubuntu, you can view the graph locally by installing GraphViz and the xdot Dot Viewer: ``` sudo apt update && sudo apt install graphviz xdot ``` -Then you can generate and view the graph by piping the text output above -straight to xdot: +Then you can generate and view the graph by piping the text output above straight to xdot: ``` -xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ +xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph) ``` -As you can see, the first stage of the sample project has a single target -that builds a single source file with no additional dependencies: +As you can see, the first stage of the sample project has a single target that builds a single source file with no additional dependencies: ![Dependency graph for 'hello-world'](/docs/images/cpp-tutorial-stage1.png "Dependency graph") -**Figure 1.** Dependency graph for `hello-world` displays a single target with a single -source file. +**Figure 1.** Dependency graph for `hello-world` displays a single target with a single source file. -After you set up your workspace, build your project, and examine its -dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. diff --git a/tutorials/cpp-labels.mdx b/tutorials/cpp-labels.mdx index 78d0dbce..17e062aa 100644 --- a/tutorials/cpp-labels.mdx +++ b/tutorials/cpp-labels.mdx @@ -2,26 +2,12 @@ title: 'Use labels to reference targets' --- - - -In `BUILD` files and at the command line, Bazel uses *labels* to reference -targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax -is: +In `BUILD` files and at the command line, Bazel uses *labels* to reference targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax is: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path from the -workspace root (the directory containing the `MODULE.bazel` file) to the directory -containing the `BUILD` file, and `target-name` is what you named the target -in the `BUILD` file (the `name` attribute). If the target is a file target, -then `path/to/package` is the path to the root of the package, and -`target-name` is the name of the target file, including its full -path relative to the root of the package (the directory containing the -package's `BUILD` file). +If the target is a rule target, then `path/to/package` is the path from the workspace root (the directory containing the `MODULE.bazel` file) to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path relative to the root of the package (the directory containing the package's `BUILD` file). -When referencing targets at the repository root, the package path is empty, -just use `//:target-name`. When referencing targets within the same `BUILD` -file, you can even skip the `//` workspace root identifier and just use -`:target-name`. +When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. diff --git a/tutorials/cpp-use-cases.mdx b/tutorials/cpp-use-cases.mdx index f25f80b3..f13e6be0 100644 --- a/tutorials/cpp-use-cases.mdx +++ b/tutorials/cpp-use-cases.mdx @@ -2,21 +2,13 @@ title: 'Common C++ Build Use Cases' --- +Here you will find some of the most common use cases for building C++ projects with Bazel. If you have not done so already, get started with building C++ projects with Bazel by completing the tutorial [Introduction to Bazel: Build a C++ Project](/start/cpp). - -Here you will find some of the most common use cases for building C++ projects -with Bazel. If you have not done so already, get started with building C++ -projects with Bazel by completing the tutorial -[Introduction to Bazel: Build a C++ Project](/start/cpp). - -For information on cc_library and hdrs header files, see -cc_library. +For information on cc\_library and hdrs header files, see [cc\_library](/reference/be/c-cpp#cc_library). ## Including multiple files in a target -You can include multiple files in a single target with -glob. -For example: +You can include multiple files in a single target with [glob](/reference/be/functions#glob). For example: ```python cc_library( @@ -26,19 +18,11 @@ cc_library( ) ``` -With this target, Bazel will build all the `.cc` and `.h` files it finds in the -same directory as the `BUILD` file that contains this target (excluding -subdirectories). +With this target, Bazel will build all the `.cc` and `.h` files it finds in the same directory as the `BUILD` file that contains this target (excluding subdirectories). ## Using transitive includes -If a file includes a header, then any rule with that file as a source (that is, -having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should -depend on the included header's library rule. Conversely, only direct -dependencies need to be specified as dependencies. For example, suppose -`sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` -doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` -file would look like this: +If a file includes a header, then any rule with that file as a source (that is, having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should depend on the included header's library rule. Conversely, only direct dependencies need to be specified as dependencies. For example, suppose `sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` file would look like this: ```python cc_library( @@ -62,15 +46,11 @@ cc_library( ) ``` -Here, the `sandwich` library depends on the `bread` library, which depends -on the `flour` library. +Here, the `sandwich` library depends on the `bread` library, which depends on the `flour` library. ## Adding include paths -Sometimes you cannot (or do not want to) root include paths at the workspace -root. Existing libraries might already have an include directory that doesn't -match its path in your workspace. For example, suppose you have the following -directory structure: +Sometimes you cannot (or do not want to) root include paths at the workspace root. Existing libraries might already have an include directory that doesn't match its path in your workspace. For example, suppose you have the following directory structure: ``` └── my-project @@ -83,11 +63,7 @@ directory structure: └── MODULE.bazel ``` -Bazel will expect `some_lib.h` to be included as -`legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes -`"some_lib.h"`. To make that include path valid, -`legacy/some_lib/BUILD` will need to specify that the `some_lib/include` -directory is an include directory: +Bazel will expect `some_lib.h` to be included as `legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes `"some_lib.h"`. To make that include path valid, `legacy/some_lib/BUILD` will need to specify that the `some_lib/include` directory is an include directory: ```python cc_library( @@ -98,15 +74,11 @@ cc_library( ) ``` -This is especially useful for external dependencies, as their header files -must otherwise be included with a `/` prefix. +This is especially useful for external dependencies, as their header files must otherwise be included with a `/` prefix. ## Include external libraries -Suppose you are using [Google Test](https://github.com/google/googletest) -. -You can add a dependency on it in the `MODULE.bazel` file to -download Google Test and make it available in your repository: +Suppose you are using [Google Test](https://github.com/google/googletest). You can add a dependency on it in the `MODULE.bazel` file to download Google Test and make it available in your repository: ```python bazel_dep(name = "googletest", version = "1.15.2") @@ -142,8 +114,7 @@ cc_test( ) ``` -To make `hello-greet` visible to `hello-test`, you must add -`"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. +To make `hello-greet` visible to `hello-test`, you must add `"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. Now you can use `bazel test` to run the test. @@ -163,11 +134,9 @@ INFO: Elapsed time: 4.497s, Critical Path: 2.53s Executed 1 out of 1 tests: 1 test passes. ``` - ## Adding dependencies on precompiled libraries -If you want to use a library of which you only have a compiled version (for -example, headers and a `.so` file) wrap it in a `cc_library` rule: +If you want to use a library of which you only have a compiled version (for example, headers and a `.so` file) wrap it in a `cc_library` rule: ```python cc_library( diff --git a/versions/index.mdx b/versions/index.mdx index 4290e57f..64608bf9 100644 --- a/versions/index.mdx +++ b/versions/index.mdx @@ -2,14 +2,8 @@ title: 'Documentation Versions' --- +The default documentation on this website represents the latest version at HEAD. Each major and minor supported release will have a snapshot of the narrative and reference documentation that follows the lifecycle of Bazel's version support. +To see documentation for stable Bazel versions, use the "Versioned docs" drop-down. -The default documentation on this website represents the latest version at HEAD. -Each major and minor supported release will have a snapshot of the narrative and -reference documentation that follows the lifecycle of Bazel's version support. - -To see documentation for stable Bazel versions, use the "Versioned docs" -drop-down. - -To see documentation for older Bazel versions prior to Feb 2022, go to -[docs.bazel.build](https://docs.bazel.build/). +To see documentation for older Bazel versions prior to Feb 2022, go to [docs.bazel.build](https://docs.bazel.build/). From 0c32ddfc16f1414ada4fc5f79f4f624a14ba0c8a Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 17:55:06 -0800 Subject: [PATCH 02/10] update workflows + testing Co-authored-by: promptless[bot] --- .github/workflows/pull-from-bazel-build.yml | 11 +- tools/mdx-transform/package-lock.json | 453 +------------------- tools/mdx-transform/package.json | 11 +- tools/mdx-transform/transform.mjs | 68 +-- 4 files changed, 60 insertions(+), 483 deletions(-) diff --git a/.github/workflows/pull-from-bazel-build.yml b/.github/workflows/pull-from-bazel-build.yml index 53f079b2..b2d12d48 100644 --- a/.github/workflows/pull-from-bazel-build.yml +++ b/.github/workflows/pull-from-bazel-build.yml @@ -48,6 +48,15 @@ jobs: with: bazelisk-cache: true repository-cache: true + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Install mdx transformer dependencies + working-directory: tools/mdx-transform + run: npm ci - name: Build reference documentation working-directory: upstream @@ -141,4 +150,4 @@ jobs: echo "Changes committed and pushed successfully" else echo "No changes detected, skipping commit" - fi \ No newline at end of file + fi diff --git a/tools/mdx-transform/package-lock.json b/tools/mdx-transform/package-lock.json index f9b006a3..f8e056b9 100644 --- a/tools/mdx-transform/package-lock.json +++ b/tools/mdx-transform/package-lock.json @@ -12,18 +12,15 @@ "hast-util-to-mdast": "^10.1.2", "hast-util-to-text": "^4.0.2", "mdast-util-to-hast": "^13.2.0", + "mdast-util-to-string": "^4.0.0", "rehype-raw": "^7.0.0", - "rehype-stringify": "^10.0.1", "remark-frontmatter": "^5.0.0", "remark-gfm": "^4.0.1", - "remark-mdx": "^3.1.1", "remark-parse": "^11.0.0", - "remark-rehype": "^11.1.2", "remark-stringify": "^11.0.0", "unified": "^11.0.5", - "unist-util-is": "^6.0.1", - "unist-util-remove": "^4.0.0", - "unist-util-visit": "^5.0.0" + "unist-util-visit": "^5.0.0", + "vfile": "^6.0.1" }, "bin": { "mdx-transform": "transform.mjs" @@ -38,21 +35,6 @@ "@types/ms": "*" } }, - "node_modules/@types/estree": { - "version": "1.0.8", - "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", - "integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==", - "license": "MIT" - }, - "node_modules/@types/estree-jsx": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/@types/estree-jsx/-/estree-jsx-1.0.5.tgz", - "integrity": "sha512-52CcUVNFyfb1A2ALocQw/Dd1BQFNmSdkuC3BkZ6iqhdMfQz7JWOFRuJFloOzjk+6WijU56m9oKXFAXc7o3Towg==", - "license": "MIT", - "dependencies": { - "@types/estree": "*" - } - }, "node_modules/@types/hast": { "version": "3.0.4", "resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.4.tgz", @@ -89,27 +71,6 @@ "integrity": "sha512-WmoN8qaIAo7WTYWbAZuG8PYEhn5fkz7dZrqTBZ7dtt//lL2Gwms1IcnQ5yHqjDfX8Ft5j4YzDM23f87zBfDe9g==", "license": "ISC" }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/acorn-jsx": { - "version": "5.3.2", - "resolved": "https://registry.npmjs.org/acorn-jsx/-/acorn-jsx-5.3.2.tgz", - "integrity": "sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ==", - "license": "MIT", - "peerDependencies": { - "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" - } - }, "node_modules/bail": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/bail/-/bail-2.0.2.tgz", @@ -160,16 +121,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/character-reference-invalid": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/character-reference-invalid/-/character-reference-invalid-2.0.1.tgz", - "integrity": "sha512-iBZ4F4wRbyORVsu0jPV7gXkOsGYjGHPmAyv+HiHG8gi5PtC9KI2j1+v8/tlibRvjoWX027ypmG/n0HtO5t7unw==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, "node_modules/comma-separated-tokens": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/comma-separated-tokens/-/comma-separated-tokens-2.0.3.tgz", @@ -256,30 +207,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/estree-util-is-identifier-name": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/estree-util-is-identifier-name/-/estree-util-is-identifier-name-3.0.0.tgz", - "integrity": "sha512-hFtqIDZTIUZ9BXLb8y4pYGyk6+wekIivNVTcmvk8NoOh+VeRn5y6cEHzbURrWbfp1fIqdVipilzj+lfaadNZmg==", - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/estree-util-visit": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/estree-util-visit/-/estree-util-visit-2.0.0.tgz", - "integrity": "sha512-m5KgiH85xAhhW8Wta0vShLcUvOsh3LLPI2YVwcbio1l7E09NTLL1EyMZFM1OyWowoH0skScNbhOPl4kcBgzTww==", - "license": "MIT", - "dependencies": { - "@types/estree-jsx": "^1.0.0", - "@types/unist": "^3.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/extend": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", @@ -586,50 +513,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/is-alphabetical": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/is-alphabetical/-/is-alphabetical-2.0.1.tgz", - "integrity": "sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/is-alphanumerical": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/is-alphanumerical/-/is-alphanumerical-2.0.1.tgz", - "integrity": "sha512-hmbYhX/9MUMF5uh7tOXyK/n0ZvWpad5caBA17GsC6vyuCqaWliRG5K1qS9inmUhEMaOBIW7/whAnSwveW/LtZw==", - "license": "MIT", - "dependencies": { - "is-alphabetical": "^2.0.0", - "is-decimal": "^2.0.0" - }, - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/is-decimal": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/is-decimal/-/is-decimal-2.0.1.tgz", - "integrity": "sha512-AAB9hiomQs5DXWcRB1rqsxGUstbRroFOPPVAomNk/3XHR5JyEZChOyTWe2oayKnsSsr/kcGqF+z6yuH6HHpN0A==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/is-hexadecimal": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/is-hexadecimal/-/is-hexadecimal-2.0.1.tgz", - "integrity": "sha512-DgZQp241c8oO6cA1SbTEWiXeoxV42vlcJxgH+B3hi1AiqqKruZR3ZGF8In3fj4+/y/7rHvlOZLZtgJ/4ttYGZg==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, "node_modules/is-plain-obj": { "version": "4.1.0", "resolved": "https://registry.npmjs.org/is-plain-obj/-/is-plain-obj-4.1.0.tgz", @@ -821,83 +704,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/mdast-util-mdx": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/mdast-util-mdx/-/mdast-util-mdx-3.0.0.tgz", - "integrity": "sha512-JfbYLAW7XnYTTbUsmpu0kdBUVe+yKVJZBItEjwyYJiDJuZ9w4eeaqks4HQO+R7objWgS2ymV60GYpI14Ug554w==", - "license": "MIT", - "dependencies": { - "mdast-util-from-markdown": "^2.0.0", - "mdast-util-mdx-expression": "^2.0.0", - "mdast-util-mdx-jsx": "^3.0.0", - "mdast-util-mdxjs-esm": "^2.0.0", - "mdast-util-to-markdown": "^2.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/mdast-util-mdx-expression": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/mdast-util-mdx-expression/-/mdast-util-mdx-expression-2.0.1.tgz", - "integrity": "sha512-J6f+9hUp+ldTZqKRSg7Vw5V6MqjATc+3E4gf3CFNcuZNWD8XdyI6zQ8GqH7f8169MM6P7hMBRDVGnn7oHB9kXQ==", - "license": "MIT", - "dependencies": { - "@types/estree-jsx": "^1.0.0", - "@types/hast": "^3.0.0", - "@types/mdast": "^4.0.0", - "devlop": "^1.0.0", - "mdast-util-from-markdown": "^2.0.0", - "mdast-util-to-markdown": "^2.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/mdast-util-mdx-jsx": { - "version": "3.2.0", - "resolved": "https://registry.npmjs.org/mdast-util-mdx-jsx/-/mdast-util-mdx-jsx-3.2.0.tgz", - "integrity": "sha512-lj/z8v0r6ZtsN/cGNNtemmmfoLAFZnjMbNyLzBafjzikOM+glrjNHPlf6lQDOTccj9n5b0PPihEBbhneMyGs1Q==", - "license": "MIT", - "dependencies": { - "@types/estree-jsx": "^1.0.0", - "@types/hast": "^3.0.0", - "@types/mdast": "^4.0.0", - "@types/unist": "^3.0.0", - "ccount": "^2.0.0", - "devlop": "^1.1.0", - "mdast-util-from-markdown": "^2.0.0", - "mdast-util-to-markdown": "^2.0.0", - "parse-entities": "^4.0.0", - "stringify-entities": "^4.0.0", - "unist-util-stringify-position": "^4.0.0", - "vfile-message": "^4.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/mdast-util-mdxjs-esm": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/mdast-util-mdxjs-esm/-/mdast-util-mdxjs-esm-2.0.1.tgz", - "integrity": "sha512-EcmOpxsZ96CvlP03NghtH1EsLtr0n9Tm4lPUJUBccV9RwUOneqSycg19n5HGzCf+10LozMRSObtVr3ee1WoHtg==", - "license": "MIT", - "dependencies": { - "@types/estree-jsx": "^1.0.0", - "@types/hast": "^3.0.0", - "@types/mdast": "^4.0.0", - "devlop": "^1.0.0", - "mdast-util-from-markdown": "^2.0.0", - "mdast-util-to-markdown": "^2.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/mdast-util-phrasing": { "version": "4.1.0", "resolved": "https://registry.npmjs.org/mdast-util-phrasing/-/mdast-util-phrasing-4.1.0.tgz", @@ -1173,108 +979,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/micromark-extension-mdx-expression": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/micromark-extension-mdx-expression/-/micromark-extension-mdx-expression-3.0.1.tgz", - "integrity": "sha512-dD/ADLJ1AeMvSAKBwO22zG22N4ybhe7kFIZ3LsDI0GlsNr2A3KYxb0LdC1u5rj4Nw+CHKY0RVdnHX8vj8ejm4Q==", - "funding": [ - { - "type": "GitHub Sponsors", - "url": "https://github.com/sponsors/unifiedjs" - }, - { - "type": "OpenCollective", - "url": "https://opencollective.com/unified" - } - ], - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0", - "devlop": "^1.0.0", - "micromark-factory-mdx-expression": "^2.0.0", - "micromark-factory-space": "^2.0.0", - "micromark-util-character": "^2.0.0", - "micromark-util-events-to-acorn": "^2.0.0", - "micromark-util-symbol": "^2.0.0", - "micromark-util-types": "^2.0.0" - } - }, - "node_modules/micromark-extension-mdx-jsx": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/micromark-extension-mdx-jsx/-/micromark-extension-mdx-jsx-3.0.2.tgz", - "integrity": "sha512-e5+q1DjMh62LZAJOnDraSSbDMvGJ8x3cbjygy2qFEi7HCeUT4BDKCvMozPozcD6WmOt6sVvYDNBKhFSz3kjOVQ==", - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0", - "devlop": "^1.0.0", - "estree-util-is-identifier-name": "^3.0.0", - "micromark-factory-mdx-expression": "^2.0.0", - "micromark-factory-space": "^2.0.0", - "micromark-util-character": "^2.0.0", - "micromark-util-events-to-acorn": "^2.0.0", - "micromark-util-symbol": "^2.0.0", - "micromark-util-types": "^2.0.0", - "vfile-message": "^4.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/micromark-extension-mdx-md": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/micromark-extension-mdx-md/-/micromark-extension-mdx-md-2.0.0.tgz", - "integrity": "sha512-EpAiszsB3blw4Rpba7xTOUptcFeBFi+6PY8VnJ2hhimH+vCQDirWgsMpz7w1XcZE7LVrSAUGb9VJpG9ghlYvYQ==", - "license": "MIT", - "dependencies": { - "micromark-util-types": "^2.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/micromark-extension-mdxjs": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/micromark-extension-mdxjs/-/micromark-extension-mdxjs-3.0.0.tgz", - "integrity": "sha512-A873fJfhnJ2siZyUrJ31l34Uqwy4xIFmvPY1oj+Ean5PHcPBYzEsvqvWGaWcfEIr11O5Dlw3p2y0tZWpKHDejQ==", - "license": "MIT", - "dependencies": { - "acorn": "^8.0.0", - "acorn-jsx": "^5.0.0", - "micromark-extension-mdx-expression": "^3.0.0", - "micromark-extension-mdx-jsx": "^3.0.0", - "micromark-extension-mdx-md": "^2.0.0", - "micromark-extension-mdxjs-esm": "^3.0.0", - "micromark-util-combine-extensions": "^2.0.0", - "micromark-util-types": "^2.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/micromark-extension-mdxjs-esm": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/micromark-extension-mdxjs-esm/-/micromark-extension-mdxjs-esm-3.0.0.tgz", - "integrity": "sha512-DJFl4ZqkErRpq/dAPyeWp15tGrcrrJho1hKK5uBS70BCtfrIFg81sqcTVu3Ta+KD1Tk5vAtBNElWxtAa+m8K9A==", - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0", - "devlop": "^1.0.0", - "micromark-core-commonmark": "^2.0.0", - "micromark-util-character": "^2.0.0", - "micromark-util-events-to-acorn": "^2.0.0", - "micromark-util-symbol": "^2.0.0", - "micromark-util-types": "^2.0.0", - "unist-util-position-from-estree": "^2.0.0", - "vfile-message": "^4.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/micromark-factory-destination": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/micromark-factory-destination/-/micromark-factory-destination-2.0.1.tgz", @@ -1318,33 +1022,6 @@ "micromark-util-types": "^2.0.0" } }, - "node_modules/micromark-factory-mdx-expression": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/micromark-factory-mdx-expression/-/micromark-factory-mdx-expression-2.0.3.tgz", - "integrity": "sha512-kQnEtA3vzucU2BkrIa8/VaSAsP+EJ3CKOvhMuJgOEGg9KDC6OAY6nSnNDVRiVNRqj7Y4SlSzcStaH/5jge8JdQ==", - "funding": [ - { - "type": "GitHub Sponsors", - "url": "https://github.com/sponsors/unifiedjs" - }, - { - "type": "OpenCollective", - "url": "https://opencollective.com/unified" - } - ], - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0", - "devlop": "^1.0.0", - "micromark-factory-space": "^2.0.0", - "micromark-util-character": "^2.0.0", - "micromark-util-events-to-acorn": "^2.0.0", - "micromark-util-symbol": "^2.0.0", - "micromark-util-types": "^2.0.0", - "unist-util-position-from-estree": "^2.0.0", - "vfile-message": "^4.0.0" - } - }, "node_modules/micromark-factory-space": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/micromark-factory-space/-/micromark-factory-space-2.0.1.tgz", @@ -1546,31 +1223,6 @@ ], "license": "MIT" }, - "node_modules/micromark-util-events-to-acorn": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/micromark-util-events-to-acorn/-/micromark-util-events-to-acorn-2.0.3.tgz", - "integrity": "sha512-jmsiEIiZ1n7X1Rr5k8wVExBQCg5jy4UXVADItHmNk1zkwEVhBuIUKRu3fqv+hs4nxLISi2DQGlqIOGiFxgbfHg==", - "funding": [ - { - "type": "GitHub Sponsors", - "url": "https://github.com/sponsors/unifiedjs" - }, - { - "type": "OpenCollective", - "url": "https://opencollective.com/unified" - } - ], - "license": "MIT", - "dependencies": { - "@types/estree": "^1.0.0", - "@types/unist": "^3.0.0", - "devlop": "^1.0.0", - "estree-util-visit": "^2.0.0", - "micromark-util-symbol": "^2.0.0", - "micromark-util-types": "^2.0.0", - "vfile-message": "^4.0.0" - } - }, "node_modules/micromark-util-html-tag-name": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/micromark-util-html-tag-name/-/micromark-util-html-tag-name-2.0.1.tgz", @@ -1706,31 +1358,6 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, - "node_modules/parse-entities": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/parse-entities/-/parse-entities-4.0.2.tgz", - "integrity": "sha512-GG2AQYWoLgL877gQIKeRPGO1xF9+eG1ujIb5soS5gPvLQ1y2o8FL90w2QWNdf9I361Mpp7726c+lj3U0qK1uGw==", - "license": "MIT", - "dependencies": { - "@types/unist": "^2.0.0", - "character-entities-legacy": "^3.0.0", - "character-reference-invalid": "^2.0.0", - "decode-named-character-reference": "^1.0.0", - "is-alphanumerical": "^2.0.0", - "is-decimal": "^2.0.0", - "is-hexadecimal": "^2.0.0" - }, - "funding": { - "type": "github", - "url": "https://github.com/sponsors/wooorm" - } - }, - "node_modules/parse-entities/node_modules/@types/unist": { - "version": "2.0.11", - "resolved": "https://registry.npmjs.org/@types/unist/-/unist-2.0.11.tgz", - "integrity": "sha512-CmBKiL6NNo/OqgmMn95Fk9Whlp2mtvIv+KNpQKN2F4SjvrEesubTRWGYSg+BnWZOnlCaSTU1sMpsBOzgbYhnsA==", - "license": "MIT" - }, "node_modules/parse5": { "version": "7.3.0", "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz", @@ -1782,21 +1409,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/rehype-stringify": { - "version": "10.0.1", - "resolved": "https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-10.0.1.tgz", - "integrity": "sha512-k9ecfXHmIPuFVI61B9DeLPN0qFHfawM6RsuX48hoqlaKSF61RskNjSm1lI8PhBEM0MRdLxVVm4WmTqJQccH9mA==", - "license": "MIT", - "dependencies": { - "@types/hast": "^3.0.0", - "hast-util-to-html": "^9.0.0", - "unified": "^11.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/remark-frontmatter": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/remark-frontmatter/-/remark-frontmatter-5.0.0.tgz", @@ -1831,20 +1443,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/remark-mdx": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/remark-mdx/-/remark-mdx-3.1.1.tgz", - "integrity": "sha512-Pjj2IYlUY3+D8x00UJsIOg5BEvfMyeI+2uLPn9VO9Wg4MEtN/VTIq2NEJQfde9PnX15KgtHyl9S0BcTnWrIuWg==", - "license": "MIT", - "dependencies": { - "mdast-util-mdx": "^3.0.0", - "micromark-extension-mdxjs": "^3.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/remark-parse": { "version": "11.0.0", "resolved": "https://registry.npmjs.org/remark-parse/-/remark-parse-11.0.0.tgz", @@ -1861,23 +1459,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/remark-rehype": { - "version": "11.1.2", - "resolved": "https://registry.npmjs.org/remark-rehype/-/remark-rehype-11.1.2.tgz", - "integrity": "sha512-Dh7l57ianaEoIpzbp0PC9UKAdCSVklD8E5Rpw7ETfbTl3FqcOOgq5q2LVDhgGCkaBv7p24JXikPdvhhmHvKMsw==", - "license": "MIT", - "dependencies": { - "@types/hast": "^3.0.0", - "@types/mdast": "^4.0.0", - "mdast-util-to-hast": "^13.0.0", - "unified": "^11.0.0", - "vfile": "^6.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/remark-stringify": { "version": "11.0.0", "resolved": "https://registry.npmjs.org/remark-stringify/-/remark-stringify-11.0.0.tgz", @@ -2006,34 +1587,6 @@ "url": "https://opencollective.com/unified" } }, - "node_modules/unist-util-position-from-estree": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/unist-util-position-from-estree/-/unist-util-position-from-estree-2.0.0.tgz", - "integrity": "sha512-KaFVRjoqLyF6YXCbVLNad/eS4+OfPQQn2yOd7zF/h5T/CSL2v8NpN6a5TPvtbXthAGw5nG+PuTtq+DdIZr+cRQ==", - "license": "MIT", - "dependencies": { - "@types/unist": "^3.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, - "node_modules/unist-util-remove": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/unist-util-remove/-/unist-util-remove-4.0.0.tgz", - "integrity": "sha512-b4gokeGId57UVRX/eVKej5gXqGlc9+trkORhFJpu9raqZkZhU0zm8Doi05+HaiBsMEIJowL+2WtQ5ItjsngPXg==", - "license": "MIT", - "dependencies": { - "@types/unist": "^3.0.0", - "unist-util-is": "^6.0.0", - "unist-util-visit-parents": "^6.0.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/unified" - } - }, "node_modules/unist-util-stringify-position": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-4.0.0.tgz", diff --git a/tools/mdx-transform/package.json b/tools/mdx-transform/package.json index 9e4a0528..9744b817 100644 --- a/tools/mdx-transform/package.json +++ b/tools/mdx-transform/package.json @@ -9,7 +9,7 @@ "description": "", "main": "index.js", "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" + "test": "node --test" }, "keywords": [], "author": "", @@ -18,17 +18,14 @@ "hast-util-to-mdast": "^10.1.2", "hast-util-to-text": "^4.0.2", "mdast-util-to-hast": "^13.2.0", + "mdast-util-to-string": "^4.0.0", "rehype-raw": "^7.0.0", - "rehype-stringify": "^10.0.1", "remark-frontmatter": "^5.0.0", "remark-gfm": "^4.0.1", - "remark-mdx": "^3.1.1", "remark-parse": "^11.0.0", - "remark-rehype": "^11.1.2", "remark-stringify": "^11.0.0", "unified": "^11.0.5", - "unist-util-is": "^6.0.1", - "unist-util-remove": "^4.0.0", - "unist-util-visit": "^5.0.0" + "unist-util-visit": "^5.0.0", + "vfile": "^6.0.1" } } diff --git a/tools/mdx-transform/transform.mjs b/tools/mdx-transform/transform.mjs index afa0861c..d4074d14 100644 --- a/tools/mdx-transform/transform.mjs +++ b/tools/mdx-transform/transform.mjs @@ -19,11 +19,33 @@ import {toText} from 'hast-util-to-text'; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); +const cloneNode = (node) => { + if (typeof structuredClone === 'function') { + return structuredClone(node); + } + return JSON.parse(JSON.stringify(node)); +}; + const remarkBase = unified() .use(remarkParse) .use(remarkFrontmatter, ['yaml']) .use(remarkGfm); +const remarkStringifyOptions = { + bullet: '-', + fences: true, + entities: {useNamedReferences: true}, + listItemIndent: 'one', + allowDangerousHtml: true, +}; + +const stringifyMarkdown = (tree) => unified() + .use(remarkGfm) + .use(remarkStringify, remarkStringifyOptions) + .stringify(tree); + +const escapeAttribute = (value) => value.replace(/"/g, '"'); + function escapeYaml(str) { return str.replace(/'/g, "''"); } @@ -120,13 +142,10 @@ function sanitizeMdast(tree) { }); if (node.type === 'code' || node.type === 'inlineCode') { value = value.replace(//g, '>'); - } - if (node.type === 'text') { - value = value - .replace(//g, '>') - .replace(/\{/g, '{') - .replace(/\}/g, '}'); + } else if (node.type === 'text') { + value = value.replace(/#include <([^>]+)>/g, (_match, header) => `#include <${header}>`); + value = value.replace(/(^|[^\\])\{/g, (_match, lead) => `${lead}\\{`); + value = value.replace(/(^|[^\\])\}/g, (_match, lead) => `${lead}\\}`); } node.value = value; } @@ -255,10 +274,17 @@ export function rehypeCustom() { if (compare) { const type = compare === 'compare-better' ? 'success' : 'warning'; const title = toText(first).replace(/[–—-]\s*$/,'').trim() || (type === 'success' ? 'Yes' : 'No'); - const bodyNodes = node.children.slice(1); - const bodyText = toText({type: 'element', tagName: 'div', children: bodyNodes}).trim(); - const callout = `${bodyText ? '\n' + bodyText + '\n' : ''}`; - parent.children[index] = {type: 'raw', value: callout}; + const bodyNodes = node.children.slice(1).map(cloneNode); + if (bodyNodes.length) { + const firstBody = bodyNodes[0]; + if (firstBody.type === 'text') { + firstBody.value = firstBody.value.replace(/^[-\s:–—]+/, '').trimStart(); + } + } + const bodyTree = toMdast({type: 'element', tagName: 'div', children: bodyNodes}, {}); + const bodyMarkdown = stringifyMarkdown(bodyTree).trim(); + const callout = `${bodyMarkdown ? `\n${bodyMarkdown}\n` : ''}`; + parent.children.splice(index, 1, {type: 'raw', value: callout}); return [visit.SKIP, index]; } } @@ -273,9 +299,9 @@ export function rehypeCustom() { if (compare) { const icon = compare === 'compare-better' ? '✅' : '⚠️'; const label = toText(first).trim(); - const rest = node.children.slice(1); - const restText = toText({type: 'element', tagName: 'div', children: rest}).trim(); - node.children = [{type: 'text', value: `${icon} ${label}:${restText ? ' ' + restText : ''}`}]; + const rest = node.children.slice(1).map(cloneNode); + const prefix = rest.length ? `${icon} ${label}: ` : `${icon} ${label}`; + node.children = [{type: 'text', value: prefix}, ...rest]; } } } @@ -309,16 +335,7 @@ export function transformContent(content) { const mdastResult = toMdast(hast, {}); sanitizeMdast(mdastResult); - let output = unified() - .use(remarkGfm) - .use(remarkStringify, { - bullet: '-', - fences: true, - entities: {useNamedReferences: true}, - listItemIndent: 'one', - allowDangerousHtml: true, - }) - .stringify(mdastResult); + let output = stringifyMarkdown(mdastResult); output = output.replace(/\n{3,}/g, '\n\n'); if (file.data.frontmatter) { output = `---\n${file.data.frontmatter}\n---\n\n${output}`; @@ -333,6 +350,7 @@ export function transformContent(content) { }) .replace(/https\\:\/\//g, 'https://') .replace(/http\\:\/\//g, 'http://'); + output = output.replace(/\\([{}])/g, '\$1'); return output; } @@ -350,7 +368,7 @@ function runCli() { fs.writeFileSync(absoluteOutput, transformed, 'utf8'); } -if (import.meta.url === pathToFileURL(process.argv[1]).href) { +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { runCli(); } From 0242e826aab1cceee52e4800c8501aa682ddf2eb Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 18:02:12 -0800 Subject: [PATCH 03/10] add tests Co-authored-by: promptless[bot] --- tools/mdx-transform/test/transform.test.mjs | 31 +++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 tools/mdx-transform/test/transform.test.mjs diff --git a/tools/mdx-transform/test/transform.test.mjs b/tools/mdx-transform/test/transform.test.mjs new file mode 100644 index 00000000..4581f8bc --- /dev/null +++ b/tools/mdx-transform/test/transform.test.mjs @@ -0,0 +1,31 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; + +import {transformContent} from '../transform.mjs'; + +const stripFrontmatter = (content) => content.replace(/^---[\s\S]*?---\n\n/, ''); + +test('compare callout preserves inline markup', () => { + const input = `# Title\n\n

Better — Keep foo and link.

`; + const output = stripFrontmatter(transformContent(input)); + + assert.ok(output.includes('')); + assert.ok(output.includes('`foo`')); + assert.ok(output.includes('[link](https://example.com)')); + assert.ok(output.includes('')); +}); + +test('compare list items retain trailing content', () => { + const input = `# Title\n\n
  • Worse still works
`; + const output = stripFrontmatter(transformContent(input)); + + assert.ok(output.includes('- ⚠️ Worse: ')); + assert.ok(output.includes('*still works*')); +}); + +test('braces are readable and escaped for MDX', () => { + const input = '# Title\n\nExample {\'cpu\': \'ppc\'}'; + const output = stripFrontmatter(transformContent(input)); + + assert.ok(output.includes("Example \\{'cpu': 'ppc'\\}")); +}); From 83bce97dfe3d98d7549916d2a1af40ea5687d8a4 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 18:04:01 -0800 Subject: [PATCH 04/10] cleanup Co-authored-by: promptless[bot] --- about/faq.mdx | 74 +- about/intro.mdx | 95 +- about/roadmap.mdx | 88 +- about/vision.mdx | 110 +- about/why.mdx | 66 +- .../build-performance-breakdown.mdx | 248 ++- .../performance/build-performance-metrics.mdx | 76 +- advanced/performance/iteration-speed.mdx | 84 +- advanced/performance/json-trace-profile.mdx | 102 +- advanced/performance/memory.mdx | 60 +- basics/artifact-based-builds.mdx | 277 ++- basics/build-systems.mdx | 136 +- basics/dependencies.mdx | 306 ++- basics/distributed-builds.mdx | 128 +- basics/hermeticity.mdx | 117 +- basics/index.mdx | 54 +- basics/task-based-builds.mdx | 238 ++- brand/index.mdx | 84 +- build/share-variables.mdx | 31 +- build/style-guide.mdx | 279 ++- community/recommended-rules.mdx | 50 +- community/remote-execution-services.mdx | 31 +- community/sig.mdx | 146 +- community/users.mdx | 533 +++-- concepts/build-ref.mdx | 100 +- concepts/platforms.mdx | 283 ++- concepts/visibility.mdx | 331 ++- configure/attributes.mdx | 338 ++- configure/best-practices.mdx | 67 +- configure/coverage.mdx | 106 +- configure/windows.mdx | 240 ++- contribute/breaking-changes.mdx | 112 +- contribute/codebase.mdx | 1881 ++++++++++++----- contribute/design-documents.mdx | 238 ++- contribute/docs.mdx | 32 +- contribute/index.mdx | 76 +- contribute/maintainers-guide.mdx | 254 ++- contribute/naming.mdx | 62 +- contribute/patch-acceptance.mdx | 65 +- contribute/policy.mdx | 103 +- contribute/release-notes.mdx | 67 +- contribute/statemachine-guide.mdx | 568 +++-- contribute/windows-chocolatey-maintenance.mdx | 50 +- contribute/windows-scoop-maintenance.mdx | 28 +- docs/android-build-performance.mdx | 70 +- docs/android-instrumentation-test.mdx | 192 +- docs/android-ndk.mdx | 147 +- docs/bazel-and-android.mdx | 43 +- docs/bazel-and-apple.mdx | 87 +- docs/bazel-and-cpp.mdx | 125 +- docs/bazel-and-java.mdx | 219 +- docs/bazel-and-javascript.mdx | 30 +- docs/configurable-attributes.mdx | 337 ++- docs/mobile-install.mdx | 183 +- docs/sandboxing.mdx | 144 +- extending/aspects.mdx | 202 +- extending/auto-exec-groups.mdx | 97 +- extending/concepts.mdx | 109 +- extending/depsets.mdx | 130 +- extending/exec-groups.mdx | 106 +- extending/legacy-macros.mdx | 137 +- extending/macros.mdx | 270 ++- extending/platforms.mdx | 170 +- extending/rules.mdx | 879 ++++++-- extending/toolchains.mdx | 314 ++- external/extension.mdx | 202 +- external/faq.mdx | 327 ++- external/lockfile.mdx | 201 +- external/migration.mdx | 979 +++++---- external/module.mdx | 222 +- external/overview.mdx | 261 ++- external/repo.mdx | 141 +- external/vendor.mdx | 136 +- help.mdx | 52 +- install/bazelisk.mdx | 53 +- install/compile-source.mdx | 262 ++- install/completion.mdx | 177 +- install/docker-container.mdx | 28 +- install/ide.mdx | 81 +- install/index.mdx | 32 +- install/os-x.mdx | 82 +- install/suse.mdx | 16 +- install/ubuntu.mdx | 82 +- install/windows.mdx | 185 +- migrate/index.mdx | 6 +- migrate/maven.mdx | 250 ++- migrate/xcode.mdx | 229 +- query/aquery.mdx | 177 +- query/cquery.mdx | 350 ++- query/guide.mdx | 206 +- reference/glossary.mdx | 516 ++++- reference/skyframe.mdx | 211 +- release/backward-compatibility.mdx | 70 +- release/index.mdx | 254 ++- release/rolling.mdx | 11 +- release/rule-compatibility.mdx | 110 +- remote/bep-examples.mdx | 110 +- remote/bep-glossary.mdx | 148 +- remote/bep.mdx | 143 +- remote/cache-local.mdx | 68 +- remote/cache-remote.mdx | 152 +- remote/caching.mdx | 330 +-- remote/ci.mdx | 173 +- remote/creating.mdx | 195 +- remote/multiplex.mdx | 108 +- remote/output-directories.mdx | 133 +- remote/persistent.mdx | 219 +- remote/rbe.mdx | 28 +- remote/rules.mdx | 196 +- remote/sandbox.mdx | 210 +- remote/workspace.mdx | 122 +- rules/bzl-style.mdx | 221 +- rules/challenges.mdx | 230 +- rules/deploying.mdx | 143 +- rules/errors/read-only-variable.mdx | 9 +- rules/faq.mdx | 57 +- rules/index.mdx | 81 +- rules/language.mdx | 127 +- rules/legacy-macro-tutorial.mdx | 49 +- rules/macro-tutorial.mdx | 37 +- rules/performance.mdx | 147 +- rules/rules-tutorial.mdx | 126 +- rules/testing.mdx | 157 +- rules/verbs-tutorial.mdx | 118 +- run/bazelrc.mdx | 243 ++- run/client-server.mdx | 54 +- run/scripts.mdx | 136 +- start/android-app.mdx | 261 ++- start/cpp.mdx | 178 +- start/go.mdx | 224 +- start/ios-app.mdx | 3 +- start/java.mdx | 233 +- tutorials/ccp-toolchain-config.mdx | 832 ++++---- tutorials/cpp-dependency.mdx | 31 +- tutorials/cpp-labels.mdx | 20 +- tutorials/cpp-use-cases.mdx | 55 +- versions/index.mdx | 12 +- 137 files changed, 17465 insertions(+), 6968 deletions(-) diff --git a/about/faq.mdx b/about/faq.mdx index fc0c31d1..dd5be8a9 100644 --- a/about/faq.mdx +++ b/about/faq.mdx @@ -2,6 +2,8 @@ title: 'FAQ' --- + + If you have questions or need support, see [Getting Help](/help). ## What is Bazel? @@ -12,19 +14,19 @@ Bazel is a tool that automates software builds and tests. Supported build tasks Bazel was designed to fit the way software is developed at Google. It has the following features: -- Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. -- High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. -- Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. -- Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. -- Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about \~200ms. +* Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. +* High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. +* Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. +* Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. +* Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about ~200ms. ## Why doesn’t Google use...? -- Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. - - Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. -- Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. -- Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. -- Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. +* Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. + * Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. +* Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. +* Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. +* Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. ## Where did Bazel come from? @@ -46,10 +48,10 @@ Bazel runs build operations locally by default. However, Bazel can also connect For our server code base, we use the following development workflow: -- All our server code is in a single, gigantic version control system. -- Everybody builds their software with Bazel. -- Different teams own different parts of the source tree, and make their components available as `BUILD` targets. -- Branching is primarily used for managing releases, so everybody develops their software at the head revision. +* All our server code is in a single, gigantic version control system. +* Everybody builds their software with Bazel. +* Different teams own different parts of the source tree, and make their components available as `BUILD` targets. +* Branching is primarily used for managing releases, so everybody develops their software at the head revision. Bazel is a cornerstone of this philosophy: since Bazel requires all dependencies to be fully specified, we can predict which programs and tests are affected by a change, and vet them before submission. @@ -61,22 +63,24 @@ Building software should be fun and easy. Slow and unpredictable builds take the ## Why would I want to use Bazel? -- Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. -- Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. -- Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. +* Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. +* Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. +* Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. ## Can I see examples? -Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. +Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) +or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. + ## What is Bazel best at? Bazel shines at building and testing projects with the following properties: -- Projects with a large codebase -- Projects written in (multiple) compiled languages -- Projects that deploy on multiple platforms -- Projects that have extensive tests +* Projects with a large codebase +* Projects written in (multiple) compiled languages +* Projects that deploy on multiple platforms +* Projects that have extensive tests ## Where can I run Bazel? @@ -86,13 +90,11 @@ Porting to other UNIX platforms should be relatively easy, as long as a JDK is a ## What should I not use Bazel for? -- Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: - - - A compilation step that fetches data from the internet. - - A test step that connects to the QA instance of your site. - - A deployment step that changes your site’s cloud configuration. - -- If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. +* Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: + * A compilation step that fetches data from the internet. + * A test step that connects to the QA instance of your site. + * A deployment step that changes your site’s cloud configuration. +* If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. ## How stable is Bazel’s feature set? @@ -130,10 +132,10 @@ Yes, you can use our [Docker rules](https://github.com/bazelbuild/rules_docker) For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (for example, executing binaries through a shell script inside a rule), you will need to take some extra care: -- Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. -- Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. -- Avoid connecting to the network. Sandboxed execution can help here too. -- Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. +* Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. +* Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. +* Avoid connecting to the network. Sandboxed execution can help here too. +* Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. ## Do you have binary releases? @@ -177,8 +179,8 @@ We still have to refactor the interfaces between the public code in Bazel and ou Open sourcing Bazel is a work-in-progress. In particular, we’re still working on open sourcing: -- Many of our unit and integration tests (which should make contributing patches easier). -- Full IDE integration. +* Many of our unit and integration tests (which should make contributing patches easier). +* Full IDE integration. Beyond code, we’d like to eventually have all code reviews, bug tracking, and design decisions happen publicly, with the Bazel community involved. We are not there yet, so some changes will simply appear in the Bazel repository without clear explanation. Despite this lack of transparency, we want to support external developers and collaborate. Thus, we are opening up the code, even though some of the development is still happening internal to Google. Please let us know if anything seems unclear or unjustified as we transition to an open model. @@ -188,7 +190,7 @@ Yes, some of the code base either integrates with Google-specific technology or ## How do I contact the team? -We are reachable at [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com). +We are reachable at bazel-discuss@googlegroups.com. ## Where do I report bugs? diff --git a/about/intro.mdx b/about/intro.mdx index a70f715b..a531ac2a 100644 --- a/about/intro.mdx +++ b/about/intro.mdx @@ -2,63 +2,110 @@ title: 'Intro to Bazel' --- -Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users. + + +Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. +It uses a human-readable, high-level build language. Bazel supports projects in +multiple languages and builds outputs for multiple platforms. Bazel supports +large codebases across multiple repositories, and large numbers of users. ## Benefits Bazel offers the following advantages: -- **High-level build language.** Bazel uses an abstract, human-readable language to describe the build properties of your project at a high semantical level. Unlike other tools, Bazel operates on the *concepts* of libraries, binaries, scripts, and data sets, shielding you from the complexity of writing individual calls to tools such as compilers and linkers. +* **High-level build language.** Bazel uses an abstract, human-readable + language to describe the build properties of your project at a high + semantical level. Unlike other tools, Bazel operates on the *concepts* + of libraries, binaries, scripts, and data sets, shielding you from the + complexity of writing individual calls to tools such as compilers and + linkers. -- **Bazel is fast and reliable.** Bazel caches all previously done work and tracks changes to both file content and build commands. This way, Bazel knows when something needs to be rebuilt, and rebuilds only that. To further speed up your builds, you can set up your project to build in a highly parallel and incremental fashion. +* **Bazel is fast and reliable.** Bazel caches all previously done work and + tracks changes to both file content and build commands. This way, Bazel + knows when something needs to be rebuilt, and rebuilds only that. To further + speed up your builds, you can set up your project to build in a highly + parallel and incremental fashion. -- **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel can build binaries and deployable packages for multiple platforms, including desktop, server, and mobile, from the same project. +* **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel + can build binaries and deployable packages for multiple platforms, including + desktop, server, and mobile, from the same project. -- **Bazel scales.** Bazel maintains agility while handling builds with 100k+ source files. It works with multiple repositories and user bases in the tens of thousands. +* **Bazel scales.** Bazel maintains agility while handling builds with 100k+ + source files. It works with multiple repositories and user bases in the tens + of thousands. -- **Bazel is extensible.** Many [languages](/rules) are supported, and you can extend Bazel to support any other language or framework. +* **Bazel is extensible.** Many [languages](/rules) are + supported, and you can extend Bazel to support any other language or + framework. ## Using Bazel To build or test a project with Bazel, you typically do the following: -1. **Set up Bazel.** Download and [install Bazel](/install). +1. **Set up Bazel.** Download and [install Bazel](/install). -2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a directory where Bazel looks for build inputs and `BUILD` files, and where it stores build outputs. +2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a + directory where Bazel looks for build inputs and `BUILD` files, and where it + stores build outputs. -3. **Write a `BUILD` file**, which tells Bazel what to build and how to build it. +3. **Write a `BUILD` file**, which tells Bazel what to build and how to + build it. - You write your `BUILD` file by declaring build targets using [Starlark](/rules/language), a domain-specific language. (See example [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) + You write your `BUILD` file by declaring build targets using + [Starlark](/rules/language), a domain-specific language. (See example + [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) - A build target specifies a set of input artifacts that Bazel will build plus their dependencies, the build rule Bazel will use to build it, and options that configure the build rule. + A build target specifies a set of input artifacts that Bazel will build plus + their dependencies, the build rule Bazel will use to build it, and options + that configure the build rule. - A build rule specifies the build tools Bazel will use, such as compilers and linkers, and their configurations. Bazel ships with a number of build rules covering the most common artifact types in the supported languages on supported platforms. + A build rule specifies the build tools Bazel will use, such as compilers and + linkers, and their configurations. Bazel ships with a number of build rules + covering the most common artifact types in the supported languages on + supported platforms. -4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel places your outputs within the workspace. +4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel + places your outputs within the workspace. -In addition to building, you can also use Bazel to run [tests](/reference/test-encyclopedia) and [query](/query/guide) the build to trace dependencies in your code. +In addition to building, you can also use Bazel to run +[tests](/reference/test-encyclopedia) and [query](/query/guide) the build +to trace dependencies in your code. ## Bazel build process When running a build or a test, Bazel does the following: -1. **Loads** the `BUILD` files relevant to the target. +1. **Loads** the `BUILD` files relevant to the target. -2. **Analyzes** the inputs and their [dependencies](/concepts/dependencies), applies the specified build rules, and produces an [action](/extending/concepts#evaluation-model) graph. +2. **Analyzes** the inputs and their + [dependencies](/concepts/dependencies), applies the specified build + rules, and produces an [action](/extending/concepts#evaluation-model) + graph. -3. **Executes** the build actions on the inputs until the final build outputs are produced. +3. **Executes** the build actions on the inputs until the final build outputs + are produced. -Since all previous build work is cached, Bazel can identify and reuse cached artifacts and only rebuild or retest what's changed. To further enforce correctness, you can set up Bazel to run builds and tests [hermetically](/basics/hermeticity) through sandboxing, minimizing skew and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). +Since all previous build work is cached, Bazel can identify and reuse cached +artifacts and only rebuild or retest what's changed. To further enforce +correctness, you can set up Bazel to run builds and tests +[hermetically](/basics/hermeticity) through sandboxing, minimizing skew +and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). ### Action graph -The action graph represents the build artifacts, the relationships between them, and the build actions that Bazel will perform. Thanks to this graph, Bazel can [track](/run/build#build-consistency) changes to file content as well as changes to actions, such as build or test commands, and know what build work has previously been done. The graph also enables you to easily [trace dependencies](/query/guide) in your code. +The action graph represents the build artifacts, the relationships between them, +and the build actions that Bazel will perform. Thanks to this graph, Bazel can +[track](/run/build#build-consistency) changes to +file content as well as changes to actions, such as build or test commands, and +know what build work has previously been done. The graph also enables you to +easily [trace dependencies](/query/guide) in your code. ## Getting started tutorials -To get started with Bazel, see [Getting Started](/start/) or jump directly to the Bazel tutorials: +To get started with Bazel, see [Getting Started](/start/) or jump +directly to the Bazel tutorials: -- [Tutorial: Build a C++ Project](/start/cpp) -- [Tutorial: Build a Java Project](/start/java) -- [Tutorial: Build an Android Application](/start/android-app) -- [Tutorial: Build an iOS Application](/start/ios-app) +* [Tutorial: Build a C++ Project](/start/cpp) +* [Tutorial: Build a Java Project](/start/java) +* [Tutorial: Build an Android Application](/start/android-app) +* [Tutorial: Build an iOS Application](/start/ios-app) diff --git a/about/roadmap.mdx b/about/roadmap.mdx index c17485d2..42e63e9b 100644 --- a/about/roadmap.mdx +++ b/about/roadmap.mdx @@ -2,50 +2,98 @@ title: 'Bazel roadmap' --- -As Bazel continues to evolve in response to your needs, we want to share our 2025 roadmap update. -We plan to bring Bazel 9.0 [long term support (LTS)](https://bazel.build/release/versioning) to you in late 2025. + +As Bazel continues to evolve in response to your needs, we want to share our +2025 roadmap update. + +We plan to bring Bazel 9.0 +[long term support (LTS)](https://bazel.build/release/versioning) to you in late +2025. ## Full transition to Bzlmod -[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) hosts more than 650 modules. +[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external +dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. +As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) +hosts more than 650 modules. -With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will be the only way to introduce external dependencies in Bazel. To minimize the migration cost for the community, we'll focus on further improving our migration [guide](https://bazel.build/external/migration) and [tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). +With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will +be the only way to introduce external dependencies in Bazel. To minimize the +migration cost for the community, we'll focus on further improving our migration +[guide](https://bazel.build/external/migration) and +[tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). -Additionally, we aim to implement an improved shared repository cache (see [#12227](https://github.com/bazelbuild/bazel/issues/12227)) with garbage collection, and may backport it to Bazel 8. The Bazel Central Registry will also support verifying SLSA attestations. +Additionally, we aim to implement an improved shared repository cache (see +[#12227](https://github.com/bazelbuild/bazel/issues/12227)) +with garbage collection, and may backport it to Bazel 8. The Bazel Central +Registry will also support verifying SLSA attestations. ## Migration of Android, C++, Java, Python, and Proto rules -With Bazel 8, we have migrated support for Android, Java, Python, and Proto rules out of the Bazel codebase into Starlark rules in their corresponding repositories. To ease the migration, we implemented the autoload features in Bazel, which can be controlled with [--incompatible\_autoload\_externally](https://github.com/bazelbuild/bazel/issues/23043) and [--incompatible\_disable\_autoloads\_in\_main\_repo](https://github.com/bazelbuild/bazel/issues/25755) flags. +With Bazel 8, we have migrated support for Android, Java, Python, and Proto +rules out of the Bazel codebase into Starlark rules in their corresponding +repositories. To ease the migration, we implemented the autoload features in +Bazel, which can be controlled with +[--incompatible_autoload_externally](https://github.com/bazelbuild/bazel/issues/23043) +and [--incompatible_disable_autoloads_in_main_repo](https://github.com/bazelbuild/bazel/issues/25755) +flags. -With Bazel 9, we aim to disable autoloads by default and require every project to explicitly load required rules in BUILD files. +With Bazel 9, we aim to disable autoloads by default and require every project +to explicitly load required rules in BUILD files. -We will rewrite most of C++ language support to Starlark, detach it from Bazel binary and move it into the [/rules\_cc](https://github.com/bazelbuild/rules_cc) repository. This is the last remaining major language support that is still part of Bazel. +We will rewrite most of C++ language support to Starlark, detach it from Bazel +binary and move it into the [/rules_cc](https://github.com/bazelbuild/rules_cc) +repository. This is the last remaining major language support that is still part +of Bazel. -We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving them to repositories next to the implementation to increase velocity of rule authors. +We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving +them to repositories next to the implementation to increase velocity of rule +authors. ## Starlark improvements -Bazel will have the ability to evaluate symbolic macros lazily. This means that a symbolic macro won't run if the targets it declares are not requested, improving performance for very large packages. +Bazel will have the ability to evaluate symbolic macros lazily. This means that +a symbolic macro won't run if the targets it declares are not requested, +improving performance for very large packages. -Starlark will have an experimental type system, similar to Python's type annotations. We expect the type system to stabilize *after* Bazel 9 is launched. +Starlark will have an experimental type system, similar to Python's type +annotations. We expect the type system to stabilize _after_ Bazel 9 is launched. ## Configurability Our main focus is reducing the cost and confusion of build flags. -We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a new project configuration model that doesn't make users have to know which build and test flags to set where. So `$ bazel test //foo` automatically sets the right flags based on `foo`'s project's policy. This will likely remain experimental in 9.0 but guiding feedback is welcome. - -[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip out Starlark flags when they leave project boundaries, so they don't break caching on transitive dependencies that don't need them. This makes builds that use [transitions](https://bazel.build/extending/config#user-defined-transitions) cheaper and faster. [Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) an example. We're extending the idea to control which flags propagate to [exec configurations](https://bazel.build/extending/rules#configurations) and are considering even more flexible support like custom Starlark to determine which dependency edges should propagate flags. - -We're up-prioritizing effort to move built-in language flags out of Bazel and into Starlark, where they can live with related rule definitions. +We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a +new project configuration model that doesn't make users have to know which build +and test flags to set where. So `$ bazel test //foo` automatically sets the +right flags based on `foo`'s project's policy. This will likely remain +experimental in 9.0 but guiding feedback is welcome. + +[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip +out Starlark flags when they leave project boundaries, so they don't break +caching on transitive dependencies that don't need them. This makes builds that +use [transitions](https://bazel.build/extending/config#user-defined-transitions) +cheaper and faster. +[Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) +an example. We're extending the idea to control which flags propagate to +[exec configurations](https://bazel.build/extending/rules#configurations) and +are considering even more flexible support like custom Starlark to determine +which dependency edges should propagate flags. + +We're up-prioritizing effort to move built-in language flags out of Bazel and +into Starlark, where they can live with related rule definitions. ## Remote execution improvements -We plan to add support for asynchronous execution, speeding up remote execution by increasing parallelism. +We plan to add support for asynchronous execution, speeding up remote execution +by increasing parallelism. -*** +--- -To follow updates to the roadmap and discuss planned features, join the community Slack server at [slack.bazel.build](https://slack.bazel.build/). +To follow updates to the roadmap and discuss planned features, join the +community Slack server at [slack.bazel.build](https://slack.bazel.build/). -*This roadmap is intended to help inform the community about the team's intentions for Bazel 9.0. Priorities are subject to change in response to developer and customer feedback, or to new market opportunities.* +*This roadmap is intended to help inform the community about the team's +intentions for Bazel 9.0. Priorities are subject to change in response to +developer and customer feedback, or to new market opportunities.* diff --git a/about/vision.mdx b/about/vision.mdx index b80ca03a..da0ed02d 100644 --- a/about/vision.mdx +++ b/about/vision.mdx @@ -2,48 +2,96 @@ title: 'Bazel Vision' --- -Any software developer can efficiently build, test, and package any project, of any size or complexity, with tooling that's easy to adopt and extend. -- **Engineers can take build fundamentals for granted.** Software developers focus on the creative process of authoring code because the mechanical process of build and test is solved. When customizing the build system to support new languages or unique organizational needs, users focus on the aspects of extensibility that are unique to their use case, without having to reinvent the basic plumbing. -- **Engineers can easily contribute to any project.** A developer who wants to start working on a new project can simply clone the project and run the build. There's no need for local configuration - it just works. With cross-platform remote execution, they can work on any machine anywhere and fully test their changes against all platforms the project targets. Engineers can quickly configure the build for a new project or incrementally migrate an existing build. +Any software developer can efficiently build, test, and package +any project, of any size or complexity, with tooling that's easy to adopt and +extend. -- **Projects can scale to any size codebase, any size team.** Fast, incremental testing allows teams to fully validate every change before it is committed. This remains true even as repos grow, projects span multiple repos, and multiple languages are introduced. Infrastructure does not force developers to trade test coverage for build speed. +* **Engineers can take build fundamentals for granted.** Software developers + focus on the creative process of authoring code because the mechanical + process of build and test is solved. When customizing the build system to + support new languages or unique organizational needs, users focus on the + aspects of extensibility that are unique to their use case, without having + to reinvent the basic plumbing. + +* **Engineers can easily contribute to any project.** A developer who wants to + start working on a new project can simply clone the project and run the + build. There's no need for local configuration - it just works. With + cross-platform remote execution, they can work on any machine anywhere and + fully test their changes against all platforms the project targets. + Engineers can quickly configure the build for a new project or incrementally + migrate an existing build. + +* **Projects can scale to any size codebase, any size team.** Fast, + incremental testing allows teams to fully validate every change before it is + committed. This remains true even as repos grow, projects span multiple + repos, and multiple languages are introduced. Infrastructure does not force + developers to trade test coverage for build speed. **We believe Bazel has the potential to fulfill this vision.** -Bazel was built from the ground up to enable builds that are reproducible (a given set of inputs will always produce the same outputs) and portable (a build can be run on any machine without affecting the output). +Bazel was built from the ground up to enable builds that are reproducible (a +given set of inputs will always produce the same outputs) and portable (a build +can be run on any machine without affecting the output). -These characteristics support safe incrementality (rebuilding only changed inputs doesn't introduce the risk of corruption) and distributability (build actions are isolated and can be offloaded). By minimizing the work needed to do a correct build and parallelizing that work across multiple cores and remote systems, Bazel can make any build fast. +These characteristics support safe incrementality (rebuilding only changed +inputs doesn't introduce the risk of corruption) and distributability (build +actions are isolated and can be offloaded). By minimizing the work needed to do +a correct build and parallelizing that work across multiple cores and remote +systems, Bazel can make any build fast. -Bazel's abstraction layer — instructions specific to languages, platforms, and toolchains implemented in a simple extensibility language — allows it to be easily applied to any context. +Bazel's abstraction layer — instructions specific to languages, platforms, and +toolchains implemented in a simple extensibility language — allows it to be +easily applied to any context. ## Bazel core competencies -1. Bazel supports **multi-language, multi-platform** builds and tests. You can run a single command to build and test your entire source tree, no matter which combination of languages and platforms you target. -2. Bazel builds are **fast and correct**. Every build and test run is incremental, on your developers' machines and on CI. -3. Bazel provides a **uniform, extensible language** to define builds for any language or platform. -4. Bazel allows your builds **to scale** by connecting to remote execution and caching services. -5. Bazel works across **all major development platforms** (Linux, MacOS, and Windows). -6. We accept that adopting Bazel requires effort, but **gradual adoption** is possible. Bazel interfaces with de-facto standard tools for a given language/platform. +1. Bazel supports **multi-language, multi-platform** builds and tests. You can + run a single command to build and test your entire source tree, no matter + which combination of languages and platforms you target. +1. Bazel builds are **fast and correct**. Every build and test run is + incremental, on your developers' machines and on CI. +1. Bazel provides a **uniform, extensible language** to define builds for any + language or platform. +1. Bazel allows your builds **to scale** by connecting to remote execution and + caching services. +1. Bazel works across **all major development platforms** (Linux, MacOS, and + Windows). +1. We accept that adopting Bazel requires effort, but **gradual adoption** is + possible. Bazel interfaces with de-facto standard tools for a given + language/platform. ## Serving language communities -Software engineering evolves in the context of language communities — typically, self-organizing groups of people who use common tools and practices. - -To be of use to members of a language community, high-quality Bazel rules must be available that integrate with the workflows and conventions of that community. - -Bazel is committed to be extensible and open, and to support good rulesets for any language. - -### Requirements of a good ruleset - -1. The rules need to support efficient **building and testing** for the language, including code coverage. -2. The rules need to **interface with a widely-used "package manager"** for the language (such as Maven for Java), and support incremental migration paths from other widely-used build systems. -3. The rules need to be **extensible and interoperable**, following ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) principles. -4. The rules need to be **remote-execution ready**. In practice, this means **configurable using the [toolchains](/extending/toolchains) mechanism**. -5. The rules (and Bazel) need to interface with a **widely-used IDE** for the language, if there is one. -6. The rules need to have **thorough, usable documentation,** with introductory material for new users, comprehensive docs for expert users. - -Each of these items is essential and only together do they deliver on Bazel's competencies for their particular ecosystem. - -They are also, by and large, sufficient - once all are fulfilled, Bazel fully delivers its value to members of that language community. +Software engineering evolves in the context of language communities — typically, +self-organizing groups of people who use common tools and practices. + +To be of use to members of a language community, high-quality Bazel rules must be +available that integrate with the workflows and conventions of that community. + +Bazel is committed to be extensible and open, and to support good rulesets for +any language. + +### Requirements of a good ruleset + +1. The rules need to support efficient **building and testing** for the + language, including code coverage. +1. The rules need to **interface with a widely-used "package manager"** for the + language (such as Maven for Java), and support incremental migration paths + from other widely-used build systems. +1. The rules need to be **extensible and interoperable**, following + ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) + principles. +1. The rules need to be **remote-execution ready**. In practice, this means + **configurable using the [toolchains](/extending/toolchains) mechanism**. +1. The rules (and Bazel) need to interface with a **widely-used IDE** for the + language, if there is one. +1. The rules need to have **thorough, usable documentation,** with introductory + material for new users, comprehensive docs for expert users. + +Each of these items is essential and only together do they deliver on Bazel's +competencies for their particular ecosystem. + +They are also, by and large, sufficient - once all are fulfilled, Bazel fully +delivers its value to members of that language community. diff --git a/about/why.mdx b/about/why.mdx index dcdec18e..6224abf6 100644 --- a/about/why.mdx +++ b/about/why.mdx @@ -2,44 +2,84 @@ title: 'Why Bazel?' --- -Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) build tool with [integrated testing](#integrated-testing) that supports multiple [languages](#multi-language), [repositories](#multi-repository), and [platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). + + +Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) +build tool with [integrated testing](#integrated-testing) that supports multiple +[languages](#multi-language), [repositories](#multi-repository), and +[platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). ## Bazel is fast -Bazel knows exactly what input files each build command needs, avoiding unnecessary work by re-running only when the set of input files have changed between each build. It runs build commands with as much parallelism as possible, either within the same computer or on [remote build nodes](/remote/rbe). If the structure of build allows for it, it can run thousands of build or test commands at the same time. +Bazel knows exactly what input files each build command needs, avoiding +unnecessary work by re-running only when the set of input files have +changed between each build. +It runs build commands with as much parallelism as possible, either within the +same computer or on [remote build nodes](/remote/rbe). If the structure of build +allows for it, it can run thousands of build or test commands at the same time. -This is supported by multiple caching layers, in memory, on disk and on the remote build farm, if available. At Google, we routinely achieve cache hit rates north of 99%. +This is supported by multiple caching layers, in memory, on disk and on the +remote build farm, if available. At Google, we routinely achieve cache hit rates +north of 99%. ## Bazel is correct -Bazel ensures that your binaries are built *only* from your own source code. Bazel actions run in individual sandboxes and Bazel tracks every input file of the build, only and always re-running build commands when it needs to. This keeps your binaries up-to-date so that the [same source code always results in the same binary](/basics/hermeticity), bit by bit. +Bazel ensures that your binaries are built *only* from your own +source code. Bazel actions run in individual sandboxes and Bazel tracks +every input file of the build, only and always re-running build +commands when it needs to. This keeps your binaries up-to-date so that the +[same source code always results in the same binary](/basics/hermeticity), bit +by bit. -Say goodbye to endless `make clean` invocations and to chasing phantom bugs that were in fact resolved in source code that never got built. +Say goodbye to endless `make clean` invocations and to chasing phantom bugs +that were in fact resolved in source code that never got built. ## Bazel is extensible -Harness the full power of Bazel by writing your own rules and macros to customize Bazel for your specific needs across a wide range of projects. +Harness the full power of Bazel by writing your own rules and macros to +customize Bazel for your specific needs across a wide range of projects. -Bazel rules are written in [Starlark](/rules/language), our in-house programming language that's a subset of Python. Starlark makes rule-writing accessible to most developers, while also creating rules that can be used across the ecosystem. +Bazel rules are written in [Starlark](/rules/language), our +in-house programming language that's a subset of Python. Starlark makes +rule-writing accessible to most developers, while also creating rules that can +be used across the ecosystem. ## Integrated testing -Bazel's [integrated test runner](/docs/user-manual#running-tests) knows and runs only those tests needing to be re-run, using remote execution (if available) to run them in parallel. Detect flakes early by using remote execution to quickly run a test thousands of times. +Bazel's [integrated test runner](/docs/user-manual#running-tests) +knows and runs only those tests needing to be re-run, using remote execution +(if available) to run them in parallel. Detect flakes early by using remote +execution to quickly run a test thousands of times. -Bazel [provides facilities](/remote/bep) to upload test results to a central location, thereby facilitating efficient communication of test outcomes, be it on CI or by individual developers. +Bazel [provides facilities](/remote/bep) to upload test results to a central +location, thereby facilitating efficient communication of test outcomes, be it +on CI or by individual developers. ## Multi-language support -Bazel supports many common programming languages including C++, Java, Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, backend, web UI and mobile app) in the same Bazel invocation without being constrained to one language's idiomatic build tool. +Bazel supports many common programming languages including C++, Java, +Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, +backend, web UI and mobile app) in the same Bazel invocation without being +constrained to one language's idiomatic build tool. ## Multi-repository support -Bazel can [gather source code from multiple locations](/external/overview): you don't need to vendor your dependencies (but you can!), you can instead point Bazel to the location of your source code or prebuilt artifacts (e.g. a git repository or Maven Central), and it takes care of the rest. +Bazel can [gather source code from multiple locations](/external/overview): you +don't need to vendor your dependencies (but you can!), you can instead point +Bazel to the location of your source code or prebuilt artifacts (e.g. a git +repository or Maven Central), and it takes care of the rest. ## Multi-platform support -Bazel can simultaneously build projects for multiple platforms including Linux, macOS, Windows, and Android. It also provides powerful [cross-compilation capabilities](/extending/platforms) to build code for one platform while running the build on another. +Bazel can simultaneously build projects for multiple platforms including Linux, +macOS, Windows, and Android. It also provides powerful +[cross-compilation capabilities](/extending/platforms) to build code for one +platform while running the build on another. ## Wide ecosystem -[Industry leaders](/community/users) love Bazel, building a large community of developers who use and contribute to Bazel. Find a tools, services and documentation, including [consulting and SaaS offerings](/community/experts) Bazel can use. Explore extensions like support for programming languages in our [open source software repositories](/rules). +[Industry leaders](/community/users) love Bazel, building a large +community of developers who use and contribute to Bazel. Find a tools, services +and documentation, including [consulting and SaaS offerings](/community/experts) +Bazel can use. Explore extensions like support for programming languages in +our [open source software repositories](/rules). diff --git a/advanced/performance/build-performance-breakdown.mdx b/advanced/performance/build-performance-breakdown.mdx index 8bebff8c..477e7578 100644 --- a/advanced/performance/build-performance-breakdown.mdx +++ b/advanced/performance/build-performance-breakdown.mdx @@ -2,120 +2,234 @@ title: 'Breaking down build performance' --- -Bazel is complex and does a lot of different things over the course of a build, some of which can have an impact on build performance. This page attempts to map some of these Bazel concepts to their implications on build performance. While not extensive, we have included some examples of how to detect build performance issues through [extracting metrics](/configure/build-performance-metrics) and what you can do to fix them. With this, we hope you can apply these concepts when investigating build performance regressions. -### Clean vs Incremental builds -A clean build is one that builds everything from scratch, while an incremental build reuses some already completed work. +Bazel is complex and does a lot of different things over the course of a build, +some of which can have an impact on build performance. This page attempts to map +some of these Bazel concepts to their implications on build performance. While +not extensive, we have included some examples of how to detect build performance +issues through [extracting metrics](/configure/build-performance-metrics) +and what you can do to fix them. With this, we hope you can apply these concepts +when investigating build performance regressions. -We suggest looking at clean and incremental builds separately, especially when you are collecting / aggregating metrics that are dependent on the state of Bazel’s caches (for example [build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) ). They also represent two different user experiences. As compared to starting a clean build from scratch (which takes longer due to a cold cache), incremental builds happen far more frequently as developers iterate on code (typically faster since the cache is usually already warm). +### Clean vs Incremental builds -You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly categorize it to likely be an incremental build - the user could have switched to different flags or different targets causing an effectively clean build. Any more rigorous definition of incrementality will likely have to come in the form of a heuristic, for example looking at the number of packages loaded (`PackageMetrics.packages_loaded`). +A clean build is one that builds everything from scratch, while an incremental +build reuses some already completed work. + +We suggest looking at clean and incremental builds separately, especially when +you are collecting / aggregating metrics that are dependent on the state of +Bazel’s caches (for example +[build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) +). They also represent two different user experiences. As compared to starting +a clean build from scratch (which takes longer due to a cold cache), incremental +builds happen far more frequently as developers iterate on code (typically +faster since the cache is usually already warm). + +You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify +builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly +categorize it to likely be an incremental build - the user could have switched +to different flags or different targets causing an effectively clean build. Any +more rigorous definition of incrementality will likely have to come in the form +of a heuristic, for example looking at the number of packages loaded +(`PackageMetrics.packages_loaded`). ### Deterministic build metrics as a proxy for build performance -Measuring build performance can be difficult due to the non-deterministic nature of certain metrics (for example Bazel’s CPU time or queue times on a remote cluster). As such, it can be useful to use deterministic metrics as a proxy for the amount of work done by Bazel, which in turn affects its performance. - -The size of a build request can have a significant implication on build performance. A larger build could represent more work in analyzing and constructing the build graphs. Organic growth of builds comes naturally with development, as more dependencies are added/created, and thus grow in complexity and become more expensive to build. - -We can slice this problem into the various build phases, and use the following metrics as proxy metrics for work done at each phase: - -1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. A regression here represents more work that needs to be done to read and parse each additional BUILD file in the loading phase. - - - This is often due to the addition of dependencies and having to load their transitive closure. - - Use [query](/query/quickstart) / [cquery](/query/cquery) to find where new dependencies might have been added. - -2. `TargetMetrics.targets_configured`: representing the number of targets and aspects configured in the build. A regression represents more work in constructing and traversing the configured target graph. - - - This is often due to the addition of dependencies and having to construct the graph of their transitive closure. - - Use [cquery](/query/cquery) to find where new dependencies might have been added. - -3. `ActionSummary.actions_created`: represents the actions created in the build, and a regression represents more work in constructing the action graph. Note that this also includes unused actions that might not have been executed. - - - Use [aquery](/query/aquery) for debugging regressions; we suggest starting with [`--output=summary`](/reference/command-line-reference#flag--output) before further drilling down with [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). - -4. `ActionSummary.actions_executed`: the number of actions executed, a regression directly represents more work in executing these actions. - - - The [BEP](/remote/bep) writes out the action statistics `ActionData` that shows the most executed action types. By default, it collects the top 20 action types, but you can pass in the [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) to collect this data for all action types that were executed. - - This should help you to figure out what kind of actions were executed (additionally). - -5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by the executed actions. - - - If the number of actions executed did not increase, then it is likely that a rule implementation was changed. - -These metrics are all affected by the state of the local cache, hence you will want to ensure that the builds you extract these metrics from are **clean builds**. - -We have noted that a regression in any of these metrics can be accompanied by regressions in wall time, cpu time and memory usage. +Measuring build performance can be difficult due to the non-deterministic nature +of certain metrics (for example Bazel’s CPU time or queue times on a remote +cluster). As such, it can be useful to use deterministic metrics as a proxy for +the amount of work done by Bazel, which in turn affects its performance. + +The size of a build request can have a significant implication on build +performance. A larger build could represent more work in analyzing and +constructing the build graphs. Organic growth of builds comes naturally with +development, as more dependencies are added/created, and thus grow in complexity +and become more expensive to build. + +We can slice this problem into the various build phases, and use the following +metrics as proxy metrics for work done at each phase: + +1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. + A regression here represents more work that needs to be done to read and parse + each additional BUILD file in the loading phase. + - This is often due to the addition of dependencies and having to load their + transitive closure. + - Use [query](/query/quickstart) / [cquery](/query/cquery) to find + where new dependencies might have been added. + +2. `TargetMetrics.targets_configured`: representing the number of targets and + aspects configured in the build. A regression represents more work in + constructing and traversing the configured target graph. + - This is often due to the addition of dependencies and having to construct + the graph of their transitive closure. + - Use [cquery](/query/cquery) to find where new + dependencies might have been added. + +3. `ActionSummary.actions_created`: represents the actions created in the build, + and a regression represents more work in constructing the action graph. Note + that this also includes unused actions that might not have been executed. + - Use [aquery](/query/aquery) for debugging regressions; + we suggest starting with + [`--output=summary`](/reference/command-line-reference#flag--output) + before further drilling down with + [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). + +4. `ActionSummary.actions_executed`: the number of actions executed, a + regression directly represents more work in executing these actions. + - The [BEP](/remote/bep) writes out the action statistics + `ActionData` that shows the most executed action types. By default, it + collects the top 20 action types, but you can pass in the + [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) + to collect this data for all action types that were executed. + - This should help you to figure out what kind of actions were executed + (additionally). + +5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by + the executed actions. + - If the number of actions executed did not increase, then it is likely that + a rule implementation was changed. + + +These metrics are all affected by the state of the local cache, hence you will +want to ensure that the builds you extract these metrics from are +**clean builds**. + +We have noted that a regression in any of these metrics can be accompanied by +regressions in wall time, cpu time and memory usage. ### Usage of local resources -Bazel consumes a variety of resources on your local machine (both for analyzing the build graph and driving the execution, and for running local actions), this can affect the performance / availability of your machine in performing the build, and also other tasks. +Bazel consumes a variety of resources on your local machine (both for analyzing +the build graph and driving the execution, and for running local actions), this +can affect the performance / availability of your machine in performing the +build, and also other tasks. #### Time spent -Perhaps the metrics most susceptible to noise (and can vary greatly from build to build) is time; in particular - wall time, cpu time and system time. You can use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get a benchmark for these metrics, and with a sufficient number of `--runs`, you can increase the statistical significance of your measurement. +Perhaps the metrics most susceptible to noise (and can vary greatly from build +to build) is time; in particular - wall time, cpu time and system time. You can +use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get +a benchmark for these metrics, and with a sufficient number of `--runs`, you can +increase the statistical significance of your measurement. - **Wall time** is the real world time elapsed. - - - If *only* wall time regresses, we suggest collecting a [JSON trace profile](/advanced/performance/json-trace-profile) and looking for differences. Otherwise, it would likely be more efficient to investigate other regressed metrics as they could have affected the wall time. + - If _only_ wall time regresses, we suggest collecting a + [JSON trace profile](/advanced/performance/json-trace-profile) and looking + for differences. Otherwise, it would likely be more efficient to + investigate other regressed metrics as they could have affected the wall + time. - **CPU time** is the time spent by the CPU executing user code. - - - If the CPU time regresses across two project commits, we suggest collecting a Starlark CPU profile. You should probably also use `--nobuild` to restrict the build to the analysis phase since that is where most of the CPU heavy work is done. + - If the CPU time regresses across two project commits, we suggest collecting + a Starlark CPU profile. You should probably also use `--nobuild` to + restrict the build to the analysis phase since that is where most of the + CPU heavy work is done. - System time is the time spent by the CPU in the kernel. - - - If system time regresses, it is mostly correlated with I/O when Bazel reads files from your file system. + - If system time regresses, it is mostly correlated with I/O when Bazel reads + files from your file system. #### System-wide load profiling -Using the [`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) flag introduced in Bazel 6.0, the [JSON trace profiler](/advanced/performance/json-trace-profile) collects the system load average during the invocation. +Using the +[`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) +flag introduced in Bazel 6.0, the +[JSON trace profiler](/advanced/performance/json-trace-profile) collects the +system load average during the invocation. ![Profile that includes system load average](/docs/images/json-trace-profile-system-load-average.png "Profile that includes system load average") **Figure 1.** Profile that includes system load average. -A high load during a Bazel invocation can be an indication that Bazel schedules too many local actions in parallel for your machine. You might want to look into adjusting [`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), especially in container environments (at least until [#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). +A high load during a Bazel invocation can be an indication that Bazel schedules +too many local actions in parallel for your machine. You might want to look into +adjusting +[`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) +and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), +especially in container environments (at least until +[#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). -#### Monitoring Bazel memory usage - -There are two main sources to get Bazel’s memory usage, Bazel `info` and the [BEP](/remote/bep). - -- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after a call to `System.gc()`. - - [Bazel bench](https://github.com/bazelbuild/bazel-bench) provides benchmarks for this metric as well. - - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` and `committed-heap-size` (see [documentation](/docs/user-manual#configuration-independent-data)), but are less relevant. - -- [BEP](/remote/bep)’s `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in bytes post GC (requires setting [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) that attempts to force a full GC). - -A regression in memory usage is usually a result of a regression in [build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), which are often due to addition of dependencies or a change in the rule implementation. +#### Monitoring Bazel memory usage -To analyze Bazel’s memory footprint on a more granular level, we recommend using the [built-in memory profiler](/rules/performance#memory-profiling) for rules. +There are two main sources to get Bazel’s memory usage, Bazel `info` and the +[BEP](/remote/bep). + +- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after + a call to `System.gc()`. + - [Bazel bench](https://github.com/bazelbuild/bazel-bench) + provides benchmarks for this metric as well. + - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` + and `committed-heap-size` (see + [documentation](/docs/user-manual#configuration-independent-data)), but are + less relevant. + +- [BEP](/remote/bep)’s + `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in + bytes post GC (requires setting + [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) + that attempts to force a full GC). + +A regression in memory usage is usually a result of a regression in +[build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), +which are often due to addition of dependencies or a change in the rule +implementation. + +To analyze Bazel’s memory footprint on a more granular level, we recommend using +the [built-in memory profiler](/rules/performance#memory-profiling) +for rules. #### Memory profiling of persistent workers -While [persistent workers](/remote/persistent) can help to speed up builds significantly (especially for interpreted languages) their memory footprint can be problematic. Bazel collects metrics on its workers, in particular, the `WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory workers use (by mnemonic). +While [persistent workers](/remote/persistent) can help to speed up builds +significantly (especially for interpreted languages) their memory footprint can +be problematic. Bazel collects metrics on its workers, in particular, the +`WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory +workers use (by mnemonic). -The [JSON trace profiler](/advanced/performance/json-trace-profile) also collects persistent worker memory usage during the invocation by passing in the [`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) flag (new in Bazel 6.0). +The [JSON trace profiler](/advanced/performance/json-trace-profile) also +collects persistent worker memory usage during the invocation by passing in the +[`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) +flag (new in Bazel 6.0). ![Profile that includes workers memory usage](/docs/images/json-trace-profile-workers-memory-usage.png "Profile that includes workers memory usage") **Figure 2.** Profile that includes workers memory usage. -Lowering the value of [`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) (default 4) might help to reduce the amount of memory used by persistent workers. We are actively working on making Bazel’s resource manager and scheduler smarter so that such fine tuning will be required less often in the future. +Lowering the value of +[`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) +(default 4) might help to reduce +the amount of memory used by persistent workers. We are actively working on +making Bazel’s resource manager and scheduler smarter so that such fine tuning +will be required less often in the future. ### Monitoring network traffic for remote builds -In remote execution, Bazel downloads artifacts that were built as a result of executing actions. As such, your network bandwidth can affect the performance of your build. +In remote execution, Bazel downloads artifacts that were built as a result of +executing actions. As such, your network bandwidth can affect the performance +of your build. -If you are using remote execution for your builds, you might want to consider monitoring the network traffic during the invocation using the `NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) (requires passing `--experimental_collect_system_network_usage`). +If you are using remote execution for your builds, you might want to consider +monitoring the network traffic during the invocation using the +`NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) +(requires passing `--experimental_collect_system_network_usage`). -Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) allow you to view system-wide network usage throughout the course of the build by passing the `--experimental_collect_system_network_usage` flag (new in Bazel 6.0). +Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) +allow you to view system-wide network usage throughout the course of the build +by passing the `--experimental_collect_system_network_usage` flag (new in Bazel +6.0). ![Profile that includes system-wide network usage](/docs/images/json-trace-profile-network-usage.png "Profile that includes system-wide network usage") **Figure 3.** Profile that includes system-wide network usage. -A high but rather flat network usage when using remote execution might indicate that network is the bottleneck in your build; if you are not using it already, consider turning on Build without the Bytes by passing [`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. +A high but rather flat network usage when using remote execution might indicate +that network is the bottleneck in your build; if you are not using it already, +consider turning on Build without the Bytes by passing +[`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). +This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. -Another option is to configure a local [disk cache](/reference/command-line-reference#flag--disk_cache) to save on download bandwidth. +Another option is to configure a local +[disk cache](/reference/command-line-reference#flag--disk_cache) to save on +download bandwidth. diff --git a/advanced/performance/build-performance-metrics.mdx b/advanced/performance/build-performance-metrics.mdx index 8f92e75a..8391ea87 100644 --- a/advanced/performance/build-performance-metrics.mdx +++ b/advanced/performance/build-performance-metrics.mdx @@ -2,50 +2,96 @@ title: 'Extracting build performance metrics' --- -Probably every Bazel user has experienced builds that were slow or slower than anticipated. Improving the performance of individual builds has particular value for targets with significant impact, such as: + + +Probably every Bazel user has experienced builds that were slow or slower than +anticipated. Improving the performance of individual builds has particular value +for targets with significant impact, such as: 1. Core developer targets that are frequently iterated on and (re)built. 2. Common libraries widely depended upon by other targets. -3. A representative target from a class of targets (e.g. custom rules), diagnosing and fixing issues in one build might help to resolve issues at the larger scale. +3. A representative target from a class of targets (e.g. custom rules), + diagnosing and fixing issues in one build might help to resolve issues at the + larger scale. -An important step to improving the performance of builds is to understand where resources are spent. This page lists different metrics you can collect. [Breaking down build performance](/configure/build-performance-breakdown) showcases how you can use these metrics to detect and fix build performance issues. +An important step to improving the performance of builds is to understand where +resources are spent. This page lists different metrics you can collect. +[Breaking down build performance](/configure/build-performance-breakdown) showcases +how you can use these metrics to detect and fix build performance issues. There are a few main ways to extract metrics from your Bazel builds, namely: ## Build Event Protocol (BEP) -Bazel outputs a variety of protocol buffers [`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) through the [Build Event Protocol (BEP)](/remote/bep), which can be aggregated by a backend specified by you. Depending on your use cases, you might decide to aggregate the metrics in various ways, but here we will go over some concepts and proto fields that would be useful in general to consider. +Bazel outputs a variety of protocol buffers +[`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) +through the [Build Event Protocol (BEP)](/remote/bep), which +can be aggregated by a backend specified by you. Depending on your use cases, +you might decide to aggregate the metrics in various ways, but here we will go +over some concepts and proto fields that would be useful in general to consider. ## Bazel’s query / cquery / aquery commands -Bazel provides 3 different query modes ([query](/query/quickstart), [cquery](/query/cquery) and [aquery](/query/aquery)) that allow users to query the target graph, configured target graph and action graph respectively. The query language provides a [suite of functions](/query/language#functions) usable across the different query modes, that allows you to customize your queries according to your needs. +Bazel provides 3 different query modes ([query](/query/quickstart), +[cquery](/query/cquery) and [aquery](/query/aquery)) that allow users +to query the target graph, configured target graph and action graph +respectively. The query language provides a +[suite of functions](/query/language#functions) usable across the different +query modes, that allows you to customize your queries according to your needs. ## JSON Trace Profiles -For every build-like Bazel invocation, Bazel writes a trace profile in JSON format. The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. +For every build-like Bazel invocation, Bazel writes a trace profile in JSON +format. The [JSON trace profile](/advanced/performance/json-trace-profile) can +be very useful to quickly understand what Bazel spent time on during the +invocation. ## Execution Log -The [execution log](/remote/cache-remote) can help you to troubleshoot and fix missing remote cache hits due to machine and environment differences or non-deterministic actions. If you pass the flag [`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) (available from Bazel 5.2) it will also contain detailed spawn metrics, both for locally and remotely executed actions. You can use these metrics for example to make comparisons between local and remote machine performance or to find out which part of the spawn execution is consistently slower than expected (for example due to queuing). +The [execution log](/remote/cache-remote) can help you to troubleshoot and fix +missing remote cache hits due to machine and environment differences or +non-deterministic actions. If you pass the flag +[`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) +(available from Bazel 5.2) it will also contain detailed spawn metrics, both for +locally and remotely executed actions. You can use these metrics for example to +make comparisons between local and remote machine performance or to find out +which part of the spawn execution is consistently slower than expected (for +example due to queuing). ## Execution Graph Log -While the JSON trace profile contains the critical path information, sometimes you need additional information on the dependency graph of the executed actions. Starting with Bazel 6.0, you can pass the flags `--experimental_execution_graph_log` and `--experimental_execution_graph_log_dep_type=all` to write out a log about the executed actions and their inter-dependencies. +While the JSON trace profile contains the critical path information, sometimes +you need additional information on the dependency graph of the executed actions. +Starting with Bazel 6.0, you can pass the flags +`--experimental_execution_graph_log` and +`--experimental_execution_graph_log_dep_type=all` to write out a log about the +executed actions and their inter-dependencies. -This information can be used to understand the drag that is added by a node on the critical path. The drag is the amount of time that can potentially be saved by removing a particular node from the execution graph. +This information can be used to understand the drag that is added by a node on +the critical path. The drag is the amount of time that can potentially be saved +by removing a particular node from the execution graph. -The data helps you predict the impact of changes to the build and action graph before you actually do them. +The data helps you predict the impact of changes to the build and action graph +before you actually do them. ## Benchmarking with bazel-bench -[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a benchmarking tool for Git projects to benchmark build performance in the following cases: +[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a +benchmarking tool for Git projects to benchmark build performance in the +following cases: -- **Project benchmark:** Benchmarking two git commits against each other at a single Bazel version. Used to detect regressions in your build (often through the addition of dependencies). +* **Project benchmark:** Benchmarking two git commits against each other at a + single Bazel version. Used to detect regressions in your build (often through + the addition of dependencies). -- **Bazel benchmark:** Benchmarking two versions of Bazel against each other at a single git commit. Used to detect regressions within Bazel itself (if you happen to maintain / fork Bazel). +* **Bazel benchmark:** Benchmarking two versions of Bazel against each other at + a single git commit. Used to detect regressions within Bazel itself (if you + happen to maintain / fork Bazel). -Benchmarks monitor wall time, CPU time and system time and Bazel’s retained heap size. +Benchmarks monitor wall time, CPU time and system time and Bazel’s retained +heap size. -It is also recommended to run Bazel bench on dedicated, physical machines that are not running other processes so as to reduce sources of variability. +It is also recommended to run Bazel bench on dedicated, physical machines that +are not running other processes so as to reduce sources of variability. diff --git a/advanced/performance/iteration-speed.mdx b/advanced/performance/iteration-speed.mdx index 79240cd0..2bbf8398 100644 --- a/advanced/performance/iteration-speed.mdx +++ b/advanced/performance/iteration-speed.mdx @@ -2,44 +2,92 @@ title: 'Optimize Iteration Speed' --- -This page describes how to optimize Bazel's build performance when running Bazel repeatedly. + + +This page describes how to optimize Bazel's build performance when running Bazel +repeatedly. ## Bazel's Runtime State A Bazel invocation involves several interacting parts. -- The `bazel` command line interface (CLI) is the user-facing front-end tool and receives commands from the user. +* The `bazel` command line interface (CLI) is the user-facing front-end tool + and receives commands from the user. -- The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) for each distinct [output base](https://bazel.build/remote/output-directories). The Bazel server is generally persistent, but will shut down after some idle time so as to not waste resources. +* The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) + for each distinct [output base](https://bazel.build/remote/output-directories). + The Bazel server is generally persistent, but will shut down after some idle + time so as to not waste resources. -- The Bazel server performs the loading and analysis steps for a given command (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts of the build graph in memory. The resulting data structures are retained in the Bazel server as part of the *analysis cache*. +* The Bazel server performs the loading and analysis steps for a given command + (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts + of the build graph in memory. The resulting data structures are retained in + the Bazel server as part of the *analysis cache*. -- The Bazel server can also perform the action execution, or it can send actions off for remote execution if it is set up to do so. The results of action executions are also cached, namely in the *action cache* (or *execution cache*, which may be either local or remote, and it may be shared among Bazel servers). +* The Bazel server can also perform the action execution, or it can send + actions off for remote execution if it is set up to do so. The results of + action executions are also cached, namely in the *action cache* (or + *execution cache*, which may be either local or remote, and it may be shared + among Bazel servers). -- The result of the Bazel invocation is made available in the output tree. +* The result of the Bazel invocation is made available in the output tree. ## Running Bazel Iteratively -In a typical developer workflow, it is common to build (or run) a piece of code repeatedly, often at a very high frequency (e.g. to resolve some compilation error or investigate a failing test). In this situation, it is important that repeated invocations of `bazel` have as little overhead as possible relative to the underlying, repeated action (e.g. invoking a compiler, or executing a test). +In a typical developer workflow, it is common to build (or run) a piece of code +repeatedly, often at a very high frequency (e.g. to resolve some compilation +error or investigate a failing test). In this situation, it is important that +repeated invocations of `bazel` have as little overhead as possible relative to +the underlying, repeated action (e.g. invoking a compiler, or executing a test). With this in mind, we take another look at Bazel's runtime state: -The analysis cache is a critical piece of data. A significant amount of time can be spent just on the loading and analysis phases of a cold run (i.e. a run just after the Bazel server was started or when the analysis cache was discarded). For a single, successful cold build (e.g. for a production release) this cost is bearable, but for repeatedly building the same target it is important that this cost be amortized and not repeated on each invocation. - -The analysis cache is rather volatile. First off, it is part of the in-process state of the Bazel server, so losing the server loses the cache. But the cache is also *invalidated* very easily: for example, many `bazel` command line flags cause the cache to be discarded. This is because many flags affect the build graph (e.g. because of [configurable attributes](https://bazel.build/configure/attributes)). Some flag changes can also cause the Bazel server to be restarted (e.g. changing [startup options](https://bazel.build/docs/user-manual#startup-options)). - -A good execution cache is also valuable for build performance. An execution cache can be kept locally [on disk](https://bazel.build/remote/caching#disk-cache), or [remotely](https://bazel.build/remote/caching). The cache can be shared among Bazel servers, and indeed among developers. +The analysis cache is a critical piece of data. A significant amount of time can +be spent just on the loading and analysis phases of a cold run (i.e. a run just +after the Bazel server was started or when the analysis cache was discarded). +For a single, successful cold build (e.g. for a production release) this cost is +bearable, but for repeatedly building the same target it is important that this +cost be amortized and not repeated on each invocation. + +The analysis cache is rather volatile. First off, it is part of the in-process +state of the Bazel server, so losing the server loses the cache. But the cache +is also *invalidated* very easily: for example, many `bazel` command line flags +cause the cache to be discarded. This is because many flags affect the build +graph (e.g. because of +[configurable attributes](https://bazel.build/configure/attributes)). Some flag +changes can also cause the Bazel server to be restarted (e.g. changing +[startup options](https://bazel.build/docs/user-manual#startup-options)). + +A good execution cache is also valuable for build performance. An execution +cache can be kept locally +[on disk](https://bazel.build/remote/caching#disk-cache), or +[remotely](https://bazel.build/remote/caching). The cache can be shared among +Bazel servers, and indeed among developers. ## Avoid discarding the analysis cache -Bazel will print a warning if either the analysis cache was discarded or the server was restarted. Either of these should be avoided during iterative use: +Bazel will print a warning if either the analysis cache was discarded or the +server was restarted. Either of these should be avoided during iterative use: -- Be mindful of changing `bazel` flags in the middle of an iterative workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` causes each command to discard the analysis cache of the other. In general, try to use a fixed set of flags for the duration of a particular workflow. +* Be mindful of changing `bazel` flags in the middle of an iterative + workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` + causes each command to discard the analysis cache of the other. In general, + try to use a fixed set of flags for the duration of a particular workflow. -- Losing the Bazel server loses the analysis cache. The Bazel server has a [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle time, after which it shuts down. You can configure this time via your bazelrc file to suit your needs. The server also restarted when startup flags change, so, again, avoid changing those flags if possible. +* Losing the Bazel server loses the analysis cache. The Bazel server has a + [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle + time, after which it shuts down. You can configure this time via your + bazelrc file to suit your needs. The server also restarted when startup + flags change, so, again, avoid changing those flags if possible. -- [Beware]() that the Bazel server is killed if you press Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time by interrupting a running build that is no longer needed, but only press Ctrl-C once to request a graceful end of the current invocation. +* Beware that the Bazel server is killed if you press + Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time + by interrupting a running build that is no longer needed, but only press + Ctrl-C once to request a graceful end of the current invocation. -- If you want to use multiple sets of flags from the same workspace, you can use multiple, distinct output bases, switched with the `--output_base` flag. Each output base gets its own Bazel server. +* If you want to use multiple sets of flags from the same workspace, you can + use multiple, distinct output bases, switched with the `--output_base` + flag. Each output base gets its own Bazel server. -To make this condition an error rather than a warning, you can use the `--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) +To make this condition an error rather than a warning, you can use the +`--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) diff --git a/advanced/performance/json-trace-profile.mdx b/advanced/performance/json-trace-profile.mdx index c452c4b0..80c698c0 100644 --- a/advanced/performance/json-trace-profile.mdx +++ b/advanced/performance/json-trace-profile.mdx @@ -2,17 +2,34 @@ title: 'JSON Trace Profile' --- -The JSON trace profile can be very useful to quickly understand what Bazel spent time on during the invocation. -By default, for all build-like commands and query, Bazel writes a profile into the output base named `command-$INVOCATION_ID.profile.gz`, where `$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates a symlink called `command.profile.gz` in the output base that points the profile of the latest command. You can configure whether a profile is written with the [`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) flag, and the location it is written to with the [`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are compressed with GZIP. Bazel keeps the last 5 profiles, configurable by [`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), in the output base by default for post-build analysis. Explicitly passing a profile path with `--profile` disables automatic garbage collection. + +The JSON trace profile can be very useful to quickly understand what Bazel spent +time on during the invocation. + +By default, for all build-like commands and query, Bazel writes a profile into +the output base named `command-$INVOCATION_ID.profile.gz`, where +`$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates +a symlink called `command.profile.gz` in the output base that points the profile +of the latest command. You can configure whether a profile is written with the +[`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) +flag, and the location it is written to with the +[`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are +compressed with GZIP. Bazel keeps the last 5 profiles, configurable by +[`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), +in the output base by default for post-build analysis. Explicitly passing a +profile path with `--profile` disables automatic garbage collection. ## Tools -You can load this profile into `chrome://tracing` or analyze and post-process it with other tools. +You can load this profile into `chrome://tracing` or analyze and +post-process it with other tools. ### `chrome://tracing` -To visualize the profile, open `chrome://tracing` in a Chrome browser tab, click "Load" and pick the (potentially compressed) profile file. For more detailed results, click the boxes in the lower left corner. +To visualize the profile, open `chrome://tracing` in a Chrome browser tab, +click "Load" and pick the (potentially compressed) profile file. For more +detailed results, click the boxes in the lower left corner. Example profile: @@ -22,19 +39,29 @@ Example profile: You can use these keyboard controls to navigate: -- Press `1` for "select" mode. In this mode, you can select particular boxes to inspect the event details (see lower left corner). Select multiple events to get a summary and aggregated statistics. -- Press `2` for "pan" mode. Then drag the mouse to move the view. You can also use `a`/`d` to move left/right. -- Press `3` for "zoom" mode. Then drag the mouse to zoom. You can also use `w`/`s` to zoom in/out. -- Press `4` for "timing" mode where you can measure the distance between two events. -- Press `?` to learn about all controls. +* Press `1` for "select" mode. In this mode, you can select + particular boxes to inspect the event details (see lower left corner). + Select multiple events to get a summary and aggregated statistics. +* Press `2` for "pan" mode. Then drag the mouse to move the view. You + can also use `a`/`d` to move left/right. +* Press `3` for "zoom" mode. Then drag the mouse to zoom. You can + also use `w`/`s` to zoom in/out. +* Press `4` for "timing" mode where you can measure the distance + between two events. +* Press `?` to learn about all controls. ### Bazel Invocation Analyzer -The open-source [Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) consumes a profile format and prints suggestions on how to improve the build’s performance. This analysis can be performed using its CLI or on [https://analyzer.engflow.com](https://analyzer.engflow.com). +The open-source +[Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) +consumes a profile format and prints suggestions on how to improve +the build’s performance. This analysis can be performed using its CLI or on +[https://analyzer.engflow.com](https://analyzer.engflow.com). ### `jq` -`jq` is like `sed` for JSON data. An example usage of `jq` to extract all durations of the sandbox creation step in local action execution: +`jq` is like `sed` for JSON data. An example usage of `jq` to extract all +durations of the sandbox creation step in local action execution: ``` $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | jq '.traceEvents | .[] | select(.name == "sandbox.createFileSystem") | .dur' @@ -51,29 +78,50 @@ $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | j ## Profile information -The profile contains multiple rows. Usually the bulk of rows represent Bazel threads and their corresponding events, but some special rows are also included. +The profile contains multiple rows. Usually the bulk of rows represent Bazel +threads and their corresponding events, but some special rows are also included. -The special rows included depend on the version of Bazel invoked when the profile was created, and may be customized by different flags. +The special rows included depend on the version of Bazel invoked when the +profile was created, and may be customized by different flags. Figure 1 shows a profile created with Bazel v5.3.1 and includes these rows: -- `action count`: Displays how many concurrent actions were in flight. Click on it to see the actual value. Should go up to the value of [`--jobs`](/reference/command-line-reference#flag--jobs) in clean builds. -- `CPU usage (Bazel)`: For each second of the build, displays the amount of CPU that was used by Bazel (a value of 1 equals one core being 100% busy). -- `Critical Path`: Displays one block for each action on the critical path. -- `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", and "runAnalysisPhase". -- `Garbage Collector`: Displays minor and major Garbage Collection (GC) pauses. +* `action count`: Displays how many concurrent actions were in flight. Click + on it to see the actual value. Should go up to the value of + [`--jobs`](/reference/command-line-reference#flag--jobs) in clean + builds. +* `CPU usage (Bazel)`: For each second of the build, displays the amount of + CPU that was used by Bazel (a value of 1 equals one core being 100% busy). +* `Critical Path`: Displays one block for each action on the critical path. +* `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of + what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", + and "runAnalysisPhase". +* `Garbage Collector`: Displays minor and major Garbage Collection (GC) + pauses. ## Common performance issues When analyzing performance profiles, look for: -- Slower than expected analysis phase (`runAnalysisPhase`), especially on incremental builds. This can be a sign of a poor rule implementation, for example one that flattens depsets. Package loading can be slow by an excessive amount of targets, complex macros or recursive globs. -- Individual slow actions, especially those on the critical path. It might be possible to split large actions into multiple smaller actions or reduce the set of (transitive) dependencies to speed them up. Also check for an unusual high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). -- Bottlenecks, that is a small number of threads is busy while all others are idling / waiting for the result (see around 22s and 29s in Figure 1). Optimizing this will most likely require touching the rule implementations or Bazel itself to introduce more parallelism. This can also happen when there is an unusual amount of GC. +* Slower than expected analysis phase (`runAnalysisPhase`), especially on + incremental builds. This can be a sign of a poor rule implementation, for + example one that flattens depsets. Package loading can be slow by an + excessive amount of targets, complex macros or recursive globs. +* Individual slow actions, especially those on the critical path. It might be + possible to split large actions into multiple smaller actions or reduce the + set of (transitive) dependencies to speed them up. Also check for an unusual + high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). +* Bottlenecks, that is a small number of threads is busy while all others are + idling / waiting for the result (see around 22s and 29s in Figure 1). + Optimizing this will most likely require touching the rule implementations + or Bazel itself to introduce more parallelism. This can also happen when + there is an unusual amount of GC. ## Profile file format -The top-level object contains metadata (`otherData`) and the actual tracing data (`traceEvents`). The metadata contains extra info, for example the invocation ID and date of the Bazel invocation. +The top-level object contains metadata (`otherData`) and the actual tracing data +(`traceEvents`). The metadata contains extra info, for example the invocation ID +and date of the Bazel invocation. Example: @@ -100,6 +148,12 @@ Example: } ``` -Timestamps (`ts`) and durations (`dur`) in the trace events are given in microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. Note that some events are merged together if they are very short and close to each other; pass [`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) if you would like to prevent event merging. +Timestamps (`ts`) and durations (`dur`) in the trace events are given in +microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. +Note that some events are merged together if they are very short and close to +each other; pass +[`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) +if you would like to prevent event merging. -See also the [Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). +See also the +[Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). diff --git a/advanced/performance/memory.mdx b/advanced/performance/memory.mdx index 8f6c8b5f..844e691b 100644 --- a/advanced/performance/memory.mdx +++ b/advanced/performance/memory.mdx @@ -2,27 +2,50 @@ title: 'Optimize Memory' --- + + This page describes how to limit and reduce the memory Bazel uses. ## Running Bazel with Limited RAM -In certain situations, you may want Bazel to use minimal memory. You can set the maximum heap via the startup flag [`--host_jvm_args`](/docs/user-manual#host-jvm-args), like `--host_jvm_args=-Xmx2g`. +In certain situations, you may want Bazel to use minimal memory. You can set the +maximum heap via the startup flag +[`--host_jvm_args`](/docs/user-manual#host-jvm-args), +like `--host_jvm_args=-Xmx2g`. ### Trade incremental build speeds for memory -If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when it doesn't have enough memory. You can make Bazel use less memory, at the cost of slower incremental builds, by passing the following command flags: [`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), [`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), and [`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). +If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when +it doesn't have enough memory. You can make Bazel use less memory, at the cost +of slower incremental builds, by passing the following command flags: +[`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), +[`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), +and +[`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). -These flags will minimize the memory that Bazel uses in a build, at the cost of making future builds slower than a standard incremental build would be. +These flags will minimize the memory that Bazel uses in a build, at the cost of +making future builds slower than a standard incremental build would be. You can also pass any one of these flags individually: -- `--discard_analysis_cache` will reduce the memory used during execution (not analysis). Incremental builds will not have to redo package loading, but will have to redo analysis and execution (although the on-disk action cache can prevent most re-execution). -- `--notrack_incremental_state` will not store any edges in Bazel's internal dependency graph, so that it is unusable for incremental builds. The next build will discard that data, but it is preserved until then, for internal debugging, unless `--nokeep_state_after_build` is specified. -- `--nokeep_state_after_build` will discard all data after the build, so that incremental builds have to build from scratch (except for the on-disk action cache). Alone, it does not affect the high-water mark of the current build. + * `--discard_analysis_cache` will reduce the memory used during execution (not +analysis). Incremental builds will not have to redo package loading, but will +have to redo analysis and execution (although the on-disk action cache can +prevent most re-execution). + * `--notrack_incremental_state` will not store any edges in Bazel's internal + dependency graph, so that it is unusable for incremental builds. The next build + will discard that data, but it is preserved until then, for internal debugging, + unless `--nokeep_state_after_build` is specified. + * `--nokeep_state_after_build` will discard all data after the build, so that + incremental builds have to build from scratch (except for the on-disk action + cache). Alone, it does not affect the high-water mark of the current build. ### Trade build flexibility for memory with Skyfocus (Experimental) -If you want to make Bazel use less memory *and* retain incremental build speeds, you can tell Bazel the working set of files that you will be modifying, and Bazel will only keep state needed to correctly incrementally rebuild changes to those files. This feature is called **Skyfocus**. +If you want to make Bazel use less memory *and* retain incremental build speeds, +you can tell Bazel the working set of files that you will be modifying, and +Bazel will only keep state needed to correctly incrementally rebuild changes to +those files. This feature is called **Skyfocus**. To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: @@ -30,16 +53,21 @@ To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: bazel build //pkg:target --experimental_enable_skyfocus ``` -By default, the working set will be the set of files next to the target being built. In the example, all files in `//pkg` will be kept in the working set, and changes to files outside of the working set will be disallowed, until you issue `bazel clean` or restart the Bazel server. +By default, the working set will be the set of files next to the target being +built. In the example, all files in `//pkg` will be kept in the working set, and +changes to files outside of the working set will be disallowed, until you issue +`bazel clean` or restart the Bazel server. -If you want to specify an exact set of files or directories, use the `--experimental_working_set` flag, like so: +If you want to specify an exact set of files or directories, use the +`--experimental_working_set` flag, like so: ```sh bazel build //pkg:target --experimental_enable_skyfocus --experimental_working_set=path/to/another/dir,path/to/tests/dir ``` -You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the memory reduction amount: +You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the +memory reduction amount: Putting it altogether, you should see something like this: @@ -51,13 +79,19 @@ INFO: Analyzed 149 targets (4533 packages loaded, 169438 targets configured). INFO: Found 25 targets and 124 test targets... INFO: Updated working set successfully. INFO: Focusing on 334 roots, 3 leafs... (use --experimental_skyfocus_dump_keys to show them) -INFO: Heap: 1237MB -> 676MB (-45.31%) +INFO: Heap: 1237MB -> 676MB (-45.31%) INFO: Elapsed time: 192.670s ... INFO: Build completed successfully, 62303 total actions ``` -For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, and incremental builds to handle changes to files under `dir1`, `dir2`, and `dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot rebuild changed files outside of these directories. +For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, +and incremental builds to handle changes to files under `dir1`, `dir2`, and +`dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot +rebuild changed files outside of these directories. ## Memory Profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. Read more about this process on the [Memory Profiling section](/rules/performance#memory-profiling) of our documentation on how to improve the performance of custom rules. +Bazel comes with a built-in memory profiler that can help you check your rule’s +memory use. Read more about this process on the +[Memory Profiling section](/rules/performance#memory-profiling) of our +documentation on how to improve the performance of custom rules. diff --git a/basics/artifact-based-builds.mdx b/basics/artifact-based-builds.mdx index 5dfc5896..4176a288 100644 --- a/basics/artifact-based-builds.mdx +++ b/basics/artifact-based-builds.mdx @@ -2,19 +2,57 @@ title: 'Artifact-Based Build Systems' --- -This page covers artifact-based build systems and the philosophy behind their creation. Bazel is an artifact-based build system. While task-based build systems are good step above build scripts, they give too much power to individual engineers by letting them define their own tasks. -Artifact-based build systems have a small number of tasks defined by the system that engineers can configure in a limited way. Engineers still tell the system **what** to build, but the build system determines **how** to build it. As with task-based build systems, artifact-based build systems, such as Bazel, still have buildfiles, but the contents of those buildfiles are very different. Rather than being an imperative set of commands in a Turing-complete scripting language describing how to produce an output, buildfiles in Bazel are a declarative manifest describing a set of artifacts to build, their dependencies, and a limited set of options that affect how they’re built. When engineers run `bazel` on the command line, they specify a set of targets to build (the **what**), and Bazel is responsible for configuring, running, and scheduling the compilation steps (the **how**). Because the build system now has full control over what tools to run when, it can make much stronger guarantees that allow it to be far more efficient while still guaranteeing correctness. -## A functional perspective +This page covers artifact-based build systems and the philosophy behind their +creation. Bazel is an artifact-based build system. While task-based build +systems are good step above build scripts, they give too much power to +individual engineers by letting them define their own tasks. + +Artifact-based build systems have a small number of tasks defined by the system +that engineers can configure in a limited way. Engineers still tell the system +**what** to build, but the build system determines **how** to build it. As with +task-based build systems, artifact-based build systems, such as Bazel, still +have buildfiles, but the contents of those buildfiles are very different. Rather +than being an imperative set of commands in a Turing-complete scripting language +describing how to produce an output, buildfiles in Bazel are a declarative +manifest describing a set of artifacts to build, their dependencies, and a +limited set of options that affect how they’re built. When engineers run `bazel` +on the command line, they specify a set of targets to build (the **what**), and +Bazel is responsible for configuring, running, and scheduling the compilation +steps (the **how**). Because the build system now has full control over what +tools to run when, it can make much stronger guarantees that allow it to be far +more efficient while still guaranteeing correctness. -It’s easy to make an analogy between artifact-based build systems and functional programming. Traditional imperative programming languages (such as, Java, C, and Python) specify lists of statements to be executed one after another, in the same way that task-based build systems let programmers define a series of steps to execute. Functional programming languages (such as, Haskell and ML), in contrast, are structured more like a series of mathematical equations. In functional languages, the programmer describes a computation to perform, but leaves the details of when and exactly how that computation is executed to the compiler. +## A functional perspective -This maps to the idea of declaring a manifest in an artifact-based build system and letting the system figure out how to execute the build. Many problems can't be easily expressed using functional programming, but the ones that do benefit greatly from it: the language is often able to trivially parallelize such programs and make strong guarantees about their correctness that would be impossible in an imperative language. The easiest problems to express using functional programming are the ones that simply involve transforming one piece of data into another using a series of rules or functions. And that’s exactly what a build system is: the whole system is effectively a mathematical function that takes source files (and tools like the compiler) as inputs and produces binaries as outputs. So, it’s not surprising that it works well to base a build system around the tenets of functional programming. +It’s easy to make an analogy between artifact-based build systems and functional +programming. Traditional imperative programming languages (such as, Java, C, and +Python) specify lists of statements to be executed one after another, in the +same way that task-based build systems let programmers define a series of steps +to execute. Functional programming languages (such as, Haskell and ML), in +contrast, are structured more like a series of mathematical equations. In +functional languages, the programmer describes a computation to perform, but +leaves the details of when and exactly how that computation is executed to the +compiler. + +This maps to the idea of declaring a manifest in an artifact-based build system +and letting the system figure out how to execute the build. Many problems can't +be easily expressed using functional programming, but the ones that do benefit +greatly from it: the language is often able to trivially parallelize such +programs and make strong guarantees about their correctness that would be +impossible in an imperative language. The easiest problems to express using +functional programming are the ones that simply involve transforming one piece +of data into another using a series of rules or functions. And that’s exactly +what a build system is: the whole system is effectively a mathematical function +that takes source files (and tools like the compiler) as inputs and produces +binaries as outputs. So, it’s not surprising that it works well to base a build +system around the tenets of functional programming. ## Understanding artifact-based build systems -Google's build system, Blaze, was the first artifact-based build system. Bazel is the open-sourced version of Blaze. +Google's build system, Blaze, was the first artifact-based build system. Bazel +is the open-sourced version of Blaze. Here’s what a buildfile (normally named `BUILD`) looks like in Bazel: @@ -37,64 +75,205 @@ java_library( ) ``` -In Bazel, `BUILD` files define targets—the two types of targets here are `java_binary` and `java_library`. Every target corresponds to an artifact that can be created by the system: binary targets produce binaries that can be executed directly, and library targets produce libraries that can be used by binaries or other libraries. Every target has: - -- `name`: how the target is referenced on the command line and by other targets -- `srcs`: the source files to be compiled to create the artifact for the target -- `deps`: other targets that must be built before this target and linked into it - -Dependencies can either be within the same package (such as `MyBinary`’s dependency on `:mylib`) or on a different package in the same source hierarchy (such as `mylib`’s dependency on `//java/com/example/common`). - -As with task-based build systems, you perform builds using Bazel’s command-line tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After entering that command for the first time in a clean repository, Bazel: - -1. Parses every `BUILD` file in the workspace to create a graph of dependencies among artifacts. -2. Uses the graph to determine the transitive dependencies of `MyBinary`; that is, every target that `MyBinary` depends on and every target that those targets depend on, recursively. -3. Builds each of those dependencies, in order. Bazel starts by building each target that has no other dependencies and keeps track of which dependencies still need to be built for each target. As soon as all of a target’s dependencies are built, Bazel starts building that target. This process continues until every one of `MyBinary`’s transitive dependencies have been built. -4. Builds `MyBinary` to produce a final executable binary that links in all of the dependencies that were built in step 3. - -Fundamentally, it might not seem like what’s happening here is that much different than what happened when using a task-based build system. Indeed, the end result is the same binary, and the process for producing it involved analyzing a bunch of steps to find dependencies among them, and then running those steps in order. But there are critical differences. The first one appears in step 3: because Bazel knows that each target only produces a Java library, it knows that all it has to do is run the Java compiler rather than an arbitrary user-defined script, so it knows that it’s safe to run these steps in parallel. This can produce an order of magnitude performance improvement over building targets one at a time on a multicore machine, and is only possible because the artifact-based approach leaves the build system in charge of its own execution strategy so that it can make stronger guarantees about parallelism. - -The benefits extend beyond parallelism, though. The next thing that this approach gives us becomes apparent when the developer types `bazel build :MyBinary` a second time without making any changes: Bazel exits in less than a second with a message saying that the target is up to date. This is possible due to the functional programming paradigm we talked about earlier—Bazel knows that each target is the result only of running a Java compiler, and it knows that the output from the Java compiler depends only on its inputs, so as long as the inputs haven’t changed, the output can be reused. And this analysis works at every level; if `MyBinary.java` changes, Bazel knows to rebuild `MyBinary` but reuse `mylib`. If a source file for `//java/com/example/common` changes, Bazel knows to rebuild that library, `mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. Because Bazel knows about the properties of the tools it runs at every step, it’s able to rebuild only the minimum set of artifacts each time while guaranteeing that it won’t produce stale builds. - -Reframing the build process in terms of artifacts rather than tasks is subtle but powerful. By reducing the flexibility exposed to the programmer, the build system can know more about what is being done at every step of the build. It can use this knowledge to make the build far more efficient by parallelizing build processes and reusing their outputs. But this is really just the first step, and these building blocks of parallelism and reuse form the basis for a distributed and highly scalable build system. +In Bazel, `BUILD` files define targets—the two types of targets here are +`java_binary` and `java_library`. Every target corresponds to an artifact that +can be created by the system: binary targets produce binaries that can be +executed directly, and library targets produce libraries that can be used by +binaries or other libraries. Every target has: + +* `name`: how the target is referenced on the command line and by other + targets +* `srcs`: the source files to be compiled to create the artifact for the target +* `deps`: other targets that must be built before this target and linked into + it + +Dependencies can either be within the same package (such as `MyBinary`’s +dependency on `:mylib`) or on a different package in the same source hierarchy +(such as `mylib`’s dependency on `//java/com/example/common`). + +As with task-based build systems, you perform builds using Bazel’s command-line +tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After +entering that command for the first time in a clean repository, Bazel: + +1. Parses every `BUILD` file in the workspace to create a graph of dependencies + among artifacts. +1. Uses the graph to determine the transitive dependencies of `MyBinary`; that + is, every target that `MyBinary` depends on and every target that those + targets depend on, recursively. +1. Builds each of those dependencies, in order. Bazel starts by building each + target that has no other dependencies and keeps track of which dependencies + still need to be built for each target. As soon as all of a target’s + dependencies are built, Bazel starts building that target. This process + continues until every one of `MyBinary`’s transitive dependencies have been + built. +1. Builds `MyBinary` to produce a final executable binary that links in all of + the dependencies that were built in step 3. + +Fundamentally, it might not seem like what’s happening here is that much +different than what happened when using a task-based build system. Indeed, the +end result is the same binary, and the process for producing it involved +analyzing a bunch of steps to find dependencies among them, and then running +those steps in order. But there are critical differences. The first one appears +in step 3: because Bazel knows that each target only produces a Java library, it +knows that all it has to do is run the Java compiler rather than an arbitrary +user-defined script, so it knows that it’s safe to run these steps in parallel. +This can produce an order of magnitude performance improvement over building +targets one at a time on a multicore machine, and is only possible because the +artifact-based approach leaves the build system in charge of its own execution +strategy so that it can make stronger guarantees about parallelism. + +The benefits extend beyond parallelism, though. The next thing that this +approach gives us becomes apparent when the developer types `bazel +build :MyBinary` a second time without making any changes: Bazel exits in less +than a second with a message saying that the target is up to date. This is +possible due to the functional programming paradigm we talked about +earlier—Bazel knows that each target is the result only of running a Java +compiler, and it knows that the output from the Java compiler depends only on +its inputs, so as long as the inputs haven’t changed, the output can be reused. +And this analysis works at every level; if `MyBinary.java` changes, Bazel knows +to rebuild `MyBinary` but reuse `mylib`. If a source file for +`//java/com/example/common` changes, Bazel knows to rebuild that library, +`mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. +Because Bazel knows about the properties of the tools it runs at every step, +it’s able to rebuild only the minimum set of artifacts each time while +guaranteeing that it won’t produce stale builds. + +Reframing the build process in terms of artifacts rather than tasks is subtle +but powerful. By reducing the flexibility exposed to the programmer, the build +system can know more about what is being done at every step of the build. It can +use this knowledge to make the build far more efficient by parallelizing build +processes and reusing their outputs. But this is really just the first step, and +these building blocks of parallelism and reuse form the basis for a distributed +and highly scalable build system. ## Other nifty Bazel tricks -Artifact-based build systems fundamentally solve the problems with parallelism and reuse that are inherent in task-based build systems. But there are still a few problems that came up earlier that we haven’t addressed. Bazel has clever ways of solving each of these, and we should discuss them before moving on. +Artifact-based build systems fundamentally solve the problems with parallelism +and reuse that are inherent in task-based build systems. But there are still a +few problems that came up earlier that we haven’t addressed. Bazel has clever +ways of solving each of these, and we should discuss them before moving on. ### Tools as dependencies -One problem we ran into earlier was that builds depended on the tools installed on our machine, and reproducing builds across systems could be difficult due to different tool versions or locations. The problem becomes even more difficult when your project uses languages that require different tools based on which platform they’re being built on or compiled for (such as, Windows versus Linux), and each of those platforms requires a slightly different set of tools to do the same job. +One problem we ran into earlier was that builds depended on the tools installed +on our machine, and reproducing builds across systems could be difficult due to +different tool versions or locations. The problem becomes even more difficult +when your project uses languages that require different tools based on which +platform they’re being built on or compiled for (such as, Windows versus Linux), +and each of those platforms requires a slightly different set of tools to do the +same job. -Bazel solves the first part of this problem by treating tools as dependencies to each target. Every `java_library` in the workspace implicitly depends on a Java compiler, which defaults to a well-known compiler. Whenever Bazel builds a `java_library`, it checks to make sure that the specified compiler is available at a known location. Just like any other dependency, if the Java compiler changes, every artifact that depends on it is rebuilt. +Bazel solves the first part of this problem by treating tools as dependencies to +each target. Every `java_library` in the workspace implicitly depends on a Java +compiler, which defaults to a well-known compiler. Whenever Bazel builds a +`java_library`, it checks to make sure that the specified compiler is available +at a known location. Just like any other dependency, if the Java compiler +changes, every artifact that depends on it is rebuilt. -Bazel solves the second part of the problem, platform independence, by setting [build configurations](/run/build#build-config-cross-compilation). Rather than targets depending directly on their tools, they depend on types of configurations: +Bazel solves the second part of the problem, platform independence, by setting +[build configurations](/run/build#build-config-cross-compilation). Rather than +targets depending directly on their tools, they depend on types of configurations: -- **Host configuration**: building tools that run during the build -- **Target configuration**: building the binary you ultimately requested +* **Host configuration**: building tools that run during the build +* **Target configuration**: building the binary you ultimately requested ### Extending the build system -Bazel comes with targets for several popular programming languages out of the box, but engineers will always want to do more—part of the benefit of task-based systems is their flexibility in supporting any kind of build process, and it would be better not to give that up in an artifact-based build system. Fortunately, Bazel allows its supported target types to be extended by [adding custom rules](/extending/rules). - -To define a rule in Bazel, the rule author declares the inputs that the rule requires (in the form of attributes passed in the `BUILD` file) and the fixed set of outputs that the rule produces. The author also defines the actions that will be generated by that rule. Each action declares its inputs and outputs, runs a particular executable or writes a particular string to a file, and can be connected to other actions via its inputs and outputs. This means that actions are the lowest-level composable unit in the build system—an action can do whatever it wants so long as it uses only its declared inputs and outputs, and Bazel takes care of scheduling actions and caching their results as appropriate. - -The system isn’t foolproof given that there’s no way to stop an action developer from doing something like introducing a nondeterministic process as part of their action. But this doesn’t happen very often in practice, and pushing the possibilities for abuse all the way down to the action level greatly decreases opportunities for errors. Rules supporting many common languages and tools are widely available online, and most projects will never need to define their own rules. Even for those that do, rule definitions only need to be defined in one central place in the repository, meaning most engineers will be able to use those rules without ever having to worry about their implementation. +Bazel comes with targets for several popular programming languages out of the +box, but engineers will always want to do more—part of the benefit of task-based +systems is their flexibility in supporting any kind of build process, and it +would be better not to give that up in an artifact-based build system. +Fortunately, Bazel allows its supported target types to be extended by +[adding custom rules](/extending/rules). + +To define a rule in Bazel, the rule author declares the inputs that the rule +requires (in the form of attributes passed in the `BUILD` file) and the fixed +set of outputs that the rule produces. The author also defines the actions that +will be generated by that rule. Each action declares its inputs and outputs, +runs a particular executable or writes a particular string to a file, and can be +connected to other actions via its inputs and outputs. This means that actions +are the lowest-level composable unit in the build system—an action can do +whatever it wants so long as it uses only its declared inputs and outputs, and +Bazel takes care of scheduling actions and caching their results as appropriate. + +The system isn’t foolproof given that there’s no way to stop an action developer +from doing something like introducing a nondeterministic process as part of +their action. But this doesn’t happen very often in practice, and pushing the +possibilities for abuse all the way down to the action level greatly decreases +opportunities for errors. Rules supporting many common languages and tools are +widely available online, and most projects will never need to define their own +rules. Even for those that do, rule definitions only need to be defined in one +central place in the repository, meaning most engineers will be able to use +those rules without ever having to worry about their implementation. ### Isolating the environment -Actions sound like they might run into the same problems as tasks in other systems—isn’t it still possible to write actions that both write to the same file and end up conflicting with one another? Actually, Bazel makes these conflicts impossible by using *[sandboxing](/docs/sandboxing)*. On supported systems, every action is isolated from every other action via a filesystem sandbox. Effectively, each action can see only a restricted view of the filesystem that includes the inputs it has declared and any outputs it has produced. This is enforced by systems such as LXC on Linux, the same technology behind Docker. This means that it’s impossible for actions to conflict with one another because they are unable to read any files they don’t declare, and any files that they write but don’t declare will be thrown away when the action finishes. Bazel also uses sandboxes to restrict actions from communicating via the network. +Actions sound like they might run into the same problems as tasks in other +systems—isn’t it still possible to write actions that both write to the same +file and end up conflicting with one another? Actually, Bazel makes these +conflicts impossible by using _[sandboxing](/docs/sandboxing)_. On supported +systems, every action is isolated from every other action via a filesystem +sandbox. Effectively, each action can see only a restricted view of the +filesystem that includes the inputs it has declared and any outputs it has +produced. This is enforced by systems such as LXC on Linux, the same technology +behind Docker. This means that it’s impossible for actions to conflict with one +another because they are unable to read any files they don’t declare, and any +files that they write but don’t declare will be thrown away when the action +finishes. Bazel also uses sandboxes to restrict actions from communicating via +the network. ### Making external dependencies deterministic -There’s still one problem remaining: build systems often need to download dependencies (whether tools or libraries) from external sources rather than directly building them. This can be seen in the example via the `@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file from Maven. - -Depending on files outside of the current workspace is risky. Those files could change at any time, potentially requiring the build system to constantly check whether they’re fresh. If a remote file changes without a corresponding change in the workspace source code, it can also lead to unreproducible builds—a build might work one day and fail the next for no obvious reason due to an unnoticed dependency change. Finally, an external dependency can introduce a huge security risk when it is owned by a third party: if an attacker is able to infiltrate that third-party server, they can replace the dependency file with something of their own design, potentially giving them full control over your build environment and its output. - -The fundamental problem is that we want the build system to be aware of these files without having to check them into source control. Updating a dependency should be a conscious choice, but that choice should be made once in a central place rather than managed by individual engineers or automatically by the system. This is because even with a “Live at Head” model, we still want builds to be deterministic, which implies that if you check out a commit from last week, you should see your dependencies as they were then rather than as they are now. - -Bazel and some other build systems address this problem by requiring a workspacewide manifest file that lists a *cryptographic hash* for every external dependency in the workspace. The hash is a concise way to uniquely represent the file without checking the entire file into source control. Whenever a new external dependency is referenced from a workspace, that dependency’s hash is added to the manifest, either manually or automatically. When Bazel runs a build, it checks the actual hash of its cached dependency against the expected hash defined in the manifest and redownloads the file only if the hash differs. - -If the artifact we download has a different hash than the one declared in the manifest, the build will fail unless the hash in the manifest is updated. This can be done automatically, but that change must be approved and checked into source control before the build will accept the new dependency. This means that there’s always a record of when a dependency was updated, and an external dependency can’t change without a corresponding change in the workspace source. It also means that, when checking out an older version of the source code, the build is guaranteed to use the same dependencies that it was using at the point when that version was checked in (or else it will fail if those dependencies are no longer available). - -Of course, it can still be a problem if a remote server becomes unavailable or starts serving corrupt data—this can cause all of your builds to begin failing if you don’t have another copy of that dependency available. To avoid this problem, we recommend that, for any nontrivial project, you mirror all of its dependencies onto servers or services that you trust and control. Otherwise you will always be at the mercy of a third party for your build system’s availability, even if the checked-in hashes guarantee its security. +There’s still one problem remaining: build systems often need to download +dependencies (whether tools or libraries) from external sources rather than +directly building them. This can be seen in the example via the +`@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file +from Maven. + +Depending on files outside of the current workspace is risky. Those files could +change at any time, potentially requiring the build system to constantly check +whether they’re fresh. If a remote file changes without a corresponding change +in the workspace source code, it can also lead to unreproducible builds—a build +might work one day and fail the next for no obvious reason due to an unnoticed +dependency change. Finally, an external dependency can introduce a huge security +risk when it is owned by a third party: if an attacker is able to infiltrate +that third-party server, they can replace the dependency file with something of +their own design, potentially giving them full control over your build +environment and its output. + +The fundamental problem is that we want the build system to be aware of these +files without having to check them into source control. Updating a dependency +should be a conscious choice, but that choice should be made once in a central +place rather than managed by individual engineers or automatically by the +system. This is because even with a “Live at Head” model, we still want builds +to be deterministic, which implies that if you check out a commit from last +week, you should see your dependencies as they were then rather than as they are +now. + +Bazel and some other build systems address this problem by requiring a +workspacewide manifest file that lists a _cryptographic hash_ for every external +dependency in the workspace. The hash is a concise way to uniquely represent the +file without checking the entire file into source control. Whenever a new +external dependency is referenced from a workspace, that dependency’s hash is +added to the manifest, either manually or automatically. When Bazel runs a +build, it checks the actual hash of its cached dependency against the expected +hash defined in the manifest and redownloads the file only if the hash differs. + +If the artifact we download has a different hash than the one declared in the +manifest, the build will fail unless the hash in the manifest is updated. This +can be done automatically, but that change must be approved and checked into +source control before the build will accept the new dependency. This means that +there’s always a record of when a dependency was updated, and an external +dependency can’t change without a corresponding change in the workspace source. +It also means that, when checking out an older version of the source code, the +build is guaranteed to use the same dependencies that it was using at the point +when that version was checked in (or else it will fail if those dependencies are +no longer available). + +Of course, it can still be a problem if a remote server becomes unavailable or +starts serving corrupt data—this can cause all of your builds to begin failing +if you don’t have another copy of that dependency available. To avoid this +problem, we recommend that, for any nontrivial project, you mirror all of its +dependencies onto servers or services that you trust and control. Otherwise you +will always be at the mercy of a third party for your build system’s +availability, even if the checked-in hashes guarantee its security. diff --git a/basics/build-systems.mdx b/basics/build-systems.mdx index e2f80cb6..b3c63389 100644 --- a/basics/build-systems.mdx +++ b/basics/build-systems.mdx @@ -2,44 +2,126 @@ title: 'Why a Build System?' --- -This page discusses what build systems are, what they do, why you should use a build system, and why compilers and build scripts aren't the best choice as your organization starts to scale. It's intended for developers who don't have much experience with a build system. + + +This page discusses what build systems are, what they do, why you should use a +build system, and why compilers and build scripts aren't the best choice as your +organization starts to scale. It's intended for developers who don't have much +experience with a build system. ## What is a build system? -Fundamentally, all build systems have a straightforward purpose: they transform the source code written by engineers into executable binaries that can be read by machines. Build systems aren't just for human-authored code; they also allow machines to create builds automatically, whether for testing or for releases to production. In an organization with thousands of engineers, it's common that most builds are triggered automatically rather than directly by engineers. +Fundamentally, all build systems have a straightforward purpose: they transform +the source code written by engineers into executable binaries that can be read +by machines. Build systems aren't just for human-authored code; they also allow +machines to create builds automatically, whether for testing or for releases to +production. In an organization with thousands of engineers, it's common that +most builds are triggered automatically rather than directly by engineers. ### Can't I just use a compiler? -The need for a build system might not be immediately obvious. Most engineers don't use a build system while learning to code: most start by invoking tools like `gcc` or `javac` directly from the command line, or the equivalent in an integrated development environment (IDE). As long as all the source code is in the same directory, a command like this works fine: +The need for a build system might not be immediately obvious. Most engineers +don't use a build system while learning to code: most start by invoking tools +like `gcc` or `javac` directly from the command line, or the equivalent in an +integrated development environment (IDE). As long as all the source code is in +the same directory, a command like this works fine: ```posix-terminal javac *.java ``` -This instructs the Java compiler to take every Java source file in the current directory and turn it into a binary class file. In the simplest case, this is all you need. - -However, as soon as code expands, the complications begin. `javac` is smart enough to look in subdirectories of the current directory to find code to import. But it has no way of finding code stored in *other parts* of the filesystem (perhaps a library shared by several projects). It also only knows how to build Java code. Large systems often involve different pieces written in a variety of programming languages with webs of dependencies among those pieces, meaning no compiler for a single language can possibly build the entire system. - -Once you're dealing with code from multiple languages or multiple compilation units, building code is no longer a one-step process. Now you must evaluate what your code depends on and build those pieces in the proper order, possibly using a different set of tools for each piece. If any dependencies change, you must repeat this process to avoid depending on stale binaries. For a codebase of even moderate size, this process quickly becomes tedious and error-prone. - -The compiler also doesn’t know anything about how to handle external dependencies, such as third-party `JAR` files in Java. Without a build system, you could manage this by downloading the dependency from the internet, sticking it in a `lib` folder on the hard drive, and configuring the compiler to read libraries from that directory. Over time, it's difficult to maintain the updates, versions, and source of these external dependencies. +This instructs the Java compiler to take every Java source file in the current +directory and turn it into a binary class file. In the simplest case, this is +all you need. + +However, as soon as code expands, the complications begin. `javac` is smart +enough to look in subdirectories of the current directory to find code to +import. But it has no way of finding code stored in _other parts_ of the +filesystem (perhaps a library shared by several projects). It also only knows +how to build Java code. Large systems often involve different pieces written in +a variety of programming languages with webs of dependencies among those pieces, +meaning no compiler for a single language can possibly build the entire system. + +Once you're dealing with code from multiple languages or multiple compilation +units, building code is no longer a one-step process. Now you must evaluate what +your code depends on and build those pieces in the proper order, possibly using +a different set of tools for each piece. If any dependencies change, you must +repeat this process to avoid depending on stale binaries. For a codebase of even +moderate size, this process quickly becomes tedious and error-prone. + +The compiler also doesn’t know anything about how to handle external +dependencies, such as third-party `JAR` files in Java. Without a build system, +you could manage this by downloading the dependency from the internet, sticking +it in a `lib` folder on the hard drive, and configuring the compiler to read +libraries from that directory. Over time, it's difficult to maintain the +updates, versions, and source of these external dependencies. ### What about shell scripts? -Suppose that your hobby project starts out simple enough that you can build it using just a compiler, but you begin running into some of the problems described previously. Maybe you still don’t think you need a build system and can automate away the tedious parts using some simple shell scripts that take care of building things in the correct order. This helps out for a while, but pretty soon you start running into even more problems: - -- It becomes tedious. As your system grows more complex, you begin spending almost as much time working on your build scripts as on real code. Debugging shell scripts is painful, with more and more hacks being layered on top of one another. - -- It’s slow. To make sure you weren’t accidentally relying on stale libraries, you have your build script build every dependency in order every time you run it. You think about adding some logic to detect which parts need to be rebuilt, but that sounds awfully complex and error prone for a script. Or you think about specifying which parts need to be rebuilt each time, but then you’re back to square one. - -- Good news: it’s time for a release! Better go figure out all the arguments you need to pass to the jar command to make your final build. And remember how to upload it and push it out to the central repository. And build and push the documentation updates, and send out a notification to users. Hmm, maybe this calls for another script... - -- Disaster! Your hard drive crashes, and now you need to recreate your entire system. You were smart enough to keep all of your source files in version control, but what about those libraries you downloaded? Can you find them all again and make sure they were the same version as when you first downloaded them? Your scripts probably depended on particular tools being installed in particular places—can you restore that same environment so that the scripts work again? What about all those environment variables you set a long time ago to get the compiler working just right and then forgot about? - -- Despite the problems, your project is successful enough that you’re able to begin hiring more engineers. Now you realize that it doesn’t take a disaster for the previous problems to arise—you need to go through the same painful bootstrapping process every time a new developer joins your team. And despite your best efforts, there are still small differences in each person’s system. Frequently, what works on one person’s machine doesn’t work on another’s, and each time it takes a few hours of debugging tool paths or library versions to figure out where the difference is. - -- You decide that you need to automate your build system. In theory, this is as simple as getting a new computer and setting it up to run your build script every night using cron. You still need to go through the painful setup process, but now you don’t have the benefit of a human brain being able to detect and resolve minor problems. Now, every morning when you get in, you see that last night’s build failed because yesterday a developer made a change that worked on their system but didn’t work on the automated build system. Each time it’s a simple fix, but it happens so often that you end up spending a lot of time each day discovering and applying these simple fixes. - -- Builds become slower and slower as the project grows. One day, while waiting for a build to complete, you gaze mournfully at the idle desktop of your coworker, who is on vacation, and wish there were a way to take advantage of all that wasted computational power. - -You’ve run into a classic problem of scale. For a single developer working on at most a couple hundred lines of code for at most a week or two (which might have been the entire experience thus far of a junior developer who just graduated university), a compiler is all you need. Scripts can maybe take you a little bit farther. But as soon as you need to coordinate across multiple developers and their machines, even a perfect build script isn’t enough because it becomes very difficult to account for the minor differences in those machines. At this point, this simple approach breaks down and it’s time to invest in a real build system. +Suppose that your hobby project starts out simple enough that you can build it +using just a compiler, but you begin running into some of the problems described +previously. Maybe you still don’t think you need a build system and can automate +away the tedious parts using some simple shell scripts that take care of +building things in the correct order. This helps out for a while, but pretty +soon you start running into even more problems: + +* It becomes tedious. As your system grows more complex, you begin spending + almost as much time working on your build scripts as on real code. Debugging + shell scripts is painful, with more and more hacks being layered on top of + one another. + +* It’s slow. To make sure you weren’t accidentally relying on stale libraries, + you have your build script build every dependency in order every time you + run it. You think about adding some logic to detect which parts need to be + rebuilt, but that sounds awfully complex and error prone for a script. Or + you think about specifying which parts need to be rebuilt each time, but + then you’re back to square one. + +* Good news: it’s time for a release! Better go figure out all the arguments + you need to pass to the jar command to make your final build. And remember + how to upload it and push it out to the central repository. And build and + push the documentation updates, and send out a notification to users. Hmm, + maybe this calls for another script... + +* Disaster! Your hard drive crashes, and now you need to recreate your entire + system. You were smart enough to keep all of your source files in version + control, but what about those libraries you downloaded? Can you find them + all again and make sure they were the same version as when you first + downloaded them? Your scripts probably depended on particular tools being + installed in particular places—can you restore that same environment so that + the scripts work again? What about all those environment variables you set a + long time ago to get the compiler working just right and then forgot about? + +* Despite the problems, your project is successful enough that you’re able to + begin hiring more engineers. Now you realize that it doesn’t take a disaster + for the previous problems to arise—you need to go through the same painful + bootstrapping process every time a new developer joins your team. And + despite your best efforts, there are still small differences in each + person’s system. Frequently, what works on one person’s machine doesn’t work + on another’s, and each time it takes a few hours of debugging tool paths or + library versions to figure out where the difference is. + +* You decide that you need to automate your build system. In theory, this is + as simple as getting a new computer and setting it up to run your build + script every night using cron. You still need to go through the painful + setup process, but now you don’t have the benefit of a human brain being + able to detect and resolve minor problems. Now, every morning when you get + in, you see that last night’s build failed because yesterday a developer + made a change that worked on their system but didn’t work on the automated + build system. Each time it’s a simple fix, but it happens so often that you + end up spending a lot of time each day discovering and applying these simple + fixes. + +* Builds become slower and slower as the project grows. One day, while waiting + for a build to complete, you gaze mournfully at the idle desktop of your + coworker, who is on vacation, and wish there were a way to take advantage of + all that wasted computational power. + +You’ve run into a classic problem of scale. For a single developer working on at +most a couple hundred lines of code for at most a week or two (which might have +been the entire experience thus far of a junior developer who just graduated +university), a compiler is all you need. Scripts can maybe take you a little bit +farther. But as soon as you need to coordinate across multiple developers and +their machines, even a perfect build script isn’t enough because it becomes very +difficult to account for the minor differences in those machines. At this point, +this simple approach breaks down and it’s time to invest in a real build system. diff --git a/basics/dependencies.mdx b/basics/dependencies.mdx index 308b627e..1d3bf8f1 100644 --- a/basics/dependencies.mdx +++ b/basics/dependencies.mdx @@ -2,84 +2,300 @@ title: 'Dependency Management' --- -In looking through the previous pages, one theme repeats over and over: managing your own code is fairly straightforward, but managing its dependencies is much more difficult. There are all sorts of dependencies: sometimes there’s a dependency on a task (such as “push the documentation before I mark a release as complete”), and sometimes there’s a dependency on an artifact (such as “I need to have the latest version of the computer vision library to build my code”). Sometimes, you have internal dependencies on another part of your codebase, and sometimes you have external dependencies on code or data owned by another team (either in your organization or a third party). But in any case, the idea of “I need that before I can have this” is something that recurs repeatedly in the design of build systems, and managing dependencies is perhaps the most fundamental job of a build system. -## Dealing with Modules and Dependencies - -Projects that use artifact-based build systems like Bazel are broken into a set of modules, with modules expressing dependencies on one another via `BUILD` files. Proper organization of these modules and dependencies can have a huge effect on both the performance of the build system and how much work it takes to maintain. -## Using Fine-Grained Modules and the 1:1:1 Rule +In looking through the previous pages, one theme repeats over and over: managing +your own code is fairly straightforward, but managing its dependencies is much +more difficult. There are all sorts of dependencies: sometimes there’s a +dependency on a task (such as “push the documentation before I mark a release as +complete”), and sometimes there’s a dependency on an artifact (such as “I need +to have the latest version of the computer vision library to build my code”). +Sometimes, you have internal dependencies on another part of your codebase, and +sometimes you have external dependencies on code or data owned by another team +(either in your organization or a third party). But in any case, the idea of “I +need that before I can have this” is something that recurs repeatedly in the +design of build systems, and managing dependencies is perhaps the most +fundamental job of a build system. -The first question that comes up when structuring an artifact-based build is deciding how much functionality an individual module should encompass. In Bazel, a *module* is represented by a target specifying a buildable unit like a `java_library` or a `go_binary`. At one extreme, the entire project could be contained in a single module by putting one `BUILD` file at the root and recursively globbing together all of that project’s source files. At the other extreme, nearly every source file could be made into its own module, effectively requiring each file to list in a `BUILD` file every other file it depends on. - -Most projects fall somewhere between these extremes, and the choice involves a trade-off between performance and maintainability. Using a single module for the entire project might mean that you never need to touch the `BUILD` file except when adding an external dependency, but it means that the build system must always build the entire project all at once. This means that it won’t be able to parallelize or distribute parts of the build, nor will it be able to cache parts that it’s already built. One-module-per-file is the opposite: the build system has the maximum flexibility in caching and scheduling steps of the build, but engineers need to expend more effort maintaining lists of dependencies whenever they change which files reference which. +## Dealing with Modules and Dependencies -Though the exact granularity varies by language (and often even within language), Google tends to favor significantly smaller modules than one might typically write in a task-based build system. A typical production binary at Google often depends on tens of thousands of targets, and even a moderate-sized team can own several hundred targets within its codebase. For languages like Java that have a strong built-in notion of packaging, each directory usually contains a single package, target, and `BUILD` file (Pants, another build system based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging conventions frequently define multiple targets per `BUILD` file. +Projects that use artifact-based build systems like Bazel are broken into a set +of modules, with modules expressing dependencies on one another via `BUILD` +files. Proper organization of these modules and dependencies can have a huge +effect on both the performance of the build system and how much work it takes to +maintain. -The benefits of smaller build targets really begin to show at scale because they lead to faster distributed builds and a less frequent need to rebuild targets. The advantages become even more compelling after testing enters the picture, as finer-grained targets mean that the build system can be much smarter about running only a limited subset of tests that could be affected by any given change. Because Google believes in the systemic benefits of using smaller targets, we’ve made some strides in mitigating the downside by investing in tooling to automatically manage `BUILD` files to avoid burdening developers. +## Using Fine-Grained Modules and the 1:1:1 Rule -Some of these tools, such as `buildifier` and `buildozer`, are available with Bazel in the [`buildtools` directory](https://github.com/bazelbuild/buildtools). +The first question that comes up when structuring an artifact-based build is +deciding how much functionality an individual module should encompass. In Bazel, +a _module_ is represented by a target specifying a buildable unit like a +`java_library` or a `go_binary`. At one extreme, the entire project could be +contained in a single module by putting one `BUILD` file at the root and +recursively globbing together all of that project’s source files. At the other +extreme, nearly every source file could be made into its own module, effectively +requiring each file to list in a `BUILD` file every other file it depends on. + +Most projects fall somewhere between these extremes, and the choice involves a +trade-off between performance and maintainability. Using a single module for the +entire project might mean that you never need to touch the `BUILD` file except +when adding an external dependency, but it means that the build system must +always build the entire project all at once. This means that it won’t be able to +parallelize or distribute parts of the build, nor will it be able to cache parts +that it’s already built. One-module-per-file is the opposite: the build system +has the maximum flexibility in caching and scheduling steps of the build, but +engineers need to expend more effort maintaining lists of dependencies whenever +they change which files reference which. + +Though the exact granularity varies by language (and often even within +language), Google tends to favor significantly smaller modules than one might +typically write in a task-based build system. A typical production binary at +Google often depends on tens of thousands of targets, and even a moderate-sized +team can own several hundred targets within its codebase. For languages like +Java that have a strong built-in notion of packaging, each directory usually +contains a single package, target, and `BUILD` file (Pants, another build system +based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging +conventions frequently define multiple targets per `BUILD` file. + +The benefits of smaller build targets really begin to show at scale because they +lead to faster distributed builds and a less frequent need to rebuild targets. +The advantages become even more compelling after testing enters the picture, as +finer-grained targets mean that the build system can be much smarter about +running only a limited subset of tests that could be affected by any given +change. Because Google believes in the systemic benefits of using smaller +targets, we’ve made some strides in mitigating the downside by investing in +tooling to automatically manage `BUILD` files to avoid burdening developers. + +Some of these tools, such as `buildifier` and `buildozer`, are available with +Bazel in the [`buildtools` +directory](https://github.com/bazelbuild/buildtools). ## Minimizing Module Visibility -Bazel and other build systems allow each target to specify a visibility — a property that determines which other targets may depend on it. A private target can only be referenced within its own `BUILD` file. A target may grant broader visibility to the targets of an explicitly defined list of `BUILD` files, or, in the case of public visibility, to every target in the workspace. - -As with most programming languages, it is usually best to minimize visibility as much as possible. Generally, teams at Google will make targets public only if those targets represent widely used libraries available to any team at Google. Teams that require others to coordinate with them before using their code will maintain an allowlist of customer targets as their target’s visibility. Each team’s internal implementation targets will be restricted to only directories owned by the team, and most `BUILD` files will have only one target that isn’t private. +Bazel and other build systems allow each target to specify a visibility — a +property that determines which other targets may depend on it. A private target +can only be referenced within its own `BUILD` file. A target may grant broader +visibility to the targets of an explicitly defined list of `BUILD` files, or, in +the case of public visibility, to every target in the workspace. + +As with most programming languages, it is usually best to minimize visibility as +much as possible. Generally, teams at Google will make targets public only if +those targets represent widely used libraries available to any team at Google. +Teams that require others to coordinate with them before using their code will +maintain an allowlist of customer targets as their target’s visibility. Each +team’s internal implementation targets will be restricted to only directories +owned by the team, and most `BUILD` files will have only one target that isn’t +private. ## Managing Dependencies -Modules need to be able to refer to one another. The downside of breaking a codebase into fine-grained modules is that you need to manage the dependencies among those modules (though tools can help automate this). Expressing these dependencies usually ends up being the bulk of the content in a `BUILD` file. +Modules need to be able to refer to one another. The downside of breaking a +codebase into fine-grained modules is that you need to manage the dependencies +among those modules (though tools can help automate this). Expressing these +dependencies usually ends up being the bulk of the content in a `BUILD` file. ### Internal dependencies -In a large project broken into fine-grained modules, most dependencies are likely to be internal; that is, on another target defined and built in the same source repository. Internal dependencies differ from external dependencies in that they are built from source rather than downloaded as a prebuilt artifact while running the build. This also means that there’s no notion of “version” for internal dependencies—a target and all of its internal dependencies are always built at the same commit/revision in the repository. One issue that should be handled carefully with regard to internal dependencies is how to treat transitive dependencies (Figure 1). Suppose target A depends on target B, which depends on a common library target C. Should target A be able to use classes defined in target C? - -[![Transitive dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) +In a large project broken into fine-grained modules, most dependencies are +likely to be internal; that is, on another target defined and built in the same +source repository. Internal dependencies differ from external dependencies in +that they are built from source rather than downloaded as a prebuilt artifact +while running the build. This also means that there’s no notion of “version” for +internal dependencies—a target and all of its internal dependencies are always +built at the same commit/revision in the repository. One issue that should be +handled carefully with regard to internal dependencies is how to treat +transitive dependencies (Figure 1). Suppose target A depends on target B, which +depends on a common library target C. Should target A be able to use classes +defined in target C? + +[![Transitive +dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) **Figure 1**. Transitive dependencies -As far as the underlying tools are concerned, there’s no problem with this; both B and C will be linked into target A when it is built, so any symbols defined in C are known to A. Bazel allowed this for many years, but as Google grew, we began to see problems. Suppose that B was refactored such that it no longer needed to depend on C. If B’s dependency on C was then removed, A and any other target that used C via a dependency on B would break. Effectively, a target’s dependencies became part of its public contract and could never be safely changed. This meant that dependencies accumulated over time and builds at Google started to slow down. - -Google eventually solved this issue by introducing a “strict transitive dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to reference a symbol without depending on it directly and, if so, fails with an error and a shell command that can be used to automatically insert the dependency. Rolling this change out across Google’s entire codebase and refactoring every one of our millions of build targets to explicitly list their dependencies was a multiyear effort, but it was well worth it. Our builds are now much faster given that targets have fewer unnecessary dependencies, and engineers are empowered to remove dependencies they don’t need without worrying about breaking targets that depend on them. - -As usual, enforcing strict transitive dependencies involved a trade-off. It made build files more verbose, as frequently used libraries now need to be listed explicitly in many places rather than pulled in incidentally, and engineers needed to spend more effort adding dependencies to `BUILD` files. We’ve since developed tools that reduce this toil by automatically detecting many missing dependencies and adding them to a `BUILD` files without any developer intervention. But even without such tools, we’ve found the trade-off to be well worth it as the codebase scales: explicitly adding a dependency to `BUILD` file is a one-time cost, but dealing with implicit transitive dependencies can cause ongoing problems as long as the build target exists. Bazel [enforces strict transitive dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) on Java code by default. +As far as the underlying tools are concerned, there’s no problem with this; both +B and C will be linked into target A when it is built, so any symbols defined in +C are known to A. Bazel allowed this for many years, but as Google grew, we +began to see problems. Suppose that B was refactored such that it no longer +needed to depend on C. If B’s dependency on C was then removed, A and any other +target that used C via a dependency on B would break. Effectively, a target’s +dependencies became part of its public contract and could never be safely +changed. This meant that dependencies accumulated over time and builds at Google +started to slow down. + +Google eventually solved this issue by introducing a “strict transitive +dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to +reference a symbol without depending on it directly and, if so, fails with an +error and a shell command that can be used to automatically insert the +dependency. Rolling this change out across Google’s entire codebase and +refactoring every one of our millions of build targets to explicitly list their +dependencies was a multiyear effort, but it was well worth it. Our builds are +now much faster given that targets have fewer unnecessary dependencies, and +engineers are empowered to remove dependencies they don’t need without worrying +about breaking targets that depend on them. + +As usual, enforcing strict transitive dependencies involved a trade-off. It made +build files more verbose, as frequently used libraries now need to be listed +explicitly in many places rather than pulled in incidentally, and engineers +needed to spend more effort adding dependencies to `BUILD` files. We’ve since +developed tools that reduce this toil by automatically detecting many missing +dependencies and adding them to a `BUILD` files without any developer +intervention. But even without such tools, we’ve found the trade-off to be well +worth it as the codebase scales: explicitly adding a dependency to `BUILD` file +is a one-time cost, but dealing with implicit transitive dependencies can cause +ongoing problems as long as the build target exists. Bazel [enforces strict +transitive +dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) +on Java code by default. ### External dependencies -If a dependency isn’t internal, it must be external. External dependencies are those on artifacts that are built and stored outside of the build system. The dependency is imported directly from an artifact repository (typically accessed over the internet) and used as-is rather than being built from source. One of the biggest differences between external and internal dependencies is that external dependencies have versions, and those versions exist independently of the project’s source code. +If a dependency isn’t internal, it must be external. External dependencies are +those on artifacts that are built and stored outside of the build system. The +dependency is imported directly from an artifact repository (typically accessed +over the internet) and used as-is rather than being built from source. One of +the biggest differences between external and internal dependencies is that +external dependencies have versions, and those versions exist independently of +the project’s source code. ### Automatic versus manual dependency management -Build systems can allow the versions of external dependencies to be managed either manually or automatically. When managed manually, the buildfile explicitly lists the version it wants to download from the artifact repository, often using a [semantic version string](https://semver.org/) such as `1.1.4`. When managed automatically, the source file specifies a range of acceptable versions, and the build system always downloads the latest one. For example, Gradle allows a dependency version to be declared as “1.+” to specify that any minor or patch version of a dependency is acceptable so long as the major version is 1. - -Automatically managed dependencies can be convenient for small projects, but they’re usually a recipe for disaster on projects of nontrivial size or that are being worked on by more than one engineer. The problem with automatically managed dependencies is that you have no control over when the version is updated. There’s no way to guarantee that external parties won’t make breaking updates (even when they claim to use semantic versioning), so a build that worked one day might be broken the next with no easy way to detect what changed or to roll it back to a working state. Even if the build doesn’t break, there can be subtle behavior or performance changes that are impossible to track down. - -In contrast, because manually managed dependencies require a change in source control, they can be easily discovered and rolled back, and it’s possible to check out an older version of the repository to build with older dependencies. Bazel requires that versions of all dependencies be specified manually. At even moderate scales, the overhead of manual version management is well worth it for the stability it provides. +Build systems can allow the versions of external dependencies to be managed +either manually or automatically. When managed manually, the buildfile +explicitly lists the version it wants to download from the artifact repository, +often using a [semantic version string](https://semver.org/) such +as `1.1.4`. When managed automatically, the source file specifies a range of +acceptable versions, and the build system always downloads the latest one. For +example, Gradle allows a dependency version to be declared as “1.+” to specify +that any minor or patch version of a dependency is acceptable so long as the +major version is 1. + +Automatically managed dependencies can be convenient for small projects, but +they’re usually a recipe for disaster on projects of nontrivial size or that are +being worked on by more than one engineer. The problem with automatically +managed dependencies is that you have no control over when the version is +updated. There’s no way to guarantee that external parties won’t make breaking +updates (even when they claim to use semantic versioning), so a build that +worked one day might be broken the next with no easy way to detect what changed +or to roll it back to a working state. Even if the build doesn’t break, there +can be subtle behavior or performance changes that are impossible to track down. + +In contrast, because manually managed dependencies require a change in source +control, they can be easily discovered and rolled back, and it’s possible to +check out an older version of the repository to build with older dependencies. +Bazel requires that versions of all dependencies be specified manually. At even +moderate scales, the overhead of manual version management is well worth it for +the stability it provides. ### The One-Version Rule -Different versions of a library are usually represented by different artifacts, so in theory there’s no reason that different versions of the same external dependency couldn’t both be declared in the build system under different names. That way, each target could choose which version of the dependency it wanted to use. This causes a lot of problems in practice, so Google enforces a strict [One-Version Rule](https://opensource.google/docs/thirdparty/oneversion/) for all third-party dependencies in our codebase. - -The biggest problem with allowing multiple versions is the diamond dependency issue. Suppose that target A depends on target B and on v1 of an external library. If target B is later refactored to add a dependency on v2 of the same external library, target A will break because it now depends implicitly on two different versions of the same library. Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, because any of that target’s users could already be depending on a different version. Following the One-Version Rule makes this conflict impossible—if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist. +Different versions of a library are usually represented by different artifacts, +so in theory there’s no reason that different versions of the same external +dependency couldn’t both be declared in the build system under different names. +That way, each target could choose which version of the dependency it wanted to +use. This causes a lot of problems in practice, so Google enforces a strict +[One-Version +Rule](https://opensource.google/docs/thirdparty/oneversion/) for +all third-party dependencies in our codebase. + +The biggest problem with allowing multiple versions is the diamond dependency +issue. Suppose that target A depends on target B and on v1 of an external +library. If target B is later refactored to add a dependency on v2 of the same +external library, target A will break because it now depends implicitly on two +different versions of the same library. Effectively, it’s never safe to add a +new dependency from a target to any third-party library with multiple versions, +because any of that target’s users could already be depending on a different +version. Following the One-Version Rule makes this conflict impossible—if a +target adds a dependency on a third-party library, any existing dependencies +will already be on that same version, so they can happily coexist. ### Transitive external dependencies -Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central, allow artifacts to specify dependencies on particular versions of other artifacts in the repository. Build tools like Maven or Gradle often recursively download each transitive dependency by default, meaning that adding a single dependency in your project could potentially cause dozens of artifacts to be downloaded in total. - -This is very convenient: when adding a dependency on a new library, it would be a big pain to have to track down each of that library’s transitive dependencies and add them all manually. But there’s also a huge downside: because different libraries can depend on different versions of the same third-party library, this strategy necessarily violates the One-Version Rule and leads to the diamond dependency problem. If your target depends on two external libraries that use different versions of the same dependency, there’s no telling which one you’ll get. This also means that updating an external dependency could cause seemingly unrelated failures throughout the codebase if the new version begins pulling in conflicting versions of some of its dependencies. - -Bazel did not use to automatically download transitive dependencies. It used to employ a `WORKSPACE` file that required all transitive dependencies to be listed, which led to a lot of pain when managing external dependencies. Bazel has since added support for automatic transitive external dependency management in the form of the `MODULE.bazel` file. See [external dependency overview](/external/overview) for more details. - -Yet again, the choice here is one between convenience and scalability. Small projects might prefer not having to worry about managing transitive dependencies themselves and might be able to get away with using automatic transitive dependencies. This strategy becomes less and less appealing as the organization and codebase grows, and conflicts and unexpected results become more and more frequent. At larger scales, the cost of manually managing dependencies is much less than the cost of dealing with issues caused by automatic dependency management. +Dealing with the transitive dependencies of an external dependency can be +particularly difficult. Many artifact repositories such as Maven Central, allow +artifacts to specify dependencies on particular versions of other artifacts in +the repository. Build tools like Maven or Gradle often recursively download each +transitive dependency by default, meaning that adding a single dependency in +your project could potentially cause dozens of artifacts to be downloaded in +total. + +This is very convenient: when adding a dependency on a new library, it would be +a big pain to have to track down each of that library’s transitive dependencies +and add them all manually. But there’s also a huge downside: because different +libraries can depend on different versions of the same third-party library, this +strategy necessarily violates the One-Version Rule and leads to the diamond +dependency problem. If your target depends on two external libraries that use +different versions of the same dependency, there’s no telling which one you’ll +get. This also means that updating an external dependency could cause seemingly +unrelated failures throughout the codebase if the new version begins pulling in +conflicting versions of some of its dependencies. + +Bazel did not use to automatically download transitive dependencies. It used to +employ a `WORKSPACE` file that required all transitive dependencies to be +listed, which led to a lot of pain when managing external dependencies. Bazel +has since added support for automatic transitive external dependency management +in the form of the `MODULE.bazel` file. See [external dependency +overview](/external/overview) for more details. + +Yet again, the choice here is one between convenience and scalability. Small +projects might prefer not having to worry about managing transitive dependencies +themselves and might be able to get away with using automatic transitive +dependencies. This strategy becomes less and less appealing as the organization +and codebase grows, and conflicts and unexpected results become more and more +frequent. At larger scales, the cost of manually managing dependencies is much +less than the cost of dealing with issues caused by automatic dependency +management. ### Caching build results using external dependencies -External dependencies are most often provided by third parties that release stable versions of libraries, perhaps without providing source code. Some organizations might also choose to make some of their own code available as artifacts, allowing other pieces of code to depend on them as third-party rather than internal dependencies. This can theoretically speed up builds if artifacts are slow to build but quick to download. - -However, this also introduces a lot of overhead and complexity: someone needs to be responsible for building each of those artifacts and uploading them to the artifact repository, and clients need to ensure that they stay up to date with the latest version. Debugging also becomes much more difficult because different parts of the system will have been built from different points in the repository, and there is no longer a consistent view of the source tree. - -A better way to solve the problem of artifacts taking a long time to build is to use a build system that supports remote caching, as described earlier. Such a build system saves the resulting artifacts from every build to a location that is shared across engineers, so if a developer depends on an artifact that was recently built by someone else, the build system automatically downloads it instead of building it. This provides all of the performance benefits of depending directly on artifacts while still ensuring that builds are as consistent as if they were always built from the same source. This is the strategy used internally by Google, and Bazel can be configured to use a remote cache. +External dependencies are most often provided by third parties that release +stable versions of libraries, perhaps without providing source code. Some +organizations might also choose to make some of their own code available as +artifacts, allowing other pieces of code to depend on them as third-party rather +than internal dependencies. This can theoretically speed up builds if artifacts +are slow to build but quick to download. + +However, this also introduces a lot of overhead and complexity: someone needs to +be responsible for building each of those artifacts and uploading them to the +artifact repository, and clients need to ensure that they stay up to date with +the latest version. Debugging also becomes much more difficult because different +parts of the system will have been built from different points in the +repository, and there is no longer a consistent view of the source tree. + +A better way to solve the problem of artifacts taking a long time to build is to +use a build system that supports remote caching, as described earlier. Such a +build system saves the resulting artifacts from every build to a location that +is shared across engineers, so if a developer depends on an artifact that was +recently built by someone else, the build system automatically downloads it +instead of building it. This provides all of the performance benefits of +depending directly on artifacts while still ensuring that builds are as +consistent as if they were always built from the same source. This is the +strategy used internally by Google, and Bazel can be configured to use a remote +cache. ### Security and reliability of external dependencies -Depending on artifacts from third-party sources is inherently risky. There’s an availability risk if the third-party source (such as an artifact repository) goes down, because your entire build might grind to a halt if it’s unable to download an external dependency. There’s also a security risk: if the third-party system is compromised by an attacker, the attacker could replace the referenced artifact with one of their own design, allowing them to inject arbitrary code into your build. Both problems can be mitigated by mirroring any artifacts you depend on onto servers you control and blocking your build system from accessing third-party artifact repositories like Maven Central. The trade-off is that these mirrors take effort and resources to maintain, so the choice of whether to use them often depends on the scale of the project. The security issue can also be completely prevented with little overhead by requiring the hash of each third-party artifact to be specified in the source repository, causing the build to fail if the artifact is tampered with. Another alternative that completely sidesteps the issue is to vendor your project’s dependencies. When a project vendors its dependencies, it checks them into source control alongside the project’s source code, either as source or as binaries. This effectively means that all of the project’s external dependencies are converted to internal dependencies. Google uses this approach internally, checking every third-party library referenced throughout Google into a `third_party` directory at the root of Google’s source tree. However, this works at Google only because Google’s source control system is custom built to handle an extremely large monorepo, so vendoring might not be an option for all organizations. +Depending on artifacts from third-party sources is inherently risky. There’s an +availability risk if the third-party source (such as an artifact repository) +goes down, because your entire build might grind to a halt if it’s unable to +download an external dependency. There’s also a security risk: if the +third-party system is compromised by an attacker, the attacker could replace the +referenced artifact with one of their own design, allowing them to inject +arbitrary code into your build. Both problems can be mitigated by mirroring any +artifacts you depend on onto servers you control and blocking your build system +from accessing third-party artifact repositories like Maven Central. The +trade-off is that these mirrors take effort and resources to maintain, so the +choice of whether to use them often depends on the scale of the project. The +security issue can also be completely prevented with little overhead by +requiring the hash of each third-party artifact to be specified in the source +repository, causing the build to fail if the artifact is tampered with. Another +alternative that completely sidesteps the issue is to vendor your project’s +dependencies. When a project vendors its dependencies, it checks them into +source control alongside the project’s source code, either as source or as +binaries. This effectively means that all of the project’s external dependencies +are converted to internal dependencies. Google uses this approach internally, +checking every third-party library referenced throughout Google into a +`third_party` directory at the root of Google’s source tree. However, this works +at Google only because Google’s source control system is custom built to handle +an extremely large monorepo, so vendoring might not be an option for all +organizations. diff --git a/basics/distributed-builds.mdx b/basics/distributed-builds.mdx index b11e7c9b..c32f44ff 100644 --- a/basics/distributed-builds.mdx +++ b/basics/distributed-builds.mdx @@ -2,46 +2,136 @@ title: 'Distributed Builds' --- -When you have a large codebase, chains of dependencies can become very deep. Even simple binaries can often depend on tens of thousands of build targets. At this scale, it’s simply impossible to complete a build in a reasonable amount of time on a single machine: no build system can get around the fundamental laws of physics imposed on a machine’s hardware. The only way to make this work is with a build system that supports distributed builds wherein the units of work being done by the system are spread across an arbitrary and scalable number of machines. Assuming we’ve broken the system’s work into small enough units (more on this later), this would allow us to complete any build of any size as quickly as we’re willing to pay for. This scalability is the holy grail we’ve been working toward by defining an artifact-based build system. + + +When you have a large codebase, chains of dependencies can become very deep. +Even simple binaries can often depend on tens of thousands of build targets. At +this scale, it’s simply impossible to complete a build in a reasonable amount +of time on a single machine: no build system can get around the fundamental +laws of physics imposed on a machine’s hardware. The only way to make this work +is with a build system that supports distributed builds wherein the units of +work being done by the system are spread across an arbitrary and scalable +number of machines. Assuming we’ve broken the system’s work into small enough +units (more on this later), this would allow us to complete any build of any +size as quickly as we’re willing to pay for. This scalability is the holy grail +we’ve been working toward by defining an artifact-based build system. ## Remote caching -The simplest type of distributed build is one that only leverages *remote caching*, which is shown in Figure 1. +The simplest type of distributed build is one that only leverages _remote +caching_, which is shown in Figure 1. [![Distributed build with remote caching](/images/distributed-build-remote-cache.png)](/images/distributed-build-remote-cache.png) **Figure 1**. A distributed build showing remote caching -Every system that performs builds, including both developer workstations and continuous integration systems, shares a reference to a common remote cache service. This service might be a fast and local short-term storage system like Redis or a cloud service like Google Cloud Storage. Whenever a user needs to build an artifact, whether directly or as a dependency, the system first checks with the remote cache to see if that artifact already exists there. If so, it can download the artifact instead of building it. If not, the system builds the artifact itself and uploads the result back to the cache. This means that low-level dependencies that don’t change very often can be built once and shared across users rather than having to be rebuilt by each user. At Google, many artifacts are served from a cache rather than built from scratch, vastly reducing the cost of running our build system. - -For a remote caching system to work, the build system must guarantee that builds are completely reproducible. That is, for any build target, it must be possible to determine the set of inputs to that target such that the same set of inputs will produce exactly the same output on any machine. This is the only way to ensure that the results of downloading an artifact are the same as the results of building it oneself. Note that this requires that each artifact in the cache be keyed on both its target and a hash of its inputs—that way, different engineers could make different modifications to the same target at the same time, and the remote cache would store all of the resulting artifacts and serve them appropriately without conflict. - -Of course, for there to be any benefit from a remote cache, downloading an artifact needs to be faster than building it. This is not always the case, especially if the cache server is far from the machine doing the build. Google’s network and build system is carefully tuned to be able to quickly share build results. +Every system that performs builds, including both developer workstations and +continuous integration systems, shares a reference to a common remote cache +service. This service might be a fast and local short-term storage system like +Redis or a cloud service like Google Cloud Storage. Whenever a user needs to +build an artifact, whether directly or as a dependency, the system first checks +with the remote cache to see if that artifact already exists there. If so, it +can download the artifact instead of building it. If not, the system builds the +artifact itself and uploads the result back to the cache. This means that +low-level dependencies that don’t change very often can be built once and shared +across users rather than having to be rebuilt by each user. At Google, many +artifacts are served from a cache rather than built from scratch, vastly +reducing the cost of running our build system. + +For a remote caching system to work, the build system must guarantee that builds +are completely reproducible. That is, for any build target, it must be possible +to determine the set of inputs to that target such that the same set of inputs +will produce exactly the same output on any machine. This is the only way to +ensure that the results of downloading an artifact are the same as the results +of building it oneself. Note that this requires that each artifact in the cache +be keyed on both its target and a hash of its inputs—that way, different +engineers could make different modifications to the same target at the same +time, and the remote cache would store all of the resulting artifacts and serve +them appropriately without conflict. + +Of course, for there to be any benefit from a remote cache, downloading an +artifact needs to be faster than building it. This is not always the case, +especially if the cache server is far from the machine doing the build. Google’s +network and build system is carefully tuned to be able to quickly share build +results. ## Remote execution -Remote caching isn’t a true distributed build. If the cache is lost or if you make a low-level change that requires everything to be rebuilt, you still need to perform the entire build locally on your machine. The true goal is to support remote execution, in which the actual work of doing the build can be spread across any number of workers. Figure 2 depicts a remote execution system. +Remote caching isn’t a true distributed build. If the cache is lost or if you +make a low-level change that requires everything to be rebuilt, you still need +to perform the entire build locally on your machine. The true goal is to support +remote execution, in which the actual work of doing the build can be spread +across any number of workers. Figure 2 depicts a remote execution system. [![Remote execution system](/images/remote-execution-system.png)](/images/remote-execution-system.png) **Figure 2**. A remote execution system -The build tool running on each user’s machine (where users are either human engineers or automated build systems) sends requests to a central build master. The build master breaks the requests into their component actions and schedules the execution of those actions over a scalable pool of workers. Each worker performs the actions asked of it with the inputs specified by the user and writes out the resulting artifacts. These artifacts are shared across the other machines executing actions that require them until the final output can be produced and sent to the user. - -The trickiest part of implementing such a system is managing the communication between the workers, the master, and the user’s local machine. Workers might depend on intermediate artifacts produced by other workers, and the final output needs to be sent back to the user’s local machine. To do this, we can build on top of the distributed cache described previously by having each worker write its results to and read its dependencies from the cache. The master blocks workers from proceeding until everything they depend on has finished, in which case they’ll be able to read their inputs from the cache. The final product is also cached, allowing the local machine to download it. Note that we also need a separate means of exporting the local changes in the user’s source tree so that workers can apply those changes before building. - -For this to work, all of the parts of the artifact-based build systems described earlier need to come together. Build environments must be completely self-describing so that we can spin up workers without human intervention. Build processes themselves must be completely self-contained because each step might be executed on a different machine. Outputs must be completely deterministic so that each worker can trust the results it receives from other workers. Such guarantees are extremely difficult for a task-based system to provide, which makes it nigh-impossible to build a reliable remote execution system on top of one. +The build tool running on each user’s machine (where users are either human +engineers or automated build systems) sends requests to a central build master. +The build master breaks the requests into their component actions and schedules +the execution of those actions over a scalable pool of workers. Each worker +performs the actions asked of it with the inputs specified by the user and +writes out the resulting artifacts. These artifacts are shared across the other +machines executing actions that require them until the final output can be +produced and sent to the user. + +The trickiest part of implementing such a system is managing the communication +between the workers, the master, and the user’s local machine. Workers might +depend on intermediate artifacts produced by other workers, and the final output +needs to be sent back to the user’s local machine. To do this, we can build on +top of the distributed cache described previously by having each worker write +its results to and read its dependencies from the cache. The master blocks +workers from proceeding until everything they depend on has finished, in which +case they’ll be able to read their inputs from the cache. The final product is +also cached, allowing the local machine to download it. Note that we also need a +separate means of exporting the local changes in the user’s source tree so that +workers can apply those changes before building. + +For this to work, all of the parts of the artifact-based build systems described +earlier need to come together. Build environments must be completely +self-describing so that we can spin up workers without human intervention. Build +processes themselves must be completely self-contained because each step might +be executed on a different machine. Outputs must be completely deterministic so +that each worker can trust the results it receives from other workers. Such +guarantees are extremely difficult for a task-based system to provide, which +makes it nigh-impossible to build a reliable remote execution system on top of +one. ## Distributed builds at Google -Since 2008, Google has been using a distributed build system that employs both remote caching and remote execution, which is illustrated in Figure 3. +Since 2008, Google has been using a distributed build system that employs both +remote caching and remote execution, which is illustrated in Figure 3. [![High-level build system](/images/high-level-build-system.png)](/images/high-level-build-system.png) **Figure 3**. Google’s distributed build system -Google’s remote cache is called ObjFS. It consists of a backend that stores build outputs in Bigtables distributed throughout our fleet of production machines and a frontend FUSE daemon named objfsd that runs on each developer’s machine. The FUSE daemon allows engineers to browse build outputs as if they were normal files stored on the workstation, but with the file content downloaded on-demand only for the few files that are directly requested by the user. Serving file contents on-demand greatly reduces both network and disk usage, and the system is able to build twice as fast compared to when we stored all build output on the developer’s local disk. - -Google’s remote execution system is called Forge. A Forge client in Blaze (Bazel's internal equivalent) called the Distributor sends requests for each action to a job running in our datacenters called the Scheduler. The Scheduler maintains a cache of action results, allowing it to return a response immediately if the action has already been created by any other user of the system. If not, it places the action into a queue. A large pool of Executor jobs continually read actions from this queue, execute them, and store the results directly in the ObjFS Bigtables. These results are available to the executors for future actions, or to be downloaded by the end user via objfsd. - -The end result is a system that scales to efficiently support all builds performed at Google. And the scale of Google’s builds is truly massive: Google runs millions of builds executing millions of test cases and producing petabytes of build outputs from billions of lines of source code every day. Not only does such a system let our engineers build complex codebases quickly, it also allows us to implement a huge number of automated tools and systems that rely on our build. +Google’s remote cache is called ObjFS. It consists of a backend that stores +build outputs in Bigtables distributed throughout our fleet of production +machines and a frontend FUSE daemon named objfsd that runs on each developer’s +machine. The FUSE daemon allows engineers to browse build outputs as if they +were normal files stored on the workstation, but with the file content +downloaded on-demand only for the few files that are directly requested by the +user. Serving file contents on-demand greatly reduces both network and disk +usage, and the system is able to build twice as fast compared to when we stored +all build output on the developer’s local disk. + +Google’s remote execution system is called Forge. A Forge client in Blaze +(Bazel's internal equivalent) called +the Distributor sends requests for each action to a job running in our +datacenters called the Scheduler. The Scheduler maintains a cache of action +results, allowing it to return a response immediately if the action has already +been created by any other user of the system. If not, it places the action into +a queue. A large pool of Executor jobs continually read actions from this queue, +execute them, and store the results directly in the ObjFS Bigtables. These +results are available to the executors for future actions, or to be downloaded +by the end user via objfsd. + +The end result is a system that scales to efficiently support all builds +performed at Google. And the scale of Google’s builds is truly massive: Google +runs millions of builds executing millions of test cases and producing petabytes +of build outputs from billions of lines of source code every day. Not only does +such a system let our engineers build complex codebases quickly, it also allows +us to implement a huge number of automated tools and systems that rely on our +build. diff --git a/basics/hermeticity.mdx b/basics/hermeticity.mdx index a9765af8..dcee3a9e 100644 --- a/basics/hermeticity.mdx +++ b/basics/hermeticity.mdx @@ -2,59 +2,108 @@ title: 'Hermeticity' --- -This page covers hermeticity, the benefits of using hermetic builds, and strategies for identifying non-hermetic behavior in your builds. + + +This page covers hermeticity, the benefits of using hermetic builds, and +strategies for identifying non-hermetic behavior in your builds. ## Overview -When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system. +When given the same input source code and product configuration, a hermetic +build system always returns the same output by isolating the build from changes +to the host system. -In order to isolate the build, hermetic builds are insensitive to libraries and other software installed on the local or remote host machine. They depend on specific versions of build tools, such as compilers, and dependencies, such as libraries. This makes the build process self-contained as it doesn't rely on services external to the build environment. +In order to isolate the build, hermetic builds are insensitive to libraries and +other software installed on the local or remote host machine. They depend on +specific versions of build tools, such as compilers, and dependencies, such as +libraries. This makes the build process self-contained as it doesn't rely on +services external to the build environment. The two important aspects of hermeticity are: -- **Isolation**: Hermetic build systems treat tools as source code. They download copies of tools and manage their storage and use inside managed file trees. This creates isolation between the host machine and local user, including installed versions of languages. -- **Source identity**: Hermetic build systems try to ensure the sameness of inputs. Code repositories, such as Git, identify sets of code mutations with a unique hash code. Hermetic build systems use this hash to identify changes to the build's input. +* **Isolation**: Hermetic build systems treat tools as source code. They + download copies of tools and manage their storage and use inside managed file + trees. This creates isolation between the host machine and local user, + including installed versions of languages. +* **Source identity**: Hermetic build systems try to ensure the sameness of + inputs. Code repositories, such as Git, identify sets of code mutations with a + unique hash code. Hermetic build systems use this hash to identify changes to + the build's input. ## Benefits The major benefits of hermetic builds are: -- **Speed**: The output of an action can be cached, and the action need not be run again unless inputs change. -- **Parallel execution**: For given input and output, the build system can construct a graph of all actions to calculate efficient and parallel execution. The build system loads the rules and calculates an action graph and hash inputs to look up in the cache. -- **Multiple builds**: You can build multiple hermetic builds on the same machine, each build using different tools and versions. -- **Reproducibility**: Hermetic builds are good for troubleshooting because you know the exact conditions that produced the build. +* **Speed**: The output of an action can be cached, and the action need not be + run again unless inputs change. +* **Parallel execution**: For given input and output, the build system can + construct a graph of all actions to calculate efficient and parallel + execution. The build system loads the rules and calculates an action graph + and hash inputs to look up in the cache. +* **Multiple builds**: You can build multiple hermetic builds on the same + machine, each build using different tools and versions. +* **Reproducibility**: Hermetic builds are good for troubleshooting because you + know the exact conditions that produced the build. ## Identifying non-hermeticity -If you are preparing to switch to Bazel, migration is easier if you improve your existing builds' hermeticity in advance. Some common sources of non-hermeticity in builds are: +If you are preparing to switch to Bazel, migration is easier if you improve +your existing builds' hermeticity in advance. Some common sources of +non-hermeticity in builds are: -- Arbitrary processing in `.mk` files -- Actions or tooling that create files non-deterministically, usually involving build IDs or timestamps -- System binaries that differ across hosts (such as `/usr/bin` binaries, absolute paths, system C++ compilers for native C++ rules autoconfiguration) -- Writing to the source tree during the build. This prevents the same source tree from being used for another target. The first build writes to the source tree, fixing the source tree for target A. Then trying to build target B may fail. +* Arbitrary processing in `.mk` files +* Actions or tooling that create files non-deterministically, usually involving + build IDs or timestamps +* System binaries that differ across hosts (such as `/usr/bin` binaries, absolute + paths, system C++ compilers for native C++ rules autoconfiguration) +* Writing to the source tree during the build. This prevents the same source + tree from being used for another target. The first build writes to the source + tree, fixing the source tree for target A. Then trying to build target B may + fail. ## Troubleshooting non-hermetic builds -Starting with local execution, issues that affect local cache hits reveal non-hermetic actions. - -- Ensure null sequential builds: If you run `make` and get a successful build, running the build again should not rebuild any targets. If you run each build step twice or on different systems, compare a hash of the file contents and get results that differ, the build is not reproducible. -- Run steps to [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) from a variety of potential client machines to ensure that you catch any cases of client environment leaking into the actions. -- Execute a build within a docker container that contains nothing but the checked-out source tree and explicit list of host tools. Build breakages and error messages will catch implicit system dependencies. -- Discover and fix hermeticity problems using [remote execution rules](/remote/rules#overview). -- Enable strict [sandboxing](/docs/sandboxing) at the per-action level, since actions in a build can be stateful and affect the build or the output. -- [Workspace rules](/remote/workspace) allow developers to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. You can get a log of some potentially non-hermetic actions in Bazel workspace rules by adding the flag `--experimental_workspace_rules_log_file=<var>PATH</var>` to your Bazel command. - -Note: Make your build fully hermetic when mixing remote and local execution, using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote Docker container will enable the build to execute the same in both environments. +Starting with local execution, issues that affect local cache hits reveal +non-hermetic actions. + +* Ensure null sequential builds: If you run `make` and get a successful build, + running the build again should not rebuild any targets. If you run each build + step twice or on different systems, compare a hash of the file contents and + get results that differ, the build is not reproducible. +* Run steps to + [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) + from a variety of potential client machines to ensure that you catch any + cases of client environment leaking into the actions. +* Execute a build within a docker container that contains nothing but the + checked-out source tree and explicit list of host tools. Build breakages and + error messages will catch implicit system dependencies. +* Discover and fix hermeticity problems using + [remote execution rules](/remote/rules#overview). +* Enable strict [sandboxing](/docs/sandboxing) + at the per-action level, since actions in a build can be stateful and affect + the build or the output. +* [Workspace rules](/remote/workspace) + allow developers to add dependencies to external workspaces, but they are + rich enough to allow arbitrary processing to happen in the process. You can + get a log of some potentially non-hermetic actions in Bazel workspace rules by + adding the flag + `--experimental_workspace_rules_log_file=PATH` to + your Bazel command. + +Note: Make your build fully hermetic when mixing remote and local execution, +using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote +Docker container will enable the build to execute the same in both environments. ## Hermeticity with Bazel -For more information about how other projects have had success using hermetic builds with Bazel, see these BazelCon talks: - -- [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) -- [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) -- [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) -- [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) -- [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) -- [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) -- [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=4\&t=0s) (BMW) -- [Building Self Driving Cars with Bazel + Q\&A](https://www.youtube.com/watch?v=fjfFe98LTm8\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=29) (GM Cruise) +For more information about how other projects have had success using hermetic +builds with Bazel, see these BazelCon talks: + +* [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) +* [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) +* [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) +* [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) +* [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) +* [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) +* [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=4&t=0s) (BMW) +* [Building Self Driving Cars with Bazel + Q&A](https://www.youtube.com/watch?v=fjfFe98LTm8&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=29) (GM Cruise) diff --git a/basics/index.mdx b/basics/index.mdx index ecd09227..f3c833fa 100644 --- a/basics/index.mdx +++ b/basics/index.mdx @@ -2,28 +2,56 @@ title: 'Build Basics' --- -A build system is one of the most important parts of an engineering organization because each developer interacts with it potentially dozens or hundreds of times per day. A fully featured build system is necessary to enable developer productivity as an organization scales. For individual developers, it's straightforward to just compile your code and so a build system might seem excessive. But at a larger scale, having a build system helps with managing shared dependencies, such as relying on another part of the code base, or an external resource, such as a library. Build systems help to make sure that you have everything you need to build your code before it starts building. Build systems also increase velocity when they're set up to help engineers share resources and results. -This section covers some history and basics of building and build systems, including design decisions that went into making Bazel. If you're familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you can skip this section, but it's a helpful overview to understand why artifact-based build systems are excellent at enabling scale. -Note: Much of this section's content comes from the *Build Systems and Build Philosophy* chapter of the [*Software Engineering at Google* book](https://abseil.io/resources/swe-book/html/ch18.html). Thank you to the original author, Erik Kuefler, for allowing its reuse and modification here! +A build system is one of the most important parts of an engineering organization +because each developer interacts with it potentially dozens or hundreds of times +per day. A fully featured build system is necessary to enable developer +productivity as an organization scales. For individual developers, it's +straightforward to just compile your code and so a build system might seem +excessive. But at a larger scale, having a build system helps with managing +shared dependencies, such as relying on another part of the code base, or an +external resource, such as a library. Build systems help to make sure that you +have everything you need to build your code before it starts building. Build +systems also increase velocity when they're set up to help engineers share +resources and results. -- **[Why a Build System?](/basics/build-systems)** +This section covers some history and basics of building and build systems, +including design decisions that went into making Bazel. If you're +familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you +can skip this section, but it's a helpful overview to understand why +artifact-based build systems are excellent at enabling scale. - If you haven't used a build system before, start here. This page covers why you should use a build system, and why compilers and build scripts aren't the best choice once your organization starts to scale beyond a few developers. +Note: Much of this section's content comes from the _Build Systems and +Build Philosophy_ chapter of the +[_Software Engineering at Google_ book](https://abseil.io/resources/swe-book/html/ch18.html). +Thank you to the original author, Erik Kuefler, for allowing its reuse and +modification here! -- **[Task-Based Build Systems](/basics/task-based-builds)** +* **[Why a Build System?](/basics/build-systems)** - This page discusses task-based build systems (such as Make, Maven, and Gradle) and some of their challenges. + If you haven't used a build system before, start here. This page covers why + you should use a build system, and why compilers and build scripts aren't + the best choice once your organization starts to scale beyond a few + developers. -- **[Artifact-Based Build Systems](/basics/artifact-based-builds)** +* **[Task-Based Build Systems](/basics/task-based-builds)** - This page discusses artifact-based build systems in response to the pain points of task-based build systems. + This page discusses task-based build systems (such as Make, Maven, and + Gradle) and some of their challenges. -- **[Distributed Builds](/basics/distributed-builds)** +* **[Artifact-Based Build Systems](/basics/artifact-based-builds)** - This page covers distributed builds, or builds that are executed outside of your local machine. This requires more robust infrastructure to share resources and build results (and is where the true wizardry happens!) + This page discusses artifact-based build systems in response to the pain + points of task-based build systems. -- **[Dependency Management](/basics/dependencies)** +* **[Distributed Builds](/basics/distributed-builds)** - This page covers some complications of dependencies at a large scale and strategies to counteract those complications. + This page covers distributed builds, or builds that are executed outside of + your local machine. This requires more robust infrastructure to share + resources and build results (and is where the true wizardry happens!) + +* **[Dependency Management](/basics/dependencies)** + + This page covers some complications of dependencies at a large scale and + strategies to counteract those complications. diff --git a/basics/task-based-builds.mdx b/basics/task-based-builds.mdx index daef39b3..9dd3f8c0 100644 --- a/basics/task-based-builds.mdx +++ b/basics/task-based-builds.mdx @@ -2,68 +2,94 @@ title: 'Task-Based Build Systems' --- -This page covers task-based build systems, how they work and some of the complications that can occur with task-based systems. After shell scripts, task-based build systems are the next logical evolution of building. + + +This page covers task-based build systems, how they work and some of the +complications that can occur with task-based systems. After shell scripts, +task-based build systems are the next logical evolution of building. + ## Understanding task-based build systems -In a task-based build system, the fundamental unit of work is the task. Each task is a script that can execute any sort of logic, and tasks specify other tasks as dependencies that must run before them. Most major build systems in use today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of shell scripts, most modern build systems require engineers to create build files that describe how to perform the build. +In a task-based build system, the fundamental unit of work is the task. Each +task is a script that can execute any sort of logic, and tasks specify other +tasks as dependencies that must run before them. Most major build systems in use +today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of +shell scripts, most modern build systems require engineers to create build files +that describe how to perform the build. -Take this example from the [Ant manual](https://ant.apache.org/manual/using.html): +Take this example from the +[Ant manual](https://ant.apache.org/manual/using.html): ```xml -<project name="MyProject" default="dist" basedir="."> - <description> + + simple example build file - </description> - - <property name="src" location="src"/> - <property name="build" location="build"/> - <property name="dist" location="dist"/> - - <target name="init"> - - <tstamp/> - - <mkdir dir="${build}"/> - </target> - <target name="compile" depends="init" - description="compile the source"> - - <javac srcdir="${src}" destdir="${build}"/> - </target> - <target name="dist" depends="compile" - description="generate the distribution"> - - <mkdir dir="${dist}/lib"/> - - <jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar" basedir="${build}"/> - </target> - <target name="clean" - description="clean up"> - - <delete dir="${build}"/> - <delete dir="${dist}"/> - </target> -</project> + + + + + + + + + + + + + + + + + + + + + + + + + + + + ``` -The buildfile is written in XML and defines some simple metadata about the build along with a list of tasks (the `<target>` tags in the XML). (Ant uses the word *target* to represent a *task*, and it uses the word *task* to refer to *commands*.) Each task executes a list of possible commands defined by Ant, which here include creating and deleting directories, running `javac`, and creating a JAR file. This set of commands can be extended by user-provided plug-ins to cover any sort of logic. Each task can also define the tasks it depends on via the depends attribute. These dependencies form an acyclic graph, as seen in Figure 1. +The buildfile is written in XML and defines some simple metadata about the build +along with a list of tasks (the `` tags in the XML). (Ant uses the word +_target_ to represent a _task_, and it uses the word _task_ to refer to +_commands_.) Each task executes a list of possible commands defined by Ant, +which here include creating and deleting directories, running `javac`, and +creating a JAR file. This set of commands can be extended by user-provided +plug-ins to cover any sort of logic. Each task can also define the tasks it +depends on via the depends attribute. These dependencies form an acyclic graph, +as seen in Figure 1. [![Acrylic graph showing dependencies](/images/task-dependencies.png)](/images/task-dependencies.png) Figure 1. An acyclic graph showing dependencies -Users perform builds by providing tasks to Ant’s command-line tool. For example, when a user types `ant dist`, Ant takes the following steps: - -1. Loads a file named `build.xml` in the current directory and parses it to create the graph structure shown in Figure 1. -2. Looks for the task named `dist` that was provided on the command line and discovers that it has a dependency on the task named `compile`. -3. Looks for the task named `compile` and discovers that it has a dependency on the task named `init`. -4. Looks for the task named `init` and discovers that it has no dependencies. -5. Executes the commands defined in the `init` task. -6. Executes the commands defined in the `compile` task given that all of that task’s dependencies have been run. -7. Executes the commands defined in the `dist` task given that all of that task’s dependencies have been run. - -In the end, the code executed by Ant when running the `dist` task is equivalent to the following shell script: +Users perform builds by providing tasks to Ant’s command-line tool. For example, +when a user types `ant dist`, Ant takes the following steps: + +1. Loads a file named `build.xml` in the current directory and parses it to + create the graph structure shown in Figure 1. +1. Looks for the task named `dist` that was provided on the command line and + discovers that it has a dependency on the task named `compile`. +1. Looks for the task named `compile` and discovers that it has a dependency on + the task named `init`. +1. Looks for the task named `init` and discovers that it has no dependencies. +1. Executes the commands defined in the `init` task. +1. Executes the commands defined in the `compile` task given that all of that + task’s dependencies have been run. +1. Executes the commands defined in the `dist` task given that all of that + task’s dependencies have been run. + +In the end, the code executed by Ant when running the `dist` task is equivalent +to the following shell script: ```posix-terminal ./createTimestamp.sh @@ -77,32 +103,114 @@ mkdir -p dist/lib/ jar cf dist/lib/MyProject-$(date --iso-8601).jar build/* ``` -When the syntax is stripped away, the buildfile and the build script actually aren’t too different. But we’ve already gained a lot by doing this. We can create new buildfiles in other directories and link them together. We can easily add new tasks that depend on existing tasks in arbitrary and complex ways. We need only pass the name of a single task to the `ant` command-line tool, and it determines everything that needs to be run. - -Ant is an old piece of software, originally released in 2000. Other tools like Maven and Gradle have improved on Ant in the intervening years and essentially replaced it by adding features like automatic management of external dependencies and a cleaner syntax without any XML. But the nature of these newer systems remains the same: they allow engineers to write build scripts in a principled and modular way as tasks and provide tools for executing those tasks and managing dependencies among them. +When the syntax is stripped away, the buildfile and the build script actually +aren’t too different. But we’ve already gained a lot by doing this. We can +create new buildfiles in other directories and link them together. We can easily +add new tasks that depend on existing tasks in arbitrary and complex ways. We +need only pass the name of a single task to the `ant` command-line tool, and it +determines everything that needs to be run. + +Ant is an old piece of software, originally released in 2000. Other tools like +Maven and Gradle have improved on Ant in the intervening years and essentially +replaced it by adding features like automatic management of external +dependencies and a cleaner syntax without any XML. But the nature of these newer +systems remains the same: they allow engineers to write build scripts in a +principled and modular way as tasks and provide tools for executing those tasks +and managing dependencies among them. ## The dark side of task-based build systems -Because these tools essentially let engineers define any script as a task, they are extremely powerful, allowing you to do pretty much anything you can imagine with them. But that power comes with drawbacks, and task-based build systems can become difficult to work with as their build scripts grow more complex. The problem with such systems is that they actually end up giving *too much power to engineers and not enough power to the system*. Because the system has no idea what the scripts are doing, performance suffers, as it must be very conservative in how it schedules and executes build steps. And there’s no way for the system to confirm that each script is doing what it should, so scripts tend to grow in complexity and end up being another thing that needs debugging. +Because these tools essentially let engineers define any script as a task, they +are extremely powerful, allowing you to do pretty much anything you can imagine +with them. But that power comes with drawbacks, and task-based build systems can +become difficult to work with as their build scripts grow more complex. The +problem with such systems is that they actually end up giving _too much power to +engineers and not enough power to the system_. Because the system has no idea +what the scripts are doing, performance suffers, as it must be very conservative +in how it schedules and executes build steps. And there’s no way for the system +to confirm that each script is doing what it should, so scripts tend to grow in +complexity and end up being another thing that needs debugging. ### Difficulty of parallelizing build steps -Modern development workstations are quite powerful, with multiple cores that are capable of executing several build steps in parallel. But task-based systems are often unable to parallelize task execution even when it seems like they should be able to. Suppose that task A depends on tasks B and C. Because tasks B and C have no dependency on each other, is it safe to run them at the same time so that the system can more quickly get to task A? Maybe, if they don’t touch any of the same resources. But maybe not—perhaps both use the same file to track their statuses and running them at the same time causes a conflict. There’s no way in general for the system to know, so either it has to risk these conflicts (leading to rare but very difficult-to-debug build problems), or it has to restrict the entire build to running on a single thread in a single process. This can be a huge waste of a powerful developer machine, and it completely rules out the possibility of distributing the build across multiple machines. +Modern development workstations are quite powerful, with multiple cores that are +capable of executing several build steps in parallel. But task-based systems are +often unable to parallelize task execution even when it seems like they should +be able to. Suppose that task A depends on tasks B and C. Because tasks B and C +have no dependency on each other, is it safe to run them at the same time so +that the system can more quickly get to task A? Maybe, if they don’t touch any +of the same resources. But maybe not—perhaps both use the same file to track +their statuses and running them at the same time causes a conflict. There’s no +way in general for the system to know, so either it has to risk these conflicts +(leading to rare but very difficult-to-debug build problems), or it has to +restrict the entire build to running on a single thread in a single process. +This can be a huge waste of a powerful developer machine, and it completely +rules out the possibility of distributing the build across multiple machines. ### Difficulty performing incremental builds -A good build system allows engineers to perform reliable incremental builds such that a small change doesn’t require the entire codebase to be rebuilt from scratch. This is especially important if the build system is slow and unable to parallelize build steps for the aforementioned reasons. But unfortunately, task-based build systems struggle here, too. Because tasks can do anything, there’s no way in general to check whether they’ve already been done. Many tasks simply take a set of source files and run a compiler to create a set of binaries; thus, they don’t need to be rerun if the underlying source files haven’t changed. But without additional information, the system can’t say this for sure—maybe the task downloads a file that could have changed, or maybe it writes a timestamp that could be different on each run. To guarantee correctness, the system typically must rerun every task during each build. Some build systems try to enable incremental builds by letting engineers specify the conditions under which a task needs to be rerun. Sometimes this is feasible, but often it’s a much trickier problem than it appears. For example, in languages like C++ that allow files to be included directly by other files, it’s impossible to determine the entire set of files that must be watched for changes without parsing the input sources. Engineers often end up taking shortcuts, and these shortcuts can lead to rare and frustrating problems where a task result is reused even when it shouldn’t be. When this happens frequently, engineers get into the habit of running clean before every build to get a fresh state, completely defeating the purpose of having an incremental build in the first place. Figuring out when a task needs to be rerun is surprisingly subtle, and is a job better handled by machines than humans. +A good build system allows engineers to perform reliable incremental builds such +that a small change doesn’t require the entire codebase to be rebuilt from +scratch. This is especially important if the build system is slow and unable to +parallelize build steps for the aforementioned reasons. But unfortunately, +task-based build systems struggle here, too. Because tasks can do anything, +there’s no way in general to check whether they’ve already been done. Many tasks +simply take a set of source files and run a compiler to create a set of +binaries; thus, they don’t need to be rerun if the underlying source files +haven’t changed. But without additional information, the system can’t say this +for sure—maybe the task downloads a file that could have changed, or maybe it +writes a timestamp that could be different on each run. To guarantee +correctness, the system typically must rerun every task during each build. Some +build systems try to enable incremental builds by letting engineers specify the +conditions under which a task needs to be rerun. Sometimes this is feasible, but +often it’s a much trickier problem than it appears. For example, in languages +like C++ that allow files to be included directly by other files, it’s +impossible to determine the entire set of files that must be watched for changes +without parsing the input sources. Engineers often end up taking shortcuts, and +these shortcuts can lead to rare and frustrating problems where a task result is +reused even when it shouldn’t be. When this happens frequently, engineers get +into the habit of running clean before every build to get a fresh state, +completely defeating the purpose of having an incremental build in the first +place. Figuring out when a task needs to be rerun is surprisingly subtle, and is +a job better handled by machines than humans. ### Difficulty maintaining and debugging scripts -Finally, the build scripts imposed by task-based build systems are often just difficult to work with. Though they often receive less scrutiny, build scripts are code just like the system being built, and are easy places for bugs to hide. Here are some examples of bugs that are very common when working with a task-based build system: - -- Task A depends on task B to produce a particular file as output. The owner of task B doesn’t realize that other tasks rely on it, so they change it to produce output in a different location. This can’t be detected until someone tries to run task A and finds that it fails. -- Task A depends on task B, which depends on task C, which is producing a particular file as output that’s needed by task A. The owner of task B decides that it doesn’t need to depend on task C any more, which causes task A to fail even though task B doesn’t care about task C at all! -- The developer of a new task accidentally makes an assumption about the machine running the task, such as the location of a tool or the value of particular environment variables. The task works on their machine, but fails whenever another developer tries it. -- A task contains a nondeterministic component, such as downloading a file from the internet or adding a timestamp to a build. Now, people get potentially different results each time they run the build, meaning that engineers won’t always be able to reproduce and fix one another’s failures or failures that occur on an automated build system. -- Tasks with multiple dependencies can create race conditions. If task A depends on both task B and task C, and task B and C both modify the same file, task A gets a different result depending on which one of tasks B and C finishes first. - -There’s no general-purpose way to solve these performance, correctness, or maintainability problems within the task-based framework laid out here. So long as engineers can write arbitrary code that runs during the build, the system can’t have enough information to always be able to run builds quickly and correctly. To solve the problem, we need to take some power out of the hands of engineers and put it back in the hands of the system and reconceptualize the role of the system not as running tasks, but as producing artifacts. - -This approach led to the creation of artifact-based build systems, like Blaze and Bazel. +Finally, the build scripts imposed by task-based build systems are often just +difficult to work with. Though they often receive less scrutiny, build scripts +are code just like the system being built, and are easy places for bugs to hide. +Here are some examples of bugs that are very common when working with a +task-based build system: + +* Task A depends on task B to produce a particular file as output. The owner + of task B doesn’t realize that other tasks rely on it, so they change it to + produce output in a different location. This can’t be detected until someone + tries to run task A and finds that it fails. +* Task A depends on task B, which depends on task C, which is producing a + particular file as output that’s needed by task A. The owner of task B + decides that it doesn’t need to depend on task C any more, which causes task + A to fail even though task B doesn’t care about task C at all! +* The developer of a new task accidentally makes an assumption about the + machine running the task, such as the location of a tool or the value of + particular environment variables. The task works on their machine, but fails + whenever another developer tries it. +* A task contains a nondeterministic component, such as downloading a file + from the internet or adding a timestamp to a build. Now, people get + potentially different results each time they run the build, meaning that + engineers won’t always be able to reproduce and fix one another’s failures + or failures that occur on an automated build system. +* Tasks with multiple dependencies can create race conditions. If task A + depends on both task B and task C, and task B and C both modify the same + file, task A gets a different result depending on which one of tasks B and C + finishes first. + +There’s no general-purpose way to solve these performance, correctness, or +maintainability problems within the task-based framework laid out here. So long +as engineers can write arbitrary code that runs during the build, the system +can’t have enough information to always be able to run builds quickly and +correctly. To solve the problem, we need to take some power out of the hands of +engineers and put it back in the hands of the system and reconceptualize the +role of the system not as running tasks, but as producing artifacts. + +This approach led to the creation of artifact-based build systems, like Blaze +and Bazel. diff --git a/brand/index.mdx b/brand/index.mdx index f6b1bd31..2a21cd43 100644 --- a/brand/index.mdx +++ b/brand/index.mdx @@ -2,57 +2,87 @@ title: 'Bazel Brand Guidelines' --- -The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and are treated separately from the copyright or patent license grants contained in the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel Trademarks other than those permitted in these guidelines must be approved in advance. + + +The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and +are treated separately from the copyright or patent license grants contained in +the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel +Trademarks other than those permitted in these guidelines must be approved in +advance. ## Purpose of the Brand Guidelines -These guidelines exist to ensure that the Bazel project can share its technology under open source licenses while making sure that the "Bazel" brand is protected as a meaningful source identifier in a way that's consistent with trademark law. By adhering to these guidelines, you help to promote the freedom to use and develop high-quality Bazel technology. +These guidelines exist to ensure that the Bazel project can share its technology +under open source licenses while making sure that the "Bazel" brand is protected +as a meaningful source identifier in a way that's consistent with trademark law. +By adhering to these guidelines, you help to promote the freedom to use and +develop high-quality Bazel technology. ## Acceptable Uses -Given the open nature of Bazel, you may use the Bazel trademark to refer to the project without prior written permission. Examples of these approved references include the following: +Given the open nature of Bazel, you may use the Bazel trademark to refer to the +project without prior written permission. Examples of these approved references +include the following: -- To refer to the Bazel Project itself; -- To link to bazel.build; -- To refer to unmodified source code or other files shared by the Bazel repositories on GitHub; -- In blog posts, news articles, or educational materials about Bazel; -- To accurately identify that your design or implementation is based on, is for use with, or is compatible with Bazel technology. +* To refer to the Bazel Project itself; +* To link to bazel.build; +* To refer to unmodified source code or other files shared by the Bazel + repositories on GitHub; +* In blog posts, news articles, or educational materials about Bazel; +* To accurately identify that your design or implementation is based on, is + for use with, or is compatible with Bazel technology. Examples: -- \[Your Product] for Bazel -- \[Your Product] is compatible with Bazel -- \[XYZ] Conference for Bazel Users +* \[Your Product\] for Bazel +* \[Your Product\] is compatible with Bazel +* \[XYZ\] Conference for Bazel Users ## General Guidelines -- The Bazel name may never be used or registered in a manner that would cause confusion as to Google's sponsorship, affiliation, or endorsement. -- Don't use the Bazel name as part of your company name, product name, domain name, or social media profile. -- Other than as permitted by these guidelines, the Bazel name should not be combined with other trademarks, terms, or source identifiers. -- Don't remove, distort or alter any element of the Bazel Trademarks. That includes modifying the Bazel Trademark, for example, through hyphenation, combination or abbreviation. Do not shorten, abbreviate, or create acronyms out of the Bazel Trademarks. -- Don't display the word Bazel using any different stylization, color, or font from the surrounding text. -- Don't use the term Bazel as a verb or use it in possessive form. -- Don't use the Bazel logo on any website, product UI, or promotional materials without prior written permission from [product@bazel.build](mailto:product@bazel.build). +* The Bazel name may never be used or registered in a manner that would cause + confusion as to Google's sponsorship, affiliation, or endorsement. +* Don't use the Bazel name as part of your company name, product name, domain + name, or social media profile. +* Other than as permitted by these guidelines, the Bazel name should not be + combined with other trademarks, terms, or source identifiers. +* Don't remove, distort or alter any element of the Bazel Trademarks. That + includes modifying the Bazel Trademark, for example, through hyphenation, + combination or abbreviation. Do not shorten, abbreviate, or create acronyms + out of the Bazel Trademarks. +* Don't display the word Bazel using any different stylization, color, or font + from the surrounding text. +* Don't use the term Bazel as a verb or use it in possessive form. +* Don't use the Bazel logo on any website, product UI, or promotional + materials without prior written permission from + [product@bazel.build](mailto:product@bazel.build). ## Usage for Events and Community Groups -The Bazel word mark may be used referentially in events, community groups, or other gatherings related to the Bazel build system, but it may not be used in a manner that implies official status or endorsement. +The Bazel word mark may be used referentially in events, community groups, or +other gatherings related to the Bazel build system, but it may not be used in a +manner that implies official status or endorsement. Examples of appropriate naming conventions are: -- \[XYZ] Bazel User Group -- Bazel Community Day at \[XYZ] -- \[XYZ] Conference for Bazel Users +* \[XYZ\] Bazel User Group +* Bazel Community Day at \[XYZ\] +* \[XYZ\] Conference for Bazel Users -where \[XYZ] represents the location and optionally other wordings. +where \[XYZ\] represents the location and optionally other wordings. -Any naming convention that may imply official status or endorsement requires review for approval from [product@bazel.build](mailto:product@bazel.build). +Any naming convention that may imply official status or endorsement requires +review for approval from [product@bazel.build](mailto:product@bazel.build). Examples of naming conventions that require prior written permission: -- BazelCon -- Bazel Conference +* BazelCon +* Bazel Conference ## Contact Us -Please do not hesitate to contact us at [product@bazel.build](mailto:product@bazel.build) if you are unsure whether your intended use of the Bazel Trademarks is in compliance with these guidelines, or to ask for permission to use the Bazel Trademarks, clearly describing the intended usage and duration. +Please do not hesitate to contact us at +[product@bazel.build](mailto:product@bazel.build) if you are unsure whether your +intended use of the Bazel Trademarks is in compliance with these guidelines, or +to ask for permission to use the Bazel Trademarks, clearly describing the +intended usage and duration. diff --git a/build/share-variables.mdx b/build/share-variables.mdx index 3bf6683b..b248034e 100644 --- a/build/share-variables.mdx +++ b/build/share-variables.mdx @@ -2,9 +2,13 @@ title: 'Sharing Variables' --- -`BUILD` files are intended to be simple and declarative. They will typically consist of a series of target declarations. As your code base and your `BUILD` files get larger, you will probably notice some duplication, such as: -```python + +`BUILD` files are intended to be simple and declarative. They will typically +consist of a series of target declarations. As your code base and your `BUILD` +files get larger, you will probably notice some duplication, such as: + +``` python cc_library( name = "foo", copts = ["-DVERSION=5"], @@ -19,11 +23,17 @@ cc_library( ) ``` -Code duplication in `BUILD` files is usually fine. This can make the file more readable: each declaration can be read and understood without any context. This is important, not only for humans, but also for external tools. For example, a tool might be able to read and update `BUILD` files to add missing dependencies. Code refactoring and code reuse might prevent this kind of automated modification. +Code duplication in `BUILD` files is usually fine. This can make the file more +readable: each declaration can be read and understood without any context. This +is important, not only for humans, but also for external tools. For example, a +tool might be able to read and update `BUILD` files to add missing dependencies. +Code refactoring and code reuse might prevent this kind of automated +modification. -If it is useful to share values (for example, if values must be kept in sync), you can introduce a variable: +If it is useful to share values (for example, if values must be kept in sync), +you can introduce a variable: -```python +``` python COPTS = ["-DVERSION=5"] cc_library( @@ -40,21 +50,24 @@ cc_library( ) ``` -Multiple declarations now use the value `COPTS`. By convention, use uppercase letters to name global constants. +Multiple declarations now use the value `COPTS`. By convention, use uppercase +letters to name global constants. ## Sharing variables across multiple BUILD files -If you need to share a value across multiple `BUILD` files, you have to put it in a `.bzl` file. `.bzl` files contain definitions (variables and functions) that can be used in `BUILD` files. +If you need to share a value across multiple `BUILD` files, you have to put it +in a `.bzl` file. `.bzl` files contain definitions (variables and functions) +that can be used in `BUILD` files. In `path/to/variables.bzl`, write: -```python +``` python COPTS = ["-DVERSION=5"] ``` Then, you can update your `BUILD` files to access the variable: -```python +``` python load("//path/to:variables.bzl", "COPTS") cc_library( diff --git a/build/style-guide.mdx b/build/style-guide.mdx index d0a01fe0..cfc3cf86 100644 --- a/build/style-guide.mdx +++ b/build/style-guide.mdx @@ -2,17 +2,31 @@ title: 'BUILD Style Guide' --- + + ## Prefer DAMP BUILD files over DRY -The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by introducing abstractions such as variables and functions to avoid redundancy in code. +The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by +introducing abstractions such as variables and functions to avoid redundancy in +code. -In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — encourages readability over uniqueness to make files easier to understand and maintain. +In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — +encourages readability over uniqueness to make files easier to understand and +maintain. -`BUILD` files aren't code, they are configurations. They aren't tested like code, but do need to be maintained by people and tools. That makes DAMP better for them than DRY. +`BUILD` files aren't code, they are configurations. They aren't tested like +code, but do need to be maintained by people and tools. That makes DAMP better +for them than DRY. ## BUILD.bazel file formatting -`BUILD` file formatting follows the same approach as Go, where a standardized tool takes care of most formatting issues. [Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and emits the source code in a standard style. Every `BUILD` file is therefore formatted in the same automated way, which makes formatting a non-issue during code reviews. It also makes it easier for tools to understand, edit, and generate `BUILD` files. +`BUILD` file formatting follows the same approach as Go, where a standardized +tool takes care of most formatting issues. +[Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and +emits the source code in a standard style. Every `BUILD` file is therefore +formatted in the same automated way, which makes formatting a non-issue during +code reviews. It also makes it easier for tools to understand, edit, and +generate `BUILD` files. `BUILD` file formatting must match the output of `buildifier`. @@ -45,15 +59,18 @@ py_test( **Recommendation**: Use the following order (every element is optional): -- Package description (a comment) +* Package description (a comment) -- All `load()` statements +* All `load()` statements -- The `package()` function. +* The `package()` function. -- Calls to rules and macros +* Calls to rules and macros -Buildifier makes a distinction between a standalone comment and a comment attached to an element. If a comment is not attached to a specific element, use an empty line after it. The distinction is important when doing automated changes (for example, to keep or remove a comment when deleting a rule). +Buildifier makes a distinction between a standalone comment and a comment +attached to an element. If a comment is not attached to a specific element, use +an empty line after it. The distinction is important when doing automated +changes (for example, to keep or remove a comment when deleting a rule). ```python # Standalone comment (such as to make a section in a file) @@ -64,7 +81,11 @@ cc_library(name = "cc") ## References to targets in the current package -Files should be referred to by their paths relative to the package directory (without ever using up-references, such as `..`). Generated files should be prefixed with "`:`" to indicate that they are not sources. Source files should not be prefixed with `:`. Rules should be prefixed with `:`. For example, assuming `x.cc` is a source file: +Files should be referred to by their paths relative to the package directory +(without ever using up-references, such as `..`). Generated files should be +prefixed with "`:`" to indicate that they are not sources. Source files +should not be prefixed with `:`. Rules should be prefixed with `:`. For +example, assuming `x.cc` is a source file: ```python cc_library( @@ -77,70 +98,100 @@ genrule( name = "gen_header", srcs = [], outs = ["x.h"], - cmd = "echo 'int x();' > $@", + cmd = "echo 'int x();' > $@", ) ``` ## Target naming -Target names should be descriptive. If a target contains one source file, the target should generally have a name derived from that source (for example, a `cc_library` for `chat.cc` could be named `chat`, or a `java_library` for `DirectMessage.java` could be named `direct_message`). - -The eponymous target for a package (the target with the same name as the containing directory) should provide the functionality described by the directory name. If there is no such target, do not create an eponymous target. - -Prefer using the short name when referring to an eponymous target (`//x` instead of `//x:x`). If you are in the same package, prefer the local reference (`:x` instead of `//x`). - -Avoid using "reserved" target names which have special meaning. This includes `all`, `__pkg__`, and `__subpackages__`, these names have special semantics and can cause confusion and unexpected behaviors when they are used. - -In the absence of a prevailing team convention these are some non-binding recommendations that are broadly used at Google: - -- In general, use ["snake\_case"](https://en.wikipedia.org/wiki/Snake_case) - - - For a `java_library` with one `src` this means using a name that is not the same as the filename without the extension - - For Java `*_binary` and `*_test` rules, use ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). This allows for the target name to match one of the `src`s. For `java_test`, this makes it possible for the `test_class` attribute to be inferred from the name of the target. - -- If there are multiple variants of a particular target then add a suffix to disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) - -- Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` - -- Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to avoid conflicts between a `_library` target and its corresponding `_binary`) - -- For proto related targets: - - - `proto_library` targets should have names ending in `_proto` - - - Languages specific `*_proto_library` rules should match the underlying proto but replace `_proto` with a language specific suffix such as: - - - **`cc_proto_library`**: `_cc_proto` - - **`java_proto_library`**: `_java_proto` - - **`java_lite_proto_library`**: `_java_proto_lite` +Target names should be descriptive. If a target contains one source file, +the target should generally have a name derived from that source (for example, a +`cc_library` for `chat.cc` could be named `chat`, or a `java_library` for +`DirectMessage.java` could be named `direct_message`). + +The eponymous target for a package (the target with the same name as the +containing directory) should provide the functionality described by the +directory name. If there is no such target, do not create an eponymous +target. + +Prefer using the short name when referring to an eponymous target (`//x` +instead of `//x:x`). If you are in the same package, prefer the local +reference (`:x` instead of `//x`). + +Avoid using "reserved" target names which have special meaning. This includes +`all`, `__pkg__`, and `__subpackages__`, these names have special +semantics and can cause confusion and unexpected behaviors when they are used. + +In the absence of a prevailing team convention these are some non-binding +recommendations that are broadly used at Google: + +* In general, use ["snake_case"](https://en.wikipedia.org/wiki/Snake_case) + * For a `java_library` with one `src` this means using a name that is not + the same as the filename without the extension + * For Java `*_binary` and `*_test` rules, use + ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). + This allows for the target name to match one of the `src`s. For + `java_test`, this makes it possible for the `test_class` attribute to be + inferred from the name of the target. +* If there are multiple variants of a particular target then add a suffix to + disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) +* Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` +* Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to + avoid conflicts between a `_library` target and its corresponding `_binary`) +* For proto related targets: + * `proto_library` targets should have names ending in `_proto` + * Languages specific `*_proto_library` rules should match the underlying + proto but replace `_proto` with a language specific suffix such as: + * **`cc_proto_library`**: `_cc_proto` + * **`java_proto_library`**: `_java_proto` + * **`java_lite_proto_library`**: `_java_proto_lite` ## Visibility -Visibility should be scoped as tightly as possible, while still allowing access by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as appropriate. +Visibility should be scoped as tightly as possible, while still allowing access +by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as +appropriate. -Avoid setting package `default_visibility` to `//visibility:public`. `//visibility:public` should be individually set only for targets in the project's public API. These could be libraries that are designed to be depended on by external projects or binaries that could be used by an external project's build process. +Avoid setting package `default_visibility` to `//visibility:public`. +`//visibility:public` should be individually set only for targets in the +project's public API. These could be libraries that are designed to be depended +on by external projects or binaries that could be used by an external project's +build process. ## Dependencies -Dependencies should be restricted to direct dependencies (dependencies needed by the sources listed in the rule). Do not list transitive dependencies. +Dependencies should be restricted to direct dependencies (dependencies +needed by the sources listed in the rule). Do not list transitive dependencies. -Package-local dependencies should be listed first and referred to in a way compatible with the [References to targets in the current package](#targets-current-package) section above (not by their absolute package name). +Package-local dependencies should be listed first and referred to in a way +compatible with the +[References to targets in the current package](#targets-current-package) +section above (not by their absolute package name). -Prefer to list dependencies directly, as a single list. Putting the "common" dependencies of several targets into a variable reduces maintainability, makes it impossible for tools to change the dependencies of a target, and can lead to unused dependencies. +Prefer to list dependencies directly, as a single list. Putting the "common" +dependencies of several targets into a variable reduces maintainability, makes +it impossible for tools to change the dependencies of a target, and can lead to +unused dependencies. ## Globs -Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it is more error-prone and less obvious than an empty list. +Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it +is more error-prone and less obvious than an empty list. ### Recursive -Do not use recursive globs to match source files (for example, `glob(["**/*.java"])`). +Do not use recursive globs to match source files (for example, +`glob(["**/*.java"])`). -Recursive globs make `BUILD` files difficult to reason about because they skip subdirectories containing `BUILD` files. +Recursive globs make `BUILD` files difficult to reason about because they skip +subdirectories containing `BUILD` files. -Recursive globs are generally less efficient than having a `BUILD` file per directory with a dependency graph defined between them as this enables better remote caching and parallelism. +Recursive globs are generally less efficient than having a `BUILD` file per +directory with a dependency graph defined between them as this enables better +remote caching and parallelism. -It is good practice to author a `BUILD` file in each directory and define a dependency graph between them. +It is good practice to author a `BUILD` file in each directory and define a +dependency graph between them. ### Non-recursive @@ -148,16 +199,21 @@ Non-recursive globs are generally acceptable. ## Avoid list comprehensions -Avoid using list comprehensions at the top level of a `BUILD.bazel` file. Automate repetitive calls by creating each named target with a separate top-level rule or macro call. Give each a short `name` parameter for clarity. +Avoid using list comprehensions at the top level of a `BUILD.bazel` file. +Automate repetitive calls by creating each named target with a separate +top-level rule or macro call. Give each a short `name` parameter for clarity. List comprehension reduces the following: -- Maintainability. It's difficult or impossible for human maintainers and large scale automated changes to update list comprehensions correctly. -- Discoverability. Since the pattern doesn't have `name` parameters, it's hard to find the rule by name. +* Maintainability. It's difficult or impossible for human maintainers and + large scale automated changes to update list comprehensions correctly. +* Discoverability. Since the pattern doesn't have `name` parameters, + it's hard to find the rule by name. -A common application of the list comprehension pattern is to generate tests. For example: +A common application of the list comprehension pattern is to generate tests. For +example: -```build +```build {.bad} [[java_test( name = "test_%s_%s" % (backend, count), srcs = [ ... ], @@ -172,7 +228,8 @@ A common application of the list comprehension pattern is to generate tests. For ]] ``` -We recommend using simpler alternatives. For example, define a macro that generates one test and invoke it for each top-level `name`: +We recommend using simpler alternatives. For example, define a macro that +generates one test and invoke it for each top-level `name`: ```build my_java_test(name = "test_fake_1", @@ -186,7 +243,7 @@ my_java_test(name = "test_fake_10", Don't use list variables to encapsulate common dependencies: -```build +```build {.bad} COMMON_DEPS = [ "//d:e", "//x/y:z", @@ -203,11 +260,12 @@ cc_library(name = "b", ) ``` -Similarly, don't use a library target with [`exports`](/reference/be/java#java_library.exports) to group dependencies. +Similarly, don't use a library target with +[`exports`](/reference/be/java#java_library.exports) to group dependencies. Instead, list the dependencies separately for each target: -```build +```build {.good} cc_library(name = "a", srcs = ["a.cc"], deps = [ @@ -227,25 +285,38 @@ cc_library(name = "b", ) ``` -Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools maintain them. There will be repetition, but you won't have to think about how to manage the dependencies. +Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools +maintain them. There will be repetition, but you won't have to think about how +to manage the dependencies. ## Prefer literal strings -Although Starlark provides string operators for concatenation (`+`) and formatting (`%`), use them with caution. It is tempting to factor out common string parts to make expressions more concise or break long lines. However, +Although Starlark provides string operators for concatenation (`+`) and +formatting (`%`), use them with caution. It is tempting to factor out common +string parts to make expressions more concise or break long lines. However, -- It is harder to read broken-up string values at a glance. +* It is harder to read broken-up string values at a glance. -- Automated tools such as [buildozer](https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md) and Code Search have trouble finding values and updating them correctly when the values broken up. +* Automated tools such as + [buildozer][buildozer] and Code Search have trouble finding values and + updating them correctly when the values broken up. -- In `BUILD` files, readability is more important than avoiding repetition (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). +* In `BUILD` files, readability is more important than avoiding repetition + (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). -- This Style Guide [warns against splitting label-valued strings](#other-conventions) and [explicitly permits long lines](#differences-python-style-guide). +* This Style Guide + [warns against splitting label-valued strings](#other-conventions) + and + [explicitly permits long lines](#differences-python-style-guide). -- Buildifier automatically fuses concatenated strings when it detects that they are labels. +* Buildifier automatically fuses concatenated strings when it detects that + they are labels. -Therefore, prefer explicit, literal strings over concatenated or formatted strings, especially in label-type attributes such as `name` and `deps`. For example, this `BUILD` fragment: +Therefore, prefer explicit, literal strings over concatenated or formatted +strings, especially in label-type attributes such as `name` and `deps`. For +example, this `BUILD` fragment: -```build +```build {.bad} NAME = "foo" PACKAGE = "//a/b" @@ -259,7 +330,7 @@ proto_library( would be better rewritten as -```build +```build {.good} proto_library( name = "foo_proto", deps = ["//a/b:other_proto"], @@ -267,32 +338,66 @@ proto_library( ) ``` +[buildozer]: https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md + ## Limit the symbols exported by each `.bzl` file -Minimize the number of symbols (rules, macros, constants, functions) exported by each public `.bzl` (Starlark) file. We recommend that a file should export multiple symbols only if they are certain to be used together. Otherwise, split it into multiple `.bzl` files, each with its own [bzl\_library](https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library). +Minimize the number of symbols (rules, macros, constants, functions) exported by +each public `.bzl` (Starlark) file. We recommend that a file should export +multiple symbols only if they are certain to be used together. Otherwise, split +it into multiple `.bzl` files, each with its own [bzl_library][bzl_library]. -Excessive symbols can cause `.bzl` files to grow into broad "libraries" of symbols, causing changes to single files to force Bazel to rebuild many targets. +Excessive symbols can cause `.bzl` files to grow into broad "libraries" of +symbols, causing changes to single files to force Bazel to rebuild many targets. + +[bzl_library]: https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library ## Other conventions -- Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), use lowercase and underscores to declare variables (such as `my_variable`). + * Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), + use lowercase and underscores to declare variables (such as `my_variable`). -- Labels should never be split, even if they are longer than 79 characters. Labels should be string literals whenever possible. *Rationale*: It makes find and replace easy. It also improves readability. + * Labels should never be split, even if they are longer than 79 characters. + Labels should be string literals whenever possible. *Rationale*: It makes + find and replace easy. It also improves readability. -- The value of the name attribute should be a literal constant string (except in macros). *Rationale*: External tools use the name attribute to refer a rule. They need to find rules without having to interpret code. + * The value of the name attribute should be a literal constant string (except + in macros). *Rationale*: External tools use the name attribute to refer a + rule. They need to find rules without having to interpret code. -- When setting boolean-type attributes, use boolean values, not integer values. For legacy reasons, rules still convert integers to booleans as needed, but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying "deflake this target by rerunning it once". `flaky = True` unambiguously says "this test is flaky". + * When setting boolean-type attributes, use boolean values, not integer values. + For legacy reasons, rules still convert integers to booleans as needed, + but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying + "deflake this target by rerunning it once". `flaky = True` unambiguously says + "this test is flaky". ## Differences with Python style guide -Although compatibility with [Python style guide](https://www.python.org/dev/peps/pep-0008/) is a goal, there are a few differences: - -- No strict line length limit. Long comments and long strings are often split to 79 columns, but it is not required. It should not be enforced in code reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this limit. It is common for `BUILD` files to be generated or edited by tools, which does not go well with a line length limit. - -- Implicit string concatenation is not supported. Use the `+` operator. *Rationale*: `BUILD` files contain many string lists. It is easy to forget a comma, which leads to a complete different result. This has created many bugs in the past. [See also this discussion.](https://lwn.net/Articles/551438/) - -- Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: Named arguments are much more frequent than in Python and are always on a separate line. Spaces improve readability. This convention has been around for a long time, and it is not worth modifying all existing `BUILD` files. - -- By default, use double quotation marks for strings. *Rationale*: This is not specified in the Python style guide, but it recommends consistency. So we decided to use only double-quoted strings. Many languages use double-quotes for string literals. - -- Use a single blank line between two top-level definitions. *Rationale*: The structure of a `BUILD` file is not like a typical Python file. It has only top-level statements. Using a single-blank line makes `BUILD` files shorter. +Although compatibility with +[Python style guide](https://www.python.org/dev/peps/pep-0008/) +is a goal, there are a few differences: + + * No strict line length limit. Long comments and long strings are often split + to 79 columns, but it is not required. It should not be enforced in code + reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this + limit. It is common for `BUILD` files to be generated or edited by tools, + which does not go well with a line length limit. + + * Implicit string concatenation is not supported. Use the `+` operator. + *Rationale*: `BUILD` files contain many string lists. It is easy to forget a + comma, which leads to a complete different result. This has created many bugs + in the past. [See also this discussion.](https://lwn.net/Articles/551438/) + + * Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: + Named arguments are much more frequent than in Python and are always on a + separate line. Spaces improve readability. This convention has been around + for a long time, and it is not worth modifying all existing `BUILD` files. + + * By default, use double quotation marks for strings. *Rationale*: This is not + specified in the Python style guide, but it recommends consistency. So we + decided to use only double-quoted strings. Many languages use double-quotes + for string literals. + + * Use a single blank line between two top-level definitions. *Rationale*: The + structure of a `BUILD` file is not like a typical Python file. It has only + top-level statements. Using a single-blank line makes `BUILD` files shorter. diff --git a/community/recommended-rules.mdx b/community/recommended-rules.mdx index 426d8784..86daa056 100644 --- a/community/recommended-rules.mdx +++ b/community/recommended-rules.mdx @@ -2,33 +2,53 @@ title: 'Recommended Rules' --- -In the documentation, we provide a list of [recommended rules](/rules). -This is a set of high quality rules, which will provide a good experience to our users. We make a distinction between the supported rules, and the hundreds of rules you can find on the Internet. + +In the documentation, we provide a list of +[recommended rules](/rules). + +This is a set of high quality rules, which will provide a good experience to our +users. We make a distinction between the supported rules, and the hundreds of +rules you can find on the Internet. ## Nomination -If a ruleset meets the requirements below, a rule maintainer can nominate it to be part of the *recommended rules* by filing a [GitHub issue](https://github.com/bazelbuild/bazel/). +If a ruleset meets the requirements below, a rule maintainer can nominate it +to be part of the _recommended rules_ by filing a +[GitHub issue](https://github.com/bazelbuild/bazel/). -After a review by the [Bazel core team](/contribute/policy), it will be recommended on the Bazel website. +After a review by the [Bazel core team](/contribute/policy), it +will be recommended on the Bazel website. ## Requirements for the rule maintainers -- The ruleset provides an important feature, useful to a large number of Bazel users (for example, support for a widely popular language). -- The ruleset is well maintained. There must be at least two active maintainers. -- The ruleset is well documented, with examples, and easy to use. -- The ruleset follows the best practices and is performant (see [the performance guide](/rules/performance)). -- The ruleset has sufficient test coverage. -- The ruleset is tested on [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) with the latest version of Bazel. Tests should always pass (when used as a presubmit check). -- The ruleset is also tested with the upcoming incompatible changes. Breakages should be fixed within two weeks. Migration issues should be reported to the Bazel team quickly. +* The ruleset provides an important feature, useful to a large number of Bazel + users (for example, support for a widely popular language). +* The ruleset is well maintained. There must be at least two active maintainers. +* The ruleset is well documented, with examples, and easy to use. +* The ruleset follows the best practices and is performant (see + [the performance guide](/rules/performance)). +* The ruleset has sufficient test coverage. +* The ruleset is tested on + [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) + with the latest version of Bazel. Tests should always pass (when used as a + presubmit check). +* The ruleset is also tested with the upcoming incompatible changes. Breakages + should be fixed within two weeks. Migration issues should be reported to the + Bazel team quickly. ## Requirements for Bazel developers -- Recommended rules are frequently tested with Bazel at head (at least once a day). -- No change in Bazel may break a recommended rule (with the default set of flags). If it happens, the change should be fixed or rolled back. +* Recommended rules are frequently tested with Bazel at head (at least once a + day). +* No change in Bazel may break a recommended rule (with the default set of + flags). If it happens, the change should be fixed or rolled back. ## Demotion -If there is a concern that a particular ruleset is no longer meeting the requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be filed. +If there is a concern that a particular ruleset is no longer meeting the +requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be +filed. -Rule maintainers will be contacted and need to respond in 2 weeks. Based on the outcome, Bazel core team might make a decision to demote the rule set. +Rule maintainers will be contacted and need to respond in 2 weeks. Based on the +outcome, Bazel core team might make a decision to demote the rule set. diff --git a/community/remote-execution-services.mdx b/community/remote-execution-services.mdx index 7a971881..6dee80fa 100644 --- a/community/remote-execution-services.mdx +++ b/community/remote-execution-services.mdx @@ -2,23 +2,28 @@ title: 'Remote Execution Services' --- + + Use the following services to run Bazel with remote execution: -- Manual +* Manual - - Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) directly to create your own remote execution service. + * Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) + directly to create your own remote execution service. -- Self-service +* Self-service - - [Buildbarn](https://github.com/buildbarn) - - [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - - [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - - [NativeLink](https://github.com/TraceMachina/nativelink) + * [Buildbarn](https://github.com/buildbarn) + * [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + * [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + * [NativeLink](https://github.com/TraceMachina/nativelink) -- Commercial +* Commercial - - [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. - - [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. - - [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, caching, and results UI. - - [EngFlow Remote Execution](https://www.engflow.com) - Remote execution and remote caching service with Build and Test UI. Can be self-hosted or hosted. - - [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. + * [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. + * [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. + * [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, + caching, and results UI. + * [EngFlow Remote Execution](https://www.engflow.com) - Remote execution + and remote caching service with Build and Test UI. Can be self-hosted or hosted. + * [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. diff --git a/community/sig.mdx b/community/sig.mdx index 33ef6b15..ae5f9189 100644 --- a/community/sig.mdx +++ b/community/sig.mdx @@ -2,23 +2,42 @@ title: 'Bazel Special Interest Groups' --- -Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular areas and to support communication and coordination between [Bazel owners, maintainers, and contributors](/contribute/policy). This policy applies to [`bazelbuild`](http://github.com/bazelbuild). -SIGs do their work in public. The ideal scope for a SIG covers a well-defined domain, where the majority of participation is from the community. SIGs may focus on community maintained repositories in `bazelbuild` (such as language rules) or focus on areas of code in the Bazel repository (such as Remote Execution). -While not all SIGs will have the same level of energy, breadth of scope, or governance models, there should be sufficient evidence that there are community members willing to engage and contribute should the interest group be established. Before joining, review the group's work, and then get in touch with the SIG leader. Membership policies vary on a per-SIG basis. +Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular +areas and to support communication and coordination between [Bazel owners, +maintainers, and contributors](/contribute/policy). This policy +applies to [`bazelbuild`](http://github.com/bazelbuild). -See the complete list of [Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). +SIGs do their work in public. The ideal scope for a SIG covers a well-defined +domain, where the majority of participation is from the community. SIGs may +focus on community maintained repositories in `bazelbuild` (such as language +rules) or focus on areas of code in the Bazel repository (such as Remote +Execution). + +While not all SIGs will have the same level of energy, breadth of scope, or +governance models, there should be sufficient evidence that there are community +members willing to engage and contribute should the interest group be +established. Before joining, review the group's work, and then get in touch +with the SIG leader. Membership policies vary on a per-SIG basis. + +See the complete list of +[Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). ### Non-goals: What a SIG is not -SIGs are intended to facilitate collaboration on shared work. A SIG is therefore: +SIGs are intended to facilitate collaboration on shared work. A SIG is +therefore: -- *Not a support forum:* a mailing list and a SIG is not the same thing -- *Not immediately required:* early on in a project's life, you may not know if you have shared work or collaborators -- *Not free labor:* energy is required to grow and coordinate the work collaboratively +- *Not a support forum:* a mailing list and a SIG is not the same thing +- *Not immediately required:* early on in a project's life, you may not know + if you have shared work or collaborators +- *Not free labor:* energy is required to grow and coordinate the work + collaboratively -Bazel Owners take a conservative approach to SIG creation—thanks to the ease of starting projects on GitHub, there are many avenues where collaboration can happen without the need for a SIG. +Bazel Owners take a conservative approach to SIG creation—thanks to the ease of +starting projects on GitHub, there are many avenues where collaboration can +happen without the need for a SIG. ## SIG lifecycle @@ -26,65 +45,114 @@ This section covers how to create a SIG. ### Research and consultation -To propose a new SIG group, first gather evidence for approval, as specified below. Some possible avenues to consider are: +To propose a new SIG group, first gather evidence for approval, as specified +below. Some possible avenues to consider are: -- A well-defined problem or set of problems the group would solve -- Consultation with community members who would benefit, assessing both the benefit and their willingness to commit -- For existing projects, evidence from issues and PRs that contributors care about the topic -- Potential goals for the group to achieve -- Resource requirements of running the group +- A well-defined problem or set of problems the group would solve +- Consultation with community members who would benefit, assessing both the + benefit and their willingness to commit +- For existing projects, evidence from issues and PRs that contributors care + about the topic +- Potential goals for the group to achieve +- Resource requirements of running the group -Even if the need for a SIG seems self-evident, the research and consultation is still important to the success of the group. +Even if the need for a SIG seems self-evident, the research and consultation is +still important to the success of the group. ### Create the new group -The new group should follow the below process for chartering. In particular, it must demonstrate: - -- A clear purpose and benefit to Bazel (either around a sub-project or application area) -- Two or more contributors willing to act as group leads, existence of other contributors, and evidence of demand for the group -- Each group needs to use at least one publicly accessible mailing list. A SIG may reuse one of the public lists, such as [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list for @bazel.build, or create their own list -- Resources the SIG initially requires (usually, mailing list and regular video call.) -- SIGs can serve documents and files from their directory in [`bazelbuild/community`](https://github.com/bazelbuild/community) or from their own repository in the [`bazelbuild`](https://github.com/bazelbuild) GitHub organization. SIGs may link to external resources if they choose to organize their work outside of the `bazelbuild` GitHub organization -- Bazel Owners approve or reject SIG applications and consult other stakeholders as necessary - -Before entering the formal parts of the process, you should consult with the Bazel product team, at [product@bazel.build](mailto:product@bazel.build). Most SIGs require conversation and iteration before approval. - -The formal request for the new group is done by submitting a charter as a PR to [`bazelbuild/community`](https://github.com/bazelbuild/community), and including the request in the comments on the PR following the template below. On approval, the PR for the group is merged and the required resources created. +The new group should follow the below process for chartering. In particular, it +must demonstrate: + +- A clear purpose and benefit to Bazel (either around a sub-project or + application area) +- Two or more contributors willing to act as group leads, existence of other + contributors, and evidence of demand for the group +- Each group needs to use at least one publicly accessible mailing list. A SIG + may reuse one of the public lists, such as + [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list + for @bazel.build, or create their own list +- Resources the SIG initially requires (usually, mailing list and regular + video call.) +- SIGs can serve documents and files from their directory in + [`bazelbuild/community`](https://github.com/bazelbuild/community) + or from their own repository in the + [`bazelbuild`](https://github.com/bazelbuild) GitHub + organization. SIGs may link to external resources if they choose to organize + their work outside of the `bazelbuild` GitHub organization +- Bazel Owners approve or reject SIG applications and consult other + stakeholders as necessary + +Before entering the formal parts of the process, you should consult with +the Bazel product team, at product@bazel.build. Most SIGs require conversation +and iteration before approval. + +The formal request for the new group is done by submitting a charter as a PR to +[`bazelbuild/community`](https://github.com/bazelbuild/community), +and including the request in the comments on the PR following the template +below. On approval, the PR for the group is merged and the required resources +created. ### Template Request for New SIG -To request a new SIG, use the template in the community repo: [SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). +To request a new SIG, use the template in the community repo: +[SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). ### Chartering -To establish a group, you need a charter and must follow the Bazel [code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). Archives of the group will be public. Membership may either be open to all without approval, or available on request, pending approval of the group administrator. +To establish a group, you need a charter and must follow the Bazel +[code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). +Archives of the group will be public. Membership may either be open to all +without approval, or available on request, pending approval of the group +administrator. -The charter must nominate an administrator. As well as an administrator, the group must include at least one person as lead (these may be the same person), who serves as point of contact for coordination as required with the Bazel product team. +The charter must nominate an administrator. As well as an administrator, the +group must include at least one person as lead (these may be the same person), +who serves as point of contact for coordination as required with the Bazel +product team. -Group creators must post their charter to the group mailing list. The community repository in the Bazel GitHub organization archives such documents and policies. As groups evolve their practices and conventions, they should update their charters within the relevant part of the community repository. +Group creators must post their charter to the group mailing list. The community +repository in the Bazel GitHub organization archives such documents and +policies. As groups evolve their practices and conventions, they should update +their charters within the relevant part of the community repository. ### Collaboration and inclusion -While not mandated, the group should choose to make use of collaboration via scheduled conference calls or chat channels to conduct meetings. Any such meetings should be advertised on the mailing list, and notes posted to the mailing list afterwards. Regular meetings help drive accountability and progress in a SIG. +While not mandated, the group should choose to make use of collaboration +via scheduled conference calls or chat channels to conduct meetings. Any such +meetings should be advertised on the mailing list, and notes posted to the +mailing list afterwards. Regular meetings help drive accountability and progress +in a SIG. -Bazel product team members may proactively monitor and encourage the group to discussion and action as appropriate. +Bazel product team members may proactively monitor and encourage the group to +discussion and action as appropriate. ### Launch a SIG Required activities: -- Notify Bazel general discussion groups ([bazel-discuss](https://groups.google.com/g/bazel-discuss), [bazel-dev](https://groups.google.com/g/bazel-dev)). +- Notify Bazel general discussion groups + ([bazel-discuss](https://groups.google.com/g/bazel-discuss), + [bazel-dev](https://groups.google.com/g/bazel-dev)). Optional activities: -- Create a blog post for the Bazel blog +- Create a blog post for the Bazel blog ### Health and termination of SIGs -The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners occasionally request the SIG lead to report on the SIG's work, to inform the broader Bazel community of the group's activity. +The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners +occasionally request the SIG lead to report on the SIG's work, to inform the +broader Bazel community of the group's activity. -If a SIG no longer has a useful purpose or interested community, it may be archived and cease operation. The Bazel product team reserves the right to archive such inactive SIGs to maintain the overall health of the project, though it is a less preferable outcome. A SIG may also opt to disband if it recognizes it has reached the end of its useful life. +If a SIG no longer has a useful purpose or interested community, it may be +archived and cease operation. The Bazel product team reserves the right to +archive such inactive SIGs to maintain the overall health of the project, +though it is a less preferable outcome. A SIG may also opt to disband if +it recognizes it has reached the end of its useful life. ## Note -*This content has been adopted from Tensorflow’s [SIG playbook](https://www.tensorflow.org/community/sig_playbook) with modifications.* +*This content has been adopted from Tensorflow’s +[SIG playbook](https://www.tensorflow.org/community/sig_playbook) +with modifications.* diff --git a/community/users.mdx b/community/users.mdx index 81a174c5..91e26c47 100644 --- a/community/users.mdx +++ b/community/users.mdx @@ -2,189 +2,276 @@ title: 'Who''s Using Bazel' --- -Note: Using Bazel? You can add your company on [StackShare](https://stackshare.io/bazel). To add yourself to this page, contact [product@bazel.build](mailto:product@bazel.build). -This page lists companies and OSS projects that are known to use Bazel. This does not constitute an endorsement. + +Note: Using Bazel? You can add your company on +[StackShare](https://stackshare.io/bazel). To add yourself to this page, +contact [product@bazel.build](mailto:product@bazel.build). + +This page lists companies and OSS projects that are known to use Bazel. +This does not constitute an endorsement. ## Companies using Bazel ### [acqio](https://acqio.com.br) -![](/community/images/acqio_logo.svg) + -Acqio is a Fintech that provides payment products and services for small and medium merchants. Acqio has a handful of monorepos and uses Bazel along with Kubernetes to deliver fast and reliable microservices. +Acqio is a Fintech that provides payment products and services for small and +medium merchants. Acqio has a handful of monorepos and uses Bazel along with +Kubernetes to deliver fast and reliable microservices. ### [Adobe](https://www.adobe.com/) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Adobe_logo_and_wordmark_%282017%29.svg/440px-Adobe_logo_and_wordmark_%282017%29.svg.png) + -Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for continuous, GitOps driven Kubernetes deployments. +Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for +continuous, GitOps driven Kubernetes deployments. ### [Asana](https://asana.com) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Asana_logo.svg/256px-Asana_logo.svg.png) + -Asana is a web and mobile application designed to help teams track their work. In their own words: +Asana is a web and mobile application designed to help teams track their work. +In their own words: -> Bazel has increased reliability, stability, and speed for all of builds/tests at Asana. We no longer need to clean because of incorrect caches. +> Bazel has increased reliability, stability, and speed for all of builds/tests +at Asana. We no longer need to clean because of incorrect caches. ### [Ascend.io](https://ascend.io) -Ascend is a Palo Alto startup that offers solutions for large data sets analysis. Their motto is *Big data is hard. We make it easy*. +Ascend is a Palo Alto startup that offers solutions for large data sets +analysis. Their motto is _Big data is hard. We make it easy_. ### [ASML](https://asml.com) -![](https://upload.wikimedia.org/wikipedia/en/6/6c/ASML_Holding_N.V._logo.svg) + -ASML is an innovation leader in the semiconductor industry. We provide chipmakers with everything they need – hardware, software and services – to mass produce patterns on silicon through lithography. +ASML is an innovation leader in the semiconductor industry. We provide chipmakers +with everything they need – hardware, software and services – to mass produce +patterns on silicon through lithography. ### [Beeswax](https://www.beeswax.com/) -> Beeswax is a New York based startup that provides real time bidding as service. Bazel powers their Jenkins based continuous integration and deployment framework. Beeswax loves Bazel because it is blazingly fast, correct and well supported across many languages and platforms. +> Beeswax is a New York based startup that provides real time bidding as +service. Bazel powers their Jenkins based continuous integration and deployment +framework. Beeswax loves Bazel because it is blazingly fast, correct and well +supported across many languages and platforms. ### [Braintree](https://www.braintreepayments.com) -![](https://upload.wikimedia.org/wikipedia/commons/0/00/Braintree-logo1.png) + -Braintree, a PayPal subsidiary, develops payment solutions for websites and applications. They use Bazel for parts of their internal build and Paul Gross even posted a [nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). +Braintree, a PayPal subsidiary, develops payment solutions for websites and +applications. They use Bazel for parts of their internal build and Paul Gross +even posted a +[nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). ### [Canva](https://www.canva.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/b/bb/Canva_Logo.svg) - -Canva leverages Bazel to manage its large polyglot codebase, which includes Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered significant developer and compute infrastructure efficiencies, for example 5-6x decreases in average CI build times, and it continues to become the foundation of fast, reproducible, and standardised software builds at the company. +Canva leverages Bazel to manage its large polyglot codebase, which includes +Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered +significant developer and compute infrastructure efficiencies, for example 5-6x +decreases in average CI build times, and it continues to become the foundation +of fast, reproducible, and standardised software builds at the company. ### [CarGurus](https://www.cargurus.com) + -![](https://www.cargurus.com/gfx/reskin/logos/logo_CarGurus.svg) - -CarGurus is on a mission to build the world's most trusted and transparent automotive marketplace and uses Bazel to build their polyglot monorepo. +CarGurus is on a mission to build the world's most trusted and transparent +automotive marketplace and uses Bazel to build their polyglot monorepo. ### [Compass](https://www.compass.com) -Compass is a tech-driven real estate platform. With an elite team of real estate, technology and business professionals, we aim to be the best and most trusted source for home seekers. +Compass is a tech-driven real estate platform. With an elite team of real +estate, technology and business professionals, we aim to be the best and most +trusted source for home seekers. ### [Databricks](https://databricks.com) -![](https://databricks.com/wp-content/uploads/2021/10/db-nav-logo.svg) Databricks provides cloud-based integrated workspaces based on Apache Spark™. + +Databricks provides cloud-based integrated workspaces based on Apache Spark™. -> The Databricks codebase is a Monorepo, containing the Scala code that powers most of our services, Javascript for front-end UI, Python for scripting, Jsonnet to configure our infrastructure, and much more \[...] Even though our monorepo contains a million lines of Scala, working with code within is fast and snappy. ([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) +> The Databricks codebase is a Monorepo, containing the Scala code that powers +most of our services, Javascript for front-end UI, Python for scripting, +Jsonnet to configure our infrastructure, and much more [...] Even though our +monorepo contains a million lines of Scala, working with code within is fast +and snappy. +([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) ### [Dataform](https://dataform.co) -Dataform provides scalable analytics for data teams. They maintain a handful of NPM packages and a documentation site in one single monorepo and they do it all with Bazel. +Dataform provides scalable analytics for data teams. They maintain a handful of +NPM packages and a documentation site in one single monorepo and they do it all +with Bazel. -After the migration to Bazel, they [reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), including: +After the migration to Bazel, they +[reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), +including: -> - Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). -> - Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes -> - Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in \< 30 minutes on a brand new, empty laptop +> * Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). +> * Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes +> * Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop ### [Deep Silver FISHLABS](https://www.dsfishlabs.com) - -Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with C++/Python/Go/C as a base for their internal build tooling and especially for baking and deploying all their 3D Assets. +Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with +C++/Python/Go/C as a base for their internal build tooling and especially for +baking and deploying all their 3D Assets. ### [Dropbox](https://www.dropbox.com/) - -![](/community/images/dropbox.png) At Dropbox, Bazel is a key component to our distributed build and test environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable production releases. + +At Dropbox, Bazel is a key component to our distributed build and test +environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable +production releases. ### [Engel & Völkers](https://www.engelvoelkers.com) -Engel & Völkers AG is a privately owned German company that, via a series of franchised offices, provides services related to real estate transactions. +Engel & Völkers AG is a privately owned German company that, via a series of +franchised offices, provides services related to real estate transactions. -> One of our internal project has seen a decrease of compilation time from 11 minutes to roughly 1 minute, this was an impressive achievement and we are currently working on bringing Bazel to more projects. ([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) +> One of our internal project has seen a decrease of compilation time from 11 +minutes to roughly 1 minute, this was an impressive achievement and we are +currently working on bringing Bazel to more projects. +([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) ### [Etsy](https://www.etsy.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/a/aa/Etsy_logo_lg_rgb.png) +Etsy is an e-commerce website focused on handmade or vintage items and supplies, +as well as unique factory-manufactured items. -Etsy is an e-commerce website focused on handmade or vintage items and supplies, as well as unique factory-manufactured items. - -They use Bazel to build and test its Java-based search platform. Bazel produces both packages for bare metal servers and repeatable Docker images. +They use Bazel to build and test its Java-based search platform. Bazel produces +both packages for bare metal servers and repeatable Docker images. ### [Evertz.io](https://www.evertz.io/) -Evertz.io is a multi-tenant, serverless SaaS platform for offering cost effective, multi-regional services worldwide to the Broadcast Media Industry, created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). +Evertz.io is a multi-tenant, serverless SaaS platform for offering cost +effective, multi-regional services worldwide to the Broadcast Media Industry, +created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). -The website is fully built and deployed with an Angular and Bazel workflow ([source](https://twitter.com/MattMackay/status/1113947685508341762)). +The website is fully built and deployed with an Angular and Bazel workflow +([source](https://twitter.com/MattMackay/status/1113947685508341762)). ### [FINDMINE](http://www.findmine.com) + -![](https://www.findmine.com/static/assets/landpage/findmine-color-logo.png) - -FINDMINE is a automation technology for the retail industry that uses machine learning to scale the currently manual and tedious process of product curation. We use Bazel to mechanize our entire python package building, testing, and deployment process. +FINDMINE is a automation technology for the retail industry that uses machine +learning to scale the currently manual and tedious process of product curation. +We use Bazel to mechanize our entire python package building, testing, and +deployment process. ### [Flexport](https://www.flexport.com/) -Flexport is a tech-enabled global freight forwarder; our mission is to make global trade easier for everyone. At Flexport, we use Bazel to build/test our Java/JavaScript services and client libraries and to generate Java and Ruby code from protobuf definitions. [Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) +Flexport is a tech-enabled global freight forwarder; our mission is to make +global trade easier for everyone. At Flexport, we use Bazel to build/test our +Java/JavaScript services and client libraries and to generate Java and Ruby +code from protobuf definitions. +[Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) ### [Foursquare](https://foursquare.com) + -![](https://upload.wikimedia.org/wikipedia/commons/9/99/FSQ_logo.png) - -Foursquare's mission is to create technology that constructs meaningful bridges between digital spaces and physical places. We manage millions of lines of primarily Scala and Python code powering data-intensive applications, including complex codegen and container build processes, with Bazel. +Foursquare's mission is to create technology that constructs meaningful +bridges between digital spaces and physical places. We manage millions of +lines of primarily Scala and Python code powering data-intensive +applications, including complex codegen and container build processes, with +Bazel. ### [GermanTechJobs](https://germantechjobs.de) + -![](https://upload.wikimedia.org/wikipedia/commons/9/98/GermanTechJobs_Logo.png) - -Bazel has simplified our workflows 10-fold and enabled shipping features at scale. +Bazel has simplified our workflows 10-fold and enabled shipping features at +scale. ### [Google](https://google.com) + -![](https://upload.wikimedia.org/wikipedia/commons/2/2f/Google_2015_logo.svg) - -Bazel was designed to be able to scale to Google's needs and meet Google's requirements of reproducibility and platform/language support. All software at Google is built using Bazel. Google uses Bazel and its rules for millions of builds every day. +Bazel was designed to be able to scale to Google's needs and meet Google's +requirements of reproducibility and platform/language support. All software at +Google is built using Bazel. Google uses Bazel and its rules for millions of +builds every day. ### [Huawei](http://www.huawei.com/) -> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go projects, except for Go projects, others originally were built by Maven. We write a simple tool to translate a Maven-built project into Bazel-built one. More and more projects will use Bazel in recent future. +> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go +projects, except for Go projects, others originally were built by Maven. We +write a simple tool to translate a Maven-built project into Bazel-built one. +More and more projects will use Bazel in recent future. ### [IMC Trading](https://imc.com) + -![](https://upload.wikimedia.org/wikipedia/commons/1/17/IMC_Logo.svg) - -> IMC is a global proprietary trading firm and market maker headquarted in Amsterdam. We are using Bazel to continuously build and test our Java/C++/Python/SystemVerilog projects. +> IMC is a global proprietary trading firm and market maker headquarted in +Amsterdam. We are using Bazel to continuously build and test our +Java/C++/Python/SystemVerilog projects. ### [Improbable.io](https://improbable.io/) -Improbable.io develops SpatialOS, a distributed operating system that enables creating huge simulations inhabited by millions of complex entities. +Improbable.io develops SpatialOS, a distributed operating system that enables +creating huge simulations inhabited by millions of complex entities. ### [Interaxon](https://www.choosemuse.com/) -InteraXon is a thought-controlled computing firm that creates hardware and software platforms to convert brainwaves into digital signals. +InteraXon is a thought-controlled computing firm that creates hardware and +software platforms to convert brainwaves into digital signals. ### [Jupiter](https://jupiter.co/) -Jupiter is a company that provides delivery of groceries and household essentials every week. +Jupiter is a company that provides delivery of groceries and household +essentials every week. -They use Bazel in their backend code, specifically to compile protos and Kotlin to JVM binaries, using remote caching. ([source](https://starship.jupiter.co/jupiter-stack/)) +They use Bazel in their backend code, specifically to compile protos and Kotlin +to JVM binaries, using remote caching. +([source](https://starship.jupiter.co/jupiter-stack/)) ### [Just](https://gojust.com/) -Just is an enterprise financial technology company, headquartered in Norway, creating software solutions to transform how global corporate treasurers manage risk and liquidity. Their entire application stack is built with Bazel. +Just is an enterprise financial technology company, headquartered in Norway, +creating software solutions to transform how global corporate treasurers manage +risk and liquidity. Their entire application stack is built with Bazel. ### [Line](https://line.me/) -Line provides an app for instant communications, which is the most popular messaging application in Japan. They use Bazel on their codebase consisting of about 60% Swift and 40% C/C++/Objective-C/Objective-C++ ([source](https://twitter.com/thi_dt/status/1253334262020886532)). +Line provides an app for instant communications, which is the most popular +messaging application in Japan. +They use Bazel on their codebase consisting of about 60% Swift and 40% +C/C++/Objective-C/Objective-C++ +([source](https://twitter.com/thi_dt/status/1253334262020886532)). -> After switching to Bazel, we were able to achieve a huge improvement in the build times. This brought a significant improvement in the turn-around time during a QA period. Distributing a new build to our testers no longer means another hour waiting for building and testing. ([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) +> After switching to Bazel, we were able to achieve a huge improvement in the +build times. This brought a significant improvement in the turn-around time +during a QA period. Distributing a new build to our testers no longer means +another hour waiting for building and testing. +([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) ### [LingoChamp](https://www.liulishuo.com/en) -![](/community/images/liulishuo.png) LingoChamp provides professional solutions to English learners. We use Bazel for our go, java and python projects. + +LingoChamp provides professional solutions to English learners. We use Bazel +for our go, java and python projects. ### [LinkedIn](https://linkedin.com/) -![](/community/images/Linkedin-Logo.png) LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social network. LinkedIn uses Bazel for building its iOS Apps. + +LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social +network. LinkedIn uses Bazel for building its iOS Apps. ### [Lucid Software](https://lucid.co/) -![](/community/images/Lucid_Software-logo.svg) + -Lucid Software is a leader in visual collaboration, helping teams see and build the future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), [Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams can align around a shared vision, clarify complexity, and collaborate visually, no matter where they’re located. +Lucid Software is a leader in visual collaboration, helping teams see and build the +future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), +[Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams +can align around a shared vision, clarify complexity, and collaborate visually, no +matter where they’re located. -Lucid uses Bazel to build millions of lines of Scala and TypeScript. Migrating to Bazel has tremendously sped up its builds, reduced external dependencies on the build environment, and simplified developers' experience with the build system. Bazel has improved developer productivity at Lucid and unlocked further growth. +Lucid uses Bazel to build millions of lines of Scala and TypeScript. +Migrating to Bazel has tremendously sped up its builds, reduced external +dependencies on the build environment, and simplified developers' experience +with the build system. Bazel has improved developer productivity at Lucid and +unlocked further growth. ### [Lyft](https://www.lyft.com/) @@ -192,81 +279,131 @@ Lyft is using Bazel for their iOS ([source](https://twitter.com/SmileyKeith/stat ### [Meetup](http://www.meetup.com/) -Meetup is an online social networking portal that facilitates offline group meetings. The Meetup engineering team contributes to [rules\_scala](https://github.com/bazelbuild/rules_scala) and is the maintainer of [rules\_avro](https://github.com/meetup/rules_avro) and [rules\_openapi](https://github.com/meetup/rules_openapi). +Meetup is an online social networking portal that facilitates offline group +meetings. +The Meetup engineering team contributes to +[rules_scala](https://github.com/bazelbuild/rules_scala) and is the +maintainer of [rules_avro](https://github.com/meetup/rules_avro) +and [rules_openapi](https://github.com/meetup/rules_openapi). + ### [Nvidia](https://www.nvidia.com/) -> At Nvidia we have been using dazel(docker bazel) for python to work around some of bazel's python short comings. Everything else runs in normal bazel (Mostly Go / Scala/ C++/ Cuda) ([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) +> At Nvidia we have been using dazel(docker bazel) for python to work around +some of bazel's python short comings. Everything else runs in normal bazel +(Mostly Go / Scala/ C++/ Cuda) +([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) + ### [Peloton Technology](http://www.peloton-tech.com) -Peloton Technology is an automated vehicle technology company that tackles truck accidents and fuel use. They use Bazel to *enable reliable builds for automotive safety systems*. +Peloton Technology is an automated vehicle technology company that tackles truck +accidents and fuel use. They use Bazel to _enable reliable builds for automotive +safety systems_. ### [Pigweed](https://pigweed.dev) -![](https://pigweed.dev/_static/pw_logo.svg) + -Pigweed is an open-source solution for sustained, robust, and rapid embedded product development for large teams. Pigweed has shipped in millions of devices, including Google's suite of Pixel devices, Nest thermostats, [satellites](https://www.spinlaunch.com/), and [autonomous aerial drones](https://www.flyzipline.com/). +Pigweed is an open-source solution for sustained, robust, and rapid embedded +product development for large teams. Pigweed has shipped in millions of +devices, including Google's suite of Pixel devices, Nest thermostats, +[satellites](https://www.spinlaunch.com/), and [autonomous aerial +drones](https://www.flyzipline.com/). -Pigweed [uses Bazel as its primary build system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for Embedded](https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded) blog post discusses why we think it's a great build system for embedded projects! +Pigweed [uses Bazel as its primary build +system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for +Embedded][pw-bazel-great] blog post discusses why we think it's a great build +system for embedded projects! + +[pw-bazel-great]: https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded ### [Pinterest](https://www.pinterest.com/) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Pinterest_Logo.svg/200px-Pinterest_Logo.svg.png) + -Pinterest is the world’s catalog of ideas. They use Bazel to build various backend services (Java/C++) and the iOS application (Objective-C/C++). +Pinterest is the world’s catalog of ideas. They use Bazel to build various +backend services (Java/C++) and the iOS application (Objective-C/C++). -> We identified Bazel was the best fit for our goals to build a foundation for an order of magnitude improvement in performance, eliminate variability in build environments and adopt incrementally. As a result, we’re now shipping all our iOS releases using Bazel. [Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) +> We identified Bazel was the best fit for our goals to build a foundation for +an order of magnitude improvement in performance, eliminate variability in +build environments and adopt incrementally. As a result, we’re now shipping all +our iOS releases using Bazel. +[Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) ### [PubRef](https://github.com/pubref) -PubRef is an emerging scientific publishing platform. They use Bazel with [rules\_closure](https://github.com/bazelbuild/rules_closure) to build the frontend, native java rules to build the main backend, [rules\_go](https://github.com/bazelbuild/rules_go), [rules\_node](https://github.com/pubref/rules_node), and [rules\_kotlin](https://github.com/pubref/rules_kotlin) to build assorted backend services. [rules\_protobuf](https://github.com/pubref/rules_protobuf) is used to assist with gRPC-based communication between backend services. PubRef.org is based in Boulder, CO. +PubRef is an emerging scientific publishing platform. They use Bazel with +[rules_closure](https://github.com/bazelbuild/rules_closure) to build the +frontend, native java rules to build the main backend, +[rules_go](https://github.com/bazelbuild/rules_go), +[rules_node](https://github.com/pubref/rules_node), and +[rules_kotlin](https://github.com/pubref/rules_kotlin) to build assorted +backend services. [rules_protobuf](https://github.com/pubref/rules_protobuf) is +used to assist with gRPC-based communication between backend services. +PubRef.org is based in Boulder, CO. ### [Redfin](https://redfin.com/) - -Redfin is a next-generation real estate brokerage with full-service local agents. They use Bazel to build and deploy the website and various backend services. - -> With the conversion mostly behind us, things are greatly improved! Our CI builds are faster (*way* faster: they used to take 40–90 minutes, and now dev builds average 5–6 minutes). Reliability is far higher, too. This is harder to quantify, but the shift from unexplained build failures being something that “just happens” to being viewed as real problems to be solved has put us on a virtuous cycle of ever-increasing reliability. ([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) +Redfin is a next-generation real estate brokerage with full-service local +agents. They use Bazel to build and deploy the website and various backend +services. + +> With the conversion mostly behind us, things are greatly improved! Our CI +builds are faster (*way* faster: they used to take 40–90 minutes, and now dev +builds average 5–6 minutes). Reliability is far higher, too. This is harder to +quantify, but the shift from unexplained build failures being something that +“just happens” to being viewed as real problems to be solved has put us on a +virtuous cycle of ever-increasing reliability. +([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) ### [Ritual](https://ritual.co) + -![](https://lh3.googleusercontent.com/7Ir6j25ROnsXhtQXveOzup33cizxLf-TiifSC1cI6op0bQVB-WePmPjJOfXUBQ0L3KpkheObAiS28e-TS8hZtDzxOIc) - -Ritual is a mobile pick up app, connecting restaurants with customers to offer a simple, time-saving tool to get the food and beverages they want, without the wait. Ritual uses Bazel for their backend services. +Ritual is a mobile pick up app, connecting restaurants with customers to offer +a simple, time-saving tool to get the food and beverages they want, without the +wait. Ritual uses Bazel for their backend services. ### [Snap](https://www.snap.com/en-US/) -Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more details about their process, see their [engineering blog](https://eng.snap.com/blog/). +Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel +in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more +details about their process, see their [engineering blog](https://eng.snap.com/blog/). ### [Stripe](https://stripe.com) - -![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Stripe_Logo%2C_revised_2016.svg/320px-Stripe_Logo%2C_revised_2016.svg.png) + Stripe provides mobile payment solutions. They use Bazel in their build and test pipelines, as detailed in their [engineering blog](https://stripe.com/blog/fast-secure-builds-choose-two). ### [Tinder](https://tinder.com) + -![](https://policies.tinder.com/static/b0327365f4c0a31c4337157c10e9fadf/c1b63/tinder_full_color_watermark.png) - -Tinder migrated its iOS app from CocoaPods to Bazel in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). +Tinder migrated its iOS app from CocoaPods to Bazel +in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). ### [Tink](https://tink.com/) + -![](https://cdn.tink.se/tink-logos/LOW/Tink_Black.png) +Tink is a european fintech, building the best way to connect to banks across +Europe. -Tink is a european fintech, building the best way to connect to banks across Europe. - -They are using Bazel to build their backend services from a polyglot monorepo. Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) meetup group. +They are using Bazel to build their backend services from a polyglot monorepo. +Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) +meetup group. ### [Tokopedia](https://www.tokopedia.com/) -Tokopedia is an Indonesian technology company specializing in e-commerce, with over 90 million monthly active users and over 7 million merchants on the platform. +Tokopedia is an Indonesian technology company specializing in e-commerce, with +over 90 million monthly active users and over 7 million merchants on the +platform. -They wrote the article [How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), where they explain how Bazel sped up their builds. The build duration went from 55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote caching. +They wrote the article +[How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), +where they explain how Bazel sped up their builds. The build duration went from +55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote +caching. ### [Trunk.io](https://trunk.io/merge/trunk-merge-and-bazel) - -![](/community/images/trunk-logo-dark.svg) + Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initialized Capital. Trunk offers a powerful pull request merge service with first-class support for the Bazel build system. By leveraging Bazel's understanding of dependencies within a codebase, Trunk's merge service intelligently creates parallel merge lanes, allowing independent changes to be tested and merged simultaneously. @@ -274,49 +411,71 @@ Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initial ### [Twitter](https://twitter.com/) -Twitter has made the decision to migrate from Pants to Bazel as their primary build tool ([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). +Twitter has made the decision to migrate from Pants to Bazel as their primary +build tool +([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). ### [Two Sigma](https://www.twosigma.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Two_Sigma_logo.svg/2880px-Two_Sigma_logo.svg.png) - -Two Sigma is a New York-headquartered technology company dedicated to finding value in the world’s data. +Two Sigma is a New York-headquartered technology company dedicated to finding +value in the world’s data. ### [TypeDB](https://typedb.com) +TypeDB Logo -![TypeDB Logo](/community/images/typedb.png) - -TypeDB is a database technology that can be used to intuitively model interconnected data. Through its type-theoretic and polymorphic query language, TypeQL, the data can be accessed with simple, human-readable queries that run at lightspeed. +TypeDB is a database technology that can be used to intuitively model +interconnected data. Through its type-theoretic and polymorphic query language, +TypeQL, the data can be accessed with simple, human-readable queries that run at +lightspeed. -Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution pipeline that manages many repositories in a wide variety of languages, and deploys to numerous platforms seamlessly. The TypeDB team has also released Bazel rules for assembling and deploying software distributions. +Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution +pipeline that manages many repositories in a wide variety of languages, and +deploys to numerous platforms seamlessly. The TypeDB team has also released +Bazel rules for assembling and deploying software distributions. ### [Uber](https://www.uber.com) -Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo is likely one of the largest Go repositories using Bazel. See the article [Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) to learn more about their experience. +Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo +is likely one of the largest Go repositories using Bazel. See the article +[Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) +to learn more about their experience. ### [Uber Advanced Technologies Group](https://www.uber.com/info/atg/) + -![](https://upload.wikimedia.org/wikipedia/commons/thumb/6/62/Uber_logo.svg/220px-Uber_logo.svg.png) - -Uber Advanced Technologies Group is focused on autonomous vehicle efforts at Uber, including trucking/freight and autonomous ride sharing. The organization uses Bazel as its primary build system. +Uber Advanced Technologies Group is focused on autonomous vehicle efforts at +Uber, including trucking/freight and autonomous ride sharing. The organization +uses Bazel as its primary build system. ### [Vistar Media](http://vistarmedia.com) - -Vistar Media is an advertising platform that enables brands to reach consumers based on their behavior in the physical world. Their engineering team is primarily based out of Philadelphia and is using Bazel for builds, deploys, to speed up testing, and to consolidate repositories written with a variety of different technologies. +Vistar Media is an advertising platform that enables brands to reach consumers +based on their behavior in the physical world. Their engineering team is +primarily based out of Philadelphia and is using Bazel for builds, deploys, to +speed up testing, and to consolidate repositories written with a variety of +different technologies. ### [VMware](https://www.vmware.com/) - -VMware uses Bazel to produce deterministic, reliable builds while developing innovative products for their customers. +VMware uses Bazel to produce deterministic, reliable builds while developing +innovative products for their customers. ### [Wix](https://www.wix.com/) -Wix is a cloud-based web development platform. Their backend uses Java and Scala code. They use remote execution with Google Cloud Build. +Wix is a cloud-based web development platform. Their backend uses Java and Scala +code. They use remote execution with Google Cloud Build. -> We have seen about 5 times faster clean builds when running with bazel remote execution which utilizes bazel’s great build/test parallelism capabilities when it dispatches build/test actions to a worker farm. Average build times are more than 10 times faster due to the utilization of bazel’s aggressive caching mechanism. ([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) +> We have seen about 5 times faster clean builds when running with bazel remote +execution which utilizes bazel’s great build/test parallelism capabilities when +it dispatches build/test actions to a worker farm. Average build times are more +than 10 times faster due to the utilization of bazel’s aggressive caching +mechanism. +([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) ### [Zenly](https://zen.ly/) -Zenly is a live map of your friends and family. It’s the most fun way to meet up — or just see what’s up! — so you can feel together, even when you're apart. +Zenly is a live map of your friends and family. It’s the most fun way to meet up +— or just see what’s up! — so you can feel together, even when you're apart. + *** @@ -324,33 +483,44 @@ Zenly is a live map of your friends and family. It’s the most fun way to meet ### [Abseil](https://abseil.io/) -Abseil is an open-source collection of C++ code (compliant to C++11) designed to augment the C++ standard library. +Abseil is an open-source collection of C++ code (compliant to C++11) designed +to augment the C++ standard library. ### [Angular](https://angular.io) -![](https://upload.wikimedia.org/wikipedia/commons/c/cf/Angular_full_color_logo.svg) + -Angular is a popular web framework. Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). +Angular is a popular web framework. +Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). ### [Apollo](https://github.com/ApolloAuto/apollo) -Apollo is a high performance, flexible architecture which accelerates the development, testing, and deployment of Autonomous Vehicles. +Apollo is a high performance, flexible architecture which accelerates the +development, testing, and deployment of Autonomous Vehicles. ### [brpc](https://github.com/brpc/brpc) -An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ instances(not counting clients) and thousands kinds of services, called "baidu-rpc" inside Baidu. +An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ +instances(not counting clients) and thousands kinds of services, called +"baidu-rpc" inside Baidu. ### [cert-manager](https://github.com/jetstack/cert-manager) -cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry. +cert-manager is a Kubernetes add-on to automate the management and issuance of +TLS certificates from various issuing sources. It will ensure certificates are +valid and up to date periodically, and attempt to renew certificates at an +appropriate time before expiry. ### [CallBuilder](https://github.com/google/CallBuilder) -A Java code generator that allows you to create a builder by writing one function. +A Java code generator that allows you to create a builder by writing one +function. ### [CPPItertools](https://github.com/ryanhaining/cppitertools) -C++ library providing range-based for loop add-ons inspired by the Python builtins and itertools library. Like itertools and the Python3 builtins, this library uses lazy evaluation wherever possible. +C++ library providing range-based for loop add-ons inspired by the Python +builtins and itertools library. Like itertools and the Python3 builtins, this +library uses lazy evaluation wherever possible. ### [Copybara](https://github.com/google/copybara) @@ -358,11 +528,13 @@ Copybara is a tool for transforming and moving code between repositories. ### [Dagger](https://google.github.io/dagger/) -Dagger is a fully static, compile-time dependency injection framework for both Java and Android. +Dagger is a fully static, compile-time dependency injection framework for both +Java and Android. ### [DAML](https://github.com/digital-asset/daml) -DAML is a smart contract language for building future-proof distributed applications on a safe, privacy-aware runtime. +DAML is a smart contract language for building future-proof distributed +applications on a safe, privacy-aware runtime. ### [DeepMind Lab](https://github.com/deepmind/lab) @@ -370,7 +542,10 @@ A customisable 3D platform for agent-based AI research. ### [Drake](https://github.com/RobotLocomotion/drake) -Drake is a C++ toolbox started at MIT and now led by the Toyota Research Institute. It is a collection of tools for analyzing the dynamics of our robots and building control systems for them, with a heavy emphasis on optimization-based design/analysis. +Drake is a C++ toolbox started at MIT and now led by the Toyota Research +Institute. It is a collection of tools for analyzing the dynamics of our robots +and building control systems for them, with a heavy emphasis on +optimization-based design/analysis. ### [Envoy](https://github.com/lyft/envoy) @@ -378,15 +553,19 @@ C++ L7 proxy and communication bus ### [Error Prone](https://github.com/google/error-prone) -Catches common Java mistakes as compile-time errors. (Migration to Bazel is in progress.) +Catches common Java mistakes as compile-time errors. (Migration to Bazel is in +progress.) ### [Extensible Service Proxy](https://github.com/cloudendpoints/esp) -Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management capabilities for JSON/REST or gRPC API services. The current implementation is based on an NGINX HTTP reverse proxy server. +Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management +capabilities for JSON/REST or gRPC API services. The current implementation is +based on an NGINX HTTP reverse proxy server. ### [FFruit](https://gitlab.com/perezd/ffruit/) -FFruit is a free & open source Android application to the popular service [Falling Fruit](https://fallingfruit.org). +FFruit is a free & open source Android application to the popular service +[Falling Fruit](https://fallingfruit.org). ### [Gerrit Code Review](https://gerritcodereview.com) @@ -398,51 +577,61 @@ Gitiles is a simple repository browser for Git repositories, built on JGit. ### [Grakn](https://github.com/graknlabs/grakn) -Grakn ([https://grakn.ai/](https://grakn.ai/)) is the knowledge graph engine to organise complex networks of data and make it queryable. +Grakn (https://grakn.ai/) is the knowledge graph engine to organise complex +networks of data and make it queryable. ### [GRPC](http://www.grpc.io) - -A language-and-platform-neutral remote procedure call system. (Bazel is a supported, although not primary, build system.) +A language-and-platform-neutral remote procedure call system. +(Bazel is a supported, although not primary, build system.) ### [gVisor](https://github.com/google/gvisor) - gVisor is a container runtime sandbox. ### [Guetzli](https://github.com/google/guetzli/) -Guetzli is a JPEG encoder that aims for excellent compression density at high visual quality. +Guetzli is a JPEG encoder that aims for excellent compression density at high +visual quality. ### [Gulava](http://www.github.com/google/gulava/) -A Java code generator that lets you write Prolog-style predicates and use them seamlessly from normal Java code. +A Java code generator that lets you write Prolog-style predicates and use them +seamlessly from normal Java code. ### [Heron](https://github.com/apache/incubator-heron) -Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter. +Heron is a realtime, distributed, fault-tolerant stream processing engine from +Twitter. ### [Internet Computer Protocol](https://internetcomputer.org/) -![](https://internetcomputer.org/img/IC_logo_horizontal_white.svg) + -The Internet Computer Protocol is a publicly available blockchain network that enables replicated execution of general-purpose computation, serving hundreds of thousands of applications and their users. +The Internet Computer Protocol is a publicly available blockchain network that +enables replicated execution of general-purpose computation, serving hundreds +of thousands of applications and their users. ### [Jazzer](https://github.com/CodeIntelligenceTesting/jazzer) -![](https://www.code-intelligence.com/hubfs/Logos/CI%20Logos/Jazzer_einfach.png) + Jazzer is a fuzzer for Java and other JVM-based languages that integrates with JUnit 5. ### [JGit](https://eclipse.org/jgit/) -JGit is a lightweight, pure Java library implementing the Git version control system. +JGit is a lightweight, pure Java library implementing the Git version control +system. ### [Jsonnet](https://jsonnet.org/) -An elegant, formally-specified config generation language for JSON. (Bazel is a supported build system.) +An elegant, formally-specified config generation language for JSON. +(Bazel is a supported build system.) ### [Kubernetes](https://github.com/kubernetes/kubernetes) -![](https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.png) Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. + +Kubernetes is an open source system for managing containerized applications +across multiple hosts, providing basic mechanisms for deployment, maintenance, +and scaling of applications. ### [Kythe](https://github.com/google/kythe) @@ -450,9 +639,10 @@ An ecosystem for building tools that work with code. ### [ls-lint](https://github.com/loeffel-io/ls-lint) -![](https://raw.githubusercontent.com/loeffel-io/ls-lint/master/assets/logo/ls-lint.png) + -An extremely fast directory and filename linter - Bring some structure to your project file system. +An extremely fast directory and filename linter - Bring some structure to your +project file system. ### [Nomulus](https://github.com/google/nomulus) @@ -460,11 +650,19 @@ Top-level domain name registry service on Google App Engine. ### [ONOS : Open Network Operating System](https://github.com/opennetworkinglab/onos) -![](https://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Logo_for_the_ONOS_open_source_project.png/175px-Logo_for_the_ONOS_open_source_project.png) ONOS is the only SDN controller platform that supports the transition from legacy “brown field” networks to SDN “green field” networks. This enables exciting new capabilities, and disruptive deployment and operational cost points for network operators. + +ONOS is the only SDN controller platform that supports the transition from +legacy “brown field” networks to SDN “green field” networks. This enables +exciting new capabilities, and disruptive deployment and operational cost points +for network operators. ### [PetitParser for Java](https://github.com/petitparser/java-petitparser) -Grammars for programming languages are traditionally specified statically. They are hard to compose and reuse due to ambiguities that inevitably arise. PetitParser combines ideas from scannnerless parsing, parser combinators, parsing expression grammars and packrat parsers to model grammars and parsers as objects that can be reconfigured dynamically. +Grammars for programming languages are traditionally specified statically. +They are hard to compose and reuse due to ambiguities that inevitably arise. +PetitParser combines ideas from scannnerless parsing, parser combinators, +parsing expression grammars and packrat parsers to model grammars and parsers +as objects that can be reconfigured dynamically. ### [PlaidML](https://github.com/plaidml/plaidml) @@ -472,11 +670,14 @@ PlaidML is a framework for making deep learning work everywhere. ### [Project V](https://www.v2ray.com/) -![](https://www.v2ray.com/resources/v2ray_1024.png) Project V is a set of tools to help you build your own privacy network over internet. + +Project V is a set of tools to help you build your own privacy network over +internet. ### [Prysmatic Labs Ethereum 2.0 Implementation](https://github.com/prysmaticlabs/prysm) -Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed computing platform. +Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed +computing platform. ### [Ray](https://github.com/ray-project/ray) @@ -484,7 +685,8 @@ Ray is a flexible, high-performance distributed execution framework. ### [Resty](https://github.com/go-resty/resty) -Resty is a Simple HTTP and REST client library for Go (inspired by Ruby rest-client). +Resty is a Simple HTTP and REST client library for Go (inspired by Ruby +rest-client). ### [Roughtime](https://roughtime.googlesource.com/roughtime) @@ -496,7 +698,9 @@ Selenium is a portable framework for testing web applications. ### [Semantic](https://github.com/github/semantic) -Semantic is a Haskell library and command line tool for parsing, analyzing, and comparing source code. It is developed by GitHub (and used for example for the code navigation). +Semantic is a Haskell library and command line tool for parsing, analyzing, and +comparing source code. It is developed by GitHub (and used for example for the +code navigation). ### [Served](https://github.com/meltwater/served) @@ -504,11 +708,13 @@ Served is a C++ library for building high performance RESTful web servers. ### [Sonnet](https://github.com/deepmind/sonnet) -Sonnet is a library built on top of TensorFlow for building complex neural networks. +Sonnet is a library built on top of TensorFlow for building complex neural +networks. ### [Sorbet](https://github.com/sorbet/sorbet) -Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to codebases with millions of lines of code and can be adopted incrementally. +Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to +codebases with millions of lines of code and can be adopted incrementally. ### [Spotify](https://spotify.com) @@ -516,11 +722,12 @@ Spotify is using Bazel to build their iOS and Android Apps ([source](https://twi ### [Tink](https://github.com/google/tink) -Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. +Tink is a multi-language, cross-platform, open source library that provides +cryptographic APIs that are secure, easy to use correctly, and hard(er) to +misuse. ### [TensorFlow](http://tensorflow.org) - -![](https://upload.wikimedia.org/wikipedia/commons/a/a4/TensorFlowLogo.png) + An open source software library for machine intelligence. @@ -534,8 +741,10 @@ Project Wycheproof tests crypto libraries against known attacks. ### [XIOSim](https://github.com/s-kanev/XIOSim) -XIOSim is a detailed user-mode microarchitectural simulator for the x86 architecture. +XIOSim is a detailed user-mode microarchitectural simulator for the x86 +architecture. ### [ZhihuDailyPurify](https://github.com/izzyleung/ZhihuDailyPurify) -ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese question-and-answer webs. +ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese +question-and-answer webs. diff --git a/concepts/build-ref.mdx b/concepts/build-ref.mdx index 33c1e76c..e8839d40 100644 --- a/concepts/build-ref.mdx +++ b/concepts/build-ref.mdx @@ -2,27 +2,51 @@ title: 'Repositories, workspaces, packages, and targets' --- -Bazel builds software from source code organized in directory trees called repositories. A defined set of repositories comprises the workspace. Source files in repositories are organized in a nested hierarchy of packages, where each package is a directory that contains a set of related source files and one `BUILD` file. The `BUILD` file specifies what software outputs can be built from the source. + + +Bazel builds software from source code organized in directory trees called +repositories. A defined set of repositories comprises the workspace. Source +files in repositories are organized in a nested hierarchy of packages, where +each package is a directory that contains a set of related source files and one +`BUILD` file. The `BUILD` file specifies what software outputs can be built from +the source. ### Repositories -Source files used in a Bazel build are organized in *repositories* (often shortened to *repos*). A repo is a directory tree with a boundary marker file at its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. +Source files used in a Bazel build are organized in _repositories_ (often +shortened to _repos_). A repo is a directory tree with a boundary marker file at +its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or +in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. -The repo in which the current Bazel command is being run is called the *main repo*. Other, (external) repos are defined by *repo rules*; see [external dependencies overview](/external/overview) for more information. +The repo in which the current Bazel command is being run is called the _main +repo_. Other, (external) repos are defined by _repo rules_; see [external +dependencies overview](/external/overview) for more information. ## Workspace -A *workspace* is the environment shared by all Bazel commands run from the same main repo. It encompasses the main repo and the set of all defined external repos. +A _workspace_ is the environment shared by all Bazel commands run from the same +main repo. It encompasses the main repo and the set of all defined external +repos. -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". ## Packages -The primary unit of code organization in a repository is the *package*. A package is a collection of related files and a specification of how they can be used to produce output artifacts. +The primary unit of code organization in a repository is the _package_. A +package is a collection of related files and a specification of how they can be +used to produce output artifacts. -A package is defined as a directory containing a [`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A package includes all files in its directory, plus all subdirectories beneath it, except those which themselves contain a `BUILD` file. From this definition, no file or directory may be a part of two different packages. +A package is defined as a directory containing a +[`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A +package includes all files in its directory, plus all subdirectories beneath it, +except those which themselves contain a `BUILD` file. From this definition, no +file or directory may be a part of two different packages. -For example, in the following directory tree there are two packages, `my/app`, and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but a directory belonging to package `my/app`. +For example, in the following directory tree there are two packages, `my/app`, +and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but +a directory belonging to package `my/app`. ``` src/my/app/BUILD @@ -34,18 +58,48 @@ src/my/app/tests/test.cc ## Targets -A package is a container of *targets*, which are defined in the package's `BUILD` file. Most targets are one of two principal kinds, *files* and *rules*. - -Files are further divided into two kinds. *Source files* are usually written by the efforts of people, and checked in to the repository. *Generated files*, sometimes called derived files or output files, are not checked in, but are generated from source files. - -The second kind of target is declared with a *rule*. Each rule instance specifies the relationship between a set of input and a set of output files. The inputs to a rule may be source files, but they also may be the outputs of other rules. - -Whether the input to a rule is a source file or a generated file is in most cases immaterial; what matters is only the contents of that file. This fact makes it easy to replace a complex source file with a generated file produced by a rule, such as happens when the burden of manually maintaining a highly structured file becomes too tiresome, and someone writes a program to derive it. No change is required to the consumers of that file. Conversely, a generated file may easily be replaced by a source file with only local changes. - -The inputs to a rule may also include *other rules*. The precise meaning of such relationships is often quite complex and language- or rule-dependent, but intuitively it is simple: a C++ library rule A might have another C++ library rule B for an input. The effect of this dependency is that B's header files are available to A during compilation, B's symbols are available to A during linking, and B's runtime data is available to A during execution. - -An invariant of all rules is that the files generated by a rule always belong to the same package as the rule itself; it is not possible to generate files into another package. It is not uncommon for a rule's inputs to come from another package, though. - -Package groups are sets of packages whose purpose is to limit accessibility of certain rules. Package groups are defined by the `package_group` function. They have three properties: the list of packages they contain, their name, and other package groups they include. The only allowed ways to refer to them are from the `visibility` attribute of rules or from the `default_visibility` attribute of the `package` function; they do not generate or consume files. For more information, refer to the [`package_group` documentation](/reference/be/functions#package_group). - -[Labels→](/concepts/labels) +A package is a container of _targets_, which are defined in the package's +`BUILD` file. Most targets are one of two principal kinds, _files_ and _rules_. + +Files are further divided into two kinds. _Source files_ are usually written by +the efforts of people, and checked in to the repository. _Generated files_, +sometimes called derived files or output files, are not checked in, but are +generated from source files. + +The second kind of target is declared with a _rule_. Each rule instance +specifies the relationship between a set of input and a set of output files. The +inputs to a rule may be source files, but they also may be the outputs of other +rules. + +Whether the input to a rule is a source file or a generated file is in most +cases immaterial; what matters is only the contents of that file. This fact +makes it easy to replace a complex source file with a generated file produced by +a rule, such as happens when the burden of manually maintaining a highly +structured file becomes too tiresome, and someone writes a program to derive it. +No change is required to the consumers of that file. Conversely, a generated +file may easily be replaced by a source file with only local changes. + +The inputs to a rule may also include _other rules_. The precise meaning of such +relationships is often quite complex and language- or rule-dependent, but +intuitively it is simple: a C++ library rule A might have another C++ library +rule B for an input. The effect of this dependency is that B's header files are +available to A during compilation, B's symbols are available to A during +linking, and B's runtime data is available to A during execution. + +An invariant of all rules is that the files generated by a rule always belong to +the same package as the rule itself; it is not possible to generate files into +another package. It is not uncommon for a rule's inputs to come from another +package, though. + +Package groups are sets of packages whose purpose is to limit accessibility of +certain rules. Package groups are defined by the `package_group` function. They +have three properties: the list of packages they contain, their name, and other +package groups they include. The only allowed ways to refer to them are from the +`visibility` attribute of rules or from the `default_visibility` attribute of +the `package` function; they do not generate or consume files. For more +information, refer to the [`package_group` +documentation](/reference/be/functions#package_group). + + + Labels + diff --git a/concepts/platforms.mdx b/concepts/platforms.mdx index e2ecbde8..e560ea4d 100644 --- a/concepts/platforms.mdx +++ b/concepts/platforms.mdx @@ -2,23 +2,30 @@ title: 'Migrating to Platforms' --- -Bazel has sophisticated [support](#background) for modeling [platforms](/extending/platforms) and [toolchains](/extending/toolchains) for multi-architecture and cross-compiled builds. + + +Bazel has sophisticated [support](#background) for modeling +[platforms][Platforms] and [toolchains][Toolchains] for multi-architecture and +cross-compiled builds. This page summarizes the state of this support. -Key Point: Bazel's platform and toolchain APIs are available today. Not all languages support them. Use these APIs with your project if you can. Bazel is migrating all major languages so eventually all builds will be platform-based. +Key Point: Bazel's platform and toolchain APIs are available today. Not all +languages support them. Use these APIs with your project if you can. Bazel is +migrating all major languages so eventually all builds will be platform-based. See also: -- [Platforms](/extending/platforms) -- [Toolchains](/extending/toolchains) -- [Background](#background) +* [Platforms][Platforms] +* [Toolchains][Toolchains] +* [Background][Background] ## Status ### C++ -C++ rules use platforms to select toolchains when `--incompatible_enable_cc_toolchain_resolution` is set. +C++ rules use platforms to select toolchains when +`--incompatible_enable_cc_toolchain_resolution` is set. This means you can configure a C++ project with: @@ -34,19 +41,23 @@ bazel build //:my_cpp_project` --cpu=... --crosstool_top=... --compiler=... This will be enabled by default in Bazel 7.0 ([#7260](https://github.com/bazelbuild/bazel/issues/7260)). -To test your C++ project with platforms, see [Migrating Your Project](#migrating-your-project) and [Configuring C++ toolchains](/tutorials/ccp-toolchain-config). +To test your C++ project with platforms, see +[Migrating Your Project](#migrating-your-project) and +[Configuring C++ toolchains]. ### Java Java rules use platforms to select toolchains. -This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, `--javabase`, and `--host_javabase`. +This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, +`--javabase`, and `--host_javabase`. See [Java and Bazel](/docs/bazel-and-java) for details. ### Android -Android rules use platforms to select toolchains when `--incompatible_enable_android_toolchain_resolution` is set. +Android rules use platforms to select toolchains when +`--incompatible_enable_android_toolchain_resolution` is set. This means you can configure an Android project with: @@ -54,42 +65,63 @@ This means you can configure an Android project with: bazel build //:my_android_project --android_platforms=//:my_android_platform ``` -instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, and `--fat_apk_cpu`. +instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, +and `--fat_apk_cpu`. This will be enabled by default in Bazel 7.0 ([#16285](https://github.com/bazelbuild/bazel/issues/16285)). -To test your Android project with platforms, see [Migrating Your Project](#migrating-your-project). +To test your Android project with platforms, see +[Migrating Your Project](#migrating-your-project). ### Apple -[Apple rules](https://github.com/bazelbuild/rules_apple) do not support platforms and are not yet scheduled for support. +[Apple rules] do not support platforms and are not yet scheduled +for support. -You can still use platform APIs with Apple builds (for example, when building with a mixture of Apple rules and pure C++) with [platform mappings](#platform-mappings). +You can still use platform APIs with Apple builds (for example, when building +with a mixture of Apple rules and pure C++) with [platform +mappings](#platform-mappings). ### Other languages -- [Go rules](https://github.com/bazelbuild/rules_go) fully support platforms -- [Rust rules](https://github.com/bazelbuild/rules_rust) fully support platforms. +* [Go rules] fully support platforms +* [Rust rules] fully support platforms. -If you own a language rule set, see [Migrating your rule set](#migrating-your-rule-set) for adding support. +If you own a language rule set, see [Migrating your rule set] for adding +support. ## Background -*Platforms* and *toolchains* were introduced to standardize how software projects target different architectures and cross-compile. +*Platforms* and *toolchains* were introduced to standardize how software +projects target different architectures and cross-compile. -This was [inspired](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) by the observation that language maintainers were already doing this in ad hoc, incompatible ways. For example, C++ rules used `--cpu` and `--crosstool_top` to declare a target CPU and toolchain. Neither of these correctly models a "platform". This produced awkward and incorrect builds. +This was +[inspired][Inspiration] +by the observation that language maintainers were already doing this in ad +hoc, incompatible ways. For example, C++ rules used `--cpu` and + `--crosstool_top` to declare a target CPU and toolchain. Neither of these +correctly models a "platform". This produced awkward and incorrect builds. -Java, Android, and other languages evolved their own flags for similar purposes, none of which interoperated with each other. This made cross-language builds confusing and complicated. +Java, Android, and other languages evolved their own flags for similar purposes, +none of which interoperated with each other. This made cross-language builds +confusing and complicated. -Bazel is intended for large, multi-language, multi-platform projects. This demands more principled support for these concepts, including a clear standard API. +Bazel is intended for large, multi-language, multi-platform projects. This +demands more principled support for these concepts, including a clear +standard API. ### Need for migration -Upgrading to the new API requires two efforts: releasing the API and upgrading rule logic to use it. +Upgrading to the new API requires two efforts: releasing the API and upgrading +rule logic to use it. -The first is done but the second is ongoing. This consists of ensuring language-specific platforms and toolchains are defined, language logic reads toolchains through the new API instead of old flags like `--crosstool_top`, and `config_setting`s select on the new API instead of old flags. +The first is done but the second is ongoing. This consists of ensuring +language-specific platforms and toolchains are defined, language logic reads +toolchains through the new API instead of old flags like `--crosstool_top`, and +`config_setting`s select on the new API instead of old flags. -This work is straightforward but requires a distinct effort for each language, plus fair warning for project owners to test against upcoming changes. +This work is straightforward but requires a distinct effort for each language, +plus fair warning for project owners to test against upcoming changes. This is why this is an ongoing migration. @@ -104,18 +136,25 @@ bazel build //:myproject --platforms=//:myplatform This implies: 1. Your project's rules choose the right toolchains for `//:myplatform`. -2. Your project's dependencies choose the right toolchains for `//:myplatform`. -3. `//:myplatform` references [common declarations](https://github.com/bazelbuild/platforms) of `CPU`, `OS`, and other generic, language-independent properties -4. All relevant [`select()`s](/docs/configurable-attributes) properly match `//:myplatform`. -5. `//:myplatform` is defined in a clear, accessible place: in your project's repo if the platform is unique to your project, or some common place all consuming projects can find it - -Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be deprecated and removed as soon as it's safe to do so. +1. Your project's dependencies choose the right toolchains for `//:myplatform`. +1. `//:myplatform` references +[common declarations][Common Platform Declarations] +of `CPU`, `OS`, and other generic, language-independent properties +1. All relevant [`select()`s][select()] properly match `//:myplatform`. +1. `//:myplatform` is defined in a clear, accessible place: in your project's +repo if the platform is unique to your project, or some common place all +consuming projects can find it + +Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be +deprecated and removed as soon as it's safe to do so. Ultimately, this will be the *sole* way to configure architectures. + ## Migrating your project -If you build with languages that support platforms, your build should already work with an invocation like: +If you build with languages that support platforms, your build should already +work with an invocation like: ```posix-terminal bazel build //:myproject --platforms=//:myplatform @@ -123,29 +162,49 @@ bazel build //:myproject --platforms=//:myplatform See [Status](#status) and your language's documentation for precise details. -If a language requires a flag to enable platform support, you also need to set that flag. See [Status](#status) for details. +If a language requires a flag to enable platform support, you also need to set +that flag. See [Status](#status) for details. For your project to build, you need to check the following: -1. `//:myplatform` must exist. It's generally the project owner's responsibility to define platforms because different projects target different machines. See [Default platforms](#default-platforms). +1. `//:myplatform` must exist. It's generally the project owner's responsibility + to define platforms because different projects target different machines. + See [Default platforms](#default-platforms). -2. The toolchains you want to use must exist. If using stock toolchains, the language owners should include instructions for how to register them. If writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). +1. The toolchains you want to use must exist. If using stock toolchains, the + language owners should include instructions for how to register them. If + writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your + `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). -3. `select()`s and [configuration transitions](/extending/config#user-defined-transitions) must resolve properly. See [select()](#select) and [Transitions](#transitions). +1. `select()`s and [configuration transitions][Starlark transitions] must + resolve properly. See [select()](#select) and [Transitions](#transitions). -4. If your build mixes languages that do and don't support platforms, you may need platform mappings to help the legacy languages work with the new API. See [Platform mappings](#platform-mappings) for details. +1. If your build mixes languages that do and don't support platforms, you may + need platform mappings to help the legacy languages work with the new API. + See [Platform mappings](#platform-mappings) for details. If you still have problems, [reach out](#questions) for support. ### Default platforms -Project owners should define explicit [platforms](/extending/platforms#constraints-platforms) to describe the architectures they want to build for. These are then triggered with `--platforms`. +Project owners should define explicit +[platforms][Defining Constraints and Platforms] to describe the architectures +they want to build for. These are then triggered with `--platforms`. -When `--platforms` isn't set, Bazel defaults to a `platform` representing the local build machine. This is auto-generated at `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`) so there's no need to explicitly define it. It maps the local machine's `OS` and `CPU` with `constraint_value`s declared in [`@platforms`](https://github.com/bazelbuild/platforms). +When `--platforms` isn't set, Bazel defaults to a `platform` representing the +local build machine. This is auto-generated at `@platforms//host` (aliased as +`@bazel_tools//tools:host_platform`) +so there's no need to explicitly define it. It maps the local machine's `OS` +and `CPU` with `constraint_value`s declared in +[`@platforms`](https://github.com/bazelbuild/platforms). ### `select()` -Projects can [`select()`](/docs/configurable-attributes) on [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value) but not complete platforms. This is intentional so `select()` supports as wide a variety of machines as possible. A library with `ARM`-specific sources should support *all* `ARM`-powered machines unless there's reason to be more specific. +Projects can [`select()`][select()] on +[`constraint_value` targets][constraint_value Rule] but not complete +platforms. This is intentional so `select()` supports as wide a variety of +machines as possible. A library with `ARM`-specific sources should support *all* +`ARM`-powered machines unless there's reason to be more specific. To select on one or more `constraint_value`s, use: @@ -169,47 +228,75 @@ config_setting( ) ``` -More details [here](/docs/configurable-attributes#platforms). +More details [here][select() Platforms]. -`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. When migrating your project to platforms, you must either convert them to `constraint_values` or use [platform mappings](#platform-mappings) to support both styles during migration. +`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. +When migrating your project to platforms, you must either convert them to +`constraint_values` or use [platform mappings](#platform-mappings) to support +both styles during migration. ### Transitions -[Starlark transitions](/extending/config#user-defined-transitions) change flags down parts of your build graph. If your project uses a transition that sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read `--platforms` won't see these changes. +[Starlark transitions][Starlark transitions] change +flags down parts of your build graph. If your project uses a transition that +sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read +`--platforms` won't see these changes. -When migrating your project to platforms, you must either convert changes like `return { "//command_line_option:cpu": "arm" }` to `return { "//command_line_option:platforms": "//:my_arm_platform" }` or use [platform mappings](#platform-mappings) to support both styles during migration. window. +When migrating your project to platforms, you must either convert changes like +`return { "//command_line_option:cpu": "arm" }` to `return { +"//command_line_option:platforms": "//:my_arm_platform" }` or use [platform +mappings](#platform-mappings) to support both styles during migration. +window. ## Migrating your rule set If you own a rule set and want to support platforms, you need to: -1. Have rule logic resolve toolchains with the toolchain API. See [toolchain API](/extending/toolchains) (`ctx.toolchains`). +1. Have rule logic resolve toolchains with the toolchain API. See + [toolchain API][Toolchains] (`ctx.toolchains`). -2. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so rule logic alternately resolves toolchains through the new API or old flags like `--crosstool_top` during migration testing. +1. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so + rule logic alternately resolves toolchains through the new API or old flags + like `--crosstool_top` during migration testing. -3. Define the relevant properties that make up platform components. See [Common platform properties](#common-platform-properties) +1. Define the relevant properties that make up platform components. See + [Common platform properties](#common-platform-properties) -4. Define standard toolchains and make them accessible to users through your rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) +1. Define standard toolchains and make them accessible to users through your + rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) -5. Ensure [`select()`s](#select) and [configuration transitions](#transitions) support platforms. This is the biggest challenge. It's particularly challenging for multi-language projects (which may fail if *all* languages can't read `--platforms`). +1. Ensure [`select()`s](#select) and + [configuration transitions](#transitions) support platforms. This is the + biggest challenge. It's particularly challenging for multi-language projects + (which may fail if *all* languages can't read `--platforms`). -If you need to mix with rules that don't support platforms, you may need [platform mappings](#platform-mappings) to bridge the gap. +If you need to mix with rules that don't support platforms, you may need +[platform mappings](#platform-mappings) to bridge the gap. ### Common platform properties -Common, cross-language platform properties like `OS` and `CPU` should be declared in [`@platforms`](https://github.com/bazelbuild/platforms). This encourages sharing, standardization, and cross-language compatibility. +Common, cross-language platform properties like `OS` and `CPU` should be +declared in [`@platforms`](https://github.com/bazelbuild/platforms). +This encourages sharing, standardization, and cross-language compatibility. -Properties unique to your rules should be declared in your rule's repo. This lets you maintain clear ownership over the specific concepts your rules are responsible for. +Properties unique to your rules should be declared in your rule's repo. This +lets you maintain clear ownership over the specific concepts your rules are +responsible for. -If your rules use custom-purpose OSes or CPUs, these should be declared in your rule's repo vs. [`@platforms`](https://github.com/bazelbuild/platforms). +If your rules use custom-purpose OSes or CPUs, these should be declared in your +rule's repo vs. +[`@platforms`](https://github.com/bazelbuild/platforms). ## Platform mappings -*Platform mappings* is a temporary API that lets platform-aware logic mix with legacy logic in the same build. This is a blunt tool that's only intended to smooth incompatibilities with different migration timeframes. +*Platform mappings* is a temporary API that lets platform-aware logic mix with +legacy logic in the same build. This is a blunt tool that's only intended to +smooth incompatibilities with different migration timeframes. -Caution: Only use this if necessary, and expect to eventually eliminate it. +Caution: Only use this if necessary, and expect to eventually eliminate it. -A platform mapping is a map of either a `platform()` to a corresponding set of legacy flags or the reverse. For example: +A platform mapping is a map of either a `platform()` to a +corresponding set of legacy flags or the reverse. For example: ```python platforms: @@ -230,15 +317,20 @@ flags: //platforms:macos ``` -Bazel uses this to guarantee all settings, both platform-based and legacy, are consistently applied throughout the build, including through [transitions](#transitions). +Bazel uses this to guarantee all settings, both platform-based and +legacy, are consistently applied throughout the build, including through +[transitions](#transitions). -By default Bazel reads mappings from the `platform_mappings` file in your workspace root. You can also set `--platform_mappings=//:my_custom_mapping`. +By default Bazel reads mappings from the `platform_mappings` file in your +workspace root. You can also set +`--platform_mappings=//:my_custom_mapping`. -See the [platform mappings design](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit) for details. +See the [platform mappings design] for details. ## API review -A [`platform`](/reference/be/platforms-and-toolchains#platform) is a collection of [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value): +A [`platform`][platform Rule] is a collection of +[`constraint_value` targets][constraint_value Rule]: ```python platform( @@ -250,7 +342,9 @@ platform( ) ``` -A [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) is a machine property. Values of the same "kind" are grouped under a common [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting): +A [`constraint_value`][constraint_value Rule] is a machine +property. Values of the same "kind" are grouped under a common +[`constraint_setting`][constraint_setting Rule]: ```python constraint_setting(name = "os") @@ -264,27 +358,72 @@ constraint_value( ) ``` -A [`toolchain`](/extending/toolchains) is a [Starlark rule](/extending/rules). Its attributes declare a language's tools (like `compiler = "//mytoolchain:custom_gcc"`). Its [providers](/extending/rules#providers) pass this information to rules that need to build with these tools. +A [`toolchain`][Toolchains] is a [Starlark rule][Starlark rule]. Its +attributes declare a language's tools (like `compiler = +"//mytoolchain:custom_gcc"`). Its [providers][Starlark Provider] pass +this information to rules that need to build with these tools. -Toolchains declare the `constraint_value`s of machines they can [target](/reference/be/platforms-and-toolchains#toolchain.target_compatible_with) (`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can [run on](/reference/be/platforms-and-toolchains#toolchain.exec_compatible_with) (`exec_compatible_with = ["@platforms//os:mac"]`). +Toolchains declare the `constraint_value`s of machines they can +[target][target_compatible_with Attribute] +(`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can +[run on][exec_compatible_with Attribute] +(`exec_compatible_with = ["@platforms//os:mac"]`). -When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel automatically selects a toolchain that can run on the build machine and build binaries for `//:myplatform`. This is known as *toolchain resolution*. +When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel +automatically selects a toolchain that can run on the build machine and +build binaries for `//:myplatform`. This is known as *toolchain resolution*. -The set of available toolchains can be registered in the `MODULE.bazel` file with [`register_toolchains`](/rules/lib/globals/module#register_toolchains) or at the command line with [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). +The set of available toolchains can be registered in the `MODULE.bazel` file +with [`register_toolchains`][register_toolchains Function] or at the +command line with [`--extra_toolchains`][extra_toolchains Flag]. -For more information see [here](/extending/toolchains). +For more information see [here][Toolchains]. ## Questions -For general support and questions about the migration timeline, contact [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) or the owners of the appropriate rules. +For general support and questions about the migration timeline, contact +[bazel-discuss] or the owners of the appropriate rules. -For discussions on the design and evolution of the platform/toolchain APIs, contact [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev). +For discussions on the design and evolution of the platform/toolchain APIs, +contact [bazel-dev]. ## See also -- [Configurable Builds - Part 1](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) -- [Platforms](/extending/platforms) -- [Toolchains](/extending/toolchains) -- [Bazel Platforms Cookbook](https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/) -- [Platforms examples](https://github.com/hlopko/bazel_platforms_examples) -- [Example C++ toolchain](https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms) +* [Configurable Builds - Part 1] +* [Platforms] +* [Toolchains] +* [Bazel Platforms Cookbook] +* [Platforms examples] +* [Example C++ toolchain] + +[Android Rules]: /docs/bazel-and-android +[Apple Rules]: https://github.com/bazelbuild/rules_apple +[Background]: #background +[Bazel platforms Cookbook]: https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/ +[bazel-dev]: https://groups.google.com/forum/#!forum/bazel-dev +[bazel-discuss]: https://groups.google.com/forum/#!forum/bazel-discuss +[Common Platform Declarations]: https://github.com/bazelbuild/platforms +[constraint_setting Rule]: /reference/be/platforms-and-toolchains#constraint_setting +[constraint_value Rule]: /reference/be/platforms-and-toolchains#constraint_value +[Configurable Builds - Part 1]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html +[Configuring C++ toolchains]: /tutorials/ccp-toolchain-config +[Defining Constraints and Platforms]: /extending/platforms#constraints-platforms +[Example C++ toolchain]: https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms +[exec_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.exec_compatible_with +[extra_toolchains Flag]: /reference/command-line-reference#flag--extra_toolchains +[Go Rules]: https://github.com/bazelbuild/rules_go +[Inspiration]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html +[Migrating your rule set]: #migrating-your-rule-set +[Platforms]: /extending/platforms +[Platforms examples]: https://github.com/hlopko/bazel_platforms_examples +[platform mappings design]: https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit +[platform Rule]: /reference/be/platforms-and-toolchains#platform +[register_toolchains Function]: /rules/lib/globals/module#register_toolchains +[Rust rules]: https://github.com/bazelbuild/rules_rust +[select()]: /docs/configurable-attributes +[select() Platforms]: /docs/configurable-attributes#platforms +[Starlark provider]: /extending/rules#providers +[Starlark rule]: /extending/rules +[Starlark transitions]: /extending/config#user-defined-transitions +[target_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.target_compatible_with +[Toolchains]: /extending/toolchains diff --git a/concepts/visibility.mdx b/concepts/visibility.mdx index 10fd9eec..982f0a0e 100644 --- a/concepts/visibility.mdx +++ b/concepts/visibility.mdx @@ -2,49 +2,97 @@ title: 'Visibility' --- -This page covers Bazel's two visibility systems: [target visibility](#target-visibility) and [load visibility](#load-visibility). -Both types of visibility help other developers distinguish between your library's public API and its implementation details, and help enforce structure as your workspace grows. You can also use visibility when deprecating a public API to allow current users while denying new ones. + +This page covers Bazel's two visibility systems: +[target visibility](#target-visibility) and [load visibility](#load-visibility). + +Both types of visibility help other developers distinguish between your +library's public API and its implementation details, and help enforce structure +as your workspace grows. You can also use visibility when deprecating a public +API to allow current users while denying new ones. ## Target visibility -**Target visibility** controls who may depend on your target — that is, who may use your target's label inside an attribute such as `deps`. A target will fail to build during the [analysis](/reference/glossary#analysis-phase) phase if it violates the visibility of one of its dependencies. +**Target visibility** controls who may depend on your target — that is, who may +use your target's label inside an attribute such as `deps`. A target will fail +to build during the [analysis](/reference/glossary#analysis-phase) phase if it +violates the visibility of one of its dependencies. -Generally, a target `A` is visible to a target `B` if they are in the same location, or if `A` grants visibility to `B`'s location. In the absence of [symbolic macros](/extending/macros), the term "location" can be simplified to just "package"; see [below](#symbolic-macros) for more on symbolic macros. +Generally, a target `A` is visible to a target `B` if they are in the same +location, or if `A` grants visibility to `B`'s location. In the absence of +[symbolic macros](/extending/macros), the term "location" can be simplified +to just "package"; see [below](#symbolic-macros) for more on symbolic macros. -Visibility is specified by listing allowed packages. Allowing a package does not necessarily mean that its subpackages are also allowed. For more details on packages and subpackages, see [Concepts and terminology](/concepts/build-ref). +Visibility is specified by listing allowed packages. Allowing a package does not +necessarily mean that its subpackages are also allowed. For more details on +packages and subpackages, see [Concepts and terminology](/concepts/build-ref). -For prototyping, you can disable target visibility enforcement by setting the flag `--check_visibility=false`. This shouldn't be done for production usage in submitted code. +For prototyping, you can disable target visibility enforcement by setting the +flag `--check_visibility=false`. This shouldn't be done for production usage in +submitted code. -The primary way to control visibility is with a rule's [`visibility`](/reference/be/common-definitions#common.visibility) attribute. The following subsections describe the attribute's format, how to apply it to various kinds of targets, and the interaction between the visibility system and symbolic macros. +The primary way to control visibility is with a rule's +[`visibility`](/reference/be/common-definitions#common.visibility) attribute. +The following subsections describe the attribute's format, how to apply it to +various kinds of targets, and the interaction between the visibility system and +symbolic macros. ### Visibility specifications -All rule targets have a `visibility` attribute that takes a list of labels. Each label has one of the following forms. With the exception of the last form, these are just syntactic placeholders that don't correspond to any actual target. +All rule targets have a `visibility` attribute that takes a list of labels. Each +label has one of the following forms. With the exception of the last form, these +are just syntactic placeholders that don't correspond to any actual target. -- `"//visibility:public"`: Grants access to all packages. +* `"//visibility:public"`: Grants access to all packages. -- `"//visibility:private"`: Does not grant any additional access; only targets in this location's package can use this target. +* `"//visibility:private"`: Does not grant any additional access; only targets + in this location's package can use this target. -- `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its subpackages). +* `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its + subpackages). -- `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its direct and indirect subpackages. +* `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its + direct and indirect subpackages. -- `"//some_pkg:my_package_group"`: Grants access to all of the packages that are part of the given [`package_group`](/reference/be/functions#package_group). +* `"//some_pkg:my_package_group"`: Grants access to all of the packages that + are part of the given [`package_group`](/reference/be/functions#package_group). - - Package groups use a [different syntax](/reference/be/functions#package_group.packages) for specifying packages. Within a package group, the forms `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, `"//visibility:public"` and `"//visibility:private"` are just `"public"` and `"private"`. + * Package groups use a + [different syntax](/reference/be/functions#package_group.packages) for + specifying packages. Within a package group, the forms + `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively + replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, + `"//visibility:public"` and `"//visibility:private"` are just `"public"` + and `"private"`. -For example, if `//some/package:mytarget` has its `visibility` set to `[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target that is part of the `//some/package/...` source tree, as well as targets declared in `//tests/BUILD`, but not by targets defined in `//tests/integration/BUILD`. +For example, if `//some/package:mytarget` has its `visibility` set to +`[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target +that is part of the `//some/package/...` source tree, as well as targets +declared in `//tests/BUILD`, but not by targets defined in +`//tests/integration/BUILD`. -**Best practice:** To make several targets visible to the same set of packages, use a `package_group` instead of repeating the list in each target's `visibility` attribute. This increases readability and prevents the lists from getting out of sync. +**Best practice:** To make several targets visible to the same set +of packages, use a `package_group` instead of repeating the list in each +target's `visibility` attribute. This increases readability and prevents the +lists from getting out of sync. -**Best practice:** When granting visibility to another team's project, prefer `__subpackages__` over `__pkg__` to avoid needless visibility churn as that project evolves and adds new subpackages. +**Best practice:** When granting visibility to another team's project, prefer +`__subpackages__` over `__pkg__` to avoid needless visibility churn as that +project evolves and adds new subpackages. -Note: The `visibility` attribute may not specify non-`package_group` targets. Doing so triggers a "Label does not refer to a package group" or "Cycle in dependency graph" error. +Note: The `visibility` attribute may not specify non-`package_group` targets. +Doing so triggers a "Label does not refer to a package group" or "Cycle in +dependency graph" error. ### Rule target visibility -A rule target's visibility is determined by taking its `visibility` attribute -- or a suitable default if not given -- and appending the location where the target was declared. For targets not declared in a symbolic macro, if the package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), this default is used; for all other packages and for targets declared in a symbolic macro, the default is just `["//visibility:private"]`. +A rule target's visibility is determined by taking its `visibility` attribute +-- or a suitable default if not given -- and appending the location where the +target was declared. For targets not declared in a symbolic macro, if the +package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), +this default is used; for all other packages and for targets declared in a +symbolic macro, the default is just `["//visibility:private"]`. ```starlark # //mypkg/BUILD @@ -82,11 +130,15 @@ package_group( ) ``` -**Best practice:** Avoid setting `default_visibility` to public. It may be convenient for prototyping or in small codebases, but the risk of inadvertently creating public targets increases as the codebase grows. It's better to be explicit about which targets are part of a package's public interface. +**Best practice:** Avoid setting `default_visibility` to public. It may be +convenient for prototyping or in small codebases, but the risk of inadvertently +creating public targets increases as the codebase grows. It's better to be +explicit about which targets are part of a package's public interface. ### Generated file target visibility -A generated file target has the same visibility as the rule target that generates it. +A generated file target has the same visibility as the rule target that +generates it. ```starlark # //mypkg/BUILD @@ -116,21 +168,36 @@ some_rule( ### Source file target visibility -Source file targets can either be explicitly declared using [`exports_files`](/reference/be/functions#exports_files), or implicitly created by referring to their filename in a label attribute of a rule (outside of a symbolic macro). As with rule targets, the location of the call to `exports_files`, or the BUILD file that referred to the input file, is always automatically appended to the file's visibility. +Source file targets can either be explicitly declared using +[`exports_files`](/reference/be/functions#exports_files), or implicitly created +by referring to their filename in a label attribute of a rule (outside of a +symbolic macro). As with rule targets, the location of the call to +`exports_files`, or the BUILD file that referred to the input file, is always +automatically appended to the file's visibility. -Files declared by `exports_files` can have their visibility set by the `visibility` parameter to that function. If this parameter is not given, the visibility is public. +Files declared by `exports_files` can have their visibility set by the +`visibility` parameter to that function. If this parameter is not given, the visibility is public. -Note: `exports_files` may not be used to override the visibility of a generated file. +Note: `exports_files` may not be used to override the visibility of a generated +file. -For files that do not appear in a call to `exports_files`, the visibility depends on the value of the flag [`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): +For files that do not appear in a call to `exports_files`, the visibility +depends on the value of the flag +[`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): -- If the flag is true, the visibility is private. +* If the flag is true, the visibility is private. -- Else, the legacy behavior applies: The visibility is the same as the `BUILD` file's `default_visibility`, or private if a default visibility is not specified. +* Else, the legacy behavior applies: The visibility is the same as the + `BUILD` file's `default_visibility`, or private if a default visibility is + not specified. -Avoid relying on the legacy behavior. Always write an `exports_files` declaration whenever a source file target needs non-private visibility. +Avoid relying on the legacy behavior. Always write an `exports_files` +declaration whenever a source file target needs non-private visibility. -**Best practice:** When possible, prefer to expose a rule target rather than a source file. For example, instead of calling `exports_files` on a `.java` file, wrap the file in a non-private `java_library` target. Generally, rule targets should only directly reference source files that live in the same package. +**Best practice:** When possible, prefer to expose a rule target rather than a +source file. For example, instead of calling `exports_files` on a `.java` file, +wrap the file in a non-private `java_library` target. Generally, rule targets +should only directly reference source files that live in the same package. #### Example @@ -151,45 +218,98 @@ cc_binary( ### Config setting visibility -Historically, Bazel has not enforced visibility for [`config_setting`](/reference/be/general#config_setting) targets that are referenced in the keys of a [`select()`](/reference/be/functions#select). There are two flags to remove this legacy behavior: +Historically, Bazel has not enforced visibility for +[`config_setting`](/reference/be/general#config_setting) targets that are +referenced in the keys of a [`select()`](/reference/be/functions#select). There +are two flags to remove this legacy behavior: -- [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) enables visibility checking for these targets. To assist with migration, it also causes any `config_setting` that does not specify a `visibility` to be considered public (regardless of package-level `default_visibility`). +* [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) + enables visibility checking for these targets. To assist with migration, it + also causes any `config_setting` that does not specify a `visibility` to be + considered public (regardless of package-level `default_visibility`). -- [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) causes `config_setting`s that do not specify a `visibility` to respect the package's `default_visibility` and to fallback on private visibility, just like any other rule target. It is a no-op if `--incompatible_enforce_config_setting_visibility` is not set. +* [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) + causes `config_setting`s that do not specify a `visibility` to respect the + package's `default_visibility` and to fallback on private visibility, just + like any other rule target. It is a no-op if + `--incompatible_enforce_config_setting_visibility` is not set. -Avoid relying on the legacy behavior. Any `config_setting` that is intended to be used outside the current package should have an explicit `visibility`, if the package does not already specify a suitable `default_visibility`. +Avoid relying on the legacy behavior. Any `config_setting` that is intended to +be used outside the current package should have an explicit `visibility`, if the +package does not already specify a suitable `default_visibility`. ### Package group target visibility -`package_group` targets do not have a `visibility` attribute. They are always publicly visible. +`package_group` targets do not have a `visibility` attribute. They are always +publicly visible. ### Visibility of implicit dependencies -Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — dependencies that are not spelled out in a `BUILD` file but are inherent to every instance of that rule. For example, a `cc_library` rule might create an implicit dependency from each of its rule targets to an executable target representing a C++ compiler. +Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — +dependencies that are not spelled out in a `BUILD` file but are inherent to +every instance of that rule. For example, a `cc_library` rule might create an +implicit dependency from each of its rule targets to an executable target +representing a C++ compiler. -The visibility of such an implicit dependency is checked with respect to the package containing the `.bzl` file in which the rule (or aspect) is defined. In our example, the C++ compiler could be private so long as it lives in the same package as the definition of the `cc_library` rule. As a fallback, if the implicit dependency is not visible from the definition, it is checked with respect to the `cc_library` target. +The visibility of such an implicit dependency is checked with respect to the +package containing the `.bzl` file in which the rule (or aspect) is defined. In +our example, the C++ compiler could be private so long as it lives in the same +package as the definition of the `cc_library` rule. As a fallback, if the +implicit dependency is not visible from the definition, it is checked with +respect to the `cc_library` target. -If you want to restrict the usage of a rule to certain packages, use [load visibility](#load-visibility) instead. +If you want to restrict the usage of a rule to certain packages, use +[load visibility](#load-visibility) instead. ### Visibility and symbolic macros -This section describes how the visibility system interacts with [symbolic macros](/extending/macros). +This section describes how the visibility system interacts with +[symbolic macros](/extending/macros). #### Locations within symbolic macros -A key detail of the visibility system is how we determine the location of a declaration. For targets that are not declared in a symbolic macro, the location is just the package where the target lives -- the package of the `BUILD` file. But for targets created in a symbolic macro, the location is the package containing the `.bzl` file where the macro's definition (the `my_macro = macro(...)` statement) appears. When a target is created inside multiple nested targets, it is always the innermost symbolic macro's definition that is used. - -The same system is used to determine what location to check against a given dependency's visibility. If the consuming target was created inside a macro, we look at the innermost macro's definition rather than the package the consuming target lives in. - -This means that all macros whose code is defined in the same package are automatically "friends" with one another. Any target directly created by a macro defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, regardless of what packages the macros are actually instantiated in. Likewise, they can see, and can be seen by, targets declared directly in `//lib/BUILD` and its legacy macros. Conversely, targets that live in the same package cannot necessarily see one another if at least one of them is created by a symbolic macro. - -Within a symbolic macro's implementation function, the `visibility` parameter has the effective value of the macro's `visibility` attribute after appending the location where the macro was called. The standard way for a macro to export one of its targets to its caller is to forward this value along to the target's declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit this attribute won't be visible to the caller of the macro unless the caller happens to be in the same package as the macro definition. This behavior composes, in the sense that a chain of nested calls to submacros may each pass `visibility = visibility`, re-exporting the inner macro's exported targets to the caller at each level, without exposing any of the macros' implementation details. +A key detail of the visibility system is how we determine the location of a +declaration. For targets that are not declared in a symbolic macro, the location +is just the package where the target lives -- the package of the `BUILD` file. +But for targets created in a symbolic macro, the location is the package +containing the `.bzl` file where the macro's definition (the +`my_macro = macro(...)` statement) appears. When a target is created inside +multiple nested targets, it is always the innermost symbolic macro's definition +that is used. + +The same system is used to determine what location to check against a given +dependency's visibility. If the consuming target was created inside a macro, we +look at the innermost macro's definition rather than the package the consuming +target lives in. + +This means that all macros whose code is defined in the same package are +automatically "friends" with one another. Any target directly created by a macro +defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, +regardless of what packages the macros are actually instantiated in. Likewise, +they can see, and can be seen by, targets declared directly in `//lib/BUILD` and +its legacy macros. Conversely, targets that live in the same package cannot +necessarily see one another if at least one of them is created by a symbolic +macro. + +Within a symbolic macro's implementation function, the `visibility` parameter +has the effective value of the macro's `visibility` attribute after appending +the location where the macro was called. The standard way for a macro to export +one of its targets to its caller is to forward this value along to the target's +declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit +this attribute won't be visible to the caller of the macro unless the caller +happens to be in the same package as the macro definition. This behavior +composes, in the sense that a chain of nested calls to submacros may each pass +`visibility = visibility`, re-exporting the inner macro's exported targets to +the caller at each level, without exposing any of the macros' implementation +details. #### Delegating privileges to a submacro -The visibility model has a special feature to allow a macro to delegate its permissions to a submacro. This is important for factoring and composing macros. +The visibility model has a special feature to allow a macro to delegate its +permissions to a submacro. This is important for factoring and composing macros. -Suppose you have a macro `my_macro` that creates a dependency edge using a rule `some_library` from another package: +Suppose you have a macro `my_macro` that creates a dependency edge using a rule +`some_library` from another package: ```starlark # //macro/defs.bzl @@ -218,7 +338,10 @@ load("//macro:defs.bzl", "my_macro") my_macro(name = "foo", ...) ``` -The `//pkg:foo_dependency` target has no `visibility` specified, so it is only visible within `//macro`, which works fine for the consuming target. Now, what happens if the author of `//lib` refactors `some_library` to instead be implemented using a macro? +The `//pkg:foo_dependency` target has no `visibility` specified, so it is only +visible within `//macro`, which works fine for the consuming target. Now, what +happens if the author of `//lib` refactors `some_library` to instead be +implemented using a macro? ```starlark # //lib:defs.bzl @@ -234,43 +357,88 @@ def _impl(name, visibility, deps, ...): some_library = macro(implementation = _impl, ...) ``` -With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than `//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's visibility. The author of `my_macro` can't be expected to pass `visibility = ["//lib"]` to the declaration of the dependency just to work around this implementation detail. +With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than +`//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's +visibility. The author of `my_macro` can't be expected to pass +`visibility = ["//lib"]` to the declaration of the dependency just to work +around this implementation detail. -For this reason, when a dependency of a target is also an attribute value of the macro that declared the target, we check the dependency's visibility against the location of the macro instead of the location of the consuming target. +For this reason, when a dependency of a target is also an attribute value of the +macro that declared the target, we check the dependency's visibility against the +location of the macro instead of the location of the consuming target. -In this example, to validate whether `//pkg:foo_consumer` can see `//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an input to the call to `some_library` inside of `my_macro`, and instead check the dependency's visibility against the location of this call, `//macro`. +In this example, to validate whether `//pkg:foo_consumer` can see +`//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an +input to the call to `some_library` inside of `my_macro`, and instead check the +dependency's visibility against the location of this call, `//macro`. -This process can repeat recursively, as long as a target or macro declaration is inside of another symbolic macro taking the dependency's label in one of its label-typed attributes. +This process can repeat recursively, as long as a target or macro declaration is +inside of another symbolic macro taking the dependency's label in one of its +label-typed attributes. -Note: Visibility delegation does not work for labels that were not passed into the macro, such as labels derived by string manipulation. +Note: Visibility delegation does not work for labels that were not passed into +the macro, such as labels derived by string manipulation. #### Finalizers -Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. +Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), +in addition to seeing targets following the usual symbolic macro visibility +rules, can *also* see all targets which are visible to the finalizer target's +package. -In other words, if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. +In other words, if you migrate a `native.existing_rules()`-based legacy macro to +a finalizer, the targets declared by the finalizer will still be able to see +their old dependencies. -It is possible to define targets that a finalizer can introspect using `native.existing_rules()`, but which it cannot use as dependencies under the visibility system. For example, if a macro-defined target is not visible to its own package or to the finalizer macro's definition, and is not delegated to the finalizer, the finalizer cannot see such a target. Note, however, that a `native.existing_rules()`-based legacy macro will also be unable to see such a target. +It is possible to define targets that a finalizer can introspect using +`native.existing_rules()`, but which it cannot use as dependencies under the +visibility system. For example, if a macro-defined target is not visible to its +own package or to the finalizer macro's definition, and is not delegated to the +finalizer, the finalizer cannot see such a target. Note, however, that a +`native.existing_rules()`-based legacy macro will also be unable to see such a +target. ## Load visibility -**Load visibility** controls whether a `.bzl` file may be loaded from other `BUILD` or `.bzl` files outside the current package. +**Load visibility** controls whether a `.bzl` file may be loaded from other +`BUILD` or `.bzl` files outside the current package. -In the same way that target visibility protects source code that is encapsulated by targets, load visibility protects build logic that is encapsulated by `.bzl` files. For instance, a `BUILD` file author might wish to factor some repetitive target declarations into a macro in a `.bzl` file. Without the protection of load visibility, they might find their macro reused by other collaborators in the same workspace, so that modifying the macro breaks other teams' builds. +In the same way that target visibility protects source code that is encapsulated +by targets, load visibility protects build logic that is encapsulated by `.bzl` +files. For instance, a `BUILD` file author might wish to factor some repetitive +target declarations into a macro in a `.bzl` file. Without the protection of +load visibility, they might find their macro reused by other collaborators in +the same workspace, so that modifying the macro breaks other teams' builds. -Note that a `.bzl` file may or may not have a corresponding source file target. If it does, there is no guarantee that the load visibility and the target visibility coincide. That is, the same `BUILD` file might be able to load the `.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), or vice versa. This can sometimes cause problems for rules that wish to consume `.bzl` files as source code, such as for documentation generation or testing. +Note that a `.bzl` file may or may not have a corresponding source file target. +If it does, there is no guarantee that the load visibility and the target +visibility coincide. That is, the same `BUILD` file might be able to load the +`.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), +or vice versa. This can sometimes cause problems for rules that wish to consume +`.bzl` files as source code, such as for documentation generation or testing. -For prototyping, you may disable load visibility enforcement by setting `--check_bzl_visibility=false`. As with `--check_visibility=false`, this should not be done for submitted code. +For prototyping, you may disable load visibility enforcement by setting +`--check_bzl_visibility=false`. As with `--check_visibility=false`, this should +not be done for submitted code. Load visibility is available as of Bazel 6.0. ### Declaring load visibility -To set the load visibility of a `.bzl` file, call the [`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. The argument to `visibility()` is a list of package specifications, just like the [`packages`](/reference/be/functions#package_group.packages) attribute of `package_group`. However, `visibility()` does not accept negative package specifications. +To set the load visibility of a `.bzl` file, call the +[`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. +The argument to `visibility()` is a list of package specifications, just like +the [`packages`](/reference/be/functions#package_group.packages) attribute of +`package_group`. However, `visibility()` does not accept negative package +specifications. -The call to `visibility()` must only occur once per file, at the top level (not inside a function), and ideally immediately following the `load()` statements. +The call to `visibility()` must only occur once per file, at the top level (not +inside a function), and ideally immediately following the `load()` statements. -Unlike target visibility, the default load visibility is always public. Files that do not call `visibility()` are always loadable from anywhere in the workspace. It is a good idea to add `visibility("private")` to the top of any new `.bzl` file that is not specifically intended for use outside the package. +Unlike target visibility, the default load visibility is always public. Files +that do not call `visibility()` are always loadable from anywhere in the +workspace. It is a good idea to add `visibility("private")` to the top of any +new `.bzl` file that is not specifically intended for use outside the package. ### Example @@ -312,7 +480,8 @@ This section describes tips for managing load visibility declarations. #### Factoring visibilities -When multiple `.bzl` files should have the same visibility, it can be helpful to factor their package specifications into a common list. For example: +When multiple `.bzl` files should have the same visibility, it can be helpful to +factor their package specifications into a common list. For example: ```starlark # //mylib/internal_defs.bzl @@ -344,13 +513,18 @@ visibility(clients) ... ``` -This helps prevent accidental skew between the various `.bzl` files' visibilities. It also is more readable when the `clients` list is large. +This helps prevent accidental skew between the various `.bzl` files' +visibilities. It also is more readable when the `clients` list is large. #### Composing visibilities -Sometimes a `.bzl` file might need to be visible to an allowlist that is composed of multiple smaller allowlists. This is analogous to how a `package_group` can incorporate other `package_group`s via its [`includes`](/reference/be/functions#package_group.includes) attribute. +Sometimes a `.bzl` file might need to be visible to an allowlist that is +composed of multiple smaller allowlists. This is analogous to how a +`package_group` can incorporate other `package_group`s via its +[`includes`](/reference/be/functions#package_group.includes) attribute. -Suppose you are deprecating a widely used macro. You want it to be visible only to existing users and to the packages owned by your own team. You might write: +Suppose you are deprecating a widely used macro. You want it to be visible only +to existing users and to the packages owned by your own team. You might write: ```starlark # //mylib/macros.bzl @@ -364,7 +538,12 @@ visibility(our_packages + their_remaining_uses) #### Deduplicating with package groups -Unlike target visibility, you cannot define a load visibility in terms of a `package_group`. If you want to reuse the same allowlist for both target visibility and load visibility, it's best to move the list of package specifications into a .bzl file, where both kinds of declarations may refer to it. Building off the example in [Factoring visibilities](#factoring-visibilities) above, you might write: +Unlike target visibility, you cannot define a load visibility in terms of a +`package_group`. If you want to reuse the same allowlist for both target +visibility and load visibility, it's best to move the list of package +specifications into a .bzl file, where both kinds of declarations may refer to +it. Building off the example in [Factoring visibilities](#factoring-visibilities) +above, you might write: ```starlark # //mylib/BUILD @@ -377,11 +556,17 @@ package_group( ) ``` -This only works if the list does not contain any negative package specifications. +This only works if the list does not contain any negative package +specifications. #### Protecting individual symbols -Any Starlark symbol whose name begins with an underscore cannot be loaded from another file. This makes it easy to create private symbols, but does not allow you to share these symbols with a limited set of trusted files. On the other hand, load visibility gives you control over what other packages may see your `.bzl file`, but does not allow you to prevent any non-underscored symbol from being loaded. +Any Starlark symbol whose name begins with an underscore cannot be loaded from +another file. This makes it easy to create private symbols, but does not allow +you to share these symbols with a limited set of trusted files. On the other +hand, load visibility gives you control over what other packages may see your +`.bzl file`, but does not allow you to prevent any non-underscored symbol from +being loaded. Luckily, you can combine these two features to get fine-grained control. @@ -418,4 +603,8 @@ public_util = _public_util #### bzl-visibility Buildifier lint -There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) that provides a warning if users load a file from a directory named `internal` or `private`, when the user's file is not itself underneath the parent of that directory. This lint predates the load visibility feature and is unnecessary in workspaces where `.bzl` files declare visibilities. +There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) +that provides a warning if users load a file from a directory named `internal` +or `private`, when the user's file is not itself underneath the parent of that +directory. This lint predates the load visibility feature and is unnecessary in +workspaces where `.bzl` files declare visibilities. diff --git a/configure/attributes.mdx b/configure/attributes.mdx index df76c4ba..7bc3f41e 100644 --- a/configure/attributes.mdx +++ b/configure/attributes.mdx @@ -2,9 +2,15 @@ title: 'Configurable Build Attributes' --- -***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. -This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. + +**_Configurable attributes_**, commonly known as [`select()`]( +/reference/be/functions#select), is a Bazel feature that lets users toggle the values +of build rule attributes at the command line. + +This can be used, for example, for a multiplatform library that automatically +chooses the appropriate implementation for the architecture, or for a +feature-configurable binary that can be customized at build time. ## Example @@ -35,30 +41,59 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: - -| | | -| ----------------------------------------------- | ------------------ | -| Command | deps = | -| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | -| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | - -`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the +command line. Specifically, `deps` becomes: + +
tags at end of lines (common in tables) -/]*>[^<]*$/ { +/]*>[[:space:]]*[^<[:space:]]+[[:space:]]*$/ { # Only add closing tag if there isn't already one if ($0 !~ /<\/td>$/) { $0 = $0 "" @@ -224,7 +513,7 @@ in_code_block { } # Be more careful with - don't match
+ + + + + + + + + + + + + + + + + + + + +
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
+ +`select()` serves as a placeholder for a value that will be chosen based on +*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) +targets. By using `select()` in a configurable attribute, the attribute +effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either +* They all resolve to the same value. For example, when running on linux x86, this is unambiguous + `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` + is an unambiguous specialization of `values = {"cpu": "x86"}`. -- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. - -The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when +nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, +`resources`, `cmd`, and most other attributes. Only a small number of attributes +are *non-configurable*, and these are clearly annotated. For example, +`config_setting`'s own +[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies +under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of +the machine running Bazel (which, thanks to cross-compilation, may be different +than the CPU the target is built for). This is known as a +[configuration transition](/reference/glossary#transition). Given @@ -102,19 +137,30 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and +`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s +build parameters, which include `--cpu=arm`. The `tools` attribute changes +`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on +`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a +[`config_setting`](/reference/be/general#config_setting) or +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of +expected command line flag settings. By encapsulating these in a target, it's +easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands +them for all builds in all projects. These are specified with +[`config_setting`](/reference/be/general#config_setting)'s +[`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -127,21 +173,31 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` +is the expected value for that flag. `:meaningful_condition_name` matches if +*every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, +[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because +it only affects how Bazel reports progress to the user. Targets can't use that +flag to construct their results. The exact set of supported flags isn't +documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with +[Starlark build settings][BuildSettings]. Unlike built-in flags, these are +defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s +[`flag_values`](/reference/be/general#config_setting.flag_values) +attribute: ```python config_setting( @@ -154,17 +210,29 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) +for a working example. -[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) +is an alternative legacy syntax for custom flags (for example +`--define foo=bar`). This can be expressed either in the +[values](/reference/be/general#config_setting.values) attribute +(`values = {"define": "foo=bar"}`) or the +[define_values](/reference/be/general#config_setting.define_values) attribute +(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards +compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The +`config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition matches. +The built-in condition `//conditions:default` matches when no other condition +matches. -Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match +and no default condition emits a `"no matching conditions"` error. This can +protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -190,11 +258,16 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s +[`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides +flexibility, it can also be burdensome to individually set each one every time +you want to build a target. + [Platforms](/extending/platforms) +let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -252,9 +325,12 @@ platform( ) ``` -The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the +`config_setting`s that contain a subset of the platform's `constraint_values`, +allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, +you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -281,9 +357,11 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to check against single values. +This saves the need for boilerplate `config_setting`s when you only need to +check against single values. -Platforms are still under development. See the [documentation](/concepts/platforms) for details. +Platforms are still under development. See the +[documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -305,12 +383,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: + - Duplicate labels can appear in different paths of the same `select`. + - Duplicate labels can *not* appear within the same path of a `select`. + - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -- Duplicate labels can appear in different paths of the same `select`. -- Duplicate labels can *not* appear within the same path of a `select`. -- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) - -`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: +`select` cannot appear inside another `select`. If you need to nest `selects` +and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -331,7 +409,8 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND +chaining](#and-chaining). ## OR chaining @@ -350,7 +429,9 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. +Most conditions evaluate to the same dep. But this syntax is hard to read and +maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple +times. One option is to predefine the value as a BUILD variable: @@ -369,13 +450,18 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary duplication. +This makes it easier to manage the dependency. But it still causes unnecessary +duplication. For more direct support, use one of the following: ### `selects.with_or` -The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: +The +[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -394,12 +480,18 @@ sh_binary( ### `selects.config_setting_group` -The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: + +The +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` + ```python config_setting( name = "config1", @@ -423,13 +515,17 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across +different attributes. -It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous +"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the +[Skylib](https://github.com/bazelbuild/bazel-skylib) macro +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -454,11 +550,13 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed +inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to fails with the error: +By default, when no condition matches, the target the `select()` is attached to +fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -468,7 +566,8 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) +attribute: ```python cc_library( @@ -491,7 +590,8 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable attributes. For example, given: +Rule implementations receive the *resolved values* of configurable +attributes. For example, given: ```python # myapp/BUILD @@ -511,7 +611,9 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert +Macros can accept `select()` clauses and pass them through to native +rules. But *they cannot directly manipulate them*. For example, there's no way +for a macro to convert ```python select({"foo": "val"}, ...) @@ -525,22 +627,32 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* +because macros are evaluated in Bazel's [loading phase](/run/build#loading), +which occurs before flag values are known. +This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. +Second, macros that just need to iterate over *all* `select` paths, while +technically feasible, lack a coherent UI. Further design is necessary to change +this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's +[loading phase](/reference/glossary#loading-phase). +This means it doesn't know what command line flags a target uses since those +flags aren't evaluated until later in the build (in the +[analysis phase](/reference/glossary#analysis-phase)). +So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has +all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` - ```python # myapp/BUILD @@ -589,9 +701,14 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for +details. -The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in +*macros*. These are different than *rules*. See the +documentation on [rules](/extending/rules) and [macros](/extending/macros) +to understand the difference. +Here's an end-to-end example: Define a rule and macro: @@ -671,7 +788,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before +Bazel reads the build's command line flags. That means there isn't enough +information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -694,7 +813,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: +Because *macros* (but not rules) by definition +[can't evaluate `select()`s](#faq-select-macro), any attempt to do so +usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -707,7 +828,8 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly vigilant with them: +Booleans are a special case that fail silently, so you should be particularly +vigilant with them: ```sh $ cat myapp/defs.bzl @@ -729,13 +851,21 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. +This happens because macros don't understand the contents of `select()`. +So what they're really evaluting is the `select()` object itself. According to +[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design +standards, all objects aside from a very small number of exceptions +automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before +Bazel knows what the build's command line parameters are. Can they at least read +the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but it isn't yet a Bazel feature. What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: +Conceptually this is possible, but it isn't yet a Bazel feature. +What you *can* do today is prepare a straight dictionary, then feed it into a +`select()`: ```sh $ cat myapp/defs.bzl @@ -747,7 +877,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -779,7 +909,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -787,11 +917,17 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo +rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in +the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't +actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in +the `actual` attribute, to perform this type of run-time determination. This +works correctly, since `alias()` is a BUILD rule, and is evaluated with a +specific configuration. ```sh $ cat WORKSPACE @@ -817,31 +953,37 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target +that depends on either `//:ssl` or `//external:ssl` will see the alternative +located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, +use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo <desired build flags> +$ bazel cquery //myapp:foo //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on +//myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> +$ bazel cquery 'somepath(//bar, //myapp:foo)' //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the +configuration that resolves `//myapp:foo`'s `select()`. You can inspect its +values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -860,13 +1002,18 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. +`//myapp:foo` may exist in different configurations in the same build. See the +[cquery docs](/query/cquery) for guidance on using `somepath` to get the right +one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the +same command line flags as the `bazel cquery`. The `config` command relies on +the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform +is the target platform because the semantics are unclear. For example: @@ -889,11 +1036,15 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the +`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the +`:x86_linux_platform` defined here? The author of the `BUILD` file and the user +who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with these constraints: +Instead, define a `config_setting` that matches **any** platform with +these constraints: ```py config_setting( @@ -914,11 +1065,13 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what +platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you +can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -938,4 +1091,7 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. +The Bazel team doesn't endorse doing this; it overly constrains your build and +confuses users when the expected condition does not match. + +[BuildSettings]: /extending/config#user-defined-build-settings diff --git a/configure/best-practices.mdx b/configure/best-practices.mdx index 90eec9fc..deecf9dd 100644 --- a/configure/best-practices.mdx +++ b/configure/best-practices.mdx @@ -2,7 +2,10 @@ title: 'Best Practices' --- -This page assumes you are familiar with Bazel and provides guidelines and advice on structuring your projects to take full advantage of Bazel's features. + + +This page assumes you are familiar with Bazel and provides guidelines and +advice on structuring your projects to take full advantage of Bazel's features. The overall goals are: @@ -11,45 +14,81 @@ The overall goals are: - To make code well-structured and testable. - To create a build configuration that is easy to understand and maintain. -These guidelines are not requirements: few projects will be able to adhere to all of them. As the man page for lint says, "A special reward will be presented to the first person to produce a real program that produces no errors with strict checking." However, incorporating as many of these principles as possible should make a project more readable, less error-prone, and faster to build. +These guidelines are not requirements: few projects will be able to adhere to +all of them. As the man page for lint says, "A special reward will be presented +to the first person to produce a real program that produces no errors with +strict checking." However, incorporating as many of these principles as possible +should make a project more readable, less error-prone, and faster to build. -This page uses the requirement levels described in [this RFC](https://www.ietf.org/rfc/rfc2119.txt). +This page uses the requirement levels described in +[this RFC](https://www.ietf.org/rfc/rfc2119.txt). ## Running builds and tests -A project should always be able to run `bazel build //...` and `bazel test //...` successfully on its stable branch. Targets that are necessary but do not build under certain circumstances (such as,require specific build flags, don't build on a certain platform, require license agreements) should be tagged as specifically as possible (for example, "`requires-osx`"). This tagging allows targets to be filtered at a more fine-grained level than the "manual" tag and allows someone inspecting the `BUILD` file to understand what a target's restrictions are. +A project should always be able to run `bazel build //...` and +`bazel test //...` successfully on its stable branch. Targets that are necessary +but do not build under certain circumstances (such as,require specific build +flags, don't build on a certain platform, require license agreements) should be +tagged as specifically as possible (for example, "`requires-osx`"). This +tagging allows targets to be filtered at a more fine-grained level than the +"manual" tag and allows someone inspecting the `BUILD` file to understand what +a target's restrictions are. ## Third-party dependencies You may declare third-party dependencies: -- Either declare them as remote repositories in the `MODULE.bazel` file. -- Or put them in a directory called `third_party/` under your workspace directory. +* Either declare them as remote repositories in the `MODULE.bazel` file. +* Or put them in a directory called `third_party/` under your workspace directory. ## Depending on binaries -Everything should be built from source whenever possible. Generally this means that, instead of depending on a library `some-library.so`, you'd create a `BUILD` file and build `some-library.so` from its sources, then depend on that target. +Everything should be built from source whenever possible. Generally this means +that, instead of depending on a library `some-library.so`, you'd create a +`BUILD` file and build `some-library.so` from its sources, then depend on that +target. -Always building from source ensures that a build is not using a library that was built with incompatible flags or a different architecture. There are also some features like coverage, static analysis, or dynamic analysis that only work on the source. +Always building from source ensures that a build is not using a library that +was built with incompatible flags or a different architecture. There are also +some features like coverage, static analysis, or dynamic analysis that only +work on the source. ## Versioning -Prefer building all code from head whenever possible. When versions must be used, avoid including the version in the target name (for example, `//guava`, not `//guava-20.0`). This naming makes the library easier to update (only one target needs to be updated). It's also more resilient to diamond dependency issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, you could end up with a library that tries to depend on two different versions. If you created a misleading alias to point both targets to one `guava` library, then the `BUILD` files are misleading. +Prefer building all code from head whenever possible. When versions must be +used, avoid including the version in the target name (for example, `//guava`, +not `//guava-20.0`). This naming makes the library easier to update (only one +target needs to be updated). It's also more resilient to diamond dependency +issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, +you could end up with a library that tries to depend on two different versions. +If you created a misleading alias to point both targets to one `guava` library, +then the `BUILD` files are misleading. ## Using the `.bazelrc` file -For project-specific options, use the configuration file your `<var>workspace</var>/.bazelrc` (see [bazelrc format](/run/bazelrc)). +For project-specific options, use the configuration file your +`workspace/.bazelrc` (see [bazelrc format](/run/bazelrc)). -If you want to support per-user options for your project that you **do not** want to check into source control, include the line: +If you want to support per-user options for your project that you **do not** +want to check into source control, include the line: ``` try-import %workspace%/user.bazelrc ``` +(or any other file name) in your `workspace/.bazelrc` +and add `user.bazelrc` to your `.gitignore`. -(or any other file name) in your `<var>workspace</var>/.bazelrc` and add `user.bazelrc` to your `.gitignore`. +The open-source +[bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) -The open-source [bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) generates a custom bazelrc file that matches your Bazel version and provides a preset of recommended flags. +generates a custom bazelrc file that matches your Bazel version and provides a +preset of recommended flags. ## Packages -Every directory that contains buildable files should be a package. If a `BUILD` file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's a sign that a `BUILD` file should be added to that subdirectory. The longer this structure exists, the more likely circular dependencies will be inadvertently created, a target's scope will creep, and an increasing number of reverse dependencies will have to be updated. +Every directory that contains buildable files should be a package. If a `BUILD` +file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's +a sign that a `BUILD` file should be added to that subdirectory. The longer +this structure exists, the more likely circular dependencies will be +inadvertently created, a target's scope will creep, and an increasing number +of reverse dependencies will have to be updated. diff --git a/configure/coverage.mdx b/configure/coverage.mdx index 72cde9c5..03a8ab9b 100644 --- a/configure/coverage.mdx +++ b/configure/coverage.mdx @@ -2,67 +2,129 @@ title: 'Code coverage with Bazel' --- -Bazel features a `coverage` sub-command to produce code coverage reports on repositories that can be tested with `bazel coverage`. Due to the idiosyncrasies of the various language ecosystems, it is not always trivial to make this work for a given project. -This page documents the general process for creating and viewing coverage reports, and also features some language-specific notes for languages whose configuration is well-known. It is best read by first reading [the general section](#creating-a-coverage-report), and then reading about the requirements for a specific language. Note also the [remote execution section](#remote-execution), which requires some additional considerations. -While a lot of customization is possible, this document focuses on producing and consuming [`lcov`](https://github.com/linux-test-project/lcov) reports, which is currently the most well-supported route. +Bazel features a `coverage` sub-command to produce code coverage +reports on repositories that can be tested with `bazel coverage`. Due +to the idiosyncrasies of the various language ecosystems, it is not +always trivial to make this work for a given project. + +This page documents the general process for creating and viewing +coverage reports, and also features some language-specific notes for +languages whose configuration is well-known. It is best read by first +reading [the general section](#creating-a-coverage-report), and then +reading about the requirements for a specific language. Note also the +[remote execution section](#remote-execution), which requires some +additional considerations. + +While a lot of customization is possible, this document focuses on +producing and consuming [`lcov`][lcov] reports, which is currently the +most well-supported route. ## Creating a coverage report ### Preparation -The basic workflow for creating coverage reports requires the following: +The basic workflow for creating coverage reports requires the +following: - A basic repository with test targets - A toolchain with the language-specific code coverage tools installed - A correct "instrumentation" configuration -The former two are language-specific and mostly straightforward, however the latter can be more difficult for complex projects. +The former two are language-specific and mostly straightforward, +however the latter can be more difficult for complex projects. -"Instrumentation" in this case refers to the coverage tools that are used for a specific target. Bazel allows turning this on for a specific subset of files using the [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) flag, which specifies a filter for targets that are tested with the instrumentation enabled. To enable instrumentation for tests, the [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) flag is required. +"Instrumentation" in this case refers to the coverage tools that are +used for a specific target. Bazel allows turning this on for a +specific subset of files using the +[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) +flag, which specifies a filter for targets that are tested with the +instrumentation enabled. To enable instrumentation for tests, the +[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) +flag is required. -By default, bazel tries to match the target package(s), and prints the relevant filter as an `INFO` message. +By default, bazel tries to match the target package(s), and prints the +relevant filter as an `INFO` message. ### Running coverage -To produce a coverage report, use [`bazel coverage --combined_report=lcov [target]`](/reference/command-line-reference#coverage). This runs the tests for the target, generating coverage reports in the lcov format for each file. +To produce a coverage report, use [`bazel coverage +--combined_report=lcov +[target]`](/reference/command-line-reference#coverage). This runs the +tests for the target, generating coverage reports in the lcov format +for each file. -Once finished, bazel runs an action that collects all the produced coverage files, and merges them into one, which is then finally created under `$(bazel info output_path)/_coverage/_coverage_report.dat`. +Once finished, bazel runs an action that collects all the produced +coverage files, and merges them into one, which is then finally +created under `$(bazel info +output_path)/_coverage/_coverage_report.dat`. -Coverage reports are also produced if tests fail, though note that this does not extend to the failed tests - only passing tests are reported. +Coverage reports are also produced if tests fail, though note that +this does not extend to the failed tests - only passing tests are +reported. ### Viewing coverage -The coverage report is only output in the non-human-readable `lcov` format. From this, we can use the `genhtml` utility (part of [the lcov project](https://github.com/linux-test-project/lcov)) to produce a report that can be viewed in a web browser: +The coverage report is only output in the non-human-readable `lcov` +format. From this, we can use the `genhtml` utility (part of [the lcov +project][lcov]) to produce a report that can be viewed in a web +browser: ```console genhtml --branch-coverage --output genhtml "$(bazel info output_path)/_coverage/_coverage_report.dat" ``` -Note that `genhtml` reads the source code as well, to annotate missing coverage in these files. For this to work, it is expected that `genhtml` is executed in the root of the bazel project. +Note that `genhtml` reads the source code as well, to annotate missing +coverage in these files. For this to work, it is expected that +`genhtml` is executed in the root of the bazel project. -To view the result, simply open the `index.html` file produced in the `genhtml` directory in any web browser. +To view the result, simply open the `index.html` file produced in the +`genhtml` directory in any web browser. -For further help and information around the `genhtml` tool, or the `lcov` coverage format, see [the lcov project](https://github.com/linux-test-project/lcov). +For further help and information around the `genhtml` tool, or the +`lcov` coverage format, see [the lcov project][lcov]. ## Remote execution Running with remote test execution currently has a few caveats: -- The report combination action cannot yet run remotely. This is because Bazel does not consider the coverage output files as part of its graph (see [this issue](https://github.com/bazelbuild/bazel/issues/4685)), and can therefore not correctly treat them as inputs to the combination action. To work around this, use `--strategy=CoverageReport=local`. - - Note: It may be necessary to specify something like `--strategy=CoverageReport=local,remote` instead, if Bazel is set up to try `local,remote`, due to how Bazel resolves strategies. -- `--remote_download_minimal` and similar flags can also not be used as a consequence of the former. -- Bazel will currently fail to create coverage information if tests have been cached previously. To work around this, `--nocache_test_results` can be set specifically for coverage runs, although this of course incurs a heavy cost in terms of test times. -- `--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs` - - Usually coverage is run as part of the test action, and so by default, we don't get all coverage back as outputs of the remote execution by default. These flags override the default and obtain the coverage data. See [this issue](https://github.com/bazelbuild/bazel/issues/4685) for more details. +- The report combination action cannot yet run remotely. This is + because Bazel does not consider the coverage output files as part of + its graph (see [this issue][remote_report_issue]), and can therefore + not correctly treat them as inputs to the combination action. To + work around this, use `--strategy=CoverageReport=local`. + - Note: It may be necessary to specify something like + `--strategy=CoverageReport=local,remote` instead, if Bazel is set + up to try `local,remote`, due to how Bazel resolves strategies. +- `--remote_download_minimal` and similar flags can also not be used + as a consequence of the former. +- Bazel will currently fail to create coverage information if tests + have been cached previously. To work around this, + `--nocache_test_results` can be set specifically for coverage runs, + although this of course incurs a heavy cost in terms of test times. +- `--experimental_split_coverage_postprocessing` and + `--experimental_fetch_all_coverage_outputs` + - Usually coverage is run as part of the test action, and so by + default, we don't get all coverage back as outputs of the remote + execution by default. These flags override the default and obtain + the coverage data. See [this issue][split_coverage_issue] for more + details. ## Language-specific configuration ### Java -Java should work out-of-the-box with the default configuration. The [bazel toolchains](https://github.com/bazelbuild/bazel-toolchains) contain everything necessary for remote execution, as well, including JUnit. +Java should work out-of-the-box with the default configuration. The +[bazel toolchains][bazel_toolchains] contain everything necessary for +remote execution, as well, including JUnit. ### Python -See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) for additional steps needed to enable coverage support in Python. +See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) +for additional steps needed to enable coverage support in Python. + +[lcov]: https://github.com/linux-test-project/lcov +[bazel_toolchains]: https://github.com/bazelbuild/bazel-toolchains +[remote_report_issue]: https://github.com/bazelbuild/bazel/issues/4685 +[split_coverage_issue]: https://github.com/bazelbuild/bazel/issues/4685 diff --git a/configure/windows.mdx b/configure/windows.mdx index 566c425d..2dbb90ee 100644 --- a/configure/windows.mdx +++ b/configure/windows.mdx @@ -2,17 +2,24 @@ title: 'Using Bazel on Windows' --- -This page covers Best Practices for using Bazel on Windows. For installation instructions, see [Install Bazel on Windows](/install/windows). + + +This page covers Best Practices for using Bazel on Windows. For installation +instructions, see [Install Bazel on Windows](/install/windows). ## Known issues -Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. [GitHub-Windows](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows). +Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. +[GitHub-Windows]. + +[GitHub-Windows]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows ## Best practices ### Avoid long path issues -Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. To avoid hitting this issue, you can specify a short output directory for Bazel by the [--output\_user\_root](/reference/command-line-reference#flag--output_user_root) flag. +Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. +To avoid hitting this issue, you can specify a short output directory for Bazel by the [\-\-output_user_root](/reference/command-line-reference#flag--output_user_root) flag. For example, add the following line to your bazelrc file: @@ -22,10 +29,14 @@ startup --output_user_root=C:/tmp ### Enable symlink support -Some features require Bazel to be able to create file symlinks on Windows, either by enabling [Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) (on Windows 10 version 1703 or newer), or by running Bazel as an administrator. This enables the following features: +Some features require Bazel to be able to create file symlinks on Windows, +either by enabling +[Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) +(on Windows 10 version 1703 or newer), or by running Bazel as an administrator. +This enables the following features: -- [--windows\_enable\_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) -- [--enable\_runfiles](/reference/command-line-reference#flag--enable_runfiles) +* [\-\-windows_enable_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) +* [\-\-enable_runfiles](/reference/command-line-reference#flag--enable_runfiles) To make it easier, add the following lines to your bazelrc file: @@ -37,11 +48,23 @@ build --enable_runfiles **Note**: Creating symlinks on Windows is an expensive operation. The `--enable_runfiles` flag can potentially create a large amount of file symlinks. Only enable this feature when you need it. +{/* TODO(pcloudy): https://github.com/bazelbuild/bazel/issues/6402 + Write a doc about runfiles library and add a link to it here */} + ### Running Bazel: MSYS2 shell vs. command prompt vs. PowerShell -**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from PowerShell. +**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from +PowerShell. -As of 2020-01-15, **do not** run Bazel from `bash` -- either from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel may work for most use cases, some things are broken, like [interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). Also, if you choose to run under MSYS2, you need to disable MSYS2's automatic path conversion, otherwise MSYS will convert command line arguments that *look like* Unix paths (such as `//foo:bar`) into Windows paths. See [this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) for details. +As of 2020-01-15, **do not** run Bazel from `bash` -- either +from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel +may work for most use cases, some things are broken, like +[interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). +Also, if you choose to run under MSYS2, you need to disable MSYS2's +automatic path conversion, otherwise MSYS will convert command line arguments +that _look like_ Unix paths (such as `//foo:bar`) into Windows paths. See +[this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) +for details. ### Using Bazel without Bash (MSYS2) @@ -55,7 +78,13 @@ Starting with Bazel 1.0, you can build any rule without Bash unless it is a: - `sh_binary` or `sh_test` rule, because these inherently need Bash - Starlark rule that uses `ctx.actions.run_shell()` or `ctx.resolve_command()` -However, `genrule` is often used for simple tasks like [copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). Instead of using `genrule` (and depending on Bash) you may find a suitable rule in the [bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). When built on Windows, **these rules do not require Bash**. +However, `genrule` is often used for simple tasks like +[copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) +or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). +Instead of using `genrule` (and depending on Bash) you may find a suitable rule +in the +[bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). +When built on Windows, **these rules do not require Bash**. #### Using bazel test without Bash @@ -75,15 +104,24 @@ Starting with Bazel 1.0, you can run any rule without Bash, except when: - you use `--run_under` or `--script_path` - the test rule itself requires Bash (because its executable is a shell script) -#### Using sh\_binary and sh\_\* rules, and ctx.actions.run\_shell() without Bash +#### Using sh\_binary and sh\_* rules, and ctx.actions.run_shell() without Bash -You need Bash to build and test `sh_*` rules, and to build and test Starlark rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This applies not only to rules in your project, but to rules in any of the external repositories your project depends on (even transitively). +You need Bash to build and test `sh_*` rules, and to build and test Starlark +rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This +applies not only to rules in your project, but to rules in any of the external +repositories your project depends on (even transitively). -In the future, there may be an option to use Windows Subsystem for Linux (WSL) to build these rules, but currently it is not a priority for the Bazel-on-Windows subteam. +In the future, there may be an option to use Windows Subsystem for +Linux (WSL) to build these rules, but currently it is not a priority for +the Bazel-on-Windows subteam. ### Setting environment variables -Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only set in that command prompt session. If you start a new `cmd.exe`, you need to set the variables again. To always set the variables when `cmd.exe` starts, you can add them to the User variables or System variables in the `Control Panel > System Properties > Advanced > Environment Variables...` dialog box. +Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only +set in that command prompt session. If you start a new `cmd.exe`, you need to +set the variables again. To always set the variables when `cmd.exe` starts, you +can add them to the User variables or System variables in the `Control Panel > +System Properties > Advanced > Environment Variables...` dialog box. ## Build on Windows @@ -91,54 +129,71 @@ Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only To build C++ targets with MSVC, you need: -- [The Visual C++ compiler](/install/windows#install-vc). +* [The Visual C++ compiler](/install/windows#install-vc). -- (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. +* (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. - Bazel automatically detects the Visual C++ compiler on your system. To tell Bazel to use a specific VC installation, you can set the following environment variables: + Bazel automatically detects the Visual C++ compiler on your system. + To tell Bazel to use a specific VC installation, you can set the + following environment variables: - For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. + For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. - - `BAZEL_VC` the Visual C++ Build Tools installation directory + * `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC + ``` - - `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel will choose the latest version. + * `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version + number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools + version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel + will choose the latest version. - ``` - set BAZEL_VC_FULL_VERSION=14.16.27023 - ``` + ``` + set BAZEL_VC_FULL_VERSION=14.16.27023 + ``` - For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) + For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) - - `BAZEL_VC` the Visual C++ Build Tools installation directory + * `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC + ``` -- The [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). +* The [Windows + SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). - The Windows SDK contains header files and libraries you need when building Windows applications, including Bazel itself. By default, the latest Windows SDK installed will be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified Windows SDK installed. + The Windows SDK contains header files and libraries you need when building + Windows applications, including Bazel itself. By default, the latest Windows SDK installed will + be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You + can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 + SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified + Windows SDK installed. - **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise `BAZEL_WINSDK_FULL_VERSION` will be ignored. + **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't + support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise + `BAZEL_WINSDK_FULL_VERSION` will be ignored. - ``` - set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 - ``` + ``` + set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 + ``` If everything is set up, you can build a C++ target now! -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` -bazel build //examples/cpp:hello-world -bazel-bin\examples\cpp\hello-world.exe +bazel build //examples/cpp:hello-world +bazel-bin\examples\cpp\hello-world.exe ``` -By default, the built binaries target x64 architecture. To build for ARM64 architecture, use +By default, the built binaries target x64 architecture. To build for ARM64 +architecture, use ```none --platforms=//:windows_arm64 --extra_toolchains=@local_config_cc//:cc-toolchain-arm64_windows @@ -152,88 +207,111 @@ cc_configure = use_extension("@rules_cc//cc:extensions.bzl", "cc_configure_exten use_repo(cc_configure, "local_config_cc") ``` -To build and use Dynamically Linked Libraries (DLL files), see [this example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). +To build and use Dynamically Linked Libraries (DLL files), see [this +example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). -**Command Line Length Limit**: To prevent the [Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), enable the compiler parameter file feature via `--features=compiler_param_file`. +**Command Line Length Limit**: To prevent the +[Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), +enable the compiler parameter file feature via `--features=compiler_param_file`. ### Build C++ with Clang From 0.29.0, Bazel supports building with LLVM's MSVC-compatible compiler driver (`clang-cl.exe`). -**Requirement**: To build with Clang, you have to install **both** [LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, because although you use `clang-cl.exe` as compiler, you still need to link to Visual C++ libraries. +**Requirement**: To build with Clang, you have to install **both** +[LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, +because although you use `clang-cl.exe` as compiler, you still need to link to +Visual C++ libraries. -Bazel can automatically detect LLVM installation on your system, or you can explicitly tell Bazel where LLVM is installed by `BAZEL_LLVM`. +Bazel can automatically detect LLVM installation on your system, or you can explicitly tell +Bazel where LLVM is installed by `BAZEL_LLVM`. -- `BAZEL_LLVM` the LLVM installation directory +* `BAZEL_LLVM` the LLVM installation directory - ```posix-terminal - set BAZEL_LLVM=C:\Program Files\LLVM - ``` + ```posix-terminal + set BAZEL_LLVM=C:\Program Files\LLVM + ``` To enable the Clang toolchain for building C++, there are several situations. -- In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the top level `BUILD` file): +* In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the + top level `BUILD` file): - ``` - platform( - name = "x64_windows-clang-cl", - constraint_values = [ - "@platforms//cpu:x86_64", - "@platforms//os:windows", - "@bazel_tools//tools/cpp:clang-cl", - ], - ) - ``` + ``` + platform( + name = "x64_windows-clang-cl", + constraint_values = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@bazel_tools//tools/cpp:clang-cl", + ], + ) + ``` - Then enable the Clang toolchain by specifying the following build flags: + Then enable the Clang toolchain by specifying the following build flags: - ``` - --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl - ``` + ``` + --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl + ``` -- In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by a build flag `--compiler=clang-cl`. +* In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by + a build flag `--compiler=clang-cl`. - If your build sets the flag [--incompatible\_enable\_cc\_toolchain\_resolution](https://github.com/bazelbuild/bazel/issues/7260) to `true`, then use the approach for Bazel 7.0.0. + If your build sets the flag + [\-\-incompatible_enable_cc_toolchain_resolution] + (https://github.com/bazelbuild/bazel/issues/7260) + to `true`, then use the approach for Bazel 7.0.0. -- In Bazel 0.28 and older: Clang is not supported. +* In Bazel 0.28 and older: Clang is not supported. ### Build Java To build Java targets, you need: -- [The Java SE Development Kit](/install/windows#install-jdk) +* [The Java SE Development Kit](/install/windows#install-jdk) On Windows, Bazel builds two output files for `java_binary` rules: -- a `.jar` file -- a `.exe` file that can set up the environment for the JVM and run the binary +* a `.jar` file +* a `.exe` file that can set up the environment for the JVM and run the binary -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world - bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe + bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world + bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe ``` ### Build Python To build Python targets, you need: -- The [Python interpreter](/install/windows#install-python) +* The [Python interpreter](/install/windows#install-python) On Windows, Bazel builds two output files for `py_binary` rules: -- a self-extracting zip file -- an executable file that can launch the Python interpreter with the self-extracting zip file as the argument +* a self-extracting zip file +* an executable file that can launch the Python interpreter with the + self-extracting zip file as the argument -You can either run the executable file (it has a `.exe` extension) or you can run Python with the self-extracting zip file as the argument. +You can either run the executable file (it has a `.exe` extension) or you can run +Python with the self-extracting zip file as the argument. -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/py_native:bin - bazel-bin\examples\py_native\bin.exe - python bazel-bin\examples\py_native\bin.zip + bazel build //examples/py_native:bin + bazel-bin\examples\py_native\bin.exe + python bazel-bin\examples\py_native\bin.zip ``` -If you are interested in details about how Bazel builds Python targets on Windows, check out this [design doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). +If you are interested in details about how Bazel builds Python targets on +Windows, check out this [design +doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). diff --git a/contribute/breaking-changes.mdx b/contribute/breaking-changes.mdx index c30b163a..5dda1b9d 100644 --- a/contribute/breaking-changes.mdx +++ b/contribute/breaking-changes.mdx @@ -2,41 +2,65 @@ title: 'Guide for rolling out breaking changes' --- -It is inevitable that we will make breaking changes to Bazel. We will have to change our designs and fix the things that do not quite work. However, we need to make sure that community and Bazel ecosystem can follow along. To that end, Bazel project has adopted a [backward compatibility policy](/release/backward-compatibility). This document describes the process for Bazel contributors to make a breaking change in Bazel to adhere to this policy. + + +It is inevitable that we will make breaking changes to Bazel. We will have to +change our designs and fix the things that do not quite work. However, we need +to make sure that community and Bazel ecosystem can follow along. To that end, +Bazel project has adopted a +[backward compatibility policy](/release/backward-compatibility). +This document describes the process for Bazel contributors to make a breaking +change in Bazel to adhere to this policy. 1. Follow the [design document policy](/contribute/design-documents). -2. [File a GitHub issue.](#github-issue) +1. [File a GitHub issue.](#github-issue) -3. [Implement the change.](#implementation) +1. [Implement the change.](#implementation) -4. [Update labels.](#labels) +1. [Update labels.](#labels) -5. [Update repositories.](#update-repos) +1. [Update repositories.](#update-repos) -6. [Flip the incompatible flag.](#flip-flag) +1. [Flip the incompatible flag.](#flip-flag) ## GitHub issue -[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) in the Bazel repository. [See example.](https://github.com/bazelbuild/bazel/issues/6611) +[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) +in the Bazel repository. +[See example.](https://github.com/bazelbuild/bazel/issues/6611) We recommend that: -- The title starts with the name of the flag (the flag name will start with `incompatible_`). +* The title starts with the name of the flag (the flag name will start with + `incompatible_`). -- You add the label [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). +* You add the label + [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). -- The description contains a description of the change and a link to relevant design documents. +* The description contains a description of the change and a link to relevant + design documents. -- The description contains a migration recipe, to explain users how they should update their code. Ideally, when the change is mechanical, include a link to a migration tool. +* The description contains a migration recipe, to explain users how they should + update their code. Ideally, when the change is mechanical, include a link to a + migration tool. -- The description includes an example of the error message users will get if they don't migrate. This will make the GitHub issue more discoverable from search engines. Make sure that the error message is helpful and actionable. When possible, the error message should include the name of the incompatible flag. +* The description includes an example of the error message users will get if + they don't migrate. This will make the GitHub issue more discoverable from + search engines. Make sure that the error message is helpful and actionable. + When possible, the error message should include the name of the incompatible + flag. -For the migration tool, consider contributing to [Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. It may also report warnings. +For the migration tool, consider contributing to +[Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). +It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. +It may also report warnings. ## Implementation -Create a new flag in Bazel. The default value must be false. The help text should contain the URL of the GitHub issue. As the flag name starts with `incompatible_`, it needs metadata tags: +Create a new flag in Bazel. The default value must be false. The help text +should contain the URL of the GitHub issue. As the flag name starts with +`incompatible_`, it needs metadata tags: ```java metadataTags = { @@ -44,60 +68,80 @@ Create a new flag in Bazel. The default value must be false. The help text shoul }, ``` -In the commit description, add a brief summary of the flag. Also add [`RELNOTES:`](release-notes.md) in the following form: `RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` +In the commit description, add a brief summary of the flag. +Also add [`RELNOTES:`](release-notes.md) in the following form: +`RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` -The commit should also update the relevant documentation, so that there is no window of commits in which the code is inconsistent with the docs. Since our documentation is versioned, changes to the docs will not be inadvertently released prematurely. +The commit should also update the relevant documentation, so that there is no +window of commits in which the code is inconsistent with the docs. Since our +documentation is versioned, changes to the docs will not be inadvertently +released prematurely. ## Labels -Once the commit is merged and the incompatible change is ready to be adopted, add the label [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) to the GitHub issue. +Once the commit is merged and the incompatible change is ready to be adopted, add the label +[`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) +to the GitHub issue. -If a problem is found with the flag and users are not expected to migrate yet: remove the flags `migration-ready`. +If a problem is found with the flag and users are not expected to migrate yet: +remove the flags `migration-ready`. -If you plan to flip the flag in the next major release, add label \`breaking-change-X.0" to the issue. +If you plan to flip the flag in the next major release, add label `breaking-change-X.0" to the issue. ## Updating repositories -Bazel CI tests a list of important projects at [Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). +Bazel CI tests a list of important projects at +[Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often +dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). +Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). Our dev support team monitors the [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) label. Once you add this label to the GitHub issue, they will handle the following: 1. Create a comment in the GitHub issue to track the list of failures and downstream projects that need to be migrated ([see example](https://github.com/bazelbuild/bazel/issues/17032#issuecomment-1353077469)) -2. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) +1. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) -3. Follow up to make sure all issues are addressed before the target release date +1. Follow up to make sure all issues are addressed before the target release date Migrating projects in the downstream pipeline is NOT entirely the responsibility of the incompatible change author, but you can do the following to accelerate the migration and make life easier for both Bazel users and the Bazel Green Team. 1. Send PRs to fix downstream projects. -2. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). +1. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). ## Flipping the flag Before flipping the default value of the flag to true, please make sure that: -- Core repositories in the ecosystem are migrated. +* Core repositories in the ecosystem are migrated. - On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. + On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), + the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. -- All issues in the checklist are marked as fixed/closed. +* All issues in the checklist are marked as fixed/closed. -- User concerns and questions have been resolved. +* User concerns and questions have been resolved. When the flag is ready to flip in Bazel, but blocked on internal migration at Google, please consider setting the flag value to false in the internal `blazerc` file to unblock the flag flip. By doing this, we can ensure Bazel users depend on the new behaviour by default as early as possible. When changing the flag default to true, please: -- Use `RELNOTES[INC]` in the commit description, with the following format: `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for details` You can include additional information in the rest of the commit description. -- Use `Fixes #xyz` in the description, so that the GitHub issue gets closed when the commit is merged. -- Review and update documentation if needed. -- File a new issue `#abc` to track the removal of the flag. +* Use `RELNOTES[INC]` in the commit description, with the + following format: + `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for + details` + You can include additional information in the rest of the commit description. +* Use `Fixes #xyz` in the description, so that the GitHub issue gets closed + when the commit is merged. +* Review and update documentation if needed. +* File a new issue `#abc` to track the removal of the flag. ## Removing the flag -After the flag is flipped at HEAD, it should be removed from Bazel eventually. When you plan to remove the incompatible flag: +After the flag is flipped at HEAD, it should be removed from Bazel eventually. +When you plan to remove the incompatible flag: -- Consider leaving more time for users to migrate if it's a major incompatible change. Ideally, the flag should be available in at least one major release. -- For the commit that removes the flag, use `Fixes #abc` in the description so that the GitHub issue gets closed when the commit is merged. +* Consider leaving more time for users to migrate if it's a major incompatible change. + Ideally, the flag should be available in at least one major release. +* For the commit that removes the flag, use `Fixes #abc` in the description + so that the GitHub issue gets closed when the commit is merged. diff --git a/contribute/codebase.mdx b/contribute/codebase.mdx index 4d12a684..44e0150d 100644 --- a/contribute/codebase.mdx +++ b/contribute/codebase.mdx @@ -2,773 +2,1652 @@ title: 'The Bazel codebase' --- -This document is a description of the codebase and how Bazel is structured. It is intended for people willing to contribute to Bazel, not for end-users. + + +This document is a description of the codebase and how Bazel is structured. It +is intended for people willing to contribute to Bazel, not for end-users. ## Introduction -The codebase of Bazel is large (\~350KLOC production code and \~260 KLOC test code) and no one is familiar with the whole landscape: everyone knows their particular valley very well, but few know what lies over the hills in every direction. +The codebase of Bazel is large (~350KLOC production code and ~260 KLOC test +code) and no one is familiar with the whole landscape: everyone knows their +particular valley very well, but few know what lies over the hills in every +direction. -In order for people midway upon the journey not to find themselves within a forest dark with the straightforward pathway being lost, this document tries to give an overview of the codebase so that it's easier to get started with working on it. +In order for people midway upon the journey not to find themselves within a +forest dark with the straightforward pathway being lost, this document tries to +give an overview of the codebase so that it's easier to get started with +working on it. -The public version of the source code of Bazel lives on GitHub at [github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not the "source of truth"; it's derived from a Google-internal source tree that contains additional functionality that is not useful outside Google. The long-term goal is to make GitHub the source of truth. +The public version of the source code of Bazel lives on GitHub at +[github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not +the "source of truth"; it's derived from a Google-internal source tree that +contains additional functionality that is not useful outside Google. The +long-term goal is to make GitHub the source of truth. -Contributions are accepted through the regular GitHub pull request mechanism, and manually imported by a Googler into the internal source tree, then re-exported back out to GitHub. +Contributions are accepted through the regular GitHub pull request mechanism, +and manually imported by a Googler into the internal source tree, then +re-exported back out to GitHub. ## Client/server architecture -The bulk of Bazel resides in a server process that stays in RAM between builds. This allows Bazel to maintain state between builds. +The bulk of Bazel resides in a server process that stays in RAM between builds. +This allows Bazel to maintain state between builds. -This is why the Bazel command line has two kinds of options: startup and command. In a command line like this: +This is why the Bazel command line has two kinds of options: startup and +command. In a command line like this: ``` bazel --host_jvm_args=-Xmx8G build -c opt //foo:bar ``` -Some options (`--host_jvm_args=`) are before the name of the command to be run and some are after (`-c opt`); the former kind is called a "startup option" and affects the server process as a whole, whereas the latter kind, the "command option", only affects a single command. - -Each server instance has a single associated workspace (collection of source trees known as "repositories") and each workspace usually has a single active server instance. This can be circumvented by specifying a custom output base (see the "Directory layout" section for more information). - -Bazel is distributed as a single ELF executable that is also a valid .zip file. When you type `bazel`, the above ELF executable implemented in C++ (the "client") gets control. It sets up an appropriate server process using the following steps: - -1. Checks whether it has already extracted itself. If not, it does that. This is where the implementation of the server comes from. -2. Checks whether there is an active server instance that works: it is running, it has the right startup options and uses the right workspace directory. It finds the running server by looking at the directory `$OUTPUT_BASE/server` where there is a lock file with the port the server is listening on. -3. If needed, kills the old server process -4. If needed, starts up a new server process - -After a suitable server process is ready, the command that needs to be run is communicated to it over a gRPC interface, then the output of Bazel is piped back to the terminal. Only one command can be running at the same time. This is implemented using an elaborate locking mechanism with parts in C++ and parts in Java. There is some infrastructure for running multiple commands in parallel, since the inability to run `bazel version` in parallel with another command is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s and some state in `BlazeRuntime`. - -At the end of a command, the Bazel server transmits the exit code the client should return. An interesting wrinkle is the implementation of `bazel run`: the job of this command is to run something Bazel just built, but it can't do that from the server process because it doesn't have a terminal. So instead it tells the client what binary it should `exec()` and with what arguments. - -When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC connection, which tries to terminate the command as soon as possible. After the third Ctrl-C, the client sends a SIGKILL to the server instead. - -The source code of the client is under `src/main/cpp` and the protocol used to communicate with the server is in `src/main/protobuf/command_server.proto` . - -The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls from the client are handled by `GrpcServerImpl.run()`. +Some options (`--host_jvm_args=`) are before the name of the command to be run +and some are after (`-c opt`); the former kind is called a "startup option" and +affects the server process as a whole, whereas the latter kind, the "command +option", only affects a single command. + +Each server instance has a single associated workspace (collection of source +trees known as "repositories") and each workspace usually has a single active +server instance. This can be circumvented by specifying a custom output base +(see the "Directory layout" section for more information). + +Bazel is distributed as a single ELF executable that is also a valid .zip file. +When you type `bazel`, the above ELF executable implemented in C++ (the +"client") gets control. It sets up an appropriate server process using the +following steps: + +1. Checks whether it has already extracted itself. If not, it does that. This + is where the implementation of the server comes from. +2. Checks whether there is an active server instance that works: it is running, + it has the right startup options and uses the right workspace directory. It + finds the running server by looking at the directory `$OUTPUT_BASE/server` + where there is a lock file with the port the server is listening on. +3. If needed, kills the old server process +4. If needed, starts up a new server process + +After a suitable server process is ready, the command that needs to be run is +communicated to it over a gRPC interface, then the output of Bazel is piped back +to the terminal. Only one command can be running at the same time. This is +implemented using an elaborate locking mechanism with parts in C++ and parts in +Java. There is some infrastructure for running multiple commands in parallel, +since the inability to run `bazel version` in parallel with another command +is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s +and some state in `BlazeRuntime`. + +At the end of a command, the Bazel server transmits the exit code the client +should return. An interesting wrinkle is the implementation of `bazel run`: the +job of this command is to run something Bazel just built, but it can't do that +from the server process because it doesn't have a terminal. So instead it tells +the client what binary it should `exec()` and with what arguments. + +When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC +connection, which tries to terminate the command as soon as possible. After the +third Ctrl-C, the client sends a SIGKILL to the server instead. + +The source code of the client is under `src/main/cpp` and the protocol used to +communicate with the server is in `src/main/protobuf/command_server.proto` . + +The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls +from the client are handled by `GrpcServerImpl.run()`. ## Directory layout -Bazel creates a somewhat complicated set of directories during a build. A full description is available in [Output directory layout](/remote/output-directories). +Bazel creates a somewhat complicated set of directories during a build. A full +description is available in [Output directory layout](/remote/output-directories). -The "main repo" is the source tree Bazel is run in. It usually corresponds to something you checked out from source control. The root of this directory is known as the "workspace root". +The "main repo" is the source tree Bazel is run in. It usually corresponds to +something you checked out from source control. The root of this directory is +known as the "workspace root". -Bazel puts all of its data under the "output user root". This is usually `$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the `--output_user_root` startup option. +Bazel puts all of its data under the "output user root". This is usually +`$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the +`--output_user_root` startup option. -The "install base" is where Bazel is extracted to. This is done automatically and each Bazel version gets a subdirectory based on its checksum under the install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed using the `--install_base` command line option. +The "install base" is where Bazel is extracted to. This is done automatically +and each Bazel version gets a subdirectory based on its checksum under the +install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed +using the `--install_base` command line option. -The "output base" is the place where the Bazel instance attached to a specific workspace writes to. Each output base has at most one Bazel server instance running at any time. It's usually at `$OUTPUT_USER_ROOT/<checksum of the path to the workspace>`. It can be changed using the `--output_base` startup option, which is, among other things, useful for getting around the limitation that only one Bazel instance can be running in any workspace at any given time. +The "output base" is the place where the Bazel instance attached to a specific +workspace writes to. Each output base has at most one Bazel server instance +running at any time. It's usually at `$OUTPUT_USER_ROOT/`. It can be changed using the `--output_base` startup option, +which is, among other things, useful for getting around the limitation that only +one Bazel instance can be running in any workspace at any given time. The output directory contains, among other things: -- The fetched external repositories at `$OUTPUT_BASE/external`. -- The exec root, a directory that contains symlinks to all the source code for the current build. It's located at `$OUTPUT_BASE/execroot`. During the build, the working directory is `$EXECROOT/<name of main repository>`. We are planning to change this to `$EXECROOT`, although it's a long term plan because it's a very incompatible change. -- Files built during the build. +* The fetched external repositories at `$OUTPUT_BASE/external`. +* The exec root, a directory that contains symlinks to all the source + code for the current build. It's located at `$OUTPUT_BASE/execroot`. During + the build, the working directory is `$EXECROOT/`. We are planning to change this to `$EXECROOT`, although it's a + long term plan because it's a very incompatible change. +* Files built during the build. ## The process of executing a command -Once the Bazel server gets control and is informed about a command it needs to execute, the following sequence of events happens: +Once the Bazel server gets control and is informed about a command it needs to +execute, the following sequence of events happens: -1. `BlazeCommandDispatcher` is informed about the new request. It decides whether the command needs a workspace to run in (almost every command except for ones that don't have anything to do with source code, such as version or help) and whether another command is running. +1. `BlazeCommandDispatcher` is informed about the new request. It decides + whether the command needs a workspace to run in (almost every command except + for ones that don't have anything to do with source code, such as version or + help) and whether another command is running. -2. The right command is found. Each command must implement the interface `BlazeCommand` and must have the `@Command` annotation (this is a bit of an antipattern, it would be nice if all the metadata a command needs was described by methods on `BlazeCommand`) +2. The right command is found. Each command must implement the interface + `BlazeCommand` and must have the `@Command` annotation (this is a bit of an + antipattern, it would be nice if all the metadata a command needs was + described by methods on `BlazeCommand`) -3. The command line options are parsed. Each command has different command line options, which are described in the `@Command` annotation. +3. The command line options are parsed. Each command has different command line + options, which are described in the `@Command` annotation. -4. An event bus is created. The event bus is a stream for events that happen during the build. Some of these are exported to outside of Bazel under the aegis of the Build Event Protocol in order to tell the world how the build goes. +4. An event bus is created. The event bus is a stream for events that happen + during the build. Some of these are exported to outside of Bazel under the + aegis of the Build Event Protocol in order to tell the world how the build + goes. -5. The command gets control. The most interesting commands are those that run a build: build, test, run, coverage and so on: this functionality is implemented by `BuildTool`. +5. The command gets control. The most interesting commands are those that run a + build: build, test, run, coverage and so on: this functionality is + implemented by `BuildTool`. -6. The set of target patterns on the command line is parsed and wildcards like `//pkg:all` and `//pkg/...` are resolved. This is implemented in `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as `TargetPatternPhaseValue`. +6. The set of target patterns on the command line is parsed and wildcards like + `//pkg:all` and `//pkg/...` are resolved. This is implemented in + `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as + `TargetPatternPhaseValue`. -7. The loading/analysis phase is run to produce the action graph (a directed acyclic graph of commands that need to be executed for the build). +7. The loading/analysis phase is run to produce the action graph (a directed + acyclic graph of commands that need to be executed for the build). -8. The execution phase is run. This means running every action required to build the top-level targets that are requested are run. +8. The execution phase is run. This means running every action required to + build the top-level targets that are requested are run. ## Command line options -The command line options for a Bazel invocation are described in an `OptionsParsingResult` object, which in turn contains a map from "option classes" to the values of the options. An "option class" is a subclass of `OptionsBase` and groups command line options together that are related to each other. For example: - -1. Options related to a programming language (`CppOptions` or `JavaOptions`). These should be a subclass of `FragmentOptions` and are eventually wrapped into a `BuildOptions` object. -2. Options related to the way Bazel executes actions (`ExecutionOptions`) - -These options are designed to be consumed in the analysis phase and (either through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). Some of them (for example, whether to do C++ include scanning or not) are read in the execution phase, but that always requires explicit plumbing since `BuildConfiguration` is not available then. For more information, see the section "Configurations". - -**WARNING:** We like to pretend that `OptionsBase` instances are immutable and use them that way (such as a part of `SkyKeys`). This is not the case and modifying them is a really good way to break Bazel in subtle ways that are hard to debug. Unfortunately, making them actually immutable is a large endeavor. (Modifying a `FragmentOptions` immediately after construction before anyone else gets a chance to keep a reference to it and before `equals()` or `hashCode()` is called on it is okay.) +The command line options for a Bazel invocation are described in an +`OptionsParsingResult` object, which in turn contains a map from "option +classes" to the values of the options. An "option class" is a subclass of +`OptionsBase` and groups command line options together that are related to each +other. For example: + +1. Options related to a programming language (`CppOptions` or `JavaOptions`). + These should be a subclass of `FragmentOptions` and are eventually wrapped + into a `BuildOptions` object. +2. Options related to the way Bazel executes actions (`ExecutionOptions`) + +These options are designed to be consumed in the analysis phase and (either +through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). +Some of them (for example, whether to do C++ include scanning or not) are read +in the execution phase, but that always requires explicit plumbing since +`BuildConfiguration` is not available then. For more information, see the +section "Configurations". + +**WARNING:** We like to pretend that `OptionsBase` instances are immutable and +use them that way (such as a part of `SkyKeys`). This is not the case and +modifying them is a really good way to break Bazel in subtle ways that are hard +to debug. Unfortunately, making them actually immutable is a large endeavor. +(Modifying a `FragmentOptions` immediately after construction before anyone else +gets a chance to keep a reference to it and before `equals()` or `hashCode()` is +called on it is okay.) Bazel learns about option classes in the following ways: -1. Some are hard-wired into Bazel (`CommonCommandOptions`) -2. From the `@Command` annotation on each Bazel command -3. From `ConfiguredRuleClassProvider` (these are command line options related to individual programming languages) -4. Starlark rules can also define their own options (see [here](/extending/config)) +1. Some are hard-wired into Bazel (`CommonCommandOptions`) +2. From the `@Command` annotation on each Bazel command +3. From `ConfiguredRuleClassProvider` (these are command line options related + to individual programming languages) +4. Starlark rules can also define their own options (see + [here](/extending/config)) -Each option (excluding Starlark-defined options) is a member variable of a `FragmentOptions` subclass that has the `@Option` annotation, which specifies the name and the type of the command line option along with some help text. +Each option (excluding Starlark-defined options) is a member variable of a +`FragmentOptions` subclass that has the `@Option` annotation, which specifies +the name and the type of the command line option along with some help text. -The Java type of the value of a command line option is usually something simple (a string, an integer, a Boolean, a label, etc.). However, we also support options of more complicated types; in this case, the job of converting from the command line string to the data type falls to an implementation of `com.google.devtools.common.options.Converter`. +The Java type of the value of a command line option is usually something simple +(a string, an integer, a Boolean, a label, etc.). However, we also support +options of more complicated types; in this case, the job of converting from the +command line string to the data type falls to an implementation of +`com.google.devtools.common.options.Converter`. ## The source tree, as seen by Bazel -Bazel is in the business of building software, which happens by reading and interpreting the source code. The totality of the source code Bazel operates on is called "the workspace" and it is structured into repositories, packages and rules. +Bazel is in the business of building software, which happens by reading and +interpreting the source code. The totality of the source code Bazel operates on +is called "the workspace" and it is structured into repositories, packages and +rules. ### Repositories -A "repository" is a source tree on which a developer works; it usually represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, that is, a single source tree that contains all source code used to run the build. Bazel, in contrast, supports projects whose source code spans multiple repositories. The repository from which Bazel is invoked is called the "main repository", the others are called "external repositories". +A "repository" is a source tree on which a developer works; it usually +represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, +that is, a single source tree that contains all source code used to run the build. +Bazel, in contrast, supports projects whose source code spans multiple +repositories. The repository from which Bazel is invoked is called the "main +repository", the others are called "external repositories". -A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The main repo is the source tree where you're invoking Bazel from. External repos are defined in various ways; see [external dependencies overview](/external/overview) for more information. +A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or +in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The +main repo is the source tree where you're invoking Bazel from. External repos +are defined in various ways; see [external dependencies +overview](/external/overview) for more information. -Code of external repositories is symlinked or downloaded under `$OUTPUT_BASE/external`. +Code of external repositories is symlinked or downloaded under +`$OUTPUT_BASE/external`. -When running the build, the whole source tree needs to be pieced together; this is done by `SymlinkForest`, which symlinks every package in the main repository to `$EXECROOT` and every external repository to either `$EXECROOT/external` or `$EXECROOT/..`. +When running the build, the whole source tree needs to be pieced together; this +is done by `SymlinkForest`, which symlinks every package in the main repository +to `$EXECROOT` and every external repository to either `$EXECROOT/external` or +`$EXECROOT/..`. ### Packages -Every repository is composed of packages, a collection of related files and a specification of the dependencies. These are specified by a file called `BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this file name. However, it turned out to be a commonly used path segment, especially on Windows, where file names are case-insensitive. - -Packages are independent of each other: changes to the `BUILD` file of a package cannot cause other packages to change. The addition or removal of `BUILD` files \_can \_change other packages, since recursive globs stop at package boundaries and thus the presence of a `BUILD` file stops the recursion. - -The evaluation of a `BUILD` file is called "package loading". It's implemented in the class `PackageFactory`, works by calling the Starlark interpreter and requires knowledge of the set of available rule classes. The result of package loading is a `Package` object. It's mostly a map from a string (the name of a target) to the target itself. - -A large chunk of complexity during package loading is globbing: Bazel does not require every source file to be explicitly listed and instead can run globs (such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that descend into subdirectories (but not into subpackages). This requires access to the file system and since that can be slow, we implement all sorts of tricks to make it run in parallel and as efficiently as possible. +Every repository is composed of packages, a collection of related files and +a specification of the dependencies. These are specified by a file called +`BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason +why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this +file name. However, it turned out to be a commonly used path segment, especially +on Windows, where file names are case-insensitive. + +Packages are independent of each other: changes to the `BUILD` file of a package +cannot cause other packages to change. The addition or removal of `BUILD` files +_can _change other packages, since recursive globs stop at package boundaries +and thus the presence of a `BUILD` file stops the recursion. + +The evaluation of a `BUILD` file is called "package loading". It's implemented +in the class `PackageFactory`, works by calling the Starlark interpreter and +requires knowledge of the set of available rule classes. The result of package +loading is a `Package` object. It's mostly a map from a string (the name of a +target) to the target itself. + +A large chunk of complexity during package loading is globbing: Bazel does not +require every source file to be explicitly listed and instead can run globs +(such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that +descend into subdirectories (but not into subpackages). This requires access to +the file system and since that can be slow, we implement all sorts of tricks to +make it run in parallel and as efficiently as possible. Globbing is implemented in the following classes: -- `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber -- `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to the legacy globber in order to avoid "Skyframe restarts" (described below) +* `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber +* `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to + the legacy globber in order to avoid "Skyframe restarts" (described below) -The `Package` class itself contains some members that are exclusively used to parse the "external" package (related to external dependencies) and which do not make sense for real packages. This is a design flaw because objects describing regular packages should not contain fields that describe something else. These include: +The `Package` class itself contains some members that are exclusively used to +parse the "external" package (related to external dependencies) and which do not +make sense for real packages. This is +a design flaw because objects describing regular packages should not contain +fields that describe something else. These include: -- The repository mappings -- The registered toolchains -- The registered execution platforms +* The repository mappings +* The registered toolchains +* The registered execution platforms -Ideally, there would be more separation between parsing the "external" package from parsing regular packages so that `Package` does not need to cater for the needs of both. This is unfortunately difficult to do because the two are intertwined quite deeply. +Ideally, there would be more separation between parsing the "external" package +from parsing regular packages so that `Package` does not need to cater for the +needs of both. This is unfortunately difficult to do because the two are +intertwined quite deeply. ### Labels, Targets, and Rules Packages are composed of targets, which have the following types: -1. **Files:** things that are either the input or the output of the build. In Bazel parlance, we call them *artifacts* (discussed elsewhere). Not all files created during the build are targets; it's common for an output of Bazel not to have an associated label. -2. **Rules:** these describe steps to derive its outputs from its inputs. They are generally associated with a programming language (such as `cc_library`, `java_library` or `py_library`), but there are some language-agnostic ones (such as `genrule` or `filegroup`) -3. **Package groups:** discussed in the [Visibility](#visibility) section. - -The name of a target is called a *Label*. The syntax of labels is `@repo//pac/kage:name`, where `repo` is the name of the repository the Label is in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of the file (if the label refers to a source file) relative to the directory of the package. When referring to a target on the command line, some parts of the label can be omitted: - -1. If the repository is omitted, the label is taken to be in the main repository. -2. If the package part is omitted (such as `name` or `:name`), the label is taken to be in the package of the current working directory (relative paths containing uplevel references (..) are not allowed) - -A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may be implemented either in Starlark (the `rule()` function) or in Java (so called "native rules", type `RuleClass`). In the long term, every language-specific rule will be implemented in Starlark, but some legacy rule families (such as Java or C++) are still in Java for the time being. - -Starlark rule classes need to be imported at the beginning of `BUILD` files using the `load()` statement, whereas Java rule classes are "innately" known by Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. +1. **Files:** things that are either the input or the output of the build. In + Bazel parlance, we call them _artifacts_ (discussed elsewhere). Not all + files created during the build are targets; it's common for an output of + Bazel not to have an associated label. +2. **Rules:** these describe steps to derive its outputs from its inputs. They + are generally associated with a programming language (such as `cc_library`, + `java_library` or `py_library`), but there are some language-agnostic ones + (such as `genrule` or `filegroup`) +3. **Package groups:** discussed in the [Visibility](#visibility) section. + +The name of a target is called a _Label_. The syntax of labels is +`@repo//pac/kage:name`, where `repo` is the name of the repository the Label is +in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of +the file (if the label refers to a source file) relative to the directory of the +package. When referring to a target on the command line, some parts of the label +can be omitted: + +1. If the repository is omitted, the label is taken to be in the main + repository. +2. If the package part is omitted (such as `name` or `:name`), the label is taken + to be in the package of the current working directory (relative paths + containing uplevel references (..) are not allowed) + +A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may +be implemented either in Starlark (the `rule()` function) or in Java (so called +"native rules", type `RuleClass`). In the long term, every language-specific +rule will be implemented in Starlark, but some legacy rule families (such as Java +or C++) are still in Java for the time being. + +Starlark rule classes need to be imported at the beginning of `BUILD` files +using the `load()` statement, whereas Java rule classes are "innately" known by +Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. Rule classes contain information such as: -1. Its attributes (such as `srcs`, `deps`): their types, default values, constraints, etc. -2. The configuration transitions and aspects attached to each attribute, if any -3. The implementation of the rule -4. The transitive info providers the rule "usually" creates +1. Its attributes (such as `srcs`, `deps`): their types, default values, + constraints, etc. +2. The configuration transitions and aspects attached to each attribute, if any +3. The implementation of the rule +4. The transitive info providers the rule "usually" creates -**Terminology note:** In the codebase, we often use "Rule" to mean the target created by a rule class. But in Starlark and in user-facing documentation, "Rule" should be used exclusively to refer to the rule class itself; the target is just a "target". Also note that despite `RuleClass` having "class" in its name, there is no Java inheritance relationship between a rule class and targets of that type. +**Terminology note:** In the codebase, we often use "Rule" to mean the target +created by a rule class. But in Starlark and in user-facing documentation, +"Rule" should be used exclusively to refer to the rule class itself; the target +is just a "target". Also note that despite `RuleClass` having "class" in its +name, there is no Java inheritance relationship between a rule class and targets +of that type. ## Skyframe -The evaluation framework underlying Bazel is called Skyframe. Its model is that everything that needs to be built during a build is organized into a directed acyclic graph with edges pointing from any pieces of data to its dependencies, that is, other pieces of data that need to be known to construct it. - -The nodes in the graph are called `SkyValue`s and their names are called `SkyKey`s. Both are deeply immutable; only immutable objects should be reachable from them. This invariant almost always holds, and in case it doesn't (such as for the individual options classes `BuildOptions`, which is a member of `BuildConfigurationValue` and its `SkyKey`) we try really hard not to change them or to change them in only ways that are not observable from the outside. From this it follows that everything that is computed within Skyframe (such as configured targets) must also be immutable. - -The most convenient way to observe the Skyframe graph is to run `bazel dump --skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best to do it for tiny builds, since it can get pretty large. - -Skyframe lives in the `com.google.devtools.build.skyframe` package. The similarly-named package `com.google.devtools.build.lib.skyframe` contains the implementation of Bazel on top of Skyframe. More information about Skyframe is available [here](/reference/skyframe). - -To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the `SkyFunction` corresponding to the type of the key. During the function's evaluation, it may request other dependencies from Skyframe by calling the various overloads of `SkyFunction.Environment.getValue()`. This has the side-effect of registering those dependencies into Skyframe's internal graph, so that Skyframe will know to re-evaluate the function when any of its dependencies change. In other words, Skyframe's caching and incremental computation work at the granularity of `SkyFunction`s and `SkyValue`s. - -Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` will return null. The function should then yield control back to Skyframe by itself returning null. At some later point, Skyframe will evaluate the unavailable dependency, then restart the function from the beginning — only this time the `getValue()` call will succeed with a non-null result. - -A consequence of this is that any computation performed inside the `SkyFunction` prior to the restart must be repeated. But this does not include work done to evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work around this issue by: - -1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to limit the number of restarts. -2. Breaking up a `SkyValue` into separate pieces computed by different `SkyFunction`s, so that they can be computed and cached independently. This should be done strategically, since it has the potential to increases memory usage. -3. Storing state between restarts, either using `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache "behind the back of Skyframe". With complex SkyFunctions, state management between restarts can get tricky, so [`StateMachine`s](/contribute/statemachine-guide) were introduced for a structured approach to logical concurrency, including hooks to suspend and resume hierarchical computations within a `SkyFunction`. Example: [`DependencyResolver#computeDependencies`](https://developers.google.com/devsite/reference/markdown/links#reference_links) uses a `StateMachine` with `getState()` to compute the potentially huge set of direct dependencies of a configured target, which otherwise can result in expensive restarts. - -Fundamentally, Bazel need these types of workarounds because hundreds of thousands of in-flight Skyframe nodes is common, and Java's support of lightweight threads [does not outperform](/contribute/statemachine-guide#epilogue_eventually_removing_callbacks) the `StateMachine` implementation as of 2023. +The evaluation framework underlying Bazel is called Skyframe. Its model is that +everything that needs to be built during a build is organized into a directed +acyclic graph with edges pointing from any pieces of data to its dependencies, +that is, other pieces of data that need to be known to construct it. + +The nodes in the graph are called `SkyValue`s and their names are called +`SkyKey`s. Both are deeply immutable; only immutable objects should be +reachable from them. This invariant almost always holds, and in case it doesn't +(such as for the individual options classes `BuildOptions`, which is a member of +`BuildConfigurationValue` and its `SkyKey`) we try really hard not to change +them or to change them in only ways that are not observable from the outside. +From this it follows that everything that is computed within Skyframe (such as +configured targets) must also be immutable. + +The most convenient way to observe the Skyframe graph is to run `bazel dump +--skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best +to do it for tiny builds, since it can get pretty large. + +Skyframe lives in the `com.google.devtools.build.skyframe` package. The +similarly-named package `com.google.devtools.build.lib.skyframe` contains the +implementation of Bazel on top of Skyframe. More information about Skyframe is +available [here](/reference/skyframe). + +To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the +`SkyFunction` corresponding to the type of the key. During the function's +evaluation, it may request other dependencies from Skyframe by calling the +various overloads of `SkyFunction.Environment.getValue()`. This has the +side-effect of registering those dependencies into Skyframe's internal graph, so +that Skyframe will know to re-evaluate the function when any of its dependencies +change. In other words, Skyframe's caching and incremental computation work at +the granularity of `SkyFunction`s and `SkyValue`s. + +Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` +will return null. The function should then yield control back to Skyframe by +itself returning null. At some later point, Skyframe will evaluate the +unavailable dependency, then restart the function from the beginning — only this +time the `getValue()` call will succeed with a non-null result. + +A consequence of this is that any computation performed inside the `SkyFunction` +prior to the restart must be repeated. But this does not include work done to +evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work +around this issue by: + +1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to + limit the number of restarts. +2. Breaking up a `SkyValue` into separate pieces computed by different + `SkyFunction`s, so that they can be computed and cached independently. This + should be done strategically, since it has the potential to increases memory + usage. +3. Storing state between restarts, either using + `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache + "behind the back of Skyframe". With complex SkyFunctions, state management + between restarts can get tricky, so + [`StateMachine`s](/contribute/statemachine-guide) were introduced for a + structured approach to logical concurrency, including hooks to suspend and + resume hierarchical computations within a `SkyFunction`. Example: + [`DependencyResolver#computeDependencies`][statemachine_example] + uses a `StateMachine` with `getState()` to compute the potentially huge set + of direct dependencies of a configured target, which otherwise can result in + expensive restarts. + +[statemachine_example]: https://developers.google.com/devsite/reference/markdown/links#reference_links + +Fundamentally, Bazel need these types of workarounds because hundreds of +thousands of in-flight Skyframe nodes is common, and Java's support of +lightweight threads [does not outperform][virtual_threads] the +`StateMachine` implementation as of 2023. + +[virtual_threads]: /contribute/statemachine-guide#epilogue_eventually_removing_callbacks ## Starlark -Starlark is the domain-specific language people use to configure and extend Bazel. It's conceived as a restricted subset of Python that has far fewer types, more restrictions on control flow, and most importantly, strong immutability guarantees to enable concurrent reads. It is not Turing-complete, which discourages some (but not all) users from trying to accomplish general programming tasks within the language. +Starlark is the domain-specific language people use to configure and extend +Bazel. It's conceived as a restricted subset of Python that has far fewer types, +more restrictions on control flow, and most importantly, strong immutability +guarantees to enable concurrent reads. It is not Turing-complete, which +discourages some (but not all) users from trying to accomplish general +programming tasks within the language. -Starlark is implemented in the `net.starlark.java` package. It also has an independent Go implementation [here](https://github.com/google/starlark-go). The Java implementation used in Bazel is currently an interpreter. +Starlark is implemented in the `net.starlark.java` package. +It also has an independent Go implementation +[here](https://github.com/google/starlark-go). The Java +implementation used in Bazel is currently an interpreter. Starlark is used in several contexts, including: -1. **`BUILD` files.** This is where new build targets are defined. Starlark code running in this context only has access to the contents of the `BUILD` file itself and `.bzl` files loaded by it. -2. **The `MODULE.bazel` file.** This is where external dependencies are defined. Starlark code running in this context only has very limited access to a few predefined directives. -3. **`.bzl` files.** This is where new build rules, repo rules, module extensions are defined. Starlark code here can define new functions and load from other `.bzl` files. +1. **`BUILD` files.** This is where new build targets are defined. Starlark + code running in this context only has access to the contents of the `BUILD` + file itself and `.bzl` files loaded by it. +2. **The `MODULE.bazel` file.** This is where external dependencies are + defined. Starlark code running in this context only has very limited access + to a few predefined directives. +3. **`.bzl` files.** This is where new build rules, repo rules, module + extensions are defined. Starlark code here can define new functions and load + from other `.bzl` files. -The dialects available for `BUILD` and `.bzl` files are slightly different because they express different things. A list of differences is available [here](/rules/language#differences-between-build-and-bzl-files). +The dialects available for `BUILD` and `.bzl` files are slightly different +because they express different things. A list of differences is available +[here](/rules/language#differences-between-build-and-bzl-files). More information about Starlark is available [here](/rules/language). ## The loading/analysis phase -The loading/analysis phase is where Bazel determines what actions are needed to build a particular rule. Its basic unit is a "configured target", which is, quite sensibly, a (target, configuration) pair. - -It's called the "loading/analysis phase" because it can be split into two distinct parts, which used to be serialized, but they can now overlap in time: - -1. Loading packages, that is, turning `BUILD` files into the `Package` objects that represent them -2. Analyzing configured targets, that is, running the implementation of the rules to produce the action graph - -Each configured target in the transitive closure of the configured targets requested on the command line must be analyzed bottom-up; that is, leaf nodes first, then up to the ones on the command line. The inputs to the analysis of a single configured target are: - -1. **The configuration.** ("how" to build that rule; for example, the target platform but also things like command line options the user wants to be passed to the C++ compiler) -2. **The direct dependencies.** Their transitive info providers are available to the rule being analyzed. They are called like that because they provide a "roll-up" of the information in the transitive closure of the configured target, such as all the .jar files on the classpath or all the .o files that need to be linked into a C++ binary) -3. **The target itself**. This is the result of loading the package the target is in. For rules, this includes its attributes, which is usually what matters. -4. **The implementation of the configured target.** For rules, this can either be in Starlark or in Java. All non-rule configured targets are implemented in Java. +The loading/analysis phase is where Bazel determines what actions are needed to +build a particular rule. Its basic unit is a "configured target", which is, +quite sensibly, a (target, configuration) pair. + +It's called the "loading/analysis phase" because it can be split into two +distinct parts, which used to be serialized, but they can now overlap in time: + +1. Loading packages, that is, turning `BUILD` files into the `Package` objects + that represent them +2. Analyzing configured targets, that is, running the implementation of the + rules to produce the action graph + +Each configured target in the transitive closure of the configured targets +requested on the command line must be analyzed bottom-up; that is, leaf nodes +first, then up to the ones on the command line. The inputs to the analysis of +a single configured target are: + +1. **The configuration.** ("how" to build that rule; for example, the target + platform but also things like command line options the user wants to be + passed to the C++ compiler) +2. **The direct dependencies.** Their transitive info providers are available + to the rule being analyzed. They are called like that because they provide a + "roll-up" of the information in the transitive closure of the configured + target, such as all the .jar files on the classpath or all the .o files that + need to be linked into a C++ binary) +3. **The target itself**. This is the result of loading the package the target + is in. For rules, this includes its attributes, which is usually what + matters. +4. **The implementation of the configured target.** For rules, this can either + be in Starlark or in Java. All non-rule configured targets are implemented + in Java. The output of analyzing a configured target is: -1. The transitive info providers that configured targets that depend on it can access -2. The artifacts it can create and the actions that produce them. +1. The transitive info providers that configured targets that depend on it can + access +2. The artifacts it can create and the actions that produce them. -The API offered to Java rules is `RuleContext`, which is the equivalent of the `ctx` argument of Starlark rules. Its API is more powerful, but at the same time, it's easier to do Bad Things™, for example to write code whose time or space complexity is quadratic (or worse), to make the Bazel server crash with a Java exception or to violate invariants (such as by inadvertently modifying an `Options` instance or by making a configured target mutable) +The API offered to Java rules is `RuleContext`, which is the equivalent of the +`ctx` argument of Starlark rules. Its API is more powerful, but at the same +time, it's easier to do Bad Things™, for example to write code whose time or +space complexity is quadratic (or worse), to make the Bazel server crash with a +Java exception or to violate invariants (such as by inadvertently modifying an +`Options` instance or by making a configured target mutable) -The algorithm that determines the direct dependencies of a configured target lives in `DependencyResolver.dependentNodeMap()`. +The algorithm that determines the direct dependencies of a configured target +lives in `DependencyResolver.dependentNodeMap()`. ### Configurations -Configurations are the "how" of building a target: for what platform, with what command line options, etc. - -The same target can be built for multiple configurations in the same build. This is useful, for example, when the same code is used for a tool that's run during the build and for the target code and we are cross-compiling or when we are building a fat Android app (one that contains native code for multiple CPU architectures) - -Conceptually, the configuration is a `BuildOptions` instance. However, in practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides additional sundry pieces of functionality. It propagates from the top of the dependency graph to the bottom. If it changes, the build needs to be re-analyzed. - -This results in anomalies like having to re-analyze the whole build if, for example, the number of requested test runs changes, even though that only affects test targets (we have plans to "trim" configurations so that this is not the case, but it's not ready yet). - -When a rule implementation needs part of the configuration, it needs to declare it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` . This is both to avoid mistakes (such as Python rules using the Java fragment) and to facilitate configuration trimming so that such as if Python options change, C++ targets don't need to be re-analyzed. - -The configuration of a rule is not necessarily the same as that of its "parent" rule. The process of changing the configuration in a dependency edge is called a "configuration transition". It can happen in two places: - -1. On a dependency edge. These transitions are specified in `Attribute.Builder.cfg()` and are functions from a `Rule` (where the transition happens) and a `BuildOptions` (the original configuration) to one or more `BuildOptions` (the output configuration). -2. On any incoming edge to a configured target. These are specified in `RuleClass.Builder.cfg()`. +Configurations are the "how" of building a target: for what platform, with what +command line options, etc. + +The same target can be built for multiple configurations in the same build. This +is useful, for example, when the same code is used for a tool that's run during +the build and for the target code and we are cross-compiling or when we are +building a fat Android app (one that contains native code for multiple CPU +architectures) + +Conceptually, the configuration is a `BuildOptions` instance. However, in +practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides +additional sundry pieces of functionality. It propagates from the top of the +dependency graph to the bottom. If it changes, the build needs to be +re-analyzed. + +This results in anomalies like having to re-analyze the whole build if, for +example, the number of requested test runs changes, even though that only +affects test targets (we have plans to "trim" configurations so that this is +not the case, but it's not ready yet). + +When a rule implementation needs part of the configuration, it needs to declare +it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` +. This is both to avoid mistakes (such as Python rules using the Java fragment) and +to facilitate configuration trimming so that such as if Python options change, C++ +targets don't need to be re-analyzed. + +The configuration of a rule is not necessarily the same as that of its "parent" +rule. The process of changing the configuration in a dependency edge is called a +"configuration transition". It can happen in two places: + +1. On a dependency edge. These transitions are specified in + `Attribute.Builder.cfg()` and are functions from a `Rule` (where the + transition happens) and a `BuildOptions` (the original configuration) to one + or more `BuildOptions` (the output configuration). +2. On any incoming edge to a configured target. These are specified in + `RuleClass.Builder.cfg()`. The relevant classes are `TransitionFactory` and `ConfigurationTransition`. Configuration transitions are used, for example: -1. To declare that a particular dependency is used during the build and it should thus be built in the execution architecture -2. To declare that a particular dependency must be built for multiple architectures (such as for native code in fat Android APKs) +1. To declare that a particular dependency is used during the build and it + should thus be built in the execution architecture +2. To declare that a particular dependency must be built for multiple + architectures (such as for native code in fat Android APKs) -If a configuration transition results in multiple configurations, it's called a *split transition.* +If a configuration transition results in multiple configurations, it's called a +_split transition._ -Configuration transitions can also be implemented in Starlark (documentation [here](/extending/config)) +Configuration transitions can also be implemented in Starlark (documentation +[here](/extending/config)) ### Transitive info providers -Transitive info providers are a way (and the \_only \_way) for configured targets to learn things about other configured targets that they depend on, and the only way to tell things about themselves to other configured targets that depend on them. The reason why "transitive" is in their name is that this is usually some sort of roll-up of the transitive closure of a configured target. - -There is generally a 1:1 correspondence between Java transitive info providers and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of `FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was deemed to be more Starlark-ish than a direct transliteration of the Java one). Their key is one of the following things: - -1. A Java Class object. This is only available for providers that are not accessible from Starlark. These providers are a subclass of `TransitiveInfoProvider`. -2. A string. This is legacy and heavily discouraged since it's susceptible to name clashes. Such transitive info providers are direct subclasses of `build.lib.packages.Info` . -3. A provider symbol. This can be created from Starlark using the `provider()` function and is the recommended way to create new providers. The symbol is represented by a `Provider.Key` instance in Java. - -New providers implemented in Java should be implemented using `BuiltinProvider`. `NativeProvider` is deprecated (we haven't had time to remove it yet) and `TransitiveInfoProvider` subclasses cannot be accessed from Starlark. +Transitive info providers are a way (and the _only _way) for configured targets +to learn things about other configured targets that they depend on, and the only +way to tell things about themselves to other configured targets that depend on +them. The reason why "transitive" is in their name is that this is usually some +sort of roll-up of the transitive closure of a configured target. + +There is generally a 1:1 correspondence between Java transitive info providers +and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of +`FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was +deemed to be more Starlark-ish than a direct transliteration of the Java one). +Their key is one of the following things: + +1. A Java Class object. This is only available for providers that are not + accessible from Starlark. These providers are a subclass of + `TransitiveInfoProvider`. +2. A string. This is legacy and heavily discouraged since it's susceptible to + name clashes. Such transitive info providers are direct subclasses of + `build.lib.packages.Info` . +3. A provider symbol. This can be created from Starlark using the `provider()` + function and is the recommended way to create new providers. The symbol is + represented by a `Provider.Key` instance in Java. + +New providers implemented in Java should be implemented using `BuiltinProvider`. +`NativeProvider` is deprecated (we haven't had time to remove it yet) and +`TransitiveInfoProvider` subclasses cannot be accessed from Starlark. ### Configured targets -Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a subclass for each rule class implemented in Java. Starlark configured targets are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . +Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a +subclass for each rule class implemented in Java. Starlark configured targets +are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . -Configured target factories should use `RuleConfiguredTargetBuilder` to construct their return value. It consists of the following things: +Configured target factories should use `RuleConfiguredTargetBuilder` to +construct their return value. It consists of the following things: -1. Their `filesToBuild`, the hazy concept of "the set of files this rule represents." These are the files that get built when the configured target is on the command line or in the srcs of a genrule. -2. Their runfiles, regular and data. -3. Their output groups. These are various "other sets of files" the rule can build. They can be accessed using the output\_group attribute of the filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. +1. Their `filesToBuild`, the hazy concept of "the set of files this rule + represents." These are the files that get built when the configured target + is on the command line or in the srcs of a genrule. +2. Their runfiles, regular and data. +3. Their output groups. These are various "other sets of files" the rule can + build. They can be accessed using the output\_group attribute of the + filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. ### Runfiles -Some binaries need data files to run. A prominent example is tests that need input files. This is represented in Bazel by the concept of "runfiles". A "runfiles tree" is a directory tree of the data files for a particular binary. It is created in the file system as a symlink tree with individual symlinks pointing to the files in the source or output trees. - -A set of runfiles is represented as a `Runfiles` instance. It is conceptually a map from the path of a file in the runfiles tree to the `Artifact` instance that represents it. It's a little more complicated than a single `Map` for two reasons: - -- Most of the time, the runfiles path of a file is the same as its execpath. We use this to save some RAM. -- There are various legacy kinds of entries in runfiles trees, which also need to be represented. - -Runfiles are collected using `RunfilesProvider`: an instance of this class represents the runfiles a configured target (such as a library) and its transitive closure needs and they are gathered like a nested set (in fact, they are implemented using nested sets under the cover): each target unions the runfiles of its dependencies, adds some of its own, then sends the resulting set upwards in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` instances, one for when the rule is depended on through the "data" attribute and one for every other kind of incoming dependency. This is because a target sometimes presents different runfiles when depended on through a data attribute than otherwise. This is undesired legacy behavior that we haven't gotten around removing yet. - -Runfiles of binaries are represented as an instance of `RunfilesSupport`. This is different from `Runfiles` because `RunfilesSupport` has the capability of actually being built (unlike `Runfiles`, which is just a mapping). This necessitates the following additional components: - -- **The input runfiles manifest.** This is a serialized description of the runfiles tree. It is used as a proxy for the contents of the runfiles tree and Bazel assumes that the runfiles tree changes if and only if the contents of the manifest change. -- **The output runfiles manifest.** This is used by runtime libraries that handle runfiles trees, notably on Windows, which sometimes doesn't support symbolic links. -- **Command line arguments** for running the binary whose runfiles the `RunfilesSupport` object represents. +Some binaries need data files to run. A prominent example is tests that need +input files. This is represented in Bazel by the concept of "runfiles". A +"runfiles tree" is a directory tree of the data files for a particular binary. +It is created in the file system as a symlink tree with individual symlinks +pointing to the files in the source or output trees. + +A set of runfiles is represented as a `Runfiles` instance. It is conceptually a +map from the path of a file in the runfiles tree to the `Artifact` instance that +represents it. It's a little more complicated than a single `Map` for two +reasons: + +* Most of the time, the runfiles path of a file is the same as its execpath. + We use this to save some RAM. +* There are various legacy kinds of entries in runfiles trees, which also need + to be represented. + +Runfiles are collected using `RunfilesProvider`: an instance of this class +represents the runfiles a configured target (such as a library) and its transitive +closure needs and they are gathered like a nested set (in fact, they are +implemented using nested sets under the cover): each target unions the runfiles +of its dependencies, adds some of its own, then sends the resulting set upwards +in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` +instances, one for when the rule is depended on through the "data" attribute and +one for every other kind of incoming dependency. This is because a target +sometimes presents different runfiles when depended on through a data attribute +than otherwise. This is undesired legacy behavior that we haven't gotten around +removing yet. + +Runfiles of binaries are represented as an instance of `RunfilesSupport`. This +is different from `Runfiles` because `RunfilesSupport` has the capability of +actually being built (unlike `Runfiles`, which is just a mapping). This +necessitates the following additional components: + +* **The input runfiles manifest.** This is a serialized description of the + runfiles tree. It is used as a proxy for the contents of the runfiles tree + and Bazel assumes that the runfiles tree changes if and only if the contents + of the manifest change. +* **The output runfiles manifest.** This is used by runtime libraries that + handle runfiles trees, notably on Windows, which sometimes doesn't support + symbolic links. +* **Command line arguments** for running the binary whose runfiles the + `RunfilesSupport` object represents. ### Aspects -Aspects are a way to "propagate computation down the dependency graph". They are described for users of Bazel [here](/extending/aspects). A good motivating example is protocol buffers: a `proto_library` rule should not know about any particular language, but building the implementation of a protocol buffer message (the "basic unit" of protocol buffers) in any programming language should be coupled to the `proto_library` rule so that if two targets in the same language depend on the same protocol buffer, it gets built only once. - -Just like configured targets, they are represented in Skyframe as a `SkyValue` and the way they are constructed is very similar to how configured targets are built: they have a factory class called `ConfiguredAspectFactory` that has access to a `RuleContext`, but unlike configured target factories, it also knows about the configured target it is attached to and its providers. - -The set of aspects propagated down the dependency graph is specified for each attribute using the `Attribute.Builder.aspects()` function. There are a few confusingly-named classes that participate in the process: - -1. `AspectClass` is the implementation of the aspect. It can be either in Java (in which case it's a subclass) or in Starlark (in which case it's an instance of `StarlarkAspectClass`). It's analogous to `RuleConfiguredTargetFactory`. -2. `AspectDefinition` is the definition of the aspect; it includes the providers it requires, the providers it provides and contains a reference to its implementation, such as the appropriate `AspectClass` instance. It's analogous to `RuleClass`. -3. `AspectParameters` is a way to parametrize an aspect that is propagated down the dependency graph. It's currently a string to string map. A good example of why it's useful is protocol buffers: if a language has multiple APIs, the information as to which API the protocol buffers should be built for should be propagated down the dependency graph. -4. `Aspect` represents all the data that's needed to compute an aspect that propagates down the dependency graph. It consists of the aspect class, its definition and its parameters. -5. `RuleAspect` is the function that determines which aspects a particular rule should propagate. It's a `Rule` -\> `Aspect` function. - -A somewhat unexpected complication is that aspects can attach to other aspects; for example, an aspect collecting the classpath for a Java IDE will probably want to know about all the .jar files on the classpath, but some of them are protocol buffers. In that case, the IDE aspect will want to attach to the (`proto_library` rule + Java proto aspect) pair. - -The complexity of aspects on aspects is captured in the class `AspectCollection`. +Aspects are a way to "propagate computation down the dependency graph". They are +described for users of Bazel +[here](/extending/aspects). A good +motivating example is protocol buffers: a `proto_library` rule should not know +about any particular language, but building the implementation of a protocol +buffer message (the "basic unit" of protocol buffers) in any programming +language should be coupled to the `proto_library` rule so that if two targets in +the same language depend on the same protocol buffer, it gets built only once. + +Just like configured targets, they are represented in Skyframe as a `SkyValue` +and the way they are constructed is very similar to how configured targets are +built: they have a factory class called `ConfiguredAspectFactory` that has +access to a `RuleContext`, but unlike configured target factories, it also knows +about the configured target it is attached to and its providers. + +The set of aspects propagated down the dependency graph is specified for each +attribute using the `Attribute.Builder.aspects()` function. There are a few +confusingly-named classes that participate in the process: + +1. `AspectClass` is the implementation of the aspect. It can be either in Java + (in which case it's a subclass) or in Starlark (in which case it's an + instance of `StarlarkAspectClass`). It's analogous to + `RuleConfiguredTargetFactory`. +2. `AspectDefinition` is the definition of the aspect; it includes the + providers it requires, the providers it provides and contains a reference to + its implementation, such as the appropriate `AspectClass` instance. It's + analogous to `RuleClass`. +3. `AspectParameters` is a way to parametrize an aspect that is propagated down + the dependency graph. It's currently a string to string map. A good example + of why it's useful is protocol buffers: if a language has multiple APIs, the + information as to which API the protocol buffers should be built for should + be propagated down the dependency graph. +4. `Aspect` represents all the data that's needed to compute an aspect that + propagates down the dependency graph. It consists of the aspect class, its + definition and its parameters. +5. `RuleAspect` is the function that determines which aspects a particular rule + should propagate. It's a `Rule` -> `Aspect` function. + +A somewhat unexpected complication is that aspects can attach to other aspects; +for example, an aspect collecting the classpath for a Java IDE will probably +want to know about all the .jar files on the classpath, but some of them are +protocol buffers. In that case, the IDE aspect will want to attach to the +(`proto_library` rule + Java proto aspect) pair. + +The complexity of aspects on aspects is captured in the class +`AspectCollection`. ### Platforms and toolchains -Bazel supports multi-platform builds, that is, builds where there may be multiple architectures where build actions run and multiple architectures for which code is built. These architectures are referred to as *platforms* in Bazel parlance (full documentation [here](/extending/platforms)) - -A platform is described by a key-value mapping from *constraint settings* (such as the concept of "CPU architecture") to *constraint values* (such as a particular CPU like x86\_64). We have a "dictionary" of the most commonly used constraint settings and values in the `@platforms` repository. - -The concept of *toolchain* comes from the fact that depending on what platforms the build is running on and what platforms are targeted, one may need to use different compilers; for example, a particular C++ toolchain may run on a specific OS and be able to target some other OSes. Bazel must determine the C++ compiler that is used based on the set execution and target platform (documentation for toolchains [here](/extending/toolchains)). - -In order to do this, toolchains are annotated with the set of execution and target platform constraints they support. In order to do this, the definition of a toolchain are split into two parts: - -1. A `toolchain()` rule that describes the set of execution and target constraints a toolchain supports and tells what kind (such as C++ or Java) of toolchain it is (the latter is represented by the `toolchain_type()` rule) -2. A language-specific rule that describes the actual toolchain (such as `cc_toolchain()`) - -This is done in this way because we need to know the constraints for every toolchain in order to do toolchain resolution and language-specific `*_toolchain()` rules contain much more information than that, so they take more time to load. +Bazel supports multi-platform builds, that is, builds where there may be +multiple architectures where build actions run and multiple architectures for +which code is built. These architectures are referred to as _platforms_ in Bazel +parlance (full documentation +[here](/extending/platforms)) + +A platform is described by a key-value mapping from _constraint settings_ (such as +the concept of "CPU architecture") to _constraint values_ (such as a particular CPU +like x86\_64). We have a "dictionary" of the most commonly used constraint +settings and values in the `@platforms` repository. + +The concept of _toolchain_ comes from the fact that depending on what platforms +the build is running on and what platforms are targeted, one may need to use +different compilers; for example, a particular C++ toolchain may run on a +specific OS and be able to target some other OSes. Bazel must determine the C++ +compiler that is used based on the set execution and target platform +(documentation for toolchains +[here](/extending/toolchains)). + +In order to do this, toolchains are annotated with the set of execution and +target platform constraints they support. In order to do this, the definition of +a toolchain are split into two parts: + +1. A `toolchain()` rule that describes the set of execution and target + constraints a toolchain supports and tells what kind (such as C++ or Java) of + toolchain it is (the latter is represented by the `toolchain_type()` rule) +2. A language-specific rule that describes the actual toolchain (such as + `cc_toolchain()`) + +This is done in this way because we need to know the constraints for every +toolchain in order to do toolchain resolution and language-specific +`*_toolchain()` rules contain much more information than that, so they take more +time to load. Execution platforms are specified in one of the following ways: -1. In the MODULE.bazel file using the `register_execution_platforms()` function -2. On the command line using the --extra\_execution\_platforms command line option - -The set of available execution platforms is computed in `RegisteredExecutionPlatformsFunction` . - -The target platform for a configured target is determined by `PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we eventually want to support multiple target platforms, but it's not implemented yet. - -The set of toolchains to be used for a configured target is determined by `ToolchainResolutionFunction`. It is a function of: - -- The set of registered toolchains (in the MODULE.bazel file and the configuration) -- The desired execution and target platforms (in the configuration) -- The set of toolchain types that are required by the configured target (in `UnloadedToolchainContextKey)` -- The set of execution platform constraints of the configured target (the `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` - -Its result is an `UnloadedToolchainContext`, which is essentially a map from toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of the selected toolchain. It's called "unloaded" because it does not contain the toolchains themselves, only their labels. - -Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` and used by the implementation of the configured target that requested them. - -We also have a legacy system that relies on there being one single "host" configuration and target configurations being represented by various configuration flags, such as `--cpu` . We are gradually transitioning to the above system. In order to handle cases where people rely on the legacy configuration values, we have implemented [platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) to translate between the legacy flags and the new-style platform constraints. Their code is in `PlatformMappingFunction` and uses a non-Starlark "little language". +1. In the MODULE.bazel file using the `register_execution_platforms()` function +2. On the command line using the --extra\_execution\_platforms command line + option + +The set of available execution platforms is computed in +`RegisteredExecutionPlatformsFunction` . + +The target platform for a configured target is determined by +`PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we +eventually want to support multiple target platforms, but it's not implemented +yet. + +The set of toolchains to be used for a configured target is determined by +`ToolchainResolutionFunction`. It is a function of: + +* The set of registered toolchains (in the MODULE.bazel file and the + configuration) +* The desired execution and target platforms (in the configuration) +* The set of toolchain types that are required by the configured target (in + `UnloadedToolchainContextKey)` +* The set of execution platform constraints of the configured target (the + `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` + +Its result is an `UnloadedToolchainContext`, which is essentially a map from +toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of +the selected toolchain. It's called "unloaded" because it does not contain the +toolchains themselves, only their labels. + +Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` +and used by the implementation of the configured target that requested them. + +We also have a legacy system that relies on there being one single "host" +configuration and target configurations being represented by various +configuration flags, such as `--cpu` . We are gradually transitioning to the above +system. In order to handle cases where people rely on the legacy configuration +values, we have implemented +[platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) +to translate between the legacy flags and the new-style platform constraints. +Their code is in `PlatformMappingFunction` and uses a non-Starlark "little +language". ### Constraints -Sometimes one wants to designate a target as being compatible with only a few platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: +Sometimes one wants to designate a target as being compatible with only a few +platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: -- Rule-specific constraints -- `environment_group()` / `environment()` -- Platform constraints +* Rule-specific constraints +* `environment_group()` / `environment()` +* Platform constraints -Rule-specific constraints are mostly used within Google for Java rules; they are on their way out and they are not available in Bazel, but the source code may contain references to it. The attribute that governs this is called `constraints=` . +Rule-specific constraints are mostly used within Google for Java rules; they are +on their way out and they are not available in Bazel, but the source code may +contain references to it. The attribute that governs this is called +`constraints=` . -#### environment\_group() and environment() +#### environment_group() and environment() These rules are a legacy mechanism and are not widely used. -All build rules can declare which "environments" they can be built for, where an "environment" is an instance of the `environment()` rule. +All build rules can declare which "environments" they can be built for, where an +"environment" is an instance of the `environment()` rule. There are various ways supported environments can be specified for a rule: -1. Through the `restricted_to=` attribute. This is the most direct form of specification; it declares the exact set of environments the rule supports. -2. Through the `compatible_with=` attribute. This declares environments a rule supports in addition to "standard" environments that are supported by default. -3. Through the package-level attributes `default_restricted_to=` and `default_compatible_with=`. -4. Through default specifications in `environment_group()` rules. Every environment belongs to a group of thematically related peers (such as "CPU architectures", "JDK versions" or "mobile operating systems"). The definition of an environment group includes which of these environments should be supported by "default" if not otherwise specified by the `restricted_to=` / `environment()` attributes. A rule with no such attributes inherits all defaults. -5. Through a rule class default. This overrides global defaults for all instances of the given rule class. This can be used, for example, to make all `*_test` rules testable without each instance having to explicitly declare this capability. - -`environment()` is implemented as a regular rule whereas `environment_group()` is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a function that is available by default from Starlark (`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous target. This is to avoid a cyclic dependency that would arise because each environment needs to declare the environment group it belongs to and each environment group needs to declare its default environments. - -A build can be restricted to a certain environment with the `--target_environment` command line option. - -The implementation of the constraint check is in `RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. +1. Through the `restricted_to=` attribute. This is the most direct form of + specification; it declares the exact set of environments the rule supports. +2. Through the `compatible_with=` attribute. This declares environments a rule + supports in addition to "standard" environments that are supported by + default. +3. Through the package-level attributes `default_restricted_to=` and + `default_compatible_with=`. +4. Through default specifications in `environment_group()` rules. Every + environment belongs to a group of thematically related peers (such as "CPU + architectures", "JDK versions" or "mobile operating systems"). The + definition of an environment group includes which of these environments + should be supported by "default" if not otherwise specified by the + `restricted_to=` / `environment()` attributes. A rule with no such + attributes inherits all defaults. +5. Through a rule class default. This overrides global defaults for all + instances of the given rule class. This can be used, for example, to make + all `*_test` rules testable without each instance having to explicitly + declare this capability. + +`environment()` is implemented as a regular rule whereas `environment_group()` +is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a +function that is available by default from Starlark +(`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous +target. This is to avoid a cyclic dependency that would arise because each +environment needs to declare the environment group it belongs to and each +environment group needs to declare its default environments. + +A build can be restricted to a certain environment with the +`--target_environment` command line option. + +The implementation of the constraint check is in +`RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. #### Platform constraints -The current "official" way to describe what platforms a target is compatible with is by using the same constraints used to describe toolchains and platforms. It was implemented in pull request [#10945](https://github.com/bazelbuild/bazel/pull/10945). +The current "official" way to describe what platforms a target is compatible +with is by using the same constraints used to describe toolchains and platforms. +It was implemented in pull request +[#10945](https://github.com/bazelbuild/bazel/pull/10945). ### Visibility -If you work on a large codebase with a lot of developers (like at Google), you want to take care to prevent everyone else from arbitrarily depending on your code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), people *will* come to rely on behaviors that you considered to be implementation details. +If you work on a large codebase with a lot of developers (like at Google), you +want to take care to prevent everyone else from arbitrarily depending on your +code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), +people _will_ come to rely on behaviors that you considered to be implementation +details. -Bazel supports this by the mechanism called *visibility*: you can limit which targets can depend on a particular target using the [visibility](/reference/be/common-definitions#common-attributes) attribute. This attribute is a little special because, although it holds a list of labels, these labels may encode a pattern over package names rather than a pointer to any particular target. (Yes, this is a design flaw.) +Bazel supports this by the mechanism called _visibility_: you can limit which +targets can depend on a particular target using the +[visibility](/reference/be/common-definitions#common-attributes) attribute. This +attribute is a little special because, although it holds a list of labels, these +labels may encode a pattern over package names rather than a pointer to any +particular target. (Yes, this is a design flaw.) This is implemented in the following places: -- The `RuleVisibility` interface represents a visibility declaration. It can be either a constant (fully public or fully private) or a list of labels. -- Labels can refer to either package groups (predefined list of packages), to packages directly (`//pkg:__pkg__`) or subtrees of packages (`//pkg:__subpackages__`). This is different from the command line syntax, which uses `//pkg:*` or `//pkg/...`. -- Package groups are implemented as their own target (`PackageGroup`) and configured target (`PackageGroupConfiguredTarget`). We could probably replace these with simple rules if we wanted to. Their logic is implemented with the help of: `PackageSpecification`, which corresponds to a single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds to a single `package_group`'s `packages` attribute; and `PackageSpecificationProvider`, which aggregates over a `package_group` and its transitive `includes`. -- The conversion from visibility label lists to dependencies is done in `DependencyResolver.visitTargetVisibility` and a few other miscellaneous places. -- The actual check is done in `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` +* The `RuleVisibility` interface represents a visibility declaration. It can + be either a constant (fully public or fully private) or a list of labels. +* Labels can refer to either package groups (predefined list of packages), to + packages directly (`//pkg:__pkg__`) or subtrees of packages + (`//pkg:__subpackages__`). This is different from the command line syntax, + which uses `//pkg:*` or `//pkg/...`. +* Package groups are implemented as their own target (`PackageGroup`) and + configured target (`PackageGroupConfiguredTarget`). We could probably + replace these with simple rules if we wanted to. Their logic is implemented + with the help of: `PackageSpecification`, which corresponds to a + single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds + to a single `package_group`'s `packages` attribute; and + `PackageSpecificationProvider`, which aggregates over a `package_group` and + its transitive `includes`. +* The conversion from visibility label lists to dependencies is done in + `DependencyResolver.visitTargetVisibility` and a few other miscellaneous + places. +* The actual check is done in + `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` ### Nested sets -Oftentimes, a configured target aggregates a set of files from its dependencies, adds its own, and wraps the aggregate set into a transitive info provider so that configured targets that depend on it can do the same. Examples: +Oftentimes, a configured target aggregates a set of files from its dependencies, +adds its own, and wraps the aggregate set into a transitive info provider so +that configured targets that depend on it can do the same. Examples: -- The C++ header files used for a build -- The object files that represent the transitive closure of a `cc_library` -- The set of .jar files that need to be on the classpath for a Java rule to compile or run -- The set of Python files in the transitive closure of a Python rule +* The C++ header files used for a build +* The object files that represent the transitive closure of a `cc_library` +* The set of .jar files that need to be on the classpath for a Java rule to + compile or run +* The set of Python files in the transitive closure of a Python rule -If we did this the naive way by using, for example, `List` or `Set`, we'd end up with quadratic memory usage: if there is a chain of N rules and each rule adds a file, we'd have 1+2+...+N collection members. +If we did this the naive way by using, for example, `List` or `Set`, we'd end up with +quadratic memory usage: if there is a chain of N rules and each rule adds a +file, we'd have 1+2+...+N collection members. -In order to get around this problem, we came up with the concept of a `NestedSet`. It's a data structure that is composed of other `NestedSet` instances and some members of its own, thereby forming a directed acyclic graph of sets. They are immutable and their members can be iterated over. We define multiple iteration order (`NestedSet.Order`): preorder, postorder, topological (a node always comes after its ancestors) and "don't care, but it should be the same each time". +In order to get around this problem, we came up with the concept of a +`NestedSet`. It's a data structure that is composed of other `NestedSet` +instances and some members of its own, thereby forming a directed acyclic graph +of sets. They are immutable and their members can be iterated over. We define +multiple iteration order (`NestedSet.Order`): preorder, postorder, topological +(a node always comes after its ancestors) and "don't care, but it should be the +same each time". The same data structure is called `depset` in Starlark. ### Artifacts and Actions -The actual build consists of a set of commands that need to be run to produce the output the user wants. The commands are represented as instances of the class `Action` and the files are represented as instances of the class `Artifact`. They are arranged in a bipartite, directed, acyclic graph called the "action graph". - -Artifacts come in two kinds: source artifacts (ones that are available before Bazel starts executing) and derived artifacts (ones that need to be built). Derived artifacts can themselves be multiple kinds: - -1. **Regular artifacts.** These are checked for up-to-dateness by computing their checksum, with mtime as a shortcut; we don't checksum the file if its ctime hasn't changed. -2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by calling readlink(). Unlike regular artifacts, these can be dangling symlinks. Usually used in cases where one then packs up some files into an archive of some sort. -3. **Tree artifacts.** These are not single files, but directory trees. They are checked for up-to-dateness by checking the set of files in it and their contents. They are represented as a `TreeArtifact`. -4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a rebuild. This is used exclusively for build stamp information: we don't want to do a rebuild just because the current time changed. - -There is no fundamental reason why source artifacts cannot be tree artifacts or unresolved symlink artifacts; it's just that we haven't implemented it yet. At the moment, source symlinks are always resolved, and while source directories are supported, their contents are entirely opaque to build rules and thus don't support the same kind of lazy command line expansion as tree artifacts do. - -Actions are best understood as a command that needs to be run, the environment it needs and the set of outputs it produces. The following things are the main components of the description of an action: - -- The command line that needs to be run -- The input artifacts it needs -- The environment variables that need to be set -- Annotations that describe the environment (such as platform) it needs to run in \\ - -There are also a few other special cases, like writing a file whose content is known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be separate classes), although Java and C++ have their own action types (`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). - -We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is pretty close, but C++ is a bit of a special-case due to .d file parsing and include scanning. - -The action graph is mostly "embedded" into the Skyframe graph: conceptually, the execution of an action is represented as an invocation of `ActionExecutionFunction`. The mapping from an action graph dependency edge to a Skyframe dependency edge is described in `ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few optimizations in order to keep the number of Skyframe edges low: - -- Derived artifacts do not have their own `SkyValue`s. Instead, `Artifact.getGeneratingActionKey()` is used to find out the key for the action that generates it -- Nested sets have their own Skyframe key. +The actual build consists of a set of commands that need to be run to produce +the output the user wants. The commands are represented as instances of the +class `Action` and the files are represented as instances of the class +`Artifact`. They are arranged in a bipartite, directed, acyclic graph called the +"action graph". + +Artifacts come in two kinds: source artifacts (ones that are available +before Bazel starts executing) and derived artifacts (ones that need to be +built). Derived artifacts can themselves be multiple kinds: + +1. **Regular artifacts.** These are checked for up-to-dateness by computing + their checksum, with mtime as a shortcut; we don't checksum the file if its + ctime hasn't changed. +2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by + calling readlink(). Unlike regular artifacts, these can be dangling + symlinks. Usually used in cases where one then packs up some files into an + archive of some sort. +3. **Tree artifacts.** These are not single files, but directory trees. They + are checked for up-to-dateness by checking the set of files in it and their + contents. They are represented as a `TreeArtifact`. +4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a + rebuild. This is used exclusively for build stamp information: we don't want + to do a rebuild just because the current time changed. + +There is no fundamental reason why source artifacts cannot be tree artifacts or +unresolved symlink artifacts; it's just that we haven't implemented it yet. At +the moment, source symlinks are always resolved, and while source directories +are supported, their contents are entirely opaque to build rules and thus don't +support the same kind of lazy command line expansion as tree artifacts do. + +Actions are best understood as a command that needs to be run, the environment +it needs and the set of outputs it produces. The following things are the main +components of the description of an action: + +* The command line that needs to be run +* The input artifacts it needs +* The environment variables that need to be set +* Annotations that describe the environment (such as platform) it needs to run in + \ + +There are also a few other special cases, like writing a file whose content is +known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are +a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be +separate classes), although Java and C++ have their own action types +(`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). + +We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is +pretty close, but C++ is a bit of a special-case due to .d file parsing and +include scanning. + +The action graph is mostly "embedded" into the Skyframe graph: conceptually, the +execution of an action is represented as an invocation of +`ActionExecutionFunction`. The mapping from an action graph dependency edge to a +Skyframe dependency edge is described in +`ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few +optimizations in order to keep the number of Skyframe edges low: + +* Derived artifacts do not have their own `SkyValue`s. Instead, + `Artifact.getGeneratingActionKey()` is used to find out the key for the + action that generates it +* Nested sets have their own Skyframe key. ### Shared actions -Some actions are generated by multiple configured targets; Starlark rules are more limited since they are only allowed to put their derived actions into a directory determined by their configuration and their package (but even so, rules in the same package can conflict), but rules implemented in Java can put derived artifacts anywhere. - -This is considered to be a misfeature, but getting rid of it is really hard because it produces significant savings in execution time when, for example, a source file needs to be processed somehow and that file is referenced by multiple rules (handwave-handwave). This comes at the cost of some RAM: each instance of a shared action needs to be stored in memory separately. - -If two actions generate the same output file, they must be exactly the same: have the same inputs, the same outputs and run the same command line. This equivalence relation is implemented in `Actions.canBeShared()` and it is verified between the analysis and execution phases by looking at every Action. This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` and is one of the few places in Bazel that requires a "global" view of the build. +Some actions are generated by multiple configured targets; Starlark rules are +more limited since they are only allowed to put their derived actions into a +directory determined by their configuration and their package (but even so, +rules in the same package can conflict), but rules implemented in Java can put +derived artifacts anywhere. + +This is considered to be a misfeature, but getting rid of it is really hard +because it produces significant savings in execution time when, for example, a +source file needs to be processed somehow and that file is referenced by +multiple rules (handwave-handwave). This comes at the cost of some RAM: each +instance of a shared action needs to be stored in memory separately. + +If two actions generate the same output file, they must be exactly the same: +have the same inputs, the same outputs and run the same command line. This +equivalence relation is implemented in `Actions.canBeShared()` and it is +verified between the analysis and execution phases by looking at every Action. +This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` +and is one of the few places in Bazel that requires a "global" view of the +build. ## The execution phase -This is when Bazel actually starts running build actions, such as commands that produce outputs. - -The first thing Bazel does after the analysis phase is to determine what Artifacts need to be built. The logic for this is encoded in `TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the configured targets on the command line and the contents of a special output group for the explicit purpose of expressing "if this target is on the command line, build these artifacts". - -The next step is creating the execution root. Since Bazel has the option to read source packages from different locations in the file system (`--package_path`), it needs to provide locally executed actions with a full source tree. This is handled by the class `SymlinkForest` and works by taking note of every target used in the analysis phase and building up a single directory tree that symlinks every package with a used target from its actual location. An alternative would be to pass the correct paths to commands (taking `--package_path` into account). This is undesirable because: - -- It changes action command lines when a package is moved from a package path entry to another (used to be a common occurrence) -- It results in different command lines if an action is run remotely than if it's run locally -- It requires a command line transformation specific to the tool in use (consider the difference between such as Java classpaths and C++ include paths) -- Changing the command line of an action invalidates its action cache entry -- `--package_path` is slowly and steadily being deprecated - -Then, Bazel starts traversing the action graph (the bipartite, directed graph composed of actions and their input and output artifacts) and running actions. The execution of each action is represented by an instance of the `SkyValue` class `ActionExecutionValue`. - -Since running an action is expensive, we have a few layers of caching that can be hit behind Skyframe: - -- `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts of `ActionExecutionFunction` cheap -- The local action cache contains data about the state of the file system -- Remote execution systems usually also contain their own cache +This is when Bazel actually starts running build actions, such as commands that +produce outputs. + +The first thing Bazel does after the analysis phase is to determine what +Artifacts need to be built. The logic for this is encoded in +`TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the +configured targets on the command line and the contents of a special output +group for the explicit purpose of expressing "if this target is on the command +line, build these artifacts". + +The next step is creating the execution root. Since Bazel has the option to read +source packages from different locations in the file system (`--package_path`), +it needs to provide locally executed actions with a full source tree. This is +handled by the class `SymlinkForest` and works by taking note of every target +used in the analysis phase and building up a single directory tree that symlinks +every package with a used target from its actual location. An alternative would +be to pass the correct paths to commands (taking `--package_path` into account). +This is undesirable because: + +* It changes action command lines when a package is moved from a package path + entry to another (used to be a common occurrence) +* It results in different command lines if an action is run remotely than if + it's run locally +* It requires a command line transformation specific to the tool in use + (consider the difference between such as Java classpaths and C++ include paths) +* Changing the command line of an action invalidates its action cache entry +* `--package_path` is slowly and steadily being deprecated + +Then, Bazel starts traversing the action graph (the bipartite, directed graph +composed of actions and their input and output artifacts) and running actions. +The execution of each action is represented by an instance of the `SkyValue` +class `ActionExecutionValue`. + +Since running an action is expensive, we have a few layers of caching that can +be hit behind Skyframe: + +* `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts + of `ActionExecutionFunction` cheap +* The local action cache contains data about the state of the file system +* Remote execution systems usually also contain their own cache ### The local action cache -This cache is another layer that sits behind Skyframe; even if an action is re-executed in Skyframe, it can still be a hit in the local action cache. It represents the state of the local file system and it's serialized to disk which means that when one starts up a new Bazel server, one can get local action cache hits even though the Skyframe graph is empty. +This cache is another layer that sits behind Skyframe; even if an action is +re-executed in Skyframe, it can still be a hit in the local action cache. It +represents the state of the local file system and it's serialized to disk which +means that when one starts up a new Bazel server, one can get local action cache +hits even though the Skyframe graph is empty. -This cache is checked for hits using the method `ActionCacheChecker.getTokenIfNeedToExecute()` . +This cache is checked for hits using the method +`ActionCacheChecker.getTokenIfNeedToExecute()` . -Contrary to its name, it's a map from the path of a derived artifact to the action that emitted it. The action is described as: +Contrary to its name, it's a map from the path of a derived artifact to the +action that emitted it. The action is described as: -1. The set of its input and output files and their checksum -2. Its "action key", which is usually the command line that was executed, but in general, represents everything that's not captured by the checksum of the input files (such as for `FileWriteAction`, it's the checksum of the data that's written) +1. The set of its input and output files and their checksum +2. Its "action key", which is usually the command line that was executed, but + in general, represents everything that's not captured by the checksum of the + input files (such as for `FileWriteAction`, it's the checksum of the data + that's written) -There is also a highly experimental "top-down action cache" that is still under development, which uses transitive hashes to avoid going to the cache as many times. +There is also a highly experimental "top-down action cache" that is still under +development, which uses transitive hashes to avoid going to the cache as many +times. ### Input discovery and input pruning -Some actions are more complicated than just having a set of inputs. Changes to the set of inputs of an action come in two forms: - -- An action may discover new inputs before its execution or decide that some of its inputs are not actually necessary. The canonical example is C++, where it's better to make an educated guess about what header files a C++ file uses from its transitive closure so that we don't heed to send every file to remote executors; therefore, we have an option not to register every header file as an "input", but scan the source file for transitively included headers and only mark those header files as inputs that are mentioned in `#include` statements (we overestimate so that we don't need to implement a full C preprocessor) This option is currently hard-wired to "false" in Bazel and is only used at Google. -- An action may realize that some files were not used during its execution. In C++, this is called ".d files": the compiler tells which header files were used after the fact, and in order to avoid the embarrassment of having worse incrementality than Make, Bazel makes use of this fact. This offers a better estimate than the include scanner because it relies on the compiler. +Some actions are more complicated than just having a set of inputs. Changes to +the set of inputs of an action come in two forms: + +* An action may discover new inputs before its execution or decide that some + of its inputs are not actually necessary. The canonical example is C++, + where it's better to make an educated guess about what header files a C++ + file uses from its transitive closure so that we don't heed to send every + file to remote executors; therefore, we have an option not to register every + header file as an "input", but scan the source file for transitively + included headers and only mark those header files as inputs that are + mentioned in `#include` statements (we overestimate so that we don't need to + implement a full C preprocessor) This option is currently hard-wired to + "false" in Bazel and is only used at Google. +* An action may realize that some files were not used during its execution. In + C++, this is called ".d files": the compiler tells which header files were + used after the fact, and in order to avoid the embarrassment of having worse + incrementality than Make, Bazel makes use of this fact. This offers a better + estimate than the include scanner because it relies on the compiler. These are implemented using methods on Action: -1. `Action.discoverInputs()` is called. It should return a nested set of Artifacts that are determined to be required. These must be source artifacts so that there are no dependency edges in the action graph that don't have an equivalent in the configured target graph. -2. The action is executed by calling `Action.execute()`. -3. At the end of `Action.execute()`, the action can call `Action.updateInputs()` to tell Bazel that not all of its inputs were needed. This can result in incorrect incremental builds if a used input is reported as unused. +1. `Action.discoverInputs()` is called. It should return a nested set of + Artifacts that are determined to be required. These must be source artifacts + so that there are no dependency edges in the action graph that don't have an + equivalent in the configured target graph. +2. The action is executed by calling `Action.execute()`. +3. At the end of `Action.execute()`, the action can call + `Action.updateInputs()` to tell Bazel that not all of its inputs were + needed. This can result in incorrect incremental builds if a used input is + reported as unused. -When an action cache returns a hit on a fresh Action instance (such as created after a server restart), Bazel calls `updateInputs()` itself so that the set of inputs reflects the result of input discovery and pruning done before. +When an action cache returns a hit on a fresh Action instance (such as created +after a server restart), Bazel calls `updateInputs()` itself so that the set of +inputs reflects the result of input discovery and pruning done before. -Starlark actions can make use of the facility to declare some inputs as unused using the `unused_inputs_list=` argument of `ctx.actions.run()`. +Starlark actions can make use of the facility to declare some inputs as unused +using the `unused_inputs_list=` argument of +`ctx.actions.run()`. ### Various ways to run actions: Strategies/ActionContexts -Some actions can be run in different ways. For example, a command line can be executed locally, locally but in various kinds of sandboxes, or remotely. The concept that embodies this is called an `ActionContext` (or `Strategy`, since we successfully went only halfway with a rename...) +Some actions can be run in different ways. For example, a command line can be +executed locally, locally but in various kinds of sandboxes, or remotely. The +concept that embodies this is called an `ActionContext` (or `Strategy`, since we +successfully went only halfway with a rename...) The life cycle of an action context is as follows: -1. When the execution phase is started, `BlazeModule` instances are asked what action contexts they have. This happens in the constructor of `ExecutionTool`. Action context types are identified by a Java `Class` instance that refers to a sub-interface of `ActionContext` and which interface the action context must implement. -2. The appropriate action context is selected from the available ones and is forwarded to `ActionExecutionContext` and `BlazeExecutor` . -3. Actions request contexts using `ActionExecutionContext.getContext()` and `BlazeExecutor.getStrategy()` (there should really be only one way to do it…) - -Strategies are free to call other strategies to do their jobs; this is used, for example, in the dynamic strategy that starts actions both locally and remotely, then uses whichever finishes first. - -One notable strategy is the one that implements persistent worker processes (`WorkerSpawnStrategy`). The idea is that some tools have a long startup time and should therefore be reused between actions instead of starting one anew for every action (This does represent a potential correctness issue, since Bazel relies on the promise of the worker process that it doesn't carry observable state between individual requests) - -If the tool changes, the worker process needs to be restarted. Whether a worker can be reused is determined by computing a checksum for the tool used using `WorkerFilesHash`. It relies on knowing which inputs of the action represent part of the tool and which represent inputs; this is determined by the creator of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are counted as parts of the tool. +1. When the execution phase is started, `BlazeModule` instances are asked what + action contexts they have. This happens in the constructor of + `ExecutionTool`. Action context types are identified by a Java `Class` + instance that refers to a sub-interface of `ActionContext` and which + interface the action context must implement. +2. The appropriate action context is selected from the available ones and is + forwarded to `ActionExecutionContext` and `BlazeExecutor` . +3. Actions request contexts using `ActionExecutionContext.getContext()` and + `BlazeExecutor.getStrategy()` (there should really be only one way to do + it…) + +Strategies are free to call other strategies to do their jobs; this is used, for +example, in the dynamic strategy that starts actions both locally and remotely, +then uses whichever finishes first. + +One notable strategy is the one that implements persistent worker processes +(`WorkerSpawnStrategy`). The idea is that some tools have a long startup time +and should therefore be reused between actions instead of starting one anew for +every action (This does represent a potential correctness issue, since Bazel +relies on the promise of the worker process that it doesn't carry observable +state between individual requests) + +If the tool changes, the worker process needs to be restarted. Whether a worker +can be reused is determined by computing a checksum for the tool used using +`WorkerFilesHash`. It relies on knowing which inputs of the action represent +part of the tool and which represent inputs; this is determined by the creator +of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are +counted as parts of the tool. More information about strategies (or action contexts!): -- Information about various strategies for running actions is available [here](https://jmmv.dev/2019/12/bazel-strategies.html). -- Information about the dynamic strategy, one where we run an action both locally and remotely to see whichever finishes first is available [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). -- Information about the intricacies of executing actions locally is available [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). +* Information about various strategies for running actions is available + [here](https://jmmv.dev/2019/12/bazel-strategies.html). +* Information about the dynamic strategy, one where we run an action both + locally and remotely to see whichever finishes first is available + [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). +* Information about the intricacies of executing actions locally is available + [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). ### The local resource manager -Bazel *can* run many actions in parallel. The number of local actions that *should* be run in parallel differs from action to action: the more resources an action requires, the less instances should be running at the same time to avoid overloading the local machine. +Bazel _can_ run many actions in parallel. The number of local actions that +_should_ be run in parallel differs from action to action: the more resources an +action requires, the less instances should be running at the same time to avoid +overloading the local machine. -This is implemented in the class `ResourceManager`: each action has to be annotated with an estimate of the local resources it requires in the form of a `ResourceSet` instance (CPU and RAM). Then when action contexts do something that requires local resources, they call `ResourceManager.acquireResources()` and are blocked until the required resources are available. +This is implemented in the class `ResourceManager`: each action has to be +annotated with an estimate of the local resources it requires in the form of a +`ResourceSet` instance (CPU and RAM). Then when action contexts do something +that requires local resources, they call `ResourceManager.acquireResources()` +and are blocked until the required resources are available. -A more detailed description of local resource management is available [here](https://jmmv.dev/2019/12/bazel-local-resources.html). +A more detailed description of local resource management is available +[here](https://jmmv.dev/2019/12/bazel-local-resources.html). ### The structure of the output directory -Each action requires a separate place in the output directory where it places its outputs. The location of derived artifacts is usually as follows: +Each action requires a separate place in the output directory where it places +its outputs. The location of derived artifacts is usually as follows: ``` -$EXECROOT/bazel-out/<configuration>/bin/<package>/<artifact name> +$EXECROOT/bazel-out//bin// ``` -How is the name of the directory that is associated with a particular configuration determined? There are two conflicting desirable properties: - -1. If two configurations can occur in the same build, they should have different directories so that both can have their own version of the same action; otherwise, if the two configurations disagree about such as the command line of an action producing the same output file, Bazel doesn't know which action to choose (an "action conflict") -2. If two configurations represent "roughly" the same thing, they should have the same name so that actions executed in one can be reused for the other if the command lines match: for example, changes to the command line options to the Java compiler should not result in C++ compile actions being re-run. - -So far, we have not come up with a principled way of solving this problem, which has similarities to the problem of configuration trimming. A longer discussion of options is available [here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). The main problematic areas are Starlark rules (whose authors usually aren't intimately familiar with Bazel) and aspects, which add another dimension to the space of things that can produce the "same" output file. - -The current approach is that the path segment for the configuration is `<CPU>-<compilation mode>` with various suffixes added so that configuration transitions implemented in Java don't result in action conflicts. In addition, a checksum of the set of Starlark configuration transitions is added so that users can't cause action conflicts. It is far from perfect. This is implemented in `OutputDirectories.buildMnemonic()` and relies on each configuration fragment adding its own part to the name of the output directory. +How is the name of the directory that is associated with a particular +configuration determined? There are two conflicting desirable properties: + +1. If two configurations can occur in the same build, they should have + different directories so that both can have their own version of the same + action; otherwise, if the two configurations disagree about such as the command + line of an action producing the same output file, Bazel doesn't know which + action to choose (an "action conflict") +2. If two configurations represent "roughly" the same thing, they should have + the same name so that actions executed in one can be reused for the other if + the command lines match: for example, changes to the command line options to + the Java compiler should not result in C++ compile actions being re-run. + +So far, we have not come up with a principled way of solving this problem, which +has similarities to the problem of configuration trimming. A longer discussion +of options is available +[here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). +The main problematic areas are Starlark rules (whose authors usually aren't +intimately familiar with Bazel) and aspects, which add another dimension to the +space of things that can produce the "same" output file. + +The current approach is that the path segment for the configuration is +`-` with various suffixes added so that configuration +transitions implemented in Java don't result in action conflicts. In addition, a +checksum of the set of Starlark configuration transitions is added so that users +can't cause action conflicts. It is far from perfect. This is implemented in +`OutputDirectories.buildMnemonic()` and relies on each configuration fragment +adding its own part to the name of the output directory. ## Tests Bazel has rich support for running tests. It supports: -- Running tests remotely (if a remote execution backend is available) -- Running tests multiple times in parallel (for deflaking or gathering timing data) -- Sharding tests (splitting test cases in same test over multiple processes for speed) -- Re-running flaky tests -- Grouping tests into test suites +* Running tests remotely (if a remote execution backend is available) +* Running tests multiple times in parallel (for deflaking or gathering timing + data) +* Sharding tests (splitting test cases in same test over multiple processes + for speed) +* Re-running flaky tests +* Grouping tests into test suites -Tests are regular configured targets that have a TestProvider, which describes how the test should be run: +Tests are regular configured targets that have a TestProvider, which describes +how the test should be run: -- The artifacts whose building result in the test being run. This is a "cache status" file that contains a serialized `TestResultData` message -- The number of times the test should be run -- The number of shards the test should be split into -- Some parameters about how the test should be run (such as the test timeout) +* The artifacts whose building result in the test being run. This is a "cache + status" file that contains a serialized `TestResultData` message +* The number of times the test should be run +* The number of shards the test should be split into +* Some parameters about how the test should be run (such as the test timeout) ### Determining which tests to run Determining which tests are run is an elaborate process. -First, during target pattern parsing, test suites are recursively expanded. The expansion is implemented in `TestsForTargetPatternFunction`. A somewhat surprising wrinkle is that if a test suite declares no tests, it refers to *every* test in its package. This is implemented in `Package.beforeBuild()` by adding an implicit attribute called `$implicit_tests` to test suite rules. - -Then, tests are filtered for size, tags, timeout and language according to the command line options. This is implemented in `TestFilter` and is called from `TargetPatternPhaseFunction.determineTests()` during target parsing and the result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason why rule attributes which can be filtered for are not configurable is that this happens before the analysis phase, therefore, the configuration is not available. - -This is then processed further in `BuildView.createResult()`: targets whose analysis failed are filtered out and tests are split into exclusive and non-exclusive tests. It's then put into `AnalysisResult`, which is how `ExecutionTool` knows which tests to run. - -In order to lend some transparency to this elaborate process, the `tests()` query operator (implemented in `TestsFunction`) is available to tell which tests are run when a particular target is specified on the command line. It's unfortunately a reimplementation, so it probably deviates from the above in multiple subtle ways. +First, during target pattern parsing, test suites are recursively expanded. The +expansion is implemented in `TestsForTargetPatternFunction`. A somewhat +surprising wrinkle is that if a test suite declares no tests, it refers to +_every_ test in its package. This is implemented in `Package.beforeBuild()` by +adding an implicit attribute called `$implicit_tests` to test suite rules. + +Then, tests are filtered for size, tags, timeout and language according to the +command line options. This is implemented in `TestFilter` and is called from +`TargetPatternPhaseFunction.determineTests()` during target parsing and the +result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason +why rule attributes which can be filtered for are not configurable is that this +happens before the analysis phase, therefore, the configuration is not +available. + +This is then processed further in `BuildView.createResult()`: targets whose +analysis failed are filtered out and tests are split into exclusive and +non-exclusive tests. It's then put into `AnalysisResult`, which is how +`ExecutionTool` knows which tests to run. + +In order to lend some transparency to this elaborate process, the `tests()` +query operator (implemented in `TestsFunction`) is available to tell which tests +are run when a particular target is specified on the command line. It's +unfortunately a reimplementation, so it probably deviates from the above in +multiple subtle ways. ### Running tests -The way the tests are run is by requesting cache status artifacts. This then results in the execution of a `TestRunnerAction`, which eventually calls the `TestActionContext` chosen by the `--test_strategy` command line option that runs the test in the requested way. - -Tests are run according to an elaborate protocol that uses environment variables to tell tests what's expected from them. A detailed description of what Bazel expects from tests and what tests can expect from Bazel is available [here](/reference/test-encyclopedia). At the simplest, an exit code of 0 means success, anything else means failure. - -In addition to the cache status file, each test process emits a number of other files. They are put in the "test log directory" which is the subdirectory called `testlogs` of the output directory of the target configuration: - -- `test.xml`, a JUnit-style XML file detailing the individual test cases in the test shard -- `test.log`, the console output of the test. stdout and stderr are not separated. -- `test.outputs`, the "undeclared outputs directory"; this is used by tests that want to output files in addition to what they print to the terminal. - -There are two things that can happen during test execution that cannot during building regular targets: exclusive test execution and output streaming. - -Some tests need to be executed in exclusive mode, for example not in parallel with other tests. This can be elicited either by adding `tags=["exclusive"]` to the test rule or running the test with `--test_strategy=exclusive` . Each exclusive test is run by a separate Skyframe invocation requesting the execution of the test after the "main" build. This is implemented in `SkyframeExecutor.runExclusiveTest()`. - -Unlike regular actions, whose terminal output is dumped when the action finishes, the user can request the output of tests to be streamed so that they get informed about the progress of a long-running test. This is specified by the `--test_output=streamed` command line option and implies exclusive test execution so that outputs of different tests are not interspersed. - -This is implemented in the aptly-named `StreamedTestOutput` class and works by polling changes to the `test.log` file of the test in question and dumping new bytes to the terminal where Bazel rules. - -Results of the executed tests are available on the event bus by observing various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). They are dumped to the Build Event Protocol and they are emitted to the console by `AggregatingTestListener`. +The way the tests are run is by requesting cache status artifacts. This then +results in the execution of a `TestRunnerAction`, which eventually calls the +`TestActionContext` chosen by the `--test_strategy` command line option that +runs the test in the requested way. + +Tests are run according to an elaborate protocol that uses environment variables +to tell tests what's expected from them. A detailed description of what Bazel +expects from tests and what tests can expect from Bazel is available +[here](/reference/test-encyclopedia). At the +simplest, an exit code of 0 means success, anything else means failure. + +In addition to the cache status file, each test process emits a number of other +files. They are put in the "test log directory" which is the subdirectory called +`testlogs` of the output directory of the target configuration: + +* `test.xml`, a JUnit-style XML file detailing the individual test cases in + the test shard +* `test.log`, the console output of the test. stdout and stderr are not + separated. +* `test.outputs`, the "undeclared outputs directory"; this is used by tests + that want to output files in addition to what they print to the terminal. + +There are two things that can happen during test execution that cannot during +building regular targets: exclusive test execution and output streaming. + +Some tests need to be executed in exclusive mode, for example not in parallel with +other tests. This can be elicited either by adding `tags=["exclusive"]` to the +test rule or running the test with `--test_strategy=exclusive` . Each exclusive +test is run by a separate Skyframe invocation requesting the execution of the +test after the "main" build. This is implemented in +`SkyframeExecutor.runExclusiveTest()`. + +Unlike regular actions, whose terminal output is dumped when the action +finishes, the user can request the output of tests to be streamed so that they +get informed about the progress of a long-running test. This is specified by the +`--test_output=streamed` command line option and implies exclusive test +execution so that outputs of different tests are not interspersed. + +This is implemented in the aptly-named `StreamedTestOutput` class and works by +polling changes to the `test.log` file of the test in question and dumping new +bytes to the terminal where Bazel rules. + +Results of the executed tests are available on the event bus by observing +various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). +They are dumped to the Build Event Protocol and they are emitted to the console +by `AggregatingTestListener`. ### Coverage collection -Coverage is reported by the tests in LCOV format in the files `bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . - -To collect coverage, each test execution is wrapped in a script called `collect_coverage.sh` . - -This script sets up the environment of the test to enable coverage collection and determine where the coverage files are written by the coverage runtime(s). It then runs the test. A test may itself run multiple subprocesses and consist of parts written in multiple different programming languages (with separate coverage collection runtimes). The wrapper script is responsible for converting the resulting files to LCOV format if necessary, and merges them into a single file. - -The interposition of `collect_coverage.sh` is done by the test strategies and requires `collect_coverage.sh` to be on the inputs of the test. This is accomplished by the implicit attribute `:coverage_support` which is resolved to the value of the configuration flag `--coverage_support` (see `TestConfiguration.TestOptions.coverageSupport`) - -Some languages do offline instrumentation, meaning that the coverage instrumentation is added at compile time (such as C++) and others do online instrumentation, meaning that coverage instrumentation is added at execution time. - -Another core concept is *baseline coverage*. This is the coverage of a library, binary, or test if no code in it was run. The problem it solves is that if you want to compute the test coverage for a binary, it is not enough to merge the coverage of all of the tests because there may be code in the binary that is not linked into any test. Therefore, what we do is to emit a coverage file for every binary which contains only the files we collect coverage for with no covered lines. The default baseline coverage file for a target is at `bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are encouraged to generate their own baseline coverage files with more meaningful content than just the names of the source files. - -We track two groups of files for coverage collection for each rule: the set of instrumented files and the set of instrumentation metadata files. - -The set of instrumented files is just that, a set of files to instrument. For online coverage runtimes, this can be used at runtime to decide which files to instrument. It is also used to implement baseline coverage. - -The set of instrumentation metadata files is the set of extra files a test needs to generate the LCOV files Bazel requires from it. In practice, this consists of runtime-specific files; for example, gcc emits .gcno files during compilation. These are added to the set of inputs of test actions if coverage mode is enabled. - -Whether or not coverage is being collected is stored in the `BuildConfiguration`. This is handy because it is an easy way to change the test action and the action graph depending on this bit, but it also means that if this bit is flipped, all targets need to be re-analyzed (some languages, such as C++ require different compiler options to emit code that can collect coverage, which mitigates this issue somewhat, since then a re-analysis is needed anyway). - -The coverage support files are depended on through labels in an implicit dependency so that they can be overridden by the invocation policy, which allows them to differ between the different versions of Bazel. Ideally, these differences would be removed, and we standardized on one of them. - -We also generate a "coverage report" which merges the coverage collected for every test in a Bazel invocation. This is handled by `CoverageReportActionFactory` and is called from `BuildView.createResult()` . It gets access to the tools it needs by looking at the `:coverage_report_generator` attribute of the first test that is executed. +Coverage is reported by the tests in LCOV format in the files +`bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . + +To collect coverage, each test execution is wrapped in a script called +`collect_coverage.sh` . + +This script sets up the environment of the test to enable coverage collection +and determine where the coverage files are written by the coverage runtime(s). +It then runs the test. A test may itself run multiple subprocesses and consist +of parts written in multiple different programming languages (with separate +coverage collection runtimes). The wrapper script is responsible for converting +the resulting files to LCOV format if necessary, and merges them into a single +file. + +The interposition of `collect_coverage.sh` is done by the test strategies and +requires `collect_coverage.sh` to be on the inputs of the test. This is +accomplished by the implicit attribute `:coverage_support` which is resolved to +the value of the configuration flag `--coverage_support` (see +`TestConfiguration.TestOptions.coverageSupport`) + +Some languages do offline instrumentation, meaning that the coverage +instrumentation is added at compile time (such as C++) and others do online +instrumentation, meaning that coverage instrumentation is added at execution +time. + +Another core concept is _baseline coverage_. This is the coverage of a library, +binary, or test if no code in it was run. The problem it solves is that if you +want to compute the test coverage for a binary, it is not enough to merge the +coverage of all of the tests because there may be code in the binary that is not +linked into any test. Therefore, what we do is to emit a coverage file for every +binary which contains only the files we collect coverage for with no covered +lines. The default baseline coverage file for a target is at +`bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are +encouraged to generate their own baseline coverage files with more meaningful +content than just the names of the source files. + +We track two groups of files for coverage collection for each rule: the set of +instrumented files and the set of instrumentation metadata files. + +The set of instrumented files is just that, a set of files to instrument. For +online coverage runtimes, this can be used at runtime to decide which files to +instrument. It is also used to implement baseline coverage. + +The set of instrumentation metadata files is the set of extra files a test needs +to generate the LCOV files Bazel requires from it. In practice, this consists of +runtime-specific files; for example, gcc emits .gcno files during compilation. +These are added to the set of inputs of test actions if coverage mode is +enabled. + +Whether or not coverage is being collected is stored in the +`BuildConfiguration`. This is handy because it is an easy way to change the test +action and the action graph depending on this bit, but it also means that if +this bit is flipped, all targets need to be re-analyzed (some languages, such as +C++ require different compiler options to emit code that can collect coverage, +which mitigates this issue somewhat, since then a re-analysis is needed anyway). + +The coverage support files are depended on through labels in an implicit +dependency so that they can be overridden by the invocation policy, which allows +them to differ between the different versions of Bazel. Ideally, these +differences would be removed, and we standardized on one of them. + +We also generate a "coverage report" which merges the coverage collected for +every test in a Bazel invocation. This is handled by +`CoverageReportActionFactory` and is called from `BuildView.createResult()` . It +gets access to the tools it needs by looking at the `:coverage_report_generator` +attribute of the first test that is executed. ## The query engine -Bazel has a [little language](/query/guide) used to ask it various things about various graphs. The following query kinds are provided: - -- `bazel query` is used to investigate the target graph -- `bazel cquery` is used to investigate the configured target graph -- `bazel aquery` is used to investigate the action graph - -Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. Additional additional query functions can be done by subclassing `QueryFunction` . In order to allow streaming query results, instead of collecting them to some data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which calls it for results it wants to return. - -The result of a query can be emitted in various ways: labels, labels and rule classes, XML, protobuf and so on. These are implemented as subclasses of `OutputFormatter`. - -A subtle requirement of some query output formats (proto, definitely) is that Bazel needs to emit \_all \_the information that package loading provides so that one can diff the output and determine whether a particular target has changed. As a consequence, attribute values need to be serializable, which is why there are only so few attribute types without any attributes having complex Starlark values. The usual workaround is to use a label, and attach the complex information to the rule with that label. It's not a very satisfying workaround and it would be very nice to lift this requirement. +Bazel has a +[little language](/query/guide) +used to ask it various things about various graphs. The following query kinds +are provided: + +* `bazel query` is used to investigate the target graph +* `bazel cquery` is used to investigate the configured target graph +* `bazel aquery` is used to investigate the action graph + +Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. +Additional additional query functions can be done by subclassing `QueryFunction` +. In order to allow streaming query results, instead of collecting them to some +data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which +calls it for results it wants to return. + +The result of a query can be emitted in various ways: labels, labels and rule +classes, XML, protobuf and so on. These are implemented as subclasses of +`OutputFormatter`. + +A subtle requirement of some query output formats (proto, definitely) is that +Bazel needs to emit _all _the information that package loading provides so that +one can diff the output and determine whether a particular target has changed. +As a consequence, attribute values need to be serializable, which is why there +are only so few attribute types without any attributes having complex Starlark +values. The usual workaround is to use a label, and attach the complex +information to the rule with that label. It's not a very satisfying workaround +and it would be very nice to lift this requirement. ## The module system -Bazel can be extended by adding modules to it. Each module must subclass `BlazeModule` (the name is a relic of the history of Bazel when it used to be called Blaze) and gets information about various events during the execution of a command. +Bazel can be extended by adding modules to it. Each module must subclass +`BlazeModule` (the name is a relic of the history of Bazel when it used to be +called Blaze) and gets information about various events during the execution of +a command. -They are mostly used to implement various pieces of "non-core" functionality that only some versions of Bazel (such as the one we use at Google) need: +They are mostly used to implement various pieces of "non-core" functionality +that only some versions of Bazel (such as the one we use at Google) need: -- Interfaces to remote execution systems -- New commands +* Interfaces to remote execution systems +* New commands -The set of extension points `BlazeModule` offers is somewhat haphazard. Don't use it as an example of good design principles. +The set of extension points `BlazeModule` offers is somewhat haphazard. Don't +use it as an example of good design principles. ## The event bus -The main way BlazeModules communicate with the rest of Bazel is by an event bus (`EventBus`): a new instance is created for every build, various parts of Bazel can post events to it and modules can register listeners for the events they are interested in. For example, the following things are represented as events: +The main way BlazeModules communicate with the rest of Bazel is by an event bus +(`EventBus`): a new instance is created for every build, various parts of Bazel +can post events to it and modules can register listeners for the events they are +interested in. For example, the following things are represented as events: -- The list of build targets to be built has been determined (`TargetParsingCompleteEvent`) -- The top-level configurations have been determined (`BuildConfigurationEvent`) -- A target was built, successfully or not (`TargetCompleteEvent`) -- A test was run (`TestAttempt`, `TestSummary`) +* The list of build targets to be built has been determined + (`TargetParsingCompleteEvent`) +* The top-level configurations have been determined + (`BuildConfigurationEvent`) +* A target was built, successfully or not (`TargetCompleteEvent`) +* A test was run (`TestAttempt`, `TestSummary`) -Some of these events are represented outside of Bazel in the [Build Event Protocol](/remote/bep) (they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things outside the Bazel process to observe the build. They are accessible either as a file that contains protocol messages or Bazel can connect to a server (called the Build Event Service) to stream events. +Some of these events are represented outside of Bazel in the +[Build Event Protocol](/remote/bep) +(they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things +outside the Bazel process to observe the build. They are accessible either as a +file that contains protocol messages or Bazel can connect to a server (called +the Build Event Service) to stream events. -This is implemented in the `build.lib.buildeventservice` and `build.lib.buildeventstream` Java packages. +This is implemented in the `build.lib.buildeventservice` and +`build.lib.buildeventstream` Java packages. ## External repositories -Note: The information in this section is out of date, as code in this area has undergone extensive change in the past couple of years. Please refer to [external dependencies overview](/external/overview) for more up-to-date information. +Note: The information in this section is out of date, as code in this area has +undergone extensive change in the past couple of years. Please refer to +[external dependencies overview](/external/overview) for more up-to-date +information. -Whereas Bazel was originally designed to be used in a monorepo (a single source tree containing everything one needs to build), Bazel lives in a world where this is not necessarily true. "External repositories" are an abstraction used to bridge these two worlds: they represent code that is necessary for the build but is not in the main source tree. +Whereas Bazel was originally designed to be used in a monorepo (a single source +tree containing everything one needs to build), Bazel lives in a world where +this is not necessarily true. "External repositories" are an abstraction used to +bridge these two worlds: they represent code that is necessary for the build but +is not in the main source tree. ### The WORKSPACE file -The set of external repositories is determined by parsing the WORKSPACE file. For example, a declaration like this: +The set of external repositories is determined by parsing the WORKSPACE file. +For example, a declaration like this: ``` local_repository(name="foo", path="/foo/bar") ``` -Results in the repository called `@foo` being available. Where this gets complicated is that one can define new repository rules in Starlark files, which can then be used to load new Starlark code, which can be used to define new repository rules and so on… +Results in the repository called `@foo` being available. Where this gets +complicated is that one can define new repository rules in Starlark files, which +can then be used to load new Starlark code, which can be used to define new +repository rules and so on… -To handle this case, the parsing of the WORKSPACE file (in `WorkspaceFileFunction`) is split up into chunks delineated by `load()` statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and computing `WorkspaceFileFunction` until index X means evaluating it until the Xth `load()` statement. +To handle this case, the parsing of the WORKSPACE file (in +`WorkspaceFileFunction`) is split up into chunks delineated by `load()` +statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and +computing `WorkspaceFileFunction` until index X means evaluating it until the +Xth `load()` statement. ### Fetching repositories -Before the code of the repository is available to Bazel, it needs to be *fetched*. This results in Bazel creating a directory under `$OUTPUT_BASE/external/<repository name>`. +Before the code of the repository is available to Bazel, it needs to be +_fetched_. This results in Bazel creating a directory under +`$OUTPUT_BASE/external/`. Fetching the repository happens in the following steps: -1. `PackageLookupFunction` realizes that it needs a repository and creates a `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` -2. `RepositoryLoaderFunction` forwards the request to `RepositoryDelegatorFunction` for unclear reasons (the code says it's to avoid re-downloading things in case of Skyframe restarts, but it's not a very solid reasoning) -3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to fetch by iterating over the chunks of the WORKSPACE file until the requested repository is found -4. The appropriate `RepositoryFunction` is found that implements the repository fetching; it's either the Starlark implementation of the repository or a hard-coded map for repositories that are implemented in Java. - -There are various layers of caching since fetching a repository can be very expensive: - -1. There is a cache for downloaded files that is keyed by their checksum (`RepositoryCache`). This requires the checksum to be available in the WORKSPACE file, but that's good for hermeticity anyway. This is shared by every Bazel server instance on the same workstation, regardless of which workspace or output base they are running in. -2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` that contains a checksum of the rule that was used to fetch it. If the Bazel server restarts but the checksum does not change, it's not re-fetched. This is implemented in `RepositoryDelegatorFunction.DigestWriter` . -3. The `--distdir` command line option designates another cache that is used to look up artifacts to be downloaded. This is useful in enterprise settings where Bazel should not fetch random things from the Internet. This is implemented by `DownloadManager` . - -Once a repository is downloaded, the artifacts in it are treated as source artifacts. This poses a problem because Bazel usually checks for up-to-dateness of source artifacts by calling stat() on them, and these artifacts are also invalidated when the definition of the repository they are in changes. Thus, `FileStateValue`s for an artifact in an external repository need to depend on their external repository. This is handled by `ExternalFilesHelper`. +1. `PackageLookupFunction` realizes that it needs a repository and creates a + `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` +2. `RepositoryLoaderFunction` forwards the request to + `RepositoryDelegatorFunction` for unclear reasons (the code says it's to + avoid re-downloading things in case of Skyframe restarts, but it's not a + very solid reasoning) +3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to + fetch by iterating over the chunks of the WORKSPACE file until the requested + repository is found +4. The appropriate `RepositoryFunction` is found that implements the repository + fetching; it's either the Starlark implementation of the repository or a + hard-coded map for repositories that are implemented in Java. + +There are various layers of caching since fetching a repository can be very +expensive: + +1. There is a cache for downloaded files that is keyed by their checksum + (`RepositoryCache`). This requires the checksum to be available in the + WORKSPACE file, but that's good for hermeticity anyway. This is shared by + every Bazel server instance on the same workstation, regardless of which + workspace or output base they are running in. +2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` + that contains a checksum of the rule that was used to fetch it. If the Bazel + server restarts but the checksum does not change, it's not re-fetched. This + is implemented in `RepositoryDelegatorFunction.DigestWriter` . +3. The `--distdir` command line option designates another cache that is used to + look up artifacts to be downloaded. This is useful in enterprise settings + where Bazel should not fetch random things from the Internet. This is + implemented by `DownloadManager` . + +Once a repository is downloaded, the artifacts in it are treated as source +artifacts. This poses a problem because Bazel usually checks for up-to-dateness +of source artifacts by calling stat() on them, and these artifacts are also +invalidated when the definition of the repository they are in changes. Thus, +`FileStateValue`s for an artifact in an external repository need to depend on +their external repository. This is handled by `ExternalFilesHelper`. ### Repository mappings -It can happen that multiple repositories want to depend on the same repository, but in different versions (this is an instance of the "diamond dependency problem"). For example, if two binaries in separate repositories in the build want to depend on Guava, they will presumably both refer to Guava with labels starting `@guava//` and expect that to mean different versions of it. - -Therefore, Bazel allows one to re-map external repository labels so that the string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the repository of one binary and another Guava repository (such as `@guava2//`) the repository of the other. - -Alternatively, this can also be used to **join** diamonds. If a repository depends on `@guava1//`, and another depends on `@guava2//`, repository mapping allows one to re-map both repositories to use a canonical `@guava//` repository. - -The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute of individual repository definitions. It then appears in Skyframe as a member of `WorkspaceFileValue`, where it is plumbed to: - -- `Package.Builder.repositoryMapping` which is used to transform label-valued attributes of rules in the package by `RuleClass.populateRuleAttributeValues()` -- `Package.repositoryMapping` which is used in the analysis phase (for resolving things like `$(location)` which are not parsed in the loading phase) -- `BzlLoadFunction` for resolving labels in load() statements +It can happen that multiple repositories want to depend on the same repository, +but in different versions (this is an instance of the "diamond dependency +problem"). For example, if two binaries in separate repositories in the build +want to depend on Guava, they will presumably both refer to Guava with labels +starting `@guava//` and expect that to mean different versions of it. + +Therefore, Bazel allows one to re-map external repository labels so that the +string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the +repository of one binary and another Guava repository (such as `@guava2//`) the +repository of the other. + +Alternatively, this can also be used to **join** diamonds. If a repository +depends on `@guava1//`, and another depends on `@guava2//`, repository mapping +allows one to re-map both repositories to use a canonical `@guava//` repository. + +The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute +of individual repository definitions. It then appears in Skyframe as a member of +`WorkspaceFileValue`, where it is plumbed to: + +* `Package.Builder.repositoryMapping` which is used to transform label-valued + attributes of rules in the package by + `RuleClass.populateRuleAttributeValues()` +* `Package.repositoryMapping` which is used in the analysis phase (for + resolving things like `$(location)` which are not parsed in the loading + phase) +* `BzlLoadFunction` for resolving labels in load() statements ## JNI bits -The server of Bazel is *mostly* written in Java. The exception is the parts that Java cannot do by itself or couldn't do by itself when we implemented it. This is mostly limited to interaction with the file system, process control and various other low-level things. +The server of Bazel is _mostly_ written in Java. The exception is the parts that +Java cannot do by itself or couldn't do by itself when we implemented it. This +is mostly limited to interaction with the file system, process control and +various other low-level things. -The C++ code lives under src/main/native and the Java classes with native methods are: +The C++ code lives under src/main/native and the Java classes with native +methods are: -- `NativePosixFiles` and `NativePosixFileSystem` -- `ProcessUtils` -- `WindowsFileOperations` and `WindowsFileProcesses` -- `com.google.devtools.build.lib.platform` +* `NativePosixFiles` and `NativePosixFileSystem` +* `ProcessUtils` +* `WindowsFileOperations` and `WindowsFileProcesses` +* `com.google.devtools.build.lib.platform` ## Console output -Emitting console output seems like a simple thing, but the confluence of running multiple processes (sometimes remotely), fine-grained caching, the desire to have a nice and colorful terminal output and having a long-running server makes it non-trivial. - -Right after the RPC call comes in from the client, two `RpcOutputStream` instances are created (for stdout and stderr) that forward the data printed into them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) pair). Anything that needs to be printed on the console goes through these streams. Then these streams are handed over to `BlazeCommandDispatcher.execExclusively()`. - -Output is by default printed with ANSI escape sequences. When these are not desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In addition, `System.out` and `System.err` are redirected to these output streams. This is so that debugging information can be printed using `System.err.println()` and still end up in the terminal output of the client (which is different from that of the server). Care is taken that if a process produces binary output (such as `bazel query --output=proto`), no munging of stdout takes place. - -Short messages (errors, warnings and the like) are expressed through the `EventHandler` interface. Notably, these are different from what one posts to the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, warning, info, and a few others) and they may have a `Location` (the place in the source code that caused the event to happen). - -Some `EventHandler` implementations store the events they received. This is used to replay information to the UI caused by various kinds of cached processing, for example, the warnings emitted by a cached configured target. - -Some `EventHandler`s also allow posting events that eventually find their way to the event bus (regular `Event`s do \_not \_appear there). These are implementations of `ExtendedEventHandler` and their main use is to replay cached `EventBus` events. These `EventBus` events all implement `Postable`, but not everything that is posted to `EventBus` necessarily implements this interface; only those that are cached by an `ExtendedEventHandler` (it would be nice and most of the things do; it's not enforced, though) - -Terminal output is *mostly* emitted through `UiEventHandler`, which is responsible for all the fancy output formatting and progress reporting Bazel does. It has two inputs: - -- The event bus -- The event stream piped into it through Reporter - -The only direct connection the command execution machinery (for example the rest of Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, which allows direct access to these streams. It's only used when a command needs to dump large amounts of possible binary data (such as `bazel query`). +Emitting console output seems like a simple thing, but the confluence of running +multiple processes (sometimes remotely), fine-grained caching, the desire to +have a nice and colorful terminal output and having a long-running server makes +it non-trivial. + +Right after the RPC call comes in from the client, two `RpcOutputStream` +instances are created (for stdout and stderr) that forward the data printed into +them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) +pair). Anything that needs to be printed on the console goes through these +streams. Then these streams are handed over to +`BlazeCommandDispatcher.execExclusively()`. + +Output is by default printed with ANSI escape sequences. When these are not +desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In +addition, `System.out` and `System.err` are redirected to these output streams. +This is so that debugging information can be printed using +`System.err.println()` and still end up in the terminal output of the client +(which is different from that of the server). Care is taken that if a process +produces binary output (such as `bazel query --output=proto`), no munging of stdout +takes place. + +Short messages (errors, warnings and the like) are expressed through the +`EventHandler` interface. Notably, these are different from what one posts to +the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, +warning, info, and a few others) and they may have a `Location` (the place in +the source code that caused the event to happen). + +Some `EventHandler` implementations store the events they received. This is used +to replay information to the UI caused by various kinds of cached processing, +for example, the warnings emitted by a cached configured target. + +Some `EventHandler`s also allow posting events that eventually find their way to +the event bus (regular `Event`s do _not _appear there). These are +implementations of `ExtendedEventHandler` and their main use is to replay cached +`EventBus` events. These `EventBus` events all implement `Postable`, but not +everything that is posted to `EventBus` necessarily implements this interface; +only those that are cached by an `ExtendedEventHandler` (it would be nice and +most of the things do; it's not enforced, though) + +Terminal output is _mostly_ emitted through `UiEventHandler`, which is +responsible for all the fancy output formatting and progress reporting Bazel +does. It has two inputs: + +* The event bus +* The event stream piped into it through Reporter + +The only direct connection the command execution machinery (for example the rest of +Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, +which allows direct access to these streams. It's only used when a command needs +to dump large amounts of possible binary data (such as `bazel query`). ## Profiling Bazel -Bazel is fast. Bazel is also slow, because builds tend to grow until just the edge of what's bearable. For this reason, Bazel includes a profiler which can be used to profile builds and Bazel itself. It's implemented in a class that's aptly named `Profiler`. It's turned on by default, although it records only abridged data so that its overhead is tolerable; The command line `--record_full_profiler_data` makes it record everything it can. - -It emits a profile in the Chrome profiler format; it's best viewed in Chrome. It's data model is that of task stacks: one can start tasks and end tasks and they are supposed to be neatly nested within each other. Each Java thread gets its own task stack. **TODO:** How does this work with actions and continuation-passing style? - -The profiler is started and stopped in `BlazeRuntime.initProfiler()` and `BlazeRuntime.afterCommand()` respectively and attempts to be live for as long as possible so that we can profile everything. To add something to the profile, call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure represents the end of the task. It's best used with try-with-resources statements. - -We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on and it mostly records maximum heap sizes and GC behavior. +Bazel is fast. Bazel is also slow, because builds tend to grow until just the +edge of what's bearable. For this reason, Bazel includes a profiler which can be +used to profile builds and Bazel itself. It's implemented in a class that's +aptly named `Profiler`. It's turned on by default, although it records only +abridged data so that its overhead is tolerable; The command line +`--record_full_profiler_data` makes it record everything it can. + +It emits a profile in the Chrome profiler format; it's best viewed in Chrome. +It's data model is that of task stacks: one can start tasks and end tasks and +they are supposed to be neatly nested within each other. Each Java thread gets +its own task stack. **TODO:** How does this work with actions and +continuation-passing style? + +The profiler is started and stopped in `BlazeRuntime.initProfiler()` and +`BlazeRuntime.afterCommand()` respectively and attempts to be live for as long +as possible so that we can profile everything. To add something to the profile, +call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure +represents the end of the task. It's best used with try-with-resources +statements. + +We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on +and it mostly records maximum heap sizes and GC behavior. ## Testing Bazel -Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and ones that only run the analysis phase. We call the former "integration tests" and the latter "unit tests", although they are more like integration tests that are, well, less integrated. We also have some actual unit tests, where they are necessary. +Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and +ones that only run the analysis phase. We call the former "integration tests" +and the latter "unit tests", although they are more like integration tests that +are, well, less integrated. We also have some actual unit tests, where they are +necessary. Of integration tests, we have two kinds: -1. Ones implemented using a very elaborate bash test framework under `src/test/shell` -2. Ones implemented in Java. These are implemented as subclasses of `BuildIntegrationTestCase` - -`BuildIntegrationTestCase` is the preferred integration testing framework as it is well-equipped for most testing scenarios. As it is a Java framework, it provides debuggability and seamless integration with many common development tools. There are many examples of `BuildIntegrationTestCase` classes in the Bazel repository. - -Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a scratch file system you can use to write `BUILD` files, then various helper methods can request configured targets, change the configuration and assert various things about the result of the analysis. +1. Ones implemented using a very elaborate bash test framework under + `src/test/shell` +2. Ones implemented in Java. These are implemented as subclasses of + `BuildIntegrationTestCase` + +`BuildIntegrationTestCase` is the preferred integration testing framework as it +is well-equipped for most testing scenarios. As it is a Java framework, it +provides debuggability and seamless integration with many common development +tools. There are many examples of `BuildIntegrationTestCase` classes in the +Bazel repository. + +Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a +scratch file system you can use to write `BUILD` files, then various helper +methods can request configured targets, change the configuration and assert +various things about the result of the analysis. diff --git a/contribute/design-documents.mdx b/contribute/design-documents.mdx index 47405539..1fe70b9d 100644 --- a/contribute/design-documents.mdx +++ b/contribute/design-documents.mdx @@ -2,94 +2,145 @@ title: 'Design Documents' --- -If you're planning to add, change, or remove a user-facing feature, or make a *significant architectural change* to Bazel, you **must** write a design document and have it reviewed before you can submit the change. + + +If you're planning to add, change, or remove a user-facing feature, or make a +*significant architectural change* to Bazel, you **must** write a design +document and have it reviewed before you can submit the change. Here are some examples of significant changes: -- Addition or deletion of native build rules -- Breaking-changes to native rules -- Changes to a native build rule semantics that affect the behavior of more than a single rule -- Changes to Bazel's rule definition API -- Changes to the APIs that Bazel uses to connect to other systems -- Changes to the Starlark language, semantics, or APIs -- Changes that could have a pervasive effect on Bazel performance or memory usage (for better or for worse) -- Changes to widely used internal APIs -- Changes to flags and command-line interface. +* Addition or deletion of native build rules +* Breaking-changes to native rules +* Changes to a native build rule semantics that affect the behavior of more + than a single rule +* Changes to Bazel's rule definition API +* Changes to the APIs that Bazel uses to connect to other systems +* Changes to the Starlark language, semantics, or APIs +* Changes that could have a pervasive effect on Bazel performance or memory + usage (for better or for worse) +* Changes to widely used internal APIs +* Changes to flags and command-line interface. ## Reasons for design reviews -When you write a design document, you can coordinate with other Bazel developers and seek guidance from Bazel's core team. For example, when a proposal adds, removes, or modifies any function or object available in BUILD, MODULE.bazel, or bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. Design documents are reviewed before submission because: - -- Bazel is a very complex system; seemingly innocuous local changes can have significant global consequences. -- The team gets many feature requests from users; such requests need to be evaluated not only for technical feasibility but importance with regards to other feature requests. -- Bazel features are frequently implemented by people outside the core team; such contributors have widely varying levels of Bazel expertise. -- The Bazel team itself has varying levels of expertise; no single team member has a complete understanding of every corner of Bazel. -- Changes to Bazel must account for backward compatibility and avoid breaking changes. +When you write a design document, you can coordinate with other Bazel developers +and seek guidance from Bazel's core team. For example, when a proposal adds, +removes, or modifies any function or object available in BUILD, MODULE.bazel, or +bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. +Design documents are reviewed before submission because: + +* Bazel is a very complex system; seemingly innocuous local changes can have + significant global consequences. +* The team gets many feature requests from users; such requests need to be + evaluated not only for technical feasibility but importance with regards to + other feature requests. +* Bazel features are frequently implemented by people outside the core team; + such contributors have widely varying levels of Bazel expertise. +* The Bazel team itself has varying levels of expertise; no single team member + has a complete understanding of every corner of Bazel. +* Changes to Bazel must account for backward compatibility and avoid breaking + changes. Bazel's design review policy helps to maximize the likelihood that: -- all feature requests get a baseline level of scrutiny. -- the right people will weigh in on designs before we've invested in an implementation that may not work. +* all feature requests get a baseline level of scrutiny. +* the right people will weigh in on designs before we've invested in an + implementation that may not work. -To help you get started, take a look at the design documents in the [Bazel Proposals Repository](https://github.com/bazelbuild/proposals). Designs are works in progress, so implementation details can change over time and with feedback. The published design documents capture the initial design, and *not* the ongoing changes as designs are implemented. Always go to the documentation for descriptions of current Bazel functionality. +To help you get started, take a look at the design documents in the +[Bazel Proposals Repository](https://github.com/bazelbuild/proposals). +Designs are works in progress, so implementation details can change over time +and with feedback. The published design documents capture the initial design, +and *not* the ongoing changes as designs are implemented. Always go to the +documentation for descriptions of current Bazel functionality. ## Contributor Workflow -As a contributor, you can write a design document, send pull requests and request reviewers for your proposal. +As a contributor, you can write a design document, send pull requests and +request reviewers for your proposal. ### Write the design document All design documents must have a header that includes: -- author -- date of last major change -- list of reviewers, including one (and only one) [lead reviewer](#lead-reviewer) -- current status (*draft*, *in review*, *approved*, *rejected*, *being implemented*, *implemented*) -- link to discussion thread (*to be added after the announcement*) +* author +* date of last major change +* list of reviewers, including one (and only one) + [lead reviewer](#lead-reviewer) +* current status (_draft_, _in review_, _approved_, _rejected_, + _being implemented_, _implemented_) +* link to discussion thread (_to be added after the announcement_) -The document can be written either [as a world-readable Google Doc](#gdocs) or [using Markdown](#markdown). Read below about for a [Markdown / Google Docs comparison](#markdown-versus-gdocs). +The document can be written either [as a world-readable Google Doc](#gdocs) +or [using Markdown](#markdown). Read below about for a +[Markdown / Google Docs comparison](#markdown-versus-gdocs). -Proposals that have a user-visible impact must have a section documenting the impact on backward compatibility (and a rollout plan if needed). +Proposals that have a user-visible impact must have a section documenting the +impact on backward compatibility (and a rollout plan if needed). ### Create a Pull Request -Share your design doc by creating a pull request (PR) to add the document to [the design index](https://github.com/bazelbuild/proposals). Add your markdown file or a document link to your PR. +Share your design doc by creating a pull request (PR) to add the document to +[the design index](https://github.com/bazelbuild/proposals). Add +your markdown file or a document link to your PR. -When possible, [choose a lead reviewer](#lead-reviewer). and cc other reviewers. If you don't choose a lead reviewer, a Bazel maintainer will assign one to your PR. +When possible, [choose a lead reviewer](#lead-reviewer). +and cc other reviewers. If you don't choose a lead reviewer, a Bazel +maintainer will assign one to your PR. -After you create your PR, reviewers can make preliminary comments during the code review. For example, the lead reviewer can suggest extra reviewers, or point out missing information. The lead reviewer approves the PR when they believe the review process can start. This doesn't mean the proposal is perfect or will be approved; it means that the proposal contains enough information to start the discussion. +After you create your PR, reviewers can make preliminary comments during the +code review. For example, the lead reviewer can suggest extra reviewers, or +point out missing information. The lead reviewer approves the PR when they +believe the review process can start. This doesn't mean the proposal is perfect +or will be approved; it means that the proposal contains enough information to +start the discussion. ### Announce the new proposal -Send an announcement to [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when the PR is submitted. +Send an announcement to +[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when +the PR is submitted. -You may copy other groups (for example, [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), to get feedback from Bazel end-users). +You may copy other groups (for example, +[bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), +to get feedback from Bazel end-users). ### Iterate with reviewers -Anyone interested can comment on your proposal. Try to answer questions, clarify the proposal, and address concerns. +Anyone interested can comment on your proposal. Try to answer questions, +clarify the proposal, and address concerns. -Discussion should happen on the announcement thread. If the proposal is in a Google Doc, comments may be used instead (Note that anonymous comments are allowed). +Discussion should happen on the announcement thread. If the proposal is in a +Google Doc, comments may be used instead (Note that anonymous comments are +allowed). ### Update the status -Create a new PR to update the status of the proposal, when iteration is complete. Send the PR to the same lead reviewer and cc the other reviewers. +Create a new PR to update the status of the proposal, when iteration is +complete. Send the PR to the same lead reviewer and cc the other reviewers. -To officially accept the proposal, the lead reviewer approves the PR after ensuring that the other reviewers agree with the decision. +To officially accept the proposal, the lead reviewer approves the PR after +ensuring that the other reviewers agree with the decision. -There must be at least 1 week between the first announcement and the approval of a proposal. This ensures that users had enough time to read the document and share their concerns. +There must be at least 1 week between the first announcement and the approval of +a proposal. This ensures that users had enough time to read the document and +share their concerns. -Implementation can begin before the proposal is accepted, for example as a proof-of-concept or an experimentation. However, you cannot submit the change before the review is complete. +Implementation can begin before the proposal is accepted, for example as a +proof-of-concept or an experimentation. However, you cannot submit the change +before the review is complete. ### Choosing a lead reviewer A lead reviewer should be a domain expert who is: -- Knowledgeable of the relevant subsystems -- Objective and capable of providing constructive feedback -- Available for the entire review period to lead the process +* Knowledgeable of the relevant subsystems +* Objective and capable of providing constructive feedback +* Available for the entire review period to lead the process -Consider checking the contacts for various [team labels](/contribute/maintainers-guide#team-labels). +Consider checking the contacts for various [team +labels](/contribute/maintainers-guide#team-labels). ## Markdown vs Google Docs @@ -97,34 +148,49 @@ Decide what works best for you, since both are accepted. Benefits of using Google Docs: -- Effective for brainstorming, since it is easy to get started with. -- Collaborative editing. -- Quick iteration. -- Easy way to suggest edits. +* Effective for brainstorming, since it is easy to get started with. +* Collaborative editing. +* Quick iteration. +* Easy way to suggest edits. Benefits of using Markdown files: -- Clean URLs for linking. -- Explicit record of revisions. -- No forgetting to set up access rights before publicizing a link. -- Easily searchable with search engines. -- Future-proof: Plain text is not at the mercy of any specific tool and doesn't require an Internet connection. -- It is possible to update them even if the author is not around anymore. -- They can be processed automatically (update/detect dead links, fetch list of authors, etc.). +* Clean URLs for linking. +* Explicit record of revisions. +* No forgetting to set up access rights before publicizing a link. +* Easily searchable with search engines. +* Future-proof: Plain text is not at the mercy of any specific tool + and doesn't require an Internet connection. +* It is possible to update them even if the author is not around anymore. +* They can be processed automatically (update/detect dead links, fetch + list of authors, etc.). -You can choose to first iterate on a Google Doc, and then convert it to Markdown for posterity. +You can choose to first iterate on a Google Doc, and then convert it to +Markdown for posterity. ### Using Google Docs -For consistency, use the [Bazel design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). It includes the necessary header and creates visual consistency with other Bazel related documents. To do that, click on **File** \> **Make a copy** or click this link to [make a copy of the design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). +For consistency, use the [Bazel design doc template]( +https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). +It includes the necessary header and creates visual +consistency with other Bazel related documents. To do that, click on **File** > +**Make a copy** or click this link to [make a copy of the design doc +template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). -To make your document readable to the world, click on **Share** \> **Advanced** \> **Change…**, and choose "On - Anyone with the link". If you allow comments on the document, anyone can comment anonymously, even without a Google account. +To make your document readable to the world, click on +**Share** > **Advanced** > **Change…**, and +choose "On - Anyone with the link". If you allow comments on the document, +anyone can comment anonymously, even without a Google account. ### Using Markdown -Documents are stored on GitHub and use the [GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) ([Specification](https://github.github.com/gfm/)). +Documents are stored on GitHub and use the +[GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) +([Specification](https://github.github.com/gfm/)). -Create a PR to update an existing document. Significant changes should be reviewed by the document reviewers. Trivial changes (such as typos, formatting) can be approved by anyone. +Create a PR to update an existing document. Significant changes should be +reviewed by the document reviewers. Trivial changes (such as typos, formatting) +can be approved by anyone. ## Reviewer workflow @@ -132,41 +198,57 @@ A reviewer comments, reviews and approves design documents. ### General reviewer responsibilities -You're responsible for reviewing design documents, asking for additional information if needed, and approving a design that passes the review process. +You're responsible for reviewing design documents, asking for additional +information if needed, and approving a design that passes the review process. #### When you receive a new proposal -1. Take a quick look at the document. -2. Comment if critical information is missing, or if the design doesn't fit with the goals of the project. -3. Suggest additional reviewers. -4. Approve the PR when it is ready for review. +1. Take a quick look at the document. +1. Comment if critical information is missing, or if the design doesn't fit + with the goals of the project. +1. Suggest additional reviewers. +1. Approve the PR when it is ready for review. #### During the review process -1. Engage in a dialogue with the design author about issues that are problematic or require clarification. -2. If appropriate, invite comments from non-reviewers who should be aware of the design. -3. Decide which comments must be addressed by the author as a prerequisite to approval. -4. Write "LGTM" (*Looks Good To Me*) in the discussion thread when you are happy with the current state of the proposal. +1. Engage in a dialogue with the design author about issues that are problematic + or require clarification. +1. If appropriate, invite comments from non-reviewers who should be aware of + the design. +1. Decide which comments must be addressed by the author as a prerequisite to + approval. +1. Write "LGTM" (_Looks Good To Me_) in the discussion thread when you are + happy with the current state of the proposal. -Follow this process for all design review requests. Do not approve designs affecting Bazel if they are not in the [design index](https://github.com/bazelbuild/proposals). +Follow this process for all design review requests. Do not approve designs +affecting Bazel if they are not in the +[design index](https://github.com/bazelbuild/proposals). ### Lead reviewer responsibilities -You're responsible for making the go / no-go decision on implementation of a pending design. If you're not able to do this, you should identify a suitable delegate (reassign the PR to the delegate), or reassign the bug to a Bazel manager for further disposition. +You're responsible for making the go / no-go decision on implementation +of a pending design. If you're not able to do this, you should identify a +suitable delegate (reassign the PR to the delegate), or reassign the bug to a +Bazel manager for further disposition. #### During the review process -1. Ensure that the comment and design iteration process moves forward constructively. -2. Prior to approval, ensure that concerns from other reviewers have been resolved. +1. Ensure that the comment and design iteration process moves forward + constructively. +1. Prior to approval, ensure that concerns from other reviewers have been + resolved. #### After approval by all reviewers -1. Make sure there has been at least 1 week since the announcement on the mailing list. -2. Make sure the PR updates the status. -3. Approve the PR sent by the proposal author. +1. Make sure there has been at least 1 week since the announcement on the + mailing list. +1. Make sure the PR updates the status. +1. Approve the PR sent by the proposal author. #### Rejecting designs -1. Make sure the PR author sends a PR; or send them a PR. -2. The PR updates the status of the document. -3. Add a comment to the document explaining why the design can't be approved in its current state, and outlining next steps, if any (such as "revisit invalid assumptions and resubmit"). +1. Make sure the PR author sends a PR; or send them a PR. +1. The PR updates the status of the document. +1. Add a comment to the document explaining why the design can't be approved in + its current state, and outlining next steps, if any (such as "revisit invalid + assumptions and resubmit"). diff --git a/contribute/docs.mdx b/contribute/docs.mdx index 940b20e2..cc240cc4 100644 --- a/contribute/docs.mdx +++ b/contribute/docs.mdx @@ -2,26 +2,42 @@ title: 'Contribute to Bazel documentation' --- -Thank you for contributing to Bazel's documentation! There are a few ways to help create better docs for our community. + + +Thank you for contributing to Bazel's documentation! There are a few ways to +help create better docs for our community. ## Documentation types This site includes a few types of content. -- *Narrative documentation*, which is written by technical writers and engineers. Most of this site is narrative documentation that covers conceptual and task-based guides. -- *Reference documentation*, which is generated documentation from code comments. You can't make changes to the reference doc pages directly, but instead need to change their source. + - *Narrative documentation*, which is written by technical writers and + engineers. Most of this site is narrative documentation that covers + conceptual and task-based guides. + - *Reference documentation*, which is generated documentation from code comments. + You can't make changes to the reference doc pages directly, but instead need + to change their source. ## Documentation infrastructure -Bazel documentation is served from Google and the source files are mirrored in Bazel's GitHub repository. You can make changes to the source files in GitHub. If approved, you can merge the changes and a Bazel maintainer will update the website source to publish your updates. +Bazel documentation is served from Google and the source files are mirrored in +Bazel's GitHub repository. You can make changes to the source files in GitHub. +If approved, you can merge the changes and a Bazel maintainer will update the +website source to publish your updates. + ## Small changes -You can approach small changes, such as fixing errors or typos, in a couple of ways. +You can approach small changes, such as fixing errors or typos, in a couple of +ways. -- **Pull request**. You can create a pull request in GitHub with the [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. -- **Bug**. You can file a bug with details and suggested changes and the Bazel documentation owners will make the update. + - **Pull request**. You can create a pull request in GitHub with the + [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. + - **Bug**. You can file a bug with details and suggested changes and the Bazel + documentation owners will make the update. ## Large changes -If you want to make substantial changes to existing documentation or propose new documentation, you can either create a pull request or start with a Google doc and contact the Bazel Owners to collaborate. +If you want to make substantial changes to existing documentation or propose +new documentation, you can either create a pull request or start with a Google +doc and contact the Bazel Owners to collaborate. diff --git a/contribute/index.mdx b/contribute/index.mdx index 18b12854..ee667729 100644 --- a/contribute/index.mdx +++ b/contribute/index.mdx @@ -2,41 +2,57 @@ title: 'Contributing to Bazel' --- + + There are many ways to help the Bazel project and ecosystem. ## Provide feedback -As you use Bazel, you may find things that can be improved. You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) when: +As you use Bazel, you may find things that can be improved. +You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) +when: -- Bazel crashes or you encounter a bug that can [only be resolved using `bazel clean`](/run/build#correct-incremental-rebuilds). -- The documentation is incomplete or unclear. You can also report issues from the page you are viewing by using the "Create issue" link at the top right corner of the page. -- An error message could be improved. + - Bazel crashes or you encounter a bug that can [only be resolved using `bazel + clean`](/run/build#correct-incremental-rebuilds). + - The documentation is incomplete or unclear. You can also report issues + from the page you are viewing by using the "Create issue" + link at the top right corner of the page. + - An error message could be improved. ## Participate in the community You can engage with the Bazel community by: -- Answering questions [on Stack Overflow](https://stackoverflow.com/questions/tagged/bazel). -- Helping other users [on Slack](https://slack.bazel.build). -- Improving documentation or [contributing examples](https://github.com/bazelbuild/examples). -- Sharing your experience or your tips, for example, on a blog or social media. + - Answering questions [on Stack Overflow]( + https://stackoverflow.com/questions/tagged/bazel). + - Helping other users [on Slack](https://slack.bazel.build). + - Improving documentation or [contributing examples]( + https://github.com/bazelbuild/examples). + - Sharing your experience or your tips, for example, on a blog or social media. ## Contribute code -Bazel is a large project and making a change to the Bazel source code can be difficult. +Bazel is a large project and making a change to the Bazel source code +can be difficult. You can contribute to the Bazel ecosystem by: -- Helping rules maintainers by contributing pull requests. -- Creating new rules and open-sourcing them. -- Contributing to Bazel-related tools, for example, migration tools. -- Improving Bazel integration with other IDEs and tools. + - Helping rules maintainers by contributing pull requests. + - Creating new rules and open-sourcing them. + - Contributing to Bazel-related tools, for example, migration tools. + - Improving Bazel integration with other IDEs and tools. -Before making a change, [create a GitHub issue](http://github.com/bazelbuild/bazel/issues) or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). +Before making a change, [create a GitHub +issue](http://github.com/bazelbuild/bazel/issues) +or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). -The most helpful contributions fix bugs or add features (as opposed to stylistic, refactoring, or "cleanup" changes). Your change should include tests and documentation, keeping in mind backward-compatibility, portability, and the impact on memory usage and performance. +The most helpful contributions fix bugs or add features (as opposed +to stylistic, refactoring, or "cleanup" changes). Your change should +include tests and documentation, keeping in mind backward-compatibility, +portability, and the impact on memory usage and performance. -To learn about how to submit a change, see the [patch acceptance process](/contribute/patch-acceptance). +To learn about how to submit a change, see the +[patch acceptance process](/contribute/patch-acceptance). ## Bazel's code description @@ -44,19 +60,23 @@ Bazel has a large codebase with code in multiple locations. See the [codebase gu Bazel is organized as follows: -- Client code is in `src/main/cpp` and provides the command-line interface. - -- Protocol buffers are in `src/main/protobuf`. - -- Server code is in `src/main/java` and `src/test/java`. - - - Core code which is mostly composed of [SkyFrame](/reference/skyframe) and some utilities. - - Built-in rules are in `com.google.devtools.build.lib.rules` and in `com.google.devtools.build.lib.bazel.rules`. You might want to read about the [Challenges of Writing Rules](/rules/challenges) first. - -- Java native interfaces are in `src/main/native`. +* Client code is in `src/main/cpp` and provides the command-line interface. +* Protocol buffers are in `src/main/protobuf`. +* Server code is in `src/main/java` and `src/test/java`. + * Core code which is mostly composed of [SkyFrame](/reference/skyframe) + and some utilities. + * Built-in rules are in `com.google.devtools.build.lib.rules` and in + `com.google.devtools.build.lib.bazel.rules`. You might want to read about + the [Challenges of Writing Rules](/rules/challenges) first. +* Java native interfaces are in `src/main/native`. +* Various tooling for language support are described in the list in the + [compiling Bazel](/install/compile-source) section. -- Various tooling for language support are described in the list in the [compiling Bazel](/install/compile-source) section. ### Searching Bazel's source code -To quickly search through Bazel's source code, use [Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's repositories, branches, and files. You can also view history, diffs, and blame information. To learn more, see the [Bazel Code Search User Guide](/contribute/search). +To quickly search through Bazel's source code, use +[Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's +repositories, branches, and files. You can also view history, diffs, and blame +information. To learn more, see the +[Bazel Code Search User Guide](/contribute/search). diff --git a/contribute/maintainers-guide.mdx b/contribute/maintainers-guide.mdx index b98a408e..9d745afb 100644 --- a/contribute/maintainers-guide.mdx +++ b/contribute/maintainers-guide.mdx @@ -2,141 +2,205 @@ title: 'Guide for Bazel Maintainers' --- + + This is a guide for the maintainers of the Bazel open source project. -If you are looking to contribute to Bazel, please read [Contributing to Bazel](/contribute) instead. +If you are looking to contribute to Bazel, please read [Contributing to +Bazel](/contribute) instead. The objectives of this page are to: -1. Serve as the maintainers' source of truth for the project’s contribution process. -2. Set expectations between the community contributors and the project maintainers. +1. Serve as the maintainers' source of truth for the project’s contribution + process. +1. Set expectations between the community contributors and the project + maintainers. -Bazel's [core group of contributors](/contribute/policy) has dedicated subteams to manage aspects of the open source project. These are: +Bazel's [core group of contributors](/contribute/policy) has dedicated +subteams to manage aspects of the open source project. These are: -- **Release Process**: Manage Bazel's release process. -- **Green Team**: Grow a healthy ecosystem of rules and tools. -- **Developer Experience Gardeners**: Encourage external contributions, review issues and pull requests, and make our development workflow more open. +* **Release Process**: Manage Bazel's release process. +* **Green Team**: Grow a healthy ecosystem of rules and tools. +* **Developer Experience Gardeners**: Encourage external contributions, review + issues and pull requests, and make our development workflow more open. ## Releases -- [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) -- [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) +* [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) +* [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) ## Continuous Integration -Read the Green team's guide to Bazel's CI infrastructure on the [bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) repository. +Read the Green team's guide to Bazel's CI infrastructure on the +[bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) +repository. ## Lifecycle of an Issue -1. A user creates an issue by choosing one of the [issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) and it enters the pool of [unreviewed open issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93\&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). - -2. A member on the Developer Experience (DevEx) subteam rotation reviews the issue. - - 1. If the issue is **not a bug** or a **feature request**, the DevEx member will usually close the issue and redirect the user to [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for higher visibility on the question. - 2. If the issue belongs in one of the rules repositories owned by the community, like [rules\_apple](https://github.com.bazelbuild/rules_apple), the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) to the correct repository. - 3. If the issue is vague or has missing information, the DevEx member will assign the issue back to the user to request for more information before continuing. This usually occurs when the user does not choose the right [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) or provides incomplete information. - -3. After reviewing the issue, the DevEx member decides if the issue requires immediate attention. If it does, they will assign the **P0** [priority](#priority) label and an owner from the list of team leads. - -4. The DevEx member assigns the `untriaged` label and exactly one [team label](#team-labels) for routing. - -5. The DevEx member also assigns exactly one `type:` label, such as `type: bug` or `type: feature request`, according to the type of the issue. - -6. For platform-specific issues, the DevEx member assigns one `platform:` label, such as `platform:apple` for Mac-specific issues. - -7. If the issue is low priority and can be worked on by a new community contributor, the DevEx member assigns the `good first issue` label. At this stage, the issue enters the pool of [untriaged open issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). - -Each Bazel subteam will triage all issues under labels they own, preferably on a weekly basis. The subteam will review and evaluate the issue and provide a resolution, if possible. If you are an owner of a team label, see [this section ](#label-own)for more information. +1. A user creates an issue by choosing one of the +[issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) + and it enters the pool of [unreviewed open + issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). +1. A member on the Developer Experience (DevEx) subteam rotation reviews the + issue. + 1. If the issue is **not a bug** or a **feature request**, the DevEx member + will usually close the issue and redirect the user to + [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and + [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for + higher visibility on the question. + 1. If the issue belongs in one of the rules repositories owned by the + community, like [rules_apple](https://github.com.bazelbuild/rules_apple), + the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) + to the correct repository. + 1. If the issue is vague or has missing information, the DevEx member will + assign the issue back to the user to request for more information before + continuing. This usually occurs when the user does not choose the right + [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) + or provides incomplete information. +1. After reviewing the issue, the DevEx member decides if the issue requires + immediate attention. If it does, they will assign the **P0** + [priority](#priority) label and an owner from the list of team leads. +1. The DevEx member assigns the `untriaged` label and exactly one [team + label](#team-labels) for routing. +1. The DevEx member also assigns exactly one `type:` label, such as `type: bug` + or `type: feature request`, according to the type of the issue. +1. For platform-specific issues, the DevEx member assigns one `platform:` label, + such as `platform:apple` for Mac-specific issues. +1. If the issue is low priority and can be worked on by a new community + contributor, the DevEx member assigns the `good first issue` label. +At this stage, the issue enters the pool of [untriaged open +issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). + +Each Bazel subteam will triage all issues under labels they own, preferably on a +weekly basis. The subteam will review and evaluate the issue and provide a +resolution, if possible. If you are an owner of a team label, see [this section +](#label-own) for more information. When an issue is resolved, it can be closed. ## Lifecycle of a Pull Request 1. A user creates a pull request. -2. If you a member of a Bazel team and sending a PR against your own area, you are responsible for assigning your team label and finding the best reviewer. -3. Otherwise, during daily triage, a DevEx member assigns one [team label](#team-labels) and the team's technical lead (TL) for routing. +1. If you a member of a Bazel team and sending a PR against your own area, + you are responsible for assigning your team label and finding the best + reviewer. +1. Otherwise, during daily triage, a DevEx member assigns one + [team label](#team-labels) and the team's technical lead (TL) for routing. 1. The TL may optionally assign someone else to review the PR. -4. The assigned reviewer reviews the PR and works with the author until it is approved or dropped. -5. If approved, the reviewer **imports** the PR's commit(s) into Google's internal version control system for further tests. As Bazel is the same build system used internally at Google, we need to test all PR commits against the internal test suite. This is the reason why we do not merge PRs directly. -6. If the imported commit passes all internal tests, the commit will be squashed and exported back out to GitHub. -7. When the commit merges into master, GitHub automatically closes the PR. +1. The assigned reviewer reviews the PR and works with the author until it is + approved or dropped. +1. If approved, the reviewer **imports** the PR's commit(s) into Google's + internal version control system for further tests. As Bazel is the same build + system used internally at Google, we need to test all PR commits against the + internal test suite. This is the reason why we do not merge PRs directly. +1. If the imported commit passes all internal tests, the commit will be squashed + and exported back out to GitHub. +1. When the commit merges into master, GitHub automatically closes the PR. + ## My team owns a label. What should I do? -Subteams need to triage all issues in the [labels they own](#team-labels), preferably on a weekly basis. +Subteams need to triage all issues in the [labels they own](#team-labels), +preferably on a weekly basis. ### Issues 1. Filter the list of issues by your team label **and** the `untriaged` label. -2. Review the issue. -3. Identify a [priority level](#priority) and assign the label. -4. The issue may have already been prioritized by the DevEx subteam if it's a P0. Re-prioritize if needed. -5. Each issue needs to have exactly one [priority label](#priority). If an issue is either P0 or P1 we assume that is actively worked on. -6. Remove the `untriaged` label. +1. Review the issue. +1. Identify a [priority level](#priority) and assign the label. + 1. The issue may have already been prioritized by the DevEx subteam if it's a + P0. Re-prioritize if needed. + 1. Each issue needs to have exactly one [priority label](#priority). If an + issue is either P0 or P1 we assume that is actively worked on. +1. Remove the `untriaged` label. -Note that you need to be in the [bazelbuild organization](https://github.com/bazelbuild) to be able to add or remove labels. +Note that you need to be in the [bazelbuild +organization](https://github.com/bazelbuild) to be able to add or remove labels. ### Pull Requests 1. Filter the list of pull requests by your team label. -2. Review open pull requests. -3. **Optional**: If you are assigned for the review but is not the right fit for it, re-assign the appropriate reviewer to perform a code review. -4. Work with the pull request creator to complete a code review. -5. Approve the PR. -6. Ensure that all tests pass. -7. Import the patch to the internal version control system and run the internal presubmits. -8. Submit the internal patch. If the patch submits and exports successfully, the PR will be closed automatically by GitHub. +1. Review open pull requests. + 1. **Optional**: If you are assigned for the review but is not the right fit + for it, re-assign the appropriate reviewer to perform a code review. +1. Work with the pull request creator to complete a code review. +1. Approve the PR. +1. Ensure that all tests pass. +1. Import the patch to the internal version control system and run the internal + presubmits. +1. Submit the internal patch. If the patch submits and exports successfully, the + PR will be closed automatically by GitHub. ## Priority -The following definitions for priority will be used by the maintainers to triage issues. - -- [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken functionality that causes a Bazel release (minus release candidates) to be unusable, or a downed service that severely impacts development of the Bazel project. This includes regressions introduced in a new release that blocks a significant number of users, or an incompatible breaking change that was not compliant to the [Breaking Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) policy. No practical workaround exists. -- [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or feature which should be addressed in the next release, or a serious issue that impacts many users (including the development of the Bazel project), but a practical workaround exists. Typically does not require immediate action. In high demand and planned in the current quarter's roadmap. -- [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature that should be addressed but we don't currently work on. Moderate live issue in a released Bazel version that is inconvenient for a user that needs to be addressed in an future release and/or an easy workaround exists. -- [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug fix or enhancement with small impact. Not prioritized into Bazel roadmaps or any imminent release, however community contributions are encouraged. -- [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect or feature request that is unlikely to get closed. Can also be kept open for a potential re-prioritization if more users are impacted. +The following definitions for priority will be used by the maintainers to triage +issues. + +* [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken + functionality that causes a Bazel release (minus release candidates) to be + unusable, or a downed service that severely impacts development of the Bazel + project. This includes regressions introduced in a new release that blocks a + significant number of users, or an incompatible breaking change that was not + compliant to the [Breaking + Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) + policy. No practical workaround exists. +* [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or + feature which should be addressed in the next release, or a serious issue that + impacts many users (including the development of the Bazel project), but a + practical workaround exists. Typically does not require immediate action. In + high demand and planned in the current quarter's roadmap. +* [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature + that should be addressed but we don't currently work on. Moderate live issue + in a released Bazel version that is inconvenient for a user that needs to be + addressed in an future release and/or an easy workaround exists. +* [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug + fix or enhancement with small impact. Not prioritized into Bazel roadmaps or + any imminent release, however community contributions are encouraged. +* [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect + or feature request that is unlikely to get closed. Can also be kept open for a + potential re-prioritization if more users are impacted. ## Team labels -- [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team - - Contact: [ahumesky](https://github.com/ahumesky) -- [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues - - Contact: [meisterT](https://github.com/meisterT) -- [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI - - Contact: [meisterT](https://github.com/meisterT) -- [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags - - Contact: [gregestren](https://github.com/gregestren) -- [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc - - Contact: [haxorz](https://github.com/haxorz) -- [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team -- [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file - - Contact: [meteorcloudy](https://github.com/meteorcloudy) -- [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob - - Contact: [brandjon](https://github.com/brandjon) -- [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team - - Contact: [meisterT](https://github.com/meisterT) -- [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure - - Contact: [meteorcloudy](https://github.com/meteorcloudy) -- [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team - - Contact: [meisterT](https://github.com/meisterT) -- [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team - - Contact: [coeuvre](https://github.com/coeuvre) -- [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts - - Contact: [comius](https://github.com/comius) -- [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic - - Contact: [pzembrod](https://github.com/pzembrod) -- [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules - - Contact: [hvadehra](https://github.com/hvadehra) -- [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules - - Contact: [rickeylev](https://github.com/rickeylev) -- [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel - - Contact: [comius](https://github.com/comius) -- [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. - - Contact: [brandjon](https://github.com/brandjon) -- [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. - - Contact: [brandjon](https://github.com/brandjon) - -For new issues, we deprecated the `category: *` labels in favor of the team labels. +* [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team + * Contact: [ahumesky](https://github.com/ahumesky) +* [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues + * Contact: [meisterT](https://github.com/meisterT) +* [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI + * Contact: [meisterT](https://github.com/meisterT) +* [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags + * Contact: [gregestren](https://github.com/gregestren) +* [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc + * Contact: [haxorz](https://github.com/haxorz) +* [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team +* [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file + * Contact: [meteorcloudy](https://github.com/meteorcloudy) +* [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob + * Contact: [brandjon](https://github.com/brandjon) +* [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team + * Contact: [meisterT](https://github.com/meisterT) +* [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure + * Contact: [meteorcloudy](https://github.com/meteorcloudy) +* [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team + * Contact: [meisterT](https://github.com/meisterT) +* [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team + * Contact: [coeuvre](https://github.com/coeuvre) +* [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts + * Contact: [comius](https://github.com/comius) +* [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic + * Contact: [pzembrod](https://github.com/pzembrod) +* [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules + * Contact: [hvadehra](https://github.com/hvadehra) +* [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules + * Contact: [rickeylev](https://github.com/rickeylev) +* [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel + * Contact: [comius](https://github.com/comius) +* [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. + * Contact: [brandjon](https://github.com/brandjon) +* [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. + * Contact: [brandjon](https://github.com/brandjon) + +For new issues, we deprecated the `category: *` labels in favor of the team +labels. See the full list of labels [here](https://github.com/bazelbuild/bazel/labels). diff --git a/contribute/naming.mdx b/contribute/naming.mdx index 806f7cd2..144b08af 100644 --- a/contribute/naming.mdx +++ b/contribute/naming.mdx @@ -2,42 +2,72 @@ title: 'Naming a Bazel related project' --- -First, thank you for contributing to the Bazel ecosystem! Please reach out to the Bazel community on the [bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss) to share your project and its suggested name. -If you are building a Bazel related tool or sharing your Skylark rules, we recommend following these guidelines for the name of your project: + +First, thank you for contributing to the Bazel ecosystem! Please reach out to +the Bazel community on the +[bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss +) to share your project and its suggested name. + +If you are building a Bazel related tool or sharing your Skylark rules, +we recommend following these guidelines for the name of your project: ## Naming Starlark rules -See [Deploying new Starlark rules](/rules/deploying) in the docs. +See [Deploying new Starlark rules](/rules/deploying) +in the docs. ## Naming other Bazel related tools -This section applies if you are building a tool to enrich the Bazel ecosystem. For example, a new IDE plugin or a new build system migrator. +This section applies if you are building a tool to enrich the Bazel ecosystem. +For example, a new IDE plugin or a new build system migrator. -Picking a good name for your tool can be hard. If we’re not careful and use too many codenames, the Bazel ecosystem could become very difficult to understand for newcomers. +Picking a good name for your tool can be hard. If we’re not careful and use too +many codenames, the Bazel ecosystem could become very difficult to understand +for newcomers. Follow these guidelines for naming Bazel tools: -1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand for our users, we should avoid confusing them with too many new names. +1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand +for our users, we should avoid confusing them with too many new names. + +2. Prefer **using a name that includes "Bazel"**: This helps to express that it +is a Bazel related tool, it also helps people find it with a search engine. -2. Prefer **using a name that includes "Bazel"**: This helps to express that it is a Bazel related tool, it also helps people find it with a search engine. +3. Prefer **using names that are descriptive about what the tool is doing**: +Ideally, the name should not need a subtitle for users to have a first good +guess at what the tool does. Using english words separated by spaces is a good +way to achieve this. -3. Prefer **using names that are descriptive about what the tool is doing**: Ideally, the name should not need a subtitle for users to have a first good guess at what the tool does. Using english words separated by spaces is a good way to achieve this. +4. **It is not a requirement to use a floral or food theme**: Bazel evokes +[basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to +look for a name that is a plant, food or that relates to "basil." -4. **It is not a requirement to use a floral or food theme**: Bazel evokes [basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to look for a name that is a plant, food or that relates to "basil." +5. **If your tool relates to another third party brand, use it only as a +descriptor**: For example, use "Bazel migrator for Cmake" instead of +"Cmake Bazel migrator". -5. **If your tool relates to another third party brand, use it only as a descriptor**: For example, use "Bazel migrator for Cmake" instead of "Cmake Bazel migrator". +These guidelines also apply to the GitHub repository URL. Reading the repository +URL should help people understand what the tool does. Of course, the repository +name can be shorter and must use dashes instead of spaces and lower case letters. -These guidelines also apply to the GitHub repository URL. Reading the repository URL should help people understand what the tool does. Of course, the repository name can be shorter and must use dashes instead of spaces and lower case letters. Examples of good names: -- *Bazel for Eclipse*: Users will understand that if they want to use Bazel with Eclipse, this is where they should be looking. It uses a third party brand as a descriptor. -- *Bazel buildfarm*: A "buildfarm" is a [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users will understand that this project relates to building on servers. +* *Bazel for Eclipse*: Users will understand that if they want to use Bazel + with Eclipse, this is where they should be looking. It uses a third party brand + as a descriptor. +* *Bazel buildfarm*: A "buildfarm" is a + [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users + will understand that this project relates to building on servers. Examples of names to avoid: -- *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) does not relate enough to the Bazel project. -- *Bazelizer*: The tool behind this name could do a lot of things, this name is not descriptive enough. +* *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) + does not relate enough to the Bazel project. +* *Bazelizer*: The tool behind this name could do a lot of things, this name is + not descriptive enough. -Note that these recommendations are aligned with the [guidelines](https://opensource.google.com/docs/releasing/preparing/#name) Google uses when open sourcing a project. +Note that these recommendations are aligned with the +[guidelines](https://opensource.google.com/docs/releasing/preparing/#name) +Google uses when open sourcing a project. diff --git a/contribute/patch-acceptance.mdx b/contribute/patch-acceptance.mdx index c877c554..87376afd 100644 --- a/contribute/patch-acceptance.mdx +++ b/contribute/patch-acceptance.mdx @@ -2,26 +2,51 @@ title: 'Patch Acceptance Process' --- -This page outlines how contributors can propose and make changes to the Bazel code base. -1. Read the [Bazel Contribution policy](/contribute/policy). - -2. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to discuss your plan and design. Pull requests that change or add behavior need a corresponding issue for tracking. - -3. If you're proposing significant changes, write a [design document](/contribute/design-documents). - -4. Ensure you've signed a [Contributor License Agreement](https://cla.developers.google.com). - -5. Prepare a git commit that implements the feature. Don't forget to add tests and update the documentation. If your change has user-visible effects, please [add release notes](/contribute/release-notes). If it is an incompatible change, read the [guide for rolling out breaking changes](/contribute/breaking-changes). -6. Create a pull request on [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, read [about pull requests](https://help.github.com/articles/about-pull-requests/). Note that we restrict permissions to create branches on the main Bazel repository, so you will need to push your commit to [your own fork of the repository](https://help.github.com/articles/working-with-forks/). +This page outlines how contributors can propose and make changes to the Bazel +code base. -7. A Bazel maintainer should assign you a reviewer within two business days (excluding holidays in the USA and Germany). If you aren't assigned a reviewer in that time, you can request one by emailing \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). - -8. Work with the reviewer to complete a code review. For each change, create a new commit and push it to make changes to your pull request. If the review takes too long (for instance, if the reviewer is unresponsive), send an email to \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). - -9. After your review is complete, a Bazel maintainer applies your patch to Google's internal version control system. - - This triggers internal presubmit checks that may suggest more changes. If you haven't expressed a preference, the maintainer submitting your change adds "trivial" changes (such as [linting](https://en.wikipedia.org/wiki/Lint_\(software\))) that don't affect design. If deeper changes are required or you'd prefer to apply changes directly, you and the reviewer should communicate preferences clearly in review comments. - - After internal submission, the patch is exported as a Git commit, at which point the GitHub pull request is closed. All final changes are attributed to you. +1. Read the [Bazel Contribution policy](/contribute/policy). +1. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to + discuss your plan and design. Pull requests that change or add behavior + need a corresponding issue for tracking. +1. If you're proposing significant changes, write a + [design document](/contribute/design-documents). +1. Ensure you've signed a [Contributor License + Agreement](https://cla.developers.google.com). +1. Prepare a git commit that implements the feature. Don't forget to add tests + and update the documentation. If your change has user-visible effects, please + [add release notes](/contribute/release-notes). If it is an incompatible change, + read the [guide for rolling out breaking changes](/contribute/breaking-changes). +1. Create a pull request on + [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, + read [about pull + requests](https://help.github.com/articles/about-pull-requests/). Note that + we restrict permissions to create branches on the main Bazel repository, so + you will need to push your commit to [your own fork of the + repository](https://help.github.com/articles/working-with-forks/). +1. A Bazel maintainer should assign you a reviewer within two business days + (excluding holidays in the USA and Germany). If you aren't assigned a + reviewer in that time, you can request one by emailing + [bazel-discuss@googlegroups.com] + (mailto:bazel-discuss@googlegroups.com). +1. Work with the reviewer to complete a code review. For each change, create a + new commit and push it to make changes to your pull request. If the review + takes too long (for instance, if the reviewer is unresponsive), send an email to + [bazel-discuss@googlegroups.com] + (mailto:bazel-discuss@googlegroups.com). +1. After your review is complete, a Bazel maintainer applies your patch to + Google's internal version control system. + + This triggers internal presubmit checks + that may suggest more changes. If you haven't expressed a preference, the + maintainer submitting your change adds "trivial" changes (such as + [linting](https://en.wikipedia.org/wiki/Lint_(software))) that don't affect + design. If deeper changes are required or you'd prefer to apply + changes directly, you and the reviewer should communicate preferences + clearly in review comments. + + After internal submission, the patch is exported as a Git commit, + at which point the GitHub pull request is closed. All final changes + are attributed to you. diff --git a/contribute/policy.mdx b/contribute/policy.mdx index 407f4c68..1bf00290 100644 --- a/contribute/policy.mdx +++ b/contribute/policy.mdx @@ -1,59 +1,78 @@ +translation: human +page_type: lcat --- title: 'Contribution policy' --- + + This page covers Bazel's governance model and contribution policy. ## Governance model -The [Bazel project](https://github.com/bazelbuild) is led and managed by Google and has a large community of contributors outside of Google. Some Bazel components (such as specific rules repositories under the [bazelbuild](https://github.com/bazelbuild) organization) are led, maintained, and managed by members of the community. The Google Bazel team reviews suggestions to add community-owned repositories (such as rules) to the [bazelbuild](https://github.com/bazelbuild) GitHub organization. +The [Bazel project](https://github.com/bazelbuild) is led and managed by Google +and has a large community of contributors outside of Google. Some Bazel +components (such as specific rules repositories under the +[bazelbuild](https://github.com/bazelbuild) organization) are led, +maintained, and managed by members of the community. The Google Bazel team +reviews suggestions to add community-owned repositories (such as rules) to the +[bazelbuild](https://github.com/bazelbuild) GitHub organization. ### Contributor roles -Here are outlines of the roles in the Bazel project, including their responsibilities: - -- **Owners**: The Google Bazel team. Owners are responsible for: - - - Strategy, maintenance, and leadership of the Bazel project. - - Building and maintaining Bazel's core functionality. - - Appointing Maintainers and approving new repositories. - -- **Maintainers**: The Google Bazel team and designated GitHub users. Maintainers are responsible for: - - - Building and maintaining the primary functionality of their repository. - - Reviewing and approving contributions to areas of the Bazel code base. - - Supporting users and contributors with timely and transparent issue management, PR review, and documentation. - - Releasing, testing and collaborating with Bazel Owners. - -- **Contributors**: All users who contribute code or documentation to the Bazel project. - - - Creating well-written PRs to contribute to Bazel's codebase and documentation. - - Using standard channels, such as GitHub Issues, to propose changes and report issues. +Here are outlines of the roles in the Bazel project, including their +responsibilities: + +* **Owners**: The Google Bazel team. Owners are responsible for: + * Strategy, maintenance, and leadership of the Bazel project. + * Building and maintaining Bazel's core functionality. + * Appointing Maintainers and approving new repositories. +* **Maintainers**: The Google Bazel team and designated GitHub users. + Maintainers are responsible for: + * Building and maintaining the primary functionality of their repository. + * Reviewing and approving contributions to areas of the Bazel code base. + * Supporting users and contributors with timely and transparent issue + management, PR review, and documentation. + * Releasing, testing and collaborating with Bazel Owners. +* **Contributors**: All users who contribute code or documentation to the + Bazel project. + * Creating well-written PRs to contribute to Bazel's codebase and + documentation. + * Using standard channels, such as GitHub Issues, to propose changes and + report issues. ### Becoming a Maintainer -Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as rule sets. Contributors with a record of consistent, responsible past contributions who are planning major contributions in the future could be considered to become qualified Maintainers. +Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as +rule sets. Contributors with a record of consistent, responsible past +contributions who are planning major contributions in the future could be +considered to become qualified Maintainers. ## Contribution policy -The Bazel project accepts contributions from external contributors. Here are the contribution policies for Google-managed and Community-managed areas of code. - -- **Licensing**. All Maintainers and Contributors must sign the [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). - -- **Contributions**. Owners and Maintainers should make every effort to accept worthwhile contributions. All contributions must be: - - - Well written and well tested - - Discussed and approved by the Maintainers of the relevant area of code. Discussions and approvals happen on GitHub Issues and in GitHub PRs. Larger contributions require a [design review](/contribute/design-documents). - - Added to Bazel's Continuous Integration system if not already present. - - Supportable and aligned with Bazel product direction - -- **Code review**. All changes in all `bazelbuild` repositories require review: - - - All PRs must be approved by an Owner or Maintainer. - - Only Owners and Maintainers can merge PRs. - -- **Compatibility**. Owners may need to reject or request modifications to PRs in the unlikely event that the change requires substantial modifications to internal Google systems. - -- **Documentation**. Where relevant, feature contributions should include documentation updates. - -For more details on contributing to Bazel, see our [contribution guidelines](/contribute/). +The Bazel project accepts contributions from external contributors. Here are the +contribution policies for Google-managed and Community-managed areas of code. + +* **Licensing**. All Maintainers and Contributors must sign the + [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). +* **Contributions**. Owners and Maintainers should make every effort to accept + worthwhile contributions. All contributions must be: + * Well written and well tested + * Discussed and approved by the Maintainers of the relevant area of code. + Discussions and approvals happen on GitHub Issues and in GitHub PRs. + Larger contributions require a + [design review](/contribute/design-documents). + * Added to Bazel's Continuous Integration system if not already present. + * Supportable and aligned with Bazel product direction +* **Code review**. All changes in all `bazelbuild` repositories require + review: + * All PRs must be approved by an Owner or Maintainer. + * Only Owners and Maintainers can merge PRs. +* **Compatibility**. Owners may need to reject or request modifications to PRs + in the unlikely event that the change requires substantial modifications to + internal Google systems. +* **Documentation**. Where relevant, feature contributions should include + documentation updates. + +For more details on contributing to Bazel, see our +[contribution guidelines](/contribute/). diff --git a/contribute/release-notes.mdx b/contribute/release-notes.mdx index bd0d9467..83e1d75b 100644 --- a/contribute/release-notes.mdx +++ b/contribute/release-notes.mdx @@ -2,44 +2,77 @@ title: 'Writing release notes' --- + + This document is targeted at Bazel contributors. -Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release note. This is used by the Bazel team to track changes in each release and write the release announcement. +Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release +note. This is used by the Bazel team to track changes in each release and write +the release announcement. ## Overview -- Is your change a bugfix? In that case, you don't need a release note. Please include a reference to the GitHub issue. +* Is your change a bugfix? In that case, you don't need a release note. Please + include a reference to the GitHub issue. -- If the change adds / removes / changes Bazel in a user-visible way, then it may be advantageous to mention it. +* If the change adds / removes / changes Bazel in a user-visible way, then it + may be advantageous to mention it. -If the change is significant, follow the [design document policy](/contribute/design-documents) first. +If the change is significant, follow the [design document +policy](/contribute/design-documents) first. ## Guidelines -The release notes will be read by our users, so it should be short (ideally one sentence), avoid jargon (Bazel-internal terminology), should focus on what the change is about. +The release notes will be read by our users, so it should be short (ideally one +sentence), avoid jargon (Bazel-internal terminology), should focus on what the +change is about. -- Include a link to the relevant documentation. Almost any release note should contain a link. If the description mentions a flag, a feature, a command name, users will probably want to know more about it. +* Include a link to the relevant documentation. Almost any release note should + contain a link. If the description mentions a flag, a feature, a command name, + users will probably want to know more about it. -- Use backquotes around code, symbols, flags, or any word containing an underscore. +* Use backquotes around code, symbols, flags, or any word containing an + underscore. -- Do not just copy and paste bug descriptions. They are often cryptic and only make sense to us and leave the user scratching their head. Release notes are meant to explain what has changed and why in user-understandable language. +* Do not just copy and paste bug descriptions. They are often cryptic and only + make sense to us and leave the user scratching their head. Release notes are + meant to explain what has changed and why in user-understandable language. -- Always use present tense and the format "Bazel now supports Y" or "X now does Z." We don't want our release notes to sound like bug entries. All release note entries should be informative and use a consistent style and language. +* Always use present tense and the format "Bazel now supports Y" or "X now does + Z." We don't want our release notes to sound like bug entries. All release + note entries should be informative and use a consistent style and language. -- If something has been deprecated or removed, use "X has been deprecated" or "X has been removed." Not "is removed" or "was removed." +* If something has been deprecated or removed, use "X has been deprecated" or "X + has been removed." Not "is removed" or "was removed." -- If Bazel now does something differently, use "X now $newBehavior instead of $oldBehavior" in present tense. This lets the user know in detail what to expect when they use the new release. +* If Bazel now does something differently, use "X now $newBehavior instead of + $oldBehavior" in present tense. This lets the user know in detail what to + expect when they use the new release. -- If Bazel now supports or no longer supports something, use "Bazel now supports / no longer supports X". +* If Bazel now supports or no longer supports something, use "Bazel now supports + / no longer supports X". -- Explain why something has been removed / deprecated / changed. One sentence is enough but we want the user to be able to evaluate impact on their builds. +* Explain why something has been removed / deprecated / changed. One sentence is + enough but we want the user to be able to evaluate impact on their builds. -- Do NOT make any promises about future functionality. Avoid "this flag will be removed" or "this will be changed." It introduces uncertainty. The first thing the user will wonder is "when?" and we don't want them to start worrying about their current builds breaking at some unknown time. +* Do NOT make any promises about future functionality. Avoid "this flag will be + removed" or "this will be changed." It introduces uncertainty. The first thing + the user will wonder is "when?" and we don't want them to start worrying about + their current builds breaking at some unknown time. ## Process -As part of the [release process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), we collect the `RELNOTES` tags of every commit. We copy everything in a [Google Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) where we review, edit, and organize the notes. +As part of the [release +process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), +we collect the `RELNOTES` tags of every commit. We copy everything in a [Google +Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) +where we review, edit, and organize the notes. -The release manager sends an email to the [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. Bazel contributors are invited to contribute to the document and make sure their changes are correctly reflected in the announcement. +The release manager sends an email to the +[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. +Bazel contributors are invited to contribute to the document and make sure +their changes are correctly reflected in the announcement. -Later, the announcement will be submitted to the [Bazel blog](https://blog.bazel.build/), using the [bazel-blog repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). +Later, the announcement will be submitted to the [Bazel +blog](https://blog.bazel.build/), using the [bazel-blog +repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). diff --git a/contribute/statemachine-guide.mdx b/contribute/statemachine-guide.mdx index 86f34c50..e98a96e8 100644 --- a/contribute/statemachine-guide.mdx +++ b/contribute/statemachine-guide.mdx @@ -1,28 +1,64 @@ --- -title: 'A Guide to Skyframe StateMachines' +title: 'A Guide to Skyframe `StateMachine`s' --- + + ## Overview -A Skyframe `StateMachine` is a *deconstructed* function-object that resides on the heap. It supports flexible and evaluation without redundancy[1](#user-content-fn-1) when required values are not immediately available but computed asynchronously. The `StateMachine` cannot tie up a thread resource while waiting, but instead has to be suspended and resumed. The deconstruction thus exposes explicit re-entry points so that prior computations can be skipped. +A Skyframe `StateMachine` is a *deconstructed* function-object that resides on +the heap. It supports flexible and evaluation without redundancy[^1] when +required values are not immediately available but computed asynchronously. The +`StateMachine` cannot tie up a thread resource while waiting, but instead has to +be suspended and resumed. The deconstruction thus exposes explicit re-entry +points so that prior computations can be skipped. -`StateMachine`s can be used to express sequences, branching, structured logical concurrency and are tailored specifically for Skyframe interaction. `StateMachine`s can be composed into larger `StateMachine`s and share sub-`StateMachine`s. Concurrency is always hierarchical by construction and purely logical. Every concurrent subtask runs in the single shared parent SkyFunction thread. +`StateMachine`s can be used to express sequences, branching, structured logical +concurrency and are tailored specifically for Skyframe interaction. +`StateMachine`s can be composed into larger `StateMachine`s and share +sub-`StateMachine`s. Concurrency is always hierarchical by construction and +purely logical. Every concurrent subtask runs in the single shared parent +SkyFunction thread. ## Introduction -This section briefly motivates and introduces `StateMachine`s, found in the [`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) package. +This section briefly motivates and introduces `StateMachine`s, found in the +[`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) +package. ### A brief introduction to Skyframe restarts -Skyframe is a framework that performs parallel evaluation of dependency graphs. Each node in the graph corresponds with the evaluation of a SkyFunction with a SkyKey specifying its parameters and SkyValue specifying its result. The computational model is such that a SkyFunction may lookup SkyValues by SkyKey, triggering recursive, parallel evaluation of additional SkyFunctions. Instead of blocking, which would tie up a thread, when a requested SkyValue is not yet ready because some subgraph of computation is incomplete, the requesting SkyFunction observes a `null` `getValue` response and should return `null` instead of a SkyValue, signaling that it is incomplete due to missing inputs. Skyframe *restarts* the SkyFunctions when all previously requested SkyValues become available. - -Before the introduction of `SkyKeyComputeState`, the traditional way of handling a restart was to fully rerun the computation. Although this has quadratic complexity, functions written this way eventually complete because each rerun, fewer lookups return `null`. With `SkyKeyComputeState` it is possible to associate hand-specified check-point data with a SkyFunction, saving significant recomputation. - -`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate virtually all recomputation when a SkyFunction restarts (assuming that `SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume execution hooks. +Skyframe is a framework that performs parallel evaluation of dependency graphs. +Each node in the graph corresponds with the evaluation of a SkyFunction with a +SkyKey specifying its parameters and SkyValue specifying its result. The +computational model is such that a SkyFunction may lookup SkyValues by SkyKey, +triggering recursive, parallel evaluation of additional SkyFunctions. Instead of +blocking, which would tie up a thread, when a requested SkyValue is not yet +ready because some subgraph of computation is incomplete, the requesting +SkyFunction observes a `null` `getValue` response and should return `null` +instead of a SkyValue, signaling that it is incomplete due to missing inputs. +Skyframe *restarts* the SkyFunctions when all previously requested SkyValues +become available. + +Before the introduction of `SkyKeyComputeState`, the traditional way of handling +a restart was to fully rerun the computation. Although this has quadratic +complexity, functions written this way eventually complete because each rerun, +fewer lookups return `null`. With `SkyKeyComputeState` it is possible to +associate hand-specified check-point data with a SkyFunction, saving significant +recomputation. + +`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate +virtually all recomputation when a SkyFunction restarts (assuming that +`SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume +execution hooks. ### Stateful computations inside `SkyKeyComputeState` -From an object-oriented design standpoint, it makes sense to consider storing computational objects inside `SkyKeyComputeState` instead of pure data values. In *Java*, the bare minimum description of a behavior carrying object is a *functional interface* and it turns out to be sufficient. A `StateMachine` has the following, curiously recursive, definition[2](#user-content-fn-2). +From an object-oriented design standpoint, it makes sense to consider storing +computational objects inside `SkyKeyComputeState` instead of pure data values. +In *Java*, the bare minimum description of a behavior carrying object is a +*functional interface* and it turns out to be sufficient. A `StateMachine` has +the following, curiously recursive, definition[^2]. ``` @FunctionalInterface @@ -31,9 +67,12 @@ public interface StateMachine { } ``` -The `Tasks` interface is analogous to `SkyFunction.Environment` but it is designed for asynchrony and adds support for logically concurrent subtasks[3](#user-content-fn-3). +The `Tasks` interface is analogous to `SkyFunction.Environment` but it is +designed for asynchrony and adds support for logically concurrent subtasks[^3]. -The return value of `step` is another `StateMachine`, allowing the specification of a sequence of steps, inductively. `step` returns `DONE` when the `StateMachine` is done. For example: +The return value of `step` is another `StateMachine`, allowing the specification +of a sequence of steps, inductively. `step` returns `DONE` when the +`StateMachine` is done. For example: ``` class HelloWorld implements StateMachine { @@ -59,66 +98,98 @@ hello world ``` -Note that the method reference `this::step2` is also a `StateMachine` due to `step2` satisfying `StateMachine`'s functional interface definition. Method references are the most common way to specify the next state in a `StateMachine`. +Note that the method reference `this::step2` is also a `StateMachine` due to +`step2` satisfying `StateMachine`'s functional interface definition. Method +references are the most common way to specify the next state in a +`StateMachine`. ![Suspending and resuming](/contribute/images/suspend-resume.svg) -Intuitively, breaking a computation down into `StateMachine` steps, instead of a monolithic function, provides the hooks needed to *suspend* and *resume* a computation. When `StateMachine.step` returns, there is an explicit *suspension* point. The continuation specified by the returned `StateMachine` value is an explicit *resume* point. Recomputation can thus be avoided because the computation can be picked up exactly where it left off. +Intuitively, breaking a computation down into `StateMachine` steps, instead of a +monolithic function, provides the hooks needed to *suspend* and *resume* a +computation. When `StateMachine.step` returns, there is an explicit *suspension* +point. The continuation specified by the returned `StateMachine` value is an +explicit *resume* point. Recomputation can thus be avoided because the +computation can be picked up exactly where it left off. ### Callbacks, continuations and asynchronous computation -In technical terms, a `StateMachine` serves as a *continuation*, determining the subsequent computation to be executed. Instead of blocking, a `StateMachine` can voluntarily *suspend* by returning from the `step` function, which transfers control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can then switch to a ready `StateMachine` or relinquish control back to Skyframe. +In technical terms, a `StateMachine` serves as a *continuation*, determining the +subsequent computation to be executed. Instead of blocking, a `StateMachine` can +voluntarily *suspend* by returning from the `step` function, which transfers +control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can +then switch to a ready `StateMachine` or relinquish control back to Skyframe. -Traditionally, *callbacks* and *continuations* are conflated into one concept. However, `StateMachine`s maintain a distinction between the two. +Traditionally, *callbacks* and *continuations* are conflated into one concept. +However, `StateMachine`s maintain a distinction between the two. -- *Callback* - describes where to store the result of an asynchronous computation. -- *Continuation* - specifies the next execution state. +* *Callback* - describes where to store the result of an asynchronous + computation. +* *Continuation* - specifies the next execution state. -Callbacks are required when invoking an asynchronous operation, which means that the actual operation doesn't occur immediately upon calling the method, as in the case of a SkyValue lookup. Callbacks should be kept as simple as possible. +Callbacks are required when invoking an asynchronous operation, which means that +the actual operation doesn't occur immediately upon calling the method, as in +the case of a SkyValue lookup. Callbacks should be kept as simple as possible. -Caution: A common pitfall of callbacks is that the asynchronous computation must ensure the callback is called by the end of every reachable path. It's possible to overlook some branches and the compiler doesn't give warnings about this. +Caution: A common pitfall of callbacks is that the asynchronous computation must +ensure the callback is called by the end of every reachable path. It's possible +to overlook some branches and the compiler doesn't give warnings about this. -*Continuations* are the `StateMachine` return values of `StateMachine`s and encapsulate the complex execution that follows once all asynchronous computations resolve. This structured approach helps to keep the complexity of callbacks manageable. +*Continuations* are the `StateMachine` return values of `StateMachine`s and +encapsulate the complex execution that follows once all asynchronous +computations resolve. This structured approach helps to keep the complexity of +callbacks manageable. ## Tasks -The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues by SkyKey and to schedule concurrent subtasks. +The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues +by SkyKey and to schedule concurrent subtasks. ``` interface Tasks { void enqueue(StateMachine subtask); - void lookUp(SkyKey key, Consumer<SkyValue> sink); + void lookUp(SkyKey key, Consumer sink); - <E extends Exception> - void lookUp(SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); + + void lookUp(SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); // lookUp overloads for 2 and 3 exception types exist, but are elided here. } ``` -Tip: When any state uses the `Tasks` interface to perform lookups or create subtasks, those lookups and subtasks will complete before the next state begins. +Tip: When any state uses the `Tasks` interface to perform lookups or create +subtasks, those lookups and subtasks will complete before the next state begins. -Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create subtasks, they all *transitively* complete before the next state begins. +Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create +subtasks, they all *transitively* complete before the next state begins. ### SkyValue lookups -`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are analogous to `SkyFunction.Environment.getValue` and `SkyFunction.Environment.getValueOrThrow` and have similar exception handling semantics. The implementation does not immediately perform the lookup, but instead, batches[4](#user-content-fn-4) as many lookups as possible before doing so. The value might not be immediately available, for example, requiring a Skyframe restart, so the caller specifies what to do with the resulting value using a callback. +`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are +analogous to `SkyFunction.Environment.getValue` and +`SkyFunction.Environment.getValueOrThrow` and have similar exception handling +semantics. The implementation does not immediately perform the lookup, but +instead, batches[^4] as many lookups as possible before doing so. The value +might not be immediately available, for example, requiring a Skyframe restart, +so the caller specifies what to do with the resulting value using a callback. -The `StateMachine` processor ([`Driver`s and bridging to SkyFrame](#drivers-and-bridging)) guarantees that the value is available before the next state begins. An example follows. +The `StateMachine` processor ([`Driver`s and bridging to +SkyFrame](#drivers-and-bridging)) guarantees that the value is available before +the next state begins. An example follows. ``` -class DoesLookup implements StateMachine, Consumer<SkyValue> { +class DoesLookup implements StateMachine, Consumer { private Value value; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new Key(), (Consumer<SkyValue>) this); + tasks.lookUp(new Key(), (Consumer) this); return this::processValue; } // The `lookUp` call in `step` causes this to be called before `processValue`. - @Override // Implementation of Consumer<SkyValue>. + @Override // Implementation of Consumer. public void accept(SkyValue value) { this.value = (Value)value; } @@ -130,15 +201,25 @@ class DoesLookup implements StateMachine, Consumer<SkyValue> { } ``` -In the above example, the first step does a lookup for `new Key()`, passing `this` as the consumer. That is possible because `DoesLookup` implements `Consumer<SkyValue>`. +In the above example, the first step does a lookup for `new Key()`, passing +`this` as the consumer. That is possible because `DoesLookup` implements +`Consumer`. -Tip: When passing `this` as a value sink, it's helpful to readers to upcast it to the receiver type to narrow down the purpose of passing `this`. The example passes `(Consumer<SkyValue>) this`. +Tip: When passing `this` as a value sink, it's helpful to readers to upcast it +to the receiver type to narrow down the purpose of passing `this`. The example +passes `(Consumer) this`. -By contract, before the next state `DoesLookup.processValue` begins, all the lookups of `DoesLookup.step` are complete. Therefore `value` is available when it is accessed in `processValue`. +By contract, before the next state `DoesLookup.processValue` begins, all the +lookups of `DoesLookup.step` are complete. Therefore `value` is available when +it is accessed in `processValue`. ### Subtasks -`Tasks.enqueue` requests the execution of logically concurrent subtasks. Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s can do, including recursively creating more subtasks or looking up SkyValues. Much like `lookUp`, the state machine driver ensures that all subtasks are complete before proceeding to the next step. An example follows. +`Tasks.enqueue` requests the execution of logically concurrent subtasks. +Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s +can do, including recursively creating more subtasks or looking up SkyValues. +Much like `lookUp`, the state machine driver ensures that all subtasks are +complete before proceeding to the next step. An example follows. ``` class Subtasks implements StateMachine { @@ -176,15 +257,22 @@ class Subtasks implements StateMachine { } ``` -Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a single thread so the "concurrent" update of `i` does not need any synchronization. +Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a +single thread so the "concurrent" update of `i` does not need any +synchronization. ### Structured concurrency -Since every `lookUp` and `enqueue` must resolve before advancing to the next state, it means that concurrency is naturally limited to tree-structures. It's possible to create hierarchical[5](#user-content-fn-5) concurrency as shown in the following example. +Since every `lookUp` and `enqueue` must resolve before advancing to the next +state, it means that concurrency is naturally limited to tree-structures. It's +possible to create hierarchical[^5] concurrency as shown in the following +example. ![Structured Concurrency](/contribute/images/structured-concurrency.svg) -It's hard to tell from the *UML* that the concurrency structure forms a tree. There's an [alternate view](#concurrency-tree-diagram) that better shows the tree structure. +It's hard to tell from the *UML* that the concurrency structure forms a tree. +There's an [alternate view](#concurrency-tree-diagram) that better shows the +tree structure. ![Unstructured Concurrency](/contribute/images/unstructured-concurrency.svg) @@ -192,15 +280,19 @@ Structured concurrency is much easier to reason about. ## Composition and control flow patterns -This section presents examples for how multiple `StateMachine`s can be composed and solutions to certain control flow problems. +This section presents examples for how multiple `StateMachine`s can be composed +and solutions to certain control flow problems. ### Sequential states -This is the most common and straightforward control flow pattern. An example of this is shown in [Stateful computations inside `SkyKeyComputeState`](#stateful-computations). +This is the most common and straightforward control flow pattern. An example of +this is shown in [Stateful computations inside +`SkyKeyComputeState`](#stateful-computations). ### Branching -Branching states in `StateMachine`s can be achieved by returning different values using regular *Java* control flow, as shown in the following example. +Branching states in `StateMachine`s can be achieved by returning different +values using regular *Java* control flow, as shown in the following example. ``` class Branch implements StateMachine { @@ -220,11 +312,18 @@ It’s very common for certain branches to return `DONE`, for early completion. ### Advanced sequential composition -Since the `StateMachine` control structure is memoryless, sharing `StateMachine` definitions as subtasks can sometimes be awkward. Let *M1* and *M2* be `StateMachine` instances that share a `StateMachine`, *S*, with *M1* and *M2* being the sequences *\<A, S, B\>* and *\<X, S, Y\>* respectively. The problem is that *S* doesn’t know whether to continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a call stack. This section reviews some techniques for achieving this. +Since the `StateMachine` control structure is memoryless, sharing `StateMachine` +definitions as subtasks can sometimes be awkward. Let *M1* and +*M2* be `StateMachine` instances that share a `StateMachine`, *S*, +with *M1* and *M2* being the sequences *<A, S, B>* and +*<X, S, Y>* respectively. The problem is that *S* doesn’t know whether to +continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a +call stack. This section reviews some techniques for achieving this. #### `StateMachine` as terminal sequence element -This doesn’t solve the initial problem posed. It only demonstrates sequential composition when the shared `StateMachine` is terminal in the sequence. +This doesn’t solve the initial problem posed. It only demonstrates sequential +composition when the shared `StateMachine` is terminal in the sequence. ``` // S is the shared state machine. @@ -251,7 +350,8 @@ This works even if *S* is itself a complex state machine. #### Subtask for sequential composition -Since enqueued subtasks are guaranteed to complete before the next state, it’s sometimes possible to slightly abuse[6](#user-content-fn-6) the subtask mechanism. +Since enqueued subtasks are guaranteed to complete before the next state, it’s +sometimes possible to slightly abuse[^6] the subtask mechanism. ``` class M1 implements StateMachine { @@ -259,7 +359,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // S starts after `step` returns and by contract must complete before `doB` - // begins. It is effectively sequential, inducing the sequence < A, S, B >. + // begins. It is effectively sequential, inducing the sequence < A, S, B >. tasks.enqueue(new S()); return this::doB; } @@ -274,7 +374,7 @@ class M2 implements StateMachine { @Override public StateMachine step(Tasks tasks) { performX(); - // Similarly, this induces the sequence < X, S, Y>. + // Similarly, this induces the sequence < X, S, Y>. tasks.enqueue(new S()); return this::doY; } @@ -288,7 +388,10 @@ class M2 implements StateMachine { #### `runAfter` injection -Sometimes, abusing `Tasks.enqueue` is impossible because there are other parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* executes. In this case, injecting a `runAfter` parameter into *S* can be used to inform *S* of what to do next. +Sometimes, abusing `Tasks.enqueue` is impossible because there are other +parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* +executes. In this case, injecting a `runAfter` parameter into *S* can be used to +inform *S* of what to do next. ``` class S implements StateMachine { @@ -315,7 +418,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // Passes `this::doB` as the `runAfter` parameter of S, resulting in the - // sequence < A, S, B >. + // sequence < A, S, B >. return new S(/* runAfter= */ this::doB); } @@ -330,7 +433,7 @@ class M2 implements StateMachine { public StateMachine step(Tasks tasks) { performX(); // Passes `this::doY` as the `runAfter` parameter of S, resulting in the - // sequence < X, S, Y >. + // sequence < X, S, Y >. return new S(/* runAfter= */ this::doY); } @@ -341,7 +444,10 @@ class M2 implements StateMachine { } ``` -This approach is cleaner than abusing subtasks. However, applying this too liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is the road to [Callback Hell](#callback-hell). It’s better to break up sequential `runAfter`s with ordinary sequential states instead. +This approach is cleaner than abusing subtasks. However, applying this too +liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is +the road to [Callback Hell](#callback-hell). It’s better to break up sequential +`runAfter`s with ordinary sequential states instead. ``` return new S(/* runAfter= */ new T(/* runAfter= */ this::nextStep)) @@ -360,21 +466,37 @@ can be replaced with the following. } ``` -Note: It's possible to pass `DONE` as the `runAfter` parameter when there's nothing to run afterwards. +Note: It's possible to pass `DONE` as the `runAfter` parameter when there's +nothing to run afterwards. -Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` to let the reader know the meaning at the callsite. +Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` +to let the reader know the meaning at the callsite. #### *Forbidden* alternative: `runAfterUnlessError` -In an earlier draft, we had considered a `runAfterUnlessError` that would abort early on errors. This was motivated by the fact that errors often end up getting checked twice, once by the `StateMachine` that has a `runAfter` reference and once by the `runAfter` machine itself. +In an earlier draft, we had considered a `runAfterUnlessError` that would abort +early on errors. This was motivated by the fact that errors often end up getting +checked twice, once by the `StateMachine` that has a `runAfter` reference and +once by the `runAfter` machine itself. -After some deliberation, we decided that uniformity of the code is more important than deduplicating the error checking. It would be confusing if the `runAfter` mechanism did not work in a consistent manner with the `tasks.enqueue` mechanism, which always requires error checking. +After some deliberation, we decided that uniformity of the code is more +important than deduplicating the error checking. It would be confusing if the +`runAfter` mechanism did not work in a consistent manner with the +`tasks.enqueue` mechanism, which always requires error checking. -Warning: When using `runAfter`, the machine that has the injected `runAfter` should invoke it unconditionally at completion, even on error, for consistency. +Warning: When using `runAfter`, the machine that has the injected `runAfter` +should invoke it unconditionally at completion, even on error, for consistency. ### Direct delegation -Each time there is a formal state transition, the main `Driver` loop advances. As per contract, advancing states means that all previously enqueued SkyValue lookups and subtasks resolve before the next state executes. Sometimes the logic of a delegate `StateMachine` makes a phase advance unnecessary or counterproductive. For example, if the first `step` of the delegate performs SkyKey lookups that could be parallelized with lookups of the delegating state then a phase advance would make them sequential. It could make more sense to perform direct delegation, as shown in the example below. +Each time there is a formal state transition, the main `Driver` loop advances. +As per contract, advancing states means that all previously enqueued SkyValue +lookups and subtasks resolve before the next state executes. Sometimes the logic +of a delegate `StateMachine` makes a phase advance unnecessary or +counterproductive. For example, if the first `step` of the delegate performs +SkyKey lookups that could be parallelized with lookups of the delegating state +then a phase advance would make them sequential. It could make more sense to +perform direct delegation, as shown in the example below. ``` class Parent implements StateMachine { @@ -420,37 +542,49 @@ class Delegate implements StateMachine { ## Data flow -The focus of the previous discussion has been on managing control flow. This section describes the propagation of data values. +The focus of the previous discussion has been on managing control flow. This +section describes the propagation of data values. ### Implementing `Tasks.lookUp` callbacks -There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue lookups](#skyvalue-lookups). This section provides rationale and suggests approaches for handling multiple SkyValues. +There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue +lookups](#skyvalue-lookups). This section provides rationale and suggests +approaches for handling multiple SkyValues. #### `Tasks.lookUp` callbacks The `Tasks.lookUp` method takes a callback, `sink`, as a parameter. ``` - void lookUp(SkyKey key, Consumer<SkyValue> sink); + void lookUp(SkyKey key, Consumer sink); ``` The idiomatic approach would be to use a *Java* lambda to implement this: ``` - tasks.lookUp(key, value -> myValue = (MyValueClass)value); + tasks.lookUp(key, value -> myValue = (MyValueClass)value); ``` -with `myValue` being a member variable of the `StateMachine` instance doing the lookup. However, the lambda requires an extra memory allocation compared to implementing the `Consumer<SkyValue>` interface in the `StateMachine` implementation. The lambda is still useful when there are multiple lookups that would be ambiguous. +with `myValue` being a member variable of the `StateMachine` instance doing the +lookup. However, the lambda requires an extra memory allocation compared to +implementing the `Consumer` interface in the `StateMachine` +implementation. The lambda is still useful when there are multiple lookups that +would be ambiguous. -Note: Bikeshed warning. There is a noticeable difference of approximately 1% end-to-end CPU usage when implementing callbacks systematically in `StateMachine` implementations compared to using lambdas, which makes this recommendation debatable. To avoid unnecessary debates, it is advised to leave the decision up to the individual implementing the solution. +Note: Bikeshed warning. There is a noticeable difference of approximately 1% +end-to-end CPU usage when implementing callbacks systematically in +`StateMachine` implementations compared to using lambdas, which makes this +recommendation debatable. To avoid unnecessary debates, it is advised to leave +the decision up to the individual implementing the solution. -There are also error handling overloads of `Tasks.lookUp`, that are analogous to `SkyFunction.Environment.getValueOrThrow`. +There are also error handling overloads of `Tasks.lookUp`, that are analogous to +`SkyFunction.Environment.getValueOrThrow`. ``` - <E extends Exception> void lookUp( - SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); + void lookUp( + SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); - interface ValueOrExceptionSink<E extends Exception> { + interface ValueOrExceptionSink { void acceptValueOrException(@Nullable SkyValue value, @Nullable E exception); } ``` @@ -458,13 +592,13 @@ There are also error handling overloads of `Tasks.lookUp`, that are analogous to An example implementation is shown below. ``` -class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyException> { +class PerformLookupWithError extends StateMachine, ValueOrExceptionSink { private MyValue value; private MyException error; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink<MyException>) this); + tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink) this); return this::processResult; } @@ -493,29 +627,33 @@ class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyExc } ``` -As with lookups without error handling, having the `StateMachine` class directly implement the callback saves a memory allocation for the lamba. +As with lookups without error handling, having the `StateMachine` class directly +implement the callback saves a memory allocation for the lamba. -[Error handling](#error-handling) provides a bit more detail, but essentially, there's not much difference between the propagation of errors and normal values. +[Error handling](#error-handling) provides a bit more detail, but essentially, +there's not much difference between the propagation of errors and normal values. #### Consuming multiple SkyValues -Multiple SkyValue lookups are often required. An approach that works much of the time is to switch on the type of SkyValue. The following is an example that has been simplified from prototype production code. +Multiple SkyValue lookups are often required. An approach that works much of the +time is to switch on the type of SkyValue. The following is an example that has +been simplified from prototype production code. ``` @Nullable private StateMachine fetchConfigurationAndPackage(Tasks tasks) { var configurationKey = configuredTarget.getConfigurationKey(); if (configurationKey != null) { - tasks.lookUp(configurationKey, (Consumer<SkyValue>) this); + tasks.lookUp(configurationKey, (Consumer) this); } var packageId = configuredTarget.getLabel().getPackageIdentifier(); - tasks.lookUp(PackageValue.key(packageId), (Consumer<SkyValue>) this); + tasks.lookUp(PackageValue.key(packageId), (Consumer) this); return this::constructResult; } - @Override // Implementation of `Consumer<SkyValue>`. + @Override // Implementation of `Consumer`. public void accept(SkyValue value) { if (value instanceof BuildConfigurationValue) { this.configurationValue = (BuildConfigurationValue) value; @@ -529,11 +667,18 @@ Multiple SkyValue lookups are often required. An approach that works much of the } ``` -The `Consumer<SkyValue>` callback implementation can be shared unambiguously because the value types are different. When that’s not the case, falling back to lambda-based implementations or full inner-class instances that implement the appropriate callbacks is viable. +The `Consumer` callback implementation can be shared unambiguously +because the value types are different. When that’s not the case, falling back to +lambda-based implementations or full inner-class instances that implement the +appropriate callbacks is viable. ### Propagating values between `StateMachine`s -So far, this document has only explained how to arrange work in a subtask, but subtasks also need to report a values back to the caller. Since subtasks are logically asynchronous, their results are communicated back to the caller using a *callback*. To make this work, the subtask defines a sink interface that is injected via its constructor. +So far, this document has only explained how to arrange work in a subtask, but +subtasks also need to report a values back to the caller. Since subtasks are +logically asynchronous, their results are communicated back to the caller using +a *callback*. To make this work, the subtask defines a sink interface that is +injected via its constructor. ``` class BarProducer implements StateMachine { @@ -564,9 +709,16 @@ class BarProducer implements StateMachine { } ``` -Tip: It would be tempting to use the more concise signature void `accept(Bar value)` rather than the stuttery `void acceptBarValue(Bar value)` above. However, `Consumer<SkyValue>` is a common overload of `void accept(Bar value)`, so doing this often leads to violations of the [Overloads: never split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) style-guide rule. +Tip: It would be tempting to use the more concise signature void `accept(Bar +value)` rather than the stuttery `void acceptBarValue(Bar value)` above. +However, `Consumer` is a common overload of `void accept(Bar value)`, +so doing this often leads to violations of the [Overloads: never +split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) +style-guide rule. -Tip: Using a custom `ResultSink` type instead of a generic one from `java.util.function` makes it easy to find implementations in the code base, improving readability. +Tip: Using a custom `ResultSink` type instead of a generic one from +`java.util.function` makes it easy to find implementations in the code base, +improving readability. A caller `StateMachine` would then look like the following. @@ -615,37 +767,70 @@ class Caller implements StateMachine, BarProducer.ResultSink { } ``` -The preceding example demonstrates a few things. `Caller` has to propagate its results back and defines its own `Caller.ResultSink`. `Caller` implements the `BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if `value` is null to determine if an error occurred. This is a common behavior pattern after accepting output from either a subtask or SkyValue lookup. +The preceding example demonstrates a few things. `Caller` has to propagate its +results back and defines its own `Caller.ResultSink`. `Caller` implements the +`BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if +`value` is null to determine if an error occurred. This is a common behavior +pattern after accepting output from either a subtask or SkyValue lookup. -Note that the implementation of `acceptBarError` eagerly forwards the result to the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). +Note that the implementation of `acceptBarError` eagerly forwards the result to +the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). -Alternatives for top-level `StateMachine`s are described in [`Driver`s and bridging to SkyFunctions](#drivers-and-bridging). +Alternatives for top-level `StateMachine`s are described in [`Driver`s and +bridging to SkyFunctions](#drivers-and-bridging). ### Error handling -There's a couple of examples of error handling already in [`Tasks.lookUp` callbacks](#tasks-lookup-callbacks) and [Propagating values between `StateMachines`](#propagating-values). Exceptions, other than `InterruptedException` are not thrown, but instead passed around through callbacks as values. Such callbacks often have exclusive-or semantics, with exactly one of a value or error being passed. +There's a couple of examples of error handling already in [`Tasks.lookUp` +callbacks](#tasks-lookup-callbacks) and [Propagating values between +`StateMachines`](#propagating-values). Exceptions, other than +`InterruptedException` are not thrown, but instead passed around through +callbacks as values. Such callbacks often have exclusive-or semantics, with +exactly one of a value or error being passed. -The next section describes a a subtle, but important interaction with Skyframe error handling. +The next section describes a a subtle, but important interaction with Skyframe +error handling. #### Error bubbling (--nokeep\_going) -Warning: Errors need to be eagerly propagated all the way back to the SkyFunction for error bubbling to function correctly. +Warning: Errors need to be eagerly propagated all the way back to the +SkyFunction for error bubbling to function correctly. -During error bubbling, a SkyFunction may be restarted even if not all requested SkyValues are available. In such cases, the subsequent state will never be reached due to the `Tasks` API contract. However, the `StateMachine` should still propagate the exception. +During error bubbling, a SkyFunction may be restarted even if not all requested +SkyValues are available. In such cases, the subsequent state will never be +reached due to the `Tasks` API contract. However, the `StateMachine` should +still propagate the exception. -Since propagation must occur regardless of whether the next state is reached, the error handling callback must perform this task. For an inner `StateMachine`, this is achieved by invoking the parent callback. +Since propagation must occur regardless of whether the next state is reached, +the error handling callback must perform this task. For an inner `StateMachine`, +this is achieved by invoking the parent callback. -At the top-level `StateMachine`, which interfaces with the SkyFunction, this can be done by calling the `setException` method of `ValueOrExceptionProducer`. `ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even if there are missing SkyValues. +At the top-level `StateMachine`, which interfaces with the SkyFunction, this can +be done by calling the `setException` method of `ValueOrExceptionProducer`. +`ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even +if there are missing SkyValues. -If a `Driver` is being utilized directly, it is essential to check for propagated errors from the SkyFunction, even if the machine has not finished processing. +If a `Driver` is being utilized directly, it is essential to check for +propagated errors from the SkyFunction, even if the machine has not finished +processing. ### Event Handling -For SkyFunctions that need to emit events, a `StoredEventHandler` is injected into SkyKeyComputeState and further injected into `StateMachine`s that require them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping certain events unless they are replayed but this was subsequently fixed. `StoredEventHandler` injection is preserved because it simplifies the implementation of events emitted from error handling callbacks. +For SkyFunctions that need to emit events, a `StoredEventHandler` is injected +into SkyKeyComputeState and further injected into `StateMachine`s that require +them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping +certain events unless they are replayed but this was subsequently fixed. +`StoredEventHandler` injection is preserved because it simplifies the +implementation of events emitted from error handling callbacks. ## `Driver`s and bridging to SkyFunctions -A `Driver` is responsible for managing the execution of `StateMachine`s, beginning with a specified root `StateMachine`. As `StateMachine`s can recursively enqueue subtask `StateMachine`s, a single `Driver` can manage numerous subtasks. These subtasks create a tree structure, a result of [Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue lookups across subtasks for improved efficiency. +A `Driver` is responsible for managing the execution of `StateMachine`s, +beginning with a specified root `StateMachine`. As `StateMachine`s can +recursively enqueue subtask `StateMachine`s, a single `Driver` can manage +numerous subtasks. These subtasks create a tree structure, a result of +[Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue +lookups across subtasks for improved efficiency. There are a number of classes built around the `Driver`, with the following API. @@ -656,15 +841,24 @@ public final class Driver { } ``` -`Driver` takes a single root `StateMachine` as a parameter. Calling `Driver.drive` executes the `StateMachine` as far as it can go without a Skyframe restart. It returns true when the `StateMachine` completes and false otherwise, indicating that not all values were available. +`Driver` takes a single root `StateMachine` as a parameter. Calling +`Driver.drive` executes the `StateMachine` as far as it can go without a +Skyframe restart. It returns true when the `StateMachine` completes and false +otherwise, indicating that not all values were available. -`Driver` maintains the concurrent state of the `StateMachine` and it is well suited for embedding in `SkyKeyComputeState`. +`Driver` maintains the concurrent state of the `StateMachine` and it is well +suited for embedding in `SkyKeyComputeState`. ### Directly instantiating `Driver` -`StateMachine` implementations conventionally communicate their results via callbacks. It's possible to directly instantiate a `Driver` as shown in the following example. +`StateMachine` implementations conventionally communicate their results via +callbacks. It's possible to directly instantiate a `Driver` as shown in the +following example. -The `Driver` is embedded in the `SkyKeyComputeState` implementation along with an implementation of the corresponding `ResultSink` to be defined a bit further down. At the top level, the `State` object is an appropriate receiver for the result of the computation as it is guaranteed to outlive `Driver`. +The `Driver` is embedded in the `SkyKeyComputeState` implementation along with +an implementation of the corresponding `ResultSink` to be defined a bit further +down. At the top level, the `State` object is an appropriate receiver for the +result of the computation as it is guaranteed to outlive `Driver`. ``` class State implements SkyKeyComputeState, ResultProducer.ResultSink { @@ -746,7 +940,8 @@ private Result computeResult(State state, Skyfunction.Environment env) ### Embedding `Driver` -If the `StateMachine` produces a value and raises no exceptions, embedding `Driver` is another possible implementation, as shown in the following example. +If the `StateMachine` produces a value and raises no exceptions, embedding +`Driver` is another possible implementation, as shown in the following example. ``` class ResultProducer implements StateMachine { @@ -775,7 +970,8 @@ class ResultProducer implements StateMachine { } ``` -The SkyFunction may have code that looks like the following (where `State` is the function specific type of `SkyKeyComputeState`). +The SkyFunction may have code that looks like the following (where `State` is +the function specific type of `SkyKeyComputeState`). ``` @Nullable // Null when a Skyframe restart is needed. @@ -796,16 +992,19 @@ Result computeResult(SkyFunction.Environment env, State state) } ``` -Embedding `Driver` in the `StateMachine` implementation is a better fit for Skyframe's synchronous coding style. +Embedding `Driver` in the `StateMachine` implementation is a better fit for +Skyframe's synchronous coding style. ### StateMachines that may produce exceptions -Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` and `ValueOrException2Producer` classes that have synchronous APIs to match synchronous SkyFunction code. +Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` +and `ValueOrException2Producer` classes that have synchronous APIs to match +synchronous SkyFunction code. The `ValueOrExceptionProducer` abstract class includes the following methods. ``` -public abstract class ValueOrExceptionProducer<V, E extends Exception> +public abstract class ValueOrExceptionProducer implements StateMachine { @Nullable public final V tryProduceValue(Environment env) @@ -818,34 +1017,65 @@ public abstract class ValueOrExceptionProducer<V, E extends Exception> } ``` -It includes an embedded `Driver` instance and closely resembles the `ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, implementations call `setValue` or `setException` when either of those occur. When both occur, the exception takes priority. The `tryProduceValue` method bridges the asynchronous callback code to synchronous code and throws an exception when one is set. +It includes an embedded `Driver` instance and closely resembles the +`ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces +with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, +implementations call `setValue` or `setException` when either of those occur. +When both occur, the exception takes priority. The `tryProduceValue` method +bridges the asynchronous callback code to synchronous code and throws an +exception when one is set. -As previously noted, during error bubbling, it's possible for an error to occur even if the machine is not yet done because not all inputs are available. To accommodate this, `tryProduceValue` throws any set exceptions, even before the machine is done. +As previously noted, during error bubbling, it's possible for an error to occur +even if the machine is not yet done because not all inputs are available. To +accommodate this, `tryProduceValue` throws any set exceptions, even before the +machine is done. ## Epilogue: Eventually removing callbacks -`StateMachine`s are a highly efficient, but boilerplate intensive way to perform asynchronous computation. Continuations (particularly in the form of `Runnable`s passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and there are no efficient asynchronous APIs for disk I/O. Eventually, it would be good to optimize away callbacks as they have a learning curve and impede readability. - -One of the most promising alternatives is *Java* virtual threads. Instead of having to write callbacks, everything is replaced with synchronous, blocking calls. This is possible because tying up a virtual thread resource, unlike a platform thread, is supposed to be cheap. However, even with virtual threads, replacing simple synchronous operations with thread creation and synchronization primitives is too expensive. We performed a migration from `StateMachine`s to *Java* virtual threads and they were orders of magnitude slower, leading to almost a 3x increase in end-to-end analysis latency. Since virtual threads are still a preview feature, it's possible that this migration can be performed at a later date when performance improves. - -Another approach to consider is waiting for *Loom* coroutines, if they ever become available. The advantage here is that it might be possible to reduce synchronization overhead by using cooperative multitasking. - -If all else fails, low-level bytecode rewriting could also be a viable alternative. With enough optimization, it might be possible to achieve performance that approaches hand-written callback code. +`StateMachine`s are a highly efficient, but boilerplate intensive way to perform +asynchronous computation. Continuations (particularly in the form of `Runnable`s +passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, +but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and +there are no efficient asynchronous APIs for disk I/O. Eventually, it would be +good to optimize away callbacks as they have a learning curve and impede +readability. + +One of the most promising alternatives is *Java* virtual threads. Instead of +having to write callbacks, everything is replaced with synchronous, blocking +calls. This is possible because tying up a virtual thread resource, unlike a +platform thread, is supposed to be cheap. However, even with virtual threads, +replacing simple synchronous operations with thread creation and synchronization +primitives is too expensive. We performed a migration from `StateMachine`s to +*Java* virtual threads and they were orders of magnitude slower, leading to +almost a 3x increase in end-to-end analysis latency. Since virtual threads are +still a preview feature, it's possible that this migration can be performed at a +later date when performance improves. + +Another approach to consider is waiting for *Loom* coroutines, if they ever +become available. The advantage here is that it might be possible to reduce +synchronization overhead by using cooperative multitasking. + +If all else fails, low-level bytecode rewriting could also be a viable +alternative. With enough optimization, it might be possible to achieve +performance that approaches hand-written callback code. ## Appendix ### Callback Hell -Callback hell is an infamous problem in asynchronous code that uses callbacks. It stems from the fact that the continuation for a subsequent step is nested within the previous step. If there are many steps, this nesting can be extremely deep. If coupled with control flow the code becomes unmanageable. +Callback hell is an infamous problem in asynchronous code that uses callbacks. +It stems from the fact that the continuation for a subsequent step is nested +within the previous step. If there are many steps, this nesting can be extremely +deep. If coupled with control flow the code becomes unmanageable. ``` class CallbackHell implements StateMachine { @Override public StateMachine step(Tasks task) { doA(); - return (t, l) -> { + return (t, l) -> { doB(); - return (t1, l2) -> { + return (t1, l2) -> { doC(); return DONE; }; @@ -854,7 +1084,11 @@ class CallbackHell implements StateMachine { } ``` -One of the advantages of nested implementations is that the stack frame of the outer step can be preserved. In *Java*, captured lambda variables must be effectively final so using such variables can be cumbersome. Deep nesting is avoided by returning method references as continuations instead of lambdas as shown as follows. +One of the advantages of nested implementations is that the stack frame of the +outer step can be preserved. In *Java*, captured lambda variables must be +effectively final so using such variables can be cumbersome. Deep nesting is +avoided by returning method references as continuations instead of lambdas as +shown as follows. ``` class CallbackHellAvoided implements StateMachine { @@ -876,18 +1110,23 @@ class CallbackHellAvoided implements StateMachine { } ``` -Callback hell may also occur if the [`runAfter` injection](#runafter-injection) pattern is used too densely, but this can be avoided by interspersing injections with sequential steps. +Callback hell may also occur if the [`runAfter` injection](#runafter-injection) +pattern is used too densely, but this can be avoided by interspersing injections +with sequential steps. #### Example: Chained SkyValue lookups -It is often the case that the application logic requires dependent chains of SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. Thinking about this naively, this would result in a complex, deeply nested callback structure. +It is often the case that the application logic requires dependent chains of +SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. +Thinking about this naively, this would result in a complex, deeply nested +callback structure. ``` private ValueType1 value1; private ValueType2 value2; private StateMachine step1(...) { - tasks.lookUp(key1, (Consumer<SkyValue>) this); // key1 has type KeyType1. + tasks.lookUp(key1, (Consumer) this); // key1 has type KeyType1. return this::step2; } @@ -907,48 +1146,91 @@ private void acceptValueType2(SkyValue value) { } ``` -However, since continuations are specified as method references, the code looks procedural across state transitions: `step2` follows `step1`. Note that here, a lambda is used to assign `value2`. This makes the ordering of the code match the ordering of the computation from top-to-bottom. +However, since continuations are specified as method references, the code looks +procedural across state transitions: `step2` follows `step1`. Note that here, a +lambda is used to assign `value2`. This makes the ordering of the code match the +ordering of the computation from top-to-bottom. ### Miscellaneous Tips #### Readability: Execution Ordering -To improve readability, strive to keep the `StateMachine.step` implementations in execution order and callback implementations immediately following where they are passed in the code. This isn't always possible where the control flow branches. Additional comments might be helpful in such cases. +To improve readability, strive to keep the `StateMachine.step` implementations +in execution order and callback implementations immediately following where they +are passed in the code. This isn't always possible where the control flow +branches. Additional comments might be helpful in such cases. -In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an intermediate method reference is created to achieve this. This trades a small amount of performance for readability, which is likely worthwhile here. +In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an +intermediate method reference is created to achieve this. This trades a small +amount of performance for readability, which is likely worthwhile here. #### Generational Hypothesis -Medium-lived *Java* objects break the generational hypothesis of the *Java* garbage collector, which is designed to handle objects that live for a very short time or objects that live forever. By definition, objects in `SkyKeyComputeState` violate this hypothesis. Such objects, containing the constructed tree of all still-running `StateMachine`s, rooted at `Driver` have an intermediate lifespan as they suspend, waiting for asynchronous computations to complete. - -It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes possible to observe an increase in GC time, even with dramatic decreases in actual garbage generated. Since `StateMachine`s have an intermediate lifespan they could be promoted to old gen, causing it to fill up more quickly, thus necessitating more expensive major or full GCs to clean up. - -The initial precaution is to minimize the use of `StateMachine` variables, but it is not always feasible, for example, if a value is needed across multiple states. Where it is possible, local stack `step` variables are young generation variables and efficiently GC'd. - -For `StateMachine` variables, breaking things down into subtasks and following the recommended pattern for [Propagating values between `StateMachine`s](#propagating-values) is also helpful. Observe that when following the pattern, only child `StateMachine`s have references to parent `StateMachine`s and not vice versa. This means that as children complete and update the parents using result callbacks, the children naturally fall out of scope and become eligible for GC. - -Finally, in some cases, a `StateMachine` variable is needed in earlier states but not in later states. It can be beneficial to null out references of large objects once it is known that they are no longer needed. +Medium-lived *Java* objects break the generational hypothesis of the *Java* +garbage collector, which is designed to handle objects that live for a very +short time or objects that live forever. By definition, objects in +`SkyKeyComputeState` violate this hypothesis. Such objects, containing the +constructed tree of all still-running `StateMachine`s, rooted at `Driver` have +an intermediate lifespan as they suspend, waiting for asynchronous computations +to complete. + +It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes +possible to observe an increase in GC time, even with dramatic decreases in +actual garbage generated. Since `StateMachine`s have an intermediate lifespan +they could be promoted to old gen, causing it to fill up more quickly, thus +necessitating more expensive major or full GCs to clean up. + +The initial precaution is to minimize the use of `StateMachine` variables, but +it is not always feasible, for example, if a value is needed across multiple +states. Where it is possible, local stack `step` variables are young generation +variables and efficiently GC'd. + +For `StateMachine` variables, breaking things down into subtasks and following +the recommended pattern for [Propagating values between +`StateMachine`s](#propagating-values) is also helpful. Observe that when +following the pattern, only child `StateMachine`s have references to parent +`StateMachine`s and not vice versa. This means that as children complete and +update the parents using result callbacks, the children naturally fall out of +scope and become eligible for GC. + +Finally, in some cases, a `StateMachine` variable is needed in earlier states +but not in later states. It can be beneficial to null out references of large +objects once it is known that they are no longer needed. #### Naming states -When naming a method, it's usually possible to name a method for the behavior that happens within that method. It's less clear how to do this in `StateMachine`s because there is no stack. For example, suppose method `foo` calls a sub-method `bar`. In a `StateMachine`, this could be translated into the state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior `bar`. As a result, method names for states tend to be narrower in scope, potentially reflecting local behavior. +When naming a method, it's usually possible to name a method for the behavior +that happens within that method. It's less clear how to do this in +`StateMachine`s because there is no stack. For example, suppose method `foo` +calls a sub-method `bar`. In a `StateMachine`, this could be translated into the +state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior +`bar`. As a result, method names for states tend to be narrower in scope, +potentially reflecting local behavior. ### Concurrency tree diagram -The following is an alternative view of the diagram in [Structured concurrency](#structured-concurrency) that better depicts the tree structure. The blocks form a small tree. +The following is an alternative view of the diagram in [Structured +concurrency](#structured-concurrency) that better depicts the tree structure. +The blocks form a small tree. ![Structured Concurrency 3D](/contribute/images/structured-concurrency-3d.svg) -## Footnotes - -1. In contrast to Skyframe's convention of restarting from the beginning when values are not available. [↩](#user-content-fnref-1) - -2. Note that `step` is permitted to throw `InterruptedException`, but the examples omit this. There are a few low methods in *Bazel* code that throw this exception and it propagates up to the `Driver`, to be described later, that runs the `StateMachine`. It's fine to not declare it to be thrown when unneeded. [↩](#user-content-fnref-2) - -3. Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which performs *independent* work for each dependency. Instead of manipulating complex data structures that process all the dependencies at once, introducing inefficiencies, each dependency has its own independent `StateMachine`. [↩](#user-content-fnref-3) - -4. Multiple `tasks.lookUp` calls within a single step are batched together. Additional batching can be created by lookups occurring within concurrent subtasks. [↩](#user-content-fnref-4) - -5. This is conceptually similar to Java’s structured concurrency [jeps/428](https://openjdk.org/jeps/428). [↩](#user-content-fnref-5) - -6. Doing this is similar to spawning a thread and joining it to achieve sequential composition. [↩](#user-content-fnref-6) +[^1]: In contrast to Skyframe's convention of restarting from the beginning when + values are not available. +[^2]: Note that `step` is permitted to throw `InterruptedException`, but the + examples omit this. There are a few low methods in *Bazel* code that throw + this exception and it propagates up to the `Driver`, to be described later, + that runs the `StateMachine`. It's fine to not declare it to be thrown when + unneeded. +[^3]: Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which + performs *independent* work for each dependency. Instead of manipulating + complex data structures that process all the dependencies at once, + introducing inefficiencies, each dependency has its own independent + `StateMachine`. +[^4]: Multiple `tasks.lookUp` calls within a single step are batched together. + Additional batching can be created by lookups occurring within concurrent + subtasks. +[^5]: This is conceptually similar to Java’s structured concurrency + [jeps/428](https://openjdk.org/jeps/428). +[^6]: Doing this is similar to spawning a thread and joining it to achieve + sequential composition. diff --git a/contribute/windows-chocolatey-maintenance.mdx b/contribute/windows-chocolatey-maintenance.mdx index a570547d..c6aee8fb 100644 --- a/contribute/windows-chocolatey-maintenance.mdx +++ b/contribute/windows-chocolatey-maintenance.mdx @@ -2,16 +2,22 @@ title: 'Maintaining Bazel Chocolatey package on Windows' --- -Note: The Chocolatey package is experimental; please provide feedback (`@petemounce` in issue tracker). + + +Note: The Chocolatey package is experimental; please provide feedback +(`@petemounce` in issue tracker). ## Prerequisites You need: -- [chocolatey package manager](https://chocolatey.org) installed -- (to publish) a chocolatey API key granting you permission to publish the `bazel` package - - [@petemounce](https://github.com/petemounce) currently maintains this unofficial package. -- (to publish) to have set up that API key for the chocolatey source locally via `choco apikey -k <your key here> -s https://chocolatey.org/` +* [chocolatey package manager](https://chocolatey.org) installed +* (to publish) a chocolatey API key granting you permission to publish the + `bazel` package + * [@petemounce](https://github.com/petemounce) currently + maintains this unofficial package. +* (to publish) to have set up that API key for the chocolatey source locally + via `choco apikey -k -s https://chocolatey.org/` ## Build @@ -23,38 +29,44 @@ pushd scripts/packages/chocolatey popd ``` -Should result in `scripts/packages/chocolatey/bazel.<version>.nupkg` being created. +Should result in `scripts/packages/chocolatey/bazel..nupkg` being +created. The `build.ps1` script supports `mode` values `local`, `rc` and `release`. ## Test -1. Build the package (with `-mode local`) +0. Build the package (with `-mode local`) - - run a webserver (`python -m SimpleHTTPServer` in `scripts/packages/chocolatey` is convenient and starts one on `http://localhost:8000`) + * run a webserver (`python -m SimpleHTTPServer` in + `scripts/packages/chocolatey` is convenient and starts one on + `http://localhost:8000`) -2. Test the install +0. Test the install - The `test.ps1` should install the package cleanly (and error if it did not install cleanly), then tell you what to do next. + The `test.ps1` should install the package cleanly (and error if it did not + install cleanly), then tell you what to do next. -3. Test the uninstall +0. Test the uninstall - ```sh - choco uninstall bazel - # should remove bazel from the system - ``` + ```sh + choco uninstall bazel + # should remove bazel from the system + ``` Chocolatey's moderation process automates checks here as well. ## Release -Modify `tools/parameters.json` for the new release's URI and checksum once the release has been published to github releases. +Modify `tools/parameters.json` for the new release's URI and checksum once the +release has been published to github releases. ```powershell -./build.ps1 -version <version> -isRelease -./test.ps1 -version <version> +./build.ps1 -version -isRelease +./test.ps1 -version # if the test.ps1 passes choco push bazel.x.y.z.nupkg --source https://chocolatey.org/ ``` -Chocolatey.org will then run automated checks and respond to the push via email to the maintainers. +Chocolatey.org will then run automated checks and respond to the push via email +to the maintainers. diff --git a/contribute/windows-scoop-maintenance.mdx b/contribute/windows-scoop-maintenance.mdx index dc4aadd9..58e2a6c4 100644 --- a/contribute/windows-scoop-maintenance.mdx +++ b/contribute/windows-scoop-maintenance.mdx @@ -2,26 +2,38 @@ title: 'Maintaining Bazel Scoop package on Windows' --- -Note: The Scoop package is experimental. To provide feedback, go to `@excitoon` in issue tracker. + + +Note: The Scoop package is experimental. To provide feedback, go to +`@excitoon` in issue tracker. ## Prerequisites You need: -- [Scoop package manager](https://scoop.sh/) installed -- GitHub account in order to publish and create pull requests to [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) - - [@excitoon](https://github.com/excitoon) currently maintains this unofficial package. Feel free to ask questions by [e-mail](mailto:vladimir.chebotarev@gmail.com) or [Telegram](http://telegram.me/excitoon). +* [Scoop package manager](https://scoop.sh/) installed +* GitHub account in order to publish and create pull requests to + [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) + * [@excitoon](https://github.com/excitoon) currently maintains this + unofficial package. Feel free to ask questions by + [e-mail](mailto:vladimir.chebotarev@gmail.com) or + [Telegram](http://telegram.me/excitoon). ## Release process -Scoop packages are very easy to maintain. Once you have the URL of released Bazel, you need to make appropriate changes in [this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): +Scoop packages are very easy to maintain. Once you have the URL of released +Bazel, you need to make appropriate changes in +[this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): - update version - update dependencies if needed - update URL - update hash (`sha256` by default) -In your filesystem, `bazel.json` is located in the directory `%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to your clone of a Git repository [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). +In your filesystem, `bazel.json` is located in the directory +`%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to +your clone of a Git repository +[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). Test the result: @@ -32,7 +44,9 @@ bazel version bazel something_else ``` -The first time, make a fork of [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and specify it as your own remote for `%UserProfile%/scoop/buckets/main`: +The first time, make a fork of +[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and +specify it as your own remote for `%UserProfile%/scoop/buckets/main`: ``` git remote add mine FORK_URL diff --git a/docs/android-build-performance.mdx b/docs/android-build-performance.mdx index a96a199d..0d5edc77 100644 --- a/docs/android-build-performance.mdx +++ b/docs/android-build-performance.mdx @@ -2,39 +2,53 @@ title: 'Android Build Performance' --- -This page contains information on optimizing build performance for Android apps specifically. For general build performance optimization with Bazel, see [Optimizing Performance](/rules/performance). + + +This page contains information on optimizing build performance for Android +apps specifically. For general build performance optimization with Bazel, see +[Optimizing Performance](/rules/performance). ## Recommended flags -The flags are in the [`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so they can be pasted directly into a `bazelrc` file and invoked with `--config=<configuration_name>` on the command line. +The flags are in the +[`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so +they can be pasted directly into a `bazelrc` file and invoked with +`--config=` on the command line. **Profiling performance** -Bazel writes a JSON trace profile by default to a file called `command.profile.gz` in Bazel's output base. See the [JSON Profile documentation](/rules/performance#performance-profiling) for how to read and interact with the profile. +Bazel writes a JSON trace profile by default to a file called +`command.profile.gz` in Bazel's output base. +See the [JSON Profile documentation](/rules/performance#performance-profiling) for +how to read and interact with the profile. **Persistent workers for Android build actions**. -A subset of Android build actions has support for [persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). +A subset of Android build actions has support for +[persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). These actions' mnemonics are: -- DexBuilder -- Javac -- Desugar -- AaptPackage -- AndroidResourceParser -- AndroidResourceValidator -- AndroidResourceCompiler -- RClassGenerator -- AndroidResourceLink -- AndroidAapt2 -- AndroidAssetMerger -- AndroidResourceMerger -- AndroidCompiledResourceMerger - -Enabling workers can result in better build performance by saving on JVM startup costs from invoking each of these tools, but at the cost of increased memory usage on the system by persisting them. - -To enable workers for these actions, apply these flags with `--config=android_workers` on the command line: +* DexBuilder +* Javac +* Desugar +* AaptPackage +* AndroidResourceParser +* AndroidResourceValidator +* AndroidResourceCompiler +* RClassGenerator +* AndroidResourceLink +* AndroidAapt2 +* AndroidAssetMerger +* AndroidResourceMerger +* AndroidCompiledResourceMerger + +Enabling workers can result in better build performance by saving on JVM +startup costs from invoking each of these tools, but at the cost of increased +memory usage on the system by persisting them. + +To enable workers for these actions, apply these flags with +`--config=android_workers` on the command line: ``` build:android_workers --strategy=DexBuilder=worker @@ -54,7 +68,11 @@ build:android_workers --strategy=Desugar=worker build:android_workers --persistent_android_resource_processor ``` -The default number of persistent workers created per action is `4`. We have [measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) by capping the number of instances for each action to `1` or `2`, although this may vary depending on the system Bazel is running on, and the project being built. +The default number of persistent workers created per action is `4`. We have +[measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) +by capping the number of instances for each action to `1` or `2`, although this +may vary depending on the system Bazel is running on, and the project being +built. To cap the number of instances for an action, apply these flags: @@ -68,8 +86,12 @@ build:android_workers --worker_max_instances=AaptPackage=2 **Using AAPT2** -[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the `--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on `android_binary` and `android_local_test`. +[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved +performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the +`--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on +`android_binary` and `android_local_test`. **SSD optimizations** -The `--experimental_multi_threaded_digest` flag is useful for optimizing digest computation on SSDs. +The `--experimental_multi_threaded_digest` flag is useful for optimizing digest +computation on SSDs. diff --git a/docs/android-instrumentation-test.mdx b/docs/android-instrumentation-test.mdx index e5b23f88..fca7b577 100644 --- a/docs/android-instrumentation-test.mdx +++ b/docs/android-instrumentation-test.mdx @@ -2,23 +2,34 @@ title: 'Android Instrumentation Tests' --- -*If you're new to Bazel, start with the [Building Android with Bazel](/start/android-app) tutorial.* + + +_If you're new to Bazel, start with the [Building Android with +Bazel](/start/android-app ) tutorial._ ![Running Android instrumentation tests in parallel](/docs/images/android_test.gif "Android instrumentation test") **Figure 1.** Running parallel Android instrumentation tests. -[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) allows developers to test their apps on Android emulators and devices. It utilizes real Android framework APIs and the Android Test Library. +[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) +allows developers to test their apps on Android emulators and devices. +It utilizes real Android framework APIs and the Android Test Library. -For hermeticity and reproducibility, Bazel creates and launches Android emulators in a sandbox, ensuring that tests always run from a clean state. Each test gets an isolated emulator instance, allowing tests to run in parallel without passing states between them. +For hermeticity and reproducibility, Bazel creates and launches Android +emulators in a sandbox, ensuring that tests always run from a clean state. Each +test gets an isolated emulator instance, allowing tests to run in parallel +without passing states between them. -For more information on Android instrumentation tests, check out the [Android developer documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). +For more information on Android instrumentation tests, check out the [Android +developer +documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). Please file issues in the [GitHub issue tracker](https://github.com/bazelbuild/bazel/issues). ## How it works -When you run `bazel test` on an `android_instrumentation_test` target for the first time, Bazel performs the following steps: +When you run `bazel test` on an `android_instrumentation_test` target for the +first time, Bazel performs the following steps: 1. Builds the test APK, APK under test, and their transitive dependencies 2. Creates, boots, and caches clean emulator states @@ -28,7 +39,9 @@ When you run `bazel test` on an `android_instrumentation_test` target for the fi 6. Shuts down the emulator 7. Reports the results -In subsequent test runs, Bazel boots the emulator from the clean, cached state created in step 2, so there are no leftover states from previous runs. Caching emulator state also speeds up test runs. +In subsequent test runs, Bazel boots the emulator from the clean, cached state +created in step 2, so there are no leftover states from previous runs. Caching +emulator state also speeds up test runs. ## Prerequisites @@ -41,14 +54,17 @@ Ensure your environment satisfies the following prerequisites: ```posix-terminal bazel info release ``` - This results in output similar to the following: -```none +```none {:.devsite-disable-click-to-copy} release 4.1.0 ``` -- **KVM**. Bazel requires emulators to have [hardware acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) with KVM on Linux. You can follow these [installation instructions](https://help.ubuntu.com/community/KVM/Installation) for Ubuntu. +- **KVM**. Bazel requires emulators to have [hardware + acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) + with KVM on Linux. You can follow these + [installation instructions](https://help.ubuntu.com/community/KVM/Installation) + for Ubuntu. To verify that KVM has the correct configuration, run: @@ -58,32 +74,34 @@ apt-get install cpu-checker && kvm-ok If it prints the following message, you have the correct configuration: -```none +```none {:.devsite-disable-click-to-copy} INFO: /dev/kvm exists KVM acceleration can be used ``` -- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). +- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires + the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). To install it, run: ```posix-terminal apt-get install xvfb ``` - -Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` by running: +Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` +by running: ```posix-terminal which Xvfb ``` - The output is the following: ```{:.devsite-disable-click-to-copy} /usr/bin/Xvfb ``` -- **32-bit Libraries**. Some of the binaries used by the test infrastructure are 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For Ubuntu, install these 32-bit libraries: +- **32-bit Libraries**. Some of the binaries used by the test infrastructure are + 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For + Ubuntu, install these 32-bit libraries: ```posix-terminal sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 libbz2-1.0:i386 @@ -97,6 +115,7 @@ Here is a typical target dependency graph of an `android_instrumentation_test`: **Figure 2.** Target dependency graph of an `android_instrumentation_test`. + ### BUILD file The graph translates into a `BUILD` file like this: @@ -150,45 +169,59 @@ android_library( The main attributes of the rule `android_instrumentation_test` are: -- `test_app`: An `android_binary` target. This target contains test code and dependencies like Espresso and UIAutomator. The selected `android_binary` target is required to specify an `instruments` attribute pointing to another `android_binary`, which is the app under test. +- `test_app`: An `android_binary` target. This target contains test code and + dependencies like Espresso and UIAutomator. The selected `android_binary` + target is required to specify an `instruments` attribute pointing to another + `android_binary`, which is the app under test. -- `target_device`: An `android_device` target. This target describes the specifications of the Android emulator which Bazel uses to create, launch and run the tests. See the [section on choosing an Android device](#android-device-target) for more information. +- `target_device`: An `android_device` target. This target describes the + specifications of the Android emulator which Bazel uses to create, launch and + run the tests. See the [section on choosing an Android + device](#android-device-target) for more information. -The test app's `AndroidManifest.xml` must include [an `<instrumentation>` tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). This tag must specify the attributes for the **package of the target app** and the **fully qualified class name of the instrumentation test runner**, `androidx.test.runner.AndroidJUnitRunner`. +The test app's `AndroidManifest.xml` must include [an `` +tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). +This tag must specify the attributes for the **package of the target app** and +the **fully qualified class name of the instrumentation test runner**, +`androidx.test.runner.AndroidJUnitRunner`. Here is an example `AndroidTestManifest.xml` for the test app: ```xml -<?xml version="1.0" encoding="UTF-8"?> -<manifest xmlns:android="http://schemas.android.com/apk/res/android" + + - <instrumentation + - <uses-sdk + - <application > - - </application> -</manifest> + + + + ``` ### WORKSPACE dependencies -In order to use this rule, your project needs to depend on these external repositories: +In order to use this rule, your project needs to depend on these external +repositories: - `@androidsdk`: The Android SDK. Download this through Android Studio. -- `@android_test_support`: Hosts the test runner, emulator launcher, and `android_device` targets. You can find the [latest release here](https://github.com/android/android-test/releases). +- `@android_test_support`: Hosts the test runner, emulator launcher, and + `android_device` targets. You can find the [latest release + here](https://github.com/android/android-test/releases). -Enable these dependencies by adding the following lines to your `WORKSPACE` file: +Enable these dependencies by adding the following lines to your `WORKSPACE` +file: ```python # Android SDK @@ -210,20 +243,25 @@ android_test_repositories() ## Maven dependencies -For managing dependencies on Maven artifacts from repositories, such as [Google Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), you should use a Maven resolver, such as [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). +For managing dependencies on Maven artifacts from repositories, such as [Google +Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), +you should use a Maven resolver, such as +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). -The rest of this page shows how to use `rules_jvm_external` to resolve and fetch dependencies from Maven repositories. +The rest of this page shows how to use `rules_jvm_external` to +resolve and fetch dependencies from Maven repositories. -## Choosing an android\_device target +## Choosing an android_device target -`android_instrumentation_test.target_device` specifies which Android device to run the tests on. These `android_device` targets are defined in [`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). +`android_instrumentation_test.target_device` specifies which Android device to +run the tests on. These `android_device` targets are defined in +[`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). For example, you can query for the sources for a particular target by running: ```posix-terminal bazel query --output=build @android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86 ``` - Which results in output that looks similar to: ```python @@ -249,22 +287,29 @@ android_device( The device target names use this template: ``` -@android_test_support//tools/android/emulated_devices/<var>device_type</var>:<var>system</var>_<var>api_level</var>_x86_qemu2 +@android_test_support//tools/android/emulated_devices/{{ "" }}device_type{{ "" }}:{{ "" }}system{{ "" }}_{{ "" }}api_level{{ "" }}_x86_qemu2 ``` -In order to launch an `android_device`, the `system_image` for the selected API level is required. To download the system image, use Android SDK's `tools/bin/sdkmanager`. For example, to download the system image for `generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager "system-images;android-23;default;x86"`. +In order to launch an `android_device`, the `system_image` for the selected API +level is required. To download the system image, use Android SDK's +`tools/bin/sdkmanager`. For example, to download the system image for +`generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager +"system-images;android-23;default;x86"`. -To see the full list of supported `android_device` targets in `@android_test_support`, run the following command: +To see the full list of supported `android_device` targets in +`@android_test_support`, run the following command: ```posix-terminal bazel query 'filter("x86_qemu2$", kind(android_device, @android_test_support//tools/android/emulated_devices/...:*))' ``` -Bazel currently supports x86-based emulators only. For better performance, use `QEMU2` `android_device` targets instead of `QEMU` ones. +Bazel currently supports x86-based emulators only. For better performance, use +`QEMU2` `android_device` targets instead of `QEMU` ones. ## Running tests -To run tests, add these lines to your project's `<var>project root</var>:<var>/.bazelrc` file. +To run tests, add these lines to your project's +`project root:/.bazelrc` file. ``` # Configurations for testing with Bazel @@ -294,11 +339,13 @@ Then, use one of the configurations to run tests: - `bazel test //my/test:target --config=headless` - `bazel test //my/test:target --config=local_device` -Use **only one configuration** or tests will fail. +Use __only one configuration__ or tests will fail. ### Headless testing -With `Xvfb`, it is possible to test with emulators without the graphical interface, also known as headless testing. To disable the graphical interface when running tests, pass the test argument `--enable_display=false` to Bazel: +With `Xvfb`, it is possible to test with emulators without the graphical +interface, also known as headless testing. To disable the graphical interface +when running tests, pass the test argument `--enable_display=false` to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=false @@ -306,7 +353,9 @@ bazel test //my/test:target --test_arg=--enable_display=false ### GUI testing -If the `$DISPLAY` environment variable is set, it's possible to enable the graphical interface of the emulator while the test is running. To do this, pass these test arguments to Bazel: +If the `$DISPLAY` environment variable is set, it's possible to enable the +graphical interface of the emulator while the test is running. To do this, pass +these test arguments to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY @@ -314,15 +363,26 @@ bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY ### Testing with a local emulator or device -Bazel also supports testing directly on a locally launched emulator or connected device. Pass the flags `--test_strategy=exclusive` and `--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. If there is more than one connected device, pass the flag `--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of the device/emulator listed in `adb devices`. +Bazel also supports testing directly on a locally launched emulator or connected +device. Pass the flags +`--test_strategy=exclusive` and +`--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. +If there is more than one connected device, pass the flag +`--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of +the device/emulator listed in `adb devices`. ## Sample projects -If you are looking for canonical project samples, see the [Android testing samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) for projects using Espresso and UIAutomator. +If you are looking for canonical project samples, see the [Android testing +samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) +for projects using Espresso and UIAutomator. ## Espresso setup -If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) (`androidx.test.espresso`), you can use the following snippets to set up your Bazel workspace with the list of commonly used Espresso artifacts and their dependencies: +If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) +(`androidx.test.espresso`), you can use the following snippets to set up your +Bazel workspace with the list of commonly used Espresso artifacts and their +dependencies: ``` androidx.test.espresso:espresso-core @@ -333,7 +393,8 @@ org.hamcrest:java-hamcrest junit:junit ``` -One way to organize these dependencies is to create a `//:test_deps` shared library in your `<var>project root</var>/BUILD.bazel` file: +One way to organize these dependencies is to create a `//:test_deps` shared +library in your `project root/BUILD.bazel` file: ```python java_library( @@ -350,7 +411,7 @@ java_library( ) ``` -Then, add the required dependencies in `<var>project root</var>/WORKSPACE`: +Then, add the required dependencies in `project root/WORKSPACE`: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -383,7 +444,8 @@ maven_install( ) ``` -Finally, in your test `android_binary` target, add the `//:test_deps` dependency: +Finally, in your test `android_binary` target, add the `//:test_deps` +dependency: ```python android_binary( @@ -401,14 +463,17 @@ android_binary( ### Reading test logs -Use `--test_output=errors` to print logs for failing tests, or `--test_output=all` to print all test output. If you're looking for an individual test log, go to `$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. +Use `--test_output=errors` to print logs for failing tests, or +`--test_output=all` to print all test output. If you're looking for an +individual test log, go to +`$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. -For example, the test logs for `BasicSample` canonical project are in `bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: +For example, the test logs for `BasicSample` canonical project are in +`bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: ```posix-terminal tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ``` - This results in the following output: ```none @@ -466,7 +531,9 @@ $ tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ### Reading emulator logs -The emulator logs for `android_device` targets are stored in the `/tmp/` directory with the name `emulator_xxxxx.log`, where `xxxxx` is a randomly-generated sequence of characters. +The emulator logs for `android_device` targets are stored in the `/tmp/` +directory with the name `emulator_xxxxx.log`, where `xxxxx` is a +randomly-generated sequence of characters. Use this command to find the latest emulator log: @@ -476,7 +543,8 @@ ls -1t /tmp/emulator_*.log | head -n 1 ### Testing against multiple API levels -If you would like to test against multiple API levels, you can use a list comprehension to create test targets for each API level. For example: +If you would like to test against multiple API levels, you can use a list +comprehension to create test targets for each API level. For example: ```python API_LEVELS = [ @@ -495,10 +563,14 @@ API_LEVELS = [ ## Known issues -- [Forked adb server processes are not terminated after tests](https://github.com/bazelbuild/bazel/issues/4853) -- While APK building works on all platforms (Linux, macOS, Windows), testing only works on Linux. -- Even with `--config=local_adb`, users still need to specify `android_instrumentation_test.target_device`. -- If using a local device or emulator, Bazel does not uninstall the APKs after the test. Clean the packages by running this command: +- [Forked adb server processes are not terminated after + tests](https://github.com/bazelbuild/bazel/issues/4853) +- While APK building works on all platforms (Linux, macOS, Windows), testing + only works on Linux. +- Even with `--config=local_adb`, users still need to specify + `android_instrumentation_test.target_device`. +- If using a local device or emulator, Bazel does not uninstall the APKs after + the test. Clean the packages by running this command: ```posix-terminal adb shell pm list diff --git a/docs/android-ndk.mdx b/docs/android-ndk.mdx index 292231dc..b434797d 100644 --- a/docs/android-ndk.mdx +++ b/docs/android-ndk.mdx @@ -2,13 +2,25 @@ title: 'Using the Android Native Development Kit with Bazel' --- -*If you're new to Bazel, please start with the [Building Android with Bazel](/start/android-app) tutorial.* + + +_If you're new to Bazel, please start with the [Building Android with +Bazel](/start/android-app ) tutorial._ ## Overview -Bazel can run in many different build configurations, including several that use the Android Native Development Kit (NDK) toolchain. This means that normal `cc_library` and `cc_binary` rules can be compiled for Android directly within Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository rule and its related bzlmod extension. +Bazel can run in many different build configurations, including several that use +the Android Native Development Kit (NDK) toolchain. This means that normal +`cc_library` and `cc_binary` rules can be compiled for Android directly within +Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository +rule and its related bzlmod extension. -For general Android compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). This tutorial demonstrates how to integrate C++ library dependencies into Android apps and uses [`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK toolchain discovery and registration. +For general Android +compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). +This tutorial demonstrates how to integrate C++ library dependencies into +Android apps and uses +[`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK +toolchain discovery and registration. ## Prerequisites @@ -16,7 +28,12 @@ Please ensure that you have installed the Android SDK and NDK. ### NDK and SDK setup -External repository setup varies depending on whether you are using WORKSPACE or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of each other. If you are using one dependency management solution, you don't need to add the boilerplate for the other. +External repository setup varies depending on whether you are using WORKSPACE +or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* +Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of +each other. +If you are using one dependency management solution, you don't need to add +the boilerplate for the other. #### Bzlmod MODULE.bazel setup @@ -60,18 +77,23 @@ android_ndk_repository( Compatibility notes for WORKSPACE: -- Both `rules_android` and `rules_android_ndk` rules require extra boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) file of `rules_android_ndk`'s basic example app. +* Both `rules_android` and `rules_android_ndk` rules require extra + boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date + and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) + file of `rules_android_ndk`'s basic example app. -For more information about the `android_ndk_repository` rule, see its [docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). +For more information about the `android_ndk_repository` rule, see its +[docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). ## Quick start -To build C++ for Android, simply add `cc_library` dependencies to your `android_binary` or `android_library` rules. +To build C++ for Android, simply add `cc_library` dependencies to your +`android_binary` or `android_library` rules. For example, given the following `BUILD` file for an Android app: ```python -# In <project>/app/src/main/BUILD.bazel +# In /app/src/main/BUILD.bazel load("@rules_cc//cc:cc_library.bzl", "cc_library") load("@rules_android//rules:rules.bzl", "android_binary", "android_library") @@ -100,17 +122,19 @@ This `BUILD` file results in the following target graph: ![Example results](/docs/images/android_ndk.png "Build graph results") -**Figure 1.** Build graph of Android project with cc\_library dependencies. +**Figure 1.** Build graph of Android project with cc_library dependencies. To build the app, simply run: ```posix-terminal -bazel build //app/src/main:app --android_platforms=<your platform> +bazel build //app/src/main:app --android_platforms= ``` -Note that if you don't specify `--android_platforms`, your build will fail with errors about missing JNI headers. +Note that if you don't specify `--android_platforms`, your build will fail with +errors about missing JNI headers. -The `bazel build` command compiles the Java files, Android resource files, and `cc_library` rules, and packages everything into an APK: +The `bazel build` command compiles the Java files, Android resource files, and +`cc_library` rules, and packages everything into an APK: ```posix-terminal $ zipinfo -1 bazel-bin/app/src/main/app.apk @@ -126,21 +150,27 @@ META-INF/CERT.RSA META-INF/MANIFEST.MF ``` -Bazel compiles all of the cc\_libraries into a single shared object (`.so`) file, targeted the architectures specified by `--android_platforms`. See the section on [configuring the target ABI](#configuring-target-abi) for more details. +Bazel compiles all of the cc_libraries into a single shared object (`.so`) file, +targeted the architectures specified by `--android_platforms`. +See the section on [configuring the target ABI](#configuring-target-abi) for +more details. ## Example setup -This example is available in the [Bazel examples repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). +This example is available in the [Bazel examples +repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). -In the `BUILD.bazel` file, three targets are defined with the `android_binary`, `android_library`, and `cc_library` rules. +In the `BUILD.bazel` file, three targets are defined with the `android_binary`, +`android_library`, and `cc_library` rules. The `android_binary` top-level target builds the APK. -The `cc_library` target contains a single C++ source file with a JNI function implementation: +The `cc_library` target contains a single C++ source file with a JNI function +implementation: ```c++ -#include <jni.h> -#include <string> +#include +#include extern "C" JNIEXPORT jstring @@ -150,11 +180,14 @@ Java_com_example_android_bazel_MainActivity_stringFromJNI( JNIEnv *env, jobject /* this */) { std::string hello = "Hello from C++"; - return env->NewStringUTF(hello.c_str()); + return env->NewStringUTF(hello.c_str()); } ``` -The `android_library` target specifies the Java sources, resource files, and the dependency on a `cc_library` target. For this example, `MainActivity.java` loads the shared object file `libapp.so`, and defines the method signature for the JNI function: +The `android_library` target specifies the Java sources, resource files, and the +dependency on a `cc_library` target. For this example, `MainActivity.java` loads +the shared object file `libapp.so`, and defines the method signature for the JNI +function: ```java public class MainActivity extends AppCompatActivity { @@ -173,19 +206,23 @@ public class MainActivity extends AppCompatActivity { } ``` -Note: The name of the native library is derived from the name of the top level `android_binary` target. In this example, it is `app`. +Note: The name of the native library is derived from the name of the top +level `android_binary` target. In this example, it is `app`. ## Configuring the target ABI To configure the target ABI, use the `--android_platforms` flag as follows: ```posix-terminal -bazel build //:app --android_platforms=<var>comma-separated list of platforms</var> +bazel build //:app --android_platforms={{ "" }}comma-separated list of platforms{{ "" }} ``` -Just like the `--platforms` flag, the values passed to `--android_platforms` are the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) targets, using standard constraint values to describe your device. +Just like the `--platforms` flag, the values passed to `--android_platforms` are +the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) +targets, using standard constraint values to describe your device. -For example, for an Android device with a 64-bit ARM processor, you'd define your platform like this: +For example, for an Android device with a 64-bit ARM processor, you'd define +your platform like this: ```py platform( @@ -197,36 +234,43 @@ platform( ) ``` -Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) OS constraint. To migrate the CPU constraint, check this chart: +Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) +OS constraint. To migrate the CPU constraint, check this chart: -| CPU Value | Platform | -| ------------- | ------------------------ | -| `armeabi-v7a` | `@platforms//cpu:armv7` | -| `arm64-v8a` | `@platforms//cpu:arm64` | -| `x86` | `@platforms//cpu:x86_32` | -| `x86_64` | `@platforms//cpu:x86_64` | +CPU Value | Platform +------------- | ------------------------------------------ +`armeabi-v7a` | `@platforms//cpu:armv7` +`arm64-v8a` | `@platforms//cpu:arm64` +`x86` | `@platforms//cpu:x86_32` +`x86_64` | `@platforms//cpu:x86_64` -And, of course, for a multi-architecture APK, you pass multiple labels, for example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in your top-level `BUILD.bazel` file). +And, of course, for a multi-architecture APK, you pass multiple labels, for +example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in +your top-level `BUILD.bazel` file). -Bazel is unable to select a default Android platform, so one must be defined and specified with `--android_platforms`. +Bazel is unable to select a default Android platform, so one must be defined and +specified with `--android_platforms`. -Depending on the NDK revision and Android API level, the following ABIs are available: +Depending on the NDK revision and Android API level, the following ABIs are +available: | NDK revision | ABIs | -| ------------ | ----------------------------------------------------------- | +|--------------|-------------------------------------------------------------| | 16 and lower | armeabi, armeabi-v7a, arm64-v8a, mips, mips64, x86, x86\_64 | | 17 and above | armeabi-v7a, arm64-v8a, x86, x86\_64 | -See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) for more information on these ABIs. +See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) +for more information on these ABIs. -Multi-ABI Fat APKs are not recommended for release builds since they increase the size of the APK, but can be useful for development and QA builds. +Multi-ABI Fat APKs are not recommended for release builds since they increase +the size of the APK, but can be useful for development and QA builds. ## Selecting a C++ standard Use the following flags to build according to a C++ standard: | C++ Standard | Flag | -| ------------ | ----------------------- | +|--------------|-------------------------| | C++98 | Default, no flag needed | | C++11 | `--cxxopt=-std=c++11` | | C++14 | `--cxxopt=-std=c++14` | @@ -238,9 +282,11 @@ For example: bazel build //:app --cxxopt=-std=c++11 ``` -Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and `--linkopt` in the [User Manual](/docs/user-manual#cxxopt). +Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and +`--linkopt` in the [User Manual](/docs/user-manual#cxxopt). -Compiler and linker flags can also be specified as attributes in `cc_library` using `copts` and `linkopts`. For example: +Compiler and linker flags can also be specified as attributes in `cc_library` +using `copts` and `linkopts`. For example: ```python cc_library( @@ -253,9 +299,11 @@ cc_library( ## Building a `cc_library` for Android without using `android_binary` -To build a standalone `cc_binary` or `cc_library` for Android without using an `android_binary`, use the `--platforms` flag. +To build a standalone `cc_binary` or `cc_library` for Android without using an +`android_binary`, use the `--platforms` flag. -For example, assuming you have defined Android platforms in `my/platforms/BUILD`: +For example, assuming you have defined Android platforms in +`my/platforms/BUILD`: ```posix-terminal bazel build //my/cc/jni:target \ @@ -264,9 +312,13 @@ bazel build //my/cc/jni:target \ With this approach, the entire build tree is affected. -Note: All of the targets on the command line must be compatible with building for Android when specifying these flags, which may make it difficult to use [Bazel wild-cards](/run/build#specifying-build-targets) like `/...` and `:all`. +Note: All of the targets on the command line must be compatible with +building for Android when specifying these flags, which may make it difficult to +use [Bazel wild-cards](/run/build#specifying-build-targets) like +`/...` and `:all`. -These flags can be put into a `bazelrc` config (one for each ABI), in `<var>project</var>/.bazelrc`: +These flags can be put into a `bazelrc` config (one for each ABI), in +`project/.bazelrc`: ``` common:android_x86 --platforms=//my/platforms:x86 @@ -274,7 +326,7 @@ common:android_x86 --platforms=//my/platforms:x86 common:android_armeabi-v7a --platforms=//my/platforms:armeabi-v7a # In general -common:android_<abi> --platforms=//my/platforms:<abi> +common:android_ --platforms=//my/platforms: ``` Then, to build a `cc_library` for `x86` for example, run: @@ -283,4 +335,7 @@ Then, to build a `cc_library` for `x86` for example, run: bazel build //my/cc/jni:target --config=android_x86 ``` -In general, use this method for low-level targets (like `cc_library`) or when you know exactly what you're building; rely on the automatic configuration transitions from `android_binary` for high-level targets where you're expecting to build a lot of targets you don't control. +In general, use this method for low-level targets (like `cc_library`) or when +you know exactly what you're building; rely on the automatic configuration +transitions from `android_binary` for high-level targets where you're expecting +to build a lot of targets you don't control. diff --git a/docs/bazel-and-android.mdx b/docs/bazel-and-android.mdx index b043a8bc..bf3625c9 100644 --- a/docs/bazel-and-android.mdx +++ b/docs/bazel-and-android.mdx @@ -2,27 +2,44 @@ title: 'Android and Bazel' --- -This page contains resources that help you use Bazel with Android projects. It links to a tutorial, build rules, and other information specific to building Android projects with Bazel. + + +This page contains resources that help you use Bazel with Android projects. It +links to a tutorial, build rules, and other information specific to building +Android projects with Bazel. ## Getting started The following resources will help you work with Bazel on Android projects: -- [Tutorial: Building an Android app](/start/android-app). This tutorial is a good place to start learning about Bazel commands and concepts, and how to build Android apps with Bazel. -- [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). This codelab explains how to build Android apps with Bazel. +* [Tutorial: Building an Android app](/start/android-app ). This + tutorial is a good place to start learning about Bazel commands and concepts, + and how to build Android apps with Bazel. +* [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). + This codelab explains how to build Android apps with Bazel. ## Features -Bazel has Android rules for building and testing Android apps, integrating with the SDK/NDK, and creating emulator images. There are also Bazel plugins for Android Studio and IntelliJ. - -- [Android rules](/reference/be/android). The Build Encyclopedia describes the rules for building and testing Android apps with Bazel. -- [Integration with Android Studio](/install/ide). Bazel is compatible with Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) plugin. -- [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` feature provides automated build-and-deploy functionality for building and testing Android apps directly on Android devices and emulators. -- [Android instrumentation testing](/docs/android-instrumentation-test) on emulators and devices. -- [Android NDK integration](/docs/android-ndk). Bazel supports compiling to native code through direct NDK integration and the C++ rules. -- [Android build performance](/docs/android-build-performance). This page provides information on optimizing build performance for Android apps. +Bazel has Android rules for building and testing Android apps, integrating with +the SDK/NDK, and creating emulator images. There are also Bazel plugins for +Android Studio and IntelliJ. + +* [Android rules](/reference/be/android). The Build Encyclopedia describes the rules + for building and testing Android apps with Bazel. +* [Integration with Android Studio](/install/ide). Bazel is compatible with + Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) + plugin. +* [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` + feature provides automated build-and-deploy functionality for building and + testing Android apps directly on Android devices and emulators. +* [Android instrumentation testing](/docs/android-instrumentation-test) on + emulators and devices. +* [Android NDK integration](/docs/android-ndk). Bazel supports compiling to + native code through direct NDK integration and the C++ rules. +* [Android build performance](/docs/android-build-performance). This page + provides information on optimizing build performance for Android apps. ## Further reading -- Integrating with dependencies from Google Maven and Maven Central with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external). -- Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). +* Integrating with dependencies from Google Maven and Maven Central with [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external). +* Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). diff --git a/docs/bazel-and-apple.mdx b/docs/bazel-and-apple.mdx index 430c0f1f..6e4a06fe 100644 --- a/docs/bazel-and-apple.mdx +++ b/docs/bazel-and-apple.mdx @@ -2,52 +2,85 @@ title: 'Apple Apps and Bazel' --- -This page contains resources that help you use Bazel to build macOS and iOS projects. It links to a tutorial, build rules, and other information specific to using Bazel to build and test for those platforms. + + +This page contains resources that help you use Bazel to build macOS and iOS +projects. It links to a tutorial, build rules, and other information specific to +using Bazel to build and test for those platforms. ## Working with Bazel The following resources will help you work with Bazel on macOS and iOS projects: -- [Tutorial: Building an iOS app](/start/ios-app) -- [Objective-C build rules](/reference/be/objective-c) -- [General Apple rules](https://github.com/bazelbuild/rules_apple) -- [Integration with Xcode](/install/ide) +* [Tutorial: Building an iOS app](/start/ios-app) +* [Objective-C build rules](/reference/be/objective-c) +* [General Apple rules](https://github.com/bazelbuild/rules_apple) +* [Integration with Xcode](/install/ide) ## Migrating to Bazel -If you currently build your macOS and iOS projects with Xcode, follow the steps in the migration guide to start building them with Bazel: +If you currently build your macOS and iOS projects with Xcode, follow the steps +in the migration guide to start building them with Bazel: -- [Migrating from Xcode to Bazel](/migrate/xcode) +* [Migrating from Xcode to Bazel](/migrate/xcode) ## Apple apps and new rules -**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. +You do not need it when getting started with Bazel. -The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your macOS and iOS projects: +The following modules, configuration fragments, and providers will help you +[extend Bazel's capabilities](/extending/concepts) +when building your macOS and iOS projects: -- Modules: +* Modules: - - [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) - - [`apple_common`](/rules/lib/toplevel/apple_common) - - [`apple_platform`](/rules/lib/builtins/apple_platform) - - [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) - - [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) + * [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) + * [`apple_common`](/rules/lib/toplevel/apple_common) + * [`apple_platform`](/rules/lib/builtins/apple_platform) + * [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) + * [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) -- Configuration fragments: +* Configuration fragments: - - [`apple`](/rules/lib/fragments/apple) + * [`apple`](/rules/lib/fragments/apple) -- Providers: +* Providers: - - [`ObjcProvider`](/rules/lib/providers/ObjcProvider) - - [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) + * [`ObjcProvider`](/rules/lib/providers/ObjcProvider) + * [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) ## Xcode selection -If your build requires Xcode, Bazel will select an appropriate version based on the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes the set of available Xcode versions and sets a default version if `--xcode_version` is not passed. This default is overridden by the `--xcode_version` flag, as long as it is set to an Xcode version that is represented in the `--xcode_config` target. - -If you do not pass `--xcode_config`, Bazel will use the autogenerated [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the Xcode versions available on your host machine. The default version is the newest available Xcode version. This is appropriate for local execution. - -If you are performing remote builds, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `versions` attribute is a list of remotely available [`xcode_version`](/reference/be/objective-c#xcode_version) targets, and whose `default` attribute is one of these [`xcode_versions`](/reference/be/objective-c#xcode_version). - -If you are using dynamic execution, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `remote_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the remotely available Xcode versions, and whose `local_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the locally available Xcode versions. For `local_versions`, you probably want to use the autogenerated `@local_config_xcode//:host_available_xcodes`. The default Xcode version is the newest mutually available version, if there is one, otherwise the default of the `local_versions` target. If you prefer to use the `local_versions` default as the default, you can pass `--experimental_prefer_mutual_default=false`. +If your build requires Xcode, Bazel will select an appropriate version based on +the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes +the set of available Xcode versions and sets a default version if +`--xcode_version` is not passed. This default is overridden by the +`--xcode_version` flag, as long as it is set to an Xcode version that is +represented in the `--xcode_config` target. + +If you do not pass `--xcode_config`, Bazel will use the autogenerated +[`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the +Xcode versions available on your host machine. The default version is +the newest available Xcode version. This is appropriate for local execution. + +If you are performing remote builds, you should set `--xcode_config` to an +[`xcode_config`](/reference/be/objective-c#xcode_config) +target whose `versions` attribute is a list of remotely available +[`xcode_version`](/reference/be/objective-c#xcode_version) +targets, and whose `default` attribute is one of these +[`xcode_versions`](/reference/be/objective-c#xcode_version). + +If you are using dynamic execution, you should set `--xcode_config` to an +[`xcode_config`](/reference/be/objective-c#xcode_config) +target whose `remote_versions` attribute is an +[`available_xcodes`](/reference/be/workspace#available_xcodes) +target containing the remotely available Xcode versions, and whose +`local_versions` attribute is an +[`available_xcodes`](/reference/be/workspace#available_xcodes) +target containing the locally available Xcode versions. For `local_versions`, +you probably want to use the autogenerated +`@local_config_xcode//:host_available_xcodes`. The default Xcode version is the +newest mutually available version, if there is one, otherwise the default of the +`local_versions` target. If you prefer to use the `local_versions` default +as the default, you can pass `--experimental_prefer_mutual_default=false`. diff --git a/docs/bazel-and-cpp.mdx b/docs/bazel-and-cpp.mdx index 17ecb7c8..73e502cb 100644 --- a/docs/bazel-and-cpp.mdx +++ b/docs/bazel-and-cpp.mdx @@ -2,78 +2,101 @@ title: 'C++ and Bazel' --- -This page contains resources that help you use Bazel with C++ projects. It links to a tutorial, build rules, and other information specific to building C++ projects with Bazel. -## Working with Bazel - -The following resources will help you work with Bazel on C++ projects: - -- [Tutorial: Building a C++ project](/start/cpp) -- [C++ common use cases](/tutorials/cpp-use-cases) +This page contains resources that help you use Bazel with C++ projects. It links +to a tutorial, build rules, and other information specific to building C++ +projects with Bazel. -- [C/C++ rules](/reference/be/c-cpp) - -- Essential Libraries - - - [Abseil](https://abseil.io/docs/cpp/quickstart) - - [Boost](https://github.com/nelhage/rules_boost) - - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) - -- [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +## Working with Bazel -- [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) +The following resources will help you work with Bazel on C++ projects: -- [Integrating with C++ rules](/configure/integrate-cpp) +* [Tutorial: Building a C++ project](/start/cpp) +* [C++ common use cases](/tutorials/cpp-use-cases) +* [C/C++ rules](/reference/be/c-cpp) +* Essential Libraries + - [Abseil](https://abseil.io/docs/cpp/quickstart) + - [Boost](https://github.com/nelhage/rules_boost) + - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) +* [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +* [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) +* [Integrating with C++ rules](/configure/integrate-cpp) ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to C++ projects. +In addition to [general Bazel best practices](/configure/best-practices), below are +best practices specific to C++ projects. ### BUILD files Follow the guidelines below when creating your BUILD files: -- Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) rule target per compilation unit in the directory. - -- You should granularize your C++ libraries as much as possible to maximize incrementality and parallelize the build. - -- If there is a single source file in `srcs`, name the library the same as that C++ file's name. This library should contain C++ file(s), any matching header file(s), and the library's direct dependencies. For example: - - ```python - cc_library( - name = "mylib", - srcs = ["mylib.cc"], - hdrs = ["mylib.h"], - deps = [":lower-level-lib"] - ) - ``` - -- Use one `cc_test` rule target per `cc_library` target in the file. Name the target `[library-name]_test` and the source file `[library-name]_test.cc`. For example, a test target for the `mylib` library target shown above would look like this: - - ```python - cc_test( - name = "mylib_test", - srcs = ["mylib_test.cc"], - deps = [":mylib"] - ) - ``` +* Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) + rule target per compilation unit in the directory. + +* You should granularize your C++ libraries as much as + possible to maximize incrementality and parallelize the build. + +* If there is a single source file in `srcs`, name the library the same as + that C++ file's name. This library should contain C++ file(s), any matching + header file(s), and the library's direct dependencies. For example: + + ```python + cc_library( + name = "mylib", + srcs = ["mylib.cc"], + hdrs = ["mylib.h"], + deps = [":lower-level-lib"] + ) + ``` + +* Use one `cc_test` rule target per `cc_library` target in the file. Name the + target `[library-name]_test` and the source file `[library-name]_test.cc`. + For example, a test target for the `mylib` library target shown above would + look like this: + + ```python + cc_test( + name = "mylib_test", + srcs = ["mylib_test.cc"], + deps = [":mylib"] + ) + ``` ### Include paths Follow these guidelines for include paths: -- Make all include paths relative to the workspace directory. +* Make all include paths relative to the workspace directory. -- Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not angle-brackets (`#include <foo/bar/baz.h>`). +* Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not + angle-brackets (`#include <foo/bar/baz.h>`). -- Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` (parent directory). +* Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` + (parent directory). -- For legacy or `third_party` code that requires includes pointing outside the project repository, such as external repository includes requiring a prefix, use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) arguments on the `cc_library` rule target. +* For legacy or `third_party` code that requires includes pointing outside the + project repository, such as external repository includes requiring a prefix, + use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and + [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) + arguments on the `cc_library` rule target. ### Toolchain features -The following optional [features](/docs/cc-toolchain-config-reference#features) can improve the hygiene of a C++ project. They can be enabled using the `--features` command-line flag or the `features` attribute of [`repo`](/external/overview#repo.bazel), [`package`](/reference/be/functions#package) or `cc_*` rules: - -- The `parse_headers` feature makes it so that the C++ compiler is used to parse (but not compile) all header files in the built targets and their dependencies when using the [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) flag. This can help catch issues in header-only libraries and ensure that headers are self-contained and independent of the order in which they are included. -- The `layering_check` feature enforces that targets only include headers provided by their direct dependencies. The default toolchain supports this feature on Linux with `clang` as the compiler. +The following optional [features](/docs/cc-toolchain-config-reference#features) +can improve the hygiene of a C++ project. They can be enabled using the +`--features` command-line flag or the `features` attribute of +[`repo`](/external/overview#repo.bazel), +[`package`](/reference/be/functions#package) or `cc_*` rules: + +* The `parse_headers` feature makes it so that the C++ compiler is used to parse + (but not compile) all header files in the built targets and their dependencies + when using the + [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) + flag. This can help catch issues in header-only libraries and ensure that + headers are self-contained and independent of the order in which they are + included. +* The `layering_check` feature enforces that targets only include headers + provided by their direct dependencies. The default toolchain supports this + feature on Linux with `clang` as the compiler. diff --git a/docs/bazel-and-java.mdx b/docs/bazel-and-java.mdx index f8aa4d9a..bd5af010 100644 --- a/docs/bazel-and-java.mdx +++ b/docs/bazel-and-java.mdx @@ -2,120 +2,164 @@ title: 'Java and Bazel' --- -This page contains resources that help you use Bazel with Java projects. It links to a tutorial, build rules, and other information specific to building Java projects with Bazel. + + +This page contains resources that help you use Bazel with Java projects. It +links to a tutorial, build rules, and other information specific to building +Java projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on Java projects: -- [Tutorial: Building a Java Project](/start/java) -- [Java rules](/reference/be/java) +* [Tutorial: Building a Java Project](/start/java) +* [Java rules](/reference/be/java) ## Migrating to Bazel -If you currently build your Java projects with Maven, follow the steps in the migration guide to start building your Maven projects with Bazel: +If you currently build your Java projects with Maven, follow the steps in the +migration guide to start building your Maven projects with Bazel: -- [Migrating from Maven to Bazel](/migrate/maven) +* [Migrating from Maven to Bazel](/migrate/maven) ## Java versions There are two relevant versions of Java that are set with configuration flags: -- the version of the source files in the repository -- the version of the Java runtime that is used to execute the code and to test it +* the version of the source files in the repository +* the version of the Java runtime that is used to execute the code and to test + it ### Configuring the version of the source code in your repository -Without an additional configuration, Bazel assumes all Java source files in the repository are written in a single Java version. To specify the version of the sources in the repository add `build --java_language_version={ver}` to `.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners should set this flag so that Bazel and its users can reference the source code's Java version number. For more details, see [Java language version flag](/docs/user-manual#java-language-version). +Without an additional configuration, Bazel assumes all Java source files in the +repository are written in a single Java version. To specify the version of the +sources in the repository add `build --java_language_version={ver}` to +`.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners +should set this flag so that Bazel and its users can reference the source code's +Java version number. For more details, see +[Java language version flag](/docs/user-manual#java-language-version). ### Configuring the JVM used to execute and test the code Bazel uses one JDK for compilation and another JVM to execute and test the code. -By default Bazel compiles the code using a JDK it downloads and it executes and tests the code with the JVM installed on the local machine. Bazel searches for the JVM using `JAVA_HOME` or path. +By default Bazel compiles the code using a JDK it downloads and it executes and +tests the code with the JVM installed on the local machine. Bazel searches for +the JVM using `JAVA_HOME` or path. -The resulting binaries are compatible with locally installed JVM in system libraries, which means the resulting binaries depend on what is installed on the machine. +The resulting binaries are compatible with locally installed JVM in system +libraries, which means the resulting binaries depend on what is installed on the +machine. -To configure the JVM used for execution and testing use `--java_runtime_version` flag. The default value is `local_jdk`. +To configure the JVM used for execution and testing use `--java_runtime_version` +flag. The default value is `local_jdk`. ### Hermetic testing and compilation -To create a hermetic compile, you can use command line flag `--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and tested on the JVM downloaded from a remote repository. For more details, see [Java runtime version flag](/docs/user-manual#java_runtime_version). +To create a hermetic compile, you can use command line flag +`--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and +tested on the JVM downloaded from a remote repository. For more details, see +[Java runtime version flag](/docs/user-manual#java_runtime_version). ### Configuring compilation and execution of build tools in Java -There is a second pair of JDK and JVM used to build and execute tools, which are used in the build process, but are not in the build results. That JDK and JVM are controlled using `--tool_java_language_version` and `--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, respectively. +There is a second pair of JDK and JVM used to build and execute tools, which are +used in the build process, but are not in the build results. That JDK and JVM +are controlled using `--tool_java_language_version` and +`--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, +respectively. #### Compiling using locally installed JDK -Bazel by default compiles using remote JDK, because it is overriding JDK's internals. The compilation toolchains using locally installed JDK are configured, however not used. +Bazel by default compiles using remote JDK, because it is overriding JDK's +internals. The compilation toolchains using locally installed JDK are configured, +however not used. -To compile using locally installed JDK, that is use the compilation toolchains for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, however, mind that this may not work on JDK of arbitrary vendors. +To compile using locally installed JDK, that is use the compilation toolchains +for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, +however, mind that this may not work on JDK of arbitrary vendors. -For more details, see [configuring Java toolchains](#config-java-toolchains). +For more details, see +[configuring Java toolchains](#config-java-toolchains). ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to Java projects. +In addition to [general Bazel best practices](/configure/best-practices), below are +best practices specific to Java projects. ### Directory structure -Prefer Maven's standard directory layout (sources under `src/main/java`, tests under `src/test/java`). +Prefer Maven's standard directory layout (sources under `src/main/java`, tests +under `src/test/java`). ### BUILD files Follow these guidelines when creating your `BUILD` files: -- Use one `BUILD` file per directory containing Java sources, because this improves build performance. +* Use one `BUILD` file per directory containing Java sources, because this + improves build performance. -- Every `BUILD` file should contain one `java_library` rule that looks like this: +* Every `BUILD` file should contain one `java_library` rule that looks like + this: - ```python - java_library( - name = "directory-name", - srcs = glob(["*.java"]), - deps = [...], - ) - ``` + ```python + java_library( + name = "directory-name", + srcs = glob(["*.java"]), + deps = [...], + ) + ``` -- The name of the library should be the name of the directory containing the `BUILD` file. This makes the label of the library shorter, that is use `"//package"` instead of `"//package:package"`. +* The name of the library should be the name of the directory containing the + `BUILD` file. This makes the label of the library shorter, that is use + `"//package"` instead of `"//package:package"`. -- The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of all Java files in the directory. +* The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of + all Java files in the directory. -- Tests should be in a matching directory under `src/test` and depend on this library. +* Tests should be in a matching directory under `src/test` and depend on this + library. ## Creating new rules for advanced Java builds -**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. - -The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your Java projects: - -- Main Java module: [`java_common`](/rules/lib/toplevel/java_common) - -- Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) +**Note**: Creating new rules is for advanced build and test scenarios. You do +not need it when getting started with Bazel. -- Configuration fragment: [`java`](/rules/lib/fragments/java) +The following modules, configuration fragments, and providers will help you +[extend Bazel's capabilities](/extending/concepts) when building your Java +projects: -- Other modules: +* Main Java module: [`java_common`](/rules/lib/toplevel/java_common) +* Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) +* Configuration fragment: [`java`](/rules/lib/fragments/java) +* Other modules: - - [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) - - [`java_compilation_info`](/rules/lib/providers/java_compilation_info) - - [`java_output_jars`](/rules/lib/providers/java_output_jars) - - [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) - - [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) + * [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) + * [`java_compilation_info`](/rules/lib/providers/java_compilation_info) + * [`java_output_jars`](/rules/lib/providers/java_output_jars) + * [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) + * [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) ## Configuring the Java toolchains Bazel uses two types of Java toolchains: -- execution, used to execute and test Java binaries, controlled with the `--java_runtime_version` flag -- compilation, used to compile Java sources, controlled with the `--java_language_version` flag +* execution, used to execute and test Java binaries, controlled with the + `--java_runtime_version` flag +* compilation, used to compile Java sources, controlled with the + `--java_language_version` flag ### Configuring additional execution toolchains -Execution toolchain is the JVM, either local or from a repository, with some additional information about its version, operating system, and CPU architecture. +Execution toolchain is the JVM, either local or from a repository, with some +additional information about its version, operating system, and CPU +architecture. -Java execution toolchains may added using the `local_java_repository` or `remote_java_repository` repo rules in a module extension. Adding the rule makes the JVM available using a flag. When multiple definitions for the same operating system and CPU architecture are given, the first one is used. +Java execution toolchains may added using the `local_java_repository` or +`remote_java_repository` repo rules in a module extension. Adding the rule makes +the JVM available using a flag. When multiple definitions for the same operating +system and CPU architecture are given, the first one is used. Example configuration of local JVM in MODULE.bazel: @@ -155,29 +199,51 @@ register_toolchains("@openjdk_canary_linux_arm_toolchain_config_repo//:all") ### Configuring additional compilation toolchains -Compilation toolchain is composed of JDK and multiple tools that Bazel uses during the compilation and that provides additional features, such as: Error Prone, strict Java dependencies, header compilation, Android desugaring, coverage instrumentation, and genclass handling for IDEs. +Compilation toolchain is composed of JDK and multiple tools that Bazel uses +during the compilation and that provides additional features, such as: Error +Prone, strict Java dependencies, header compilation, Android desugaring, +coverage instrumentation, and genclass handling for IDEs. -JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the aforementioned features. Actual compilation is executed using the internal compiler by the JDK. The JDK used for compilation is specified by `java_runtime` attribute of the toolchain. +JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the +aforementioned features. Actual compilation is executed using the internal +compiler by the JDK. The JDK used for compilation is specified by `java_runtime` +attribute of the toolchain. -Bazel overrides some JDK internals. In case of JDK version \> 9, `java.compiler` and `jdk.compiler` modules are patched using JDK's flag `--patch_module`. In case of JDK version 8, the Java compiler is patched using `-Xbootclasspath` flag. +Bazel overrides some JDK internals. In case of JDK version > 9, +`java.compiler` and `jdk.compiler` modules are patched using JDK's flag +`--patch_module`. In case of JDK version 8, the Java compiler is patched using +`-Xbootclasspath` flag. -VanillaJavaBuilder is a second implementation of JavaBuilder, which does not modify JDK's internal compiler and does not have any of the additional features. VanillaJavaBuilder is not used by any of the built-in toolchains. +VanillaJavaBuilder is a second implementation of JavaBuilder, +which does not modify JDK's internal compiler and does not have any of the +additional features. VanillaJavaBuilder is not used by any of the built-in +toolchains. In addition to JavaBuilder, Bazel uses several other tools during compilation. -The `ijar` tool processes `jar` files to remove everything except call signatures. Resulting jars are called header jars. They are used to improve the compilation incrementality by only recompiling downstream dependents when the body of a function changes. +The `ijar` tool processes `jar` files to remove everything except call +signatures. Resulting jars are called header jars. They are used to improve the +compilation incrementality by only recompiling downstream dependents when the +body of a function changes. The `singlejar` tool packs together multiple `jar` files into a single one. -The `genclass` tool post-processes the output of a Java compilation, and produces a `jar` containing only the class files for sources that were generated by annotation processors. +The `genclass` tool post-processes the output of a Java compilation, and produces +a `jar` containing only the class files for sources that were generated by +annotation processors. -The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in LCOV format. +The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in +LCOV format. The `TestRunner` tool executes JUnit 4 tests in a controlled environment. -You can reconfigure the compilation by adding `default_java_toolchain` macro to a `BUILD` file and registering it either by adding `register_toolchains` rule to the `MODULE.bazel` file or by using [`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. +You can reconfigure the compilation by adding `default_java_toolchain` macro to +a `BUILD` file and registering it either by adding `register_toolchains` rule to +the `MODULE.bazel` file or by using +[`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. -The toolchain is only used when the `source_version` attribute matches the value specified by `--java_language_version` flag. +The toolchain is only used when the `source_version` attribute matches the +value specified by `--java_language_version` flag. Example toolchain configuration: @@ -198,26 +264,37 @@ default_java_toolchain( ) ``` -which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` or by adding `register_toolchains("//:repository_default_toolchain_definition")` to the workpace. +which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` +or by adding `register_toolchains("//:repository_default_toolchain_definition")` +to the workpace. Predefined configurations: -- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions \>= 9 -- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of arbitrary vendors. -- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt tools (`ijar`, `singlejar`) -- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are built from sources (this may be useful on operating system with different libc) +- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 +- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of + arbitrary vendors. +- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt + tools (`ijar`, `singlejar`) +- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are + built from sources (this may be useful on operating system with different + libc) #### Configuring JVM and Java compiler flags -You may configure JVM and javac flags either with flags or with `default_java_toolchain` attributes. +You may configure JVM and javac flags either with flags or with + `default_java_toolchain` attributes. -The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and `--host_javacopt`. +The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and +`--host_javacopt`. -The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, `javabuilder_jvm_opts`, and `turbine_jvm_opts`. +The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, +`javabuilder_jvm_opts`, and `turbine_jvm_opts`. #### Package specific Java compiler flags configuration -You can configure different Java compiler flags for specific source files using `package_configuration` attribute of `default_java_toolchain`. Please refer to the example below. +You can configure different Java compiler flags for specific source +files using `package_configuration` attribute of `default_java_toolchain`. +Please refer to the example below. ```python load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolchain") @@ -252,11 +329,14 @@ package_group( #### Multiple versions of Java source code in a single repository -Bazel only supports compiling a single version of Java sources in a build. build. This means that when building a Java test or an application, all dependencies are built against the same Java version. +Bazel only supports compiling a single version of Java sources in a build. +build. This means that when building a Java test or an application, all + dependencies are built against the same Java version. However, separate builds may be executed using different flags. -To make the task of using different flags easier, sets of flags for a specific version may be grouped with `.bazelrc` configs": +To make the task of using different flags easier, sets of flags for a specific +version may be grouped with `.bazelrc` configs": ```python build:java8 --java_language_version=8 @@ -265,4 +345,5 @@ build:java11 --java_language_version=11 build:java11 --java_runtime_version=remotejdk_11 ``` -These configs can be used with the `--config` flag, for example `bazel test --config=java11 //:java11_test`. +These configs can be used with the `--config` flag, for example +`bazel test --config=java11 //:java11_test`. diff --git a/docs/bazel-and-javascript.mdx b/docs/bazel-and-javascript.mdx index 4a2c176a..63d80189 100644 --- a/docs/bazel-and-javascript.mdx +++ b/docs/bazel-and-javascript.mdx @@ -2,19 +2,23 @@ title: 'JavaScript and Bazel' --- -This page contains resources that help you use Bazel with JavaScript projects. It links to build rules and other information specific to building JavaScript with Bazel. + + +This page contains resources that help you use Bazel with JavaScript projects. +It links to build rules and other information specific to building JavaScript +with Bazel. The following resources will help you work with Bazel on JavaScript projects: -- [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) -- [rules\_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs -- [rules\_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler -- [rules\_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier -- [rules\_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) -- [rules\_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) -- [rules\_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) -- [rules\_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler -- [rules\_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) -- [rules\_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) -- [rules\_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) -- [rules\_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) +* [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) +* [rules_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs +* [rules_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler +* [rules_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier +* [rules_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) +* [rules_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) +* [rules_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) +* [rules_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler +* [rules_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) +* [rules_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) +* [rules_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) +* [rules_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) diff --git a/docs/configurable-attributes.mdx b/docs/configurable-attributes.mdx index 6ec62150..958341f3 100644 --- a/docs/configurable-attributes.mdx +++ b/docs/configurable-attributes.mdx @@ -2,9 +2,15 @@ title: 'Configurable Build Attributes' --- -***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. -This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. + +**_Configurable attributes_**, commonly known as [`select()`]( +/reference/be/functions#select), is a Bazel feature that lets users toggle the values +of build rule attributes at the command line. + +This can be used, for example, for a multiplatform library that automatically +chooses the appropriate implementation for the architecture, or for a +feature-configurable binary that can be customized at build time. ## Example @@ -35,30 +41,60 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: - -| | | -| ----------------------------------------------- | ------------------ | -| Command | deps = | -| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | -| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | - -`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the +command line. Specifically, `deps` becomes: + + + + + + + + + + + + + + + + + + + + + + +
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
+ +`select()` serves as a placeholder for a value that will be chosen based on +*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) +targets. By using `select()` in a configurable attribute, the attribute +effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either: -- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. +* They all resolve to the same value. For example, when running on linux x86, this is unambiguous + `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` + is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when +nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, +`resources`, `cmd`, and most other attributes. Only a small number of attributes +are *non-configurable*, and these are clearly annotated. For example, +`config_setting`'s own +[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies +under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of +the machine running Bazel (which, thanks to cross-compilation, may be different +than the CPU the target is built for). This is known as a +[configuration transition](/reference/glossary#transition). Given @@ -102,19 +138,30 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and +`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s +build parameters, which include `--cpu=arm`. The `tools` attribute changes +`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on +`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a +[`config_setting`](/reference/be/general#config_setting) or +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of +expected command line flag settings. By encapsulating these in a target, it's +easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands +them for all builds in all projects. These are specified with +[`config_setting`](/reference/be/general#config_setting)'s +[`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -127,21 +174,31 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` +is the expected value for that flag. `:meaningful_condition_name` matches if +*every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, +[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because +it only affects how Bazel reports progress to the user. Targets can't use that +flag to construct their results. The exact set of supported flags isn't +documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with +[Starlark build settings][BuildSettings]. Unlike built-in flags, these are +defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s +[`flag_values`](/reference/be/general#config_setting.flag_values) +attribute: ```python config_setting( @@ -154,17 +211,29 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) +for a working example. -[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) +is an alternative legacy syntax for custom flags (for example +`--define foo=bar`). This can be expressed either in the +[values](/reference/be/general#config_setting.values) attribute +(`values = {"define": "foo=bar"}`) or the +[define_values](/reference/be/general#config_setting.define_values) attribute +(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards +compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The +`config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition matches. +The built-in condition `//conditions:default` matches when no other condition +matches. -Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match +and no default condition emits a `"no matching conditions"` error. This can +protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -190,11 +259,16 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s +[`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides +flexibility, it can also be burdensome to individually set each one every time +you want to build a target. + [Platforms](/extending/platforms) +let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -252,9 +326,12 @@ platform( ) ``` -The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the +`config_setting`s that contain a subset of the platform's `constraint_values`, +allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, +you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -281,9 +358,11 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to check against single values. +This saves the need for boilerplate `config_setting`s when you only need to +check against single values. -Platforms are still under development. See the [documentation](/concepts/platforms) for details. +Platforms are still under development. See the +[documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -305,12 +384,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: + - Duplicate labels can appear in different paths of the same `select`. + - Duplicate labels can *not* appear within the same path of a `select`. + - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -- Duplicate labels can appear in different paths of the same `select`. -- Duplicate labels can *not* appear within the same path of a `select`. -- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) - -`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: +`select` cannot appear inside another `select`. If you need to nest `selects` +and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -331,7 +410,8 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND +chaining](#and-chaining). ## OR chaining @@ -350,7 +430,9 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. +Most conditions evaluate to the same dep. But this syntax is hard to read and +maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple +times. One option is to predefine the value as a BUILD variable: @@ -369,13 +451,18 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary duplication. +This makes it easier to manage the dependency. But it still causes unnecessary +duplication. For more direct support, use one of the following: ### `selects.with_or` -The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: +The +[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -394,12 +481,18 @@ sh_binary( ### `selects.config_setting_group` -The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: + +The +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` + ```python config_setting( name = "config1", @@ -423,13 +516,17 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across +different attributes. -It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous +"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the +[Skylib](https://github.com/bazelbuild/bazel-skylib) macro +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -454,11 +551,13 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed +inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to fails with the error: +By default, when no condition matches, the target the `select()` is attached to +fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -468,7 +567,8 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) +attribute: ```python cc_library( @@ -491,7 +591,8 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable attributes. For example, given: +Rule implementations receive the *resolved values* of configurable +attributes. For example, given: ```python # myapp/BUILD @@ -511,7 +612,9 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert +Macros can accept `select()` clauses and pass them through to native +rules. But *they cannot directly manipulate them*. For example, there's no way +for a macro to convert ```python select({"foo": "val"}, ...) @@ -525,22 +628,32 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* +because macros are evaluated in Bazel's [loading phase](/run/build#loading), +which occurs before flag values are known. +This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. +Second, macros that just need to iterate over *all* `select` paths, while +technically feasible, lack a coherent UI. Further design is necessary to change +this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's +[loading phase](/reference/glossary#loading-phase). +This means it doesn't know what command line flags a target uses since those +flags aren't evaluated until later in the build (in the +[analysis phase](/reference/glossary#analysis-phase)). +So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has +all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` - ```python # myapp/BUILD @@ -589,9 +702,14 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for +details. -The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in +*macros*. These are different than *rules*. See the +documentation on [rules](/extending/rules) and [macros](/extending/macros) +to understand the difference. +Here's an end-to-end example: Define a rule and macro: @@ -671,7 +789,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before +Bazel reads the build's command line flags. That means there isn't enough +information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -694,7 +814,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: +Because *macros* (but not rules) by definition +[can't evaluate `select()`s](#faq-select-macro), any attempt to do so +usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -707,7 +829,8 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly vigilant with them: +Booleans are a special case that fail silently, so you should be particularly +vigilant with them: ```sh $ cat myapp/defs.bzl @@ -729,13 +852,21 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. +This happens because macros don't understand the contents of `select()`. +So what they're really evaluting is the `select()` object itself. According to +[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design +standards, all objects aside from a very small number of exceptions +automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before +Bazel knows what the build's command line parameters are. Can they at least read +the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: +Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). +What you *can* do today is prepare a straight dictionary, then feed it into a +`select()`: ```sh $ cat myapp/defs.bzl @@ -747,7 +878,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -779,7 +910,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -787,11 +918,17 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo +rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in +the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't +actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in +the `actual` attribute, to perform this type of run-time determination. This +works correctly, since `alias()` is a BUILD rule, and is evaluated with a +specific configuration. You can even have a `bind()` target point to an `alias()`, if needed. @@ -819,31 +956,37 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target +that depends on either `//:ssl` or `//external:ssl` will see the alternative +located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, +use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo <desired build flags> +$ bazel cquery //myapp:foo //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on +//myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> +$ bazel cquery 'somepath(//bar, //myapp:foo)' //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the +configuration that resolves `//myapp:foo`'s `select()`. You can inspect its +values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -862,13 +1005,18 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. +`//myapp:foo` may exist in different configurations in the same build. See the +[cquery docs](/query/cquery) for guidance on using `somepath` to get the right +one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the +same command line flags as the `bazel cquery`. The `config` command relies on +the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform +is the target platform because the semantics are unclear. For example: @@ -891,11 +1039,15 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the +`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the +`:x86_linux_platform` defined here? The author of the `BUILD` file and the user +who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with these constraints: +Instead, define a `config_setting` that matches **any** platform with +these constraints: ```py config_setting( @@ -916,11 +1068,13 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what +platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you +can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -940,4 +1094,7 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. +The Bazel team doesn't endorse doing this; it overly constrains your build and +confuses users when the expected condition does not match. + +[BuildSettings]: /extending/config#user-defined-build-settings diff --git a/docs/mobile-install.mdx b/docs/mobile-install.mdx index 1d08dcd0..8bc4a679 100644 --- a/docs/mobile-install.mdx +++ b/docs/mobile-install.mdx @@ -2,100 +2,203 @@ title: 'bazel mobile-install' --- -Fast iterative development for Android -This page describes how `bazel mobile-install` makes iterative development for Android much faster. It describes the benefits of this approach versus the drawbacks of separate build and install steps. + + +

Fast iterative development for Android

+ +This page describes how `bazel mobile-install` makes iterative development +for Android much faster. It describes the benefits of this approach versus the +drawbacks of separate build and install steps. ## Summary To install small changes to an Android app very quickly, do the following: -1. Find the `android_binary` rule of the app you want to install. -2. Connect your device to `adb`. -3. Run `bazel mobile-install :your_target`. App startup will be a little slower than usual. -4. Edit the code or Android resources. -5. Run `bazel mobile-install :your_target`. -6. Enjoy a fast and minimal incremental installation! + 1. Find the `android_binary` rule of the app you want to install. + 2. Connect your device to `adb`. + 3. Run `bazel mobile-install :your_target`. App startup will be a little + slower than usual. + 4. Edit the code or Android resources. + 5. Run `bazel mobile-install :your_target`. + 6. Enjoy a fast and minimal incremental installation! Some command line options to Bazel that may be useful: -- `--adb` tells Bazel which adb binary to use -- `--adb_arg` can be used to add extra arguments to the command line of `adb`. One useful application of this is to select which device you want to install to if you have multiple devices connected to your workstation: `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=<SERIAL>` + - `--adb` tells Bazel which adb binary to use + - `--adb_arg` can be used to add extra arguments to the command line of `adb`. + One useful application of this is to select which device you want to install + to if you have multiple devices connected to your workstation: + `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=` -When in doubt, look at the [example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) +When in doubt, look at the +[example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), +contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), +or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) ## Introduction -One of the most important attributes of a developer's toolchain is speed: there is a world of difference between changing the code and seeing it run within a second and having to wait minutes, sometimes hours, before you get any feedback on whether your changes do what you expect them to. +One of the most important attributes of a developer's toolchain is speed: there +is a world of difference between changing the code and seeing it run within a +second and having to wait minutes, sometimes hours, before you get any feedback +on whether your changes do what you expect them to. -Unfortunately, the traditional Android toolchain for building an .apk entails many monolithic, sequential steps and all of these have to be done in order to build an Android app. At Google, waiting five minutes to build a single-line change was not unusual on larger projects like Google Maps. +Unfortunately, the traditional Android toolchain for building an .apk entails +many monolithic, sequential steps and all of these have to be done in order to +build an Android app. At Google, waiting five minutes to build a single-line +change was not unusual on larger projects like Google Maps. -`bazel mobile-install` makes iterative development for Android much faster by using a combination of change pruning, work sharding, and clever manipulation of Android internals, all without changing any of your app's code. +`bazel mobile-install` makes iterative development for Android much faster by +using a combination of change pruning, work sharding, and clever manipulation of +Android internals, all without changing any of your app's code. ## Problems with traditional app installation Building an Android app has some issues, including: -- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) is invoked exactly once in the build and it does not know how to reuse work from previous builds: it dexes every method again, even though only one method was changed. +- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) +is invoked exactly once in the build and it does not know how to reuse work from +previous builds: it dexes every method again, even though only one method was +changed. -- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 connection, and larger apps can take a lot of time to upload. The entire app is uploaded, even if only small parts have changed, for example, a resource or a single method, so this can be a major bottleneck. +- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 +connection, and larger apps can take a lot of time to upload. The entire app is +uploaded, even if only small parts have changed, for example, a resource or a +single method, so this can be a major bottleneck. -- Compilation to native code. Android L introduced ART, a new Android runtime, which compiles apps ahead-of-time rather than compiling them just-in-time like Dalvik. This makes apps much faster at the cost of longer installation time. This is a good tradeoff for users because they typically install an app once and use it many times, but results in slower development where an app is installed many times and each version is run at most a handful of times. +- Compilation to native code. Android L introduced ART, a new Android runtime, +which compiles apps ahead-of-time rather than compiling them just-in-time like +Dalvik. This makes apps much faster at the cost of longer installation +time. This is a good tradeoff for users because they typically install an app +once and use it many times, but results in slower development where an app is +installed many times and each version is run at most a handful of times. ## The approach of `bazel mobile-install` `bazel mobile-install `makes the following improvements: -- Sharded desugaring and dexing. After building the app's Java code, Bazel shards the class files into approximately equal-sized parts and invokes `d8` separately on them. `d8` is not invoked on shards that did not change since the last build. These shards are then compiled into separate sharded APKs. + - Sharded desugaring and dexing. After building the app's Java code, Bazel + shards the class files into approximately equal-sized parts and invokes `d8` + separately on them. `d8` is not invoked on shards that did not change since + the last build. These shards are then compiled into separate sharded APKs. -- Incremental file transfer. Android resources, .dex files, and native libraries are removed from the main .apk and are stored in under a separate mobile-install directory. This makes it possible to update code and Android resources independently without reinstalling the whole app. Thus, transferring the files takes less time and only the .dex files that have changed are recompiled on-device. + - Incremental file transfer. Android resources, .dex files, and native + libraries are removed from the main .apk and are stored in under a separate + mobile-install directory. This makes it possible to update code and Android + resources independently without reinstalling the whole app. Thus, + transferring the files takes less time and only the .dex files that have + changed are recompiled on-device. -- Sharded installation. Mobile-install uses Android Studio's [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) tool to combine sharded APKs on the connected device and provide a cohesive experience. + - Sharded installation. Mobile-install uses Android Studio's + [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) + tool to combine sharded APKs on the connected device and provide a cohesive + experience. ### Sharded Dexing -Sharded dexing is reasonably straightforward: once the .jar files are built, a [tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) shards them into separate .jar files of approximately equal size, then invokes `d8` on those that were changed since the previous build. The logic that determines which shards to dex is not specific to Android: it just uses the general change pruning algorithm of Bazel. - -The first version of the sharding algorithm simply ordered the .class files alphabetically, then cut the list up into equal-sized parts, but this proved to be suboptimal: if a class was added or removed (even a nested or an anonymous one), it would cause all the classes alphabetically after it to shift by one, resulting in dexing those shards again. Thus, it was decided to shard Java packages rather than individual classes. Of course, this still results in dexing many shards if a new package is added or removed, but that is much less frequent than adding or removing a single class. - -The number of shards is controlled by command-line configuration, using the `--define=num_dex_shards=N` flag. In an ideal world, Bazel would automatically determine how many shards are best, but Bazel currently must know the set of actions (for example, commands to be executed during the build) before executing any of them, so it cannot determine the optimal number of shards because it doesn't know how many Java classes there will eventually be in the app. Generally speaking, the more shards, the faster the build and the installation will be, but the slower app startup becomes, because the dynamic linker has to do more work. The sweet spot is usually between 10 and 50 shards. +Sharded dexing is reasonably straightforward: once the .jar files are built, a +[tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) +shards them into separate .jar files of approximately equal size, then invokes +`d8` on those that were changed since the previous build. The logic that +determines which shards to dex is not specific to Android: it just uses the +general change pruning algorithm of Bazel. + +The first version of the sharding algorithm simply ordered the .class files +alphabetically, then cut the list up into equal-sized parts, but this proved to +be suboptimal: if a class was added or removed (even a nested or an anonymous +one), it would cause all the classes alphabetically after it to shift by one, +resulting in dexing those shards again. Thus, it was decided to shard Java +packages rather than individual classes. Of course, this still results in +dexing many shards if a new package is added or removed, but that is much less +frequent than adding or removing a single class. + +The number of shards is controlled by command-line configuration, using the +`--define=num_dex_shards=N` flag. In an ideal world, Bazel would +automatically determine how many shards are best, but Bazel currently must know +the set of actions (for example, commands to be executed during the build) before +executing any of them, so it cannot determine the optimal number of shards +because it doesn't know how many Java classes there will eventually be in the +app. Generally speaking, the more shards, the faster the build and the +installation will be, but the slower app startup becomes, because the dynamic +linker has to do more work. The sweet spot is usually between 10 and 50 shards. ### Incremental deployment -Incremental APK shard transfer and installation is now handled by the `apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). Whereas earlier (native) versions of mobile-install required manually tracking first-time installations and selectively apply the `--incremental` flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) has been greatly simplified. The same mobile-install invocation can be used regardless of how many times the app has been installed or reinstalled. - -At a high level, the `apkdeployer` tool is a wrapper around various `adb` sub-commands. The main entrypoint logic can be found in the [`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) class, with other utility classes colocated in the same package. The `Deployer` class ingests, among other things, a list of paths to split APKs and a protobuf with information about the installation, and leverages deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) in order to create an install session and incrementally deploy app splits. See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) classes for implementation details. +Incremental APK shard transfer and installation is now handled by the +`apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). +Whereas earlier (native) versions of mobile-install required manually tracking +first-time installations and selectively apply the `--incremental` +flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) +has been greatly simplified. The same mobile-install +invocation can be used regardless of how many times the app has been installed +or reinstalled. + +At a high level, the `apkdeployer` tool is a wrapper around various `adb` +sub-commands. The main entrypoint logic can be found in the +[`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) +class, with other utility classes colocated in the same package. +The `Deployer` class ingests, among other things, a list of paths to split +APKs and a protobuf with information about the installation, and leverages +deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) +in order to create an install session and incrementally deploy app splits. +See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) +and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) +classes for implementation details. ## Results ### Performance -In general, `bazel mobile-install` results in a 4x to 10x speedup of building and installing large apps after a small change. +In general, `bazel mobile-install` results in a 4x to 10x speedup of building +and installing large apps after a small change. The following numbers were computed for a few Google products: -![](/docs/images/mobile-install-performance.svg) + -This, of course, depends on the nature of the change: recompilation after changing a base library takes more time. +This, of course, depends on the nature of the change: recompilation after +changing a base library takes more time. ### Limitations -The tricks the stub application plays don't work in every case. The following cases highlight where it does not work as expected: +The tricks the stub application plays don't work in every case. +The following cases highlight where it does not work as expected: -- Mobile-install is only supported via the Starlark rules of `rules_android`. See the ["brief history of mobile-install"](#mobile-install-history) for more detail. + - Mobile-install is only supported via the Starlark rules of `rules_android`. + See the ["brief history of mobile-install"](#mobile-install-history) for + more detail. -- Only devices running ART are supported. Mobile-install uses API and runtime features that only exist on devices running ART, not Dalvik. Any Android runtime more recent than Android L (API 21+) should be compatible. + - Only devices running ART are supported. Mobile-install uses API and runtime features + that only exist on devices running ART, not Dalvik. Any Android runtime more + recent than Android L (API 21+) should be compatible. -- Bazel itself must be run with a tool Java runtime *and* language version of 17 or higher. + - Bazel itself must be run with a tool Java runtime _and_ language version + of 17 or higher. -- Bazel versions prior to 8.4.0 must specify some additional flags for mobile-install. See [the Bazel Android tutorial](/start/android-app). These flags inform Bazel where the Starlark mobile-install aspect is and which rules are supported. + - Bazel versions prior to 8.4.0 must specify some additional flags for + mobile-install. See [the Bazel Android tutorial](/start/android-app). These + flags inform Bazel where the Starlark mobile-install aspect is and which + rules are supported. ### A brief history of mobile-install +Earlier Bazel versions _natively_ included built-in build and test rules for +popular languages and ecosystems such as C++, Java, and Android. These rules +were therefore referred to as _native_ rules. Bazel 8 (released in 2024) removed +support for these rules because many of them had been migrated to the +[Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) +for more details. + +The legacy native Android rules also supported a legacy _native_ version of +mobile-install functionality. This is referred to as "mobile-install v1" or +"native mobile-install" now. This functionality was deleted in Bazel 8, along +with the built-in Android rules. -Earlier Bazel versions *natively* included built-in build and test rules for popular languages and ecosystems such as C++, Java, and Android. These rules were therefore referred to as *native* rules. Bazel 8 (released in 2024) removed support for these rules because many of them had been migrated to the [Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) for more details. +Now, all mobile-install functionality, as well as all Android build and test +rules, are implemented in Starlark and reside in the `rules_android` GitHub +repository. The latest version is known as "mobile-install v3" or "MIv3". -The legacy native Android rules also supported a legacy *native* version of mobile-install functionality. This is referred to as "mobile-install v1" or "native mobile-install" now. This functionality was deleted in Bazel 8, along with the built-in Android rules. +_Naming note_: There was a "mobile-install **v2**" available only internally +at Google at one point, but this was never published externally, and only v3 +continues to be used for both Google-internal and OSS rules_android deployment. -Now, all mobile-install functionality, as well as all Android build and test rules, are implemented in Starlark and reside in the `rules_android` GitHub repository. The latest version is known as "mobile-install v3" or "MIv3". -*Naming note*: There was a "mobile-install **v2**" available only internally at Google at one point, but this was never published externally, and only v3 continues to be used for both Google-internal and OSS rules\_android deployment. diff --git a/docs/sandboxing.mdx b/docs/sandboxing.mdx index b12027ec..68697953 100644 --- a/docs/sandboxing.mdx +++ b/docs/sandboxing.mdx @@ -2,49 +2,119 @@ title: 'Sandboxing' --- -This article covers sandboxing in Bazel and debugging your sandboxing environment. -*Sandboxing* is a permission restricting strategy that isolates processes from each other or from resources in a system. For Bazel, this means restricting file system access. -Bazel's file system sandbox runs processes in a working directory that only contains known inputs, such that compilers and other tools don't see source files they should not access, unless they know the absolute paths to them. +This article covers sandboxing in Bazel and debugging your sandboxing +environment. -Sandboxing doesn't hide the host environment in any way. Processes can freely access all files on the file system. However, on platforms that support user namespaces, processes can't modify any files outside their working directory. This ensures that the build graph doesn't have hidden dependencies that could affect the reproducibility of the build. +*Sandboxing* is a permission restricting strategy that isolates processes from +each other or from resources in a system. For Bazel, this means restricting file +system access. -More specifically, Bazel constructs an `execroot/` directory for each action, which acts as the action's work directory at execution time. `execroot/` contains all input files to the action and serves as the container for any generated outputs. Bazel then uses an operating-system-provided technique, containers on Linux and `sandbox-exec` on macOS, to constrain the action within `execroot/`. +Bazel's file system sandbox runs processes in a working directory that only +contains known inputs, such that compilers and other tools don't see source +files they should not access, unless they know the absolute paths to them. -## Reasons for sandboxing - -- Without action sandboxing, Bazel doesn't know if a tool uses undeclared input files (files that are not explicitly listed in the dependencies of an action). When one of the undeclared input files changes, Bazel still believes that the build is up-to-date and won’t rebuild the action. This can result in an incorrect incremental build. - -- Incorrect reuse of cache entries creates problems during remote caching. A bad cache entry in a shared cache affects every developer on the project, and wiping the entire remote cache is not a feasible solution. +Sandboxing doesn't hide the host environment in any way. Processes can freely +access all files on the file system. However, on platforms that support user +namespaces, processes can't modify any files outside their working directory. +This ensures that the build graph doesn't have hidden dependencies that could +affect the reproducibility of the build. -- Sandboxing mimics the behavior of remote execution — if a build works well with sandboxing, it will likely also work with remote execution. By making remote execution upload all necessary files (including local tools), you can significantly reduce maintenance costs for compile clusters compared to having to install the tools on every machine in the cluster every time you want to try out a new compiler or make a change to an existing tool. - -## What sandbox strategy to use +More specifically, Bazel constructs an `execroot/` directory for each action, +which acts as the action's work directory at execution time. `execroot/` +contains all input files to the action and serves as the container for any +generated outputs. Bazel then uses an operating-system-provided technique, +containers on Linux and `sandbox-exec` on macOS, to constrain the action within +`execroot/`. -You can choose which kind of sandboxing to use, if any, with the [strategy flags](user-manual.html#strategy-options). Using the `sandboxed` strategy makes Bazel pick one of the sandbox implementations listed below, preferring an OS-specific sandbox to the less hermetic generic one. [Persistent workers](/remote/persistent) run in a generic sandbox if you pass the `--worker_sandboxing` flag. - -The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. It simply executes the action's command line with the working directory set to the execroot of your workspace. - -`processwrapper-sandbox` is a sandboxing strategy that does not require any "advanced" features - it should work on any POSIX system out of the box. It builds a sandbox directory consisting of symlinks that point to the original source files, executes the action's command line with the working directory set to this directory instead of the execroot, then moves the known output artifacts out of the sandbox into the execroot and deletes the sandbox. This prevents the action from accidentally using any input files that are not declared and from littering the execroot with unknown output files. +## Reasons for sandboxing -`linux-sandbox` goes one step further and builds on top of the `processwrapper-sandbox`. Similar to what Docker does under the hood, it uses Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the action from the host. That is, it makes the entire filesystem read-only except for the sandbox directory, so the action cannot accidentally modify anything on the host filesystem. This prevents situations like a buggy test accidentally rm -rf'ing your $HOME directory. Optionally, you can also prevent the action from accessing the network. `linux-sandbox` uses PID namespaces to prevent the action from seeing any other processes and to reliably kill all processes (even daemons spawned by the action) at the end. +- Without action sandboxing, Bazel doesn't know if a tool uses undeclared + input files (files that are not explicitly listed in the dependencies of an + action). When one of the undeclared input files changes, Bazel still + believes that the build is up-to-date and won’t rebuild the action. This can + result in an incorrect incremental build. -`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool to achieve roughly the same as the Linux sandbox. +- Incorrect reuse of cache entries creates problems during remote caching. A + bad cache entry in a shared cache affects every developer on the project, + and wiping the entire remote cache is not a feasible solution. -Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" scenario due to restrictions in the mechanisms provided by the operating systems. Because Docker also uses Linux namespaces for its container magic, you cannot easily run `linux-sandbox` inside a Docker container, unless you use `docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a process that's already being sandboxed. Thus, in these cases, Bazel automatically falls back to using `processwrapper-sandbox`. +- Sandboxing mimics the behavior of remote execution — if a build works well + with sandboxing, it will likely also work with remote execution. By making + remote execution upload all necessary files (including local tools), you can + significantly reduce maintenance costs for compile clusters compared to + having to install the tools on every machine in the cluster every time you + want to try out a new compiler or make a change to an existing tool. -If you would rather get a build error — such as to not accidentally build with a less strict execution strategy — explicitly modify the list of execution strategies that Bazel tries to use (for example, `bazel build --spawn_strategy=worker,linux-sandbox`). +## What sandbox strategy to use -Dynamic execution usually requires sandboxing for local execution. To opt out, pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently sandboxes [persistent workers](/remote/persistent). +You can choose which kind of sandboxing to use, if any, with the +[strategy flags](user-manual.html#strategy-options). Using the `sandboxed` +strategy makes Bazel pick one of the sandbox implementations listed below, +preferring an OS-specific sandbox to the less hermetic generic one. +[Persistent workers](/remote/persistent) run in a generic sandbox if you pass +the `--worker_sandboxing` flag. + +The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. +It simply executes the action's command line with the working directory set to +the execroot of your workspace. + +`processwrapper-sandbox` is a sandboxing strategy that does not require any +"advanced" features - it should work on any POSIX system out of the box. It +builds a sandbox directory consisting of symlinks that point to the original +source files, executes the action's command line with the working directory set +to this directory instead of the execroot, then moves the known output artifacts +out of the sandbox into the execroot and deletes the sandbox. This prevents the +action from accidentally using any input files that are not declared and from +littering the execroot with unknown output files. + +`linux-sandbox` goes one step further and builds on top of the +`processwrapper-sandbox`. Similar to what Docker does under the hood, it uses +Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the +action from the host. That is, it makes the entire filesystem read-only except +for the sandbox directory, so the action cannot accidentally modify anything on +the host filesystem. This prevents situations like a buggy test accidentally rm +-rf'ing your $HOME directory. Optionally, you can also prevent the action from +accessing the network. `linux-sandbox` uses PID namespaces to prevent the action +from seeing any other processes and to reliably kill all processes (even daemons +spawned by the action) at the end. + +`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool +to achieve roughly the same as the Linux sandbox. + +Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" +scenario due to restrictions in the mechanisms provided by the operating +systems. Because Docker also uses Linux namespaces for its container magic, you +cannot easily run `linux-sandbox` inside a Docker container, unless you use +`docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a +process that's already being sandboxed. Thus, in these cases, Bazel +automatically falls back to using `processwrapper-sandbox`. + +If you would rather get a build error — such as to not accidentally build with a +less strict execution strategy — explicitly modify the list of execution +strategies that Bazel tries to use (for example, `bazel build +--spawn_strategy=worker,linux-sandbox`). + +Dynamic execution usually requires sandboxing for local execution. To opt out, +pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently +sandboxes [persistent workers](/remote/persistent). ## Downsides to sandboxing -- Sandboxing incurs extra setup and teardown cost. How big this cost is depends on many factors, including the shape of the build and the performance of the host OS. For Linux, sandboxed builds are rarely more than a few percent slower. Setting `--reuse_sandbox_directories` can mitigate the setup and teardown cost. +- Sandboxing incurs extra setup and teardown cost. How big this cost is + depends on many factors, including the shape of the build and the + performance of the host OS. For Linux, sandboxed builds are rarely more than + a few percent slower. Setting `--reuse_sandbox_directories` can + mitigate the setup and teardown cost. -- Sandboxing effectively disables any cache the tool may have. You can mitigate this by using [persistent workers](/remote/persistent), at the cost of weaker sandbox guarantees. +- Sandboxing effectively disables any cache the tool may have. You can + mitigate this by using [persistent workers](/remote/persistent), at + the cost of weaker sandbox guarantees. -- [Multiplex workers](/remote/multiplex) require explicit worker support to be sandboxed. Workers that do not support multiplex sandboxing run as singleplex workers under dynamic execution, which can cost extra memory. +- [Multiplex workers](/remote/multiplex) require explicit worker support + to be sandboxed. Workers that do not support multiplex sandboxing run as + singleplex workers under dynamic execution, which can cost extra memory. ## Debugging @@ -52,7 +122,11 @@ Follow the strategies below to debug issues with sandboxing. ### Deactivated namespaces -On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file exists and contains a 0, you can activate user namespaces by running: +On some platforms, such as +[Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) +cluster nodes or Debian, user namespaces are deactivated by default due to +security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file +exists and contains a 0, you can activate user namespaces by running: ```posix-terminal sudo sysctl kernel.unprivileged_userns_clone=1 @@ -60,11 +134,16 @@ On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/k ### Rule execution failures -The sandbox may fail to execute rules because of the system setup. If you see a message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for genrules, and `--spawn_strategy=local` for other rules. +The sandbox may fail to execute rules because of the system setup. If you see a +message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or +directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for +genrules, and `--spawn_strategy=local` for other rules. ### Detailed debugging for build failures -If your build failed, use `--verbose_failures` and `--sandbox_debug` to make Bazel show the exact command it ran when your build failed, including the part that sets up the sandbox. +If your build failed, use `--verbose_failures` and `--sandbox_debug` to make +Bazel show the exact command it ran when your build failed, including the part +that sets up the sandbox. Example error message: @@ -87,6 +166,9 @@ namespace-sandbox failed: error executing command /some/path/to/your/some-compiler --some-params some-target) ``` -You can now inspect the generated sandbox directory and see which files Bazel created and run the command again to see how it behaves. +You can now inspect the generated sandbox directory and see which files Bazel +created and run the command again to see how it behaves. -Note that Bazel does not delete the sandbox directory when you use `--sandbox_debug`. Unless you are actively debugging, you should disable `--sandbox_debug` because it fills up your disk over time. +Note that Bazel does not delete the sandbox directory when you use +`--sandbox_debug`. Unless you are actively debugging, you should disable +`--sandbox_debug` because it fills up your disk over time. diff --git a/extending/aspects.mdx b/extending/aspects.mdx index c956a8ef..ac9a0273 100644 --- a/extending/aspects.mdx +++ b/extending/aspects.mdx @@ -2,16 +2,32 @@ title: 'Aspects' --- -This page explains the basics and benefits of using [aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced examples. -Aspects allow augmenting build dependency graphs with additional information and actions. Some typical scenarios when aspects can be useful: -- IDEs that integrate Bazel can use aspects to collect information about the project. -- Code generation tools can leverage aspects to execute on their inputs in *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy of [protobuf](https://developers.google.com/protocol-buffers/) library definitions, and language-specific rules can use aspects to attach actions generating protobuf support code for a particular language. +This page explains the basics and benefits of using +[aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced +examples. + +Aspects allow augmenting build dependency graphs with additional information +and actions. Some typical scenarios when aspects can be useful: + +* IDEs that integrate Bazel can use aspects to collect information about the + project. +* Code generation tools can leverage aspects to execute on their inputs in + *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy + of [protobuf](https://developers.google.com/protocol-buffers/) library + definitions, and language-specific rules can use aspects to attach + actions generating protobuf support code for a particular language. ## Aspect basics -`BUILD` files provide a description of a project’s source code: what source files are part of the project, what artifacts (*targets*) should be built from those files, what the dependencies between those files are, etc. Bazel uses this information to perform a build, that is, it figures out the set of actions needed to produce the artifacts (such as running compiler or linker) and executes those actions. Bazel accomplishes this by constructing a *dependency graph* between targets and visiting this graph to collect those actions. +`BUILD` files provide a description of a project’s source code: what source +files are part of the project, what artifacts (_targets_) should be built from +those files, what the dependencies between those files are, etc. Bazel uses +this information to perform a build, that is, it figures out the set of actions +needed to produce the artifacts (such as running compiler or linker) and +executes those actions. Bazel accomplishes this by constructing a _dependency +graph_ between targets and visiting this graph to collect those actions. Consider the following `BUILD` file: @@ -30,21 +46,41 @@ This `BUILD` file defines a dependency graph shown in the following figure: **Figure 1.** `BUILD` file dependency graph. -Bazel analyzes this dependency graph by calling an implementation function of the corresponding [rule](/extending/rules) (in this case "java\_library") for every target in the above example. Rule implementation functions generate actions that build artifacts, such as `.jar` files, and pass information, such as locations and names of those artifacts, to the reverse dependencies of those targets in [providers](/extending/rules#providers). - -Aspects are similar to rules in that they have an implementation function that generates actions and returns providers. However, their power comes from the way the dependency graph is built for them. An aspect has an implementation and a list of all attributes it propagates along. Consider an aspect A that propagates along attributes named "deps". This aspect can be applied to a target X, yielding an aspect application node A(X). During its application, aspect A is applied recursively to all targets that X refers to in its "deps" attribute (all attributes in A's propagation list). - -Thus a single act of applying aspect A to a target X yields a "shadow graph" of the original dependency graph of targets shown in the following figure: +Bazel analyzes this dependency graph by calling an implementation function of +the corresponding [rule](/extending/rules) (in this case "java_library") for every +target in the above example. Rule implementation functions generate actions that +build artifacts, such as `.jar` files, and pass information, such as locations +and names of those artifacts, to the reverse dependencies of those targets in +[providers](/extending/rules#providers). + +Aspects are similar to rules in that they have an implementation function that +generates actions and returns providers. However, their power comes from +the way the dependency graph is built for them. An aspect has an implementation +and a list of all attributes it propagates along. Consider an aspect A that +propagates along attributes named "deps". This aspect can be applied to +a target X, yielding an aspect application node A(X). During its application, +aspect A is applied recursively to all targets that X refers to in its "deps" +attribute (all attributes in A's propagation list). + +Thus a single act of applying aspect A to a target X yields a "shadow graph" of +the original dependency graph of targets shown in the following figure: ![Build Graph with Aspect](/rules/build-graph-aspects.png "Build graph with aspects") **Figure 2.** Build graph with aspects. -The only edges that are shadowed are the edges along the attributes in the propagation set, thus the `runtime_deps` edge is not shadowed in this example. An aspect implementation function is then invoked on all nodes in the shadow graph similar to how rule implementations are invoked on the nodes of the original graph. +The only edges that are shadowed are the edges along the attributes in +the propagation set, thus the `runtime_deps` edge is not shadowed in this +example. An aspect implementation function is then invoked on all nodes in +the shadow graph similar to how rule implementations are invoked on the nodes +of the original graph. ## Simple example -This example demonstrates how to recursively print the source files for a rule and all of its dependencies that have a `deps` attribute. It shows an aspect implementation, an aspect definition, and how to invoke the aspect from the Bazel command line. +This example demonstrates how to recursively print the source files for a +rule and all of its dependencies that have a `deps` attribute. It shows +an aspect implementation, an aspect definition, and how to invoke the aspect +from the Bazel command line. ```python def _print_aspect_impl(target, ctx): @@ -75,16 +111,25 @@ print_aspect = aspect( required_providers = [CcInfo], ) ``` +Aspect definitions are similar to rule definitions, and defined using +the [`aspect`](/rules/lib/globals/bzl#aspect) function. -Aspect definitions are similar to rule definitions, and defined using the [`aspect`](/rules/lib/globals/bzl#aspect) function. - -Just like a rule, an aspect has an implementation function which in this case is `_print_aspect_impl`. +Just like a rule, an aspect has an implementation function which in this case is +``_print_aspect_impl``. -`attr_aspects` is a list of rule attributes along which the aspect propagates. In this case, the aspect will propagate along the `deps` attribute of the rules that it is applied to. +``attr_aspects`` is a list of rule attributes along which the aspect propagates. +In this case, the aspect will propagate along the ``deps`` attribute of the +rules that it is applied to. -Another common argument for `attr_aspects` is `['*']` which would propagate the aspect to all attributes of a rule. +Another common argument for `attr_aspects` is `['*']` which would propagate the +aspect to all attributes of a rule. -`required_providers` is a list of providers that allows the aspect to limit its propagation to only the targets whose rules advertise its required providers. For more details consult [the documentation of the aspect function](/rules/lib/globals/bzl#aspect). In this case, the aspect will only apply on targets that declare `CcInfo` provider. +``required_providers`` is a list of providers that allows the aspect to limit +its propagation to only the targets whose rules advertise its required +providers. For more details consult +[the documentation of the aspect function](/rules/lib/globals/bzl#aspect). +In this case, the aspect will only apply on targets that declare `CcInfo` +provider. ### Aspect implementation @@ -100,32 +145,48 @@ def _print_aspect_impl(target, ctx): return [] ``` -Aspect implementation functions are similar to the rule implementation functions. They return [providers](/extending/rules#providers), can generate [actions](/extending/rules#actions), and take two arguments: +Aspect implementation functions are similar to the rule implementation +functions. They return [providers](/extending/rules#providers), can generate +[actions](/extending/rules#actions), and take two arguments: -- `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. -- `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes and generate outputs and actions. +* `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. +* `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes + and generate outputs and actions. -The implementation function can access the attributes of the target rule via [`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are provided by the target to which it is applied (via the `target` argument). +The implementation function can access the attributes of the target rule via +[`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are +provided by the target to which it is applied (via the `target` argument). -Aspects are required to return a list of providers. In this example, the aspect does not provide anything, so it returns an empty list. +Aspects are required to return a list of providers. In this example, the aspect +does not provide anything, so it returns an empty list. ### Invoking the aspect using the command line -The simplest way to apply an aspect is from the command line using the [`--aspects`](/reference/command-line-reference#flag--aspects) argument. Assuming the aspect above were defined in a file named `print.bzl` this: +The simplest way to apply an aspect is from the command line using the +[`--aspects`](/reference/command-line-reference#flag--aspects) +argument. Assuming the aspect above were defined in a file named `print.bzl` +this: ```bash bazel build //MyExample:example --aspects print.bzl%print_aspect ``` -would apply the `print_aspect` to the target `example` and all of the target rules that are accessible recursively via the `deps` attribute. +would apply the `print_aspect` to the target `example` and all of the +target rules that are accessible recursively via the `deps` attribute. -The `--aspects` flag takes one argument, which is a specification of the aspect in the format `<extension file label>%<aspect top-level name>`. +The `--aspects` flag takes one argument, which is a specification of the aspect +in the format `%`. ## Advanced example -The following example demonstrates using an aspect from a target rule that counts files in targets, potentially filtering them by extension. It shows how to use a provider to return values, how to use parameters to pass an argument into an aspect implementation, and how to invoke an aspect from a rule. +The following example demonstrates using an aspect from a target rule +that counts files in targets, potentially filtering them by extension. +It shows how to use a provider to return values, how to use parameters to pass +an argument into an aspect implementation, and how to invoke an aspect from a rule. -Note: Aspects added in rules' attributes are called *rule-propagated aspects* as opposed to *command-line aspects* that are specified using the `--aspects` flag. +Note: Aspects added in rules' attributes are called *rule-propagated aspects* as +opposed to *command-line aspects* that are specified using the ``--aspects`` +flag. `file_count.bzl` file: @@ -213,15 +274,28 @@ file_count_aspect = aspect( ) ``` -This example shows how the aspect propagates through the `deps` attribute. +This example shows how the aspect propagates through the ``deps`` attribute. -`attrs` defines a set of attributes for an aspect. Public aspect attributes define parameters and can only be of types `bool`, `int` or `string`. For rule-propagated aspects, `int` and `string` parameters must have `values` specified on them. This example has a parameter called `extension` that is allowed to have '`*`', '`h`', or '`cc`' as a value. +``attrs`` defines a set of attributes for an aspect. Public aspect attributes +define parameters and can only be of types ``bool``, ``int`` or ``string``. +For rule-propagated aspects, ``int`` and ``string`` parameters must have +``values`` specified on them. This example has a parameter called ``extension`` +that is allowed to have '``*``', '``h``', or '``cc``' as a value. -For rule-propagated aspects, parameter values are taken from the rule requesting the aspect, using the attribute of the rule that has the same name and type. (see the definition of `file_count_rule`). +For rule-propagated aspects, parameter values are taken from the rule requesting +the aspect, using the attribute of the rule that has the same name and type. +(see the definition of ``file_count_rule``). -For command-line aspects, the parameters values can be passed using [`--aspects_parameters`](/reference/command-line-reference#flag--aspects_parameters) flag. The `values` restriction of `int` and `string` parameters may be omitted. +For command-line aspects, the parameters values can be passed using +[``--aspects_parameters``](/reference/command-line-reference#flag--aspects_parameters) +flag. The ``values`` restriction of ``int`` and ``string`` parameters may be +omitted. -Aspects are also allowed to have private attributes of types `label` or `label_list`. Private label attributes can be used to specify dependencies on tools or libraries that are needed for actions generated by aspects. There is not a private attribute defined in this example, but the following code snippet demonstrates how you could pass in a tool to an aspect: +Aspects are also allowed to have private attributes of types ``label`` or +``label_list``. Private label attributes can be used to specify dependencies on +tools or libraries that are needed for actions generated by aspects. There is not +a private attribute defined in this example, but the following code snippet +demonstrates how you could pass in a tool to an aspect: ```python ... @@ -259,17 +333,40 @@ def _file_count_aspect_impl(target, ctx): return [FileCountInfo(count = count)] ``` -Just like a rule implementation function, an aspect implementation function returns a struct of providers that are accessible to its dependencies. - -In this example, the `FileCountInfo` is defined as a provider that has one field `count`. It is best practice to explicitly define the fields of a provider using the `fields` attribute. - -The set of providers for an aspect application A(X) is the union of providers that come from the implementation of a rule for target X and from the implementation of aspect A. The providers that a rule implementation propagates are created and frozen before aspects are applied and cannot be modified from an aspect. It is an error if a target and an aspect that is applied to it each provide a provider with the same type, with the exceptions of [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) (which is merged, so long as the rule and aspect specify different output groups) and [`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) (which is taken from the aspect). This means that aspect implementations may never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). - -The parameters and private attributes are passed in the attributes of the `ctx`. This example references the `extension` parameter and determines what files to count. - -For returning providers, the values of attributes along which the aspect is propagated (from the `attr_aspects` list) are replaced with the results of an application of the aspect to them. For example, if target X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be \[A(Y), A(Z)]. In this example, `ctx.rule.attr.deps` are Target objects that are the results of applying the aspect to the 'deps' of the original target to which the aspect has been applied. - -In the example, the aspect accesses the `FileCountInfo` provider from the target's dependencies to accumulate the total transitive number of files. +Just like a rule implementation function, an aspect implementation function +returns a struct of providers that are accessible to its dependencies. + +In this example, the ``FileCountInfo`` is defined as a provider that has one +field ``count``. It is best practice to explicitly define the fields of a +provider using the ``fields`` attribute. + +The set of providers for an aspect application A(X) is the union of providers +that come from the implementation of a rule for target X and from the +implementation of aspect A. The providers that a rule implementation propagates +are created and frozen before aspects are applied and cannot be modified from an +aspect. It is an error if a target and an aspect that is applied to it each +provide a provider with the same type, with the exceptions of +[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) +(which is merged, so long as the +rule and aspect specify different output groups) and +[`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) +(which is taken from the aspect). This means that aspect implementations may +never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). + +The parameters and private attributes are passed in the attributes of the +``ctx``. This example references the ``extension`` parameter and determines +what files to count. + +For returning providers, the values of attributes along which +the aspect is propagated (from the `attr_aspects` list) are replaced with +the results of an application of the aspect to them. For example, if target +X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be [A(Y), A(Z)]. +In this example, ``ctx.rule.attr.deps`` are Target objects that are the +results of applying the aspect to the 'deps' of the original target to which +the aspect has been applied. + +In the example, the aspect accesses the ``FileCountInfo`` provider from the +target's dependencies to accumulate the total transitive number of files. ### Invoking the aspect from a rule @@ -287,9 +384,13 @@ file_count_rule = rule( ) ``` -The rule implementation demonstrates how to access the `FileCountInfo` via the `ctx.attr.deps`. +The rule implementation demonstrates how to access the ``FileCountInfo`` +via the ``ctx.attr.deps``. -The rule definition demonstrates how to define a parameter (`extension`) and give it a default value (`*`). Note that having a default value that was not one of '`cc`', '`h`', or '`*`' would be an error due to the restrictions placed on the parameter in the aspect definition. +The rule definition demonstrates how to define a parameter (``extension``) +and give it a default value (``*``). Note that having a default value that +was not one of '``cc``', '``h``', or '``*``' would be an error due to the +restrictions placed on the parameter in the aspect definition. ### Invoking an aspect through a target rule @@ -308,10 +409,13 @@ file_count_rule( ) ``` -This demonstrates how to pass the `extension` parameter into the aspect via the rule. Since the `extension` parameter has a default value in the rule implementation, `extension` would be considered an optional parameter. +This demonstrates how to pass the ``extension`` parameter into the aspect +via the rule. Since the ``extension`` parameter has a default value in the +rule implementation, ``extension`` would be considered an optional parameter. -When the `file_count` target is built, our aspect will be evaluated for itself, and all of the targets accessible recursively via `deps`. +When the ``file_count`` target is built, our aspect will be evaluated for +itself, and all of the targets accessible recursively via ``deps``. ## References -- [`aspect` API reference](/rules/lib/globals/bzl#aspect) +* [`aspect` API reference](/rules/lib/globals/bzl#aspect) diff --git a/extending/auto-exec-groups.mdx b/extending/auto-exec-groups.mdx index 72af2c3a..9fee9783 100644 --- a/extending/auto-exec-groups.mdx +++ b/extending/auto-exec-groups.mdx @@ -2,11 +2,18 @@ title: 'Automatic Execution Groups (AEGs)' --- -Automatic execution groups select an [execution platform](https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs.) for each toolchain type. In other words, one target can have multiple execution platforms without defining execution groups. + + +Automatic execution groups select an [execution platform][exec_platform] +for each toolchain type. In other words, one target can have multiple +execution platforms without defining execution groups. ## Quick summary -Automatic execution groups are closely connected to toolchains. If you are using toolchains, you need to set them on the affected actions (actions which use an executable or a tool from a toolchain) by adding `toolchain` parameter. For example: +Automatic execution groups are closely connected to toolchains. If you are using +toolchains, you need to set them on the affected actions (actions which use an +executable or a tool from a toolchain) by adding `toolchain` parameter. For +example: ```python ctx.actions.run( @@ -16,10 +23,15 @@ ctx.actions.run( toolchain = '@bazel_tools//tools/jdk:toolchain_type', ) ``` +If the action does not use a tool or executable from a toolchain, and Blaze +doesn't detect that ([the error](#first-error-message) is raised), you can set +`toolchain = None`. -If the action does not use a tool or executable from a toolchain, and Blaze doesn't detect that ([the error](#first-error-message) is raised), you can set `toolchain = None`. - -If you need to use multiple toolchains on a single execution platform (an action uses executable or tools from two or more toolchains), you need to manually define [exec\_groups](https://bazel.build/extending/exec-groups) (check [When should I use a custom exec\_group?](/extending/auto-exec-groups#when-should-use-exec-groups) section). +If you need to use multiple toolchains on a single execution platform (an action +uses executable or tools from two or more toolchains), you need to manually +define [exec_groups][exec_groups] (check +[When should I use a custom exec_group?][multiple_toolchains_exec_groups] +section). ## History @@ -32,11 +44,20 @@ my_rule = rule( ) ``` -Rule `my_rule` registers two toolchain types. This means that the [Toolchain Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used to find an execution platform which supports both toolchain types. The selected execution platform was used for each registered action inside the rule, unless specified differently with [exec\_groups](https://bazel.build/extending/exec-groups). In other words, all actions inside the rule used to have a single execution platform even if they used tools from different toolchains (execution platform is selected for each target). This resulted in failures when there was no execution platform supporting all toolchains. +Rule `my_rule` registers two toolchain types. This means that the [Toolchain +Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used +to find an execution platform which supports both toolchain types. The selected +execution platform was used for each registered action inside the rule, unless +specified differently with [exec_groups][exec_groups]. +In other words, all actions inside the rule used to have a single execution +platform even if they used tools from different toolchains (execution platform +is selected for each target). This resulted in failures when there was no +execution platform supporting all toolchains. ## Current state -With AEGs, the execution platform is selected for each toolchain type. The implementation function of the earlier example, `my_rule`, would look like: +With AEGs, the execution platform is selected for each toolchain type. The +implementation function of the earlier example, `my_rule`, would look like: ```python def _impl(ctx): @@ -53,17 +74,29 @@ def _impl(ctx): ) ``` -This rule creates two actions, the `First action` which uses executable from a `//tools:toolchain_type_1` and the `Second action` which uses executable from a `//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed on a single execution platform which supports both toolchain types. With AEGs, by adding the `toolchain` parameter inside the actions, each action executes on the execution platform that provides the toolchain. The actions may be executed on different execution platforms. +This rule creates two actions, the `First action` which uses executable from a +`//tools:toolchain_type_1` and the `Second action` which uses executable from a +`//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed +on a single execution platform which supports both toolchain types. With AEGs, +by adding the `toolchain` parameter inside the actions, each action executes on +the execution platform that provides the toolchain. The actions may be executed +on different execution platforms. -The same is effective with [ctx.actions.run\_shell](https://bazel.build/rules/lib/builtins/actions#run_shell) where `toolchain` parameter should be added when `tools` are from a toolchain. +The same is effective with [ctx.actions.run_shell][run_shell] where `toolchain` +parameter should be added when `tools` are from a toolchain. ## Difference between custom exec groups and automatic exec groups -As the name suggests, AEGs are exec groups created automatically for each toolchain type registered on a rule. There is no need to manually specify them, unlike the "classic" exec groups. Moreover, name of AEG is automatically set to its toolchain type (e.g. `//tools:toolchain_type_1`). +As the name suggests, AEGs are exec groups created automatically for each +toolchain type registered on a rule. There is no need to manually specify them, +unlike the "classic" exec groups. Moreover, name of AEG is automatically set to +its toolchain type (e.g. `//tools:toolchain_type_1`). -### When should I use a custom exec\_group? +### When should I use a custom exec_group? -Custom exec\_groups are needed only in case where multiple toolchains need to execute on a single execution platform. In all other cases there's no need to define custom exec\_groups. For example: +Custom exec_groups are needed only in case where multiple toolchains need to +execute on a single execution platform. In all other cases there's no need to +define custom exec_groups. For example: ```python def _impl(ctx): @@ -88,7 +121,9 @@ my_rule = rule( ## Migration of AEGs -Internally in google3, Blaze is already using AEGs. Externally for Bazel, migration is in the process. Some rules are already using this feature (e.g. Java and C++ rules). +Internally in google3, Blaze is already using AEGs. +Externally for Bazel, migration is in the process. Some rules are already using +this feature (e.g. Java and C++ rules). ### Which Bazel versions support this migration? @@ -96,7 +131,8 @@ AEGs are fully supported from Bazel 7. ### How to enable AEGs? -Set `--incompatible_auto_exec_groups` to true. More information about the flag on [the GitHub issue](https://github.com/bazelbuild/bazel/issues/17134). +Set `--incompatible_auto_exec_groups` to true. More information about the flag +on [the GitHub issue][github_flag]. ### How to enable AEGs inside a particular rule? @@ -110,23 +146,38 @@ my_rule = rule( } ) ``` - -This enables AEGs only in `my_rule` and its actions start using the new logic when selecting the execution platform. Incompatible flag is overridden with this attribute. +This enables AEGs only in `my_rule` and its actions start using the new logic +when selecting the execution platform. Incompatible flag is overridden with this +attribute. ### How to disable AEGs in case of an error? -Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in your project ([flag's GitHub issue](https://github.com/bazelbuild/bazel/issues/17134)), or disable a particular rule by setting `_use_auto_exec_groups` attribute to `False` ([more details about the attribute](#how-enable-particular-rule)). +Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in +your project ([flag's GitHub issue][github_flag]), or disable a particular rule +by setting `_use_auto_exec_groups` attribute to `False` +([more details about the attribute](#how-enable-particular-rule)). ### Error messages while migrating to AEGs #### Couldn't identify if tools are from implicit dependencies or a toolchain. Please set the toolchain parameter. If you're not using a toolchain, set it to 'None'. + * In this case you get a stack of calls before the error happened and you can + clearly see which exact action needs the toolchain parameter. Check which + toolchain is used for the action and set it with the toolchain param. If no + toolchain is used inside the action for tools or executable, set it to + `None`. -- In this case you get a stack of calls before the error happened and you can clearly see which exact action needs the toolchain parameter. Check which toolchain is used for the action and set it with the toolchain param. If no toolchain is used inside the action for tools or executable, set it to `None`. - -#### Action declared for non-existent toolchain '\[toolchain\_type]'. - -- This means that you've set the toolchain parameter on the action but didn't register it on the rule. Register the toolchain or set `None` inside the action. +#### Action declared for non-existent toolchain '[toolchain_type]'. + * This means that you've set the toolchain parameter on the action but didn't +register it on the rule. Register the toolchain or set `None` inside the action. ## Additional material -For more information, check design document: [Automatic exec groups for toolchains](https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch). +For more information, check design document: +[Automatic exec groups for toolchains][aegs_design_doc]. + +[exec_platform]: https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs. +[exec_groups]: https://bazel.build/extending/exec-groups +[github_flag]: https://github.com/bazelbuild/bazel/issues/17134 +[aegs_design_doc]: https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch +[run_shell]: https://bazel.build/rules/lib/builtins/actions#run_shell +[multiple_toolchains_exec_groups]: /extending/auto-exec-groups#when-should-use-exec-groups diff --git a/extending/concepts.mdx b/extending/concepts.mdx index da9fbd1d..e634c611 100644 --- a/extending/concepts.mdx +++ b/extending/concepts.mdx @@ -2,62 +2,111 @@ title: 'Extension Overview' --- -This page describes how to extend the BUILD language using macros and rules. -Bazel extensions are files ending in `.bzl`. Use a [load statement](/concepts/build-files#load) to import a symbol from an extension. -Before learning the more advanced concepts, first: +{/* [TOC] */} -- Read about the [Starlark language](/rules/language), used in both the `BUILD` and `.bzl` files. +This page describes how to extend the BUILD language using macros +and rules. -- Learn how you can [share variables](/build/share-variables) between two `BUILD` files. +Bazel extensions are files ending in `.bzl`. Use a +[load statement](/concepts/build-files#load) to import a symbol from an extension. -## Macros and rules +Before learning the more advanced concepts, first: -A macro is a function that instantiates rules. Macros come in two flavors: [symbolic macros](/extending/macros) (new in Bazel 8) and [legacy macros](/extending/legacy-macros). The two flavors of macros are defined differently, but behave almost the same from the point of view of a user. A macro is useful when a `BUILD` file is getting too repetitive or too complex, as it lets you reuse some code. The function is evaluated as soon as the `BUILD` file is read. After the evaluation of the `BUILD` file, Bazel has little information about macros. If your macro generates a `genrule`, Bazel will behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one exception is that targets declared in a symbolic macro have [special visibility semantics](/extending/macros#visibility): a symbolic macro can hide its internal targets from the rest of the package.) +* Read about the [Starlark language](/rules/language), used in both the + `BUILD` and `.bzl` files. -A [rule](/extending/rules) is more powerful than a macro. It can access Bazel internals and have full control over what is going on. It may for example pass information to other rules. +* Learn how you can [share variables](/build/share-variables) + between two `BUILD` files. + +## Macros and rules -If you want to reuse simple logic, start with a macro; we recommend a symbolic macro, unless you need to support older Bazel versions. If a macro becomes complex, it is often a good idea to make it a rule. Support for a new language is typically done with a rule. Rules are for advanced users, and most users will never have to write one; they will only load and call existing rules. +A macro is a function that instantiates rules. Macros come in two flavors: +[symbolic macros](/extending/macros) (new in Bazel 8) and [legacy +macros](/extending/legacy-macros). The two flavors of macros are defined +differently, but behave almost the same from the point of view of a user. A +macro is useful when a `BUILD` file is getting too repetitive or too complex, as +it lets you reuse some code. The function is evaluated as soon as the `BUILD` +file is read. After the evaluation of the `BUILD` file, Bazel has little +information about macros. If your macro generates a `genrule`, Bazel will +behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one +exception is that targets declared in a symbolic macro have [special visibility +semantics](/extending/macros#visibility): a symbolic macro can hide its internal +targets from the rest of the package.) + +A [rule](/extending/rules) is more powerful than a macro. It can access Bazel +internals and have full control over what is going on. It may for example pass +information to other rules. + +If you want to reuse simple logic, start with a macro; we recommend a symbolic +macro, unless you need to support older Bazel versions. If a macro becomes +complex, it is often a good idea to make it a rule. Support for a new language +is typically done with a rule. Rules are for advanced users, and most users will +never have to write one; they will only load and call existing rules. ## Evaluation model A build consists of three phases. -- **Loading phase**. First, load and evaluate all extensions and all `BUILD` files that are needed for the build. The execution of the `BUILD` files simply instantiates rules (each time a rule is called, it gets added to a graph). This is where macros are evaluated. - -- **Analysis phase**. The code of the rules is executed (their `implementation` function), and actions are instantiated. An action describes how to generate a set of outputs from a set of inputs, such as "run gcc on hello.c and get hello.o". You must list explicitly which files will be generated before executing the actual commands. In other words, the analysis phase takes the graph generated by the loading phase and generates an action graph. - -- **Execution phase**. Actions are executed, when at least one of their outputs is required. If a file is missing or if a command fails to generate one output, the build fails. Tests are also run during this phase. - -Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` files. A file is read at most once per build and the result of the evaluation is cached and reused. A file is evaluated only once all its dependencies (`load()` statements) have been resolved. By design, loading a `.bzl` file has no visible side-effect, it only defines values and functions. - -Bazel tries to be clever: it uses dependency analysis to know which files must be loaded, which rules must be analyzed, and which actions must be executed. For example, if a rule generates actions that you don't need for the current build, they will not be executed. +* **Loading phase**. First, load and evaluate all extensions and all `BUILD` + files that are needed for the build. The execution of the `BUILD` files simply + instantiates rules (each time a rule is called, it gets added to a graph). + This is where macros are evaluated. + +* **Analysis phase**. The code of the rules is executed (their `implementation` + function), and actions are instantiated. An action describes how to generate + a set of outputs from a set of inputs, such as "run gcc on hello.c and get + hello.o". You must list explicitly which files will be generated before + executing the actual commands. In other words, the analysis phase takes + the graph generated by the loading phase and generates an action graph. + +* **Execution phase**. Actions are executed, when at least one of their outputs is + required. If a file is missing or if a command fails to generate one output, + the build fails. Tests are also run during this phase. + +Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` +files. A file is read at most once per build and the result of the evaluation is +cached and reused. A file is evaluated only once all its dependencies (`load()` +statements) have been resolved. By design, loading a `.bzl` file has no visible +side-effect, it only defines values and functions. + +Bazel tries to be clever: it uses dependency analysis to know which files must +be loaded, which rules must be analyzed, and which actions must be executed. For +example, if a rule generates actions that you don't need for the current build, +they will not be executed. ## Creating extensions -- [Create your first macro](/rules/macro-tutorial) in order to reuse some code. Then [learn more about macros](/extending/macros) and [using them to create "custom verbs"](/rules/verbs-tutorial). +* [Create your first macro](/rules/macro-tutorial) in order to reuse some code. + Then [learn more about macros](/extending/macros) and [using them to create + "custom verbs"](/rules/verbs-tutorial). -- [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. Next, you can read more about the [rules concepts](/extending/rules). +* [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. + Next, you can read more about the [rules concepts](/extending/rules). -The two links below will be very useful when writing your own extensions. Keep them within reach: +The two links below will be very useful when writing your own extensions. Keep +them within reach: -- The [API reference](/rules/lib) +* The [API reference](/rules/lib) -- [Examples](https://github.com/bazelbuild/examples/tree/master/rules) +* [Examples](https://github.com/bazelbuild/examples/tree/master/rules) ## Going further -In addition to [macros](/extending/macros) and [rules](/extending/rules), you may want to write [aspects](/extending/aspects) and [repository rules](/external/repo). +In addition to [macros](/extending/macros) and [rules](/extending/rules), you +may want to write [aspects](/extending/aspects) and [repository +rules](/external/repo). -- Use [Buildifier](https://github.com/bazelbuild/buildtools) consistently to format and lint your code. +* Use [Buildifier](https://github.com/bazelbuild/buildtools) + consistently to format and lint your code. -- Follow the [`.bzl` style guide](/rules/bzl-style). +* Follow the [`.bzl` style guide](/rules/bzl-style). -- [Test](/rules/testing) your code. +* [Test](/rules/testing) your code. -- [Generate documentation](https://skydoc.bazel.build/) to help your users. +* [Generate documentation](https://skydoc.bazel.build/) to help your users. -- [Optimize the performance](/rules/performance) of your code. +* [Optimize the performance](/rules/performance) of your code. -- [Deploy](/rules/deploying) your extensions to other people. +* [Deploy](/rules/deploying) your extensions to other people. diff --git a/extending/depsets.mdx b/extending/depsets.mdx index 1f0d18c4..4e84d3f2 100644 --- a/extending/depsets.mdx +++ b/extending/depsets.mdx @@ -2,21 +2,40 @@ title: 'Depsets' --- -[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently collecting data across a target’s transitive dependencies. They are an essential element of rule processing. -The defining feature of depset is its time- and space-efficient union operation. The depset constructor accepts a list of elements ("direct") and a list of other depsets ("transitive"), and returns a depset representing a set containing all the direct elements and the union of all the transitive sets. Conceptually, the constructor creates a new graph node that has the direct and transitive nodes as its successors. Depsets have a well-defined ordering semantics, based on traversal of this graph. + +[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently +collecting data across a target’s transitive dependencies. They are an essential +element of rule processing. + +The defining feature of depset is its time- and space-efficient union operation. +The depset constructor accepts a list of elements ("direct") and a list of other +depsets ("transitive"), and returns a depset representing a set containing all the +direct elements and the union of all the transitive sets. Conceptually, the +constructor creates a new graph node that has the direct and transitive nodes +as its successors. Depsets have a well-defined ordering semantics, based on +traversal of this graph. Example uses of depsets include: -- Storing the paths of all object files for a program’s libraries, which can then be passed to a linker action through a provider. +* Storing the paths of all object files for a program’s libraries, which can + then be passed to a linker action through a provider. -- For an interpreted language, storing the transitive source files that are included in an executable's runfiles. +* For an interpreted language, storing the transitive source files that are + included in an executable's runfiles. ## Description and operations -Conceptually, a depset is a directed acyclic graph (DAG) that typically looks similar to the target graph. It is constructed from the leaves up to the root. Each target in a dependency chain can add its own contents on top of the previous without having to read or copy them. +Conceptually, a depset is a directed acyclic graph (DAG) that typically looks +similar to the target graph. It is constructed from the leaves up to the root. +Each target in a dependency chain can add its own contents on top of the +previous without having to read or copy them. -Each node in the DAG holds a list of direct elements and a list of child nodes. The contents of the depset are the transitive elements, such as the direct elements of all the nodes. A new depset can be created using the [depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct elements and another list of child nodes. +Each node in the DAG holds a list of direct elements and a list of child nodes. +The contents of the depset are the transitive elements, such as the direct elements +of all the nodes. A new depset can be created using the +[depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct +elements and another list of child nodes. ```python s = depset(["a", "b", "c"]) @@ -26,7 +45,11 @@ print(s) # depset(["a", "b", "c"]) print(t) # depset(["d", "e", "a", "b", "c"]) ``` -To retrieve the contents of a depset, use the [to\_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive elements, not including duplicates. There is no way to directly inspect the precise structure of the DAG, although this structure does affect the order in which the elements are returned. +To retrieve the contents of a depset, use the +[to_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive +elements, not including duplicates. There is no way to directly inspect the +precise structure of the DAG, although this structure does affect the order in +which the elements are returned. ```python s = depset(["a", "b", "c"]) @@ -35,9 +58,11 @@ print("c" in s.to_list()) # True print(s.to_list() == ["a", "b", "c"]) # True ``` -The allowed items in a depset are restricted, just as the allowed keys in dictionaries are restricted. In particular, depset contents may not be mutable. +The allowed items in a depset are restricted, just as the allowed keys in +dictionaries are restricted. In particular, depset contents may not be mutable. -Depsets use reference equality: a depset is equal to itself, but unequal to any other depset, even if they have the same contents and same internal structure. +Depsets use reference equality: a depset is equal to itself, but unequal to any +other depset, even if they have the same contents and same internal structure. ```python s = depset(["a", "b", "c"]) @@ -61,7 +86,9 @@ t = depset(["c", "b", "a"]) print(sorted(s.to_list()) == sorted(t.to_list())) # True ``` -There is no ability to remove elements from a depset. If this is needed, you must read out the entire contents of the depset, filter the elements you want to remove, and reconstruct a new depset. This is not particularly efficient. +There is no ability to remove elements from a depset. If this is needed, you +must read out the entire contents of the depset, filter the elements you want to +remove, and reconstruct a new depset. This is not particularly efficient. ```python s = depset(["a", "b", "c"]) @@ -78,9 +105,24 @@ print(s) # depset(["a"]) ### Order -The `to_list` operation performs a traversal over the DAG. The kind of traversal depends on the *order* that was specified at the time the depset was constructed. It is useful for Bazel to support multiple orders because sometimes tools care about the order of their inputs. For example, a linker action may need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the linker’s command line. Other tools might have the opposite requirement. - -Three traversal orders are supported: `postorder`, `preorder`, and `topological`. The first two work exactly like [tree traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) except that they operate on DAGs and skip already visited nodes. The third order works as a topological sort from root to leaves, essentially the same as preorder except that shared children are listed only after all of their parents. Preorder and postorder operate as left-to-right traversals, but note that within each node direct elements have no order relative to children. For topological order, there is no left-to-right guarantee, and even the all-parents-before-child guarantee does not apply in the case that there are duplicate elements in different nodes of the DAG. +The `to_list` operation performs a traversal over the DAG. The kind of traversal +depends on the *order* that was specified at the time the depset was +constructed. It is useful for Bazel to support multiple orders because sometimes +tools care about the order of their inputs. For example, a linker action may +need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the +linker’s command line. Other tools might have the opposite requirement. + +Three traversal orders are supported: `postorder`, `preorder`, and +`topological`. The first two work exactly like [tree +traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) +except that they operate on DAGs and skip already visited nodes. The third order +works as a topological sort from root to leaves, essentially the same as +preorder except that shared children are listed only after all of their parents. +Preorder and postorder operate as left-to-right traversals, but note that within +each node direct elements have no order relative to children. For topological +order, there is no left-to-right guarantee, and even the +all-parents-before-child guarantee does not apply in the case that there are +duplicate elements in different nodes of the DAG. ```python # This demonstrates different traversal orders. @@ -109,13 +151,20 @@ print(create("preorder").to_list()) # ["d", "b", "a", "c"] print(create("topological").to_list()) # ["d", "b", "c", "a"] ``` -Due to how traversals are implemented, the order must be specified at the time the depset is created with the constructor’s `order` keyword argument. If this argument is omitted, the depset has the special `default` order, in which case there are no guarantees about the order of any of its elements (except that it is deterministic). +Due to how traversals are implemented, the order must be specified at the time +the depset is created with the constructor’s `order` keyword argument. If this +argument is omitted, the depset has the special `default` order, in which case +there are no guarantees about the order of any of its elements (except that it +is deterministic). ## Full example -This example is available at [https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). +This example is available at +[https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). -Suppose there is a hypothetical interpreted language Foo. In order to build each `foo_binary` you need to know all the `*.foo` files that it directly or indirectly depends on. +Suppose there is a hypothetical interpreted language Foo. In order to build +each `foo_binary` you need to know all the `*.foo` files that it directly or +indirectly depends on. ```python # //depsets:BUILD @@ -159,14 +208,21 @@ foo_binary( import sys if __name__ == "__main__": - assert len(sys.argv) >= 1 + assert len(sys.argv) >= 1 output = open(sys.argv[1], "wt") for path in sys.argv[2:]: input = open(path, "rt") output.write(input.read()) ``` -Here, the transitive sources of the binary `d` are all of the `*.foo` files in the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` target to know about any file besides `d.foo`, the `foo_library` targets need to pass them along in a provider. Each library receives the providers from its own dependencies, adds its own immediate sources, and passes on a new provider with the augmented contents. The `foo_binary` rule does the same, except that instead of returning a provider, it uses the complete list of sources to construct a command line for an action. +Here, the transitive sources of the binary `d` are all of the `*.foo` files in +the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` +target to know about any file besides `d.foo`, the `foo_library` targets need to +pass them along in a provider. Each library receives the providers from its own +dependencies, adds its own immediate sources, and passes on a new provider with +the augmented contents. The `foo_binary` rule does the same, except that instead +of returning a provider, it uses the complete list of sources to construct a +command line for an action. Here’s a complete implementation of the `foo_library` and `foo_binary` rules. @@ -223,11 +279,15 @@ foo_binary = rule( ) ``` -You can test this by copying these files into a fresh package, renaming the labels appropriately, creating the source `*.foo` files with dummy content, and building the `d` target. +You can test this by copying these files into a fresh package, renaming the +labels appropriately, creating the source `*.foo` files with dummy content, and +building the `d` target. + ## Performance -To see the motivation for using depsets, consider what would happen if `get_transitive_srcs()` collected its sources in a list. +To see the motivation for using depsets, consider what would happen if +`get_transitive_srcs()` collected its sources in a list. ```python def get_transitive_srcs(srcs, deps): @@ -238,9 +298,12 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This does not take into account duplicates, so the source files for `a` will appear twice on the command line and twice in the contents of the output file. +This does not take into account duplicates, so the source files for `a` +will appear twice on the command line and twice in the contents of the output +file. -An alternative is using a general set, which can be simulated by a dictionary where the keys are the elements and all the keys map to `True`. +An alternative is using a general set, which can be simulated by a +dictionary where the keys are the elements and all the keys map to `True`. ```python def get_transitive_srcs(srcs, deps): @@ -253,16 +316,31 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This gets rid of the duplicates, but it makes the order of the command line arguments (and therefore the contents of the files) unspecified, although still deterministic. +This gets rid of the duplicates, but it makes the order of the command line +arguments (and therefore the contents of the files) unspecified, although still +deterministic. -Moreover, both approaches are asymptotically worse than the depset-based approach. Consider the case where there is a long chain of dependencies on Foo libraries. Processing every rule requires copying all of the transitive sources that came before it into a new data structure. This means that the time and space cost for analyzing an individual library or binary target is proportional to its own height in the chain. For a chain of length n, foolib\_1 ← foolib\_2 ← … ← foolib\_n, the overall cost is effectively O(n^2). +Moreover, both approaches are asymptotically worse than the depset-based +approach. Consider the case where there is a long chain of dependencies on +Foo libraries. Processing every rule requires copying all of the transitive +sources that came before it into a new data structure. This means that the +time and space cost for analyzing an individual library or binary target +is proportional to its own height in the chain. For a chain of length n, +foolib_1 ← foolib_2 ← … ← foolib_n, the overall cost is effectively O(n^2). -Generally speaking, depsets should be used whenever you are accumulating information through your transitive dependencies. This helps ensure that your build scales well as your target graph grows deeper. +Generally speaking, depsets should be used whenever you are accumulating +information through your transitive dependencies. This helps ensure that +your build scales well as your target graph grows deeper. -Finally, it’s important to not retrieve the contents of the depset unnecessarily in rule implementations. One call to `to_list()` at the end in a binary rule is fine, since the overall cost is just O(n). It’s when many non-terminal targets try to call `to_list()` that quadratic behavior occurs. +Finally, it’s important to not retrieve the contents of the depset +unnecessarily in rule implementations. One call to `to_list()` +at the end in a binary rule is fine, since the overall cost is just O(n). It’s +when many non-terminal targets try to call `to_list()` that quadratic behavior +occurs. For more information about using depsets efficiently, see the [performance](/rules/performance) page. ## API Reference Please see [here](/rules/lib/builtins/depset) for more details. + diff --git a/extending/exec-groups.mdx b/extending/exec-groups.mdx index 6f7e7198..a8b45b92 100644 --- a/extending/exec-groups.mdx +++ b/extending/exec-groups.mdx @@ -2,21 +2,40 @@ title: 'Execution Groups' --- -Execution groups allow for multiple execution platforms within a single target. Each execution group has its own [toolchain](/extending/toolchains) dependencies and performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). + + +Execution groups allow for multiple execution platforms within a single target. +Each execution group has its own [toolchain](/extending/toolchains) dependencies and +performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). ## Current status -Execution groups for certain natively declared actions, like `CppLink`, can be used inside `exec_properties` to set per-action, per-target execution requirements. For more details, see the [Default execution groups](#exec-groups-for-native-rules) section. +Execution groups for certain natively declared actions, like `CppLink`, can be +used inside `exec_properties` to set per-action, per-target execution +requirements. For more details, see the +[Default execution groups](#exec-groups-for-native-rules) section. ## Background -Execution groups allow the rule author to define sets of actions, each with a potentially different execution platform. Multiple execution platforms can allow actions to execution differently, for example compiling an iOS app on a remote (linux) worker and then linking/code signing on a local mac worker. +Execution groups allow the rule author to define sets of actions, each with a +potentially different execution platform. Multiple execution platforms can allow +actions to execution differently, for example compiling an iOS app on a remote +(linux) worker and then linking/code signing on a local mac worker. -Being able to define groups of actions also helps alleviate the usage of action mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be unique and can only reference a single action. This is especially helpful in allocating extra resources to specific memory and processing intensive actions like linking in C++ builds without over-allocating to less demanding tasks. +Being able to define groups of actions also helps alleviate the usage of action +mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be +unique and can only reference a single action. This is especially helpful in +allocating extra resources to specific memory and processing intensive actions +like linking in C++ builds without over-allocating to less demanding tasks. ## Defining execution groups -During rule definition, rule authors can [declare](/rules/lib/globals/bzl#exec_group) a set of execution groups. On each execution group, the rule author can specify everything needed to select an execution platform for that execution group, namely any constraints via `exec_compatible_with` and toolchain types via `toolchain`. +During rule definition, rule authors can +[declare](/rules/lib/globals/bzl#exec_group) +a set of execution groups. On each execution group, the rule author can specify +everything needed to select an execution platform for that execution group, +namely any constraints via `exec_compatible_with` and toolchain types via +`toolchain`. ```python # foo.bzl @@ -37,13 +56,25 @@ my_rule = rule( ) ``` -In the code snippet above, you can see that tool dependencies can also specify transition for an exec group using the [`cfg`](/rules/lib/toplevel/attr#label) attribute param and the [`config`](/rules/lib/toplevel/config) module. The module exposes an `exec` function which takes a single string parameter which is the name of the exec group for which the dependency should be built. +In the code snippet above, you can see that tool dependencies can also specify +transition for an exec group using the +[`cfg`](/rules/lib/toplevel/attr#label) +attribute param and the +[`config`](/rules/lib/toplevel/config) +module. The module exposes an `exec` function which takes a single string +parameter which is the name of the exec group for which the dependency should be +built. -As on native rules, the `test` execution group is present by default on Starlark test rules. +As on native rules, the `test` execution group is present by default on Starlark +test rules. ## Accessing execution groups -In the rule implementation, you can declare that actions should be run on the execution platform of an execution group. You can do this by using the `exec_group` param of action generating methods, specifically \[`ctx.actions.run`] (/rules/lib/builtins/actions#run) and [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). +In the rule implementation, you can declare that actions should be run on the +execution platform of an execution group. You can do this by using the `exec_group` +param of action generating methods, specifically [`ctx.actions.run`] +(/rules/lib/builtins/actions#run) and +[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). ```python # foo.bzl @@ -55,7 +86,9 @@ def _impl(ctx): ) ``` -Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) of execution groups, similarly to how you can access the resolved toolchain of a target: +Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) +of execution groups, similarly to how you +can access the resolved toolchain of a target: ```python # foo.bzl @@ -68,18 +101,28 @@ def _impl(ctx): ) ``` -Note: If an action uses a toolchain from an execution group, but doesn't specify that execution group in the action declaration, that may potentially cause issues. A mismatch like this may not immediately cause failures, but is a latent problem. +Note: If an action uses a toolchain from an execution group, but doesn't specify +that execution group in the action declaration, that may potentially cause +issues. A mismatch like this may not immediately cause failures, but is a latent +problem. ### Default execution groups The following execution groups are predefined: -- `test`: Test runner actions (for more details, see the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). -- `cpp_link`: C++ linking actions. +* `test`: Test runner actions (for more details, see + the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). +* `cpp_link`: C++ linking actions. ## Using execution groups to set execution properties -Execution groups are integrated with the [`exec_properties`](/reference/be/common-definitions#common-attributes) attribute that exists on every rule and allows the target writer to specify a string dict of properties that is then passed to the execution machinery. For example, if you wanted to set some property, say memory, for the target and give certain actions a higher memory allocation, you would write an `exec_properties` entry with an execution-group-augmented key, such as: +Execution groups are integrated with the +[`exec_properties`](/reference/be/common-definitions#common-attributes) +attribute that exists on every rule and allows the target writer to specify a +string dict of properties that is then passed to the execution machinery. For +example, if you wanted to set some property, say memory, for the target and give +certain actions a higher memory allocation, you would write an `exec_properties` +entry with an execution-group-augmented key, such as: ```python # BUILD @@ -93,13 +136,24 @@ my_rule( ) ``` -All actions with `exec_group = "link"` would see the exec properties dictionary as `{"mem": "16g"}`. As you see here, execution-group-level settings override target-level settings. +All actions with `exec_group = "link"` would see the exec properties +dictionary as `{"mem": "16g"}`. As you see here, execution-group-level +settings override target-level settings. ## Using execution groups to set platform constraints -Execution groups are also integrated with the [`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and [`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) attributes that exist on every rule and allow the target writer to specify additional constraints that must be satisfied by the execution platforms selected for the target's actions. +Execution groups are also integrated with the +[`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and +[`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) +attributes that exist on every rule and allow the target writer to specify +additional constraints that must be satisfied by the execution platforms +selected for the target's actions. -For example, if the rule `my_test` defines the `link` execution group in addition to the default and the `test` execution group, then the following usage of these attributes would run actions in the default execution group on a platform with a high number of CPUs, the test action on Linux, and the link action on the default execution platform: +For example, if the rule `my_test` defines the `link` execution group in +addition to the default and the `test` execution group, then the following +usage of these attributes would run actions in the default execution group on +a platform with a high number of CPUs, the test action on Linux, and the link +action on the default execution platform: ```python # BUILD @@ -126,16 +180,23 @@ my_test( ### Execution groups for native rules -The following execution groups are available for actions defined by native rules: +The following execution groups are available for actions defined by native +rules: -- `test`: Test runner actions. -- `cpp_link`: C++ linking actions. +* `test`: Test runner actions. +* `cpp_link`: C++ linking actions. ### Execution groups and platform execution properties -It is possible to define `exec_properties` for arbitrary execution groups on platform targets (unlike `exec_properties` set directly on a target, where properties for unknown execution groups are rejected). Targets then inherit the execution platform's `exec_properties` that affect the default execution group and any other relevant execution groups. +It is possible to define `exec_properties` for arbitrary execution groups on +platform targets (unlike `exec_properties` set directly on a target, where +properties for unknown execution groups are rejected). Targets then inherit the +execution platform's `exec_properties` that affect the default execution group +and any other relevant execution groups. -For example, suppose running tests on the exec platform requires some resource to be available, but it isn't required for compiling and linking; this can be modelled as follows: +For example, suppose running tests on the exec platform requires some resource +to be available, but it isn't required for compiling and linking; this can be +modelled as follows: ```python constraint_setting(name = "resource") @@ -156,4 +217,5 @@ cc_test( ) ``` -`exec_properties` defined directly on targets take precedence over those that are inherited from the execution platform. +`exec_properties` defined directly on targets take precedence over those that +are inherited from the execution platform. diff --git a/extending/legacy-macros.mdx b/extending/legacy-macros.mdx index 910cc46a..9c2b8e28 100644 --- a/extending/legacy-macros.mdx +++ b/extending/legacy-macros.mdx @@ -2,7 +2,12 @@ title: 'Legacy Macros' --- -Legacy macros are unstructured functions called from `BUILD` files that can create targets. By the end of the [loading phase](/extending/concepts#evaluation-model), legacy macros don't exist anymore, and Bazel sees only the concrete set of instantiated rules. + + +Legacy macros are unstructured functions called from `BUILD` files that can +create targets. By the end of the +[loading phase](/extending/concepts#evaluation-model), legacy macros don't exist +anymore, and Bazel sees only the concrete set of instantiated rules. ## Why you shouldn't use legacy macros (and should use Symbolic macros instead) @@ -10,30 +15,38 @@ Where possible you should use [symbolic macros](macros.md#macros). Symbolic macros -- Prevent action at a distance -- Make it possible to hide implementation details through granular visibility -- Take typed attributes, which in turn means automatic label and select conversion. -- Are more readable -- Will soon have [lazy evaluation](macros.md#laziness) +* Prevent action at a distance +* Make it possible to hide implementation details through granular visibility +* Take typed attributes, which in turn means automatic label and select + conversion. +* Are more readable +* Will soon have [lazy evaluation](macros.md#laziness) ## Usage The typical use case for a macro is when you want to reuse a rule. -For example, genrule in a `BUILD` file generates a file using `//:generator` with a `some_arg` argument hardcoded in the command: +For example, genrule in a `BUILD` file generates a file using `//:generator` +with a `some_arg` argument hardcoded in the command: ```python genrule( name = "file", outs = ["file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", tools = ["//:generator"], ) ``` -Note: `$@` is a [Make variable](/reference/be/make-variables#predefined_genrule_variables) that refers to the execution-time locations of the files in the `outs` attribute list. It is equivalent to `$(locations :file.txt)`. +Note: `$@` is a +[Make variable](/reference/be/make-variables#predefined_genrule_variables) that +refers to the execution-time locations of the files in the `outs` attribute +list. It is equivalent to `$(locations :file.txt)`. -If you want to generate more files with different arguments, you may want to extract this code to a macro function. To create a macro called `file_generator`, which has `name` and `arg` parameters, we can replace the genrule with the following: +If you want to generate more files with different arguments, you may want to +extract this code to a macro function. To create a macro called +`file_generator`, which has `name` and `arg` parameters, we can replace the +genrule with the following: ```python load("//path:generator.bzl", "file_generator") @@ -54,22 +67,27 @@ file_generator( ) ``` -Here, you load the `file_generator` symbol from a `.bzl` file located in the `//path` package. By putting macro function definitions in a separate `.bzl` file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be loaded from any package in the workspace. +Here, you load the `file_generator` symbol from a `.bzl` file located in the +`//path` package. By putting macro function definitions in a separate `.bzl` +file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be +loaded from any package in the workspace. -Finally, in `path/generator.bzl`, write the definition of the macro to encapsulate and parameterize the original genrule definition: +Finally, in `path/generator.bzl`, write the definition of the macro to +encapsulate and parameterize the original genrule definition: ```python def file_generator(name, arg, visibility=None): native.genrule( name = name, outs = [name + ".txt"], - cmd = "$(location //:generator) %s > $@" % arg, + cmd = "$(location //:generator) %s > $@" % arg, tools = ["//:generator"], visibility = visibility, ) ``` -You can also use macros to chain rules together. This example shows chained genrules, where a genrule uses the outputs of a previous genrule as inputs: +You can also use macros to chain rules together. This example shows chained +genrules, where a genrule uses the outputs of a previous genrule as inputs: ```python def chained_genrules(name, visibility=None): @@ -85,19 +103,23 @@ def chained_genrules(name, visibility=None): name = name + "-two", srcs = [name + ".one"], outs = [name + ".two"], - cmd = "$(location :tool-two) $< $@", + cmd = "$(location :tool-two) $< $@", tools = [":tool-two"], visibility = visibility, ) ``` -The example only assigns a visibility value to the second genrule. This allows macro authors to hide the outputs of intermediate rules from being depended upon by other targets in the workspace. +The example only assigns a visibility value to the second genrule. This allows +macro authors to hide the outputs of intermediate rules from being depended upon +by other targets in the workspace. -Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the `srcs` attribute list. +Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the +`srcs` attribute list. ## Expanding macros -When you want to investigate what a macro does, use the `query` command with `--output=build` to see the expanded form: +When you want to investigate what a macro does, use the `query` command with +`--output=build` to see the expanded form: ```none $ bazel query --output=build :file @@ -106,13 +128,14 @@ genrule( name = "file", tools = ["//:generator"], outs = ["//test:file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", ) ``` ## Instantiating native rules -Native rules (rules that don't need a `load()` statement) can be instantiated from the [native](/rules/lib/toplevel/native) module: +Native rules (rules that don't need a `load()` statement) can be instantiated +from the [native](/rules/lib/toplevel/native) module: ```python def my_macro(name, visibility=None): @@ -123,13 +146,23 @@ def my_macro(name, visibility=None): ) ``` -If you need to know the package name (for example, which `BUILD` file is calling the macro), use the function [native.package\_name()](/rules/lib/toplevel/native#package_name). Note that `native` can only be used in `.bzl` files, and not in `BUILD` files. +If you need to know the package name (for example, which `BUILD` file is calling +the macro), use the function +[native.package_name()](/rules/lib/toplevel/native#package_name). Note that +`native` can only be used in `.bzl` files, and not in `BUILD` files. ## Label resolution in macros -Since legacy macros are evaluated in the [loading phase](concepts.md#evaluation-model), label strings such as `"//foo:bar"` that occur in a legacy macro are interpreted relative to the `BUILD` file in which the macro is used rather than relative to the `.bzl` file in which it is defined. This behavior is generally undesirable for macros that are meant to be used in other repositories, such as because they are part of a published Starlark ruleset. +Since legacy macros are evaluated in the +[loading phase](concepts.md#evaluation-model), label strings such as +`"//foo:bar"` that occur in a legacy macro are interpreted relative to the +`BUILD` file in which the macro is used rather than relative to the `.bzl` file +in which it is defined. This behavior is generally undesirable for macros that +are meant to be used in other repositories, such as because they are part of a +published Starlark ruleset. -To get the same behavior as for Starlark rules, wrap the label strings with the [`Label`](/rules/lib/builtins/Label#Label) constructor: +To get the same behavior as for Starlark rules, wrap the label strings with the +[`Label`](/rules/lib/builtins/Label#Label) constructor: ```python # @my_ruleset//rules:defs.bzl @@ -151,39 +184,71 @@ def my_cc_wrapper(name, deps = [], **kwargs): ) ``` -With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys that are label strings will be automatically resolved to `Label` objects relative to the package of the file that contains the `select` call. If this is not chosen, wrap the label string with [native.package\_relative\_label()](/rules/lib/toplevel/native#package_relative_label). +With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys +that are label strings will be automatically resolved to `Label` objects +relative to the package of the file that contains the `select` call. If this is +not chosen, wrap the label string with +[native.package_relative_label()](/rules/lib/toplevel/native#package_relative_label). ## Debugging -- `bazel query --output=build //my/path:all` will show you how the `BUILD` file looks after evaluation. All legacy macros, globs, loops are expanded. Known limitation: `select` expressions are not shown in the output. +* `bazel query --output=build //my/path:all` will show you how the `BUILD` + file looks after evaluation. All legacy macros, globs, loops are expanded. + Known limitation: `select` expressions are not shown in the output. -- You may filter the output based on `generator_function` (which function generated the rules) or `generator_name` (the name attribute of the macro): `bash $ bazel query --output=build 'attr(generator_function, my_macro, //my/path:all)'` +* You may filter the output based on `generator_function` (which function + generated the rules) or `generator_name` (the name attribute of the macro): + `bash $ bazel query --output=build 'attr(generator_function, my_macro, + //my/path:all)'` -- To find out where exactly the rule `foo` is generated in a `BUILD` file, you can try the following trick. Insert this line near the top of the `BUILD` file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when the rule `foo` is created (due to a name conflict), which will show you the full stack trace. +* To find out where exactly the rule `foo` is generated in a `BUILD` file, you + can try the following trick. Insert this line near the top of the `BUILD` + file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when + the rule `foo` is created (due to a name conflict), which will show you the + full stack trace. -- You can also use [print](/rules/lib/globals/all#print) for debugging. It displays the message as a `DEBUG` log line during the loading phase. Except in rare cases, either remove `print` calls, or make them conditional under a `debugging` parameter that defaults to `False` before submitting the code to the depot. +* You can also use [print](/rules/lib/globals/all#print) for debugging. It + displays the message as a `DEBUG` log line during the loading phase. Except + in rare cases, either remove `print` calls, or make them conditional under a + `debugging` parameter that defaults to `False` before submitting the code to + the depot. ## Errors -If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) function. Explain clearly to the user what went wrong and how to fix their `BUILD` file. It is not possible to catch an error. +If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) +function. Explain clearly to the user what went wrong and how to fix their +`BUILD` file. It is not possible to catch an error. ```python def my_macro(name, deps, visibility=None): - if len(deps) < 2: + if len(deps) < 2: fail("Expected at least two values in deps") # ... ``` ## Conventions -- All public functions (functions that don't start with underscore) that instantiate rules must have a `name` argument. This argument should not be optional (don't give a default value). +* All public functions (functions that don't start with underscore) that + instantiate rules must have a `name` argument. This argument should not be + optional (don't give a default value). -- Public functions should use a docstring following [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). +* Public functions should use a docstring following + [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). -- In `BUILD` files, the `name` argument of the macros must be a keyword argument (not a positional argument). +* In `BUILD` files, the `name` argument of the macros must be a keyword + argument (not a positional argument). -- The `name` attribute of rules generated by a macro should include the name argument as a prefix. For example, `macro(name = "foo")` can generate a `cc_library` `foo` and a genrule `foo_gen`. +* The `name` attribute of rules generated by a macro should include the name + argument as a prefix. For example, `macro(name = "foo")` can generate a + `cc_library` `foo` and a genrule `foo_gen`. -- In most cases, optional parameters should have a default value of `None`. `None` can be passed directly to native rules, which treat it the same as if you had not passed in any argument. Thus, there is no need to replace it with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer to the rules it creates, as their defaults may be complex or may change over time. Additionally, a parameter that is explicitly set to its default value looks different than one that is never set (or set to `None`) when accessed through the query language or build-system internals. +* In most cases, optional parameters should have a default value of `None`. + `None` can be passed directly to native rules, which treat it the same as if + you had not passed in any argument. Thus, there is no need to replace it + with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer + to the rules it creates, as their defaults may be complex or may change over + time. Additionally, a parameter that is explicitly set to its default value + looks different than one that is never set (or set to `None`) when accessed + through the query language or build-system internals. -- Macros should have an optional `visibility` argument. +* Macros should have an optional `visibility` argument. diff --git a/extending/macros.mdx b/extending/macros.mdx index 1e4ac1e3..136665d0 100644 --- a/extending/macros.mdx +++ b/extending/macros.mdx @@ -2,23 +2,42 @@ title: 'Macros' --- -This page covers the basics of using macros and includes typical use cases, debugging, and conventions. -A macro is a function called from the `BUILD` file that can instantiate rules. Macros are mainly used for encapsulation and code reuse of existing rules and other macros. -Macros come in two flavors: symbolic macros, which are described on this page, and [legacy macros](legacy-macros.md). Where possible, we recommend using symbolic macros for code clarity. +This page covers the basics of using macros and includes typical use cases, +debugging, and conventions. -Symbolic macros offer typed arguments (string to label conversion, relative to where the macro was called) and the ability to restrict and specify the visibility of targets created. They are designed to be amenable to lazy evaluation (which will be added in a future Bazel release). Symbolic macros are available by default in Bazel 8. Where this document mentions `macros`, it's referring to **symbolic macros**. +A macro is a function called from the `BUILD` file that can instantiate rules. +Macros are mainly used for encapsulation and code reuse of existing rules and +other macros. -An executable example of symbolic macros can be found in the [examples repository](https://github.com/bazelbuild/examples/tree/main/macros). +Macros come in two flavors: symbolic macros, which are described on this page, +and [legacy macros](legacy-macros.md). Where possible, we recommend using +symbolic macros for code clarity. + +Symbolic macros offer typed arguments (string to label conversion, relative to +where the macro was called) and the ability to restrict and specify the +visibility of targets created. They are designed to be amenable to lazy +evaluation (which will be added in a future Bazel release). Symbolic macros are +available by default in Bazel 8. Where this document mentions `macros`, it's +referring to **symbolic macros**. + +An executable example of symbolic macros can be found in the +[examples repository](https://github.com/bazelbuild/examples/tree/main/macros). ## Usage -Macros are defined in `.bzl` files by calling the [`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with two required parameters: `attrs` and `implementation`. +Macros are defined in `.bzl` files by calling the +[`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with +two required parameters: `attrs` and `implementation`. ### Attributes -`attrs` accepts a dictionary of attribute name to [attribute types](https://bazel.build/rules/lib/toplevel/attr#members), which represents the arguments to the macro. Two common attributes – `name` and `visibility` – are implicitly added to all macros and are not included in the dictionary passed to `attrs`. +`attrs` accepts a dictionary of attribute name to [attribute +types](https://bazel.build/rules/lib/toplevel/attr#members), which represents +the arguments to the macro. Two common attributes – `name` and `visibility` – +are implicitly added to all macros and are not included in the dictionary passed +to `attrs`. ```starlark # macro/macro.bzl @@ -31,13 +50,31 @@ my_macro = macro( ) ``` -Attribute type declarations accept the [parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), `mandatory`, `default`, and `doc`. Most attribute types also accept the `configurable` parameter, which determines whether the attribute accepts `select`s. If an attribute is `configurable`, it will parse non-`select` values as an unconfigurable `select` – `"foo"` will become `select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). +Attribute type declarations accept the +[parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), +`mandatory`, `default`, and `doc`. Most attribute types also accept the +`configurable` parameter, which determines whether the attribute accepts +`select`s. If an attribute is `configurable`, it will parse non-`select` values +as an unconfigurable `select` – `"foo"` will become +`select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). #### Attribute inheritance -Macros are often intended to wrap a rule (or another macro), and the macro's author often wants to forward the bulk of the wrapped symbol's attributes unchanged, using `**kwargs`, to the macro's main target (or main inner macro). - -To support this pattern, a macro can *inherit attributes* from a rule or another macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or [macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s `inherit_attrs` argument. (You can also use the special string `"common"` instead of a rule or macro symbol to inherit the [common attributes defined for all Starlark build rules](https://bazel.build/reference/be/common-definitions#common-attributes).) Only public attributes get inherited, and the attributes in the macro's own `attrs` dictionary override inherited attributes with the same name. You can also *remove* inherited attributes by using `None` as a value in the `attrs` dictionary: +Macros are often intended to wrap a rule (or another macro), and the macro's +author often wants to forward the bulk of the wrapped symbol's attributes +unchanged, using `**kwargs`, to the macro's main target (or main inner macro). + +To support this pattern, a macro can *inherit attributes* from a rule or another +macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or +[macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s +`inherit_attrs` argument. (You can also use the special string `"common"` +instead of a rule or macro symbol to inherit the [common attributes defined for +all Starlark build +rules](https://bazel.build/reference/be/common-definitions#common-attributes).) +Only public attributes get inherited, and the attributes in the macro's own +`attrs` dictionary override inherited attributes with the same name. You can +also *remove* inherited attributes by using `None` as a value in the `attrs` +dictionary: ```starlark # macro/macro.bzl @@ -53,7 +90,11 @@ my_macro = macro( ) ``` -The default value of non-mandatory inherited attributes is always overridden to be `None`, regardless of the original attribute definition's default value. If you need to examine or modify an inherited non-mandatory attribute – for example, if you want to add a tag to an inherited `tags` attribute – you must make sure to handle the `None` case in your macro's implementation function: +The default value of non-mandatory inherited attributes is always overridden to +be `None`, regardless of the original attribute definition's default value. If +you need to examine or modify an inherited non-mandatory attribute – for +example, if you want to add a tag to an inherited `tags` attribute – you must +make sure to handle the `None` case in your macro's implementation function: ```starlark # macro/macro.bzl @@ -71,9 +112,15 @@ def _my_macro_impl(name, visibility, tags, **kwargs): ### Implementation -`implementation` accepts a function which contains the logic of the macro. Implementation functions often create targets by calling one or more rules, and they are usually private (named with a leading underscore). Conventionally, they are named the same as their macro, but prefixed with `_` and suffixed with `_impl`. +`implementation` accepts a function which contains the logic of the macro. +Implementation functions often create targets by calling one or more rules, and +they are usually private (named with a leading underscore). Conventionally, +they are named the same as their macro, but prefixed with `_` and suffixed with +`_impl`. -Unlike rule implementation functions, which take a single argument (`ctx`) that contains a reference to the attributes, macro implementation functions accept a parameter for each argument. +Unlike rule implementation functions, which take a single argument (`ctx`) that +contains a reference to the attributes, macro implementation functions accept a +parameter for each argument. ```starlark # macro/macro.bzl @@ -91,7 +138,11 @@ def _my_macro_impl(name, visibility, deps, create_test): ) ``` -If a macro inherits attributes, its implementation function *must* have a `**kwargs` residual keyword parameter, which can be forwarded to the call that invokes the inherited rule or submacro. (This helps ensure that your macro won't be broken if the rule or macro which from which you are inheriting adds a new attribute.) +If a macro inherits attributes, its implementation function *must* have a +`**kwargs` residual keyword parameter, which can be forwarded to the call that +invokes the inherited rule or submacro. (This helps ensure that your macro won't +be broken if the rule or macro which from which you are inheriting adds a new +attribute.) ### Declaration @@ -113,9 +164,12 @@ my_macro( ) ``` -This would create targets `//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. +This would create targets +`//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. -Just like in rule calls, if an attribute value in a macro call is set to `None`, that attribute is treated as if it was omitted by the macro's caller. For example, the following two macro calls are equivalent: +Just like in rule calls, if an attribute value in a macro call is set to `None`, +that attribute is treated as if it was omitted by the macro's caller. For +example, the following two macro calls are equivalent: ```starlark # pkg/BUILD @@ -123,17 +177,27 @@ my_macro(name = "abc", srcs = ["src.cc"], deps = None) my_macro(name = "abc", srcs = ["src.cc"]) ``` -This is generally not useful in `BUILD` files, but is helpful when programmatically wrapping a macro inside another macro. +This is generally not useful in `BUILD` files, but is helpful when +programmatically wrapping a macro inside another macro. ## Details ### Naming conventions for targets created -The names of any targets or submacros created by a symbolic macro must either match the macro's `name` parameter or must be prefixed by `name` followed by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, for example, `foo_bar`. +The names of any targets or submacros created by a symbolic macro must +either match the macro's `name` parameter or must be prefixed by `name` followed +by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only +create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, +for example, `foo_bar`. -Targets or files that violate macro naming convention can be declared, but cannot be built and cannot be used as dependencies. +Targets or files that violate macro naming convention can be declared, but +cannot be built and cannot be used as dependencies. -Non-macro files and targets within the same package as a macro instance should *not* have names that conflict with potential macro target names, though this exclusivity is not enforced. We are in the progress of implementing [lazy evaluation](#laziness) as a performance improvement for Symbolic macros, which will be impaired in packages that violate the naming schema. +Non-macro files and targets within the same package as a macro instance should +*not* have names that conflict with potential macro target names, though this +exclusivity is not enforced. We are in the progress of implementing +[lazy evaluation](#laziness) as a performance improvement for Symbolic macros, +which will be impaired in packages that violate the naming schema. ### Restrictions @@ -141,35 +205,54 @@ Symbolic macros have some additional restrictions compared to legacy macros. Symbolic macros -- must take a `name` argument and a `visibility` argument -- must have an `implementation` function -- may not return values -- may not mutate their arguments -- may not call `native.existing_rules()` unless they are special `finalizer` macros -- may not call `native.package()` -- may not call `glob()` -- may not call `native.environment_group()` -- must create targets whose names adhere to the [naming schema](#naming) -- can't refer to input files that weren't declared or passed in as an argument -- can't refer to private targets of their callers (see [visibility and macros](#visibility) for more details). +* must take a `name` argument and a `visibility` argument +* must have an `implementation` function +* may not return values +* may not mutate their arguments +* may not call `native.existing_rules()` unless they are special `finalizer` + macros +* may not call `native.package()` +* may not call `glob()` +* may not call `native.environment_group()` +* must create targets whose names adhere to the [naming schema](#naming) +* can't refer to input files that weren't declared or passed in as an argument +* can't refer to private targets of their callers (see + [visibility and macros](#visibility) for more details). ### Visibility and macros -The [visibility](/concepts/visibility) system helps protect the implementation details of both (symbolic) macros and their callers. +The [visibility](/concepts/visibility) system helps protect the implementation +details of both (symbolic) macros and their callers. -By default, targets created in a symbolic macro are visible within the macro itself, but not necessarily to the macro's caller. The macro can "export" a target as a public API by forwarding the value of its own `visibility` attribute, as in `some_rule(..., visibility = visibility)`. +By default, targets created in a symbolic macro are visible within the macro +itself, but not necessarily to the macro's caller. The macro can "export" a +target as a public API by forwarding the value of its own `visibility` +attribute, as in `some_rule(..., visibility = visibility)`. The key ideas of macro visibility are: -1. Visibility is checked based on what macro declared the target, not what package called the macro. +1. Visibility is checked based on what macro declared the target, not what + package called the macro. - - In other words, being in the same package does not by itself make one target visible to another. This protects the macro's internal targets from becoming dependencies of other macros or top-level targets in the package. + * In other words, being in the same package does not by itself make one + target visible to another. This protects the macro's internal targets + from becoming dependencies of other macros or top-level targets in the + package. -2. All `visibility` attributes, on both rules and macros, automatically include the place where the rule or macro was called. +1. All `visibility` attributes, on both rules and macros, automatically + include the place where the rule or macro was called. - - Thus, a target is unconditionally visible to other targets declared in the same macro (or the `BUILD` file, if not in a macro). + * Thus, a target is unconditionally visible to other targets declared in the + same macro (or the `BUILD` file, if not in a macro). -In practice, this means that when a macro declares a target without setting its `visibility`, the target defaults to being internal to the macro. (The package's [default visibility](/reference/be/functions#package.default_visibility) does not apply within a macro.) Exporting the target means that the target is visible to whatever the macro's caller specified in the macro's `visibility` attribute, plus the package of the macro's caller itself, as well as the macro's own code. Another way of thinking of it is that the visibility of a macro determines who (aside from the macro itself) can see the macro's exported targets. +In practice, this means that when a macro declares a target without setting its +`visibility`, the target defaults to being internal to the macro. (The package's +[default visibility](/reference/be/functions#package.default_visibility) does +not apply within a macro.) Exporting the target means that the target is visible +to whatever the macro's caller specified in the macro's `visibility` attribute, +plus the package of the macro's caller itself, as well as the macro's own code. +Another way of thinking of it is that the visibility of a macro determines who +(aside from the macro itself) can see the macro's exported targets. ```starlark # tool/BUILD @@ -231,25 +314,51 @@ some_rule( ) ``` -If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if the `//pkg` package had set its `default_visibility` to that value, then `//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain protected. - -A macro can declare that a target is visible to a friend package by passing `visibility = ["//some_friend:__pkg__"]` (for an internal target) or `visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). Note that it is an antipattern for a macro to declare a target with public visibility (`visibility = ["//visibility:public"]`). This is because it makes the target unconditionally visible to every package, even if the caller specified a more restricted visibility. - -All visibility checking is done with respect to the innermost currently running symbolic macro. However, there is a visibility delegation mechanism: If a macro passes a label as an attribute value to an inner macro, any usages of the label in the inner macro are checked with respect to the outer macro. See the [visibility page](/concepts/visibility#symbolic-macros) for more details. - -Remember that legacy macros are entirely transparent to the visibility system, and behave as though their location is whatever BUILD file or symbolic macro they were called from. +If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if +the `//pkg` package had set its `default_visibility` to that value, then +`//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a +macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain +protected. + +A macro can declare that a target is visible to a friend package by passing +`visibility = ["//some_friend:__pkg__"]` (for an internal target) or +`visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). +Note that it is an antipattern for a macro to declare a target with public +visibility (`visibility = ["//visibility:public"]`). This is because it makes +the target unconditionally visible to every package, even if the caller +specified a more restricted visibility. + +All visibility checking is done with respect to the innermost currently running +symbolic macro. However, there is a visibility delegation mechanism: If a macro +passes a label as an attribute value to an inner macro, any usages of the label +in the inner macro are checked with respect to the outer macro. See the +[visibility page](/concepts/visibility#symbolic-macros) for more details. + +Remember that legacy macros are entirely transparent to the visibility system, +and behave as though their location is whatever BUILD file or symbolic macro +they were called from. #### Finalizers and visibility -Targets declared in a rule finalizer, in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. +Targets declared in a rule finalizer, in addition to seeing targets following +the usual symbolic macro visibility rules, can *also* see all targets which are +visible to the finalizer target's package. -This means that if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. +This means that if you migrate a `native.existing_rules()`-based legacy macro to +a finalizer, the targets declared by the finalizer will still be able to see +their old dependencies. -However, note that it's possible to declare a target in a symbolic macro such that a finalizer's targets cannot see it under the visibility system – even though the finalizer can *introspect* its attributes using `native.existing_rules()`. +However, note that it's possible to declare a target in a symbolic macro such +that a finalizer's targets cannot see it under the visibility system – even +though the finalizer can *introspect* its attributes using +`native.existing_rules()`. ### Selects -If an attribute is `configurable` (the default) and its value is not `None`, then the macro implementation function will see the attribute value as wrapped in a trivial `select`. This makes it easier for the macro author to catch bugs where they did not anticipate that the attribute value could be a `select`. +If an attribute is `configurable` (the default) and its value is not `None`, +then the macro implementation function will see the attribute value as wrapped +in a trivial `select`. This makes it easier for the macro author to catch bugs +where they did not anticipate that the attribute value could be a `select`. For example, consider the following macro: @@ -260,15 +369,38 @@ my_macro = macro( ) ``` -If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` to be invoked with its `deps` parameter set to `select({"//conditions:default": ["//a"]})`. If this causes the implementation function to fail (say, because the code tried to index into the value as in `deps[0]`, which is not allowed for `select`s), the macro author can then make a choice: either they can rewrite their macro to only use operations compatible with `select`, or they can mark the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The latter ensures that users are not permitted to pass a `select` value in. - -Rule targets reverse this transformation, and store trivial `select`s as their unconditional values; in the above example, if `_my_macro_impl` declares a rule target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as `["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` values to be stored in all targets instantiated by macros. - -If the value of a configurable attribute is `None`, it does not get wrapped in a `select`. This ensures that tests like `my_attr == None` still work, and that when the attribute is forwarded to a rule with a computed default, the rule behaves properly (that is, as if the attribute were not passed in at all). It is not always possible for an attribute to take on a `None` value, but it can happen for the `attr.label()` type, and for any inherited non-mandatory attribute. +If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` +to be invoked with its `deps` parameter set to `select({"//conditions:default": +["//a"]})`. If this causes the implementation function to fail (say, because the +code tried to index into the value as in `deps[0]`, which is not allowed for +`select`s), the macro author can then make a choice: either they can rewrite +their macro to only use operations compatible with `select`, or they can mark +the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The +latter ensures that users are not permitted to pass a `select` value in. + +Rule targets reverse this transformation, and store trivial `select`s as their +unconditional values; in the above example, if `_my_macro_impl` declares a rule +target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as +`["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` +values to be stored in all targets instantiated by macros. + +If the value of a configurable attribute is `None`, it does not get wrapped in a +`select`. This ensures that tests like `my_attr == None` still work, and that +when the attribute is forwarded to a rule with a computed default, the rule +behaves properly (that is, as if the attribute were not passed in at all). It is +not always possible for an attribute to take on a `None` value, but it can +happen for the `attr.label()` type, and for any inherited non-mandatory +attribute. ## Finalizers -A rule finalizer is a special symbolic macro which – regardless of its lexical position in a BUILD file – is evaluated in the final stage of loading a package, after all non-finalizer targets have been defined. Unlike ordinary symbolic macros, a finalizer can call `native.existing_rules()`, where it behaves slightly differently than in legacy macros: it only returns the set of non-finalizer rule targets. The finalizer may assert on the state of that set or define new targets. +A rule finalizer is a special symbolic macro which – regardless of its lexical +position in a BUILD file – is evaluated in the final stage of loading a package, +after all non-finalizer targets have been defined. Unlike ordinary symbolic +macros, a finalizer can call `native.existing_rules()`, where it behaves +slightly differently than in legacy macros: it only returns the set of +non-finalizer rule targets. The finalizer may assert on the state of that set or +define new targets. To declare a finalizer, call `macro()` with `finalizer = True`: @@ -294,17 +426,24 @@ my_finalizer = macro( ## Laziness -IMPORTANT: We are in the process of implementing lazy macro expansion and evaluation. This feature is not available yet. +IMPORTANT: We are in the process of implementing lazy macro expansion and +evaluation. This feature is not available yet. -Currently, all macros are evaluated as soon as the BUILD file is loaded, which can negatively impact performance for targets in packages that also have costly unrelated macros. In the future, non-finalizer symbolic macros will only be evaluated if they're required for the build. The prefix naming schema helps Bazel determine which macro to expand given a requested target. +Currently, all macros are evaluated as soon as the BUILD file is loaded, which +can negatively impact performance for targets in packages that also have costly +unrelated macros. In the future, non-finalizer symbolic macros will only be +evaluated if they're required for the build. The prefix naming schema helps +Bazel determine which macro to expand given a requested target. ## Migration troubleshooting Here are some common migration headaches and how to fix them. -- Legacy macro calls `glob()` +* Legacy macro calls `glob()` -Move the `glob()` call to your BUILD file (or to a legacy macro called from the BUILD file), and pass the `glob()` value to the symbolic macro using a label-list attribute: +Move the `glob()` call to your BUILD file (or to a legacy macro called from the +BUILD file), and pass the `glob()` value to the symbolic macro using a +label-list attribute: ```starlark # BUILD file @@ -314,10 +453,13 @@ my_macro( ) ``` -- Legacy macro has a parameter that isn't a valid starlark `attr` type. +* Legacy macro has a parameter that isn't a valid starlark `attr` type. + +Pull as much logic as possible into a nested symbolic macro, but keep the +top level macro a legacy macro. -Pull as much logic as possible into a nested symbolic macro, but keep the top level macro a legacy macro. +* Legacy macro calls a rule that creates a target that breaks the naming schema -- Legacy macro calls a rule that creates a target that breaks the naming schema +That's okay, just don't depend on the "offending" target. The naming check will +be quietly ignored. -That's okay, just don't depend on the "offending" target. The naming check will be quietly ignored. diff --git a/extending/platforms.mdx b/extending/platforms.mdx index e6e349fd..94e6290f 100644 --- a/extending/platforms.mdx +++ b/extending/platforms.mdx @@ -2,29 +2,60 @@ title: 'Platforms' --- -Bazel can build and test code on a variety of hardware, operating systems, and system configurations, using many different versions of build tools such as linkers and compilers. To help manage this complexity, Bazel has a concept of *constraints* and *platforms*. A constraint is a dimension in which build or production environments may differ, such as CPU architecture, the presence or absence of a GPU, or the version of a system-installed compiler. A platform is a named collection of choices for these constraints, representing the particular resources that are available in some environment. -Modeling the environment as a platform helps Bazel to automatically select the appropriate [toolchains](/extending/toolchains) for build actions. Platforms can also be used in combination with the [config\_setting](/reference/be/general#config_setting) rule to write [configurable attributes](/docs/configurable-attributes). + +Bazel can build and test code on a variety of hardware, operating systems, and +system configurations, using many different versions of build tools such as +linkers and compilers. To help manage this complexity, Bazel has a concept of +*constraints* and *platforms*. A constraint is a dimension in which build or +production environments may differ, such as CPU architecture, the presence or +absence of a GPU, or the version of a system-installed compiler. A platform is a +named collection of choices for these constraints, representing the particular +resources that are available in some environment. + +Modeling the environment as a platform helps Bazel to automatically select the +appropriate +[toolchains](/extending/toolchains) +for build actions. Platforms can also be used in combination with the +[config_setting](/reference/be/general#config_setting) +rule to write [configurable attributes](/docs/configurable-attributes). Bazel recognizes three roles that a platform may serve: -- **Host** - the platform on which Bazel itself runs. -- **Execution** - a platform on which build tools execute build actions to produce intermediate and final outputs. -- **Target** - a platform on which a final output resides and executes. +* **Host** - the platform on which Bazel itself runs. +* **Execution** - a platform on which build tools execute build actions to + produce intermediate and final outputs. +* **Target** - a platform on which a final output resides and executes. Bazel supports the following build scenarios regarding platforms: -- **Single-platform builds** (default) - host, execution, and target platforms are the same. For example, building a Linux executable on Ubuntu running on an Intel x64 CPU. +* **Single-platform builds** (default) - host, execution, and target platforms + are the same. For example, building a Linux executable on Ubuntu running on + an Intel x64 CPU. -- **Cross-compilation builds** - host and execution platforms are the same, but the target platform is different. For example, building an iOS app on macOS running on a MacBook Pro. +* **Cross-compilation builds** - host and execution platforms are the same, but + the target platform is different. For example, building an iOS app on macOS + running on a MacBook Pro. -- **Multi-platform builds** - host, execution, and target platforms are all different. +* **Multi-platform builds** - host, execution, and target platforms are all + different. -Tip: for detailed instructions on migrating your project to platforms, see [Migrating to Platforms](/concepts/platforms). +Tip: for detailed instructions on migrating your project to platforms, see +[Migrating to Platforms](/concepts/platforms). ## Defining constraints and platforms -The space of possible choices for platforms is defined by using the [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) and [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) rules within `BUILD` files. `constraint_setting` creates a new dimension, while `constraint_value` creates a new value for a given dimension; together they effectively define an enum and its possible values. For example, the following snippet of a `BUILD` file introduces a constraint for the system's glibc version with two possible values. +The space of possible choices for platforms is defined by using the +[`constraint_setting`][constraint_setting] and +[`constraint_value`][constraint_value] rules within `BUILD` files. +`constraint_setting` creates a new dimension, while +`constraint_value` creates a new value for a given dimension; together they +effectively define an enum and its possible values. For example, the following +snippet of a `BUILD` file introduces a constraint for the system's glibc version +with two possible values. + +[constraint_setting]: /reference/be/platforms-and-toolchains#constraint_setting +[constraint_value]: /reference/be/platforms-and-toolchains#constraint_value ```python constraint_setting(name = "glibc_version") @@ -40,9 +71,16 @@ constraint_value( ) ``` -Constraints and their values may be defined across different packages in the workspace. They are referenced by label and subject to the usual visibility controls. If visibility allows, you can extend an existing constraint setting by defining your own value for it. +Constraints and their values may be defined across different packages in the +workspace. They are referenced by label and subject to the usual visibility +controls. If visibility allows, you can extend an existing constraint setting by +defining your own value for it. -The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with certain choices of constraint values. The following creates a platform named `linux_x86`, and says that it describes any environment that runs a Linux operating system on an x86\_64 architecture with a glibc version of 2.25. (See below for more on Bazel's built-in constraints.) +The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with +certain choices of constraint values. The +following creates a platform named `linux_x86`, and says that it describes any +environment that runs a Linux operating system on an x86_64 architecture with a +glibc version of 2.25. (See below for more on Bazel's built-in constraints.) ```python platform( @@ -55,33 +93,52 @@ platform( ) ``` -Note: It is an error for a platform to specify more than one value of the same constraint setting, such as `@platforms//cpu:x86_64` and `@platforms//cpu:arm` for `@platforms//cpu:cpu`. +Note: It is an error for a platform to specify more than one value of the +same constraint setting, such as `@platforms//cpu:x86_64` and +`@platforms//cpu:arm` for `@platforms//cpu:cpu`. ## Generally useful constraints and platforms -To keep the ecosystem consistent, Bazel team maintains a repository with constraint definitions for the most popular CPU architectures and operating systems. These are all located in [https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). +To keep the ecosystem consistent, Bazel team maintains a repository with +constraint definitions for the most popular CPU architectures and operating +systems. These are all located in +[https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). -Bazel ships with the following special platform definition: `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the autodetected host platform value - represents autodetected platform for the system Bazel is running on. +Bazel ships with the following special platform definition: +`@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the +autodetected host platform value - +represents autodetected platform for the system Bazel is running on. ## Specifying a platform for a build -You can specify the host and target platforms for a build using the following command-line flags: - -- `--host_platform` - defaults to `@bazel_tools//tools:host_platform` - - - This target is aliased to `@platforms//host`, which is backed by a repo rule that detects the host OS and CPU and writes the platform target. - - There's also `@platforms//host:constraints.bzl`, which exposes an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and Starlark files. - -- `--platforms` - defaults to the host platform - - - This means that when no other flags are set, `@platforms//host` is the target platform. - - If `--host_platform` is set and not `--platforms`, the value of `--host_platform` is both the host and target platform. +You can specify the host and target platforms for a build using the following +command-line flags: + +* `--host_platform` - defaults to `@bazel_tools//tools:host_platform` + * This target is aliased to `@platforms//host`, which is backed by a repo + rule that detects the host OS and CPU and writes the platform target. + * There's also `@platforms//host:constraints.bzl`, which exposes + an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and + Starlark files. +* `--platforms` - defaults to the host platform + * This means that when no other flags are set, + `@platforms//host` is the target platform. + * If `--host_platform` is set and not `--platforms`, the value of + `--host_platform` is both the host and target platform. ## Skipping incompatible targets -When building for a specific target platform it is often desirable to skip targets that will never work on that platform. For example, your Windows device driver is likely going to generate lots of compiler errors when building on a Linux machine with `//...`. Use the [`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) attribute to tell Bazel what target platform constraints your code has. +When building for a specific target platform it is often desirable to skip +targets that will never work on that platform. For example, your Windows device +driver is likely going to generate lots of compiler errors when building on a +Linux machine with `//...`. Use the +[`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) +attribute to tell Bazel what target platform constraints your code has. -The simplest use of this attribute restricts a target to a single platform. The target will not be built for any platform that doesn't satisfy all of the constraints. The following example restricts `win_driver_lib.cc` to 64-bit Windows. +The simplest use of this attribute restricts a target to a single platform. +The target will not be built for any platform that doesn't satisfy all of the +constraints. The following example restricts `win_driver_lib.cc` to 64-bit +Windows. ```python cc_library( @@ -94,11 +151,16 @@ cc_library( ) ``` -`:win_driver_lib` is *only* compatible for building with 64-bit Windows and incompatible with all else. Incompatibility is transitive. Any targets that transitively depend on an incompatible target are themselves considered incompatible. +`:win_driver_lib` is *only* compatible for building with 64-bit Windows and +incompatible with all else. Incompatibility is transitive. Any targets +that transitively depend on an incompatible target are themselves considered +incompatible. ### When are targets skipped? -Targets are skipped when they are considered incompatible and included in the build as part of a target pattern expansion. For example, the following two invocations skip any incompatible targets found in a target pattern expansion. +Targets are skipped when they are considered incompatible and included in the +build as part of a target pattern expansion. For example, the following two +invocations skip any incompatible targets found in a target pattern expansion. ```console $ bazel build --platforms=//:myplatform //... @@ -108,9 +170,15 @@ $ bazel build --platforms=//:myplatform //... $ bazel build --platforms=//:myplatform //:all ``` -Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are similarly skipped if the `test_suite` is specified on the command line with [`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). In other words, `test_suite` targets on the command line behave like `:all` and `...`. Using `--noexpand_test_suites` prevents expansion and causes `test_suite` targets with incompatible tests to also be incompatible. +Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are +similarly skipped if the `test_suite` is specified on the command line with +[`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). +In other words, `test_suite` targets on the command line behave like `:all` and +`...`. Using `--noexpand_test_suites` prevents expansion and causes +`test_suite` targets with incompatible tests to also be incompatible. -Explicitly specifying an incompatible target on the command line results in an error message and a failed build. +Explicitly specifying an incompatible target on the command line results in an +error message and a failed build. ```console $ bazel build --platforms=//:myplatform //:target_incompatible_with_myplatform @@ -120,13 +188,20 @@ ERROR: Target //:target_incompatible_with_myplatform is incompatible and cannot FAILED: Build did NOT complete successfully ``` -Incompatible explicit targets are silently skipped if `--skip_incompatible_explicit_targets` is enabled. +Incompatible explicit targets are silently skipped if +`--skip_incompatible_explicit_targets` is enabled. ### More expressive constraints -For more flexibility in expressing constraints, use the `@platforms//:incompatible` [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) that no platform satisfies. +For more flexibility in expressing constraints, use the +`@platforms//:incompatible` +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) +that no platform satisfies. -Use [`select()`](/reference/be/functions#select) in combination with `@platforms//:incompatible` to express more complicated restrictions. For example, use it to implement basic OR logic. The following marks a library compatible with macOS and Linux, but no other platforms. +Use [`select()`](/reference/be/functions#select) in combination with +`@platforms//:incompatible` to express more complicated restrictions. For +example, use it to implement basic OR logic. The following marks a library +compatible with macOS and Linux, but no other platforms. Note: An empty constraints list is equivalent to "compatible with everything". @@ -146,11 +221,16 @@ The above can be interpreted as follows: 1. When targeting macOS, the target has no constraints. 2. When targeting Linux, the target has no constraints. -3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because `@platforms//:incompatible` is not part of any platform, the target is deemed incompatible. +3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because + `@platforms//:incompatible` is not part of any platform, the target is + deemed incompatible. -To make your constraints more readable, use [skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). +To make your constraints more readable, use +[skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). -You can express inverse compatibility in a similar way. The following example describes a library that is compatible with everything *except* for ARM. +You can express inverse compatibility in a similar way. The following example +describes a library that is compatible with everything _except_ for ARM. ```python cc_library( @@ -165,9 +245,15 @@ cc_library( ### Detecting incompatible targets using `bazel cquery` -You can use the [`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) in `bazel cquery`'s [Starlark output format](/query/cquery#output-format-definition) to distinguish incompatible targets from compatible ones. +You can use the +[`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) +in `bazel cquery`'s [Starlark output +format](/query/cquery#output-format-definition) to distinguish +incompatible targets from compatible ones. -This can be used to filter out incompatible targets. The example below will only print the labels for targets that are compatible. Incompatible targets are not printed. +This can be used to filter out incompatible targets. The example below will +only print the labels for targets that are compatible. Incompatible targets are +not printed. ```console $ cat example.cquery @@ -177,9 +263,11 @@ def format(target): return target.label return "" + $ bazel cquery //... --output=starlark --starlark:file=example.cquery ``` ### Known Issues -Incompatible targets [ignore visibility restrictions](https://github.com/bazelbuild/bazel/issues/16044). +Incompatible targets [ignore visibility +restrictions](https://github.com/bazelbuild/bazel/issues/16044). diff --git a/extending/rules.mdx b/extending/rules.mdx index b7afa71c..248ee328 100644 --- a/extending/rules.mdx +++ b/extending/rules.mdx @@ -2,28 +2,57 @@ title: 'Rules' --- -A **rule** defines a series of [**actions**](#actions) that Bazel performs on inputs to produce a set of outputs, which are referenced in [**providers**](#providers) returned by the rule's [**implementation function**](#implementation_function). For example, a C++ binary rule might: -1. Take a set of `.cpp` source files (inputs). -2. Run `g++` on the source files (action). -3. Return the `DefaultInfo` provider with the executable output and other files to make available at runtime. -4. Return the `CcInfo` provider with C++-specific information gathered from the target and its dependencies. -From Bazel's perspective, `g++` and the standard C++ libraries are also inputs to this rule. As a rule writer, you must consider not only the user-provided inputs to a rule, but also all of the tools and libraries required to execute the actions. - -Before creating or modifying any rule, ensure you are familiar with Bazel's [build phases](/extending/concepts). It is important to understand the three phases of a build (loading, analysis, and execution). It is also useful to learn about [macros](/extending/macros) to understand the difference between rules and macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). Then, use this page as a reference. - -A few rules are built into Bazel itself. These *native rules*, such as `genrule` and `filegroup`, provide some core support. By defining your own rules, you can add support for languages and tools that Bazel doesn't support natively. - -Bazel provides an extensibility model for writing rules using the [Starlark](/rules/language) language. These rules are written in `.bzl` files, which can be loaded directly from `BUILD` files. - -When defining your own rule, you get to decide what attributes it supports and how it generates its outputs. - -The rule's `implementation` function defines its exact behavior during the [analysis phase](/extending/concepts#evaluation-model). This function doesn't run any external commands. Rather, it registers [actions](#actions) that will be used later during the execution phase to build the rule's outputs, if they are needed. +A **rule** defines a series of [**actions**](#actions) that Bazel performs on +inputs to produce a set of outputs, which are referenced in +[**providers**](#providers) returned by the rule's +[**implementation function**](#implementation_function). For example, a C++ +binary rule might: + +1. Take a set of `.cpp` source files (inputs). +2. Run `g++` on the source files (action). +3. Return the `DefaultInfo` provider with the executable output and other files + to make available at runtime. +4. Return the `CcInfo` provider with C++-specific information gathered from the + target and its dependencies. + +From Bazel's perspective, `g++` and the standard C++ libraries are also inputs +to this rule. As a rule writer, you must consider not only the user-provided +inputs to a rule, but also all of the tools and libraries required to execute +the actions. + +Before creating or modifying any rule, ensure you are familiar with Bazel's +[build phases](/extending/concepts). It is important to understand the three +phases of a build (loading, analysis, and execution). It is also useful to +learn about [macros](/extending/macros) to understand the difference between rules and +macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). +Then, use this page as a reference. + +A few rules are built into Bazel itself. These *native rules*, such as +`genrule` and `filegroup`, provide some core support. +By defining your own rules, you can add support for languages and tools +that Bazel doesn't support natively. + +Bazel provides an extensibility model for writing rules using the +[Starlark](/rules/language) language. These rules are written in `.bzl` files, which +can be loaded directly from `BUILD` files. + +When defining your own rule, you get to decide what attributes it supports and +how it generates its outputs. + +The rule's `implementation` function defines its exact behavior during the +[analysis phase](/extending/concepts#evaluation-model). This function doesn't run any +external commands. Rather, it registers [actions](#actions) that will be used +later during the execution phase to build the rule's outputs, if they are +needed. ## Rule creation -In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new rule, and store the result in a global variable. The call to `rule` specifies [attributes](#attributes) and an [implementation function](#implementation_function): +In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new +rule, and store the result in a global variable. The call to `rule` specifies +[attributes](#attributes) and an +[implementation function](#implementation_function): ```python example_library = rule( @@ -37,7 +66,10 @@ example_library = rule( This defines a [rule kind](/query/language#kind) named `example_library`. -The call to `rule` also must specify if the rule creates an [executable](#executable-rules) output (with `executable = True`), or specifically a test executable (with `test = True`). If the latter, the rule is a *test rule*, and the name of the rule must end in `_test`. +The call to `rule` also must specify if the rule creates an +[executable](#executable-rules) output (with `executable = True`), or specifically +a test executable (with `test = True`). If the latter, the rule is a *test rule*, +and the name of the rule must end in `_test`. ## Target instantiation @@ -53,23 +85,48 @@ example_library( ) ``` -Each call to a build rule returns no value, but has the side effect of defining a target. This is called *instantiating* the rule. This specifies a name for the new target and values for the target's [attributes](#attributes). +Each call to a build rule returns no value, but has the side effect of defining +a target. This is called *instantiating* the rule. This specifies a name for the +new target and values for the target's [attributes](#attributes). -Rules can also be called from Starlark functions and loaded in `.bzl` files. Starlark functions that call rules are called [Starlark macros](/extending/macros). Starlark macros must ultimately be called from `BUILD` files, and can only be called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` files are evaluated to instantiate targets. +Rules can also be called from Starlark functions and loaded in `.bzl` files. +Starlark functions that call rules are called [Starlark macros](/extending/macros). +Starlark macros must ultimately be called from `BUILD` files, and can only be +called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` +files are evaluated to instantiate targets. ## Attributes -An *attribute* is a rule argument. Attributes can provide specific values to a target's [implementation](#implementation_function), or they can refer to other targets, creating a graph of dependencies. +An *attribute* is a rule argument. Attributes can provide specific values to a +target's [implementation](#implementation_function), or they can refer to other +targets, creating a graph of dependencies. -Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) module) to the `attrs` parameter of `rule`. [Common attributes](/reference/be/common-definitions#common-attributes), such as `name` and `visibility`, are implicitly added to all rules. Additional attributes are implicitly added to [executable and test rules](#executable-rules) specifically. Attributes which are implicitly added to a rule can't be included in the dictionary passed to `attrs`. +Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map +from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) +module) to the `attrs` parameter of `rule`. +[Common attributes](/reference/be/common-definitions#common-attributes), such as +`name` and `visibility`, are implicitly added to all rules. Additional +attributes are implicitly added to +[executable and test rules](#executable-rules) specifically. Attributes which +are implicitly added to a rule can't be included in the dictionary passed to +`attrs`. ### Dependency attributes -Rules that process source code usually define the following attributes to handle various [types of dependencies](/concepts/dependencies#types_of_dependencies): - -- `srcs` specifies source files processed by a target's actions. Often, the attribute schema specifies which file extensions are expected for the sort of source file the rule processes. Rules for languages with header files generally specify a separate `hdrs` attribute for headers processed by a target and its consumers. -- `deps` specifies code dependencies for a target. The attribute schema should specify which [providers](#providers) those dependencies must provide. (For example, `cc_library` provides `CcInfo`.) -- `data` specifies files to be made available at runtime to any executable which depends on a target. That should allow arbitrary files to be specified. +Rules that process source code usually define the following attributes to handle +various [types of dependencies](/concepts/dependencies#types_of_dependencies): + +* `srcs` specifies source files processed by a target's actions. Often, the + attribute schema specifies which file extensions are expected for the sort + of source file the rule processes. Rules for languages with header files + generally specify a separate `hdrs` attribute for headers processed by a + target and its consumers. +* `deps` specifies code dependencies for a target. The attribute schema should + specify which [providers](#providers) those dependencies must provide. (For + example, `cc_library` provides `CcInfo`.) +* `data` specifies files to be made available at runtime to any executable + which depends on a target. That should allow arbitrary files to be + specified. ```python example_library = rule( @@ -84,7 +141,16 @@ example_library = rule( ) ``` -These are examples of *dependency attributes*. Any attribute that specifies an input label (those defined with [`attr.label_list`](/rules/lib/toplevel/attr#label_list), [`attr.label`](/rules/lib/toplevel/attr#label), or [`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) specifies dependencies of a certain type between a target and the targets whose labels (or the corresponding [`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target is defined. The repository, and possibly the path, for these labels is resolved relative to the defined target. +These are examples of *dependency attributes*. Any attribute that specifies +an input label (those defined with +[`attr.label_list`](/rules/lib/toplevel/attr#label_list), +[`attr.label`](/rules/lib/toplevel/attr#label), or +[`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) +specifies dependencies of a certain type +between a target and the targets whose labels (or the corresponding +[`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target +is defined. The repository, and possibly the path, for these labels is resolved +relative to the defined target. ```python example_library( @@ -98,15 +164,27 @@ example_library( ) ``` -In this example, `other_target` is a dependency of `my_target`, and therefore `other_target` is analyzed first. It is an error if there is a cycle in the dependency graph of targets. +In this example, `other_target` is a dependency of `my_target`, and therefore +`other_target` is analyzed first. It is an error if there is a cycle in the +dependency graph of targets. -[]() + ### Private attributes and implicit dependencies -A dependency attribute with a default value creates an *implicit dependency*. It is implicit because it's a part of the target graph that the user doesn't specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a relationship between a rule and a *tool* (a build-time dependency, such as a compiler), since most of the time a user is not interested in specifying what tool the rule uses. Inside the rule's implementation function, this is treated the same as other dependencies. +A dependency attribute with a default value creates an *implicit dependency*. It +is implicit because it's a part of the target graph that the user doesn't +specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a +relationship between a rule and a *tool* (a build-time dependency, such as a +compiler), since most of the time a user is not interested in specifying what +tool the rule uses. Inside the rule's implementation function, this is treated +the same as other dependencies. -If you want to provide an implicit dependency without allowing the user to override that value, you can make the attribute *private* by giving it a name that begins with an underscore (`_`). Private attributes must have default values. It generally only makes sense to use private attributes for implicit dependencies. +If you want to provide an implicit dependency without allowing the user to +override that value, you can make the attribute *private* by giving it a name +that begins with an underscore (`_`). Private attributes must have default +values. It generally only makes sense to use private attributes for implicit +dependencies. ```python example_library = rule( @@ -123,34 +201,67 @@ example_library = rule( ) ``` -In this example, every target of type `example_library` has an implicit dependency on the compiler `//tools:example_compiler`. This allows `example_library`'s implementation function to generate actions that invoke the compiler, even though the user did not pass its label as an input. Since `_compiler` is a private attribute, it follows that `ctx.attr._compiler` will always point to `//tools:example_compiler` in all targets of this rule type. Alternatively, you can name the attribute `compiler` without the underscore and keep the default value. This allows users to substitute a different compiler if necessary, but it requires no awareness of the compiler's label. - -Implicit dependencies are generally used for tools that reside in the same repository as the rule implementation. If the tool comes from the [execution platform](/extending/platforms) or a different repository instead, the rule should obtain that tool from a [toolchain](/extending/toolchains). +In this example, every target of type `example_library` has an implicit +dependency on the compiler `//tools:example_compiler`. This allows +`example_library`'s implementation function to generate actions that invoke the +compiler, even though the user did not pass its label as an input. Since +`_compiler` is a private attribute, it follows that `ctx.attr._compiler` +will always point to `//tools:example_compiler` in all targets of this rule +type. Alternatively, you can name the attribute `compiler` without the +underscore and keep the default value. This allows users to substitute a +different compiler if necessary, but it requires no awareness of the compiler's +label. + +Implicit dependencies are generally used for tools that reside in the same +repository as the rule implementation. If the tool comes from the +[execution platform](/extending/platforms) or a different repository instead, the +rule should obtain that tool from a [toolchain](/extending/toolchains). ### Output attributes -*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and [`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the target generates. These differ from dependency attributes in two ways: +*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and +[`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the +target generates. These differ from dependency attributes in two ways: -- They define output file targets instead of referring to targets defined elsewhere. -- The output file targets depend on the instantiated rule target, instead of the other way around. +* They define output file targets instead of referring to targets defined + elsewhere. +* The output file targets depend on the instantiated rule target, instead of + the other way around. -Typically, output attributes are only used when a rule needs to create outputs with user-defined names which can't be based on the target name. If a rule has one output attribute, it is typically named `out` or `outs`. +Typically, output attributes are only used when a rule needs to create outputs +with user-defined names which can't be based on the target name. If a rule has +one output attribute, it is typically named `out` or `outs`. -Output attributes are the preferred way of creating *predeclared outputs*, which can be specifically depended upon or [requested at the command line](#requesting_output_files). +Output attributes are the preferred way of creating *predeclared outputs*, which +can be specifically depended upon or +[requested at the command line](#requesting_output_files). ## Implementation function -Every rule requires an `implementation` function. These functions are executed strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the graph of targets generated in the loading phase into a graph of [actions](#actions) to be performed during the execution phase. As such, implementation functions can't actually read or write files. +Every rule requires an `implementation` function. These functions are executed +strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the +graph of targets generated in the loading phase into a graph of +[actions](#actions) to be performed during the execution phase. As such, +implementation functions can't actually read or write files. -Rule implementation functions are usually private (named with a leading underscore). Conventionally, they are named the same as their rule, but suffixed with `_impl`. +Rule implementation functions are usually private (named with a leading +underscore). Conventionally, they are named the same as their rule, but suffixed +with `_impl`. -Implementation functions take exactly one parameter: a [rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of [providers](#providers). +Implementation functions take exactly one parameter: a +[rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of +[providers](#providers). ### Targets -Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) objects. These objects contain the [providers](#providers) generated when the target's implementation function was executed. +Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) +objects. These objects contain the [providers](#providers) generated when the +target's implementation function was executed. -[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each dependency attribute, containing `Target` objects representing each direct dependency using that attribute. For `label_list` attributes, this is a list of `Targets`. For `label` attributes, this is a single `Target` or `None`. +[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each +dependency attribute, containing `Target` objects representing each direct +dependency using that attribute. For `label_list` attributes, this is a list of +`Targets`. For `label` attributes, this is a single `Target` or `None`. A list of provider objects are returned by a target's implementation function: @@ -158,9 +269,14 @@ A list of provider objects are returned by a target's implementation function: return [ExampleInfo(headers = depset(...))] ``` -Those can be accessed using index notation (`[]`), with the type of provider as a key. These can be [custom providers](#custom_providers) defined in Starlark or [providers for native rules](/rules/lib/providers) available as Starlark global variables. +Those can be accessed using index notation (`[]`), with the type of provider as +a key. These can be [custom providers](#custom_providers) defined in Starlark or +[providers for native rules](/rules/lib/providers) available as Starlark +global variables. -For example, if a rule takes header files using a `hdrs` attribute and provides them to the compilation actions of the target and its consumers, it could collect them like so: +For example, if a rule takes header files using a `hdrs` attribute and provides +them to the compilation actions of the target and its consumers, it could +collect them like so: ```python def _example_library_impl(ctx): @@ -168,15 +284,24 @@ def _example_library_impl(ctx): transitive_headers = [hdr[ExampleInfo].headers for hdr in ctx.attr.hdrs] ``` -There's a legacy struct style, which is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). +There's a legacy struct style, which is strongly discouraged and rules should be +[migrated away from it](#migrating_from_legacy_providers). ### Files -Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't perform file I/O during the analysis phase, these objects can't be used to directly read or write file content. Rather, they are passed to action-emitting functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the action graph. +Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't +perform file I/O during the analysis phase, these objects can't be used to +directly read or write file content. Rather, they are passed to action-emitting +functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the +action graph. -A `File` can either be a source file or a generated file. Each generated file must be an output of exactly one action. Source files can't be the output of any action. +A `File` can either be a source file or a generated file. Each generated file +must be an output of exactly one action. Source files can't be the output of +any action. -For each dependency attribute, the corresponding field of [`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all dependencies using that attribute: +For each dependency attribute, the corresponding field of +[`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all +dependencies using that attribute: ```python def _example_library_impl(ctx): @@ -186,11 +311,20 @@ def _example_library_impl(ctx): ... ``` -[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for dependency attributes whose specs set `allow_single_file = True`. [`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only contains fields for dependency attributes whose specs set `executable = True`. +[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for +dependency attributes whose specs set `allow_single_file = True`. +[`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only +contains fields for dependency attributes whose specs set `executable = True`. ### Declaring outputs -During the analysis phase, a rule's implementation function can create outputs. Since all labels have to be known during the loading phase, these additional outputs have no labels. `File` objects for outputs can be created using [`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and [`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). Often, the names of outputs are based on the target's name, [`ctx.label.name`](/rules/lib/builtins/ctx#label): +During the analysis phase, a rule's implementation function can create outputs. +Since all labels have to be known during the loading phase, these additional +outputs have no labels. `File` objects for outputs can be created using +[`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and +[`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). +Often, the names of outputs are based on the target's name, +[`ctx.label.name`](/rules/lib/builtins/ctx#label): ```python def _example_library_impl(ctx): @@ -199,20 +333,31 @@ def _example_library_impl(ctx): ... ``` -For *predeclared outputs*, like those created for [output attributes](#output_attributes), `File` objects instead can be retrieved from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). +For *predeclared outputs*, like those created for +[output attributes](#output_attributes), `File` objects instead can be retrieved +from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). ### Actions -An action describes how to generate a set of outputs from a set of inputs, for example "run gcc on hello.c and get hello.o". When an action is created, Bazel doesn't run the command immediately. It registers it in a graph of dependencies, because an action can depend on the output of another action. For example, in C, the linker must be called after the compiler. +An action describes how to generate a set of outputs from a set of inputs, for +example "run gcc on hello.c and get hello.o". When an action is created, Bazel +doesn't run the command immediately. It registers it in a graph of dependencies, +because an action can depend on the output of another action. For example, in C, +the linker must be called after the compiler. -General-purpose functions that create actions are defined in [`ctx.actions`](/rules/lib/builtins/actions): +General-purpose functions that create actions are defined in +[`ctx.actions`](/rules/lib/builtins/actions): -- [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. -- [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell command. -- [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. -- [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to generate a file from a template. +* [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. +* [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell + command. +* [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. +* [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to + generate a file from a template. -[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently accumulate the arguments for actions. It avoids flattening depsets until execution time: +[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently +accumulate the arguments for actions. It avoids flattening depsets until +execution time: ```python def _example_library_impl(ctx): @@ -239,31 +384,61 @@ def _example_library_impl(ctx): ... ``` -Actions take a list or depset of input files and generate a (non-empty) list of output files. The set of input and output files must be known during the [analysis phase](/extending/concepts#evaluation-model). It might depend on the value of attributes, including providers from dependencies, but it can't depend on the result of the execution. For example, if your action runs the unzip command, you must specify which files you expect to be inflated (before running unzip). Actions which create a variable number of files internally can wrap those in a single file (such as a zip, tar, or other archive format). +Actions take a list or depset of input files and generate a (non-empty) list of +output files. The set of input and output files must be known during the +[analysis phase](/extending/concepts#evaluation-model). It might depend on the value of +attributes, including providers from dependencies, but it can't depend on the +result of the execution. For example, if your action runs the unzip command, you +must specify which files you expect to be inflated (before running unzip). +Actions which create a variable number of files internally can wrap those in a +single file (such as a zip, tar, or other archive format). -Actions must list all of their inputs. Listing inputs that are not used is permitted, but inefficient. +Actions must list all of their inputs. Listing inputs that are not used is +permitted, but inefficient. -Actions must create all of their outputs. They may write other files, but anything not in outputs won't be available to consumers. All declared outputs must be written by some action. +Actions must create all of their outputs. They may write other files, but +anything not in outputs won't be available to consumers. All declared outputs +must be written by some action. -Actions are comparable to pure functions: They should depend only on the provided inputs, and avoid accessing computer information, username, clock, network, or I/O devices (except for reading inputs and writing outputs). This is important because the output will be cached and reused. +Actions are comparable to pure functions: They should depend only on the +provided inputs, and avoid accessing computer information, username, clock, +network, or I/O devices (except for reading inputs and writing outputs). This is +important because the output will be cached and reused. -Dependencies are resolved by Bazel, which decides which actions to execute. It is an error if there is a cycle in the dependency graph. Creating an action doesn't guarantee that it will be executed, that depends on whether its outputs are needed for the build. +Dependencies are resolved by Bazel, which decides which actions to +execute. It is an error if there is a cycle in the dependency graph. Creating +an action doesn't guarantee that it will be executed, that depends on whether +its outputs are needed for the build. ### Providers -Providers are pieces of information that a rule exposes to other rules that depend on it. This data can include output files, libraries, parameters to pass on a tool's command line, or anything else a target's consumers should know about. +Providers are pieces of information that a rule exposes to other rules that +depend on it. This data can include output files, libraries, parameters to pass +on a tool's command line, or anything else a target's consumers should know +about. -Since a rule's implementation function can only read providers from the instantiated target's immediate dependencies, rules need to forward any information from a target's dependencies that needs to be known by a target's consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). +Since a rule's implementation function can only read providers from the +instantiated target's immediate dependencies, rules need to forward any +information from a target's dependencies that needs to be known by a target's +consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). -A target's providers are specified by a list of provider objects returned by the implementation function. +A target's providers are specified by a list of provider objects returned by +the implementation function. -Old implementation functions can also be written in a legacy style where the implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of provider objects. This style is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). +Old implementation functions can also be written in a legacy style where the +implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of +provider objects. This style is strongly discouraged and rules should be +[migrated away from it](#migrating_from_legacy_providers). #### Default outputs -A target's *default outputs* are the outputs that are requested by default when the target is requested for build at the command line. For example, a `java_library` target `//pkg:foo` has `foo.jar` as a default output, so that will be built by the command `bazel build //pkg:foo`. +A target's *default outputs* are the outputs that are requested by default when +the target is requested for build at the command line. For example, a +`java_library` target `//pkg:foo` has `foo.jar` as a default output, so that +will be built by the command `bazel build //pkg:foo`. -Default outputs are specified by the `files` parameter of [`DefaultInfo`](/rules/lib/providers/DefaultInfo): +Default outputs are specified by the `files` parameter of +[`DefaultInfo`](/rules/lib/providers/DefaultInfo): ```python def _example_library_impl(ctx): @@ -274,17 +449,37 @@ def _example_library_impl(ctx): ] ``` -If `DefaultInfo` is not returned by a rule implementation or the `files` parameter is not specified, `DefaultInfo.files` defaults to all *predeclared outputs* (generally, those created by [output attributes](#output_attributes)). +If `DefaultInfo` is not returned by a rule implementation or the `files` +parameter is not specified, `DefaultInfo.files` defaults to all +*predeclared outputs* (generally, those created by [output +attributes](#output_attributes)). -Rules that perform actions should provide default outputs, even if those outputs are not expected to be directly used. Actions that are not in the graph of the requested outputs are pruned. If an output is only used by a target's consumers, those actions won't be performed when the target is built in isolation. This makes debugging more difficult because rebuilding just the failing target won't reproduce the failure. +Rules that perform actions should provide default outputs, even if those outputs +are not expected to be directly used. Actions that are not in the graph of the +requested outputs are pruned. If an output is only used by a target's consumers, +those actions won't be performed when the target is built in isolation. This +makes debugging more difficult because rebuilding just the failing target won't +reproduce the failure. #### Runfiles -Runfiles are a set of files used by a target at runtime (as opposed to build time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates a directory tree containing symlinks pointing to the runfiles. This stages the environment for the binary so it can access the runfiles during runtime. +Runfiles are a set of files used by a target at runtime (as opposed to build +time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates +a directory tree containing symlinks pointing to the runfiles. This stages the +environment for the binary so it can access the runfiles during runtime. -Runfiles can be added manually during rule creation. [`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the `runfiles` parameter on `DefaultInfo`. The executable output of [executable rules](#executable-rules) is implicitly added to the runfiles. +Runfiles can be added manually during rule creation. +[`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method +on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the +`runfiles` parameter on `DefaultInfo`. The executable output of +[executable rules](#executable-rules) is implicitly added to the runfiles. -Some rules specify attributes, generally named [`data`](/reference/be/common-definitions#common.data), whose outputs are added to a targets' runfiles. Runfiles should also be merged in from `data`, as well as from any attributes which might provide code for eventual execution, generally `srcs` (which might contain `filegroup` targets with associated `data`) and `deps`. +Some rules specify attributes, generally named +[`data`](/reference/be/common-definitions#common.data), whose outputs are added to +a targets' runfiles. Runfiles should also be merged in from `data`, as well as +from any attributes which might provide code for eventual execution, generally +`srcs` (which might contain `filegroup` targets with associated `data`) and +`deps`. ```python def _example_library_impl(ctx): @@ -308,7 +503,8 @@ def _example_library_impl(ctx): #### Custom providers -Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) function to convey rule-specific information: +Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) +function to convey rule-specific information: ```python ExampleInfo = provider( @@ -341,11 +537,23 @@ def _example_library_impl(ctx): ##### Custom initialization of providers -It's possible to guard the instantiation of a provider with custom preprocessing and validation logic. This can be used to ensure that all provider instances satisfy certain invariants, or to give users a cleaner API for obtaining an instance. +It's possible to guard the instantiation of a provider with custom +preprocessing and validation logic. This can be used to ensure that all +provider instances satisfy certain invariants, or to give users a cleaner API for +obtaining an instance. -This is done by passing an `init` callback to the [`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the return type of `provider()` changes to be a tuple of two values: the provider symbol that is the ordinary return value when `init` is not used, and a "raw constructor". +This is done by passing an `init` callback to the +[`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the +return type of `provider()` changes to be a tuple of two values: the provider +symbol that is the ordinary return value when `init` is not used, and a "raw +constructor". -In this case, when the provider symbol is called, instead of directly returning a new instance, it will forward the arguments along to the `init` callback. The callback's return value must be a dict mapping field names (strings) to values; this is used to initialize the fields of the new instance. Note that the callback may have any signature, and if the arguments don't match the signature an error is reported as if the callback were invoked directly. +In this case, when the provider symbol is called, instead of directly returning +a new instance, it will forward the arguments along to the `init` callback. The +callback's return value must be a dict mapping field names (strings) to values; +this is used to initialize the fields of the new instance. Note that the +callback may have any signature, and if the arguments don't match the signature +an error is reported as if the callback were invoked directly. The raw constructor, by contrast, will bypass the `init` callback. @@ -378,7 +586,9 @@ ExampleInfo( ) ``` -The raw constructor can be used to define alternative public factory functions that don't go through the `init` logic. For example, exampleinfo.bzl could define: +The raw constructor can be used to define alternative public factory functions +that don't go through the `init` logic. For example, exampleinfo.bzl +could define: ```python def make_barebones_exampleinfo(headers): @@ -386,9 +596,12 @@ def make_barebones_exampleinfo(headers): return _new_exampleinfo(files_to_link = depset(), headers = all_headers) ``` -Typically, the raw constructor is bound to a variable whose name begins with an underscore (`_new_exampleinfo` above), so that user code can't load it and generate arbitrary provider instances. +Typically, the raw constructor is bound to a variable whose name begins with an +underscore (`_new_exampleinfo` above), so that user code can't load it and +generate arbitrary provider instances. -Another use for `init` is to prevent the user from calling the provider symbol altogether, and force them to use a factory function instead: +Another use for `init` is to prevent the user from calling the provider +symbol altogether, and force them to use a factory function instead: ```python def _exampleinfo_init_banned(*args, **kwargs): @@ -403,11 +616,15 @@ def make_exampleinfo(...): return _new_exampleinfo(...) ``` -[]() + ## Executable rules and test rules -Executable rules define targets that can be invoked by a `bazel run` command. Test rules are a special kind of executable rule whose targets can also be invoked by a `bazel test` command. Executable and test rules are created by setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or [`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: +Executable rules define targets that can be invoked by a `bazel run` command. +Test rules are a special kind of executable rule whose targets can also be +invoked by a `bazel test` command. Executable and test rules are created by +setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or +[`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: ```python example_binary = rule( @@ -423,9 +640,17 @@ example_test = rule( ) ``` -Test rules must have names that end in `_test`. (Test *target* names also often end in `_test` by convention, but this is not required.) Non-test rules must not have this suffix. +Test rules must have names that end in `_test`. (Test *target* names also often +end in `_test` by convention, but this is not required.) Non-test rules must not +have this suffix. -Both kinds of rules must produce an executable output file (which may or may not be predeclared) that will be invoked by the `run` or `test` commands. To tell Bazel which of a rule's outputs to use as this executable, pass it as the `executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) provider. That `executable` is added to the default outputs of the rule (so you don't need to pass that to both `executable` and `files`). It's also implicitly added to the [runfiles](#runfiles): +Both kinds of rules must produce an executable output file (which may or may not +be predeclared) that will be invoked by the `run` or `test` commands. To tell +Bazel which of a rule's outputs to use as this executable, pass it as the +`executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) +provider. That `executable` is added to the default outputs of the rule (so you +don't need to pass that to both `executable` and `files`). It's also implicitly +added to the [runfiles](#runfiles): ```python def _example_binary_impl(ctx): @@ -437,13 +662,30 @@ def _example_binary_impl(ctx): ] ``` -The action that generates this file must set the executable bit on the file. For a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done by the underlying tool that is invoked by the action. For a [`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. - -As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a special `ctx.outputs.executable` predeclared output. This file serves as the default executable if you don't specify one using `DefaultInfo`; it must not be used otherwise. This output mechanism is deprecated because it doesn't support customizing the executable file's name at analysis time. - -See examples of an [executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) and a [test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). - -[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and [test rules](/reference/be/common-definitions#common-attributes-tests) have additional attributes implicitly defined, in addition to those added for [all rules](/reference/be/common-definitions#common-attributes). The defaults of implicitly-added attributes can't be changed, though this can be worked around by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the default: +The action that generates this file must set the executable bit on the file. For +a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or +[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done +by the underlying tool that is invoked by the action. For a +[`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. + +As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a +special `ctx.outputs.executable` predeclared output. This file serves as the +default executable if you don't specify one using `DefaultInfo`; it must not be +used otherwise. This output mechanism is deprecated because it doesn't support +customizing the executable file's name at analysis time. + +See examples of an +[executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) +and a +[test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). + +[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and +[test rules](/reference/be/common-definitions#common-attributes-tests) have additional +attributes implicitly defined, in addition to those added for +[all rules](/reference/be/common-definitions#common-attributes). The defaults of +implicitly-added attributes can't be changed, though this can be worked around +by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the +default: ```python def example_test(size = "small", **kwargs): @@ -456,7 +698,8 @@ _example_test = rule( ### Runfiles location -When an executable target is run with `bazel run` (or `test`), the root of the runfiles directory is adjacent to the executable. The paths relate as follows: +When an executable target is run with `bazel run` (or `test`), the root of the +runfiles directory is adjacent to the executable. The paths relate as follows: ```python # Given launcher_path and runfile_file: @@ -467,21 +710,50 @@ execution_root_relative_path = "%s/%s/%s" % ( runfiles_root, workspace_name, runfile_path) ``` -The path to a `File` under the runfiles directory corresponds to [`File.short_path`](/rules/lib/builtins/File#short_path). +The path to a `File` under the runfiles directory corresponds to +[`File.short_path`](/rules/lib/builtins/File#short_path). -The binary executed directly by `bazel` is adjacent to the root of the `runfiles` directory. However, binaries called *from* the runfiles can't make the same assumption. To mitigate this, each binary should provide a way to accept its runfiles root as a parameter using an environment, or command line argument or flag. This allows binaries to pass the correct canonical runfiles root to the binaries it calls. If that's not set, a binary can guess that it was the first binary called and look for an adjacent runfiles directory. +The binary executed directly by `bazel` is adjacent to the root of the +`runfiles` directory. However, binaries called *from* the runfiles can't make +the same assumption. To mitigate this, each binary should provide a way to +accept its runfiles root as a parameter using an environment, or command line +argument or flag. This allows binaries to pass the correct canonical runfiles root +to the binaries it calls. If that's not set, a binary can guess that it was the +first binary called and look for an adjacent runfiles directory. ## Advanced topics ### Requesting output files -A single target can have several output files. When a `bazel build` command is run, some of the outputs of the targets given to the command are considered to be *requested*. Bazel only builds these requested files and the files that they directly or indirectly depend on. (In terms of the action graph, Bazel only executes the actions that are reachable as transitive dependencies of the requested files.) - -In addition to [default outputs](#default_outputs), any *predeclared output* can be explicitly requested on the command line. Rules can specify predeclared outputs using [output attributes](#output_attributes). In that case, the user explicitly chooses labels for outputs when they instantiate the rule. To obtain [`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can [implicitly define predeclared outputs](#deprecated_predeclared_outputs) based on the target name as well, but this feature is deprecated. - -In addition to default outputs, there are *output groups*, which are collections of output files that may be requested together. These can be requested with [`--output_groups`](/reference/command-line-reference#flag--output_groups). For example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` output group, these files can be built by running `bazel build //pkg:mytarget --output_groups=debug_files`. Since non-predeclared outputs don't have labels, they can only be requested by appearing in the default outputs or an output group. - -Output groups can be specified with the [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many built-in providers, `OutputGroupInfo` can take parameters with arbitrary names to define output groups with that name: +A single target can have several output files. When a `bazel build` command is +run, some of the outputs of the targets given to the command are considered to +be *requested*. Bazel only builds these requested files and the files that they +directly or indirectly depend on. (In terms of the action graph, Bazel only +executes the actions that are reachable as transitive dependencies of the +requested files.) + +In addition to [default outputs](#default_outputs), any *predeclared output* can +be explicitly requested on the command line. Rules can specify predeclared +outputs using [output attributes](#output_attributes). In that case, the user +explicitly chooses labels for outputs when they instantiate the rule. To obtain +[`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding +attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can +[implicitly define predeclared outputs](#deprecated_predeclared_outputs) based +on the target name as well, but this feature is deprecated. + +In addition to default outputs, there are *output groups*, which are collections +of output files that may be requested together. These can be requested with +[`--output_groups`](/reference/command-line-reference#flag--output_groups). For +example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` +output group, these files can be built by running `bazel build //pkg:mytarget +--output_groups=debug_files`. Since non-predeclared outputs don't have labels, +they can only be requested by appearing in the default outputs or an output +group. + +Output groups can be specified with the +[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many +built-in providers, `OutputGroupInfo` can take parameters with arbitrary names +to define output groups with that name: ```python def _example_library_impl(ctx): @@ -498,37 +770,82 @@ def _example_library_impl(ctx): ] ``` -Also unlike most providers, `OutputGroupInfo` can be returned by both an [aspect](/extending/aspects) and the rule target to which that aspect is applied, as long as they don't define the same output groups. In that case, the resulting providers are merged. +Also unlike most providers, `OutputGroupInfo` can be returned by both an +[aspect](/extending/aspects) and the rule target to which that aspect is applied, as +long as they don't define the same output groups. In that case, the resulting +providers are merged. -Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts of files from a target to the actions of its consumers. Define [rule-specific providers](#custom_providers) for that instead. +Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts +of files from a target to the actions of its consumers. Define +[rule-specific providers](#custom_providers) for that instead. ### Configurations -Imagine that you want to build a C++ binary for a different architecture. The build can be complex and involve multiple steps. Some of the intermediate binaries, like compilers and code generators, have to run on [the execution platform](/extending/platforms#overview) (which could be your host, or a remote executor). Some binaries like the final output must be built for the target architecture. - -For this reason, Bazel has a concept of "configurations" and transitions. The topmost targets (the ones requested on the command line) are built-in the "target" configuration, while tools that should run on the execution platform are built-in an "exec" configuration. Rules may generate different actions based on the configuration, for instance to change the cpu architecture that is passed to the compiler. In some cases, the same library may be needed for different configurations. If this happens, it will be analyzed and potentially built multiple times. - -By default, Bazel builds a target's dependencies in the same configuration as the target itself, in other words without transitions. When a dependency is a tool that's needed to help build the target, the corresponding attribute should specify a transition to an exec configuration. This causes the tool and all its dependencies to build for the execution platform. - -For each dependency attribute, you can use `cfg` to decide if dependencies should build in the same configuration or transition to an exec configuration. If a dependency attribute has the flag `executable = True`, `cfg` must be set explicitly. This is to guard against accidentally building a tool for the wrong configuration. [See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) - -In general, sources, dependent libraries, and executables that will be needed at runtime can use the same configuration. - -Tools that are executed as part of the build (such as compilers or code generators) should be built for an exec configuration. In this case, specify `cfg = "exec"` in the attribute. - -Otherwise, executables that are used at runtime (such as as part of a test) should be built for the target configuration. In this case, specify `cfg = "target"` in the attribute. - -`cfg = "target"` doesn't actually do anything: it's purely a convenience value to help rule designers be explicit about their intentions. When `executable = False`, which means `cfg` is optional, only set this when it truly helps readability. - -You can also use `cfg = my_transition` to use [user-defined transitions](/extending/config#user-defined-transitions), which allow rule authors a great deal of flexibility in changing configurations, with the drawback of [making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). - -**Note**: Historically, Bazel didn't have the concept of execution platforms, and instead all build actions were considered to run on the host machine. Bazel versions before 6.0 created a distinct "host" configuration to represent this. If you see references to "host" in code or old documentation, that's what this refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual overhead. - -[]() +Imagine that you want to build a C++ binary for a different architecture. The +build can be complex and involve multiple steps. Some of the intermediate +binaries, like compilers and code generators, have to run on +[the execution platform](/extending/platforms#overview) (which could be your host, +or a remote executor). Some binaries like the final output must be built for the +target architecture. + +For this reason, Bazel has a concept of "configurations" and transitions. The +topmost targets (the ones requested on the command line) are built-in the +"target" configuration, while tools that should run on the execution platform +are built-in an "exec" configuration. Rules may generate different actions based +on the configuration, for instance to change the cpu architecture that is passed +to the compiler. In some cases, the same library may be needed for different +configurations. If this happens, it will be analyzed and potentially built +multiple times. + +By default, Bazel builds a target's dependencies in the same configuration as +the target itself, in other words without transitions. When a dependency is a +tool that's needed to help build the target, the corresponding attribute should +specify a transition to an exec configuration. This causes the tool and all its +dependencies to build for the execution platform. + +For each dependency attribute, you can use `cfg` to decide if dependencies +should build in the same configuration or transition to an exec configuration. +If a dependency attribute has the flag `executable = True`, `cfg` must be set +explicitly. This is to guard against accidentally building a tool for the wrong +configuration. +[See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) + +In general, sources, dependent libraries, and executables that will be needed at +runtime can use the same configuration. + +Tools that are executed as part of the build (such as compilers or code generators) +should be built for an exec configuration. In this case, specify `cfg = "exec"` in +the attribute. + +Otherwise, executables that are used at runtime (such as as part of a test) should +be built for the target configuration. In this case, specify `cfg = "target"` in +the attribute. + +`cfg = "target"` doesn't actually do anything: it's purely a convenience value to +help rule designers be explicit about their intentions. When `executable = False`, +which means `cfg` is optional, only set this when it truly helps readability. + +You can also use `cfg = my_transition` to use +[user-defined transitions](/extending/config#user-defined-transitions), which allow +rule authors a great deal of flexibility in changing configurations, with the +drawback of +[making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). + +**Note**: Historically, Bazel didn't have the concept of execution platforms, +and instead all build actions were considered to run on the host machine. Bazel +versions before 6.0 created a distinct "host" configuration to represent this. +If you see references to "host" in code or old documentation, that's what this +refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual +overhead. + + ### Configuration fragments -Rules may access [configuration fragments](/rules/lib/fragments) such as `cpp` and `java`. However, all required fragments must be declared in order to avoid access errors: +Rules may access +[configuration fragments](/rules/lib/fragments) such as +`cpp` and `java`. However, all required fragments must be declared in +order to avoid access errors: ```python def _impl(ctx): @@ -545,7 +862,14 @@ my_rule = rule( ### Runfiles symlinks -Normally, the relative path of a file in the runfiles tree is the same as the relative path of that file in the source tree or generated output tree. If these need to be different for some reason, you can specify the `root_symlinks` or `symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to files, where the paths are relative to the root of the runfiles directory. The `symlinks` dictionary is the same, but paths are implicitly prefixed with the name of the main workspace (*not* the name of the repository containing the current target). +Normally, the relative path of a file in the runfiles tree is the same as the +relative path of that file in the source tree or generated output tree. If these +need to be different for some reason, you can specify the `root_symlinks` or +`symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to +files, where the paths are relative to the root of the runfiles directory. The +`symlinks` dictionary is the same, but paths are implicitly prefixed with the +name of the main workspace (*not* the name of the repository containing the +current target). ```python ... @@ -557,20 +881,37 @@ Normally, the relative path of a file in the runfiles tree is the same as the re # sometarget.runfiles/ # some/ # path/ - # here.foo -> some_data_file2 - # <workspace_name>/ + # here.foo -> some_data_file2 + # / # some/ # path/ - # here.bar -> some_data_file3 + # here.bar -> some_data_file3 ``` -If `symlinks` or `root_symlinks` is used, be careful not to map two different files to the same path in the runfiles tree. This will cause the build to fail with an error describing the conflict. To fix, you will need to modify your `ctx.runfiles` arguments to remove the collision. This checking will be done for any targets using your rule, as well as targets of any kind that depend on those targets. This is especially risky if your tool is likely to be used transitively by another tool; symlink names must be unique across the runfiles of a tool and all of its dependencies. +If `symlinks` or `root_symlinks` is used, be careful not to map two different +files to the same path in the runfiles tree. This will cause the build to fail +with an error describing the conflict. To fix, you will need to modify your +`ctx.runfiles` arguments to remove the collision. This checking will be done for +any targets using your rule, as well as targets of any kind that depend on those +targets. This is especially risky if your tool is likely to be used transitively +by another tool; symlink names must be unique across the runfiles of a tool and +all of its dependencies. ### Code coverage -When the [`coverage`](/reference/command-line-reference#coverage) command is run, the build may need to add coverage instrumentation for certain targets. The build also gathers the list of source files that are instrumented. The subset of targets that are considered is controlled by the flag [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). Test targets are excluded, unless [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) is specified. +When the [`coverage`](/reference/command-line-reference#coverage) command is run, +the build may need to add coverage instrumentation for certain targets. The +build also gathers the list of source files that are instrumented. The subset of +targets that are considered is controlled by the flag +[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). +Test targets are excluded, unless +[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) +is specified. -If a rule implementation adds coverage instrumentation at build time, it needs to account for that in its implementation function. [ctx.coverage\_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns `True` in coverage mode if a target's sources should be instrumented: +If a rule implementation adds coverage instrumentation at build time, it needs +to account for that in its implementation function. +[ctx.coverage_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns +`True` in coverage mode if a target's sources should be instrumented: ```python # Are this rule's sources instrumented? @@ -578,9 +919,13 @@ if ctx.coverage_instrumented(): # Do something to turn on coverage for this compile action ``` -Logic that always needs to be on in coverage mode (whether a target's sources specifically are instrumented or not) can be conditioned on [ctx.configuration.coverage\_enabled](/rules/lib/builtins/configuration#coverage_enabled). +Logic that always needs to be on in coverage mode (whether a target's sources +specifically are instrumented or not) can be conditioned on +[ctx.configuration.coverage_enabled](/rules/lib/builtins/configuration#coverage_enabled). -If the rule directly includes sources from its dependencies before compilation (such as header files), it may also need to turn on compile-time instrumentation if the dependencies' sources should be instrumented: +If the rule directly includes sources from its dependencies before compilation +(such as header files), it may also need to turn on compile-time instrumentation if +the dependencies' sources should be instrumented: ```python # Are this rule's sources or any of the sources for its direct dependencies @@ -589,7 +934,13 @@ if ctx.coverage_instrumented() or any([ctx.coverage_instrumented(dep) for dep in # Do something to turn on coverage for this compile action ``` -Rules also should provide information about which attributes are relevant for coverage with the `InstrumentedFilesInfo` provider, constructed using [`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). The `dependency_attributes` parameter of `instrumented_files_info` should list all runtime dependency attributes, including code dependencies like `deps` and data dependencies like `data`. The `source_attributes` parameter should list the rule's source files attributes if coverage instrumentation might be added: +Rules also should provide information about which attributes are relevant for +coverage with the `InstrumentedFilesInfo` provider, constructed using +[`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). +The `dependency_attributes` parameter of `instrumented_files_info` should list +all runtime dependency attributes, including code dependencies like `deps` and +data dependencies like `data`. The `source_attributes` parameter should list the +rule's source files attributes if coverage instrumentation might be added: ```python def _example_library_impl(ctx): @@ -606,11 +957,18 @@ def _example_library_impl(ctx): ] ``` -If `InstrumentedFilesInfo` is not returned, a default one is created with each non-tool [dependency attribute](#dependency_attributes) that doesn't set [`cfg`](#configuration) to `"exec"` in the attribute schema. in `dependency_attributes`. (This isn't ideal behavior, since it puts attributes like `srcs` in `dependency_attributes` instead of `source_attributes`, but it avoids the need for explicit coverage configuration for all rules in the dependency chain.) +If `InstrumentedFilesInfo` is not returned, a default one is created with each +non-tool [dependency attribute](#dependency_attributes) that doesn't set +[`cfg`](#configuration) to `"exec"` in the attribute schema. in +`dependency_attributes`. (This isn't ideal behavior, since it puts attributes +like `srcs` in `dependency_attributes` instead of `source_attributes`, but it +avoids the need for explicit coverage configuration for all rules in the +dependency chain.) #### Test rules -Test rules require additional setup to generate coverage reports. The rule itself has to add the following implicit attributes: +Test rules require additional setup to generate coverage reports. The rule +itself has to add the following implicit attributes: ```python my_test = rule( @@ -633,59 +991,115 @@ my_test = rule( ) ``` -By using `configuration_field`, the dependency on the Java LCOV merger tool can be avoided as long as coverage is not requested. +By using `configuration_field`, the dependency on the Java LCOV merger tool can +be avoided as long as coverage is not requested. -When the test is run, it should emit coverage information in the form of one or more \[LCOV files] ([https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT](https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT)) with unique names into the directory specified by the `COVERAGE_DIR` environment variable. Bazel will then merge these files into a single LCOV file using the `_lcov_merger` tool. If present, it will also collect C/C++ coverage using the `_collect_cc_coverage` tool. +When the test is run, it should emit coverage information in the form of one or +more [LCOV files] +(https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT) +with unique names into the directory specified by the `COVERAGE_DIR` environment +variable. Bazel will then merge these files into a single LCOV file using the +`_lcov_merger` tool. If present, it will also collect C/C++ coverage using the +`_collect_cc_coverage` tool. ### Baseline coverage -Since coverage is only collected for code that ends up in the dependency tree of a test, coverage reports can be misleading as they don't necessarily cover all the code matched by the `--instrumentation_filter` flag. +Since coverage is only collected for code that ends up in the dependency tree of +a test, coverage reports can be misleading as they don't necessarily cover all +the code matched by the `--instrumentation_filter` flag. -For this reason, Bazel allows rules to specify baseline coverage files using the `baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These files must be generated in LCOV format by a user-defined action and are supposed to list all lines, branches, functions and/or blocks in the target's source files (according to the `sources_attributes` and `extensions` parameters). For source files in targets that are instrumented for coverage, Bazel merges their baseline coverage into the combined coverage report generated with `--combined_report` and thus ensures that untested files still show up as uncovered. +For this reason, Bazel allows rules to specify baseline coverage files using the +`baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These +files must be generated in LCOV format by a user-defined action and are supposed +to list all lines, branches, functions and/or blocks in the target's source +files (according to the `sources_attributes` and `extensions` parameters). For +source files in targets that are instrumented for coverage, Bazel merges their +baseline coverage into the combined coverage report generated with +`--combined_report` and thus ensures that untested files still show up as +uncovered. -If a rule doesn't provide any baseline coverage files, Bazel generates synthetic coverage information that only mentions the source file paths, but doesn't contain any information about their contents. +If a rule doesn't provide any baseline coverage files, Bazel generates synthetic +coverage information that only mentions the source file paths, but doesn't +contain any information about their contents. ### Validation Actions -Sometimes you need to validate something about the build, and the information required to do that validation is available only in artifacts (source files or generated files). Because this information is in artifacts, rules can't do this validation at analysis time because rules can't read files. Instead, actions must do this validation at execution time. When validation fails, the action will fail, and hence so will the build. - -Examples of validations that might be run are static analysis, linting, dependency and consistency checks, and style checks. - -Validation actions can also help to improve build performance by moving parts of actions that are not required for building artifacts into separate actions. For example, if a single action that does compilation and linting can be separated into a compilation action and a linting action, then the linting action can be run as a validation action and run in parallel with other actions. - -These "validation actions" often don't produce anything that is used elsewhere in the build, since they only need to assert things about their inputs. This presents a problem though: If a validation action doesn't produce anything that is used elsewhere in the build, how does a rule get the action to run? Historically, the approach was to have the validation action output an empty file, and artificially add that output to the inputs of some other important action in the build: - -![](/rules/validation_action_historical.svg) - -This works, because Bazel will always run the validation action when the compile action is run, but this has significant drawbacks: - -1. The validation action is in the critical path of the build. Because Bazel thinks the empty output is required to run the compile action, it will run the validation action first, even though the compile action will ignore the input. This reduces parallelism and slows down builds. - -2. If other actions in the build might run instead of the compile action, then the empty outputs of validation actions need to be added to those actions as well (`java_library`'s source jar output, for example). This is also a problem if new actions that might run instead of the compile action are added later, and the empty validation output is accidentally left off. +Sometimes you need to validate something about the build, and the +information required to do that validation is available only in artifacts +(source files or generated files). Because this information is in artifacts, +rules can't do this validation at analysis time because rules can't read +files. Instead, actions must do this validation at execution time. When +validation fails, the action will fail, and hence so will the build. + +Examples of validations that might be run are static analysis, linting, +dependency and consistency checks, and style checks. + +Validation actions can also help to improve build performance by moving parts +of actions that are not required for building artifacts into separate actions. +For example, if a single action that does compilation and linting can be +separated into a compilation action and a linting action, then the linting +action can be run as a validation action and run in parallel with other actions. + +These "validation actions" often don't produce anything that is used elsewhere +in the build, since they only need to assert things about their inputs. This +presents a problem though: If a validation action doesn't produce anything that +is used elsewhere in the build, how does a rule get the action to run? +Historically, the approach was to have the validation action output an empty +file, and artificially add that output to the inputs of some other important +action in the build: + + + +This works, because Bazel will always run the validation action when the compile +action is run, but this has significant drawbacks: + +1. The validation action is in the critical path of the build. Because Bazel +thinks the empty output is required to run the compile action, it will run the +validation action first, even though the compile action will ignore the input. +This reduces parallelism and slows down builds. + +2. If other actions in the build might run instead of the +compile action, then the empty outputs of validation actions need to be added to +those actions as well (`java_library`'s source jar output, for example). This is +also a problem if new actions that might run instead of the compile action are +added later, and the empty validation output is accidentally left off. The solution to these problems is to use the Validations Output Group. #### Validations Output Group -The Validations Output Group is an output group designed to hold the otherwise unused outputs of validation actions, so that they don't need to be artificially added to the inputs of other actions. +The Validations Output Group is an output group designed to hold the otherwise +unused outputs of validation actions, so that they don't need to be artificially +added to the inputs of other actions. -This group is special in that its outputs are always requested, regardless of the value of the `--output_groups` flag, and regardless of how the target is depended upon (for example, on the command line, as a dependency, or through implicit outputs of the target). Note that normal caching and incrementality still apply: if the inputs to the validation action have not changed and the validation action previously succeeded, then the validation action won't be run. +This group is special in that its outputs are always requested, regardless of +the value of the `--output_groups` flag, and regardless of how the target is +depended upon (for example, on the command line, as a dependency, or through +implicit outputs of the target). Note that normal caching and incrementality +still apply: if the inputs to the validation action have not changed and the +validation action previously succeeded, then the validation action won't be +run. -![](/rules/validation_action.svg) + -Using this output group still requires that validation actions output some file, even an empty one. This might require wrapping some tools that normally don't create outputs so that a file is created. +Using this output group still requires that validation actions output some file, +even an empty one. This might require wrapping some tools that normally don't +create outputs so that a file is created. A target's validation actions are not run in three cases: -- When the target is depended upon as a tool -- When the target is depended upon as an implicit dependency (for example, an attribute that starts with "\_") -- When the target is built in the exec configuration. +* When the target is depended upon as a tool +* When the target is depended upon as an implicit dependency (for example, an + attribute that starts with "_") +* When the target is built in the exec configuration. -It is assumed that these targets have their own separate builds and tests that would uncover any validation failures. +It is assumed that these targets have their own +separate builds and tests that would uncover any validation failures. #### Using the Validations Output Group -The Validations Output Group is named `_validation` and is used like any other output group: +The Validations Output Group is named `_validation` and is used like any other +output group: ```python def _rule_with_validation_impl(ctx): @@ -705,6 +1119,7 @@ def _rule_with_validation_impl(ctx): OutputGroupInfo(_validation = depset([validation_output])), ] + rule_with_validation = rule( implementation = _rule_with_validation_impl, outputs = { @@ -721,9 +1136,17 @@ rule_with_validation = rule( ) ``` -Notice that the validation output file is not added to the `DefaultInfo` or the inputs to any other action. The validation action for a target of this rule kind will still run if the target is depended upon by label, or any of the target's implicit outputs are directly or indirectly depended upon. +Notice that the validation output file is not added to the `DefaultInfo` or the +inputs to any other action. The validation action for a target of this rule kind +will still run if the target is depended upon by label, or any of the target's +implicit outputs are directly or indirectly depended upon. -It is usually important that the outputs of validation actions only go into the validation output group, and are not added to the inputs of other actions, as this could defeat parallelism gains. Note however that Bazel doesn't have any special checking to enforce this. Therefore, you should test that validation action outputs are not added to the inputs of any actions in the tests for Starlark rules. For example: +It is usually important that the outputs of validation actions only go into the +validation output group, and are not added to the inputs of other actions, as +this could defeat parallelism gains. Note however that Bazel doesn't +have any special checking to enforce this. Therefore, you should test +that validation action outputs are not added to the inputs of any actions in the +tests for Starlark rules. For example: ```python load("@bazel_skylib//lib:unittest.bzl", "analysistest") @@ -748,7 +1171,8 @@ validation_outputs_test = analysistest.make(_validation_outputs_test_impl) #### Validation Actions Flag -Running validation actions is controlled by the `--run_validations` command line flag, which defaults to true. +Running validation actions is controlled by the `--run_validations` command line +flag, which defaults to true. ## Deprecated features @@ -756,23 +1180,52 @@ Running validation actions is controlled by the `--run_validations` command line There are two **deprecated** ways of using predeclared outputs: -- The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies a mapping between output attribute names and string templates for generating predeclared output labels. Prefer using non-predeclared outputs and explicitly adding outputs to `DefaultInfo.files`. Use the rule target's label as input for rules which consume the output instead of a predeclared output's label. +* The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies + a mapping between output attribute names and string templates for generating + predeclared output labels. Prefer using non-predeclared outputs and + explicitly adding outputs to `DefaultInfo.files`. Use the rule target's + label as input for rules which consume the output instead of a predeclared + output's label. -- For [executable rules](#executable-rules), `ctx.outputs.executable` refers to a predeclared executable output with the same name as the rule target. Prefer declaring the output explicitly, for example with `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that generates the executable sets its permissions to allow execution. Explicitly pass the executable output to the `executable` parameter of `DefaultInfo`. +* For [executable rules](#executable-rules), `ctx.outputs.executable` refers + to a predeclared executable output with the same name as the rule target. + Prefer declaring the output explicitly, for example with + `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that + generates the executable sets its permissions to allow execution. Explicitly + pass the executable output to the `executable` parameter of `DefaultInfo`. ### Runfiles features to avoid -[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) type have a complex set of features, many of which are kept for legacy reasons. The following recommendations help reduce complexity: - -- **Avoid** use of the `collect_data` and `collect_default` modes of [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect runfiles across certain hardcoded dependency edges in confusing ways. Instead, add files using the `files` or `transitive_files` parameters of `ctx.runfiles`, or by merging in runfiles from dependencies with `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. - -- **Avoid** use of the `data_runfiles` and `default_runfiles` of the `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. The distinction between "default" and "data" runfiles is maintained for legacy reasons. For example, some rules put their default outputs in `data_runfiles`, but not `default_runfiles`. Instead of using `data_runfiles`, rules should *both* include default outputs and merge in `default_runfiles` from attributes which provide runfiles (often [`data`](/reference/be/common-definitions#common-attributes.data)). - -- When retrieving `runfiles` from `DefaultInfo` (generally only for merging runfiles between the current rule and its dependencies), use `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. +[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) +type have a complex set of features, many of which are kept for legacy reasons. +The following recommendations help reduce complexity: + +* **Avoid** use of the `collect_data` and `collect_default` modes of + [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect + runfiles across certain hardcoded dependency edges in confusing ways. + Instead, add files using the `files` or `transitive_files` parameters of + `ctx.runfiles`, or by merging in runfiles from dependencies with + `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. + +* **Avoid** use of the `data_runfiles` and `default_runfiles` of the + `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. + The distinction between "default" and "data" runfiles is maintained for + legacy reasons. For example, some rules put their default outputs in + `data_runfiles`, but not `default_runfiles`. Instead of using + `data_runfiles`, rules should *both* include default outputs and merge in + `default_runfiles` from attributes which provide runfiles (often + [`data`](/reference/be/common-definitions#common-attributes.data)). + +* When retrieving `runfiles` from `DefaultInfo` (generally only for merging + runfiles between the current rule and its dependencies), use + `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. ### Migrating from legacy providers -Historically, Bazel providers were simple fields on the `Target` object. They were accessed using the dot operator, and they were created by putting the field in a [`struct`](/rules/lib/builtins/struct) returned by the rule's implementation function instead of a list of provider objects: +Historically, Bazel providers were simple fields on the `Target` object. They +were accessed using the dot operator, and they were created by putting the field +in a [`struct`](/rules/lib/builtins/struct) returned by the rule's +implementation function instead of a list of provider objects: ```python return struct(example_info = struct(headers = depset(...))) @@ -784,9 +1237,13 @@ Such providers can be retrieved from the corresponding field of the `Target` obj transitive_headers = [hdr.example_info.headers for hdr in ctx.attr.hdrs] ``` -*This style is deprecated and should not be used in new code;* see following for information that may help you migrate. The new provider mechanism avoids name clashes. It also supports data hiding, by requiring any code accessing a provider instance to retrieve it using the provider symbol. +*This style is deprecated and should not be used in new code;* see following for +information that may help you migrate. The new provider mechanism avoids name +clashes. It also supports data hiding, by requiring any code accessing a +provider instance to retrieve it using the provider symbol. -For the moment, legacy providers are still supported. A rule can return both legacy and modern providers as follows: +For the moment, legacy providers are still supported. A rule can return both +legacy and modern providers as follows: ```python def _old_rule_impl(ctx): @@ -803,18 +1260,40 @@ def _old_rule_impl(ctx): providers = [modern_data, ...]) ``` -If `dep` is the resulting `Target` object for an instance of this rule, the providers and their contents can be retrieved as `dep.legacy_info.x` and `dep[MyInfo].y`. - -In addition to `providers`, the returned struct can also take several other fields that have special meaning (and thus don't create a corresponding legacy provider): - -- The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and `executable` correspond to the same-named fields of [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of these fields while also returning a `DefaultInfo` provider. - -- The field `output_groups` takes a struct value and corresponds to an [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). - -In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in [`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency attributes, legacy providers are passed in as strings and modern providers are passed in by their `Info` symbol. Be sure to change from strings to symbols when migrating. For complex or large rule sets where it is difficult to update all rules atomically, you may have an easier time if you follow this sequence of steps: - -1. Modify the rules that produce the legacy provider to produce both the legacy and modern providers, using the preceding syntax. For rules that declare they return the legacy provider, update that declaration to include both the legacy and modern providers. - -2. Modify the rules that consume the legacy provider to instead consume the modern provider. If any attribute declarations require the legacy provider, also update them to instead require the modern provider. Optionally, you can interleave this work with step 1 by having consumers accept or require either provider: Test for the presence of the legacy provider using `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. - -3. Fully remove the legacy provider from all rules. +If `dep` is the resulting `Target` object for an instance of this rule, the +providers and their contents can be retrieved as `dep.legacy_info.x` and +`dep[MyInfo].y`. + +In addition to `providers`, the returned struct can also take several other +fields that have special meaning (and thus don't create a corresponding legacy +provider): + +* The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and + `executable` correspond to the same-named fields of + [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of + these fields while also returning a `DefaultInfo` provider. + +* The field `output_groups` takes a struct value and corresponds to an + [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). + +In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in +[`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency +attributes, legacy providers are passed in as strings and modern providers are +passed in by their `Info` symbol. Be sure to change from strings to symbols +when migrating. For complex or large rule sets where it is difficult to update +all rules atomically, you may have an easier time if you follow this sequence of +steps: + +1. Modify the rules that produce the legacy provider to produce both the legacy + and modern providers, using the preceding syntax. For rules that declare they + return the legacy provider, update that declaration to include both the + legacy and modern providers. + +2. Modify the rules that consume the legacy provider to instead consume the + modern provider. If any attribute declarations require the legacy provider, + also update them to instead require the modern provider. Optionally, you can + interleave this work with step 1 by having consumers accept or require either + provider: Test for the presence of the legacy provider using + `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. + +3. Fully remove the legacy provider from all rules. diff --git a/extending/toolchains.mdx b/extending/toolchains.mdx index ef07d948..0d92d6e1 100644 --- a/extending/toolchains.mdx +++ b/extending/toolchains.mdx @@ -2,11 +2,23 @@ title: 'Toolchains' --- -This page describes the toolchain framework, which is a way for rule authors to decouple their rule logic from platform-based selection of tools. It is recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) pages before continuing. This page covers why toolchains are needed, how to define and use them, and how Bazel selects an appropriate toolchain based on platform constraints. + + +This page describes the toolchain framework, which is a way for rule authors to +decouple their rule logic from platform-based selection of tools. It is +recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) +pages before continuing. This page covers why toolchains are needed, how to +define and use them, and how Bazel selects an appropriate toolchain based on +platform constraints. ## Motivation -Let's first look at the problem toolchains are designed to solve. Suppose you are writing rules to support the "bar" programming language. Your `bar_binary` rule would compile `*.bar` files using the `barc` compiler, a tool that itself is built as another target in your workspace. Since users who write `bar_binary` targets shouldn't have to specify a dependency on the compiler, you make it an implicit dependency by adding it to the rule definition as a private attribute. +Let's first look at the problem toolchains are designed to solve. Suppose you +are writing rules to support the "bar" programming language. Your `bar_binary` +rule would compile `*.bar` files using the `barc` compiler, a tool that itself +is built as another target in your workspace. Since users who write `bar_binary` +targets shouldn't have to specify a dependency on the compiler, you make it an +implicit dependency by adding it to the rule definition as a private attribute. ```python bar_binary = rule( @@ -22,7 +34,9 @@ bar_binary = rule( ) ``` -`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so it'll be built before any `bar_binary` target. It can be accessed by the rule's implementation function just like any other attribute: +`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so +it'll be built before any `bar_binary` target. It can be accessed by the rule's +implementation function just like any other attribute: ```python BarcInfo = provider( @@ -44,9 +58,16 @@ def _bar_binary_impl(ctx): ... ``` -The issue here is that the compiler's label is hardcoded into `bar_binary`, yet different targets may need different compilers depending on what platform they are being built for and what platform they are being built on -- called the *target platform* and *execution platform*, respectively. Furthermore, the rule author does not necessarily even know all the available tools and platforms, so it is not feasible to hardcode them in the rule's definition. +The issue here is that the compiler's label is hardcoded into `bar_binary`, yet +different targets may need different compilers depending on what platform they +are being built for and what platform they are being built on -- called the +*target platform* and *execution platform*, respectively. Furthermore, the rule +author does not necessarily even know all the available tools and platforms, so +it is not feasible to hardcode them in the rule's definition. -A less-than-ideal solution would be to shift the burden onto users, by making the `_compiler` attribute non-private. Then individual targets could be hardcoded to build for one platform or another. +A less-than-ideal solution would be to shift the burden onto users, by making +the `_compiler` attribute non-private. Then individual targets could be +hardcoded to build for one platform or another. ```python bar_binary( @@ -62,7 +83,8 @@ bar_binary( ) ``` -You can improve on this solution by using `select` to choose the `compiler` [based on the platform](/docs/configurable-attributes): +You can improve on this solution by using `select` to choose the `compiler` +[based on the platform](/docs/configurable-attributes): ```python config_setting( @@ -89,13 +111,26 @@ bar_binary( ) ``` -But this is tedious and a bit much to ask of every single `bar_binary` user. If this style is not used consistently throughout the workspace, it leads to builds that work fine on a single platform but fail when extended to multi-platform scenarios. It also does not address the problem of adding support for new platforms and compilers without modifying existing rules or targets. +But this is tedious and a bit much to ask of every single `bar_binary` user. +If this style is not used consistently throughout the workspace, it leads to +builds that work fine on a single platform but fail when extended to +multi-platform scenarios. It also does not address the problem of adding support +for new platforms and compilers without modifying existing rules or targets. -The toolchain framework solves this problem by adding an extra level of indirection. Essentially, you declare that your rule has an abstract dependency on *some* member of a family of targets (a toolchain type), and Bazel automatically resolves this to a particular target (a toolchain) based on the applicable platform constraints. Neither the rule author nor the target author need know the complete set of available platforms and toolchains. +The toolchain framework solves this problem by adding an extra level of +indirection. Essentially, you declare that your rule has an abstract dependency +on *some* member of a family of targets (a toolchain type), and Bazel +automatically resolves this to a particular target (a toolchain) based on the +applicable platform constraints. Neither the rule author nor the target author +need know the complete set of available platforms and toolchains. ## Writing rules that use toolchains -Under the toolchain framework, instead of having rules depend directly on tools, they instead depend on *toolchain types*. A toolchain type is a simple target that represents a class of tools that serve the same role for different platforms. For instance, you can declare a type that represents the bar compiler: +Under the toolchain framework, instead of having rules depend directly on tools, +they instead depend on *toolchain types*. A toolchain type is a simple target +that represents a class of tools that serve the same role for different +platforms. For instance, you can declare a type that represents the bar +compiler: ```python # By convention, toolchain_type targets are named "toolchain_type" and @@ -104,7 +139,9 @@ Under the toolchain framework, instead of having rules depend directly on tools, toolchain_type(name = "toolchain_type") ``` -The rule definition in the previous section is modified so that instead of taking in the compiler as an attribute, it declares that it consumes a `//bar_tools:toolchain_type` toolchain. +The rule definition in the previous section is modified so that instead of +taking in the compiler as an attribute, it declares that it consumes a +`//bar_tools:toolchain_type` toolchain. ```python bar_binary = rule( @@ -118,7 +155,8 @@ bar_binary = rule( ) ``` -The implementation function now accesses this dependency under `ctx.toolchains` instead of `ctx.attr`, using the toolchain type as the key. +The implementation function now accesses this dependency under `ctx.toolchains` +instead of `ctx.attr`, using the toolchain type as the key. ```python def _bar_binary_impl(ctx): @@ -133,15 +171,28 @@ def _bar_binary_impl(ctx): ... ``` -`ctx.toolchains["//bar_tools:toolchain_type"]` returns the [`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) of whatever target Bazel resolved the toolchain dependency to. The fields of the `ToolchainInfo` object are set by the underlying tool's rule; in the next section, this rule is defined such that there is a `barcinfo` field that wraps a `BarcInfo` object. +`ctx.toolchains["//bar_tools:toolchain_type"]` returns the +[`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) +of whatever target Bazel resolved the toolchain dependency to. The fields of the +`ToolchainInfo` object are set by the underlying tool's rule; in the next +section, this rule is defined such that there is a `barcinfo` field that wraps +a `BarcInfo` object. -Bazel's procedure for resolving toolchains to targets is described [below](#toolchain-resolution). Only the resolved toolchain target is actually made a dependency of the `bar_binary` target, not the whole space of candidate toolchains. +Bazel's procedure for resolving toolchains to targets is described +[below](#toolchain-resolution). Only the resolved toolchain target is actually +made a dependency of the `bar_binary` target, not the whole space of candidate +toolchains. ### Mandatory and Optional Toolchains -By default, when a rule expresses a toolchain type dependency using a bare label (as shown above), the toolchain type is considered to be **mandatory**. If Bazel is unable to find a matching toolchain (see [Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain type, this is an error and analysis halts. +By default, when a rule expresses a toolchain type dependency using a bare label +(as shown above), the toolchain type is considered to be **mandatory**. If Bazel +is unable to find a matching toolchain (see +[Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain +type, this is an error and analysis halts. -It is possible instead to declare an **optional** toolchain type dependency, as follows: +It is possible instead to declare an **optional** toolchain type dependency, as +follows: ```python bar_binary = rule( @@ -152,20 +203,20 @@ bar_binary = rule( ) ``` -When an optional toolchain type cannot be resolved, analysis continues, and the result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. +When an optional toolchain type cannot be resolved, analysis continues, and the +result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. -The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) function defaults to mandatory. +The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) +function defaults to mandatory. The following forms can be used: -- Mandatory toolchain types: - - - `toolchains = ["//bar_tools:toolchain_type"]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` - +- Mandatory toolchain types: + - `toolchains = ["//bar_tools:toolchain_type"]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` - Optional toolchain types: - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` ```python bar_binary = rule( @@ -177,11 +228,15 @@ bar_binary = rule( ) ``` -You can mix and match forms in the same rule, also. However, if the same toolchain type is listed multiple times, it will take the most strict version, where mandatory is more strict than optional. +You can mix and match forms in the same rule, also. However, if the same +toolchain type is listed multiple times, it will take the most strict version, +where mandatory is more strict than optional. ### Writing aspects that use toolchains -Aspects have access to the same toolchain API as rules: you can define required toolchain types, access toolchains via the context, and use them to generate new actions using the toolchain. +Aspects have access to the same toolchain API as rules: you can define required +toolchain types, access toolchains via the context, and use them to generate new +actions using the toolchain. ```py bar_aspect = aspect( @@ -200,15 +255,28 @@ def _bar_aspect_impl(target, ctx): To define some toolchains for a given toolchain type, you need three things: -1. A language-specific rule representing the kind of tool or tool suite. By convention this rule's name is suffixed with "\_toolchain". +1. A language-specific rule representing the kind of tool or tool suite. By + convention this rule's name is suffixed with "\_toolchain". - 1. **Note:** The `\_toolchain` rule cannot create any build actions. Rather, it collects artifacts from other rules and forwards them to the rule that uses the toolchain. That rule is responsible for creating all build actions. + 1. **Note:** The `\_toolchain` rule cannot create any build actions. + Rather, it collects artifacts from other rules and forwards them to the + rule that uses the toolchain. That rule is responsible for creating all + build actions. -2. Several targets of this rule type, representing versions of the tool or tool suite for different platforms. +2. Several targets of this rule type, representing versions of the tool or tool + suite for different platforms. -3. For each such target, an associated target of the generic [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) rule, to provide metadata used by the toolchain framework. This `toolchain` target also refers to the `toolchain_type` associated with this toolchain. This means that a given `_toolchain` rule could be associated with any `toolchain_type`, and that only in a `toolchain` instance that uses this `_toolchain` rule that the rule is associated with a `toolchain_type`. +3. For each such target, an associated target of the generic + [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) + rule, to provide metadata used by the toolchain framework. This `toolchain` + target also refers to the `toolchain_type` associated with this toolchain. + This means that a given `_toolchain` rule could be associated with any + `toolchain_type`, and that only in a `toolchain` instance that uses + this `_toolchain` rule that the rule is associated with a `toolchain_type`. -For our running example, here's a definition for a `bar_toolchain` rule. Our example has only a compiler, but other tools such as a linker could also be grouped underneath it. +For our running example, here's a definition for a `bar_toolchain` rule. Our +example has only a compiler, but other tools such as a linker could also be +grouped underneath it. ```python def _bar_toolchain_impl(ctx): @@ -231,7 +299,13 @@ bar_toolchain = rule( ) ``` -The rule must return a `ToolchainInfo` provider, which becomes the object that the consuming rule retrieves using `ctx.toolchains` and the label of the toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value pairs. The specification of exactly what fields are added to the `ToolchainInfo` should be clearly documented at the toolchain type. In this example, the values return wrapped in a `BarcInfo` object to reuse the schema defined above; this style may be useful for validation and code reuse. +The rule must return a `ToolchainInfo` provider, which becomes the object that +the consuming rule retrieves using `ctx.toolchains` and the label of the +toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value +pairs. The specification of exactly what fields are added to the `ToolchainInfo` +should be clearly documented at the toolchain type. In this example, the values +return wrapped in a `BarcInfo` object to reuse the schema defined above; this +style may be useful for validation and code reuse. Now you can define targets for specific `barc` compilers. @@ -257,7 +331,10 @@ bar_toolchain( ) ``` -Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. These definitions link the language-specific targets to the toolchain type and provide the constraint information that tells Bazel when the toolchain is appropriate for a given platform. +Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. +These definitions link the language-specific targets to the toolchain type and +provide the constraint information that tells Bazel when the toolchain is +appropriate for a given platform. ```python toolchain( @@ -289,13 +366,21 @@ toolchain( ) ``` -The use of relative path syntax above suggests these definitions are all in the same package, but there's no reason the toolchain type, language-specific toolchain targets, and `toolchain` definition targets can't all be in separate packages. +The use of relative path syntax above suggests these definitions are all in the +same package, but there's no reason the toolchain type, language-specific +toolchain targets, and `toolchain` definition targets can't all be in separate +packages. -See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) for a real-world example. +See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) +for a real-world example. ### Toolchains and configurations -An important question for rule authors is, when a `bar_toolchain` target is analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions should be used for dependencies? The example above uses string attributes, but what would happen for a more complicated toolchain that depends on other targets in the Bazel repository? +An important question for rule authors is, when a `bar_toolchain` target is +analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions +should be used for dependencies? The example above uses string attributes, but +what would happen for a more complicated toolchain that depends on other targets +in the Bazel repository? Let's see a more complex version of `bar_toolchain`: @@ -321,13 +406,32 @@ bar_toolchain = rule( ) ``` -The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, but the meaning of the `cfg` parameter is slightly different. - -The dependency from a target (called the "parent") to a toolchain via toolchain resolution uses a special configuration transition called the "toolchain transition". The toolchain transition keeps the configuration the same, except that it forces the execution platform to be the same for the toolchain as for the parent (otherwise, toolchain resolution for the toolchain could pick any execution platform, and wouldn't necessarily be the same as for parent). This allows any `exec` dependencies of the toolchain to also be executable for the parent's build actions. Any of the toolchain's dependencies which use `cfg = "target"` (or which don't specify `cfg`, since "target" is the default) are built for the same target platform as the parent. This allows toolchain rules to contribute both libraries (the `system_lib` attribute above) and tools (the `compiler` attribute) to the build rules which need them. The system libraries are linked into the final artifact, and so need to be built for the same platform, whereas the compiler is a tool invoked during the build, and needs to be able to run on the execution platform. +The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, +but the meaning of the `cfg` parameter is slightly different. + +The dependency from a target (called the "parent") to a toolchain via toolchain +resolution uses a special configuration transition called the "toolchain +transition". The toolchain transition keeps the configuration the same, except +that it forces the execution platform to be the same for the toolchain as for +the parent (otherwise, toolchain resolution for the toolchain could pick any +execution platform, and wouldn't necessarily be the same as for parent). This +allows any `exec` dependencies of the toolchain to also be executable for the +parent's build actions. Any of the toolchain's dependencies which use `cfg = +"target"` (or which don't specify `cfg`, since "target" is the default) are +built for the same target platform as the parent. This allows toolchain rules to +contribute both libraries (the `system_lib` attribute above) and tools (the +`compiler` attribute) to the build rules which need them. The system libraries +are linked into the final artifact, and so need to be built for the same +platform, whereas the compiler is a tool invoked during the build, and needs to +be able to run on the execution platform. ## Registering and building with toolchains -At this point all the building blocks are assembled, and you just need to make the toolchains available to Bazel's resolution procedure. This is done by registering the toolchain, either in a `MODULE.bazel` file using `register_toolchains()`, or by passing the toolchains' labels on the command line using the `--extra_toolchains` flag. +At this point all the building blocks are assembled, and you just need to make +the toolchains available to Bazel's resolution procedure. This is done by +registering the toolchain, either in a `MODULE.bazel` file using +`register_toolchains()`, or by passing the toolchains' labels on the command +line using the `--extra_toolchains` flag. ```python register_toolchains( @@ -340,12 +444,16 @@ register_toolchains( ) ``` -When using target patterns to register toolchains, the order in which the individual toolchains are registered is determined by the following rules: +When using target patterns to register toolchains, the order in which the +individual toolchains are registered is determined by the following rules: -- The toolchains defined in a subpackage of a package are registered before the toolchains defined in the package itself. -- Within a package, toolchains are registered in the lexicographical order of their names. +* The toolchains defined in a subpackage of a package are registered before the + toolchains defined in the package itself. +* Within a package, toolchains are registered in the lexicographical order of + their names. -Now when you build a target that depends on a toolchain type, an appropriate toolchain will be selected based on the target and execution platforms. +Now when you build a target that depends on a toolchain type, an appropriate +toolchain will be selected based on the target and execution platforms. ```python # my_pkg/BUILD @@ -367,52 +475,109 @@ bar_binary( bazel build //my_pkg:my_bar_binary --platforms=//my_pkg:my_target_platform ``` -Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that has `@platforms//os:linux` and therefore resolve the `//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. This will end up building `//bar_tools:barc_linux` but not `//bar_tools:barc_windows`. +Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that +has `@platforms//os:linux` and therefore resolve the +`//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. +This will end up building `//bar_tools:barc_linux` but not +`//bar_tools:barc_windows`. ## Toolchain resolution -Note: [Some Bazel rules](/concepts/platforms#status) do not yet support toolchain resolution. - -For each target that uses toolchains, Bazel's toolchain resolution procedure determines the target's concrete toolchain dependencies. The procedure takes as input a set of required toolchain types, the target platform, the list of available execution platforms, and the list of available toolchains. Its outputs are a selected toolchain for each toolchain type as well as a selected execution platform for the current target. - -The available execution platforms and toolchains are gathered from the external dependency graph via [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) and [`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in `MODULE.bazel` files. Additional execution platforms and toolchains may also be specified on the command line via [`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) and [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). The host platform is automatically included as an available execution platform. Available platforms and toolchains are tracked as ordered lists for determinism, with preference given to earlier items in the list. - -The set of available toolchains, in priority order, is created from `--extra_toolchains` and `register_toolchains`: - -1. Toolchains registered using `--extra_toolchains` are added first. (Within these, the **last** toolchain has highest priority.) -2. Toolchains registered using `register_toolchains` in the transitive external dependency graph, in the following order: (Within these, the **first** mentioned toolchain has highest priority.) -3. Toolchains registered by the root module (as in, the `MODULE.bazel` at the workspace root); -4. Toolchains registered in the user's `WORKSPACE` file, including in any macros invoked from there; -5. Toolchains registered by non-root modules (as in, dependencies specified by the root module, and their dependencies, and so forth); -6. Toolchains registered in the "WORKSPACE suffix"; this is only used by certain native rules bundled with the Bazel installation. - -**NOTE:** [Pseudo-targets like `:all`, `:*`, and `/...`](/run/build#specifying-build-targets) are ordered by Bazel's package loading mechanism, which uses a lexicographic ordering. +Note: [Some Bazel rules](/concepts/platforms#status) do not yet support +toolchain resolution. + +For each target that uses toolchains, Bazel's toolchain resolution procedure +determines the target's concrete toolchain dependencies. The procedure takes as +input a set of required toolchain types, the target platform, the list of +available execution platforms, and the list of available toolchains. Its outputs +are a selected toolchain for each toolchain type as well as a selected execution +platform for the current target. + +The available execution platforms and toolchains are gathered from the +external dependency graph via +[`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) +and +[`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in +`MODULE.bazel` files. +Additional execution platforms and toolchains may also be specified on the +command line via +[`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) +and +[`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). +The host platform is automatically included as an available execution platform. +Available platforms and toolchains are tracked as ordered lists for determinism, +with preference given to earlier items in the list. + +The set of available toolchains, in priority order, is created from +`--extra_toolchains` and `register_toolchains`: + +1. Toolchains registered using `--extra_toolchains` are added first. (Within + these, the **last** toolchain has highest priority.) +2. Toolchains registered using `register_toolchains` in the transitive external + dependency graph, in the following order: (Within these, the **first** + mentioned toolchain has highest priority.) + 1. Toolchains registered by the root module (as in, the `MODULE.bazel` at the + workspace root); + 2. Toolchains registered in the user's `WORKSPACE` file, including in any + macros invoked from there; + 3. Toolchains registered by non-root modules (as in, dependencies specified by + the root module, and their dependencies, and so forth); + 4. Toolchains registered in the "WORKSPACE suffix"; this is only used by + certain native rules bundled with the Bazel installation. + +**NOTE:** [Pseudo-targets like `:all`, `:*`, and +`/...`](/run/build#specifying-build-targets) are ordered by Bazel's package +loading mechanism, which uses a lexicographic ordering. The resolution steps are as follows. -1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a platform if, for each `constraint_value` in its list, the platform also has that `constraint_value` (either explicitly or as a default). +1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a + platform if, for each `constraint_value` in its list, the platform also has + that `constraint_value` (either explicitly or as a default). - If the platform has `constraint_value`s from `constraint_setting`s not referenced by the clause, these do not affect matching. + If the platform has `constraint_value`s from `constraint_setting`s not + referenced by the clause, these do not affect matching. -2. If the target being built specifies the [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) (or its rule definition specifies the [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), the list of available execution platforms is filtered to remove any that do not match the execution constraints. +1. If the target being built specifies the + [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) + (or its rule definition specifies the + [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), + the list of available execution platforms is filtered to remove + any that do not match the execution constraints. -3. The list of available toolchains is filtered to remove any toolchains specifying `target_settings` that don't match the current configuration. +1. The list of available toolchains is filtered to remove any toolchains + specifying `target_settings` that don't match the current configuration. -4. For each available execution platform, you associate each toolchain type with the first available toolchain, if any, that is compatible with this execution platform and the target platform. +1. For each available execution platform, you associate each toolchain type with + the first available toolchain, if any, that is compatible with this execution + platform and the target platform. -5. Any execution platform that failed to find a compatible mandatory toolchain for one of its toolchain types is ruled out. Of the remaining platforms, the first one becomes the current target's execution platform, and its associated toolchains (if any) become dependencies of the target. +1. Any execution platform that failed to find a compatible mandatory toolchain + for one of its toolchain types is ruled out. Of the remaining platforms, the + first one becomes the current target's execution platform, and its associated + toolchains (if any) become dependencies of the target. -The chosen execution platform is used to run all actions that the target generates. +The chosen execution platform is used to run all actions that the target +generates. -In cases where the same target can be built in multiple configurations (such as for different CPUs) within the same build, the resolution procedure is applied independently to each version of the target. +In cases where the same target can be built in multiple configurations (such as +for different CPUs) within the same build, the resolution procedure is applied +independently to each version of the target. -If the rule uses [execution groups](/extending/exec-groups), each execution group performs toolchain resolution separately, and each has its own execution platform and toolchains. +If the rule uses [execution groups](/extending/exec-groups), each execution +group performs toolchain resolution separately, and each has its own execution +platform and toolchains. ## Debugging toolchains -If you are adding toolchain support to an existing rule, use the `--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag provides verbose output for toolchain types or target names that match the regex variable. You can use `.*` to output all information. Bazel will output names of toolchains it checks and skips during the resolution process. +If you are adding toolchain support to an existing rule, use the +`--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag +provides verbose output for toolchain types or target names that match the regex variable. You +can use `.*` to output all information. Bazel will output names of toolchains it +checks and skips during the resolution process. -For example, to debug toolchain selection for all actions created directly by `//my:target`: +For example, to debug toolchain selection for all actions created directly by +`//my:target`: ```sh $ bazel build //my:all --toolchain_resolution_debug=//my:target @@ -424,7 +589,8 @@ To debug toolchain selection for all actions over all build targets: $ bazel build //my:all --toolchain_resolution_debug=.* ``` -If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: +If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain +resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: ``` # Find all direct dependencies of //cc:my_cc_lib. This includes explicitly @@ -443,5 +609,5 @@ $ bazel cquery 'deps(//cc:my_cc_lib, 1)' # Which of these are from toolchain resolution? $ bazel cquery 'deps(//cc:my_cc_lib, 1)' --transitions=lite | grep "toolchain dependency" - [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 + [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 ``` diff --git a/external/extension.mdx b/external/extension.mdx index 679096e2..ababc7fe 100644 --- a/external/extension.mdx +++ b/external/extension.mdx @@ -2,20 +2,42 @@ title: 'Module extensions' --- -Module extensions allow users to extend the module system by reading input data from modules across the dependency graph, performing necessary logic to resolve dependencies, and finally creating repos by calling [repo rules](/external/repo). These extensions have capabilities similar to repo rules, which enables them to perform file I/O, send network requests, and so on. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. -You can define module extensions in `.bzl` files, just like repo rules. They're not invoked directly; rather, each module specifies pieces of data called *tags* for extensions to read. Bazel runs module resolution before evaluating any extensions. The extension reads all the tags belonging to it across the entire dependency graph. + +Module extensions allow users to extend the module system by reading input data +from modules across the dependency graph, performing necessary logic to resolve +dependencies, and finally creating repos by calling [repo +rules](/external/repo). These extensions have capabilities similar to repo +rules, which enables them to perform file I/O, send network requests, and so on. +Among other things, they allow Bazel to interact with other package management +systems while also respecting the dependency graph built out of Bazel modules. + +You can define module extensions in `.bzl` files, just like repo rules. They're +not invoked directly; rather, each module specifies pieces of data called *tags* +for extensions to read. Bazel runs module resolution before evaluating any +extensions. The extension reads all the tags belonging to it across the entire +dependency graph. ## Extension usage -Extensions are hosted in Bazel modules themselves. To use an extension in a module, first add a `bazel_dep` on the module hosting the extension, and then call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function to bring it into scope. Consider the following example — a snippet from a `MODULE.bazel` file to use the "maven" extension defined in the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) module: +Extensions are hosted in Bazel modules themselves. To use an extension in a +module, first add a `bazel_dep` on the module hosting the extension, and then +call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function +to bring it into scope. Consider the following example — a snippet from a +`MODULE.bazel` file to use the "maven" extension defined in the +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) +module: ```python bazel_dep(name = "rules_jvm_external", version = "4.5") maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -This binds the return value of `use_extension` to a variable, which allows the user to use dot-syntax to specify tags for the extension. The tags must follow the schema defined by the corresponding *tag classes* specified in the [extension definition](#extension_definition). For an example specifying some `maven.install` and `maven.artifact` tags: +This binds the return value of `use_extension` to a variable, which allows the +user to use dot-syntax to specify tags for the extension. The tags must follow +the schema defined by the corresponding *tag classes* specified in the +[extension definition](#extension_definition). For an example specifying some +`maven.install` and `maven.artifact` tags: ```python maven.install(artifacts = ["org.junit:junit:4.13.2"]) @@ -25,19 +47,33 @@ maven.artifact(group = "com.google.guava", exclusions = ["com.google.j2objc:j2objc-annotations"]) ``` -Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos generated by the extension into the scope of the current module. +Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos +generated by the extension into the scope of the current module. ```python use_repo(maven, "maven") ``` -Repos generated by an extension are part of its API. In this example, the "maven" module extension promises to generate a repo called `maven`. With the declaration above, the extension properly resolves labels such as `@maven//:org_junit_junit` to point to the repo generated by the "maven" extension. +Repos generated by an extension are part of its API. In this example, the +"maven" module extension promises to generate a repo called `maven`. With the +declaration above, the extension properly resolves labels such as +`@maven//:org_junit_junit` to point to the repo generated by the "maven" +extension. -Note: Module extensions are evaluated lazily. This means that an extension will typically not be evaluated unless some module brings one of its repositories into scope using `use_repo` and that repository is referenced in a build. While testing a module extension, `bazel mod deps` can be useful as it unconditionally evaluates all module extensions. +Note: Module extensions are evaluated lazily. This means that an extension will +typically not be evaluated unless some module brings one of its repositories +into scope using `use_repo` and that repository is referenced in a build. While +testing a module extension, `bazel mod deps` can be useful as it +unconditionally evaluates all module extensions. ## Extension definition -You can define module extensions similarly to [repo rules](/external/repo), using the [`module_extension`](/rules/lib/globals/bzl#module_extension) function. However, while repo rules have a number of attributes, module extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which has a number of attributes. The tag classes define schemas for tags used by this extension. For example, the "maven" extension above might be defined like this: +You can define module extensions similarly to [repo rules](/external/repo), +using the [`module_extension`](/rules/lib/globals/bzl#module_extension) +function. However, while repo rules have a number of attributes, module +extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which +has a number of attributes. The tag classes define schemas for tags used by this +extension. For example, the "maven" extension above might be defined like this: ```python # @rules_jvm_external//:extensions.bzl @@ -50,9 +86,13 @@ maven = module_extension( ) ``` -These declarations show that `maven.install` and `maven.artifact` tags can be specified using the specified attribute schema. +These declarations show that `maven.install` and `maven.artifact` tags can be +specified using the specified attribute schema. -The implementation function of module extensions are similar to those of repo rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, which grants access to all modules using the extension and all pertinent tags. The implementation function then calls repo rules to generate repos. +The implementation function of module extensions are similar to those of repo +rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, +which grants access to all modules using the extension and all pertinent tags. +The implementation function then calls repo rules to generate repos. ```python # @rules_jvm_external//:extensions.bzl @@ -80,33 +120,65 @@ def _maven_impl(ctx): ### Extension identity -Module extensions are identified by the name and the `.bzl` file that appears in the call to `use_extension`. In the following example, the extension `maven` is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the name `maven`: +Module extensions are identified by the name and the `.bzl` file that appears +in the call to `use_extension`. In the following example, the extension `maven` +is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the +name `maven`: ```python maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -Re-exporting an extension from a different `.bzl` file gives it a new identity and if both versions of the extension are used in the transitive module graph, then they will be evaluated separately and will only see the tags associated with that particular identity. +Re-exporting an extension from a different `.bzl` file gives it a new identity +and if both versions of the extension are used in the transitive module graph, +then they will be evaluated separately and will only see the tags associated +with that particular identity. -As an extension author you should make sure that users will only use your module extension from one single `.bzl` file. +As an extension author you should make sure that users will only use your +module extension from one single `.bzl` file. ## Repository names and visibility -Repos generated by extensions have canonical names in the form of `<var>module_repo_canonical_name</var>+<var>extension_name</var>+<var>repo_name</var>`. Note that the canonical name format is not an API you should depend on — it's subject to change at any time. - -This naming policy means that each extension has its own "repo namespace"; two distinct extensions can each define a repo with the same name without risking any clashes. It also means that `repository_ctx.name` reports the canonical name of the repo, which is *not* the same as the name specified in the repo rule call. - -Taking repos generated by module extensions into consideration, there are several repo visibility rules: - -- A Bazel module repo can see all repos introduced in its `MODULE.bazel` file via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_repo`](/rules/lib/globals/module#use_repo). -- A repo generated by a module extension can see all repos visible to the module that hosts the extension, *plus* all other repos generated by the same module extension (using the names specified in the repo rule calls as their apparent names). - - This might result in a conflict. If the module repo can see a repo with the apparent name `foo`, and the extension generates a repo with the specified name `foo`, then for all repos generated by that extension `foo` refers to the former. -- Similarly, in a module extension's implementation function, repos created by the extension can refer to each other by their apparent names in attributes, regardless of the order in which they are created. - - In case of a conflict with a repository visible to the module, labels passed to repository rule attributes can be wrapped in a call to [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to the repo visible to the module instead of the extension-generated repo of the same name. +Repos generated by extensions have canonical names in the form of `{{ "" +}}module_repo_canonical_name+extension_name{{ +"" }}+repo_name`. Note that the canonical name +format is not an API you should depend on — it's subject to change at any time. + +This naming policy means that each extension has its own "repo namespace"; two +distinct extensions can each define a repo with the same name without risking +any clashes. It also means that `repository_ctx.name` reports the canonical name +of the repo, which is *not* the same as the name specified in the repo rule +call. + +Taking repos generated by module extensions into consideration, there are +several repo visibility rules: + +* A Bazel module repo can see all repos introduced in its `MODULE.bazel` file + via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and + [`use_repo`](/rules/lib/globals/module#use_repo). +* A repo generated by a module extension can see all repos visible to the + module that hosts the extension, *plus* all other repos generated by the + same module extension (using the names specified in the repo rule calls as + their apparent names). + * This might result in a conflict. If the module repo can see a repo with + the apparent name `foo`, and the extension generates a repo with the + specified name `foo`, then for all repos generated by that extension + `foo` refers to the former. +* Similarly, in a module extension's implementation function, repos created + by the extension can refer to each other by their apparent names in + attributes, regardless of the order in which they are created. + * In case of a conflict with a repository visible to the module, labels + passed to repository rule attributes can be wrapped in a call to + [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to + the repo visible to the module instead of the extension-generated repo + of the same name. ### Overriding and injecting module extension repos -The root module can use [`override_repo`](/rules/lib/globals/module#override_repo) and [`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject module extension repos. +The root module can use +[`override_repo`](/rules/lib/globals/module#override_repo) and +[`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject +module extension repos. #### Example: Replacing `rules_java`'s `java_tools` with a vendored copy @@ -160,32 +232,80 @@ inject_repo(go_deps, "zlib") ## Best practices -This section describes best practices when writing extensions so they are straightforward to use, maintainable, and adapt well to changes over time. +This section describes best practices when writing extensions so they are +straightforward to use, maintainable, and adapt well to changes over time. ### Put each extension in a separate file -When extensions are in a different files, it allows one extension to load repositories generated by another extension. Even if you don't use this functionality, it's best to put them in separate files in case you need it later. This is because the extension's identify is based on its file, so moving the extension into another file later changes your public API and is a backwards incompatible change for your users. +When extensions are in a different files, it allows one extension to load +repositories generated by another extension. Even if you don't use this +functionality, it's best to put them in separate files in case you need it +later. This is because the extension's identify is based on its file, so moving +the extension into another file later changes your public API and is a backwards +incompatible change for your users. ### Specify reproducibility and use facts -If your extension always defines the same repositories given the same inputs (extension tags, files it reads, etc.) and in particular doesn't rely on any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by a checksum, consider returning [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with `reproducible = True`. This allows Bazel to skip this extension when writing to the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces the chance of merge conflicts. Note that Bazel still caches the results of reproducible extensions in a way that persists across server restarts, so even a long-running extension can be marked as reproducible without a performance penalty. - -If your extension relies on effectively immutable data obtained from outside the build, most commonly from the network, but you don't have a checksum available to guard the download, consider using the `facts` parameter of [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to persistently record such data and thus allow your extension to become reproducible. `facts` is expected to be a dictionary with string keys and arbitrary JSON-like Starlark values that is always persisted in the lockfile and available to future evaluations of the extension via the [`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. - -`facts` are not invalidated even when the code of your module extension changes, so be prepared to handle the case where the structure of `facts` changes. Bazel also assumes that two different `facts` dicts produced by two different evaluations of the same extension can be shallowly merged (i.e., as if by using the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` not supporting enumeration of its entries, just lookups by key. - -An example of using `facts` would be to record a mapping from version numbers of some SDK to the an object containing the download URL and checksum of that version. The first time the extension is evaluated, it can fetch this mapping from the network, but on later evaluations it can use the mapping from `facts` to avoid the network requests. +If your extension always defines the same repositories given the same inputs +(extension tags, files it reads, etc.) and in particular doesn't rely on +any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by +a checksum, consider returning +[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with +`reproducible = True`. This allows Bazel to skip this extension when writing to +the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces +the chance of merge conflicts. Note that Bazel still caches the results of +reproducible extensions in a way that persists across server restarts, so even +a long-running extension can be marked as reproducible without a performance +penalty. + +If your extension relies on effectively immutable data obtained from outside +the build, most commonly from the network, but you don't have a checksum +available to guard the download, consider using the `facts` parameter of +[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to +persistently record such data and thus allow your extension to become +reproducible. `facts` is expected to be a dictionary with string keys and +arbitrary JSON-like Starlark values that is always persisted in the lockfile and +available to future evaluations of the extension via the +[`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. + +`facts` are not invalidated even when the code of your module extension changes, +so be prepared to handle the case where the structure of `facts` changes. +Bazel also assumes that two different `facts` dicts produced by two different +evaluations of the same extension can be shallowly merged (i.e., as if by using +the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` +not supporting enumeration of its entries, just lookups by key. + +An example of using `facts` would be to record a mapping from version numbers of +some SDK to the an object containing the download URL and checksum of that +version. The first time the extension is evaluated, it can fetch this mapping +from the network, but on later evaluations it can use the mapping from `facts` +to avoid the network requests. ### Specify dependence on operating system and architecture -If your extension relies on the operating system or its architecture type, ensure to indicate this in the extension definition using the `os_dependent` and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the need for re-evaluation if there are changes to either of them. +If your extension relies on the operating system or its architecture type, +ensure to indicate this in the extension definition using the `os_dependent` +and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the +need for re-evaluation if there are changes to either of them. -Since this kind of dependence on the host makes it more difficult to maintain the lockfile entry for this extension, consider [marking the extension reproducible](#specify_reproducibility) if possible. +Since this kind of dependence on the host makes it more difficult to maintain +the lockfile entry for this extension, consider +[marking the extension reproducible](#specify_reproducibility) if possible. ### Only the root module should directly affect repository names -Remember that when an extension creates repositories, they are created within the namespace of the extension. This means collisions can occur if different modules use the same extension and end up creating a repository with the same name. This often manifests as a module extension's `tag_class` having a `name` argument that is passed as a repository rule's `name` value. - -For example, say the root module, `A`, depends on module `B`. Both modules depend on module `mylang`. If both `A` and `B` call `mylang.toolchain(name="foo")`, they will both try to create a repository named `foo` within the `mylang` module and an error will occur. - -To avoid this, either remove the ability to set the repository name directly, or only allow the root module to do so. It's OK to allow the root module this ability because nothing will depend on it, so it doesn't have to worry about another module creating a conflicting name. +Remember that when an extension creates repositories, they are created within +the namespace of the extension. This means collisions can occur if different +modules use the same extension and end up creating a repository with the same +name. This often manifests as a module extension's `tag_class` having a `name` +argument that is passed as a repository rule's `name` value. + +For example, say the root module, `A`, depends on module `B`. Both modules +depend on module `mylang`. If both `A` and `B` call +`mylang.toolchain(name="foo")`, they will both try to create a repository named +`foo` within the `mylang` module and an error will occur. + +To avoid this, either remove the ability to set the repository name directly, +or only allow the root module to do so. It's OK to allow the root module this +ability because nothing will depend on it, so it doesn't have to worry about +another module creating a conflicting name. diff --git a/external/faq.mdx b/external/faq.mdx index 58d6c521..a0239cb5 100644 --- a/external/faq.mdx +++ b/external/faq.mdx @@ -2,63 +2,145 @@ title: 'Frequently asked questions' --- -This page answers some frequently asked questions about external dependencies in Bazel. -## MODULE.bazel - -### How should I version a Bazel module? -Setting `version` with the [`module`](/rules/lib/globals/module#module) directive in the source archive `MODULE.bazel` can have several downsides and unintended side effects if not managed carefully: -- Duplication: releasing a new version of a module typically involves both incrementing the version in `MODULE.bazel` and tagging the release, two separate steps that can fall out of sync. While automation can reduce this risk, it's simpler and safer to avoid it altogether. -- Inconsistency: users overriding a module with a specific commit using a [non-registry override](module.md#non-registry_overrides) will see an incorrect version. for example, if the `MODULE.bazel` in the source archive sets `version = "0.3.0"` but additional commits have been made since that release, a user overriding with one of those commits would still see `0.3.0`. In reality, the version should reflect that it's ahead of the release, for example `0.3.1-rc1`. +This page answers some frequently asked questions about external dependencies in +Bazel. -- Non-registry override issues: using placeholder values can cause issues when users override a module with a non-registry override. For example, `0.0.0` doesn't sort as the highest version, which is usually the expected behavior users want when doing a non-registry override. - -Thus, it's best to avoid setting the version in the source archive `MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry (e.g., the [Bazel Central Registry](https://registry.bazel.build/)), which is the actual source of truth for the module version during Bazel's external dependency resolution (see [Bazel registries](https://bazel.build/external/registry)). +## MODULE.bazel -This is usually automated, for example the [`rules-template`](https://github.com/bazel-contrib/rules-template) example rule repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml) to publish the release to the BCR. The action [generates a patch for the source archive `MODULE.bazel`](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216) with the release version. This patch is stored in the registry and is applied when the module is fetched during Bazel's external dependency resolution. +### How should I version a Bazel module? -This way, the version in the releases in the registry will be correctly set to the released version and thus, `bazel_dep`, `single_version_override` and `multiple_version_override` will work as expected, while avoiding potential issues when doing a non-registry override because the version in the source archive will be the default value (`''`), which will always be handled correctly (it's the default version value after all) and will behave as expected when sorting (the empty string is treated as the highest version). +Setting `version` with the [`module`] directive in the source archive +`MODULE.bazel` can have several downsides and unintended side effects if not +managed carefully: + +* Duplication: releasing a new version of a module typically involves both + incrementing the version in `MODULE.bazel` and tagging the release, two + separate steps that can fall out of sync. While automation can + reduce this risk, it's simpler and safer to avoid it altogether. + +* Inconsistency: users overriding a module with a specific commit using a + [non-registry override] will see an incorrect version. for example, if the + `MODULE.bazel` in the source archive sets `version = "0.3.0"` but + additional commits have been made since that release, a user overriding + with one of those commits would still see `0.3.0`. In reality, the version + should reflect that it's ahead of the release, for example `0.3.1-rc1`. + +* Non-registry override issues: using placeholder values can cause issues + when users override a module with a non-registry override. For example, + `0.0.0` doesn't sort as the highest version, which is usually the expected + behavior users want when doing a non-registry override. + +Thus, it's best to avoid setting the version in the source archive +`MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry +(e.g., the [Bazel Central Registry]), which is the actual source of truth for +the module version during Bazel's external dependency resolution (see [Bazel +registries]). + +This is usually automated, for example the [`rules-template`] example rule +repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action] to +publish the release to the BCR. The action [generates a patch for the source +archive `MODULE.bazel`] with the release version. This patch is stored in the +registry and is applied when the module is fetched during Bazel's external +dependency resolution. + +This way, the version in the releases in the registry will be correctly set to +the released version and thus, `bazel_dep`, `single_version_override` and +`multiple_version_override` will work as expected, while avoiding potential +issues when doing a non-registry override because the version in the source +archive will be the default value (`''`), which will always be handled +correctly (it's the default version value after all) and will behave as +expected when sorting (the empty string is treated as the highest version). + +[Bazel Central Registry]: https://registry.bazel.build/ +[Bazel registries]: https://bazel.build/external/registry +[bazel-contrib/publish-to-bcr publish.yaml GitHub Action]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml +[generates a patch for the source archive `MODULE.bazel`]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216 +[`module`]: /rules/lib/globals/module#module +[non-registry override]: module.md#non-registry_overrides +[`rules-template`]: https://github.com/bazel-contrib/rules-template ### When should I increment the compatibility level? -The [`compatibility_level`](module.md#compatibility_level) of a Bazel module should be incremented *in the same commit* that introduces a backwards incompatible ("breaking") change. +The [`compatibility_level`](module.md#compatibility_level) of a Bazel module +should be incremented _in the same commit_ that introduces a backwards +incompatible ("breaking") change. -However, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. This can happen when for example' two modules depend on versions of a third module with different compatibility levels. +However, Bazel can throw an error if it detects that versions of the _same +module_ with _different compatibility levels_ exist in the resolved dependency +graph. This can happen when for example' two modules depend on versions of a +third module with different compatibility levels. -Thus, incrementing `compatibility_level` too frequently can be very disruptive and is discouraged. To avoid this situation, the `compatibility_level` should be incremented *only* when the breaking change affects most use cases and isn't easy to migrate and/or work-around. +Thus, incrementing `compatibility_level` too frequently can be very disruptive +and is discouraged. To avoid this situation, the `compatibility_level` should be +incremented _only_ when the breaking change affects most use cases and isn't +easy to migrate and/or work-around. ### Why does MODULE.bazel not support `load`s? -During dependency resolution, the MODULE.bazel file of all referenced external dependencies are fetched from registries. At this stage, the source archives of the dependencies are not fetched yet; so if the MODULE.bazel file `load`s another file, there is no way for Bazel to actually fetch that file without fetching the entire source archive. Note the MODULE.bazel file itself is special, as it's directly hosted on the registry. - -There are a few use cases that people asking for `load`s in MODULE.bazel are generally interested in, and they can be solved without `load`s: - -- Ensuring that the version listed in MODULE.bazel is consistent with build metadata stored elsewhere, for example in a .bzl file: This can be achieved by using the [`native.module_version`](/rules/lib/toplevel/native#module_version) method in a .bzl file loaded from a BUILD file. -- Splitting up a very large MODULE.bazel file into manageable sections, particularly for monorepos: The root module can use the [`include`](/rules/lib/globals/module#include) directive to split its MODULE.bazel file into multiple segments. For the same reason we don't allow `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. -- Users of the old WORKSPACE system might remember declaring a repo, and then immediately `load`ing from that repo to perform complex logic. This capability has been replaced by [module extensions](extension). +During dependency resolution, the MODULE.bazel file of all referenced external +dependencies are fetched from registries. At this stage, the source archives of +the dependencies are not fetched yet; so if the MODULE.bazel file `load`s +another file, there is no way for Bazel to actually fetch that file without +fetching the entire source archive. Note the MODULE.bazel file itself is +special, as it's directly hosted on the registry. + +There are a few use cases that people asking for `load`s in MODULE.bazel are +generally interested in, and they can be solved without `load`s: + +* Ensuring that the version listed in MODULE.bazel is consistent with build + metadata stored elsewhere, for example in a .bzl file: This can be achieved + by using the + [`native.module_version`](/rules/lib/toplevel/native#module_version) method + in a .bzl file loaded from a BUILD file. +* Splitting up a very large MODULE.bazel file into manageable sections, + particularly for monorepos: The root module can use the + [`include`](/rules/lib/globals/module#include) directive to split its + MODULE.bazel file into multiple segments. For the same reason we don't allow + `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. +* Users of the old WORKSPACE system might remember declaring a repo, and then + immediately `load`ing from that repo to perform complex logic. This + capability has been replaced by [module extensions](extension). ### Can I specify a SemVer range for a `bazel_dep`? -No. Some other package managers like [npm](https://docs.npmjs.com/about-semantic-versioning) and [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax) support version ranges (implicitly or explicitly), and this often requires a constraint solver (making the output harder to predict for users) and makes version resolution nonreproducible without a lockfile. +No. Some other package managers like [npm][npm-semver] and [Cargo][cargo-semver] +support version ranges (implicitly or explicitly), and this often requires a +constraint solver (making the output harder to predict for users) and makes +version resolution nonreproducible without a lockfile. -Bazel instead uses [Minimal Version Selection](module#version-selection) like Go, which in contrast makes the output easy to predict and guarantees reproducibility. This is a tradeoff that matches Bazel's design goals. +Bazel instead uses [Minimal Version Selection](module#version-selection) like +Go, which in contrast makes the output easy to predict and guarantees +reproducibility. This is a tradeoff that matches Bazel's design goals. -Furthermore, Bazel module versions are [a superset of SemVer](module#version-format), so what makes sense in a strict SemVer environment doesn't always carry over to Bazel module versions. +Furthermore, Bazel module versions are [a superset of +SemVer](module#version-format), so what makes sense in a strict SemVer +environment doesn't always carry over to Bazel module versions. ### Can I automatically get the latest version for a `bazel_dep`? -Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", version = "latest")` to automatically get the latest version of a dep. This is similar to [the question about SemVer ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also no. +Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", +version = "latest")` to automatically get the latest version of a dep. This is +similar to [the question about SemVer +ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also +no. -The recommended solution here is to have automation take care of this. For example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports Bazel modules. +The recommended solution here is to have automation take care of this. For +example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports +Bazel modules. -Sometimes, users asking this question are really looking for a way to quickly iterate during local development. This can be achieved by using a [`local_path_override`](/rules/lib/globals/module#local_path_override). +Sometimes, users asking this question are really looking for a way to quickly +iterate during local development. This can be achieved by using a +[`local_path_override`](/rules/lib/globals/module#local_path_override). ### Why all these `use_repo`s? -Module extension usages in MODULE.bazel files sometimes come with a big `use_repo` directive. For example, a typical usage of the [`go_deps` extension](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies) from `gazelle` might look like: +Module extension usages in MODULE.bazel files sometimes come with a big +`use_repo` directive. For example, a typical usage of the +[`go_deps` extension][go_deps] from `gazelle` might look like: ```python go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") @@ -73,82 +155,175 @@ use_repo( ) ``` -The long `use_repo` directive may seem redundant, since the information is arguably already in the referenced `go.mod` file. +The long `use_repo` directive may seem redundant, since the information is +arguably already in the referenced `go.mod` file. -The reason Bazel needs this `use_repo` directive is that it runs module extensions lazily. That is, a module extension is only run if its result is observed. Since a module extension's "output" is repo definitions, this means that we only run a module extension if a repo it defines is requested (for instance, if the target `@org_golang_x_net//:foo` is built, in the example above). However, we don't know which repos a module extension would define until after we run it. This is where the `use_repo` directive comes in; the user can tell Bazel which repos they expect the extension to generate, and Bazel would then only run the extension when these specific repos are used. +The reason Bazel needs this `use_repo` directive is that it runs module +extensions lazily. That is, a module extension is only run if its result is +observed. Since a module extension's "output" is repo definitions, this means +that we only run a module extension if a repo it defines is requested (for +instance, if the target `@org_golang_x_net//:foo` is built, in the example +above). However, we don't know which repos a module extension would define until +after we run it. This is where the `use_repo` directive comes in; the user can +tell Bazel which repos they expect the extension to generate, and Bazel would +then only run the extension when these specific repos are used. -To help the maintain this `use_repo` directive, a module extension can return an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) object from its implementation function. The user can run the `bazel mod tidy` command to update the `use_repo` directives for these module extensions. +To help the maintain this `use_repo` directive, a module extension can return +an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) +object from its implementation function. The user can run the `bazel mod tidy` +command to update the `use_repo` directives for these module extensions. ## Bzlmod migration ### Which is evaluated first, MODULE.bazel or WORKSPACE? -When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to wonder which system is consulted first. The short answer is that MODULE.bazel (Bzlmod) is evaluated first. - -The long answer is that "which evaluates first" is not the right question to ask; rather, the right question to ask is: in the context of the repo with [canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what is the repo mapping of `@@base`? - -Labels with apparent repo names (a single leading `@`) can refer to different things based on the context they're resolved from. When you see a label `@bar//:baz` and wonder what it actually points to, you need to first find out what the context repo is: for example, if the label is in a BUILD file located in the repo `@@foo`, then the context repo is `@@foo`. - -Then, depending on what the context repo is, the ["repository visibility" table](migration#repository-visibility) in the migration guide can be used to find out which repo an apparent name actually resolves to. - -- If the context repo is the main repo (`@@`): - - 1. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file (through any of [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), [`use_repo`](/rules/lib/globals/module#use_repo), [`module`](/rules/lib/globals/module#module.repo_name), [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` resolves to what that MODULE.bazel file claims. - 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its canonical name is `@@bar`), then `@bar` resolves to `@@bar`. - 3. Otherwise, `@bar` resolves to something like `@@[unknown repo 'bar' requested from @@]`, and this will ultimately result in an error. - -- If the context repo is a Bzlmod-world repo (that is, it corresponds to a non-root Bazel module, or is generated by a module extension), then it will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. - - Notably, this includes any repos introduced in a `non_module_deps`-like module extension in the root module, or `use_repo_rule` instantiations in the root module. - -- If the context repo is defined in WORKSPACE: - - 1. First, check if the context repo definition has the magical `repo_mapping` attribute. If so, go through the mapping first (so for a repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking at `@baz` below). - 2. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file claims. (This is the same as item 1 in the main repo case.) - 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel will throw an error. +When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to +wonder which system is consulted first. The short answer is that MODULE.bazel +(Bzlmod) is evaluated first. + +The long answer is that "which evaluates first" is not the right question to +ask; rather, the right question to ask is: in the context of the repo with +[canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent +repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what +is the repo mapping of `@@base`? + +Labels with apparent repo names (a single leading `@`) can refer to different +things based on the context they're resolved from. When you see a label +`@bar//:baz` and wonder what it actually points to, you need to first find out +what the context repo is: for example, if the label is in a BUILD file located +in the repo `@@foo`, then the context repo is `@@foo`. + +Then, depending on what the context repo is, the ["repository +visibility" table](migration#repository-visibility) in the migration guide can +be used to find out which repo an apparent name actually resolves to. + +* If the context repo is the main repo (`@@`): + 1. If `bar` is an apparent repo name introduced by the root module's + MODULE.bazel file (through any of + [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), + [`use_repo`](/rules/lib/globals/module#use_repo), + [`module`](/rules/lib/globals/module#module.repo_name), + [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` + resolves to what that MODULE.bazel file claims. + 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its + canonical name is `@@bar`), then `@bar` resolves to `@@bar`. + 3. Otherwise, `@bar` resolves to something like + `@@[unknown repo 'bar' requested from @@]`, and this will ultimately + result in an error. +* If the context repo is a Bzlmod-world repo (that is, it corresponds to a + non-root Bazel module, or is generated by a module extension), then it + will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. + * Notably, this includes any repos introduced in a `non_module_deps`-like + module extension in the root module, or `use_repo_rule` instantiations + in the root module. +* If the context repo is defined in WORKSPACE: + 1. First, check if the context repo definition has the magical + `repo_mapping` attribute. If so, go through the mapping first (so for a + repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking + at `@baz` below). + 2. If `bar` is an apparent repo name introduced by the root module's + MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file + claims. (This is the same as item 1 in the main repo case.) + 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a + repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel + will throw an error. For a more succinct version: -- Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world repos. -- WORKSPACE-world repos (including the main repo) will first see what the root module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world repos. +* Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world + repos. +* WORKSPACE-world repos (including the main repo) will first see what the root + module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world + repos. -Of note, labels in the Bazel command line (including Starlark flags, label-typed flag values, and build/test target patterns) are treated as having the main repo as the context repo. +Of note, labels in the Bazel command line (including Starlark flags, label-typed +flag values, and build/test target patterns) are treated as having the main repo +as the context repo. ## Other ### How do I prepare and run an offline build? -Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag (like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the context of the main repo, see [question above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of `@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). +Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag +(like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the +context of the main repo, see [question +above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target +pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of +`@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). -The make sure no fetches happen during a build, use `--nofetch`. More precisely, this makes any attempt to run a non-local repository rule fail. +The make sure no fetches happen during a build, use `--nofetch`. More precisely, +this makes any attempt to run a non-local repository rule fail. -If you want to fetch repos *and* modify them to test locally, consider using the [`bazel vendor`](vendor) command. +If you want to fetch repos _and_ modify them to test locally, consider using +the [`bazel vendor`](vendor) command. ### How do I use HTTP proxies? -Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly accepted by other programs, such as [curl](https://everything.curl.dev/usingcurl/proxies/env.html). +Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly +accepted by other programs, such as +[curl](https://everything.curl.dev/usingcurl/proxies/env.html). ### How do I make Bazel prefer IPv6 in dual-stack IPv4/IPv6 setups? -On IPv6-only machines, Bazel can download dependencies with no changes. However, on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, preferring IPv4 if enabled. In some situations, for example when the IPv4 network cannot resolve/reach external addresses, this can cause `Network unreachable` exceptions and build failures. In these cases, you can override Bazel's behavior to prefer IPv6 by using the [`java.net.preferIPv6Addresses=true` system property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). Specifically: +On IPv6-only machines, Bazel can download dependencies with no changes. However, +on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, +preferring IPv4 if enabled. In some situations, for example when the IPv4 +network cannot resolve/reach external addresses, this can cause `Network +unreachable` exceptions and build failures. In these cases, you can override +Bazel's behavior to prefer IPv6 by using the +[`java.net.preferIPv6Addresses=true` system +property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). +Specifically: + +* Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup + option](/docs/user-manual#startup-options), for example by adding the + following line in your [`.bazelrc` file](/run/bazelrc): + + `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` + +* When running Java build targets that need to connect to the internet (such + as for integration tests), use the + `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool + flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` + file](/run/bazelrc): + + `build --jvmopt=-Djava.net.preferIPv6Addresses` + +* If you are using + [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for + dependency version resolution, also add + `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment + variable to [provide JVM options for + Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). -- Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup option](/docs/user-manual#startup-options), for example by adding the following line in your [`.bazelrc` file](/run/bazelrc): - - `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` - -- When running Java build targets that need to connect to the internet (such as for integration tests), use the `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` file](/run/bazelrc): - - `build --jvmopt=-Djava.net.preferIPv6Addresses` +### Can repo rules be run remotely with remote execution? -- If you are using [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for dependency version resolution, also add `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment variable to [provide JVM options for Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). +No; or at least, not yet. Users employing remote execution services to speed up +their builds may notice that repo rules are still run locally. For example, an +`http_archive` would be first downloaded onto the local machine (using any local +download cache if applicable), extracted, and then each source file would be +uploaded to the remote execution service as an input file. It's natural to ask +why the remote execution service doesn't just download and extract that archive, +saving a useless roundtrip. -### Can repo rules be run remotely with remote execution? +Part of the reason is that repo rules (and module extensions) are akin to +"scripts" that are run by Bazel itself. A remote executor doesn't necessarily +even have a Bazel installed. -No; or at least, not yet. Users employing remote execution services to speed up their builds may notice that repo rules are still run locally. For example, an `http_archive` would be first downloaded onto the local machine (using any local download cache if applicable), extracted, and then each source file would be uploaded to the remote execution service as an input file. It's natural to ask why the remote execution service doesn't just download and extract that archive, saving a useless roundtrip. +Another reason is that Bazel often needs the BUILD files in the downloaded and +extracted archives to perform loading and analysis, which _are_ performed +locally. -Part of the reason is that repo rules (and module extensions) are akin to "scripts" that are run by Bazel itself. A remote executor doesn't necessarily even have a Bazel installed. +There are preliminary ideas to solve this problem by re-imagining repo rules as +build rules, which would naturally allow them to be run remotely, but conversely +raise new architectural concerns (for example, the `query` commands would +potentially need to run actions, complicating their design). -Another reason is that Bazel often needs the BUILD files in the downloaded and extracted archives to perform loading and analysis, which *are* performed locally. +For more previous discussion on this topic, see [A way to support repositories +that need Bazel for being +fetched](https://github.com/bazelbuild/bazel/discussions/20464). -There are preliminary ideas to solve this problem by re-imagining repo rules as build rules, which would naturally allow them to be run remotely, but conversely raise new architectural concerns (for example, the `query` commands would potentially need to run actions, complicating their design). +[npm-semver]: https://docs.npmjs.com/about-semantic-versioning +[cargo-semver]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax +[go_deps]: https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies -For more previous discussion on this topic, see [A way to support repositories that need Bazel for being fetched](https://github.com/bazelbuild/bazel/discussions/20464). diff --git a/external/lockfile.mdx b/external/lockfile.mdx index a20423c3..9f312615 100644 --- a/external/lockfile.mdx +++ b/external/lockfile.mdx @@ -1,46 +1,98 @@ +keywords: product:Bazel,lockfile,Bzlmod --- title: 'Bazel Lockfile' --- -The lockfile feature in Bazel enables the recording of specific versions or dependencies of software libraries or packages required by a project. It achieves this by storing the result of module resolution and extension evaluation. The lockfile promotes reproducible builds, ensuring consistent development environments. Additionally, it enhances build efficiency by allowing Bazel to skip the parts of the resolution process that are unaffected by changes in project dependencies. Furthermore, the lockfile improves stability by preventing unexpected updates or breaking changes in external libraries, thereby reducing the risk of introducing bugs. + + +The lockfile feature in Bazel enables the recording of specific versions or +dependencies of software libraries or packages required by a project. It +achieves this by storing the result of module resolution and extension +evaluation. The lockfile promotes reproducible builds, ensuring consistent +development environments. Additionally, it enhances build efficiency by allowing +Bazel to skip the parts of the resolution process that are unaffected by changes +in project dependencies. Furthermore, the lockfile improves stability by +preventing unexpected updates or breaking changes in external libraries, thereby +reducing the risk of introducing bugs. ## Lockfile Generation -The lockfile is generated under the workspace root with the name `MODULE.bazel.lock`. It is created or updated during the build process, specifically after module resolution and extension evaluation. Importantly, it only includes dependencies that are included in the current invocation of the build. +The lockfile is generated under the workspace root with the name +`MODULE.bazel.lock`. It is created or updated during the build process, +specifically after module resolution and extension evaluation. Importantly, it +only includes dependencies that are included in the current invocation of the +build. -When changes occur in the project that affect its dependencies, the lockfile is automatically updated to reflect the new state. This ensures that the lockfile remains focused on the specific set of dependencies required for the current build, providing an accurate representation of the project's resolved dependencies. +When changes occur in the project that affect its dependencies, the lockfile is +automatically updated to reflect the new state. This ensures that the lockfile +remains focused on the specific set of dependencies required for the current +build, providing an accurate representation of the project's resolved +dependencies. ## Lockfile Usage -The lockfile can be controlled by the flag [`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to customize the behavior of Bazel when the project state differs from the lockfile. The available modes are: - -- `update` (Default): Use the information that is present in the lockfile to skip downloads of known registry files and to avoid re-evaluating extensions whose results are still up-to-date. If information is missing, it will be added to the lockfile. In this mode, Bazel also avoids refreshing mutable information, such as yanked versions, for dependencies that haven't changed. -- `refresh`: Like `update`, but mutable information is always refreshed when switching to this mode and roughly every hour while in this mode. -- `error`: Like `update`, but if any information is missing or out-of-date, Bazel will fail with an error. This mode never changes the lockfile or performs network requests during resolution. Module extensions that marked themselves as `reproducible` may still perform network requests, but are expected to always produce the same result. -- `off`: The lockfile is neither checked nor updated. +The lockfile can be controlled by the flag +[`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to +customize the behavior of Bazel when the project state differs from the +lockfile. The available modes are: + +* `update` (Default): Use the information that is present in the lockfile to + skip downloads of known registry files and to avoid re-evaluating extensions + whose results are still up-to-date. If information is missing, it will + be added to the lockfile. In this mode, Bazel also avoids refreshing + mutable information, such as yanked versions, for dependencies that haven't + changed. +* `refresh`: Like `update`, but mutable information is always refreshed when + switching to this mode and roughly every hour while in this mode. +* `error`: Like `update`, but if any information is missing or out-of-date, + Bazel will fail with an error. This mode never changes the lockfile or + performs network requests during resolution. Module extensions that marked + themselves as `reproducible` may still perform network requests, but are + expected to always produce the same result. +* `off`: The lockfile is neither checked nor updated. ## Lockfile Benefits The lockfile offers several benefits and can be utilized in various ways: -- **Reproducible builds.** By capturing the specific versions or dependencies of software libraries, the lockfile ensures that builds are reproducible across different environments and over time. Developers can rely on consistent and predictable results when building their projects. +- **Reproducible builds.** By capturing the specific versions or dependencies + of software libraries, the lockfile ensures that builds are reproducible + across different environments and over time. Developers can rely on + consistent and predictable results when building their projects. -- **Fast incremental resolutions.** The lockfile enables Bazel to avoid downloading registry files that were already used in a previous build. This significantly improves build efficiency, especially in scenarios where resolution can be time-consuming. +- **Fast incremental resolutions.** The lockfile enables Bazel to avoid + downloading registry files that were already used in a previous build. + This significantly improves build efficiency, especially in scenarios where + resolution can be time-consuming. -- **Stability and risk reduction.** The lockfile helps maintain stability by preventing unexpected updates or breaking changes in external libraries. By locking the dependencies to specific versions, the risk of introducing bugs due to incompatible or untested updates is reduced. +- **Stability and risk reduction.** The lockfile helps maintain stability by + preventing unexpected updates or breaking changes in external libraries. By + locking the dependencies to specific versions, the risk of introducing bugs + due to incompatible or untested updates is reduced. ### Hidden lockfile -Bazel also maintains another lockfile at `"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this lockfile are explicitly unspecified. It is only used as a performance optimization. While it can be deleted together with the output base via `bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a module extension. +Bazel also maintains another lockfile at +`"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this +lockfile are explicitly unspecified. It is only used as a performance +optimization. While it can be deleted together with the output base via +`bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a +module extension. ## Lockfile Contents -The lockfile contains all the necessary information to determine whether the project state has changed. It also includes the result of building the project in the current state. The lockfile consists of two main parts: +The lockfile contains all the necessary information to determine whether the +project state has changed. It also includes the result of building the project +in the current state. The lockfile consists of two main parts: -1. Hashes of all remote files that are inputs to module resolution. -2. For each module extension, the lockfile includes inputs that affect it, represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as well as the output of running that extension, referred to as `generatedRepoSpecs` +1. Hashes of all remote files that are inputs to module resolution. +2. For each module extension, the lockfile includes inputs that affect it, + represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as + well as the output of running that extension, referred to as + `generatedRepoSpecs` -Here is an example that demonstrates the structure of the lockfile, along with explanations for each section: +Here is an example that demonstrates the structure of the lockfile, along with +explanations for each section: ```json { @@ -100,53 +152,106 @@ Here is an example that demonstrates the structure of the lockfile, along with e ### Registry File Hashes -The `registryFileHashes` section contains the hashes of all files from remote registries accessed during module resolution. Since the resolution algorithm is fully deterministic when given the same inputs and all remote inputs are hashed, this ensures a fully reproducible resolution result while avoiding excessive duplication of remote information in the lockfile. Note that this also requires recording when a particular registry didn't contain a certain module, but a registry with lower precedence did (see the "not found" entry in the example). This inherently mutable information can be updated via `bazel mod deps --lockfile_mode=refresh`. +The `registryFileHashes` section contains the hashes of all files from +remote registries accessed during module resolution. Since the resolution +algorithm is fully deterministic when given the same inputs and all remote +inputs are hashed, this ensures a fully reproducible resolution result while +avoiding excessive duplication of remote information in the lockfile. Note that +this also requires recording when a particular registry didn't contain a certain +module, but a registry with lower precedence did (see the "not found" entry in +the example). This inherently mutable information can be updated via +`bazel mod deps --lockfile_mode=refresh`. -Bazel uses the hashes from the lockfile to look up registry files in the repository cache before downloading them, which speeds up subsequent resolutions. +Bazel uses the hashes from the lockfile to look up registry files in the +repository cache before downloading them, which speeds up subsequent +resolutions. ### Selected Yanked Versions -The `selectedYankedVersions` section contains the yanked versions of modules that were selected by module resolution. Since this usually result in an error when trying to build, this section is only non-empty when yanked versions are explicitly allowed via `--allow_yanked_versions` or `BZLMOD_ALLOW_YANKED_VERSIONS`. +The `selectedYankedVersions` section contains the yanked versions of modules +that were selected by module resolution. Since this usually result in an error +when trying to build, this section is only non-empty when yanked versions are +explicitly allowed via `--allow_yanked_versions` or +`BZLMOD_ALLOW_YANKED_VERSIONS`. -This field is needed since, compared to module files, yanked version information is inherently mutable and thus can't be referenced by a hash. This information can be updated via `bazel mod deps --lockfile_mode=refresh`. +This field is needed since, compared to module files, yanked version information +is inherently mutable and thus can't be referenced by a hash. This information +can be updated via `bazel mod deps --lockfile_mode=refresh`. ### Module Extensions -The `moduleExtensions` section is a map that includes only the extensions used in the current invocation or previously invoked, while excluding any extensions that are no longer utilized. In other words, if an extension is not being used anymore across the dependency graph, it is removed from the `moduleExtensions` map. - -If an extension is independent of the operating system or architecture type, this section features only a single "general" entry. Otherwise, multiple entries are included, named after the OS, architecture, or both, with each corresponding to the result of evaluating the extension on those specifics. - -Each entry in the extension map corresponds to a used extension and is identified by its containing file and name. The corresponding value for each entry contains the relevant information associated with that extension: - -1. The `bzlTransitiveDigest` is the digest of the extension implementation and the .bzl files transitively loaded by it. -2. The `usagesDigest` is the digest of the *usages* of the extension in the dependency graph, which includes all tags. -3. Further unspecified fields that track other inputs to the extension, such as contents of files or directories it reads or environment variables it uses. -4. The `generatedRepoSpecs` encode the repositories created by the extension with the current input. -5. The optional `moduleExtensionMetadata` field contains metadata provided by the extension such as whether certain repositories it created should be imported via `use_repo` by the root module. This information powers the `bazel mod tidy` command. - -Module extensions can opt out of being included in the lockfile by setting the returning metadata with `reproducible = True`. By doing so, they promise that they will always create the same repositories when given the same inputs. +The `moduleExtensions` section is a map that includes only the extensions used +in the current invocation or previously invoked, while excluding any extensions +that are no longer utilized. In other words, if an extension is not being used +anymore across the dependency graph, it is removed from the `moduleExtensions` +map. + +If an extension is independent of the operating system or architecture type, +this section features only a single "general" entry. Otherwise, multiple +entries are included, named after the OS, architecture, or both, with each +corresponding to the result of evaluating the extension on those specifics. + +Each entry in the extension map corresponds to a used extension and is +identified by its containing file and name. The corresponding value for each +entry contains the relevant information associated with that extension: + +1. The `bzlTransitiveDigest` is the digest of the extension implementation + and the .bzl files transitively loaded by it. +2. The `usagesDigest` is the digest of the _usages_ of the extension in the + dependency graph, which includes all tags. +3. Further unspecified fields that track other inputs to the extension, + such as contents of files or directories it reads or environment + variables it uses. +4. The `generatedRepoSpecs` encode the repositories created by the + extension with the current input. +5. The optional `moduleExtensionMetadata` field contains metadata provided by + the extension such as whether certain repositories it created should be + imported via `use_repo` by the root module. This information powers the + `bazel mod tidy` command. + +Module extensions can opt out of being included in the lockfile by setting the +returning metadata with `reproducible = True`. By doing so, they promise that +they will always create the same repositories when given the same inputs. ## Best Practices -To maximize the benefits of the lockfile feature, consider the following best practices: +To maximize the benefits of the lockfile feature, consider the following best +practices: -- Regularly update the lockfile to reflect changes in project dependencies or configuration. This ensures that subsequent builds are based on the most up-to-date and accurate set of dependencies. To lock down all extensions at once, run `bazel mod deps --lockfile_mode=update`. +* Regularly update the lockfile to reflect changes in project dependencies or + configuration. This ensures that subsequent builds are based on the most + up-to-date and accurate set of dependencies. To lock down all extensions + at once, run `bazel mod deps --lockfile_mode=update`. -- Include the lockfile in version control to facilitate collaboration and ensure that all team members have access to the same lockfile, promoting consistent development environments across the project. +* Include the lockfile in version control to facilitate collaboration and + ensure that all team members have access to the same lockfile, promoting + consistent development environments across the project. -- Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a `.bazelversion` file in version control that specifies the Bazel version corresponding to the lockfile. Because Bazel itself is a dependency of your build, the lockfile is specific to the Bazel version, and will change even between [backwards compatible](/release/backward-compatibility) Bazel releases. Using `bazelisk` ensures that all developers are using a Bazel version that matches the lockfile. +* Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a + `.bazelversion` file in version control that specifies the Bazel version + corresponding to the lockfile. Because Bazel itself is a dependency of + your build, the lockfile is specific to the Bazel version, and will + change even between [backwards compatible](/release/backward-compatibility) + Bazel releases. Using `bazelisk` ensures that all developers are using + a Bazel version that matches the lockfile. -By following these best practices, you can effectively utilize the lockfile feature in Bazel, leading to more efficient, reliable, and collaborative software development workflows. +By following these best practices, you can effectively utilize the lockfile +feature in Bazel, leading to more efficient, reliable, and collaborative +software development workflows. ## Merge Conflicts -The lockfile format is designed to minimize merge conflicts, but they can still happen. +The lockfile format is designed to minimize merge conflicts, but they can still +happen. ### Automatic Resolution -Bazel provides a custom [git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) to help resolve these conflicts automatically. +Bazel provides a custom +[git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) +to help resolve these conflicts automatically. -Set up the driver by adding this line to a `.gitattributes` file in the root of your git repository: +Set up the driver by adding this line to a `.gitattributes` file in the root of +your git repository: ```gitattributes # A custom merge driver for the Bazel lockfile. @@ -154,7 +259,8 @@ Set up the driver by adding this line to a `.gitattributes` file in the root of MODULE.bazel.lock merge=bazel-lockfile-merge ``` -Then each developer who wants to use the driver has to register it once by following these steps: +Then each developer who wants to use the driver has to register it once by +following these steps: 1. Install [jq](https://jqlang.github.io/jq/download/) (1.5 or higher). 2. Run the following commands: @@ -163,15 +269,18 @@ Then each developer who wants to use the driver has to register it once by follo jq_script=$(curl https://raw.githubusercontent.com/bazelbuild/bazel/master/scripts/bazel-lockfile-merge.jq) printf '%s\n' "${jq_script}" | less # to optionally inspect the jq script git config --global merge.bazel-lockfile-merge.name "Merge driver for the Bazel lockfile (MODULE.bazel.lock)" -git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" +git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" ``` ### Manual Resolution -Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` fields can be safely resolved by keeping all the entries from both sides of the conflict. +Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` +fields can be safely resolved by keeping all the entries from both sides of the +conflict. Other types of merge conflicts should not be resolved manually. Instead: -1. Restore the previous state of the lockfile via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. +1. Restore the previous state of the lockfile + via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. 2. Resolve any conflicts in the `MODULE.bazel` file. 3. Run `bazel mod deps` to update the lockfile. diff --git a/external/migration.mdx b/external/migration.mdx index 49ba462f..8e70b1f3 100644 --- a/external/migration.mdx +++ b/external/migration.mdx @@ -1,200 +1,267 @@ +keywords: bzlmod --- title: 'Bzlmod Migration Guide' --- -Due to the [shortcomings of WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate your project to Bzlmod and drop WORKSPACE for managing external dependencies. + + +Due to the [shortcomings of +WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the +legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late +2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate +your project to Bzlmod and drop WORKSPACE for managing external dependencies. ## Why migrate to Bzlmod? -- There are many [advantages](overview#benefits) compared to the legacy WORKSPACE system, which helps to ensure a healthy growth of the Bazel ecosystem. +* There are many [advantages](overview#benefits) compared to the legacy + WORKSPACE system, which helps to ensure a healthy growth of the Bazel + ecosystem. -- If your project is a dependency of other projects, migrating to Bzlmod will unblock their migration and make it easier for them to depend on your project. +* If your project is a dependency of other projects, migrating to Bzlmod will + unblock their migration and make it easier for them to depend on your + project. -- Migration to Bzlmod is a necessary step in order to use future Bazel versions (mandatory in Bazel 9). +* Migration to Bzlmod is a necessary step in order to use future Bazel + versions (mandatory in Bazel 9). ## How to migrate to Bzlmod? Recommended migration process: -1. Use [migration tool](/external/migration_tool) as a helper tool to streamline the migration process as much as possible. -2. If there are errors left after using the migration tool, resolve them manually. For understanding the main differences between concepts inside `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus Bzlmod](#workspace-vs-bzlmod) section. +1. Use [migration tool](/external/migration_tool) as a helper tool to + streamline the migration process as much as possible. +2. If there are errors left after using the migration tool, resolve them + manually. For understanding the main differences between concepts inside + `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus + Bzlmod](#workspace-vs-bzlmod) section. ## WORKSPACE vs Bzlmod -Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This section explains how to migrate from specific WORKSPACE functionalities to Bzlmod. +Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This +section explains how to migrate from specific WORKSPACE functionalities to +Bzlmod. ### Define the root of a Bazel workspace -The WORKSPACE file marks the source root of a Bazel project, this responsibility is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at your workspace root, maybe with comments like: +The WORKSPACE file marks the source root of a Bazel project, this responsibility +is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions +prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at +your workspace root, maybe with comments like: -- **WORKSPACE** +* **WORKSPACE** - ```python - # This file marks the root of the Bazel workspace. - # See MODULE.bazel for external dependencies setup. - ``` + ```python + # This file marks the root of the Bazel workspace. + # See MODULE.bazel for external dependencies setup. + ``` ### Enable Bzlmod in your bazelrc -`.bazelrc` lets you set flags that apply every time your run Bazel. To enable Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so it applies to every command: +`.bazelrc` lets you set flags that apply every time your run Bazel. To enable +Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so +it applies to every command: -- **.bazelrc** +* **.bazelrc** - ``` - # Enable Bzlmod for every Bazel command - common --enable_bzlmod - ``` + ``` + # Enable Bzlmod for every Bazel command + common --enable_bzlmod + ``` ### Specify repository name for your workspace -- **WORKSPACE** +* **WORKSPACE** - The [`workspace`](/rules/lib/globals/workspace#workspace) function is used to specify a repository name for your workspace. This allows a target `//foo:bar` in the workspace to be referenced as `@<workspace name>//foo:bar`. If not specified, the default repository name for your workspace is `__main__`. + The [`workspace`](/rules/lib/globals/workspace#workspace) function is used + to specify a repository name for your workspace. This allows a target + `//foo:bar` in the workspace to be referenced as `@//foo:bar`. If not specified, the default repository name for your + workspace is `__main__`. - ```python - ## WORKSPACE - workspace(name = "com_foo_bar") - ``` + ```python + ## WORKSPACE + workspace(name = "com_foo_bar") + ``` -- **Bzlmod** +* **Bzlmod** - It's recommended to reference targets in the same workspace with the `//foo:bar` syntax without `@<repo name>`. But if you do need the old syntax , you can use the module name specified by the [`module`](/rules/lib/globals/module#module) function as the repository name. If the module name is different from the needed repository name, you can use `repo_name` attribute of the [`module`](/rules/lib/globals/module#module) function to override the repository name. + It's recommended to reference targets in the same workspace with the + `//foo:bar` syntax without `@`. But if you do need the old syntax + , you can use the module name specified by the + [`module`](/rules/lib/globals/module#module) function as the repository + name. If the module name is different from the needed repository name, you + can use `repo_name` attribute of the + [`module`](/rules/lib/globals/module#module) function to override the + repository name. - ```python - ## MODULE.bazel - module( - name = "bar", - repo_name = "com_foo_bar", - ) - ``` + ```python + ## MODULE.bazel + module( + name = "bar", + repo_name = "com_foo_bar", + ) + ``` ### Fetch external dependencies as Bazel modules -If your dependency is a Bazel project, you should be able to depend on it as a Bazel module when it also adopts Bzlmod. +If your dependency is a Bazel project, you should be able to depend on it as a +Bazel module when it also adopts Bzlmod. -- **WORKSPACE** +* **WORKSPACE** - With WORKSPACE, it's common to use the [`http_archive`](/rules/lib/repo/http#http_archive) or [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to download the sources of the Bazel project. + With WORKSPACE, it's common to use the + [`http_archive`](/rules/lib/repo/http#http_archive) or + [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to + download the sources of the Bazel project. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "bazel_skylib", - urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], - sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", - ) - load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") - bazel_skylib_workspace() + http_archive( + name = "bazel_skylib", + urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], + sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", + ) + load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") + bazel_skylib_workspace() - http_archive( - name = "rules_java", - urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], - sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", - ) - load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") - rules_java_dependencies() - rules_java_toolchains() - ``` + http_archive( + name = "rules_java", + urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], + sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", + ) + load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") + rules_java_dependencies() + rules_java_toolchains() + ``` - As you can see, it's a common pattern that users need to load transitive dependencies from a macro of the dependency. Assume both `bazel_skylib` and `rules_java` depends on `platform`, the exact version of the `platform` dependency is determined by the order of the macros. + As you can see, it's a common pattern that users need to load transitive + dependencies from a macro of the dependency. Assume both `bazel_skylib` and + `rules_java` depends on `platform`, the exact version of the `platform` + dependency is determined by the order of the macros. -- **Bzlmod** +* **Bzlmod** - With Bzlmod, as long as your dependency is available in [Bazel Central Registry](https://registry.bazel.build) or your custom [Bazel registry](/external/registry), you can simply depend on it with a [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. + With Bzlmod, as long as your dependency is available in [Bazel Central + Registry](https://registry.bazel.build) or your custom [Bazel + registry](/external/registry), you can simply depend on it with a + [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. - ```python - ## MODULE.bazel - bazel_dep(name = "bazel_skylib", version = "1.4.2") - bazel_dep(name = "rules_java", version = "6.1.1") - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "bazel_skylib", version = "1.4.2") + bazel_dep(name = "rules_java", version = "6.1.1") + ``` - Bzlmod resolves Bazel module dependencies transitively using the [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal required version of `platform` is selected automatically. + Bzlmod resolves Bazel module dependencies transitively using the + [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal + required version of `platform` is selected automatically. ### Override a dependency as a Bazel module -As the root module, you can override Bazel module dependencies in different ways. +As the root module, you can override Bazel module dependencies in different +ways. -Please read the [overrides](/external/module#overrides) section for more information. +Please read the [overrides](/external/module#overrides) section for more +information. -You can find some example usages in the [examples](https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module) repository. +You can find some example usages in the +[examples][override-examples] +repository. + +[override-examples]: https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module ### Fetch external dependencies with module extensions -If your dependency is not a Bazel project or not yet available in any Bazel registry, you can introduce it using [`use_repo_rule`](/external/module#use_repo_rule) or [module extensions](/external/extension). +If your dependency is not a Bazel project or not yet available in any Bazel +registry, you can introduce it using +[`use_repo_rule`](/external/module#use_repo_rule) or [module +extensions](/external/extension). -- **WORKSPACE** +* **WORKSPACE** - Download a file using the [`http_file`](/rules/lib/repo/http#http_file) repository rule. + Download a file using the [`http_file`](/rules/lib/repo/http#http_file) + repository rule. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel file to directly instantiate repos: + With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel + file to directly instantiate repos: - ```python - ## MODULE.bazel - http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## MODULE.bazel + http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Under the hood, this is implemented using a module extension. If you need to perform more complex logic than simply invoking a repo rule, you could also implement a module extension yourself. You'll need to move the definition into a `.bzl` file, which also lets you share the definition between WORKSPACE and Bzlmod during the migration period. + Under the hood, this is implemented using a module extension. If you need to + perform more complex logic than simply invoking a repo rule, you could also + implement a module extension yourself. You'll need to move the definition + into a `.bzl` file, which also lets you share the definition between + WORKSPACE and Bzlmod during the migration period. - ```python - ## repositories.bzl - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - def my_data_dependency(): - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## repositories.bzl + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + def my_data_dependency(): + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Implement a module extension to load the dependencies macro. You can define it in the same `.bzl` file of the macro, but to keep compatibility with older Bazel versions, it's better to define it in a separate `.bzl` file. + Implement a module extension to load the dependencies macro. You can define + it in the same `.bzl` file of the macro, but to keep compatibility with + older Bazel versions, it's better to define it in a separate `.bzl` file. - ```python - ## extensions.bzl - load("//:repositories.bzl", "my_data_dependency") - def _non_module_dependencies_impl(_ctx): - my_data_dependency() + ```python + ## extensions.bzl + load("//:repositories.bzl", "my_data_dependency") + def _non_module_dependencies_impl(_ctx): + my_data_dependency() - non_module_dependencies = module_extension( - implementation = _non_module_dependencies_impl, - ) - ``` + non_module_dependencies = module_extension( + implementation = _non_module_dependencies_impl, + ) + ``` - To make the repository visible to the root project, you should declare the usages of the module extension and the repository in the MODULE.bazel file. + To make the repository visible to the root project, you should declare the + usages of the module extension and the repository in the MODULE.bazel file. - ```python - ## MODULE.bazel - non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") - use_repo(non_module_dependencies, "data_file") - ``` + ```python + ## MODULE.bazel + non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") + use_repo(non_module_dependencies, "data_file") + ``` ### Resolve conflict external dependencies with module extension -A project can provide a macro that introduces external repositories based on inputs from its callers. But what if there are multiple callers in the dependency graph and they cause a conflict? +A project can provide a macro that introduces external repositories based on +inputs from its callers. But what if there are multiple callers in the +dependency graph and they cause a conflict? -Assume the project `foo` provides the following macro which takes `version` as an argument. +Assume the project `foo` provides the following macro which takes `version` as +an argument. ```python -## repositories.bzl in foo +## repositories.bzl in foo {:#repositories.bzl-foo} load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") def data_deps(version = "1.0"): http_file( @@ -204,239 +271,297 @@ def data_deps(version = "1.0"): ) ``` -- **WORKSPACE** - - With WORKSPACE, you can load the macro from `@foo` and specify the version of the data dependency you need. Assume you have another dependency `@bar`, which also depends on `@foo` but requires a different version of the data dependency. +* **WORKSPACE** - ```python - ## WORKSPACE + With WORKSPACE, you can load the macro from `@foo` and specify the version + of the data dependency you need. Assume you have another dependency `@bar`, + which also depends on `@foo` but requires a different version of the data + dependency. - # Introduce @foo and @bar. - ... - - load("@foo//:repositories.bzl", "data_deps") - data_deps(version = "2.0") - - load("@bar//:repositories.bzl", "bar_deps") - bar_deps() # -> which calls data_deps(version = "3.0") - ``` + ```python + ## WORKSPACE - In this case, the end user must carefully adjust the order of macros in the WORKSPACE to get the version they need. This is one of the biggest pain points with WORKSPACE since it doesn't really provide a sensible way to resolve dependencies. + # Introduce @foo and @bar. + ... -- **Bzlmod** + load("@foo//:repositories.bzl", "data_deps") + data_deps(version = "2.0") - With Bzlmod, the author of project `foo` can use module extension to resolve conflicts. For example, let's assume it makes sense to always select the maximal required version of the data dependency among all Bazel modules. + load("@bar//:repositories.bzl", "bar_deps") + bar_deps() # -> which calls data_deps(version = "3.0") + ``` - ```python - ## extensions.bzl in foo - load("//:repositories.bzl", "data_deps") + In this case, the end user must carefully adjust the order of macros in the + WORKSPACE to get the version they need. This is one of the biggest pain + points with WORKSPACE since it doesn't really provide a sensible way to + resolve dependencies. - data = tag_class(attrs={"version": attr.string()}) +* **Bzlmod** - def _data_deps_extension_impl(module_ctx): - # Select the maximal required version in the dependency graph. - version = "1.0" - for mod in module_ctx.modules: - for data in mod.tags.data: - version = max(version, data.version) - data_deps(version) + With Bzlmod, the author of project `foo` can use module extension to resolve + conflicts. For example, let's assume it makes sense to always select the + maximal required version of the data dependency among all Bazel modules. - data_deps_extension = module_extension( - implementation = _data_deps_extension_impl, - tag_classes = {"data": data}, - ) - ``` + ```python + ## extensions.bzl in foo + load("//:repositories.bzl", "data_deps") + + data = tag_class(attrs={"version": attr.string()}) + + def _data_deps_extension_impl(module_ctx): + # Select the maximal required version in the dependency graph. + version = "1.0" + for mod in module_ctx.modules: + for data in mod.tags.data: + version = max(version, data.version) + data_deps(version) + + data_deps_extension = module_extension( + implementation = _data_deps_extension_impl, + tag_classes = {"data": data}, + ) + ``` - ```python - ## MODULE.bazel in bar - bazel_dep(name = "foo", version = "1.0") + ```python + ## MODULE.bazel in bar + bazel_dep(name = "foo", version = "1.0") - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "3.0") - use_repo(foo_data_deps, "data_file") - ``` + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "3.0") + use_repo(foo_data_deps, "data_file") + ``` - ```python - ## MODULE.bazel in root module - bazel_dep(name = "foo", version = "1.0") - bazel_dep(name = "bar", version = "1.0") + ```python + ## MODULE.bazel in root module + bazel_dep(name = "foo", version = "1.0") + bazel_dep(name = "bar", version = "1.0") - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "2.0") - use_repo(foo_data_deps, "data_file") - ``` + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "2.0") + use_repo(foo_data_deps, "data_file") + ``` - In this case, the root module requires data version `2.0`, while its dependency `bar` requires `3.0`. The module extension in `foo` can correctly resolve this conflict and automatically select version `3.0` for the data dependency. + In this case, the root module requires data version `2.0`, while its + dependency `bar` requires `3.0`. The module extension in `foo` can correctly + resolve this conflict and automatically select version `3.0` for the data + dependency. ### Integrate third party package manager -Following the last section, since module extension provides a way to collect information from the dependency graph, perform custom logic to resolve dependencies and call repository rules to introduce external repositories, this provides a great way for rules authors to enhance the rulesets that integrate package managers for specific languages. +Following the last section, since module extension provides a way to collect +information from the dependency graph, perform custom logic to resolve +dependencies and call repository rules to introduce external repositories, this +provides a great way for rules authors to enhance the rulesets that integrate +package managers for specific languages. -Please read the [module extensions](/external/extension) page to learn more about how to use module extensions. +Please read the [module extensions](/external/extension) page to learn more +about how to use module extensions. -Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies from different package managers: +Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies +from different package managers: -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) -- [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) -- [rules\_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) - -A minimal example that integrates a pseudo package manager is available at the [examples](https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager) repository. - -### Detect toolchains on the host machine +- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) +- [rules_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) +- [rules_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) -When Bazel build rules need to detect what toolchains are available on your host machine, they use repository rules to inspect the host machine and generate toolchain info as external repositories. +A minimal example that integrates a pseudo package manager is available at the +[examples][pkg-mgr-example] +repository. -- **WORKSPACE** +[pkg-mgr-example]: https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager - Given the following repository rule to detect a shell toolchain. +### Detect toolchains on the host machine - ```python - ## local_config_sh.bzl - def _sh_config_rule_impl(repository_ctx): - sh_path = get_sh_path_from_env("SH_BIN_PATH") +When Bazel build rules need to detect what toolchains are available on your host +machine, they use repository rules to inspect the host machine and generate +toolchain info as external repositories. - if not sh_path: - sh_path = detect_sh_from_path() +* **WORKSPACE** - if not sh_path: - sh_path = "/shell/binary/not/found" + Given the following repository rule to detect a shell toolchain. - repository_ctx.file("BUILD", """ - load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") - sh_toolchain( - name = "local_sh", - path = "{sh_path}", - visibility = ["//visibility:public"], - ) - toolchain( - name = "local_sh_toolchain", - toolchain = ":local_sh", - toolchain_type = "@bazel_tools//tools/sh:toolchain_type", - ) - """.format(sh_path = sh_path)) + ```python + ## local_config_sh.bzl + def _sh_config_rule_impl(repository_ctx): + sh_path = get_sh_path_from_env("SH_BIN_PATH") + + if not sh_path: + sh_path = detect_sh_from_path() + + if not sh_path: + sh_path = "/shell/binary/not/found" + + repository_ctx.file("BUILD", """ + load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") + sh_toolchain( + name = "local_sh", + path = "{sh_path}", + visibility = ["//visibility:public"], + ) + toolchain( + name = "local_sh_toolchain", + toolchain = ":local_sh", + toolchain_type = "@bazel_tools//tools/sh:toolchain_type", + ) + """.format(sh_path = sh_path)) - sh_config_rule = repository_rule( - environ = ["SH_BIN_PATH"], - local = True, - implementation = _sh_config_rule_impl, - ) - ``` + sh_config_rule = repository_rule( + environ = ["SH_BIN_PATH"], + local = True, + implementation = _sh_config_rule_impl, + ) + ``` - You can load the repository rule in WORKSPACE. + You can load the repository rule in WORKSPACE. - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can introduce the same repository using a module extension, which is similar to introducing the `@data_file` repository in the last section. + With Bzlmod, you can introduce the same repository using a module extension, + which is similar to introducing the `@data_file` repository in the last + section. - ``` - ## local_config_sh_extension.bzl - load("//:local_config_sh.bzl", "sh_config_rule") + ``` + ## local_config_sh_extension.bzl + load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_extension = module_extension( - implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), - ) - ``` + sh_config_extension = module_extension( + implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), + ) + ``` - Then use the extension in the MODULE.bazel file. + Then use the extension in the MODULE.bazel file. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - ``` + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + ``` ### Register toolchains & execution platforms -Following the last section, after introducing a repository hosting toolchain information (e.g. `local_config_sh`), you probably want to register the toolchain. +Following the last section, after introducing a repository hosting toolchain +information (e.g. `local_config_sh`), you probably want to register the +toolchain. + +* **WORKSPACE** -- **WORKSPACE** + With WORKSPACE, you can register the toolchain in the following ways. - With WORKSPACE, you can register the toolchain in the following ways. + 1. You can register the toolchain the `.bzl` file and load the macro in the + WORKSPACE file. - 1. You can register the toolchain the `.bzl` file and load the macro in the WORKSPACE file. + ```python + ## local_config_sh.bzl + def sh_configure(): + sh_config_rule(name = "local_config_sh") + native.register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## local_config_sh.bzl - def sh_configure(): - sh_config_rule(name = "local_config_sh") - native.register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_configure") + sh_configure() + ``` - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_configure") - sh_configure() - ``` + 2. Or register the toolchain in the WORKSPACE file directly. - 2. Or register the toolchain in the WORKSPACE file directly. + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +* **Bzlmod** -- **Bzlmod** + With Bzlmod, the + [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and + [`register_execution_platforms`][register_execution_platforms] + APIs are only available in the MODULE.bazel file. You cannot call + `native.register_toolchains` in a module extension. - With Bzlmod, the [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) APIs are only available in the MODULE.bazel file. You cannot call `native.register_toolchains` in a module extension. + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +The toolchains and execution platforms registered in `WORKSPACE`, +`WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this +order of precedence during toolchain selection (from highest to lowest): -The toolchains and execution platforms registered in `WORKSPACE`, `WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this order of precedence during toolchain selection (from highest to lowest): +1. toolchains and execution platforms registered in the root module's + `MODULE.bazel` file. +2. toolchains and execution platforms registered in the `WORKSPACE` or + `WORKSPACE.bzlmod` file. +3. toolchains and execution platforms registered by modules that are + (transitive) dependencies of the root module. +4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` + [suffix](/external/migration#builtin-default-deps). -1. toolchains and execution platforms registered in the root module's `MODULE.bazel` file. -2. toolchains and execution platforms registered in the `WORKSPACE` or `WORKSPACE.bzlmod` file. -3. toolchains and execution platforms registered by modules that are (transitive) dependencies of the root module. -4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` [suffix](/external/migration#builtin-default-deps). +[register_execution_platforms]: /rules/lib/globals/module#register_execution_platforms ### Introduce local repositories -You may need to introduce a dependency as a local repository when you need a local version of the dependency for debugging or you want to incorporate a directory in your workspace as external repository. +You may need to introduce a dependency as a local repository when you need a +local version of the dependency for debugging or you want to incorporate a +directory in your workspace as external repository. -- **WORKSPACE** +* **WORKSPACE** - With WORKSPACE, this is achieved by two native repository rules, [`local_repository`](/reference/be/workspace#local_repository) and [`new_local_repository`](/reference/be/workspace#new_local_repository). + With WORKSPACE, this is achieved by two native repository rules, + [`local_repository`](/reference/be/workspace#local_repository) and + [`new_local_repository`](/reference/be/workspace#new_local_repository). - ```python - ## WORKSPACE - local_repository( - name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## WORKSPACE + local_repository( + name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can use [`local_path_override`](/rules/lib/globals/module#local_path_override) to override a module with a local path. + With Bzlmod, you can use + [`local_path_override`](/rules/lib/globals/module#local_path_override) to + override a module with a local path. - ```python - ## MODULE.bazel - bazel_dep(name = "rules_java") - local_path_override( - module_name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "rules_java") + local_path_override( + module_name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` - Note: With `local_path_override`, you can only introduce a local directory as a Bazel module, which means it should have a MODULE.bazel file and its transitive dependencies are taken into consideration during dependency resolution. In addition, all module override directives can only be used by the root module. + Note: With `local_path_override`, you can only introduce a local directory + as a Bazel module, which means it should have a MODULE.bazel file and its + transitive dependencies are taken into consideration during dependency + resolution. In addition, all module override directives can only be used by + the root module. - It is also possible to introduce a local repository with module extension. However, you cannot call `native.local_repository` in module extension, there is ongoing effort on starlarkifying all native repository rules (check [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). Then you can call the corresponding starlark `local_repository` in a module extension. It's also trivial to implement a custom version of `local_repository` repository rule if this is a blocking issue for you. + It is also possible to introduce a local repository with module extension. + However, you cannot call `native.local_repository` in module extension, + there is ongoing effort on starlarkifying all native repository rules (check + [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). + Then you can call the corresponding starlark `local_repository` in a module + extension. It's also trivial to implement a custom version of + `local_repository` repository rule if this is a blocking issue for you. ### Bind targets -The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and not supported in Bzlmod. It was introduced to give a target an alias in the special `//external` package. All users depending on this should migrate away. +The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and +not supported in Bzlmod. It was introduced to give a target an alias in the +special `//external` package. All users depending on this should migrate away. For example, if you have @@ -448,180 +573,290 @@ bind( ) ``` -This allows other targets to depend on `//external:openssl`. You can migrate away from this by: +This allows other targets to depend on `//external:openssl`. You can migrate +away from this by: -- Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. +* Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. + * Tip: Use `bazel query --output=build --noenable_bzlmod + --enable_workspace [target]` command to find relevant info + about the target. - - Tip: Use `bazel query --output=build --noenable_bzlmod --enable_workspace [target]` command to find relevant info about the target. +* Or use the [`alias`](/reference/be/general#alias) build rule + * Define the following target in a package (e.g. `//third_party`) -- Or use the [`alias`](/reference/be/general#alias) build rule - - - Define the following target in a package (e.g. `//third_party`) - - ```python - ## third_party/BUILD - alias( - name = "openssl", - actual = "@my-ssl//src:openssl-lib", - ) - ``` + ```python + ## third_party/BUILD + alias( + name = "openssl", + actual = "@my-ssl//src:openssl-lib", + ) + ``` - - Replace all usages of `//external:openssl` with `//third_party:openssl`. + * Replace all usages of `//external:openssl` with `//third_party:openssl`. ### Fetch versus Sync -Fetch and sync commands are used to download external repos locally and keep them updated. Sometimes also to allow building offline using the `--nofetch` flag after fetching all repos needed for a build. +Fetch and sync commands are used to download external repos locally and keep +them updated. Sometimes also to allow building offline using the `--nofetch` +flag after fetching all repos needed for a build. -- **WORKSPACE** +* **WORKSPACE** - Sync performs a force fetch for all repositories, or for a specific configured set of repos, while fetch is *only* used to fetch for a specific target. + Sync performs a force fetch for all repositories, or for a specific + configured set of repos, while fetch is _only_ used to fetch for a specific + target. -- **Bzlmod** +* **Bzlmod** - The sync command is no longer applicable, but fetch offers [various options](/reference/command-line-reference#fetch-options). You can fetch a target, a repository, a set of configured repos or all repositories involved in your dependency resolution and module extensions. The fetch result is cached and to force a fetch you must include the `--force` option during the fetch process. + The sync command is no longer applicable, but fetch offers + [various options](/reference/command-line-reference#fetch-options). + You can fetch a target, a repository, a set of configured repos or all + repositories involved in your dependency resolution and module extensions. + The fetch result is cached and to force a fetch you must include the + `--force` option during the fetch process. ## Manual migration -This section provides useful information and guidance for your **manual** Bzlmod migration process. For more automatized migration process, check [recommended migration process](#how-migrate-to-bzlmod) section. +This section provides useful information and guidance for your **manual** Bzlmod +migration process. For more automatized migration process, check [recommended +migration process](#how-migrate-to-bzlmod) section. ### Know your dependencies in WORKSPACE -The first step of migration is to understand what dependencies you have. It could be hard to figure out what exact dependencies are introduced in the WORKSPACE file because transitive dependencies are often loaded with `*_deps` macros. +The first step of migration is to understand what dependencies you have. It +could be hard to figure out what exact dependencies are introduced in the +WORKSPACE file because transitive dependencies are often loaded with `*_deps` +macros. #### Inspect external dependency with workspace resolved file -Fortunately, the flag [`--experimental_repository_resolved_file`](/reference/command-line-reference#flag--experimental_repository_resolved_file) can help. This flag essentially generates a "lock file" of all fetched external dependencies in your last Bazel command. You can find more details in this [blog post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). +Fortunately, the flag +[`--experimental_repository_resolved_file`][resolved_file_flag] +can help. This flag essentially generates a "lock file" of all fetched external +dependencies in your last Bazel command. You can find more details in this [blog +post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). + +[resolved_file_flag]: /reference/command-line-reference#flag--experimental_repository_resolved_file It can be used in two ways: -1. To fetch info of external dependencies needed for building certain targets. +1. To fetch info of external dependencies needed for building certain targets. - ```shell - bazel clean --expunge - bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar - ``` + ```shell + bazel clean --expunge + bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar + ``` -2. To fetch info of all external dependencies defined in the WORKSPACE file. +2. To fetch info of all external dependencies defined in the WORKSPACE file. - ```shell - bazel clean --expunge - bazel sync --experimental_repository_resolved_file=resolved.bzl - ``` + ```shell + bazel clean --expunge + bazel sync --experimental_repository_resolved_file=resolved.bzl + ``` - With the `bazel sync` command, you can fetch all dependencies defined in the WORKSPACE file, which include: + With the `bazel sync` command, you can fetch all dependencies defined in the + WORKSPACE file, which include: - - `bind` usages - - `register_toolchains` & `register_execution_platforms` usages + * `bind` usages + * `register_toolchains` & `register_execution_platforms` usages - However, if your project is cross platforms, bazel sync may break on certain platforms because some repository rules may only run correctly on supported platforms. + However, if your project is cross platforms, bazel sync may break on certain + platforms because some repository rules may only run correctly on supported + platforms. -After running the command, you should have information of your external dependencies in the `resolved.bzl` file. +After running the command, you should have information of your external +dependencies in the `resolved.bzl` file. #### Inspect external dependency with `bazel query` You may also know `bazel query` can be used for inspecting repository rules with ```shell -bazel query --output=build //external:<repo name> +bazel query --output=build //external: ``` -While it is more convenient and much faster, [bazel query can lie about external dependency version](https://github.com/bazelbuild/bazel/issues/12947), so be careful using it! Querying and inspecting external dependencies with Bzlmod is going to achieved by a [new subcommand](https://github.com/bazelbuild/bazel/issues/15365). +While it is more convenient and much faster, [bazel query can lie about +external dependency version](https://github.com/bazelbuild/bazel/issues/12947), +so be careful using it! Querying and inspecting external +dependencies with Bzlmod is going to achieved by a [new +subcommand](https://github.com/bazelbuild/bazel/issues/15365). #### Built-in default dependencies -If you check the file generated by `--experimental_repository_resolved_file`, you are going to find many dependencies that are not defined in your WORKSPACE. This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE file content to inject some default dependencies, which are usually required by native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With Bzlmod, those dependencies are introduced with a built-in module [`bazel_tools`](https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools) , which is a default dependency for every other Bazel module. +If you check the file generated by `--experimental_repository_resolved_file`, +you are going to find many dependencies that are not defined in your WORKSPACE. +This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE +file content to inject some default dependencies, which are usually required by +native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With +Bzlmod, those dependencies are introduced with a built-in module +[`bazel_tools`][bazel_tools] , which is a default dependency for every other +Bazel module. + +[bazel_tools]: https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools ### Hybrid mode for gradual migration -Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies from the WORKSPACE file to Bzlmod to be a gradual process. +Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies +from the WORKSPACE file to Bzlmod to be a gradual process. -Note: In practice, loading "\*\_deps" macros in WORKSPACE often causes confusions with Bzlmod dependencies, therefore we recommend starting with a WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. +Note: In practice, loading "*_deps" macros in WORKSPACE often causes confusions +with Bzlmod dependencies, therefore we recommend starting with a +WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. #### WORKSPACE.bzlmod -During the migration, Bazel users may need to switch between builds with and without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the process smoother. +During the migration, Bazel users may need to switch between builds with and +without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the +process smoother. -WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, if a WORKSPACE.bzlmod file also exists at the workspace root: +WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, +if a WORKSPACE.bzlmod file also exists at the workspace root: -- `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. -- No [prefixes or suffixes](/external/migration#builtin-default-deps) are added to the WORKSPACE.bzlmod file. +* `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. +* No [prefixes or suffixes](/external/migration#builtin-default-deps) are + added to the WORKSPACE.bzlmod file. Using the WORKSPACE.bzlmod file can make the migration easier because: -- When Bzlmod is disabled, you fall back to fetching dependencies from the original WORKSPACE file. -- When Bzlmod is enabled, you can better track what dependencies are left to migrate with WORKSPACE.bzlmod. +* When Bzlmod is disabled, you fall back to fetching dependencies from the + original WORKSPACE file. +* When Bzlmod is enabled, you can better track what dependencies are left to + migrate with WORKSPACE.bzlmod. #### Repository visibility -Bzlmod is able to control which other repositories are visible from a given repository, check [repository names and strict deps](/external/module#repository_names_and_strict_deps) for more details. +Bzlmod is able to control which other repositories are visible from a given +repository, check [repository names and strict +deps](/external/module#repository_names_and_strict_deps) for more details. -Here is a summary of repository visibilities from different types of repositories when also taking WORKSPACE into consideration. +Here is a summary of repository visibilities from different types of +repositories when also taking WORKSPACE into consideration. -| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | -| ---------------------- | ------------------ | ----------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------ | -| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | -| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | -| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | -| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | +| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | +|----------------|--------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------|----------------------| +| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | +| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | +| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | +| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | -Note: For the root module, if a repository `@foo` is defined in WORKSPACE and `@foo` is also used as an [apparent repository name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` refers to the one introduced in MODULE.bazel. +Note: For the root module, if a repository `@foo` is defined in WORKSPACE and +`@foo` is also used as an [apparent repository +name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` +refers to the one introduced in MODULE.bazel. -Note: For a module extension generated repository `@bar`, if `@foo` is used as an [apparent repository name](/external/overview#apparent-repo-name) of another repository generated by the same module extension and direct dependencies of the module hosting the module extension, then for repository `@bar`, `@foo` refers to the latter. +Note: For a module extension generated repository `@bar`, if `@foo` is used as +an [apparent repository name](/external/overview#apparent-repo-name) of +another repository generated by the same module extension and direct +dependencies of the module hosting the module extension, then for repository +`@bar`, `@foo` refers to the latter. ### Manual migration process A typical Bzlmod migration process can look like this: -1. Understand what dependencies you have in WORKSPACE. -2. Add an empty MODULE.bazel file at your project root. -3. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. -4. Build your targets with Bzlmod enabled and check which repository is missing. -5. Check the definition of the missing repository in the resolved dependency file. -6. Introduce the missing dependency as a Bazel module, through a module extension, or leave it in the WORKSPACE.bzlmod for later migration. -7. Go back to 4 and repeat until all dependencies are available. +1. Understand what dependencies you have in WORKSPACE. +1. Add an empty MODULE.bazel file at your project root. +1. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. +1. Build your targets with Bzlmod enabled and check which repository is + missing. +1. Check the definition of the missing repository in the resolved dependency + file. +1. Introduce the missing dependency as a Bazel module, through a module + extension, or leave it in the WORKSPACE.bzlmod for later migration. +1. Go back to 4 and repeat until all dependencies are available. ## Publish Bazel modules -If your Bazel project is a dependency for other projects, you can publish your project in the [Bazel Central Registry](https://registry.bazel.build/). +If your Bazel project is a dependency for other projects, you can publish your +project in the [Bazel Central Registry](https://registry.bazel.build/). + +To be able to check in your project in the BCR, you need a source archive URL of +the project. Take note of a few things when creating the source archive: -To be able to check in your project in the BCR, you need a source archive URL of the project. Take note of a few things when creating the source archive: +* **Make sure the archive is pointing to a specific version.** -- **Make sure the archive is pointing to a specific version.** + The BCR can only accept versioned source archives because Bzlmod needs to + conduct version comparison during dependency resolution. - The BCR can only accept versioned source archives because Bzlmod needs to conduct version comparison during dependency resolution. +* **Make sure the archive URL is stable.** -- **Make sure the archive URL is stable.** + Bazel verifies the content of the archive by a hash value, so you should + make sure the checksum of the downloaded file never changes. If the URL is + from GitHub, please create and upload a release archive in the release page. + GitHub isn't going to guarantee the checksum of source archives generated on + demand. In short, URLs in the form of + `https://github.com///releases/download/...` is considered stable + while `https://github.com///archive/...` is not. Check [GitHub + Archive Checksum + Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) + for more context. - Bazel verifies the content of the archive by a hash value, so you should make sure the checksum of the downloaded file never changes. If the URL is from GitHub, please create and upload a release archive in the release page. GitHub isn't going to guarantee the checksum of source archives generated on demand. In short, URLs in the form of `https://github.com/<org>/<repo>/releases/download/...` is considered stable while `https://github.com/<org>/<repo>/archive/...` is not. Check [GitHub Archive Checksum Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) for more context. +* **Make sure the source tree follows the layout of the original repository.** -- **Make sure the source tree follows the layout of the original repository.** + In case your repository is very large and you want to create a distribution + archive with reduced size by stripping out unnecessary sources, please make + sure the stripped source tree is a subset of the original source tree. This + makes it easier for end users to override the module to a non-release + version by [`archive_override`](/rules/lib/globals/module#archive_override) + and [`git_override`](/rules/lib/globals/module#git_override). - In case your repository is very large and you want to create a distribution archive with reduced size by stripping out unnecessary sources, please make sure the stripped source tree is a subset of the original source tree. This makes it easier for end users to override the module to a non-release version by [`archive_override`](/rules/lib/globals/module#archive_override) and [`git_override`](/rules/lib/globals/module#git_override). +* **Include a test module in a subdirectory that tests your most common + APIs.** -- **Include a test module in a subdirectory that tests your most common APIs.** + A test module is a Bazel project with its own WORKSPACE and MODULE.bazel + file located in a subdirectory of the source archive which depends on the + actual module to be published. It should contain examples or some + integration tests that cover your most common APIs. Check + [test module][test_module] to learn how to set it up. - A test module is a Bazel project with its own WORKSPACE and MODULE.bazel file located in a subdirectory of the source archive which depends on the actual module to be published. It should contain examples or some integration tests that cover your most common APIs. Check [test module](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module) to learn how to set it up. +[test_module]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module -When you have your source archive URL ready, follow the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module) to submit your module to the BCR with a GitHub Pull Request. +When you have your source archive URL ready, follow the [BCR contribution +guidelines][bcr_contrib_guide] to submit your module to the BCR with a GitHub +Pull Request. -It is **highly recommended** to set up the [Publish to BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your repository to automate the process of submitting your module to the BCR. +[bcr_contrib_guide]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module + +It is **highly recommended** to set up the [Publish to +BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your +repository to automate the process of submitting your module to the BCR. ## Best practices -This section documents a few best practices you should follow for better managing your external dependencies. +This section documents a few best practices you should follow for better +managing your external dependencies. #### Split targets into different packages to avoid fetching unnecessary dependencies. -Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev dependencies for tests are forced to be fetched unnecessarily for building targets that don't need them. This is actually not Bzlmod specific, but following this practices makes it easier to specify dev dependencies correctly. +Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev +dependencies for tests are forced to be fetched unnecessarily for building +targets that don't need them. This is actually not Bzlmod specific, but +following this practices makes it easier to specify dev dependencies correctly. #### Specify dev dependencies -You can set the `dev_dependency` attribute to true for [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_extension`](/rules/lib/globals/module#use_extension) directives so that they don't propagate to dependent projects. As the root module, you can use the [`--ignore_dev_dependency`](/reference/command-line-reference#flag--ignore_dev_dependency) flag to verify if your targets still build without dev dependencies and overrides. +You can set the `dev_dependency` attribute to true for +[`bazel_dep`](/rules/lib/globals/module#bazel_dep) and +[`use_extension`](/rules/lib/globals/module#use_extension) directives so that +they don't propagate to dependent projects. As the root module, you can use the +[`--ignore_dev_dependency`][ignore_dev_dep_flag] flag to verify if your targets +still build without dev dependencies and overrides. + +[ignore_dev_dep_flag]: /reference/command-line-reference#flag--ignore_dev_dependency + + ## Community migration progress -You can check the [Bazel Central Registry](https://registry.bazel.build) to find out if your dependencies are already available. Otherwise feel free to join this [GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to upvote or post the dependencies that are blocking your migration. +You can check the [Bazel Central Registry](https://registry.bazel.build) to find +out if your dependencies are already available. Otherwise feel free to join this +[GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to +upvote or post the dependencies that are blocking your migration. ## Report issues -Please check the [Bazel GitHub issue list](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod) for known Bzlmod issues. Feel free to file new issues or feature requests that can help unblock your migration! +Please check the [Bazel GitHub issue list][bzlmod_github_issue] for known Bzlmod +issues. Feel free to file new issues or feature requests that can help unblock +your migration! + +[bzlmod_github_issue]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod diff --git a/external/module.mdx b/external/module.mdx index 9ac802a4..0499c4cd 100644 --- a/external/module.mdx +++ b/external/module.mdx @@ -2,9 +2,16 @@ title: 'Bazel modules' --- -A Bazel **module** is a Bazel project that can have multiple versions, each of which publishes metadata about other modules that it depends on. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. -A module must have a `MODULE.bazel` file at its repo root. This file is the module's manifest, declaring its name, version, list of direct dependencies, and other information. For a basic example: + +A Bazel **module** is a Bazel project that can have multiple versions, each of +which publishes metadata about other modules that it depends on. This is +analogous to familiar concepts in other dependency management systems, such as a +Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. + +A module must have a `MODULE.bazel` file at its repo root. This file is the +module's manifest, declaring its name, version, list of direct dependencies, and +other information. For a basic example: ```python module(name = "my-module", version = "1.0") @@ -13,29 +20,50 @@ bazel_dep(name = "rules_cc", version = "0.0.1") bazel_dep(name = "protobuf", version = "3.19.0") ``` -See the [full list](/rules/lib/globals/module) of directives available in `MODULE.bazel` files. +See the [full list](/rules/lib/globals/module) of directives available in +`MODULE.bazel` files. -To perform module resolution, Bazel starts by reading the root module's `MODULE.bazel` file, and then repeatedly requests any dependency's `MODULE.bazel` file from a [Bazel registry](/external/registry) until it discovers the entire dependency graph. +To perform module resolution, Bazel starts by reading the root module's +`MODULE.bazel` file, and then repeatedly requests any dependency's +`MODULE.bazel` file from a [Bazel registry](/external/registry) until it +discovers the entire dependency graph. -By default, Bazel then [selects](#version-selection) one version of each module to use. Bazel represents each module with a repo, and consults the registry again to learn how to define each of the repos. +By default, Bazel then [selects](#version-selection) one version of each module +to use. Bazel represents each module with a repo, and consults the registry +again to learn how to define each of the repos. ## Version format -Bazel has a diverse ecosystem and projects use various versioning schemes. The most popular by far is [SemVer](https://semver.org), but there are also prominent projects using different schemes such as [Abseil](https://github.com/abseil/abseil-cpp/releases), whose versions are date-based, for example `20210324.2`). +Bazel has a diverse ecosystem and projects use various versioning schemes. The +most popular by far is [SemVer](https://semver.org), but there are +also prominent projects using different schemes such as +[Abseil](https://github.com/abseil/abseil-cpp/releases), whose +versions are date-based, for example `20210324.2`). -For this reason, Bazel adopts a more relaxed version of the SemVer spec. The differences include: +For this reason, Bazel adopts a more relaxed version of the SemVer spec. The +differences include: -- SemVer prescribes that the "release" part of the version must consist of 3 segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so that any number of segments is allowed. -- In SemVer, each of the segments in the "release" part must be digits only. In Bazel, this is loosened to allow letters too, and the comparison semantics match the "identifiers" in the "prerelease" part. -- Additionally, the semantics of major, minor, and patch version increases are not enforced. However, see [compatibility level](#compatibility_level) for details on how we denote backwards compatibility. +* SemVer prescribes that the "release" part of the version must consist of 3 + segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so + that any number of segments is allowed. +* In SemVer, each of the segments in the "release" part must be digits only. + In Bazel, this is loosened to allow letters too, and the comparison + semantics match the "identifiers" in the "prerelease" part. +* Additionally, the semantics of major, minor, and patch version increases are + not enforced. However, see [compatibility level](#compatibility_level) for + details on how we denote backwards compatibility. -Any valid SemVer version is a valid Bazel module version. Additionally, two SemVer versions `a` and `b` compare `a < b` if and only if the same holds when they're compared as Bazel module versions. +Any valid SemVer version is a valid Bazel module version. Additionally, two +SemVer versions `a` and `b` compare `a < b` if and only if the same holds when +they're compared as Bazel module versions. -Finally, to learn more about module versioning, [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). +Finally, to learn more about module versioning, [see the `MODULE.bazel` +FAQ](faq#module-versioning-best-practices). ## Version selection -Consider the diamond dependency problem, a staple in the versioned dependency management space. Suppose you have the dependency graph: +Consider the diamond dependency problem, a staple in the versioned dependency +management space. Suppose you have the dependency graph: ``` A 1.0 @@ -45,78 +73,164 @@ Consider the diamond dependency problem, a staple in the versioned dependency ma D 1.0 D 1.1 ``` -Which version of `D` should be used? To resolve this question, Bazel uses the [Minimal Version Selection](https://research.swtch.com/vgo-mvs) (MVS) algorithm introduced in the Go module system. MVS assumes that all new versions of a module are backwards compatible, and so picks the highest version specified by any dependent (`D 1.1` in our example). It's called "minimal" because `D 1.1` is the earliest version that could satisfy our requirements — even if `D 1.2` or newer exists, we don't select them. Using MVS creates a version selection process that is *high-fidelity* and *reproducible*. +Which version of `D` should be used? To resolve this question, Bazel uses the +[Minimal Version Selection](https://research.swtch.com/vgo-mvs) +(MVS) algorithm introduced in the Go module system. MVS assumes that all new +versions of a module are backwards compatible, and so picks the highest version +specified by any dependent (`D 1.1` in our example). It's called "minimal" +because `D 1.1` is the earliest version that could satisfy our requirements — +even if `D 1.2` or newer exists, we don't select them. Using MVS creates a +version selection process that is *high-fidelity* and *reproducible*. ### Yanked versions -The registry can declare certain versions as *yanked* if they should be avoided (such as for security vulnerabilities). Bazel throws an error when selecting a yanked version of a module. To fix this error, either upgrade to a newer, non-yanked version, or use the [`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) flag to explicitly allow the yanked version. +The registry can declare certain versions as *yanked* if they should be avoided +(such as for security vulnerabilities). Bazel throws an error when selecting a +yanked version of a module. To fix this error, either upgrade to a newer, +non-yanked version, or use the +[`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) +flag to explicitly allow the yanked version. ## Compatibility level -In Go, MVS's assumption about backwards compatibility works because it treats backwards incompatible versions of a module as a separate module. In terms of SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can coexist in the resolved dependency graph. This is, in turn, made possible by encoding the major version in the package path in Go, so there aren't any compile-time or linking-time conflicts. Bazel, however, cannot provide such guarantees because it follows [a relaxed version of SemVer](#version-format). - -Thus, Bazel needs the equivalent of the SemVer major version number to detect backwards incompatible ("breaking") versions. This number is called the *compatibility level*, and is specified by each module version in its [`module()`](/rule/lib/globals/module#module) directive. With this information, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. - -Finally, incrementing the compatibility level can be disruptive to the users. To learn more about when and how to increment it, [check the `MODULE.bazel` FAQ](faq#incrementing-compatibility-level). +In Go, MVS's assumption about backwards compatibility works because it treats +backwards incompatible versions of a module as a separate module. In terms of +SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can +coexist in the resolved dependency graph. This is, in turn, made possible by +encoding the major version in the package path in Go, so there aren't any +compile-time or linking-time conflicts. Bazel, however, cannot provide such +guarantees because it follows [a relaxed version of SemVer](#version-format). + +Thus, Bazel needs the equivalent of the SemVer major version number to detect +backwards incompatible ("breaking") versions. This number is called the +*compatibility level*, and is specified by each module version in its +[`module()`](/rule/lib/globals/module#module) directive. With this information, +Bazel can throw an error if it detects that versions of the _same module_ with +_different compatibility levels_ exist in the resolved dependency graph. + +Finally, incrementing the compatibility level can be disruptive to the users. +To learn more about when and how to increment it, [check the `MODULE.bazel` +FAQ](faq#incrementing-compatibility-level). ## Overrides -Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel module resolution. Only the root module's overrides take effect — if a module is used as a dependency, its overrides are ignored. +Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel +module resolution. Only the root module's overrides take effect — if a module is +used as a dependency, its overrides are ignored. -Each override is specified for a certain module name, affecting all of its versions in the dependency graph. Although only the root module's overrides take effect, they can be for transitive dependencies that the root module does not directly depend on. +Each override is specified for a certain module name, affecting all of its +versions in the dependency graph. Although only the root module's overrides take +effect, they can be for transitive dependencies that the root module does not +directly depend on. ### Single-version override -The [`single_version_override`](/rules/lib/globals/module#single_version_override) serves multiple purposes: +The [`single_version_override`](/rules/lib/globals/module#single_version_override) +serves multiple purposes: -- With the `version` attribute, you can pin a dependency to a specific version, regardless of which versions of the dependency are requested in the dependency graph. -- With the `registry` attribute, you can force this dependency to come from a specific registry, instead of following the normal [registry selection](/external/registry#selecting_registries) process. -- With the `patch*` attributes, you can specify a set of patches to apply to the downloaded module. +* With the `version` attribute, you can pin a dependency to a specific + version, regardless of which versions of the dependency are requested in the + dependency graph. +* With the `registry` attribute, you can force this dependency to come from a + specific registry, instead of following the normal [registry + selection](/external/registry#selecting_registries) process. +* With the `patch*` attributes, you can specify a set of patches to apply to + the downloaded module. These attributes are all optional and can be mixed and matched with each other. ### Multiple-version override -A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) can be specified to allow multiple versions of the same module to coexist in the resolved dependency graph. - -You can specify an explicit list of allowed versions for the module, which must all be present in the dependency graph before resolution — there must exist *some* transitive dependency depending on each allowed version. After resolution, only the allowed versions of the module remain, while Bazel upgrades other versions of the module to the nearest higher allowed version at the same compatibility level. If no higher allowed version at the same compatibility level exists, Bazel throws an error. - -For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the dependency graph before resolution and the major version is the compatibility level: - -- A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other versions remaining the same. -- A multiple-version override allowing `1.5` and `2.0` results in an error, as `1.7` has no higher version at the same compatibility level to upgrade to. -- A multiple-version override allowing `1.9` and `2.0` results in an error, as `1.9` is not present in the dependency graph before resolution. - -Additionally, users can also override the registry using the `registry` attribute, similarly to single-version overrides. +A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) +can be specified to allow multiple versions of the same module to coexist in the +resolved dependency graph. + +You can specify an explicit list of allowed versions for the module, which must +all be present in the dependency graph before resolution — there must exist +*some* transitive dependency depending on each allowed version. After +resolution, only the allowed versions of the module remain, while Bazel upgrades +other versions of the module to the nearest higher allowed version at the same +compatibility level. If no higher allowed version at the same compatibility +level exists, Bazel throws an error. + +For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the +dependency graph before resolution and the major version is the compatibility +level: + +* A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in + `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other + versions remaining the same. +* A multiple-version override allowing `1.5` and `2.0` results in an error, as + `1.7` has no higher version at the same compatibility level to upgrade to. +* A multiple-version override allowing `1.9` and `2.0` results in an error, as + `1.9` is not present in the dependency graph before resolution. + +Additionally, users can also override the registry using the `registry` +attribute, similarly to single-version overrides. ### Non-registry overrides -Non-registry overrides completely remove a module from version resolution. Bazel does not request these `MODULE.bazel` files from a registry, but instead from the repo itself. +Non-registry overrides completely remove a module from version resolution. Bazel +does not request these `MODULE.bazel` files from a registry, but instead from +the repo itself. Bazel supports the following non-registry overrides: -- [`archive_override`](/rules/lib/globals/module#archive_override) -- [`git_override`](/rules/lib/globals/module#git_override) -- [`local_path_override`](/rules/lib/globals/module#local_path_override) +* [`archive_override`](/rules/lib/globals/module#archive_override) +* [`git_override`](/rules/lib/globals/module#git_override) +* [`local_path_override`](/rules/lib/globals/module#local_path_override) -Note that setting a version value in the source archive `MODULE.bazel` can have downsides when the module is being overridden with a non-registry override. To learn more about this [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). +Note that setting a version value in the source archive `MODULE.bazel` can have +downsides when the module is being overridden with a non-registry override. To +learn more about this [see the `MODULE.bazel` +FAQ](faq#module-versioning-best-practices). ## Define repos that don't represent Bazel modules -With `bazel_dep`, you can define repos that represent other Bazel modules. Sometimes there is a need to define a repo that does *not* represent a Bazel module; for example, one that contains a plain JSON file to be read as data. +With `bazel_dep`, you can define repos that represent other Bazel modules. +Sometimes there is a need to define a repo that does _not_ represent a Bazel +module; for example, one that contains a plain JSON file to be read as data. -In this case, you could use the [`use_repo_rule` directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo by invoking a repo rule. This repo will only be visible to the module it's defined in. +In this case, you could use the [`use_repo_rule` +directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo +by invoking a repo rule. This repo will only be visible to the module it's +defined in. -Under the hood, this is implemented using the same mechanism as [module extensions](/external/extension), which lets you define repos with more flexibility. +Under the hood, this is implemented using the same mechanism as [module +extensions](/external/extension), which lets you define repos with more +flexibility. ## Repository names and strict deps -The [apparent name](/external/overview#apparent-repo-name) of a repo backing a module to its direct dependents defaults to its module name, unless the `repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive says otherwise. Note that this means a module can only find its direct dependencies. This helps prevent accidental breakages due to changes in transitive dependencies. - -The [canonical name](/external/overview#canonical-repo-name) of a repo backing a module is either `<var>module_name</var>+<var>version</var>` (for example, `bazel_skylib+1.0.3`) or `<var>module_name</var>+` (for example, `bazel_features+`), depending on whether there are multiple versions of the module in the entire dependency graph (see [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). Note that **the canonical name format** is not an API you should depend on and **is subject to change at any time**. Instead of hard-coding the canonical name, use a supported way to get it directly from Bazel: - -- In BUILD and `.bzl` files, use [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance constructed from a label string given by the apparent name of the repo, e.g., `Label("@bazel_skylib").repo_name`. -- When looking up runfiles, use [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) or one of the runfiles libraries in `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, in `@rules_foo//foo/runfiles`. -- When interacting with Bazel from an external tool such as an IDE or language server, use the `bazel mod dump_repo_mapping` command to get the mapping from apparent names to canonical names for a given set of repositories. - -[Module extensions](/external/extension) can also introduce additional repos into the visible scope of a module. +The [apparent name](/external/overview#apparent-repo-name) of a repo backing a +module to its direct dependents defaults to its module name, unless the +`repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) +directive says otherwise. Note that this means a module can only find its direct +dependencies. This helps prevent accidental breakages due to changes in +transitive dependencies. + +The [canonical name](/external/overview#canonical-repo-name) of a repo backing a +module is either `module_name+version{{ +"" }}` (for example, `bazel_skylib+1.0.3`) or `module_name{{ +"" }}+` (for example, `bazel_features+`), depending on whether there are +multiple versions of the module in the entire dependency graph (see +[`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). +Note that **the canonical name format** is not an API you should depend on and +**is subject to change at any time**. Instead of hard-coding the canonical name, +use a supported way to get it directly from Bazel: + +* In BUILD and `.bzl` files, use + [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance + constructed from a label string given by the apparent name of the repo, e.g., + `Label("@bazel_skylib").repo_name`. +* When looking up runfiles, use + [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) + or one of the runfiles libraries in + `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, + in `@rules_foo//foo/runfiles`. +* When interacting with Bazel from an external tool such as an IDE or language + server, use the `bazel mod dump_repo_mapping` command to get the mapping from + apparent names to canonical names for a given set of repositories. + +[Module extensions](/external/extension) can also introduce additional repos +into the visible scope of a module. diff --git a/external/overview.mdx b/external/overview.mdx index 84593001..6e55b579 100644 --- a/external/overview.mdx +++ b/external/overview.mdx @@ -2,15 +2,28 @@ title: 'External dependencies overview' --- -Bazel supports *external dependencies*, source files (both text and binary) used in your build that are not from your workspace. For example, they could be a ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local machine outside your current workspace. -This document gives an overview of the system before examining some of the concepts in more detail. + + +Bazel supports *external dependencies*, source files (both text and binary) used +in your build that are not from your workspace. For example, they could be a +ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local +machine outside your current workspace. + +This document gives an overview of the system before examining some of the +concepts in more detail. ## Overview of the system -Bazel's external dependency system works on the basis of [*Bazel modules*](#module), each of which is a versioned Bazel project, and [*repositories*](#repository) (or repos), which are directory trees containing source files. +Bazel's external dependency system works on the basis of [*Bazel +modules*](#module), each of which is a versioned Bazel project, and +[*repositories*](#repository) (or repos), which are directory trees containing +source files. -Bazel starts from the root module -- that is, the project you're working on. Like all modules, it needs to have a `MODULE.bazel` file at its directory root, declaring its basic metadata and direct dependencies. The following is a basic example: +Bazel starts from the root module -- that is, the project you're working on. +Like all modules, it needs to have a `MODULE.bazel` file at its directory root, +declaring its basic metadata and direct dependencies. The following is a basic +example: ```python module(name = "my-module", version = "1.0") @@ -19,13 +32,29 @@ bazel_dep(name = "rules_cc", version = "0.1.1") bazel_dep(name = "platforms", version = "0.0.11") ``` -From there, Bazel looks up all transitive dependency modules in a [*Bazel registry*](registry) — by default, the [Bazel Central Registry](https://bcr.bazel.build/). The registry provides the dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire transitive dependency graph before performing version resolution. - -After version resolution, in which one version is selected for each module, Bazel consults the registry again to learn how to define a repo for each module -- that is, how the sources for each dependency module should be fetched. Most of the time, these are just archives downloaded from the internet and extracted. - -Modules can also specify customized pieces of data called *tags*, which are consumed by [*module extensions*](extension) after module resolution to define additional repos. These extensions can perform actions like file I/O and sending network requests. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. - -The three kinds of repos -- the main repo (which is the source tree you're working in), the repos representing transitive dependency modules, and the repos created by module extensions -- form the [*workspace*](#workspace) together. External repos (non-main repos) are fetched on demand, for example when they're referred to by labels (like `@repo//pkg:target`) in BUILD files. +From there, Bazel looks up all transitive dependency modules in a +[*Bazel registry*](registry) — by default, the [Bazel Central +Registry](https://bcr.bazel.build/). The registry provides the +dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire +transitive dependency graph before performing version resolution. + +After version resolution, in which one version is selected for each module, +Bazel consults the registry again to learn how to define a repo for each module +-- that is, how the sources for each dependency module should be fetched. Most +of the time, these are just archives downloaded from the internet and extracted. + +Modules can also specify customized pieces of data called *tags*, which are +consumed by [*module extensions*](extension) after module resolution +to define additional repos. These extensions can perform actions like file I/O +and sending network requests. Among other things, they allow Bazel to interact +with other package management systems while also respecting the dependency graph +built out of Bazel modules. + +The three kinds of repos -- the main repo (which is the source tree you're +working in), the repos representing transitive dependency modules, and the repos +created by module extensions -- form the [*workspace*](#workspace) together. +External repos (non-main repos) are fetched on demand, for example when they're +referred to by labels (like `@repo//pkg:target`) in BUILD files. ## Benefits @@ -33,30 +62,54 @@ Bazel's external dependency system offers a wide range of benefits. ### Automatic Dependency Resolution -- **Deterministic Version Resolution**: Bazel adopts the deterministic [MVS](module#version-selection) version resolution algorithm, minimizing conflicts and addressing diamond dependency issues. -- **Simplified Dependency Management**: `MODULE.bazel` declares only direct dependencies, while transitive dependencies are automatically resolved, providing a clearer overview of the project's dependencies. -- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: Only direct dependencies are visible, ensuring correctness and predictability. +- **Deterministic Version Resolution**: Bazel adopts the deterministic + [MVS](module#version-selection) version resolution algorithm, + minimizing conflicts and addressing diamond dependency issues. +- **Simplified Dependency Management**: `MODULE.bazel` declares only direct + dependencies, while transitive dependencies are automatically resolved, + providing a clearer overview of the project's dependencies. +- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: + Only direct dependencies are visible, ensuring correctness and + predictability. ### Ecosystem Integration -- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized repository for discovering and managing common dependencies as Bazel modules. - -- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ library) is adapted for Bazel and made available in BCR, it streamlines its integration for the whole community and eliminates duplicated effort and conflicts of custom BUILD files. - -- **Unified Integration with Language-Specific Package Managers**: Rulesets streamline integration with external package managers for non-Bazel dependencies, including: - - - [rules\_jvm\_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) for Maven, - - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) for PyPi, - - [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) for Go Modules, - - [rules\_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) for Cargo. +- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized + repository for discovering and managing common dependencies as Bazel + modules. +- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ + library) is adapted for Bazel and made available in BCR, it streamlines its + integration for the whole community and eliminates duplicated effort and + conflicts of custom BUILD files. +- **Unified Integration with Language-Specific Package Managers**: Rulesets + streamline integration with external package managers for non-Bazel + dependencies, including: + * [rules_jvm_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) + for Maven, + * [rules_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) + for PyPi, + * [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) + for Go Modules, + * [rules_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) + for Cargo. ### Advanced Features -- **[Module Extensions](extension)**: The [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module extension features allow flexible use of custom repository rules and resolution logic to introduce any non-Bazel dependencies. -- **[`bazel mod` Command](mod-command)**: The sub-command offers powerful ways to inspect external dependencies. You know exactly how an external dependency is defined and where it comes from. -- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you need to facilitate offline builds. -- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and accelerates dependency resolution. -- **(Upcoming) [BCR Provenance Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: Strengthen supply chain security by ensuring verified provenance of dependencies. +- **[Module Extensions](extension)**: The + [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module + extension features allow flexible use of custom repository rules and + resolution logic to introduce any non-Bazel dependencies. +- **[`bazel mod` Command](mod-command)**: The sub-command offers + powerful ways to inspect external dependencies. You know exactly how an + external dependency is defined and where it comes from. +- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you + need to facilitate offline builds. +- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and + accelerates dependency resolution. +- **(Upcoming) [BCR Provenance + Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: + Strengthen supply chain security by ensuring verified provenance of + dependencies. ## Concepts @@ -64,7 +117,8 @@ This section gives more detail on concepts related to external dependencies. ### Module -A Bazel project that can have multiple versions, each of which can have dependencies on other modules. +A Bazel project that can have multiple versions, each of which can have +dependencies on other modules. In a local Bazel workspace, a module is represented by a repository. @@ -72,71 +126,118 @@ For more details, see [Bazel modules](module). ### Repository -A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source +files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo +represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in +legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file +will signify the boundary of a repo; multiple such files can coexist in a +directory. ### Main repository The repository in which the current Bazel command is being run. -The root of the main repository is also known as the [](). +The root of the main repository is also known as the +**workspace root**. ### Workspace -The environment shared by all Bazel commands run in the same main repository. It encompasses the main repo and the set of all defined external repos. +The environment shared by all Bazel commands run in the same main repository. It +encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". ### Canonical repository name -The name by which a repository is always addressable. Within the context of a workspace, each repository has a single canonical name. A target inside a repo whose canonical name is `canonical_name` can be addressed by the label `@@canonical_name//package:target` (note the double `@`). +The name by which a repository is always addressable. Within the context of a +workspace, each repository has a single canonical name. A target inside a repo +whose canonical name is `canonical_name` can be addressed by the label +`@@canonical_name//package:target` (note the double `@`). The main repository always has the empty string as the canonical name. ### Apparent repository name -The name by which a repository is addressable in the context of a certain other repo. This can be thought of as a repo's "nickname": The repo with the canonical name `michael` might have the apparent name `mike` in the context of the repo `alice`, but might have the apparent name `mickey` in the context of the repo `bob`. In this case, a target inside `michael` can be addressed by the label `@mike//package:target` in the context of `alice` (note the single `@`). +The name by which a repository is addressable in the context of a certain other +repo. This can be thought of as a repo's "nickname": The repo with the canonical +name `michael` might have the apparent name `mike` in the context of the repo +`alice`, but might have the apparent name `mickey` in the context of the repo +`bob`. In this case, a target inside `michael` can be addressed by the label +`@mike//package:target` in the context of `alice` (note the single `@`). -Conversely, this can be understood as a **repository mapping**: each repo maintains a mapping from "apparent repo name" to a "canonical repo name". +Conversely, this can be understood as a **repository mapping**: each repo +maintains a mapping from "apparent repo name" to a "canonical repo name". ### Repository rule -A schema for repository definitions that tells Bazel how to materialize a repository. For example, it could be "download a zip archive from a certain URL and extract it", or "fetch a certain Maven artifact and make it available as a `java_import` target", or simply "symlink a local directory". Every repo is **defined** by calling a repo rule with an appropriate number of arguments. +A schema for repository definitions that tells Bazel how to materialize a +repository. For example, it could be "download a zip archive from a certain URL +and extract it", or "fetch a certain Maven artifact and make it available as a +`java_import` target", or simply "symlink a local directory". Every repo is +**defined** by calling a repo rule with an appropriate number of arguments. -See [Repository rules](repo) for more information about how to write your own repository rules. +See [Repository rules](repo) for more information about how to write +your own repository rules. -The most common repo rules by far are [`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive from a URL and extracts it, and [`local_repository`](/reference/be/workspace#local_repository), which symlinks a local directory that is already a Bazel repository. +The most common repo rules by far are +[`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive +from a URL and extracts it, and +[`local_repository`](/reference/be/workspace#local_repository), which symlinks a +local directory that is already a Bazel repository. ### Fetch a repository -The action of making a repo available on local disk by running its associated repo rule. The repos defined in a workspace are not available on local disk before they are fetched. +The action of making a repo available on local disk by running its associated +repo rule. The repos defined in a workspace are not available on local disk +before they are fetched. -Normally, Bazel only fetches a repo when it needs something from the repo, and the repo hasn't already been fetched. If the repo has already been fetched before, Bazel only re-fetches it if its definition has changed. +Normally, Bazel only fetches a repo when it needs something from the repo, +and the repo hasn't already been fetched. If the repo has already been fetched +before, Bazel only re-fetches it if its definition has changed. -The `fetch` command can be used to initiate a pre-fetch for a repository, target, or all necessary repositories to perform any build. This capability enables offline builds using the `--nofetch` option. +The `fetch` command can be used to initiate a pre-fetch for a repository, +target, or all necessary repositories to perform any build. This capability +enables offline builds using the `--nofetch` option. -The `--fetch` option serves to manage network access. Its default value is true. However, when set to false (`--nofetch`), the command will utilize any cached version of the dependency, and if none exists, the command will result in failure. +The `--fetch` option serves to manage network access. Its default value is true. +However, when set to false (`--nofetch`), the command will utilize any cached +version of the dependency, and if none exists, the command will result in +failure. -See [fetch options](/reference/command-line-reference#fetch-options) for more information about controlling fetch. +See [fetch options](/reference/command-line-reference#fetch-options) for more +information about controlling fetch. ### Directory layout -After being fetched, the repo can be found in the subdirectory `external` in the [output base](/remote/output-directories), under its canonical name. +After being fetched, the repo can be found in the subdirectory `external` in the +[output base](/remote/output-directories), under its canonical name. -You can run the following command to see the contents of the repo with the canonical name `canonical_name`: +You can run the following command to see the contents of the repo with the +canonical name `canonical_name`: ```posix-terminal -ls $(bazel info output_base)/external/<var> canonical_name </var> +ls $(bazel info output_base)/external/{{ '' }} canonical_name {{ '' }} ``` ### REPO.bazel file -The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost boundary of the directory tree that constitutes a repo. It doesn't need to contain anything to serve as a repo boundary file; however, it can also be used to specify some common attributes for all build targets inside the repo. +The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost +boundary of the directory tree that constitutes a repo. It doesn't need to +contain anything to serve as a repo boundary file; however, it can also be used +to specify some common attributes for all build targets inside the repo. -The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no `load` statements are supported. The `repo()` function takes the same arguments as the [`package()` function](/reference/be/functions#package) in `BUILD` files; whereas `package()` specifies common attributes for all build targets inside the package, `repo()` analogously does so for all build targets inside the repo. +The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no +`load` statements are supported. The `repo()` function takes the same arguments as the [`package()` +function](/reference/be/functions#package) in `BUILD` files; whereas `package()` +specifies common attributes for all build targets inside the package, `repo()` +analogously does so for all build targets inside the repo. -For example, you can specify a common license for all targets in your repo by having the following `REPO.bazel` file: +For example, you can specify a common license for all targets in your repo by +having the following `REPO.bazel` file: ```python repo( @@ -146,9 +247,12 @@ repo( ## The legacy WORKSPACE system -In older Bazel versions (before 9.0), external dependencies were introduced by defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a similar syntax to `BUILD` files, employing repo rules instead of build rules. +In older Bazel versions (before 9.0), external dependencies were introduced by +defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a +similar syntax to `BUILD` files, employing repo rules instead of build rules. -The following snippet is an example to use the `http_archive` repo rule in the `WORKSPACE` file: +The following snippet is an example to use the `http_archive` repo rule in the +`WORKSPACE` file: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -159,26 +263,43 @@ http_archive( ) ``` -The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` system, by default, the canonical name of a repo is also its apparent name to all other repos. +The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` +system, by default, the canonical name of a repo is also its apparent name to +all other repos. -See the [full list](/rules/lib/globals/workspace) of functions available in `WORKSPACE` files. +See the [full list](/rules/lib/globals/workspace) of functions available in +`WORKSPACE` files. ### Shortcomings of the `WORKSPACE` system -In the years after the `WORKSPACE` system was introduced, users reported many pain points, including: - -- Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all transitive dependencies must be defined in the `WORKSPACE` file of the main repo, in addition to direct dependencies. - -- To work around this, projects have adopted the "deps.bzl" pattern, in which they define a macro which in turn defines multiple repos, and ask users to call this macro in their `WORKSPACE` files. - - - This has its own problems: macros cannot `load` other `.bzl` files, so these projects have to define their transitive dependencies in this "deps" macro, or work around this issue by having the user call multiple layered "deps" macros. - - Bazel evaluates the `WORKSPACE` file sequentially. Additionally, dependencies are specified using `http_archive` with URLs, without any version information. This means that there is no reliable way to perform version resolution in the case of diamond dependencies (`A` depends on `B` and `C`; `B` and `C` both depend on different versions of `D`). - -Due to the shortcomings of WORKSPACE, the new module-based system (codenamed "Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. Read the [Bzlmod migration guide](migration) on how to migrate to Bzlmod. +In the years after the `WORKSPACE` system was introduced, users reported many +pain points, including: + +* Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all + transitive dependencies must be defined in the `WORKSPACE` file of the main + repo, in addition to direct dependencies. +* To work around this, projects have adopted the "deps.bzl" pattern, in which + they define a macro which in turn defines multiple repos, and ask users to + call this macro in their `WORKSPACE` files. + * This has its own problems: macros cannot `load` other `.bzl` files, so + these projects have to define their transitive dependencies in this + "deps" macro, or work around this issue by having the user call multiple + layered "deps" macros. + * Bazel evaluates the `WORKSPACE` file sequentially. Additionally, + dependencies are specified using `http_archive` with URLs, without any + version information. This means that there is no reliable way to perform + version resolution in the case of diamond dependencies (`A` depends on + `B` and `C`; `B` and `C` both depend on different versions of `D`). + +Due to the shortcomings of WORKSPACE, the new module-based system (codenamed +"Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. +Read the [Bzlmod migration guide](migration) on how to migrate +to Bzlmod. ### External links on Bzlmod -- [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) -- [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) (original Bzlmod design doc) -- [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) -- [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) +* [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) +* [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) + (original Bzlmod design doc) +* [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) +* [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) diff --git a/external/repo.mdx b/external/repo.mdx index bbf37aee..b878f030 100644 --- a/external/repo.mdx +++ b/external/repo.mdx @@ -2,19 +2,39 @@ title: 'Repository Rules' --- -This page covers how to define repository rules and provides examples for more details. -An [external repository](/external/overview#repository) is a directory tree, containing source files usable in a Bazel build, which is generated on demand by running its corresponding **repo rule**. Repos can be defined in a multitude of ways, but ultimately, each repo is defined by invoking a repo rule, just as build targets are defined by invoking build rules. They can be used to depend on third-party libraries (such as Maven packaged libraries) but also to generate `BUILD` files specific to the host Bazel is running on. + +This page covers how to define repository rules and provides examples for more +details. + +An [external repository](/external/overview#repository) is a directory tree, +containing source files usable in a Bazel build, which is generated on demand by +running its corresponding **repo rule**. Repos can be defined in a multitude of +ways, but ultimately, each repo is defined by invoking a repo rule, just as +build targets are defined by invoking build rules. They can be used to depend on +third-party libraries (such as Maven packaged libraries) but also to generate +`BUILD` files specific to the host Bazel is running on. ## Repository rule definition -In a `.bzl` file, use the [repository\_rule](/rules/lib/globals/bzl#repository_rule) function to define a new repo rule and store it in a global variable. After a repo rule is defined, it can be invoked as a function to define repos. This invocation is usually performed from inside a [module extension](/external/extension) implementation function. +In a `.bzl` file, use the +[repository_rule](/rules/lib/globals/bzl#repository_rule) function to define a +new repo rule and store it in a global variable. After a repo rule is defined, +it can be invoked as a function to define repos. This invocation is usually +performed from inside a [module extension](/external/extension) implementation +function. -The two major components of a repo rule definition are its attribute schema and implementation function. The attribute schema determines the names and types of attributes passed to a repo rule invocation, and the implementation function is run when the repo needs to be fetched. +The two major components of a repo rule definition are its attribute schema and +implementation function. The attribute schema determines the names and types of +attributes passed to a repo rule invocation, and the implementation function is +run when the repo needs to be fetched. ## Attributes -Attributes are arguments passed to the repo rule invocation. The schema of attributes accepted by a repo rule is specified using the `attrs` argument when the repo rule is defined with a call to `repository_rule`. An example defining `url` and `sha256` attributes as strings: +Attributes are arguments passed to the repo rule invocation. The schema of +attributes accepted by a repo rule is specified using the `attrs` argument when +the repo rule is defined with a call to `repository_rule`. An example defining +`url` and `sha256` attributes as strings: ```python http_archive = repository_rule( @@ -26,7 +46,8 @@ http_archive = repository_rule( ) ``` -To access an attribute within the implementation function, use `repository_ctx.attr.<attribute_name>`: +To access an attribute within the implementation function, use +`repository_ctx.attr.`: ```python def _impl(repository_ctx): @@ -34,15 +55,29 @@ def _impl(repository_ctx): checksum = repository_ctx.attr.sha256 ``` -All `repository_rule`s have the implicitly defined attribute `name`. This is a string attribute that behaves somewhat magically: when specified as an input to a repo rule invocation, it takes an apparent repo name; but when read from the repo rule's implementation function using `repository_ctx.attr.name`, it returns the canonical repo name. +All `repository_rule`s have the implicitly defined attribute `name`. This is a +string attribute that behaves somewhat magically: when specified as an input to +a repo rule invocation, it takes an apparent repo name; but when read from the +repo rule's implementation function using `repository_ctx.attr.name`, it returns +the canonical repo name. ## Implementation function -Every repo rule requires an `implementation` function. It contains the actual logic of the rule and is executed strictly in the Loading Phase. +Every repo rule requires an `implementation` function. It contains the actual +logic of the rule and is executed strictly in the Loading Phase. -The function has exactly one input parameter, `repository_ctx`. The function returns either `None` to signify that the rule is reproducible given the specified parameters, or a dict with a set of parameters for that rule that would turn that rule into a reproducible one generating the same repo. For example, for a rule tracking a git repository that would mean returning a specific commit identifier instead of a floating branch that was originally specified. +The function has exactly one input parameter, `repository_ctx`. The function +returns either `None` to signify that the rule is reproducible given the +specified parameters, or a dict with a set of parameters for that rule that +would turn that rule into a reproducible one generating the same repo. For +example, for a rule tracking a git repository that would mean returning a +specific commit identifier instead of a floating branch that was originally +specified. -The input parameter `repository_ctx` can be used to access attribute values, and non-hermetic functions (finding a binary, executing a binary, creating a file in the repository or downloading a file from the Internet). See [the API docs](/rules/lib/builtins/repository_ctx) for more context. Example: +The input parameter `repository_ctx` can be used to access attribute values, and +non-hermetic functions (finding a binary, executing a binary, creating a file in +the repository or downloading a file from the Internet). See [the API +docs](/rules/lib/builtins/repository_ctx) for more context. Example: ```python def _impl(repository_ctx): @@ -55,38 +90,72 @@ local_repository = repository_rule( ## When is the implementation function executed? -The implementation function of a repo rule is executed when Bazel needs a target from that repository, for example when another target (in another repo) depends on it or if it is mentioned on the command line. The implementation function is then expected to create the repo in the file system. This is called "fetching" the repo. - -In contrast to regular targets, repos are not necessarily re-fetched when something changes that would cause the repo to be different. This is because there are things that Bazel either cannot detect changes to or it would cause too much overhead on every build (for example, things that are fetched from the network). Therefore, repos are re-fetched only if one of the following things changes: - -- The attributes passed to the repo rule invocation. - -- The Starlark code comprising the implementation of the repo rule. - -- The value of any environment variable passed to `repository_ctx`'s `getenv()` method or declared with the `environ` attribute of the [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of these environment variables can be hard-wired on the command line with the [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. - -- The existence, contents, and type of any paths being [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation function of the repo rule. - - - Certain other methods of `repository_ctx` with a `watch` parameter, such as `read()`, `execute()`, and `extract()`, can also cause paths to be watched. - - Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths to be watched in other ways. - -- When `bazel fetch --force` is executed. - -There are two parameters of `repository_rule` that control when the repositories are re-fetched: - -- If the `configure` flag is set, the repository is re-fetched on `bazel fetch --force --configure` (non-`configure` repositories are not re-fetched). -- If the `local` flag is set, in addition to the above cases, the repo is also re-fetched when the Bazel server restarts. +The implementation function of a repo rule is executed when Bazel needs a target +from that repository, for example when another target (in another repo) depends +on it or if it is mentioned on the command line. The implementation function is +then expected to create the repo in the file system. This is called "fetching" +the repo. + +In contrast to regular targets, repos are not necessarily re-fetched when +something changes that would cause the repo to be different. This is because +there are things that Bazel either cannot detect changes to or it would cause +too much overhead on every build (for example, things that are fetched from the +network). Therefore, repos are re-fetched only if one of the following things +changes: + +* The attributes passed to the repo rule invocation. +* The Starlark code comprising the implementation of the repo rule. +* The value of any environment variable passed to `repository_ctx`'s + `getenv()` method or declared with the `environ` attribute of the + [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of + these environment variables can be hard-wired on the command line with the + [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. +* The existence, contents, and type of any paths being + [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation + function of the repo rule. + * Certain other methods of `repository_ctx` with a `watch` parameter, such + as `read()`, `execute()`, and `extract()`, can also cause paths to be + watched. + * Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) + and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths + to be watched in other ways. +* When `bazel fetch --force` is executed. + +There are two parameters of `repository_rule` that control when the repositories +are re-fetched: + +* If the `configure` flag is set, the repository is re-fetched on `bazel + fetch --force --configure` (non-`configure` repositories are not + re-fetched). +* If the `local` flag is set, in addition to the above cases, the repo is also + re-fetched when the Bazel server restarts. ## Forcing refetch of external repos -Sometimes, an external repo can become outdated without any change to its definition or dependencies. For example, a repo fetching sources might follow a particular branch of a third-party repository, and new commits are available on that branch. In this case, you can ask bazel to refetch all external repos unconditionally by calling `bazel fetch --force --all`. +Sometimes, an external repo can become outdated without any change to its +definition or dependencies. For example, a repo fetching sources might follow a +particular branch of a third-party repository, and new commits are available on +that branch. In this case, you can ask bazel to refetch all external repos +unconditionally by calling `bazel fetch --force --all`. -Moreover, some repo rules inspect the local machine and might become outdated if the local machine was upgraded. Here you can ask Bazel to only refetch those external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) definition has the `configure` attribute set, use `bazel fetch --force --configure`. +Moreover, some repo rules inspect the local machine and might become outdated if +the local machine was upgraded. Here you can ask Bazel to only refetch those +external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) +definition has the `configure` attribute set, use `bazel fetch --force +--configure`. ## Examples -- [C++ auto-configured toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): it uses a repo rule to automatically create the C++ configuration files for Bazel by looking for the local C++ compiler, the environment and the flags the C++ compiler supports. +- [C++ auto-configured + toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): + it uses a repo rule to automatically create the C++ configuration files for + Bazel by looking for the local C++ compiler, the environment and the flags + the C++ compiler supports. -- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) uses several `repository_rule` to defines the list of dependencies needed to use the Go rules. +- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) + uses several `repository_rule` to defines the list of dependencies needed to + use the Go rules. -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) creates an external repository called `@maven` by default that generates build targets for every Maven artifact in the transitive dependency tree. +- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) + creates an external repository called `@maven` by default that generates + build targets for every Maven artifact in the transitive dependency tree. diff --git a/external/vendor.mdx b/external/vendor.mdx index a2e79e8d..653c1942 100644 --- a/external/vendor.mdx +++ b/external/vendor.mdx @@ -1,8 +1,13 @@ +keywords: product:Bazel,Bzlmod,vendor --- title: 'Vendor Mode' --- -Vendor mode is a feature that lets you create a local copy of external dependencies. This is useful for offline builds, or when you want to control the source of an external dependency. + + +Vendor mode is a feature that lets you create a local copy of +external dependencies. This is useful for offline builds, or when you want to +control the source of an external dependency. ## Enable vendor mode @@ -11,15 +16,19 @@ You can enable vendor mode by specifying `--vendor_dir` flag. For example, by adding it to your `.bazelrc` file: ```none -# Enable vendor mode with vendor directory under <workspace>/vendor_src +# Enable vendor mode with vendor directory under /vendor_src common --vendor_dir=vendor_src ``` -The vendor directory can be either a relative path to your workspace root or an absolute path. +The vendor directory can be either a relative path to your workspace root or an +absolute path. ## Vendor a specific external repository -You can use the `vendor` command with the `--repo` flag to specify which repo to vendor, it accepts both [canonical repo name](/external/overview#canonical-repo-name) and [apparent repo name](/external/overview#apparent-repo-name). +You can use the `vendor` command with the `--repo` flag to specify which repo +to vendor, it accepts both [canonical repo +name](/external/overview#canonical-repo-name) and [apparent repo +name](/external/overview#apparent-repo-name). For example, running: @@ -33,11 +42,13 @@ or bazel vendor --vendor_dir=vendor_src --repo=@@rules_cc+ ``` -will both get rules\_cc to be vendored under `<workspace root>/vendor_src/rules_cc+`. +will both get rules_cc to be vendored under +`/vendor_src/rules_cc+`. ## Vendor external dependencies for given targets -To vendor all external dependencies required for building given target patterns, you can run `bazel vendor <target patterns>`. +To vendor all external dependencies required for building given target patterns, +you can run `bazel vendor `. For example @@ -45,9 +56,12 @@ For example bazel vendor --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -will vendor all repos required for building the `//src/main:hello-world` target and all targets under `//src/test/...` with the current configuration. +will vendor all repos required for building the `//src/main:hello-world` target +and all targets under `//src/test/...` with the current configuration. -Under the hood, it's doing a `bazel build --nobuild` command to analyze the target patterns, therefore build flags could be applied to this command and affect the result. +Under the hood, it's doing a `bazel build --nobuild` command to analyze the +target patterns, therefore build flags could be applied to this command and +affect the result. ### Build the target offline @@ -57,15 +71,20 @@ With the external dependencies vendored, you can build the target offline by bazel build --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -The build should work in a clean build environment without network access and repository cache. +The build should work in a clean build environment without network access and +repository cache. -Therefore, you should be able to check in the vendored source and build the same targets offline on another machine. +Therefore, you should be able to check in the vendored source and build the same +targets offline on another machine. -Note: If you make changes to the targets to build, the external dependencies, the build configuration, or the Bazel version, you may need to re-vendor to make sure offline build still works. +Note: If you make changes to the targets to build, the external dependencies, +the build configuration, or the Bazel version, you may need to re-vendor to make +sure offline build still works. ## Vendor all external dependencies -To vendor all repos in your transitive external dependencies graph, you can run: +To vendor all repos in your transitive external dependencies graph, you can +run: ```none bazel vendor --vendor_dir=vendor_src @@ -73,20 +92,25 @@ bazel vendor --vendor_dir=vendor_src Note that vendoring all dependencies has a few **disadvantages**: -- Fetching all repos, including those introduced transitively, can be time-consuming. -- The vendor directory can become very large. -- Some repos may fail to fetch if they are not compatible with the current platform or environment. +- Fetching all repos, including those introduced transitively, can be time-consuming. +- The vendor directory can become very large. +- Some repos may fail to fetch if they are not compatible with the current platform or environment. Therefore, consider vendoring for specific targets first. ## Configure vendor mode with VENDOR.bazel -You can control how given repos are handled with the VENDOR.bazel file located under the vendor directory. +You can control how given repos are handled with the VENDOR.bazel file located +under the vendor directory. -There are two directives available, both accepting a list of [canonical repo names](/external/overview#canonical-repo-name) as arguments: +There are two directives available, both accepting a list of +[canonical repo names](/external/overview#canonical-repo-name) as arguments: - `ignore()`: to completely ignore a repository from vendor mode. -- `pin()`: to pin a repository to its current vendored source as if there is a `--override_repository` flag for this repo. Bazel will NOT update the vendored source for this repo while running the vendor command unless it's unpinned. The user can modify and maintain the vendored source for this repo manually. +- `pin()`: to pin a repository to its current vendored source as if there is a + `--override_repository` flag for this repo. Bazel will NOT update the vendored + source for this repo while running the vendor command unless it's unpinned. + The user can modify and maintain the vendored source for this repo manually. For example @@ -97,59 +121,93 @@ pin("@@bazel_skylib+") With this configuration -- Both repos will be excluded from subsequent vendor commands. -- Repo `bazel_skylib` will be overridden to the source located under the vendor directory. -- The user can safely modify the vendored source of `bazel_skylib`. -- To re-vendor `bazel_skylib`, the user has to disable the pin statement first. +- Both repos will be excluded from subsequent vendor commands. +- Repo `bazel_skylib` will be overridden to the source located under the + vendor directory. +- The user can safely modify the vendored source of `bazel_skylib`. +- To re-vendor `bazel_skylib`, the user has to disable the pin statement + first. -Note: Repository rules with [`local`](/rules/lib/globals/bzl#repository_rule.local) or [`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are always excluded from vendoring. +Note: Repository rules with +[`local`](/rules/lib/globals/bzl#repository_rule.local) or +[`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are +always excluded from vendoring. ## Understand how vendor mode works -Bazel fetches external dependencies of a project under `$(bazel info output_base)/external`. Vendoring external dependencies means moving out relevant files and directories to the given vendor directory and use the vendored source for later builds. +Bazel fetches external dependencies of a project under `$(bazel info +output_base)/external`. Vendoring external dependencies means moving out +relevant files and directories to the given vendor directory and use the +vendored source for later builds. The content being vendored includes: -- The repo directory -- The repo marker file +- The repo directory +- The repo marker file -During a build, if the vendored marker file is up-to-date or the repo is pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating a symlink to it under `$(bazel info output_base)/external` instead of actually running the repository rule. Otherwise, a warning is printed and Bazel will fallback to fetching the latest version of the repo. +During a build, if the vendored marker file is up-to-date or the repo is +pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating +a symlink to it under `$(bazel info output_base)/external` instead of actually +running the repository rule. Otherwise, a warning is printed and Bazel will +fallback to fetching the latest version of the repo. -Note: Bazel assumes the vendored source is not changed by users unless the repo is pinned in the VENDOR.bazel file. If a user does change the vendored source without pinning the repo, the changed vendored source will be used, but it will be overwritten if its existing marker file is outdated and the repo is vendored again. +Note: Bazel assumes the vendored source is not changed by users unless the repo +is pinned in the VENDOR.bazel file. If a user does change the vendored source +without pinning the repo, the changed vendored source will be used, but it will +be overwritten if its existing marker file is +outdated and the repo is vendored again. ### Vendor registry files -Bazel has to perform the Bazel module resolution in order to fetch external dependencies, which may require accessing registry files through internet. To achieve offline build, Bazel vendors all registry files fetched from network under the `<vendor_dir>/_registries` directory. +Bazel has to perform the Bazel module resolution in order to fetch external +dependencies, which may require accessing registry files through internet. To +achieve offline build, Bazel vendors all registry files fetched from +network under the `/_registries` directory. ### Vendor symlinks -External repositories may contain symlinks pointing to other files or directories. To make sure symlinks work correctly, Bazel uses the following strategy to rewrite symlinks in the vendored source: +External repositories may contain symlinks pointing to other files or +directories. To make sure symlinks work correctly, Bazel uses the following +strategy to rewrite symlinks in the vendored source: -- Create a symlink `<vendor_dir>/bazel-external` that points to `$(bazel info output_base)/external`. It is refreshed by every Bazel command automatically. -- For the vendored source, rewrite all symlinks that originally point to a path under `$(bazel info output_base)/external` to a relative path under `<vendor_dir>/bazel-external`. +- Create a symlink `/bazel-external` that points to `$(bazel info + output_base)/external`. It is refreshed by every Bazel command + automatically. +- For the vendored source, rewrite all symlinks that originally point to a + path under `$(bazel info output_base)/external` to a relative path under + `/bazel-external`. For example, if the original symlink is ```none -<vendor_dir>/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file +/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file ``` It will be rewritten to ```none -<vendor_dir>/repo_foo+/link => ../../bazel-external/repo_bar+/file +/repo_foo+/link => ../../bazel-external/repo_bar+/file ``` where ```none -<vendor_dir>/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed +/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed ``` -Since `<vendor_dir>/bazel-external` is generated by Bazel automatically, it's recommended to add it to `.gitignore` or equivalent to avoid checking it in. +Since `/bazel-external` is generated by Bazel automatically, it's +recommended to add it to `.gitignore` or equivalent to avoid checking it in. + +With this strategy, symlinks in the vendored source should work correctly even +after the vendored source is moved to another location or the bazel output base +is changed. -With this strategy, symlinks in the vendored source should work correctly even after the vendored source is moved to another location or the bazel output base is changed. +Note: symlinks that point to an absolute path outside of $(bazel info +output_base)/external are not rewritten. Therefore, it could still break +cross-machine compatibility. -Note: symlinks that point to an absolute path outside of $(bazel info output\_base)/external are not rewritten. Therefore, it could still break cross-machine compatibility. +Note: On Windows, vendoring symlinks only works with +[`--windows_enable_symlinks`][windows_enable_symlinks] +flag enabled. -Note: On Windows, vendoring symlinks only works with [`--windows_enable_symlinks`](/reference/command-line-reference#flag--windows_enable_symlinks) flag enabled. +[windows_enable_symlinks]: /reference/command-line-reference#flag--windows_enable_symlinks diff --git a/help.mdx b/help.mdx index af11fb80..754b4bc2 100644 --- a/help.mdx +++ b/help.mdx @@ -2,49 +2,53 @@ title: 'Getting Help' --- -This page lists Bazel resources beyond the documentation and covers how to get support from the Bazel team and community. + + +This page lists Bazel resources beyond the documentation and covers how to get +support from the Bazel team and community. ## Search existing material In addition to the documentation, you can find helpful information by searching: -- [Bazel user group](https://groups.google.com/g/bazel-discuss) -- [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) -- [Bazel blog](https://blog.bazel.build/) -- [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -- [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) +* [Bazel user group](https://groups.google.com/g/bazel-discuss) +* [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) +* [Bazel blog](https://blog.bazel.build/) +* [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +* [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) ## Watch videos There are recordings of Bazel talks at various conferences, such as: -- Bazel’s annual conference, BazelCon: - - - [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) - - [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) - - [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) - - [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) - - [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) - - [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) - - [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) - - [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +* Bazel’s annual conference, BazelCon: + * [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) + * [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) + * [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) + * [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) + * [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) + * [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) + * [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) + * [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +* Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) -- Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) ## Ask the Bazel community If there are no existing answers, you can ask the community by: -- Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) -- Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) -- Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -- Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) -- Consulting a [Bazel community expert](/community/experts) +* Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) +* Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) +* Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +* Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) +* Consulting a [Bazel community expert](/community/experts) ## Understand Bazel's support level -Please read the [release page](/release) to understand Bazel's release model and what level of support Bazel provides. +Please read the [release page](/release) to understand Bazel's release model and +what level of support Bazel provides. ## File a bug -If you encounter a bug or want to request a feature, file a [GitHub Issue](https://github.com/bazelbuild/bazel/issues). +If you encounter a bug or want to request a feature, file a [GitHub +Issue](https://github.com/bazelbuild/bazel/issues). diff --git a/install/bazelisk.mdx b/install/bazelisk.mdx index 4bc9f3e5..a3189cb7 100644 --- a/install/bazelisk.mdx +++ b/install/bazelisk.mdx @@ -2,39 +2,58 @@ title: 'Installing / Updating Bazel using Bazelisk' --- + + ## Installing Bazel -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically downloads and installs the appropriate version of Bazel. Use Bazelisk if you need to switch between different versions of Bazel depending on the current working directory, or to always keep Bazel updated to the latest release. +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the +recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically +downloads and installs the appropriate version of Bazel. Use Bazelisk if you +need to switch between different versions of Bazel depending on the current +working directory, or to always keep Bazel updated to the latest release. -For more details, see [the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). +For more details, see +[the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). ## Updating Bazel -Bazel has a [backward compatibility policy](/release/backward-compatibility) (see [guidance for rolling out incompatible changes](/contribute/breaking-changes) if you are the author of one). That page summarizes best practices on how to test and migrate your project with upcoming incompatible changes and how to provide feedback to the incompatible change authors. +Bazel has a [backward compatibility policy](/release/backward-compatibility) +(see [guidance for rolling out incompatible +changes](/contribute/breaking-changes) if you +are the author of one). That page summarizes best practices on how to test and +migrate your project with upcoming incompatible changes and how to provide +feedback to the incompatible change authors. ### Managing Bazel versions with Bazelisk -[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage Bazel versions. +[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage +Bazel versions. Bazelisk can: -- Auto-update Bazel to the latest LTS or rolling release. -- Build the project with a Bazel version specified in the .bazelversion file. Check in that file into your version control to ensure reproducibility of your builds. -- Help migrate your project for incompatible changes (see above) -- Easily try release candidates +* Auto-update Bazel to the latest LTS or rolling release. +* Build the project with a Bazel version specified in the .bazelversion + file. Check in that file into your version control to ensure reproducibility + of your builds. +* Help migrate your project for incompatible changes (see above) +* Easily try release candidates ### Recommended migration process -Within minor updates to any LTS release, any project can be prepared for the next release without breaking compatibility with the current release. However, there may be backward-incompatible changes between major LTS versions. +Within minor updates to any LTS release, any +project can be prepared for the next release without breaking +compatibility with the current release. However, there may be +backward-incompatible changes between major LTS versions. Follow this process to migrate from one major version to another: 1. Read the release notes to get advice on how to migrate to the next version. - -2. Major incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue: - - - Migration guidance is available in the associated GitHub issue. - - Tooling is available for some of incompatible changes migration. For example, [buildifier](https://github.com/bazelbuild/buildtools/releases). - - Report migration problems by commenting on the associated GitHub issue. - -After migration, you can continue to build your projects without worrying about backward-compatibility until the next major release. +1. Major incompatible changes should have an associated `--incompatible_*` flag + and a corresponding GitHub issue: + * Migration guidance is available in the associated GitHub issue. + * Tooling is available for some of incompatible changes migration. For + example, [buildifier](https://github.com/bazelbuild/buildtools/releases). + * Report migration problems by commenting on the associated GitHub issue. + +After migration, you can continue to build your projects without worrying about +backward-compatibility until the next major release. diff --git a/install/compile-source.mdx b/install/compile-source.mdx index 8ba134d8..3b87883d 100644 --- a/install/compile-source.mdx +++ b/install/compile-source.mdx @@ -2,67 +2,91 @@ title: 'Compiling Bazel from Source' --- -This page describes how to install Bazel from source and provides troubleshooting tips for common issues. + + +This page describes how to install Bazel from source and provides +troubleshooting tips for common issues. To build Bazel from source, you can do one of the following: -- Build it [using an existing Bazel binary](#build-bazel-using-bazel) +* Build it [using an existing Bazel binary](#build-bazel-using-bazel) -- Build it [without an existing Bazel binary](#bootstrap-bazel) which is known as *bootstrapping*. +* Build it [without an existing Bazel binary](#bootstrap-bazel) which is known + as _bootstrapping_. ## Build Bazel using Bazel ### Summary -1. Get the latest Bazel release from the [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with [Bazelisk](https://github.com/bazelbuild/bazelisk). +1. Get the latest Bazel release from the + [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with + [Bazelisk](https://github.com/bazelbuild/bazelisk). -2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) and extract somewhere. Alternatively you can git clone the source tree from [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) +2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) + and extract somewhere. + Alternatively you can git clone the source tree from https://github.com/bazelbuild/bazel -3. Install the same prerequisites as for bootstrapping (see [for Unix-like systems](#bootstrap-unix-prereq) or [for Windows](#bootstrap-windows-prereq)) +3. Install the same prerequisites as for bootstrapping (see + [for Unix-like systems](#bootstrap-unix-prereq) or + [for Windows](#bootstrap-windows-prereq)) -4. Build a development build of Bazel using Bazel: `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on Windows) +4. Build a development build of Bazel using Bazel: + `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on + Windows) -5. The resulting binary is at `bazel-bin/src/bazel-dev` (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you like and use immediately without further installation. +5. The resulting binary is at `bazel-bin/src/bazel-dev` + (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you + like and use immediately without further installation. Detailed instructions follow below. ### Step 1: Get the latest Bazel release -**Goal**: Install or download a release version of Bazel. Make sure you can run it by typing `bazel` in a terminal. +**Goal**: Install or download a release version of Bazel. Make sure you can run +it by typing `bazel` in a terminal. -**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing Bazel binary. You can install one from a package manager or download one from GitHub. See [Installing Bazel](/install). (Or you can [build from scratch (bootstrap)](#bootstrap-bazel).) +**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing +Bazel binary. You can install one from a package manager or download one from +GitHub. See [Installing Bazel](/install). (Or you can [build from +scratch (bootstrap)](#bootstrap-bazel).) **Troubleshooting**: -- If you cannot run Bazel by typing `bazel` in a terminal: +* If you cannot run Bazel by typing `bazel` in a terminal: - - Maybe your Bazel binary's directory is not on the PATH. + * Maybe your Bazel binary's directory is not on the PATH. - This is not a big problem. Instead of typing `bazel`, you will need to type the full path. + This is not a big problem. Instead of typing `bazel`, you will need to + type the full path. - - Maybe the Bazel binary itself is not called `bazel` (on Unixes) or `bazel.exe` (on Windows). + * Maybe the Bazel binary itself is not called `bazel` (on Unixes) or + `bazel.exe` (on Windows). - This is not a big problem. You can either rename the binary, or type the binary's name instead of `bazel`. + This is not a big problem. You can either rename the binary, or type the + binary's name instead of `bazel`. - - Maybe the binary is not executable (on Unixes). + * Maybe the binary is not executable (on Unixes). - You must make the binary executable by running `chmod +x /path/to/bazel`. + You must make the binary executable by running `chmod +x /path/to/bazel`. ### Step 2: Download Bazel's sources from GitHub -If you are familiar with Git, then just git clone [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) +If you are familiar with Git, then just git clone https://github.com/bazelbuild/bazel Otherwise: -1. Download the [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). +1. Download the + [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). -2. Extract the contents somewhere. +2. Extract the contents somewhere. - For example create a `bazel-src` directory under your home directory and extract there. + For example create a `bazel-src` directory under your home directory and + extract there. ### Step 3: Install prerequisites -Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ compiler, MSYS2 (if you are building on Windows), etc. +Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ +compiler, MSYS2 (if you are building on Windows), etc. ### Step 4a: Build Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -72,63 +96,66 @@ For instructions for Windows, see [Build Bazel on Windows](#build-bazel-on-windo **Instructions**: -1. Start a Bash terminal +1. Start a Bash terminal -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - ``` - cd ~/bazel-src - ``` + cd ~/bazel-src -3. Build Bazel from source: +3. Build Bazel from source: - ``` - bazel build //src:bazel-dev - ``` + bazel build //src:bazel-dev - Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` + to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel + version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). +4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). ### Step 4b: Build Bazel on Windows -For instructions for Unix-like systems, see [Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). +For instructions for Unix-like systems, see +[Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). -**Goal**: Run Bazel to build a custom Bazel binary (`bazel-bin\src\bazel-dev.exe`). +**Goal**: Run Bazel to build a custom Bazel binary +(`bazel-bin\src\bazel-dev.exe`). **Instructions**: -1. Start Command Prompt (Start Menu \> Run \> "cmd.exe") +1. Start Command Prompt (Start Menu > Run > "cmd.exe") -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - ``` - cd %USERPROFILE%\bazel-src - ``` + cd %USERPROFILE%\bazel-src -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev.exe + bazel build //src:bazel-dev.exe - Alternatively you can run `bazel build //src:bazel.exe --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel.exe + --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel + version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin\src\bazel-dev.exe` (or `bazel-bin\src\bazel.exe`). +4. The output will be at `bazel-bin\src\bazel-dev.exe` (or + `bazel-bin\src\bazel.exe`). ### Step 5: Install the built binary Actually, there's nothing to install. -The output of the previous step is a self-contained Bazel binary. You can copy it to any directory and use immediately. (It's useful if that directory is on your PATH so that you can run "bazel" everywhere.) +The output of the previous step is a self-contained Bazel binary. You can copy +it to any directory and use immediately. (It's useful if that directory is on +your PATH so that you can run "bazel" everywhere.) -*** +--- ## Build Bazel from scratch (bootstrapping) @@ -138,16 +165,24 @@ You can also build Bazel from scratch, without using an existing Bazel binary. (This step is the same for all platforms.) -1. Download `bazel-<version>-dist.zip` from [GitHub](https://github.com/bazelbuild/bazel/releases), for example `bazel-0.28.1-dist.zip`. +1. Download `bazel--dist.zip` from + [GitHub](https://github.com/bazelbuild/bazel/releases), for example + `bazel-0.28.1-dist.zip`. - **Attention**: + **Attention**: - - There is a **single, architecture-independent** distribution archive. There are no architecture-specific or OS-specific distribution archives. - - These sources are **not the same as the GitHub source tree**. You have to use the distribution archive to bootstrap Bazel. You cannot use a source tree cloned from GitHub. (The distribution archive contains generated source files that are required for bootstrapping and are not part of the normal Git source tree.) + - There is a **single, architecture-independent** distribution archive. + There are no architecture-specific or OS-specific distribution archives. + - These sources are **not the same as the GitHub source tree**. You + have to use the distribution archive to bootstrap Bazel. You cannot + use a source tree cloned from GitHub. (The distribution archive contains + generated source files that are required for bootstrapping and are not part + of the normal Git source tree.) -2. Unpack the distribution archive somewhere on disk. +2. Unpack the distribution archive somewhere on disk. - You should verify the signature made by Bazel's [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. + You should verify the signature made by Bazel's + [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. ### Step 2a: Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -155,17 +190,18 @@ For instructions for Windows, see [Bootstrap Bazel on Windows](#bootstrap-window #### 2.1. Install the prerequisites -- **Bash** +* **Bash** -- **zip, unzip** +* **zip, unzip** -- **C++ build toolchain** +* **C++ build toolchain** -- **JDK.** Version 21 is required. +* **JDK.** Version 21 is required. -- **Python**. Version 3 is required. +* **Python**. Version 3 is required. -For example on Ubuntu Linux you can install these requirements using the following command: +For example on Ubuntu Linux you can install these requirements using the +following command: ```sh sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip @@ -173,76 +209,90 @@ sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip #### 2.2. Bootstrap Bazel on Unix -1. Open a shell or Terminal window. +1. Open a shell or Terminal window. -2. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. +3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. -The compiled output is placed into `output/bazel`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH` (such as `/usr/local/bin` on Linux). +The compiled output is placed into `output/bazel`. This is a self-contained +Bazel binary, without an embedded JDK. You can copy it anywhere or use it +in-place. For convenience, copy this binary to a directory that's on your +`PATH` (such as `/usr/local/bin` on Linux). -To build the `bazel` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. +To build the `bazel` binary in a reproducible way, also set +[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) +in the "Run the compilation script" step. ### Step 2b: Bootstrap Bazel on Windows -For instructions for Unix-like systems, see [Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). +For instructions for Unix-like systems, see +[Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). #### 2.1. Install the prerequisites -- [MSYS2 shell](https://msys2.github.io/) +* [MSYS2 shell](https://msys2.github.io/) -- **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: +* **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: - ``` - pacman -S zip unzip patch - ``` + ``` + pacman -S zip unzip patch + ``` -- **The Visual C++ compiler.** Install the Visual C++ compiler either as part of Visual Studio 2015 or newer, or by installing the latest [Build Tools for Visual Studio 2017](https://aka.ms/BuildTools). +* **The Visual C++ compiler.** Install the Visual C++ compiler either as part + of Visual Studio 2015 or newer, or by installing the latest [Build Tools + for Visual Studio 2017](https://aka.ms/BuildTools). -- **JDK.** Version 21 is required. +* **JDK.** Version 21 is required. -- **Python**. Versions 2 and 3 are supported, installing one of them is enough. You need the Windows-native version (downloadable from [https://www.python.org](https://www.python.org)). Versions installed via pacman in MSYS2 will not work. +* **Python**. Versions 2 and 3 are supported, installing one of them is + enough. You need the Windows-native version (downloadable from + [https://www.python.org](https://www.python.org)). Versions installed via + pacman in MSYS2 will not work. #### 2.2. Bootstrap Bazel on Windows -1. Open the MSYS2 shell. - -2. Set the following environment variables: - - - Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the path to the Visual Studio directory (BAZEL\_V**S**) or to the Visual C++ directory (BAZEL\_V**C**). Setting one of them is enough. - - - `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the examples below. - - Do not set this to `C:\Windows\System32\bash.exe`. (You have that file if you installed Windows Subsystem for Linux.) Bazel does not support this version of `bash.exe`. +1. Open the MSYS2 shell. - - `PATH`: Add the Python directory. +2. Set the following environment variables: + * Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the + path to the Visual Studio directory (BAZEL\_VS) or to the Visual + C++ directory (BAZEL\_VC). Setting one of them is enough. + * `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the + examples below. - - `JAVA_HOME`: Set to the JDK directory. + Do not set this to `C:\Windows\System32\bash.exe`. (You have that file + if you installed Windows Subsystem for Linux.) Bazel does not support + this version of `bash.exe`. + * `PATH`: Add the Python directory. + * `JAVA_HOME`: Set to the JDK directory. - **Example** (using BAZEL\_V**S**): + **Example** (using BAZEL\_VS): - ``` - export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" - ``` + export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" - or (using BAZEL\_V**C**): + or (using BAZEL\_VC): - ``` - export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" - ``` + export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" -3. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` +4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` -The compiled output is placed into `output/bazel.exe`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH`. +The compiled output is placed into `output/bazel.exe`. This is a self-contained +Bazel binary, without an embedded JDK. You can copy it anywhere or use it +in-place. For convenience, copy this binary to a directory that's on +your `PATH`. -To build the `bazel.exe` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. +To build the `bazel.exe` binary in a reproducible way, also set +[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) +in the "Run the compilation script" step. -You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the Command Prompt (`cmd.exe`) or PowerShell. +You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the +Command Prompt (`cmd.exe`) or PowerShell. diff --git a/install/completion.mdx b/install/completion.mdx index 309ce3f8..1d1d1b76 100644 --- a/install/completion.mdx +++ b/install/completion.mdx @@ -2,7 +2,11 @@ title: 'Command-Line Completion' --- -You can enable command-line completion (also known as tab-completion) in Bash and Zsh. This lets you tab-complete command names, flags names and flag values, and target names. + + +You can enable command-line completion (also known as tab-completion) in Bash +and Zsh. This lets you tab-complete command names, flags names and flag values, +and target names. ## Bash @@ -10,51 +14,55 @@ Bazel comes with a Bash completion script. If you installed Bazel: -- From the APT repository, then you're done -- the Bash completion script is already installed in `/etc/bash_completion.d`. - -- From Homebrew, then you're done -- the Bash completion script is already installed in `$(brew --prefix)/etc/bash_completion.d`. - -- From the installer downloaded from GitHub, then: - - 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. - - 2. Do one of the following: - - - Either copy this file to your completion directory (if you have one). - - Example: on Ubuntu this is the `/etc/bash_completion.d` directory. - - - Or source the completion file from Bash's RC file. +* From the APT repository, then you're done -- the Bash completion script is + already installed in `/etc/bash_completion.d`. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: +* From Homebrew, then you're done -- the Bash completion script is + already installed in `$(brew --prefix)/etc/bash_completion.d`. - ``` - source /path/to/bazel-complete.bash - ``` +* From the installer downloaded from GitHub, then: + 1. Locate the absolute path of the completion file. The installer copied it + to the `bin` directory. -- Via [bootstrapping](/install/compile-source), then: + Example: if you ran the installer with `--user`, this will be + `$HOME/.bazel/bin`. If you ran the installer as root, this will be + `/usr/local/lib/bazel/bin`. + 2. Do one of the following: + * Either copy this file to your completion directory (if you have + one). - 1. Emit the completion script into a file: + Example: on Ubuntu this is the `/etc/bash_completion.d` directory. + * Or source the completion file from Bash's RC file. - ``` - bazel help completion bash > bazel-complete.bash - ``` + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) + or `~/.bash_profile` (on macOS), using the path to your completion + file's absolute path: - 2. Do one of the following: + ``` + source /path/to/bazel-complete.bash + ``` - - Copy this file to your completion directory, if you have one. +* Via [bootstrapping](/install/compile-source), then: + 1. Emit the completion script into a file: - Example: on Ubuntu this is the `/etc/bash_completion.d` directory + ``` + bazel help completion bash > bazel-complete.bash + ``` + 2. Do one of the following: + * Copy this file to your completion directory, if you have + one. - - Copy it somewhere on your local disk, such as to `$HOME`, and source the completion file from Bash's RC file. + Example: on Ubuntu this is the `/etc/bash_completion.d` directory + * Copy it somewhere on your local disk, such as to `$HOME`, and + source the completion file from Bash's RC file. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) + or `~/.bash_profile` (on macOS), using the path to your completion + file's absolute path: - ``` - source /path/to/bazel-complete.bash - ``` + ``` + source /path/to/bazel-complete.bash + ``` ## Zsh @@ -62,48 +70,57 @@ Bazel comes with a Zsh completion script. If you installed Bazel: -- From the APT repository, then you're done -- the Zsh completion script is already installed in `/usr/share/zsh/vendor-completions`. - - > If you have a heavily customized `.zshrc` and the autocomplete does not function, try one of the following solutions: - > - > Add the following to your `.zshrc`: - > - > ``` - > zstyle :compinstall filename '/home/tradical/.zshrc' - > - > autoload -Uz compinit - > compinit - > ``` - > - > or - > - > Follow the instructions [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) - > - > If you are using `oh-my-zsh`, you may want to install and enable the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the solutions described above. - -- From Homebrew, then you're done -- the Zsh completion script is already installed in `$(brew --prefix)/share/zsh/site-functions`. - -- From the installer downloaded from GitHub, then: - - 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. - - 2. Add this script to a directory on your `$fpath`: - - ``` - fpath[1,0]=~/.zsh/completion/ - mkdir -p ~/.zsh/completion/ - cp /path/from/above/step/_bazel ~/.zsh/completion - ``` - - You may have to call `rm -f ~/.zcompdump; compinit` the first time to make it work. - - 3. Optionally, add the following to your .zshrc. - - ``` - # This way the completion script does not have to parse Bazel's options - # repeatedly. The directory in cache-path must be created manually. - zstyle ':completion:*' use-cache on - zstyle ':completion:*' cache-path ~/.zsh/cache - ``` +* From the APT repository, then you're done -- the Zsh completion script is + already installed in `/usr/share/zsh/vendor-completions`. + + > If you have a heavily customized `.zshrc` and the autocomplete + > does not function, try one of the following solutions: + > + > Add the following to your `.zshrc`: + > + > ``` + > zstyle :compinstall filename '/home/tradical/.zshrc' + > + > autoload -Uz compinit + > compinit + > ``` + > + > or + > + > Follow the instructions + > [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) + > + > If you are using `oh-my-zsh`, you may want to install and enable + > the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the + > solutions described above. + +* From Homebrew, then you're done -- the Zsh completion script is + already installed in `$(brew --prefix)/share/zsh/site-functions`. + +* From the installer downloaded from GitHub, then: + 1. Locate the absolute path of the completion file. The installer copied it + to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be + `$HOME/.bazel/bin`. If you ran the installer as root, this will be + `/usr/local/lib/bazel/bin`. + + 2. Add this script to a directory on your `$fpath`: + + ``` + fpath[1,0]=~/.zsh/completion/ + mkdir -p ~/.zsh/completion/ + cp /path/from/above/step/_bazel ~/.zsh/completion + ``` + + You may have to call `rm -f ~/.zcompdump; compinit` + the first time to make it work. + + 3. Optionally, add the following to your .zshrc. + + ``` + # This way the completion script does not have to parse Bazel's options + # repeatedly. The directory in cache-path must be created manually. + zstyle ':completion:*' use-cache on + zstyle ':completion:*' cache-path ~/.zsh/cache + ``` diff --git a/install/docker-container.mdx b/install/docker-container.mdx index 2e97dc2b..b1efb76c 100644 --- a/install/docker-container.mdx +++ b/install/docker-container.mdx @@ -2,11 +2,19 @@ title: 'Getting Started with Bazel Docker Container' --- -This page provides details on the contents of the Bazel container, how to build the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel inside the Bazel container, and how to build this project directly from the host machine using the Bazel container with directory mounting. + + +This page provides details on the contents of the Bazel container, how to build +the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel +inside the Bazel container, and how to build this project directly +from the host machine using the Bazel container with directory mounting. ## Build Abseil project from your host machine with directory mounting -The instructions in this section allow you to build using the Bazel container with the sources checked out in your host environment. A container is started up for each build command you execute. Build results are cached in your host environment so they can be reused across builds. +The instructions in this section allow you to build using the Bazel container +with the sources checked out in your host environment. A container is started up +for each build command you execute. Build results are cached in your host +environment so they can be reused across builds. Clone the project to a directory in your host machine. @@ -20,7 +28,8 @@ Create a folder that will have cached results to be shared across builds. mkdir -p /tmp/build_output/ ``` -Use the Bazel container to build the project and make the build outputs available in the output folder in your host machine. +Use the Bazel container to build the project and make the build +outputs available in the output folder in your host machine. ```posix-terminal docker run \ @@ -34,7 +43,9 @@ docker run \ build //absl/... ``` -Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=asan|tsan|msan` build +flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or +MemorySanitizer (msan) accordingly. ```posix-terminal docker run \ @@ -50,7 +61,10 @@ docker run \ ## Build Abseil project from inside the container -The instructions in this section allow you to build using the Bazel container with the sources inside the container. By starting a container at the beginning of your development workflow and doing changes in the worskpace within the container, build results will be cached. +The instructions in this section allow you to build using the Bazel container +with the sources inside the container. By starting a container at the beginning +of your development workflow and doing changes in the worskpace within the +container, build results will be cached. Start a shell in the Bazel container: @@ -72,7 +86,9 @@ Do a regular build. ubuntu@5a99103747c6:~/abseil-cpp$ bazel build //absl/... ``` -Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=asan|tsan|msan` +build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or +MemorySanitizer (msan) accordingly. ```posix-terminal ubuntu@5a99103747c6:~/abseil-cpp$ bazel build --config={asan | tsan | msan} -- //absl/... -//absl/types:variant_test diff --git a/install/ide.mdx b/install/ide.mdx index 5c6252da..aa6210b3 100644 --- a/install/ide.mdx +++ b/install/ide.mdx @@ -2,39 +2,56 @@ title: 'Integrating Bazel with IDEs' --- -This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android Studio, and CLion (or build your own IDE plugin). It also includes links to installation and plugin details. -IDEs integrate with Bazel in a variety of ways, from features that allow Bazel executions from within the IDE, to awareness of Bazel structures such as syntax highlighting of the `BUILD` files. -If you are interested in developing an editor or IDE plugin for Bazel, please join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). +This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android +Studio, and CLion (or build your own IDE plugin). It also includes links to +installation and plugin details. + +IDEs integrate with Bazel in a variety of ways, from features that allow Bazel +executions from within the IDE, to awareness of Bazel structures such as syntax +highlighting of the `BUILD` files. + +If you are interested in developing an editor or IDE plugin for Bazel, please +join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start +a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). ## IDEs and editors ### IntelliJ, Android Studio, and CLion -[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). +[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and +CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). This is the open source version of the plugin used internally at Google. Features: -- Interop with language-specific plugins. Supported languages include Java, Scala, and Python. -- Import `BUILD` files into the IDE with semantic awareness of Bazel targets. -- Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and `.bzl`files -- Build, test, and execute binaries directly from the IDE -- Create configurations for debugging and running binaries. +* Interop with language-specific plugins. Supported languages include Java, + Scala, and Python. +* Import `BUILD` files into the IDE with semantic awareness of Bazel targets. +* Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and + `.bzl`files +* Build, test, and execute binaries directly from the IDE +* Create configurations for debugging and running binaries. To install, go to the IDE's plugin browser and search for `Bazel`. -To manually install older versions, download the zip files from JetBrains' Plugin Repository and install the zip file from the IDE's plugin browser: +To manually install older versions, download the zip files from JetBrains' +Plugin Repository and install the zip file from the IDE's plugin browser: -- [Android Studio plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) -- [IntelliJ plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) -- [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) +* [Android Studio + plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) +* [IntelliJ + plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) +* [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) ### Xcode -[rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), [Tulsi](https://tulsi.bazel.build), and [XCHammer](https://github.com/pinterest/xchammer) generate Xcode projects from Bazel `BUILD` files. +[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), +[Tulsi](https://tulsi.bazel.build), and +[XCHammer](https://github.com/pinterest/xchammer) generate Xcode +projects from Bazel `BUILD` files. ### Visual Studio Code @@ -42,16 +59,21 @@ Official plugin for VS Code. Features: -- Bazel Build Targets tree -- Starlark debugger for `.bzl` files during a build (set breakpoints, step through code, inspect variables, and so on) +* Bazel Build Targets tree +* Starlark debugger for `.bzl` files during a build (set breakpoints, step + through code, inspect variables, and so on) -Find [the plugin on the Visual Studio marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). +Find [the plugin on the Visual Studio +marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) +or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). +The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Atom -Find the [`language-bazel` package](https://atom.io/packages/language-bazel) on the Atom package manager. +Find the [`language-bazel` package](https://atom.io/packages/language-bazel) +on the Atom package manager. See also: [Autocomplete for Source Code](#autocomplete-for-source-code) @@ -63,23 +85,31 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Emacs -See [`bazelbuild/bazel-emacs-mode` on GitHub](https://github.com/bazelbuild/emacs-bazel-mode) +See [`bazelbuild/bazel-emacs-mode` on +GitHub](https://github.com/bazelbuild/emacs-bazel-mode) See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Visual Studio -[Lavender](https://github.com/tmandry/lavender) is an experimental project for generating Visual Studio projects that use Bazel for building. +[Lavender](https://github.com/tmandry/lavender) is an experimental project for +generating Visual Studio projects that use Bazel for building. ### Eclipse -[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) is a set of plugins for importing Bazel packages into an Eclipse workspace as Eclipse projects. +[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) +is a set of plugins for importing Bazel packages into an Eclipse workspace as +Eclipse projects. ## Autocomplete for Source Code ### C Language Family (C++, C, Objective-C, Objective-C++, and CUDA) -[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and consumes the Protobuf output of Bazel to extract the compile commands. +[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) +run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. +The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to +provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and +consumes the Protobuf output of Bazel to extract the compile commands. [`hedronvision/bazel-compile-commands-extractor`](https://github.com/hedronvision/bazel-compile-commands-extractor) enables autocomplete, smart navigation, quick fixes, and more in a wide variety of extensible editors, including VSCode, Vim, Emacs, Atom, and Sublime. It lets language servers, like clangd and ccls, and other types of tooling, draw upon Bazel's understanding of how `cc` and `objc` code will be compiled, including how it configures cross-compilation for other platforms. @@ -89,8 +119,11 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ## Automatically run build and test on file change -[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a tool for building Bazel targets when source files change. +[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a +tool for building Bazel targets when source files change. ## Building your own IDE plugin -Read the [**IDE support** blog post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about the Bazel APIs to use when building an IDE plugin. +Read the [**IDE support** blog +post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about +the Bazel APIs to use when building an IDE plugin. diff --git a/install/index.mdx b/install/index.mdx index 898cd417..29b46972 100644 --- a/install/index.mdx +++ b/install/index.mdx @@ -2,7 +2,10 @@ title: 'Installing Bazel' --- -This page describes the various platforms supported by Bazel and links to the packages for more details. + + +This page describes the various platforms supported by Bazel and links +to the packages for more details. [Bazelisk](/install/bazelisk) is the recommended way to install Bazel on [Ubuntu Linux](/install/ubuntu), [macOS](/install/os-x), and [Windows](/install/windows). @@ -10,21 +13,24 @@ You can find available Bazel releases on our [release page](/release). ## Community-supported packages -Bazel community members maintain these packages. The Bazel team doesn't officially support them. Contact the package maintainers for support. +Bazel community members maintain these packages. The Bazel team doesn't +officially support them. Contact the package maintainers for support. -- [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*\&branch=edge\&repo=\&arch=\&origin=\&flagged=\&maintainer=) -- [Arch Linux](https://archlinux.org/packages/extra/x86_64/bazel/) -- [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) -- [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) -- [FreeBSD](https://www.freshports.org/devel/bazel) -- [Homebrew](https://formulae.brew.sh/formula/bazel) -- [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) -- [openSUSE](/install/suse) -- [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) -- [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) +* [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*&branch=edge&repo=&arch=&origin=&flagged=&maintainer=) +* [Arch Linux][arch] +* [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) +* [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) +* [FreeBSD](https://www.freshports.org/devel/bazel) +* [Homebrew](https://formulae.brew.sh/formula/bazel) +* [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) +* [openSUSE](/install/suse) +* [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) +* [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) ## Community-supported architectures -- [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) +* [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) For other platforms, you can try to [compile from source](/install/compile-source). + +[arch]: https://archlinux.org/packages/extra/x86_64/bazel/ diff --git a/install/os-x.mdx b/install/os-x.mdx index 980d2d81..b8d5bca2 100644 --- a/install/os-x.mdx +++ b/install/os-x.mdx @@ -2,21 +2,23 @@ title: 'Installing Bazel on macOS' --- + + This page describes how to install Bazel on macOS and set up your environment. You can install Bazel on macOS using one of the following methods: -- *Recommended*: [Use Bazelisk](/install/bazelisk) -- [Use Homebrew](#install-on-mac-os-x-homebrew) -- [Use the binary installer](#install-with-installer-mac-os-x) -- [Compile Bazel from source](/install/compile-source) +* *Recommended*: [Use Bazelisk](/install/bazelisk) +* [Use Homebrew](#install-on-mac-os-x-homebrew) +* [Use the binary installer](#install-with-installer-mac-os-x) +* [Compile Bazel from source](/install/compile-source) Bazel comes with two completion scripts. After installing Bazel, you can: -- Access the [bash completion script](/install/completion#bash) -- Install the [zsh completion script](/install/completion#zsh) +* Access the [bash completion script](/install/completion#bash) +* Install the [zsh completion script](/install/completion#zsh) -## Installing using Homebrew +

Installing using Homebrew

### Step 1: Install Homebrew on macOS @@ -34,37 +36,46 @@ Install the Bazel package via Homebrew as follows: brew install bazel ``` -All set! You can confirm Bazel is installed successfully by running the following command: +All set! You can confirm Bazel is installed successfully by running the +following command: ```posix-terminal bazel --version ``` -Once installed, you can upgrade to a newer version of Bazel using the following command: +Once installed, you can upgrade to a newer version of Bazel using the +following command: ```posix-terminal brew upgrade bazel ``` -## Installing using the binary installer +

Installing using the binary installer

-The binary installers are on Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). +The binary installers are on Bazel's +[GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary. Some additional libraries must also be installed for Bazel to work. +The installer contains the Bazel binary. Some additional libraries +must also be installed for Bazel to work. ### Step 1: Install Xcode command line tools -If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode command line tools package by using `xcode-select`: +If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode +command line tools package by using `xcode-select`: ```posix-terminal xcode-select --install ``` -Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS SDK 8.1 installed on your system. +Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS +SDK 8.1 installed on your system. -Download Xcode from the [App Store](https://apps.apple.com/us/app/xcode/id497799835) or the [Apple Developer site](https://developer.apple.com/download/more/?=xcode). +Download Xcode from the +[App Store](https://apps.apple.com/us/app/xcode/id497799835) or the +[Apple Developer site](https://developer.apple.com/download/more/?=xcode). -Once Xcode is installed, accept the license agreement for all users with the following command: +Once Xcode is installed, accept the license agreement for all users with the +following command: ```posix-terminal sudo xcodebuild -license accept @@ -72,46 +83,59 @@ sudo xcodebuild -license accept ### Step 2: Download the Bazel installer -Next, download the Bazel binary installer named `bazel-<version>-installer-darwin-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named +`bazel--installer-darwin-x86_64.sh` from the +[Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). -**On macOS Catalina or newer (macOS \>= 11)**, due to Apple's new app signing requirements, you need to download the installer from the terminal using `curl`, replacing the version variable with the Bazel version you want to download: +**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, +you need to download the installer from the terminal using `curl`, replacing +the version variable with the Bazel version you want to download: ```posix-terminal export BAZEL_VERSION=5.2.0 -curl -fLO "https://github.com/bazelbuild/bazel/releases/download/<var>$BAZEL_VERSION</var>/bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" +curl -fLO "https://github.com/bazelbuild/bazel/releases/download/{{ '' }}$BAZEL_VERSION{{ '' }}/bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" ``` -This is a temporary workaround until the macOS release flow supports signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). +This is a temporary workaround until the macOS release flow supports +signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). ### Step 3: Run the installer Run the Bazel installer as follows: ```posix-terminal -chmod +x "bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" +chmod +x "bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" -./bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh --user +./bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and +sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see +additional installation options. -If you are **on macOS Catalina or newer (macOS \>= 11)** and get an error that ***“bazel-real” cannot be opened because the developer cannot be verified***, you need to re-download the installer from the terminal using `curl` as a workaround; see Step 2 above. +If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that _**“bazel-real” cannot be +opened because the developer cannot be verified**_, you need to re-download +the installer from the terminal using `curl` as a workaround; see Step 2 above. ### Step 4: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `<var>HOME</var>/bin` directory. It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel +executable is installed in your `HOME/bin` directory. +It's a good idea to add this directory to your default paths, as follows: ```posix-terminal -export PATH="<var>PATH</var>:<var>HOME</var>/bin" +export PATH="{{ '' }}PATH{{ '' }}:{{ '' }}HOME{{ '' }}/bin" ``` -You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` file. +You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` +file. -All set! You can confirm Bazel is installed successfully by running the following command: +All set! You can confirm Bazel is installed successfully by running the +following command: ```posix-terminal bazel --version ``` - To update to a newer release of Bazel, download and install the desired version. + diff --git a/install/suse.mdx b/install/suse.mdx index 73debc99..741b108f 100644 --- a/install/suse.mdx +++ b/install/suse.mdx @@ -2,17 +2,23 @@ title: 'Installing Bazel on openSUSE Tumbleweed & Leap' --- + + This page describes how to install Bazel on openSUSE Tumbleweed and Leap. -`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). +`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues +using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). -Packages are provided for openSUSE Tumbleweed and Leap. You can find all available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93\&baseproject=ALL\&q=bazel). +Packages are provided for openSUSE Tumbleweed and Leap. You can find all +available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93&baseproject=ALL&q=bazel). The commands below must be run either via `sudo` or while logged in as `root`. ## Installing Bazel on openSUSE -Run the following commands to install the package. If you need a specific version, you can install it via the specific `bazelXXX` package, otherwise, just `bazel` is enough: +Run the following commands to install the package. If you need a specific +version, you can install it via the specific `bazelXXX` package, otherwise, +just `bazel` is enough: To install the latest version of Bazel, run: @@ -20,7 +26,9 @@ To install the latest version of Bazel, run: zypper install bazel ``` -You can also install a specific version of Bazel by specifying the package version with `bazel<var>version</var>`. For example, to install Bazel 4.2, run: +You can also install a specific version of Bazel by specifying the package +version with `bazelversion`. For example, to install +Bazel 4.2, run: ```posix-terminal zypper install bazel4.2 diff --git a/install/ubuntu.mdx b/install/ubuntu.mdx index b498f42f..73d42751 100644 --- a/install/ubuntu.mdx +++ b/install/ubuntu.mdx @@ -2,30 +2,37 @@ title: 'Installing Bazel on Ubuntu' --- -This page describes the options for installing Bazel on Ubuntu. It also provides links to the Bazel completion scripts and the binary installer, if needed as a backup option (for example, if you don't have admin access). + + +This page describes the options for installing Bazel on Ubuntu. +It also provides links to the Bazel completion scripts and the binary installer, +if needed as a backup option (for example, if you don't have admin access). Supported Ubuntu Linux platforms: -- 22.04 (LTS) -- 20.04 (LTS) -- 18.04 (LTS) +* 22.04 (LTS) +* 20.04 (LTS) +* 18.04 (LTS) -Bazel should be compatible with other Ubuntu releases and Debian "stretch" and above, but is untested and not guaranteed to work. +Bazel should be compatible with other Ubuntu releases and Debian +"stretch" and above, but is untested and not guaranteed to work. Install Bazel on Ubuntu using one of the following methods: -- *Recommended*: [Use Bazelisk](/install/bazelisk) -- [Use our custom APT repository](#install-on-ubuntu) -- [Use the binary installer](#binary-installer) -- [Use the Bazel Docker container](#docker-container) -- [Compile Bazel from source](/install/compile-source) +* *Recommended*: [Use Bazelisk](/install/bazelisk) +* [Use our custom APT repository](#install-on-ubuntu) +* [Use the binary installer](#binary-installer) +* [Use the Bazel Docker container](#docker-container) +* [Compile Bazel from source](/install/compile-source) -**Note:** For Arm-based systems, the APT repository does not contain an `arm64` release, and there is no binary installer available. Either use Bazelisk or compile from source. +**Note:** For Arm-based systems, the APT repository does not contain an `arm64` +release, and there is no binary installer available. Either use Bazelisk or +compile from source. Bazel comes with two completion scripts. After installing Bazel, you can: -- Access the [bash completion script](/install/completion#bash) -- Install the [zsh completion script](/install/completion#zsh) +* Access the [bash completion script](/install/completion#bash) +* Install the [zsh completion script](/install/completion#zsh) ## Using Bazel's apt repository @@ -36,14 +43,16 @@ Bazel comes with two completion scripts. After installing Bazel, you can: ```posix-terminal sudo apt install apt-transport-https curl gnupg -y -curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg +curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg sudo mv bazel-archive-keyring.gpg /usr/share/keyrings echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list ``` -The component name "jdk1.8" is kept only for legacy reasons and doesn't relate to supported or included JDK versions. Bazel releases are Java-version agnostic. Changing the "jdk1.8" component name would break existing users of the repo. +The component name "jdk1.8" is kept only for legacy reasons and doesn't relate +to supported or included JDK versions. Bazel releases are Java-version agnostic. +Changing the "jdk1.8" component name would break existing users of the repo. ### Step 2: Install and update Bazel @@ -57,13 +66,18 @@ Once installed, you can upgrade to a newer version of Bazel as part of your norm sudo apt update && sudo apt full-upgrade ``` -The `bazel` package always installs the latest stable version of Bazel. You can install specific, older versions of Bazel in addition to the latest one, such as this: +The `bazel` package always installs the latest stable version of Bazel. You +can install specific, older versions of Bazel in addition to the latest one, +such as this: ```posix-terminal sudo apt install bazel-1.0.0 ``` -This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This can be useful if you need a specific version of Bazel to build a project, for example because it uses a `.bazelversion` file to explicitly state with which Bazel version it should be built. +This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This +can be useful if you need a specific version of Bazel to build a project, for +example because it uses a `.bazelversion` file to explicitly state with which +Bazel version it should be built. Optionally, you can set `bazel` to a specific version by creating a symlink: @@ -75,7 +89,8 @@ bazel --version # 1.0.0 ### Step 3: Install a JDK (optional) -Bazel includes a private, bundled JRE as its runtime and doesn't require you to install any specific version of Java. +Bazel includes a private, bundled JRE as its runtime and doesn't require you to +install any specific version of Java. However, if you want to build Java code using Bazel, you have to install a JDK. @@ -85,11 +100,14 @@ sudo apt install default-jdk ## Using the binary installer -Generally, you should use the apt repository, but the binary installer can be useful if you don't have admin permissions on your machine or can't add custom repositories. +Generally, you should use the apt repository, but the binary installer +can be useful if you don't have admin permissions on your machine or +can't add custom repositories. The binary installers can be downloaded from Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary and extracts it into your `$HOME/bin` folder. Some additional libraries must be installed manually for Bazel to work. +The installer contains the Bazel binary and extracts it into your `$HOME/bin` +folder. Some additional libraries must be installed manually for Bazel to work. ### Step 1: Install required packages @@ -107,34 +125,42 @@ sudo apt-get install default-jdk ### Step 2: Run the installer -Next, download the Bazel binary installer named `bazel-<var>version</var>-installer-linux-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-version-installer-linux-x86_64.sh` +from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). Run it as follows: ```posix-terminal -chmod +x bazel-<var>version</var>-installer-linux-x86_64.sh +chmod +x bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh -./bazel-<var>version</var>-installer-linux-x86_64.sh --user +./bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and +sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see +additional installation options. ### Step 3: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `$HOME/bin` directory. It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel +executable is installed in your `$HOME/bin` directory. +It's a good idea to add this directory to your default paths, as follows: ```posix-terminal export PATH="$PATH:$HOME/bin" ``` -You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it permanent. +You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it +permanent. ## Using the Bazel Docker container -We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. You can use the Docker container as follows: +We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. +You can use the Docker container as follows: ``` -$ docker pull gcr.io/bazel-public/bazel:<bazel version> +$ docker pull gcr.io/bazel-public/bazel: ``` The Docker container is built by [these steps](https://github.com/bazelbuild/continuous-integration/tree/master/bazel/oci). + diff --git a/install/windows.mdx b/install/windows.mdx index 2f808353..b38c0986 100644 --- a/install/windows.mdx +++ b/install/windows.mdx @@ -2,11 +2,16 @@ title: 'Installing Bazel on Windows' --- -This page describes the requirements and steps to install Bazel on Windows. It also includes troubleshooting and other ways to install Bazel, such as using Chocolatey or Scoop. + + +This page describes the requirements and steps to install Bazel on Windows. +It also includes troubleshooting and other ways to install Bazel, such as +using Chocolatey or Scoop. ## Installing Bazel -This section covers the prerequisites, environment setup, and detailed steps during installation on Windows. +This section covers the prerequisites, environment setup, and detailed +steps during installation on Windows. ### Check your system @@ -14,31 +19,33 @@ Recommended: 64 bit Windows 10, version 1703 (Creators Update) or newer To check your Windows version: -- Click the Start button. -- Type `winver` in the search box and press Enter. -- You should see the About Windows box with your Windows version information. +* Click the Start button. +* Type `winver` in the search box and press Enter. +* You should see the About Windows box with your Windows version information. ### Install the prerequisites -- [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) +* [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) ### Download Bazel *Recommended*: [Use Bazelisk](/install/bazelisk) + Alternatively you can: -- [Download the Bazel binary (`bazel-<var>version</var>-windows-x86_64.exe`) from GitHub](https://github.com/bazelbuild/bazel/releases). -- [Install Bazel from Chocolatey](#chocolately) -- [Install Bazel from Scoop](#scoop) -- [Build Bazel from source](/install/compile-source) +* [Download the Bazel binary (`bazel-version-windows-x86_64.exe`) from + GitHub](https://github.com/bazelbuild/bazel/releases). +* [Install Bazel from Chocolatey](#chocolately) +* [Install Bazel from Scoop](#scoop) +* [Build Bazel from source](/install/compile-source) ### Set up your environment To make Bazel easily accessible from command prompts or PowerShell by default, you can rename the Bazel binary to `bazel.exe` and add it to your default paths. ```posix-terminal -set PATH=%PATH%;<var>path to the Bazel binary</var> +set PATH=%PATH%;{{ '' }}path to the Bazel binary{{ '' }} ``` You can also change your system `PATH` environment variable to make it permanent. Check out how to [set environment variables](/configure/windows#set-environment-variables). @@ -50,57 +57,64 @@ You can also change your system `PATH` environment variable to make it permanent To check the installation is correct, try to run: ```posix-terminal -bazel <var>version</var> +bazel {{ '' }}version{{ '' }} ``` Next, you can check out more tips and guidance here: -- [Installing compilers and language runtimes](#install-compilers) -- [Troubleshooting](#troubleshooting) -- [Best practices on Windows](/configure/windows#best-practices) -- [Tutorials](/start/#tutorials) +* [Installing compilers and language runtimes](#install-compilers) +* [Troubleshooting](#troubleshooting) +* [Best practices on Windows](/configure/windows#best-practices) +* [Tutorials](/start/#tutorials) ## Installing compilers and language runtimes Depending on which languages you want to build, you will need: -- [MSYS2 x86\_64](https://www.msys2.org/) +* [MSYS2 x86_64](https://www.msys2.org/) - MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix tools (like `grep`, `tar`, `git`). + MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix + tools (like `grep`, `tar`, `git`). - You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an error if a build target needs Bash but Bazel could not locate it. + You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are + `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an + error if a build target needs Bash but Bazel could not locate it. -- Common MSYS2 packages +* Common MSYS2 packages - You will likely need these to build and run targets that depend on Bash. MSYS2 does not install these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). + You will likely need these to build and run targets that depend on Bash. MSYS2 does not install + these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). - Open the MSYS2 terminal and run this command: + Open the MSYS2 terminal and run this command: - ```posix-terminal - pacman -S zip unzip patch diffutils git - ``` + ```posix-terminal + pacman -S zip unzip patch diffutils git + ``` - Optional: If you want to use Bazel from CMD or Powershell and still be able to use Bash tools, make sure to add `<var>MSYS2_INSTALL_PATH</var>/usr/bin` to your `PATH` environment variable. + Optional: If you want to use Bazel from CMD or Powershell and still be able + to use Bash tools, make sure to add + `MSYS2_INSTALL_PATH/usr/bin` to your + `PATH` environment variable. -- [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) +* [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) - You will need this to build C++ code on Windows. + You will need this to build C++ code on Windows. - Also supported: + Also supported: - - Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK + * Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK -- [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) +* [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) - You will need this to build Java code on Windows. + You will need this to build Java code on Windows. - Also supported: Java 8, 9, and 10 + Also supported: Java 8, 9, and 10 -- [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) +* [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) - You will need this to build Python code on Windows. + You will need this to build Python code on Windows. - Also supported: Python 2.7 or newer for Windows x86-64 + Also supported: Python 2.7 or newer for Windows x86-64 ## Troubleshooting @@ -108,11 +122,11 @@ Depending on which languages you want to build, you will need: **Possible reasons**: -- you installed MSYS2 not under the default install path +* you installed MSYS2 not under the default install path -- you installed MSYS2 i686 instead of MSYS2 x86\_64 +* you installed MSYS2 i686 instead of MSYS2 x86\_64 -- you installed MSYS instead of MSYS2 +* you installed MSYS instead of MSYS2 **Solution**: @@ -120,95 +134,104 @@ Ensure you installed MSYS2 x86\_64. If that doesn't help: -1. Go to Start Menu \> Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for \<username\>"), and click the "New\..." button below it. +3. Look at the list on the top ("User variables for <username>"), and click the "New..." + button below it. -4. For "Variable name", enter `BAZEL_SH` +4. For "Variable name", enter `BAZEL_SH` -5. Click "Browse File..." +5. Click "Browse File..." -6. Navigate to the MSYS2 directory, then to `usr\bin` below it. +6. Navigate to the MSYS2 directory, then to `usr\bin` below it. - For example, this might be `C:\msys64\usr\bin` on your system. + For example, this might be `C:\msys64\usr\bin` on your system. -7. Select the `bash.exe` or `bash` file and click OK +7. Select the `bash.exe` or `bash` file and click OK -8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. +8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. ### Bazel does not find Visual Studio or Visual C++ **Possible reasons**: -- you installed multiple versions of Visual Studio +* you installed multiple versions of Visual Studio -- you installed and removed various versions of Visual Studio +* you installed and removed various versions of Visual Studio -- you installed various versions of the Windows SDK +* you installed various versions of the Windows SDK -- you installed Visual Studio not under the default install path +* you installed Visual Studio not under the default install path **Solution**: -1. Go to Start Menu \> Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for \<username\>"), and click the "New\..." button below it. +3. Look at the list on the top ("User variables for <username>"), and click the "New..." + button below it. -4. For "Variable name", enter `BAZEL_VC` +4. For "Variable name", enter `BAZEL_VC` -5. Click "Browse Directory..." +5. Click "Browse Directory..." -6. Navigate to the `VC` directory of Visual Studio. +6. Navigate to the `VC` directory of Visual Studio. - For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` on your system. + For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` + on your system. -7. Select the `VC` folder and click OK +7. Select the `VC` folder and click OK -8. The "Variable value" field now has the path to `VC`. Click OK to close the window. +8. The "Variable value" field now has the path to `VC`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. ## Other ways to install Bazel ### Using Chocolatey -1. Install the [Chocolatey](https://chocolatey.org) package manager +1. Install the [Chocolatey](https://chocolatey.org) package manager -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - choco install bazel - ``` + ```posix-terminal + choco install bazel + ``` - This command will install the latest available version of Bazel and its dependencies, such as the MSYS2 shell. This will not install Visual C++ though. + This command will install the latest available version of Bazel and + its dependencies, such as the MSYS2 shell. This will not install Visual C++ + though. -See [Chocolatey installation and package maintenance guide](/contribute/windows-chocolatey-maintenance) for more information about the Chocolatey package. +See [Chocolatey installation and package maintenance +guide](/contribute/windows-chocolatey-maintenance) for more +information about the Chocolatey package. ### Using Scoop -1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: +1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: - ```posix-terminal - iex (new-object net.webclient).downloadstring('https://get.scoop.sh') - ``` + ```posix-terminal + iex (new-object net.webclient).downloadstring('https://get.scoop.sh') + ``` -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - scoop install bazel - ``` + ```posix-terminal + scoop install bazel + ``` -See [Scoop installation and package maintenance guide](/contribute/windows-scoop-maintenance) for more information about the Scoop package. +See [Scoop installation and package maintenance +guide](/contribute/windows-scoop-maintenance) for more +information about the Scoop package. ### Build from source diff --git a/migrate/index.mdx b/migrate/index.mdx index 8683aedb..2a574067 100644 --- a/migrate/index.mdx +++ b/migrate/index.mdx @@ -2,7 +2,9 @@ title: 'Migrating to Bazel' --- + + This page links to migration guides for Bazel. -- [Maven](/migrate/maven) -- [Xcode](/migrate/xcode) +* [Maven](/migrate/maven) +* [Xcode](/migrate/xcode) diff --git a/migrate/maven.mdx b/migrate/maven.mdx index 0673ef53..38aaffc0 100644 --- a/migrate/maven.mdx +++ b/migrate/maven.mdx @@ -2,34 +2,55 @@ title: 'Migrating from Maven to Bazel' --- -This page describes how to migrate from Maven to Bazel, including the prerequisites and installation steps. It describes the differences between Maven and Bazel, and provides a migration example using the Guava project. -When migrating from any build tool to Bazel, it's best to have both build tools running in parallel until you have fully migrated your development team, CI system, and any other relevant systems. You can run Maven and Bazel in the same repository. -Note: While Bazel supports downloading and publishing Maven artifacts with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) , it does not directly support Maven-based plugins. Maven plugins can't be directly run by Bazel since there's no Maven compatibility layer. +This page describes how to migrate from Maven to Bazel, including the +prerequisites and installation steps. It describes the differences between Maven +and Bazel, and provides a migration example using the Guava project. + +When migrating from any build tool to Bazel, it's best to have both build tools +running in parallel until you have fully migrated your development team, CI +system, and any other relevant systems. You can run Maven and Bazel in the same +repository. + +Note: While Bazel supports downloading and publishing Maven artifacts with +[rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) +, it does not directly support Maven-based plugins. Maven plugins can't be +directly run by Bazel since there's no Maven compatibility layer. ## Before you begin -- [Install Bazel](/install) if it's not yet installed. -- If you're new to Bazel, go through the tutorial [Introduction to Bazel: Build Java](/start/java) before you start migrating. The tutorial explains Bazel's concepts, structure, and label syntax. +* [Install Bazel](/install) if it's not yet installed. +* If you're new to Bazel, go through the tutorial [Introduction to Bazel: + Build Java](/start/java) before you start migrating. The tutorial explains + Bazel's concepts, structure, and label syntax. ## Differences between Maven and Bazel -- Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files and multiple targets per `BUILD` file, allowing for builds that are more incremental than Maven's. -- Maven takes charge of steps for the deployment process. Bazel does not automate deployment. -- Bazel enables you to express dependencies between languages. -- As you add new sections to the project, with Bazel you may need to add new `BUILD` files. Best practice is to add a `BUILD` file to each new Java package. +* Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files + and multiple targets per `BUILD` file, allowing for builds that are more + incremental than Maven's. +* Maven takes charge of steps for the deployment process. Bazel does not + automate deployment. +* Bazel enables you to express dependencies between languages. +* As you add new sections to the project, with Bazel you may need to add new + `BUILD` files. Best practice is to add a `BUILD` file to each new Java + package. ## Migrate from Maven to Bazel The steps below describe how to migrate your project to Bazel: -1. [Create the MODULE.bazel file](#1-build) -2. [Create one BUILD file](#2-build) -3. [Create more BUILD files](#3-build) -4. [Build using Bazel](#4-build) +1. [Create the MODULE.bazel file](#1-build) +2. [Create one BUILD file](#2-build) +3. [Create more BUILD files](#3-build) +4. [Build using Bazel](#4-build) -Examples below come from a migration of the [Guava project](https://github.com/google/guava) from Maven to Bazel. The Guava project used is release `v31.1`. The examples using Guava do not walk through each step in the migration, but they do show the files and contents that are generated or added manually for the migration. +Examples below come from a migration of the [Guava +project](https://github.com/google/guava) from Maven to Bazel. The +Guava project used is release `v31.1`. The examples using Guava do not walk +through each step in the migration, but they do show the files and contents that +are generated or added manually for the migration. ``` $ git clone https://github.com/google/guava.git && cd guava @@ -38,13 +59,22 @@ $ git checkout v31.1 ### 1. Create the MODULE.bazel file -Create a file named `MODULE.bazel` at the root of your project. If your project has no external dependencies, this file can be empty. +Create a file named `MODULE.bazel` at the root of your project. If your project +has no external dependencies, this file can be empty. -If your project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the MODULE.bazel file. You can use `rules_jvm_external` to manage dependencies from Maven. For instructions about using this ruleset, see [the README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) . +If your project depends on files or packages that are not in one of the +project's directories, specify these external dependencies in the MODULE.bazel +file. You can use `rules_jvm_external` to manage dependencies from Maven. For +instructions about using this ruleset, see [the +README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) +. #### Guava project example: external dependencies -You can list the external dependencies of the [Guava project](https://github.com/google/guava) with the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) ruleset. +You can list the external dependencies of the [Guava +project](https://github.com/google/guava) with the +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) +ruleset. Add the following snippet to the `MODULE.bazel` file: @@ -68,74 +98,89 @@ use_repo(maven, "maven") ### 2. Create one BUILD file -Now that you have your workspace defined and external dependencies (if applicable) listed, you need to create `BUILD` files to describe how your project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use many `BUILD` files to build a project. These files specify multiple build targets, which allow Bazel to produce incremental builds. - -Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of your project and using it to do an initial build using Bazel. Then, you refine your build by adding more `BUILD` files with more granular targets. - -1. In the same directory as your `MODULE.bazel` file, create a text file and name it `BUILD`. - -2. In this `BUILD` file, use the appropriate rule to create one target to build your project. Here are some tips: - - - Use the appropriate rule: - - - To build projects with a single Maven module, use the `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - ) - ``` - - - To build projects with multiple Maven modules, use the `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob([ - "Module1/src/main/java/**/*.java", - "Module2/src/main/java/**/*.java", - ... - ]), - resources = glob([ - "Module1/src/main/resources/**", - "Module2/src/main/resources/**", - ... - ]), - deps = ["//:all-external-targets"], - ) - ``` - - - To build binaries, use the `java_binary` rule: - - ```python - java_binary( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - main_class = "com.example.Main" - ) - ``` - - - Specify the attributes: - - - `name`: Give the target a meaningful name. In the examples above, the target is called "everything." - - `srcs`: Use globbing to list all .java files in your project. - - `resources`: Use globbing to list all resources in your project. - - `deps`: You need to determine which external dependencies your project needs. - - - Take a look at the [example below of this top-level BUILD file](#guava-2) from the migration of the Guava project. - -3. Now that you have a `BUILD` file at the root of your project, build your project to ensure that it works. On the command line, from your workspace directory, use `bazel build //:everything` to build your project with Bazel. - - The project has now been successfully built with Bazel. You will need to add more `BUILD` files to allow incremental builds of the project. +Now that you have your workspace defined and external dependencies (if +applicable) listed, you need to create `BUILD` files to describe how your +project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use +many `BUILD` files to build a project. These files specify multiple build +targets, which allow Bazel to produce incremental builds. + +Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of +your project and using it to do an initial build using Bazel. Then, you refine +your build by adding more `BUILD` files with more granular targets. + +1. In the same directory as your `MODULE.bazel` file, create a text file and + name it `BUILD`. + +2. In this `BUILD` file, use the appropriate rule to create one target to build + your project. Here are some tips: + + * Use the appropriate rule: + * To build projects with a single Maven module, use the + `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + ) + ``` + + * To build projects with multiple Maven modules, use the + `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob([ + "Module1/src/main/java/**/*.java", + "Module2/src/main/java/**/*.java", + ... + ]), + resources = glob([ + "Module1/src/main/resources/**", + "Module2/src/main/resources/**", + ... + ]), + deps = ["//:all-external-targets"], + ) + ``` + + * To build binaries, use the `java_binary` rule: + + ```python + java_binary( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + main_class = "com.example.Main" + ) + ``` + + * Specify the attributes: + * `name`: Give the target a meaningful name. In the examples + above, the target is called "everything." + * `srcs`: Use globbing to list all .java files in your project. + * `resources`: Use globbing to list all resources in your project. + * `deps`: You need to determine which external dependencies your + project needs. + * Take a look at the [example below of this top-level BUILD + file](#guava-2) from the migration of the Guava project. + +3. Now that you have a `BUILD` file at the root of your project, build your + project to ensure that it works. On the command line, from your workspace + directory, use `bazel build //:everything` to build your project with Bazel. + + The project has now been successfully built with Bazel. You will need to add + more `BUILD` files to allow incremental builds of the project. #### Guava project example: start with one BUILD file -When migrating the Guava project to Bazel, initially one `BUILD` file is used to build the entire project. Here are the contents of this initial `BUILD` file in the workspace directory: +When migrating the Guava project to Bazel, initially one `BUILD` file is used to +build the entire project. Here are the contents of this initial `BUILD` file in +the workspace directory: ```python java_library( @@ -157,25 +202,40 @@ java_library( ### 3. Create more BUILD files (optional) -Bazel does work with just one `BUILD file`, as you saw after completing your first build. You should still consider breaking the build into smaller chunks by adding more `BUILD` files with granular targets. +Bazel does work with just one `BUILD file`, as you saw after completing your +first build. You should still consider breaking the build into smaller chunks by +adding more `BUILD` files with granular targets. -Multiple `BUILD` files with multiple targets will give the build increased granularity, allowing: +Multiple `BUILD` files with multiple targets will give the build increased +granularity, allowing: -- increased incremental builds of the project, -- increased parallel execution of the build, -- better maintainability of the build for future users, and -- control over visibility of targets between packages, which can prevent issues such as libraries containing implementation details leaking into public APIs. +* increased incremental builds of the project, +* increased parallel execution of the build, +* better maintainability of the build for future users, and +* control over visibility of targets between packages, which can prevent + issues such as libraries containing implementation details leaking into + public APIs. Tips for adding more `BUILD` files: -- You can start by adding a `BUILD` file to each Java package. Start with Java packages that have the fewest dependencies and work you way up to packages with the most dependencies. -- As you add `BUILD` files and specify targets, add these new targets to the `deps` sections of targets that depend on them. Note that the `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. -- Any time you add a `BUILD` file to a `main` directory, ensure that you add a `BUILD` file to the corresponding `test` directory. -- Take care to limit visibility properly between packages. -- To simplify troubleshooting errors in your setup of `BUILD` files, ensure that the project continues to build with Bazel as you add each build file. Run `bazel build //...` to ensure all of your targets still build. +* You can start by adding a `BUILD` file to each Java package. Start with Java + packages that have the fewest dependencies and work you way up to packages + with the most dependencies. +* As you add `BUILD` files and specify targets, add these new targets to the + `deps` sections of targets that depend on them. Note that the `glob()` + function does not cross package boundaries, so as the number of packages + grows the files matched by `glob()` will shrink. +* Any time you add a `BUILD` file to a `main` directory, ensure that you add a + `BUILD` file to the corresponding `test` directory. +* Take care to limit visibility properly between packages. +* To simplify troubleshooting errors in your setup of `BUILD` files, ensure + that the project continues to build with Bazel as you add each build file. + Run `bazel build //...` to ensure all of your targets still build. ### 4. Build using Bazel -You've been building using Bazel as you add `BUILD` files to validate the setup of the build. +You've been building using Bazel as you add `BUILD` files to validate the setup +of the build. -When you have `BUILD` files at the desired granularity, you can use Bazel to produce all of your builds. +When you have `BUILD` files at the desired granularity, you can use Bazel to +produce all of your builds. diff --git a/migrate/xcode.mdx b/migrate/xcode.mdx index 4cd229c9..986cd115 100644 --- a/migrate/xcode.mdx +++ b/migrate/xcode.mdx @@ -2,163 +2,243 @@ title: 'Migrating from Xcode to Bazel' --- -This page describes how to build or test an Xcode project with Bazel. It describes the differences between Xcode and Bazel, and provides the steps for converting an Xcode project to a Bazel project. It also provides troubleshooting solutions to address common errors. + + +This page describes how to build or test an Xcode project with Bazel. It +describes the differences between Xcode and Bazel, and provides the steps for +converting an Xcode project to a Bazel project. It also provides troubleshooting +solutions to address common errors. ## Differences between Xcode and Bazel -- Bazel requires you to explicitly specify every build target and its dependencies, plus the corresponding build settings via build rules. +* Bazel requires you to explicitly specify every build target and its + dependencies, plus the corresponding build settings via build rules. -- Bazel requires all files on which the project depends to be present within the workspace directory or specified as dependencies in the `MODULE.bazel` file. +* Bazel requires all files on which the project depends to be present within + the workspace directory or specified as dependencies in the `MODULE.bazel` + file. -- When building Xcode projects with Bazel, the `BUILD` file(s) become the source of truth. If you work on the project in Xcode, you must generate a new version of the Xcode project that matches the `BUILD` files using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) whenever you update the `BUILD` files. Certain changes to the `BUILD` files such as adding dependencies to a target don't require regenerating the project which can speed up development. If you're not using Xcode, the `bazel build` and `bazel test` commands provide build and test capabilities with certain limitations described later in this guide. +* When building Xcode projects with Bazel, the `BUILD` file(s) become the + source of truth. If you work on the project in Xcode, you must generate a + new version of the Xcode project that matches the `BUILD` files using + [rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) + whenever you update the `BUILD` files. Certain changes to the `BUILD` files + such as adding dependencies to a target don't require regenerating the + project which can speed up development. If you're not using Xcode, the + `bazel build` and `bazel test` commands provide build and test capabilities + with certain limitations described later in this guide. ## Before you begin Before you begin, do the following: -1. [Install Bazel](/install) if you have not already done so. +1. [Install Bazel](/install) if you have not already done so. -2. If you're not familiar with Bazel and its concepts, complete the [iOS app tutorial](/start/ios-app)). You should understand the Bazel workspace, including the `MODULE.bazel` and `BUILD` files, as well as the concepts of targets, build rules, and Bazel packages. +2. If you're not familiar with Bazel and its concepts, complete the [iOS app + tutorial](/start/ios-app)). You should understand the Bazel workspace, + including the `MODULE.bazel` and `BUILD` files, as well as the concepts of + targets, build rules, and Bazel packages. -3. Analyze and understand the project's dependencies. +3. Analyze and understand the project's dependencies. ### Analyze project dependencies -Unlike Xcode, Bazel requires you to explicitly declare all dependencies for every target in the `BUILD` file. +Unlike Xcode, Bazel requires you to explicitly declare all dependencies for +every target in the `BUILD` file. -For more information on external dependencies, see [Working with external dependencies](/docs/external). +For more information on external dependencies, see [Working with external +dependencies](/docs/external). ## Build or test an Xcode project with Bazel To build or test an Xcode project with Bazel, do the following: -1. [Create the `MODULE.bazel` file](#create-workspace) +1. [Create the `MODULE.bazel` file](#create-workspace) -2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) +2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) -3. [Create a `BUILD` file:](#create-build-file) +3. [Create a `BUILD` file:](#create-build-file) - a. [Add the application target](#add-app-target) + a. [Add the application target](#add-app-target) - b. [(Optional) Add the test target(s)](#add-test-target) + b. [(Optional) Add the test target(s)](#add-test-target) - c. [Add the library target(s)](#add-library-target) + c. [Add the library target(s)](#add-library-target) -4. [(Optional) Granularize the build](#granularize-build) +4. [(Optional) Granularize the build](#granularize-build) -5. [Run the build](#run-build) +5. [Run the build](#run-build) -6. [Generate the Xcode project with rules\_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) +6. [Generate the Xcode project with rules_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) ### Step 1: Create the `MODULE.bazel` file -Create a `MODULE.bazel` file in a new directory. This directory becomes the Bazel workspace root. If the project uses no external dependencies, this file can be empty. If the project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the `MODULE.bazel` file. +Create a `MODULE.bazel` file in a new directory. This directory becomes the +Bazel workspace root. If the project uses no external dependencies, this file +can be empty. If the project depends on files or packages that are not in one of +the project's directories, specify these external dependencies in the +`MODULE.bazel` file. -Note: Place the project source code within the directory tree containing the `MODULE.bazel` file. +Note: Place the project source code within the directory tree containing the +`MODULE.bazel` file. ### Step 2: (Experimental) Integrate SwiftPM dependencies -To integrate SwiftPM dependencies into the Bazel workspace with [swift\_bazel](https://github.com/cgrindel/swift_bazel), you must convert them into Bazel packages as described in the [following tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) . +To integrate SwiftPM dependencies into the Bazel workspace with +[swift_bazel](https://github.com/cgrindel/swift_bazel), you must +convert them into Bazel packages as described in the [following +tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) +. -Note: SwiftPM support is a manual process with many variables. SwiftPM integration with Bazel has not been fully verified and is not officially supported. +Note: SwiftPM support is a manual process with many variables. SwiftPM +integration with Bazel has not been fully verified and is not officially +supported. ### Step 3: Create a `BUILD` file -Once you have defined the workspace and external dependencies, you need to create a `BUILD` file that tells Bazel how the project is structured. Create the `BUILD` file at the root of the Bazel workspace and configure it to do an initial build of the project as follows: +Once you have defined the workspace and external dependencies, you need to +create a `BUILD` file that tells Bazel how the project is structured. Create the +`BUILD` file at the root of the Bazel workspace and configure it to do an +initial build of the project as follows: -- [Step 3a: Add the application target](#step-3a-add-the-application-target) -- [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) -- [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) +* [Step 3a: Add the application target](#step-3a-add-the-application-target) +* [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) +* [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) -**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, packages, and targets](/concepts/build-ref). +**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, +packages, and targets](/concepts/build-ref). #### Step 3a: Add the application target -Add a [`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) or an [`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) rule target. This target builds a macOS or iOS application bundle, respectively. In the target, specify the following at the minimum: +Add a +[`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) +or an +[`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) +rule target. This target builds a macOS or iOS application bundle, respectively. +In the target, specify the following at the minimum: -- `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the binary. +* `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the + binary. -- `provisioning_profile` - provisioning profile from your Apple Developer account (if building for an iOS device device). +* `provisioning_profile` - provisioning profile from your Apple Developer + account (if building for an iOS device device). -- `families` (iOS only) - whether to build the application for iPhone, iPad, or both. +* `families` (iOS only) - whether to build the application for iPhone, iPad, + or both. -- `infoplists` - list of .plist files to merge into the final Info.plist file. +* `infoplists` - list of .plist files to merge into the final Info.plist file. -- `minimum_os_version` - the minimum version of macOS or iOS that the application supports. This ensures Bazel builds the application with the correct API levels. +* `minimum_os_version` - the minimum version of macOS or iOS that the + application supports. This ensures Bazel builds the application with the + correct API levels. #### Step 3b: (Optional) Add the test target(s) -Bazel's [Apple build rules](https://github.com/bazelbuild/rules_apple) support running unit and UI tests on all Apple platforms. Add test targets as follows: +Bazel's [Apple build +rules](https://github.com/bazelbuild/rules_apple) support running +unit and UI tests on all Apple platforms. Add test targets as follows: -- [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) to run library-based and application-based unit tests on a macOS. +* [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) + to run library-based and application-based unit tests on a macOS. -- [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) to build and run library-based unit tests on iOS. +* [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) + to build and run library-based unit tests on iOS. -- [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) to build and run user interface tests in the iOS simulator. +* [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) + to build and run user interface tests in the iOS simulator. -- Similar test rules exist for [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) and [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). +* Similar test rules exist for + [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), + [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) + and + [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). -At the minimum, specify a value for the `minimum_os_version` attribute. While other packaging attributes, such as `bundle_identifier` and `infoplists`, default to most commonly used values, ensure that those defaults are compatible with the project and adjust them as necessary. For tests that require the iOS simulator, also specify the `ios_application` target name as the value of the `test_host` attribute. +At the minimum, specify a value for the `minimum_os_version` attribute. While +other packaging attributes, such as `bundle_identifier` and `infoplists`, +default to most commonly used values, ensure that those defaults are compatible +with the project and adjust them as necessary. For tests that require the iOS +simulator, also specify the `ios_application` target name as the value of the +`test_host` attribute. #### Step 3c: Add the library target(s) -Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each Objective-C library and a [`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) target for each Swift library on which the application and/or tests depend. +Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each +Objective-C library and a +[`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) +target for each Swift library on which the application and/or tests depend. Add the library targets as follows: -- Add the application library targets as dependencies to the application targets. +* Add the application library targets as dependencies to the application + targets. -- Add the test library targets as dependencies to the test targets. +* Add the test library targets as dependencies to the test targets. -- List the implementation sources in the `srcs` attribute. +* List the implementation sources in the `srcs` attribute. -- List the headers in the `hdrs` attribute. +* List the headers in the `hdrs` attribute. -Note: You can use the [`glob`](/reference/be/functions#glob) function to include all sources and/or headers of a certain type. Use it carefully as it might include files you do not want Bazel to build. +Note: You can use the [`glob`](/reference/be/functions#glob) function to include +all sources and/or headers of a certain type. Use it carefully as it might +include files you do not want Bazel to build. -You can browse existing examples for various types of applications directly in the [rules\_apple examples directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For example: +You can browse existing examples for various types of applications directly in +the [rules_apple examples +directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For +example: -- [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) +* [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) -- [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) +* [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) -- [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) +* [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) -For more information on build rules, see [Apple Rules for Bazel](https://github.com/bazelbuild/rules_apple). +For more information on build rules, see [Apple Rules for +Bazel](https://github.com/bazelbuild/rules_apple). At this point, it is a good idea to test the build: -`bazel build //:<application_target>` +`bazel build //:` ### Step 4: (Optional) Granularize the build -If the project is large, or as it grows, consider chunking it into multiple Bazel packages. This increased granularity provides: +If the project is large, or as it grows, consider chunking it into multiple +Bazel packages. This increased granularity provides: -- Increased incrementality of builds, +* Increased incrementality of builds, -- Increased parallelization of build tasks, +* Increased parallelization of build tasks, -- Better maintainability for future users, +* Better maintainability for future users, -- Better control over source code visibility across targets and packages. This prevents issues such as libraries containing implementation details leaking into public APIs. +* Better control over source code visibility across targets and packages. This + prevents issues such as libraries containing implementation details leaking + into public APIs. Tips for granularizing the project: -- Put each library in its own Bazel package. Start with those requiring the fewest dependencies and work your way up the dependency tree. +* Put each library in its own Bazel package. Start with those requiring the + fewest dependencies and work your way up the dependency tree. -- As you add `BUILD` files and specify targets, add these new targets to the `deps` attributes of targets that depend on them. +* As you add `BUILD` files and specify targets, add these new targets to the + `deps` attributes of targets that depend on them. -- The `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. +* The `glob()` function does not cross package boundaries, so as the number of + packages grows the files matched by `glob()` will shrink. -- When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to the corresponding `test` directory. +* When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to + the corresponding `test` directory. -- Enforce healthy visibility limits across packages. +* Enforce healthy visibility limits across packages. -- Build the project after each major change to the `BUILD` files and fix build errors as you encounter them. +* Build the project after each major change to the `BUILD` files and fix build + errors as you encounter them. ### Step 5: Run the build -Run the fully migrated build to ensure it completes with no errors or warnings. Run every application and test target individually to more easily find sources of any errors that occur. +Run the fully migrated build to ensure it completes with no errors or warnings. +Run every application and test target individually to more easily find sources +of any errors that occur. For example: @@ -166,17 +246,25 @@ For example: bazel build //:my-target ``` -### Step 6: Generate the Xcode project with rules\_xcodeproj +### Step 6: Generate the Xcode project with rules_xcodeproj -When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source of truth about the build. To make Xcode aware of this, you must generate a Bazel-compatible Xcode project using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) . +When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source +of truth about the build. To make Xcode aware of this, you must generate a +Bazel-compatible Xcode project using +[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) +. ### Troubleshooting -Bazel errors can arise when it gets out of sync with the selected Xcode version, like when you apply an update. Here are some things to try if you're experiencing errors with Xcode, for example "Xcode version must be specified to use an Apple CROSSTOOL". +Bazel errors can arise when it gets out of sync with the selected Xcode version, +like when you apply an update. Here are some things to try if you're +experiencing errors with Xcode, for example "Xcode version must be specified to +use an Apple CROSSTOOL". -- Manually run Xcode and accept any terms and conditions. +* Manually run Xcode and accept any terms and conditions. -- Use Xcode select to indicate the correct version, accept the license, and clear Bazel's state. +* Use Xcode select to indicate the correct version, accept the license, and + clear Bazel's state. ```posix-terminal sudo xcode-select -s /Applications/Xcode.app/Contents/Developer @@ -186,6 +274,7 @@ Bazel errors can arise when it gets out of sync with the selected Xcode version, bazel sync --configure ``` -- If this does not work, you may also try running `bazel clean --expunge`. +* If this does not work, you may also try running `bazel clean --expunge`. -Note: If you've saved your Xcode to a different path, you can use `xcode-select -s` to point to that path. +Note: If you've saved your Xcode to a different path, you can use `xcode-select +-s` to point to that path. diff --git a/query/aquery.mdx b/query/aquery.mdx index 446855d1..151565b7 100644 --- a/query/aquery.mdx +++ b/query/aquery.mdx @@ -2,13 +2,23 @@ title: 'Action Graph Query (aquery)' --- -The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis Configured Target Graph and exposes information about **Actions, Artifacts and their relationships.** -`aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs/outputs/mnemonics. -The tool accepts several command-line [options](#command-options). Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. +The `aquery` command allows you to query for actions in your build graph. +It operates on the post-analysis Configured Target Graph and exposes +information about **Actions, Artifacts and their relationships.** -It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. +`aquery` is useful when you are interested in the properties of the Actions/Artifacts +generated from the Configured Target Graph. For example, the actual commands run +and their inputs/outputs/mnemonics. + +The tool accepts several command-line [options](#command-options). +Notably, the aquery command runs on top of a regular Bazel build and inherits +the set of options available during a build. + +It supports the same set of functions that is also available to traditional +`query` but `siblings`, `buildfiles` and +`tests`. An example `aquery` output (without specific details): @@ -31,9 +41,11 @@ A simple example of the syntax for `aquery` is as follows: The query expression (in quotes) consists of the following: -- `aquery_function(...)`: functions specific to `aquery`. More details [below](#using-aquery-functions). -- `function(...)`: the standard [functions](/query/language#functions) as traditional `query`. -- `//target` is the label to the interested target. +* `aquery_function(...)`: functions specific to `aquery`. + More details [below](#using-aquery-functions). +* `function(...)`: the standard [functions](/query/language#functions) + as traditional `query`. +* `//target` is the label to the interested target. ``` # aquery examples: @@ -52,13 +64,14 @@ $ bazel aquery 'inputs(".*cpp", deps(//src/target_a))' There are three `aquery` functions: -- `inputs`: filter actions by inputs. -- `outputs`: filter actions by outputs -- `mnemonic`: filter actions by mnemonic +* `inputs`: filter actions by inputs. +* `outputs`: filter actions by outputs +* `mnemonic`: filter actions by mnemonic `expr ::= inputs(word, expr)` -The `inputs` operator returns the actions generated from building `expr`, whose input filenames match the regex provided by `word`. + The `inputs` operator returns the actions generated from building `expr`, + whose input filenames match the regex provided by `word`. `$ bazel aquery 'inputs(".*cpp", deps(//src/target_a))'` @@ -70,9 +83,13 @@ You can also combine functions to achieve the AND operation. For example: $ bazel aquery 'mnemonic("Cpp.*", (inputs(".*cpp", inputs("foo.*", //src/target_a))))' ``` -The above command would find all actions involved in building `//src/target_a`, whose mnemonics match `"Cpp.*"` and inputs match the patterns `".*cpp"` and `"foo.*"`. + The above command would find all actions involved in building `//src/target_a`, + whose mnemonics match `"Cpp.*"` and inputs match the patterns + `".*cpp"` and `"foo.*"`. -Important: aquery functions can't be nested inside non-aquery functions. Conceptually, this makes sense since the output of aquery functions is Actions, not Configured Targets. +Important: aquery functions can't be nested inside non-aquery functions. +Conceptually, this makes sense since the output of aquery functions is Actions, +not Configured Targets. An example of the syntax error produced: @@ -87,17 +104,23 @@ An example of the syntax error produced: ### Build options -`aquery` runs on top of a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. +`aquery` runs on top of a regular Bazel build and thus inherits the set of +[options](/reference/command-line-reference#build-options) +available during a build. ### Aquery options #### `--output=(text|summary|commands|proto|jsonproto|textproto), default=text` -The default output format (`text`) is human-readable, use `proto`, `textproto`, or `jsonproto` for machine-readable format. The proto message is `analysis.ActionGraphContainer`. +The default output format (`text`) is human-readable, +use `proto`, `textproto`, or `jsonproto` for machine-readable format. +The proto message is `analysis.ActionGraphContainer`. -The `commands` output format prints a list of build commands with one command per line. +The `commands` output format prints a list of build commands with +one command per line. -In general, do not depend on the order of output. For more information, see the [core query ordering contract](/query/language#graph-order). +In general, do not depend on the order of output. For more information, +see the [core query ordering contract](/query/language#graph-order). #### `--include_commandline, default=true` @@ -119,35 +142,45 @@ Warning: Enabling this flag will automatically enable the `--include_commandline #### `--include_file_write_contents, default=false` -Include file contents for the `actions.write()` action and the contents of the manifest file for the `SourceSymlinkManifest` action The file contents is returned in the `file_contents` field with `--output=`xxx`proto`. With `--output=text`, the output has - +Include file contents for the `actions.write()` action and the contents of the +manifest file for the `SourceSymlinkManifest` action The file contents is +returned in the `file_contents` field with `--output=`xxx`proto`. +With `--output=text`, the output has ``` -FileWriteContents: [<base64-encoded file contents>] +FileWriteContents: [] ``` - line #### `--skyframe_state, default=false` Without performing extra analysis, dump the Action Graph from Skyframe. -Note: Specifying a target with `--skyframe_state` is currently not supported. This flag is only available with `--output=proto` or `--output=textproto`. +Note: Specifying a target with `--skyframe_state` is currently not supported. +This flag is only available with `--output=proto` or `--output=textproto`. ## Other tools and features ### Querying against the state of Skyframe -[Skyframe](/reference/skyframe) is the evaluation and incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph constructed from the previous runs of the [Analysis phase](/run/build#analysis). +[Skyframe](/reference/skyframe) is the evaluation and +incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph +constructed from the previous runs of the [Analysis phase](/run/build#analysis). -In some cases, it is useful to query the Action Graph on Skyframe. An example use case would be: +In some cases, it is useful to query the Action Graph on Skyframe. +An example use case would be: -1. Run `bazel build //target_a` -2. Run `bazel build //target_b` -3. File `foo.out` was generated. +1. Run `bazel build //target_a` +2. Run `bazel build //target_b` +3. File `foo.out` was generated. -*As a Bazel user, I want to determine if `foo.out` was generated from building `//target_a` or `//target_b`*. +_As a Bazel user, I want to determine if `foo.out` was generated from building +`//target_a` or `//target_b`_. -One could run `bazel aquery 'outputs("foo.out", //target_a)'` and `bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible for creating `foo.out`, and in turn the target. However, the number of different targets previously built can be larger than 2, which makes running multiple `aquery` commands a hassle. +One could run `bazel aquery 'outputs("foo.out", //target_a)'` and +`bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible +for creating `foo.out`, and in turn the target. However, the number of different +targets previously built can be larger than 2, which makes running multiple `aquery` +commands a hassle. As an alternative, the `--skyframe_state` flag can be used: @@ -161,17 +194,22 @@ As an alternative, the `--skyframe_state` flag can be used: $ bazel aquery --output=proto --skyframe_state 'outputs("foo.out")' ``` -With `--skyframe_state` mode, `aquery` takes the content of the Action Graph that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and outputs the content, without re-running the analysis phase. +With `--skyframe_state` mode, `aquery` takes the content of the Action Graph +that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and +outputs the content, without re-running the analysis phase. #### Special considerations ##### Output format -`--skyframe_state` is currently only available for `--output=proto` and `--output=textproto` +`--skyframe_state` is currently only available for `--output=proto` +and `--output=textproto` ##### Non-inclusion of target labels in the query expression -Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, regardless of the targets. Having the target label specified in the query together with `--skyframe_state` is considered a syntax error: +Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, +regardless of the targets. Having the target label specified in the query together with +`--skyframe_state` is considered a syntax error: ``` # WRONG: Target Included @@ -189,9 +227,12 @@ Currently, `--skyframe_state` queries the whole action graph that exists on Skyf ### Comparing aquery outputs -You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. For instance: when you make some changes to your rule definition and want to verify that the command lines being run did not change. `aquery_differ` is the tool for that. +You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. +For instance: when you make some changes to your rule definition and want to verify that the +command lines being run did not change. `aquery_differ` is the tool for that. -The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. To use it, clone the repository to your local machine. An example usage: +The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. +To use it, clone the repository to your local machine. An example usage: ``` $ bazel run //tools/aquery_differ -- \ @@ -202,7 +243,9 @@ The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/ba --attrs=inputs ``` -The above command returns the difference between the `before` and `after` aquery outputs: which actions were present in one but not the other, which actions have different command line/inputs in each aquery output, ...). The result of running the above command would be: +The above command returns the difference between the `before` and `after` aquery outputs: +which actions were present in one but not the other, which actions have different +command line/inputs in each aquery output, ...). The result of running the above command would be: ``` Aquery output 'after' change contains an action that generates the following outputs that aquery output 'before' change doesn't: @@ -225,29 +268,36 @@ The above command returns the difference between the `before` and `after` aquery `--before, --after`: The aquery output files to be compared -`--input_type=(proto|text_proto), default=proto`: the format of the input files. Support is provided for `proto` and `textproto` aquery output. +`--input_type=(proto|text_proto), default=proto`: the format of the input +files. Support is provided for `proto` and `textproto` aquery output. -`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions to be compared. +`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions +to be compared. ### Aspect-on-aspect -It is possible for [Aspects](/extending/aspects) to be applied on top of each other. The aquery output of the action generated by these Aspects would then include the *Aspect path*, which is the sequence of Aspects applied to the target which generated the action. +It is possible for [Aspects](/extending/aspects) +to be applied on top of each other. The aquery output of the action generated by +these Aspects would then include the _Aspect path_, which is the sequence of +Aspects applied to the target which generated the action. An example of Aspect-on-Aspect: ``` t0 ^ - | <- a1 + | <- a1 t1 ^ - | <- a2 + | <- a2 t2 ``` -Let ti be a target of rule ri, which applies an Aspect ai to its dependencies. +Let ti be a target of rule ri, which applies an Aspect ai +to its dependencies. -Assume that a2 generates an action X when applied to target t0. The text output of `bazel aquery --include_aspects 'deps(//t2)'` for action X would be: +Assume that a2 generates an action X when applied to target t0. The text output of +`bazel aquery --include_aspects 'deps(//t2)'` for action X would be: ``` action ... @@ -255,11 +305,13 @@ Assume that a2 generates an action X when applied to target t0. The text output Target: //my_pkg:t0 Configuration: ... AspectDescriptors: [//my_pkg:rule.bzl%**a2**(foo=...) - -> //my_pkg:rule.bzl%**a1**(bar=...)] + -> //my_pkg:rule.bzl%**a1**(bar=...)] ... ``` -This means that action `X` was generated by Aspect `a2` applied onto `a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied onto target `t0`. +This means that action `X` was generated by Aspect `a2` applied onto +`a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied +onto target `t0`. Each `AspectDescriptor` has the following format: @@ -267,31 +319,49 @@ Each `AspectDescriptor` has the following format: AspectClass([param=value,...]) ``` -`AspectClass` could be the name of the Aspect class (for native Aspects) or `bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are sorted in topological order of the [dependency graph](/extending/aspects#aspect_basics). +`AspectClass` could be the name of the Aspect class (for native Aspects) or +`bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are +sorted in topological order of the +[dependency graph](/extending/aspects#aspect_basics). ### Linking with the JSON profile -While aquery provides information about the actions being run in a build (why they're being run, their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) tells us the timing and duration of their execution. It is possible to combine these 2 sets of information via a common denominator: an action's primary output. +While aquery provides information about the actions being run in a build (why they're being run, +their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) +tells us the timing and duration of their execution. +It is possible to combine these 2 sets of information via a common denominator: an action's primary output. -To include actions' outputs in the JSON profile, generate the profile with `--experimental_include_primary_output --noslim_profile`. Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output is included by default by aquery. +To include actions' outputs in the JSON profile, generate the profile with +`--experimental_include_primary_output --noslim_profile`. +Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output +is included by default by aquery. -We don't currently provide a canonical tool to combine these 2 data sources, but you should be able to build your own script with the above information. +We don't currently provide a canonical tool to combine these 2 data sources, but you should be +able to build your own script with the above information. ## Known issues ### Handling shared actions -Sometimes actions are [shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) between configured targets. +Sometimes actions are +[shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) +between configured targets. -In the execution phase, those shared actions are [simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these like separate actions whose output Artifacts have the exact same `execPath`. As a result, equivalent Artifacts appear duplicated. +In the execution phase, those shared actions are +[simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. +However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these +like separate actions whose output Artifacts have the exact same `execPath`. As a result, +equivalent Artifacts appear duplicated. -The list of aquery issues/planned features can be found on [GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). +The list of aquery issues/planned features can be found on +[GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). ## FAQs ### The ActionKey remains the same even though the content of an input file changed. -In the context of aquery, the `ActionKey` refers to the `String` gotten from [ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): +In the context of aquery, the `ActionKey` refers to the `String` gotten from +[ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): ``` Returns a string encoding all of the significant behaviour of this Action that might affect the @@ -313,7 +383,8 @@ In the context of aquery, the `ActionKey` refers to the `String` gotten from [Ac input names change or else action validation may falsely validate. ``` -This excludes the changes to the content of the input files, and is not to be confused with [RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). +This excludes the changes to the content of the input files, and is not to be confused with +[RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). ## Updates diff --git a/query/cquery.mdx b/query/cquery.mdx index 6a073058..d35ea829 100644 --- a/query/cquery.mdx +++ b/query/cquery.mdx @@ -2,14 +2,21 @@ title: 'Configurable Query (cquery)' --- -`cquery` is a variant of [`query`](/query/language) that correctly handles [`select()`](/docs/configurable-attributes) and build options' effects on the build graph. -It achieves this by running over the results of Bazel's [analysis phase](/extending/concepts#evaluation-model), which integrates these effects. `query`, by contrast, runs over the results of Bazel's loading phase, before options are evaluated. + +`cquery` is a variant of [`query`](/query/language) that correctly handles +[`select()`](/docs/configurable-attributes) and build options' effects on the +build graph. + +It achieves this by running over the results of Bazel's [analysis +phase](/extending/concepts#evaluation-model), +which integrates these effects. `query`, by contrast, runs over the results of +Bazel's loading phase, before options are evaluated. For example: ``` -$ cat > tree/BUILD <<EOF +$ cat > tree/BUILD <<EOF sh_library( name = "ash", deps = select({ @@ -23,11 +30,11 @@ sh_library(name = "white-ash") sh_library(name = "common-ash") config_setting( name = "excelsior", - values = {"define": "species=excelsior"}, + values = \{"define": "species=excelsior"\}, ) config_setting( name = "americana", - values = {"define": "species=americana"}, + values = \{"define": "species=americana"\}, ) EOF ``` @@ -52,9 +59,13 @@ $ bazel cquery "deps(//tree:ash)" --define species=excelsior --noimplicit_deps //tree:excelsior (9f87702) ``` -Each result includes a [unique identifier](#configurations) `(9f87702)` of the [configuration](/reference/glossary#configuration) the target is built with. +Each result includes a [unique identifier](#configurations) `(9f87702)` of +the [configuration](/reference/glossary#configuration) the +target is built with. -Since `cquery` runs over the configured target graph. it doesn't have insight into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). +Since `cquery` runs over the configured target graph. it doesn't have insight +into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) +rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). ## Basic syntax @@ -64,10 +75,19 @@ A simple `cquery` call looks like: The query expression `"function(//target)"` consists of the following: -- **`function(...)`** is the function to run on the target. `cquery` supports most of `query`'s [functions](/query/language#functions), plus a few new ones. -- **`//target`** is the expression fed to the function. In this example, the expression is a simple target. But the query language also allows nesting of functions. See the [Query guide](/query/guide) for examples. +* **`function(...)`** is the function to run on the target. `cquery` + supports most + of `query`'s [functions](/query/language#functions), plus a + few new ones. +* **`//target`** is the expression fed to the function. In this example, the + expression is a simple target. But the query language also allows nesting of functions. + See the [Query guide](/query/guide) for examples. + -`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) phases. Unless otherwise specified, `cquery` parses the target(s) listed in the query expression. See [`--universe_scope`](#universe-scope) for querying dependencies of top-level build targets. +`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) +phases. Unless otherwise specified, `cquery` parses the target(s) listed in the +query expression. See [`--universe_scope`](#universe-scope) +for querying dependencies of top-level build targets. ## Configurations @@ -77,7 +97,9 @@ The line: //tree:ash (9f87702) ``` -means `//tree:ash` was built in a configuration with ID `9f87702`. For most targets, this is an opaque hash of the build option values defining the configuration. +means `//tree:ash` was built in a configuration with ID `9f87702`. For most +targets, this is an opaque hash of the build option values defining the +configuration. To see the configuration's complete contents, run: @@ -85,15 +107,25 @@ To see the configuration's complete contents, run: $ bazel config 9f87702 ``` -`9f87702` is a prefix of the complete ID. This is because complete IDs are SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid prefix of a complete ID, similar to [Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). To see complete IDs, run `$ bazel config`. +`9f87702` is a prefix of the complete ID. This is because complete IDs are +SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid +prefix of a complete ID, similar to +[Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). + To see complete IDs, run `$ bazel config`. ## Target pattern evaluation -`//foo` has a different meaning for `cquery` than for `query`. This is because `cquery` evaluates *configured* targets and the build graph may have multiple configured versions of `//foo`. +`//foo` has a different meaning for `cquery` than for `query`. This is because +`cquery` evaluates _configured_ targets and the build graph may have multiple +configured versions of `//foo`. -For `cquery`, a target pattern in the query expression evaluates to every configured target with a label that matches that pattern. Output is deterministic, but `cquery` makes no ordering guarantee beyond the [core query ordering contract](/query/language#graph-order). +For `cquery`, a target pattern in the query expression evaluates +to every configured target with a label that matches that pattern. Output is +deterministic, but `cquery` makes no ordering guarantee beyond the +[core query ordering contract](/query/language#graph-order). -This produces subtler results for query expressions than with `query`. For example, the following can produce multiple results: +This produces subtler results for query expressions than with `query`. +For example, the following can produce multiple results: ``` # Analyzes //foo in the target configuration, but also analyzes @@ -105,13 +137,22 @@ $ bazel cquery //foo --universe_scope=//foo,//genrule_with_foo_as_tool //foo (exec) ``` -If you want to precisely declare which instance to query over, use the [`config`](#config) function. +If you want to precisely declare which instance to query over, use +the [`config`](#config) function. -See `query`'s [target pattern documentation](/query/language#target-patterns) for more information on target patterns. +See `query`'s [target pattern +documentation](/query/language#target-patterns) for more information on target +patterns. ## Functions -Of the [set of functions](/query/language#functions "list of query functions") supported by `query`, `cquery` supports all but [`allrdeps`](/query/language#allrdeps), [`buildfiles`](/query/language#buildfiles), [`rbuildfiles`](/query/language#rbuildfiles), [`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and [`visible`](/query/language#visible). +Of the [set of functions](/query/language#functions "list of query functions") +supported by `query`, `cquery` supports all but +[`allrdeps`](/query/language#allrdeps), +[`buildfiles`](/query/language#buildfiles), +[`rbuildfiles`](/query/language#rbuildfiles), +[`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and +[`visible`](/query/language#visible). `cquery` also introduces the following new functions: @@ -119,9 +160,13 @@ Of the [set of functions](/query/language#functions "list of query functions") s `expr ::= config(expr, word)` -The `config` operator attempts to find the configured target for the label denoted by the first argument and configuration specified by the second argument. +The `config` operator attempts to find the configured target for +the label denoted by the first argument and configuration specified by the +second argument. -Valid values for the second argument are `null` or a [custom configuration hash](#configurations). Hashes can be retrieved from `$ bazel config` or a previous `cquery`'s output. +Valid values for the second argument are `null` or a +[custom configuration hash](#configurations). Hashes can be retrieved from `$ +bazel config` or a previous `cquery`'s output. Examples: @@ -137,19 +182,27 @@ $ bazel cquery "deps(//foo)" $ bazel cquery "config(//baz, 3732cc8)" ``` -If not all results of the first argument can be found in the specified configuration, only those that can be found are returned. If no results can be found in the specified configuration, the query fails. +If not all results of the first argument can be found in the specified +configuration, only those that can be found are returned. If no results +can be found in the specified configuration, the query fails. ## Options ### Build options -`cquery` runs over a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. +`cquery` runs over a regular Bazel build and thus inherits the set of +[options](/reference/command-line-reference#build-options) available during a +build. -### Using cquery options +### Using cquery options #### `--universe_scope` (comma-separated list) -Often, the dependencies of configured targets go through [transitions](/extending/rules#configurations), which causes their configuration to differ from their dependent. This flag allows you to query a target as if it were built as a dependency or a transitive dependency of another target. For example: +Often, the dependencies of configured targets go through +[transitions](/extending/rules#configurations), +which causes their configuration to differ from their dependent. This flag +allows you to query a target as if it were built as a dependency or a transitive +dependency of another target. For example: ``` # x/BUILD @@ -157,7 +210,7 @@ genrule( name = "my_gen", srcs = ["x.in"], outs = ["x.cc"], - cmd = "$(locations :tool) $< >$@", + cmd = "$(locations :tool) $< >$@", tools = [":tool"], ) cc_binary( @@ -166,34 +219,75 @@ cc_binary( ) ``` -Genrules configure their tools in the [exec configuration](/extending/rules#configurations) so the following queries would produce the following outputs: - -| Query | Target Built | Output | -| ------------------------------------------------------- | ------------ | ---------------------- | -| bazel cquery "//x:tool" | //x:tool | //x:tool(targetconfig) | -| bazel cquery "//x:tool" --universe\_scope="//x:my\_gen" | //x:my\_gen | //x:tool(execconfig) | - -If this flag is set, its contents are built. *If it's not set, all targets mentioned in the query expression are built* instead. The transitive closure of the built targets are used as the universe of the query. Either way, the targets to be built must be buildable at the top level (that is, compatible with top-level options). `cquery` returns results in the transitive closure of these top-level targets. - -Even if it's possible to build all targets in a query expression at the top level, it may be beneficial to not do so. For example, explicitly setting `--universe_scope` could prevent building targets multiple times in configurations you don't care about. It could also help specify which configuration version of a target you're looking for. You should set this flag if your query expression is more complex than `deps(//foo)`. +Genrules configure their tools in the +[exec configuration](/extending/rules#configurations) +so the following queries would produce the following outputs: + + + + + + + + + + + + + + + + + + + + + +
QueryTarget BuiltOutput
bazel cquery "//x:tool"//x:tool//x:tool(targetconfig)
bazel cquery "//x:tool" --universe_scope="//x:my_gen"//x:my_gen//x:tool(execconfig)
+ +If this flag is set, its contents are built. _If it's not set, all targets +mentioned in the query expression are built_ instead. The transitive closure of +the built targets are used as the universe of the query. Either way, the targets +to be built must be buildable at the top level (that is, compatible with +top-level options). `cquery` returns results in the transitive closure of these +top-level targets. + +Even if it's possible to build all targets in a query expression at the top +level, it may be beneficial to not do so. For example, explicitly setting +`--universe_scope` could prevent building targets multiple times in +configurations you don't care about. It could also help specify which +configuration version of a target you're looking for. You should set this flag +if your query expression is more complex than `deps(//foo)`. #### `--implicit_deps` (boolean, default=True) -Setting this flag to false filters out all results that aren't explicitly set in the BUILD file and instead set elsewhere by Bazel. This includes filtering resolved toolchains. +Setting this flag to false filters out all results that aren't explicitly set in +the BUILD file and instead set elsewhere by Bazel. This includes filtering +resolved toolchains. #### `--tool_deps` (boolean, default=True) -Setting this flag to false filters out all configured targets for which the path from the queried target to them crosses a transition between the target configuration and the [non-target configurations](/extending/rules#configurations). If the queried target is in the target configuration, setting `--notool_deps` will only return targets that also are in the target configuration. If the queried target is in a non-target configuration, setting `--notool_deps` will only return targets also in non-target configurations. This setting generally does not affect filtering of resolved toolchains. +Setting this flag to false filters out all configured targets for which the +path from the queried target to them crosses a transition between the target +configuration and the +[non-target configurations](/extending/rules#configurations). If the queried +target is in the target configuration, setting `--notool_deps` will only return +targets that also are in the target configuration. If the queried target is in a +non-target configuration, setting `--notool_deps` will only return targets also +in non-target configurations. This setting generally does not affect filtering +of resolved toolchains. #### `--include_aspects` (boolean, default=True) Include dependencies added by [aspects](/extending/aspects). -If this flag is disabled, `cquery somepath(X, Y)` and `cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. +If this flag is disabled, `cquery somepath(X, Y)` and +`cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. ## Output formats -By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. There are other options for exposing the results as well. +By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. +There are other options for exposing the results as well. ### Transitions @@ -202,11 +296,22 @@ By default, cquery outputs results in a dependency-ordered list of label and con --transitions=full ``` -Configuration [transitions](/extending/rules#configurations) are used to build targets underneath the top level targets in different configurations than the top level targets. +Configuration [transitions](/extending/rules#configurations) +are used to build targets underneath the top level targets in different +configurations than the top level targets. -For example, a target might impose a transition to the exec configuration on all dependencies in its `tools` attribute. These are known as attribute transitions. Rules can also impose transitions on their own configurations, known as rule class transitions. This output format outputs information about these transitions such as what type they are and the effect they have on build options. +For example, a target might impose a transition to the exec configuration on all +dependencies in its `tools` attribute. These are known as attribute +transitions. Rules can also impose transitions on their own configurations, +known as rule class transitions. This output format outputs information about +these transitions such as what type they are and the effect they have on build +options. -This output format is triggered by the `--transitions` flag which by default is set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs information about rule class transitions and attribute transitions including a detailed diff of the options before and after the transition. `LITE` mode outputs the same information without the options diff. +This output format is triggered by the `--transitions` flag which by default is +set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs +information about rule class transitions and attribute transitions including a +detailed diff of the options before and after the transition. `LITE` mode +outputs the same information without the options diff. ### Protocol message output @@ -214,17 +319,27 @@ This output format is triggered by the `--transitions` flag which by default is --output=proto ``` -This option causes the resulting targets to be printed in a binary protocol buffer form. The definition of the protocol buffer can be found at [src/main/protobuf/analysis\_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). +This option causes the resulting targets to be printed in a binary protocol +buffer form. The definition of the protocol buffer can be found at +[src/main/protobuf/analysis_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). -`CqueryResult` is the top level message containing the results of the cquery. It has a list of `ConfiguredTarget` messages and a list of `Configuration` messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal to that of the `id` field from the corresponding `Configuration` message. +`CqueryResult` is the top level message containing the results of the cquery. It +has a list of `ConfiguredTarget` messages and a list of `Configuration` +messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal +to that of the `id` field from the corresponding `Configuration` message. -#### --\[no]proto:include\_configurations +#### --[no]proto:include_configurations -By default, cquery results return configuration information as part of each configured target. If you'd like to omit this information and get proto output that is formatted exactly like query's proto output, set this flag to false. +By default, cquery results return configuration information as part of each +configured target. If you'd like to omit this information and get proto output +that is formatted exactly like query's proto output, set this flag to false. -See [query's proto output documentation](/query/language#output-formats) for more proto output-related options. +See [query's proto output documentation](/query/language#output-formats) +for more proto output-related options. -Note: While selects are resolved both at the top level of returned targets and within attributes, all possible inputs for selects are still included as `rule_input` fields. +Note: While selects are resolved both at the top level of returned +targets and within attributes, all possible inputs for selects are still +included as `rule_input` fields. ### Graph output @@ -232,7 +347,10 @@ Note: While selects are resolved both at the top level of returned targets and w --output=graph ``` -This option generates output as a Graphviz-compatible .dot file. See `query`'s [graph output documentation](/query/language#display-result-graph) for details. `cquery` also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and [`--graph:factored`](/query/language#graph-factored). +This option generates output as a Graphviz-compatible .dot file. See `query`'s +[graph output documentation](/query/language#display-result-graph) for details. `cquery` +also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and +[`--graph:factored`](/query/language#graph-factored). ### Files output @@ -240,11 +358,23 @@ This option generates output as a Graphviz-compatible .dot file. See `query`'s [ --output=files ``` -This option prints a list of the output files produced by each target matched by the query similar to the list printed at the end of a `bazel build` invocation. The output contains only the files advertised in the requested output groups as determined by the [`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. It does include source files. +This option prints a list of the output files produced by each target matched +by the query similar to the list printed at the end of a `bazel build` +invocation. The output contains only the files advertised in the requested +output groups as determined by the +[`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. +It does include source files. -All paths emitted by this output format are relative to the [execroot](https://bazel.build/remote/output-directories), which can be obtained via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, paths to files in the main repository also resolve relative to the workspace directory. +All paths emitted by this output format are relative to the +[execroot](https://bazel.build/remote/output-directories), which can be obtained +via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, +paths to files in the main repository also resolve relative to the workspace +directory. -Note: The output of `bazel cquery --output=files //pkg:foo` contains the output files of `//pkg:foo` in *all* configurations that occur in the build (also see the [section on target pattern evaluation](#target-pattern-evaluation)). If that is not desired, wrap you query in [`config(..., target)`](#config). +Note: The output of `bazel cquery --output=files //pkg:foo` contains the output +files of `//pkg:foo` in *all* configurations that occur in the build (also see +the [section on target pattern evaluation](#target-pattern-evaluation)). If that +is not desired, wrap you query in [`config(..., target)`](#config). ### Defining the output format using Starlark @@ -252,21 +382,39 @@ Note: The output of `bazel cquery --output=files //pkg:foo` contains the output --output=starlark ``` -This output format calls a [Starlark](/rules/language) function for each configured target in the query result, and prints the value returned by the call. The `--starlark:file` flag specifies the location of a Starlark file that defines a function named `format` with a single parameter, `target`. This function is called for each [Target](/rules/lib/builtins/Target) in the query result. Alternatively, for convenience, you may specify just the body of a function declared as `def format(target): return expr` by using the `--starlark:expr` flag. +This output format calls a [Starlark](/rules/language) +function for each configured target in the query result, and prints the value +returned by the call. The `--starlark:file` flag specifies the location of a +Starlark file that defines a function named `format` with a single parameter, +`target`. This function is called for each [Target](/rules/lib/builtins/Target) +in the query result. Alternatively, for convenience, you may specify just the +body of a function declared as `def format(target): return expr` by using the +`--starlark:expr` flag. #### 'cquery' Starlark dialect -The cquery Starlark environment differs from a BUILD or .bzl file. It includes all core Starlark [built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), plus a few cquery-specific ones described below, but not (for example) `glob`, `native`, or `rule`, and it does not support load statements. +The cquery Starlark environment differs from a BUILD or .bzl file. It includes +all core Starlark +[built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), +plus a few cquery-specific ones described below, but not (for example) `glob`, +`native`, or `rule`, and it does not support load statements. -##### build\_options(target) +##### build_options(target) -`build_options(target)` returns a map whose keys are build option identifiers (see [Configurations](/extending/config)) and whose values are their Starlark values. Build options whose values are not legal Starlark values are omitted from this map. +`build_options(target)` returns a map whose keys are build option identifiers +(see [Configurations](/extending/config)) and whose values are their Starlark +values. Build options whose values are not legal Starlark values are omitted +from this map. -If the target is an input file, `build_options(target)` returns None, as input file targets have a null configuration. +If the target is an input file, `build_options(target)` returns None, as input +file targets have a null configuration. ##### providers(target) -`providers(target)` returns a map whose keys are names of [providers](/extending/rules#providers) (for example, `"DefaultInfo"`) and whose values are their Starlark values. Providers whose values are not legal Starlark values are omitted from this map. +`providers(target)` returns a map whose keys are names of +[providers](/extending/rules#providers) +(for example, `"DefaultInfo"`) and whose values are their Starlark values. +Providers whose values are not legal Starlark values are omitted from this map. #### Examples @@ -277,7 +425,8 @@ Print a space-separated list of the base names of all files produced by `//foo`: --starlark:expr="' '.join([f.basename for f in providers(target)['DefaultInfo'].files.to_list()])" ``` -Print a space-separated list of the paths of all files produced by **rule** targets in `//bar` and its subpackages: +Print a space-separated list of the paths of all files produced by **rule** targets in +`//bar` and its subpackages: ``` bazel cquery 'kind(rule, //bar/...)' --output=starlark \ @@ -305,7 +454,8 @@ Print the value of the command line option `--javacopt` when building `//foo`. --starlark:expr="build_options(target)['//command_line_option:javacopt']" ``` -Print the label of each target with exactly one output. This example uses Starlark functions defined in a file. +Print the label of each target with exactly one output. This example uses +Starlark functions defined in a file. ``` $ cat example.cquery @@ -322,7 +472,8 @@ Print the label of each target with exactly one output. This example uses Starla $ bazel cquery //baz --output=starlark --starlark:file=example.cquery ``` -Print the label of each target which is strictly Python 3. This example uses Starlark functions defined in a file. +Print the label of each target which is strictly Python 3. This example uses +Starlark functions defined in a file. ``` $ cat example.cquery @@ -343,7 +494,7 @@ Extract a value from a user defined Provider. ``` $ cat some_package/my_rule.bzl - MyRuleInfo = provider(fields={"color": "the name of a color"}) + MyRuleInfo = provider(fields=\{"color": "the name of a color"\}) def _my_rule_impl(ctx): ... @@ -368,51 +519,96 @@ Extract a value from a user defined Provider. ## cquery vs. query -`cquery` and `query` complement each other and excel in different niches. Consider the following to decide which is right for you: - -- `cquery` follows specific `select()` branches to model the exact graph you build. `query` doesn't know which branch the build chooses, so overapproximates by including all branches. -- `cquery`'s precision requires building more of the graph than `query` does. Specifically, `cquery` evaluates *configured targets* while `query` only evaluates *targets*. This takes more time and uses more memory. -- `cquery`'s interpretation of the [query language](/query/language) introduces ambiguity that `query` avoids. For example, if `"//foo"` exists in two configurations, which one should `cquery "deps(//foo)"` use? The [`config`](#config) function can help with this. +`cquery` and `query` complement each other and excel in +different niches. Consider the following to decide which is right for you: + +* `cquery` follows specific `select()` branches to + model the exact graph you build. `query` doesn't know which + branch the build chooses, so overapproximates by including all branches. +* `cquery`'s precision requires building more of the graph than + `query` does. Specifically, `cquery` + evaluates _configured targets_ while `query` only + evaluates _targets_. This takes more time and uses more memory. +* `cquery`'s interpretation of + the [query language](/query/language) introduces ambiguity + that `query` avoids. For example, + if `"//foo"` exists in two configurations, which one + should `cquery "deps(//foo)"` use? + The [`config`](#config) function can help with this. ## Non-deterministic output -`cquery` does not automatically wipe the build graph from previous commands. It's therefore prone to picking up results from past queries. +`cquery` does not automatically wipe the build graph from previous commands. +It's therefore prone to picking up results from past queries. -For example, `genrule` exerts an exec transition on its `tools` attribute - that is, it configures its tools in the \[exec configuration] ([https://bazel.build/rules/rules#configurations](https://bazel.build/rules/rules#configurations)). +For example, `genrule` exerts an exec transition on its `tools` attribute - +that is, it configures its tools in the [exec configuration] +(https://bazel.build/rules/rules#configurations). You can see the lingering effects of that transition below. ``` -$ cat > foo/BUILD << +$ cat > foo/BUILD << /tmp/deps.svg ``` -Note: `dot` supports other image formats, just replace `svg` with the format identifier, for example, `png`. +Note: `dot` supports other image formats, just replace `svg` with the +format identifier, for example, `png`. When a dependency graph is big and complicated, it can be helpful start with a single path: @@ -46,7 +62,8 @@ $ bazel query "somepath(//foo:foo, third_party/zlib:zlibonly)" //third_party/zlib:zlibonly ``` -If you do not specify `--output graph` with `allpaths`, you will get a flattened list of the dependency graph. +If you do not specify `--output graph` with `allpaths`, +you will get a flattened list of the dependency graph. ``` $ bazel query "allpaths(//foo, third_party/...)" @@ -76,15 +93,27 @@ $ bazel query "allpaths(//foo, third_party/...)" ### Aside: implicit dependencies -The BUILD file for `//foo` never references `//translations/tools:aggregator`. So, where's the direct dependency? +The BUILD file for `//foo` never references +`//translations/tools:aggregator`. So, where's the direct dependency? -Certain rules include implicit dependencies on additional libraries or tools. For example, to build a `genproto` rule, you need first to build the Protocol Compiler, so every `genproto` rule carries an implicit dependency on the protocol compiler. These dependencies are not mentioned in the build file, but added in by the build tool. The full set of implicit dependencies is currently undocumented. Using `--noimplicit_deps` allows you to filter out these deps from your query results. For cquery, this will include resolved toolchains. +Certain rules include implicit dependencies on additional libraries or tools. +For example, to build a `genproto` rule, you need first to build the Protocol +Compiler, so every `genproto` rule carries an implicit dependency on the +protocol compiler. These dependencies are not mentioned in the build file, +but added in by the build tool. The full set of implicit dependencies is + currently undocumented. Using `--noimplicit_deps` allows you to filter out + these deps from your query results. For cquery, this will include resolved toolchains. ## Reverse dependencies -You might want to know the set of targets that depends on some target. For instance, if you're going to change some code, you might want to know what other code you're about to break. You can use `rdeps(u, x)` to find the reverse dependencies of the targets in `x` within the transitive closure of `u`. +You might want to know the set of targets that depends on some target. For instance, +if you're going to change some code, you might want to know what other code +you're about to break. You can use `rdeps(u, x)` to find the reverse +dependencies of the targets in `x` within the transitive closure of `u`. -Bazel's [Sky Query](/query/language#sky-query) supports the `allrdeps` function which allows you to query reverse dependencies in a universe you specify. +Bazel's [Sky Query](/query/language#sky-query) +supports the `allrdeps` function which allows you to query reverse dependencies +in a universe you specify. ## Miscellaneous uses @@ -94,47 +123,33 @@ You can use `bazel query` to analyze many dependency relationships. #### What packages exist beneath `foo`? -``` -bazel query 'foo/...' --output package -``` +```bazel query 'foo/...' --output package``` #### What rules are defined in the `foo` package? -``` -bazel query 'kind(rule, foo:*)' --output label_kind -``` +```bazel query 'kind(rule, foo:*)' --output label_kind``` #### What files are generated by rules in the `foo` package? -``` -bazel query 'kind("generated file", //foo:*)' -``` +```bazel query 'kind("generated file", //foo:*)'``` #### What targets are generated by starlark macro `foo`? -``` -bazel query 'attr(generator_function, foo, //path/to/search/...)' -``` +```bazel query 'attr(generator_function, foo, //path/to/search/...)'``` #### What's the set of BUILD files needed to build `//foo`? -``` -bazel query 'buildfiles(deps(//foo))' | cut -f1 -d: -``` +```bazel query 'buildfiles(deps(//foo))' | cut -f1 -d:``` #### What are the individual tests that a `test_suite` expands to? -``` -bazel query 'tests(//foo:smoke_tests)' -``` +```bazel query 'tests(//foo:smoke_tests)'``` #### Which of those are C++ tests? -``` -bazel query 'kind(cc_.*, tests(//foo:smoke_tests))' -``` +```bazel query 'kind(cc_.*, tests(//foo:smoke_tests))'``` -#### Which of those are small? Medium? Large? +#### Which of those are small? Medium? Large? ``` bazel query 'attr(size, small, tests(//foo:smoke_tests))' @@ -146,27 +161,19 @@ bazel query 'attr(size, large, tests(//foo:smoke_tests))' #### What are the tests beneath `foo` that match a pattern? -``` -bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))' -``` +```bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))'``` The pattern is a regex and is applied to the full name of the rule. It's similar to doing -``` -bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t' -``` +```bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t'``` #### What package contains file `path/to/file/bar.java`? -``` - bazel query path/to/file/bar.java --output=package -``` +``` bazel query path/to/file/bar.java --output=package``` #### What is the build label for `path/to/file/bar.java?` -``` -bazel query path/to/file/bar.java -``` +```bazel query path/to/file/bar.java``` #### What rule target(s) contain file `path/to/file/bar.java` as a source? @@ -179,36 +186,28 @@ bazel query "attr('srcs', $fullname, ${fullname//:*/}:*)" #### What packages does `foo` depend on? (What do I need to check out to build `foo`) -``` -bazel query 'buildfiles(deps(//foo:foo))' --output package -``` +```bazel query 'buildfiles(deps(//foo:foo))' --output package``` -Note: `buildfiles` is required in order to correctly obtain all files referenced by `subinclude`; see the reference manual for details. +Note: `buildfiles` is required in order to correctly obtain all files +referenced by `subinclude`; see the reference manual for details. #### What packages does the `foo` tree depend on, excluding `foo/contrib`? -``` -bazel query 'deps(foo/... except foo/contrib/...)' --output package -``` +```bazel query 'deps(foo/... except foo/contrib/...)' --output package``` ### What rule dependencies exist ... #### What genproto rules does bar depend upon? -``` -bazel query 'kind(genproto, deps(bar/...))' -``` +```bazel query 'kind(genproto, deps(bar/...))'``` #### Find the definition of some JNI (C++) library that is transitively depended upon by a Java binary rule in the servlet tree. -``` -bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location -``` +```bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location``` ##### ...Now find the definitions of all the Java binaries that depend on them -``` -bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in +```bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in let cls = kind(cc_.*library, deps($jbs)) in $jbs intersect allpaths($jbs, $cls)' ``` @@ -219,67 +218,54 @@ bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in Source files: -``` -bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$ -``` +```bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$``` Generated files: -``` -bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$ -``` +```bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$``` -#### What is the complete set of Java source files required to build QUX's tests? +#### What is the complete set of Java source files required to build QUX's tests? Source files: -``` -bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ -``` +```bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` Generated files: -``` -bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ -``` +```bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` ### What differences in dependencies between X and Y exist ... #### What targets does `//foo` depend on that `//foo:foolib` does not? -``` -bazel query 'deps(//foo) except deps(//foo:foolib)' -``` +```bazel query 'deps(//foo) except deps(//foo:foolib)'``` -#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does *not* depend on? +#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does _not_ depend on? -``` -bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))' -``` +```bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))'``` ### Why does this dependency exist ... #### Why does `bar` depend on `groups2`? -``` -bazel query 'somepath(bar/...,groups2/...:*)' -``` +```bazel query 'somepath(bar/...,groups2/...:*)'``` -Once you have the results of this query, you will often find that a single target stands out as being an unexpected or egregious and undesirable dependency of `bar`. The query can then be further refined to: +Once you have the results of this query, you will often find that a single +target stands out as being an unexpected or egregious and undesirable +dependency of `bar`. The query can then be further refined to: #### Show me a path from `docker/updater:updater_systest` (a `py_test`) to some `cc_library` that it depends upon: -``` -bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in - somepath(docker/updater:updater_systest, $cc)' -``` +```bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in + somepath(docker/updater:updater_systest, $cc)'``` #### Why does library `//photos/frontend:lib` depend on two variants of the same library `//third_party/jpeglib` and `//third_party/jpeg`? -This query boils down to: "show me the subgraph of `//photos/frontend:lib` that depends on both libraries". When shown in topological order, the last element of the result is the most likely culprit. +This query boils down to: "show me the subgraph of `//photos/frontend:lib` that +depends on both libraries". When shown in topological order, the last element +of the result is the most likely culprit. -``` -bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) +```bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) intersect allpaths(//photos/frontend:lib, //third_party/jpeg)' //photos/frontend:lib @@ -291,41 +277,41 @@ bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) //third_party/jpeg/img:renderer ``` -### What depends on ... +### What depends on ... #### What rules under bar depend on Y? -``` -bazel query 'bar/... intersect allpaths(bar/..., Y)' -``` +```bazel query 'bar/... intersect allpaths(bar/..., Y)'``` -Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X depend on Y?" If expression X is non-trivial, it may be convenient to bind a name to it using `let` to avoid duplication. +Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X +depend on Y?" If expression X is non-trivial, it may be convenient to bind a +name to it using `let` to avoid duplication. #### What targets directly depend on T, in T's package? -``` -bazel query 'same_pkg_direct_rdeps(T)' -``` +```bazel query 'same_pkg_direct_rdeps(T)'``` ### How do I break a dependency ... +{/* TODO find a convincing value of X to plug in here */} + #### What dependency paths do I have to break to make `bar` no longer depend on X? To output the graph to a `svg` file: -``` -bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg -``` +```bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg``` ### Misc #### How many sequential steps are there in the `//foo-tests` build? -Unfortunately, the query language can't currently give you the longest path from x to y, but it can find the (or rather *a*) most distant node from the starting point, or show you the *lengths* of the longest path from x to every y that it depends on. Use `maxrank`: +Unfortunately, the query language can't currently give you the longest path +from x to y, but it can find the (or rather _a_) most distant node from the +starting point, or show you the _lengths_ of the longest path from x to every +y that it depends on. Use `maxrank`: -``` -bazel query 'deps(//foo-tests)' --output maxrank | tail -1 -85 //third_party/zlib:zutil.c -``` +```bazel query 'deps(//foo-tests)' --output maxrank | tail -1 +85 //third_party/zlib:zutil.c``` -The result indicates that there exist paths of length 85 that must occur in order in this build. +The result indicates that there exist paths of length 85 that must occur in +order in this build. diff --git a/reference/glossary.mdx b/reference/glossary.mdx index f045f9e6..da7478c3 100644 --- a/reference/glossary.mdx +++ b/reference/glossary.mdx @@ -2,155 +2,289 @@ title: 'Bazel Glossary' --- + + ### Action -A command to run during the build, for example, a call to a compiler that takes [artifacts](#artifact) as inputs and produces other artifacts as outputs. Includes metadata like the command line arguments, action key, environment variables, and declared input/output artifacts. +A command to run during the build, for example, a call to a compiler that takes +[artifacts](#artifact) as inputs and produces other artifacts as outputs. +Includes metadata like the command line arguments, action key, environment +variables, and declared input/output artifacts. **See also:** [Rules documentation](/extending/rules#actions) ### Action cache -An on-disk cache that stores a mapping of executed [actions](#action) to the outputs they created. The cache key is known as the [action key](#action-key). A core component for Bazel's incrementality model. The cache is stored in the output base directory and thus survives Bazel server restarts. +An on-disk cache that stores a mapping of executed [actions](#action) to the +outputs they created. The cache key is known as the [action key](#action-key). A +core component for Bazel's incrementality model. The cache is stored in the +output base directory and thus survives Bazel server restarts. ### Action graph -An in-memory graph of [actions](#action) and the [artifacts](#artifact) that these actions read and generate. The graph might include artifacts that exist as source files (for example, in the file system) as well as generated intermediate/final artifacts that are not mentioned in `BUILD` files. Produced during the [analysis phase](#analysis-phase) and used during the [execution phase](#execution-phase). +An in-memory graph of [actions](#action) and the [artifacts](#artifact) that +these actions read and generate. The graph might include artifacts that exist as +source files (for example, in the file system) as well as generated +intermediate/final artifacts that are not mentioned in `BUILD` files. Produced +during the [analysis phase](#analysis-phase) and used during the [execution +phase](#execution-phase). ### Action graph query (aquery) -A [query](#query-concept) tool that can query over build [actions](#action). This provides the ability to analyze how [build rules](#rule) translate into the actual work builds do. +A [query](#query-concept) tool that can query over build [actions](#action). +This provides the ability to analyze how [build rules](#rule) translate into the +actual work builds do. ### Action key -The cache key of an [action](#action). Computed based on action metadata, which might include the command to be executed in the action, compiler flags, library locations, or system headers, depending on the action. Enables Bazel to cache or invalidate individual actions deterministically. +The cache key of an [action](#action). Computed based on action metadata, which +might include the command to be executed in the action, compiler flags, library +locations, or system headers, depending on the action. Enables Bazel to cache or +invalidate individual actions deterministically. ### Analysis phase -The second phase of a build. Processes the [target graph](#target-graph) specified in [`BUILD` files](#build-file) to produce an in-memory [action graph](#action-graph) that determines the order of actions to run during the [execution phase](#execution-phase). This is the phase in which rule implementations are evaluated. +The second phase of a build. Processes the [target graph](#target-graph) +specified in [`BUILD` files](#build-file) to produce an in-memory [action +graph](#action-graph) that determines the order of actions to run during the +[execution phase](#execution-phase). This is the phase in which rule +implementations are evaluated. ### Artifact -A source file or a generated file. Can also be a directory of files, known as [tree artifacts](#tree-artifact). +A source file or a generated file. Can also be a directory of files, known as +[tree artifacts](#tree-artifact). -An artifact may be an input to multiple actions, but must only be generated by at most one action. +An artifact may be an input to multiple actions, but must only be generated by +at most one action. -An artifact that corresponds to a [file target](#target) can be addressed by a label. +An artifact that corresponds to a [file target](#target) can be addressed by a +label. ### Aspect -A mechanism for rules to create additional [actions](#action) in their dependencies. For example, if target A depends on B, one can apply an aspect on A that traverses *up* a dependency edge to B, and runs additional actions in B to generate and collect additional output files. These additional actions are cached and reused between targets requiring the same aspect. Created with the `aspect()` Starlark Build API function. Can be used, for example, to generate metadata for IDEs, and create actions for linting. +A mechanism for rules to create additional [actions](#action) in their +dependencies. For example, if target A depends on B, one can apply an aspect on +A that traverses *up* a dependency edge to B, and runs additional actions in B +to generate and collect additional output files. These additional actions are +cached and reused between targets requiring the same aspect. Created with the +`aspect()` Starlark Build API function. Can be used, for example, to generate +metadata for IDEs, and create actions for linting. **See also:** [Aspects documentation](/extending/aspects) ### Aspect-on-aspect -A composition mechanism whereby aspects can be applied to the results of other aspects. For example, an aspect that generates information for use by IDEs can be applied on top of an aspect that generates `.java` files from a proto. +A composition mechanism whereby aspects can be applied to the results +of other aspects. For example, an aspect that generates information for use by +IDEs can be applied on top of an aspect that generates `.java` files from a +proto. -For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that `B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) attribute. +For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that +`B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute +must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) +attribute. ### Attribute -A parameter to a [rule](#rule), used to express per-target build information. Examples include `srcs`, `deps`, and `copts`, which respectively declare a target's source files, dependencies, and custom compiler options. The particular attributes available for a given target depend on its rule type. +A parameter to a [rule](#rule), used to express per-target build information. +Examples include `srcs`, `deps`, and `copts`, which respectively declare a +target's source files, dependencies, and custom compiler options. The particular +attributes available for a given target depend on its rule type. ### .bazelrc -Bazel’s configuration file used to change the default values for [startup flags](#startup-flags) and [command flags](#command-flags), and to define common groups of options that can then be set together on the Bazel command line using a `--config` flag. Bazel can combine settings from multiple bazelrc files (systemwide, per-workspace, per-user, or from a custom location), and a `bazelrc` file may also import settings from other `bazelrc` files. +Bazel’s configuration file used to change the default values for [startup +flags](#startup-flags) and [command flags](#command-flags), and to define common +groups of options that can then be set together on the Bazel command line using +a `--config` flag. Bazel can combine settings from multiple bazelrc files +(systemwide, per-workspace, per-user, or from a custom location), and a +`bazelrc` file may also import settings from other `bazelrc` files. ### Blaze -The Google-internal version of Bazel. Google’s main build system for its mono-repository. +The Google-internal version of Bazel. Google’s main build system for its +mono-repository. ### BUILD File -A `BUILD` file is the main configuration file that tells Bazel what software outputs to build, what their dependencies are, and how to build them. Bazel takes a `BUILD` file as input and uses the file to create a graph of dependencies and to derive the actions that must be completed to build intermediate and final software outputs. A `BUILD` file marks a directory and any sub-directories not containing a `BUILD` file as a [package](#package), and can contain [targets](#target) created by [rules](#rule). The file can also be named `BUILD.bazel`. +A `BUILD` file is the main configuration file that tells Bazel what software +outputs to build, what their dependencies are, and how to build them. Bazel +takes a `BUILD` file as input and uses the file to create a graph of dependencies +and to derive the actions that must be completed to build intermediate and final +software outputs. A `BUILD` file marks a directory and any sub-directories not +containing a `BUILD` file as a [package](#package), and can contain +[targets](#target) created by [rules](#rule). The file can also be named +`BUILD.bazel`. ### BUILD.bazel File -See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same directory. +See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same +directory. ### .bzl File -A file that defines rules, [macros](#macro), and constants written in [Starlark](#starlark). These can then be imported into [`BUILD` files](#build-file) using the `load()` function. +A file that defines rules, [macros](#macro), and constants written in +[Starlark](#starlark). These can then be imported into [`BUILD` +files](#build-file) using the `load()` function. + +{/* TODO: ### Build event protocol */} + +{/* TODO: ### Build flag */} ### Build graph -The dependency graph that Bazel constructs and traverses to perform a build. Includes nodes like [targets](#target), [configured targets](#configured-target), [actions](#action), and [artifacts](#artifact). A build is considered complete when all [artifacts](#artifact) on which a set of requested targets depend are verified as up-to-date. +The dependency graph that Bazel constructs and traverses to perform a build. +Includes nodes like [targets](#target), [configured +targets](#configured-target), [actions](#action), and [artifacts](#artifact). A +build is considered complete when all [artifacts](#artifact) on which a set of +requested targets depend are verified as up-to-date. ### Build setting -A Starlark-defined piece of [configuration](#configuration). [Transitions](#transition) can set build settings to change a subgraph's configuration. If exposed to the user as a [command-line flag](#command-flags), also known as a build flag. +A Starlark-defined piece of [configuration](#configuration). +[Transitions](#transition) can set build settings to change a subgraph's +configuration. If exposed to the user as a [command-line flag](#command-flags), +also known as a build flag. ### Clean build -A build that doesn't use the results of earlier builds. This is generally slower than an [incremental build](#incremental-build) but commonly considered to be more [correct](#correctness). Bazel guarantees both clean and incremental builds are always correct. +A build that doesn't use the results of earlier builds. This is generally slower +than an [incremental build](#incremental-build) but commonly considered to be +more [correct](#correctness). Bazel guarantees both clean and incremental builds +are always correct. ### Client-server model -The `bazel` command-line client automatically starts a background server on the local machine to execute Bazel [commands](#command). The server persists across commands but automatically stops after a period of inactivity (or explicitly via bazel shutdown). Splitting Bazel into a server and client helps amortize JVM startup time and supports faster [incremental builds](#incremental-build) because the [action graph](#action-graph) remains in memory across commands. +The `bazel` command-line client automatically starts a background server on the +local machine to execute Bazel [commands](#command). The server persists across +commands but automatically stops after a period of inactivity (or explicitly via +bazel shutdown). Splitting Bazel into a server and client helps amortize JVM +startup time and supports faster [incremental builds](#incremental-build) +because the [action graph](#action-graph) remains in memory across commands. ### Command -Used on the command line to invoke different Bazel functions, like `bazel build`, `bazel test`, `bazel run`, and `bazel query`. +Used on the command line to invoke different Bazel functions, like `bazel +build`, `bazel test`, `bazel run`, and `bazel query`. ### Command flags -A set of flags specific to a [command](#command). Command flags are specified *after* the command (`bazel build <command flags>`). Flags can be applicable to one or more commands. For example, `--configure` is a flag exclusively for the `bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, `test` and more. Flags are often used for [configuration](#configuration) purposes, so changes in flag values can cause Bazel to invalidate in-memory graphs and restart the [analysis phase](#analysis-phase). +A set of flags specific to a [command](#command). Command flags are specified +*after* the command (`bazel build `). Flags can be applicable to +one or more commands. For example, `--configure` is a flag exclusively for the +`bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, +`test` and more. Flags are often used for [configuration](#configuration) +purposes, so changes in flag values can cause Bazel to invalidate in-memory +graphs and restart the [analysis phase](#analysis-phase). ### Configuration -Information outside of [rule](#rule) definitions that impacts how rules generate [actions](#action). Every build has at least one configuration specifying the target platform, action environment variables, and command-line [build flags](#command-flags). [Transitions](#transition) may create additional configurations, such as for host tools or cross-compilation. +Information outside of [rule](#rule) definitions that impacts how rules generate +[actions](#action). Every build has at least one configuration specifying the +target platform, action environment variables, and command-line [build +flags](#command-flags). [Transitions](#transition) may create additional +configurations, such as for host tools or cross-compilation. **See also:** [Configurations](/extending/rules#configurations) +{/* TODO: ### Configuration fragment */} + ### Configuration trimming -The process of only including the pieces of [configuration](#configuration) a target actually needs. For example, if you build Java binary `//:j` with C++ dependency `//:c`, it's wasteful to include the value of `--javacopt` in the configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ build cacheability. +The process of only including the pieces of [configuration](#configuration) a +target actually needs. For example, if you build Java binary `//:j` with C++ +dependency `//:c`, it's wasteful to include the value of `--javacopt` in the +configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ +build cacheability. ### Configured query (cquery) -A [query](#query-concept) tool that queries over [configured targets](#configured-target) (after the [analysis phase](#analysis-phase) completes). This means `select()` and [build flags](#command-flags) (such as `--platforms`) are accurately reflected in the results. +A [query](#query-concept) tool that queries over [configured +targets](#configured-target) (after the [analysis phase](#analysis-phase) +completes). This means `select()` and [build flags](#command-flags) (such as +`--platforms`) are accurately reflected in the results. **See also:** [cquery documentation](/query/cquery) ### Configured target -The result of evaluating a [target](#target) with a [configuration](#configuration). The [analysis phase](#analysis-phase) produces this by combining the build's options with the targets that need to be built. For example, if `//:foo` builds for two different architectures in the same build, it has two configured targets: `<//:foo, x86>` and `<//:foo, arm>`. +The result of evaluating a [target](#target) with a +[configuration](#configuration). The [analysis phase](#analysis-phase) produces +this by combining the build's options with the targets that need to be built. +For example, if `//:foo` builds for two different architectures in the same +build, it has two configured targets: `` and ``. ### Correctness -A build is correct when its output faithfully reflects the state of its transitive inputs. To achieve correct builds, Bazel strives to be [hermetic](#hermeticity), reproducible, and making [build analysis](#analysis-phase) and [action execution](#execution-phase) deterministic. +A build is correct when its output faithfully reflects the state of its +transitive inputs. To achieve correct builds, Bazel strives to be +[hermetic](#hermeticity), reproducible, and making [build +analysis](#analysis-phase) and [action execution](#execution-phase) +deterministic. ### Dependency -A directed edge between two [targets](#target). A target `//:foo` has a *target dependency* on target `//:bar` if `//:foo`'s attribute values contain a reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an action in `//:foo` depends on an input [artifact](#artifact) created by an action in `//:bar`. +A directed edge between two [targets](#target). A target `//:foo` has a *target +dependency* on target `//:bar` if `//:foo`'s attribute values contain a +reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an +action in `//:foo` depends on an input [artifact](#artifact) created by an +action in `//:bar`. -In certain contexts, it could also refer to an *external dependency*; see [modules](#module). +In certain contexts, it could also refer to an _external dependency_; see +[modules](#module). ### Depset -A data structure for collecting data on transitive dependencies. Optimized so that merging depsets is time and space efficient, because it’s common to have very large depsets (hundreds of thousands of files). Implemented to recursively refer to other depsets for space efficiency reasons. [Rule](#rule) implementations should not "flatten" depsets by converting them to lists unless the rule is at the top level of the build graph. Flattening large depsets incurs huge memory consumption. Also known as *nested sets* in Bazel's internal implementation. +A data structure for collecting data on transitive dependencies. Optimized so +that merging depsets is time and space efficient, because it’s common to have +very large depsets (hundreds of thousands of files). Implemented to +recursively refer to other depsets for space efficiency reasons. [Rule](#rule) +implementations should not "flatten" depsets by converting them to lists unless +the rule is at the top level of the build graph. Flattening large depsets incurs +huge memory consumption. Also known as *nested sets* in Bazel's internal +implementation. **See also:** [Depset documentation](/extending/depsets) ### Disk cache -A local on-disk blob store for the remote caching feature. Can be used in conjunction with an actual remote blob store. +A local on-disk blob store for the remote caching feature. Can be used in +conjunction with an actual remote blob store. ### Distdir -A read-only directory containing files that Bazel would otherwise fetch from the internet using repository rules. Enables builds to run fully offline. +A read-only directory containing files that Bazel would otherwise fetch from the +internet using repository rules. Enables builds to run fully offline. ### Dynamic execution -An execution strategy that selects between local and remote execution based on various heuristics, and uses the execution results of the faster successful method. Certain [actions](#action) are executed faster locally (for example, linking) and others are faster remotely (for example, highly parallelizable compilation). A dynamic execution strategy can provide the best possible incremental and clean build times. +An execution strategy that selects between local and remote execution based on +various heuristics, and uses the execution results of the faster successful +method. Certain [actions](#action) are executed faster locally (for example, +linking) and others are faster remotely (for example, highly parallelizable +compilation). A dynamic execution strategy can provide the best possible +incremental and clean build times. ### Execution phase -The third phase of a build. Executes the [actions](#action) in the [action graph](#action-graph) created during the [analysis phase](#analysis-phase). These actions invoke executables (compilers, scripts) to read and write [artifacts](#artifact). *Spawn strategies* control how these actions are executed: locally, remotely, dynamically, sandboxed, docker, and so on. +The third phase of a build. Executes the [actions](#action) in the [action +graph](#action-graph) created during the [analysis phase](#analysis-phase). +These actions invoke executables (compilers, scripts) to read and write +[artifacts](#artifact). *Spawn strategies* control how these actions are +executed: locally, remotely, dynamically, sandboxed, docker, and so on. ### Execution root -A directory in the [workspace](#workspace)’s [output base](#output-base) directory where local [actions](#action) are executed in non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks of input [artifacts](#artifact) from the workspace. The execution root also contains symlinks to external repositories as other inputs and the `bazel-out` directory to store outputs. Prepared during the [loading phase](#loading-phase) by creating a *symlink forest* of the directories that represent the transitive closure of packages on which a build depends. Accessible with `bazel info execution_root` on the command line. +A directory in the [workspace](#workspace)’s [output base](#output-base) +directory where local [actions](#action) are executed in +non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks +of input [artifacts](#artifact) from the workspace. The execution root also +contains symlinks to external repositories as other inputs and the `bazel-out` +directory to store outputs. Prepared during the [loading phase](#loading-phase) +by creating a *symlink forest* of the directories that represent the transitive +closure of packages on which a build depends. Accessible with `bazel info +execution_root` on the command line. ### File @@ -158,27 +292,52 @@ See [Artifact](#artifact). ### Hermeticity -A build is hermetic if there are no external influences on its build and test operations, which helps to make sure that results are deterministic and [correct](#correctness). For example, hermetic builds typically disallow network access to actions, restrict access to declared inputs, use fixed timestamps and timezones, restrict access to environment variables, and use fixed seeds for random number generators +A build is hermetic if there are no external influences on its build and test +operations, which helps to make sure that results are deterministic and +[correct](#correctness). For example, hermetic builds typically disallow network +access to actions, restrict access to declared inputs, use fixed timestamps and +timezones, restrict access to environment variables, and use fixed seeds for +random number generators ### Incremental build -An incremental build reuses the results of earlier builds to reduce build time and resource usage. Dependency checking and caching aim to produce correct results for this type of build. An incremental build is the opposite of a clean build. +An incremental build reuses the results of earlier builds to reduce build time +and resource usage. Dependency checking and caching aim to produce correct +results for this type of build. An incremental build is the opposite of a clean +build. + +{/* TODO: ### Install base */} ### Label -An identifier for a [target](#target). Generally has the form `@repo//path/to/package:target`, where `repo` is the (apparent) name of the [repository](#repository) containing the target, `path/to/package` is the path to the directory that contains the [`BUILD` file](#build-file) declaring the target (this directory is also known as the [package](#package)), and `target` is the name of the target itself. Depending on the situation, parts of this syntax may be omitted. +An identifier for a [target](#target). Generally has the form +`@repo//path/to/package:target`, where `repo` is the (apparent) name of the +[repository](#repository) containing the target, `path/to/package` is the path +to the directory that contains the [`BUILD` file](#build-file) declaring the +target (this directory is also known as the [package](#package)), and `target` +is the name of the target itself. Depending on the situation, parts of this +syntax may be omitted. **See also**: [Labels](/concepts/labels) ### Loading phase -The first phase of a build where Bazel executes [`BUILD` files](#build-file) to create [packages](#package). [Macros](#macro) and certain functions like `glob()` are evaluated in this phase. Interleaved with the second phase of the build, the [analysis phase](#analysis-phase), to build up a [target graph](#target-graph). +The first phase of a build where Bazel executes [`BUILD` files](#build-file) to +create [packages](#package). [Macros](#macro) and certain functions like +`glob()` are evaluated in this phase. Interleaved with the second phase of the +build, the [analysis phase](#analysis-phase), to build up a [target +graph](#target-graph). ### Legacy macro -A flavor of [macro](#macro) which is declared as an ordinary [Starlark](#starlark) function, and which runs as a side effect of executing a `BUILD` file. +A flavor of [macro](#macro) which is declared as an ordinary +[Starlark](#starlark) function, and which runs as a side effect of executing a +`BUILD` file. -Legacy macros can do anything a function can. This means they can be convenient, but they can also be harder to read, write, and use. A legacy macro might unexpectedly mutate its arguments or fail when given a `select()` or ill-typed argument. +Legacy macros can do anything a function can. This means they can be convenient, +but they can also be harder to read, write, and use. A legacy macro might +unexpectedly mutate its arguments or fail when given a `select()` or ill-typed +argument. Contrast with [symbolic macros](#symbolic-macro). @@ -186,19 +345,34 @@ Contrast with [symbolic macros](#symbolic-macro). ### Macro -A mechanism to compose multiple [rule](#rule) target declarations together under a single [Starlark](#starlark) callable. Enables reusing common rule declaration patterns across `BUILD` files. Expanded to the underlying rule target declarations during the [loading phase](#loading-phase). +A mechanism to compose multiple [rule](#rule) target declarations together under +a single [Starlark](#starlark) callable. Enables reusing common rule declaration +patterns across `BUILD` files. Expanded to the underlying rule target +declarations during the [loading phase](#loading-phase). -Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and [legacy macros](#legacy-macro). +Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and +[legacy macros](#legacy-macro). ### Mnemonic -A short, human-readable string selected by a rule author to quickly understand what an [action](#action) in the rule is doing. Mnemonics can be used as identifiers for *spawn strategy* selections. Some examples of action mnemonics are `Javac` from Java rules, `CppCompile` from C++ rules, and `AndroidManifestMerger` from Android rules. +A short, human-readable string selected by a rule author to quickly understand +what an [action](#action) in the rule is doing. Mnemonics can be used as +identifiers for *spawn strategy* selections. Some examples of action mnemonics +are `Javac` from Java rules, `CppCompile` from C++ rules, and +`AndroidManifestMerger` from Android rules. ### Module -A Bazel project that can have multiple versions, each of which can have dependencies on other modules. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. Modules form the backbone of Bazel's external dependency management system. +A Bazel project that can have multiple versions, each of which can have +dependencies on other modules. This is analogous to familiar concepts in other +dependency management systems, such as a Maven _artifact_, an npm _package_, a +Go _module_, or a Cargo _crate_. Modules form the backbone of Bazel's external +dependency management system. -Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its root. This file contains metadata about the module itself (such as its name and version), its direct dependencies, and various other data including toolchain registrations and [module extension](#module-extension) input. +Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its +root. This file contains metadata about the module itself (such as its name and +version), its direct dependencies, and various other data including toolchain +registrations and [module extension](#module-extension) input. Module metadata is hosted in Bazel registries. @@ -206,180 +380,336 @@ Module metadata is hosted in Bazel registries. ### Module Extension -A piece of logic that can be run to generate [repos](#repository) by reading inputs from across the [module](#module) dependency graph and invoking [repo rules](#repository-rule). Module extensions have capabilities similar to repo rules, allowing them to access the internet, perform file I/O, and so on. +A piece of logic that can be run to generate [repos](#repository) by reading +inputs from across the [module](#module) dependency graph and invoking [repo +rules](#repository-rule). Module extensions have capabilities similar to repo +rules, allowing them to access the internet, perform file I/O, and so on. **See also:** [Module extensions](/external/extension) ### Native rules -[Rules](#rule) that are built into Bazel and implemented in Java. Such rules appear in [`.bzl` files](#bzl-file) as functions in the native module (for example, `native.cc_library` or `native.java_library`). User-defined rules (non-native) are created using [Starlark](#starlark). +[Rules](#rule) that are built into Bazel and implemented in Java. Such rules +appear in [`.bzl` files](#bzl-file) as functions in the native module (for +example, `native.cc_library` or `native.java_library`). User-defined rules +(non-native) are created using [Starlark](#starlark). ### Output base -A [workspace](#workspace)-specific directory to store Bazel output files. Used to separate outputs from the *workspace*'s source tree (the [main repo](#repository)). Located in the [output user root](#output-user-root). +A [workspace](#workspace)-specific directory to store Bazel output files. Used +to separate outputs from the *workspace*'s source tree (the [main +repo](#repository)). Located in the [output user root](#output-user-root). ### Output groups -A group of files that is expected to be built when Bazel finishes building a target. [Rules](#rule) put their usual outputs in the "default output group" (e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` targets). The default output group is the output group whose [artifacts](#artifact) are built when a target is requested on the command line. Rules can define more named output groups that can be explicitly specified in [`BUILD` files](#build-file) (`filegroup` rule) or the command line (`--output_groups` flag). +A group of files that is expected to be built when Bazel finishes building a +target. [Rules](#rule) put their usual outputs in the "default output group" +(e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` +targets). The default output group is the output group whose +[artifacts](#artifact) are built when a target is requested on the command line. +Rules can define more named output groups that can be explicitly specified in +[`BUILD` files](#build-file) (`filegroup` rule) or the command line +(`--output_groups` flag). ### Output user root -A user-specific directory to store Bazel's outputs. The directory name is derived from the user's system username. Prevents output file collisions if multiple users are building the same project on the system at the same time. Contains subdirectories corresponding to build outputs of individual workspaces, also known as [output bases](#output-base). +A user-specific directory to store Bazel's outputs. The directory name is +derived from the user's system username. Prevents output file collisions if +multiple users are building the same project on the system at the same time. +Contains subdirectories corresponding to build outputs of individual workspaces, +also known as [output bases](#output-base). ### Package -The set of [targets](#target) defined by a [`BUILD` file](#build-file). A package's name is the `BUILD` file's path relative to the [repo](#repository) root. A package can contain subpackages, or subdirectories containing `BUILD` files, thus forming a package hierarchy. +The set of [targets](#target) defined by a [`BUILD` file](#build-file). A +package's name is the `BUILD` file's path relative to the [repo](#repository) +root. A package can contain subpackages, or subdirectories containing `BUILD` +files, thus forming a package hierarchy. ### Package group -A [target](#target) representing a set of packages. Often used in `visibility` attribute values. +A [target](#target) representing a set of packages. Often used in `visibility` +attribute values. ### Platform -A "machine type" involved in a build. This includes the machine Bazel runs on (the "host" platform), the machines build tools execute on ("exec" platforms), and the machines targets are built for ("target platforms"). +A "machine type" involved in a build. This includes the machine Bazel runs on +(the "host" platform), the machines build tools execute on ("exec" platforms), +and the machines targets are built for ("target platforms"). ### Provider -A schema describing a unit of information to pass between [rule targets](#rule-target) along dependency relationships. Typically this contains information like compiler options, transitive source or output files, and build metadata. Frequently used in conjunction with [depsets](#depset) to efficiently store accumulated transitive data. An example of a built-in provider is `DefaultInfo`. +A schema describing a unit of information to pass between +[rule targets](#rule-target) along dependency relationships. Typically this +contains information like compiler options, transitive source or output files, +and build metadata. Frequently used in conjunction with [depsets](#depset) to +efficiently store accumulated transitive data. An example of a built-in provider +is `DefaultInfo`. -Note: The object holding specific data for a given rule target is referred to as a "provider instance", although sometimes this is conflated with "provider". +Note: The object holding specific data for a given rule target is +referred to as a "provider instance", although sometimes this is conflated with +"provider". **See also:** [Provider documentation](/extending/rules#providers) ### Query (concept) -The process of analyzing a [build graph](#build-graph) to understand [target](#target) properties and dependency structures. Bazel supports three query variants: [query](#query-command), [cquery](#configured-query), and [aquery](#action-graph-query). +The process of analyzing a [build graph](#build-graph) to understand +[target](#target) properties and dependency structures. Bazel supports three +query variants: [query](#query-command), [cquery](#configured-query), and +[aquery](#action-graph-query). ### query (command) -A [query](#query-concept) tool that operates over the build's post-[loading phase](#loading-phase) [target graph](#target-graph). This is relatively fast, but can't analyze the effects of `select()`, [build flags](#command-flags), [artifacts](#artifact), or build [actions](#action). +A [query](#query-concept) tool that operates over the build's post-[loading +phase](#loading-phase) [target graph](#target-graph). This is relatively fast, +but can't analyze the effects of `select()`, [build flags](#command-flags), +[artifacts](#artifact), or build [actions](#action). **See also:** [Query how-to](/query/guide), [Query reference](/query/language) ### Repository -A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source +files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo +represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or +`WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a +repo; multiple such files can coexist in a directory. The *main repo* is the repo in which the current Bazel command is being run. -*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` files, or invoking [repo rules](#repository-rule) in [module extensions](#module-extension). They can be fetched on demand to a predetermined "magical" location on disk. +*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` +files, or invoking [repo rules](#repository-rule) in [module +extensions](#module-extension). They can be fetched on demand to a predetermined +"magical" location on disk. -Each repo has a unique, constant *canonical* name, and potentially different *apparent* names when viewed from other repos. +Each repo has a unique, constant *canonical* name, and potentially different +*apparent* names when viewed from other repos. **See also**: [External dependencies overview](/external/overview) ### Repository cache -A shared content-addressable cache of files downloaded by Bazel for builds, shareable across [workspaces](#workspace). Enables offline builds after the initial download. Commonly used to cache files downloaded through [repository rules](#repository-rule) like `http_archive` and repository rule APIs like `repository_ctx.download`. Files are cached only if their SHA-256 checksums are specified for the download. +A shared content-addressable cache of files downloaded by Bazel for builds, +shareable across [workspaces](#workspace). Enables offline builds after the +initial download. Commonly used to cache files downloaded through [repository +rules](#repository-rule) like `http_archive` and repository rule APIs like +`repository_ctx.download`. Files are cached only if their SHA-256 checksums are +specified for the download. ### Repository rule -A schema for repository definitions that tells Bazel how to materialize (or "fetch") a [repository](#repository). Often shortened to just **repo rule**. Repo rules are invoked by Bazel internally to define repos backed by [modules](#module), or can be invoked by [module extensions](#module-extension). Repo rules can access the internet or perform file I/O; the most common repo rule is `http_archive` to download an archive containing source files from the internet. +A schema for repository definitions that tells Bazel how to materialize (or +"fetch") a [repository](#repository). Often shortened to just **repo rule**. +Repo rules are invoked by Bazel internally to define repos backed by +[modules](#module), or can be invoked by [module extensions](#module-extension). +Repo rules can access the internet or perform file I/O; the most common repo +rule is `http_archive` to download an archive containing source files from the +internet. **See also:** [Repo rule documentation](/external/repo) ### Reproducibility -The property of a build or test that a set of inputs to the build or test will always produce the same set of outputs every time, regardless of time, method, or environment. Note that this does not necessarily imply that the outputs are [correct](#correctness) or the desired outputs. +The property of a build or test that a set of inputs to the build or test will +always produce the same set of outputs every time, regardless of time, method, +or environment. Note that this does not necessarily imply that the outputs are +[correct](#correctness) or the desired outputs. ### Rule -A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as `cc_library`. From the perspective of a `BUILD` file author, a rule consists of a set of [attributes](#attributes) and black box logic. The logic tells the rule target how to produce output [artifacts](#artifact) and pass information to other rule targets. From the perspective of `.bzl` authors, rules are the primary way to extend Bazel to support new programming languages and environments. - -Rules are instantiated to produce rule targets in the [loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule targets communicate information to their downstream dependencies in the form of [providers](#provider), and register [actions](#action) describing how to generate their output artifacts. These actions are run in the [execution phase](#execution-phase). - -Note: Historically the term "rule" has been used to refer to a rule target. This usage was inherited from tools like Make, but causes confusion and should be avoided for Bazel. +A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as +`cc_library`. From the perspective of a `BUILD` file author, a rule consists of +a set of [attributes](#attributes) and black box logic. The logic tells the +rule target how to produce output [artifacts](#artifact) and pass information to +other rule targets. From the perspective of `.bzl` authors, rules are the +primary way to extend Bazel to support new programming languages and +environments. + +Rules are instantiated to produce rule targets in the +[loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule +targets communicate information to their downstream dependencies in the form of +[providers](#provider), and register [actions](#action) describing how to +generate their output artifacts. These actions are run in the [execution +phase](#execution-phase). + +Note: Historically the term "rule" has been used to refer to a rule target. +This usage was inherited from tools like Make, but causes confusion and should +be avoided for Bazel. **See also:** [Rules documentation](/extending/rules) ### Rule target -A [target](#target) that is an instance of a rule. Contrasts with file targets and package groups. Not to be confused with [rule](#rule). +A [target](#target) that is an instance of a rule. Contrasts with file targets +and package groups. Not to be confused with [rule](#rule). ### Runfiles -The runtime dependencies of an executable [target](#target). Most commonly, the executable is the executable output of a test rule, and the runfiles are runtime data dependencies of the test. Before the invocation of the executable (during bazel test), Bazel prepares the tree of runfiles alongside the test executable according to their source directory structure. +The runtime dependencies of an executable [target](#target). Most commonly, the +executable is the executable output of a test rule, and the runfiles are runtime +data dependencies of the test. Before the invocation of the executable (during +bazel test), Bazel prepares the tree of runfiles alongside the test executable +according to their source directory structure. **See also:** [Runfiles documentation](/extending/rules#runfiles) ### Sandboxing -A technique to isolate a running [action](#action) inside a restricted and temporary [execution root](#execution-root), helping to ensure that it doesn’t read undeclared inputs or write undeclared outputs. Sandboxing greatly improves [hermeticity](#hermeticity), but usually has a performance cost, and requires support from the operating system. The performance cost depends on the platform. On Linux, it's not significant, but on macOS it can make sandboxing unusable. +A technique to isolate a running [action](#action) inside a restricted and +temporary [execution root](#execution-root), helping to ensure that it doesn’t +read undeclared inputs or write undeclared outputs. Sandboxing greatly improves +[hermeticity](#hermeticity), but usually has a performance cost, and requires +support from the operating system. The performance cost depends on the platform. +On Linux, it's not significant, but on macOS it can make sandboxing unusable. ### Skyframe [Skyframe](/reference/skyframe) is the core parallel, functional, and incremental evaluation framework of Bazel. +{/* TODO: ### Spawn strategy */} + ### Stamping -A feature to embed additional information into Bazel-built [artifacts](#artifact). For example, this can be used for source control, build time and other workspace or environment-related information for release builds. Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that support the stamp attribute. +A feature to embed additional information into Bazel-built +[artifacts](#artifact). For example, this can be used for source control, build +time and other workspace or environment-related information for release builds. +Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that +support the stamp attribute. ### Starlark -The extension language for writing [rules](/extending/rules) and [macros](#macro). A restricted subset of Python (syntactically and grammatically) aimed for the purpose of configuration, and for better performance. Uses the [`.bzl` file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more restricted version of Starlark (such as no `def` function definitions), formerly known as Skylark. +The extension language for writing [rules](/extending/rules) and [macros](#macro). A +restricted subset of Python (syntactically and grammatically) aimed for the +purpose of configuration, and for better performance. Uses the [`.bzl` +file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more +restricted version of Starlark (such as no `def` function definitions), formerly +known as Skylark. **See also:** [Starlark language documentation](/rules/language) +{/* TODO: ### Starlark rules */} + +{/* TODO: ### Starlark rule sandwich */} + ### Startup flags -The set of flags specified between `bazel` and the [command](#query-command), for example, bazel `--host_jvm_debug` build. These flags modify the [configuration](#configuration) of the Bazel server, so any modification to startup flags causes a server restart. Startup flags are not specific to any command. +The set of flags specified between `bazel` and the [command](#query-command), +for example, bazel `--host_jvm_debug` build. These flags modify the +[configuration](#configuration) of the Bazel server, so any modification to +startup flags causes a server restart. Startup flags are not specific to any +command. ### Symbolic macro -A flavor of [macro](#macro) which is declared with a [rule](#rule)-like [attribute](#attribute) schema, allows hiding internal declared [targets](#target) from their own package, and enforces a predictable naming pattern on the targets that the macro declares. Designed to avoid some of the problems seen in large [legacy macro](#legacy-macro) codebases. +A flavor of [macro](#macro) which is declared with a [rule](#rule)-like +[attribute](#attribute) schema, allows hiding internal declared +[targets](#target) from their own package, and enforces a predictable naming +pattern on the targets that the macro declares. Designed to avoid some of the +problems seen in large [legacy macro](#legacy-macro) codebases. **See also:** [Symbolic macro documentation](/extending/macros) ### Target -An object that is defined in a [`BUILD` file](#build-file) and identified by a [label](#label). Targets represent the buildable units of a workspace from the perspective of the end user. +An object that is defined in a [`BUILD` file](#build-file) and identified by a +[label](#label). Targets represent the buildable units of a workspace from +the perspective of the end user. -A target that is declared by instantiating a [rule](#rule) is called a [rule target](#rule-target). Depending on the rule, these may be runnable (like `cc_binary`) or testable (like `cc_test`). Rule targets typically depend on other targets via their [attributes](#attribute) (such as `deps`); these dependencies form the basis of the [target graph](#target-graph). +A target that is declared by instantiating a [rule](#rule) is called a [rule +target](#rule-target). Depending on the rule, these may be runnable (like +`cc_binary`) or testable (like `cc_test`). Rule targets typically depend on +other targets via their [attributes](#attribute) (such as `deps`); these +dependencies form the basis of the [target graph](#target-graph). -Aside from rule targets, there are also file targets and [package group](#package-group) targets. File targets correspond to [artifacts](#artifact) that are referenced within a `BUILD` file. As a special case, the `BUILD` file of any package is always considered a source file target in that package. +Aside from rule targets, there are also file targets and [package group](#package-group) +targets. File targets correspond to [artifacts](#artifact) that are referenced +within a `BUILD` file. As a special case, the `BUILD` file of any package is +always considered a source file target in that package. -Targets are discovered during the [loading phase](#loading-phase). During the [analysis phase](#analysis-phase), targets are associated with [build configurations](#configuration) to form [configured targets](#configured-target). +Targets are discovered during the [loading phase](#loading-phase). During the +[analysis phase](#analysis-phase), targets are associated with [build +configurations](#configuration) to form [configured +targets](#configured-target). ### Target graph -An in-memory graph of [targets](#target) and their dependencies. Produced during the [loading phase](#loading-phase) and used as an input to the [analysis phase](#analysis-phase). +An in-memory graph of [targets](#target) and their dependencies. Produced during +the [loading phase](#loading-phase) and used as an input to the [analysis +phase](#analysis-phase). ### Target pattern -A way to specify a group of [targets](#target) on the command line. Commonly used patterns are `:all` (all rule targets), `:*` (all rule + file targets), `...` (current [package](#package) and all subpackages recursively). Can be used in combination, for example, `//...:*` means all rule and file targets in all packages recursively from the root of the [workspace](#workspace). +A way to specify a group of [targets](#target) on the command line. Commonly +used patterns are `:all` (all rule targets), `:*` (all rule + file targets), +`...` (current [package](#package) and all subpackages recursively). Can be used +in combination, for example, `//...:*` means all rule and file targets in all +packages recursively from the root of the [workspace](#workspace). ### Tests -Rule [targets](#target) instantiated from test rules, and therefore contains a test executable. A return code of zero from the completion of the executable indicates test success. The exact contract between Bazel and tests (such as test environment variables, test result collection methods) is specified in the [Test Encyclopedia](/reference/test-encyclopedia). +Rule [targets](#target) instantiated from test rules, and therefore contains a +test executable. A return code of zero from the completion of the executable +indicates test success. The exact contract between Bazel and tests (such as test +environment variables, test result collection methods) is specified in the [Test +Encyclopedia](/reference/test-encyclopedia). ### Toolchain -A set of tools to build outputs for a language. Typically, a toolchain includes compilers, linkers, interpreters or/and linters. A toolchain can also vary by platform, that is, a Unix compiler toolchain's components may differ for the Windows variant, even though the toolchain is for the same language. Selecting the right toolchain for the platform is known as toolchain resolution. +A set of tools to build outputs for a language. Typically, a toolchain includes +compilers, linkers, interpreters or/and linters. A toolchain can also vary by +platform, that is, a Unix compiler toolchain's components may differ for the +Windows variant, even though the toolchain is for the same language. Selecting +the right toolchain for the platform is known as toolchain resolution. ### Top-level target -A build [target](#target) is top-level if it’s requested on the Bazel command line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is called, then for this build, `//:foo` is top-level, and `//:bar` isn’t top-level, although both targets will need to be built. An important difference between top-level and non-top-level targets is that [command flags](#command-flags) set on the Bazel command line (or via [.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level targets, but might be modified by a [transition](#transition) for non-top-level targets. +A build [target](#target) is top-level if it’s requested on the Bazel command +line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is +called, then for this build, `//:foo` is top-level, and `//:bar` isn’t +top-level, although both targets will need to be built. An important difference +between top-level and non-top-level targets is that [command +flags](#command-flags) set on the Bazel command line (or via +[.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level +targets, but might be modified by a [transition](#transition) for non-top-level +targets. ### Transition -A mapping of [configuration](#configuration) state from one value to another. Enables [targets](#target) in the [build graph](#build-graph) to have different configurations, even if they were instantiated from the same [rule](#rule). A common usage of transitions is with *split* transitions, where certain parts of the [target graph](#target-graph) is forked with distinct configurations for each fork. For example, one can build an Android APK with native binaries compiled for ARM and x86 using split transitions in a single build. +A mapping of [configuration](#configuration) state from one value to another. +Enables [targets](#target) in the [build graph](#build-graph) to have different +configurations, even if they were instantiated from the same [rule](#rule). A +common usage of transitions is with *split* transitions, where certain parts of +the [target graph](#target-graph) is forked with distinct configurations for +each fork. For example, one can build an Android APK with native binaries +compiled for ARM and x86 using split transitions in a single build. **See also:** [User-defined transitions](/extending/config#user-defined-transitions) ### Tree artifact -An [artifact](#artifact) that represents a collection of files. Since these files are not themselves artifacts, an [action](#action) operating on them must instead register the tree artifact as its input or output. +An [artifact](#artifact) that represents a collection of files. Since these +files are not themselves artifacts, an [action](#action) operating on them must +instead register the tree artifact as its input or output. ### Visibility -One of two mechanisms for preventing unwanted dependencies in the build system: *target visibility* for controlling whether a [target](#target) can be depended upon by other targets; and *load visibility* for controlling whether a `BUILD` or `.bzl` file may load a given `.bzl` file. Without context, usually "visibility" refers to target visibility. +One of two mechanisms for preventing unwanted dependencies in the build system: +*target visibility* for controlling whether a [target](#target) can be depended +upon by other targets; and *load visibility* for controlling whether a `BUILD` +or `.bzl` file may load a given `.bzl` file. Without context, usually +"visibility" refers to target visibility. **See also:** [Visibility documentation](/concepts/visibility) ### Workspace -The environment shared by all Bazel commands run from the same [main repository](#repository). +The environment shared by all Bazel commands run from the same [main +repository](#repository). -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". Such usage should be avoided for clarity. +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". Such usage +should be avoided for clarity. diff --git a/reference/skyframe.mdx b/reference/skyframe.mdx index 7f99346a..ba9149fa 100644 --- a/reference/skyframe.mdx +++ b/reference/skyframe.mdx @@ -2,86 +2,197 @@ title: 'Skyframe' --- + + The parallel evaluation and incrementality model of Bazel. ## Data model The data model consists of the following items: -- `SkyValue`. Also called nodes. `SkyValues` are immutable objects that contain all the data built over the course of the build and the inputs of the build. Examples are: input files, output files, targets and configured targets. -- `SkyKey`. A short immutable name to reference a `SkyValue`, for example, `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. -- `SkyFunction`. Builds nodes based on their keys and dependent nodes. -- Node graph. A data structure containing the dependency relationship between nodes. -- `Skyframe`. Code name for the incremental evaluation framework Bazel is based on. +* `SkyValue`. Also called nodes. `SkyValues` are immutable objects that + contain all the data built over the course of the build and the inputs of + the build. Examples are: input files, output files, targets and configured + targets. +* `SkyKey`. A short immutable name to reference a `SkyValue`, for example, + `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. +* `SkyFunction`. Builds nodes based on their keys and dependent nodes. +* Node graph. A data structure containing the dependency relationship between + nodes. +* `Skyframe`. Code name for the incremental evaluation framework Bazel is + based on. ## Evaluation A build is achieved by evaluating the node that represents the build request. -First, Bazel finds the `SkyFunction` corresponding to the key of the top-level `SkyKey`. The function then requests the evaluation of the nodes it needs to evaluate the top-level node, which in turn result in other `SkyFunction` calls, until the leaf nodes are reached. Leaf nodes are usually ones that represent input files in the file system. Finally, Bazel ends up with the value of the top-level `SkyValue`, some side effects (such as output files in the file system) and a directed acyclic graph of the dependencies between the nodes involved in the build. - -A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in advance all of the nodes it needs to do its job. A simple example is evaluating an input file node that turns out to be a symlink: the function tries to read the file, realizes that it is a symlink, and thus fetches the file system node representing the target of the symlink. But that itself can be a symlink, in which case the original function will need to fetch its target, too. - -The functions are represented in the code by the interface `SkyFunction` and the services provided to it by an interface called `SkyFunction.Environment`. These are the things functions can do: - -- Request the evaluation of another node by way of calling `env.getValue`. If the node is available, its value is returned, otherwise, `null` is returned and the function itself is expected to return `null`. In the latter case, the dependent node is evaluated, and then the original node builder is invoked again, but this time the same `env.getValue` call will return a non-`null` value. -- Request the evaluation of multiple other nodes by calling `env.getValues()`. This does essentially the same, except that the dependent nodes are evaluated in parallel. -- Do computation during their invocation -- Have side effects, for example, writing files to the file system. Care needs to be taken that two different functions avoid stepping on each other's toes. In general, write side effects (where data flows outwards from Bazel) are okay, read side effects (where data flows inwards into Bazel without a registered dependency) are not, because they are an unregistered dependency and as such, can cause incorrect incremental builds. - -Well-behaved `SkyFunction` implementations avoid accessing data in any other way than requesting dependencies (such as by directly reading the file system), because that results in Bazel not registering the data dependency on the file that was read, thus resulting in incorrect incremental builds. - -Once a function has enough data to do its job, it should return a non-`null` value indicating completion. +First, Bazel finds the `SkyFunction` corresponding to the key of the top-level +`SkyKey`. The function then requests the evaluation of the nodes it needs to +evaluate the top-level node, which in turn result in other `SkyFunction` calls, +until the leaf nodes are reached. Leaf nodes are usually ones that represent +input files in the file system. Finally, Bazel ends up with the value of the +top-level `SkyValue`, some side effects (such as output files in the file +system) and a directed acyclic graph of the dependencies between the nodes +involved in the build. + +A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in +advance all of the nodes it needs to do its job. A simple example is evaluating +an input file node that turns out to be a symlink: the function tries to read +the file, realizes that it is a symlink, and thus fetches the file system node +representing the target of the symlink. But that itself can be a symlink, in +which case the original function will need to fetch its target, too. + +The functions are represented in the code by the interface `SkyFunction` and the +services provided to it by an interface called `SkyFunction.Environment`. These +are the things functions can do: + +* Request the evaluation of another node by way of calling `env.getValue`. If + the node is available, its value is returned, otherwise, `null` is returned + and the function itself is expected to return `null`. In the latter case, + the dependent node is evaluated, and then the original node builder is + invoked again, but this time the same `env.getValue` call will return a + non-`null` value. +* Request the evaluation of multiple other nodes by calling `env.getValues()`. + This does essentially the same, except that the dependent nodes are + evaluated in parallel. +* Do computation during their invocation +* Have side effects, for example, writing files to the file system. Care needs + to be taken that two different functions avoid stepping on each other's + toes. In general, write side effects (where data flows outwards from Bazel) + are okay, read side effects (where data flows inwards into Bazel without a + registered dependency) are not, because they are an unregistered dependency + and as such, can cause incorrect incremental builds. + +Well-behaved `SkyFunction` implementations avoid accessing data in any other way +than requesting dependencies (such as by directly reading the file system), +because that results in Bazel not registering the data dependency on the file +that was read, thus resulting in incorrect incremental builds. + +Once a function has enough data to do its job, it should return a non-`null` +value indicating completion. This evaluation strategy has a number of benefits: -- Hermeticity. If functions only request input data by way of depending on other nodes, Bazel can guarantee that if the input state is the same, the same data is returned. If all sky functions are deterministic, this means that the whole build will also be deterministic. -- Correct and perfect incrementality. If all the input data of all functions is recorded, Bazel can invalidate only the exact set of nodes that need to be invalidated when the input data changes. -- Parallelism. Since functions can only interact with each other by way of requesting dependencies, functions that don't depend on each other can be run in parallel and Bazel can guarantee that the result is the same as if they were run sequentially. +* Hermeticity. If functions only request input data by way of depending on + other nodes, Bazel can guarantee that if the input state is the same, the + same data is returned. If all sky functions are deterministic, this means + that the whole build will also be deterministic. +* Correct and perfect incrementality. If all the input data of all functions + is recorded, Bazel can invalidate only the exact set of nodes that need to + be invalidated when the input data changes. +* Parallelism. Since functions can only interact with each other by way of + requesting dependencies, functions that don't depend on each other can be + run in parallel and Bazel can guarantee that the result is the same as if + they were run sequentially. ## Incrementality -Since functions can only access input data by depending on other nodes, Bazel can build up a complete data flow graph from the input files to the output files, and use this information to only rebuild those nodes that actually need to be rebuilt: the reverse transitive closure of the set of changed input files. - -In particular, two possible incrementality strategies exist: the bottom-up one and the top-down one. Which one is optimal depends on how the dependency graph looks like. - -- During bottom-up invalidation, after a graph is built and the set of changed inputs is known, all the nodes are invalidated that transitively depend on changed files. This is optimal if the same top-level node will be built again. Note that bottom-up invalidation requires running `stat()` on all input files of the previous build to determine if they were changed. This can be improved by using `inotify` or a similar mechanism to learn about changed files. - -- During top-down invalidation, the transitive closure of the top-level node is checked and only those nodes are kept whose transitive closure is clean. This is better if the node graph is large, but the next build only needs a small subset of it: bottom-up invalidation would invalidate the larger graph of the first build, unlike top-down invalidation, which just walks the small graph of second build. +Since functions can only access input data by depending on other nodes, Bazel +can build up a complete data flow graph from the input files to the output +files, and use this information to only rebuild those nodes that actually need +to be rebuilt: the reverse transitive closure of the set of changed input files. + +In particular, two possible incrementality strategies exist: the bottom-up one +and the top-down one. Which one is optimal depends on how the dependency graph +looks like. + +* During bottom-up invalidation, after a graph is built and the set of changed + inputs is known, all the nodes are invalidated that transitively depend on + changed files. This is optimal if the same top-level node will be built + again. Note that bottom-up invalidation requires running `stat()` on all + input files of the previous build to determine if they were changed. This + can be improved by using `inotify` or a similar mechanism to learn about + changed files. + +* During top-down invalidation, the transitive closure of the top-level node + is checked and only those nodes are kept whose transitive closure is clean. + This is better if the node graph is large, but the next build only needs a + small subset of it: bottom-up invalidation would invalidate the larger graph + of the first build, unlike top-down invalidation, which just walks the small + graph of second build. Bazel only does bottom-up invalidation. -To get further incrementality, Bazel uses *change pruning*: if a node is invalidated, but upon rebuild, it is discovered that its new value is the same as its old value, the nodes that were invalidated due to a change in this node are "resurrected". +To get further incrementality, Bazel uses _change pruning_: if a node is +invalidated, but upon rebuild, it is discovered that its new value is the same +as its old value, the nodes that were invalidated due to a change in this node +are "resurrected". -This is useful, for example, if one changes a comment in a C++ file: then the `.o` file generated from it will be the same, thus, it is unnecessary to call the linker again. +This is useful, for example, if one changes a comment in a C++ file: then the +`.o` file generated from it will be the same, thus, it is unnecessary to call +the linker again. ## Incremental Linking / Compilation -The main limitation of this model is that the invalidation of a node is an all-or-nothing affair: when a dependency changes, the dependent node is always rebuilt from scratch, even if a better algorithm would exist that would mutate the old value of the node based on the changes. A few examples where this would be useful: +The main limitation of this model is that the invalidation of a node is an +all-or-nothing affair: when a dependency changes, the dependent node is always +rebuilt from scratch, even if a better algorithm would exist that would mutate +the old value of the node based on the changes. A few examples where this would +be useful: -- Incremental linking -- When a single class file changes in a JAR file, it is possible modify the JAR file in-place instead of building it from scratch again. +* Incremental linking +* When a single class file changes in a JAR file, it is possible + modify the JAR file in-place instead of building it from scratch again. -The reason why Bazel does not support these things in a principled way is twofold: +The reason why Bazel does not support these things in a principled way +is twofold: -- There were limited performance gains. -- Difficulty to validate that the result of the mutation is the same as that of a clean rebuild would be, and Google values builds that are bit-for-bit repeatable. +* There were limited performance gains. +* Difficulty to validate that the result of the mutation is the same as that + of a clean rebuild would be, and Google values builds that are bit-for-bit + repeatable. -Until now, it was possible to achieve good enough performance by decomposing an expensive build step and achieving partial re-evaluation that way. For example, in an Android app, you can split all the classes into multiple groups and dex them separately. This way, if classes in a group are unchanged, the dexing does not have to be redone. +Until now, it was possible to achieve good enough performance by decomposing an +expensive build step and achieving partial re-evaluation that way. For example, +in an Android app, you can split all the classes into multiple groups and dex +them separately. This way, if classes in a group are unchanged, the dexing does +not have to be redone. ## Mapping to Bazel concepts -This is high level summary of the key `SkyFunction` and `SkyValue` implementations Bazel uses to perform a build: - -- **FileStateValue**. The result of an `lstat()`. For existent files, the function also computes additional information in order to detect changes to the file. This is the lowest level node in the Skyframe graph and has no dependencies. -- **FileValue**. Used by anything that cares about the actual contents or resolved path of a file. Depends on the corresponding `FileStateValue` and any symlinks that need to be resolved (such as the `FileValue` for `a/b` needs the resolved path of `a` and the resolved path of `a/b`). The distinction between `FileValue` and `FileStateValue` is important because the latter can be used in cases where the contents of the file are not actually needed. For example, the file contents are irrelevant when evaluating file system globs (such as `srcs=glob(["*/*.java"])`). -- **DirectoryListingStateValue**. The result of `readdir()`. Like `FileStateValue`, this is the lowest level node and has no dependencies. -- **DirectoryListingValue**. Used by anything that cares about the entries of a directory. Depends on the corresponding `DirectoryListingStateValue`, as well as the associated `FileValue` of the directory. -- **PackageValue**. Represents the parsed version of a BUILD file. Depends on the `FileValue` of the associated `BUILD` file, and also transitively on any `DirectoryListingValue` that is used to resolve the globs in the package (the data structure representing the contents of a `BUILD` file internally). -- **ConfiguredTargetValue**. Represents a configured target, which is a tuple of the set of actions generated during the analysis of a target and information provided to dependent configured targets. Depends on the `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` of direct dependencies, and a special node representing the build configuration. -- **ArtifactValue**. Represents a file in the build, be it a source or an output artifact. Artifacts are almost equivalent to files, and are used to refer to files during the actual execution of build steps. Source files depends on the `FileValue` of the associated node, and output artifacts depend on the `ActionExecutionValue` of whatever action generates the artifact. -- **ActionExecutionValue**. Represents the execution of an action. Depends on the `ArtifactValues` of its input files. The action it executes is contained within its SkyKey, which is contrary to the concept that SkyKeys should be small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if the execution phase does not run. - -As a visual aid, this diagram shows the relationships between SkyFunction implementations after a build of Bazel itself: +This is high level summary of the key `SkyFunction` and `SkyValue` +implementations Bazel uses to perform a build: + +* **FileStateValue**. The result of an `lstat()`. For existent files, the + function also computes additional information in order to detect changes to + the file. This is the lowest level node in the Skyframe graph and has no + dependencies. +* **FileValue**. Used by anything that cares about the actual contents or + resolved path of a file. Depends on the corresponding `FileStateValue` and + any symlinks that need to be resolved (such as the `FileValue` for `a/b` + needs the resolved path of `a` and the resolved path of `a/b`). The + distinction between `FileValue` and `FileStateValue` is important because + the latter can be used in cases where the contents of the file are not + actually needed. For example, the file contents are irrelevant when + evaluating file system globs (such as `srcs=glob(["*/*.java"])`). +* **DirectoryListingStateValue**. The result of `readdir()`. Like + `FileStateValue`, this is the lowest level node and has no dependencies. +* **DirectoryListingValue**. Used by anything that cares about the entries of + a directory. Depends on the corresponding `DirectoryListingStateValue`, as + well as the associated `FileValue` of the directory. +* **PackageValue**. Represents the parsed version of a BUILD file. Depends on + the `FileValue` of the associated `BUILD` file, and also transitively on any + `DirectoryListingValue` that is used to resolve the globs in the package + (the data structure representing the contents of a `BUILD` file internally). +* **ConfiguredTargetValue**. Represents a configured target, which is a tuple + of the set of actions generated during the analysis of a target and + information provided to dependent configured targets. Depends on the + `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` + of direct dependencies, and a special node representing the build + configuration. +* **ArtifactValue**. Represents a file in the build, be it a source or an + output artifact. Artifacts are almost equivalent to files, and are used to + refer to files during the actual execution of build steps. Source files + depends on the `FileValue` of the associated node, and output artifacts + depend on the `ActionExecutionValue` of whatever action generates the + artifact. +* **ActionExecutionValue**. Represents the execution of an action. Depends on + the `ArtifactValues` of its input files. The action it executes is contained + within its SkyKey, which is contrary to the concept that SkyKeys should be + small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if + the execution phase does not run. + +As a visual aid, this diagram shows the relationships between +SkyFunction implementations after a build of Bazel itself: ![A graph of SkyFunction implementation relationships](/reference/skyframe.png) diff --git a/release/backward-compatibility.mdx b/release/backward-compatibility.mdx index 42f40956..af653ccd 100644 --- a/release/backward-compatibility.mdx +++ b/release/backward-compatibility.mdx @@ -2,51 +2,75 @@ title: 'Backward Compatibility' --- -This page provides information about how to handle backward compatibility, including migrating from one release to another and how to communicate incompatible changes. -Bazel is evolving. Minor versions released as part of an [LTS major version](/release#bazel-versioning) are fully backward-compatible. New major LTS releases may contain incompatible changes that require some migration effort. For more information about Bazel's release model, please check out the [Release Model](/release) page. + +This page provides information about how to handle backward compatibility, +including migrating from one release to another and how to communicate +incompatible changes. + +Bazel is evolving. Minor versions released as part of an [LTS major +version](/release#bazel-versioning) are fully backward-compatible. New major LTS +releases may contain incompatible changes that require some migration effort. +For more information about Bazel's release model, please check out the [Release +Model](/release) page. ## Summary -1. It is recommended to use `--incompatible_*` flags for breaking changes. -2. For every `--incompatible_*` flag, a GitHub issue explains the change in behavior and aims to provide a migration recipe. -3. Incompatible flags are recommended to be back-ported to the latest LTS release without enabling the flag by default. -4. APIs and behavior guarded by an `--experimental_*` flag can change at any time. -5. Never run production builds with `--experimental_*` or `--incompatible_*` flags. +1. It is recommended to use `--incompatible_*` flags for breaking changes. +1. For every `--incompatible_*` flag, a GitHub issue explains the change in + behavior and aims to provide a migration recipe. +1. Incompatible flags are recommended to be back-ported to the latest LTS + release without enabling the flag by default. +1. APIs and behavior guarded by an `--experimental_*` flag can change at any + time. +1. Never run production builds with `--experimental_*` or `--incompatible_*` + flags. ## How to follow this policy -- [For Bazel users - how to update Bazel](/install/bazelisk) -- [For contributors - best practices for incompatible changes](/contribute/breaking-changes) -- [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) +* [For Bazel users - how to update Bazel](/install/bazelisk) +* [For contributors - best practices for incompatible changes](/contribute/breaking-changes) +* [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) ## What is stable functionality? -In general, APIs or behaviors without `--experimental_...` flags are considered stable, supported features in Bazel. +In general, APIs or behaviors without `--experimental_...` flags are considered +stable, supported features in Bazel. This includes: -- Starlark language and APIs -- Rules bundled with Bazel -- Bazel APIs such as Remote Execution APIs or Build Event Protocol -- Flags and their semantics +* Starlark language and APIs +* Rules bundled with Bazel +* Bazel APIs such as Remote Execution APIs or Build Event Protocol +* Flags and their semantics ## Incompatible changes and migration recipes -For every incompatible change in a new release, the Bazel team aims to provide a *migration recipe* that helps you update your code (`BUILD` and `.bzl` files, as well as any Bazel usage in scripts, usage of Bazel API, and so on). +For every incompatible change in a new release, the Bazel team aims to provide a +_migration recipe_ that helps you update your code (`BUILD` and `.bzl` files, as +well as any Bazel usage in scripts, usage of Bazel API, and so on). -Incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue. +Incompatible changes should have an associated `--incompatible_*` flag and a +corresponding GitHub issue. -The incompatible flag and relevant changes are recommended to be back-ported to the latest LTS release without enabling the flag by default. This allows users to migrate for the incompatible changes before the next LTS release is available. +The incompatible flag and relevant changes are recommended to be back-ported to +the latest LTS release without enabling the flag by default. This allows users +to migrate for the incompatible changes before the next LTS release is +available. ## Communicating incompatible changes -The primary source of information about incompatible changes are GitHub issues marked with an ["incompatible-change" label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). +The primary source of information about incompatible changes are GitHub issues +marked with an ["incompatible-change" +label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). For every incompatible change, the issue specifies the following: -- Name of the flag controlling the incompatible change -- Description of the changed functionality -- Migration recipe +* Name of the flag controlling the incompatible change +* Description of the changed functionality +* Migration recipe -When an incompatible change is ready for migration with Bazel at HEAD (therefore, also with the next Bazel rolling release), it should be marked with the `migration-ready` label. The incompatible change issue is closed when the incompatible flag is flipped at HEAD. +When an incompatible change is ready for migration with Bazel at HEAD +(therefore, also with the next Bazel rolling release), it should be marked with +the `migration-ready` label. The incompatible change issue is closed when the +incompatible flag is flipped at HEAD. diff --git a/release/index.mdx b/release/index.mdx index 0d2629e7..f58e8f5f 100644 --- a/release/index.mdx +++ b/release/index.mdx @@ -2,48 +2,66 @@ title: 'Release Model' --- -As announced in [the original blog post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel 4.0 and higher versions provides support for two release tracks: rolling releases and long term support (LTS) releases. This page covers the latest information about Bazel's release model. + + +As announced in [the original blog +post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel +4.0 and higher versions provides support for two release tracks: rolling +releases and long term support (LTS) releases. This page covers the latest +information about Bazel's release model. ## Support matrix -| LTS release | Support stage | Latest version | End of support | -| ----------- | ------------- | --------------------------------------------------------------- | -------------- | -| Bazel 9 | Rolling | [Check rolling release page](/release/rolling) | N/A | -| Bazel 8 | Active | [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | -| Bazel 7 | Maintenance | [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | -| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | -| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | -| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | +| LTS release | Support stage | Latest version | End of support | +| ----------- | ------------- | -------------- | -------------- | +| Bazel 9 | Rolling| [Check rolling release page](/release/rolling) | N/A | +| Bazel 8 | Active| [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | +| Bazel 7 | Maintenance| [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | +| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | +| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | +| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | -All Bazel LTS releases can be found on the [release page](https://github.com/bazelbuild/bazel/releases) on GitHub. +All Bazel LTS releases can be found on the [release +page](https://github.com/bazelbuild/bazel/releases) on GitHub. -Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are recommended to upgrade to the latest LTS release or use rolling releases if you want to keep up with the latest changes at HEAD. +Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are +recommended to upgrade to the latest LTS release or use rolling releases if you +want to keep up with the latest changes at HEAD. ## Release versioning -Bazel uses a *major.minor.patch* [Semantic Versioning](https://semver.org/) scheme. +Bazel uses a _major.minor.patch_ [Semantic +Versioning](https://semver.org/) scheme. -- A *major release* contains features that are not backward compatible with the previous release. Each major Bazel version is an LTS release. -- A *minor release* contains backward-compatible bug fixes and features back-ported from the main branch. -- A *patch release* contains critical bug fixes. +* A _major release_ contains features that are not backward compatible with + the previous release. Each major Bazel version is an LTS release. +* A _minor release_ contains backward-compatible bug fixes and features + back-ported from the main branch. +* A _patch release_ contains critical bug fixes. -Additionally, pre-release versions are indicated by appending a hyphen and a date suffix to the next major version number. +Additionally, pre-release versions are indicated by appending a hyphen and a +date suffix to the next major version number. For example, a new release of each type would result in these version numbers: -- Major: 6.0.0 -- Minor: 6.1.0 -- Patch: 6.1.2 -- Pre-release: 7.0.0-pre.20230502.1 +* Major: 6.0.0 +* Minor: 6.1.0 +* Patch: 6.1.2 +* Pre-release: 7.0.0-pre.20230502.1 ## Support stages For each major Bazel version, there are four support stages: -- **Rolling**: This major version is still in pre-release, the Bazel team publishes rolling releases from HEAD. -- **Active**: This major version is the current active LTS release. The Bazel team backports important features and bug fixes into its minor releases. -- **Maintenance**: This major version is an old LTS release in maintenance mode. The Bazel team only promises to backport critical bug fixes for security issues and OS-compatibility issues into this LTS release. -- **Deprecated**: The Bazel team no longer provides support for this major version, all users should migrate to newer Bazel LTS releases. +* **Rolling**: This major version is still in pre-release, the Bazel team + publishes rolling releases from HEAD. +* **Active**: This major version is the current active LTS release. The Bazel + team backports important features and bug fixes into its minor releases. +* **Maintenance**: This major version is an old LTS release in maintenance + mode. The Bazel team only promises to backport critical bug fixes for + security issues and OS-compatibility issues into this LTS release. +* **Deprecated**: The Bazel team no longer provides support for this major + version, all users should migrate to newer Bazel LTS releases. ## Release cadence @@ -51,90 +69,148 @@ Bazel regularly publish releases for two release tracks. ### Rolling releases -- Rolling releases are coordinated with Google Blaze release and are released from HEAD around every two weeks. It is a preview of the next Bazel LTS release. -- Rolling releases can ship incompatible changes. Incompatible flags are recommended for major breaking changes, rolling out incompatible changes should follow our [backward compatibility policy](/release/backward-compatibility). +* Rolling releases are coordinated with Google Blaze release and are released + from HEAD around every two weeks. It is a preview of the next Bazel LTS + release. +* Rolling releases can ship incompatible changes. Incompatible flags are + recommended for major breaking changes, rolling out incompatible changes + should follow our [backward compatibility + policy](/release/backward-compatibility). ### LTS releases -- *Major release*: A new LTS release is expected to be cut from HEAD roughly every 12 months. Once a new LTS release is out, it immediately enters the Active stage, and the previous LTS release enters the Maintenance stage. -- *Minor release*: New minor verions on the Active LTS track are expected to be released once every 2 months. -- *Patch release*: New patch versions for LTS releases in Active and Maintenance stages are expected to be released on demand for critical bug fixes. -- A Bazel LTS release enters the Deprecated stage after being in ​​the Maintenance stage for 2 years. - -For planned releases, please check our [release issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) on Github. +* _Major release_: A new LTS release is expected to be cut from HEAD roughly + every + 12 months. Once a new LTS release is out, it immediately enters the Active + stage, and the previous LTS release enters the Maintenance stage. +* _Minor release_: New minor verions on the Active LTS track are expected to + be released once every 2 months. +* _Patch release_: New patch versions for LTS releases in Active and + Maintenance stages are expected to be released on demand for critical bug + fixes. +* A Bazel LTS release enters the Deprecated stage after being in ​​the + Maintenance stage for 2 years. + +For planned releases, please check our [release +issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) +on Github. ## Release procedure & policies -For rolling releases, the process is straightforward: about every two weeks, a new release is created, aligning with the same baseline as the Google internal Blaze release. Due to the rapid release schedule, we don't backport any changes to rolling releases. +For rolling releases, the process is straightforward: about every two weeks, a +new release is created, aligning with the same baseline as the Google internal +Blaze release. Due to the rapid release schedule, we don't backport any changes +to rolling releases. For LTS releases, the procedure and policies below are followed: -1. Determine a baseline commit for the release. - - - For a new major LTS release, the baseline commit is the HEAD of the main branch. - - For a minor or patch release, the baseline commit is the HEAD of the current latest version of the same LTS release. - -2. Create a release branch in the name of `release-<version>` from the baseline commit. - -3. Backport changes via PRs to the release branch. - - - The community can suggest certain commits to be back-ported by replying "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential release blockers, the Bazel team triages them and decide whether to back-port the commits. - - Only backward-compatible commits on the main branch can be back-ported, additional minor changes to resolve merge conflicts are acceptable. - -4. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. - - - Bazel maintainers can request to cherry-pick specific commit(s) to a release branch. This process is initiated by creating a cherry-pick request on GitHub. Here's how to do it. - - 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=\&labels=\&projects=\&template=cherry_pick_request.yml) - - 2. Fill in the request details - - - Title: Provide a concise and descriptive title for the request. - - Commit ID(s): Enter the ID(s) of the commit(s) you want to cherry-pick. If there are multiple commits, then separate them with commas. - - Category: Specify the category of the request. - - Reviewer(s): For multiple reviewers, separate their GitHub ID's with commas. - - 3. Set the milestone - - - Find the "Milestone" section and click the setting. - - Select the appropriate X.Y.Z release blockers. This action triggers the cherry-pick bot to process your request for the "release-X.Y.Z" branch. - - 4. Submit the Issue - - Once all details are filled in and the miestone is set, submit the issue. - - - The cherry-pick bot will process the request and notify if the commit(s) are eligible for cherry-picking. If the commits are cherry-pickable, which means there's no merge conflict while cherry-picking the commit, then the bot will create a new pull request. When the pull request is approved by a member of the Bazel team, the commits are cherry-picked and merged to the release branch. For a visual example of a completed cherry-pick request, refer to this [example](https://github.com/bazelbuild/bazel/issues/20230) . - -5. Identify release blockers and fix issues found on the release branch. - - - The release branch is tested with the same test suite in [postsubmit](https://buildkite.com/bazel/bazel-bazel) and [downstream test pipeline](https://buildkite.com/bazel/bazel-at-head-plus-downstream) on Bazel CI. The Bazel team monitors testing results of the release branch and fixes any regressions found. - -6. Create a new release candidate from the release branch when all known release blockers are resolved. - - - The release candidate is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors community bug reports for the candidate. - - If new release blockers are identified, go back to the last step and create a new release candidate after resolving all the issues. - - New features are not allowed to be added to the release branch after the first release candidate is created; cherry-picks are limited to critical fixes only. If a cherry-pick is needed, the requester must answer the following questions: Why is this change critical, and what benefits does it provide? What is the likelihood of this change introducing a regression? - -7. Push the release candidate as the official release if no further release blockers are found - - - For patch releases, push the release at least two business days after the last release candidate is out. - - For major and minor releases, push the release two business days after the last release candidate is out, but not earlier than one week after the first release candidate is out. - - The release is only pushed on a day where the next day is a business day. - - The release is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors and addresses community bug reports for the new release. +1. Determine a baseline commit for the release. + * For a new major LTS release, the baseline commit is the HEAD of the main + branch. + * For a minor or patch release, the baseline commit is the HEAD of the + current latest version of the same LTS release. +1. Create a release branch in the name of `release-` from the baseline + commit. +1. Backport changes via PRs to the release branch. + * The community can suggest certain commits to be back-ported by replying + "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential + release blockers, the Bazel team triages them and decide whether to + back-port the commits. + * Only backward-compatible commits on the main branch can be back-ported, + additional minor changes to resolve merge conflicts are acceptable. +1. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. + * Bazel maintainers can request to cherry-pick specific commit(s) + to a release branch. This process is initiated by creating a + cherry-pick request on GitHub. Here's how to do it. + 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=&labels=&projects=&template=cherry_pick_request.yml) + 2. Fill in the request details + * Title: Provide a concise and descriptive title for the request. + * Commit ID(s): Enter the ID(s) of the commit(s) you want to + cherry-pick. If there are multiple commits, then separate + them with commas. + * Category: Specify the category of the request. + * Reviewer(s): For multiple reviewers, separate their GitHub + ID's with commas. + 3. Set the milestone + * Find the "Milestone" section and click the setting. + * Select the appropriate X.Y.Z release blockers. This action + triggers the cherry-pick bot to process your request + for the "release-X.Y.Z" branch. + 4. Submit the Issue + * Once all details are filled in and the miestone is set, + submit the issue. + + * The cherry-pick bot will process the request and notify + if the commit(s) are eligible for cherry-picking. If + the commits are cherry-pickable, which means there's no + merge conflict while cherry-picking the commit, then + the bot will create a new pull request. When the pull + request is approved by a member of the Bazel team, the + commits are cherry-picked and merged to the release branch. + For a visual example of a completed cherry-pick request, + refer to this + [example](https://github.com/bazelbuild/bazel/issues/20230) + . + +1. Identify release blockers and fix issues found on the release branch. + * The release branch is tested with the same test suite in + [postsubmit](https://buildkite.com/bazel/bazel-bazel) and + [downstream test pipeline] + (https://buildkite.com/bazel/bazel-at-head-plus-downstream) + on Bazel CI. The Bazel team monitors testing results of the release + branch and fixes any regressions found. +1. Create a new release candidate from the release branch when all known + release blockers are resolved. + * The release candidate is announced on + [bazel-discuss](https://groups.google.com/g/bazel-discuss), + the Bazel team monitors community bug reports for the candidate. + * If new release blockers are identified, go back to the last step and + create a new release candidate after resolving all the issues. + * New features are not allowed to be added to the release branch after the + first release candidate is created; cherry-picks are limited to critical + fixes only. If a cherry-pick is needed, the requester must answer the + following questions: Why is this change critical, and what benefits does + it provide? What is the likelihood of this change introducing a + regression? +1. Push the release candidate as the official release if no further release + blockers are found + * For patch releases, push the release at least two business days after + the last release candidate is out. + * For major and minor releases, push the release two business days after + the last release candidate is out, but not earlier than one week after + the first release candidate is out. + * The release is only pushed on a day where the next day is a business + day. + * The release is announced on + [bazel-discuss](https://groups.google.com/g/bazel-discuss), + the Bazel team monitors and addresses community bug reports for the new + release. ## Report regressions -If a user finds a regression in a new Bazel release, release candidate or even Bazel at HEAD, please file a bug on [GitHub](https://github.com/bazelbuild/bazel/issues). You can use Bazelisk to bisect the culprit commit and include this information in the bug report. +If a user finds a regression in a new Bazel release, release candidate or even +Bazel at HEAD, please file a bug on +[GitHub](https://github.com/bazelbuild/bazel/issues). You can use +Bazelisk to bisect the culprit commit and include this information in the bug +report. -For example, if your build succeeds with Bazel 6.1.0 but fails with the second release candidate of 6.2.0, you can do bisect via +For example, if your build succeeds with Bazel 6.1.0 but fails with the second +release candidate of 6.2.0, you can do bisect via ```bash bazelisk --bisect=6.1.0..release-6.2.0rc2 build //foo:bar ``` -You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run corresponding bazel commands to reset the build state if it's needed to reproduce the issue. For more details, check out documentation about Bazelisk [bisect feature](https://github.com/bazelbuild/bazelisk#--bisect). +You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run +corresponding bazel commands to reset the build state if it's needed to +reproduce the issue. For more details, check out documentation about Bazelisk +[bisect feature] (https://github.com/bazelbuild/bazelisk#--bisect). -Remember to upgrade Bazelisk to the latest version to use the bisect feature. +Remember to upgrade Bazelisk to the latest version to use the bisect +feature. ## Rule compatibility -If you are a rule authors and want to maintain compatibility with different Bazel versions, please check out the [Rule Compatibility](/release/rule-compatibility) page. +If you are a rule authors and want to maintain compatibility with different +Bazel versions, please check out the [Rule +Compatibility](/release/rule-compatibility) page. diff --git a/release/rolling.mdx b/release/rolling.mdx index 4cb413ad..8e79d6f5 100644 --- a/release/rolling.mdx +++ b/release/rolling.mdx @@ -2,6 +2,13 @@ title: 'Rolling Releases' --- -This page contains an overview of all rolling releases, as per our [release policy](https://bazel.build/release#rolling-releases). [Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use these releases. -## Index + +This page contains an overview of all rolling releases, as per our +[release policy](https://bazel.build/release#rolling-releases). +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use +these releases. + +## Index + + diff --git a/release/rule-compatibility.mdx b/release/rule-compatibility.mdx index 93027953..05a8a95e 100644 --- a/release/rule-compatibility.mdx +++ b/release/rule-compatibility.mdx @@ -2,55 +2,89 @@ title: 'Rule Compatibility' --- -Bazel Starlark rules can break compatibility with Bazel LTS releases in the following two scenarios: -1. The rule breaks compatibility with future LTS releases because a feature it depends on is removed from Bazel at HEAD. -2. The rule breaks compatibility with the current or older LTS releases because a feature it depends on is only available in newer Bazel LTS releases. -Meanwhile, the rule itself can ship incompatible changes for their users as well. When combined with breaking changes in Bazel, upgrading the rule version and Bazel version can often be a source of frustration for Bazel users. This page covers how rules authors should maintain rule compatibility with Bazel to make it easier for users to upgrade Bazel and rules. +Bazel Starlark rules can break compatibility with Bazel LTS releases in the +following two scenarios: + +1. The rule breaks compatibility with future LTS releases because a feature it + depends on is removed from Bazel at HEAD. +1. The rule breaks compatibility with the current or older LTS releases because + a feature it depends on is only available in newer Bazel LTS releases. + +Meanwhile, the rule itself can ship incompatible changes for their users as +well. When combined with breaking changes in Bazel, upgrading the rule version +and Bazel version can often be a source of frustration for Bazel users. This +page covers how rules authors should maintain rule compatibility with Bazel to +make it easier for users to upgrade Bazel and rules. ## Manageable migration process -While it's obviously not feasible to guarantee compatibility between every version of Bazel and every version of the rule, our aim is to ensure that the migration process remains manageable for Bazel users. A manageable migration process is defined as a process where **users are not forced to upgrade the rule's major version and Bazel's major version simultaneously**, thereby allowing users to handle incompatible changes from one source at a time. +While it's obviously not feasible to guarantee compatibility between every +version of Bazel and every version of the rule, our aim is to ensure that the +migration process remains manageable for Bazel users. A manageable migration +process is defined as a process where **users are not forced to upgrade the +rule's major version and Bazel's major version simultaneously**, thereby +allowing users to handle incompatible changes from one source at a time. For example, with the following compatibility matrix: -- Migrating from rules\_foo 1.x + Bazel 4.x to rules\_foo 2.x + Bazel 5.x is not considered manageable, as the users need to upgrade the major version of rules\_foo and Bazel at the same time. -- Migrating from rules\_foo 2.x + Bazel 5.x to rules\_foo 3.x + Bazel 6.x is considered manageable, as the users can first upgrade rules\_foo from 2.x to 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to 6.x. +* Migrating from rules_foo 1.x + Bazel 4.x to rules_foo 2.x + Bazel 5.x is not + considered manageable, as the users need to upgrade the major version of + rules_foo and Bazel at the same time. +* Migrating from rules_foo 2.x + Bazel 5.x to rules_foo 3.x + Bazel 6.x is + considered manageable, as the users can first upgrade rules_foo from 2.x to + 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to + 6.x. -| | rules\_foo 1.x | rules\_foo 2.x | rules\_foo 3.x | HEAD | -| --------- | -------------- | -------------- | -------------- | ---- | -| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | -| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | -| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | -| HEAD | ❌ | ❌ | ❌ | ✅ | +| | rules_foo 1.x | rules_foo 2.x | rules_foo 3.x | HEAD | +| --- | --- | --- | --- | --- | +| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | +| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | +| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | +| HEAD | ❌ | ❌ | ❌ | ✅ | -❌: No version of the major rule version is compatible with the Bazel LTS release. +❌: No version of the major rule version is compatible with the Bazel LTS +release. -✅: At least one version of the rule is compatible with the latest version of the Bazel LTS release. +✅: At least one version of the rule is compatible with the latest version of the +Bazel LTS release. ## Best practices -As Bazel rules authors, you can ensure a manageable migration process for users by following these best practices: - -1. The rule should follow [Semantic Versioning](https://semver.org/): minor versions of the same major version are backward compatible. - -2. The rule at HEAD should be compatible with the latest Bazel LTS release. - -3. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, you can - - - Set up your own CI testing with Bazel at HEAD - - Add your project to [Bazel downstream testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); the Bazel team files issues to your project if breaking changes in Bazel affect your project, and you must follow our [downstream project policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) to address issues timely. - -4. The latest major version of the rule must be compatible with the latest Bazel LTS release. - -5. A new major version of the rule should be compatible with the last Bazel LTS release supported by the previous major version of the rule. - -Achieving 2. and 3. is the most important task since it allows achieving 4. and 5. naturally. - -To make it easier to keep compatibility with both Bazel at HEAD and the latest Bazel LTS release, rules authors can: - -- Request backward-compatible features to be back-ported to the latest LTS release, check out [release process](/release#release-procedure-policies) for more details. -- Use [bazel\_features](https://github.com/bazel-contrib/bazel_features) to do Bazel feature detection. - -In general, with the recommended approaches, rules should be able to migrate for Bazel incompatible changes and make use of new Bazel features at HEAD without dropping compatibility with the latest Bazel LTS release. +As Bazel rules authors, you can ensure a manageable migration process for users +by following these best practices: + +1. The rule should follow [Semantic + Versioning](https://semver.org/): minor versions of the same + major version are backward compatible. +1. The rule at HEAD should be compatible with the latest Bazel LTS release. +1. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, + you can + * Set up your own CI testing with Bazel at HEAD + * Add your project to [Bazel downstream + testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); + the Bazel team files issues to your project if breaking changes in Bazel + affect your project, and you must follow our [downstream project + policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) + to address issues timely. +1. The latest major version of the rule must be compatible with the latest + Bazel LTS release. +1. A new major version of the rule should be compatible with the last Bazel LTS + release supported by the previous major version of the rule. + +Achieving 2. and 3. is the most important task since it allows achieving 4. and +5. naturally. + +To make it easier to keep compatibility with both Bazel at HEAD and the latest +Bazel LTS release, rules authors can: + +* Request backward-compatible features to be back-ported to the latest LTS + release, check out [release process](/release#release-procedure-policies) + for more details. +* Use [bazel_features](https://github.com/bazel-contrib/bazel_features) + to do Bazel feature detection. + +In general, with the recommended approaches, rules should be able to migrate for +Bazel incompatible changes and make use of new Bazel features at HEAD without +dropping compatibility with the latest Bazel LTS release. diff --git a/remote/bep-examples.mdx b/remote/bep-examples.mdx index 68e4d42c..faf11bf9 100644 --- a/remote/bep-examples.mdx +++ b/remote/bep-examples.mdx @@ -2,9 +2,14 @@ title: 'Build Event Protocol Examples' --- -The full specification of the Build Event Protocol can be found in its protocol buffer definition. However, it might be helpful to build up some intuition before looking at the specification. -Consider a simple Bazel workspace that consists of two empty shell scripts `foo.sh` and `foo_test.sh` and the following `BUILD` file: + +The full specification of the Build Event Protocol can be found in its protocol +buffer definition. However, it might be helpful to build up some intuition +before looking at the specification. + +Consider a simple Bazel workspace that consists of two empty shell scripts +`foo.sh` and `foo_test.sh` and the following `BUILD` file: ```bash sh_library( @@ -19,32 +24,58 @@ sh_test( ) ``` -When running `bazel test ...` on this project the build graph of the generated build events will resemble the graph below. The arrows indicate the aforementioned parent and child relationship. Note that some build events and most fields have been omitted for brevity. +When running `bazel test ...` on this project the build graph of the generated +build events will resemble the graph below. The arrows indicate the +aforementioned parent and child relationship. Note that some build events and +most fields have been omitted for brevity. ![bep-graph](/docs/images/bep-graph.png "BEP graph") **Figure 1.** BEP graph. -Initially, a `BuildStarted` event is published. The event informs us that the build was invoked through the `bazel test` command and announces child events: +Initially, a `BuildStarted` event is published. The event informs us that the +build was invoked through the `bazel test` command and announces child events: -- `OptionsParsed` -- `WorkspaceStatus` -- `CommandLine` -- `UnstructuredCommandLine` -- `BuildMetadata` -- `BuildFinished` -- `PatternExpanded` -- `Progress` +* `OptionsParsed` +* `WorkspaceStatus` +* `CommandLine` +* `UnstructuredCommandLine` +* `BuildMetadata` +* `BuildFinished` +* `PatternExpanded` +* `Progress` The first three events provide information about how Bazel was invoked. -The `PatternExpanded` build event provides insight into which specific targets the `...` pattern expanded to: `//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two `TargetConfigured` events as children. Note that the `TargetConfigured` event declares the `Configuration` event as a child event, even though `Configuration` has been posted before the `TargetConfigured` event. - -Besides the parent and child relationship, events may also refer to each other using their build event identifiers. For example, in the above graph the `TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` field. - -Build events that refer to files don’t usually embed the file names and paths in the event. Instead, they contain the build event identifier of a `NamedSetOfFiles` event, which will then contain the actual file names and paths. The `NamedSetOfFiles` event allows a set of files to be reported once and referred to by many targets. This structure is necessary because otherwise in some cases the Build Event Protocol output size would grow quadratically with the number of files. A `NamedSetOfFiles` event may also not have all its files embedded, but instead refer to other `NamedSetOfFiles` events through their build event identifiers. - -Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` target from the above graph, printed in protocol buffer’s JSON representation. The build event identifier contains the target as an opaque string and refers to the `Configuration` event using its build event identifier. The event does not announce any child events. The payload contains information about whether the target was built successfully, the set of output files, and the kind of target built. +The `PatternExpanded` build event provides insight +into which specific targets the `...` pattern expanded to: +`//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two +`TargetConfigured` events as children. Note that the `TargetConfigured` event +declares the `Configuration` event as a child event, even though `Configuration` +has been posted before the `TargetConfigured` event. + +Besides the parent and child relationship, events may also refer to each other +using their build event identifiers. For example, in the above graph the +`TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` +field. + +Build events that refer to files don’t usually embed the file +names and paths in the event. Instead, they contain the build event identifier +of a `NamedSetOfFiles` event, which will then contain the actual file names and +paths. The `NamedSetOfFiles` event allows a set of files to be reported once and +referred to by many targets. This structure is necessary because otherwise in +some cases the Build Event Protocol output size would grow quadratically with +the number of files. A `NamedSetOfFiles` event may also not have all its files +embedded, but instead refer to other `NamedSetOfFiles` events through their +build event identifiers. + +Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` +target from the above graph, printed in protocol buffer’s JSON representation. +The build event identifier contains the target as an opaque string and refers to +the `Configuration` event using its build event identifier. The event does not +announce any child events. The payload contains information about whether the +target was built successfully, the set of output files, and the kind of target +built. ```json { @@ -71,9 +102,18 @@ Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` targe ## Aspect Results in BEP -Ordinary builds evaluate actions associated with `(target, configuration)` pairs. When building with [aspects](/extending/aspects) enabled, Bazel additionally evaluates targets associated with `(target, configuration, aspect)` triples, for each target affected by a given enabled aspect. +Ordinary builds evaluate actions associated with `(target, configuration)` +pairs. When building with [aspects](/extending/aspects) enabled, Bazel +additionally evaluates targets associated with `(target, configuration, +aspect)` triples, for each target affected by a given enabled aspect. -Evaluation results for aspects are available in BEP despite the absence of aspect-specific event types. For each `(target, configuration)` pair with an applicable aspect, Bazel publishes an additional `TargetConfigured` and `TargetComplete` event bearing the result from applying the aspect to the target. For example, if `//:foo_lib` is built with `--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in the BEP: +Evaluation results for aspects are available in BEP despite the absence of +aspect-specific event types. For each `(target, configuration)` pair with an +applicable aspect, Bazel publishes an additional `TargetConfigured` and +`TargetComplete` event bearing the result from applying the aspect to the +target. For example, if `//:foo_lib` is built with +`--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in +the BEP: ```json { @@ -98,21 +138,37 @@ Evaluation results for aspects are available in BEP despite the absence of aspec } ``` -Note: The only difference between the IDs is the presence of the `aspect` field. A tool that does not check the `aspect` ID field and accumulates output files by target may conflate target outputs with aspect outputs. +Note: The only difference between the IDs is the presence of the `aspect` +field. A tool that does not check the `aspect` ID field and accumulates output +files by target may conflate target outputs with aspect outputs. ## Consuming `NamedSetOfFiles` -Determining the artifacts produced by a given target (or aspect) is a common BEP use-case that can be done efficiently with some preparation. This section discusses the recursive, shared structure offered by the `NamedSetOfFiles` event, which matches the structure of a Starlark [Depset](/extending/depsets). +Determining the artifacts produced by a given target (or aspect) is a common +BEP use-case that can be done efficiently with some preparation. This section +discusses the recursive, shared structure offered by the `NamedSetOfFiles` +event, which matches the structure of a Starlark [Depset](/extending/depsets). -Consumers must take care to avoid quadratic algorithms when processing `NamedSetOfFiles` events because large builds can contain tens of thousands of such events, requiring hundreds of millions operations in a traversal with quadratic complexity. +Consumers must take care to avoid quadratic algorithms when processing +`NamedSetOfFiles` events because large builds can contain tens of thousands of +such events, requiring hundreds of millions operations in a traversal with +quadratic complexity. ![namedsetoffiles-bep-graph](/docs/images/namedsetoffiles-bep-graph.png "NamedSetOfFiles BEP graph") **Figure 2.** `NamedSetOfFiles` BEP graph. -A `NamedSetOfFiles` event always appears in the BEP stream *before* a `TargetComplete` or `NamedSetOfFiles` event that references it. This is the inverse of the "parent-child" event relationship, where all but the first event appears after at least one event announcing it. A `NamedSetOfFiles` event is announced by a `Progress` event with no semantics. - -Given these ordering and sharing constraints, a typical consumer must buffer all `NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON event stream and Python code demonstrate how to populate a map from target/aspect to built artifacts in the "default" output group, and how to process the outputs for a subset of built targets/aspects: +A `NamedSetOfFiles` event always appears in the BEP stream *before* a +`TargetComplete` or `NamedSetOfFiles` event that references it. This is the +inverse of the "parent-child" event relationship, where all but the first event +appears after at least one event announcing it. A `NamedSetOfFiles` event is +announced by a `Progress` event with no semantics. + +Given these ordering and sharing constraints, a typical consumer must buffer all +`NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON +event stream and Python code demonstrate how to populate a map from +target/aspect to built artifacts in the "default" output group, and how to +process the outputs for a subset of built targets/aspects: ```python named_sets = {} # type: dict[str, NamedSetOfFiles] diff --git a/remote/bep-glossary.mdx b/remote/bep-glossary.mdx index 66f77ea5..3bd11eea 100644 --- a/remote/bep-glossary.mdx +++ b/remote/bep-glossary.mdx @@ -2,13 +2,22 @@ title: 'Build Event Protocol Glossary' --- -Each BEP event type has its own semantics, minimally documented in [build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). The following glossary describes each event type. + + +Each BEP event type has its own semantics, minimally documented in +[build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). +The following glossary describes each event type. ## Aborted -Unlike other events, `Aborted` does not have a corresponding ID type, because the `Aborted` event *replaces* events of other types. This event indicates that the build terminated early and the event ID it appears under was not produced normally. `Aborted` contains an enum and human-friendly description to explain why the build did not complete. +Unlike other events, `Aborted` does not have a corresponding ID type, because +the `Aborted` event *replaces* events of other types. This event indicates that +the build terminated early and the event ID it appears under was not produced +normally. `Aborted` contains an enum and human-friendly description to explain +why the build did not complete. -For example, if a build is evaluating a target when the user interrupts Bazel, BEP contains an event like the following: +For example, if a build is evaluating a target when the user interrupts Bazel, +BEP contains an event like the following: ```json { @@ -28,21 +37,35 @@ For example, if a build is evaluating a target when the user interrupts Bazel, B ## ActionExecuted -Provides details about the execution of a specific [Action](/rules/lib/actions) in a build. By default, this event is included in the BEP only for failed actions, to support identifying the root cause of build failures. Users may set the `--build_event_publish_all_actions` flag to include all `ActionExecuted` events. +Provides details about the execution of a specific +[Action](/rules/lib/actions) in a build. By default, this event is +included in the BEP only for failed actions, to support identifying the root cause +of build failures. Users may set the `--build_event_publish_all_actions` flag +to include all `ActionExecuted` events. ## BuildFinished -A single `BuildFinished` event is sent after the command is complete and includes the exit code for the command. This event provides authoritative success/failure information. +A single `BuildFinished` event is sent after the command is complete and +includes the exit code for the command. This event provides authoritative +success/failure information. ## BuildMetadata -Contains the parsed contents of the `--build_metadata` flag. This event exists to support Bazel integration with other tooling by plumbing external data (such as identifiers). +Contains the parsed contents of the `--build_metadata` flag. This event exists +to support Bazel integration with other tooling by plumbing external data (such as +identifiers). ## BuildMetrics -A single `BuildMetrics` event is sent at the end of every command and includes counters/gauges useful for quantifying the build tool's behavior during the command. These metrics indicate work actually done and does not count cached work that is reused. +A single `BuildMetrics` event is sent at the end of every command and includes +counters/gauges useful for quantifying the build tool's behavior during the +command. These metrics indicate work actually done and does not count cached +work that is reused. -Note that `memory_metrics` may not be populated if there was no Java garbage collection during the command's execution. Users may set the `--memory_profile=/dev/null` option which forces the garbage collector to run at the end of the command to populate `memory_metrics`. +Note that `memory_metrics` may not be populated if there was no Java garbage +collection during the command's execution. Users may set the +`--memory_profile=/dev/null` option which forces the garbage +collector to run at the end of the command to populate `memory_metrics`. ```json { @@ -71,11 +94,14 @@ Note that `memory_metrics` may not be populated if there was no Java garbage col ## BuildStarted -The first event in a BEP stream, `BuildStarted` includes metadata describing the command before any meaningful work begins. +The first event in a BEP stream, `BuildStarted` includes metadata describing the +command before any meaningful work begins. ## BuildToolLogs -A single `BuildToolLogs` event is sent at the end of a command, including URIs of files generated by the build tool that may aid in understanding or debugging build tool behavior. Some information may be included inline. +A single `BuildToolLogs` event is sent at the end of a command, including URIs +of files generated by the build tool that may aid in understanding or debugging +build tool behavior. Some information may be included inline. ```json { @@ -104,15 +130,28 @@ A single `BuildToolLogs` event is sent at the end of a command, including URIs o ## CommandLine -The BEP contains multiple `CommandLine` events containing representations of all command-line arguments (including options and uninterpreted arguments). Each `CommandLine` event has a label in its `StructuredCommandLineId` that indicates which representation it conveys; three such events appear in the BEP: - -- `"original"`: Reconstructed commandline as Bazel received it from the Bazel client, without startup options sourced from .rc files. -- `"canonical"`: The effective commandline with .rc files expanded and invocation policy applied. -- `"tool"`: Populated from the `--experimental_tool_command_line` option. This is useful to convey the command-line of a tool wrapping Bazel through the BEP. This could be a base64-encoded `CommandLine` binary protocol buffer message which is used directly, or a string which is parsed but not interpreted (as the tool's options may differ from Bazel's). +The BEP contains multiple `CommandLine` events containing representations of all +command-line arguments (including options and uninterpreted arguments). +Each `CommandLine` event has a label in its `StructuredCommandLineId` that +indicates which representation it conveys; three such events appear in the BEP: + +* `"original"`: Reconstructed commandline as Bazel received it from the Bazel + client, without startup options sourced from .rc files. +* `"canonical"`: The effective commandline with .rc files expanded and + invocation policy applied. +* `"tool"`: Populated from the `--experimental_tool_command_line` option. This + is useful to convey the command-line of a tool wrapping Bazel through the BEP. + This could be a base64-encoded `CommandLine` binary protocol buffer message + which is used directly, or a string which is parsed but not interpreted (as + the tool's options may differ from Bazel's). ## Configuration -A `Configuration` event is sent for every [`configuration`](/extending/config) used in the top-level targets in a build. At least one configuration event is always be present. The `id` is reused by the `TargetConfigured` and `TargetComplete` event IDs and is necessary to disambiguate those events in multi-configuration builds. +A `Configuration` event is sent for every [`configuration`](/extending/config) +used in the top-level targets in a build. At least one configuration event is +always be present. The `id` is reused by the `TargetConfigured` and +`TargetComplete` event IDs and is necessary to disambiguate those events in +multi-configuration builds. ```json { @@ -137,7 +176,11 @@ A `Configuration` event is sent for every [`configuration`](/extending/config) u ## ConvenienceSymlinksIdentified -**Experimental.** If the `--experimental_convenience_symlinks_bep_event` option is set, a single `ConvenienceSymlinksIdentified` event is produced by `build` commands to indicate how symlinks in the workspace should be managed. This enables building tools that invoke Bazel remotely then arrange the local workspace as if Bazel had been run locally. +**Experimental.** If the `--experimental_convenience_symlinks_bep_event` +option is set, a single `ConvenienceSymlinksIdentified` event is produced by +`build` commands to indicate how symlinks in the workspace should be managed. +This enables building tools that invoke Bazel remotely then arrange the local +workspace as if Bazel had been run locally. ```json { @@ -168,17 +211,24 @@ A `Configuration` event is sent for every [`configuration`](/extending/config) u ## Fetch -Indicates that a Fetch operation occurred as a part of the command execution. Unlike other events, if a cached fetch result is re-used, this event does not appear in the BEP stream. +Indicates that a Fetch operation occurred as a part of the command execution. +Unlike other events, if a cached fetch result is re-used, this event does not +appear in the BEP stream. ## NamedSetOfFiles -`NamedSetOfFiles` events report a structure matching a [`depset`](/extending/depsets) of files produced during command evaluation. Transitively included depsets are identified by `NamedSetOfFilesId`. +`NamedSetOfFiles` events report a structure matching a +[`depset`](/extending/depsets) of files produced during command evaluation. +Transitively included depsets are identified by `NamedSetOfFilesId`. -For more information on interpreting a stream's `NamedSetOfFiles` events, see the [BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). +For more information on interpreting a stream's `NamedSetOfFiles` events, see the +[BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). ## OptionsParsed -A single `OptionsParsed` event lists all options applied to the command, separating startup options from command options. It also includes the [InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. +A single `OptionsParsed` event lists all options applied to the command, +separating startup options from command options. It also includes the +[InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. ```json { @@ -214,7 +264,13 @@ A single `OptionsParsed` event lists all options applied to the command, separat ## PatternExpanded -`PatternExpanded` events indicate the set of all targets that match the patterns supplied on the commandline. For successful commands, a single event is present with all patterns in the `PatternExpandedId` and all targets in the `PatternExpanded` event's *children*. If the pattern expands to any `test_suite`s the set of test targets included by the `test_suite`. For each pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) event with a `PatternExpandedId` identifying the pattern. +`PatternExpanded` events indicate the set of all targets that match the patterns +supplied on the commandline. For successful commands, a single event is present +with all patterns in the `PatternExpandedId` and all targets in the +`PatternExpanded` event's *children*. If the pattern expands to any +`test_suite`s the set of test targets included by the `test_suite`. For each +pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) +event with a `PatternExpandedId` identifying the pattern. ```json { @@ -238,11 +294,16 @@ A single `OptionsParsed` event lists all options applied to the command, separat ## Progress -Progress events contain the standard output and standard error produced by Bazel during command execution. These events are also auto-generated as needed to announce events that have not been announced by a logical "parent" event (in particular, [NamedSetOfFiles](#namedsetoffiles).) +Progress events contain the standard output and standard error produced by Bazel +during command execution. These events are also auto-generated as needed to +announce events that have not been announced by a logical "parent" event (in +particular, [NamedSetOfFiles](#namedsetoffiles).) ## TargetComplete -For each `(target, configuration, aspect)` combination that completes the execution phase, a `TargetComplete` event is included in BEP. The event contains the target's success/failure and the target's requested output groups. +For each `(target, configuration, aspect)` combination that completes the +execution phase, a `TargetComplete` event is included in BEP. The event contains +the target's success/failure and the target's requested output groups. ```json { @@ -272,9 +333,14 @@ For each `(target, configuration, aspect)` combination that completes the execut ## TargetConfigured -For each Target that completes the analysis phase, a `TargetConfigured` event is included in BEP. This is the authoritative source for a target's "rule kind" attribute. The configuration(s) applied to the target appear in the announced *children* of the event. +For each Target that completes the analysis phase, a `TargetConfigured` event is +included in BEP. This is the authoritative source for a target's "rule kind" +attribute. The configuration(s) applied to the target appear in the announced +*children* of the event. -For example, building with the `--experimental_multi_cpu` options may produce the following `TargetConfigured` event for a single target with two configurations: +For example, building with the `--experimental_multi_cpu` options may produce +the following `TargetConfigured` event for a single target with two +configurations: ```json { @@ -309,26 +375,42 @@ For example, building with the `--experimental_multi_cpu` options may produce th ## TargetSummary -For each `(target, configuration)` pair that is executed, a `TargetSummary` event is included with an aggregate success result encompassing the configured target's execution and all aspects applied to that configured target. +For each `(target, configuration)` pair that is executed, a `TargetSummary` +event is included with an aggregate success result encompassing the configured +target's execution and all aspects applied to that configured target. ## TestResult -If testing is requested, a `TestResult` event is sent for each test attempt, shard, and run per test. This allows BEP consumers to identify precisely which test actions failed their tests and identify the test outputs (such as logs, test.xml files) for each test action. +If testing is requested, a `TestResult` event is sent for each test attempt, +shard, and run per test. This allows BEP consumers to identify precisely which +test actions failed their tests and identify the test outputs (such as logs, +test.xml files) for each test action. ## TestSummary -If testing is requested, a `TestSummary` event is sent for each test `(target, configuration)`, containing information necessary to interpret the test's results. The number of attempts, shards and runs per test are included to enable BEP consumers to differentiate artifacts across these dimensions. The attempts and runs per test are considered while producing the aggregate `TestStatus` to differentiate `FLAKY` tests from `FAILED` tests. +If testing is requested, a `TestSummary` event is sent for each test `(target, +configuration)`, containing information necessary to interpret the test's +results. The number of attempts, shards and runs per test are included to enable +BEP consumers to differentiate artifacts across these dimensions. The attempts +and runs per test are considered while producing the aggregate `TestStatus` to +differentiate `FLAKY` tests from `FAILED` tests. ## UnstructuredCommandLine -Unlike [CommandLine](#commandline), this event carries the unparsed commandline flags in string form as encountered by the build tool after expanding all [`.bazelrc`](/run/bazelrc) files and considering the `--config` flag. +Unlike [CommandLine](#commandline), this event carries the unparsed commandline +flags in string form as encountered by the build tool after expanding all +[`.bazelrc`](/run/bazelrc) files and +considering the `--config` flag. -The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a given command execution. +The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a +given command execution. ## WorkspaceConfig -A single `WorkspaceConfig` event contains configuration information regarding the workspace, such as the execution root. +A single `WorkspaceConfig` event contains configuration information regarding the +workspace, such as the execution root. ## WorkspaceStatus -A single `WorkspaceStatus` event contains the result of the [workspace status command](/docs/user-manual#workspace-status). +A single `WorkspaceStatus` event contains the result of the [workspace status +command](/docs/user-manual#workspace-status). diff --git a/remote/bep.mdx b/remote/bep.mdx index 4589f8b0..bafdaa9e 100644 --- a/remote/bep.mdx +++ b/remote/bep.mdx @@ -2,28 +2,67 @@ title: 'Build Event Protocol' --- -The [Build Event Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) (BEP) allows third-party programs to gain insight into a Bazel invocation. For example, you could use the BEP to gather information for an IDE plugin or a dashboard that displays build results. -The protocol is a set of [protocol buffer](https://developers.google.com/protocol-buffers/) messages with some semantics defined on top of it. It includes information about build and test results, build progress, the build configuration and much more. The BEP is intended to be consumed programmatically and makes parsing Bazel’s command line output a thing of the past. -The Build Event Protocol represents information about a build as events. A build event is a protocol buffer message consisting of a build event identifier, a set of child event identifiers, and a payload. - -- **Build Event Identifier:** Depending on the kind of build event, it might be an [opaque string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) or [structured information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) revealing more about the build event. A build event identifier is unique within a build. - -- **Children:** A build event may announce other build events, by including their build event identifiers in its [children field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). For example, the `PatternExpanded` build event announces the targets it expands to as children. The protocol guarantees that all events, except for the first event, are announced by a previous event. - -- **Payload:** The payload contains structured information about a build event, encoded as a protocol buffer message specific to that event. Note that the payload might not be the expected type, but could be an `Aborted` message if the build aborted prematurely. +The [Build Event +Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) +(BEP) allows third-party programs to gain insight into a Bazel invocation. For +example, you could use the BEP to gather information for an IDE +plugin or a dashboard that displays build results. + +The protocol is a set of [protocol +buffer](https://developers.google.com/protocol-buffers/) messages with some +semantics defined on top of it. It includes information about build and test +results, build progress, the build configuration and much more. The BEP is +intended to be consumed programmatically and makes parsing Bazel’s +command line output a thing of the past. + +The Build Event Protocol represents information about a build as events. A +build event is a protocol buffer message consisting of a build event identifier, +a set of child event identifiers, and a payload. + +* __Build Event Identifier:__ Depending on the kind of build event, it might be +an [opaque +string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) +or [structured +information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) +revealing more about the build event. A build event identifier is unique within +a build. + +* __Children:__ A build event may announce other build events, by including +their build event identifiers in its [children +field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). +For example, the `PatternExpanded` build event announces the targets it expands +to as children. The protocol guarantees that all events, except for the first +event, are announced by a previous event. + +* __Payload:__ The payload contains structured information about a build event, +encoded as a protocol buffer message specific to that event. Note that the +payload might not be the expected type, but could be an `Aborted` message +if the build aborted prematurely. ### Build event graph -All build events form a directed acyclic graph through their parent and child relationship. Every build event except for the initial build event has one or more parent events. Please note that not all parent events of a child event must necessarily be posted before it. When a build is complete (succeeded or failed) all announced events will have been posted. In case of a Bazel crash or a failed network transport, some announced build events may never be posted. - -The event graph's structure reflects the lifecycle of a command. Every BEP graph has the following characteristic shape: - -1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) event. All other events are its descendants. -2. Immediate children of the BuildStarted event contain metadata about the command. -3. Events containing data produced by the command, such as files built and test results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) event. -4. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed by events containing summary information about the build (for example, metric or profiling data). +All build events form a directed acyclic graph through their parent and child +relationship. Every build event except for the initial build event has one or +more parent events. Please note that not all parent events of a child event must +necessarily be posted before it. When a build is complete (succeeded or failed) +all announced events will have been posted. In case of a Bazel crash or a failed +network transport, some announced build events may never be posted. + +The event graph's structure reflects the lifecycle of a command. Every BEP +graph has the following characteristic shape: + +1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) + event. All other events are its descendants. +1. Immediate children of the BuildStarted event contain metadata about the + command. +1. Events containing data produced by the command, such as files built and test + results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) + event. +1. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed + by events containing summary information about the build (for example, metric + or profiling data). ## Consuming Build Event Protocol @@ -31,13 +70,21 @@ The event graph's structure reflects the lifecycle of a command. Every BEP graph To consume the BEP in a binary format: -1. Have Bazel serialize the protocol buffer messages to a file by specifying the option `--build_event_binary_file=/path/to/file`. The file will contain serialized protocol buffer messages with each message being length delimited. Each message is prefixed with its length encoded as a variable length integer. This format can be read using the protocol buffer library’s [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) method. +1. Have Bazel serialize the protocol buffer messages to a file by specifying the + option `--build_event_binary_file=/path/to/file`. The file will contain + serialized protocol buffer messages with each message being length delimited. + Each message is prefixed with its length encoded as a variable length integer. + This format can be read using the protocol buffer library’s + [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) + method. -2. Then, write a program that extracts the relevant information from the serialized protocol buffer message. +2. Then, write a program that extracts the relevant information from the + serialized protocol buffer message. ### Consume in text or JSON formats -The following Bazel command line flags will output the BEP in human-readable formats, such as text and JSON: +The following Bazel command line flags will output the BEP in +human-readable formats, such as text and JSON: ``` --build_event_text_file @@ -46,34 +93,56 @@ The following Bazel command line flags will output the BEP in human-readable for ## Build Event Service -The [Build Event Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event Service protocol is independent of the BEP and treats BEP events as opaque bytes. Bazel ships with a gRPC client implementation of the Build Event Service protocol that publishes Build Event Protocol events. One can specify the endpoint to send the events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, you must prefix the address with the appropriate scheme: `grpc://` for plaintext gRPC and `grpcs://` for gRPC with TLS enabled. +The [Build Event +Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) +Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event +Service protocol is independent of the BEP and treats BEP events as opaque bytes. +Bazel ships with a gRPC client implementation of the Build Event Service protocol that +publishes Build Event Protocol events. One can specify the endpoint to send the +events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, +you must prefix the address with the appropriate scheme: `grpc://` for plaintext +gRPC and `grpcs://` for gRPC with TLS enabled. ### Build Event Service flags Bazel has several flags related to the Build Event Service protocol, including: -- `--bes_backend` -- `--[no]bes_lifecycle_events` -- `--bes_results_url` -- `--bes_timeout` -- `--bes_instance_name` +* `--bes_backend` +* `--[no]bes_lifecycle_events` +* `--bes_results_url` +* `--bes_timeout` +* `--bes_instance_name` -For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the +[Command-Line Reference](/reference/command-line-reference). ### Authentication and security -Bazel’s Build Event Service implementation also supports authentication and TLS. These settings can be controlled using the below flags. Please note that these flags are also used for Bazel’s Remote Execution. This implies that the Build Event Service and Remote Execution Endpoints need to share the same authentication and TLS infrastructure. +Bazel’s Build Event Service implementation also supports authentication and TLS. +These settings can be controlled using the below flags. Please note that these +flags are also used for Bazel’s Remote Execution. This implies that the Build +Event Service and Remote Execution Endpoints need to share the same +authentication and TLS infrastructure. -- `--[no]google_default_credentials` -- `--google_credentials` -- `--google_auth_scopes` -- `--tls_certificate` -- `--[no]tls_enabled` +* `--[no]google_default_credentials` +* `--google_credentials` +* `--google_auth_scopes` +* `--tls_certificate` +* `--[no]tls_enabled` -For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the +[Command-Line Reference](/reference/command-line-reference). ### Build Event Service and remote caching -The BEP typically contains many references to log files (test.log, test.xml, etc. ) stored on the machine where Bazel is running. A remote BES server typically can't access these files as they are on different machines. A way to work around this issue is to use Bazel with [remote caching](/remote/caching). Bazel will upload all output files to the remote cache (including files referenced in the BEP) and the BES server can then fetch the referenced files from the cache. - -See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for more details. +The BEP typically contains many references to log files (test.log, test.xml, +etc. ) stored on the machine where Bazel is running. A remote BES server +typically can't access these files as they are on different machines. A way to +work around this issue is to use Bazel with [remote +caching](/remote/caching). +Bazel will upload all output files to the remote cache (including files +referenced in the BEP) and the BES server can then fetch the referenced files +from the cache. + +See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for +more details. diff --git a/remote/cache-local.mdx b/remote/cache-local.mdx index 49b8d33b..a3415a64 100644 --- a/remote/cache-local.mdx +++ b/remote/cache-local.mdx @@ -2,23 +2,39 @@ title: 'Debugging Remote Cache Hits for Local Execution' --- -This page describes how to investigate cache misses in the context of local execution. -This page assumes that you have a build and/or test that successfully builds locally and is set up to utilize remote caching, and that you want to ensure that the remote cache is being effectively utilized. -For tips on how to check your cache hit rate and how to compare the execution logs between two Bazel invocations, see [Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). Everything presented in that guide also applies to remote caching with local execution. However, local execution presents some additional challenges. +This page describes how to investigate cache misses in the context of local +execution. + +This page assumes that you have a build and/or test that successfully builds +locally and is set up to utilize remote caching, and that you want to ensure +that the remote cache is being effectively utilized. + +For tips on how to check your cache hit rate and how to compare the execution +logs between two Bazel invocations, see +[Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). +Everything presented in that guide also applies to remote caching with local +execution. However, local execution presents some additional challenges. ## Checking your cache hit rate -Successful remote cache hits will show up in the status line, similar to [Cache Hits rate with Remote Execution](/remote/cache-remote#check-cache-hits). +Successful remote cache hits will show up in the status line, similar to +[Cache Hits rate with Remote +Execution](/remote/cache-remote#check-cache-hits). -In the standard output of your Bazel run, you will see something like the following: +In the standard output of your Bazel run, you will see something like the +following: -```none +```none {:.devsite-disable-click-to-copy} INFO: 7 processes: 3 remote cache hit, 4 linux-sandbox. ``` -This means that out of 7 attempted actions, 3 got a remote cache hit and 4 actions did not have cache hits and were executed locally using `linux-sandbox` strategy. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. +This means that out of 7 attempted actions, 3 got a remote cache hit and 4 +actions did not have cache hits and were executed locally using `linux-sandbox` +strategy. Local cache hits are not included in this summary. If you are getting +0 processes (or a number lower than expected), run `bazel clean` followed by +your build/test command. ## Troubleshooting cache hits @@ -26,30 +42,50 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure successful communication with the remote endpoint -To ensure your build is successfully communicating with the remote cache, follow the steps in this section. +To ensure your build is successfully communicating with the remote cache, follow +the steps in this section. 1. Check your output for warnings - With remote execution, a failure to talk to the remote endpoint would fail your build. On the other hand, a cacheable local build would not fail if it cannot cache. Check the output of your Bazel invocation for warnings, such as: + With remote execution, a failure to talk to the remote endpoint would fail + your build. On the other hand, a cacheable local build would not fail if it + cannot cache. Check the output of your Bazel invocation for warnings, such + as: - ```none + ```none WARNING: Error reading from the remote cache: ``` + or - ```none + ```none WARNING: Error writing to the remote cache: ``` - Such warnings will be followed by the error message detailing the connection problem that should help you debug: for example, mistyped endpoint name or incorrectly set credentials. Find and address any such errors. If the error message you see does not give you enough information, try adding `--verbose_failures`. -2. Follow the steps from [Troubleshooting cache hits for remote execution](/remote/cache-remote#troubleshooting_cache_hits) to ensure that your cache-writing Bazel invocations are able to get cache hits on the same machine and across machines. + Such warnings will be followed by the error message detailing the connection + problem that should help you debug: for example, mistyped endpoint name or + incorrectly set credentials. Find and address any such errors. If the error + message you see does not give you enough information, try adding + `--verbose_failures`. + +2. Follow the steps from [Troubleshooting cache hits for remote + execution](/remote/cache-remote#troubleshooting_cache_hits) to + ensure that your cache-writing Bazel invocations are able to get cache hits + on the same machine and across machines. 3. Ensure your cache-reading Bazel invocations can get cache hits. - a. Since cache-reading Bazel invocations will have a different command-line set up, take additional care to ensure that they are properly set up to communicate with the remote cache. Ensure the `--remote_cache` flag is set and there are no warnings in the output. + a. Since cache-reading Bazel invocations will have a different command-line set + up, take additional care to ensure that they are properly set up to + communicate with the remote cache. Ensure the `--remote_cache` flag is set + and there are no warnings in the output. - b. Ensure your cache-reading Bazel invocations build the same targets as the cache-writing Bazel invocations. + b. Ensure your cache-reading Bazel invocations build the same targets as the + cache-writing Bazel invocations. - c. Follow the same steps as to [ensure caching across machines](/remote/cache-remote#caching-across-machines), to ensure caching from your cache-writing Bazel invocation to your cache-reading Bazel invocation. + c. Follow the same steps as to [ensure caching across + machines](/remote/cache-remote#caching-across-machines), + to ensure caching from your cache-writing Bazel invocation to your + cache-reading Bazel invocation. diff --git a/remote/cache-remote.mdx b/remote/cache-remote.mdx index 3b96276a..896f1e58 100644 --- a/remote/cache-remote.mdx +++ b/remote/cache-remote.mdx @@ -2,21 +2,36 @@ title: 'Debugging Remote Cache Hits for Remote Execution' --- -This page describes how to check your cache hit rate and how to investigate cache misses in the context of remote execution. -This page assumes that you have a build and/or test that successfully utilizes remote execution, and you want to ensure that you are effectively utilizing remote cache. + +This page describes how to check your cache hit rate and how to investigate +cache misses in the context of remote execution. + +This page assumes that you have a build and/or test that successfully +utilizes remote execution, and you want to ensure that you are effectively +utilizing remote cache. ## Checking your cache hit rate -In the standard output of your Bazel run, look at the `INFO` line that lists processes, which roughly correspond to Bazel actions. That line details where the action was run. Look for the `remote` label, which indicates an action executed remotely, `linux-sandbox` for actions executed in a local sandbox, and other values for other execution strategies. An action whose result came from a remote cache is displayed as `remote cache hit`. +In the standard output of your Bazel run, look at the `INFO` line that lists +processes, which roughly correspond to Bazel actions. That line details +where the action was run. Look for the `remote` label, which indicates an action +executed remotely, `linux-sandbox` for actions executed in a local sandbox, +and other values for other execution strategies. An action whose result came +from a remote cache is displayed as `remote cache hit`. For example: -```none +```none {:.devsite-disable-click-to-copy} INFO: 11 processes: 6 remote cache hit, 3 internal, 2 remote. ``` -In this example there were 6 remote cache hits, and 2 actions did not have cache hits and were executed remotely. The 3 internal part can be ignored. It is typically tiny internal actions, such as creating symbolic links. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. +In this example there were 6 remote cache hits, and 2 actions did not have +cache hits and were executed remotely. The 3 internal part can be ignored. +It is typically tiny internal actions, such as creating symbolic links. Local +cache hits are not included in this summary. If you are getting 0 processes +(or a number lower than expected), run `bazel clean` followed by your build/test +command. ## Troubleshooting cache hits @@ -24,47 +39,80 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure re-running the same build/test command produces cache hits -1. Run the build(s) and/or test(s) that you expect to populate the cache. The first time a new build is run on a particular stack, you can expect no remote cache hits. As part of remote execution, action results are stored in the cache and a subsequent run should pick them up. +1. Run the build(s) and/or test(s) that you expect to populate the cache. The + first time a new build is run on a particular stack, you can expect no remote + cache hits. As part of remote execution, action results are stored in the + cache and a subsequent run should pick them up. -2. Run `bazel clean`. This command cleans your local cache, which allows you to investigate remote cache hits without the results being masked by local cache hits. +2. Run `bazel clean`. This command cleans your local cache, which allows + you to investigate remote cache hits without the results being masked by + local cache hits. -3. Run the build(s) and test(s) that you are investigating again (on the same machine). +3. Run the build(s) and test(s) that you are investigating again (on the same + machine). -4. Check the `INFO` line for cache hit rate. If you see no processes except `remote cache hit` and `internal`, then your cache is being correctly populated and accessed. In that case, skip to the next section. +4. Check the `INFO` line for cache hit rate. If you see no processes except + `remote cache hit` and `internal`, then your cache is being correctly populated and + accessed. In that case, skip to the next section. -5. A likely source of discrepancy is something non-hermetic in the build causing the actions to receive different action keys across the two runs. To find those actions, do the following: +5. A likely source of discrepancy is something non-hermetic in the build causing + the actions to receive different action keys across the two runs. To find + those actions, do the following: a. Re-run the build(s) or test(s) in question to obtain execution logs: - ```posix-terminal - bazel clean + ```posix-terminal + bazel clean - bazel <var>--optional-flags</var> build //<var>your:target</var> --execution_log_compact_file=/tmp/exec1.log - ``` + bazel --optional-flags build //your:target --execution_log_compact_file=/tmp/exec1.log + ``` - b. [Compare the execution logs](#compare-logs) between the two runs. Ensure that the actions are identical across the two log files. Discrepancies provide a clue about the changes that occurred between the runs. Update your build to eliminate those discrepancies. + b. [Compare the execution logs](#compare-logs) between the + two runs. Ensure that the actions are identical across the two log files. + Discrepancies provide a clue about the changes that occurred between the + runs. Update your build to eliminate those discrepancies. - If you are able to resolve the caching problems and now the repeated run produces all cache hits, skip to the next section. + If you are able to resolve the caching problems and now the repeated run + produces all cache hits, skip to the next section. - If your action IDs are identical but there are no cache hits, then something in your configuration is preventing caching. Continue with this section to check for common problems. + If your action IDs are identical but there are no cache hits, then something + in your configuration is preventing caching. Continue with this section to + check for common problems. -6. Check that all actions in the execution log have `cacheable` set to true. If `cacheable` does not appear in the execution log for a give action, that means that the corresponding rule may have a `no-cache` tag in its definition in the `BUILD` file. Look at the `mnemonic` and `target_label` fields in the execution log to help determine where the action is coming from. +5. Check that all actions in the execution log have `cacheable` set to true. If + `cacheable` does not appear in the execution log for a give action, that + means that the corresponding rule may have a `no-cache` tag in its + definition in the `BUILD` file. Look at the `mnemonic` and `target_label` + fields in the execution log to help determine where the action is coming + from. -7. If the actions are identical and `cacheable` but there are no cache hits, it is possible that your command line includes `--noremote_accept_cached` which would disable cache lookups for a build. +6. If the actions are identical and `cacheable` but there are no cache hits, it + is possible that your command line includes `--noremote_accept_cached` which + would disable cache lookups for a build. - If figuring out the actual command line is difficult, use the canonical command line from the [Build Event Protocol](/remote/bep) as follows: + If figuring out the actual command line is difficult, use the canonical + command line from the + [Build Event Protocol](/remote/bep) + as follows: - a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get the text version of the log. + a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get + the text version of the log. - b. Open the text version of the log and search for the `structured_command_line` message with `command_line_label: "canonical"`. It will list all the options after expansion. + b. Open the text version of the log and search for the + `structured_command_line` message with `command_line_label: "canonical"`. + It will list all the options after expansion. c. Search for `remote_accept_cached` and check whether it's set to `false`. - d. If `remote_accept_cached` is `false`, determine where it is being set to `false`: either at the command line or in a [bazelrc](/run/bazelrc#bazelrc-file-locations) file. + d. If `remote_accept_cached` is `false`, determine where it is being + set to `false`: either at the command line or in a + [bazelrc](/run/bazelrc#bazelrc-file-locations) file. ### Ensure caching across machines -After cache hits are happening as expected on the same machine, run the same build(s)/test(s) on a different machine. If you suspect that caching is not happening across machines, do the following: +After cache hits are happening as expected on the same machine, run the +same build(s)/test(s) on a different machine. If you suspect that caching is +not happening across machines, do the following: 1. Make a small modification to your build to avoid hitting existing caches. @@ -76,7 +124,8 @@ After cache hits are happening as expected on the same machine, run the same bui bazel ... build ... --execution_log_compact_file=/tmp/exec1.log ``` -3. Run the build on the second machine, ensuring the modification from step 1 is included: +3. Run the build on the second machine, ensuring the modification from step 1 + is included: ```posix-terminal bazel clean @@ -84,34 +133,47 @@ After cache hits are happening as expected on the same machine, run the same bui bazel ... build ... --execution_log_compact_file=/tmp/exec2.log ``` -4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two runs. If the logs are not identical, investigate your build configurations for discrepancies as well as properties from the host environment leaking into either of the builds. +4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two + runs. If the logs are not identical, investigate your build configurations + for discrepancies as well as properties from the host environment leaking + into either of the builds. ## Comparing the execution logs -The execution log contains records of actions executed during the build. Each record describes both the inputs (not only files, but also command line arguments, environment variables, etc) and the outputs of the action. Thus, examination of the log can reveal why an action was reexecuted. +The execution log contains records of actions executed during the build. +Each record describes both the inputs (not only files, but also command line +arguments, environment variables, etc) and the outputs of the action. Thus, +examination of the log can reveal why an action was reexecuted. -The execution log can be produced in one of three formats: compact (`--execution_log_compact_file`), binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). The compact format is recommended, as it produces much smaller files with very little runtime overhead. The following instructions work for any format. You can also convert between them using the `//src/tools/execlog:converter` tool. +The execution log can be produced in one of three formats: +compact (`--execution_log_compact_file`), +binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). +The compact format is recommended, as it produces much smaller files with very +little runtime overhead. The following instructions work for any format. You +can also convert between them using the `//src/tools/execlog:converter` tool. -To compare logs for two builds that are not sharing cache hits as expected, do the following: +To compare logs for two builds that are not sharing cache hits as expected, +do the following: -1. Get the execution logs from each build and store them as `/tmp/exec1.log` and `/tmp/exec2.log`. +1. Get the execution logs from each build and store them as `/tmp/exec1.log` and + `/tmp/exec2.log`. -2. Download the Bazel source code and build the `//src/tools/execlog:parser` tool: +2. Download the Bazel source code and build the `//src/tools/execlog:parser` + tool: - ``` - git clone https://github.com/bazelbuild/bazel.git - cd bazel - bazel build //src/tools/execlog:parser - ``` + git clone https://github.com/bazelbuild/bazel.git + cd bazel + bazel build //src/tools/execlog:parser -3. Use the `//src/tools/execlog:parser` tool to convert the logs into a human-readable text format. In this format, the actions in the second log are sorted to match the order in the first log, making a comparison easier. +3. Use the `//src/tools/execlog:parser` tool to convert the logs into a + human-readable text format. In this format, the actions in the second log are + sorted to match the order in the first log, making a comparison easier. - ``` - bazel-bin/src/tools/execlog/parser \ - --log_path=/tmp/exec1.log \ - --log_path=/tmp/exec2.log \ - --output_path=/tmp/exec1.log.txt \ - --output_path=/tmp/exec2.log.txt - ``` + bazel-bin/src/tools/execlog/parser \ + --log_path=/tmp/exec1.log \ + --log_path=/tmp/exec2.log \ + --output_path=/tmp/exec1.log.txt \ + --output_path=/tmp/exec2.log.txt -4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and `/tmp/exec2.log.txt`. +4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and + `/tmp/exec2.log.txt`. diff --git a/remote/caching.mdx b/remote/caching.mdx index 1729ee02..f85f2c8e 100644 --- a/remote/caching.mdx +++ b/remote/caching.mdx @@ -2,65 +2,94 @@ title: 'Remote Caching' --- -This page covers remote caching, setting up a server to host the cache, and running builds using the remote cache. -A remote cache is used by a team of developers and/or a continuous integration (CI) system to share build outputs. If your build is reproducible, the outputs from one machine can be safely reused on another machine, which can make builds significantly faster. + +This page covers remote caching, setting up a server to host the cache, and +running builds using the remote cache. + +A remote cache is used by a team of developers and/or a continuous integration +(CI) system to share build outputs. If your build is reproducible, the +outputs from one machine can be safely reused on another machine, which can +make builds significantly faster. ## Overview -Bazel breaks a build into discrete steps, which are called actions. Each action has inputs, output names, a command line, and environment variables. Required inputs and expected outputs are declared explicitly for each action. +Bazel breaks a build into discrete steps, which are called actions. Each action +has inputs, output names, a command line, and environment variables. Required +inputs and expected outputs are declared explicitly for each action. -You can set up a server to be a remote cache for build outputs, which are these action outputs. These outputs consist of a list of output file names and the hashes of their contents. With a remote cache, you can reuse build outputs from another user's build rather than building each new output locally. +You can set up a server to be a remote cache for build outputs, which are these +action outputs. These outputs consist of a list of output file names and the +hashes of their contents. With a remote cache, you can reuse build outputs +from another user's build rather than building each new output locally. To use remote caching: -- Set up a server as the cache's backend -- Configure the Bazel build to use the remote cache -- Use Bazel version 0.10.0 or later +* Set up a server as the cache's backend +* Configure the Bazel build to use the remote cache +* Use Bazel version 0.10.0 or later The remote cache stores two types of data: -- The action cache, which is a map of action hashes to action result metadata. -- A content-addressable store (CAS) of output files. +* The action cache, which is a map of action hashes to action result metadata. +* A content-addressable store (CAS) of output files. -Note that the remote cache additionally stores the stdout and stderr for every action. Inspecting the stdout/stderr of Bazel thus is not a good signal for [estimating cache hits](/remote/cache-local). +Note that the remote cache additionally stores the stdout and stderr for every +action. Inspecting the stdout/stderr of Bazel thus is not a good signal for +[estimating cache hits](/remote/cache-local). ### How a build uses remote caching -Once a server is set up as the remote cache, you use the cache in multiple ways: - -- Read and write to the remote cache -- Read and/or write to the remote cache except for specific targets -- Only read from the remote cache -- Not use the remote cache at all - -When you run a Bazel build that can read and write to the remote cache, the build follows these steps: - -1. Bazel creates the graph of targets that need to be built, and then creates a list of required actions. Each of these actions has declared inputs and output filenames. -2. Bazel checks your local machine for existing build outputs and reuses any that it finds. -3. Bazel checks the cache for existing build outputs. If the output is found, Bazel retrieves the output. This is a cache hit. -4. For required actions where the outputs were not found, Bazel executes the actions locally and creates the required build outputs. +Once a server is set up as the remote cache, you use the cache in multiple +ways: + +* Read and write to the remote cache +* Read and/or write to the remote cache except for specific targets +* Only read from the remote cache +* Not use the remote cache at all + +When you run a Bazel build that can read and write to the remote cache, +the build follows these steps: + +1. Bazel creates the graph of targets that need to be built, and then creates +a list of required actions. Each of these actions has declared inputs +and output filenames. +2. Bazel checks your local machine for existing build outputs and reuses any +that it finds. +3. Bazel checks the cache for existing build outputs. If the output is found, +Bazel retrieves the output. This is a cache hit. +4. For required actions where the outputs were not found, Bazel executes the +actions locally and creates the required build outputs. 5. New build outputs are uploaded to the remote cache. ## Setting up a server as the cache's backend -You need to set up a server to act as the cache's backend. A HTTP/1.1 server can treat Bazel's data as opaque bytes and so many existing servers can be used as a remote caching backend. Bazel's [HTTP Caching Protocol](#http-caching) is what supports remote caching. +You need to set up a server to act as the cache's backend. A HTTP/1.1 +server can treat Bazel's data as opaque bytes and so many existing servers +can be used as a remote caching backend. Bazel's +[HTTP Caching Protocol](#http-caching) is what supports remote +caching. -You are responsible for choosing, setting up, and maintaining the backend server that will store the cached outputs. When choosing a server, consider: +You are responsible for choosing, setting up, and maintaining the backend +server that will store the cached outputs. When choosing a server, consider: -- Networking speed. For example, if your team is in the same office, you may want to run your own local server. -- Security. The remote cache will have your binaries and so needs to be secure. -- Ease of management. For example, Google Cloud Storage is a fully managed service. +* Networking speed. For example, if your team is in the same office, you may +want to run your own local server. +* Security. The remote cache will have your binaries and so needs to be secure. +* Ease of management. For example, Google Cloud Storage is a fully managed service. -There are many backends that can be used for a remote cache. Some options include: +There are many backends that can be used for a remote cache. Some options +include: -- [nginx](#nginx) -- [bazel-remote](#bazel-remote) -- [Google Cloud Storage](#cloud-storage) +* [nginx](#nginx) +* [bazel-remote](#bazel-remote) +* [Google Cloud Storage](#cloud-storage) ### nginx -nginx is an open source web server. With its \[WebDAV module], it can be used as a remote cache for Bazel. On Debian and Ubuntu you can install the `nginx-extras` package. On macOS nginx is available via Homebrew: +nginx is an open source web server. With its [WebDAV module], it can be +used as a remote cache for Bazel. On Debian and Ubuntu you can install the +`nginx-extras` package. On macOS nginx is available via Homebrew: ```posix-terminal brew tap denji/nginx @@ -68,7 +97,12 @@ brew tap denji/nginx brew install nginx-full --with-webdav ``` -Below is an example configuration for nginx. Note that you will need to change `/path/to/cache/dir` to a valid directory where nginx has permission to write and read. You may need to change `client_max_body_size` option to a larger value if you have larger output files. The server will require other configuration such as authentication. +Below is an example configuration for nginx. Note that you will need to +change `/path/to/cache/dir` to a valid directory where nginx has permission +to write and read. You may need to change `client_max_body_size` option to a +larger value if you have larger output files. The server will require other +configuration such as authentication. + Example configuration for `server` section in `nginx.conf`: @@ -88,44 +122,72 @@ location /cache/ { ### bazel-remote -bazel-remote is an open source remote build cache that you can use on your infrastructure. It has been successfully used in production at several companies since early 2018. Note that the Bazel project does not provide technical support for bazel-remote. +bazel-remote is an open source remote build cache that you can use on +your infrastructure. It has been successfully used in production at +several companies since early 2018. Note that the Bazel project does +not provide technical support for bazel-remote. -This cache stores contents on disk and also provides garbage collection to enforce an upper storage limit and clean unused artifacts. The cache is available as a \[docker image] and its code is available on [GitHub](https://github.com/buchgr/bazel-remote/). Both the REST and gRPC remote cache APIs are supported. +This cache stores contents on disk and also provides garbage collection +to enforce an upper storage limit and clean unused artifacts. The cache is +available as a [docker image] and its code is available on +[GitHub](https://github.com/buchgr/bazel-remote/). +Both the REST and gRPC remote cache APIs are supported. -Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) page for instructions on how to use it. +Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) +page for instructions on how to use it. ### Google Cloud Storage -\[Google Cloud Storage] is a fully managed object store which provides an HTTP API that is compatible with Bazel's remote caching protocol. It requires that you have a Google Cloud account with billing enabled. +[Google Cloud Storage] is a fully managed object store which provides an +HTTP API that is compatible with Bazel's remote caching protocol. It requires +that you have a Google Cloud account with billing enabled. To use Cloud Storage as the cache: -1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). Ensure that you select a bucket location that's closest to you, as network bandwidth is important for the remote cache. +1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). +Ensure that you select a bucket location that's closest to you, as network bandwidth +is important for the remote cache. -2. Create a service account for Bazel to authenticate to Cloud Storage. See [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). +2. Create a service account for Bazel to authenticate to Cloud Storage. See +[Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). -3. Generate a secret JSON key and then pass it to Bazel for authentication. Store the key securely, as anyone with the key can read and write arbitrary data to/from your GCS bucket. +3. Generate a secret JSON key and then pass it to Bazel for authentication. Store +the key securely, as anyone with the key can read and write arbitrary data +to/from your GCS bucket. 4. Connect to Cloud Storage by adding the following flags to your Bazel command: + * Pass the following URL to Bazel by using the flag: + `--remote_cache=https://storage.googleapis.com/bucket-name` where `bucket-name` is the name of your storage bucket. + * Pass the authentication key using the flag: `--google_credentials=/path/to/your/secret-key.json`, or + `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). - - Pass the following URL to Bazel by using the flag: `--remote_cache=https://storage.googleapis.com<var>/bucket-name</var>` where `bucket-name` is the name of your storage bucket. - - Pass the authentication key using the flag: `--google_credentials=<var>/path/to/your/secret-key</var>.json`, or `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). - -5. You can configure Cloud Storage to automatically delete old files. To do so, see [Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). +5. You can configure Cloud Storage to automatically delete old files. To do so, see +[Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). ### Other servers -You can set up any HTTP/1.1 server that supports PUT and GET as the cache's backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), [Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). +You can set up any HTTP/1.1 server that supports PUT and GET as the cache's +backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), +[Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). ## Authentication -As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. You can pass a username and password to Bazel via the remote cache URL. The syntax is `https://username:password@hostname.com:port/path`. Note that HTTP Basic Authentication transmits username and password in plaintext over the network and it's thus critical to always use it with HTTPS. +As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. +You can pass a username and password to Bazel via the remote cache URL. The +syntax is `https://username:password@hostname.com:port/path`. Note that +HTTP Basic Authentication transmits username and password in plaintext over the +network and it's thus critical to always use it with HTTPS. ## HTTP caching protocol -Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. Action result metadata is stored under the path `/ac/` and output files are stored under the path `/cas/`. +Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: +Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. +Action result metadata is stored under the path `/ac/` and output files are stored +under the path `/cas/`. -For example, consider a remote cache running under `http://localhost:8080/cache`. A Bazel request to download action result metadata for an action with the SHA256 hash `01ba4719...` will look as follows: +For example, consider a remote cache running under `http://localhost:8080/cache`. +A Bazel request to download action result metadata for an action with the SHA256 +hash `01ba4719...` will look as follows: ```http GET /cache/ac/01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b HTTP/1.1 @@ -134,7 +196,8 @@ Accept: */* Connection: Keep-Alive ``` -A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to the CAS will look as follows: +A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to +the CAS will look as follows: ```http PUT /cache/cas/15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225 HTTP/1.1 @@ -148,29 +211,36 @@ Connection: Keep-Alive ## Run Bazel using the remote cache -Once a server is set up as the remote cache, to use the remote cache you need to add flags to your Bazel command. See list of configurations and their flags below. +Once a server is set up as the remote cache, to use the remote cache you +need to add flags to your Bazel command. See list of configurations and +their flags below. -You may also need configure authentication, which is specific to your chosen server. +You may also need configure authentication, which is specific to your +chosen server. -You may want to add these flags in a `.bazelrc` file so that you don't need to specify them every time you run Bazel. Depending on your project and team dynamics, you can add flags to a `.bazelrc` file that is: +You may want to add these flags in a `.bazelrc` file so that you don't +need to specify them every time you run Bazel. Depending on your project and +team dynamics, you can add flags to a `.bazelrc` file that is: -- On your local machine -- In your project's workspace, shared with the team -- On the CI system +* On your local machine +* In your project's workspace, shared with the team +* On the CI system ### Read from and write to the remote cache -Take care in who has the ability to write to the remote cache. You may want only your CI system to be able to write to the remote cache. +Take care in who has the ability to write to the remote cache. You may want +only your CI system to be able to write to the remote cache. Use the following flag to read from and write to the remote cache: ```posix-terminal -build --remote_cache=http://<var>your.host:port</var> +build --remote_cache=http://{{ '' }}your.host:port{{ '' }} ``` Besides `HTTP`, the following protocols are also supported: `HTTPS`, `grpc`, `grpcs`. -Use the following flag in addition to the one above to only read from the remote cache: +Use the following flag in addition to the one above to only read from the +remote cache: ```posix-terminal build --remote_upload_local_results=false @@ -178,7 +248,8 @@ build --remote_upload_local_results=false ### Exclude specific targets from using the remote cache -To exclude specific targets from using the remote cache, tag the target with `no-remote-cache`. For example: +To exclude specific targets from using the remote cache, tag the target with +`no-remote-cache`. For example: ```starlark java_library( @@ -189,100 +260,121 @@ java_library( ### Delete content from the remote cache -Deleting content from the remote cache is part of managing your server. How you delete content from the remote cache depends on the server you have set up as the cache. When deleting outputs, either delete the entire cache, or delete old outputs. +Deleting content from the remote cache is part of managing your server. +How you delete content from the remote cache depends on the server you have +set up as the cache. When deleting outputs, either delete the entire cache, +or delete old outputs. -The cached outputs are stored as a set of names and hashes. When deleting content, there's no way to distinguish which output belongs to a specific build. +The cached outputs are stored as a set of names and hashes. When deleting +content, there's no way to distinguish which output belongs to a specific +build. You may want to delete content from the cache to: -- Create a clean cache after a cache was poisoned -- Reduce the amount of storage used by deleting old outputs +* Create a clean cache after a cache was poisoned +* Reduce the amount of storage used by deleting old outputs ### Unix sockets -The remote HTTP cache supports connecting over unix domain sockets. The behavior is similar to curl's `--unix-socket` flag. Use the following to configure unix domain socket: +The remote HTTP cache supports connecting over unix domain sockets. The behavior +is similar to curl's `--unix-socket` flag. Use the following to configure unix +domain socket: ```posix-terminal - build --remote_cache=http://<var>your.host:port</var> - build --remote_proxy=unix:/<var>path/to/socket</var> + build --remote_cache=http://{{ '' }}your.host:port{{ '' }} + build --remote_proxy=unix:/{{ '' }}path/to/socket{{ '' }} ``` This feature is unsupported on Windows. ## Disk cache -Bazel can use a directory on the file system as a remote cache. This is useful for sharing build artifacts when switching branches and/or working on multiple workspaces of the same project, such as multiple checkouts. Enable the disk cache as follows: +Bazel can use a directory on the file system as a remote cache. This is +useful for sharing build artifacts when switching branches and/or working +on multiple workspaces of the same project, such as multiple checkouts. +Enable the disk cache as follows: ```posix-terminal -build --disk_cache=<var>path/to/build/cache</var> +build --disk_cache={{ '' }}path/to/build/cache{{ '' }} ``` -You can pass a user-specific path to the `--disk_cache` flag using the `~` alias (Bazel will substitute the current user's home directory). This comes in handy when enabling the disk cache for all developers of a project via the project's checked in `.bazelrc` file. +You can pass a user-specific path to the `--disk_cache` flag using the `~` alias +(Bazel will substitute the current user's home directory). This comes in handy +when enabling the disk cache for all developers of a project via the project's +checked in `.bazelrc` file. ### Garbage collection -Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and `--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache or for the age of individual cache entries. Bazel will automatically garbage collect the disk cache while idling between builds; the idle timer can be set with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). +Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and +`--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache +or for the age of individual cache entries. Bazel will automatically garbage +collect the disk cache while idling between builds; the idle timer can be set +with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). -As an alternative to automatic garbage collection, we also provide a [tool](https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a garbage collection on demand. +As an alternative to automatic garbage collection, we also provide a [tool]( +https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a +garbage collection on demand. ## Known issues **Input file modification during a build** -When an input file is modified during a build, Bazel might upload invalid results to the remote cache. You can enable a change detection with the `--experimental_guard_against_concurrent_changes` flag. There are no known issues and it will be enabled by default in a future release. See \[issue #3360] for updates. Generally, avoid modifying source files during a build. +When an input file is modified during a build, Bazel might upload invalid +results to the remote cache. You can enable a change detection with +the `--experimental_guard_against_concurrent_changes` flag. There +are no known issues and it will be enabled by default in a future release. +See [issue #3360] for updates. Generally, avoid modifying source files during a +build. **Environment variables leaking into an action** -An action definition contains environment variables. This can be a problem for sharing remote cache hits across machines. For example, environments with different `$PATH` variables won't share cache hits. Only environment variables explicitly whitelisted via `--action_env` are included in an action definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` with a whitelist of environment variables including `$PATH`. If you are getting fewer cache hits than expected, check that your environment doesn't have an old `/etc/bazel.bazelrc` file. +An action definition contains environment variables. This can be a problem for +sharing remote cache hits across machines. For example, environments with +different `$PATH` variables won't share cache hits. Only environment variables +explicitly whitelisted via `--action_env` are included in an action +definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` +with a whitelist of environment variables including `$PATH`. If you are getting +fewer cache hits than expected, check that your environment doesn't have an old +`/etc/bazel.bazelrc` file. **Bazel does not track tools outside a workspace** -Bazel currently does not track tools outside a workspace. This can be a problem if, for example, an action uses a compiler from `/usr/bin/`. Then, two users with different compilers installed will wrongly share cache hits because the outputs are different but they have the same action hash. See [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. +Bazel currently does not track tools outside a workspace. This can be a +problem if, for example, an action uses a compiler from `/usr/bin/`. Then, +two users with different compilers installed will wrongly share cache hits +because the outputs are different but they have the same action hash. See +[issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. -**Incremental in-memory state is lost when running builds inside docker containers** Bazel uses server/client architecture even when running in single docker container. On the server side, Bazel maintains an in-memory state which speeds up builds. When running builds inside docker containers such as in CI, the in-memory state is lost and Bazel must rebuild it before using the remote cache. +**Incremental in-memory state is lost when running builds inside docker containers** +Bazel uses server/client architecture even when running in single docker container. +On the server side, Bazel maintains an in-memory state which speeds up builds. +When running builds inside docker containers such as in CI, the in-memory state is lost +and Bazel must rebuild it before using the remote cache. ## External links -- **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. - -- **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) in which he benchmarks remote caching in Bazel. - -- [Adapting Rules for Remote Execution](/remote/rules) - -- [Troubleshooting Remote Execution](/remote/sandbox) - -- [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) - -- [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) - -- [bazel-remote](https://github.com/buchgr/bazel-remote/) - -- [Google Cloud Storage](https://cloud.google.com/storage) - -- [Google Cloud Console](https://cloud.google.com/console) - -- [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) - -- [Hazelcast](https://hazelcast.com) - -- [Apache httpd](http://httpd.apache.org) - -- [AWS S3](https://aws.amazon.com/s3) - -- [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) - -- [gRPC](https://grpc.io/) - -- [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) - -- [Buildbarn](https://github.com/buildbarn) - -- [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - -- [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - -- [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) - -- [Application Authentication](https://cloud.google.com/docs/authentication/production) - -- [NativeLink](https://github.com/TraceMachina/nativelink) +* **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. + +* **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) +in which he benchmarks remote caching in Bazel. + +* [Adapting Rules for Remote Execution](/remote/rules) +* [Troubleshooting Remote Execution](/remote/sandbox) +* [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) +* [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) +* [bazel-remote](https://github.com/buchgr/bazel-remote/) +* [Google Cloud Storage](https://cloud.google.com/storage) +* [Google Cloud Console](https://cloud.google.com/console) +* [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) +* [Hazelcast](https://hazelcast.com) +* [Apache httpd](http://httpd.apache.org) +* [AWS S3](https://aws.amazon.com/s3) +* [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) +* [gRPC](https://grpc.io/) +* [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) +* [Buildbarn](https://github.com/buildbarn) +* [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) +* [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) +* [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) +* [Application Authentication](https://cloud.google.com/docs/authentication/production) +* [NativeLink](https://github.com/TraceMachina/nativelink) diff --git a/remote/ci.mdx b/remote/ci.mdx index 9e5d786a..0a4e4488 100644 --- a/remote/ci.mdx +++ b/remote/ci.mdx @@ -2,24 +2,37 @@ title: 'Configuring Bazel CI to Test Rules for Remote Execution' --- -This page is for owners and maintainers of Bazel rule repositories. It describes how to configure the Bazel Continuous Integration (CI) system for your repository to test your rules for compatibility against a remote execution scenario. The instructions on this page apply to projects stored in GitHub repositories. + + +This page is for owners and maintainers of Bazel rule repositories. It +describes how to configure the Bazel Continuous Integration (CI) system for +your repository to test your rules for compatibility against a remote execution +scenario. The instructions on this page apply to projects stored in +GitHub repositories. ## Prerequisites Before completing the steps on this page, ensure the following: -- Your GitHub repository is part of the [Bazel GitHub organization](https://github.com/bazelbuild). -- You have configured Buildkite for your repository as described in [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). +* Your GitHub repository is part of the + [Bazel GitHub organization](https://github.com/bazelbuild). +* You have configured Buildkite for your repository as described in + [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). ## Setting up the Bazel CI for testing -1. In your `.bazelci/presubmit.yml` file, do the following: +1. In your `.bazelci/presubmit.yml` file, do the following: - a. Add a config named `rbe_ubuntu1604`. + a. Add a config named `rbe_ubuntu1604`. - b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. + b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. -2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) GitHub repository to your `WORKSPACE` file, pinned to the [latest release](https://releases.bazel.build/bazel-toolchains.html). Also add an `rbe_autoconfig` target with name `buildkite_config`. This example creates toolchain configuration for remote execution with BuildKite CI for `rbe_ubuntu1604`. +2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) + GitHub repository to your `WORKSPACE` file, pinned to the + [latest release](https://releases.bazel.build/bazel-toolchains.html). Also + add an `rbe_autoconfig` target with name `buildkite_config`. This example + creates toolchain configuration for remote execution with BuildKite CI + for `rbe_ubuntu1604`. ```posix-terminal load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") @@ -27,25 +40,43 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") rbe_autoconfig(name = "buildkite_config") ``` -3. Send a pull request with your changes to the `presubmit.yml` file. (See [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) +3. Send a pull request with your changes to the `presubmit.yml` file. (See + [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) -4. To view build results, click **Details** for the RBE (Ubuntu 16.04) pull request check in GitHub, as shown in the figure below. This link becomes available after the pull request has been merged and the CI tests have run. (See [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) +4. To view build results, click **Details** for the RBE (Ubuntu + 16.04) pull request check in GitHub, as shown in the figure below. This link + becomes available after the pull request has been merged and the CI tests + have run. (See + [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) - ![Example results](/docs/images/rbe-ci-1.png "Example results") + ![Example results](/docs/images/rbe-ci-1.png "Example results") -5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test required to pass before merging in your branch protection rule. The setting is located in GitHub in **Settings \> Branches \> Branch protection rules**, as shown in the following figure. +5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test + required to pass before merging in your branch protection rule. The setting + is located in GitHub in **Settings > Branches > Branch protection rules**, + as shown in the following figure. - ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") + ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") ## Troubleshooting failed builds and tests If your build or tests fail, it's likely due to the following: -- **Required build or test tools are not installed in the default container.** Builds using the `rbe_ubuntu1604` config run by default inside an [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container, which includes tools common to many Bazel builds. However, if your rules require tools not present in the default container, you must create a custom container based on the [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container and include those tools as described later. +* **Required build or test tools are not installed in the default container.** + Builds using the `rbe_ubuntu1604` config run by default inside an + [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) + container, which includes tools common to many Bazel builds. However, if + your rules require tools not present in the default container, you must + create a custom container based on the + [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) + container and include those tools as described later. -- **Build or test targets are using rules that are incompatible with remote execution.** See [Adapting Bazel Rules for Remote Execution](/remote/rules) for details about compatibility with remote execution. +* **Build or test targets are using rules that are incompatible with remote + execution.** See + [Adapting Bazel Rules for Remote Execution](/remote/rules) for + details about compatibility with remote execution. -## Using a custom container in the rbe\_ubuntu1604 CI config +## Using a custom container in the rbe_ubuntu1604 CI config The `rbe-ubuntu16-04` container is publicly available at the following URL: @@ -53,99 +84,121 @@ The `rbe-ubuntu16-04` container is publicly available at the following URL: http://gcr.io/cloud-marketplace/google/rbe-ubuntu16-04 ``` -You can pull it directly from Container Registry or build it from source. The next sections describe both options. +You can pull it directly from Container Registry or build it from source. The +next sections describe both options. -Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. If you are building the container from source, you must also install the latest version of Bazel. +Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. +If you are building the container from source, you must also install the latest +version of Bazel. ### Pulling the rbe-ubuntu16-04 from Container Registry -To pull the `rbe-ubuntu16-04` container from Container Registry, run the following command: +To pull the `rbe-ubuntu16-04` container from Container Registry, run the +following command: ```posix-terminal -gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> +gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} ``` -Replace \{\{ '`' }}sha256-checksum{{ '`' \}\} with the SHA256 checksum value for [the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). +Replace sha256-checksum with the SHA256 checksum value for +[the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). ### Building the rbe-ubuntu16-04 container from source To build the `rbe-ubuntu16-04` container from source, do the following: -1. Clone the `bazel-toolchains` repository: +1. Clone the `bazel-toolchains` repository: - ```posix-terminal - git clone https://github.com/bazelbuild/bazel-toolchains - ``` + ```posix-terminal + git clone https://github.com/bazelbuild/bazel-toolchains + ``` -2. Set up toolchain container targets and build the container as explained in [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). +2. Set up toolchain container targets and build the container as explained in + [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). -3. Pull the freshly built container: +3. Pull the freshly built container: - ```posix-terminal - ``` - -gcloud docker -- pull gcr.io/\{\{ '`' }}project-id{{ '`' \}\}/\{\{ '`' }}custom-container-name{{ '`' \}\}\{\{ '`' }}sha256-checksum{{ '`' \}\} \`\`\` + ```posix-terminal +gcloud docker -- pull gcr.io/project-id/custom-container-namesha256-checksum + ``` ### Running the custom container To run the custom container, do one of the following: -- If you pulled the container from Container Registry, run the following command: +* If you pulled the container from Container Registry, run the following + command: - ```posix-terminal - docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var>/bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:sha256-checksum/bin/bash + ``` - Replace `sha256-checksum` with the SHA256 checksum value for the [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). + Replace `sha256-checksum` with the SHA256 checksum value for the + [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). -- If you built the container from source, run the following command: +* If you built the container from source, run the following command: - ```posix-terminal - docker run -it gcr.io/<var>project-id</var>/<var>custom-container-name</var>@sha256:<var>sha256sum</var> /bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/project-id/custom-container-name@sha256:sha256sum /bin/bash + ``` ### Adding resources to the custom container -Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or [`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or alternate versions of the original resources to the `rbe-ubuntu16-04` container. If you are new to Docker, read the following: +Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or +[`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or +alternate versions of the original resources to the `rbe-ubuntu16-04` container. +If you are new to Docker, read the following: -- [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) -- [Docker Samples](https://docs.docker.com/samples/) +* [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) +* [Docker Samples](https://docs.docker.com/samples/) -For example, the following `Dockerfile` snippet installs `<var>my_tool_package</var>`: +For example, the following `Dockerfile` snippet installs `my_tool_package`: ``` -FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> -RUN apt-get update && yes | apt-get install -y <var>my_tool_package</var> +FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} +RUN apt-get update && yes | apt-get install -y {{ '' }}my_tool_package{{ '' }} ``` ### Pushing the custom container to Container Registry -Once you have customized the container, build the container image and push it to Container Registry as follows: +Once you have customized the container, build the container image and push it to +Container Registry as follows: 1. Build the container image: - ```posix-terminal - docker build -t <var>custom-container-name</var>. + ```posix-terminal + docker build -t custom-container-name. + + docker tag custom-container-name gcr.io/project-id/custom-container-name + ``` - docker tag <var>custom-container-name</var> gcr.io/<var>project-id</var>/<var>custom-container-name</var> - ``` +2. Push the container image to Container Registry: -2. Push the container image to Container Registry: + ```posix-terminal + gcloud docker -- push gcr.io/project-id/custom-container-name + ``` - ```posix-terminal - gcloud docker -- push gcr.io/<var>project-id</var>/<var>custom-container-name</var> - ``` +3. Navigate to the following URL to verify the container has been pushed: -3. Navigate to the following URL to verify the container has been pushed: + https://console.cloud.google.com/gcr/images/project-id/GLOBAL/custom-container-name - [https://console.cloud.google.com/gcr/images/\{\{](https://console.cloud.google.com/gcr/images/%7B%7B) '`' }}project-id{{ '`' \}\}/GLOBAL/\{\{ '`' }}custom-container-name{{ '`' \}\} +4. Take note of the SHA256 checksum of your custom container. You will need to + provide it in your build platform definition later. -4. Take note of the SHA256 checksum of your custom container. You will need to provide it in your build platform definition later. +5. Configure the container for public access as described in publicly + accessible as explained in + [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). -5. Configure the container for public access as described in publicly accessible as explained in [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). + For more information, see + [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). - For more information, see [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). ### Specifying the build platform definition -You must include a [Bazel platform](/extending/platforms) configuration in your custom toolchain configuration, which allows Bazel to select a toolchain appropriate to the desired hardware/software platform. To generate automatically a valid platform, you can add to your `WORKSPACE` an `rbe_autoconfig` target with name `buildkite_config` which includes additional attrs to select your custom container. For details on this setup, read the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). +You must include a [Bazel platform](/extending/platforms) configuration in your +custom toolchain configuration, which allows Bazel to select a toolchain +appropriate to the desired hardware/software platform. To generate +automatically a valid platform, you can add to your `WORKSPACE` an +`rbe_autoconfig` target with name `buildkite_config` which includes additional +attrs to select your custom container. For details on this setup, read +the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). diff --git a/remote/creating.mdx b/remote/creating.mdx index 4b2cf226..0e46a07a 100644 --- a/remote/creating.mdx +++ b/remote/creating.mdx @@ -2,30 +2,49 @@ title: 'Creating Persistent Workers' --- -[Persistent workers](/remote/persistent) can make your build faster. If you have repeated actions in your build that have a high startup cost or would benefit from cross-action caching, you may want to implement your own persistent worker to perform these actions. -The Bazel server communicates with the worker using `stdin`/`stdout`. It supports the use of protocol buffers or JSON strings. + +[Persistent workers](/remote/persistent) can make your build faster. If +you have repeated actions in your build that have a high startup cost or would +benefit from cross-action caching, you may want to implement your own persistent +worker to perform these actions. + +The Bazel server communicates with the worker using `stdin`/`stdout`. It +supports the use of protocol buffers or JSON strings. The worker implementation has two parts: -- The [worker](#making-worker). -- The [rule that uses the worker](#rule-uses-worker). +* The [worker](#making-worker). +* The [rule that uses the worker](#rule-uses-worker). ## Making the worker A persistent worker upholds a few requirements: -- It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) from its `stdin`. -- It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) (and only `WorkResponse`s) to its `stdout`. -- It accepts the `--persistent_worker` flag. The wrapper must recognize the `--persistent_worker` command-line flag and only make itself persistent if that flag is passed, otherwise it must do a one-shot compilation and exit. +* It reads + [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) + from its `stdin`. +* It writes + [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) + (and only `WorkResponse`s) to its `stdout`. +* It accepts the `--persistent_worker` flag. The wrapper must recognize the + `--persistent_worker` command-line flag and only make itself persistent if + that flag is passed, otherwise it must do a one-shot compilation and exit. -If your program upholds these requirements, it can be used as a persistent worker! +If your program upholds these requirements, it can be used as a persistent +worker! ### Work requests -A `WorkRequest` contains a list of arguments to the worker, a list of path-digest pairs representing the inputs the worker can access (this isn’t enforced, but you can use this info for caching), and a request id, which is 0 for singleplex workers. +A `WorkRequest` contains a list of arguments to the worker, a list of +path-digest pairs representing the inputs the worker can access (this isn’t +enforced, but you can use this info for caching), and a request id, which is 0 +for singleplex workers. -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). This document uses camel case in the JSON examples, but snake case when talking about the field regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), +the JSON protocol uses "camel case" (`requestId`). This document uses camel case +in the JSON examples, but snake case when talking about the field regardless of +protocol. ```json { @@ -38,13 +57,24 @@ NOTE: While the protocol buffer specification uses "snake case" (`request_id`), } ``` -The optional `verbosity` field can be used to request extra debugging output from the worker. It is entirely up to the worker what and how to output. Higher values indicate more verbose output. Passing the `--worker_verbose` flag to Bazel sets the `verbosity` field to 10, but smaller or larger values can be used manually for different amounts of output. +The optional `verbosity` field can be used to request extra debugging output +from the worker. It is entirely up to the worker what and how to output. Higher +values indicate more verbose output. Passing the `--worker_verbose` flag to +Bazel sets the `verbosity` field to 10, but smaller or larger values can be used +manually for different amounts of output. -The optional `sandbox_dir` field is used only by workers that support [multiplex sandboxing](/remote/multiplex). +The optional `sandbox_dir` field is used only by workers that support +[multiplex sandboxing](/remote/multiplex). ### Work responses -A `WorkResponse` contains a request id, a zero or nonzero exit code, and an output message describing any errors encountered in processing or executing the request. A worker should capture the `stdout` and `stderr` of any tool it calls and report them through the `WorkResponse`. Writing it to the `stdout` of the worker process is unsafe, as it will interfere with the worker protocol. Writing it to the `stderr` of the worker process is safe, but the result is collected in a per-worker log file instead of ascribed to individual actions. +A `WorkResponse` contains a request id, a zero or nonzero exit code, and an +output message describing any errors encountered in processing or executing +the request. A worker should capture the `stdout` and `stderr` of any tool it +calls and report them through the `WorkResponse`. Writing it to the `stdout` of +the worker process is unsafe, as it will interfere with the worker protocol. +Writing it to the `stderr` of the worker process is safe, but the result is +collected in a per-worker log file instead of ascribed to individual actions. ```json { @@ -55,7 +85,10 @@ A `WorkResponse` contains a request id, a zero or nonzero exit code, and an outp } ``` -As per the norm for protobufs, all fields are optional. However, Bazel requires the `WorkRequest` and the corresponding `WorkResponse`, to have the same request id, so the request id must be specified if it is nonzero. This is a valid `WorkResponse`. +As per the norm for protobufs, all fields are optional. However, Bazel requires +the `WorkRequest` and the corresponding `WorkResponse`, to have the same request +id, so the request id must be specified if it is nonzero. This is a valid +`WorkResponse`. ```json { @@ -63,35 +96,69 @@ As per the norm for protobufs, all fields are optional. However, Bazel requires } ``` -A `request_id` of 0 indicates a "singleplex" request, used when this request cannot be processed in parallel with other requests. The server guarantees that a given worker receives requests with either only `request_id` 0 or only `request_id` greater than zero. Singleplex requests are sent in serial, for example if the server doesn't send another request until it has received a response (except for cancel requests, see below). +A `request_id` of 0 indicates a "singleplex" request, used when this request +cannot be processed in parallel with other requests. The server guarantees that +a given worker receives requests with either only `request_id` 0 or only +`request_id` greater than zero. Singleplex requests are sent in serial, for +example if the server doesn't send another request until it has received a +response (except for cancel requests, see below). **Notes** -- Each protocol buffer is preceded by its length in `varint` format (see [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). -- JSON requests and responses are not preceded by a size indicator. -- JSON requests uphold the same structure as the protobuf, but use standard JSON and use camel case for all field names. -- In order to maintain the same backward and forward compatibility properties as protobuf, JSON workers must tolerate unknown fields in these messages, and use the protobuf defaults for missing values. -- Bazel stores requests as protobufs and converts them to JSON using [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) +* Each protocol buffer is preceded by its length in `varint` format (see + [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). +* JSON requests and responses are not preceded by a size indicator. +* JSON requests uphold the same structure as the protobuf, but use standard + JSON and use camel case for all field names. +* In order to maintain the same backward and forward compatibility properties + as protobuf, JSON workers must tolerate unknown fields in these messages, + and use the protobuf defaults for missing values. +* Bazel stores requests as protobufs and converts them to JSON using + [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) ### Cancellation -Workers can optionally allow work requests to be cancelled before they finish. This is particularly useful in connection with dynamic execution, where local execution can regularly be interrupted by a faster remote execution. To allow cancellation, add `supports-worker-cancellation: 1` to the `execution-requirements` field (see below) and set the `--experimental_worker_cancellation` flag. - -A **cancel request** is a `WorkRequest` with the `cancel` field set (and similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` field set). The only other field that must be in a cancel request or cancel response is `request_id`, indicating which request to cancel. The `request_id` field will be 0 for singleplex workers or the non-0 `request_id` of a previously sent `WorkRequest` for multiplex workers. The server may send cancel requests for requests that the worker has already responded to, in which case the cancel request must be ignored. - -Each non-cancel `WorkRequest` message must be answered exactly once, whether or not it was cancelled. Once the server has sent a cancel request, the worker may respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` field set to true. Sending a regular `WorkResponse` is also accepted, but the `output` and `exit_code` fields will be ignored. - -Once a response has been sent for a `WorkRequest`, the worker must not touch the files in its working directory. The server is free to clean up the files, including temporary files. +Workers can optionally allow work requests to be cancelled before they finish. +This is particularly useful in connection with dynamic execution, where local +execution can regularly be interrupted by a faster remote execution. To allow +cancellation, add `supports-worker-cancellation: 1` to the +`execution-requirements` field (see below) and set the +`--experimental_worker_cancellation` flag. + +A **cancel request** is a `WorkRequest` with the `cancel` field set (and +similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` +field set). The only other field that must be in a cancel request or cancel +response is `request_id`, indicating which request to cancel. The `request_id` +field will be 0 for singleplex workers or the non-0 `request_id` of a previously +sent `WorkRequest` for multiplex workers. The server may send cancel requests +for requests that the worker has already responded to, in which case the cancel +request must be ignored. + +Each non-cancel `WorkRequest` message must be answered exactly once, whether or +not it was cancelled. Once the server has sent a cancel request, the worker may +respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` +field set to true. Sending a regular `WorkResponse` is also accepted, but the +`output` and `exit_code` fields will be ignored. + +Once a response has been sent for a `WorkRequest`, the worker must not touch the +files in its working directory. The server is free to clean up the files, +including temporary files. ## Making the rule that uses the worker -You'll also need to create a rule that generates actions to be performed by the worker. Making a Starlark rule that uses a worker is just like [creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). +You'll also need to create a rule that generates actions to be performed by the +worker. Making a Starlark rule that uses a worker is just like +[creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). -In addition, the rule needs to contain a reference to the worker itself, and there are some requirements for the actions it produces. +In addition, the rule needs to contain a reference to the worker itself, and +there are some requirements for the actions it produces. ### Referring to the worker -The rule that uses the worker needs to contain a field that refers to the worker itself, so you'll need to create an instance of a `\*\_binary` rule to define your worker. If your worker is called `MyWorker.Java`, this might be the associated rule: +The rule that uses the worker needs to contain a field that refers to the worker +itself, so you'll need to create an instance of a `\*\_binary` rule to define +your worker. If your worker is called `MyWorker.Java`, this might be the +associated rule: ```python java_binary( @@ -100,9 +167,12 @@ java_binary( ) ``` -This creates the "worker" label, which refers to the worker binary. You'll then define a rule that *uses* the worker. This rule should define an attribute that refers to the worker binary. +This creates the "worker" label, which refers to the worker binary. You'll then +define a rule that *uses* the worker. This rule should define an attribute that +refers to the worker binary. -If the worker binary you built is in a package named "work", which is at the top level of the build, this might be the attribute definition: +If the worker binary you built is in a package named "work", which is at the top +level of the build, this might be the attribute definition: ```python "worker": attr.label( @@ -112,25 +182,46 @@ If the worker binary you built is in a package named "work", which is at the top ) ``` -`cfg = "exec"` indicates that the worker should be built to run on your execution platform rather than on the target platform (i.e., the worker is used as tool during the build). +`cfg = "exec"` indicates that the worker should be built to run on your +execution platform rather than on the target platform (i.e., the worker is used +as tool during the build). ### Work action requirements -The rule that uses the worker creates actions for the worker to perform. These actions have a couple of requirements. +The rule that uses the worker creates actions for the worker to perform. These +actions have a couple of requirements. -- The *"arguments"* field. This takes a list of strings, all but the last of which are arguments passed to the worker upon startup. The last element in the "arguments" list is a `flag-file` (@-preceded) argument. Workers read the arguments from the specified flagfile on a per-WorkRequest basis. Your rule can write non-startup arguments for the worker to this flagfile. +* The *"arguments"* field. This takes a list of strings, all but the last of + which are arguments passed to the worker upon startup. The last element in + the "arguments" list is a `flag-file` (@-preceded) argument. Workers read + the arguments from the specified flagfile on a per-WorkRequest basis. Your + rule can write non-startup arguments for the worker to this flagfile. -- The *"execution-requirements"* field, which takes a dictionary containing `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. +* The *"execution-requirements"* field, which takes a dictionary containing + `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. - The "arguments" and "execution-requirements" fields are required for all actions sent to workers. Additionally, actions that should be executed by JSON workers need to include `"requires-worker-protocol" : "json"` in the execution requirements field. `"requires-worker-protocol" : "proto"` is also a valid execution requirement, though it’s not required for proto workers, since they are the default. + The "arguments" and "execution-requirements" fields are required for all + actions sent to workers. Additionally, actions that should be executed by + JSON workers need to include `"requires-worker-protocol" : "json"` in the + execution requirements field. `"requires-worker-protocol" : "proto"` is also + a valid execution requirement, though it’s not required for proto workers, + since they are the default. - You can also set a `worker-key-mnemonic` in the execution requirements. This may be useful if you're reusing the executable for multiple action types and want to distinguish actions by this worker. + You can also set a `worker-key-mnemonic` in the execution requirements. This + may be useful if you're reusing the executable for multiple action types and + want to distinguish actions by this worker. -- Temporary files generated in the course of the action should be saved to the worker's directory. This enables sandboxing. +* Temporary files generated in the course of the action should be saved to the + worker's directory. This enables sandboxing. -Note: To pass an argument starting with a literal `@`, start the argument with `@@` instead. If an argument is also an external repository label, it will not be considered a flagfile argument. +Note: To pass an argument starting with a literal `@`, start the argument with +`@@` instead. If an argument is also an external repository label, it will not +be considered a flagfile argument. -Assuming a rule definition with "worker" attribute described above, in addition to a "srcs" attribute representing the inputs, an "output" attribute representing the outputs, and an "args" attribute representing the worker startup args, the call to `ctx.actions.run` might be: +Assuming a rule definition with "worker" attribute described above, in addition +to a "srcs" attribute representing the inputs, an "output" attribute +representing the outputs, and an "args" attribute representing the worker +startup args, the call to `ctx.actions.run` might be: ```python ctx.actions.run( @@ -145,14 +236,26 @@ ctx.actions.run( ) ``` -For another example, see [Implementing persistent workers](/remote/persistent#implementation). +For another example, see +[Implementing persistent workers](/remote/persistent#implementation). ## Examples -The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), in addition to an [example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) that is used in our integration tests. +The Bazel code base uses +[Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), +in addition to an +[example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) +that is used in our integration tests. -You can use their [scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) to make any Java-based tool into a worker by passing in the correct callback. +You can use their +[scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) +to make any Java-based tool into a worker by passing in the correct callback. -For an example of a rule that uses a worker, take a look at Bazel's [worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). +For an example of a rule that uses a worker, take a look at Bazel's +[worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). -External contributors have implemented workers in a variety of languages; take a look at [Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). You can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest\&type=Code)! +External contributors have implemented workers in a variety of languages; take a +look at +[Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). +You can +[find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)! diff --git a/remote/multiplex.mdx b/remote/multiplex.mdx index d9075a8a..b4b0a0d4 100644 --- a/remote/multiplex.mdx +++ b/remote/multiplex.mdx @@ -2,38 +2,112 @@ title: 'Multiplex Workers (Experimental Feature)' --- -This page describes multiplex workers, how to write multiplex-compatible rules, and workarounds for certain limitations. + + +This page describes multiplex workers, how to write multiplex-compatible +rules, and workarounds for certain limitations. Caution: Experimental features are subject to change at any time. -*Multiplex workers* allow Bazel to handle multiple requests with a single worker process. For multi-threaded workers, Bazel can use fewer resources to achieve the same, or better performance. For example, instead of having one worker process per worker, Bazel can have four multiplexed workers talking to the same worker process, which can then handle requests in parallel. For languages like Java and Scala, this saves JVM warm-up time and JIT compilation time, and in general it allows using one shared cache between all workers of the same type. +_Multiplex workers_ allow Bazel to handle multiple requests with a single worker +process. For multi-threaded workers, Bazel can use fewer resources to +achieve the same, or better performance. For example, instead of having one +worker process per worker, Bazel can have four multiplexed workers talking to +the same worker process, which can then handle requests in parallel. For +languages like Java and Scala, this saves JVM warm-up time and JIT compilation +time, and in general it allows using one shared cache between all workers of +the same type. ## Overview -There are two layers between the Bazel server and the worker process. For certain mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from the worker pool. The `WorkerProxy` forwards requests to the worker process sequentially along with a `request_id`, the worker process processes the request and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` receives a response, it parses the `request_id` and then forwards the responses back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all communication is done over standard in/out, but the tool cannot just use `stderr` for user-visible output ([see below](#output)). - -Each worker has a key. Bazel uses the key's hash code (composed of environment variables, the execution root, and the mnemonic) to determine which `WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same `WorkerMultiplexer` if they have the same hash code. Therefore, assuming environment variables and the execution root are the same in a single Bazel invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one worker process. The total number of workers, including regular workers and `WorkerProxy`s, is still limited by `--worker_max_instances`. +There are two layers between the Bazel server and the worker process. For certain +mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from +the worker pool. The `WorkerProxy` forwards requests to the worker process +sequentially along with a `request_id`, the worker process processes the request +and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` +receives a response, it parses the `request_id` and then forwards the responses +back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all +communication is done over standard in/out, but the tool cannot just use +`stderr` for user-visible output ([see below](#output)). + +Each worker has a key. Bazel uses the key's hash code (composed of environment +variables, the execution root, and the mnemonic) to determine which +`WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same +`WorkerMultiplexer` if they have the same hash code. Therefore, assuming +environment variables and the execution root are the same in a single Bazel +invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one +worker process. The total number of workers, including regular workers and +`WorkerProxy`s, is still limited by `--worker_max_instances`. ## Writing multiplex-compatible rules -The rule's worker process should be multi-threaded to take advantage of multiplex workers. Protobuf allows a ruleset to parse a single request even though there might be multiple requests piling up in the stream. Whenever the worker process parses a request from the stream, it should handle the request in a new thread. Because different thread could complete and write to the stream at the same time, the worker process needs to make sure the responses are written atomically (messages don't overlap). Responses must contain the `request_id` of the request they're handling. +The rule's worker process should be multi-threaded to take advantage of +multiplex workers. Protobuf allows a ruleset to parse a single request even +though there might be multiple requests piling up in the stream. Whenever the +worker process parses a request from the stream, it should handle the request in +a new thread. Because different thread could complete and write to the stream at +the same time, the worker process needs to make sure the responses are written +atomically (messages don't overlap). Responses must contain the +`request_id` of the request they're handling. ### Handling multiplex output -Multiplex workers need to be more careful about handling their output than singleplex workers. Anything sent to `stderr` will go into a single log file shared among all `WorkerProxy`s of the same type, randomly interleaved between concurrent requests. While redirecting `stdout` into `stderr` is a good idea, do not collect that output into the `output` field of `WorkResponse`, as that could show the user mangled pieces of output. If your tool only sends user-oriented output to `stdout` or `stderr`, you will need to change that behaviour before you can enable multiplex workers. +Multiplex workers need to be more careful about handling their output than +singleplex workers. Anything sent to `stderr` will go into a single log file +shared among all `WorkerProxy`s of the same type, +randomly interleaved between concurrent requests. While redirecting `stdout` +into `stderr` is a good idea, do not collect that output into the `output` +field of `WorkResponse`, as that could show the user mangled pieces of output. +If your tool only sends user-oriented output to `stdout` or `stderr`, you will +need to change that behaviour before you can enable multiplex workers. ## Enabling multiplex workers -Multiplex workers are not enabled by default. A ruleset can turn on multiplex workers by using the `supports-multiplex-workers` tag in the `execution_requirements` of an action (just like the `supports-workers` tag enables regular workers). As is the case when using regular workers, a worker strategy needs to be specified, either at the ruleset level (for example, `--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are necessary, and `supports-multiplex-workers` takes precedence over `supports-workers`, if both are set. You can turn off multiplex workers globally by passing `--noworker_multiplex`. - -A ruleset is encouraged to use multiplex workers if possible, to reduce memory pressure and improve performance. However, multiplex workers are not currently compatible with [dynamic execution](/remote/dynamic) unless they implement multiplex sandboxing. Attempting to run non-sandboxed multiplex workers with dynamic execution will silently use sandboxed singleplex workers instead. +Multiplex workers are not enabled by default. A ruleset can turn on multiplex +workers by using the `supports-multiplex-workers` tag in the +`execution_requirements` of an action (just like the `supports-workers` tag +enables regular workers). As is the case when using regular workers, a worker +strategy needs to be specified, either at the ruleset level (for example, +`--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for +example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are +necessary, and `supports-multiplex-workers` takes precedence over +`supports-workers`, if both are set. You can turn off multiplex workers +globally by passing `--noworker_multiplex`. + +A ruleset is encouraged to use multiplex workers if possible, to reduce memory +pressure and improve performance. However, multiplex workers are not currently +compatible with [dynamic execution](/remote/dynamic) unless they +implement multiplex sandboxing. Attempting to run non-sandboxed multiplex +workers with dynamic execution will silently use sandboxed +singleplex workers instead. ## Multiplex sandboxing -Multiplex workers can be sandboxed by adding explicit support for it in the worker implementations. While singleplex worker sandboxing can be done by running each worker process in its own sandbox, multiplex workers share the process working directory between multiple parallel requests. To allow sandboxing of multiplex workers, the worker must support reading from and writing to a subdirectory specified in each request, instead of directly in its working directory. - -To support multiplex sandboxing, the worker must use the `sandbox_dir` field from the `WorkRequest` and use that as a prefix for all file reads and writes. While the `arguments` and `inputs` fields remain unchanged from an unsandboxed request, the actual inputs are relative to the `sandbox_dir`. The worker must translate file paths found in `arguments` and `inputs` to read from this modified path, and must also write all outputs relative to the `sandbox_dir`. This includes paths such as '.', as well as paths found in files specified in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). - -Once a worker supports multiplex sandboxing, the ruleset can declare this support by adding `supports-multiplex-sandboxing` to the `execution_requirements` of an action. Bazel will then use multiplex sandboxing if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if the worker is used with dynamic execution. - -The worker files of a sandboxed multiplex worker are still relative to the working directory of the worker process. Thus, if a file is used both for running the worker and as an input, it must be specified both as an input in the flagfile argument as well as in `tools`, `executable`, or `runfiles`. +Multiplex workers can be sandboxed by adding explicit support for it in the +worker implementations. While singleplex worker sandboxing can be done by +running each worker process in its own sandbox, multiplex workers share the +process working directory between multiple parallel requests. To allow +sandboxing of multiplex workers, the worker must support reading from and +writing to a subdirectory specified in each request, instead of directly in +its working directory. + +To support multiplex sandboxing, the worker must use the `sandbox_dir` field +from the `WorkRequest` and use that as a prefix for all file reads and writes. +While the `arguments` and `inputs` fields remain unchanged from an unsandboxed +request, the actual inputs are relative to the `sandbox_dir`. The worker must +translate file paths found in `arguments` and `inputs` to read from this +modified path, and must also write all outputs relative to the `sandbox_dir`. +This includes paths such as '.', as well as paths found in files specified +in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). + +Once a worker supports multiplex sandboxing, the ruleset can declare this +support by adding `supports-multiplex-sandboxing` to the +`execution_requirements` of an action. Bazel will then use multiplex sandboxing +if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if +the worker is used with dynamic execution. + +The worker files of a sandboxed multiplex worker are still relative to the +working directory of the worker process. Thus, if a file is +used both for running the worker and as an input, it must be specified both as +an input in the flagfile argument as well as in `tools`, `executable`, or +`runfiles`. diff --git a/remote/output-directories.mdx b/remote/output-directories.mdx index 190892d8..9c1ba329 100644 --- a/remote/output-directories.mdx +++ b/remote/output-directories.mdx @@ -2,108 +2,141 @@ title: 'Output Directory Layout' --- + + This page covers requirements and layout for output directories. ## Requirements Requirements for an output directory layout: -- Doesn't collide if multiple users are building on the same box. -- Supports building in multiple workspaces at the same time. -- Supports building for multiple target configurations in the same workspace. -- Doesn't collide with any other tools. -- Is easy to access. -- Is easy to clean, even selectively. -- Is unambiguous, even if the user relies on symbolic links when changing into their client directory. -- All the build state per user should be underneath one directory ("I'd like to clean all the .o files from all my clients.") +* Doesn't collide if multiple users are building on the same box. +* Supports building in multiple workspaces at the same time. +* Supports building for multiple target configurations in the same workspace. +* Doesn't collide with any other tools. +* Is easy to access. +* Is easy to clean, even selectively. +* Is unambiguous, even if the user relies on symbolic links when changing into + their client directory. +* All the build state per user should be underneath one directory ("I'd like to + clean all the .o files from all my clients.") ## Current layout The solution that's currently implemented: -- Bazel must be invoked from a directory containing a repo boundary file, or a subdirectory thereof. In other words, Bazel must be invoked from inside a [repository](../external/overview#repository). Otherwise, an error is reported. -- The *outputRoot* directory defaults to `~/.cache/bazel` on Linux, `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if set, else `%USERPROFILE%` if set, else the result of calling `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the environment variable `$XDG_CACHE_HOME` is set on either Linux or macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel itself, then that value overrides any defaults. -- The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. This is called the *outputUserRoot* directory. -- Beneath the `outputUserRoot` directory there is an `install` directory, and in it is an `installBase` directory whose name is the MD5 hash of the Bazel installation manifest. -- Beneath the `outputUserRoot` directory, an `outputBase` directory is also created whose name is the MD5 hash of the path name of the workspace root. So, for example, if Bazel is running in the workspace root `/home/user/src/my-project` (or in a directory symlinked to that one), then an output base directory is created called: `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. -- You can use Bazel's `--output_base` startup option to override the default output base directory. For example, `bazel --output_base=/tmp/bazel/output build x/y:z`. -- You can also use Bazel's `--output_user_root` startup option to override the default install base and output base directories. For example: `bazel --output_user_root=/tmp/bazel build x/y:z`. - -The symlinks for "bazel-\<workspace-name\>", "bazel-out", "bazel-testlogs", and "bazel-bin" are put in the workspace directory; these symlinks point to some directories inside a target-specific directory inside the output directory. These symlinks are only for the user's convenience, as Bazel itself does not use them. Also, this is done only if the workspace root is writable. +* Bazel must be invoked from a directory containing a repo boundary file, or a + subdirectory thereof. In other words, Bazel must be invoked from inside a + [repository](../external/overview#repository). Otherwise, an error is + reported. +* The _outputRoot_ directory defaults to `~/.cache/bazel` on Linux, + `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if + set, else `%USERPROFILE%` if set, else the result of calling + `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the + environment variable `$XDG_CACHE_HOME` is set on either Linux or + macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. + If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel + itself, then that value overrides any defaults. +* The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. + This is called the _outputUserRoot_ directory. +* Beneath the `outputUserRoot` directory there is an `install` directory, and in + it is an `installBase` directory whose name is the MD5 hash of the Bazel + installation manifest. +* Beneath the `outputUserRoot` directory, an `outputBase` directory + is also created whose name is the MD5 hash of the path name of the workspace + root. So, for example, if Bazel is running in the workspace root + `/home/user/src/my-project` (or in a directory symlinked to that one), then + an output base directory is created called: + `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You + can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. +* You can use Bazel's `--output_base` startup option to override the default + output base directory. For example, + `bazel --output_base=/tmp/bazel/output build x/y:z`. +* You can also use Bazel's `--output_user_root` startup option to override the + default install base and output base directories. For example: + `bazel --output_user_root=/tmp/bazel build x/y:z`. + +The symlinks for "bazel-<workspace-name>", "bazel-out", "bazel-testlogs", +and "bazel-bin" are put in the workspace directory; these symlinks point to some +directories inside a target-specific directory inside the output directory. +These symlinks are only for the user's convenience, as Bazel itself does not +use them. Also, this is done only if the workspace root is writable. ## Layout diagram The directories are laid out as follows: ``` -<workspace-name>/ <== The workspace root - bazel-my-project => <..._main> <== Symlink to execRoot - bazel-out => <...bazel-out> <== Convenience symlink to outputPath - bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) - bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory - -/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot - _bazel_$USER/ <== Top level directory for a given user depends on the user name: +<workspace-name>/ <== The workspace root + bazel-my-project => <..._main> <== Symlink to execRoot + bazel-out => <...bazel-out> <== Convenience symlink to outputPath + bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) + bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory + +/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot + _bazel_$USER/ <== Top level directory for a given user depends on the user name: outputUserRoot install/ - fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase - _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of + fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase + _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of the bazel executable on first run (such as helper scripts and the main Java file BazelServer_deploy.jar) - 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as + 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as /home/user/src/my-project): outputBase - action_cache/ <== Action cache directory hierarchy + action_cache/ <== Action cache directory hierarchy This contains the persistent record of the file metadata (timestamps, and perhaps eventually also MD5 sums) used by the FilesystemValueChecker. - command.log <== A copy of the stdout/stderr output from the most + command.log <== A copy of the stdout/stderr output from the most recent bazel command. - external/ <== The directory that remote repositories are + external/ <== The directory that remote repositories are downloaded/symlinked into. - server/ <== The Bazel server puts all server-related files (such + server/ <== The Bazel server puts all server-related files (such as socket file, logs, etc) here. - jvm.out <== The debugging output for the server. - execroot/ <== The working directory for all actions. For special + jvm.out <== The debugging output for the server. + execroot/ <== The working directory for all actions. For special cases such as sandboxing and remote execution, the actions run in a directory that mimics execroot. Implementation details, such as where the directories are created, are intentionally hidden from the action. Every action can access its inputs and outputs relative to the execroot directory. - _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot - _bin/ <== Helper tools are linked from or copied to here. + _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot + _bin/ <== Helper tools are linked from or copied to here. - bazel-out/ <== All actual output of the build is under here: outputPath - _tmp/actions/ <== Action output directory. This contains a file with the + bazel-out/ <== All actual output of the build is under here: outputPath + _tmp/actions/ <== Action output directory. This contains a file with the stdout/stderr for every action from the most recent bazel run that produced output. - local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; + local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; this is currently encoded - bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) - foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz - foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc - other_package/other.o <== Object files from source //other_package:other.cc - foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named + bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) + foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz + foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc + other_package/other.o <== Object files from source //other_package:other.cc + foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named //foo/bar:baz - foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. + foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. MANIFEST _main/ ... - genfiles/ <== Bazel puts generated source for the target configuration here: + genfiles/ <== Bazel puts generated source for the target configuration here: $(GENDIR) foo/bar.h such as foo/bar.h might be a headerfile generated by //foo:bargen - testlogs/ <== Bazel internal test runner puts test log files here + testlogs/ <== Bazel internal test runner puts test log files here foo/bartest.log such as foo/bar.log might be an output of the //foo:bartest test with foo/bartest.status foo/bartest.status containing exit status of the test (such as PASSED or FAILED (Exit 1), etc) - host/ <== BuildConfiguration for build host (user's workstation), for + host/ <== BuildConfiguration for build host (user's workstation), for building prerequisite tools, that will be used in later stages of the build (ex: Protocol Compiler) - <packages>/ <== Packages referenced in the build appear as if under a regular workspace + <packages>/ <== Packages referenced in the build appear as if under a regular workspace ``` The layout of the \*.runfiles directories is documented in more detail in the places pointed to by RunfilesSupport. ## `bazel clean` -`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` directory. It also removes the workspace symlinks. The `--expunge` option will clean the entire outputBase. +`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` +directory. It also removes the workspace symlinks. The `--expunge` option +will clean the entire outputBase. diff --git a/remote/persistent.mdx b/remote/persistent.mdx index bd9029f5..1a56946a 100644 --- a/remote/persistent.mdx +++ b/remote/persistent.mdx @@ -2,65 +2,158 @@ title: 'Persistent Workers' --- -This page covers how to use persistent workers, the benefits, requirements, and how workers affect sandboxing. -A persistent worker is a long-running process started by the Bazel server, which functions as a *wrapper* around the actual *tool* (typically a compiler), or is the *tool* itself. In order to benefit from persistent workers, the tool must support doing a sequence of compilations, and the wrapper needs to translate between the tool's API and the request/response format described below. The same worker might be called with and without the `--persistent_worker` flag in the same build, and is responsible for appropriately starting and talking to the tool, as well as shutting down workers on exit. Each worker instance is assigned (but not chrooted to) a separate working directory under `<outputBase>/bazel-workers`. -Using persistent workers is an [execution strategy](/docs/user-manual#execution-strategy) that decreases start-up overhead, allows more JIT compilation, and enables caching of for example the abstract syntax trees in the action execution. This strategy achieves these improvements by sending multiple requests to a long-running process. +This page covers how to use persistent workers, the benefits, requirements, and +how workers affect sandboxing. -Persistent workers are implemented for multiple languages, including Java, [Scala](https://github.com/bazelbuild/rules_scala), [Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. +A persistent worker is a long-running process started by the Bazel server, which +functions as a *wrapper* around the actual *tool* (typically a compiler), or is +the *tool* itself. In order to benefit from persistent workers, the tool must +support doing a sequence of compilations, and the wrapper needs to translate +between the tool's API and the request/response format described below. The same +worker might be called with and without the `--persistent_worker` flag in the +same build, and is responsible for appropriately starting and talking to the +tool, as well as shutting down workers on exit. Each worker instance is assigned +(but not chrooted to) a separate working directory under +`/bazel-workers`. -Programs using a NodeJS runtime can use the [@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to implement the worker protocol. +Using persistent workers is an +[execution strategy](/docs/user-manual#execution-strategy) that decreases +start-up overhead, allows more JIT compilation, and enables caching of for +example the abstract syntax trees in the action execution. This strategy +achieves these improvements by sending multiple requests to a long-running +process. + +Persistent workers are implemented for multiple languages, including Java, +[Scala](https://github.com/bazelbuild/rules_scala), +[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. + +Programs using a NodeJS runtime can use the +[@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to +implement the worker protocol. ## Using persistent workers -[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) uses persistent workers by default when executing builds, though remote execution takes precedence. For actions that do not support persistent workers, Bazel falls back to starting a tool instance for each action. You can explicitly set your build to use persistent workers by setting the `worker` [strategy](/docs/user-manual#execution-strategy) for the applicable tool mnemonics. As a best practice, this example includes specifying `local` as a fallback to the `worker` strategy: +[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) +uses persistent workers by default when executing builds, though remote +execution takes precedence. For actions that do not support persistent workers, +Bazel falls back to starting a tool instance for each action. You can explicitly +set your build to use persistent workers by setting the `worker` +[strategy](/docs/user-manual#execution-strategy) for the applicable tool +mnemonics. As a best practice, this example includes specifying `local` as a +fallback to the `worker` strategy: ```posix-terminal -bazel build //<var>my:target</var> --strategy=Javac=worker,local +bazel build //{{ '' }}my:target{{ '' }} --strategy=Javac=worker,local ``` -Using the workers strategy instead of the local strategy can boost compilation speed significantly, depending on implementation. For Java, builds can be 2–4 times faster, sometimes more for incremental compilation. Compiling Bazel is about 2.5 times as fast with workers. For more details, see the "[Choosing number of workers](#number-of-workers)" section. - -If you also have a remote build environment that matches your local build environment, you can use the experimental [*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), which races a remote execution and a worker execution. To enable the dynamic strategy, pass the [--experimental\_spawn\_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) flag. This strategy automatically enables workers, so there is no need to specify the `worker` strategy, but you can still use `local` or `sandboxed` as fallbacks. +Using the workers strategy instead of the local strategy can boost compilation +speed significantly, depending on implementation. For Java, builds can be 2–4 +times faster, sometimes more for incremental compilation. Compiling Bazel is +about 2.5 times as fast with workers. For more details, see the +"[Choosing number of workers](#number-of-workers)" section. + +If you also have a remote build environment that matches your local build +environment, you can use the experimental +[*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), +which races a remote execution and a worker execution. To enable the dynamic +strategy, pass the +[--experimental_spawn_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) +flag. This strategy automatically enables workers, so there is no need to +specify the `worker` strategy, but you can still use `local` or `sandboxed` as +fallbacks. ## Choosing number of workers -The default number of worker instances per mnemonic is 4, but can be adjusted with the [`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) flag. There is a trade-off between making good use of the available CPUs and the amount of JIT compilation and cache hits you get. With more workers, more targets will pay start-up costs of running non-JITted code and hitting cold caches. If you have a small number of targets to build, a single worker may give the best trade-off between compilation speed and resource usage (for example, see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). The `worker_max_instances` flag sets the maximum number of worker instances per mnemonic and flag set (see below), so in a mixed system you could end up using quite a lot of memory if you keep the default value. For incremental builds the benefit of multiple worker instances is even smaller. - -This graph shows the from-scratch compilation times for Bazel (target `//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation with 64 GB of RAM. For each worker configuration, five clean builds are run and the average of the last four are taken. +The default number of worker instances per mnemonic is 4, but can be adjusted +with the +[`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) +flag. There is a trade-off between making good use of the available CPUs and the +amount of JIT compilation and cache hits you get. With more workers, more +targets will pay start-up costs of running non-JITted code and hitting cold +caches. If you have a small number of targets to build, a single worker may give +the best trade-off between compilation speed and resource usage (for example, +see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). +The `worker_max_instances` flag sets the maximum number of worker instances per +mnemonic and flag set (see below), so in a mixed system you could end up using +quite a lot of memory if you keep the default value. For incremental builds the +benefit of multiple worker instances is even smaller. + +This graph shows the from-scratch compilation times for Bazel (target +`//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation +with 64 GB of RAM. For each worker configuration, five clean builds are run and +the average of the last four are taken. ![Graph of performance improvements of clean builds](/docs/images/workers-clean-chart.png "Performance improvements of clean builds") **Figure 1.** Graph of performance improvements of clean builds. -For this configuration, two workers give the fastest compile, though at only 14% improvement compared to one worker. One worker is a good option if you want to use less memory. +For this configuration, two workers give the fastest compile, though at only 14% +improvement compared to one worker. One worker is a good option if you want to +use less memory. -Incremental compilation typically benefits even more. Clean builds are relatively rare, but changing a single file between compiles is common, in particular in test-driven development. The above example also has some non-Java packaging actions to it that can overshadow the incremental compile time. +Incremental compilation typically benefits even more. Clean builds are +relatively rare, but changing a single file between compiles is common, in +particular in test-driven development. The above example also has some non-Java +packaging actions to it that can overshadow the incremental compile time. -Recompiling the Java sources only (`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) after changing an internal string constant in [AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) gives a 3x speed-up (average of 20 incremental builds with one warmup build discarded): +Recompiling the Java sources only +(`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) +after changing an internal string constant in +[AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) +gives a 3x speed-up (average of 20 incremental builds with one warmup build +discarded): ![Graph of performance improvements of incremental builds](/docs/images/workers-incremental-chart.png "Performance improvements of incremental builds") **Figure 2.** Graph of performance improvements of incremental builds. -The speed-up depends on the change being made. A speed-up of a factor 6 is measured in the above situation when a commonly used constant is changed. +The speed-up depends on the change being made. A speed-up of a factor 6 is +measured in the above situation when a commonly used constant is changed. ## Modifying persistent workers -You can pass the [`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) flag to specify start-up flags to workers, keyed by mnemonic. For instance, passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. Only one worker flag can be set per use of this flag, and only for one mnemonic. Workers are not just created separately for each mnemonic, but also for variations in their start-up flags. Each combination of mnemonic and start-up flags is combined into a `WorkerKey`, and for each `WorkerKey` up to `worker_max_instances` workers may be created. See the next section for how the action configuration can also specify set-up flags. - -Passing the [`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) flag makes each worker request use a separate sandbox directory for all its inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, especially on macOS, but gives a better correctness guarantee. - -The [`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) flag is mainly useful for debugging and profiling. This flag forces all workers to quit once a build is done. You can also pass [`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to get more output about what the workers are doing. This flag is reflected in the `verbosity` field in `WorkRequest`, allowing worker implementations to also be more verbose. - -Workers store their logs in the `<outputBase>/bazel-workers` directory, for example `/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. The file name includes the worker id and the mnemonic. Since there can be more than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` log files for a given mnemonic. - -For Android builds, see details at the [Android Build Performance page](/docs/android-build-performance). +You can pass the +[`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) +flag to specify start-up flags to workers, keyed by mnemonic. For instance, +passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. +Only one worker flag can be set per use of this flag, and only for one mnemonic. +Workers are not just created separately for each mnemonic, but also for +variations in their start-up flags. Each combination of mnemonic and start-up +flags is combined into a `WorkerKey`, and for each `WorkerKey` up to +`worker_max_instances` workers may be created. See the next section for how the +action configuration can also specify set-up flags. + +Passing the +[`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) +flag makes each worker request use a separate sandbox directory for all its +inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, +especially on macOS, but gives a better correctness guarantee. + +The +[`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) +flag is mainly useful for debugging and profiling. This flag forces all workers +to quit once a build is done. You can also pass +[`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to +get more output about what the workers are doing. This flag is reflected in the +`verbosity` field in `WorkRequest`, allowing worker implementations to also be +more verbose. + +Workers store their logs in the `/bazel-workers` directory, for +example +`/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. +The file name includes the worker id and the mnemonic. Since there can be more +than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` +log files for a given mnemonic. + +For Android builds, see details at the +[Android Build Performance page](/docs/android-build-performance). ## Implementing persistent workers -See the [creating persistent workers](/remote/creating) page for more information on how to make a worker. +See the [creating persistent workers](/remote/creating) page for more +information on how to make a worker. This example shows a Starlark configuration for a worker that uses JSON: @@ -81,9 +174,14 @@ ctx.actions.run( ) ``` -With this definition, the first use of this action would start with executing the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request to compile `Foo.java` would then look like: +With this definition, the first use of this action would start with executing +the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request +to compile `Foo.java` would then look like: -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). In this document, we will use camel case in the JSON examples, but snake case when talking about the field regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), +the JSON protocol uses "camel case" (`requestId`). In this document, we will use +camel case in the JSON examples, but snake case when talking about the field +regardless of protocol. ```json { @@ -95,7 +193,12 @@ NOTE: While the protocol buffer specification uses "snake case" (`request_id`), } ``` -The worker receives this on `stdin` in newline-delimited JSON format (because `requires-worker-protocol` is set to JSON). The worker then performs the action, and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then parses this response and manually converts it to a `WorkResponse` proto. To communicate with the associated worker using binary-encoded protobuf instead of JSON, `requires-worker-protocol` would be set to `proto`, like this: +The worker receives this on `stdin` in newline-delimited JSON format (because +`requires-worker-protocol` is set to JSON). The worker then performs the action, +and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then +parses this response and manually converts it to a `WorkResponse` proto. To +communicate with the associated worker using binary-encoded protobuf instead of +JSON, `requires-worker-protocol` would be set to `proto`, like this: ``` execution_requirements = { @@ -104,31 +207,59 @@ The worker receives this on `stdin` in newline-delimited JSON format (because `r } ``` -If you do not include `requires-worker-protocol` in the execution requirements, Bazel will default the worker communication to use protobuf. +If you do not include `requires-worker-protocol` in the execution requirements, +Bazel will default the worker communication to use protobuf. -Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this configuration allowed changing the `max_mem` parameter, a separate worker would be spawned for each value used. This can lead to excessive memory consumption if too many variations are used. +Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this +configuration allowed changing the `max_mem` parameter, a separate worker would +be spawned for each value used. This can lead to excessive memory consumption if +too many variations are used. -Each worker can currently only process one request at a time. The experimental [multiplex workers](/remote/multiplex) feature allows using multiple threads, if the underlying tool is multithreaded and the wrapper is set up to understand this. +Each worker can currently only process one request at a time. The experimental +[multiplex workers](/remote/multiplex) feature allows using multiple +threads, if the underlying tool is multithreaded and the wrapper is set up to +understand this. -In [this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), you can see example worker wrappers written in Java as well as in Python. If you are working in JavaScript or TypeScript, the [@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) and [nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) might be helpful. +In +[this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), +you can see example worker wrappers written in Java as well as in Python. If you +are working in JavaScript or TypeScript, the +[@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) +and +[nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) +might be helpful. ## How do workers affect sandboxing? -Using the `worker` strategy by default does not run the action in a [sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the `--worker_sandboxing` flag to run all workers inside sandboxes, making sure each execution of the tool only sees the input files it's supposed to have. The tool may still leak information between requests internally, for instance through a cache. Using `dynamic` strategy [requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). +Using the `worker` strategy by default does not run the action in a +[sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the +`--worker_sandboxing` flag to run all workers inside sandboxes, making sure each +execution of the tool only sees the input files it's supposed to have. The tool +may still leak information between requests internally, for instance through a +cache. Using `dynamic` strategy +[requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). -To allow correct use of compiler caches with workers, a digest is passed along with each input file. Thus the compiler or the wrapper can check if the input is still valid without having to read the file. +To allow correct use of compiler caches with workers, a digest is passed along +with each input file. Thus the compiler or the wrapper can check if the input is +still valid without having to read the file. -Even when using the input digests to guard against unwanted caching, sandboxed workers offer less strict sandboxing than a pure sandbox, because the tool may keep other internal state that has been affected by previous requests. +Even when using the input digests to guard against unwanted caching, sandboxed +workers offer less strict sandboxing than a pure sandbox, because the tool may +keep other internal state that has been affected by previous requests. -Multiplex workers can only be sandboxed if the worker implementation support it, and this sandboxing must be separately enabled with the `--experimental_worker_multiplex_sandboxing` flag. See more details in [the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). +Multiplex workers can only be sandboxed if the worker implementation support it, +and this sandboxing must be separately enabled with the +`--experimental_worker_multiplex_sandboxing` flag. See more details in +[the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). ## Further reading For more information on persistent workers, see: -- [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) -- [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) -- [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) -- [Front End Development with Bazel: Angular/TypeScript and Persistent Workers w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) -- [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) -- [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) +* [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) +* [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) +* [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) +* [Front End Development with Bazel: Angular/TypeScript and Persistent Workers + w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) +* [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) +* [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) diff --git a/remote/rbe.mdx b/remote/rbe.mdx index 0d8bf255..75d4a155 100644 --- a/remote/rbe.mdx +++ b/remote/rbe.mdx @@ -2,20 +2,32 @@ title: 'Remote Execution Overview' --- -This page covers the benefits, requirements, and options for running Bazel with remote execution. -By default, Bazel executes builds and tests on your local machine. Remote execution of a Bazel build allows you to distribute build and test actions across multiple machines, such as a datacenter. + +This page covers the benefits, requirements, and options for running Bazel +with remote execution. + +By default, Bazel executes builds and tests on your local machine. Remote +execution of a Bazel build allows you to distribute build and test actions +across multiple machines, such as a datacenter. Remote execution provides the following benefits: -- Faster build and test execution through scaling of nodes available for parallel actions -- A consistent execution environment for a development team -- Reuse of build outputs across a development team +* Faster build and test execution through scaling of nodes available + for parallel actions +* A consistent execution environment for a development team +* Reuse of build outputs across a development team -Bazel uses an open-source [gRPC protocol](https://github.com/bazelbuild/remote-apis) to allow for remote execution and remote caching. +Bazel uses an open-source +[gRPC protocol](https://github.com/bazelbuild/remote-apis) +to allow for remote execution and remote caching. -For a list of commercially supported remote execution services as well as self-service tools, see [Remote Execution Services](https://www.bazel.build/remote-execution-services.html) +For a list of commercially supported remote execution services as well as +self-service tools, see +[Remote Execution Services](https://www.bazel.build/remote-execution-services.html) ## Requirements -Remote execution of Bazel builds imposes a set of mandatory configuration constraints on the build. For more information, see [Adapting Bazel Rules for Remote Execution](/remote/rules). +Remote execution of Bazel builds imposes a set of mandatory configuration +constraints on the build. For more information, see +[Adapting Bazel Rules for Remote Execution](/remote/rules). diff --git a/remote/rules.mdx b/remote/rules.mdx index 4ea4c9c3..340ab02c 100644 --- a/remote/rules.mdx +++ b/remote/rules.mdx @@ -2,79 +2,179 @@ title: 'Adapting Bazel Rules for Remote Execution' --- -This page is intended for Bazel users writing custom build and test rules who want to understand the requirements for Bazel rules in the context of remote execution. -Remote execution allows Bazel to execute actions on a separate platform, such as a datacenter. Bazel uses a [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) for its remote execution. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source project that aims to provide a distributed remote execution platform. -This page uses the following terminology when referring to different environment types or *platforms*: +This page is intended for Bazel users writing custom build and test rules +who want to understand the requirements for Bazel rules in the context of +remote execution. -- **Host platform** - where Bazel runs. -- **Execution platform** - where Bazel actions run. -- **Target platform** - where the build outputs (and some actions) run. +Remote execution allows Bazel to execute actions on a separate platform, such as +a datacenter. Bazel uses a +[gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) +for its remote execution. You can try remote execution with +[bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), +an open-source project that aims to provide a distributed remote execution +platform. + +This page uses the following terminology when referring to different +environment types or *platforms*: + +* **Host platform** - where Bazel runs. +* **Execution platform** - where Bazel actions run. +* **Target platform** - where the build outputs (and some actions) run. ## Overview -When configuring a Bazel build for remote execution, you must follow the guidelines described in this page to ensure the build executes remotely error-free. This is due to the nature of remote execution, namely: +When configuring a Bazel build for remote execution, you must follow the +guidelines described in this page to ensure the build executes remotely +error-free. This is due to the nature of remote execution, namely: -- **Isolated build actions.** Build tools do not retain state and dependencies cannot leak between them. +* **Isolated build actions.** Build tools do not retain state and dependencies + cannot leak between them. -- **Diverse execution environments.** Local build configuration is not always suitable for remote execution environments. +* **Diverse execution environments.** Local build configuration is not always + suitable for remote execution environments. -This page describes the issues that can arise when implementing custom Bazel build and test rules for remote execution and how to avoid them. It covers the following topics: +This page describes the issues that can arise when implementing custom Bazel +build and test rules for remote execution and how to avoid them. It covers the +following topics: -- [Invoking build tools through toolchain rules](#toolchain-rules) -- [Managing implicit dependencies](#manage-dependencies) -- [Managing platform-dependent binaries](#manage-binaries) -- [Managing configure-style WORKSPACE rules](#manage-workspace-rules) +* [Invoking build tools through toolchain rules](#toolchain-rules) +* [Managing implicit dependencies](#manage-dependencies) +* [Managing platform-dependent binaries](#manage-binaries) +* [Managing configure-style WORKSPACE rules](#manage-workspace-rules) ## Invoking build tools through toolchain rules -A Bazel toolchain rule is a configuration provider that tells a build rule what build tools, such as compilers and linkers, to use and how to configure them using parameters defined by the rule's creator. A toolchain rule allows build and test rules to invoke build tools in a predictable, preconfigured manner that's compatible with remote execution. For example, use a toolchain rule instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local variables that may not be set to equivalent values (or at all) in the remote execution environment. - -Toolchain rules currently exist for Bazel build and test rules for \[Scala]\([https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch) ain.bzl), [Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), and new toolchain rules are under way for other languages and tools such as [bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). If a toolchain rule does not exist for the tool your rule uses, consider [creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). +A Bazel toolchain rule is a configuration provider that tells a build rule what +build tools, such as compilers and linkers, to use and how to configure them +using parameters defined by the rule's creator. A toolchain rule allows build +and test rules to invoke build tools in a predictable, preconfigured manner +that's compatible with remote execution. For example, use a toolchain rule +instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local +variables that may not be set to equivalent values (or at all) in the remote +execution environment. + +Toolchain rules currently exist for Bazel build and test rules for +[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch +ain.bzl), +[Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), +and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), +and new toolchain rules are under way for other languages and tools such as +[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). +If a toolchain rule does not exist for the tool your rule uses, consider +[creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). ## Managing implicit dependencies -If a build tool can access dependencies across build actions, those actions will fail when remotely executed because each remote build action is executed separately from others. Some build tools retain state across build actions and access dependencies that have not been explicitly included in the tool invocation, which will cause remotely executed build actions to fail. - -For example, when Bazel instructs a stateful compiler to locally build *foo*, the compiler retains references to foo's build outputs. When Bazel then instructs the compiler to build *bar*, which depends on *foo*, without explicitly stating that dependency in the BUILD file for inclusion in the compiler invocation, the action executes successfully as long as the same compiler instance executes for both actions (as is typical for local execution). However, since in a remote execution scenario each build action executes a separate compiler instance, compiler state and *bar*'s implicit dependency on *foo* will be lost and the build will fail. - -To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the local Docker sandbox, which has the same restrictions for dependencies as remote execution. Use the sandbox to prepare your build for remote execution by identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) for more information. +If a build tool can access dependencies across build actions, those actions will +fail when remotely executed because each remote build action is executed +separately from others. Some build tools retain state across build actions and +access dependencies that have not been explicitly included in the tool +invocation, which will cause remotely executed build actions to fail. + +For example, when Bazel instructs a stateful compiler to locally build _foo_, +the compiler retains references to foo's build outputs. When Bazel then +instructs the compiler to build _bar_, which depends on _foo_, without +explicitly stating that dependency in the BUILD file for inclusion in the +compiler invocation, the action executes successfully as long as the same +compiler instance executes for both actions (as is typical for local execution). +However, since in a remote execution scenario each build action executes a +separate compiler instance, compiler state and _bar_'s implicit dependency on +_foo_ will be lost and the build will fail. + +To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the +local Docker sandbox, which has the same restrictions for dependencies as remote +execution. Use the sandbox to prepare your build for remote execution by +identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) +for more information. ## Managing platform-dependent binaries -Typically, a binary built on the host platform cannot safely execute on an arbitrary remote execution platform due to potentially mismatched dependencies. For example, the SingleJar binary supplied with Bazel targets the host platform. However, for remote execution, SingleJar must be compiled as part of the process of building your code so that it targets the remote execution platform. (See the [target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) +Typically, a binary built on the host platform cannot safely execute on an +arbitrary remote execution platform due to potentially mismatched dependencies. +For example, the SingleJar binary supplied with Bazel targets the host platform. +However, for remote execution, SingleJar must be compiled as part of the process +of building your code so that it targets the remote execution platform. (See the +[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) -Do not ship binaries of build tools required by your build with your source code unless you are sure they will safely run in your execution platform. Instead, do one of the following: +Do not ship binaries of build tools required by your build with your source code +unless you are sure they will safely run in your execution platform. Instead, do +one of the following: -- Ship or externally reference the source code for the tool so that it can be built for the remote execution platform. +* Ship or externally reference the source code for the tool so that it can be + built for the remote execution platform. -- Pre-install the tool into the remote execution environment (for example, a toolchain container) if it's stable enough and use toolchain rules to run it in your build. +* Pre-install the tool into the remote execution environment (for example, a + toolchain container) if it's stable enough and use toolchain rules to run it + in your build. ## Managing configure-style WORKSPACE rules -Bazel's `WORKSPACE` rules can be used for probing the host platform for tools and libraries required by the build, which, for local builds, is also Bazel's execution platform. If the build explicitly depends on local build tools and artifacts, it will fail during remote execution if the remote execution platform is not identical to the host platform. - -The following actions performed by `WORKSPACE` rules are not compatible with remote execution: - -- **Building binaries.** Executing compilation actions in `WORKSPACE` rules results in binaries that are incompatible with the remote execution platform if different from the host platform. - -- **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` rules require that their dependencies be pre-installed on the host platform. Such packages, built specifically for the host platform, will be incompatible with the remote execution platform if different from the host platform. - -- **Symlinking to local tools or artifacts.** Symlinks to tools or libraries installed on the host platform created via `WORKSPACE` rules will cause the build to fail on the remote execution platform as Bazel will not be able to locate them. Instead, create symlinks using standard build actions so that the symlinked tools and libraries are accessible from Bazel's `runfiles` tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) to symlink target files outside of the external repo directory. - -- **Mutating the host platform.** Avoid creating files outside of the Bazel `runfiles` tree, creating environment variables, and similar actions, as they may behave unexpectedly on the remote execution platform. +Bazel's `WORKSPACE` rules can be used for probing the host platform for tools +and libraries required by the build, which, for local builds, is also Bazel's +execution platform. If the build explicitly depends on local build tools and +artifacts, it will fail during remote execution if the remote execution platform +is not identical to the host platform. + +The following actions performed by `WORKSPACE` rules are not compatible with +remote execution: + +* **Building binaries.** Executing compilation actions in `WORKSPACE` rules + results in binaries that are incompatible with the remote execution platform + if different from the host platform. + +* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` + rules require that their dependencies be pre-installed on the host platform. + Such packages, built specifically for the host platform, will be + incompatible with the remote execution platform if different from the host + platform. + +* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries + installed on the host platform created via `WORKSPACE` rules will cause the + build to fail on the remote execution platform as Bazel will not be able to + locate them. Instead, create symlinks using standard build actions so that + the symlinked tools and libraries are accessible from Bazel's `runfiles` + tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) + to symlink target files outside of the external repo directory. + +* **Mutating the host platform.** Avoid creating files outside of the Bazel + `runfiles` tree, creating environment variables, and similar actions, as + they may behave unexpectedly on the remote execution platform. To help find potential non-hermetic behavior you can use [Workspace rules log](/remote/workspace). -If an external dependency executes specific operations dependent on the host platform, you should split those operations between `WORKSPACE` and build rules as follows: - -- **Platform inspection and dependency enumeration.** These operations are safe to execute locally via `WORKSPACE` rules, which can check which libraries are installed, download packages that must be built, and prepare required artifacts for compilation. For remote execution, these rules must also support using pre-checked artifacts to provide the information that would normally be obtained during host platform inspection. Pre-checked artifacts allow Bazel to describe dependencies as if they were local. Use conditional statements or the `--override_repository` flag for this. - -- **Generating or compiling target-specific artifacts and platform mutation**. These operations must be executed via regular build rules. Actions that produce target-specific artifacts for external dependencies must execute during the build. - -To more easily generate pre-checked artifacts for remote execution, you can use `WORKSPACE` rules to emit generated files. You can run those rules on each new execution environment, such as inside each toolchain container, and check the outputs of your remote execution build in to your source repo to reference. - -For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). For local execution, files produced by checking the host environment are used. For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) on an environment variable allows the rule to use files that are checked into the repo. - -The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) that can run both locally and remotely, and perform the necessary processing that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). +If an external dependency executes specific operations dependent on the host +platform, you should split those operations between `WORKSPACE` and build +rules as follows: + +* **Platform inspection and dependency enumeration.** These operations are + safe to execute locally via `WORKSPACE` rules, which can check which + libraries are installed, download packages that must be built, and prepare + required artifacts for compilation. For remote execution, these rules must + also support using pre-checked artifacts to provide the information that + would normally be obtained during host platform inspection. Pre-checked + artifacts allow Bazel to describe dependencies as if they were local. Use + conditional statements or the `--override_repository` flag for this. + +* **Generating or compiling target-specific artifacts and platform mutation**. + These operations must be executed via regular build rules. Actions that + produce target-specific artifacts for external dependencies must execute + during the build. + +To more easily generate pre-checked artifacts for remote execution, you can use +`WORKSPACE` rules to emit generated files. You can run those rules on each new +execution environment, such as inside each toolchain container, and check the +outputs of your remote execution build in to your source repo to reference. + +For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) +and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), +the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). +For local execution, files produced by checking the host environment are used. +For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) +on an environment variable allows the rule to use files that are checked into +the repo. + +The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) +that can run both locally and remotely, and perform the necessary processing +that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/remote/sandbox.mdx b/remote/sandbox.mdx index 98485efc..8c3f9565 100644 --- a/remote/sandbox.mdx +++ b/remote/sandbox.mdx @@ -2,143 +2,223 @@ title: 'Troubleshooting Bazel Remote Execution with Docker Sandbox' --- -Bazel builds that succeed locally may fail when executed remotely due to restrictions and requirements that do not affect local builds. The most common causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). -This page describes how to identify and resolve the most common issues that arise with remote execution using the Docker sandbox feature, which imposes restrictions upon the build equal to those of remote execution. This allows you to troubleshoot your build without the need for a remote execution service. -The Docker sandbox feature mimics the restrictions of remote execution as follows: +Bazel builds that succeed locally may fail when executed remotely due to +restrictions and requirements that do not affect local builds. The most common +causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). -- **Build actions execute in toolchain containers.** You can use the same toolchain containers to run your build locally and remotely via a service supporting containerized remote execution. +This page describes how to identify and resolve the most common issues that +arise with remote execution using the Docker sandbox feature, which imposes +restrictions upon the build equal to those of remote execution. This allows you +to troubleshoot your build without the need for a remote execution service. -- **No extraneous data crosses the container boundary.** Only explicitly declared inputs and outputs enter and leave the container, and only after the associated build action successfully completes. +The Docker sandbox feature mimics the restrictions of remote execution as +follows: -- **Each action executes in a fresh container.** A new, unique container is created for each spawned build action. +* **Build actions execute in toolchain containers.** You can use the same + toolchain containers to run your build locally and remotely via a service + supporting containerized remote execution. -Note: Builds take noticeably more time to complete when the Docker sandbox feature is enabled. This is normal. +* **No extraneous data crosses the container boundary.** Only explicitly + declared inputs and outputs enter and leave the container, and only after + the associated build action successfully completes. + +* **Each action executes in a fresh container.** A new, unique container is + created for each spawned build action. + +Note: Builds take noticeably more time to complete when the Docker sandbox +feature is enabled. This is normal. You can troubleshoot these issues using one of the following methods: -- **[Troubleshooting natively.](#troubleshooting-natively)** With this method, Bazel and its build actions run natively on your local machine. The Docker sandbox feature imposes restrictions upon the build equal to those of remote execution. However, this method will not detect local tools, states, and data leaking into your build, which will cause problems with remote execution. +* **[Troubleshooting natively.](#troubleshooting-natively)** With this method, + Bazel and its build actions run natively on your local machine. The Docker + sandbox feature imposes restrictions upon the build equal to those of remote + execution. However, this method will not detect local tools, states, and + data leaking into your build, which will cause problems with remote execution. -- **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** With this method, Bazel and its build actions run inside a Docker container, which allows you to detect tools, states, and data leaking from the local machine into the build in addition to imposing restrictions equal to those of remote execution. This method provides insight into your build even if portions of the build are failing. This method is experimental and not officially supported. +* **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** + With this method, Bazel and its build actions run inside a Docker container, + which allows you to detect tools, states, and data leaking from the local + machine into the build in addition to imposing restrictions + equal to those of remote execution. This method provides insight into your + build even if portions of the build are failing. This method is experimental + and not officially supported. ## Prerequisites Before you begin troubleshooting, do the following if you have not already done so: -- Install Docker and configure the permissions required to run it. -- Install Bazel 0.14.1 or later. Earlier versions do not support the Docker sandbox feature. -- Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) repo, pinned to the latest release version, to your build's `WORKSPACE` file as described [here](https://releases.bazel.build/bazel-toolchains.html). -- Add flags to your `.bazelrc` file to enable the feature. Create the file in the root directory of your Bazel project if it does not exist. Flags below are a reference sample. Please see the latest [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) file in the bazel-toolchains repo and copy the values of the flags defined there for config `docker-sandbox`. +* Install Docker and configure the permissions required to run it. +* Install Bazel 0.14.1 or later. Earlier versions do not support the Docker + sandbox feature. +* Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) + repo, pinned to the latest release version, to your build's `WORKSPACE` file + as described [here](https://releases.bazel.build/bazel-toolchains.html). +* Add flags to your `.bazelrc` file to enable the feature. Create the file in + the root directory of your Bazel project if it does not exist. Flags below + are a reference sample. Please see the latest + [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) + file in the bazel-toolchains repo and copy the values of the flags defined + there for config `docker-sandbox`. ``` # Docker Sandbox Mode -build:docker-sandbox --host_javabase=<...> -build:docker-sandbox --javabase=<...> -build:docker-sandbox --crosstool_top=<...> -build:docker-sandbox --experimental_docker_image=<...> +build:docker-sandbox --host_javabase=<...> +build:docker-sandbox --javabase=<...> +build:docker-sandbox --crosstool_top=<...> +build:docker-sandbox --experimental_docker_image=<...> build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker build:docker-sandbox --experimental_docker_verbose build:docker-sandbox --experimental_enable_docker_sandbox ``` -Note: The flags referenced in the `.bazelrc` file shown above are configured to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) container. +Note: The flags referenced in the `.bazelrc` file shown above are configured +to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) +container. If your rules require additional tools, do the following: -1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) and [building](https://docs.docker.com/engine/reference/commandline/build/) the image locally. +1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) + and [building](https://docs.docker.com/engine/reference/commandline/build/) + the image locally. + +2. Replace the value of the `--experimental_docker_image` flag above with the + name of your custom container image. -2. Replace the value of the `--experimental_docker_image` flag above with the name of your custom container image. ## Troubleshooting natively -This method executes Bazel and all of its build actions directly on the local machine and is a reliable way to confirm whether your build will succeed when executed remotely. +This method executes Bazel and all of its build actions directly on the local +machine and is a reliable way to confirm whether your build will succeed when +executed remotely. -However, with this method, locally installed tools, binaries, and data may leak into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) in addition to troubleshooting natively. +However, with this method, locally installed tools, binaries, and data may leak +into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). +Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) +in addition to troubleshooting natively. ### Step 1: Run the build -1. Add the `--config=docker-sandbox` flag to the Bazel command that executes your build. For example: +1. Add the `--config=docker-sandbox` flag to the Bazel command that executes + your build. For example: - ```posix-terminal - bazel --bazelrc=.bazelrc build --config=docker-sandbox <var>target</var> - ``` + ```posix-terminal + bazel --bazelrc=.bazelrc build --config=docker-sandbox target + ``` -2. Run the build and wait for it to complete. The build will run up to four times slower than normal due to the Docker sandbox feature. +2. Run the build and wait for it to complete. The build will run up to four + times slower than normal due to the Docker sandbox feature. You may encounter the following error: -```none +```none {:.devsite-disable-click-to-copy} ERROR: 'docker' is an invalid value for docker spawn strategy. ``` -If you do, run the build again with the `--experimental_docker_verbose` flag. This flag enables verbose error messages. This error is typically caused by a faulty Docker installation or lack of permissions to execute it under the current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). +If you do, run the build again with the `--experimental_docker_verbose` flag. +This flag enables verbose error messages. This error is typically caused by a +faulty Docker installation or lack of permissions to execute it under the +current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) +for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). ### Step 2: Resolve detected issues The following are the most commonly encountered issues and their workarounds. -- **A file, tool, binary, or resource referenced by the Bazel runfiles tree is missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/concepts/dependencies). See [Managing implicit dependencies](/remote/rules#manage-dependencies) for more information. +* **A file, tool, binary, or resource referenced by the Bazel runfiles tree is + missing.**. Confirm that all dependencies of the affected targets have been + [explicitly declared](/concepts/dependencies). See + [Managing implicit dependencies](/remote/rules#manage-dependencies) + for more information. -- **A file, tool, binary, or resource referenced by an absolute path or the `PATH` variable is missing.** Confirm that all required tools are installed within the toolchain container and use [toolchain rules](/extending/toolchains) to properly declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) for more information. +* **A file, tool, binary, or resource referenced by an absolute path or the `PATH` + variable is missing.** Confirm that all required tools are installed within + the toolchain container and use [toolchain rules](/extending/toolchains) to properly + declare dependencies pointing to the missing resource. See + [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) + for more information. -- **A binary execution fails.** One of the build rules is referencing a binary incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote/rules#manage-binaries) for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +* **A binary execution fails.** One of the build rules is referencing a binary + incompatible with the execution environment (the Docker container). See + [Managing platform-dependent binaries](/remote/rules#manage-binaries) + for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) + for help. -- **A file from `@local-jdk` is missing or causing errors.** The Java binaries on your local machine are leaking into the build while being incompatible with it. Use [`java_toolchain`](/reference/be/java#java_toolchain) in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. +* **A file from `@local-jdk` is missing or causing errors.** The Java binaries + on your local machine are leaking into the build while being incompatible with + it. Use [`java_toolchain`](/reference/be/java#java_toolchain) + in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. -- **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +* **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. ## Troubleshooting in a Docker container -With this method, Bazel runs inside a host Docker container, and Bazel's build actions execute inside individual toolchain containers spawned by the Docker sandbox feature. The sandbox spawns a brand new toolchain container for each build action and only one action executes in each toolchain container. +With this method, Bazel runs inside a host Docker container, and Bazel's build +actions execute inside individual toolchain containers spawned by the Docker +sandbox feature. The sandbox spawns a brand new toolchain container for each +build action and only one action executes in each toolchain container. -This method provides more granular control of tools installed in the host environment. By separating the execution of the build from the execution of its build actions and keeping the installed tooling to a minimum, you can verify whether your build has any dependencies on the local execution environment. +This method provides more granular control of tools installed in the host +environment. By separating the execution of the build from the execution of its +build actions and keeping the installed tooling to a minimum, you can verify +whether your build has any dependencies on the local execution environment. ### Step 1: Build the container -Note: The commands below are tailored specifically for a `debian:stretch` base. For other bases, modify them as necessary. +Note: The commands below are tailored specifically for a `debian:stretch` base. +For other bases, modify them as necessary. -1. Create a `Dockerfile` that creates the Docker container and installs Bazel with a minimal set of build tools: +1. Create a `Dockerfile` that creates the Docker container and installs Bazel + with a minimal set of build tools: - ``` - FROM debian:stretch + ``` + FROM debian:stretch - RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim + RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim - RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - + RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - - RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" - RUN apt-get update && apt-get install -y docker-ce + RUN apt-get update && apt-get install -y docker-ce - RUN wget https://releases.bazel.build/<latest Bazel version>/release/bazel-<latest Bazel version>-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh + RUN wget https://releases.bazel.build//release/bazel--installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh - RUN ./bazel-installer.sh - ``` + RUN ./bazel-installer.sh + ``` -2. Build the container as `bazel_container`: +2. Build the container as `bazel_container`: - ```posix-terminal - docker build -t bazel_container - < Dockerfile - ``` + ```posix-terminal + docker build -t bazel_container - < Dockerfile + ``` ### Step 2: Start the container -Start the Docker container using the command shown below. In the command, substitute the path to the source code on your host that you want to build. +Start the Docker container using the command shown below. In the command, +substitute the path to the source code on your host that you want to build. ```posix-terminal docker run -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp:/tmp \ - -v <var>your source code directory</var>:/src \ + -v {{ '' }}your source code directory{{ '' }}:/src \ -w /src \ bazel_container \ /bin/bash ``` -This command runs the container as root, mapping the docker socket, and mounting the `/tmp` directory. This allows Bazel to spawn other Docker containers and to use directories under `/tmp` to share files with those containers. Your source code is available at `/src` inside the container. +This command runs the container as root, mapping the docker socket, and mounting +the `/tmp` directory. This allows Bazel to spawn other Docker containers and to +use directories under `/tmp` to share files with those containers. Your source +code is available at `/src` inside the container. -The command intentionally starts from a `debian:stretch` base container that includes binaries incompatible with the `rbe-ubuntu16-04` container used as a toolchain container. If binaries from the local environment are leaking into the toolchain container, they will cause build errors. +The command intentionally starts from a `debian:stretch` base container that +includes binaries incompatible with the `rbe-ubuntu16-04` container used as a +toolchain container. If binaries from the local environment are leaking into the +toolchain container, they will cause build errors. ### Step 3: Test the container @@ -152,18 +232,28 @@ bazel version ### Step 4: Run the build -Run the build as shown below. The output user is root so that it corresponds to a directory that is accessible with the same absolute path from inside the host container in which Bazel runs, from the toolchain containers spawned by the Docker sandbox feature in which Bazel's build actions are running, and from the local machine on which the host and action containers run. +Run the build as shown below. The output user is root so that it corresponds to +a directory that is accessible with the same absolute path from inside the host +container in which Bazel runs, from the toolchain containers spawned by the Docker +sandbox feature in which Bazel's build actions are running, and from the local +machine on which the host and action containers run. ```posix-terminal -bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <var>target</var> +bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox {{ '' }}target{{ '' }} ``` ### Step 5: Resolve detected issues You can resolve build failures as follows: -- If the build fails with an "out of disk space" error, you can increase this limit by starting the host container with the flag `--memory=XX` where `XX` is the allocated disk space in gigabytes. This is experimental and may result in unpredictable behavior. +* If the build fails with an "out of disk space" error, you can increase this + limit by starting the host container with the flag `--memory=XX` where `XX` + is the allocated disk space in gigabytes. This is experimental and may + result in unpredictable behavior. -- If the build fails during the analysis or loading phases, one or more of your build rules declared in the WORKSPACE file are not compatible with remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) for possible causes and workarounds. +* If the build fails during the analysis or loading phases, one or more of + your build rules declared in the WORKSPACE file are not compatible with + remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) + for possible causes and workarounds. -- If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). +* If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). diff --git a/remote/workspace.mdx b/remote/workspace.mdx index 726f8bd3..ae0aea50 100644 --- a/remote/workspace.mdx +++ b/remote/workspace.mdx @@ -2,78 +2,126 @@ title: 'Finding Non-Hermetic Behavior in WORKSPACE Rules' --- + + In the following, a host machine is the machine where Bazel runs. -When using remote execution, the actual build and/or test steps are not happening on the host machine, but are instead sent off to the remote execution system. However, the steps involved in resolving workspace rules are happening on the host machine. If your workspace rules access information about the host machine for use during execution, your build is likely to break due to incompatibilities between the environments. +When using remote execution, the actual build and/or test steps are not +happening on the host machine, but are instead sent off to the remote execution +system. However, the steps involved in resolving workspace rules are happening +on the host machine. If your workspace rules access information about the +host machine for use during execution, your build is likely to break due to +incompatibilities between the environments. + +As part of [adapting Bazel rules for remote +execution](/remote/rules), you need to find such workspace rules +and fix them. This page describes how to find potentially problematic workspace +rules using the workspace log. -As part of [adapting Bazel rules for remote execution](/remote/rules), you need to find such workspace rules and fix them. This page describes how to find potentially problematic workspace rules using the workspace log. ## Finding non-hermetic rules -[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. All related commands are happening locally and can be a potential source of non-hermeticity. Usually non-hermetic behavior is introduced through [`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting with the host machine. +[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to +external workspaces, but they are rich enough to allow arbitrary processing to +happen in the process. All related commands are happening locally and can be a +potential source of non-hermeticity. Usually non-hermetic behavior is +introduced through +[`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting +with the host machine. -Starting with Bazel 0.18, you can get a log of some potentially non-hermetic actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to your Bazel command. Here `[PATH]` is a filename under which the log will be created. +Starting with Bazel 0.18, you can get a log of some potentially non-hermetic +actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to +your Bazel command. Here `[PATH]` is a filename under which the log will be +created. Things to note: -- the log captures the events as they are executed. If some steps are cached, they will not show up in the log, so to get a full result, don't forget to run `bazel clean --expunge` beforehand. +* the log captures the events as they are executed. If some steps are + cached, they will not show up in the log, so to get a full result, don't + forget to run `bazel clean --expunge` beforehand. -- Sometimes functions might be re-executed, in which case the related events will show up in the log multiple times. +* Sometimes functions might be re-executed, in which case the related + events will show up in the log multiple times. -- Workspace rules currently only log Starlark events. +* Workspace rules currently only log Starlark events. - Note: These particular rules do not cause hermiticity concerns as long as a hash is specified. + Note: These particular rules do not cause hermiticity concerns as long + as a hash is specified. To find what was executed during workspace initialization: -1. Run `bazel clean --expunge`. This command will clean your local cache and any cached repositories, ensuring that all initialization will be re-run. +1. Run `bazel clean --expunge`. This command will clean your local cache and + any cached repositories, ensuring that all initialization will be re-run. -2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your Bazel command and run the build. +2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your + Bazel command and run the build. - This produces a binary proto file listing messages of type [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) + This produces a binary proto file listing messages of type + [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -3. Download the Bazel source code and navigate to the Bazel folder by using the command below. You need the source code to be able to parse the workspace log with the [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). +3. Download the Bazel source code and navigate to the Bazel folder by using + the command below. You need the source code to be able to parse the + workspace log with the + [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). - ```posix-terminal - git clone https://github.com/bazelbuild/bazel.git + ```posix-terminal + git clone https://github.com/bazelbuild/bazel.git - cd bazel - ``` + cd bazel + ``` -4. In the Bazel source code repo, convert the whole workspace log to text. +4. In the Bazel source code repo, convert the whole workspace log to text. - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt + ``` -5. The output may be quite verbose and include output from built in Bazel rules. +5. The output may be quite verbose and include output from built in Bazel + rules. - To exclude specific rules from the output, use `--exclude_rule` option. For example: + To exclude specific rules from the output, use `--exclude_rule` option. + For example: - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ - --exclude_rule "//external:local_config_cc" \ - --exclude_rule "//external:dep" > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ + --exclude_rule "//external:local_config_cc" \ + --exclude_rule "//external:dep" > /tmp/workspacelog.txt + ``` -6. Open `/tmp/workspacelog.txt` and check for unsafe operations. +5. Open `/tmp/workspacelog.txt` and check for unsafe operations. -The log consists of [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) messages outlining certain potentially non-hermetic actions performed on a [`repository_ctx`](/rules/lib/builtins/repository_ctx). +The log consists of +[WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) +messages outlining certain potentially non-hermetic actions performed on a +[`repository_ctx`](/rules/lib/builtins/repository_ctx). The actions that have been highlighted as potentially non-hermetic are as follows: -- `execute`: executes an arbitrary command on the host environment. Check if these may introduce any dependencies on the host environment. +* `execute`: executes an arbitrary command on the host environment. Check if + these may introduce any dependencies on the host environment. -- `download`, `download_and_extract`: to ensure hermetic builds, make sure that sha256 is specified +* `download`, `download_and_extract`: to ensure hermetic builds, make sure + that sha256 is specified -- `file`, `template`: this is not non-hermetic in itself, but may be a mechanism for introducing dependencies on the host environment into the repository. Ensure that you understand where the input comes from, and that it does not depend on the host environment. +* `file`, `template`: this is not non-hermetic in itself, but may be a mechanism + for introducing dependencies on the host environment into the repository. + Ensure that you understand where the input comes from, and that it does not + depend on the host environment. -- `os`: this is not non-hermetic in itself, but an easy way to get dependencies on the host environment. A hermetic build would generally not call this. In evaluating whether your usage is hermetic, keep in mind that this is running on the host and not on the workers. Getting environment specifics from the host is generally not a good idea for remote builds. +* `os`: this is not non-hermetic in itself, but an easy way to get dependencies + on the host environment. A hermetic build would generally not call this. + In evaluating whether your usage is hermetic, keep in mind that this is + running on the host and not on the workers. Getting environment specifics + from the host is generally not a good idea for remote builds. -- `symlink`: this is normally safe, but look for red flags. Any symlinks to outside the repository or to an absolute path would cause problems on the remote worker. If the symlink is created based on host machine properties it would probably be problematic as well. +* `symlink`: this is normally safe, but look for red flags. Any symlinks to + outside the repository or to an absolute path would cause problems on the + remote worker. If the symlink is created based on host machine properties + it would probably be problematic as well. -- `which`: checking for programs installed on the host is usually problematic since the workers may have different configurations. +* `which`: checking for programs installed on the host is usually problematic + since the workers may have different configurations. diff --git a/rules/bzl-style.mdx b/rules/bzl-style.mdx index 8d85a7b5..65f589f4 100644 --- a/rules/bzl-style.mdx +++ b/rules/bzl-style.mdx @@ -2,49 +2,92 @@ title: '.bzl style guide' --- -This page covers basic style guidelines for Starlark and also includes information on macros and rules. -[Starlark](/rules/language) is a language that defines how software is built, and as such it is both a programming and a configuration language. -You will use Starlark to write `BUILD` files, macros, and build rules. Macros and rules are essentially meta-languages - they define how `BUILD` files are written. `BUILD` files are intended to be simple and repetitive. +This page covers basic style guidelines for Starlark and also includes +information on macros and rules. -All software is read more often than it is written. This is especially true for Starlark, as engineers read `BUILD` files to understand dependencies of their targets and details of their builds. This reading will often happen in passing, in a hurry, or in parallel to accomplishing some other task. Consequently, simplicity and readability are very important so that users can parse and comprehend `BUILD` files quickly. +[Starlark](/rules/language) is a +language that defines how software is built, and as such it is both a +programming and a configuration language. -When a user opens a `BUILD` file, they quickly want to know the list of targets in the file; or review the list of sources of that C++ library; or remove a dependency from that Java binary. Each time you add a layer of abstraction, you make it harder for a user to do these tasks. +You will use Starlark to write `BUILD` files, macros, and build rules. Macros and +rules are essentially meta-languages - they define how `BUILD` files are written. +`BUILD` files are intended to be simple and repetitive. -`BUILD` files are also analyzed and updated by many different tools. Tools may not be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` files simple will allow you to get better tooling. As a code base grows, it becomes more and more frequent to do changes across many `BUILD` files in order to update a library or do a cleanup. +All software is read more often than it is written. This is especially true for +Starlark, as engineers read `BUILD` files to understand dependencies of their +targets and details of their builds. This reading will often happen in passing, +in a hurry, or in parallel to accomplishing some other task. Consequently, +simplicity and readability are very important so that users can parse and +comprehend `BUILD` files quickly. -Important: Do not create a variable or macro just to avoid some amount of repetition in `BUILD` files. Your `BUILD` file should be easily readable both by developers and tools. The [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't really apply here. +When a user opens a `BUILD` file, they quickly want to know the list of targets in +the file; or review the list of sources of that C++ library; or remove a +dependency from that Java binary. Each time you add a layer of abstraction, you +make it harder for a user to do these tasks. + +`BUILD` files are also analyzed and updated by many different tools. Tools may not +be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` +files simple will allow you to get better tooling. As a code base grows, it +becomes more and more frequent to do changes across many `BUILD` files in order to +update a library or do a cleanup. + +Important: Do not create a variable or macro just to avoid some amount of +repetition in `BUILD` files. Your `BUILD` file should be easily readable both by +developers and tools. The +[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't +really apply here. ## General advice -- Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) as a formatter and linter. -- Follow [testing guidelines](/rules/testing). +* Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) + as a formatter and linter. +* Follow [testing guidelines](/rules/testing). ## Style ### Python style -When in doubt, follow the [PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. In particular, use four rather than two spaces for indentation to follow the Python convention. +When in doubt, follow the +[PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. +In particular, use four rather than two spaces for indentation to follow the +Python convention. + +Since +[Starlark is not Python](/rules/language#differences-with-python), +some aspects of Python style do not apply. For example, PEP 8 advises that +comparisons to singletons be done with `is`, which is not an operator in +Starlark. -Since [Starlark is not Python](/rules/language#differences-with-python), some aspects of Python style do not apply. For example, PEP 8 advises that comparisons to singletons be done with `is`, which is not an operator in Starlark. ### Docstring -Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Use a docstring at the top of each `.bzl` file, and a docstring for each public function. +Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). +Use a docstring at the top of each `.bzl` file, and a docstring for each public +function. ### Document rules and aspects -Rules and aspects, along with their attributes, as well as providers and their fields, should be documented using the `doc` argument. +Rules and aspects, along with their attributes, as well as providers and their +fields, should be documented using the `doc` argument. ### Naming convention -- Variables and function names use lowercase with words separated by underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. -- Top-level private values start with one underscore. Bazel enforces that private values cannot be used from other files. Local variables should not use the underscore prefix. +* Variables and function names use lowercase with words separated by + underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. +* Top-level private values start with one underscore. Bazel enforces that + private values cannot be used from other files. Local variables should not + use the underscore prefix. ### Line length -As in `BUILD` files, there is no strict line length limit as labels can be long. When possible, try to use at most 79 characters per line (following Python's style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline should not be enforced strictly: editors should display more than 80 columns, automated changes will frequently introduce longer lines, and humans shouldn't spend time splitting lines that are already readable. +As in `BUILD` files, there is no strict line length limit as labels can be long. +When possible, try to use at most 79 characters per line (following Python's +style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline +should not be enforced strictly: editors should display more than 80 columns, +automated changes will frequently introduce longer lines, and humans shouldn't +spend time splitting lines that are already readable. ### Keyword arguments @@ -62,41 +105,67 @@ def fct(name, srcs): ### Boolean values -Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values (such as when using a boolean attribute in a rule). +Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values +(such as when using a boolean attribute in a rule). ### Use print only for debugging -Do not use the `print()` function in production code; it is only intended for debugging, and will spam all direct and indirect users of your `.bzl` file. The only exception is that you may submit code that uses `print()` if it is disabled by default and can only be enabled by editing the source -- for example, if all uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to `False`. Be mindful of whether these statements are useful enough to justify their impact on readability. +Do not use the `print()` function in production code; it is only intended for +debugging, and will spam all direct and indirect users of your `.bzl` file. The +only exception is that you may submit code that uses `print()` if it is disabled +by default and can only be enabled by editing the source -- for example, if all +uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to +`False`. Be mindful of whether these statements are useful enough to justify +their impact on readability. ## Macros -A macro is a function which instantiates one or more rules during the loading phase. In general, use rules whenever possible instead of macros. The build graph seen by the user is not the same as the one used by Bazel during the build - macros are expanded *before Bazel does any build graph analysis.* - -Because of this, when something goes wrong, the user will need to understand your macro's implementation to troubleshoot build problems. Additionally, `bazel query` results can be hard to interpret because targets shown in the results come from macro expansion. Finally, aspects are not aware of macros, so tooling depending on aspects (IDEs and others) might fail. - -A safe use for macros is for defining additional targets intended to be referenced directly at the Bazel CLI or in BUILD files: In that case, only the *end users* of those targets need to know about them, and any build problems introduced by macros are never far from their usage. - -For macros that define generated targets (implementation details of the macro which are not supposed to be referred to at the CLI or depended on by targets not instantiated by that macro), follow these best practices: - -- A macro should take a `name` argument and define a target with that name. That target becomes that macro's *main target*. - -- Generated targets, that is all other targets defined by a macro, should: - - - Have their names prefixed by `<name>`. For example, using `name = '%s_bar' % (name)`. - - Have restricted visibility (`//visibility:private`), and - - Have a `manual` tag to avoid expansion in wildcard targets (`:all`, `...`, `:*`, etc). - -- The `name` should only be used to derive names of targets defined by the macro, and not for anything else. For example, don't use the name to derive a dependency or input file that is not generated by the macro itself. - -- All the targets created in the macro should be coupled in some way to the main target. - -- Conventionally, `name` should be the first argument when defining a macro. - -- Keep the parameter names in the macro consistent. If a parameter is passed as an attribute value to the main target, keep its name the same. If a macro parameter serves the same purpose as a common rule attribute, such as `deps`, name as you would the attribute (see below). - -- When calling a macro, use only keyword arguments. This is consistent with rules, and greatly improves readability. - -Engineers often write macros when the Starlark API of relevant rules is insufficient for their specific use case, regardless of whether the rule is defined within Bazel in native code, or in Starlark. If you're facing this problem, ask the rule author if they can extend the API to accomplish your goals. +A macro is a function which instantiates one or more rules during the loading +phase. In general, use rules whenever possible instead of macros. The build +graph seen by the user is not the same as the one used by Bazel during the +build - macros are expanded *before Bazel does any build graph analysis.* + +Because of this, when something goes wrong, the user will need to understand +your macro's implementation to troubleshoot build problems. Additionally, `bazel +query` results can be hard to interpret because targets shown in the results +come from macro expansion. Finally, aspects are not aware of macros, so tooling +depending on aspects (IDEs and others) might fail. + +A safe use for macros is for defining additional targets intended to be +referenced directly at the Bazel CLI or in BUILD files: In that case, only the +*end users* of those targets need to know about them, and any build problems +introduced by macros are never far from their usage. + +For macros that define generated targets (implementation details of the macro +which are not supposed to be referred to at the CLI or depended on by targets +not instantiated by that macro), follow these best practices: + +* A macro should take a `name` argument and define a target with that name. + That target becomes that macro's *main target*. +* Generated targets, that is all other targets defined by a macro, should: + * Have their names prefixed by ``. For example, using + `name = '%s_bar' % (name)`. + * Have restricted visibility (`//visibility:private`), and + * Have a `manual` tag to avoid expansion in wildcard targets (`:all`, + `...`, `:*`, etc). +* The `name` should only be used to derive names of targets defined by the + macro, and not for anything else. For example, don't use the name to derive + a dependency or input file that is not generated by the macro itself. +* All the targets created in the macro should be coupled in some way to the + main target. +* Conventionally, `name` should be the first argument when defining a macro. +* Keep the parameter names in the macro consistent. If a parameter is passed + as an attribute value to the main target, keep its name the same. If a macro + parameter serves the same purpose as a common rule attribute, such as + `deps`, name as you would the attribute (see below). +* When calling a macro, use only keyword arguments. This is consistent with + rules, and greatly improves readability. + +Engineers often write macros when the Starlark API of relevant rules is +insufficient for their specific use case, regardless of whether the rule is +defined within Bazel in native code, or in Starlark. If you're facing this +problem, ask the rule author if they can extend the API to accomplish your +goals. As a rule of thumb, the more macros resemble the rules, the better. @@ -104,28 +173,40 @@ See also [macros](/extending/macros#conventions). ## Rules -- Rules, aspects, and their attributes should use lower\_case names ("snake case"). - -- Rule names are nouns that describe the main kind of artifact produced by the rule, from the point of view of its dependencies (or for leaf rules, the user). This is not necessarily a file suffix. For instance, a rule that produces C++ artifacts meant to be used as Python extensions might be called `py_extension`. For most languages, typical rules include: - - - `*_library` - a compilation unit or "module". - - `*_binary` - a target producing an executable or a deployment unit. - - `*_test` - a test target. This can include multiple tests. Expect all tests in a `*_test` target to be variations on the same theme, for example, testing a single library. - - `*_import`: a target encapsulating a pre-compiled artifact, such as a `.jar`, or a `.dll` that is used during compilation. - -- Use consistent names and types for attributes. Some generally applicable attributes include: - - - `srcs`: `label_list`, allowing files: source files, typically human-authored. - - `deps`: `label_list`, typically *not* allowing files: compilation dependencies. - - `data`: `label_list`, allowing files: data files, such as test data etc. - - `runtime_deps`: `label_list`: runtime dependencies that are not needed for compilation. - -- For any attributes with non-obvious behavior (for example, string templates with special substitutions, or tools that are invoked with specific requirements), provide documentation using the `doc` keyword argument to the attribute's declaration (`attr.label_list()` or similar). - -- Rule implementation functions should almost always be private functions (named with a leading underscore). A common style is to give the implementation function for `myrule` the name `_myrule_impl`. - -- Pass information between your rules using a well-defined [provider](/extending/rules#providers) interface. Declare and document provider fields. - -- Design your rule with extensibility in mind. Consider that other rules might want to interact with your rule, access your providers, and reuse the actions you create. - -- Follow [performance guidelines](/rules/performance) in your rules. +* Rules, aspects, and their attributes should use lower_case names ("snake + case"). +* Rule names are nouns that describe the main kind of artifact produced by the + rule, from the point of view of its dependencies (or for leaf rules, the + user). This is not necessarily a file suffix. For instance, a rule that + produces C++ artifacts meant to be used as Python extensions might be called + `py_extension`. For most languages, typical rules include: + * `*_library` - a compilation unit or "module". + * `*_binary` - a target producing an executable or a deployment unit. + * `*_test` - a test target. This can include multiple tests. Expect all + tests in a `*_test` target to be variations on the same theme, for + example, testing a single library. + * `*_import`: a target encapsulating a pre-compiled artifact, such as a + `.jar`, or a `.dll` that is used during compilation. +* Use consistent names and types for attributes. Some generally applicable + attributes include: + * `srcs`: `label_list`, allowing files: source files, typically + human-authored. + * `deps`: `label_list`, typically *not* allowing files: compilation + dependencies. + * `data`: `label_list`, allowing files: data files, such as test data etc. + * `runtime_deps`: `label_list`: runtime dependencies that are not needed + for compilation. +* For any attributes with non-obvious behavior (for example, string templates + with special substitutions, or tools that are invoked with specific + requirements), provide documentation using the `doc` keyword argument to the + attribute's declaration (`attr.label_list()` or similar). +* Rule implementation functions should almost always be private functions + (named with a leading underscore). A common style is to give the + implementation function for `myrule` the name `_myrule_impl`. +* Pass information between your rules using a well-defined + [provider](/extending/rules#providers) interface. Declare and document provider + fields. +* Design your rule with extensibility in mind. Consider that other rules might + want to interact with your rule, access your providers, and reuse the + actions you create. +* Follow [performance guidelines](/rules/performance) in your rules. diff --git a/rules/challenges.mdx b/rules/challenges.mdx index e31c5d45..10ff7371 100644 --- a/rules/challenges.mdx +++ b/rules/challenges.mdx @@ -2,88 +2,222 @@ title: 'Challenges of Writing Rules' --- -This page gives a high-level overview of the specific issues and challenges of writing efficient Bazel rules. + + +This page gives a high-level overview of the specific issues and challenges +of writing efficient Bazel rules. ## Summary Requirements -- Assumption: Aim for Correctness, Throughput, Ease of Use & Latency -- Assumption: Large Scale Repositories -- Assumption: BUILD-like Description Language -- Historic: Hard Separation between Loading, Analysis, and Execution is Outdated, but still affects the API -- Intrinsic: Remote Execution and Caching are Hard -- Intrinsic: Using Change Information for Correct and Fast Incremental Builds requires Unusual Coding Patterns -- Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard +* Assumption: Aim for Correctness, Throughput, Ease of Use & Latency +* Assumption: Large Scale Repositories +* Assumption: BUILD-like Description Language +* Historic: Hard Separation between Loading, Analysis, and Execution is + Outdated, but still affects the API +* Intrinsic: Remote Execution and Caching are Hard +* Intrinsic: Using Change Information for Correct and Fast Incremental Builds + requires Unusual Coding Patterns +* Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard ## Assumptions -Here are some assumptions made about the build system, such as need for correctness, ease of use, throughput, and large scale repositories. The following sections address these assumptions and offer guidelines to ensure rules are written in an effective manner. +Here are some assumptions made about the build system, such as need for +correctness, ease of use, throughput, and large scale repositories. The +following sections address these assumptions and offer guidelines to ensure +rules are written in an effective manner. ### Aim for correctness, throughput, ease of use & latency -We assume that the build system needs to be first and foremost correct with respect to incremental builds. For a given source tree, the output of the same build should always be the same, regardless of what the output tree looks like. In the first approximation, this means Bazel needs to know every single input that goes into a given build step, such that it can rerun that step if any of the inputs change. There are limits to how correct Bazel can get, as it leaks some information such as date / time of the build, and ignores certain types of changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) helps ensure correctness by preventing reads to undeclared input files. Besides the intrinsic limits of the system, there are a few known correctness issues, most of which are related to Fileset or the C++ rules, which are both hard problems. We have long-term efforts to fix these. - -The second goal of the build system is to have high throughput; we are permanently pushing the boundaries of what can be done within the current machine allocation for a remote execution service. If the remote execution service gets overloaded, nobody can get work done. - -Ease of use comes next. Of multiple correct approaches with the same (or similar) footprint of the remote execution service, we choose the one that is easier to use. - -Latency denotes the time it takes from starting a build to getting the intended result, whether that is a test log from a passing or failing test, or an error message that a `BUILD` file has a typo. - -Note that these goals often overlap; latency is as much a function of throughput of the remote execution service as is correctness relevant for ease of use. +We assume that the build system needs to be first and foremost correct with +respect to incremental builds. For a given source tree, the output of the +same build should always be the same, regardless of what the output tree looks +like. In the first approximation, this means Bazel needs to know every single +input that goes into a given build step, such that it can rerun that step if any +of the inputs change. There are limits to how correct Bazel can get, as it leaks +some information such as date / time of the build, and ignores certain types of +changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) +helps ensure correctness by preventing reads to undeclared input files. Besides +the intrinsic limits of the system, there are a few known correctness issues, +most of which are related to Fileset or the C++ rules, which are both hard +problems. We have long-term efforts to fix these. + +The second goal of the build system is to have high throughput; we are +permanently pushing the boundaries of what can be done within the current +machine allocation for a remote execution service. If the remote execution +service gets overloaded, nobody can get work done. + +Ease of use comes next. Of multiple correct approaches with the same (or +similar) footprint of the remote execution service, we choose the one that is +easier to use. + +Latency denotes the time it takes from starting a build to getting the intended +result, whether that is a test log from a passing or failing test, or an error +message that a `BUILD` file has a typo. + +Note that these goals often overlap; latency is as much a function of throughput +of the remote execution service as is correctness relevant for ease of use. ### Large scale repositories -The build system needs to operate at the scale of large repositories where large scale means that it does not fit on a single hard drive, so it is impossible to do a full checkout on virtually all developer machines. A medium-sized build will need to read and parse tens of thousands of `BUILD` files, and evaluate hundreds of thousands of globs. While it is theoretically possible to read all `BUILD` files on a single machine, we have not yet been able to do so within a reasonable amount of time and memory. As such, it is critical that `BUILD` files can be loaded and parsed independently. +The build system needs to operate at the scale of large repositories where large +scale means that it does not fit on a single hard drive, so it is impossible to +do a full checkout on virtually all developer machines. A medium-sized build +will need to read and parse tens of thousands of `BUILD` files, and evaluate +hundreds of thousands of globs. While it is theoretically possible to read all +`BUILD` files on a single machine, we have not yet been able to do so within a +reasonable amount of time and memory. As such, it is critical that `BUILD` files +can be loaded and parsed independently. ### BUILD-like description language -In this context, we assume a configuration language that is roughly similar to `BUILD` files in declaration of library and binary rules and their interdependencies. `BUILD` files can be read and parsed independently, and we avoid even looking at source files whenever we can (except for existence). +In this context, we assume a configuration language that is +roughly similar to `BUILD` files in declaration of library and binary rules +and their interdependencies. `BUILD` files can be read and parsed independently, +and we avoid even looking at source files whenever we can (except for +existence). ## Historic -There are differences between Bazel versions that cause challenges and some of these are outlined in the following sections. +There are differences between Bazel versions that cause challenges and some +of these are outlined in the following sections. ### Hard separation between loading, analysis, and execution is outdated but still affects the API -Technically, it is sufficient for a rule to know the input and output files of an action just before the action is sent to remote execution. However, the original Bazel code base had a strict separation of loading packages, then analyzing rules using a configuration (command-line flags, essentially), and only then running any actions. This distinction is still part of the rules API today, even though the core of Bazel no longer requires it (more details below). - -That means that the rules API requires a declarative description of the rule interface (what attributes it has, types of attributes). There are some exceptions where the API allows custom code to run during the loading phase to compute implicit names of output files and implicit values of attributes. For example, a java\_library rule named 'foo' implicitly generates an output named 'libfoo.jar', which can be referenced from other rules in the build graph. - -Furthermore, the analysis of a rule cannot read any source files or inspect the output of an action; instead, it needs to generate a partial directed bipartite graph of build steps and output file names that is only determined from the rule itself and its dependencies. +Technically, it is sufficient for a rule to know the input and output files of +an action just before the action is sent to remote execution. However, the +original Bazel code base had a strict separation of loading packages, then +analyzing rules using a configuration (command-line flags, essentially), and +only then running any actions. This distinction is still part of the rules API +today, even though the core of Bazel no longer requires it (more details below). + +That means that the rules API requires a declarative description of the rule +interface (what attributes it has, types of attributes). There are some +exceptions where the API allows custom code to run during the loading phase to +compute implicit names of output files and implicit values of attributes. For +example, a java_library rule named 'foo' implicitly generates an output named +'libfoo.jar', which can be referenced from other rules in the build graph. + +Furthermore, the analysis of a rule cannot read any source files or inspect the +output of an action; instead, it needs to generate a partial directed bipartite +graph of build steps and output file names that is only determined from the rule +itself and its dependencies. ## Intrinsic -There are some intrinsic properties that make writing rules challenging and some of the most common ones are described in the following sections. +There are some intrinsic properties that make writing rules challenging and +some of the most common ones are described in the following sections. ### Remote execution and caching are hard -Remote execution and caching improve build times in large repositories by roughly two orders of magnitude compared to running the build on a single machine. However, the scale at which it needs to perform is staggering: Google's remote execution service is designed to handle a huge number of requests per second, and the protocol carefully avoids unnecessary roundtrips as well as unnecessary work on the service side. +Remote execution and caching improve build times in large repositories by +roughly two orders of magnitude compared to running the build on a single +machine. However, the scale at which it needs to perform is staggering: Google's +remote execution service is designed to handle a huge number of requests per +second, and the protocol carefully avoids unnecessary roundtrips as well as +unnecessary work on the service side. -At this time, the protocol requires that the build system knows all inputs to a given action ahead of time; the build system then computes a unique action fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, the scheduler replies with the digests of the output files; the files itself are addressed by digest later on. However, this imposes restrictions on the Bazel rules, which need to declare all input files ahead of time. +At this time, the protocol requires that the build system knows all inputs to a +given action ahead of time; the build system then computes a unique action +fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, +the scheduler replies with the digests of the output files; the files itself are +addressed by digest later on. However, this imposes restrictions on the Bazel +rules, which need to declare all input files ahead of time. ### Using change information for correct and fast incremental builds requires unusual coding patterns -Above, we argued that in order to be correct, Bazel needs to know all the input files that go into a build step in order to detect whether that build step is still up-to-date. The same is true for package loading and rule analysis, and we have designed [Skyframe](/reference/skyframe) to handle this in general. Skyframe is a graph library and evaluation framework that takes a goal node (such as 'build //foo with these options'), and breaks it down into its constituent parts, which are then evaluated and combined to yield this result. As part of this process, Skyframe reads packages, analyzes rules, and executes actions. - -At each node, Skyframe tracks exactly which nodes any given node used to compute its own output, all the way from the goal node down to the input files (which are also Skyframe nodes). Having this graph explicitly represented in memory allows the build system to identify exactly which nodes are affected by a given change to an input file (including creation or deletion of an input file), doing the minimal amount of work to restore the output tree to its intended state. - -As part of this, each node performs a dependency discovery process. Each node can declare dependencies, and then use the contents of those dependencies to declare even further dependencies. In principle, this maps well to a thread-per-node model. However, medium-sized builds contain hundreds of thousands of Skyframe nodes, which isn't easily possible with current Java technology (and for historical reasons, we're currently tied to using Java, so no lightweight threads and no continuations). - -Instead, Bazel uses a fixed-size thread pool. However, that means that if a node declares a dependency that isn't available yet, we may have to abort that evaluation and restart it (possibly in another thread), when the dependency is available. This, in turn, means that nodes should not do this excessively; a node that declares N dependencies serially can potentially be restarted N times, costing O(N^2) time. Instead, we aim for up-front bulk declaration of dependencies, which sometimes requires reorganizing the code, or even splitting a node into multiple nodes to limit the number of restarts. - -Note that this technology isn't currently available in the rules API; instead, the rules API is still defined using the legacy concepts of loading, analysis, and execution phases. However, a fundamental restriction is that all accesses to other nodes have to go through the framework so that it can track the corresponding dependencies. Regardless of the language in which the build system is implemented or in which the rules are written (they don't have to be the same), rule authors must not use standard libraries or patterns that bypass Skyframe. For Java, that means avoiding java.io.File as well as any form of reflection, and any library that does either. Libraries that support dependency injection of these low-level interfaces still need to be setup correctly for Skyframe. - -This strongly suggests to avoid exposing rule authors to a full language runtime in the first place. The danger of accidental use of such APIs is just too big - several Bazel bugs in the past were caused by rules using unsafe APIs, even though the rules were written by the Bazel team or other domain experts. +Above, we argued that in order to be correct, Bazel needs to know all the input +files that go into a build step in order to detect whether that build step is +still up-to-date. The same is true for package loading and rule analysis, and we +have designed [Skyframe](/reference/skyframe) to handle this +in general. Skyframe is a graph library and evaluation framework that takes a +goal node (such as 'build //foo with these options'), and breaks it down into +its constituent parts, which are then evaluated and combined to yield this +result. As part of this process, Skyframe reads packages, analyzes rules, and +executes actions. + +At each node, Skyframe tracks exactly which nodes any given node used to compute +its own output, all the way from the goal node down to the input files (which +are also Skyframe nodes). Having this graph explicitly represented in memory +allows the build system to identify exactly which nodes are affected by a given +change to an input file (including creation or deletion of an input file), doing +the minimal amount of work to restore the output tree to its intended state. + +As part of this, each node performs a dependency discovery process. Each +node can declare dependencies, and then use the contents of those dependencies +to declare even further dependencies. In principle, this maps well to a +thread-per-node model. However, medium-sized builds contain hundreds of +thousands of Skyframe nodes, which isn't easily possible with current Java +technology (and for historical reasons, we're currently tied to using Java, so +no lightweight threads and no continuations). + +Instead, Bazel uses a fixed-size thread pool. However, that means that if a node +declares a dependency that isn't available yet, we may have to abort that +evaluation and restart it (possibly in another thread), when the dependency is +available. This, in turn, means that nodes should not do this excessively; a +node that declares N dependencies serially can potentially be restarted N times, +costing O(N^2) time. Instead, we aim for up-front bulk declaration of +dependencies, which sometimes requires reorganizing the code, or even splitting +a node into multiple nodes to limit the number of restarts. + +Note that this technology isn't currently available in the rules API; instead, +the rules API is still defined using the legacy concepts of loading, analysis, +and execution phases. However, a fundamental restriction is that all accesses to +other nodes have to go through the framework so that it can track the +corresponding dependencies. Regardless of the language in which the build system +is implemented or in which the rules are written (they don't have to be the +same), rule authors must not use standard libraries or patterns that bypass +Skyframe. For Java, that means avoiding java.io.File as well as any form of +reflection, and any library that does either. Libraries that support dependency +injection of these low-level interfaces still need to be setup correctly for +Skyframe. + +This strongly suggests to avoid exposing rule authors to a full language runtime +in the first place. The danger of accidental use of such APIs is just too big - +several Bazel bugs in the past were caused by rules using unsafe APIs, even +though the rules were written by the Bazel team or other domain experts. ### Avoiding quadratic time and memory consumption is hard -To make matters worse, apart from the requirements imposed by Skyframe, the historical constraints of using Java, and the outdatedness of the rules API, accidentally introducing quadratic time or memory consumption is a fundamental problem in any build system based on library and binary rules. There are two very common patterns that introduce quadratic memory consumption (and therefore quadratic time consumption). - -1. Chains of Library Rules - Consider the case of a chain of library rules A depends on B, depends on C, and so on. Then, we want to compute some property over the transitive closure of these rules, such as the Java runtime classpath, or the C++ linker command for each library. Naively, we might take a standard list implementation; however, this already introduces quadratic memory consumption: the first library contains one entry on the classpath, the second two, the third three, and so on, for a total of 1+2+3+...+N = O(N^2) entries. - -2. Binary Rules Depending on the Same Library Rules - Consider the case where a set of binaries that depend on the same library rules — such as if you have a number of test rules that test the same library code. Let's say out of N rules, half the rules are binary rules, and the other half library rules. Now consider that each binary makes a copy of some property computed over the transitive closure of library rules, such as the Java runtime classpath, or the C++ linker command line. For example, it could expand the command line string representation of the C++ link action. N/2 copies of N/2 elements is O(N^2) memory. +To make matters worse, apart from the requirements imposed by Skyframe, the +historical constraints of using Java, and the outdatedness of the rules API, +accidentally introducing quadratic time or memory consumption is a fundamental +problem in any build system based on library and binary rules. There are two +very common patterns that introduce quadratic memory consumption (and therefore +quadratic time consumption). + +1. Chains of Library Rules - +Consider the case of a chain of library rules A depends on B, depends on C, and +so on. Then, we want to compute some property over the transitive closure of +these rules, such as the Java runtime classpath, or the C++ linker command for +each library. Naively, we might take a standard list implementation; however, +this already introduces quadratic memory consumption: the first library +contains one entry on the classpath, the second two, the third three, and so +on, for a total of 1+2+3+...+N = O(N^2) entries. + +2. Binary Rules Depending on the Same Library Rules - +Consider the case where a set of binaries that depend on the same library +rules — such as if you have a number of test rules that test the same +library code. Let's say out of N rules, half the rules are binary rules, and +the other half library rules. Now consider that each binary makes a copy of +some property computed over the transitive closure of library rules, such as +the Java runtime classpath, or the C++ linker command line. For example, it +could expand the command line string representation of the C++ link action. N/2 +copies of N/2 elements is O(N^2) memory. #### Custom collections classes to avoid quadratic complexity -Bazel is heavily affected by both of these scenarios, so we introduced a set of custom collection classes that effectively compress the information in memory by avoiding the copy at each step. Almost all of these data structures have set semantics, so we called it [depset](/rules/lib/depset) (also known as `NestedSet` in the internal implementation). The majority of changes to reduce Bazel's memory consumption over the past several years were changes to use depsets instead of whatever was previously used. - -Unfortunately, usage of depsets does not automatically solve all the issues; in particular, even just iterating over a depset in each rule re-introduces quadratic time consumption. Internally, NestedSets also has some helper methods to facilitate interoperability with normal collections classes; unfortunately, accidentally passing a NestedSet to one of these methods leads to copying behavior, and reintroduces quadratic memory consumption. +Bazel is heavily affected by both of these scenarios, so we introduced a set of +custom collection classes that effectively compress the information in memory by +avoiding the copy at each step. Almost all of these data structures have set +semantics, so we called it +[depset](/rules/lib/depset) +(also known as `NestedSet` in the internal implementation). The majority of +changes to reduce Bazel's memory consumption over the past several years were +changes to use depsets instead of whatever was previously used. + +Unfortunately, usage of depsets does not automatically solve all the issues; +in particular, even just iterating over a depset in each rule re-introduces +quadratic time consumption. Internally, NestedSets also has some helper methods +to facilitate interoperability with normal collections classes; unfortunately, +accidentally passing a NestedSet to one of these methods leads to copying +behavior, and reintroduces quadratic memory consumption. diff --git a/rules/deploying.mdx b/rules/deploying.mdx index 4395dd28..3fe2c869 100644 --- a/rules/deploying.mdx +++ b/rules/deploying.mdx @@ -2,30 +2,48 @@ title: 'Deploying Rules' --- -This page is for rule writers who are planning to make their rules available to others. -We recommend you start a new ruleset from the template repository: [https://github.com/bazel-contrib/rules-template](https://github.com/bazel-contrib/rules-template) That template follows the recommendations below, and includes API documentation generation and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. + +This page is for rule writers who are planning to make their rules available +to others. + +We recommend you start a new ruleset from the template repository: +https://github.com/bazel-contrib/rules-template +That template follows the recommendations below, and includes API documentation generation +and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. ## Hosting and naming rules -New rules should go into their own GitHub repository under your organization. Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) organization. +New rules should go into their own GitHub repository under your organization. +Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) +if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) +organization. -Repository names for Bazel rules are standardized on the following format: `$ORGANIZATION/rules_$NAME`. See [examples on GitHub](https://github.com/search?q=rules+bazel\&type=Repositories). For consistency, you should follow this same format when publishing your Bazel rules. +Repository names for Bazel rules are standardized on the following format: +`$ORGANIZATION/rules_$NAME`. +See [examples on GitHub](https://github.com/search?q=rules+bazel&type=Repositories). +For consistency, you should follow this same format when publishing your Bazel rules. -Make sure to use a descriptive GitHub repository description and `README.md` title, example: +Make sure to use a descriptive GitHub repository description and `README.md` +title, example: -- Repository name: `bazelbuild/rules_go` -- Repository description: *Go rules for Bazel* -- Repository tags: `golang`, `bazel` -- `README.md` header: *Go rules for [Bazel](https://bazel.build)* (note the link to [https://bazel.build](https://bazel.build) which will guide users who are unfamiliar with Bazel to the right place) +* Repository name: `bazelbuild/rules_go` +* Repository description: *Go rules for Bazel* +* Repository tags: `golang`, `bazel` +* `README.md` header: *Go rules for [Bazel](https://bazel.build)* +(note the link to https://bazel.build which will guide users who are unfamiliar +with Bazel to the right place) -Rules can be grouped either by language (such as Scala), runtime platform (such as Android), or framework (such as Spring). +Rules can be grouped either by language (such as Scala), runtime platform +(such as Android), or framework (such as Spring). ## Repository content -Every rule repository should have a certain layout so that users can quickly understand new rules. +Every rule repository should have a certain layout so that users can quickly +understand new rules. -For example, when writing new rules for the (make-believe) `mockascript` language, the rule repository would have the following structure: +For example, when writing new rules for the (make-believe) +`mockascript` language, the rule repository would have the following structure: ``` / @@ -53,9 +71,17 @@ For example, when writing new rules for the (make-believe) `mockascript` languag ### MODULE.bazel -In the project's `MODULE.bazel`, you should define the name that users will use to reference your rules. If your rules belong to the [bazelbuild](https://github.com/bazelbuild) organization, you must use `rules_<lang>` (such as `rules_mockascript`). Otherwise, you should name your repository `<org>_rules_<lang>` (such as `build_stack_rules_proto`). Please start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules should follow the convention for rules in the [bazelbuild](https://github.com/bazelbuild) organization. +In the project's `MODULE.bazel`, you should define the name that users will use +to reference your rules. If your rules belong to the +[bazelbuild](https://github.com/bazelbuild) organization, you must use +`rules_` (such as `rules_mockascript`). Otherwise, you should name your +repository `_rules_` (such as `build_stack_rules_proto`). Please +start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) +if you feel like your rules should follow the convention for rules in the +[bazelbuild](https://github.com/bazelbuild) organization. -In the following sections, assume the repository belongs to the [bazelbuild](https://github.com/bazelbuild) organization. +In the following sections, assume the repository belongs to the +[bazelbuild](https://github.com/bazelbuild) organization. ``` module(name = "rules_mockascript") @@ -63,11 +89,16 @@ module(name = "rules_mockascript") ### README -At the top level, there should be a `README` that contains a brief description of your ruleset, and the API users should expect. +At the top level, there should be a `README` that contains a brief description +of your ruleset, and the API users should expect. ### Rules -Often times there will be multiple rules provided by your repository. Create a directory named by the language and provide an entry point - `defs.bzl` file exporting all rules (also include a `BUILD` file so the directory is a package). For `rules_mockascript` that means there will be a directory named `mockascript`, and a `BUILD` file and a `defs.bzl` file inside: +Often times there will be multiple rules provided by your repository. Create a +directory named by the language and provide an entry point - `defs.bzl` file +exporting all rules (also include a `BUILD` file so the directory is a package). +For `rules_mockascript` that means there will be a directory named +`mockascript`, and a `BUILD` file and a `defs.bzl` file inside: ``` / @@ -78,7 +109,11 @@ Often times there will be multiple rules provided by your repository. Create a d ### Constraints -If your rule defines [toolchain](/extending/toolchains) rules, it's possible that you'll need to define custom `constraint_setting`s and/or `constraint_value`s. Put these into a `//<LANG>/constraints` package. Your directory structure will look like this: +If your rule defines +[toolchain](/extending/toolchains) rules, +it's possible that you'll need to define custom `constraint_setting`s and/or +`constraint_value`s. Put these into a `///constraints` package. Your +directory structure will look like this: ``` / @@ -89,58 +124,100 @@ If your rule defines [toolchain](/extending/toolchains) rules, it's possible tha defs.bzl ``` -Please read [github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) for best practices, and to see what constraints are already present, and consider contributing your constraints there if they are language independent. Be mindful of introducing custom constraints, all users of your rules will use them to perform platform specific logic in their `BUILD` files (for example, using [selects](/reference/be/functions#select)). With custom constraints, you define a language that the whole Bazel ecosystem will speak. +Please read +[github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) +for best practices, and to see what constraints are already present, and +consider contributing your constraints there if they are language independent. +Be mindful of introducing custom constraints, all users of your rules will +use them to perform platform specific logic in their `BUILD` files (for example, +using [selects](/reference/be/functions#select)). +With custom constraints, you define a language that the whole Bazel ecosystem +will speak. ### Runfiles library -If your rule provides a standard library for accessing runfiles, it should be in the form of a library target located at `//<LANG>/runfiles` (an abbreviation of `//<LANG>/runfiles:runfiles`). User targets that need to access their data dependencies will typically add this target to their `deps` attribute. +If your rule provides a standard library for accessing runfiles, it should be +in the form of a library target located at `///runfiles` (an abbreviation +of `///runfiles:runfiles`). User targets that need to access their data +dependencies will typically add this target to their `deps` attribute. ### Repository rules #### Dependencies -Your rules might have external dependencies, which you'll need to specify in your MODULE.bazel file. +Your rules might have external dependencies, which you'll need to specify in +your MODULE.bazel file. #### Registering toolchains -Your rules might also register toolchains, which you can also specify in the MODULE.bazel file. +Your rules might also register toolchains, which you can also specify in the +MODULE.bazel file. + +Note that in order to resolve toolchains in the analysis phase Bazel needs to +analyze all `toolchain` targets that are registered. Bazel will not need to +analyze all targets referenced by `toolchain.toolchain` attribute. If in order +to register toolchains you need to perform complex computation in the +repository, consider splitting the repository with `toolchain` targets from the +repository with `_toolchain` targets. Former will be always fetched, and +the latter will only be fetched when user actually needs to build `` code. -Note that in order to resolve toolchains in the analysis phase Bazel needs to analyze all `toolchain` targets that are registered. Bazel will not need to analyze all targets referenced by `toolchain.toolchain` attribute. If in order to register toolchains you need to perform complex computation in the repository, consider splitting the repository with `toolchain` targets from the repository with `<LANG>_toolchain` targets. Former will be always fetched, and the latter will only be fetched when user actually needs to build `<LANG>` code. #### Release snippet -In your release announcement provide a snippet that your users can copy-paste into their `MODULE.bazel` file. This snippet in general will look as follows: +In your release announcement provide a snippet that your users can copy-paste +into their `MODULE.bazel` file. This snippet in general will look as follows: ``` -bazel_dep(name = "rules_<LANG>", version = "<VERSION>") +bazel_dep(name = "rules_", version = "") ``` + ### Tests -There should be tests that verify that the rules are working as expected. This can either be in the standard location for the language the rules are for or a `tests/` directory at the top level. +There should be tests that verify that the rules are working as expected. This +can either be in the standard location for the language the rules are for or a +`tests/` directory at the top level. ### Examples (optional) -It is useful to users to have an `examples/` directory that shows users a couple of basic ways that the rules can be used. +It is useful to users to have an `examples/` directory that shows users a couple +of basic ways that the rules can be used. ## CI/CD -Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. See comments in the rules-template repo for more information. +Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib +org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. +See comments in the rules-template repo for more information. -If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md\&title=Request+to+add+new+project+%5BPROJECT_NAME%5D\&labels=new-project) it to [ci.bazel.build](http://ci.bazel.build). +If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), +you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md&title=Request+to+add+new+project+%5BPROJECT_NAME%5D&labels=new-project) +it to [ci.bazel.build](http://ci.bazel.build). ## Documentation -See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for instructions on how to comment your rules so that documentation can be generated automatically. +See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for +instructions on how to comment your rules so that documentation can be generated +automatically. -The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date as Starlark files are updated. +The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) +shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date +as Starlark files are updated. ## FAQs ### Why can't we add our rule to the main Bazel GitHub repository? -We want to decouple rules from Bazel releases as much as possible. It's clearer who owns individual rules, reducing the load on Bazel developers. For our users, decoupling makes it easier to modify, upgrade, downgrade, and replace rules. Contributing to rules can be lighter weight than contributing to Bazel - depending on the rules -, including full submit access to the corresponding GitHub repository. Getting submit access to Bazel itself is a much more involved process. +We want to decouple rules from Bazel releases as much as possible. It's clearer +who owns individual rules, reducing the load on Bazel developers. For our users, +decoupling makes it easier to modify, upgrade, downgrade, and replace rules. +Contributing to rules can be lighter weight than contributing to Bazel - +depending on the rules -, including full submit access to the corresponding +GitHub repository. Getting submit access to Bazel itself is a much more involved +process. -The downside is a more complicated one-time installation process for our users: they have to add a dependency on your ruleset in their `MODULE.bazel` file. +The downside is a more complicated one-time installation process for our users: +they have to add a dependency on your ruleset in their `MODULE.bazel` file. -We used to have all of the rules in the Bazel repository (under `//tools/build_rules` or `//tools/build_defs`). We still have a couple rules there, but we are working on moving the remaining rules out. +We used to have all of the rules in the Bazel repository (under +`//tools/build_rules` or `//tools/build_defs`). We still have a couple rules +there, but we are working on moving the remaining rules out. diff --git a/rules/errors/read-only-variable.mdx b/rules/errors/read-only-variable.mdx index ab4e4619..2bfde654 100644 --- a/rules/errors/read-only-variable.mdx +++ b/rules/errors/read-only-variable.mdx @@ -2,7 +2,11 @@ title: 'Error: Variable x is read only' --- -A global variable cannot be reassigned. It will always point to the same object. However, its content might change, if the value is mutable (for example, the content of a list). Local variables don't have this restriction. + + +A global variable cannot be reassigned. It will always point to the same object. +However, its content might change, if the value is mutable (for example, the +content of a list). Local variables don't have this restriction. ```python a = [1, 2] @@ -16,7 +20,8 @@ b = 4 # forbidden `ERROR: /path/ext.bzl:7:1: Variable b is read only` -You will get a similar error if you try to redefine a function (function overloading is not supported), for example: +You will get a similar error if you try to redefine a function (function +overloading is not supported), for example: ```python def foo(x): return x + 1 diff --git a/rules/faq.mdx b/rules/faq.mdx index 89481fa7..5321f0b7 100644 --- a/rules/faq.mdx +++ b/rules/faq.mdx @@ -2,48 +2,79 @@ title: 'Frequently Asked Questions' --- + + These are some common issues and questions with writing extensions. ## Why is my file not produced / my action never executed? Bazel only executes the actions needed to produce the *requested* output files. -- If the file you want has a label, you can request it directly: `bazel build //pkg:myfile.txt` +* If the file you want has a label, you can request it directly: + `bazel build //pkg:myfile.txt` -- If the file is in an output group of the target, you may need to specify that output group on the command line: `bazel build //pkg:mytarget --output_groups=foo` +* If the file is in an output group of the target, you may need to specify that + output group on the command line: + `bazel build //pkg:mytarget --output_groups=foo` -- If you want the file to be built automatically whenever your target is mentioned on the command line, add it to your rule's default outputs by returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. +* If you want the file to be built automatically whenever your target is + mentioned on the command line, add it to your rule's default outputs by + returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. See the [Rules page](/extending/rules#requesting-output-files) for more information. ## Why is my implementation function not executed? -Bazel analyzes only the targets that are requested for the build. You should either name the target on the command line, or something that depends on the target. +Bazel analyzes only the targets that are requested for the build. You should +either name the target on the command line, or something that depends on the +target. ## A file is missing when my action or binary is executed -Make sure that 1) the file has been registered as an input to the action or binary, and 2) the script or tool being executed is accessing the file using the correct path. +Make sure that 1) the file has been registered as an input to the action or +binary, and 2) the script or tool being executed is accessing the file using the +correct path. -For actions, you declare inputs by passing them to the `ctx.actions.*` function that creates the action. The proper path for the file can be obtained using [`File.path`](lib/File#path). +For actions, you declare inputs by passing them to the `ctx.actions.*` function +that creates the action. The proper path for the file can be obtained using +[`File.path`](lib/File#path). -For binaries (the executable outputs run by a `bazel run` or `bazel test` command), you declare inputs by including them in the [runfiles](/extending/rules#runfiles). Instead of using the `path` field, use [`File.short_path`](lib/File#short_path), which is file's path relative to the runfiles directory in which the binary executes. +For binaries (the executable outputs run by a `bazel run` or `bazel test` +command), you declare inputs by including them in the +[runfiles](/extending/rules#runfiles). Instead of using the `path` field, use +[`File.short_path`](lib/File#short_path), which is file's path relative to +the runfiles directory in which the binary executes. ## How can I control which files are built by `bazel build //pkg:mytarget`? -Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to [set the default outputs](/extending/rules#requesting-output-files). +Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to +[set the default outputs](/extending/rules#requesting-output-files). ## How can I run a program or do file I/O as part of my build? -A tool can be declared as a target, just like any other part of your build, and run during the execution phase to help build other targets. To create an action that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the tool as the `executable` parameter. +A tool can be declared as a target, just like any other part of your build, and +run during the execution phase to help build other targets. To create an action +that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the +tool as the `executable` parameter. -During the loading and analysis phases, a tool *cannot* run, nor can you perform file I/O. This means that tools and file contents (except the contents of BUILD and .bzl files) cannot affect how the target and action graphs get created. +During the loading and analysis phases, a tool *cannot* run, nor can you perform +file I/O. This means that tools and file contents (except the contents of BUILD +and .bzl files) cannot affect how the target and action graphs get created. ## What if I need to access the same structured data both before and during the execution phase? -You can format the structured data as a .bzl file. You can `load()` the file to access it during the loading and analysis phases. You can pass it as an input or runfile to actions and executables that need it during the execution phase. +You can format the structured data as a .bzl file. You can `load()` the file to +access it during the loading and analysis phases. You can pass it as an input or +runfile to actions and executables that need it during the execution phase. ## How should I document Starlark code? -For rules and rule attributes, you can pass a docstring literal (possibly triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper functions and macros, use a triple-quoted docstring literal following the format given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Rule implementation functions generally do not need their own docstring. +For rules and rule attributes, you can pass a docstring literal (possibly +triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper +functions and macros, use a triple-quoted docstring literal following the format +given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). +Rule implementation functions generally do not need their own docstring. -Using string literals in the expected places makes it easier for automated tooling to extract documentation. Feel free to use standard non-string comments wherever it may help the reader of your code. +Using string literals in the expected places makes it easier for automated +tooling to extract documentation. Feel free to use standard non-string comments +wherever it may help the reader of your code. diff --git a/rules/index.mdx b/rules/index.mdx index 44590f11..2a6c3eb7 100644 --- a/rules/index.mdx +++ b/rules/index.mdx @@ -2,7 +2,11 @@ title: 'Rules' --- -The Bazel ecosystem has a growing and evolving set of rules to support popular languages and packages. Much of Bazel's strength comes from the ability to [define new rules](/extending/concepts) that can be used by others. + + +The Bazel ecosystem has a growing and evolving set of rules to support popular +languages and packages. Much of Bazel's strength comes from the ability to +[define new rules](/extending/concepts) that can be used by others. This page describes the recommended, native, and non-native Bazel rules. @@ -10,57 +14,58 @@ This page describes the recommended, native, and non-native Bazel rules. Here is a selection of recommended rules: -- [Android](/docs/bazel-and-android) -- [C / C++](/docs/bazel-and-cpp) -- [Docker/OCI](https://github.com/bazel-contrib/rules_oci) -- [Go](https://github.com/bazelbuild/rules_go) -- [Haskell](https://github.com/tweag/rules_haskell) -- [Java](/docs/bazel-and-java) -- [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) -- [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) -- [Objective-C](/docs/bazel-and-apple) -- [Package building](https://github.com/bazelbuild/rules_pkg) -- [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) -- [Python](https://github.com/bazelbuild/rules_python) -- [Rust](https://github.com/bazelbuild/rules_rust) -- [Scala](https://github.com/bazelbuild/rules_scala) -- [Shell](/reference/be/shell) -- [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) - -The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains additional functions that can be useful when writing new rules and new macros. - -The rules above were reviewed and follow our [requirements for recommended rules](/community/recommended-rules). Contact the respective rule set's maintainers regarding issues and feature requests. - -To find more Bazel rules, use a search engine, take a look on [awesomebazel.com](https://awesomebazel.com/), or search on [GitHub](https://github.com/search?o=desc\&q=bazel+rules\&s=stars\&type=Repositories). +* [Android](/docs/bazel-and-android) +* [C / C++](/docs/bazel-and-cpp) +* [Docker/OCI](https://github.com/bazel-contrib/rules_oci) +* [Go](https://github.com/bazelbuild/rules_go) +* [Haskell](https://github.com/tweag/rules_haskell) +* [Java](/docs/bazel-and-java) +* [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) +* [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) +* [Objective-C](/docs/bazel-and-apple) +* [Package building](https://github.com/bazelbuild/rules_pkg) +* [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) +* [Python](https://github.com/bazelbuild/rules_python) +* [Rust](https://github.com/bazelbuild/rules_rust) +* [Scala](https://github.com/bazelbuild/rules_scala) +* [Shell](/reference/be/shell) +* [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) + +The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains +additional functions that can be useful when writing new rules and new +macros. + +The rules above were reviewed and follow our +[requirements for recommended rules](/community/recommended-rules). +Contact the respective rule set's maintainers regarding issues and feature +requests. + +To find more Bazel rules, use a search engine, take a look on +[awesomebazel.com](https://awesomebazel.com/), or search on +[GitHub](https://github.com/search?o=desc&q=bazel+rules&s=stars&type=Repositories). ## Native rules that do not apply to a specific programming language -Native rules are shipped with the Bazel binary, they are always available in BUILD files without a `load` statement. - -- Extra actions +Native rules are shipped with the Bazel binary, they are always available in +BUILD files without a `load` statement. +* Extra actions - [`extra_action`](/reference/be/extra-actions#extra_action) - [`action_listener`](/reference/be/extra-actions#action_listener) - -- General - +* General - [`filegroup`](/reference/be/general#filegroup) - [`genquery`](/reference/be/general#genquery) - [`test_suite`](/reference/be/general#test_suite) - [`alias`](/reference/be/general#alias) - [`config_setting`](/reference/be/general#config_setting) - [`genrule`](/reference/be/general#genrule) - -- Platform - +* Platform - [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) - [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) - [`platform`](/reference/be/platforms-and-toolchains#platform) - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - [`toolchain_type`](/reference/be/platforms-and-toolchains#toolchain_type) - -- Workspace - +* Workspace - [`bind`](/reference/be/workspace#bind) - [`local_repository`](/reference/be/workspace#local_repository) - [`new_local_repository`](/reference/be/workspace#new_local_repository) @@ -69,10 +74,10 @@ Native rules are shipped with the Bazel binary, they are always available in BUI ## Embedded non-native rules -Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from the `@bazel_tools` built-in external repository. - -- Repository rules +Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from +the `@bazel_tools` built-in external repository. +* Repository rules - [`git_repository`](/rules/lib/repo/git#git_repository) - [`http_archive`](/rules/lib/repo/http#http_archive) - [`http_file`](/rules/lib/repo/http#http_archive) diff --git a/rules/language.mdx b/rules/language.mdx index 9fad7248..13e33a4b 100644 --- a/rules/language.mdx +++ b/rules/language.mdx @@ -2,11 +2,18 @@ title: 'Starlark Language' --- -This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), formerly known as Skylark, the language used in Bazel. For a complete list of functions and types, see the [Bazel API reference](/rules/lib/overview). + + +{/* [TOC] */} + +This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), +formerly known as Skylark, the language used in Bazel. For a complete list of +functions and types, see the [Bazel API reference](/rules/lib/overview). For more information about the language, see [Starlark's GitHub repo](https://github.com/bazelbuild/starlark/). -For the authoritative specification of the Starlark syntax and behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). +For the authoritative specification of the Starlark syntax and +behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). ## Syntax @@ -26,32 +33,44 @@ def fizz_buzz(n): fizz_buzz(20) ``` -Starlark's semantics can differ from Python, but behavioral differences are rare, except for cases where Starlark raises an error. The following Python types are supported: +Starlark's semantics can differ from Python, but behavioral differences are +rare, except for cases where Starlark raises an error. The following Python +types are supported: -- [None](lib/globals#None) -- [bool](lib/bool) -- [dict](lib/dict) -- [tuple](lib/tuple) -- [function](lib/function) -- [int](lib/int) -- [list](lib/list) -- [string](lib/string) +* [None](lib/globals#None) +* [bool](lib/bool) +* [dict](lib/dict) +* [tuple](lib/tuple) +* [function](lib/function) +* [int](lib/int) +* [list](lib/list) +* [string](lib/string) ## Type annotations -**Experimental**. Type annotations are an experimental feature and may change at any time. Don't depend on it. It may be enabled in Bazel at HEAD by using the `--experimental_starlark_types` flag. +**Experimental**. Type annotations are an experimental feature and may change +at any time. Don't depend on it. It may be enabled in Bazel at HEAD +by using the `--experimental_starlark_types` flag. -Starlark in Bazel at HEAD is incrementally adding support for type annotations with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). +Starlark in Bazel at HEAD is incrementally adding support for type annotations +with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). -- Starlark type annotations are under active development. The progress is tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). +- Starlark type annotations are under active development. The progress is + tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). - The specification is incrementally extended: [starlark-with-types/spec.md](https://github.com/bazelbuild/starlark/blob/starlark-with-types/spec.md) - Initial proposal: [SEP-001 Bootstrapping Starlark types](https://docs.google.com/document/d/1Sid7EAbBd_w_T7D94Li_f_bK3zMTztFbzIMvcpzo1wY/edit?tab=t.0#heading=h.5mcn15i0e1ch) ## Mutability -Starlark favors immutability. Two mutable data structures are available: [lists](lib/list) and [dicts](lib/dict). Changes to mutable data-structures, such as appending a value to a list or deleting an entry in a dictionary are valid only for objects created in the current context. After a context finishes, its values become immutable. +Starlark favors immutability. Two mutable data structures are available: +[lists](lib/list) and [dicts](lib/dict). Changes to mutable +data-structures, such as appending a value to a list or deleting an entry in a +dictionary are valid only for objects created in the current context. After a +context finishes, its values become immutable. -This is because Bazel builds use parallel execution. During a build, each `.bzl` file and each `BUILD` file get their own execution context. Each rule is also analyzed in its own context. +This is because Bazel builds use parallel execution. During a build, each `.bzl` +file and each `BUILD` file get their own execution context. Each rule is also +analyzed in its own context. Let's go through an example with the file `foo.bzl`: @@ -65,9 +84,13 @@ def fct(): # declare a function fct() # execute the fct function ``` -Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s context. When `fct()` runs, it does so within the context of `foo.bzl`. After evaluation for `foo.bzl` completes, the environment contains an immutable entry, `var`, with the value `[5]`. +Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s +context. When `fct()` runs, it does so within the context of `foo.bzl`. After +evaluation for `foo.bzl` completes, the environment contains an immutable entry, +`var`, with the value `[5]`. -When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain immutable. For this reason, the following code in `bar.bzl` is illegal: +When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain +immutable. For this reason, the following code in `bar.bzl` is illegal: ```python # `bar.bzl` @@ -78,52 +101,66 @@ var.append(6) # runtime error, the list stored in var is frozen fct() # runtime error, fct() attempts to modify a frozen list ``` -Global variables defined in `bzl` files cannot be changed outside of the `bzl` file that defined them. Just like the above example using `bzl` files, values returned by rules are immutable. +Global variables defined in `bzl` files cannot be changed outside of the +`bzl` file that defined them. Just like the above example using `bzl` files, +values returned by rules are immutable. ## Differences between BUILD and .bzl files -`BUILD` files register targets via making calls to rules. `.bzl` files provide definitions for constants, rules, macros, and functions. +`BUILD` files register targets via making calls to rules. `.bzl` files provide +definitions for constants, rules, macros, and functions. -[Native functions](/reference/be/functions) and [native rules](/reference/be/overview#language-specific-native-rules) are global symbols in `BUILD` files. `bzl` files need to load them using the [`native` module](/rules/lib/toplevel/native). +[Native functions](/reference/be/functions) and [native rules]( +/reference/be/overview#language-specific-native-rules) are global symbols in +`BUILD` files. `bzl` files need to load them using the [`native` module]( +/rules/lib/toplevel/native). -There are two syntactic restrictions in `BUILD` files: 1) declaring functions is illegal, and 2) `*args` and `**kwargs` arguments are not allowed. +There are two syntactic restrictions in `BUILD` files: 1) declaring functions is +illegal, and 2) `*args` and `**kwargs` arguments are not allowed. ## Differences with Python -- Global variables are immutable. +* Global variables are immutable. -- `for` statements are not allowed at the top-level. Use them within functions instead. In `BUILD` files, you may use list comprehensions. +* `for` statements are not allowed at the top-level. Use them within functions + instead. In `BUILD` files, you may use list comprehensions. -- `if` statements are not allowed at the top-level. However, `if` expressions can be used: `first = data[0] if len(data) > 0 else None`. +* `if` statements are not allowed at the top-level. However, `if` expressions + can be used: `first = data[0] if len(data) > 0 else None`. -- Deterministic order for iterating through Dictionaries. +* Deterministic order for iterating through Dictionaries. -- Recursion is not allowed. +* Recursion is not allowed. -- Int type is limited to 32-bit signed integers. Overflows will throw an error. +* Int type is limited to 32-bit signed integers. Overflows will throw an error. -- Modifying a collection during iteration is an error. +* Modifying a collection during iteration is an error. -- Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are not defined across value types. In short: `5 < 'foo'` will throw an error and `5 == "5"` will return false. +* Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are +not defined across value types. In short: `5 < 'foo'` will throw an error and +`5 == "5"` will return false. -- In tuples, a trailing comma is valid only when the tuple is between parentheses — when you write `(1,)` instead of `1,`. +* In tuples, a trailing comma is valid only when the tuple is between + parentheses — when you write `(1,)` instead of `1,`. -- Dictionary literals cannot have duplicated keys. For example, this is an error: `{"a": 4, "b": 7, "a": 1}`. +* Dictionary literals cannot have duplicated keys. For example, this is an + error: `{"a": 4, "b": 7, "a": 1}`. -- Strings are represented with double-quotes (such as when you call [repr](lib/globals#repr)). +* Strings are represented with double-quotes (such as when you call + [repr](lib/globals#repr)). -- Strings aren't iterable. +* Strings aren't iterable. The following Python features are not supported: -- implicit string concatenation (use explicit `+` operator). -- Chained comparisons (such as `1 < x < 5`). -- `class` (see [`struct`](lib/struct#struct) function). -- `import` (see [`load`](/extending/concepts#loading-an-extension) statement). -- `while`, `yield`. -- float and set types. -- generators and generator expressions. -- `is` (use `==` instead). -- `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). -- `global`, `nonlocal`. -- most builtin functions, most methods. +* implicit string concatenation (use explicit `+` operator). +* Chained comparisons (such as `1 < x < 5`). +* `class` (see [`struct`](lib/struct#struct) function). +* `import` (see [`load`](/extending/concepts#loading-an-extension) statement). +* `while`, `yield`. +* float and set types. +* generators and generator expressions. +* `is` (use `==` instead). +* `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). +* `global`, `nonlocal`. +* most builtin functions, most methods. diff --git a/rules/legacy-macro-tutorial.mdx b/rules/legacy-macro-tutorial.mdx index 8322ec42..28b0fca8 100644 --- a/rules/legacy-macro-tutorial.mdx +++ b/rules/legacy-macro-tutorial.mdx @@ -2,11 +2,20 @@ title: 'Creating a Legacy Macro' --- -IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If you only need to support Bazel 8 or newer, we recommend using [symbolic macros](/extending/macros) instead; take a look at [Creating a Symbolic Macro](macro-tutorial). -Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a legacy macro that resizes an image. -Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. +IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If +you only need to support Bazel 8 or newer, we recommend using [symbolic +macros](/extending/macros) instead; take a look at [Creating a Symbolic +Macro](macro-tutorial). + +Imagine that you need to run a tool as part of your build. For example, you +may want to generate or preprocess a source file, or compress a binary. In this +tutorial, you are going to create a legacy macro that resizes an image. + +Macros are suitable for simple tasks. If you want to do anything more +complicated, for example add support for a new programming language, consider +creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -15,7 +24,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -25,7 +34,8 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, define a function in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, +define a function in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark def miniature(name, src, size = "100x100", **kwargs): @@ -38,20 +48,25 @@ def miniature(name, src, size = "100x100", **kwargs): srcs = [src], # Note that the line below will fail if `src` is not a filename string outs = ["small_" + src], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs ) ``` A few remarks: -- By convention, legacy macros have a `name` argument, just like rules. + * By convention, legacy macros have a `name` argument, just like rules. -- To document the behavior of a legacy macro, use [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. + * To document the behavior of a legacy macro, use + [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. -- To call a `genrule`, or any other native rule, prefix with `native.`. + * To call a `genrule`, or any other native rule, prefix with `native.`. -- Use `**kwargs` to forward the extra arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful, so that a user can use standard attributes like `visibility`, or `tags`. + * Use `**kwargs` to forward the extra arguments to the underlying `genrule` + (it works just like in + [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). + This is useful, so that a user can use standard attributes like + `visibility`, or `tags`. Now, use the macro from the `BUILD` file: @@ -70,6 +85,14 @@ cc_binary( ) ``` -And finally, a **warning note**: the macro assumes that `src` is a filename string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` works; but what happens if the `BUILD` file instead used `src = "//other/package:image.png"`, or even `src = select(...)`? - -You should make sure to declare such assumptions in your macro's documentation. Unfortunately, legacy macros, especially large ones, tend to be fragile because it can be hard to notice and document all such assumptions in your code – and, of course, some users of the macro won't read the documentation. We recommend, if possible, instead using [symbolic macros](/extending/macros), which have built-in checks on attribute types. +And finally, a **warning note**: the macro assumes that `src` is a filename +string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` +works; but what happens if the `BUILD` file instead used `src = +"//other/package:image.png"`, or even `src = select(...)`? + +You should make sure to declare such assumptions in your macro's documentation. +Unfortunately, legacy macros, especially large ones, tend to be fragile because +it can be hard to notice and document all such assumptions in your code – and, +of course, some users of the macro won't read the documentation. We recommend, +if possible, instead using [symbolic macros](/extending/macros), which have +built\-in checks on attribute types. diff --git a/rules/macro-tutorial.mdx b/rules/macro-tutorial.mdx index 4cf91617..2b3d8e4a 100644 --- a/rules/macro-tutorial.mdx +++ b/rules/macro-tutorial.mdx @@ -2,11 +2,20 @@ title: 'Creating a Symbolic Macro' --- -IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new macro system introduced in Bazel 8. If you need to support older Bazel versions, you will want to write a [legacy macro](/extending/legacy-macros) instead; take a look at [Creating a Legacy Macro](legacy-macro-tutorial). -Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a symbolic macro that resizes an image. -Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. +IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new +macro system introduced in Bazel 8. If you need to support older Bazel versions, +you will want to write a [legacy macro](/extending/legacy-macros) instead; take +a look at [Creating a Legacy Macro](legacy-macro-tutorial). + +Imagine that you need to run a tool as part of your build. For example, you +may want to generate or preprocess a source file, or compress a binary. In this +tutorial, you are going to create a symbolic macro that resizes an image. + +Macros are suitable for simple tasks. If you want to do anything more +complicated, for example add support for a new programming language, consider +creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -15,7 +24,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -25,7 +34,9 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, define an *implementation function* and a *macro declaration* in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, +define an *implementation function* and a *macro declaration* in a separate +`.bzl` file, and call the file `miniature.bzl`: ```starlark # Implementation function @@ -35,7 +46,7 @@ def _miniature_impl(name, visibility, src, size, **kwargs): visibility = visibility, srcs = [src], outs = [name + "_small_" + src.name], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs, ) @@ -73,13 +84,19 @@ miniature = macro( A few remarks: -- Symbolic macro implementation functions must have `name` and `visibility` parameters. They should used for the macro's main target. + * Symbolic macro implementation functions must have `name` and `visibility` + parameters. They should used for the macro's main target. -- To document the behavior of a symbolic macro, use `doc` parameters for `macro()` and its attributes. + * To document the behavior of a symbolic macro, use `doc` parameters for + `macro()` and its attributes. -- To call a `genrule`, or any other native rule, use `native.`. + * To call a `genrule`, or any other native rule, use `native.`. -- Use `**kwargs` to forward the extra inherited arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful so that a user can set standard attributes like `tags` or `testonly`. + * Use `**kwargs` to forward the extra inherited arguments to the underlying + `genrule` (it works just like in + [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). + This is useful so that a user can set standard attributes like `tags` or + `testonly`. Now, use the macro from the `BUILD` file: diff --git a/rules/performance.mdx b/rules/performance.mdx index edc0a2fd..c415cf14 100644 --- a/rules/performance.mdx +++ b/rules/performance.mdx @@ -2,20 +2,30 @@ title: 'Optimizing Performance' --- -When writing rules, the most common performance pitfall is to traverse or copy data that is accumulated from dependencies. When aggregated over the whole build, these operations can easily take O(N^2) time or space. To avoid this, it is crucial to understand how to use depsets effectively. -This can be hard to get right, so Bazel also provides a memory profiler that assists you in finding spots where you might have made a mistake. Be warned: The cost of writing an inefficient rule may not be evident until it is in widespread use. + +When writing rules, the most common performance pitfall is to traverse or copy +data that is accumulated from dependencies. When aggregated over the whole +build, these operations can easily take O(N^2) time or space. To avoid this, it +is crucial to understand how to use depsets effectively. + +This can be hard to get right, so Bazel also provides a memory profiler that +assists you in finding spots where you might have made a mistake. Be warned: +The cost of writing an inefficient rule may not be evident until it is in +widespread use. ## Use depsets -Whenever you are rolling up information from rule dependencies you should use [depsets](lib/depset). Only use plain lists or dicts to publish information local to the current rule. +Whenever you are rolling up information from rule dependencies you should use +[depsets](lib/depset). Only use plain lists or dicts to publish information +local to the current rule. A depset represents information as a nested graph which enables sharing. Consider the following graph: ``` -C -> B -> A +C -> B -> A D ---^ ``` @@ -37,9 +47,12 @@ c = ['c', 'b', 'a'] d = ['d', 'b', 'a'] ``` -Note that in this case `'a'` is mentioned four times! With larger graphs this problem will only get worse. +Note that in this case `'a'` is mentioned four times! With larger graphs this +problem will only get worse. -Here is an example of a rule implementation that uses depsets correctly to publish transitive information. Note that it is OK to publish rule-local information using lists if you want since this is not O(N^2). +Here is an example of a rule implementation that uses depsets correctly to +publish transitive information. Note that it is OK to publish rule-local +information using lists if you want since this is not O(N^2). ``` MyProvider = provider() @@ -61,13 +74,21 @@ See the [depset overview](/extending/depsets) page for more information. ### Avoid calling `depset.to_list()` -You can coerce a depset to a flat list using [`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) cost. If at all possible, avoid any flattening of depsets except for debugging purposes. +You can coerce a depset to a flat list using +[`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) +cost. If at all possible, avoid any flattening of depsets except for debugging +purposes. -A common misconception is that you can freely flatten depsets if you only do it at top-level targets, such as an `<xx>_binary` rule, since then the cost is not accumulated over each level of the build graph. But this is *still* O(N^2) when you build a set of targets with overlapping dependencies. This happens when building your tests `//foo/tests/...`, or when importing an IDE project. +A common misconception is that you can freely flatten depsets if you only do it +at top-level targets, such as an `_binary` rule, since then the cost is not +accumulated over each level of the build graph. But this is *still* O(N^2) when +you build a set of targets with overlapping dependencies. This happens when +building your tests `//foo/tests/...`, or when importing an IDE project. ### Reduce the number of calls to `depset` -Calling `depset` inside a loop is often a mistake. It can lead to depsets with very deep nesting, which perform poorly. For example: +Calling `depset` inside a loop is often a mistake. It can lead to depsets with +very deep nesting, which perform poorly. For example: ```python x = depset() @@ -76,7 +97,8 @@ for i in inputs: x = depset(transitive = [x, i.deps]) ``` -This code can be replaced easily. First, collect the transitive depsets and merge them all at once: +This code can be replaced easily. First, collect the transitive depsets and +merge them all at once: ```python transitive = [] @@ -95,19 +117,33 @@ x = depset(transitive = [i.deps for i in inputs]) ## Use ctx.actions.args() for command lines -When building command lines you should use [ctx.actions.args()](lib/Args). This defers expansion of any depsets to the execution phase. +When building command lines you should use [ctx.actions.args()](lib/Args). +This defers expansion of any depsets to the execution phase. -Apart from being strictly faster, this will reduce the memory consumption of your rules -- sometimes by 90% or more. +Apart from being strictly faster, this will reduce the memory consumption of +your rules -- sometimes by 90% or more. Here are some tricks: -- Pass depsets and lists directly as arguments, instead of flattening them yourself. They will get expanded by `ctx.actions.args()` for you. If you need any transformations on the depset contents, look at [ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. +* Pass depsets and lists directly as arguments, instead of flattening them +yourself. They will get expanded by `ctx.actions.args()` for you. +If you need any transformations on the depset contents, look at +[ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. -- Are you passing `File#path` as arguments? No need. Any [File](lib/File) is automatically turned into its [path](lib/File#path), deferred to expansion time. +* Are you passing `File#path` as arguments? No need. Any +[File](lib/File) is automatically turned into its +[path](lib/File#path), deferred to expansion time. -- Avoid constructing strings by concatenating them together. The best string argument is a constant as its memory will be shared between all instances of your rule. +* Avoid constructing strings by concatenating them together. +The best string argument is a constant as its memory will be shared between +all instances of your rule. -- If the args are too long for the command line an `ctx.actions.args()` object can be conditionally or unconditionally written to a param file using [`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is done behind the scenes when the action is executed. If you need to explicitly control the params file you can write it manually using [`ctx.actions.write`](lib/actions#write). +* If the args are too long for the command line an `ctx.actions.args()` object +can be conditionally or unconditionally written to a param file using +[`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is +done behind the scenes when the action is executed. If you need to explicitly +control the params file you can write it manually using +[`ctx.actions.write`](lib/actions#write). Example: @@ -118,15 +154,15 @@ def _impl(ctx): file = ctx.declare_file(...) files = depset(...) - # Bad, constructs a full string "--foo=<file path>" for each rule instance + # Bad, constructs a full string "--foo=" for each rule instance args.add("--foo=" + file.path) # Good, shares "--foo" among all rule instances, and defers file.path to later - # It will however pass ["--foo", <file path>] to the action command line, - # instead of ["--foo=<file_path>"] + # It will however pass ["--foo", ] to the action command line, + # instead of ["--foo="] args.add("--foo", file) - # Use format if you prefer ["--foo=<file path>"] to ["--foo", <file path>] + # Use format if you prefer ["--foo="] to ["--foo", ] args.add(file, format="--foo=%s") # Bad, makes a giant string of a whole depset @@ -142,7 +178,9 @@ def _to_short_path(f): ## Transitive action inputs should be depsets -When building an action using [ctx.actions.run](lib/actions?#run), do not forget that the `inputs` field accepts a depset. Use this whenever inputs are collected from dependencies transitively. +When building an action using [ctx.actions.run](lib/actions?#run), do not +forget that the `inputs` field accepts a depset. Use this whenever inputs are +collected from dependencies transitively. ``` inputs = depset(...) @@ -154,39 +192,54 @@ ctx.actions.run( ## Hanging -If Bazel appears to be hung, you can hit `Ctrl-\` or send Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread dump in the file `$(bazel info output_base)/server/jvm.out`. +If Bazel appears to be hung, you can hit Ctrl-\ or send +Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread +dump in the file `$(bazel info output_base)/server/jvm.out`. -Since you may not be able to run `bazel info` if bazel is hung, the `output_base` directory is usually the parent of the `bazel-<workspace>` symlink in your workspace directory. +Since you may not be able to run `bazel info` if bazel is hung, the +`output_base` directory is usually the parent of the `bazel-` +symlink in your workspace directory. ## Performance profiling -The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. +The [JSON trace profile](/advanced/performance/json-trace-profile) can be very +useful to quickly understand what Bazel spent time on during the invocation. -The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) flag may be used to capture Java Flight Recorder profiles of various kinds (cpu time, wall time, memory allocations and lock contention). +The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) +flag may be used to capture Java Flight Recorder profiles of various kinds +(cpu time, wall time, memory allocations and lock contention). -The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) flag may be used to write a pprof profile of CPU usage by all Starlark threads. +The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) +flag may be used to write a pprof profile of CPU usage by all Starlark threads. ## Memory profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. If there is a problem you can dump the heap to find the exact line of code that is causing the problem. +Bazel comes with a built-in memory profiler that can help you check your rule’s +memory use. If there is a problem you can dump the heap to find the +exact line of code that is causing the problem. ### Enabling memory tracking You must pass these two startup flags to *every* Bazel invocation: -``` -STARTUP_FLAGS=\ ---host_jvm_args=-javaagent:<path to java-allocation-instrumenter-3.3.4.jar> \ ---host_jvm_args=-DRULE_MEMORY_TRACKER=1 -``` + ``` + STARTUP_FLAGS=\ + --host_jvm_args=-javaagent: \

+ --host_jvm_args=-DRULE_MEMORY_TRACKER=1 + ``` +Note: You can download the allocation instrumenter jar file from [Maven Central +Repository][allocation-instrumenter-link]. -Note: You can download the allocation instrumenter jar file from [Maven Central Repository](https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4). +[allocation-instrumenter-link]: https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4 -These start the server in memory tracking mode. If you forget these for even one Bazel invocation the server will restart and you will have to start over. +These start the server in memory tracking mode. If you forget these for even +one Bazel invocation the server will restart and you will have to start over. ### Using the Memory Tracker -As an example, look at the target `foo` and see what it does. To only run the analysis and not run the build execution phase, add the `--nobuild` flag. +As an example, look at the target `foo` and see what it does. To only +run the analysis and not run the build execution phase, add the +`--nobuild` flag. ``` $ bazel $(STARTUP_FLAGS) build --nobuild //foo:foo @@ -196,14 +249,14 @@ Next, see how much memory the whole Bazel instance consumes: ``` $ bazel $(STARTUP_FLAGS) info used-heap-size-after-gc -> 2594MB +> 2594MB ``` Break it down by rule class by using `bazel dump --rules`: ``` $ bazel $(STARTUP_FLAGS) dump --rules -> +> RULE COUNT ACTIONS BYTES EACH genrule 33,762 33,801 291,538,824 8,635 @@ -218,14 +271,16 @@ _check_proto_library_deps 719 668 1,835,288 2,5 ... (more output) ``` -Look at where the memory is going by producing a `pprof` file using `bazel dump --skylark_memory`: +Look at where the memory is going by producing a `pprof` file +using `bazel dump --skylark_memory`: ``` $ bazel $(STARTUP_FLAGS) dump --skylark_memory=$HOME/prof.gz -> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz +> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz ``` -Use the `pprof` tool to investigate the heap. A good starting point is getting a flame graph by using `pprof -flame $HOME/prof.gz`. +Use the `pprof` tool to investigate the heap. A good starting point is +getting a flame graph by using `pprof -flame $HOME/prof.gz`. Get `pprof` from [https://github.com/google/pprof](https://github.com/google/pprof). @@ -233,13 +288,13 @@ Get a text dump of the hottest call sites annotated with lines: ``` $ pprof -text -lines $HOME/prof.gz -> +> flat flat% sum% cum cum% - 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library <native>:-1 - 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule <native>:-1 - 74.11MB 9.96% 44.80% 74.11MB 9.96% glob <native>:-1 - 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup <native>:-1 - 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test <native>:-1 + 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library :-1 + 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule :-1 + 74.11MB 9.96% 44.80% 74.11MB 9.96% glob :-1 + 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup :-1 + 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test :-1 26.55MB 3.57% 63.07% 26.55MB 3.57% _generate_foo_files /foo/tc/tc.bzl:491 26.01MB 3.50% 66.57% 26.01MB 3.50% _build_foo_impl /foo/build_test.bzl:78 22.01MB 2.96% 69.53% 22.01MB 2.96% _build_foo_impl /foo/build_test.bzl:73 diff --git a/rules/rules-tutorial.mdx b/rules/rules-tutorial.mdx index 30ad839d..7700f20c 100644 --- a/rules/rules-tutorial.mdx +++ b/rules/rules-tutorial.mdx @@ -2,9 +2,24 @@ title: 'Rules Tutorial' --- -[Starlark](https://github.com/bazelbuild/starlark) is a Python-like configuration language originally developed for use in Bazel and since adopted by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of Starlark properly known as the "Build Language", though it is often simply referred to as "Starlark", especially when emphasizing that a feature is expressed in the Build Language as opposed to being a built-in or "native" part of Bazel. Bazel augments the core language with numerous build-related functions such as `glob`, `genrule`, `java_binary`, and so on. -See the [Bazel](/start/) and [Starlark](/extending/concepts) documentation for more details, and the [Rules SIG template](https://github.com/bazel-contrib/rules-template) as a starting point for new rulesets. + +{/* [TOC] */} + +[Starlark](https://github.com/bazelbuild/starlark) is a Python-like +configuration language originally developed for use in Bazel and since adopted +by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of +Starlark properly known as the "Build Language", though it is often simply +referred to as "Starlark", especially when emphasizing that a feature is +expressed in the Build Language as opposed to being a built-in or "native" part +of Bazel. Bazel augments the core language with numerous build-related functions +such as `glob`, `genrule`, `java_binary`, and so on. + +See the +[Bazel](/start/) and [Starlark](/extending/concepts) documentation for +more details, and the +[Rules SIG template](https://github.com/bazel-contrib/rules-template) as a +starting point for new rulesets. ## The empty rule @@ -19,7 +34,10 @@ foo_binary = rule( ) ``` -When you call the [`rule`](lib/globals#rule) function, you must define a callback function. The logic will go there, but you can leave the function empty for now. The [`ctx`](lib/ctx) argument provides information about the target. +When you call the [`rule`](lib/globals#rule) function, you +must define a callback function. The logic will go there, but you +can leave the function empty for now. The [`ctx`](lib/ctx) argument +provides information about the target. You can load the rule and use it from a `BUILD` file. @@ -40,7 +58,9 @@ INFO: Found 1 target... Target //:bin up-to-date (nothing to build) ``` -Even though the rule does nothing, it already behaves like other rules: it has a mandatory name, it supports common attributes like `visibility`, `testonly`, and `tags`. +Even though the rule does nothing, it already behaves like other rules: it has a +mandatory name, it supports common attributes like `visibility`, `testonly`, and +`tags`. ## Evaluation model @@ -69,7 +89,10 @@ foo_binary(name = "bin1") foo_binary(name = "bin2") ``` -[`ctx.label`](lib/ctx#label) corresponds to the label of the target being analyzed. The `ctx` object has many useful fields and methods; you can find an exhaustive list in the [API reference](lib/ctx). +[`ctx.label`](lib/ctx#label) +corresponds to the label of the target being analyzed. The `ctx` object has +many useful fields and methods; you can find an exhaustive list in the +[API reference](lib/ctx). Query the code: @@ -83,10 +106,15 @@ DEBUG: /usr/home/bazel-codelab/BUILD:2:1: BUILD file Make a few observations: -- "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, Bazel evaluates all the files it loads. If multiple `BUILD` files are loading foo.bzl, you would see only one occurrence of "bzl file evaluation" because Bazel caches the result of the evaluation. -- The callback function `_foo_binary_impl` is not called. Bazel query loads `BUILD` files, but doesn't analyze targets. +* "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, + Bazel evaluates all the files it loads. If multiple `BUILD` files are loading + foo.bzl, you would see only one occurrence of "bzl file evaluation" because + Bazel caches the result of the evaluation. +* The callback function `_foo_binary_impl` is not called. Bazel query loads + `BUILD` files, but doesn't analyze targets. -To analyze the targets, use the [`cquery`](/query/cquery) ("configured query") or the `build` command: +To analyze the targets, use the [`cquery`](/query/cquery) ("configured +query") or the `build` command: ``` $ bazel build :all @@ -98,11 +126,15 @@ INFO: Found 2 targets... As you can see, `_foo_binary_impl` is now called twice - once for each target. -Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, because the evaluation of `foo.bzl` is cached after the call to `bazel query`. Bazel only emits `print` statements when they are actually executed. +Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, +because the evaluation of `foo.bzl` is cached after the call to `bazel query`. +Bazel only emits `print` statements when they are actually executed. ## Creating a file -To make your rule more useful, update it to generate a file. First, declare the file and give it a name. In this example, create a file with the same name as the target: +To make your rule more useful, update it to generate a file. First, declare the +file and give it a name. In this example, create a file with the same name as +the target: ```python ctx.actions.declare_file(ctx.label.name) @@ -115,7 +147,9 @@ The following files have no generating action: bin2 ``` -Whenever you declare a file, you have to tell Bazel how to generate it by creating an action. Use [`ctx.actions.write`](lib/actions#write), to create a file with the given content. +Whenever you declare a file, you have to tell Bazel how to generate it by +creating an action. Use [`ctx.actions.write`](lib/actions#write), +to create a file with the given content. ```python def _foo_binary_impl(ctx): @@ -133,7 +167,11 @@ $ bazel build bin1 Target //:bin1 up-to-date (nothing to build) ``` -The `ctx.actions.write` function registered an action, which taught Bazel how to generate the file. But Bazel won't create the file until it is actually requested. So the last thing to do is tell Bazel that the file is an output of the rule, and not a temporary file used within the rule implementation. +The `ctx.actions.write` function registered an action, which taught Bazel +how to generate the file. But Bazel won't create the file until it is +actually requested. So the last thing to do is tell Bazel that the file +is an output of the rule, and not a temporary file used within the rule +implementation. ```python def _foo_binary_impl(ctx): @@ -145,7 +183,8 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Look at the `DefaultInfo` and `depset` functions later. For now, assume that the last line is the way to choose the outputs of a rule. +Look at the `DefaultInfo` and `depset` functions later. For now, +assume that the last line is the way to choose the outputs of a rule. Now, run Bazel: @@ -163,7 +202,8 @@ You have successfully generated a file! ## Attributes -To make the rule more useful, add new attributes using [the `attr` module](lib/attr) and update the rule definition. +To make the rule more useful, add new attributes using +[the `attr` module](lib/attr) and update the rule definition. Add a string attribute called `username`: @@ -185,7 +225,8 @@ foo_binary( ) ``` -To access the value in the callback function, use `ctx.attr.username`. For example: +To access the value in the callback function, use `ctx.attr.username`. For +example: ```python def _foo_binary_impl(ctx): @@ -197,23 +238,38 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Note that you can make the attribute mandatory or set a default value. Look at the documentation of [`attr.string`](lib/attr#string). You may also use other types of attributes, such as [boolean](lib/attr#bool) or [list of integers](lib/attr#int_list). +Note that you can make the attribute mandatory or set a default value. Look at +the documentation of [`attr.string`](lib/attr#string). +You may also use other types of attributes, such as [boolean](lib/attr#bool) +or [list of integers](lib/attr#int_list). ## Dependencies -Dependency attributes, such as [`attr.label`](lib/attr#label) and [`attr.label_list`](lib/attr#label_list), declare a dependency from the target that owns the attribute to the target whose label appears in the attribute's value. This kind of attribute forms the basis of the target graph. +Dependency attributes, such as [`attr.label`](lib/attr#label) +and [`attr.label_list`](lib/attr#label_list), +declare a dependency from the target that owns the attribute to the target whose +label appears in the attribute's value. This kind of attribute forms the basis +of the target graph. -In the `BUILD` file, the target label appears as a string object, such as `//pkg:name`. In the implementation function, the target will be accessible as a [`Target`](lib/Target) object. For example, view the files returned by the target using [`Target.files`](lib/Target#modules.Target.files). +In the `BUILD` file, the target label appears as a string object, such as +`//pkg:name`. In the implementation function, the target will be accessible as a +[`Target`](lib/Target) object. For example, view the files returned +by the target using [`Target.files`](lib/Target#modules.Target.files). ### Multiple files -By default, only targets created by rules may appear as dependencies (such as a `foo_library()` target). If you want the attribute to accept targets that are input files (such as source files in the repository), you can do it with `allow_files` and specify the list of accepted file extensions (or `True` to allow any file extension): +By default, only targets created by rules may appear as dependencies (such as a +`foo_library()` target). If you want the attribute to accept targets that are +input files (such as source files in the repository), you can do it with +`allow_files` and specify the list of accepted file extensions (or `True` to +allow any file extension): ```python "srcs": attr.label_list(allow_files = [".java"]), ``` -The list of files can be accessed with `ctx.files.<attribute name>`. For example, the list of files in the `srcs` attribute can be accessed through +The list of files can be accessed with `ctx.files.`. For +example, the list of files in the `srcs` attribute can be accessed through ```python ctx.files.srcs @@ -227,7 +283,7 @@ If you need only one file, use `allow_single_file`: "src": attr.label(allow_single_file = [".java"]) ``` -This file is then accessible under `ctx.file.<attribute name>`: +This file is then accessible under `ctx.file.`: ```python ctx.file.src @@ -235,9 +291,17 @@ ctx.file.src ## Create a file with a template -You can create a rule that generates a .cc file based on a template. Also, you can use `ctx.actions.write` to output a string constructed in the rule implementation function, but this has two problems. First, as the template gets bigger, it becomes more memory efficient to put it in a separate file and avoid constructing large strings during the analysis phase. Second, using a separate file is more convenient for the user. Instead, use [`ctx.actions.expand_template`](lib/actions#expand_template), which performs substitutions on a template file. +You can create a rule that generates a .cc file based on a template. Also, you +can use `ctx.actions.write` to output a string constructed in the rule +implementation function, but this has two problems. First, as the template gets +bigger, it becomes more memory efficient to put it in a separate file and avoid +constructing large strings during the analysis phase. Second, using a separate +file is more convenient for the user. Instead, use +[`ctx.actions.expand_template`](lib/actions#expand_template), +which performs substitutions on a template file. -Create a `template` attribute to declare a dependency on the template file: +Create a `template` attribute to declare a dependency on the template +file: ```python def _hello_world_impl(ctx): @@ -276,7 +340,8 @@ cc_binary( ) ``` -If you don't want to expose the template to the end-user and always use the same one, you can set a default value and make the attribute private: +If you don't want to expose the template to the end-user and always use the +same one, you can set a default value and make the attribute private: ```python "_template": attr.label( @@ -285,7 +350,11 @@ If you don't want to expose the template to the end-user and always use the same ), ``` -Attributes that start with an underscore are private and cannot be set in a `BUILD` file. The template is now an *implicit dependency*: Every `hello_world` target has a dependency on this file. Don't forget to make this file visible to other packages by updating the `BUILD` file and using [`exports_files`](/reference/be/functions#exports_files): +Attributes that start with an underscore are private and cannot be set in a +`BUILD` file. The template is now an _implicit dependency_: Every `hello_world` +target has a dependency on this file. Don't forget to make this file visible +to other packages by updating the `BUILD` file and using +[`exports_files`](/reference/be/functions#exports_files): ```python exports_files(["file.cc.tpl"]) @@ -293,6 +362,7 @@ exports_files(["file.cc.tpl"]) ## Going further -- Take a look at the [reference documentation for rules](/extending/rules#contents). -- Get familiar with [depsets](/extending/depsets). -- Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) which includes additional examples of rules. +* Take a look at the [reference documentation for rules](/extending/rules#contents). +* Get familiar with [depsets](/extending/depsets). +* Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) + which includes additional examples of rules. diff --git a/rules/testing.mdx b/rules/testing.mdx index 4a191c15..2996e08c 100644 --- a/rules/testing.mdx +++ b/rules/testing.mdx @@ -2,23 +2,47 @@ title: 'Testing' --- -There are several different approaches to testing Starlark code in Bazel. This page gathers the current best practices and frameworks by use case. -## Testing rules - -[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) for checking the analysis-time behavior of rules, such as their actions and providers. Such tests are called "analysis tests" and are currently the best option for testing the inner workings of rules. -Some caveats: +There are several different approaches to testing Starlark code in Bazel. This +page gathers the current best practices and frameworks by use case. -- Test assertions occur within the build, not a separate test runner process. Targets that are created by the test must be named such that they do not collide with targets from other tests or from the build. An error that occurs during the test is seen by Bazel as a build breakage rather than a test failure. - -- It requires a fair amount of boilerplate to set up the rules under test and the rules containing test assertions. This boilerplate may seem daunting at first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros are evaluated and targets generated during the loading phase, while rule implementation functions don't run until later, during the analysis phase. +## Testing rules -- Analysis tests are intended to be fairly small and lightweight. Certain features of the analysis testing framework are restricted to verifying targets with a maximum number of transitive dependencies (currently 500). This is due to performance implications of using these features with larger tests. +[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called +[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) +for checking the analysis-time behavior of rules, such as their actions and +providers. Such tests are called "analysis tests" and are currently the best +option for testing the inner workings of rules. -The basic principle is to define a testing rule that depends on the rule-under-test. This gives the testing rule access to the rule-under-test's providers. +Some caveats: -The testing rule's implementation function carries out assertions. If there are any failures, these are not raised immediately by calling `fail()` (which would trigger an analysis-time build error), but rather by storing the errors in a generated script that fails at test execution time. +* Test assertions occur within the build, not a separate test runner process. + Targets that are created by the test must be named such that they do not + collide with targets from other tests or from the build. An error that + occurs during the test is seen by Bazel as a build breakage rather than a + test failure. + +* It requires a fair amount of boilerplate to set up the rules under test and + the rules containing test assertions. This boilerplate may seem daunting at + first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros + are evaluated and targets generated during the loading phase, while rule + implementation functions don't run until later, during the analysis phase. + +* Analysis tests are intended to be fairly small and lightweight. Certain + features of the analysis testing framework are restricted to verifying + targets with a maximum number of transitive dependencies (currently 500). + This is due to performance implications of using these features with larger + tests. + +The basic principle is to define a testing rule that depends on the +rule-under-test. This gives the testing rule access to the rule-under-test's +providers. + +The testing rule's implementation function carries out assertions. If there are +any failures, these are not raised immediately by calling `fail()` (which would +trigger an analysis-time build error), but rather by storing the errors in a +generated script that fails at test execution time. See below for a minimal toy example, followed by an example that checks actions. @@ -45,6 +69,7 @@ myrule = rule( `//mypkg/myrules_test.bzl`: + ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest") load(":myrules.bzl", "myrule", "MyInfo") @@ -114,31 +139,47 @@ myrules_test_suite(name = "myrules_test") The test can be run with `bazel test //mypkg:myrules_test`. -Aside from the initial `load()` statements, there are two main parts to the file: +Aside from the initial `load()` statements, there are two main parts to the +file: -- The tests themselves, each of which consists of 1) an analysis-time implementation function for the testing rule, 2) a declaration of the testing rule via `analysistest.make()`, and 3) a loading-time function (macro) for declaring the rule-under-test (and its dependencies) and testing rule. If the assertions do not change between test cases, 1) and 2) may be shared by multiple test cases. +* The tests themselves, each of which consists of 1) an analysis-time + implementation function for the testing rule, 2) a declaration of the + testing rule via `analysistest.make()`, and 3) a loading-time function + (macro) for declaring the rule-under-test (and its dependencies) and testing + rule. If the assertions do not change between test cases, 1) and 2) may be + shared by multiple test cases. -- The test suite function, which calls the loading-time functions for each test, and declares a `test_suite` target bundling all tests together. +* The test suite function, which calls the loading-time functions for each + test, and declares a `test_suite` target bundling all tests together. -For consistency, follow the recommended naming convention: Let `foo` stand for the part of the test name that describes what the test is checking (`provider_contents` in the above example). For example, a JUnit test method would be named `testFoo`. +For consistency, follow the recommended naming convention: Let `foo` stand for +the part of the test name that describes what the test is checking +(`provider_contents` in the above example). For example, a JUnit test method +would be named `testFoo`. Then: -- the macro which generates the test and target under test should should be named `_test_foo` (`_test_provider_contents`) +* the macro which generates the test and target under test should should be + named `_test_foo` (`_test_provider_contents`) -- its test rule type should be named `foo_test` (`provider_contents_test`) +* its test rule type should be named `foo_test` (`provider_contents_test`) -- the label of the target of this rule type should be `foo_test` (`provider_contents_test`) +* the label of the target of this rule type should be `foo_test` + (`provider_contents_test`) -- the implementation function for the testing rule should be named `_foo_test_impl` (`_provider_contents_test_impl`) +* the implementation function for the testing rule should be named + `_foo_test_impl` (`_provider_contents_test_impl`) -- the labels of the targets of the rules under test and their dependencies should be prefixed with `foo_` (`provider_contents_`) +* the labels of the targets of the rules under test and their dependencies + should be prefixed with `foo_` (`provider_contents_`) -Note that the labels of all targets can conflict with other labels in the same BUILD package, so it's helpful to use a unique name for the test. +Note that the labels of all targets can conflict with other labels in the same +BUILD package, so it's helpful to use a unique name for the test. ### Failure testing -It may be useful to verify that a rule fails given certain inputs or in certain state. This can be done using the analysis test framework: +It may be useful to verify that a rule fails given certain inputs or in certain +state. This can be done using the analysis test framework: The test rule created with `analysistest.make` should specify `expect_failure`: @@ -149,7 +190,8 @@ failure_testing_test = analysistest.make( ) ``` -The test rule implementation should make assertions on the nature of the failure that took place (specifically, the failure message): +The test rule implementation should make assertions on the nature of the failure +that took place (specifically, the failure message): ```python def _failure_testing_test_impl(ctx): @@ -158,7 +200,11 @@ def _failure_testing_test_impl(ctx): return analysistest.end(env) ``` -Also make sure that your target under test is specifically tagged 'manual'. Without this, building all targets in your package using `:all` will result in a build of the intentionally-failing target and will exhibit a build failure. With 'manual', your target under test will build only if explicitly specified, or as a dependency of a non-manual target (such as your test rule): +Also make sure that your target under test is specifically tagged 'manual'. +Without this, building all targets in your package using `:all` will result in a +build of the intentionally-failing target and will exhibit a build failure. With +'manual', your target under test will build only if explicitly specified, or as +a dependency of a non-manual target (such as your test rule): ```python def _test_failure(): @@ -173,7 +219,9 @@ def _test_failure(): ### Verifying registered actions -You may want to write tests which make assertions about the actions that your rule registers, for example, using `ctx.actions.run()`. This can be done in your analysis test rule implementation function. An example: +You may want to write tests which make assertions about the actions that your +rule registers, for example, using `ctx.actions.run()`. This can be done in your +analysis test rule implementation function. An example: ```python def _inspect_actions_test_impl(ctx): @@ -188,11 +236,14 @@ def _inspect_actions_test_impl(ctx): return analysistest.end(env) ``` -Note that `analysistest.target_actions(env)` returns a list of [`Action`](lib/Action) objects which represent actions registered by the target under test. +Note that `analysistest.target_actions(env)` returns a list of +[`Action`](lib/Action) objects which represent actions registered by the +target under test. ### Verifying rule behavior under different flags -You may want to verify your real rule behaves a certain way given certain build flags. For example, your rule may behave differently if a user specifies: +You may want to verify your real rule behaves a certain way given certain build +flags. For example, your rule may behave differently if a user specifies: ```shell bazel build //mypkg:real_target -c opt @@ -204,15 +255,20 @@ versus bazel build //mypkg:real_target -c dbg ``` -At first glance, this could be done by testing the target under test using the desired build flags: +At first glance, this could be done by testing the target under test using the +desired build flags: ```shell bazel test //mypkg:myrules_test -c opt ``` -But then it becomes impossible for your test suite to simultaneously contain a test which verifies the rule behavior under `-c opt` and another test which verifies the rule behavior under `-c dbg`. Both tests would not be able to run in the same build! +But then it becomes impossible for your test suite to simultaneously contain a +test which verifies the rule behavior under `-c opt` and another test which +verifies the rule behavior under `-c dbg`. Both tests would not be able to run +in the same build! -This can be solved by specifying the desired build flags when defining the test rule: +This can be solved by specifying the desired build flags when defining the test +rule: ```python myrule_c_opt_test = analysistest.make( @@ -223,21 +279,33 @@ myrule_c_opt_test = analysistest.make( ) ``` -Normally, a target under test is analyzed given the current build flags. Specifying `config_settings` overrides the values of the specified command line options. (Any unspecified options will retain their values from the actual command line). +Normally, a target under test is analyzed given the current build flags. +Specifying `config_settings` overrides the values of the specified command line +options. (Any unspecified options will retain their values from the actual +command line). + +In the specified `config_settings` dictionary, command line flags must be +prefixed with a special placeholder value `//command_line_option:`, as is shown +above. -In the specified `config_settings` dictionary, command line flags must be prefixed with a special placeholder value `//command_line_option:`, as is shown above. ## Validating artifacts The main ways to check that your generated files are correct are: -- You can write a test script in shell, Python, or another language, and create a target of the appropriate `*_test` rule type. +* You can write a test script in shell, Python, or another language, and + create a target of the appropriate `*_test` rule type. -- You can use a specialized rule for the kind of test you want to perform. +* You can use a specialized rule for the kind of test you want to perform. ### Using a test target -The most straightforward way to validate an artifact is to write a script and add a `*_test` target to your BUILD file. The specific artifacts you want to check should be data dependencies of this target. If your validation logic is reusable for multiple tests, it should be a script that takes command line arguments that are controlled by the test target's `args` attribute. Here's an example that validates that the output of `myrule` from above is `"abc"`. +The most straightforward way to validate an artifact is to write a script and +add a `*_test` target to your BUILD file. The specific artifacts you want to +check should be data dependencies of this target. If your validation logic is +reusable for multiple tests, it should be a script that takes command line +arguments that are controlled by the test target's `args` attribute. Here's an +example that validates that the output of `myrule` from above is `"abc"`. `//mypkg/myrule_validator.sh`: @@ -273,7 +341,12 @@ sh_test( ### Using a custom rule -A more complicated alternative is to write the shell script as a template that gets instantiated by a new rule. This involves more indirection and Starlark logic, but leads to cleaner BUILD files. As a side-benefit, any argument preprocessing can be done in Starlark instead of the script, and the script is slightly more self-documenting since it uses symbolic placeholders (for substitutions) instead of numeric ones (for arguments). +A more complicated alternative is to write the shell script as a template that +gets instantiated by a new rule. This involves more indirection and Starlark +logic, but leads to cleaner BUILD files. As a side-benefit, any argument +preprocessing can be done in Starlark instead of the script, and the script is +slightly more self-documenting since it uses symbolic placeholders (for +substitutions) instead of numeric ones (for arguments). `//mypkg/myrule_validator.sh.template`: @@ -344,11 +417,18 @@ myrule_validation_test( ) ``` -Alternatively, instead of using a template expansion action, you could have inlined the template into the .bzl file as a string and expanded it during the analysis phase using the `str.format` method or `%`-formatting. +Alternatively, instead of using a template expansion action, you could have +inlined the template into the .bzl file as a string and expanded it during the +analysis phase using the `str.format` method or `%`-formatting. ## Testing Starlark utilities -[Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) framework can be used to test utility functions (that is, functions that are neither macros nor rule implementations). Instead of using `unittest.bzl`'s `analysistest` library, `unittest` may be used. For such test suites, the convenience function `unittest.suite()` can be used to reduce boilerplate. +[Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) +framework can be used to test utility functions (that is, functions that are +neither macros nor rule implementations). Instead of using `unittest.bzl`'s +`analysistest` library, `unittest` may be used. For such test suites, the +convenience function `unittest.suite()` can be used to reduce boilerplate. `//mypkg/myhelpers.bzl`: @@ -359,6 +439,7 @@ def myhelper(): `//mypkg/myhelpers_test.bzl`: + ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":myhelpers.bzl", "myhelper") diff --git a/rules/verbs-tutorial.mdx b/rules/verbs-tutorial.mdx index 4cd6c990..db7757e1 100644 --- a/rules/verbs-tutorial.mdx +++ b/rules/verbs-tutorial.mdx @@ -2,17 +2,29 @@ title: 'Using Macros to Create Custom Verbs' --- -Day-to-day interaction with Bazel happens primarily through a few commands: `build`, `test`, and `run`. At times, though, these can feel limited: you may want to push packages to a repository, publish documentation for end-users, or deploy an application with Kubernetes. But Bazel doesn't have a `publish` or `deploy` command – where do these actions fit in? -## The bazel run command -Bazel's focus on hermeticity, reproducibility, and incrementality means the `build` and `test` commands aren't helpful for the above tasks. These actions may run in a sandbox, with limited network access, and aren't guaranteed to be re-run with every `bazel build`. +Day-to-day interaction with Bazel happens primarily through a few commands: +`build`, `test`, and `run`. At times, though, these can feel limited: you may +want to push packages to a repository, publish documentation for end-users, or +deploy an application with Kubernetes. But Bazel doesn't have a `publish` or +`deploy` command – where do these actions fit in? + +## The bazel run command -Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have side effects. Bazel users are accustomed to rules that create executables, and rule authors can follow a common set of patterns to extend this to "custom verbs". +Bazel's focus on hermeticity, reproducibility, and incrementality means the +`build` and `test` commands aren't helpful for the above tasks. These actions +may run in a sandbox, with limited network access, and aren't guaranteed to be +re-run with every `bazel build`. -### In the wild: rules\_k8s +Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have +side effects. Bazel users are accustomed to rules that create executables, and +rule authors can follow a common set of patterns to extend this to +"custom verbs". -For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), the Kubernetes rules for Bazel. Suppose you have the following target: +### In the wild: rules_k8s +For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), +the Kubernetes rules for Bazel. Suppose you have the following target: ```python # BUILD file in //application/k8s @@ -24,21 +36,51 @@ k8s_object( ) ``` -The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a standard Kubernetes YAML file when `bazel build` is used on the `staging` target. However, the additional targets are also created by the `k8s_object` macro with names like `staging.apply` and `:staging.delete`. These build scripts to perform those actions, and when executed with `bazel run staging.apply`, these behave like our own `bazel k8s-apply` or `bazel k8s-delete` commands. - -### Another example: ts\_api\_guardian\_test - -This pattern can also be seen in the Angular project. The [`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) produces two targets. The first is a standard `nodejs_test` target which compares some generated output against a "golden" file (that is, a file containing the expected output). This can be built and run with a normal `bazel test` invocation. In `angular-cli`, you can run [one such target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) with `bazel test //etc/api:angular_devkit_core_api`. - -Over time, this golden file may need to be updated for legitimate reasons. Updating this manually is tedious and error-prone, so this macro also provides a `nodejs_binary` target that updates the golden file, instead of comparing against it. Effectively, the same test script can be written to run in "verify" or "accept" mode, based on how it's invoked. This follows the same pattern you've learned already: there is no native `bazel test-accept` command, but the same effect can be achieved with `bazel run //etc/api:angular_devkit_core_api.accept`. - -This pattern can be quite powerful, and turns out to be quite common once you learn to recognize it. +The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a +standard Kubernetes YAML file when `bazel build` is used on the `staging` +target. However, the additional targets are also created by the `k8s_object` +macro with names like `staging.apply` and `:staging.delete`. These build +scripts to perform those actions, and when executed with `bazel run +staging.apply`, these behave like our own `bazel k8s-apply` or `bazel +k8s-delete` commands. + +### Another example: ts_api_guardian_test + +This pattern can also be seen in the Angular project. The +[`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) +produces two targets. The first is a standard `nodejs_test` target which compares +some generated output against a "golden" file (that is, a file containing the +expected output). This can be built and run with a normal `bazel +test` invocation. In `angular-cli`, you can run [one such +target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) +with `bazel test //etc/api:angular_devkit_core_api`. + +Over time, this golden file may need to be updated for legitimate reasons. +Updating this manually is tedious and error-prone, so this macro also provides +a `nodejs_binary` target that updates the golden file, instead of comparing +against it. Effectively, the same test script can be written to run in "verify" +or "accept" mode, based on how it's invoked. This follows the same pattern +you've learned already: there is no native `bazel test-accept` command, but the +same effect can be achieved with +`bazel run //etc/api:angular_devkit_core_api.accept`. + +This pattern can be quite powerful, and turns out to be quite common once you +learn to recognize it. ## Adapting your own rules -[Macros](/extending/macros) are the heart of this pattern. Macros are used like rules, but they can create several targets. Typically, they will create a target with the specified name which performs the primary build action: perhaps it builds a normal binary, a Docker image, or an archive of source code. In this pattern, additional targets are created to produce scripts performing side effects based on the output of the primary target, like publishing the resulting binary or updating the expected test output. +[Macros](/extending/macros) are the heart of this pattern. Macros are used like +rules, but they can create several targets. Typically, they will create a +target with the specified name which performs the primary build action: perhaps +it builds a normal binary, a Docker image, or an archive of source code. In +this pattern, additional targets are created to produce scripts performing side +effects based on the output of the primary target, like publishing the +resulting binary or updating the expected test output. -To illustrate this, wrap an imaginary rule that generates a website with [Sphinx](https://www.sphinx-doc.org) with a macro to create an additional target that allows the user to publish it when ready. Consider the following existing rule for generating a website with Sphinx: +To illustrate this, wrap an imaginary rule that generates a website with +[Sphinx](https://www.sphinx-doc.org) with a macro to create an additional +target that allows the user to publish it when ready. Consider the following +existing rule for generating a website with Sphinx: ```python _sphinx_site = rule( @@ -47,7 +89,8 @@ _sphinx_site = rule( ) ``` -Next, consider a rule like the following, which builds a script that, when run, publishes the generated pages: +Next, consider a rule like the following, which builds a script that, when run, +publishes the generated pages: ```python _sphinx_publisher = rule( @@ -63,7 +106,8 @@ _sphinx_publisher = rule( ) ``` -Finally, define the following symbolic macro (available in Bazel 8 or newer) to create targets for both of the above rules together: +Finally, define the following symbolic macro (available in Bazel 8 or newer) to +create targets for both of the above rules together: ```starlark def _sphinx_site_impl(name, visibility, srcs, **kwargs): @@ -84,7 +128,8 @@ sphinx_site = macro( ) ``` -Or, if you need to support Bazel releases older than Bazel 8, you would instead define a legacy macro: +Or, if you need to support Bazel releases older than Bazel 8, you would instead +define a legacy macro: ```starlark def sphinx_site(name, srcs = [], **kwargs): @@ -95,7 +140,8 @@ def sphinx_site(name, srcs = [], **kwargs): _sphinx_publisher(name = "%s.publish" % name, site = name, **kwargs) ``` -In the `BUILD` files, use the macro as though it just creates the primary target: +In the `BUILD` files, use the macro as though it just creates the primary +target: ```python sphinx_site( @@ -104,8 +150,28 @@ sphinx_site( ) ``` -In this example, a "docs" target is created, just as though the macro were a standard, single Bazel rule. When built, the rule generates some configuration and runs Sphinx to produce an HTML site, ready for manual inspection. However, an additional "docs.publish" target is also created, which builds a script for publishing the site. Once you check the output of the primary target, you can use `bazel run :docs.publish` to publish it for public consumption, just like an imaginary `bazel publish` command. - -It's not immediately obvious what the implementation of the `_sphinx_publisher` rule might look like. Often, actions like this write a *launcher* shell script. This method typically involves using [`ctx.actions.expand_template`](lib/actions#expand_template) to write a very simple shell script, in this case invoking the publisher binary with a path to the output of the primary target. This way, the publisher implementation can remain generic, the `_sphinx_site` rule can just produce HTML, and this small script is all that's necessary to combine the two together. - -In `rules_k8s`, this is indeed what `.apply` does: [`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) writes a very simple Bash script, based on [`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), which runs `kubectl` with the output of the primary target. This script can then be build and run with `bazel run :staging.apply`, effectively providing a `k8s-apply` command for `k8s_object` targets. +In this example, a "docs" target is created, just as though the macro were a +standard, single Bazel rule. When built, the rule generates some configuration +and runs Sphinx to produce an HTML site, ready for manual inspection. However, +an additional "docs.publish" target is also created, which builds a script for +publishing the site. Once you check the output of the primary target, you can +use `bazel run :docs.publish` to publish it for public consumption, just like +an imaginary `bazel publish` command. + +It's not immediately obvious what the implementation of the `_sphinx_publisher` +rule might look like. Often, actions like this write a _launcher_ shell script. +This method typically involves using +[`ctx.actions.expand_template`](lib/actions#expand_template) +to write a very simple shell script, in this case invoking the publisher binary +with a path to the output of the primary target. This way, the publisher +implementation can remain generic, the `_sphinx_site` rule can just produce +HTML, and this small script is all that's necessary to combine the two +together. + +In `rules_k8s`, this is indeed what `.apply` does: +[`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) +writes a very simple Bash script, based on +[`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), +which runs `kubectl` with the output of the primary target. This script can +then be build and run with `bazel run :staging.apply`, effectively providing a +`k8s-apply` command for `k8s_object` targets. diff --git a/run/bazelrc.mdx b/run/bazelrc.mdx index 2ca427d7..90d4464a 100644 --- a/run/bazelrc.mdx +++ b/run/bazelrc.mdx @@ -2,85 +2,129 @@ title: 'Write bazelrc configuration files' --- -Bazel accepts many options. Some options are varied frequently (for example, `--subcommands`) while others stay the same across several builds (such as `--package_path`). To avoid specifying these unchanged options for every build (and other commands), you can specify options in a configuration file, called `.bazelrc`. + + +Bazel accepts many options. Some options are varied frequently (for example, +`--subcommands`) while others stay the same across several builds (such as +`--package_path`). To avoid specifying these unchanged options for every build +(and other commands), you can specify options in a configuration file, called +`.bazelrc`. ### Where are the `.bazelrc` files? -Bazel looks for optional configuration files in the following locations, in the order shown below. The options are interpreted in this order, so options in later files can override a value from an earlier file if a conflict arises. All options that control which of these files are loaded are startup options, which means they must occur after `bazel` and before the command (`build`, `test`, etc). +Bazel looks for optional configuration files in the following locations, +in the order shown below. The options are interpreted in this order, so +options in later files can override a value from an earlier file if a +conflict arises. All options that control which of these files are loaded are +startup options, which means they must occur after `bazel` and +before the command (`build`, `test`, etc). + +1. **The system RC file**, unless `--nosystem_rc` is present. -1. **The system RC file**, unless `--nosystem_rc` is present. + Path: - Path: + - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` + - On Windows: `%ProgramData%\bazel.bazelrc` - - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` - - On Windows: `%ProgramData%\bazel.bazelrc` + It is not an error if this file does not exist. - It is not an error if this file does not exist. + If another system-specified location is required, you must build a custom + Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in + [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). + The system-specified location may contain environment variable references, + such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. - If another system-specified location is required, you must build a custom Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). The system-specified location may contain environment variable references, such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. +2. **The workspace RC file**, unless `--noworkspace_rc` is present. -2. **The workspace RC file**, unless `--noworkspace_rc` is present. + Path: `.bazelrc` in your workspace directory (next to the main + `MODULE.bazel` file). - Path: `.bazelrc` in your workspace directory (next to the main `MODULE.bazel` file). + It is not an error if this file does not exist. - It is not an error if this file does not exist. +3. **The home RC file**, unless `--nohome_rc` is present. -3. **The home RC file**, unless `--nohome_rc` is present. + Path: - Path: + - On Linux/macOS/Unixes: `$HOME/.bazelrc` + - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` - - On Linux/macOS/Unixes: `$HOME/.bazelrc` - - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` + It is not an error if this file does not exist. - It is not an error if this file does not exist. +4. **The environment variable RC file**, if its path is set with the `BAZELRC` + environment variable. -4. **The environment variable RC file**, if its path is set with the `BAZELRC` environment variable. + The environment variable can include multiple comma-separated paths. - The environment variable can include multiple comma-separated paths. +5. **The user-specified RC file**, if specified with + --bazelrc=file -5. **The user-specified RC file**, if specified with `--bazelrc=file` + This flag is optional but can also be specified multiple times. - This flag is optional but can also be specified multiple times. + `/dev/null` indicates that all further `--bazelrc`s will be ignored, which + is useful to disable the search for a user rc file, such as in release + builds. - `/dev/null` indicates that all further `--bazelrc`s will be ignored, which is useful to disable the search for a user rc file, such as in release builds. + For example: - For example: + ``` + --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc + ``` - ``` - --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc - ``` + - `x.rc` and `y.rc` are read. + - `z.rc` is ignored due to the prior `/dev/null`. - - `x.rc` and `y.rc` are read. - - `z.rc` is ignored due to the prior `/dev/null`. +In addition to this optional configuration file, Bazel looks for a global rc +file. For more details, see the [global bazelrc section](#global-bazelrc). -In addition to this optional configuration file, Bazel looks for a global rc file. For more details, see the [global bazelrc section](#global-bazelrc). ### `.bazelrc` syntax and semantics -Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based grammar. Empty lines and lines starting with `#` (comments) are ignored. Each line contains a sequence of words, which are tokenized according to the same rules as the Bourne shell. +Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based +grammar. Empty lines and lines starting with `#` (comments) are ignored. Each +line contains a sequence of words, which are tokenized according to the same +rules as the Bourne shell. #### Imports -Lines that start with `import` or `try-import` are special: use these to load other "rc" files. To specify a path that is relative to the workspace root, write `import %workspace%/path/to/bazelrc`. +Lines that start with `import` or `try-import` are special: use these to load +other "rc" files. To specify a path that is relative to the workspace root, +write `import %workspace%/path/to/bazelrc`. -The difference between `import` and `try-import` is that Bazel fails if the `import`'ed file is missing (or can't be read), but not so for a `try-import`'ed file. +The difference between `import` and `try-import` is that Bazel fails if the +`import`'ed file is missing (or can't be read), but not so for a `try-import`'ed +file. Import precedence: -- Options in the imported file take precedence over options specified before the import statement. -- Options specified after the import statement take precedence over the options in the imported file. -- Options in files imported later take precedence over files imported earlier. +- Options in the imported file take precedence over options specified before + the import statement. +- Options specified after the import statement take precedence over the + options in the imported file. +- Options in files imported later take precedence over files imported earlier. #### Option defaults -Most lines of a bazelrc define default option values. The first word on each line specifies when these defaults are applied: - -- `startup`: startup options, which go before the command, and are described in `bazel help startup_options`. -- `common`: options that should be applied to all Bazel commands that support them. If a command does not support an option specified in this way, the option is ignored so long as it is valid for *some* other Bazel command. Note that this only applies to option names: If the current command accepts an option with the specified name, but doesn't support the specified value, it will fail. -- `always`: options that apply to all Bazel commands. If a command does not support an option specified in this way, it will fail. -- *`command`*: Bazel command, such as `build` or `query` to which the options apply. These options also apply to all commands that inherit from the specified command. (For example, `test` inherits from `build`.) - -Each of these lines may be used more than once and the arguments that follow the first word are combined as if they had appeared on a single line. (Users of CVS, another tool with a "Swiss army knife" command-line interface, will find the syntax similar to that of `.cvsrc`.) For example, the lines: +Most lines of a bazelrc define default option values. The first word on each +line specifies when these defaults are applied: + +- `startup`: startup options, which go before the command, and are described + in `bazel help startup_options`. +- `common`: options that should be applied to all Bazel commands that support + them. If a command does not support an option specified in this way, the + option is ignored so long as it is valid for *some* other Bazel command. + Note that this only applies to option names: If the current command accepts + an option with the specified name, but doesn't support the specified value, + it will fail. +- `always`: options that apply to all Bazel commands. If a command does not + support an option specified in this way, it will fail. +- _`command`_: Bazel command, such as `build` or `query` to which the options + apply. These options also apply to all commands that inherit from the + specified command. (For example, `test` inherits from `build`.) + +Each of these lines may be used more than once and the arguments that follow the +first word are combined as if they had appeared on a single line. (Users of CVS, +another tool with a "Swiss army knife" command-line interface, will find the +syntax similar to that of `.cvsrc`.) For example, the lines: ```posix-terminal build --test_tmpdir=/tmp/foo --verbose_failures @@ -98,49 +142,89 @@ so the effective flags are `--verbose_failures` and `--test_tmpdir=/tmp/bar`. Option precedence: -- Options on the command line always take precedence over those in rc files. For example, if a rc file says `build -c opt` but the command line flag is `-c dbg`, the command line flag takes precedence. +- Options on the command line always take precedence over those in rc files. + For example, if a rc file says `build -c opt` but the command line flag is + `-c dbg`, the command line flag takes precedence. +- Within the rc file, precedence is governed by specificity: lines for a more + specific command take precedence over lines for a less specific command. -- Within the rc file, precedence is governed by specificity: lines for a more specific command take precedence over lines for a less specific command. + Specificity is defined by inheritance. Some commands inherit options from + other commands, making the inheriting command more specific than the base + command. For example `test` inherits from the `build` command, so all `bazel + build` flags are valid for `bazel test`, and all `build` lines apply also to + `bazel test` unless there's a `test` line for the same option. If the rc + file says: - Specificity is defined by inheritance. Some commands inherit options from other commands, making the inheriting command more specific than the base command. For example `test` inherits from the `build` command, so all `bazel build` flags are valid for `bazel test`, and all `build` lines apply also to `bazel test` unless there's a `test` line for the same option. If the rc file says: + ```posix-terminal + test -c dbg --test_env=PATH - ```posix-terminal - test -c dbg --test_env=PATH + build -c opt --verbose_failures + ``` - build -c opt --verbose_failures - ``` + then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel + test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. - then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. + The inheritance (specificity) graph is: - The inheritance (specificity) graph is: + * Every command inherits from `common` + * The following commands inherit from (and are more specific than) + `build`: `test`, `run`, `clean`, `mobile-install`, `info`, + `print_action`, `config`, `cquery`, and `aquery` + * `coverage`, `fetch`, and `vendor` inherit from `test` - - Every command inherits from `common` - - The following commands inherit from (and are more specific than) `build`: `test`, `run`, `clean`, `mobile-install`, `info`, `print_action`, `config`, `cquery`, and `aquery` - - `coverage`, `fetch`, and `vendor` inherit from `test` +- Two lines specifying options for the same command at equal specificity are + parsed in the order in which they appear within the file. -- Two lines specifying options for the same command at equal specificity are parsed in the order in which they appear within the file. +- Because this precedence rule does not match the file order, it helps + readability if you follow the precedence order within rc files: start with + `common` options at the top, and end with the most-specific commands at the + bottom of the file. This way, the order in which the options are read is the + same as the order in which they are applied, which is more intuitive. -- Because this precedence rule does not match the file order, it helps readability if you follow the precedence order within rc files: start with `common` options at the top, and end with the most-specific commands at the bottom of the file. This way, the order in which the options are read is the same as the order in which they are applied, which is more intuitive. - -The arguments specified on a line of an rc file may include arguments that are not options, such as the names of build targets, and so on. These, like the options specified in the same files, have lower precedence than their siblings on the command line, and are always prepended to the explicit list of non- option arguments. +The arguments specified on a line of an rc file may include arguments that are +not options, such as the names of build targets, and so on. These, like the +options specified in the same files, have lower precedence than their siblings +on the command line, and are always prepended to the explicit list of non- +option arguments. #### `--config` -In addition to setting option defaults, the rc file can be used to group options and provide a shorthand for common groupings. This is done by adding a `:name` suffix to the command. These options are ignored by default, but will be included when the option `--config=name` is present, either on the command line or in a `.bazelrc` file, recursively, even inside of another config definition. The options specified by `command:name` will only be expanded for applicable commands, in the precedence order described above. - -Note: Configs can be defined in any `.bazelrc` file, and that all lines of the form `command:name` (for applicable commands) will be expanded, across the different rc files. In order to avoid name conflicts, we suggest that configs defined in personal rc files start with an underscore (`_`) to avoid unintentional name sharing. - -`--config=foo` expands to the options defined in [the rc files](#bazelrc-file-locations) "in-place" so that the options specified for the config have the same precedence that the `--config=foo` option had. - -This syntax does not extend to the use of `startup` to set [startup options](#option-defaults). Setting `startup:config-name --some_startup_option` in the .bazelrc will be ignored. +In addition to setting option defaults, the rc file can be used to group options +and provide a shorthand for common groupings. This is done by adding a `:name` +suffix to the command. These options are ignored by default, but will be +included when the option --config=name is present, +either on the command line or in a `.bazelrc` file, recursively, even inside of +another config definition. The options specified by `command:name` will only be +expanded for applicable commands, in the precedence order described above. + +Note: Configs can be defined in any `.bazelrc` file, and that all lines of +the form `command:name` (for applicable commands) will be expanded, across the +different rc files. In order to avoid name conflicts, we suggest that configs +defined in personal rc files start with an underscore (`_`) to avoid +unintentional name sharing. + +`--config=foo` expands to the options defined in +[the rc files](#bazelrc-file-locations) "in-place" so that the options +specified for the config have the same precedence that the `--config=foo` option +had. + +This syntax does not extend to the use of `startup` to set +[startup options](#option-defaults). Setting +`startup:config-name --some_startup_option` in the .bazelrc will be ignored. #### `--enable_platform_specific_config` -In the `.bazelrc` you can use platform specific configs that will be automatically enabled based on the host OS. For example, if the host OS is Linux and the `build` command is run, the `build:linux` configuration will be automatically enabled. Supported OS identifiers are `linux`, `macos`, `windows`, `freebsd`, and `openbsd`. +In the `.bazelrc` you can use platform specific configs that will be +automatically enabled based on the host OS. For example, if the host OS +is Linux and the `build` command is run, the `build:linux` configuration +will be automatically enabled. Supported OS identifiers are `linux`, +`macos`, `windows`, `freebsd`, and `openbsd`. -This is equivalent to using `--config=linux` on Linux, `--config=windows` on Windows, and so on. This can be disabled with `--enable_platform_specific_config=false`. +This is equivalent to using `--config=linux` on Linux, +`--config=windows` on Windows, and so on. This can be disabled with +`--enable_platform_specific_config=false`. -See [--enable\_platform\_specific\_config](/reference/command-line-reference#flag--enable_platform_specific_config). +See [--enable_platform_specific_config](/reference/command-line-reference#flag--enable_platform_specific_config). #### Example @@ -163,16 +247,27 @@ build:memcheck --strip=never --test_timeout=3600 #### `.bazelignore` -You can specify directories within the workspace that you want Bazel to ignore, such as related projects that use other build systems. Place a file called `.bazelignore` at the root of the workspace and add the directories you want Bazel to ignore, one per line. Entries are relative to the workspace root. +You can specify directories within the workspace +that you want Bazel to ignore, such as related projects +that use other build systems. Place a file called +`.bazelignore` at the root of the workspace +and add the directories you want Bazel to ignore, one per +line. Entries are relative to the workspace root. -The `.bazelignore` file does not permit glob semantics. Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. See [#24203](https://github.com/bazelbuild/bazel/pull/24203). +The `.bazelignore` file does not permit glob semantics. +Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. +It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. +See [#24203](https://github.com/bazelbuild/bazel/pull/24203). ### The global bazelrc file Bazel reads optional bazelrc files in this order: -1. System rc-file located at `/etc/bazel.bazelrc`. -2. Workspace rc-file located at `$workspace/tools/bazel.rc`. -3. Home rc-file located at `$HOME/.bazelrc` +1. System rc-file located at `/etc/bazel.bazelrc`. +2. Workspace rc-file located at `$workspace/tools/bazel.rc`. +3. Home rc-file located at `$HOME/.bazelrc` -Each bazelrc file listed here has a corresponding flag which can be used to disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` startup option. +Each bazelrc file listed here has a corresponding flag which can be used to +disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can +also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` +startup option. diff --git a/run/client-server.mdx b/run/client-server.mdx index b335b0ca..1868635f 100644 --- a/run/client-server.mdx +++ b/run/client-server.mdx @@ -2,21 +2,57 @@ title: 'Client/server implementation' --- -The Bazel system is implemented as a long-lived server process. This allows it to perform many optimizations not possible with a batch-oriented implementation, such as caching of BUILD files, dependency graphs, and other metadata from one build to the next. This improves the speed of incremental builds, and allows different commands, such as `build` and `query` to share the same cache of loaded packages, making queries very fast. Each server can handle at most one invocation at a time; further concurrent invocations will either block or fail-fast (see `--block_for_lock`). -When you run `bazel`, you're running the client. The client finds the server based on the [output base](/run/scripts#output-base-option), which by default is determined by the path of the base workspace directory and your userid, so if you build in multiple workspaces, you'll have multiple output bases and thus multiple Bazel server processes. Multiple users on the same workstation can build concurrently in the same workspace because their output bases will differ (different userids). -If the client cannot find a running server instance, it starts a new one. It does this by checking if the output base already exists, implying the blaze archive has already been unpacked. Otherwise if the output base doesn't exist, the client unzips the archive's files and sets their `mtime`s to a date 9 years in the future. Once installed, the client confirms that the `mtime`s of the unzipped files are equal to the far off date to ensure no installation tampering has occurred. - -The server process will stop after a period of inactivity (3 hours, by default, which can be modified using the startup option `--max_idle_secs`). For the most part, the fact that there is a server running is invisible to the user, but sometimes it helps to bear this in mind. For example, if you're running scripts that perform a lot of automated builds in different directories, it's important to ensure that you don't accumulate a lot of idle servers; you can do this by explicitly shutting them down when you're finished with them, or by specifying a short timeout period. - -The name of a Bazel server process appears in the output of `ps x` or `ps -e f` as `bazel(dirname)`, where *dirname* is the basename of the directory enclosing the root of your workspace directory. For example: +The Bazel system is implemented as a long-lived server process. This allows it +to perform many optimizations not possible with a batch-oriented implementation, +such as caching of BUILD files, dependency graphs, and other metadata from one +build to the next. This improves the speed of incremental builds, and allows +different commands, such as `build` and `query` to share the same cache of +loaded packages, making queries very fast. Each server can handle at most one +invocation at a time; further concurrent invocations will either block or +fail-fast (see `--block_for_lock`). + +When you run `bazel`, you're running the client. The client finds the server +based on the [output base](/run/scripts#output-base-option), which by default is +determined by the path of the base workspace directory and your userid, so if +you build in multiple workspaces, you'll have multiple output bases and thus +multiple Bazel server processes. Multiple users on the same workstation can +build concurrently in the same workspace because their output bases will differ +(different userids). + +If the client cannot find a running server instance, it starts a new one. It +does this by checking if the output base already exists, implying the blaze +archive has already been unpacked. Otherwise if the output base doesn't exist, +the client unzips the archive's files and sets their `mtime`s to a date 9 years +in the future. Once installed, the client confirms that the `mtime`s of the +unzipped files are equal to the far off date to ensure no installation tampering +has occurred. + +The server process will stop after a period of inactivity (3 hours, by default, +which can be modified using the startup option `--max_idle_secs`). For the most +part, the fact that there is a server running is invisible to the user, but +sometimes it helps to bear this in mind. For example, if you're running scripts +that perform a lot of automated builds in different directories, it's important +to ensure that you don't accumulate a lot of idle servers; you can do this by +explicitly shutting them down when you're finished with them, or by specifying +a short timeout period. + +The name of a Bazel server process appears in the output of `ps x` or `ps -e f` +as bazel(dirname), where _dirname_ is the basename of the +directory enclosing the root of your workspace directory. For example: ```posix-terminal ps -e f 16143 ? Sl 3:00 bazel(src-johndoe2) -server -Djava.library.path=... ``` -This makes it easier to find out which server process belongs to a given workspace. (Beware that with certain other options to `ps`, Bazel server processes may be named just `java`.) Bazel servers can be stopped using the [shutdown](/docs/user-manual#shutdown) command. +This makes it easier to find out which server process belongs to a given +workspace. (Beware that with certain other options to `ps`, Bazel server +processes may be named just `java`.) Bazel servers can be stopped using the +[shutdown](/docs/user-manual#shutdown) command. -When running `bazel`, the client first checks that the server is the appropriate version; if not, the server is stopped and a new one started. This ensures that the use of a long-running server process doesn't interfere with proper versioning. +When running `bazel`, the client first checks that the server is the appropriate +version; if not, the server is stopped and a new one started. This ensures that +the use of a long-running server process doesn't interfere with proper +versioning. diff --git a/run/scripts.mdx b/run/scripts.mdx index 25cb1f65..f267c903 100644 --- a/run/scripts.mdx +++ b/run/scripts.mdx @@ -2,93 +2,129 @@ title: 'Calling Bazel from scripts' --- -You can call Bazel from scripts to perform a build, run tests, or query the dependency graph. Bazel has been designed to enable effective scripting, but this section lists some details to bear in mind to make your scripts more robust. -### Choosing the output base -The `--output_base` option controls where the Bazel process should write the outputs of a build to, as well as various working files used internally by Bazel, one of which is a lock that guards against concurrent mutation of the output base by multiple Bazel processes. +You can call Bazel from scripts to perform a build, run tests, or query +the dependency graph. Bazel has been designed to enable effective scripting, but +this section lists some details to bear in mind to make your scripts more +robust. -Choosing the correct output base directory for your script depends on several factors. If you need to put the build outputs in a specific location, this will dictate the output base you need to use. If you are making a "read only" call to Bazel (such as `bazel query`), the locking factors will be more important. In particular, if you need to run multiple instances of your script concurrently, you should be mindful that each Blaze server process can handle at most one invocation [at a time](/run/client-server#clientserver-implementation). Depending on your situation it may make sense for each instance of your script to wait its turn, or it may make sense to use `--output_base` to run multiple Blaze servers and use those. +### Choosing the output base -If you use the default output base value, you will be contending for the same lock used by the user's interactive Bazel commands. If the user issues long-running commands such as builds, your script will have to wait for those commands to complete before it can continue. +The `--output_base` option controls where the Bazel process should write the +outputs of a build to, as well as various working files used internally by +Bazel, one of which is a lock that guards against concurrent mutation of the +output base by multiple Bazel processes. + +Choosing the correct output base directory for your script depends on several +factors. If you need to put the build outputs in a specific location, this will +dictate the output base you need to use. If you are making a "read only" call to +Bazel (such as `bazel query`), the locking factors will be more important. In +particular, if you need to run multiple instances of your script concurrently, +you should be mindful that each Blaze server process can handle at most one +invocation [at a time](/run/client-server#clientserver-implementation). +Depending on your situation it may make sense for each instance of your script +to wait its turn, or it may make sense to use `--output_base` to run multiple +Blaze servers and use those. + +If you use the default output base value, you will be contending for the same +lock used by the user's interactive Bazel commands. If the user issues +long-running commands such as builds, your script will have to wait for those +commands to complete before it can continue. ### Notes about server mode -By default, Bazel uses a long-running [server process](/run/client-server) as an optimization. When running Bazel in a script, don't forget to call `shutdown` when you're finished with the server, or, specify `--max_idle_secs=5` so that idle servers shut themselves down promptly. +By default, Bazel uses a long-running [server process](/run/client-server) as an +optimization. When running Bazel in a script, don't forget to call `shutdown` +when you're finished with the server, or, specify `--max_idle_secs=5` so that +idle servers shut themselves down promptly. ### What exit code will I get? -Bazel attempts to differentiate failures due to the source code under consideration from external errors that prevent Bazel from executing properly. Bazel execution can result in following exit codes: +Bazel attempts to differentiate failures due to the source code under +consideration from external errors that prevent Bazel from executing properly. +Bazel execution can result in following exit codes: **Exit Codes common to all commands:** -- `0` - Success - -- `2` - Command Line Problem, Bad or Illegal flags or command combination, or Bad Environment Variables. Your command line must be modified. - -- `8` - Build Interrupted but we terminated with an orderly shutdown. - -- `9` - The server lock is held and `--noblock_for_lock` was passed. - -- `32` - External Environment Failure not on this machine. - -- `33` - Bazel ran out of memory and crashed. You need to modify your command line. - -- `34` - Reserved for Google-internal use. - -- `35` - Reserved for Google-internal use. - -- `36` - Local Environmental Issue, suspected permanent. - -- `37` - Unhandled Exception / Internal Bazel Error. - -- `38` - Transient error publishing results to the Build Event Service. - -- `39` - Blobs required by Bazel are evicted from Remote Cache. - -- `41-44` - Reserved for Google-internal use. - -- `45` - Persistent error publishing results to the Build Event Service. - -- `47` - Reserved for Google-internal use. - -- `49` - Reserved for Google-internal use. +- `0` - Success +- `2` - Command Line Problem, Bad or Illegal flags or command combination, or + Bad Environment Variables. Your command line must be modified. +- `8` - Build Interrupted but we terminated with an orderly shutdown. +- `9` - The server lock is held and `--noblock_for_lock` was passed. +- `32` - External Environment Failure not on this machine. + +- `33` - Bazel ran out of memory and crashed. You need to modify your command line. +- `34` - Reserved for Google-internal use. +- `35` - Reserved for Google-internal use. +- `36` - Local Environmental Issue, suspected permanent. +- `37` - Unhandled Exception / Internal Bazel Error. +- `38` - Transient error publishing results to the Build Event Service. +- `39` - Blobs required by Bazel are evicted from Remote Cache. +- `41-44` - Reserved for Google-internal use. +- `45` - Persistent error publishing results to the Build Event Service. +- `47` - Reserved for Google-internal use. +- `49` - Reserved for Google-internal use. **Return codes for commands `bazel build`, `bazel test`:** -- `1` - Build failed. -- `3` - Build OK, but some tests failed or timed out. -- `4` - Build successful but no tests were found even though testing was requested. +- `1` - Build failed. +- `3` - Build OK, but some tests failed or timed out. +- `4` - Build successful but no tests were found even though testing was + requested. + **For `bazel run`:** -- `1` - Build failed. -- If the build succeeds but the executed subprocess returns a non-zero exit code it will be the exit code of the command as well. +- `1` - Build failed. +- If the build succeeds but the executed subprocess returns a non-zero exit + code it will be the exit code of the command as well. **For `bazel query`:** -- `3` - Partial success, but the query encountered 1 or more errors in the input BUILD file set and therefore the results of the operation are not 100% reliable. This is likely due to a `--keep_going` option on the command line. -- `7` - Command failure. +- `3` - Partial success, but the query encountered 1 or more errors in the + input BUILD file set and therefore the results of the operation are not 100% + reliable. This is likely due to a `--keep_going` option on the command line. +- `7` - Command failure. + +Future Bazel versions may add additional exit codes, replacing generic failure +exit code `1` with a different non-zero value with a particular meaning. +However, all non-zero exit values will always constitute an error. -Future Bazel versions may add additional exit codes, replacing generic failure exit code `1` with a different non-zero value with a particular meaning. However, all non-zero exit values will always constitute an error. ### Reading the .bazelrc file -By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base workspace directory or the user's home directory. Whether or not this is desirable is a choice for your script; if your script needs to be perfectly hermetic (such as when doing release builds), you should disable reading the .bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform a build using the user's preferred settings, the default behavior is better. +By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base +workspace directory or the user's home directory. Whether or not this is +desirable is a choice for your script; if your script needs to be perfectly +hermetic (such as when doing release builds), you should disable reading the +.bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform +a build using the user's preferred settings, the default behavior is better. ### Command log -The Bazel output is also available in a command log file which you can find with the following command: +The Bazel output is also available in a command log file which you can find with +the following command: ```posix-terminal bazel info command_log ``` -The command log file contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. +The command log file contains the interleaved stdout and stderr streams of the +most recent Bazel command. Note that running `bazel info` will overwrite the +contents of this file, since it then becomes the most recent Bazel command. +However, the location of the command log file will not change unless you change +the setting of the `--output_base` or `--output_user_root` options. ### Parsing output -The Bazel output is quite easy to parse for many purposes. Two options that may be helpful for your script are `--noshow_progress` which suppresses progress messages, and `--show_result n`, which controls whether or not "build up-to-date" messages are printed; these messages may be parsed to discover which targets were successfully built, and the location of the output files they created. Be sure to specify a very large value of *n* if you rely on these messages. +The Bazel output is quite easy to parse for many purposes. Two options that may +be helpful for your script are `--noshow_progress` which suppresses progress +messages, and --show_result n, which controls whether or +not "build up-to-date" messages are printed; these messages may be parsed to +discover which targets were successfully built, and the location of the output +files they created. Be sure to specify a very large value of _n_ if you rely on +these messages. ## Troubleshooting performance by profiling diff --git a/start/android-app.mdx b/start/android-app.mdx index 3ae739fd..35da5828 100644 --- a/start/android-app.mdx +++ b/start/android-app.mdx @@ -2,22 +2,36 @@ title: 'Bazel Tutorial: Build an Android App' --- + +**Note:** There are known limitations on using Bazel for building Android apps. +Visit the [rules_android issues page](https://github.com/bazelbuild/rules_android/issues) +to see the list of known issues. While the Bazel team and Open Source Software +(OSS) contributors work actively to address known issues, users should be aware +that Android Studio does not officially support Bazel projects. + This tutorial covers how to build a simple Android app using Bazel. -Bazel supports building Android apps using the [Android rules](/reference/be/android). +Bazel supports building Android apps using the +[Android rules](/reference/be/android). -This tutorial is intended for Windows, macOS and Linux users and does not require experience with Bazel or Android app development. You do not need to write any Android code in this tutorial. +This tutorial is intended for Windows, macOS and Linux users and does not +require experience with Bazel or Android app development. You do not need to +write any Android code in this tutorial. ## What you'll learn In this tutorial you learn how to: -- Set up your environment by installing Bazel and Android Studio, and downloading the sample project. -- Set up a Bazel workspace that contains the source code for the app and a `MODULE.bazel` file that identifies the top level of the workspace directory. -- Update the `MODULE.bazel` file to contain references to the required external dependencies, like the Android SDK. -- Create a `BUILD` file. -- Build the app with Bazel. -- Deploy and run the app on an Android emulator or physical device. +* Set up your environment by installing Bazel and Android Studio, and + downloading the sample project. +* Set up a Bazel workspace that contains the source code + for the app and a `MODULE.bazel` file that identifies the top level of the + workspace directory. +* Update the `MODULE.bazel` file to contain references to the required + external dependencies, like the Android SDK. +* Create a `BUILD` file. +* Build the app with Bazel. +* Deploy and run the app on an Android emulator or physical device. ## Before you begin @@ -25,13 +39,16 @@ In this tutorial you learn how to: Before you begin the tutorial, install the following software: -- **Bazel.** To install, follow the [installation instructions](/install). -- **Android Studio.** To install, follow the steps to [download Android Studio](https://developer.android.com/sdk/index.html). Execute the setup wizard to download the SDK and configure your environment. -- (Optional) **Git.** Use `git` to download the Android app project. +* **Bazel.** To install, follow the [installation instructions](/install). +* **Android Studio.** To install, follow the steps to [download Android + Studio](https://developer.android.com/sdk/index.html). + Execute the setup wizard to download the SDK and configure your environment. +* (Optional) **Git.** Use `git` to download the Android app project. ### Get the sample project -For the sample project, use the tutorial Android app project in [Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). +For the sample project, use the tutorial Android app project in +[Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). This app has a single button that prints a greeting when clicked: @@ -39,13 +56,15 @@ This app has a single button that prints a greeting when clicked: **Figure 1.** Android app button greeting. -Clone the repository with `git` (or [download the ZIP file directly](https://github.com/bazelbuild/examples/archive/master.zip)): +Clone the repository with `git` (or [download the ZIP file +directly](https://github.com/bazelbuild/examples/archive/master.zip)): ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in `examples/android/tutorial`. For the rest of the tutorial, you will be executing commands in this directory. +The sample project for this tutorial is in `examples/android/tutorial`. For +the rest of the tutorial, you will be executing commands in this directory. ### Review the source files @@ -80,20 +99,24 @@ The key files and directories are: | Android source files | `src/main/java/com/example/bazel/MainActivity.java` and `Greeter.java` | | Resource file directory | `src/main/java/com/example/bazel/res/` | + ## Build with Bazel ### Set up the workspace -A [workspace](/concepts/build-ref#workspace) is a directory that contains the source files for one or more software projects, and has a `MODULE.bazel` file at its root. +A [workspace](/concepts/build-ref#workspace) is a directory that contains the +source files for one or more software projects, and has a `MODULE.bazel` file at +its root. -The `MODULE.bazel` file may be empty or may contain references to [external dependencies](/external/overview) required to build your project. +The `MODULE.bazel` file may be empty or may contain references to [external +dependencies](/external/overview) required to build your project. First, run the following command to create an empty `MODULE.bazel` file: -| OS | Command | -| ------------------------ | -------------------------------------- | +| OS | Command | +| ------------------------ | ----------------------------------- | | Linux, macOS | `touch MODULE.bazel` | -| Windows (Command Prompt) | `type nul > MODULE.bazel` | +| Windows (Command Prompt) | `type nul > MODULE.bazel` | | Windows (PowerShell) | `New-Item MODULE.bazel -ItemType file` | ### Running Bazel @@ -104,7 +127,8 @@ You can now check if Bazel is running correctly with the command: bazel info workspace ``` -If Bazel prints the path of the current directory, you're good to go! If the `MODULE.bazel` file does not exist, you may see an error message like: +If Bazel prints the path of the current directory, you're good to go! If the +`MODULE.bazel` file does not exist, you may see an error message like: ``` ERROR: The 'info' command is only supported from within a workspace. @@ -112,7 +136,10 @@ ERROR: The 'info' command is only supported from within a workspace. ### Integrate with the Android SDK -Bazel needs to run the Android SDK [build tools](https://developer.android.com/tools/revisions/build-tools.html) to build the app. This means that you need to add some information to your `MODULE.bazel` file so that Bazel knows where to find them. +Bazel needs to run the Android SDK +[build tools](https://developer.android.com/tools/revisions/build-tools.html) +to build the app. This means that you need to add some information to your +`MODULE.bazel` file so that Bazel knows where to find them. Add the following line to your `MODULE.bazel` file: @@ -128,53 +155,94 @@ android_sdk_repository_extension = use_extension("@rules_android//rules/android_ use_repo(android_sdk_repository_extension, "androidsdk") ``` -This will use the Android SDK at the path referenced by the `ANDROID_HOME` environment variable, and automatically detect the highest API level and the latest version of build tools installed within that location. +This will use the Android SDK at the path referenced by the `ANDROID_HOME` +environment variable, and automatically detect the highest API level and the +latest version of build tools installed within that location. -You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find the path to the installed SDK using Android Studio's [SDK Manager](https://developer.android.com/studio/intro/update#sdk-manager). Assuming the SDK is installed to default locations, you can use the following commands to set the `ANDROID_HOME` variable: +You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find +the path to the installed SDK using Android Studio's [SDK +Manager](https://developer.android.com/studio/intro/update#sdk-manager). +Assuming the SDK is installed to default locations, you can use the following +commands to set the `ANDROID_HOME` variable: -| OS | Command | +| OS | Command | | ------------------------ | --------------------------------------------------- | | Linux | `export ANDROID_HOME=$HOME/Android/Sdk/` | | macOS | `export ANDROID_HOME=$HOME/Library/Android/sdk` | | Windows (Command Prompt) | `set ANDROID_HOME=%LOCALAPPDATA%\Android\Sdk` | | Windows (PowerShell) | `$env:ANDROID_HOME="$env:LOCALAPPDATA\Android\Sdk"` | -The above commands set the variable only for the current shell session. To make them permanent, run the following commands: +The above commands set the variable only for the current shell session. To make +them permanent, run the following commands: -| OS | Command | -| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | -| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | +| OS | Command | +| ------------------------ | --------------------------------------------------- | +| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | +| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | | Windows (Command Prompt) | `setx ANDROID_HOME "%LOCALAPPDATA%\Android\Sdk"` | | Windows (PowerShell) | `[System.Environment]::SetEnvironmentVariable('ANDROID_HOME', "$env:LOCALAPPDATA\Android\Sdk", [System.EnvironmentVariableTarget]::User)` | -**Optional:** If you want to compile native code into your Android app, you also need to download the [Android NDK](https://developer.android.com/ndk/downloads/index.html) and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: + +**Optional:** If you want to compile native code into your Android app, you +also need to download the [Android +NDK](https://developer.android.com/ndk/downloads/index.html) +and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: ```python bazel_dep(name = "rules_android_ndk", version = "0.1.3") ``` -For more information, read [Using the Android Native Development Kit with Bazel](/docs/android-ndk). - -It's not necessary to set the API levels to the same value for the SDK and NDK. [This page](https://developer.android.com/ndk/guides/stable_apis.html) contains a map from Android releases to NDK-supported API levels. - -### Create a BUILD file - -A [`BUILD` file](/concepts/build-files) describes the relationship between a set of build outputs, like compiled Android resources from `aapt` or class files from `javac`, and their dependencies. These dependencies may be source files (Java, C++) in your workspace or other build outputs. `BUILD` files are written in a language called **Starlark**. - -`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. The package hierarchy is a logical structure that overlays the directory structure in your workspace. Each [package](/concepts/build-ref#packages) is a directory (and its subdirectories) that contains a related set of source files and a `BUILD` file. The package also includes any subdirectories, excluding those that contain their own `BUILD` file. The *package name* is the path to the `BUILD` file relative to the `MODULE.bazel` file. -Note that Bazel's package hierarchy is conceptually different from the Java package hierarchy of your Android App directory where the `BUILD` file is located, although the directories may be organized identically. +For more information, read [Using the Android Native Development Kit with +Bazel](/docs/android-ndk). -For the simple Android app in this tutorial, the source files in `src/main/` comprise a single Bazel package. A more complex project may have many nested packages. +It's not necessary to set the API levels to the same value for the SDK and NDK. +[This page](https://developer.android.com/ndk/guides/stable_apis.html) +contains a map from Android releases to NDK-supported API levels. -#### Add an android\_library rule - -A `BUILD` file contains several different types of declarations for Bazel. The most important type is the [build rule](/concepts/build-files#types-of-build-rules), which tells Bazel how to build an intermediate or final software output from a set of source files or other dependencies. Bazel provides two build rules, [`android_library`](/reference/be/android#android_library) and [`android_binary`](/reference/be/android#android_binary), that you can use to build an Android app. - -For this tutorial, you'll first use the `android_library` rule to tell Bazel to build an [Android library module](http://developer.android.com/tools/projects/index.html#LibraryProjects) from the app source code and resource files. You'll then use the `android_binary` rule to tell Bazel how to build the Android application package. +### Create a BUILD file -Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, and declare a new `android_library` target: +A [`BUILD` file](/concepts/build-files) describes the relationship +between a set of build outputs, like compiled Android resources from `aapt` or +class files from `javac`, and their dependencies. These dependencies may be +source files (Java, C++) in your workspace or other build outputs. `BUILD` files +are written in a language called **Starlark**. + +`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. +The package hierarchy is a logical structure that overlays the directory +structure in your workspace. Each [package](/concepts/build-ref#packages) is a +directory (and its subdirectories) that contains a related set of source files +and a `BUILD` file. The package also includes any subdirectories, excluding +those that contain their own `BUILD` file. The *package name* is the path to the +`BUILD` file relative to the `MODULE.bazel` file. + +Note that Bazel's package hierarchy is conceptually different from the Java +package hierarchy of your Android App directory where the `BUILD` file is +located, although the directories may be organized identically. + +For the simple Android app in this tutorial, the source files in `src/main/` +comprise a single Bazel package. A more complex project may have many nested +packages. + +#### Add an android_library rule + +A `BUILD` file contains several different types of declarations for Bazel. The +most important type is the +[build rule](/concepts/build-files#types-of-build-rules), which tells +Bazel how to build an intermediate or final software output from a set of source +files or other dependencies. Bazel provides two build rules, +[`android_library`](/reference/be/android#android_library) and +[`android_binary`](/reference/be/android#android_binary), that you can use to +build an Android app. + +For this tutorial, you'll first use the +`android_library` rule to tell Bazel to build an [Android library +module](http://developer.android.com/tools/projects/index.html#LibraryProjects) +from the app source code and resource files. You'll then use the +`android_binary` rule to tell Bazel how to build the Android application package. + +Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, +and declare a new `android_library` target: `src/main/java/com/example/bazel/BUILD`: @@ -196,13 +264,18 @@ android_library( ) ``` -The `android_library` build rule contains a set of attributes that specify the information that Bazel needs to build a library module from the source files. Note also that the name of the rule is `greeter_activity`. You'll reference the rule using this name as a dependency in the `android_binary` rule. +The `android_library` build rule contains a set of attributes that specify the +information that Bazel needs to build a library module from the source files. +Note also that the name of the rule is `greeter_activity`. You'll reference the +rule using this name as a dependency in the `android_binary` rule. -#### Add an android\_binary rule +#### Add an android_binary rule -The [`android_binary`](/reference/be/android#android_binary) rule builds the Android application package (`.apk` file) for your app. +The [`android_binary`](/reference/be/android#android_binary) rule builds +the Android application package (`.apk` file) for your app. -Create a new `BUILD` file in the `src/main/` directory, and declare a new `android_binary` target: +Create a new `BUILD` file in the `src/main/` directory, +and declare a new `android_binary` target: `src/main/BUILD`: @@ -216,23 +289,35 @@ android_binary( ) ``` -Here, the `deps` attribute references the output of the `greeter_activity` rule you added to the `BUILD` file above. This means that when Bazel builds the output of this rule it checks first to see if the output of the `greeter_activity` library rule has been built and is up-to-date. If not, Bazel builds it and then uses that output to build the application package file. +Here, the `deps` attribute references the output of the `greeter_activity` rule +you added to the `BUILD` file above. This means that when Bazel builds the +output of this rule it checks first to see if the output of the +`greeter_activity` library rule has been built and is up-to-date. If not, Bazel +builds it and then uses that output to build the application package file. Now, save and close the file. ### Build the app -Try building the app! Run the following command to build the `android_binary` target: +Try building the app! Run the following command to build the +`android_binary` target: ```posix-terminal bazel build //src/main:app ``` -The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the target that follows. The target is specified as the name of a build rule inside a `BUILD` file, with along with the package path relative to your workspace directory. For this example, the target is `app` and the package path is `//src/main/`. +The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the +target that follows. The target is specified as the name of a build rule inside +a `BUILD` file, with along with the package path relative to your workspace +directory. For this example, the target is `app` and the package path is +`//src/main/`. -Note that you can sometimes omit the package path or target name, depending on your current working directory at the command line and the name of the target. For more details about target labels and paths, see [Labels](/concepts/labels). +Note that you can sometimes omit the package path or target name, depending on +your current working directory at the command line and the name of the target. +For more details about target labels and paths, see [Labels](/concepts/labels). -Bazel will start to build the sample app. During the build process, its output will appear similar to the following: +Bazel will start to build the sample app. During the build process, its output +will appear similar to the following: ```bash INFO: Analysed target //src/main:app (0 packages loaded, 0 targets configured). @@ -245,25 +330,40 @@ Target //src/main:app up-to-date: #### Locate the build outputs -Bazel puts the outputs of both intermediate and final build operations in a set of per-user, per-workspace output directories. These directories are symlinked from the following locations at the top-level of the project directory, where the `MODULE.bazel` file is: +Bazel puts the outputs of both intermediate and final build operations in a set +of per-user, per-workspace output directories. These directories are symlinked +from the following locations at the top-level of the project directory, where +the `MODULE.bazel` file is: -- `bazel-bin` stores binary executables and other runnable build outputs -- `bazel-genfiles` stores intermediary source files that are generated by Bazel rules -- `bazel-out` stores other types of build outputs +* `bazel-bin` stores binary executables and other runnable build outputs +* `bazel-genfiles` stores intermediary source files that are generated by + Bazel rules +* `bazel-out` stores other types of build outputs -Bazel stores the Android `.apk` file generated using the `android_binary` rule in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is derived from the name of the Bazel package. +Bazel stores the Android `.apk` file generated using the `android_binary` rule +in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is +derived from the name of the Bazel package. -At a command prompt, list the contents of this directory and find the `app.apk` file: +At a command prompt, list the contents of this directory and find the `app.apk` +file: -| OS | Command | +| OS | Command | | ------------------------ | ------------------------ | | Linux, macOS | `ls bazel-bin/src/main` | | Windows (Command Prompt) | `dir bazel-bin\src\main` | | Windows (PowerShell) | `ls bazel-bin\src\main` | + ### Run the app -You can now deploy the app to a connected Android device or emulator from the command line using `bazel mobile-install`. This command uses the Android Debug Bridge (`adb`) to communicate with the device. You must set up your device to use `adb` following the instructions in [Android Debug Bridge](http://developer.android.com/tools/help/adb.html) before deployment. You can also choose to install the app on the Android emulator included in Android Studio. Make sure the emulator is running before executing the command below. +You can now deploy the app to a connected Android device or emulator from the +command line using `bazel mobile-install`. +This command uses the Android Debug Bridge (`adb`) to communicate with the +device. You must set up your device to use `adb` following the instructions in +[Android Debug Bridge](http://developer.android.com/tools/help/adb.html) +before deployment. You can also choose to install the app on the Android emulator +included in Android Studio. Make sure the emulator is running before executing +the command below. Enter the following: @@ -278,7 +378,12 @@ bazel mobile-install //src/main:app \ --tool_java_language_version=17 ``` -Note that the extra flags required for mobile-install can be added to your project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags (`--mode`, `--mobile_install*`) will no longer be required starting from Bazel 8.4.0 and onwards. The various Java flags for language and runtime version may be required depending on your workspace's Java configuration. *Mobile-install sub-tools require a language and runtime level of 17 or higher.* +Note that the extra flags required for mobile-install can be added to your +project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags +(`--mode`, `--mobile_install*`) will no longer be required starting from +Bazel 8.4.0 and onwards. The various Java flags for language and runtime version +may be required depending on your workspace's Java configuration. +_Mobile-install sub-tools require a language and runtime level of 17 or higher._ Now the "Bazel Tutorial App" should install and launch automatically: @@ -292,20 +397,16 @@ Now the "Bazel Tutorial App" should install and launch automatically: For more details, see these pages: -- Open issues on [rules\_android GitHub](https://github.com/bazelbuild/rules_android/issues) - -- More information on [mobile-install](/docs/mobile-install) - -- Integrate external dependencies like AppCompat, Guava and JUnit from Maven repositories using [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) - -- Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) integration. - -- Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) - -- See more Bazel example projects of: - - - [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) - - [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) - - [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) +* Open issues on [rules_android GitHub](https://github.com/bazelbuild/rules_android/issues) +* More information on [mobile-install](/docs/mobile-install) +* Integrate external dependencies like AppCompat, Guava and JUnit from Maven + repositories using [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) +* Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) + integration. +* Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) +* See more Bazel example projects of: + * [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) + * [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) + * [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) Happy building! diff --git a/start/cpp.mdx b/start/cpp.mdx index df9ec54e..adb7c71a 100644 --- a/start/cpp.mdx +++ b/start/cpp.mdx @@ -2,25 +2,37 @@ title: 'Bazel Tutorial: Build a C++ Project' --- + + ## Introduction -New to Bazel? You're in the right place. Follow this First Build tutorial for a simplified introduction to using Bazel. This tutorial defines key terms as they are used in Bazel's context and walks you through the basics of the Bazel workflow. Starting with the tools you need, you will build and run three projects with increasing complexity and learn how and why they get more complex. +New to Bazel? You're in the right place. Follow this First Build tutorial for a +simplified introduction to using Bazel. This tutorial defines key terms as they +are used in Bazel's context and walks you through the basics of the Bazel +workflow. Starting with the tools you need, you will build and run three +projects with increasing complexity and learn how and why they get more complex. -While Bazel is a [build system](https://bazel.build/basics/build-systems) that supports multi-language builds, this tutorial uses a C++ project as an example and provides the general guidelines and flow that apply to most languages. +While Bazel is a [build system](https://bazel.build/basics/build-systems) that +supports multi-language builds, this tutorial uses a C++ project as an example +and provides the general guidelines and flow that apply to most languages. Estimated completion time: 30 minutes. ### Prerequisites -Start by [installing Bazel](https://bazel.build/install), if you haven't already. This tutorial uses Git for source control, so for best results [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. +Start by [installing Bazel](https://bazel.build/install), if you haven't +already. This tutorial uses Git for source control, so for best results [install +Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. -Next, retrieve the sample project from Bazel's GitHub repository by running the following in your command-line tool of choice: +Next, retrieve the sample project from Bazel's GitHub repository by running the +following in your command-line tool of choice: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/cpp-tutorial` directory. +The sample project for this tutorial is in the `examples/cpp-tutorial` +directory. Take a look at how it's structured: @@ -52,24 +64,52 @@ examples └── MODULE.bazel ``` -There are three sets of files, each set representing a stage in this tutorial. In the first stage, you will build a single \[target] ([https://bazel.build/reference/glossary#target](https://bazel.build/reference/glossary#target)) residing in a single \[package] ([https://bazel.build/reference/glossary#package](https://bazel.build/reference/glossary#package)). In the second stage, you will build both a binary and a library from a single package. In the third and final stage, you will build a project with multiple packages and build it with multiple targets. +There are three sets of files, each set representing a stage in this tutorial. +In the first stage, you will build a single [target] +(https://bazel.build/reference/glossary#target) residing in a single [package] +(https://bazel.build/reference/glossary#package). In the second stage, you will +build both a binary and a library from a single package. In the third and final +stage, you will build a project with multiple packages and build it with +multiple targets. ### Summary: Introduction -By installing Bazel (and Git) and cloning the repository for this tutorial, you have laid the foundation for your first build with Bazel. Continue to the next section to define some terms and set up your [workspace](https://bazel.build/reference/glossary#workspace). +By installing Bazel (and Git) and cloning the repository for this tutorial, you +have laid the foundation for your first build with Bazel. Continue to the next +section to define some terms and set up your +[workspace](https://bazel.build/reference/glossary#workspace). ## Getting started -Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains these significant files: +Before you can build a project, you need to set up its workspace. A workspace +is a directory that holds your project's source files and Bazel's build outputs. +It also contains these significant files: -- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure. It's also where you specify your external dependencies. -- One or more [`BUILD` files](https://bazel.build/reference/glossary#build-file), which tell Bazel how to build different parts of the project. A directory within the workspace that contains a `BUILD` file is a [package](https://bazel.build/reference/glossary#package). (More on packages later in this tutorial.) +* The `MODULE.bazel` file, which identifies the directory and its contents as + a Bazel workspace and lives at the root of the project's directory + structure. It's also where you specify your external dependencies. +* One or more [`BUILD` + files](https://bazel.build/reference/glossary#build-file), which tell Bazel + how to build different parts of the project. A directory within the + workspace that contains a `BUILD` file is a + [package](https://bazel.build/reference/glossary#package). (More on packages + later in this tutorial.) -In future projects, to designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. For the purposes of this tutorial, a `MODULE.bazel` file is already present in each stage. +In future projects, to designate a directory as a Bazel workspace, create an +empty file named `MODULE.bazel` in that directory. For the purposes of this +tutorial, a `MODULE.bazel` file is already present in each stage. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. Each `BUILD` file requires at least one [rule](https://bazel.build/reference/glossary#rule) as a set of instructions, which tells Bazel how to build the outputs you want, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a [target](https://bazel.build/reference/glossary#target) and points to a specific set of source files and [dependencies](https://bazel.build/reference/glossary#dependency). A target can also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. Each +`BUILD` file requires at least one +[rule](https://bazel.build/reference/glossary#rule) as a set of instructions, +which tells Bazel how to build the outputs you want, such as executable binaries +or libraries. Each instance of a build rule in the `BUILD` file is called a +[target](https://bazel.build/reference/glossary#target) and points to a specific +set of source files and +[dependencies](https://bazel.build/reference/glossary#dependency). A target can +also point to other targets. Take a look at the `BUILD` file in the `cpp-tutorial/stage1/main` directory: @@ -80,15 +120,21 @@ cc_binary( ) ``` -In our example, the `hello-world` target instantiates Bazel's built-in [`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule tells Bazel to build a self-contained executable binary from the `hello-world.cc`\> source file with no dependencies. +In our example, the `hello-world` target instantiates Bazel's built-in +[`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule +tells Bazel to build a self-contained executable binary from the +`hello-world.cc`> source file with no dependencies. ### Summary: getting started -Now you are familiar with some key terms, and what they mean in the context of this project and Bazel in general. In the next section, you will build and test Stage 1 of the project. +Now you are familiar with some key terms, and what they mean in the context of +this project and Bazel in general. In the next section, you will build and test +Stage 1 of the project. ## Stage 1: single target, single package -It's time to build the first part of the project. For a visual reference, the structure of the Stage 1 section of the project is: +It's time to build the first part of the project. For a visual reference, the +structure of the Stage 1 section of the project is: ```none examples @@ -112,7 +158,9 @@ Next, run: bazel build //main:hello-world ``` -In the target label, the `//main:` part is the location of the `BUILD` file relative to the root of the workspace, and `hello-world` is the target name in the `BUILD` file. +In the target label, the `//main:` part is the location of the `BUILD` file +relative to the root of the workspace, and `hello-world` is the target name in +the `BUILD` file. Bazel produces something that looks like this: @@ -123,7 +171,8 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.267s, Critical Path: 0.25s ``` -You just built your first Bazel target. Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. +You just built your first Bazel target. Bazel places build outputs in the +`bazel-bin` directory at the root of the workspace. Now test your freshly built binary, which is: @@ -135,15 +184,23 @@ This results in a printed "`Hello world`" message. Here's the dependency graph of Stage 1: -![Dependency graph for hello-world displays a single target with a single source file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world displays a single target with a single source file.") +![Dependency graph for hello-world displays a single target with a single source +file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world +displays a single target with a single source file.") ### Summary: stage 1 -Now that you have completed your first build, you have a basic idea of how a build is structured. In the next stage, you will add complexity by adding another target. +Now that you have completed your first build, you have a basic idea of how a +build is structured. In the next stage, you will add complexity by adding +another target. ## Stage 2: multiple build targets -While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages. This allows for fast incremental builds – that is, Bazel only rebuilds what's changed – and speeds up your builds by building multiple parts of a project at once. This stage of the tutorial adds a target, and the next adds a package. +While a single target is sufficient for small projects, you may want to split +larger projects into multiple targets and packages. This allows for fast +incremental builds – that is, Bazel only rebuilds what's changed – and speeds up +your builds by building multiple parts of a project at once. This stage of the +tutorial adds a target, and the next adds a package. This is the directory you are working with for Stage 2: @@ -175,9 +232,15 @@ cc_binary( ) ``` -With this `BUILD` file, Bazel first builds the `hello-greet` library (using Bazel's built-in [`cc_library` rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the `hello-world` binary. The `deps` attribute in the `hello-world` target tells Bazel that the `hello-greet` library is required to build the `hello-world` binary. +With this `BUILD` file, Bazel first builds the `hello-greet` library (using +Bazel's built-in [`cc_library` +rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the +`hello-world` binary. The `deps` attribute in the `hello-world` target tells +Bazel that the `hello-greet` library is required to build the `hello-world` +binary. -Before you can build this new version of the project, you need to change directories, switching to the `cpp-tutorial/stage2` directory by running: +Before you can build this new version of the project, you need to change +directories, switching to the `cpp-tutorial/stage2` directory by running: ```posix-terminal cd ../stage2 @@ -198,25 +261,36 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.399s, Critical Path: 0.30s ``` -Now you can test your freshly built binary, which returns another "`Hello world`": +Now you can test your freshly built binary, which returns another "`Hello +world`": ```posix-terminal bazel-bin/main/hello-world ``` -If you now modify `hello-greet.cc` and rebuild the project, Bazel only recompiles that file. +If you now modify `hello-greet.cc` and rebuild the project, Bazel only +recompiles that file. -Looking at the dependency graph, you can see that `hello-world` depends on an extra input named `hello-greet`: +Looking at the dependency graph, you can see that `hello-world` depends on an +extra input named `hello-greet`: -![Dependency graph for hello-world displays dependency changes after modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency graph for `hello-world` displays dependency changes after modification to the file.") +![Dependency graph for `hello-world` displays dependency changes after +modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency +graph for `hello-world` displays dependency changes after modification to the +file.") ### Summary: stage 2 -You've now built the project with two targets. The `hello-world` target builds one source file and depends on one other target (`//main:hello-greet`), which builds two additional source files. In the next section, take it a step further and add another package. +You've now built the project with two targets. The `hello-world` target builds +one source file and depends on one other target (`//main:hello-greet`), which +builds two additional source files. In the next section, take it a step further +and add another package. ## Stage 3: multiple packages -This next stage adds another layer of complication and builds a project with multiple packages. Take a look at the structure and contents of the `cpp-tutorial/stage3` directory: +This next stage adds another layer of complication and builds a project with +multiple packages. Take a look at the structure and contents of the +`cpp-tutorial/stage3` directory: ```none └──stage3 @@ -232,7 +306,9 @@ This next stage adds another layer of complication and builds a project with mul └── MODULE.bazel ``` -You can see that now there are two sub-directories, and each contains a `BUILD` file. Therefore, to Bazel, the workspace now contains two packages: `lib` and `main`. +You can see that now there are two sub-directories, and each contains a `BUILD` +file. Therefore, to Bazel, the workspace now contains two packages: `lib` and +`main`. Take a look at the `lib/BUILD` file: @@ -264,13 +340,25 @@ cc_binary( ) ``` -The `hello-world` target in the main package depends on the` hello-time` target in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows this through the `deps` attribute. You can see this reflected in the dependency graph: +The `hello-world` target in the main package depends on the` hello-time` target +in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows +this through the `deps` attribute. You can see this reflected in the dependency +graph: -![Dependency graph for hello-world displays how the target in the main package depends on the target in the lib package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for `hello-world` displays how the target in the main package depends on the target in the `lib` package.") +![Dependency graph for `hello-world` displays how the target in the main package +depends on the target in the `lib` +package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for +`hello-world` displays how the target in the main package depends on the target +in the `lib` package.") -For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` explicitly visible to targets in `main/BUILD` using the visibility attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs. +For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` +explicitly visible to targets in `main/BUILD` using the visibility attribute. +This is because by default targets are only visible to other targets in the same +`BUILD` file. Bazel uses target visibility to prevent issues such as libraries +containing implementation details leaking into public APIs. -Now build this final version of the project. Switch to the `cpp-tutorial/stage3` directory by running: +Now build this final version of the project. Switch to the `cpp-tutorial/stage3` +directory by running: ```posix-terminal cd ../stage3 @@ -299,15 +387,25 @@ bazel-bin/main/hello-world ### Summary: stage 3 -You've now built the project as two packages with three targets and understand the dependencies between them, which equips you to go forth and build future projects with Bazel. In the next section, take a look at how to continue your Bazel journey. +You've now built the project as two packages with three targets and understand +the dependencies between them, which equips you to go forth and build future +projects with Bazel. In the next section, take a look at how to continue your +Bazel journey. ## Next steps -You've now completed your first basic build with Bazel, but this is just the start. Here are some more resources to continue learning with Bazel: - -- To keep focusing on C++, read about common [C++ build use cases](https://bazel.build/tutorials/cpp-use-cases). -- To get started with building other applications with Bazel, see the tutorials for [Java](https://bazel.build/start/java), [Android application](https://bazel.build/start/android-app), or [iOS application](https://bazel.build/start/ios-app). -- To learn more about working with local and remote repositories, read about [external dependencies](https://bazel.build/docs/external). -- To learn more about Bazel's other rules, see this [reference guide](https://bazel.build/rules). +You've now completed your first basic build with Bazel, but this is just the +start. Here are some more resources to continue learning with Bazel: + +* To keep focusing on C++, read about common [C++ build use + cases](https://bazel.build/tutorials/cpp-use-cases). +* To get started with building other applications with Bazel, see the + tutorials for [Java](https://bazel.build/start/java), [Android + application](https://bazel.build/start/android-app), or [iOS + application](https://bazel.build/start/ios-app). +* To learn more about working with local and remote repositories, read about + [external dependencies](https://bazel.build/docs/external). +* To learn more about Bazel's other rules, see this [reference + guide](https://bazel.build/rules). Happy building! diff --git a/start/go.mdx b/start/go.mdx index d5742314..3a095681 100644 --- a/start/go.mdx +++ b/start/go.mdx @@ -2,7 +2,12 @@ title: 'Bazel Tutorial: Build a Go Project' --- -This tutorial introduces you to the basics of Bazel by showing you how to build a Go (Golang) project. You'll learn how to set up your workspace, build a small program, import a library, and run its test. Along the way, you'll learn key Bazel concepts, such as targets and `BUILD` files. + + +This tutorial introduces you to the basics of Bazel by showing you how to build +a Go (Golang) project. You'll learn how to set up your workspace, build a small +program, import a library, and run its test. Along the way, you'll learn key +Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes @@ -10,27 +15,35 @@ Estimated completion time: 30 minutes ### Install Bazel -Before you get started, first [install bazel](/install) if you haven't done so already. +Before you get started, first [install bazel](/install) if you haven't done so +already. You can check if Bazel is installed by running `bazel version` in any directory. ### Install Go (optional) -You don't need to [install Go](https://go.dev/doc/install) to build Go projects with Bazel. The Bazel Go rule set automatically downloads and uses a Go toolchain instead of using the toolchain installed on your machine. This ensures all developers on a project build with same version of Go. +You don't need to [install Go](https://go.dev/doc/install) to build Go projects +with Bazel. The Bazel Go rule set automatically downloads and uses a Go +toolchain instead of using the toolchain installed on your machine. This ensures +all developers on a project build with same version of Go. -However, you may still want to install a Go toolchain to run commands like `go get` and `go mod tidy`. +However, you may still want to install a Go toolchain to run commands like `go +get` and `go mod tidy`. You can check if Go is installed by running `go version` in any directory. ### Get the sample project -The Bazel examples are stored in a Git repository, so you'll need to [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you haven't already. To download the examples repository, run this command: +The Bazel examples are stored in a Git repository, so you'll need to [install +Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you +haven't already. To download the examples repository, run this command: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/go-tutorial` directory. See what it contains: +The sample project for this tutorial is in the `examples/go-tutorial` directory. +See what it contains: ```none go-tutorial/ @@ -39,11 +52,13 @@ go-tutorial/ └── stage3 ``` -There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a different section of this tutorial. Each stage builds on the previous one. +There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a +different section of this tutorial. Each stage builds on the previous one. ## Build with Bazel -Start in the `stage1` directory, where we'll find a program. We can build it with `bazel build`, then run it: +Start in the `stage1` directory, where we'll find a program. We can +build it with `bazel build`, then run it: ```posix-shell $ cd go-tutorial/stage1/ @@ -92,7 +107,9 @@ func main() { } ``` -`BUILD` contains some instructions for Bazel, telling it what we want to build. You'll typically write a file like this in each directory. For this project, we have a single `go_binary` target that builds our program from `hello.go`. +`BUILD` contains some instructions for Bazel, telling it what we want to build. +You'll typically write a file like this in each directory. For this project, we +have a single `go_binary` target that builds our program from `hello.go`. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -103,9 +120,17 @@ go_binary( ) ``` -`MODULE.bazel` tracks your project's dependencies. It also marks your project's root directory, so you'll only write one `MODULE.bazel` file per project. It serves a similar purpose to Go's `go.mod` file. You don't actually need a `go.mod` file in a Bazel project, but it may still be useful to have one so that you can continue using `go get` and `go mod tidy` for dependency management. The Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in another tutorial. +`MODULE.bazel` tracks your project's dependencies. It also marks your project's +root directory, so you'll only write one `MODULE.bazel` file per project. It +serves a similar purpose to Go's `go.mod` file. You don't actually need a +`go.mod` file in a Bazel project, but it may still be useful to have one so that +you can continue using `go get` and `go mod tidy` for dependency management. The +Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in +another tutorial. -Our `MODULE.bazel` file contains a single dependency on [rules\_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need this dependency because Bazel doesn't have built-in support for Go. +Our `MODULE.bazel` file contains a single dependency on +[rules_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need +this dependency because Bazel doesn't have built-in support for Go. ```bazel bazel_dep( @@ -114,25 +139,50 @@ bazel_dep( ) ``` -Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes and other metadata about our dependencies. It includes implicit dependencies added by Bazel itself, so it's quite long, and we won't show it here. Just like `go.sum`, you should commit your `MODULE.bazel.lock` file to source control to ensure everyone on your project gets the same version of each dependency. You shouldn't need to edit `MODULE.bazel.lock` manually. +Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes +and other metadata about our dependencies. It includes implicit dependencies +added by Bazel itself, so it's quite long, and we won't show it here. Just like +`go.sum`, you should commit your `MODULE.bazel.lock` file to source control to +ensure everyone on your project gets the same version of each dependency. You +shouldn't need to edit `MODULE.bazel.lock` manually. ### Understand the BUILD file -Most of your interaction with Bazel will be through `BUILD` files (or equivalently, `BUILD.bazel` files), so it's important to understand what they do. +Most of your interaction with Bazel will be through `BUILD` files (or +equivalently, `BUILD.bazel` files), so it's important to understand what they +do. -`BUILD` files are written in a scripting language called [Starlark](https://bazel.build/rules/language), a limited subset of Python. +`BUILD` files are written in a scripting language called +[Starlark](https://bazel.build/rules/language), a limited subset of Python. -A `BUILD` file contains a list of [targets](https://bazel.build/reference/glossary#target). A target is something Bazel can build, like a binary, library, or test. +A `BUILD` file contains a list of +[targets](https://bazel.build/reference/glossary#target). A target is something +Bazel can build, like a binary, library, or test. -A target calls a rule function with a list of [attributes](https://bazel.build/reference/glossary#attribute) to describe what should be built. Our example has two attributes: `name` identifies the target on the command line, and `srcs` is a list of source file paths (slash-separated, relative to the directory containing the `BUILD` file). +A target calls a rule function with a list of +[attributes](https://bazel.build/reference/glossary#attribute) to describe what +should be built. Our example has two attributes: `name` identifies the target on +the command line, and `srcs` is a list of source file paths (slash-separated, +relative to the directory containing the `BUILD` file). -A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a target. In our example, we used the [`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) (commands) that generate a set of output files. For example, `go_binary` defines Go compile and link actions that produce an executable output file. +A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a +target. In our example, we used the +[`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) +rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) +(commands) that generate a set of output files. For example, `go_binary` defines +Go compile and link actions that produce an executable output file. -Bazel has built-in rules for a few languages like Java and C++. You can find their [documentation in the Build Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find rule sets for many other languages and tools on the [Bazel Central Registry (BCR)](https://registry.bazel.build/). +Bazel has built-in rules for a few languages like Java and C++. You can find +their [documentation in the Build +Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find +rule sets for many other languages and tools on the [Bazel Central Registry +(BCR)](https://registry.bazel.build/). ## Add a library -Move onto the `stage2` directory, where we'll build a new program that prints your fortune. This program uses a separate Go package as a library that selects a fortune from a predefined list of messages. +Move onto the `stage2` directory, where we'll build a new program that +prints your fortune. This program uses a separate Go package as a library that +selects a fortune from a predefined list of messages. ```none go-tutorial/stage2 @@ -145,7 +195,11 @@ go-tutorial/stage2 └── print_fortune.go ``` -`fortune.go` is the source file for the library. The `fortune` library is a separate Go package, so its source files are in a separate directory. Bazel doesn't require you to keep Go packages in separate directories, but it's a strong convention in the Go ecosystem, and following it will help you stay compatible with other Go tools. +`fortune.go` is the source file for the library. The `fortune` library is a +separate Go package, so its source files are in a separate directory. Bazel +doesn't require you to keep Go packages in separate directories, but it's a +strong convention in the Go ecosystem, and following it will help you stay +compatible with other Go tools. ```go package fortune @@ -163,11 +217,19 @@ func Get() string { } ``` -The `fortune` directory has its own `BUILD` file that tells Bazel how to build this package. We use `go_library` here instead of `go_binary`. +The `fortune` directory has its own `BUILD` file that tells Bazel how to build +this package. We use `go_library` here instead of `go_binary`. -We also need to set the `importpath` attribute to a string with which the library can be imported into other Go source files. This name should be the repository path (or module path) concatenated with the directory within the repository. +We also need to set the `importpath` attribute to a string with which the +library can be imported into other Go source files. This name should be the +repository path (or module path) concatenated with the directory within the +repository. -Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. [`visibility`](https://bazel.build/concepts/visibility) may be set on any target. It determines which Bazel packages may depend on this target. In our case, we want any target to be able to depend on this library, so we use the special value `//visibility:public`. +Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. +[`visibility`](https://bazel.build/concepts/visibility) may be set on any +target. It determines which Bazel packages may depend on this target. In our +case, we want any target to be able to depend on this library, so we use the +special value `//visibility:public`. ```bazel load("@rules_go//go:def.bzl", "go_library") @@ -202,9 +264,11 @@ func main() { } ``` -`print_fortune.go` imports the package using the same string declared in the `importpath` attribute of the `fortune` library. +`print_fortune.go` imports the package using the same string declared in the +`importpath` attribute of the `fortune` library. -We also need to declare this dependency to Bazel. Here's the `BUILD` file in the `stage2` directory. +We also need to declare this dependency to Bazel. Here's the `BUILD` file in the +`stage2` directory. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -222,25 +286,57 @@ You can run this with the command below. bazel run //:print_fortune ``` -The `print_fortune` target has a `deps` attribute, a list of other targets that it depends on. It contains `"//fortune"`, a label string referring to the target in the `fortune` directory named `fortune`. +The `print_fortune` target has a `deps` attribute, a list of other targets that +it depends on. It contains `"//fortune"`, a label string referring to the target +in the `fortune` directory named `fortune`. -Bazel requires that all targets declare their dependencies explicitly with attributes like `deps`. This may seem cumbersome since dependencies are *also* specified in source files, but Bazel's explictness gives it an advantage. Bazel builds an [action graph](https://bazel.build/reference/glossary#action-graph) containing all commands, inputs, and outputs before running any commands, without reading any source files. Bazel can then cache action results or send actions for [remote execution](https://bazel.build/remote/rbe) without built-in language-specific logic. +Bazel requires that all targets declare their dependencies explicitly with +attributes like `deps`. This may seem cumbersome since dependencies are *also* +specified in source files, but Bazel's explictness gives it an advantage. Bazel +builds an [action graph](https://bazel.build/reference/glossary#action-graph) +containing all commands, inputs, and outputs before running any commands, +without reading any source files. Bazel can then cache action results or send +actions for [remote execution](https://bazel.build/remote/rbe) without built-in +language-specific logic. ### Understanding labels -A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses to identify a target or a file. Labels are used in command line arguments and in `BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, `//:print-fortune`, and `@rules_go//go:def.bzl`. - -A label has three parts: a repository name, a package name, and a target (or file) name. - -The repository name is written between `@` and `//` and is used to refer to a target from a different Bazel module (for historical reasons, *module* and *repository* are sometimes used synonymously). In the label, `@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name can be omitted when referring to targets in the same repository. - -The package name is written between `//` and `:` and is used to refer to a target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, the package name is `go`. A Bazel [package](https://bazel.build/reference/glossary#package) is a set of files and targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. Its package name is a slash-separated path from the module root directory (containing `MODULE.bazel`) to the directory containing the `BUILD` file. A package may include subdirectories, but only if they don't also contain `BUILD` files defining their own packages. - -Most Go projects have one `BUILD` file per directory and one Go package per `BUILD` file. The package name in a label may be omitted when referring to targets in the same directory. - -The target name is written after `:` and refers to a target within a package. The target name may be omitted if it's the same as the last component of the package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is the same as `//fortune`). - -On the command-line, you can use `...` as a wildcard to refer to all the targets within a package. This is useful for building or testing all the targets in a repository. +A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses +to identify a target or a file. Labels are used in command line arguments and in +`BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, +`//:print-fortune`, and `@rules_go//go:def.bzl`. + +A label has three parts: a repository name, a package name, and a target (or +file) name. + +The repository name is written between `@` and `//` and is used to refer to a +target from a different Bazel module (for historical reasons, *module* and +*repository* are sometimes used synonymously). In the label, +`@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name +can be omitted when referring to targets in the same repository. + +The package name is written between `//` and `:` and is used to refer to a +target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, +the package name is `go`. A Bazel +[package](https://bazel.build/reference/glossary#package) is a set of files and +targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. +Its package name is a slash-separated path from the module root directory +(containing `MODULE.bazel`) to the directory containing the `BUILD` file. A +package may include subdirectories, but only if they don't also contain `BUILD` +files defining their own packages. + +Most Go projects have one `BUILD` file per directory and one Go package per +`BUILD` file. The package name in a label may be omitted when referring to +targets in the same directory. + +The target name is written after `:` and refers to a target within a package. +The target name may be omitted if it's the same as the last component of the +package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is +the same as `//fortune`). + +On the command-line, you can use `...` as a wildcard to refer to all the targets +within a package. This is useful for building or testing all the targets in a +repository. ```posix-shell # Build everything @@ -276,13 +372,15 @@ import ( // TestGet checks that Get returns one of the strings from fortunes. func TestGet(t *testing.T) { msg := Get() - if i := slices.Index(fortunes, msg); i < 0 { + if i := slices.Index(fortunes, msg); i < 0 { t.Errorf("Get returned %q, not one the expected messages", msg) } } ``` -This file uses the unexported `fortunes` variable, so it needs to be compiled into the same Go package as `fortune.go`. Look at the `BUILD` file to see how that works: +This file uses the unexported `fortunes` variable, so it needs to be compiled +into the same Go package as `fortune.go`. Look at the `BUILD` file to see +how that works: ```bazel load("@rules_go//go:def.bzl", "go_library", "go_test") @@ -301,9 +399,19 @@ go_test( ) ``` -We have a new `fortune_test` target that uses the `go_test` rule to compile and link a test executable. `go_test` needs to compile `fortune.go` and `fortune_test.go` together with the same command, so we use the `embed` attribute here to incorporate the attributes of the `fortune` target into `fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, but it also works with `go_library`, which is sometimes useful for generated code. +We have a new `fortune_test` target that uses the `go_test` rule to compile and +link a test executable. `go_test` needs to compile `fortune.go` and +`fortune_test.go` together with the same command, so we use the `embed` +attribute here to incorporate the attributes of the `fortune` target into +`fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, +but it also works with `go_library`, which is sometimes useful for generated +code. -You may be wondering if the `embed` attribute is related to Go's [`embed`](https://pkg.go.dev/embed) package, which is used to access data files copied into an executable. This is an unfortunate name collision: rules\_go's `embed` attribute was introduced before Go's `embed` package. Instead, rules\_go uses the `embedsrcs` to list files that can be loaded with the `embed` package. +You may be wondering if the `embed` attribute is related to Go's +[`embed`](https://pkg.go.dev/embed) package, which is used to access data files +copied into an executable. This is an unfortunate name collision: rules_go's +`embed` attribute was introduced before Go's `embed` package. Instead, rules_go +uses the `embedsrcs` to list files that can be loaded with the `embed` package. Try running our test with `bazel test`: @@ -322,7 +430,9 @@ Executed 0 out of 1 test: 1 test passes. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. ``` -You can use the `...` wildcard to run all tests. Bazel will also build targets that aren't tests, so this can catch compile errors even in packages that don't have tests. +You can use the `...` wildcard to run all tests. Bazel will also build targets +that aren't tests, so this can catch compile errors even in packages that don't +have tests. ```posix-shell $ bazel test //... @@ -330,9 +440,21 @@ $ bazel test //... ## Conclusion and further reading -In this tutorial, we built and tested a small Go project with Bazel, and we learned some core Bazel concepts along the way. - -- To get started building other applications with Bazel, see the tutorials for [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and [iOS](/start/ios-app). -- You can also check the list of [recommended rules](/rules) for other languages. -- For more information on Go, see the [rules\_go](https://github.com/bazel-contrib/rules_go) module, especially the [Core Go rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) documentation. -- To learn more about working with Bazel modules outside your project, see [external dependencies](/docs/external). In particular, for information on how to depend on Go modules and toolchains through Bazel's module system, see [Go with bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). +In this tutorial, we built and tested a small Go project with Bazel, and we +learned some core Bazel concepts along the way. + +- To get started building other applications with Bazel, see the tutorials for + [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and + [iOS](/start/ios-app). +- You can also check the list of [recommended rules](/rules) for other + languages. +- For more information on Go, see the + [rules_go](https://github.com/bazel-contrib/rules_go) module, especially the + [Core Go + rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) + documentation. +- To learn more about working with Bazel modules outside your project, see + [external dependencies](/docs/external). In particular, for information on + how to depend on Go modules and toolchains through Bazel's module system, + see [Go with + bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). diff --git a/start/ios-app.mdx b/start/ios-app.mdx index 5004bbd2..0b860ab6 100644 --- a/start/ios-app.mdx +++ b/start/ios-app.mdx @@ -2,4 +2,5 @@ title: 'Bazel Tutorial: Build an iOS App' --- -This tutorial has been moved into the [bazelbuild/rules\_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. + +This tutorial has been moved into the [bazelbuild/rules_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. diff --git a/start/java.mdx b/start/java.mdx index edb9bc0d..b892917d 100644 --- a/start/java.mdx +++ b/start/java.mdx @@ -2,7 +2,11 @@ title: 'Bazel Tutorial: Build a Java Project' --- -This tutorial covers the basics of building Java applications with Bazel. You will set up your workspace and build a simple Java project that illustrates key Bazel concepts, such as targets and `BUILD` files. + + +This tutorial covers the basics of building Java applications with +Bazel. You will set up your workspace and build a simple Java project that +illustrates key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes. @@ -10,40 +14,36 @@ Estimated completion time: 30 minutes. In this tutorial you learn how to: -- Build a target -- Visualize the project's dependencies -- Split the project into multiple targets and packages -- Control target visibility across packages -- Reference targets through labels -- Deploy a target +* Build a target +* Visualize the project's dependencies +* Split the project into multiple targets and packages +* Control target visibility across packages +* Reference targets through labels +* Deploy a target ## Before you begin ### Install Bazel -To prepare for the tutorial, first [Install Bazel](/install) if you don't have it installed already. +To prepare for the tutorial, first [Install Bazel](/install) if +you don't have it installed already. ### Install the JDK -1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). - -2. Set the JAVA\_HOME environment variable to point to the JDK. - - - On Linux/macOS: - - ``` - export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" - ``` +1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). - - On Windows: +2. Set the JAVA\_HOME environment variable to point to the JDK. + * On Linux/macOS: - 1. Open Control Panel. - 2. Go to "System and Security" \> "System" \> "Advanced System Settings" \> "Advanced" tab \> "Environment Variables..." . - 3. Under the "User variables" list (the one on the top), click "New\...". - 4. In the "Variable name" field, enter `JAVA_HOME`. - 5. Click "Browse Directory...". - 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). - 7. Click "OK" on all dialog windows. + export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" + * On Windows: + 1. Open Control Panel. + 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . + 3. Under the "User variables" list (the one on the top), click "New...". + 4. In the "Variable name" field, enter `JAVA_HOME`. + 5. Click "Browse Directory...". + 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). + 7. Click "OK" on all dialog windows. ### Get the sample project @@ -53,7 +53,8 @@ Retrieve the sample project from Bazel's GitHub repository: git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/java-tutorial` directory and is structured as follows: +The sample project for this tutorial is in the `examples/java-tutorial` +directory and is structured as follows: ``` java-tutorial @@ -75,19 +76,32 @@ java-tutorial ### Set up the workspace -Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains files that Bazel recognizes as special: +Before you can build a project, you need to set up its workspace. A workspace is +a directory that holds your project's source files and Bazel's build outputs. It +also contains files that Bazel recognizes as special: -- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure, +* The `MODULE.bazel` file, which identifies the directory and its contents as a + Bazel workspace and lives at the root of the project's directory structure, -- One or more `BUILD` files, which tell Bazel how to build different parts of the project. (A directory within the workspace that contains a `BUILD` file is a *package*. You will learn about packages later in this tutorial.) +* One or more `BUILD` files, which tell Bazel how to build different parts of + the project. (A directory within the workspace that contains a `BUILD` file + is a *package*. You will learn about packages later in this tutorial.) -To designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. +To designate a directory as a Bazel workspace, create an empty file named +`MODULE.bazel` in that directory. -When Bazel builds the project, all inputs and dependencies must be in the same workspace. Files residing in different workspaces are independent of one another unless linked, which is beyond the scope of this tutorial. +When Bazel builds the project, all inputs and dependencies must be in the same +workspace. Files residing in different workspaces are independent of one +another unless linked, which is beyond the scope of this tutorial. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. The most important type is the *build rule*, which tells Bazel how to build the desired outputs, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a *target* and points to a specific set of source files and dependencies. A target can also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. +The most important type is the *build rule*, which tells Bazel how to build the +desired outputs, such as executable binaries or libraries. Each instance +of a build rule in the `BUILD` file is called a *target* and points to a +specific set of source files and dependencies. A target can also point to other +targets. Take a look at the `java-tutorial/BUILD` file: @@ -98,19 +112,30 @@ java_binary( ) ``` -In our example, the `ProjectRunner` target instantiates Bazel's built-in [`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to build a `.jar` file and a wrapper shell script (both named after the target). +In our example, the `ProjectRunner` target instantiates Bazel's built-in +[`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to +build a `.jar` file and a wrapper shell script (both named after the target). -The attributes in the target explicitly state its dependencies and options. While the `name` attribute is mandatory, many are optional. For example, in the `ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies the source files that Bazel uses to build the target, and `main_class` specifies the class that contains the main method. (You may have noticed that our example uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel instead of listing them one by one.) +The attributes in the target explicitly state its dependencies and options. +While the `name` attribute is mandatory, many are optional. For example, in the +`ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies +the source files that Bazel uses to build the target, and `main_class` specifies +the class that contains the main method. (You may have noticed that our example +uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel +instead of listing them one by one.) ### Build the project -To build your sample project, navigate to the `java-tutorial` directory and run: +To build your sample project, navigate to the `java-tutorial` directory +and run: ```posix-terminal bazel build //:ProjectRunner ``` - -In the target label, the `//` part is the location of the `BUILD` file relative to the root of the workspace (in this case, the root itself), and `ProjectRunner` is the target name in the `BUILD` file. (You will learn about target labels in more detail at the end of this tutorial.) +In the target label, the `//` part is the location of the `BUILD` file +relative to the root of the workspace (in this case, the root itself), +and `ProjectRunner` is the target name in the `BUILD` file. (You will +learn about target labels in more detail at the end of this tutorial.) Bazel produces output similar to the following: @@ -122,7 +147,9 @@ Bazel produces output similar to the following: INFO: Elapsed time: 1.021s, Critical Path: 0.83s ``` -Congratulations, you just built your first Bazel target! Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Browse through its contents to get an idea for Bazel's output structure. +Congratulations, you just built your first Bazel target! Bazel places build +outputs in the `bazel-bin` directory at the root of the workspace. Browse +through its contents to get an idea for Bazel's output structure. Now test your freshly built binary: @@ -132,31 +159,43 @@ bazel-bin/ProjectRunner ### Review the dependency graph -Bazel requires build dependencies to be explicitly declared in BUILD files. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. +Bazel requires build dependencies to be explicitly declared in BUILD files. +Bazel uses those statements to create the project's dependency graph, which +enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: +To visualize the sample project's dependencies, you can generate a text +representation of the dependency graph by running this command at the +workspace root: ```posix-terminal bazel query --notool_deps --noimplicit_deps "deps(//:ProjectRunner)" --output graph ``` -The above command tells Bazel to look for all dependencies for the target `//:ProjectRunner` (excluding host and implicit dependencies) and format the output as a graph. +The above command tells Bazel to look for all dependencies for the target +`//:ProjectRunner` (excluding host and implicit dependencies) and format the +output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -As you can see, the project has a single target that build two source files with no additional dependencies: +As you can see, the project has a single target that build two source files with +no additional dependencies: ![Dependency graph of the target 'ProjectRunner'](/docs/images/tutorial_java_01.svg) -After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its +dependencies, then you can add some complexity. ## Refine your Bazel build -While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages to allow for fast incremental builds (that is, only rebuild what's changed) and to speed up your builds by building multiple parts of a project at once. +While a single target is sufficient for small projects, you may want to split +larger projects into multiple targets and packages to allow for fast incremental +builds (that is, only rebuild what's changed) and to speed up your builds by +building multiple parts of a project at once. ### Specify multiple build targets -You can split the sample project build into two targets. Replace the contents of the `java-tutorial/BUILD` file with the following: +You can split the sample project build into two targets. Replace the contents of +the `java-tutorial/BUILD` file with the following: ```python java_binary( @@ -172,7 +211,9 @@ java_library( ) ``` -With this configuration, Bazel first builds the `greeter` library, then the `ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that the `greeter` library is required to build the `ProjectRunner` binary. +With this configuration, Bazel first builds the `greeter` library, then the +`ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that +the `greeter` library is required to build the `ProjectRunner` binary. To build this new version of the project, run the following command: @@ -196,17 +237,26 @@ Now test your freshly built binary: bazel-bin/ProjectRunner ``` -If you now modify `ProjectRunner.java` and rebuild the project, Bazel only recompiles that file. +If you now modify `ProjectRunner.java` and rebuild the project, Bazel only +recompiles that file. -Looking at the dependency graph, you can see that `ProjectRunner` depends on the same inputs as it did before, but the structure of the build is different: +Looking at the dependency graph, you can see that `ProjectRunner` depends on the +same inputs as it did before, but the structure of the build is different: -![Dependency graph of the target 'ProjectRunner' after adding a dependency](/docs/images/tutorial_java_02.svg) +![Dependency graph of the target 'ProjectRunner' after adding a dependency]( +/docs/images/tutorial_java_02.svg) -You've now built the project with two targets. The `ProjectRunner` target builds one source files and depends on one other target (`:greeter`), which builds one additional source file. +You've now built the project with two targets. The `ProjectRunner` target builds +one source files and depends on one other target (`:greeter`), which builds +one additional source file. ### Use multiple packages -Let’s now split the project into multiple packages. If you take a look at the `src/main/java/com/example/cmdline` directory, you can see that it also contains a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now contains two packages, `//src/main/java/com/example/cmdline` and `//` (since there is a `BUILD` file at the root of the workspace). +Let’s now split the project into multiple packages. If you take a look at the +`src/main/java/com/example/cmdline` directory, you can see that it also contains +a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now +contains two packages, `//src/main/java/com/example/cmdline` and `//` (since +there is a `BUILD` file at the root of the workspace). Take a look at the `src/main/java/com/example/cmdline/BUILD` file: @@ -219,13 +269,21 @@ java_binary( ) ``` -The `runner` target depends on the `greeter` target in the `//` package (hence the target label `//:greeter`) - Bazel knows this through the `deps` attribute. Take a look at the dependency graph: +The `runner` target depends on the `greeter` target in the `//` package (hence +the target label `//:greeter`) - Bazel knows this through the `deps` attribute. +Take a look at the dependency graph: ![Dependency graph of the target 'runner'](/docs/images/tutorial_java_03.svg) -However, for the build to succeed, you must explicitly give the `runner` target in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in `//BUILD` using the `visibility` attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. (Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs.) +However, for the build to succeed, you must explicitly give the `runner` target +in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in +`//BUILD` using the `visibility` attribute. This is because by default targets +are only visible to other targets in the same `BUILD` file. (Bazel uses target +visibility to prevent issues such as libraries containing implementation details +leaking into public APIs.) -To do this, add the `visibility` attribute to the `greeter` target in `java-tutorial/BUILD` as shown below: +To do this, add the `visibility` attribute to the `greeter` target in +`java-tutorial/BUILD` as shown below: ```python java_library( @@ -235,7 +293,8 @@ java_library( ) ``` -Now you can build the new package by running the following command at the root of the workspace: +Now you can build the new package by running the following command at the root +of the workspace: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner @@ -257,29 +316,48 @@ Now test your freshly built binary: ./bazel-bin/src/main/java/com/example/cmdline/runner ``` -You've now modified the project to build as two packages, each containing one target, and understand the dependencies between them. +You've now modified the project to build as two packages, each containing one +target, and understand the dependencies between them. + ## Use labels to reference targets -In `BUILD` files and at the command line, Bazel uses target labels to reference targets - for example, `//:ProjectRunner` or `//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: +In `BUILD` files and at the command line, Bazel uses target labels to reference +targets - for example, `//:ProjectRunner` or +`//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path. +If the target is a rule target, then `path/to/package` is the path to the +directory containing the `BUILD` file, and `target-name` is what you named the +target in the `BUILD` file (the `name` attribute). If the target is a file +target, then `path/to/package` is the path to the root of the package, and +`target-name` is the name of the target file, including its full path. -When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. +When referencing targets at the repository root, the package path is empty, +just use `//:target-name`. When referencing targets within the same `BUILD` +file, you can even skip the `//` workspace root identifier and just use +`:target-name`. -For example, for targets in the `java-tutorial/BUILD` file, you did not have to specify a package path, since the workspace root is itself a package (`//`), and your two target labels were simply `//:ProjectRunner` and `//:greeter`. +For example, for targets in the `java-tutorial/BUILD` file, you did not have to +specify a package path, since the workspace root is itself a package (`//`), and +your two target labels were simply `//:ProjectRunner` and `//:greeter`. -However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you had to specify the full package path of `//src/main/java/com/example/cmdline` and your target label was `//src/main/java/com/example/cmdline:runner`. +However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you +had to specify the full package path of `//src/main/java/com/example/cmdline` +and your target label was `//src/main/java/com/example/cmdline:runner`. ## Package a Java target for deployment -Let’s now package a Java target for deployment by building the binary with all of its runtime dependencies. This lets you run the binary outside of your development environment. +Let’s now package a Java target for deployment by building the binary with all +of its runtime dependencies. This lets you run the binary outside of your +development environment. -As you remember, the [java\_binary](/reference/be/java#java_binary) build rule produces a `.jar` and a wrapper shell script. Take a look at the contents of `runner.jar` using this command: +As you remember, the [java_binary](/reference/be/java#java_binary) build rule +produces a `.jar` and a wrapper shell script. Take a look at the contents of +`runner.jar` using this command: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner.jar @@ -295,8 +373,12 @@ com/example/ com/example/cmdline/ com/example/cmdline/Runner.class ``` - -As you can see, `runner.jar` contains `Runner.class`, but not its dependency, `Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` to the classpath, so if you leave it like this, it will run locally, but it won't run standalone on another machine. Fortunately, the `java_binary` rule allows you to build a self-contained, deployable binary. To build it, append `_deploy.jar` to the target name: +As you can see, `runner.jar` contains `Runner.class`, but not its dependency, +`Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` +to the classpath, so if you leave it like this, it will run locally, but it +won't run standalone on another machine. Fortunately, the `java_binary` rule +allows you to build a self-contained, deployable binary. To build it, append +`_deploy.jar` to the target name: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner_deploy.jar @@ -310,8 +392,10 @@ Target //src/main/java/com/example/cmdline:runner_deploy.jar up-to-date: bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar INFO: Elapsed time: 1.700s, Critical Path: 0.23s ``` - -You have just built `runner_deploy.jar`, which you can run standalone away from your development environment since it contains the required runtime dependencies. Take a look at the contents of this standalone JAR using the same command as before: +You have just built `runner_deploy.jar`, which you can run standalone away from +your development environment since it contains the required runtime +dependencies. Take a look at the contents of this standalone JAR using the +same command as before: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar @@ -334,14 +418,19 @@ com/example/Greeting.class For more details, see: -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) for rules to manage transitive Maven dependencies. +* [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) for + rules to manage transitive Maven dependencies. -- [External Dependencies](/docs/external) to learn more about working with local and remote repositories. +* [External Dependencies](/docs/external) to learn more about working with + local and remote repositories. -- The [other rules](/rules) to learn more about Bazel. +* The [other rules](/rules) to learn more about Bazel. -- The [C++ build tutorial](/start/cpp) to get started with building C++ projects with Bazel. +* The [C++ build tutorial](/start/cpp) to get started with building + C++ projects with Bazel. -- The [Android application tutorial](/start/android-app) and [iOS application tutorial](/start/ios-app)) to get started with building mobile applications for Android and iOS with Bazel. +* The [Android application tutorial](/start/android-app ) and + [iOS application tutorial](/start/ios-app)) to get started with + building mobile applications for Android and iOS with Bazel. Happy building! diff --git a/tutorials/ccp-toolchain-config.mdx b/tutorials/ccp-toolchain-config.mdx index 3595614f..0b14cb55 100644 --- a/tutorials/ccp-toolchain-config.mdx +++ b/tutorials/ccp-toolchain-config.mdx @@ -2,407 +2,473 @@ title: 'Bazel Tutorial: Configure C++ Toolchains' --- -This tutorial uses an example scenario to describe how to configure C++ toolchains for a project. -## What you'll learn -In this tutorial you learn how to: - -- Set up the build environment -- Use `--toolchain_resolution_debug` to debug toolchain resolution -- Configure the C++ toolchain -- Create a Starlark rule that provides additional configuration for the `cc_toolchain` so that Bazel can build the application with `clang` -- Build the C++ binary by running `bazel build //main:hello-world` on a Linux machine -- Cross-compile the binary for android by running `bazel build //main:hello-world --platforms=//:android_x86_64` - -## Before you begin - -This tutorial assumes you are on Linux and have successfully built C++ applications and installed the appropriate tooling and libraries. The tutorial uses `clang version 19`, which you can install on your system. - -### Set up the build environment - -Set up your build environment as follows: - -1. If you have not already done so, [download and install Bazel 7.0.2](https://bazel.build/install) or later. +This tutorial uses an example scenario to describe how to configure C++ +toolchains for a project. -2. Add an empty `MODULE.bazel` file at the root folder. +## What you'll learn -3. Add the following `cc_binary` target to the `main/BUILD` file: - - ```python - cc_binary( - name = "hello-world", - srcs = ["hello-world.cc"], - ) - ``` - - Because Bazel uses many internal tools written in C++ during the build, such as `process-wrapper`, the pre-existing default C++ toolchain is specified for the host platform. This enables these internal tools to build using that toolchain of the one created in this tutorial. Hence, the `cc_binary` target is also built with the default toolchain. - -4. Run the build with the following command: +In this tutorial you learn how to: - ```bash - bazel build //main:hello-world - ``` +* Set up the build environment +* Use `--toolchain_resolution_debug` to debug toolchain resolution +* Configure the C++ toolchain +* Create a Starlark rule that provides additional configuration for the + `cc_toolchain` so that Bazel can build the application with `clang` +* Build the C++ binary by running `bazel build //main:hello-world` on a + Linux machine +* Cross-compile the binary for android by running `bazel build + //main:hello-world --platforms=//:android_x86_64` - The build succeeds without any toolchain registered in `MODULE.bazel`. +## Before you begin - To further see what's under the hood, run: +This tutorial assumes you are on Linux and have successfully built C++ +applications and installed the appropriate tooling and libraries. The tutorial +uses `clang version 19`, which you can install on your system. - ```bash - bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' +### Set up the build environment - INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 - ``` +Set up your build environment as follows: - Without specifying `--platforms`, Bazel builds the target for `@platforms//host` using `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. +1. If you have not already done so, [download and install Bazel + 7.0.2](https://bazel.build/install) or later. + +2. Add an empty `MODULE.bazel` file at the root folder. -## Configure the C++ toolchain +3. Add the following `cc_binary` target to the `main/BUILD` file: -To configure the C++ toolchain, repeatedly build the application and eliminate each error one by one as described as following. + ```python + cc_binary( + name = "hello-world", + srcs = ["hello-world.cc"], + ) + ``` -Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` flag to enable C++ toolchain resolution. + Because Bazel uses many internal tools written in C++ during the build, such + as `process-wrapper`, the pre-existing default C++ toolchain is specified + for the host platform. This enables these internal tools to build using that + toolchain of the one created in this tutorial. Hence, the `cc_binary` target + is also built with the default toolchain. -It also assumes `clang version 9.0.1`, although the details should only change slightly between different versions of clang. +4. Run the build with the following command: -1. Add `toolchain/BUILD` with + ```bash + bazel build //main:hello-world + ``` - ```python - filegroup(name = "empty") + The build succeeds without any toolchain registered in `MODULE.bazel`. - cc_toolchain( - name = "linux_x86_64_toolchain", - toolchain_identifier = "linux_x86_64-toolchain", - toolchain_config = ":linux_x86_64_toolchain_config", - all_files = ":empty", - compiler_files = ":empty", - dwp_files = ":empty", - linker_files = ":empty", - objcopy_files = ":empty", - strip_files = ":empty", - supports_param_files = 0, - ) + To further see what's under the hood, run: - toolchain( - name = "cc_toolchain_for_linux_x86_64", - toolchain = ":linux_x86_64_toolchain", - toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", - exec_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - target_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - ) - ``` - - Then add appropriate dependencies and register the toolchain with `MODULE.bazel` with - - ```python - bazel_dep(name = "platforms", version = "0.0.10") - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64" - ) - ``` - - This step defines a `cc_toolchain` and binds it to a `toolchain` target for the host configuration. - -2. Run the build again. Because the `toolchain` package doesn't yet define the `linux_x86_64_toolchain_config` target, Bazel throws the following error: - - ```bash - ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. - ``` - -3. In the `toolchain/BUILD` file, define an empty filegroup as follows: - - ```python - package(default_visibility = ["//visibility:public"]) - - filegroup(name = "linux_x86_64_toolchain_config") - ``` - -4. Run the build again. Bazel throws the following error: - - ```bash - '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. - ``` - - `CcToolchainConfigInfo` is a provider that you use to configure your C++ toolchains. To fix this error, create a Starlark rule that provides `CcToolchainConfigInfo` to Bazel by making a `toolchain/cc_toolchain_config.bzl` file with the following content: - - ```python - def _impl(ctx): - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "k8-toolchain", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - `cc_common.create_cc_toolchain_config_info()` creates the needed provider `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load statement to `toolchain/BUILD` right below the package statement: - - ```python - load(":cc_toolchain_config.bzl", "cc_toolchain_config") - ``` - - And replace the "linux\_x86\_64\_toolchain\_config" filegroup with a declaration of a `cc_toolchain_config` rule: - - ```python - cc_toolchain_config(name = "linux_x86_64_toolchain_config") - ``` - -5. Run the build again. Bazel throws the following error: - - ```bash - .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) - src/main/tools/linux-sandbox-pid1.cc:421: - "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory - Target //:hello-world failed to build` - ``` - - At this point, Bazel has enough information to attempt building the code but it still does not know what tools to use to complete the required build actions. You will modify the Starlark rule implementation to tell Bazel what tools to use. For that, you need the `tool_path()` constructor from [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): - - ```python - # toolchain/cc_toolchain_config.bzl: - # NEW - load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") - - def _impl(ctx): - tool_paths = [ # NEW - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/usr/bin/ar", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, # NEW - ) - ``` - - Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for your system. Note that the compiler is referenced by the name "gcc" for historic reasons. - -6. Run the build again. Bazel throws the following error: - - ```bash - ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': - the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): - '/usr/include/c++/13/ctime' - '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' - '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' - ... - ``` - - Bazel needs to know where to search for included headers. There are multiple ways to solve this, such as using the `includes` attribute of `cc_binary`, but here this is solved at the toolchain level with the [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you are using a different version of `clang`, the include path will be different. These paths may also be different depending on the distribution. - - Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like this: - - ```python - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - cxx_builtin_include_directories = [ # NEW - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - ``` - -7. Run the build command again, you will see an error like: - - ```bash - /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': - hello-world.cc:(.text+0x68): undefined reference to `std::cout' - ``` - - The reason for this is because the linker is missing the C++ standard library and it can't find its symbols. There are many ways to solve this, such as using the `linkopts` attribute of `cc_binary`. Here it is solved by making sure that any target using the toolchain doesn't have to specify this flag. - - Copy the following code to `toolchain/cc_toolchain_config.bzl`: - - ```python - # NEW - load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") - # NEW - load( - "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", - "feature", # NEW - "flag_group", # NEW - "flag_set", # NEW - "tool_path", - ) - - all_link_actions = [ # NEW - ACTION_NAMES.cpp_link_executable, - ACTION_NAMES.cpp_link_dynamic_library, - ACTION_NAMES.cpp_link_nodeps_dynamic_library, - ] - - def _impl(ctx): - tool_paths = [ - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/bin/false", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - features = [ # NEW - feature( - name = "default_linker_flags", - enabled = True, - flag_sets = [ - flag_set( - actions = all_link_actions, - flag_groups = ([ - flag_group( - flags = [ - "-lstdc++", - ], - ), - ]), - ), - ], - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - features = features, # NEW - cxx_builtin_include_directories = [ - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - Note that this code uses the GNU C++ library libstdc++. If you want to use the LLVM C++ library, use "-lc++" instead of "-lstdc++". - -8. Running `bazel build //main:hello-world`, it should finally build the binary successfully for host. - -9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in target names. - - In `MODULE.bazel`, register the toolchain for android - - ```python - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64", - "//toolchain:cc_toolchain_for_android_x86_64" - ) - ``` - -10. Run `bazel build //main:hello-world --android_platforms=//toolchain:android_x86_64` to build the binary for Android. - -In practice, Linux and Android should have different C++ toolchain configs. You can either modify the existing `cc_toolchain_config` for the differences or create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate platforms. - -## Review your work - -In this tutorial you learned how to configure a basic C++ toolchain, but toolchains are more powerful than this example. + ```bash + bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' + + INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 + ``` + + Without specifying `--platforms`, Bazel builds the target for + `@platforms//host` using + `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. + +## Configure the C++ toolchain + +To configure the C++ toolchain, repeatedly build the application and eliminate +each error one by one as described as following. + +Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using +an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` +flag to enable C++ toolchain resolution. + +It also assumes `clang version 9.0.1`, although the details should only change +slightly between different versions of clang. + +1. Add `toolchain/BUILD` with + + ```python + filegroup(name = "empty") + + cc_toolchain( + name = "linux_x86_64_toolchain", + toolchain_identifier = "linux_x86_64-toolchain", + toolchain_config = ":linux_x86_64_toolchain_config", + all_files = ":empty", + compiler_files = ":empty", + dwp_files = ":empty", + linker_files = ":empty", + objcopy_files = ":empty", + strip_files = ":empty", + supports_param_files = 0, + ) + + toolchain( + name = "cc_toolchain_for_linux_x86_64", + toolchain = ":linux_x86_64_toolchain", + toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", + exec_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + ) + ``` + + Then add appropriate dependencies and register the toolchain with + `MODULE.bazel` with + + ```python + bazel_dep(name = "platforms", version = "0.0.10") + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64" + ) + ``` + + This step defines a `cc_toolchain` and binds it to a `toolchain` target for + the host configuration. + +2. Run the build again. Because the `toolchain` package doesn't yet define the + `linux_x86_64_toolchain_config` target, Bazel throws the following error: + + ```bash + ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. + ``` + +3. In the `toolchain/BUILD` file, define an empty filegroup as follows: + + ```python + package(default_visibility = ["//visibility:public"]) + + filegroup(name = "linux_x86_64_toolchain_config") + ``` + +4. Run the build again. Bazel throws the following error: + + ```bash + '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. + ``` + + `CcToolchainConfigInfo` is a provider that you use to configure your C++ + toolchains. To fix this error, create a Starlark rule that provides + `CcToolchainConfigInfo` to Bazel by making a + `toolchain/cc_toolchain_config.bzl` file with the following content: + + ```python + def _impl(ctx): + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "k8-toolchain", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + `cc_common.create_cc_toolchain_config_info()` creates the needed provider + `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load + statement to `toolchain/BUILD` right below the package statement: + + ```python + load(":cc_toolchain_config.bzl", "cc_toolchain_config") + ``` + + And replace the "linux_x86_64_toolchain_config" filegroup with a declaration + of a `cc_toolchain_config` rule: + + ```python + cc_toolchain_config(name = "linux_x86_64_toolchain_config") + ``` + +5. Run the build again. Bazel throws the following error: + + ```bash + .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) + src/main/tools/linux-sandbox-pid1.cc:421: + "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory + Target //:hello-world failed to build` + ``` + + At this point, Bazel has enough information to attempt building the code but + it still does not know what tools to use to complete the required build + actions. You will modify the Starlark rule implementation to tell Bazel what + tools to use. For that, you need the `tool_path()` constructor from + [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): + + ```python + # toolchain/cc_toolchain_config.bzl: + # NEW + load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") + + def _impl(ctx): + tool_paths = [ # NEW + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/usr/bin/ar", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, # NEW + ) + ``` + + Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for + your system. Note that the compiler is referenced by the name "gcc" for + historic reasons. + +6. Run the build again. Bazel throws the following error: + + ```bash + ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': + the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): + '/usr/include/c++/13/ctime' + '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' + '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' + ... + ``` + + Bazel needs to know where to search for included headers. There are multiple + ways to solve this, such as using the `includes` attribute of `cc_binary`, + but here this is solved at the toolchain level with the + [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) + parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you + are using a different version of `clang`, the include path will be + different. These paths may also be different depending on the distribution. + + Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like + this: + + ```python + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + cxx_builtin_include_directories = [ # NEW + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + ``` + +7. Run the build command again, you will see an error like: + + ```bash + /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': + hello-world.cc:(.text+0x68): undefined reference to `std::cout' + ``` + + The reason for this is because the linker is missing the C++ standard + library and it can't find its symbols. There are many ways to solve this, + such as using the `linkopts` attribute of `cc_binary`. Here it is solved by + making sure that any target using the toolchain doesn't have to specify this + flag. + + Copy the following code to `toolchain/cc_toolchain_config.bzl`: + + ```python + # NEW + load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") + # NEW + load( + "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", + "feature", # NEW + "flag_group", # NEW + "flag_set", # NEW + "tool_path", + ) + + all_link_actions = [ # NEW + ACTION_NAMES.cpp_link_executable, + ACTION_NAMES.cpp_link_dynamic_library, + ACTION_NAMES.cpp_link_nodeps_dynamic_library, + ] + + def _impl(ctx): + tool_paths = [ + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/bin/false", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + features = [ # NEW + feature( + name = "default_linker_flags", + enabled = True, + flag_sets = [ + flag_set( + actions = all_link_actions, + flag_groups = ([ + flag_group( + flags = [ + "-lstdc++", + ], + ), + ]), + ), + ], + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + features = features, # NEW + cxx_builtin_include_directories = [ + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + Note that this code uses the GNU C++ library libstdc++. If you want to use + the LLVM C++ library, use "-lc++" instead of "-lstdc++". + +8. Running `bazel build //main:hello-world`, it should finally build the binary + successfully for host. + +9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and + `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in + target names. + + In `MODULE.bazel`, register the toolchain for android + + ```python + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64", + "//toolchain:cc_toolchain_for_android_x86_64" + ) + ``` + +10. Run `bazel build //main:hello-world + --android_platforms=//toolchain:android_x86_64` to build the binary for + Android. + +In practice, Linux and Android should have different C++ toolchain configs. You +can either modify the existing `cc_toolchain_config` for the differences or +create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate +platforms. + +## Review your work + +In this tutorial you learned how to configure a basic C++ toolchain, but +toolchains are more powerful than this example. The key takeaways are: -- You need to specify a matching `platforms` flag in the command line for Bazel to resolve to the toolchain for the same constraint values on the platform. The documentation holds more [information about language specific configuration flags](/concepts/platforms). -- You have to let the toolchain know where the tools live. In this tutorial there is a simplified version where you access the tools from the system. If you are interested in a more self-contained approach, you can read about [external dependencies](/external/overview). Your tools could come from a different module and you would have to make their files available to the `cc_toolchain` with target dependencies on attributes, such as `compiler_files`. The `tool_paths` would need to be changed as well. -- You can create features to customize which flags should be passed to different actions, be it linking or any other type of action. - -## Further reading - -For more details, see [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +- You need to specify a matching `platforms` flag in the command line for + Bazel to resolve to the toolchain for the same constraint values on the + platform. The documentation holds more [information about language specific + configuration flags](/concepts/platforms). +- You have to let the toolchain know where the tools live. In this tutorial + there is a simplified version where you access the tools from the system. If + you are interested in a more self-contained approach, you can read about + [external dependencies](/external/overview). Your tools could come from a + different module and you would have to make their files available to the + `cc_toolchain` with target dependencies on attributes, such as + `compiler_files`. The `tool_paths` would need to be changed as well. +- You can create features to customize which flags should be passed to + different actions, be it linking or any other type of action. + +## Further reading + +For more details, see [C++ toolchain +configuration](/docs/cc-toolchain-config-reference) diff --git a/tutorials/cpp-dependency.mdx b/tutorials/cpp-dependency.mdx index 24872bb8..194cc73c 100644 --- a/tutorials/cpp-dependency.mdx +++ b/tutorials/cpp-dependency.mdx @@ -2,36 +2,49 @@ title: 'Review the dependency graph' --- -A successful build has all of its dependencies explicitly stated in the `BUILD` file. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: + +A successful build has all of its dependencies explicitly stated in the `BUILD` +file. Bazel uses those statements to create the project's dependency graph, +which enables accurate incremental builds. + +To visualize the sample project's dependencies, you can generate a text +representation of the dependency graph by running this command at the +workspace root: ``` bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph ``` -The above command tells Bazel to look for all dependencies for the target `//main:hello-world` (excluding host and implicit dependencies) and format the output as a graph. +The above command tells Bazel to look for all dependencies for the target +`//main:hello-world` (excluding host and implicit dependencies) and format the +output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -On Ubuntu, you can view the graph locally by installing GraphViz and the xdot Dot Viewer: +On Ubuntu, you can view the graph locally by installing GraphViz and the xdot +Dot Viewer: ``` sudo apt update && sudo apt install graphviz xdot ``` -Then you can generate and view the graph by piping the text output above straight to xdot: +Then you can generate and view the graph by piping the text output above +straight to xdot: ``` -xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ +xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph) ``` -As you can see, the first stage of the sample project has a single target that builds a single source file with no additional dependencies: +As you can see, the first stage of the sample project has a single target +that builds a single source file with no additional dependencies: ![Dependency graph for 'hello-world'](/docs/images/cpp-tutorial-stage1.png "Dependency graph") -**Figure 1.** Dependency graph for `hello-world` displays a single target with a single source file. +**Figure 1.** Dependency graph for `hello-world` displays a single target with a single +source file. -After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its +dependencies, then you can add some complexity. diff --git a/tutorials/cpp-labels.mdx b/tutorials/cpp-labels.mdx index 17e062aa..78d0dbce 100644 --- a/tutorials/cpp-labels.mdx +++ b/tutorials/cpp-labels.mdx @@ -2,12 +2,26 @@ title: 'Use labels to reference targets' --- -In `BUILD` files and at the command line, Bazel uses *labels* to reference targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax is: + + +In `BUILD` files and at the command line, Bazel uses *labels* to reference +targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax +is: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path from the workspace root (the directory containing the `MODULE.bazel` file) to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path relative to the root of the package (the directory containing the package's `BUILD` file). +If the target is a rule target, then `path/to/package` is the path from the +workspace root (the directory containing the `MODULE.bazel` file) to the directory +containing the `BUILD` file, and `target-name` is what you named the target +in the `BUILD` file (the `name` attribute). If the target is a file target, +then `path/to/package` is the path to the root of the package, and +`target-name` is the name of the target file, including its full +path relative to the root of the package (the directory containing the +package's `BUILD` file). -When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. +When referencing targets at the repository root, the package path is empty, +just use `//:target-name`. When referencing targets within the same `BUILD` +file, you can even skip the `//` workspace root identifier and just use +`:target-name`. diff --git a/tutorials/cpp-use-cases.mdx b/tutorials/cpp-use-cases.mdx index f13e6be0..f25f80b3 100644 --- a/tutorials/cpp-use-cases.mdx +++ b/tutorials/cpp-use-cases.mdx @@ -2,13 +2,21 @@ title: 'Common C++ Build Use Cases' --- -Here you will find some of the most common use cases for building C++ projects with Bazel. If you have not done so already, get started with building C++ projects with Bazel by completing the tutorial [Introduction to Bazel: Build a C++ Project](/start/cpp). -For information on cc\_library and hdrs header files, see [cc\_library](/reference/be/c-cpp#cc_library). + +Here you will find some of the most common use cases for building C++ projects +with Bazel. If you have not done so already, get started with building C++ +projects with Bazel by completing the tutorial +[Introduction to Bazel: Build a C++ Project](/start/cpp). + +For information on cc_library and hdrs header files, see +cc_library. ## Including multiple files in a target -You can include multiple files in a single target with [glob](/reference/be/functions#glob). For example: +You can include multiple files in a single target with +glob. +For example: ```python cc_library( @@ -18,11 +26,19 @@ cc_library( ) ``` -With this target, Bazel will build all the `.cc` and `.h` files it finds in the same directory as the `BUILD` file that contains this target (excluding subdirectories). +With this target, Bazel will build all the `.cc` and `.h` files it finds in the +same directory as the `BUILD` file that contains this target (excluding +subdirectories). ## Using transitive includes -If a file includes a header, then any rule with that file as a source (that is, having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should depend on the included header's library rule. Conversely, only direct dependencies need to be specified as dependencies. For example, suppose `sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` file would look like this: +If a file includes a header, then any rule with that file as a source (that is, +having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should +depend on the included header's library rule. Conversely, only direct +dependencies need to be specified as dependencies. For example, suppose +`sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` +doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` +file would look like this: ```python cc_library( @@ -46,11 +62,15 @@ cc_library( ) ``` -Here, the `sandwich` library depends on the `bread` library, which depends on the `flour` library. +Here, the `sandwich` library depends on the `bread` library, which depends +on the `flour` library. ## Adding include paths -Sometimes you cannot (or do not want to) root include paths at the workspace root. Existing libraries might already have an include directory that doesn't match its path in your workspace. For example, suppose you have the following directory structure: +Sometimes you cannot (or do not want to) root include paths at the workspace +root. Existing libraries might already have an include directory that doesn't +match its path in your workspace. For example, suppose you have the following +directory structure: ``` └── my-project @@ -63,7 +83,11 @@ Sometimes you cannot (or do not want to) root include paths at the workspace roo └── MODULE.bazel ``` -Bazel will expect `some_lib.h` to be included as `legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes `"some_lib.h"`. To make that include path valid, `legacy/some_lib/BUILD` will need to specify that the `some_lib/include` directory is an include directory: +Bazel will expect `some_lib.h` to be included as +`legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes +`"some_lib.h"`. To make that include path valid, +`legacy/some_lib/BUILD` will need to specify that the `some_lib/include` +directory is an include directory: ```python cc_library( @@ -74,11 +98,15 @@ cc_library( ) ``` -This is especially useful for external dependencies, as their header files must otherwise be included with a `/` prefix. +This is especially useful for external dependencies, as their header files +must otherwise be included with a `/` prefix. ## Include external libraries -Suppose you are using [Google Test](https://github.com/google/googletest). You can add a dependency on it in the `MODULE.bazel` file to download Google Test and make it available in your repository: +Suppose you are using [Google Test](https://github.com/google/googletest) +. +You can add a dependency on it in the `MODULE.bazel` file to +download Google Test and make it available in your repository: ```python bazel_dep(name = "googletest", version = "1.15.2") @@ -114,7 +142,8 @@ cc_test( ) ``` -To make `hello-greet` visible to `hello-test`, you must add `"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. +To make `hello-greet` visible to `hello-test`, you must add +`"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. Now you can use `bazel test` to run the test. @@ -134,9 +163,11 @@ INFO: Elapsed time: 4.497s, Critical Path: 2.53s Executed 1 out of 1 tests: 1 test passes. ``` + ## Adding dependencies on precompiled libraries -If you want to use a library of which you only have a compiled version (for example, headers and a `.so` file) wrap it in a `cc_library` rule: +If you want to use a library of which you only have a compiled version (for +example, headers and a `.so` file) wrap it in a `cc_library` rule: ```python cc_library( diff --git a/versions/index.mdx b/versions/index.mdx index 64608bf9..4290e57f 100644 --- a/versions/index.mdx +++ b/versions/index.mdx @@ -2,8 +2,14 @@ title: 'Documentation Versions' --- -The default documentation on this website represents the latest version at HEAD. Each major and minor supported release will have a snapshot of the narrative and reference documentation that follows the lifecycle of Bazel's version support. -To see documentation for stable Bazel versions, use the "Versioned docs" drop-down. -To see documentation for older Bazel versions prior to Feb 2022, go to [docs.bazel.build](https://docs.bazel.build/). +The default documentation on this website represents the latest version at HEAD. +Each major and minor supported release will have a snapshot of the narrative and +reference documentation that follows the lifecycle of Bazel's version support. + +To see documentation for stable Bazel versions, use the "Versioned docs" +drop-down. + +To see documentation for older Bazel versions prior to Feb 2022, go to +[docs.bazel.build](https://docs.bazel.build/). From f71a0cf7b7e885b4a5ef69478e9fd9a26f2c8351 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 18:10:40 -0800 Subject: [PATCH 05/10] remove legacy awk script and refs Co-authored-by: promptless[bot] --- transform-docs.awk | 605 --------------------------------------------- 1 file changed, 605 deletions(-) delete mode 100755 transform-docs.awk diff --git a/transform-docs.awk b/transform-docs.awk deleted file mode 100755 index d55e9c67..00000000 --- a/transform-docs.awk +++ /dev/null @@ -1,605 +0,0 @@ -#!/usr/bin/awk -f -# Transform script for converting Bazel docs to Mintlify MDX format -# Usage: awk -f transform-docs.awk input.md > output.mdx - -# Trim leading/trailing whitespace -function trim(str, s) { - s = str - gsub(/^[ \t\r\n]+/, "", s) - gsub(/[ \t\r\n]+$/, "", s) - return s -} - -# Decode common HTML entities -function html_decode(text, t) { - t = text - gsub(/&/, "&", t) - gsub(/</, "<", t) - gsub(/>/, ">", t) - gsub(/"/, "\"", t) - gsub(/'/, "'", t) - gsub(/ /, " ", t) - return t -} - -# Convert inline HTML tags to Markdown/MDX-friendly syntax -function inline_to_md(text, tmp) { - tmp = text - gsub(//, "`", tmp) - gsub(/<\/code>/, "`", tmp) - gsub(//, "**", tmp) - gsub(/<\/strong>/, "**", tmp) - gsub(//, "*", tmp) - gsub(/<\/em>/, "*", tmp) - gsub(/

/, "", tmp) - gsub(/<\/p>/, "", tmp) - tmp = html_decode(tmp) - return trim(tmp) -} - -# Map PrettyPrint language labels to code fence languages -function map_lang(lang) { - lang = tolower(lang) - if (lang == "py" || lang == "python") return "python" - if (lang == "shell" || lang == "sh" || lang == "bash") return "bash" - if (lang == "java" || lang == "lang-java") return "java" - if (lang == "lang") return "" - return lang -} - -# Emit a Mintlify Callout component -function emit_callout(type, title, body) { - print "" - print body - print "" - print "" -} - -# Convert navigation tables into simple Markdown links -function emit_navigation(text, sanitized, count, hrefs, labels, match_str, href_val, label, prev_label, next_label, output_line, i) { - sanitized = text - gsub(/\n/, " ", sanitized) - gsub(/]*>[^<]*<\/span>/, "", sanitized) - gsub(/<\/?(tr|td|table)>/, "", sanitized) - gsub(/class="[^"]*"/, "", sanitized) - - count = 0 - while (match(sanitized, /]*href="[^"]*"[^>]*>[^<]*<\/a>/)) { - match_str = substr(sanitized, RSTART, RLENGTH) - sanitized = substr(sanitized, RSTART + RLENGTH) - - href_val = "" - if (match(match_str, /href="[^"]*"/)) { - href_val = substr(match_str, RSTART + 6, RLENGTH - 7) - } - - label = match_str - sub(/^[^>]*>/, "", label) - sub(/<\/a>.*$/, "", label) - gsub(/[[:space:]]+/, " ", label) - label = trim(label) - gsub(/arrow_forward/, "", label) - gsub(/arrow_back/, "", label) - - count++ - hrefs[count] = href_val - labels[count] = label - } - - if (count == 0) { - return - } - - prev_label = trim(labels[1]) - if (prev_label !~ /←/) { - prev_label = "← " prev_label - } - - output_line = "[" prev_label "](" hrefs[1] ")" - - if (count > 1 && hrefs[2] != "") { - next_label = trim(labels[2]) - if (next_label !~ /→/) { - next_label = next_label " →" - } - output_line = output_line " · [" next_label "](" hrefs[2] ")" - } - - print "" - print output_line - print "" - - for (i in hrefs) { delete hrefs[i] } - for (i in labels) { delete labels[i] } -} -# Convert compare paragraphs into Mintlify Callouts -function handle_compare(text, type, title_segment, title, body_segment, content) { - type = (index(text, "compare-better") > 0) ? "success" : "warning" - - title_segment = text - sub(/^

/, "", title_segment) - sub(/<\/span>.*/, "", title_segment) - title = inline_to_md(title_segment) - - body_segment = text - sub(/^

[^<]*<\/span>/, "", body_segment) - sub(/<\/p>[[:space:]]*$/, "", body_segment) - gsub(/\n[ \t]*/, " ", body_segment) - sub(/^[[:space:]]*(—|--|-)?[[:space:]]*/, "", body_segment) - content = inline_to_md(body_segment) - - emit_callout(type, title, content) -} - -BEGIN { - in_frontmatter = 0 - first_h1_found = 0 - frontmatter_printed = 0 - before_first_h1 = 1 - in_code_block = 0 - in_pre_block = 0 - meta_index = 0 - capture_compare = 0 - compare_buffer = "" - capture_nav_table = 0 - nav_buffer = "" -} - -# Skip Jekyll front-matter lines -/^Project: \/_project\.yaml$/ { next } -/^Book: \/_book\.yaml$/ { next } - -# Skip Starlark lint directives embedded as comments -/^\{#.*#\}$/ { next } - -# Stash metadata lines before the first H1 so we can emit them as frontmatter -before_first_h1 && /^[A-Za-z0-9_-]+: / { - meta_lines[meta_index++] = $0 - next -} - -# Remove lines that contain only '{% include "_buttons.html" %}' -/^{% include "_buttons\.html" %}$/ { next } - -# Remove lines containing '{% dynamic setvar' -/{% dynamic setvar/ { next } - -# Remove any lines that start with '{%' -/^{%/ { next } - -# Convert

 blocks (with optional classes) into fenced code blocks
-/^[ \t]*
]*>.*<\/pre>/) {
-        content = line
-        sub(/^[ \t]*]*>/, "", content)
-        sub(/<\/pre>[ \t]*$/, "", content)
-        gsub(/<\/?span[^>]*>/, "", content)
-        gsub(/<\/?div[^>]*>/, "", content)
-        gsub(/<\/?code[^>]*>/, "", content)
-        content = html_decode(content)
-        print "```" lang
-        print content
-        print "```"
-        next
-    }
-    print "```" lang
-    in_pre_block = 1
-    next
-}
-
-in_pre_block && /<\/pre>/ {
-    print "```"
-    in_pre_block = 0
-    next
-}
-
-in_pre_block {
-    line = $0
-    gsub(/<\/?span[^>]*>/, "", line)
-    gsub(/<\/?div[^>]*>/, "", line)
-    gsub(/<\/?code[^>]*>/, "", line)
-    line = html_decode(line)
-    print line
-    next
-}
-
-# Track code blocks to avoid processing their content
-/^```/ {
-    in_code_block = !in_code_block
-    print
-    next
-}
-
-# Don't process lines inside code blocks
-in_code_block {
-    print
-    next
-}
-
-# Convert HTML comments to MDX comments
-//) {
-        # Single line comment
-        gsub(//, "*/}", $0)
-        print
-        next
-    } else {
-        # Start of multi-line comment
-        gsub(/ *$/ {
-    gsub(/-->/, "*/}", $0)
-    print
-    next
-}
-
-# Convert navigation tables into inline Markdown
-capture_nav_table == 1 {
-    nav_buffer = nav_buffer "\n" $0
-    if ($0 ~ /<\/table>/) {
-        emit_navigation(nav_buffer)
-        nav_buffer = ""
-        capture_nav_table = 0
-    }
-    next
-}
-
-/^/ {
-    capture_nav_table = 1
-    nav_buffer = $0
-    next
-}
-
-# Convert compare callouts (Wrong/Correct) to Mintlify Callout components
-capture_compare == 1 {
-    compare_buffer = compare_buffer "\n" $0
-    if ($0 ~ /<\/p>/) {
-        handle_compare(compare_buffer)
-        compare_buffer = ""
-        capture_compare = 0
-    }
-    next
-}
-
-/^

/ { - compare_buffer = $0 - if ($0 ~ /<\/p>/) { - handle_compare(compare_buffer) - compare_buffer = "" - } else { - capture_compare = 1 - } - next -} - -# Handle inline compare badges inside lists -/^[ \t]*[-*][ \t]*/ { - line = $0 - icon = (index(line, "compare-better") > 0) ? "✅" : "⚠️" - label_segment = line - sub(/^[ \t]*[-*][ \t]*/, "", label_segment) - sub(/<\/span>.*/, "", label_segment) - label = inline_to_md(label_segment) - rest_segment = line - sub(/^.*<\/span>:[[:space:]]*/, "", rest_segment) - rest = inline_to_md(rest_segment) - print "- " icon " **" label "**: " rest - next -} - -# Promote **Note:** style callouts to Mintlify Callout components -/^\*\*Note\*\*:/ { - line = $0 - sub(/^\*\*Note\*\*:[ \t]*/, "", line) - body = inline_to_md(line) - emit_callout("info", "Note", body) - next -} - -/^[ \t]*

[ \t]*$/ { next } - -# Convert styled horizontal rules to Markdown -/^
" }} and {{ '' }} patterns -{ - # These double curly braces with quotes break acorn parser - gsub(/\{\{ *"" *\}\}/, "", $0) - gsub(/\{\{ *"<\/var>" *\}\}/, "", $0) - gsub(/\{\{ *'' *\}\}/, "", $0) - gsub(/\{\{ *'<\/var>' *\}\}/, "", $0) -} - -# Fix < and > that should be escaped differently in MDX -{ - # In attribute values, convert < and > to actual < > - # This is safer in JSX/MDX - while (match($0, /="[^"]*&[lg]t;[^"]*"/)) { - gsub(/</, "<", $0) - gsub(/>/, ">", $0) - } -} - -# Fix empty thead tags -
should be -{ - # Multiple variations of broken thead - gsub(/<\/th>/, "", $0) - gsub(/<\/thead>/, "", $0) - - # Also fix lines that are ONLY after a thead - if ($0 ~ /^<\/th>$/ || $0 ~ /^[ \t]*<\/th>[ \t]*$/) { - next # Skip this line entirely - } -} - -# Fix malformed tags with align attribute without quotes -{ - # align=right should be align="right" - gsub(/align=right/, "align=\"right\"", $0) - gsub(/align=left/, "align=\"left\"", $0) - gsub(/align=center/, "align=\"center\"", $0) -} - -# Fix malformed tags - CORRECTED VERSION -{ - # First, handle with no attributes - gsub(//, "", $0) - - # Then handle with attributes but no self-closing slash - # We need to find (not />) - while (match($0, /\/]*>/)) { - # Get the matched string - pre = substr($0, 1, RSTART - 1) - matched = substr($0, RSTART, RLENGTH) - post = substr($0, RSTART + RLENGTH) - - # Remove the trailing > and add /> - matched = substr(matched, 1, length(matched) - 1) " />" - $0 = pre matched post - } -} - -# Close other self-closing HTML tags properly -{ - # Fix
tags - gsub(/
/, "
", $0) - - # Fix tags - ensure they're self-closing - while (match($0, /]*[^\/]>/)) { - pre = substr($0, 1, RSTART - 1) - tag = substr($0, RSTART, RLENGTH) - post = substr($0, RSTART + RLENGTH) - # Remove the trailing > and add /> - tag = substr(tag, 1, length(tag) - 1) " />" - $0 = pre tag post - } - - # Fix
tags - gsub(/
/, "
", $0) -} - -# Fix unclosed

tags -{ - # If we have

but no closing tag on the same line, and line ends with text - if (/]*>/ && !/<\/p>/ && $0 !~ /<\/(div|table|ul|ol|blockquote)>$/) { - # Check if it's just a

with content and no closing - if ($0 ~ /]*>[^<]+$/) { - $0 = $0 "

" - } - } -} - -# Fix unclosed tags in patterns -{ - # Fix escaped underscores in code tags like \_ - # These appear in patterns like noimplicit\_deps - if (/.*\\_.*<\/code>/) { - gsub(/\\_/, "_", $0) - } - - # If we have opening but line doesn't end with - if (/]*>/ && !/<\/code>/) { - # Look for the pattern and close it properly - if ($0 ~ /]*>[^<]*$/) { - $0 = $0 "" - } - } -} - -# Fix unclosed
-/^[^<]*]*>[[:space:]]*[^<[:space:]]+[[:space:]]*$/ { - if ($0 !~ /<\/th>$/) { - $0 = $0 "" - } -} - -# Fix malformed tags -{ - # Ensure href has quotes - while (match($0, /]*)href=([^"'][^ >]+)/)) { - pre = substr($0, 1, RSTART - 1) - match_str = substr($0, RSTART, RLENGTH) - post = substr($0, RSTART + RLENGTH) - gsub(/href=([^"'][^ >]+)/, "href=\"\\1\"", match_str) - $0 = pre match_str post - } - - # Fix broken ) -{ - # Escape angle brackets in text that looks like C++ includes - # Pattern: #include or e.g. #include - if ($0 ~ /#include and escape it too - gsub(/\.h>/, ".h\\>", $0) - } -} - -# Skip blank lines before first H1 -/^[ \t]*$/ && before_first_h1 == 1 { next } - -# Convert first H1 to front-matter -/^# / && first_h1_found == 0 { - title = substr($0, 3) # Remove "# " prefix - gsub(/^[ \t]+|[ \t]+$/, "", title) # Trim whitespace - # Escape single quotes in title by doubling them for YAML - gsub(/'/, "''", title) - print "---" - print "title: '" title "'" - for (i = 0; i < meta_index; i++) { - print meta_lines[i] - } - print "---" - print "" - first_h1_found = 1 - before_first_h1 = 0 - next -} - -{ - gsub(/ class="/, " className=\"", $0) - gsub(/ style="[^"]*"/, "", $0) - gsub(/<([A-Za-z0-9]+) /, "<\\1 ", $0) - gsub(/ (href|target|rel|aria|data)/, " \\1", $0) - gsub(/[ \t]+>/, ">", $0) -} - -{ - gsub(/]*>arrow_back<\/span>/, "←", $0) - gsub(/]*>arrow_forward<\/span>/, "→", $0) - gsub(/]*>arrow_forward_ios<\/span>/, "→", $0) -} - -{ - gsub(/ className="[^"]*"/, "", $0) - gsub(/<\/?span[^>]*>/, "", $0) -} - -{ - $0 = html_decode($0) - print -} From 2bd8eef3b53be1c7dfaddb9bc6931c5818dfc9e6 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Tue, 4 Nov 2025 18:38:06 -0800 Subject: [PATCH 06/10] burn down Co-authored-by: promptless[bot] --- copy-upstream-docs.sh | 33 +-------------------------------- 1 file changed, 1 insertion(+), 32 deletions(-) diff --git a/copy-upstream-docs.sh b/copy-upstream-docs.sh index 06659e22..a07716ed 100755 --- a/copy-upstream-docs.sh +++ b/copy-upstream-docs.sh @@ -22,38 +22,7 @@ LOCAL_FILES=" # Files that are not valid MDX syntax # This output pasted from a CI job - we should burn it down to zero -BROKEN_FILES=" -community/roadmaps-configurability.mdx -concepts/build-files.mdx -concepts/dependencies.mdx -concepts/labels.mdx -configure/integrate-cpp.mdx -contribute/docs-style-guide.mdx -contribute/search.mdx -docs/cc-toolchain-config-reference.mdx -docs/user-manual.mdx -extending/config.mdx -external/mod-command.mdx -external/registry.mdx -external/migration_tool.mdx -query/language.mdx -query/quickstart.mdx -reference/be/functions.mdx -reference/be/platforms-and-toolchains.mdx -reference/command-line-reference.mdx -reference/flag-cheatsheet.mdx -reference/test-encyclopedia.mdx -remote/dynamic.mdx -rules/lib/globals/bzl.mdx -rules/lib/repo/cache.mdx -rules/lib/repo/git.mdx -rules/lib/repo/http.mdx -rules/lib/repo/local.mdx -rules/lib/repo/utils.mdx -rules/lib/globals/module.mdx -rules/windows.mdx -run/build.mdx -" +BROKEN_FILES="" # Verify that at least one source exists if [ ! -d "$UPSTREAM_SITE" ] && [ ! -d "$REFERENCE_DOCS" ]; then From a9a43590e4af2ad15c85d8303ebdb0bd0fe02a6a Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Wed, 5 Nov 2025 14:27:53 -0800 Subject: [PATCH 07/10] feat: add Node.js-based MDX transformation tooling to replace AWK script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This commit introduces a comprehensive Node.js-based MDX transformation tool that successfully processes all previously broken documentation files from the upstream Bazel repository. ## What Changed - Created `tools/mdx-transform/` with a new Node.js transformer - Replaces the legacy `transform-docs.awk` script - Uses unified/remark/rehype ecosystem for robust parsing - Handles complex HTML, liquid templates, and MDX edge cases - Updated `copy-upstream-docs.sh` to use the new transformer - Cleared BROKEN_FILES list (30 → 0 files) - All files now transform successfully - Updated CI workflow to install Node.js dependencies ## Key Features The new transformer handles: - Liquid template syntax removal ({% %}, {{ }}) - Curly brace escaping for MDX - Compare callouts → Mintlify Callout components - Navigation tables → clean link navigation - Material icons → arrow symbols - HTML entity preservation in tables - Email and URL autolink conversion - Frontmatter extraction from H1 titles ## Test Results ✅ All unit tests passing (3/3) ✅ 156 files transformed successfully ✅ 19/19 previously broken files now working ✅ Zero Mintlify validation errors on transformed files Co-authored-by: promptless[bot] <179508745+promptless[bot]@users.noreply.github.com> --- about/faq.mdx | 74 +- about/intro.mdx | 95 +- about/roadmap.mdx | 88 +- about/vision.mdx | 110 +- about/why.mdx | 66 +- .../build-performance-breakdown.mdx | 248 +-- .../performance/build-performance-metrics.mdx | 76 +- advanced/performance/iteration-speed.mdx | 84 +- advanced/performance/json-trace-profile.mdx | 102 +- advanced/performance/memory.mdx | 60 +- basics/artifact-based-builds.mdx | 277 +-- basics/build-systems.mdx | 136 +- basics/dependencies.mdx | 306 +-- basics/distributed-builds.mdx | 128 +- basics/hermeticity.mdx | 117 +- basics/index.mdx | 54 +- basics/task-based-builds.mdx | 238 +-- brand/index.mdx | 84 +- build/share-variables.mdx | 31 +- build/style-guide.mdx | 279 +-- community/recommended-rules.mdx | 50 +- community/remote-execution-services.mdx | 31 +- community/sig.mdx | 146 +- community/users.mdx | 533 ++--- concepts/build-files.mdx | 73 + concepts/build-ref.mdx | 100 +- concepts/dependencies.mdx | 192 ++ concepts/labels.mdx | 148 ++ concepts/platforms.mdx | 283 +-- concepts/visibility.mdx | 331 +-- configure/attributes.mdx | 338 +-- configure/best-practices.mdx | 67 +- configure/coverage.mdx | 106 +- configure/integrate-cpp.mdx | 37 + configure/windows.mdx | 240 +-- contribute/breaking-changes.mdx | 112 +- contribute/codebase.mdx | 1881 +++++------------ contribute/design-documents.mdx | 238 +-- contribute/docs-style-guide.mdx | 176 ++ contribute/docs.mdx | 32 +- contribute/index.mdx | 76 +- contribute/maintainers-guide.mdx | 254 +-- contribute/naming.mdx | 62 +- contribute/patch-acceptance.mdx | 65 +- contribute/policy.mdx | 103 +- contribute/release-notes.mdx | 67 +- contribute/search.mdx | 167 ++ contribute/statemachine-guide.mdx | 568 ++--- contribute/windows-chocolatey-maintenance.mdx | 50 +- contribute/windows-scoop-maintenance.mdx | 28 +- docs/android-build-performance.mdx | 70 +- docs/android-instrumentation-test.mdx | 192 +- docs/android-ndk.mdx | 147 +- docs/bazel-and-android.mdx | 43 +- docs/bazel-and-apple.mdx | 87 +- docs/bazel-and-cpp.mdx | 125 +- docs/bazel-and-java.mdx | 219 +- docs/bazel-and-javascript.mdx | 30 +- docs/cc-toolchain-config-reference.mdx | 488 +++++ docs/configurable-attributes.mdx | 337 +-- docs/mobile-install.mdx | 183 +- docs/sandboxing.mdx | 144 +- docs/user-manual.mdx | 1443 +++++++++++++ extending/aspects.mdx | 202 +- extending/auto-exec-groups.mdx | 97 +- extending/concepts.mdx | 109 +- extending/config.mdx | 631 ++++++ extending/depsets.mdx | 130 +- extending/exec-groups.mdx | 106 +- extending/legacy-macros.mdx | 137 +- extending/macros.mdx | 270 +-- extending/platforms.mdx | 170 +- extending/rules.mdx | 879 ++------ extending/toolchains.mdx | 314 +-- external/extension.mdx | 202 +- external/faq.mdx | 327 +-- external/lockfile.mdx | 201 +- external/migration.mdx | 979 ++++----- external/migration_tool.mdx | 642 ++++++ external/mod-command.mdx | 418 ++++ external/module.mdx | 222 +- external/overview.mdx | 261 +-- external/registry.mdx | 83 + external/repo.mdx | 141 +- external/vendor.mdx | 136 +- help.mdx | 52 +- install/bazelisk.mdx | 53 +- install/compile-source.mdx | 262 +-- install/completion.mdx | 177 +- install/docker-container.mdx | 28 +- install/ide.mdx | 81 +- install/index.mdx | 32 +- install/os-x.mdx | 82 +- install/suse.mdx | 16 +- install/ubuntu.mdx | 82 +- install/windows.mdx | 185 +- migrate/index.mdx | 6 +- migrate/maven.mdx | 250 +-- migrate/xcode.mdx | 229 +- query/aquery.mdx | 177 +- query/cquery.mdx | 350 +-- query/guide.mdx | 206 +- query/language.mdx | 872 ++++++++ query/quickstart.mdx | 500 +++++ reference/flag-cheatsheet.mdx | 91 + reference/glossary.mdx | 516 +---- reference/skyframe.mdx | 211 +- reference/test-encyclopedia.mdx | 308 +++ release/backward-compatibility.mdx | 70 +- release/index.mdx | 254 +-- release/rolling.mdx | 11 +- release/rule-compatibility.mdx | 110 +- remote/bep-examples.mdx | 110 +- remote/bep-glossary.mdx | 148 +- remote/bep.mdx | 143 +- remote/cache-local.mdx | 68 +- remote/cache-remote.mdx | 152 +- remote/caching.mdx | 330 ++- remote/ci.mdx | 173 +- remote/creating.mdx | 195 +- remote/dynamic.mdx | 63 + remote/multiplex.mdx | 108 +- remote/output-directories.mdx | 133 +- remote/persistent.mdx | 219 +- remote/rbe.mdx | 28 +- remote/rules.mdx | 196 +- remote/sandbox.mdx | 210 +- remote/workspace.mdx | 122 +- rules/bzl-style.mdx | 221 +- rules/challenges.mdx | 230 +- rules/deploying.mdx | 143 +- rules/errors/read-only-variable.mdx | 9 +- rules/faq.mdx | 57 +- rules/index.mdx | 81 +- rules/language.mdx | 127 +- rules/legacy-macro-tutorial.mdx | 49 +- rules/macro-tutorial.mdx | 37 +- rules/performance.mdx | 147 +- rules/rules-tutorial.mdx | 126 +- rules/testing.mdx | 157 +- rules/verbs-tutorial.mdx | 118 +- rules/windows.mdx | 189 ++ run/bazelrc.mdx | 243 +-- run/build.mdx | 369 ++++ run/client-server.mdx | 54 +- run/scripts.mdx | 136 +- start/android-app.mdx | 261 +-- start/cpp.mdx | 178 +- start/go.mdx | 224 +- start/ios-app.mdx | 3 +- start/java.mdx | 233 +- tools/mdx-transform/transform.mjs | 23 + tutorials/ccp-toolchain-config.mdx | 832 ++++---- tutorials/cpp-dependency.mdx | 31 +- tutorials/cpp-labels.mdx | 20 +- tutorials/cpp-use-cases.mdx | 55 +- versions/index.mdx | 12 +- 157 files changed, 13881 insertions(+), 17465 deletions(-) create mode 100644 concepts/build-files.mdx create mode 100644 concepts/dependencies.mdx create mode 100644 concepts/labels.mdx create mode 100644 configure/integrate-cpp.mdx create mode 100644 contribute/docs-style-guide.mdx create mode 100644 contribute/search.mdx create mode 100644 docs/cc-toolchain-config-reference.mdx create mode 100644 docs/user-manual.mdx create mode 100644 extending/config.mdx create mode 100644 external/migration_tool.mdx create mode 100644 external/mod-command.mdx create mode 100644 external/registry.mdx create mode 100644 query/language.mdx create mode 100644 query/quickstart.mdx create mode 100644 reference/flag-cheatsheet.mdx create mode 100644 reference/test-encyclopedia.mdx create mode 100644 remote/dynamic.mdx create mode 100644 rules/windows.mdx create mode 100644 run/build.mdx diff --git a/about/faq.mdx b/about/faq.mdx index dd5be8a9..fc0c31d1 100644 --- a/about/faq.mdx +++ b/about/faq.mdx @@ -2,8 +2,6 @@ title: 'FAQ' --- - - If you have questions or need support, see [Getting Help](/help). ## What is Bazel? @@ -14,19 +12,19 @@ Bazel is a tool that automates software builds and tests. Supported build tasks Bazel was designed to fit the way software is developed at Google. It has the following features: -* Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. -* High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. -* Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. -* Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. -* Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about ~200ms. +- Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. +- High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. +- Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. +- Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. +- Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about \~200ms. ## Why doesn’t Google use...? -* Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. - * Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. -* Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. -* Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. -* Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. +- Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. + - Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. +- Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. +- Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. +- Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. ## Where did Bazel come from? @@ -48,10 +46,10 @@ Bazel runs build operations locally by default. However, Bazel can also connect For our server code base, we use the following development workflow: -* All our server code is in a single, gigantic version control system. -* Everybody builds their software with Bazel. -* Different teams own different parts of the source tree, and make their components available as `BUILD` targets. -* Branching is primarily used for managing releases, so everybody develops their software at the head revision. +- All our server code is in a single, gigantic version control system. +- Everybody builds their software with Bazel. +- Different teams own different parts of the source tree, and make their components available as `BUILD` targets. +- Branching is primarily used for managing releases, so everybody develops their software at the head revision. Bazel is a cornerstone of this philosophy: since Bazel requires all dependencies to be fully specified, we can predict which programs and tests are affected by a change, and vet them before submission. @@ -63,24 +61,22 @@ Building software should be fun and easy. Slow and unpredictable builds take the ## Why would I want to use Bazel? -* Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. -* Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. -* Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. +- Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. +- Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. +- Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. ## Can I see examples? -Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) -or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. - +Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. ## What is Bazel best at? Bazel shines at building and testing projects with the following properties: -* Projects with a large codebase -* Projects written in (multiple) compiled languages -* Projects that deploy on multiple platforms -* Projects that have extensive tests +- Projects with a large codebase +- Projects written in (multiple) compiled languages +- Projects that deploy on multiple platforms +- Projects that have extensive tests ## Where can I run Bazel? @@ -90,11 +86,13 @@ Porting to other UNIX platforms should be relatively easy, as long as a JDK is a ## What should I not use Bazel for? -* Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: - * A compilation step that fetches data from the internet. - * A test step that connects to the QA instance of your site. - * A deployment step that changes your site’s cloud configuration. -* If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. +- Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: + + - A compilation step that fetches data from the internet. + - A test step that connects to the QA instance of your site. + - A deployment step that changes your site’s cloud configuration. + +- If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. ## How stable is Bazel’s feature set? @@ -132,10 +130,10 @@ Yes, you can use our [Docker rules](https://github.com/bazelbuild/rules_docker) For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (for example, executing binaries through a shell script inside a rule), you will need to take some extra care: -* Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. -* Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. -* Avoid connecting to the network. Sandboxed execution can help here too. -* Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. +- Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. +- Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. +- Avoid connecting to the network. Sandboxed execution can help here too. +- Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. ## Do you have binary releases? @@ -179,8 +177,8 @@ We still have to refactor the interfaces between the public code in Bazel and ou Open sourcing Bazel is a work-in-progress. In particular, we’re still working on open sourcing: -* Many of our unit and integration tests (which should make contributing patches easier). -* Full IDE integration. +- Many of our unit and integration tests (which should make contributing patches easier). +- Full IDE integration. Beyond code, we’d like to eventually have all code reviews, bug tracking, and design decisions happen publicly, with the Bazel community involved. We are not there yet, so some changes will simply appear in the Bazel repository without clear explanation. Despite this lack of transparency, we want to support external developers and collaborate. Thus, we are opening up the code, even though some of the development is still happening internal to Google. Please let us know if anything seems unclear or unjustified as we transition to an open model. @@ -190,7 +188,7 @@ Yes, some of the code base either integrates with Google-specific technology or ## How do I contact the team? -We are reachable at bazel-discuss@googlegroups.com. +We are reachable at [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com). ## Where do I report bugs? diff --git a/about/intro.mdx b/about/intro.mdx index a531ac2a..a70f715b 100644 --- a/about/intro.mdx +++ b/about/intro.mdx @@ -2,110 +2,63 @@ title: 'Intro to Bazel' --- - - -Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. -It uses a human-readable, high-level build language. Bazel supports projects in -multiple languages and builds outputs for multiple platforms. Bazel supports -large codebases across multiple repositories, and large numbers of users. +Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users. ## Benefits Bazel offers the following advantages: -* **High-level build language.** Bazel uses an abstract, human-readable - language to describe the build properties of your project at a high - semantical level. Unlike other tools, Bazel operates on the *concepts* - of libraries, binaries, scripts, and data sets, shielding you from the - complexity of writing individual calls to tools such as compilers and - linkers. +- **High-level build language.** Bazel uses an abstract, human-readable language to describe the build properties of your project at a high semantical level. Unlike other tools, Bazel operates on the *concepts* of libraries, binaries, scripts, and data sets, shielding you from the complexity of writing individual calls to tools such as compilers and linkers. -* **Bazel is fast and reliable.** Bazel caches all previously done work and - tracks changes to both file content and build commands. This way, Bazel - knows when something needs to be rebuilt, and rebuilds only that. To further - speed up your builds, you can set up your project to build in a highly - parallel and incremental fashion. +- **Bazel is fast and reliable.** Bazel caches all previously done work and tracks changes to both file content and build commands. This way, Bazel knows when something needs to be rebuilt, and rebuilds only that. To further speed up your builds, you can set up your project to build in a highly parallel and incremental fashion. -* **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel - can build binaries and deployable packages for multiple platforms, including - desktop, server, and mobile, from the same project. +- **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel can build binaries and deployable packages for multiple platforms, including desktop, server, and mobile, from the same project. -* **Bazel scales.** Bazel maintains agility while handling builds with 100k+ - source files. It works with multiple repositories and user bases in the tens - of thousands. +- **Bazel scales.** Bazel maintains agility while handling builds with 100k+ source files. It works with multiple repositories and user bases in the tens of thousands. -* **Bazel is extensible.** Many [languages](/rules) are - supported, and you can extend Bazel to support any other language or - framework. +- **Bazel is extensible.** Many [languages](/rules) are supported, and you can extend Bazel to support any other language or framework. ## Using Bazel To build or test a project with Bazel, you typically do the following: -1. **Set up Bazel.** Download and [install Bazel](/install). +1. **Set up Bazel.** Download and [install Bazel](/install). -2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a - directory where Bazel looks for build inputs and `BUILD` files, and where it - stores build outputs. +2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a directory where Bazel looks for build inputs and `BUILD` files, and where it stores build outputs. -3. **Write a `BUILD` file**, which tells Bazel what to build and how to - build it. +3. **Write a `BUILD` file**, which tells Bazel what to build and how to build it. - You write your `BUILD` file by declaring build targets using - [Starlark](/rules/language), a domain-specific language. (See example - [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) + You write your `BUILD` file by declaring build targets using [Starlark](/rules/language), a domain-specific language. (See example [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) - A build target specifies a set of input artifacts that Bazel will build plus - their dependencies, the build rule Bazel will use to build it, and options - that configure the build rule. + A build target specifies a set of input artifacts that Bazel will build plus their dependencies, the build rule Bazel will use to build it, and options that configure the build rule. - A build rule specifies the build tools Bazel will use, such as compilers and - linkers, and their configurations. Bazel ships with a number of build rules - covering the most common artifact types in the supported languages on - supported platforms. + A build rule specifies the build tools Bazel will use, such as compilers and linkers, and their configurations. Bazel ships with a number of build rules covering the most common artifact types in the supported languages on supported platforms. -4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel - places your outputs within the workspace. +4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel places your outputs within the workspace. -In addition to building, you can also use Bazel to run -[tests](/reference/test-encyclopedia) and [query](/query/guide) the build -to trace dependencies in your code. +In addition to building, you can also use Bazel to run [tests](/reference/test-encyclopedia) and [query](/query/guide) the build to trace dependencies in your code. ## Bazel build process When running a build or a test, Bazel does the following: -1. **Loads** the `BUILD` files relevant to the target. +1. **Loads** the `BUILD` files relevant to the target. -2. **Analyzes** the inputs and their - [dependencies](/concepts/dependencies), applies the specified build - rules, and produces an [action](/extending/concepts#evaluation-model) - graph. +2. **Analyzes** the inputs and their [dependencies](/concepts/dependencies), applies the specified build rules, and produces an [action](/extending/concepts#evaluation-model) graph. -3. **Executes** the build actions on the inputs until the final build outputs - are produced. +3. **Executes** the build actions on the inputs until the final build outputs are produced. -Since all previous build work is cached, Bazel can identify and reuse cached -artifacts and only rebuild or retest what's changed. To further enforce -correctness, you can set up Bazel to run builds and tests -[hermetically](/basics/hermeticity) through sandboxing, minimizing skew -and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). +Since all previous build work is cached, Bazel can identify and reuse cached artifacts and only rebuild or retest what's changed. To further enforce correctness, you can set up Bazel to run builds and tests [hermetically](/basics/hermeticity) through sandboxing, minimizing skew and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). ### Action graph -The action graph represents the build artifacts, the relationships between them, -and the build actions that Bazel will perform. Thanks to this graph, Bazel can -[track](/run/build#build-consistency) changes to -file content as well as changes to actions, such as build or test commands, and -know what build work has previously been done. The graph also enables you to -easily [trace dependencies](/query/guide) in your code. +The action graph represents the build artifacts, the relationships between them, and the build actions that Bazel will perform. Thanks to this graph, Bazel can [track](/run/build#build-consistency) changes to file content as well as changes to actions, such as build or test commands, and know what build work has previously been done. The graph also enables you to easily [trace dependencies](/query/guide) in your code. ## Getting started tutorials -To get started with Bazel, see [Getting Started](/start/) or jump -directly to the Bazel tutorials: +To get started with Bazel, see [Getting Started](/start/) or jump directly to the Bazel tutorials: -* [Tutorial: Build a C++ Project](/start/cpp) -* [Tutorial: Build a Java Project](/start/java) -* [Tutorial: Build an Android Application](/start/android-app) -* [Tutorial: Build an iOS Application](/start/ios-app) +- [Tutorial: Build a C++ Project](/start/cpp) +- [Tutorial: Build a Java Project](/start/java) +- [Tutorial: Build an Android Application](/start/android-app) +- [Tutorial: Build an iOS Application](/start/ios-app) diff --git a/about/roadmap.mdx b/about/roadmap.mdx index 42e63e9b..c17485d2 100644 --- a/about/roadmap.mdx +++ b/about/roadmap.mdx @@ -2,98 +2,50 @@ title: 'Bazel roadmap' --- +As Bazel continues to evolve in response to your needs, we want to share our 2025 roadmap update. - -As Bazel continues to evolve in response to your needs, we want to share our -2025 roadmap update. - -We plan to bring Bazel 9.0 -[long term support (LTS)](https://bazel.build/release/versioning) to you in late -2025. +We plan to bring Bazel 9.0 [long term support (LTS)](https://bazel.build/release/versioning) to you in late 2025. ## Full transition to Bzlmod -[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external -dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. -As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) -hosts more than 650 modules. +[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) hosts more than 650 modules. -With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will -be the only way to introduce external dependencies in Bazel. To minimize the -migration cost for the community, we'll focus on further improving our migration -[guide](https://bazel.build/external/migration) and -[tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). +With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will be the only way to introduce external dependencies in Bazel. To minimize the migration cost for the community, we'll focus on further improving our migration [guide](https://bazel.build/external/migration) and [tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). -Additionally, we aim to implement an improved shared repository cache (see -[#12227](https://github.com/bazelbuild/bazel/issues/12227)) -with garbage collection, and may backport it to Bazel 8. The Bazel Central -Registry will also support verifying SLSA attestations. +Additionally, we aim to implement an improved shared repository cache (see [#12227](https://github.com/bazelbuild/bazel/issues/12227)) with garbage collection, and may backport it to Bazel 8. The Bazel Central Registry will also support verifying SLSA attestations. ## Migration of Android, C++, Java, Python, and Proto rules -With Bazel 8, we have migrated support for Android, Java, Python, and Proto -rules out of the Bazel codebase into Starlark rules in their corresponding -repositories. To ease the migration, we implemented the autoload features in -Bazel, which can be controlled with -[--incompatible_autoload_externally](https://github.com/bazelbuild/bazel/issues/23043) -and [--incompatible_disable_autoloads_in_main_repo](https://github.com/bazelbuild/bazel/issues/25755) -flags. +With Bazel 8, we have migrated support for Android, Java, Python, and Proto rules out of the Bazel codebase into Starlark rules in their corresponding repositories. To ease the migration, we implemented the autoload features in Bazel, which can be controlled with [--incompatible\_autoload\_externally](https://github.com/bazelbuild/bazel/issues/23043) and [--incompatible\_disable\_autoloads\_in\_main\_repo](https://github.com/bazelbuild/bazel/issues/25755) flags. -With Bazel 9, we aim to disable autoloads by default and require every project -to explicitly load required rules in BUILD files. +With Bazel 9, we aim to disable autoloads by default and require every project to explicitly load required rules in BUILD files. -We will rewrite most of C++ language support to Starlark, detach it from Bazel -binary and move it into the [/rules_cc](https://github.com/bazelbuild/rules_cc) -repository. This is the last remaining major language support that is still part -of Bazel. +We will rewrite most of C++ language support to Starlark, detach it from Bazel binary and move it into the [/rules\_cc](https://github.com/bazelbuild/rules_cc) repository. This is the last remaining major language support that is still part of Bazel. -We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving -them to repositories next to the implementation to increase velocity of rule -authors. +We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving them to repositories next to the implementation to increase velocity of rule authors. ## Starlark improvements -Bazel will have the ability to evaluate symbolic macros lazily. This means that -a symbolic macro won't run if the targets it declares are not requested, -improving performance for very large packages. +Bazel will have the ability to evaluate symbolic macros lazily. This means that a symbolic macro won't run if the targets it declares are not requested, improving performance for very large packages. -Starlark will have an experimental type system, similar to Python's type -annotations. We expect the type system to stabilize _after_ Bazel 9 is launched. +Starlark will have an experimental type system, similar to Python's type annotations. We expect the type system to stabilize *after* Bazel 9 is launched. ## Configurability Our main focus is reducing the cost and confusion of build flags. -We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a -new project configuration model that doesn't make users have to know which build -and test flags to set where. So `$ bazel test //foo` automatically sets the -right flags based on `foo`'s project's policy. This will likely remain -experimental in 9.0 but guiding feedback is welcome. - -[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip -out Starlark flags when they leave project boundaries, so they don't break -caching on transitive dependencies that don't need them. This makes builds that -use [transitions](https://bazel.build/extending/config#user-defined-transitions) -cheaper and faster. -[Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) -an example. We're extending the idea to control which flags propagate to -[exec configurations](https://bazel.build/extending/rules#configurations) and -are considering even more flexible support like custom Starlark to determine -which dependency edges should propagate flags. - -We're up-prioritizing effort to move built-in language flags out of Bazel and -into Starlark, where they can live with related rule definitions. +We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a new project configuration model that doesn't make users have to know which build and test flags to set where. So `$ bazel test //foo` automatically sets the right flags based on `foo`'s project's policy. This will likely remain experimental in 9.0 but guiding feedback is welcome. + +[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip out Starlark flags when they leave project boundaries, so they don't break caching on transitive dependencies that don't need them. This makes builds that use [transitions](https://bazel.build/extending/config#user-defined-transitions) cheaper and faster. [Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) an example. We're extending the idea to control which flags propagate to [exec configurations](https://bazel.build/extending/rules#configurations) and are considering even more flexible support like custom Starlark to determine which dependency edges should propagate flags. + +We're up-prioritizing effort to move built-in language flags out of Bazel and into Starlark, where they can live with related rule definitions. ## Remote execution improvements -We plan to add support for asynchronous execution, speeding up remote execution -by increasing parallelism. +We plan to add support for asynchronous execution, speeding up remote execution by increasing parallelism. ---- +*** -To follow updates to the roadmap and discuss planned features, join the -community Slack server at [slack.bazel.build](https://slack.bazel.build/). +To follow updates to the roadmap and discuss planned features, join the community Slack server at [slack.bazel.build](https://slack.bazel.build/). -*This roadmap is intended to help inform the community about the team's -intentions for Bazel 9.0. Priorities are subject to change in response to -developer and customer feedback, or to new market opportunities.* +*This roadmap is intended to help inform the community about the team's intentions for Bazel 9.0. Priorities are subject to change in response to developer and customer feedback, or to new market opportunities.* diff --git a/about/vision.mdx b/about/vision.mdx index da0ed02d..b80ca03a 100644 --- a/about/vision.mdx +++ b/about/vision.mdx @@ -2,96 +2,48 @@ title: 'Bazel Vision' --- +Any software developer can efficiently build, test, and package any project, of any size or complexity, with tooling that's easy to adopt and extend. +- **Engineers can take build fundamentals for granted.** Software developers focus on the creative process of authoring code because the mechanical process of build and test is solved. When customizing the build system to support new languages or unique organizational needs, users focus on the aspects of extensibility that are unique to their use case, without having to reinvent the basic plumbing. -Any software developer can efficiently build, test, and package -any project, of any size or complexity, with tooling that's easy to adopt and -extend. +- **Engineers can easily contribute to any project.** A developer who wants to start working on a new project can simply clone the project and run the build. There's no need for local configuration - it just works. With cross-platform remote execution, they can work on any machine anywhere and fully test their changes against all platforms the project targets. Engineers can quickly configure the build for a new project or incrementally migrate an existing build. -* **Engineers can take build fundamentals for granted.** Software developers - focus on the creative process of authoring code because the mechanical - process of build and test is solved. When customizing the build system to - support new languages or unique organizational needs, users focus on the - aspects of extensibility that are unique to their use case, without having - to reinvent the basic plumbing. - -* **Engineers can easily contribute to any project.** A developer who wants to - start working on a new project can simply clone the project and run the - build. There's no need for local configuration - it just works. With - cross-platform remote execution, they can work on any machine anywhere and - fully test their changes against all platforms the project targets. - Engineers can quickly configure the build for a new project or incrementally - migrate an existing build. - -* **Projects can scale to any size codebase, any size team.** Fast, - incremental testing allows teams to fully validate every change before it is - committed. This remains true even as repos grow, projects span multiple - repos, and multiple languages are introduced. Infrastructure does not force - developers to trade test coverage for build speed. +- **Projects can scale to any size codebase, any size team.** Fast, incremental testing allows teams to fully validate every change before it is committed. This remains true even as repos grow, projects span multiple repos, and multiple languages are introduced. Infrastructure does not force developers to trade test coverage for build speed. **We believe Bazel has the potential to fulfill this vision.** -Bazel was built from the ground up to enable builds that are reproducible (a -given set of inputs will always produce the same outputs) and portable (a build -can be run on any machine without affecting the output). +Bazel was built from the ground up to enable builds that are reproducible (a given set of inputs will always produce the same outputs) and portable (a build can be run on any machine without affecting the output). -These characteristics support safe incrementality (rebuilding only changed -inputs doesn't introduce the risk of corruption) and distributability (build -actions are isolated and can be offloaded). By minimizing the work needed to do -a correct build and parallelizing that work across multiple cores and remote -systems, Bazel can make any build fast. +These characteristics support safe incrementality (rebuilding only changed inputs doesn't introduce the risk of corruption) and distributability (build actions are isolated and can be offloaded). By minimizing the work needed to do a correct build and parallelizing that work across multiple cores and remote systems, Bazel can make any build fast. -Bazel's abstraction layer — instructions specific to languages, platforms, and -toolchains implemented in a simple extensibility language — allows it to be -easily applied to any context. +Bazel's abstraction layer — instructions specific to languages, platforms, and toolchains implemented in a simple extensibility language — allows it to be easily applied to any context. ## Bazel core competencies -1. Bazel supports **multi-language, multi-platform** builds and tests. You can - run a single command to build and test your entire source tree, no matter - which combination of languages and platforms you target. -1. Bazel builds are **fast and correct**. Every build and test run is - incremental, on your developers' machines and on CI. -1. Bazel provides a **uniform, extensible language** to define builds for any - language or platform. -1. Bazel allows your builds **to scale** by connecting to remote execution and - caching services. -1. Bazel works across **all major development platforms** (Linux, MacOS, and - Windows). -1. We accept that adopting Bazel requires effort, but **gradual adoption** is - possible. Bazel interfaces with de-facto standard tools for a given - language/platform. +1. Bazel supports **multi-language, multi-platform** builds and tests. You can run a single command to build and test your entire source tree, no matter which combination of languages and platforms you target. +2. Bazel builds are **fast and correct**. Every build and test run is incremental, on your developers' machines and on CI. +3. Bazel provides a **uniform, extensible language** to define builds for any language or platform. +4. Bazel allows your builds **to scale** by connecting to remote execution and caching services. +5. Bazel works across **all major development platforms** (Linux, MacOS, and Windows). +6. We accept that adopting Bazel requires effort, but **gradual adoption** is possible. Bazel interfaces with de-facto standard tools for a given language/platform. ## Serving language communities -Software engineering evolves in the context of language communities — typically, -self-organizing groups of people who use common tools and practices. - -To be of use to members of a language community, high-quality Bazel rules must be -available that integrate with the workflows and conventions of that community. - -Bazel is committed to be extensible and open, and to support good rulesets for -any language. - -### Requirements of a good ruleset - -1. The rules need to support efficient **building and testing** for the - language, including code coverage. -1. The rules need to **interface with a widely-used "package manager"** for the - language (such as Maven for Java), and support incremental migration paths - from other widely-used build systems. -1. The rules need to be **extensible and interoperable**, following - ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) - principles. -1. The rules need to be **remote-execution ready**. In practice, this means - **configurable using the [toolchains](/extending/toolchains) mechanism**. -1. The rules (and Bazel) need to interface with a **widely-used IDE** for the - language, if there is one. -1. The rules need to have **thorough, usable documentation,** with introductory - material for new users, comprehensive docs for expert users. - -Each of these items is essential and only together do they deliver on Bazel's -competencies for their particular ecosystem. - -They are also, by and large, sufficient - once all are fulfilled, Bazel fully -delivers its value to members of that language community. +Software engineering evolves in the context of language communities — typically, self-organizing groups of people who use common tools and practices. + +To be of use to members of a language community, high-quality Bazel rules must be available that integrate with the workflows and conventions of that community. + +Bazel is committed to be extensible and open, and to support good rulesets for any language. + +### Requirements of a good ruleset + +1. The rules need to support efficient **building and testing** for the language, including code coverage. +2. The rules need to **interface with a widely-used "package manager"** for the language (such as Maven for Java), and support incremental migration paths from other widely-used build systems. +3. The rules need to be **extensible and interoperable**, following ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) principles. +4. The rules need to be **remote-execution ready**. In practice, this means **configurable using the [toolchains](/extending/toolchains) mechanism**. +5. The rules (and Bazel) need to interface with a **widely-used IDE** for the language, if there is one. +6. The rules need to have **thorough, usable documentation,** with introductory material for new users, comprehensive docs for expert users. + +Each of these items is essential and only together do they deliver on Bazel's competencies for their particular ecosystem. + +They are also, by and large, sufficient - once all are fulfilled, Bazel fully delivers its value to members of that language community. diff --git a/about/why.mdx b/about/why.mdx index 6224abf6..dcdec18e 100644 --- a/about/why.mdx +++ b/about/why.mdx @@ -2,84 +2,44 @@ title: 'Why Bazel?' --- - - -Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) -build tool with [integrated testing](#integrated-testing) that supports multiple -[languages](#multi-language), [repositories](#multi-repository), and -[platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). +Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) build tool with [integrated testing](#integrated-testing) that supports multiple [languages](#multi-language), [repositories](#multi-repository), and [platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). ## Bazel is fast -Bazel knows exactly what input files each build command needs, avoiding -unnecessary work by re-running only when the set of input files have -changed between each build. -It runs build commands with as much parallelism as possible, either within the -same computer or on [remote build nodes](/remote/rbe). If the structure of build -allows for it, it can run thousands of build or test commands at the same time. +Bazel knows exactly what input files each build command needs, avoiding unnecessary work by re-running only when the set of input files have changed between each build. It runs build commands with as much parallelism as possible, either within the same computer or on [remote build nodes](/remote/rbe). If the structure of build allows for it, it can run thousands of build or test commands at the same time. -This is supported by multiple caching layers, in memory, on disk and on the -remote build farm, if available. At Google, we routinely achieve cache hit rates -north of 99%. +This is supported by multiple caching layers, in memory, on disk and on the remote build farm, if available. At Google, we routinely achieve cache hit rates north of 99%. ## Bazel is correct -Bazel ensures that your binaries are built *only* from your own -source code. Bazel actions run in individual sandboxes and Bazel tracks -every input file of the build, only and always re-running build -commands when it needs to. This keeps your binaries up-to-date so that the -[same source code always results in the same binary](/basics/hermeticity), bit -by bit. +Bazel ensures that your binaries are built *only* from your own source code. Bazel actions run in individual sandboxes and Bazel tracks every input file of the build, only and always re-running build commands when it needs to. This keeps your binaries up-to-date so that the [same source code always results in the same binary](/basics/hermeticity), bit by bit. -Say goodbye to endless `make clean` invocations and to chasing phantom bugs -that were in fact resolved in source code that never got built. +Say goodbye to endless `make clean` invocations and to chasing phantom bugs that were in fact resolved in source code that never got built. ## Bazel is extensible -Harness the full power of Bazel by writing your own rules and macros to -customize Bazel for your specific needs across a wide range of projects. +Harness the full power of Bazel by writing your own rules and macros to customize Bazel for your specific needs across a wide range of projects. -Bazel rules are written in [Starlark](/rules/language), our -in-house programming language that's a subset of Python. Starlark makes -rule-writing accessible to most developers, while also creating rules that can -be used across the ecosystem. +Bazel rules are written in [Starlark](/rules/language), our in-house programming language that's a subset of Python. Starlark makes rule-writing accessible to most developers, while also creating rules that can be used across the ecosystem. ## Integrated testing -Bazel's [integrated test runner](/docs/user-manual#running-tests) -knows and runs only those tests needing to be re-run, using remote execution -(if available) to run them in parallel. Detect flakes early by using remote -execution to quickly run a test thousands of times. +Bazel's [integrated test runner](/docs/user-manual#running-tests) knows and runs only those tests needing to be re-run, using remote execution (if available) to run them in parallel. Detect flakes early by using remote execution to quickly run a test thousands of times. -Bazel [provides facilities](/remote/bep) to upload test results to a central -location, thereby facilitating efficient communication of test outcomes, be it -on CI or by individual developers. +Bazel [provides facilities](/remote/bep) to upload test results to a central location, thereby facilitating efficient communication of test outcomes, be it on CI or by individual developers. ## Multi-language support -Bazel supports many common programming languages including C++, Java, -Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, -backend, web UI and mobile app) in the same Bazel invocation without being -constrained to one language's idiomatic build tool. +Bazel supports many common programming languages including C++, Java, Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, backend, web UI and mobile app) in the same Bazel invocation without being constrained to one language's idiomatic build tool. ## Multi-repository support -Bazel can [gather source code from multiple locations](/external/overview): you -don't need to vendor your dependencies (but you can!), you can instead point -Bazel to the location of your source code or prebuilt artifacts (e.g. a git -repository or Maven Central), and it takes care of the rest. +Bazel can [gather source code from multiple locations](/external/overview): you don't need to vendor your dependencies (but you can!), you can instead point Bazel to the location of your source code or prebuilt artifacts (e.g. a git repository or Maven Central), and it takes care of the rest. ## Multi-platform support -Bazel can simultaneously build projects for multiple platforms including Linux, -macOS, Windows, and Android. It also provides powerful -[cross-compilation capabilities](/extending/platforms) to build code for one -platform while running the build on another. +Bazel can simultaneously build projects for multiple platforms including Linux, macOS, Windows, and Android. It also provides powerful [cross-compilation capabilities](/extending/platforms) to build code for one platform while running the build on another. ## Wide ecosystem -[Industry leaders](/community/users) love Bazel, building a large -community of developers who use and contribute to Bazel. Find a tools, services -and documentation, including [consulting and SaaS offerings](/community/experts) -Bazel can use. Explore extensions like support for programming languages in -our [open source software repositories](/rules). +[Industry leaders](/community/users) love Bazel, building a large community of developers who use and contribute to Bazel. Find a tools, services and documentation, including [consulting and SaaS offerings](/community/experts) Bazel can use. Explore extensions like support for programming languages in our [open source software repositories](/rules). diff --git a/advanced/performance/build-performance-breakdown.mdx b/advanced/performance/build-performance-breakdown.mdx index 477e7578..8bebff8c 100644 --- a/advanced/performance/build-performance-breakdown.mdx +++ b/advanced/performance/build-performance-breakdown.mdx @@ -2,234 +2,120 @@ title: 'Breaking down build performance' --- +Bazel is complex and does a lot of different things over the course of a build, some of which can have an impact on build performance. This page attempts to map some of these Bazel concepts to their implications on build performance. While not extensive, we have included some examples of how to detect build performance issues through [extracting metrics](/configure/build-performance-metrics) and what you can do to fix them. With this, we hope you can apply these concepts when investigating build performance regressions. +### Clean vs Incremental builds -Bazel is complex and does a lot of different things over the course of a build, -some of which can have an impact on build performance. This page attempts to map -some of these Bazel concepts to their implications on build performance. While -not extensive, we have included some examples of how to detect build performance -issues through [extracting metrics](/configure/build-performance-metrics) -and what you can do to fix them. With this, we hope you can apply these concepts -when investigating build performance regressions. +A clean build is one that builds everything from scratch, while an incremental build reuses some already completed work. -### Clean vs Incremental builds +We suggest looking at clean and incremental builds separately, especially when you are collecting / aggregating metrics that are dependent on the state of Bazel’s caches (for example [build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) ). They also represent two different user experiences. As compared to starting a clean build from scratch (which takes longer due to a cold cache), incremental builds happen far more frequently as developers iterate on code (typically faster since the cache is usually already warm). -A clean build is one that builds everything from scratch, while an incremental -build reuses some already completed work. - -We suggest looking at clean and incremental builds separately, especially when -you are collecting / aggregating metrics that are dependent on the state of -Bazel’s caches (for example -[build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) -). They also represent two different user experiences. As compared to starting -a clean build from scratch (which takes longer due to a cold cache), incremental -builds happen far more frequently as developers iterate on code (typically -faster since the cache is usually already warm). - -You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify -builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly -categorize it to likely be an incremental build - the user could have switched -to different flags or different targets causing an effectively clean build. Any -more rigorous definition of incrementality will likely have to come in the form -of a heuristic, for example looking at the number of packages loaded -(`PackageMetrics.packages_loaded`). +You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly categorize it to likely be an incremental build - the user could have switched to different flags or different targets causing an effectively clean build. Any more rigorous definition of incrementality will likely have to come in the form of a heuristic, for example looking at the number of packages loaded (`PackageMetrics.packages_loaded`). ### Deterministic build metrics as a proxy for build performance -Measuring build performance can be difficult due to the non-deterministic nature -of certain metrics (for example Bazel’s CPU time or queue times on a remote -cluster). As such, it can be useful to use deterministic metrics as a proxy for -the amount of work done by Bazel, which in turn affects its performance. - -The size of a build request can have a significant implication on build -performance. A larger build could represent more work in analyzing and -constructing the build graphs. Organic growth of builds comes naturally with -development, as more dependencies are added/created, and thus grow in complexity -and become more expensive to build. - -We can slice this problem into the various build phases, and use the following -metrics as proxy metrics for work done at each phase: - -1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. - A regression here represents more work that needs to be done to read and parse - each additional BUILD file in the loading phase. - - This is often due to the addition of dependencies and having to load their - transitive closure. - - Use [query](/query/quickstart) / [cquery](/query/cquery) to find - where new dependencies might have been added. - -2. `TargetMetrics.targets_configured`: representing the number of targets and - aspects configured in the build. A regression represents more work in - constructing and traversing the configured target graph. - - This is often due to the addition of dependencies and having to construct - the graph of their transitive closure. - - Use [cquery](/query/cquery) to find where new - dependencies might have been added. - -3. `ActionSummary.actions_created`: represents the actions created in the build, - and a regression represents more work in constructing the action graph. Note - that this also includes unused actions that might not have been executed. - - Use [aquery](/query/aquery) for debugging regressions; - we suggest starting with - [`--output=summary`](/reference/command-line-reference#flag--output) - before further drilling down with - [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). - -4. `ActionSummary.actions_executed`: the number of actions executed, a - regression directly represents more work in executing these actions. - - The [BEP](/remote/bep) writes out the action statistics - `ActionData` that shows the most executed action types. By default, it - collects the top 20 action types, but you can pass in the - [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) - to collect this data for all action types that were executed. - - This should help you to figure out what kind of actions were executed - (additionally). - -5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by - the executed actions. - - If the number of actions executed did not increase, then it is likely that - a rule implementation was changed. - - -These metrics are all affected by the state of the local cache, hence you will -want to ensure that the builds you extract these metrics from are -**clean builds**. - -We have noted that a regression in any of these metrics can be accompanied by -regressions in wall time, cpu time and memory usage. +Measuring build performance can be difficult due to the non-deterministic nature of certain metrics (for example Bazel’s CPU time or queue times on a remote cluster). As such, it can be useful to use deterministic metrics as a proxy for the amount of work done by Bazel, which in turn affects its performance. + +The size of a build request can have a significant implication on build performance. A larger build could represent more work in analyzing and constructing the build graphs. Organic growth of builds comes naturally with development, as more dependencies are added/created, and thus grow in complexity and become more expensive to build. + +We can slice this problem into the various build phases, and use the following metrics as proxy metrics for work done at each phase: + +1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. A regression here represents more work that needs to be done to read and parse each additional BUILD file in the loading phase. + + - This is often due to the addition of dependencies and having to load their transitive closure. + - Use [query](/query/quickstart) / [cquery](/query/cquery) to find where new dependencies might have been added. + +2. `TargetMetrics.targets_configured`: representing the number of targets and aspects configured in the build. A regression represents more work in constructing and traversing the configured target graph. + + - This is often due to the addition of dependencies and having to construct the graph of their transitive closure. + - Use [cquery](/query/cquery) to find where new dependencies might have been added. + +3. `ActionSummary.actions_created`: represents the actions created in the build, and a regression represents more work in constructing the action graph. Note that this also includes unused actions that might not have been executed. + + - Use [aquery](/query/aquery) for debugging regressions; we suggest starting with [`--output=summary`](/reference/command-line-reference#flag--output) before further drilling down with [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). + +4. `ActionSummary.actions_executed`: the number of actions executed, a regression directly represents more work in executing these actions. + + - The [BEP](/remote/bep) writes out the action statistics `ActionData` that shows the most executed action types. By default, it collects the top 20 action types, but you can pass in the [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) to collect this data for all action types that were executed. + - This should help you to figure out what kind of actions were executed (additionally). + +5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by the executed actions. + + - If the number of actions executed did not increase, then it is likely that a rule implementation was changed. + +These metrics are all affected by the state of the local cache, hence you will want to ensure that the builds you extract these metrics from are **clean builds**. + +We have noted that a regression in any of these metrics can be accompanied by regressions in wall time, cpu time and memory usage. ### Usage of local resources -Bazel consumes a variety of resources on your local machine (both for analyzing -the build graph and driving the execution, and for running local actions), this -can affect the performance / availability of your machine in performing the -build, and also other tasks. +Bazel consumes a variety of resources on your local machine (both for analyzing the build graph and driving the execution, and for running local actions), this can affect the performance / availability of your machine in performing the build, and also other tasks. #### Time spent -Perhaps the metrics most susceptible to noise (and can vary greatly from build -to build) is time; in particular - wall time, cpu time and system time. You can -use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get -a benchmark for these metrics, and with a sufficient number of `--runs`, you can -increase the statistical significance of your measurement. +Perhaps the metrics most susceptible to noise (and can vary greatly from build to build) is time; in particular - wall time, cpu time and system time. You can use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get a benchmark for these metrics, and with a sufficient number of `--runs`, you can increase the statistical significance of your measurement. - **Wall time** is the real world time elapsed. - - If _only_ wall time regresses, we suggest collecting a - [JSON trace profile](/advanced/performance/json-trace-profile) and looking - for differences. Otherwise, it would likely be more efficient to - investigate other regressed metrics as they could have affected the wall - time. + + - If *only* wall time regresses, we suggest collecting a [JSON trace profile](/advanced/performance/json-trace-profile) and looking for differences. Otherwise, it would likely be more efficient to investigate other regressed metrics as they could have affected the wall time. - **CPU time** is the time spent by the CPU executing user code. - - If the CPU time regresses across two project commits, we suggest collecting - a Starlark CPU profile. You should probably also use `--nobuild` to - restrict the build to the analysis phase since that is where most of the - CPU heavy work is done. + + - If the CPU time regresses across two project commits, we suggest collecting a Starlark CPU profile. You should probably also use `--nobuild` to restrict the build to the analysis phase since that is where most of the CPU heavy work is done. - System time is the time spent by the CPU in the kernel. - - If system time regresses, it is mostly correlated with I/O when Bazel reads - files from your file system. + + - If system time regresses, it is mostly correlated with I/O when Bazel reads files from your file system. #### System-wide load profiling -Using the -[`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) -flag introduced in Bazel 6.0, the -[JSON trace profiler](/advanced/performance/json-trace-profile) collects the -system load average during the invocation. +Using the [`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) flag introduced in Bazel 6.0, the [JSON trace profiler](/advanced/performance/json-trace-profile) collects the system load average during the invocation. ![Profile that includes system load average](/docs/images/json-trace-profile-system-load-average.png "Profile that includes system load average") **Figure 1.** Profile that includes system load average. -A high load during a Bazel invocation can be an indication that Bazel schedules -too many local actions in parallel for your machine. You might want to look into -adjusting -[`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) -and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), -especially in container environments (at least until -[#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). - +A high load during a Bazel invocation can be an indication that Bazel schedules too many local actions in parallel for your machine. You might want to look into adjusting [`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), especially in container environments (at least until [#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). #### Monitoring Bazel memory usage -There are two main sources to get Bazel’s memory usage, Bazel `info` and the -[BEP](/remote/bep). - -- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after - a call to `System.gc()`. - - [Bazel bench](https://github.com/bazelbuild/bazel-bench) - provides benchmarks for this metric as well. - - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` - and `committed-heap-size` (see - [documentation](/docs/user-manual#configuration-independent-data)), but are - less relevant. - -- [BEP](/remote/bep)’s - `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in - bytes post GC (requires setting - [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) - that attempts to force a full GC). - -A regression in memory usage is usually a result of a regression in -[build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), -which are often due to addition of dependencies or a change in the rule -implementation. - -To analyze Bazel’s memory footprint on a more granular level, we recommend using -the [built-in memory profiler](/rules/performance#memory-profiling) -for rules. +There are two main sources to get Bazel’s memory usage, Bazel `info` and the [BEP](/remote/bep). + +- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after a call to `System.gc()`. + + - [Bazel bench](https://github.com/bazelbuild/bazel-bench) provides benchmarks for this metric as well. + - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` and `committed-heap-size` (see [documentation](/docs/user-manual#configuration-independent-data)), but are less relevant. + +- [BEP](/remote/bep)’s `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in bytes post GC (requires setting [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) that attempts to force a full GC). + +A regression in memory usage is usually a result of a regression in [build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), which are often due to addition of dependencies or a change in the rule implementation. + +To analyze Bazel’s memory footprint on a more granular level, we recommend using the [built-in memory profiler](/rules/performance#memory-profiling) for rules. #### Memory profiling of persistent workers -While [persistent workers](/remote/persistent) can help to speed up builds -significantly (especially for interpreted languages) their memory footprint can -be problematic. Bazel collects metrics on its workers, in particular, the -`WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory -workers use (by mnemonic). +While [persistent workers](/remote/persistent) can help to speed up builds significantly (especially for interpreted languages) their memory footprint can be problematic. Bazel collects metrics on its workers, in particular, the `WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory workers use (by mnemonic). -The [JSON trace profiler](/advanced/performance/json-trace-profile) also -collects persistent worker memory usage during the invocation by passing in the -[`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) -flag (new in Bazel 6.0). +The [JSON trace profiler](/advanced/performance/json-trace-profile) also collects persistent worker memory usage during the invocation by passing in the [`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) flag (new in Bazel 6.0). ![Profile that includes workers memory usage](/docs/images/json-trace-profile-workers-memory-usage.png "Profile that includes workers memory usage") **Figure 2.** Profile that includes workers memory usage. -Lowering the value of -[`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) -(default 4) might help to reduce -the amount of memory used by persistent workers. We are actively working on -making Bazel’s resource manager and scheduler smarter so that such fine tuning -will be required less often in the future. +Lowering the value of [`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) (default 4) might help to reduce the amount of memory used by persistent workers. We are actively working on making Bazel’s resource manager and scheduler smarter so that such fine tuning will be required less often in the future. ### Monitoring network traffic for remote builds -In remote execution, Bazel downloads artifacts that were built as a result of -executing actions. As such, your network bandwidth can affect the performance -of your build. +In remote execution, Bazel downloads artifacts that were built as a result of executing actions. As such, your network bandwidth can affect the performance of your build. -If you are using remote execution for your builds, you might want to consider -monitoring the network traffic during the invocation using the -`NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) -(requires passing `--experimental_collect_system_network_usage`). +If you are using remote execution for your builds, you might want to consider monitoring the network traffic during the invocation using the `NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) (requires passing `--experimental_collect_system_network_usage`). -Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) -allow you to view system-wide network usage throughout the course of the build -by passing the `--experimental_collect_system_network_usage` flag (new in Bazel -6.0). +Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) allow you to view system-wide network usage throughout the course of the build by passing the `--experimental_collect_system_network_usage` flag (new in Bazel 6.0). ![Profile that includes system-wide network usage](/docs/images/json-trace-profile-network-usage.png "Profile that includes system-wide network usage") **Figure 3.** Profile that includes system-wide network usage. -A high but rather flat network usage when using remote execution might indicate -that network is the bottleneck in your build; if you are not using it already, -consider turning on Build without the Bytes by passing -[`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). -This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. +A high but rather flat network usage when using remote execution might indicate that network is the bottleneck in your build; if you are not using it already, consider turning on Build without the Bytes by passing [`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. -Another option is to configure a local -[disk cache](/reference/command-line-reference#flag--disk_cache) to save on -download bandwidth. +Another option is to configure a local [disk cache](/reference/command-line-reference#flag--disk_cache) to save on download bandwidth. diff --git a/advanced/performance/build-performance-metrics.mdx b/advanced/performance/build-performance-metrics.mdx index 8391ea87..8f92e75a 100644 --- a/advanced/performance/build-performance-metrics.mdx +++ b/advanced/performance/build-performance-metrics.mdx @@ -2,96 +2,50 @@ title: 'Extracting build performance metrics' --- - - -Probably every Bazel user has experienced builds that were slow or slower than -anticipated. Improving the performance of individual builds has particular value -for targets with significant impact, such as: +Probably every Bazel user has experienced builds that were slow or slower than anticipated. Improving the performance of individual builds has particular value for targets with significant impact, such as: 1. Core developer targets that are frequently iterated on and (re)built. 2. Common libraries widely depended upon by other targets. -3. A representative target from a class of targets (e.g. custom rules), - diagnosing and fixing issues in one build might help to resolve issues at the - larger scale. +3. A representative target from a class of targets (e.g. custom rules), diagnosing and fixing issues in one build might help to resolve issues at the larger scale. -An important step to improving the performance of builds is to understand where -resources are spent. This page lists different metrics you can collect. -[Breaking down build performance](/configure/build-performance-breakdown) showcases -how you can use these metrics to detect and fix build performance issues. +An important step to improving the performance of builds is to understand where resources are spent. This page lists different metrics you can collect. [Breaking down build performance](/configure/build-performance-breakdown) showcases how you can use these metrics to detect and fix build performance issues. There are a few main ways to extract metrics from your Bazel builds, namely: ## Build Event Protocol (BEP) -Bazel outputs a variety of protocol buffers -[`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) -through the [Build Event Protocol (BEP)](/remote/bep), which -can be aggregated by a backend specified by you. Depending on your use cases, -you might decide to aggregate the metrics in various ways, but here we will go -over some concepts and proto fields that would be useful in general to consider. +Bazel outputs a variety of protocol buffers [`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) through the [Build Event Protocol (BEP)](/remote/bep), which can be aggregated by a backend specified by you. Depending on your use cases, you might decide to aggregate the metrics in various ways, but here we will go over some concepts and proto fields that would be useful in general to consider. ## Bazel’s query / cquery / aquery commands -Bazel provides 3 different query modes ([query](/query/quickstart), -[cquery](/query/cquery) and [aquery](/query/aquery)) that allow users -to query the target graph, configured target graph and action graph -respectively. The query language provides a -[suite of functions](/query/language#functions) usable across the different -query modes, that allows you to customize your queries according to your needs. +Bazel provides 3 different query modes ([query](/query/quickstart), [cquery](/query/cquery) and [aquery](/query/aquery)) that allow users to query the target graph, configured target graph and action graph respectively. The query language provides a [suite of functions](/query/language#functions) usable across the different query modes, that allows you to customize your queries according to your needs. ## JSON Trace Profiles -For every build-like Bazel invocation, Bazel writes a trace profile in JSON -format. The [JSON trace profile](/advanced/performance/json-trace-profile) can -be very useful to quickly understand what Bazel spent time on during the -invocation. +For every build-like Bazel invocation, Bazel writes a trace profile in JSON format. The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. ## Execution Log -The [execution log](/remote/cache-remote) can help you to troubleshoot and fix -missing remote cache hits due to machine and environment differences or -non-deterministic actions. If you pass the flag -[`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) -(available from Bazel 5.2) it will also contain detailed spawn metrics, both for -locally and remotely executed actions. You can use these metrics for example to -make comparisons between local and remote machine performance or to find out -which part of the spawn execution is consistently slower than expected (for -example due to queuing). +The [execution log](/remote/cache-remote) can help you to troubleshoot and fix missing remote cache hits due to machine and environment differences or non-deterministic actions. If you pass the flag [`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) (available from Bazel 5.2) it will also contain detailed spawn metrics, both for locally and remotely executed actions. You can use these metrics for example to make comparisons between local and remote machine performance or to find out which part of the spawn execution is consistently slower than expected (for example due to queuing). ## Execution Graph Log -While the JSON trace profile contains the critical path information, sometimes -you need additional information on the dependency graph of the executed actions. -Starting with Bazel 6.0, you can pass the flags -`--experimental_execution_graph_log` and -`--experimental_execution_graph_log_dep_type=all` to write out a log about the -executed actions and their inter-dependencies. +While the JSON trace profile contains the critical path information, sometimes you need additional information on the dependency graph of the executed actions. Starting with Bazel 6.0, you can pass the flags `--experimental_execution_graph_log` and `--experimental_execution_graph_log_dep_type=all` to write out a log about the executed actions and their inter-dependencies. -This information can be used to understand the drag that is added by a node on -the critical path. The drag is the amount of time that can potentially be saved -by removing a particular node from the execution graph. +This information can be used to understand the drag that is added by a node on the critical path. The drag is the amount of time that can potentially be saved by removing a particular node from the execution graph. -The data helps you predict the impact of changes to the build and action graph -before you actually do them. +The data helps you predict the impact of changes to the build and action graph before you actually do them. ## Benchmarking with bazel-bench -[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a -benchmarking tool for Git projects to benchmark build performance in the -following cases: +[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a benchmarking tool for Git projects to benchmark build performance in the following cases: -* **Project benchmark:** Benchmarking two git commits against each other at a - single Bazel version. Used to detect regressions in your build (often through - the addition of dependencies). +- **Project benchmark:** Benchmarking two git commits against each other at a single Bazel version. Used to detect regressions in your build (often through the addition of dependencies). -* **Bazel benchmark:** Benchmarking two versions of Bazel against each other at - a single git commit. Used to detect regressions within Bazel itself (if you - happen to maintain / fork Bazel). +- **Bazel benchmark:** Benchmarking two versions of Bazel against each other at a single git commit. Used to detect regressions within Bazel itself (if you happen to maintain / fork Bazel). -Benchmarks monitor wall time, CPU time and system time and Bazel’s retained -heap size. +Benchmarks monitor wall time, CPU time and system time and Bazel’s retained heap size. -It is also recommended to run Bazel bench on dedicated, physical machines that -are not running other processes so as to reduce sources of variability. +It is also recommended to run Bazel bench on dedicated, physical machines that are not running other processes so as to reduce sources of variability. diff --git a/advanced/performance/iteration-speed.mdx b/advanced/performance/iteration-speed.mdx index 2bbf8398..79240cd0 100644 --- a/advanced/performance/iteration-speed.mdx +++ b/advanced/performance/iteration-speed.mdx @@ -2,92 +2,44 @@ title: 'Optimize Iteration Speed' --- - - -This page describes how to optimize Bazel's build performance when running Bazel -repeatedly. +This page describes how to optimize Bazel's build performance when running Bazel repeatedly. ## Bazel's Runtime State A Bazel invocation involves several interacting parts. -* The `bazel` command line interface (CLI) is the user-facing front-end tool - and receives commands from the user. +- The `bazel` command line interface (CLI) is the user-facing front-end tool and receives commands from the user. -* The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) - for each distinct [output base](https://bazel.build/remote/output-directories). - The Bazel server is generally persistent, but will shut down after some idle - time so as to not waste resources. +- The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) for each distinct [output base](https://bazel.build/remote/output-directories). The Bazel server is generally persistent, but will shut down after some idle time so as to not waste resources. -* The Bazel server performs the loading and analysis steps for a given command - (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts - of the build graph in memory. The resulting data structures are retained in - the Bazel server as part of the *analysis cache*. +- The Bazel server performs the loading and analysis steps for a given command (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts of the build graph in memory. The resulting data structures are retained in the Bazel server as part of the *analysis cache*. -* The Bazel server can also perform the action execution, or it can send - actions off for remote execution if it is set up to do so. The results of - action executions are also cached, namely in the *action cache* (or - *execution cache*, which may be either local or remote, and it may be shared - among Bazel servers). +- The Bazel server can also perform the action execution, or it can send actions off for remote execution if it is set up to do so. The results of action executions are also cached, namely in the *action cache* (or *execution cache*, which may be either local or remote, and it may be shared among Bazel servers). -* The result of the Bazel invocation is made available in the output tree. +- The result of the Bazel invocation is made available in the output tree. ## Running Bazel Iteratively -In a typical developer workflow, it is common to build (or run) a piece of code -repeatedly, often at a very high frequency (e.g. to resolve some compilation -error or investigate a failing test). In this situation, it is important that -repeated invocations of `bazel` have as little overhead as possible relative to -the underlying, repeated action (e.g. invoking a compiler, or executing a test). +In a typical developer workflow, it is common to build (or run) a piece of code repeatedly, often at a very high frequency (e.g. to resolve some compilation error or investigate a failing test). In this situation, it is important that repeated invocations of `bazel` have as little overhead as possible relative to the underlying, repeated action (e.g. invoking a compiler, or executing a test). With this in mind, we take another look at Bazel's runtime state: -The analysis cache is a critical piece of data. A significant amount of time can -be spent just on the loading and analysis phases of a cold run (i.e. a run just -after the Bazel server was started or when the analysis cache was discarded). -For a single, successful cold build (e.g. for a production release) this cost is -bearable, but for repeatedly building the same target it is important that this -cost be amortized and not repeated on each invocation. - -The analysis cache is rather volatile. First off, it is part of the in-process -state of the Bazel server, so losing the server loses the cache. But the cache -is also *invalidated* very easily: for example, many `bazel` command line flags -cause the cache to be discarded. This is because many flags affect the build -graph (e.g. because of -[configurable attributes](https://bazel.build/configure/attributes)). Some flag -changes can also cause the Bazel server to be restarted (e.g. changing -[startup options](https://bazel.build/docs/user-manual#startup-options)). - -A good execution cache is also valuable for build performance. An execution -cache can be kept locally -[on disk](https://bazel.build/remote/caching#disk-cache), or -[remotely](https://bazel.build/remote/caching). The cache can be shared among -Bazel servers, and indeed among developers. +The analysis cache is a critical piece of data. A significant amount of time can be spent just on the loading and analysis phases of a cold run (i.e. a run just after the Bazel server was started or when the analysis cache was discarded). For a single, successful cold build (e.g. for a production release) this cost is bearable, but for repeatedly building the same target it is important that this cost be amortized and not repeated on each invocation. + +The analysis cache is rather volatile. First off, it is part of the in-process state of the Bazel server, so losing the server loses the cache. But the cache is also *invalidated* very easily: for example, many `bazel` command line flags cause the cache to be discarded. This is because many flags affect the build graph (e.g. because of [configurable attributes](https://bazel.build/configure/attributes)). Some flag changes can also cause the Bazel server to be restarted (e.g. changing [startup options](https://bazel.build/docs/user-manual#startup-options)). + +A good execution cache is also valuable for build performance. An execution cache can be kept locally [on disk](https://bazel.build/remote/caching#disk-cache), or [remotely](https://bazel.build/remote/caching). The cache can be shared among Bazel servers, and indeed among developers. ## Avoid discarding the analysis cache -Bazel will print a warning if either the analysis cache was discarded or the -server was restarted. Either of these should be avoided during iterative use: +Bazel will print a warning if either the analysis cache was discarded or the server was restarted. Either of these should be avoided during iterative use: -* Be mindful of changing `bazel` flags in the middle of an iterative - workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` - causes each command to discard the analysis cache of the other. In general, - try to use a fixed set of flags for the duration of a particular workflow. +- Be mindful of changing `bazel` flags in the middle of an iterative workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` causes each command to discard the analysis cache of the other. In general, try to use a fixed set of flags for the duration of a particular workflow. -* Losing the Bazel server loses the analysis cache. The Bazel server has a - [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle - time, after which it shuts down. You can configure this time via your - bazelrc file to suit your needs. The server also restarted when startup - flags change, so, again, avoid changing those flags if possible. +- Losing the Bazel server loses the analysis cache. The Bazel server has a [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle time, after which it shuts down. You can configure this time via your bazelrc file to suit your needs. The server also restarted when startup flags change, so, again, avoid changing those flags if possible. -* Beware that the Bazel server is killed if you press - Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time - by interrupting a running build that is no longer needed, but only press - Ctrl-C once to request a graceful end of the current invocation. +- [Beware]() that the Bazel server is killed if you press Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time by interrupting a running build that is no longer needed, but only press Ctrl-C once to request a graceful end of the current invocation. -* If you want to use multiple sets of flags from the same workspace, you can - use multiple, distinct output bases, switched with the `--output_base` - flag. Each output base gets its own Bazel server. +- If you want to use multiple sets of flags from the same workspace, you can use multiple, distinct output bases, switched with the `--output_base` flag. Each output base gets its own Bazel server. -To make this condition an error rather than a warning, you can use the -`--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) +To make this condition an error rather than a warning, you can use the `--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) diff --git a/advanced/performance/json-trace-profile.mdx b/advanced/performance/json-trace-profile.mdx index 80c698c0..c452c4b0 100644 --- a/advanced/performance/json-trace-profile.mdx +++ b/advanced/performance/json-trace-profile.mdx @@ -2,34 +2,17 @@ title: 'JSON Trace Profile' --- +The JSON trace profile can be very useful to quickly understand what Bazel spent time on during the invocation. - -The JSON trace profile can be very useful to quickly understand what Bazel spent -time on during the invocation. - -By default, for all build-like commands and query, Bazel writes a profile into -the output base named `command-$INVOCATION_ID.profile.gz`, where -`$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates -a symlink called `command.profile.gz` in the output base that points the profile -of the latest command. You can configure whether a profile is written with the -[`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) -flag, and the location it is written to with the -[`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are -compressed with GZIP. Bazel keeps the last 5 profiles, configurable by -[`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), -in the output base by default for post-build analysis. Explicitly passing a -profile path with `--profile` disables automatic garbage collection. +By default, for all build-like commands and query, Bazel writes a profile into the output base named `command-$INVOCATION_ID.profile.gz`, where `$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates a symlink called `command.profile.gz` in the output base that points the profile of the latest command. You can configure whether a profile is written with the [`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) flag, and the location it is written to with the [`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are compressed with GZIP. Bazel keeps the last 5 profiles, configurable by [`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), in the output base by default for post-build analysis. Explicitly passing a profile path with `--profile` disables automatic garbage collection. ## Tools -You can load this profile into `chrome://tracing` or analyze and -post-process it with other tools. +You can load this profile into `chrome://tracing` or analyze and post-process it with other tools. ### `chrome://tracing` -To visualize the profile, open `chrome://tracing` in a Chrome browser tab, -click "Load" and pick the (potentially compressed) profile file. For more -detailed results, click the boxes in the lower left corner. +To visualize the profile, open `chrome://tracing` in a Chrome browser tab, click "Load" and pick the (potentially compressed) profile file. For more detailed results, click the boxes in the lower left corner. Example profile: @@ -39,29 +22,19 @@ Example profile: You can use these keyboard controls to navigate: -* Press `1` for "select" mode. In this mode, you can select - particular boxes to inspect the event details (see lower left corner). - Select multiple events to get a summary and aggregated statistics. -* Press `2` for "pan" mode. Then drag the mouse to move the view. You - can also use `a`/`d` to move left/right. -* Press `3` for "zoom" mode. Then drag the mouse to zoom. You can - also use `w`/`s` to zoom in/out. -* Press `4` for "timing" mode where you can measure the distance - between two events. -* Press `?` to learn about all controls. +- Press `1` for "select" mode. In this mode, you can select particular boxes to inspect the event details (see lower left corner). Select multiple events to get a summary and aggregated statistics. +- Press `2` for "pan" mode. Then drag the mouse to move the view. You can also use `a`/`d` to move left/right. +- Press `3` for "zoom" mode. Then drag the mouse to zoom. You can also use `w`/`s` to zoom in/out. +- Press `4` for "timing" mode where you can measure the distance between two events. +- Press `?` to learn about all controls. ### Bazel Invocation Analyzer -The open-source -[Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) -consumes a profile format and prints suggestions on how to improve -the build’s performance. This analysis can be performed using its CLI or on -[https://analyzer.engflow.com](https://analyzer.engflow.com). +The open-source [Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) consumes a profile format and prints suggestions on how to improve the build’s performance. This analysis can be performed using its CLI or on [https://analyzer.engflow.com](https://analyzer.engflow.com). ### `jq` -`jq` is like `sed` for JSON data. An example usage of `jq` to extract all -durations of the sandbox creation step in local action execution: +`jq` is like `sed` for JSON data. An example usage of `jq` to extract all durations of the sandbox creation step in local action execution: ``` $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | jq '.traceEvents | .[] | select(.name == "sandbox.createFileSystem") | .dur' @@ -78,50 +51,29 @@ $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | j ## Profile information -The profile contains multiple rows. Usually the bulk of rows represent Bazel -threads and their corresponding events, but some special rows are also included. +The profile contains multiple rows. Usually the bulk of rows represent Bazel threads and their corresponding events, but some special rows are also included. -The special rows included depend on the version of Bazel invoked when the -profile was created, and may be customized by different flags. +The special rows included depend on the version of Bazel invoked when the profile was created, and may be customized by different flags. Figure 1 shows a profile created with Bazel v5.3.1 and includes these rows: -* `action count`: Displays how many concurrent actions were in flight. Click - on it to see the actual value. Should go up to the value of - [`--jobs`](/reference/command-line-reference#flag--jobs) in clean - builds. -* `CPU usage (Bazel)`: For each second of the build, displays the amount of - CPU that was used by Bazel (a value of 1 equals one core being 100% busy). -* `Critical Path`: Displays one block for each action on the critical path. -* `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of - what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", - and "runAnalysisPhase". -* `Garbage Collector`: Displays minor and major Garbage Collection (GC) - pauses. +- `action count`: Displays how many concurrent actions were in flight. Click on it to see the actual value. Should go up to the value of [`--jobs`](/reference/command-line-reference#flag--jobs) in clean builds. +- `CPU usage (Bazel)`: For each second of the build, displays the amount of CPU that was used by Bazel (a value of 1 equals one core being 100% busy). +- `Critical Path`: Displays one block for each action on the critical path. +- `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", and "runAnalysisPhase". +- `Garbage Collector`: Displays minor and major Garbage Collection (GC) pauses. ## Common performance issues When analyzing performance profiles, look for: -* Slower than expected analysis phase (`runAnalysisPhase`), especially on - incremental builds. This can be a sign of a poor rule implementation, for - example one that flattens depsets. Package loading can be slow by an - excessive amount of targets, complex macros or recursive globs. -* Individual slow actions, especially those on the critical path. It might be - possible to split large actions into multiple smaller actions or reduce the - set of (transitive) dependencies to speed them up. Also check for an unusual - high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). -* Bottlenecks, that is a small number of threads is busy while all others are - idling / waiting for the result (see around 22s and 29s in Figure 1). - Optimizing this will most likely require touching the rule implementations - or Bazel itself to introduce more parallelism. This can also happen when - there is an unusual amount of GC. +- Slower than expected analysis phase (`runAnalysisPhase`), especially on incremental builds. This can be a sign of a poor rule implementation, for example one that flattens depsets. Package loading can be slow by an excessive amount of targets, complex macros or recursive globs. +- Individual slow actions, especially those on the critical path. It might be possible to split large actions into multiple smaller actions or reduce the set of (transitive) dependencies to speed them up. Also check for an unusual high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). +- Bottlenecks, that is a small number of threads is busy while all others are idling / waiting for the result (see around 22s and 29s in Figure 1). Optimizing this will most likely require touching the rule implementations or Bazel itself to introduce more parallelism. This can also happen when there is an unusual amount of GC. ## Profile file format -The top-level object contains metadata (`otherData`) and the actual tracing data -(`traceEvents`). The metadata contains extra info, for example the invocation ID -and date of the Bazel invocation. +The top-level object contains metadata (`otherData`) and the actual tracing data (`traceEvents`). The metadata contains extra info, for example the invocation ID and date of the Bazel invocation. Example: @@ -148,12 +100,6 @@ Example: } ``` -Timestamps (`ts`) and durations (`dur`) in the trace events are given in -microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. -Note that some events are merged together if they are very short and close to -each other; pass -[`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) -if you would like to prevent event merging. +Timestamps (`ts`) and durations (`dur`) in the trace events are given in microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. Note that some events are merged together if they are very short and close to each other; pass [`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) if you would like to prevent event merging. -See also the -[Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). +See also the [Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). diff --git a/advanced/performance/memory.mdx b/advanced/performance/memory.mdx index 844e691b..8f6c8b5f 100644 --- a/advanced/performance/memory.mdx +++ b/advanced/performance/memory.mdx @@ -2,50 +2,27 @@ title: 'Optimize Memory' --- - - This page describes how to limit and reduce the memory Bazel uses. ## Running Bazel with Limited RAM -In certain situations, you may want Bazel to use minimal memory. You can set the -maximum heap via the startup flag -[`--host_jvm_args`](/docs/user-manual#host-jvm-args), -like `--host_jvm_args=-Xmx2g`. +In certain situations, you may want Bazel to use minimal memory. You can set the maximum heap via the startup flag [`--host_jvm_args`](/docs/user-manual#host-jvm-args), like `--host_jvm_args=-Xmx2g`. ### Trade incremental build speeds for memory -If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when -it doesn't have enough memory. You can make Bazel use less memory, at the cost -of slower incremental builds, by passing the following command flags: -[`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), -[`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), -and -[`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). +If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when it doesn't have enough memory. You can make Bazel use less memory, at the cost of slower incremental builds, by passing the following command flags: [`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), [`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), and [`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). -These flags will minimize the memory that Bazel uses in a build, at the cost of -making future builds slower than a standard incremental build would be. +These flags will minimize the memory that Bazel uses in a build, at the cost of making future builds slower than a standard incremental build would be. You can also pass any one of these flags individually: - * `--discard_analysis_cache` will reduce the memory used during execution (not -analysis). Incremental builds will not have to redo package loading, but will -have to redo analysis and execution (although the on-disk action cache can -prevent most re-execution). - * `--notrack_incremental_state` will not store any edges in Bazel's internal - dependency graph, so that it is unusable for incremental builds. The next build - will discard that data, but it is preserved until then, for internal debugging, - unless `--nokeep_state_after_build` is specified. - * `--nokeep_state_after_build` will discard all data after the build, so that - incremental builds have to build from scratch (except for the on-disk action - cache). Alone, it does not affect the high-water mark of the current build. +- `--discard_analysis_cache` will reduce the memory used during execution (not analysis). Incremental builds will not have to redo package loading, but will have to redo analysis and execution (although the on-disk action cache can prevent most re-execution). +- `--notrack_incremental_state` will not store any edges in Bazel's internal dependency graph, so that it is unusable for incremental builds. The next build will discard that data, but it is preserved until then, for internal debugging, unless `--nokeep_state_after_build` is specified. +- `--nokeep_state_after_build` will discard all data after the build, so that incremental builds have to build from scratch (except for the on-disk action cache). Alone, it does not affect the high-water mark of the current build. ### Trade build flexibility for memory with Skyfocus (Experimental) -If you want to make Bazel use less memory *and* retain incremental build speeds, -you can tell Bazel the working set of files that you will be modifying, and -Bazel will only keep state needed to correctly incrementally rebuild changes to -those files. This feature is called **Skyfocus**. +If you want to make Bazel use less memory *and* retain incremental build speeds, you can tell Bazel the working set of files that you will be modifying, and Bazel will only keep state needed to correctly incrementally rebuild changes to those files. This feature is called **Skyfocus**. To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: @@ -53,21 +30,16 @@ To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: bazel build //pkg:target --experimental_enable_skyfocus ``` -By default, the working set will be the set of files next to the target being -built. In the example, all files in `//pkg` will be kept in the working set, and -changes to files outside of the working set will be disallowed, until you issue -`bazel clean` or restart the Bazel server. +By default, the working set will be the set of files next to the target being built. In the example, all files in `//pkg` will be kept in the working set, and changes to files outside of the working set will be disallowed, until you issue `bazel clean` or restart the Bazel server. -If you want to specify an exact set of files or directories, use the -`--experimental_working_set` flag, like so: +If you want to specify an exact set of files or directories, use the `--experimental_working_set` flag, like so: ```sh bazel build //pkg:target --experimental_enable_skyfocus --experimental_working_set=path/to/another/dir,path/to/tests/dir ``` -You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the -memory reduction amount: +You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the memory reduction amount: Putting it altogether, you should see something like this: @@ -79,19 +51,13 @@ INFO: Analyzed 149 targets (4533 packages loaded, 169438 targets configured). INFO: Found 25 targets and 124 test targets... INFO: Updated working set successfully. INFO: Focusing on 334 roots, 3 leafs... (use --experimental_skyfocus_dump_keys to show them) -INFO: Heap: 1237MB -> 676MB (-45.31%) +INFO: Heap: 1237MB -> 676MB (-45.31%) INFO: Elapsed time: 192.670s ... INFO: Build completed successfully, 62303 total actions ``` -For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, -and incremental builds to handle changes to files under `dir1`, `dir2`, and -`dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot -rebuild changed files outside of these directories. +For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, and incremental builds to handle changes to files under `dir1`, `dir2`, and `dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot rebuild changed files outside of these directories. ## Memory Profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s -memory use. Read more about this process on the -[Memory Profiling section](/rules/performance#memory-profiling) of our -documentation on how to improve the performance of custom rules. +Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. Read more about this process on the [Memory Profiling section](/rules/performance#memory-profiling) of our documentation on how to improve the performance of custom rules. diff --git a/basics/artifact-based-builds.mdx b/basics/artifact-based-builds.mdx index 4176a288..5dfc5896 100644 --- a/basics/artifact-based-builds.mdx +++ b/basics/artifact-based-builds.mdx @@ -2,57 +2,19 @@ title: 'Artifact-Based Build Systems' --- +This page covers artifact-based build systems and the philosophy behind their creation. Bazel is an artifact-based build system. While task-based build systems are good step above build scripts, they give too much power to individual engineers by letting them define their own tasks. - -This page covers artifact-based build systems and the philosophy behind their -creation. Bazel is an artifact-based build system. While task-based build -systems are good step above build scripts, they give too much power to -individual engineers by letting them define their own tasks. - -Artifact-based build systems have a small number of tasks defined by the system -that engineers can configure in a limited way. Engineers still tell the system -**what** to build, but the build system determines **how** to build it. As with -task-based build systems, artifact-based build systems, such as Bazel, still -have buildfiles, but the contents of those buildfiles are very different. Rather -than being an imperative set of commands in a Turing-complete scripting language -describing how to produce an output, buildfiles in Bazel are a declarative -manifest describing a set of artifacts to build, their dependencies, and a -limited set of options that affect how they’re built. When engineers run `bazel` -on the command line, they specify a set of targets to build (the **what**), and -Bazel is responsible for configuring, running, and scheduling the compilation -steps (the **how**). Because the build system now has full control over what -tools to run when, it can make much stronger guarantees that allow it to be far -more efficient while still guaranteeing correctness. +Artifact-based build systems have a small number of tasks defined by the system that engineers can configure in a limited way. Engineers still tell the system **what** to build, but the build system determines **how** to build it. As with task-based build systems, artifact-based build systems, such as Bazel, still have buildfiles, but the contents of those buildfiles are very different. Rather than being an imperative set of commands in a Turing-complete scripting language describing how to produce an output, buildfiles in Bazel are a declarative manifest describing a set of artifacts to build, their dependencies, and a limited set of options that affect how they’re built. When engineers run `bazel` on the command line, they specify a set of targets to build (the **what**), and Bazel is responsible for configuring, running, and scheduling the compilation steps (the **how**). Because the build system now has full control over what tools to run when, it can make much stronger guarantees that allow it to be far more efficient while still guaranteeing correctness. ## A functional perspective -It’s easy to make an analogy between artifact-based build systems and functional -programming. Traditional imperative programming languages (such as, Java, C, and -Python) specify lists of statements to be executed one after another, in the -same way that task-based build systems let programmers define a series of steps -to execute. Functional programming languages (such as, Haskell and ML), in -contrast, are structured more like a series of mathematical equations. In -functional languages, the programmer describes a computation to perform, but -leaves the details of when and exactly how that computation is executed to the -compiler. - -This maps to the idea of declaring a manifest in an artifact-based build system -and letting the system figure out how to execute the build. Many problems can't -be easily expressed using functional programming, but the ones that do benefit -greatly from it: the language is often able to trivially parallelize such -programs and make strong guarantees about their correctness that would be -impossible in an imperative language. The easiest problems to express using -functional programming are the ones that simply involve transforming one piece -of data into another using a series of rules or functions. And that’s exactly -what a build system is: the whole system is effectively a mathematical function -that takes source files (and tools like the compiler) as inputs and produces -binaries as outputs. So, it’s not surprising that it works well to base a build -system around the tenets of functional programming. +It’s easy to make an analogy between artifact-based build systems and functional programming. Traditional imperative programming languages (such as, Java, C, and Python) specify lists of statements to be executed one after another, in the same way that task-based build systems let programmers define a series of steps to execute. Functional programming languages (such as, Haskell and ML), in contrast, are structured more like a series of mathematical equations. In functional languages, the programmer describes a computation to perform, but leaves the details of when and exactly how that computation is executed to the compiler. + +This maps to the idea of declaring a manifest in an artifact-based build system and letting the system figure out how to execute the build. Many problems can't be easily expressed using functional programming, but the ones that do benefit greatly from it: the language is often able to trivially parallelize such programs and make strong guarantees about their correctness that would be impossible in an imperative language. The easiest problems to express using functional programming are the ones that simply involve transforming one piece of data into another using a series of rules or functions. And that’s exactly what a build system is: the whole system is effectively a mathematical function that takes source files (and tools like the compiler) as inputs and produces binaries as outputs. So, it’s not surprising that it works well to base a build system around the tenets of functional programming. ## Understanding artifact-based build systems -Google's build system, Blaze, was the first artifact-based build system. Bazel -is the open-sourced version of Blaze. +Google's build system, Blaze, was the first artifact-based build system. Bazel is the open-sourced version of Blaze. Here’s what a buildfile (normally named `BUILD`) looks like in Bazel: @@ -75,205 +37,64 @@ java_library( ) ``` -In Bazel, `BUILD` files define targets—the two types of targets here are -`java_binary` and `java_library`. Every target corresponds to an artifact that -can be created by the system: binary targets produce binaries that can be -executed directly, and library targets produce libraries that can be used by -binaries or other libraries. Every target has: - -* `name`: how the target is referenced on the command line and by other - targets -* `srcs`: the source files to be compiled to create the artifact for the target -* `deps`: other targets that must be built before this target and linked into - it - -Dependencies can either be within the same package (such as `MyBinary`’s -dependency on `:mylib`) or on a different package in the same source hierarchy -(such as `mylib`’s dependency on `//java/com/example/common`). - -As with task-based build systems, you perform builds using Bazel’s command-line -tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After -entering that command for the first time in a clean repository, Bazel: - -1. Parses every `BUILD` file in the workspace to create a graph of dependencies - among artifacts. -1. Uses the graph to determine the transitive dependencies of `MyBinary`; that - is, every target that `MyBinary` depends on and every target that those - targets depend on, recursively. -1. Builds each of those dependencies, in order. Bazel starts by building each - target that has no other dependencies and keeps track of which dependencies - still need to be built for each target. As soon as all of a target’s - dependencies are built, Bazel starts building that target. This process - continues until every one of `MyBinary`’s transitive dependencies have been - built. -1. Builds `MyBinary` to produce a final executable binary that links in all of - the dependencies that were built in step 3. - -Fundamentally, it might not seem like what’s happening here is that much -different than what happened when using a task-based build system. Indeed, the -end result is the same binary, and the process for producing it involved -analyzing a bunch of steps to find dependencies among them, and then running -those steps in order. But there are critical differences. The first one appears -in step 3: because Bazel knows that each target only produces a Java library, it -knows that all it has to do is run the Java compiler rather than an arbitrary -user-defined script, so it knows that it’s safe to run these steps in parallel. -This can produce an order of magnitude performance improvement over building -targets one at a time on a multicore machine, and is only possible because the -artifact-based approach leaves the build system in charge of its own execution -strategy so that it can make stronger guarantees about parallelism. - -The benefits extend beyond parallelism, though. The next thing that this -approach gives us becomes apparent when the developer types `bazel -build :MyBinary` a second time without making any changes: Bazel exits in less -than a second with a message saying that the target is up to date. This is -possible due to the functional programming paradigm we talked about -earlier—Bazel knows that each target is the result only of running a Java -compiler, and it knows that the output from the Java compiler depends only on -its inputs, so as long as the inputs haven’t changed, the output can be reused. -And this analysis works at every level; if `MyBinary.java` changes, Bazel knows -to rebuild `MyBinary` but reuse `mylib`. If a source file for -`//java/com/example/common` changes, Bazel knows to rebuild that library, -`mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. -Because Bazel knows about the properties of the tools it runs at every step, -it’s able to rebuild only the minimum set of artifacts each time while -guaranteeing that it won’t produce stale builds. - -Reframing the build process in terms of artifacts rather than tasks is subtle -but powerful. By reducing the flexibility exposed to the programmer, the build -system can know more about what is being done at every step of the build. It can -use this knowledge to make the build far more efficient by parallelizing build -processes and reusing their outputs. But this is really just the first step, and -these building blocks of parallelism and reuse form the basis for a distributed -and highly scalable build system. +In Bazel, `BUILD` files define targets—the two types of targets here are `java_binary` and `java_library`. Every target corresponds to an artifact that can be created by the system: binary targets produce binaries that can be executed directly, and library targets produce libraries that can be used by binaries or other libraries. Every target has: + +- `name`: how the target is referenced on the command line and by other targets +- `srcs`: the source files to be compiled to create the artifact for the target +- `deps`: other targets that must be built before this target and linked into it + +Dependencies can either be within the same package (such as `MyBinary`’s dependency on `:mylib`) or on a different package in the same source hierarchy (such as `mylib`’s dependency on `//java/com/example/common`). + +As with task-based build systems, you perform builds using Bazel’s command-line tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After entering that command for the first time in a clean repository, Bazel: + +1. Parses every `BUILD` file in the workspace to create a graph of dependencies among artifacts. +2. Uses the graph to determine the transitive dependencies of `MyBinary`; that is, every target that `MyBinary` depends on and every target that those targets depend on, recursively. +3. Builds each of those dependencies, in order. Bazel starts by building each target that has no other dependencies and keeps track of which dependencies still need to be built for each target. As soon as all of a target’s dependencies are built, Bazel starts building that target. This process continues until every one of `MyBinary`’s transitive dependencies have been built. +4. Builds `MyBinary` to produce a final executable binary that links in all of the dependencies that were built in step 3. + +Fundamentally, it might not seem like what’s happening here is that much different than what happened when using a task-based build system. Indeed, the end result is the same binary, and the process for producing it involved analyzing a bunch of steps to find dependencies among them, and then running those steps in order. But there are critical differences. The first one appears in step 3: because Bazel knows that each target only produces a Java library, it knows that all it has to do is run the Java compiler rather than an arbitrary user-defined script, so it knows that it’s safe to run these steps in parallel. This can produce an order of magnitude performance improvement over building targets one at a time on a multicore machine, and is only possible because the artifact-based approach leaves the build system in charge of its own execution strategy so that it can make stronger guarantees about parallelism. + +The benefits extend beyond parallelism, though. The next thing that this approach gives us becomes apparent when the developer types `bazel build :MyBinary` a second time without making any changes: Bazel exits in less than a second with a message saying that the target is up to date. This is possible due to the functional programming paradigm we talked about earlier—Bazel knows that each target is the result only of running a Java compiler, and it knows that the output from the Java compiler depends only on its inputs, so as long as the inputs haven’t changed, the output can be reused. And this analysis works at every level; if `MyBinary.java` changes, Bazel knows to rebuild `MyBinary` but reuse `mylib`. If a source file for `//java/com/example/common` changes, Bazel knows to rebuild that library, `mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. Because Bazel knows about the properties of the tools it runs at every step, it’s able to rebuild only the minimum set of artifacts each time while guaranteeing that it won’t produce stale builds. + +Reframing the build process in terms of artifacts rather than tasks is subtle but powerful. By reducing the flexibility exposed to the programmer, the build system can know more about what is being done at every step of the build. It can use this knowledge to make the build far more efficient by parallelizing build processes and reusing their outputs. But this is really just the first step, and these building blocks of parallelism and reuse form the basis for a distributed and highly scalable build system. ## Other nifty Bazel tricks -Artifact-based build systems fundamentally solve the problems with parallelism -and reuse that are inherent in task-based build systems. But there are still a -few problems that came up earlier that we haven’t addressed. Bazel has clever -ways of solving each of these, and we should discuss them before moving on. +Artifact-based build systems fundamentally solve the problems with parallelism and reuse that are inherent in task-based build systems. But there are still a few problems that came up earlier that we haven’t addressed. Bazel has clever ways of solving each of these, and we should discuss them before moving on. ### Tools as dependencies -One problem we ran into earlier was that builds depended on the tools installed -on our machine, and reproducing builds across systems could be difficult due to -different tool versions or locations. The problem becomes even more difficult -when your project uses languages that require different tools based on which -platform they’re being built on or compiled for (such as, Windows versus Linux), -and each of those platforms requires a slightly different set of tools to do the -same job. +One problem we ran into earlier was that builds depended on the tools installed on our machine, and reproducing builds across systems could be difficult due to different tool versions or locations. The problem becomes even more difficult when your project uses languages that require different tools based on which platform they’re being built on or compiled for (such as, Windows versus Linux), and each of those platforms requires a slightly different set of tools to do the same job. -Bazel solves the first part of this problem by treating tools as dependencies to -each target. Every `java_library` in the workspace implicitly depends on a Java -compiler, which defaults to a well-known compiler. Whenever Bazel builds a -`java_library`, it checks to make sure that the specified compiler is available -at a known location. Just like any other dependency, if the Java compiler -changes, every artifact that depends on it is rebuilt. +Bazel solves the first part of this problem by treating tools as dependencies to each target. Every `java_library` in the workspace implicitly depends on a Java compiler, which defaults to a well-known compiler. Whenever Bazel builds a `java_library`, it checks to make sure that the specified compiler is available at a known location. Just like any other dependency, if the Java compiler changes, every artifact that depends on it is rebuilt. -Bazel solves the second part of the problem, platform independence, by setting -[build configurations](/run/build#build-config-cross-compilation). Rather than -targets depending directly on their tools, they depend on types of configurations: +Bazel solves the second part of the problem, platform independence, by setting [build configurations](/run/build#build-config-cross-compilation). Rather than targets depending directly on their tools, they depend on types of configurations: -* **Host configuration**: building tools that run during the build -* **Target configuration**: building the binary you ultimately requested +- **Host configuration**: building tools that run during the build +- **Target configuration**: building the binary you ultimately requested ### Extending the build system -Bazel comes with targets for several popular programming languages out of the -box, but engineers will always want to do more—part of the benefit of task-based -systems is their flexibility in supporting any kind of build process, and it -would be better not to give that up in an artifact-based build system. -Fortunately, Bazel allows its supported target types to be extended by -[adding custom rules](/extending/rules). - -To define a rule in Bazel, the rule author declares the inputs that the rule -requires (in the form of attributes passed in the `BUILD` file) and the fixed -set of outputs that the rule produces. The author also defines the actions that -will be generated by that rule. Each action declares its inputs and outputs, -runs a particular executable or writes a particular string to a file, and can be -connected to other actions via its inputs and outputs. This means that actions -are the lowest-level composable unit in the build system—an action can do -whatever it wants so long as it uses only its declared inputs and outputs, and -Bazel takes care of scheduling actions and caching their results as appropriate. - -The system isn’t foolproof given that there’s no way to stop an action developer -from doing something like introducing a nondeterministic process as part of -their action. But this doesn’t happen very often in practice, and pushing the -possibilities for abuse all the way down to the action level greatly decreases -opportunities for errors. Rules supporting many common languages and tools are -widely available online, and most projects will never need to define their own -rules. Even for those that do, rule definitions only need to be defined in one -central place in the repository, meaning most engineers will be able to use -those rules without ever having to worry about their implementation. +Bazel comes with targets for several popular programming languages out of the box, but engineers will always want to do more—part of the benefit of task-based systems is their flexibility in supporting any kind of build process, and it would be better not to give that up in an artifact-based build system. Fortunately, Bazel allows its supported target types to be extended by [adding custom rules](/extending/rules). + +To define a rule in Bazel, the rule author declares the inputs that the rule requires (in the form of attributes passed in the `BUILD` file) and the fixed set of outputs that the rule produces. The author also defines the actions that will be generated by that rule. Each action declares its inputs and outputs, runs a particular executable or writes a particular string to a file, and can be connected to other actions via its inputs and outputs. This means that actions are the lowest-level composable unit in the build system—an action can do whatever it wants so long as it uses only its declared inputs and outputs, and Bazel takes care of scheduling actions and caching their results as appropriate. + +The system isn’t foolproof given that there’s no way to stop an action developer from doing something like introducing a nondeterministic process as part of their action. But this doesn’t happen very often in practice, and pushing the possibilities for abuse all the way down to the action level greatly decreases opportunities for errors. Rules supporting many common languages and tools are widely available online, and most projects will never need to define their own rules. Even for those that do, rule definitions only need to be defined in one central place in the repository, meaning most engineers will be able to use those rules without ever having to worry about their implementation. ### Isolating the environment -Actions sound like they might run into the same problems as tasks in other -systems—isn’t it still possible to write actions that both write to the same -file and end up conflicting with one another? Actually, Bazel makes these -conflicts impossible by using _[sandboxing](/docs/sandboxing)_. On supported -systems, every action is isolated from every other action via a filesystem -sandbox. Effectively, each action can see only a restricted view of the -filesystem that includes the inputs it has declared and any outputs it has -produced. This is enforced by systems such as LXC on Linux, the same technology -behind Docker. This means that it’s impossible for actions to conflict with one -another because they are unable to read any files they don’t declare, and any -files that they write but don’t declare will be thrown away when the action -finishes. Bazel also uses sandboxes to restrict actions from communicating via -the network. +Actions sound like they might run into the same problems as tasks in other systems—isn’t it still possible to write actions that both write to the same file and end up conflicting with one another? Actually, Bazel makes these conflicts impossible by using *[sandboxing](/docs/sandboxing)*. On supported systems, every action is isolated from every other action via a filesystem sandbox. Effectively, each action can see only a restricted view of the filesystem that includes the inputs it has declared and any outputs it has produced. This is enforced by systems such as LXC on Linux, the same technology behind Docker. This means that it’s impossible for actions to conflict with one another because they are unable to read any files they don’t declare, and any files that they write but don’t declare will be thrown away when the action finishes. Bazel also uses sandboxes to restrict actions from communicating via the network. ### Making external dependencies deterministic -There’s still one problem remaining: build systems often need to download -dependencies (whether tools or libraries) from external sources rather than -directly building them. This can be seen in the example via the -`@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file -from Maven. - -Depending on files outside of the current workspace is risky. Those files could -change at any time, potentially requiring the build system to constantly check -whether they’re fresh. If a remote file changes without a corresponding change -in the workspace source code, it can also lead to unreproducible builds—a build -might work one day and fail the next for no obvious reason due to an unnoticed -dependency change. Finally, an external dependency can introduce a huge security -risk when it is owned by a third party: if an attacker is able to infiltrate -that third-party server, they can replace the dependency file with something of -their own design, potentially giving them full control over your build -environment and its output. - -The fundamental problem is that we want the build system to be aware of these -files without having to check them into source control. Updating a dependency -should be a conscious choice, but that choice should be made once in a central -place rather than managed by individual engineers or automatically by the -system. This is because even with a “Live at Head” model, we still want builds -to be deterministic, which implies that if you check out a commit from last -week, you should see your dependencies as they were then rather than as they are -now. - -Bazel and some other build systems address this problem by requiring a -workspacewide manifest file that lists a _cryptographic hash_ for every external -dependency in the workspace. The hash is a concise way to uniquely represent the -file without checking the entire file into source control. Whenever a new -external dependency is referenced from a workspace, that dependency’s hash is -added to the manifest, either manually or automatically. When Bazel runs a -build, it checks the actual hash of its cached dependency against the expected -hash defined in the manifest and redownloads the file only if the hash differs. - -If the artifact we download has a different hash than the one declared in the -manifest, the build will fail unless the hash in the manifest is updated. This -can be done automatically, but that change must be approved and checked into -source control before the build will accept the new dependency. This means that -there’s always a record of when a dependency was updated, and an external -dependency can’t change without a corresponding change in the workspace source. -It also means that, when checking out an older version of the source code, the -build is guaranteed to use the same dependencies that it was using at the point -when that version was checked in (or else it will fail if those dependencies are -no longer available). - -Of course, it can still be a problem if a remote server becomes unavailable or -starts serving corrupt data—this can cause all of your builds to begin failing -if you don’t have another copy of that dependency available. To avoid this -problem, we recommend that, for any nontrivial project, you mirror all of its -dependencies onto servers or services that you trust and control. Otherwise you -will always be at the mercy of a third party for your build system’s -availability, even if the checked-in hashes guarantee its security. +There’s still one problem remaining: build systems often need to download dependencies (whether tools or libraries) from external sources rather than directly building them. This can be seen in the example via the `@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file from Maven. + +Depending on files outside of the current workspace is risky. Those files could change at any time, potentially requiring the build system to constantly check whether they’re fresh. If a remote file changes without a corresponding change in the workspace source code, it can also lead to unreproducible builds—a build might work one day and fail the next for no obvious reason due to an unnoticed dependency change. Finally, an external dependency can introduce a huge security risk when it is owned by a third party: if an attacker is able to infiltrate that third-party server, they can replace the dependency file with something of their own design, potentially giving them full control over your build environment and its output. + +The fundamental problem is that we want the build system to be aware of these files without having to check them into source control. Updating a dependency should be a conscious choice, but that choice should be made once in a central place rather than managed by individual engineers or automatically by the system. This is because even with a “Live at Head” model, we still want builds to be deterministic, which implies that if you check out a commit from last week, you should see your dependencies as they were then rather than as they are now. + +Bazel and some other build systems address this problem by requiring a workspacewide manifest file that lists a *cryptographic hash* for every external dependency in the workspace. The hash is a concise way to uniquely represent the file without checking the entire file into source control. Whenever a new external dependency is referenced from a workspace, that dependency’s hash is added to the manifest, either manually or automatically. When Bazel runs a build, it checks the actual hash of its cached dependency against the expected hash defined in the manifest and redownloads the file only if the hash differs. + +If the artifact we download has a different hash than the one declared in the manifest, the build will fail unless the hash in the manifest is updated. This can be done automatically, but that change must be approved and checked into source control before the build will accept the new dependency. This means that there’s always a record of when a dependency was updated, and an external dependency can’t change without a corresponding change in the workspace source. It also means that, when checking out an older version of the source code, the build is guaranteed to use the same dependencies that it was using at the point when that version was checked in (or else it will fail if those dependencies are no longer available). + +Of course, it can still be a problem if a remote server becomes unavailable or starts serving corrupt data—this can cause all of your builds to begin failing if you don’t have another copy of that dependency available. To avoid this problem, we recommend that, for any nontrivial project, you mirror all of its dependencies onto servers or services that you trust and control. Otherwise you will always be at the mercy of a third party for your build system’s availability, even if the checked-in hashes guarantee its security. diff --git a/basics/build-systems.mdx b/basics/build-systems.mdx index b3c63389..e2f80cb6 100644 --- a/basics/build-systems.mdx +++ b/basics/build-systems.mdx @@ -2,126 +2,44 @@ title: 'Why a Build System?' --- - - -This page discusses what build systems are, what they do, why you should use a -build system, and why compilers and build scripts aren't the best choice as your -organization starts to scale. It's intended for developers who don't have much -experience with a build system. +This page discusses what build systems are, what they do, why you should use a build system, and why compilers and build scripts aren't the best choice as your organization starts to scale. It's intended for developers who don't have much experience with a build system. ## What is a build system? -Fundamentally, all build systems have a straightforward purpose: they transform -the source code written by engineers into executable binaries that can be read -by machines. Build systems aren't just for human-authored code; they also allow -machines to create builds automatically, whether for testing or for releases to -production. In an organization with thousands of engineers, it's common that -most builds are triggered automatically rather than directly by engineers. +Fundamentally, all build systems have a straightforward purpose: they transform the source code written by engineers into executable binaries that can be read by machines. Build systems aren't just for human-authored code; they also allow machines to create builds automatically, whether for testing or for releases to production. In an organization with thousands of engineers, it's common that most builds are triggered automatically rather than directly by engineers. ### Can't I just use a compiler? -The need for a build system might not be immediately obvious. Most engineers -don't use a build system while learning to code: most start by invoking tools -like `gcc` or `javac` directly from the command line, or the equivalent in an -integrated development environment (IDE). As long as all the source code is in -the same directory, a command like this works fine: +The need for a build system might not be immediately obvious. Most engineers don't use a build system while learning to code: most start by invoking tools like `gcc` or `javac` directly from the command line, or the equivalent in an integrated development environment (IDE). As long as all the source code is in the same directory, a command like this works fine: ```posix-terminal javac *.java ``` -This instructs the Java compiler to take every Java source file in the current -directory and turn it into a binary class file. In the simplest case, this is -all you need. - -However, as soon as code expands, the complications begin. `javac` is smart -enough to look in subdirectories of the current directory to find code to -import. But it has no way of finding code stored in _other parts_ of the -filesystem (perhaps a library shared by several projects). It also only knows -how to build Java code. Large systems often involve different pieces written in -a variety of programming languages with webs of dependencies among those pieces, -meaning no compiler for a single language can possibly build the entire system. - -Once you're dealing with code from multiple languages or multiple compilation -units, building code is no longer a one-step process. Now you must evaluate what -your code depends on and build those pieces in the proper order, possibly using -a different set of tools for each piece. If any dependencies change, you must -repeat this process to avoid depending on stale binaries. For a codebase of even -moderate size, this process quickly becomes tedious and error-prone. - -The compiler also doesn’t know anything about how to handle external -dependencies, such as third-party `JAR` files in Java. Without a build system, -you could manage this by downloading the dependency from the internet, sticking -it in a `lib` folder on the hard drive, and configuring the compiler to read -libraries from that directory. Over time, it's difficult to maintain the -updates, versions, and source of these external dependencies. +This instructs the Java compiler to take every Java source file in the current directory and turn it into a binary class file. In the simplest case, this is all you need. + +However, as soon as code expands, the complications begin. `javac` is smart enough to look in subdirectories of the current directory to find code to import. But it has no way of finding code stored in *other parts* of the filesystem (perhaps a library shared by several projects). It also only knows how to build Java code. Large systems often involve different pieces written in a variety of programming languages with webs of dependencies among those pieces, meaning no compiler for a single language can possibly build the entire system. + +Once you're dealing with code from multiple languages or multiple compilation units, building code is no longer a one-step process. Now you must evaluate what your code depends on and build those pieces in the proper order, possibly using a different set of tools for each piece. If any dependencies change, you must repeat this process to avoid depending on stale binaries. For a codebase of even moderate size, this process quickly becomes tedious and error-prone. + +The compiler also doesn’t know anything about how to handle external dependencies, such as third-party `JAR` files in Java. Without a build system, you could manage this by downloading the dependency from the internet, sticking it in a `lib` folder on the hard drive, and configuring the compiler to read libraries from that directory. Over time, it's difficult to maintain the updates, versions, and source of these external dependencies. ### What about shell scripts? -Suppose that your hobby project starts out simple enough that you can build it -using just a compiler, but you begin running into some of the problems described -previously. Maybe you still don’t think you need a build system and can automate -away the tedious parts using some simple shell scripts that take care of -building things in the correct order. This helps out for a while, but pretty -soon you start running into even more problems: - -* It becomes tedious. As your system grows more complex, you begin spending - almost as much time working on your build scripts as on real code. Debugging - shell scripts is painful, with more and more hacks being layered on top of - one another. - -* It’s slow. To make sure you weren’t accidentally relying on stale libraries, - you have your build script build every dependency in order every time you - run it. You think about adding some logic to detect which parts need to be - rebuilt, but that sounds awfully complex and error prone for a script. Or - you think about specifying which parts need to be rebuilt each time, but - then you’re back to square one. - -* Good news: it’s time for a release! Better go figure out all the arguments - you need to pass to the jar command to make your final build. And remember - how to upload it and push it out to the central repository. And build and - push the documentation updates, and send out a notification to users. Hmm, - maybe this calls for another script... - -* Disaster! Your hard drive crashes, and now you need to recreate your entire - system. You were smart enough to keep all of your source files in version - control, but what about those libraries you downloaded? Can you find them - all again and make sure they were the same version as when you first - downloaded them? Your scripts probably depended on particular tools being - installed in particular places—can you restore that same environment so that - the scripts work again? What about all those environment variables you set a - long time ago to get the compiler working just right and then forgot about? - -* Despite the problems, your project is successful enough that you’re able to - begin hiring more engineers. Now you realize that it doesn’t take a disaster - for the previous problems to arise—you need to go through the same painful - bootstrapping process every time a new developer joins your team. And - despite your best efforts, there are still small differences in each - person’s system. Frequently, what works on one person’s machine doesn’t work - on another’s, and each time it takes a few hours of debugging tool paths or - library versions to figure out where the difference is. - -* You decide that you need to automate your build system. In theory, this is - as simple as getting a new computer and setting it up to run your build - script every night using cron. You still need to go through the painful - setup process, but now you don’t have the benefit of a human brain being - able to detect and resolve minor problems. Now, every morning when you get - in, you see that last night’s build failed because yesterday a developer - made a change that worked on their system but didn’t work on the automated - build system. Each time it’s a simple fix, but it happens so often that you - end up spending a lot of time each day discovering and applying these simple - fixes. - -* Builds become slower and slower as the project grows. One day, while waiting - for a build to complete, you gaze mournfully at the idle desktop of your - coworker, who is on vacation, and wish there were a way to take advantage of - all that wasted computational power. - -You’ve run into a classic problem of scale. For a single developer working on at -most a couple hundred lines of code for at most a week or two (which might have -been the entire experience thus far of a junior developer who just graduated -university), a compiler is all you need. Scripts can maybe take you a little bit -farther. But as soon as you need to coordinate across multiple developers and -their machines, even a perfect build script isn’t enough because it becomes very -difficult to account for the minor differences in those machines. At this point, -this simple approach breaks down and it’s time to invest in a real build system. +Suppose that your hobby project starts out simple enough that you can build it using just a compiler, but you begin running into some of the problems described previously. Maybe you still don’t think you need a build system and can automate away the tedious parts using some simple shell scripts that take care of building things in the correct order. This helps out for a while, but pretty soon you start running into even more problems: + +- It becomes tedious. As your system grows more complex, you begin spending almost as much time working on your build scripts as on real code. Debugging shell scripts is painful, with more and more hacks being layered on top of one another. + +- It’s slow. To make sure you weren’t accidentally relying on stale libraries, you have your build script build every dependency in order every time you run it. You think about adding some logic to detect which parts need to be rebuilt, but that sounds awfully complex and error prone for a script. Or you think about specifying which parts need to be rebuilt each time, but then you’re back to square one. + +- Good news: it’s time for a release! Better go figure out all the arguments you need to pass to the jar command to make your final build. And remember how to upload it and push it out to the central repository. And build and push the documentation updates, and send out a notification to users. Hmm, maybe this calls for another script... + +- Disaster! Your hard drive crashes, and now you need to recreate your entire system. You were smart enough to keep all of your source files in version control, but what about those libraries you downloaded? Can you find them all again and make sure they were the same version as when you first downloaded them? Your scripts probably depended on particular tools being installed in particular places—can you restore that same environment so that the scripts work again? What about all those environment variables you set a long time ago to get the compiler working just right and then forgot about? + +- Despite the problems, your project is successful enough that you’re able to begin hiring more engineers. Now you realize that it doesn’t take a disaster for the previous problems to arise—you need to go through the same painful bootstrapping process every time a new developer joins your team. And despite your best efforts, there are still small differences in each person’s system. Frequently, what works on one person’s machine doesn’t work on another’s, and each time it takes a few hours of debugging tool paths or library versions to figure out where the difference is. + +- You decide that you need to automate your build system. In theory, this is as simple as getting a new computer and setting it up to run your build script every night using cron. You still need to go through the painful setup process, but now you don’t have the benefit of a human brain being able to detect and resolve minor problems. Now, every morning when you get in, you see that last night’s build failed because yesterday a developer made a change that worked on their system but didn’t work on the automated build system. Each time it’s a simple fix, but it happens so often that you end up spending a lot of time each day discovering and applying these simple fixes. + +- Builds become slower and slower as the project grows. One day, while waiting for a build to complete, you gaze mournfully at the idle desktop of your coworker, who is on vacation, and wish there were a way to take advantage of all that wasted computational power. + +You’ve run into a classic problem of scale. For a single developer working on at most a couple hundred lines of code for at most a week or two (which might have been the entire experience thus far of a junior developer who just graduated university), a compiler is all you need. Scripts can maybe take you a little bit farther. But as soon as you need to coordinate across multiple developers and their machines, even a perfect build script isn’t enough because it becomes very difficult to account for the minor differences in those machines. At this point, this simple approach breaks down and it’s time to invest in a real build system. diff --git a/basics/dependencies.mdx b/basics/dependencies.mdx index 1d3bf8f1..308b627e 100644 --- a/basics/dependencies.mdx +++ b/basics/dependencies.mdx @@ -2,300 +2,84 @@ title: 'Dependency Management' --- - - -In looking through the previous pages, one theme repeats over and over: managing -your own code is fairly straightforward, but managing its dependencies is much -more difficult. There are all sorts of dependencies: sometimes there’s a -dependency on a task (such as “push the documentation before I mark a release as -complete”), and sometimes there’s a dependency on an artifact (such as “I need -to have the latest version of the computer vision library to build my code”). -Sometimes, you have internal dependencies on another part of your codebase, and -sometimes you have external dependencies on code or data owned by another team -(either in your organization or a third party). But in any case, the idea of “I -need that before I can have this” is something that recurs repeatedly in the -design of build systems, and managing dependencies is perhaps the most -fundamental job of a build system. +In looking through the previous pages, one theme repeats over and over: managing your own code is fairly straightforward, but managing its dependencies is much more difficult. There are all sorts of dependencies: sometimes there’s a dependency on a task (such as “push the documentation before I mark a release as complete”), and sometimes there’s a dependency on an artifact (such as “I need to have the latest version of the computer vision library to build my code”). Sometimes, you have internal dependencies on another part of your codebase, and sometimes you have external dependencies on code or data owned by another team (either in your organization or a third party). But in any case, the idea of “I need that before I can have this” is something that recurs repeatedly in the design of build systems, and managing dependencies is perhaps the most fundamental job of a build system. ## Dealing with Modules and Dependencies -Projects that use artifact-based build systems like Bazel are broken into a set -of modules, with modules expressing dependencies on one another via `BUILD` -files. Proper organization of these modules and dependencies can have a huge -effect on both the performance of the build system and how much work it takes to -maintain. +Projects that use artifact-based build systems like Bazel are broken into a set of modules, with modules expressing dependencies on one another via `BUILD` files. Proper organization of these modules and dependencies can have a huge effect on both the performance of the build system and how much work it takes to maintain. ## Using Fine-Grained Modules and the 1:1:1 Rule -The first question that comes up when structuring an artifact-based build is -deciding how much functionality an individual module should encompass. In Bazel, -a _module_ is represented by a target specifying a buildable unit like a -`java_library` or a `go_binary`. At one extreme, the entire project could be -contained in a single module by putting one `BUILD` file at the root and -recursively globbing together all of that project’s source files. At the other -extreme, nearly every source file could be made into its own module, effectively -requiring each file to list in a `BUILD` file every other file it depends on. - -Most projects fall somewhere between these extremes, and the choice involves a -trade-off between performance and maintainability. Using a single module for the -entire project might mean that you never need to touch the `BUILD` file except -when adding an external dependency, but it means that the build system must -always build the entire project all at once. This means that it won’t be able to -parallelize or distribute parts of the build, nor will it be able to cache parts -that it’s already built. One-module-per-file is the opposite: the build system -has the maximum flexibility in caching and scheduling steps of the build, but -engineers need to expend more effort maintaining lists of dependencies whenever -they change which files reference which. - -Though the exact granularity varies by language (and often even within -language), Google tends to favor significantly smaller modules than one might -typically write in a task-based build system. A typical production binary at -Google often depends on tens of thousands of targets, and even a moderate-sized -team can own several hundred targets within its codebase. For languages like -Java that have a strong built-in notion of packaging, each directory usually -contains a single package, target, and `BUILD` file (Pants, another build system -based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging -conventions frequently define multiple targets per `BUILD` file. - -The benefits of smaller build targets really begin to show at scale because they -lead to faster distributed builds and a less frequent need to rebuild targets. -The advantages become even more compelling after testing enters the picture, as -finer-grained targets mean that the build system can be much smarter about -running only a limited subset of tests that could be affected by any given -change. Because Google believes in the systemic benefits of using smaller -targets, we’ve made some strides in mitigating the downside by investing in -tooling to automatically manage `BUILD` files to avoid burdening developers. - -Some of these tools, such as `buildifier` and `buildozer`, are available with -Bazel in the [`buildtools` -directory](https://github.com/bazelbuild/buildtools). +The first question that comes up when structuring an artifact-based build is deciding how much functionality an individual module should encompass. In Bazel, a *module* is represented by a target specifying a buildable unit like a `java_library` or a `go_binary`. At one extreme, the entire project could be contained in a single module by putting one `BUILD` file at the root and recursively globbing together all of that project’s source files. At the other extreme, nearly every source file could be made into its own module, effectively requiring each file to list in a `BUILD` file every other file it depends on. + +Most projects fall somewhere between these extremes, and the choice involves a trade-off between performance and maintainability. Using a single module for the entire project might mean that you never need to touch the `BUILD` file except when adding an external dependency, but it means that the build system must always build the entire project all at once. This means that it won’t be able to parallelize or distribute parts of the build, nor will it be able to cache parts that it’s already built. One-module-per-file is the opposite: the build system has the maximum flexibility in caching and scheduling steps of the build, but engineers need to expend more effort maintaining lists of dependencies whenever they change which files reference which. + +Though the exact granularity varies by language (and often even within language), Google tends to favor significantly smaller modules than one might typically write in a task-based build system. A typical production binary at Google often depends on tens of thousands of targets, and even a moderate-sized team can own several hundred targets within its codebase. For languages like Java that have a strong built-in notion of packaging, each directory usually contains a single package, target, and `BUILD` file (Pants, another build system based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging conventions frequently define multiple targets per `BUILD` file. + +The benefits of smaller build targets really begin to show at scale because they lead to faster distributed builds and a less frequent need to rebuild targets. The advantages become even more compelling after testing enters the picture, as finer-grained targets mean that the build system can be much smarter about running only a limited subset of tests that could be affected by any given change. Because Google believes in the systemic benefits of using smaller targets, we’ve made some strides in mitigating the downside by investing in tooling to automatically manage `BUILD` files to avoid burdening developers. + +Some of these tools, such as `buildifier` and `buildozer`, are available with Bazel in the [`buildtools` directory](https://github.com/bazelbuild/buildtools). ## Minimizing Module Visibility -Bazel and other build systems allow each target to specify a visibility — a -property that determines which other targets may depend on it. A private target -can only be referenced within its own `BUILD` file. A target may grant broader -visibility to the targets of an explicitly defined list of `BUILD` files, or, in -the case of public visibility, to every target in the workspace. - -As with most programming languages, it is usually best to minimize visibility as -much as possible. Generally, teams at Google will make targets public only if -those targets represent widely used libraries available to any team at Google. -Teams that require others to coordinate with them before using their code will -maintain an allowlist of customer targets as their target’s visibility. Each -team’s internal implementation targets will be restricted to only directories -owned by the team, and most `BUILD` files will have only one target that isn’t -private. +Bazel and other build systems allow each target to specify a visibility — a property that determines which other targets may depend on it. A private target can only be referenced within its own `BUILD` file. A target may grant broader visibility to the targets of an explicitly defined list of `BUILD` files, or, in the case of public visibility, to every target in the workspace. + +As with most programming languages, it is usually best to minimize visibility as much as possible. Generally, teams at Google will make targets public only if those targets represent widely used libraries available to any team at Google. Teams that require others to coordinate with them before using their code will maintain an allowlist of customer targets as their target’s visibility. Each team’s internal implementation targets will be restricted to only directories owned by the team, and most `BUILD` files will have only one target that isn’t private. ## Managing Dependencies -Modules need to be able to refer to one another. The downside of breaking a -codebase into fine-grained modules is that you need to manage the dependencies -among those modules (though tools can help automate this). Expressing these -dependencies usually ends up being the bulk of the content in a `BUILD` file. +Modules need to be able to refer to one another. The downside of breaking a codebase into fine-grained modules is that you need to manage the dependencies among those modules (though tools can help automate this). Expressing these dependencies usually ends up being the bulk of the content in a `BUILD` file. ### Internal dependencies -In a large project broken into fine-grained modules, most dependencies are -likely to be internal; that is, on another target defined and built in the same -source repository. Internal dependencies differ from external dependencies in -that they are built from source rather than downloaded as a prebuilt artifact -while running the build. This also means that there’s no notion of “version” for -internal dependencies—a target and all of its internal dependencies are always -built at the same commit/revision in the repository. One issue that should be -handled carefully with regard to internal dependencies is how to treat -transitive dependencies (Figure 1). Suppose target A depends on target B, which -depends on a common library target C. Should target A be able to use classes -defined in target C? - -[![Transitive -dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) +In a large project broken into fine-grained modules, most dependencies are likely to be internal; that is, on another target defined and built in the same source repository. Internal dependencies differ from external dependencies in that they are built from source rather than downloaded as a prebuilt artifact while running the build. This also means that there’s no notion of “version” for internal dependencies—a target and all of its internal dependencies are always built at the same commit/revision in the repository. One issue that should be handled carefully with regard to internal dependencies is how to treat transitive dependencies (Figure 1). Suppose target A depends on target B, which depends on a common library target C. Should target A be able to use classes defined in target C? + +[![Transitive dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) **Figure 1**. Transitive dependencies -As far as the underlying tools are concerned, there’s no problem with this; both -B and C will be linked into target A when it is built, so any symbols defined in -C are known to A. Bazel allowed this for many years, but as Google grew, we -began to see problems. Suppose that B was refactored such that it no longer -needed to depend on C. If B’s dependency on C was then removed, A and any other -target that used C via a dependency on B would break. Effectively, a target’s -dependencies became part of its public contract and could never be safely -changed. This meant that dependencies accumulated over time and builds at Google -started to slow down. - -Google eventually solved this issue by introducing a “strict transitive -dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to -reference a symbol without depending on it directly and, if so, fails with an -error and a shell command that can be used to automatically insert the -dependency. Rolling this change out across Google’s entire codebase and -refactoring every one of our millions of build targets to explicitly list their -dependencies was a multiyear effort, but it was well worth it. Our builds are -now much faster given that targets have fewer unnecessary dependencies, and -engineers are empowered to remove dependencies they don’t need without worrying -about breaking targets that depend on them. - -As usual, enforcing strict transitive dependencies involved a trade-off. It made -build files more verbose, as frequently used libraries now need to be listed -explicitly in many places rather than pulled in incidentally, and engineers -needed to spend more effort adding dependencies to `BUILD` files. We’ve since -developed tools that reduce this toil by automatically detecting many missing -dependencies and adding them to a `BUILD` files without any developer -intervention. But even without such tools, we’ve found the trade-off to be well -worth it as the codebase scales: explicitly adding a dependency to `BUILD` file -is a one-time cost, but dealing with implicit transitive dependencies can cause -ongoing problems as long as the build target exists. Bazel [enforces strict -transitive -dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) -on Java code by default. +As far as the underlying tools are concerned, there’s no problem with this; both B and C will be linked into target A when it is built, so any symbols defined in C are known to A. Bazel allowed this for many years, but as Google grew, we began to see problems. Suppose that B was refactored such that it no longer needed to depend on C. If B’s dependency on C was then removed, A and any other target that used C via a dependency on B would break. Effectively, a target’s dependencies became part of its public contract and could never be safely changed. This meant that dependencies accumulated over time and builds at Google started to slow down. + +Google eventually solved this issue by introducing a “strict transitive dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to reference a symbol without depending on it directly and, if so, fails with an error and a shell command that can be used to automatically insert the dependency. Rolling this change out across Google’s entire codebase and refactoring every one of our millions of build targets to explicitly list their dependencies was a multiyear effort, but it was well worth it. Our builds are now much faster given that targets have fewer unnecessary dependencies, and engineers are empowered to remove dependencies they don’t need without worrying about breaking targets that depend on them. + +As usual, enforcing strict transitive dependencies involved a trade-off. It made build files more verbose, as frequently used libraries now need to be listed explicitly in many places rather than pulled in incidentally, and engineers needed to spend more effort adding dependencies to `BUILD` files. We’ve since developed tools that reduce this toil by automatically detecting many missing dependencies and adding them to a `BUILD` files without any developer intervention. But even without such tools, we’ve found the trade-off to be well worth it as the codebase scales: explicitly adding a dependency to `BUILD` file is a one-time cost, but dealing with implicit transitive dependencies can cause ongoing problems as long as the build target exists. Bazel [enforces strict transitive dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) on Java code by default. ### External dependencies -If a dependency isn’t internal, it must be external. External dependencies are -those on artifacts that are built and stored outside of the build system. The -dependency is imported directly from an artifact repository (typically accessed -over the internet) and used as-is rather than being built from source. One of -the biggest differences between external and internal dependencies is that -external dependencies have versions, and those versions exist independently of -the project’s source code. +If a dependency isn’t internal, it must be external. External dependencies are those on artifacts that are built and stored outside of the build system. The dependency is imported directly from an artifact repository (typically accessed over the internet) and used as-is rather than being built from source. One of the biggest differences between external and internal dependencies is that external dependencies have versions, and those versions exist independently of the project’s source code. ### Automatic versus manual dependency management -Build systems can allow the versions of external dependencies to be managed -either manually or automatically. When managed manually, the buildfile -explicitly lists the version it wants to download from the artifact repository, -often using a [semantic version string](https://semver.org/) such -as `1.1.4`. When managed automatically, the source file specifies a range of -acceptable versions, and the build system always downloads the latest one. For -example, Gradle allows a dependency version to be declared as “1.+” to specify -that any minor or patch version of a dependency is acceptable so long as the -major version is 1. - -Automatically managed dependencies can be convenient for small projects, but -they’re usually a recipe for disaster on projects of nontrivial size or that are -being worked on by more than one engineer. The problem with automatically -managed dependencies is that you have no control over when the version is -updated. There’s no way to guarantee that external parties won’t make breaking -updates (even when they claim to use semantic versioning), so a build that -worked one day might be broken the next with no easy way to detect what changed -or to roll it back to a working state. Even if the build doesn’t break, there -can be subtle behavior or performance changes that are impossible to track down. - -In contrast, because manually managed dependencies require a change in source -control, they can be easily discovered and rolled back, and it’s possible to -check out an older version of the repository to build with older dependencies. -Bazel requires that versions of all dependencies be specified manually. At even -moderate scales, the overhead of manual version management is well worth it for -the stability it provides. +Build systems can allow the versions of external dependencies to be managed either manually or automatically. When managed manually, the buildfile explicitly lists the version it wants to download from the artifact repository, often using a [semantic version string](https://semver.org/) such as `1.1.4`. When managed automatically, the source file specifies a range of acceptable versions, and the build system always downloads the latest one. For example, Gradle allows a dependency version to be declared as “1.+” to specify that any minor or patch version of a dependency is acceptable so long as the major version is 1. + +Automatically managed dependencies can be convenient for small projects, but they’re usually a recipe for disaster on projects of nontrivial size or that are being worked on by more than one engineer. The problem with automatically managed dependencies is that you have no control over when the version is updated. There’s no way to guarantee that external parties won’t make breaking updates (even when they claim to use semantic versioning), so a build that worked one day might be broken the next with no easy way to detect what changed or to roll it back to a working state. Even if the build doesn’t break, there can be subtle behavior or performance changes that are impossible to track down. + +In contrast, because manually managed dependencies require a change in source control, they can be easily discovered and rolled back, and it’s possible to check out an older version of the repository to build with older dependencies. Bazel requires that versions of all dependencies be specified manually. At even moderate scales, the overhead of manual version management is well worth it for the stability it provides. ### The One-Version Rule -Different versions of a library are usually represented by different artifacts, -so in theory there’s no reason that different versions of the same external -dependency couldn’t both be declared in the build system under different names. -That way, each target could choose which version of the dependency it wanted to -use. This causes a lot of problems in practice, so Google enforces a strict -[One-Version -Rule](https://opensource.google/docs/thirdparty/oneversion/) for -all third-party dependencies in our codebase. - -The biggest problem with allowing multiple versions is the diamond dependency -issue. Suppose that target A depends on target B and on v1 of an external -library. If target B is later refactored to add a dependency on v2 of the same -external library, target A will break because it now depends implicitly on two -different versions of the same library. Effectively, it’s never safe to add a -new dependency from a target to any third-party library with multiple versions, -because any of that target’s users could already be depending on a different -version. Following the One-Version Rule makes this conflict impossible—if a -target adds a dependency on a third-party library, any existing dependencies -will already be on that same version, so they can happily coexist. +Different versions of a library are usually represented by different artifacts, so in theory there’s no reason that different versions of the same external dependency couldn’t both be declared in the build system under different names. That way, each target could choose which version of the dependency it wanted to use. This causes a lot of problems in practice, so Google enforces a strict [One-Version Rule](https://opensource.google/docs/thirdparty/oneversion/) for all third-party dependencies in our codebase. + +The biggest problem with allowing multiple versions is the diamond dependency issue. Suppose that target A depends on target B and on v1 of an external library. If target B is later refactored to add a dependency on v2 of the same external library, target A will break because it now depends implicitly on two different versions of the same library. Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, because any of that target’s users could already be depending on a different version. Following the One-Version Rule makes this conflict impossible—if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist. ### Transitive external dependencies -Dealing with the transitive dependencies of an external dependency can be -particularly difficult. Many artifact repositories such as Maven Central, allow -artifacts to specify dependencies on particular versions of other artifacts in -the repository. Build tools like Maven or Gradle often recursively download each -transitive dependency by default, meaning that adding a single dependency in -your project could potentially cause dozens of artifacts to be downloaded in -total. - -This is very convenient: when adding a dependency on a new library, it would be -a big pain to have to track down each of that library’s transitive dependencies -and add them all manually. But there’s also a huge downside: because different -libraries can depend on different versions of the same third-party library, this -strategy necessarily violates the One-Version Rule and leads to the diamond -dependency problem. If your target depends on two external libraries that use -different versions of the same dependency, there’s no telling which one you’ll -get. This also means that updating an external dependency could cause seemingly -unrelated failures throughout the codebase if the new version begins pulling in -conflicting versions of some of its dependencies. - -Bazel did not use to automatically download transitive dependencies. It used to -employ a `WORKSPACE` file that required all transitive dependencies to be -listed, which led to a lot of pain when managing external dependencies. Bazel -has since added support for automatic transitive external dependency management -in the form of the `MODULE.bazel` file. See [external dependency -overview](/external/overview) for more details. - -Yet again, the choice here is one between convenience and scalability. Small -projects might prefer not having to worry about managing transitive dependencies -themselves and might be able to get away with using automatic transitive -dependencies. This strategy becomes less and less appealing as the organization -and codebase grows, and conflicts and unexpected results become more and more -frequent. At larger scales, the cost of manually managing dependencies is much -less than the cost of dealing with issues caused by automatic dependency -management. +Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central, allow artifacts to specify dependencies on particular versions of other artifacts in the repository. Build tools like Maven or Gradle often recursively download each transitive dependency by default, meaning that adding a single dependency in your project could potentially cause dozens of artifacts to be downloaded in total. + +This is very convenient: when adding a dependency on a new library, it would be a big pain to have to track down each of that library’s transitive dependencies and add them all manually. But there’s also a huge downside: because different libraries can depend on different versions of the same third-party library, this strategy necessarily violates the One-Version Rule and leads to the diamond dependency problem. If your target depends on two external libraries that use different versions of the same dependency, there’s no telling which one you’ll get. This also means that updating an external dependency could cause seemingly unrelated failures throughout the codebase if the new version begins pulling in conflicting versions of some of its dependencies. + +Bazel did not use to automatically download transitive dependencies. It used to employ a `WORKSPACE` file that required all transitive dependencies to be listed, which led to a lot of pain when managing external dependencies. Bazel has since added support for automatic transitive external dependency management in the form of the `MODULE.bazel` file. See [external dependency overview](/external/overview) for more details. + +Yet again, the choice here is one between convenience and scalability. Small projects might prefer not having to worry about managing transitive dependencies themselves and might be able to get away with using automatic transitive dependencies. This strategy becomes less and less appealing as the organization and codebase grows, and conflicts and unexpected results become more and more frequent. At larger scales, the cost of manually managing dependencies is much less than the cost of dealing with issues caused by automatic dependency management. ### Caching build results using external dependencies -External dependencies are most often provided by third parties that release -stable versions of libraries, perhaps without providing source code. Some -organizations might also choose to make some of their own code available as -artifacts, allowing other pieces of code to depend on them as third-party rather -than internal dependencies. This can theoretically speed up builds if artifacts -are slow to build but quick to download. - -However, this also introduces a lot of overhead and complexity: someone needs to -be responsible for building each of those artifacts and uploading them to the -artifact repository, and clients need to ensure that they stay up to date with -the latest version. Debugging also becomes much more difficult because different -parts of the system will have been built from different points in the -repository, and there is no longer a consistent view of the source tree. - -A better way to solve the problem of artifacts taking a long time to build is to -use a build system that supports remote caching, as described earlier. Such a -build system saves the resulting artifacts from every build to a location that -is shared across engineers, so if a developer depends on an artifact that was -recently built by someone else, the build system automatically downloads it -instead of building it. This provides all of the performance benefits of -depending directly on artifacts while still ensuring that builds are as -consistent as if they were always built from the same source. This is the -strategy used internally by Google, and Bazel can be configured to use a remote -cache. +External dependencies are most often provided by third parties that release stable versions of libraries, perhaps without providing source code. Some organizations might also choose to make some of their own code available as artifacts, allowing other pieces of code to depend on them as third-party rather than internal dependencies. This can theoretically speed up builds if artifacts are slow to build but quick to download. + +However, this also introduces a lot of overhead and complexity: someone needs to be responsible for building each of those artifacts and uploading them to the artifact repository, and clients need to ensure that they stay up to date with the latest version. Debugging also becomes much more difficult because different parts of the system will have been built from different points in the repository, and there is no longer a consistent view of the source tree. + +A better way to solve the problem of artifacts taking a long time to build is to use a build system that supports remote caching, as described earlier. Such a build system saves the resulting artifacts from every build to a location that is shared across engineers, so if a developer depends on an artifact that was recently built by someone else, the build system automatically downloads it instead of building it. This provides all of the performance benefits of depending directly on artifacts while still ensuring that builds are as consistent as if they were always built from the same source. This is the strategy used internally by Google, and Bazel can be configured to use a remote cache. ### Security and reliability of external dependencies -Depending on artifacts from third-party sources is inherently risky. There’s an -availability risk if the third-party source (such as an artifact repository) -goes down, because your entire build might grind to a halt if it’s unable to -download an external dependency. There’s also a security risk: if the -third-party system is compromised by an attacker, the attacker could replace the -referenced artifact with one of their own design, allowing them to inject -arbitrary code into your build. Both problems can be mitigated by mirroring any -artifacts you depend on onto servers you control and blocking your build system -from accessing third-party artifact repositories like Maven Central. The -trade-off is that these mirrors take effort and resources to maintain, so the -choice of whether to use them often depends on the scale of the project. The -security issue can also be completely prevented with little overhead by -requiring the hash of each third-party artifact to be specified in the source -repository, causing the build to fail if the artifact is tampered with. Another -alternative that completely sidesteps the issue is to vendor your project’s -dependencies. When a project vendors its dependencies, it checks them into -source control alongside the project’s source code, either as source or as -binaries. This effectively means that all of the project’s external dependencies -are converted to internal dependencies. Google uses this approach internally, -checking every third-party library referenced throughout Google into a -`third_party` directory at the root of Google’s source tree. However, this works -at Google only because Google’s source control system is custom built to handle -an extremely large monorepo, so vendoring might not be an option for all -organizations. +Depending on artifacts from third-party sources is inherently risky. There’s an availability risk if the third-party source (such as an artifact repository) goes down, because your entire build might grind to a halt if it’s unable to download an external dependency. There’s also a security risk: if the third-party system is compromised by an attacker, the attacker could replace the referenced artifact with one of their own design, allowing them to inject arbitrary code into your build. Both problems can be mitigated by mirroring any artifacts you depend on onto servers you control and blocking your build system from accessing third-party artifact repositories like Maven Central. The trade-off is that these mirrors take effort and resources to maintain, so the choice of whether to use them often depends on the scale of the project. The security issue can also be completely prevented with little overhead by requiring the hash of each third-party artifact to be specified in the source repository, causing the build to fail if the artifact is tampered with. Another alternative that completely sidesteps the issue is to vendor your project’s dependencies. When a project vendors its dependencies, it checks them into source control alongside the project’s source code, either as source or as binaries. This effectively means that all of the project’s external dependencies are converted to internal dependencies. Google uses this approach internally, checking every third-party library referenced throughout Google into a `third_party` directory at the root of Google’s source tree. However, this works at Google only because Google’s source control system is custom built to handle an extremely large monorepo, so vendoring might not be an option for all organizations. diff --git a/basics/distributed-builds.mdx b/basics/distributed-builds.mdx index c32f44ff..b11e7c9b 100644 --- a/basics/distributed-builds.mdx +++ b/basics/distributed-builds.mdx @@ -2,136 +2,46 @@ title: 'Distributed Builds' --- - - -When you have a large codebase, chains of dependencies can become very deep. -Even simple binaries can often depend on tens of thousands of build targets. At -this scale, it’s simply impossible to complete a build in a reasonable amount -of time on a single machine: no build system can get around the fundamental -laws of physics imposed on a machine’s hardware. The only way to make this work -is with a build system that supports distributed builds wherein the units of -work being done by the system are spread across an arbitrary and scalable -number of machines. Assuming we’ve broken the system’s work into small enough -units (more on this later), this would allow us to complete any build of any -size as quickly as we’re willing to pay for. This scalability is the holy grail -we’ve been working toward by defining an artifact-based build system. +When you have a large codebase, chains of dependencies can become very deep. Even simple binaries can often depend on tens of thousands of build targets. At this scale, it’s simply impossible to complete a build in a reasonable amount of time on a single machine: no build system can get around the fundamental laws of physics imposed on a machine’s hardware. The only way to make this work is with a build system that supports distributed builds wherein the units of work being done by the system are spread across an arbitrary and scalable number of machines. Assuming we’ve broken the system’s work into small enough units (more on this later), this would allow us to complete any build of any size as quickly as we’re willing to pay for. This scalability is the holy grail we’ve been working toward by defining an artifact-based build system. ## Remote caching -The simplest type of distributed build is one that only leverages _remote -caching_, which is shown in Figure 1. +The simplest type of distributed build is one that only leverages *remote caching*, which is shown in Figure 1. [![Distributed build with remote caching](/images/distributed-build-remote-cache.png)](/images/distributed-build-remote-cache.png) **Figure 1**. A distributed build showing remote caching -Every system that performs builds, including both developer workstations and -continuous integration systems, shares a reference to a common remote cache -service. This service might be a fast and local short-term storage system like -Redis or a cloud service like Google Cloud Storage. Whenever a user needs to -build an artifact, whether directly or as a dependency, the system first checks -with the remote cache to see if that artifact already exists there. If so, it -can download the artifact instead of building it. If not, the system builds the -artifact itself and uploads the result back to the cache. This means that -low-level dependencies that don’t change very often can be built once and shared -across users rather than having to be rebuilt by each user. At Google, many -artifacts are served from a cache rather than built from scratch, vastly -reducing the cost of running our build system. - -For a remote caching system to work, the build system must guarantee that builds -are completely reproducible. That is, for any build target, it must be possible -to determine the set of inputs to that target such that the same set of inputs -will produce exactly the same output on any machine. This is the only way to -ensure that the results of downloading an artifact are the same as the results -of building it oneself. Note that this requires that each artifact in the cache -be keyed on both its target and a hash of its inputs—that way, different -engineers could make different modifications to the same target at the same -time, and the remote cache would store all of the resulting artifacts and serve -them appropriately without conflict. - -Of course, for there to be any benefit from a remote cache, downloading an -artifact needs to be faster than building it. This is not always the case, -especially if the cache server is far from the machine doing the build. Google’s -network and build system is carefully tuned to be able to quickly share build -results. +Every system that performs builds, including both developer workstations and continuous integration systems, shares a reference to a common remote cache service. This service might be a fast and local short-term storage system like Redis or a cloud service like Google Cloud Storage. Whenever a user needs to build an artifact, whether directly or as a dependency, the system first checks with the remote cache to see if that artifact already exists there. If so, it can download the artifact instead of building it. If not, the system builds the artifact itself and uploads the result back to the cache. This means that low-level dependencies that don’t change very often can be built once and shared across users rather than having to be rebuilt by each user. At Google, many artifacts are served from a cache rather than built from scratch, vastly reducing the cost of running our build system. + +For a remote caching system to work, the build system must guarantee that builds are completely reproducible. That is, for any build target, it must be possible to determine the set of inputs to that target such that the same set of inputs will produce exactly the same output on any machine. This is the only way to ensure that the results of downloading an artifact are the same as the results of building it oneself. Note that this requires that each artifact in the cache be keyed on both its target and a hash of its inputs—that way, different engineers could make different modifications to the same target at the same time, and the remote cache would store all of the resulting artifacts and serve them appropriately without conflict. + +Of course, for there to be any benefit from a remote cache, downloading an artifact needs to be faster than building it. This is not always the case, especially if the cache server is far from the machine doing the build. Google’s network and build system is carefully tuned to be able to quickly share build results. ## Remote execution -Remote caching isn’t a true distributed build. If the cache is lost or if you -make a low-level change that requires everything to be rebuilt, you still need -to perform the entire build locally on your machine. The true goal is to support -remote execution, in which the actual work of doing the build can be spread -across any number of workers. Figure 2 depicts a remote execution system. +Remote caching isn’t a true distributed build. If the cache is lost or if you make a low-level change that requires everything to be rebuilt, you still need to perform the entire build locally on your machine. The true goal is to support remote execution, in which the actual work of doing the build can be spread across any number of workers. Figure 2 depicts a remote execution system. [![Remote execution system](/images/remote-execution-system.png)](/images/remote-execution-system.png) **Figure 2**. A remote execution system -The build tool running on each user’s machine (where users are either human -engineers or automated build systems) sends requests to a central build master. -The build master breaks the requests into their component actions and schedules -the execution of those actions over a scalable pool of workers. Each worker -performs the actions asked of it with the inputs specified by the user and -writes out the resulting artifacts. These artifacts are shared across the other -machines executing actions that require them until the final output can be -produced and sent to the user. - -The trickiest part of implementing such a system is managing the communication -between the workers, the master, and the user’s local machine. Workers might -depend on intermediate artifacts produced by other workers, and the final output -needs to be sent back to the user’s local machine. To do this, we can build on -top of the distributed cache described previously by having each worker write -its results to and read its dependencies from the cache. The master blocks -workers from proceeding until everything they depend on has finished, in which -case they’ll be able to read their inputs from the cache. The final product is -also cached, allowing the local machine to download it. Note that we also need a -separate means of exporting the local changes in the user’s source tree so that -workers can apply those changes before building. - -For this to work, all of the parts of the artifact-based build systems described -earlier need to come together. Build environments must be completely -self-describing so that we can spin up workers without human intervention. Build -processes themselves must be completely self-contained because each step might -be executed on a different machine. Outputs must be completely deterministic so -that each worker can trust the results it receives from other workers. Such -guarantees are extremely difficult for a task-based system to provide, which -makes it nigh-impossible to build a reliable remote execution system on top of -one. +The build tool running on each user’s machine (where users are either human engineers or automated build systems) sends requests to a central build master. The build master breaks the requests into their component actions and schedules the execution of those actions over a scalable pool of workers. Each worker performs the actions asked of it with the inputs specified by the user and writes out the resulting artifacts. These artifacts are shared across the other machines executing actions that require them until the final output can be produced and sent to the user. + +The trickiest part of implementing such a system is managing the communication between the workers, the master, and the user’s local machine. Workers might depend on intermediate artifacts produced by other workers, and the final output needs to be sent back to the user’s local machine. To do this, we can build on top of the distributed cache described previously by having each worker write its results to and read its dependencies from the cache. The master blocks workers from proceeding until everything they depend on has finished, in which case they’ll be able to read their inputs from the cache. The final product is also cached, allowing the local machine to download it. Note that we also need a separate means of exporting the local changes in the user’s source tree so that workers can apply those changes before building. + +For this to work, all of the parts of the artifact-based build systems described earlier need to come together. Build environments must be completely self-describing so that we can spin up workers without human intervention. Build processes themselves must be completely self-contained because each step might be executed on a different machine. Outputs must be completely deterministic so that each worker can trust the results it receives from other workers. Such guarantees are extremely difficult for a task-based system to provide, which makes it nigh-impossible to build a reliable remote execution system on top of one. ## Distributed builds at Google -Since 2008, Google has been using a distributed build system that employs both -remote caching and remote execution, which is illustrated in Figure 3. +Since 2008, Google has been using a distributed build system that employs both remote caching and remote execution, which is illustrated in Figure 3. [![High-level build system](/images/high-level-build-system.png)](/images/high-level-build-system.png) **Figure 3**. Google’s distributed build system -Google’s remote cache is called ObjFS. It consists of a backend that stores -build outputs in Bigtables distributed throughout our fleet of production -machines and a frontend FUSE daemon named objfsd that runs on each developer’s -machine. The FUSE daemon allows engineers to browse build outputs as if they -were normal files stored on the workstation, but with the file content -downloaded on-demand only for the few files that are directly requested by the -user. Serving file contents on-demand greatly reduces both network and disk -usage, and the system is able to build twice as fast compared to when we stored -all build output on the developer’s local disk. - -Google’s remote execution system is called Forge. A Forge client in Blaze -(Bazel's internal equivalent) called -the Distributor sends requests for each action to a job running in our -datacenters called the Scheduler. The Scheduler maintains a cache of action -results, allowing it to return a response immediately if the action has already -been created by any other user of the system. If not, it places the action into -a queue. A large pool of Executor jobs continually read actions from this queue, -execute them, and store the results directly in the ObjFS Bigtables. These -results are available to the executors for future actions, or to be downloaded -by the end user via objfsd. - -The end result is a system that scales to efficiently support all builds -performed at Google. And the scale of Google’s builds is truly massive: Google -runs millions of builds executing millions of test cases and producing petabytes -of build outputs from billions of lines of source code every day. Not only does -such a system let our engineers build complex codebases quickly, it also allows -us to implement a huge number of automated tools and systems that rely on our -build. +Google’s remote cache is called ObjFS. It consists of a backend that stores build outputs in Bigtables distributed throughout our fleet of production machines and a frontend FUSE daemon named objfsd that runs on each developer’s machine. The FUSE daemon allows engineers to browse build outputs as if they were normal files stored on the workstation, but with the file content downloaded on-demand only for the few files that are directly requested by the user. Serving file contents on-demand greatly reduces both network and disk usage, and the system is able to build twice as fast compared to when we stored all build output on the developer’s local disk. + +Google’s remote execution system is called Forge. A Forge client in Blaze (Bazel's internal equivalent) called the Distributor sends requests for each action to a job running in our datacenters called the Scheduler. The Scheduler maintains a cache of action results, allowing it to return a response immediately if the action has already been created by any other user of the system. If not, it places the action into a queue. A large pool of Executor jobs continually read actions from this queue, execute them, and store the results directly in the ObjFS Bigtables. These results are available to the executors for future actions, or to be downloaded by the end user via objfsd. + +The end result is a system that scales to efficiently support all builds performed at Google. And the scale of Google’s builds is truly massive: Google runs millions of builds executing millions of test cases and producing petabytes of build outputs from billions of lines of source code every day. Not only does such a system let our engineers build complex codebases quickly, it also allows us to implement a huge number of automated tools and systems that rely on our build. diff --git a/basics/hermeticity.mdx b/basics/hermeticity.mdx index dcee3a9e..a9765af8 100644 --- a/basics/hermeticity.mdx +++ b/basics/hermeticity.mdx @@ -2,108 +2,59 @@ title: 'Hermeticity' --- - - -This page covers hermeticity, the benefits of using hermetic builds, and -strategies for identifying non-hermetic behavior in your builds. +This page covers hermeticity, the benefits of using hermetic builds, and strategies for identifying non-hermetic behavior in your builds. ## Overview -When given the same input source code and product configuration, a hermetic -build system always returns the same output by isolating the build from changes -to the host system. +When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system. -In order to isolate the build, hermetic builds are insensitive to libraries and -other software installed on the local or remote host machine. They depend on -specific versions of build tools, such as compilers, and dependencies, such as -libraries. This makes the build process self-contained as it doesn't rely on -services external to the build environment. +In order to isolate the build, hermetic builds are insensitive to libraries and other software installed on the local or remote host machine. They depend on specific versions of build tools, such as compilers, and dependencies, such as libraries. This makes the build process self-contained as it doesn't rely on services external to the build environment. The two important aspects of hermeticity are: -* **Isolation**: Hermetic build systems treat tools as source code. They - download copies of tools and manage their storage and use inside managed file - trees. This creates isolation between the host machine and local user, - including installed versions of languages. -* **Source identity**: Hermetic build systems try to ensure the sameness of - inputs. Code repositories, such as Git, identify sets of code mutations with a - unique hash code. Hermetic build systems use this hash to identify changes to - the build's input. +- **Isolation**: Hermetic build systems treat tools as source code. They download copies of tools and manage their storage and use inside managed file trees. This creates isolation between the host machine and local user, including installed versions of languages. +- **Source identity**: Hermetic build systems try to ensure the sameness of inputs. Code repositories, such as Git, identify sets of code mutations with a unique hash code. Hermetic build systems use this hash to identify changes to the build's input. ## Benefits The major benefits of hermetic builds are: -* **Speed**: The output of an action can be cached, and the action need not be - run again unless inputs change. -* **Parallel execution**: For given input and output, the build system can - construct a graph of all actions to calculate efficient and parallel - execution. The build system loads the rules and calculates an action graph - and hash inputs to look up in the cache. -* **Multiple builds**: You can build multiple hermetic builds on the same - machine, each build using different tools and versions. -* **Reproducibility**: Hermetic builds are good for troubleshooting because you - know the exact conditions that produced the build. +- **Speed**: The output of an action can be cached, and the action need not be run again unless inputs change. +- **Parallel execution**: For given input and output, the build system can construct a graph of all actions to calculate efficient and parallel execution. The build system loads the rules and calculates an action graph and hash inputs to look up in the cache. +- **Multiple builds**: You can build multiple hermetic builds on the same machine, each build using different tools and versions. +- **Reproducibility**: Hermetic builds are good for troubleshooting because you know the exact conditions that produced the build. ## Identifying non-hermeticity -If you are preparing to switch to Bazel, migration is easier if you improve -your existing builds' hermeticity in advance. Some common sources of -non-hermeticity in builds are: +If you are preparing to switch to Bazel, migration is easier if you improve your existing builds' hermeticity in advance. Some common sources of non-hermeticity in builds are: -* Arbitrary processing in `.mk` files -* Actions or tooling that create files non-deterministically, usually involving - build IDs or timestamps -* System binaries that differ across hosts (such as `/usr/bin` binaries, absolute - paths, system C++ compilers for native C++ rules autoconfiguration) -* Writing to the source tree during the build. This prevents the same source - tree from being used for another target. The first build writes to the source - tree, fixing the source tree for target A. Then trying to build target B may - fail. +- Arbitrary processing in `.mk` files +- Actions or tooling that create files non-deterministically, usually involving build IDs or timestamps +- System binaries that differ across hosts (such as `/usr/bin` binaries, absolute paths, system C++ compilers for native C++ rules autoconfiguration) +- Writing to the source tree during the build. This prevents the same source tree from being used for another target. The first build writes to the source tree, fixing the source tree for target A. Then trying to build target B may fail. ## Troubleshooting non-hermetic builds -Starting with local execution, issues that affect local cache hits reveal -non-hermetic actions. - -* Ensure null sequential builds: If you run `make` and get a successful build, - running the build again should not rebuild any targets. If you run each build - step twice or on different systems, compare a hash of the file contents and - get results that differ, the build is not reproducible. -* Run steps to - [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) - from a variety of potential client machines to ensure that you catch any - cases of client environment leaking into the actions. -* Execute a build within a docker container that contains nothing but the - checked-out source tree and explicit list of host tools. Build breakages and - error messages will catch implicit system dependencies. -* Discover and fix hermeticity problems using - [remote execution rules](/remote/rules#overview). -* Enable strict [sandboxing](/docs/sandboxing) - at the per-action level, since actions in a build can be stateful and affect - the build or the output. -* [Workspace rules](/remote/workspace) - allow developers to add dependencies to external workspaces, but they are - rich enough to allow arbitrary processing to happen in the process. You can - get a log of some potentially non-hermetic actions in Bazel workspace rules by - adding the flag - `--experimental_workspace_rules_log_file=PATH` to - your Bazel command. - -Note: Make your build fully hermetic when mixing remote and local execution, -using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote -Docker container will enable the build to execute the same in both environments. +Starting with local execution, issues that affect local cache hits reveal non-hermetic actions. + +- Ensure null sequential builds: If you run `make` and get a successful build, running the build again should not rebuild any targets. If you run each build step twice or on different systems, compare a hash of the file contents and get results that differ, the build is not reproducible. +- Run steps to [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) from a variety of potential client machines to ensure that you catch any cases of client environment leaking into the actions. +- Execute a build within a docker container that contains nothing but the checked-out source tree and explicit list of host tools. Build breakages and error messages will catch implicit system dependencies. +- Discover and fix hermeticity problems using [remote execution rules](/remote/rules#overview). +- Enable strict [sandboxing](/docs/sandboxing) at the per-action level, since actions in a build can be stateful and affect the build or the output. +- [Workspace rules](/remote/workspace) allow developers to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. You can get a log of some potentially non-hermetic actions in Bazel workspace rules by adding the flag `--experimental_workspace_rules_log_file=<var>PATH</var>` to your Bazel command. + +Note: Make your build fully hermetic when mixing remote and local execution, using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote Docker container will enable the build to execute the same in both environments. ## Hermeticity with Bazel -For more information about how other projects have had success using hermetic -builds with Bazel, see these BazelCon talks: - -* [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) -* [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) -* [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) -* [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) -* [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) -* [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) -* [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=4&t=0s) (BMW) -* [Building Self Driving Cars with Bazel + Q&A](https://www.youtube.com/watch?v=fjfFe98LTm8&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=29) (GM Cruise) +For more information about how other projects have had success using hermetic builds with Bazel, see these BazelCon talks: + +- [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) +- [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) +- [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) +- [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) +- [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) +- [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) +- [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=4\&t=0s) (BMW) +- [Building Self Driving Cars with Bazel + Q\&A](https://www.youtube.com/watch?v=fjfFe98LTm8\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=29) (GM Cruise) diff --git a/basics/index.mdx b/basics/index.mdx index f3c833fa..ecd09227 100644 --- a/basics/index.mdx +++ b/basics/index.mdx @@ -2,56 +2,28 @@ title: 'Build Basics' --- +A build system is one of the most important parts of an engineering organization because each developer interacts with it potentially dozens or hundreds of times per day. A fully featured build system is necessary to enable developer productivity as an organization scales. For individual developers, it's straightforward to just compile your code and so a build system might seem excessive. But at a larger scale, having a build system helps with managing shared dependencies, such as relying on another part of the code base, or an external resource, such as a library. Build systems help to make sure that you have everything you need to build your code before it starts building. Build systems also increase velocity when they're set up to help engineers share resources and results. +This section covers some history and basics of building and build systems, including design decisions that went into making Bazel. If you're familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you can skip this section, but it's a helpful overview to understand why artifact-based build systems are excellent at enabling scale. -A build system is one of the most important parts of an engineering organization -because each developer interacts with it potentially dozens or hundreds of times -per day. A fully featured build system is necessary to enable developer -productivity as an organization scales. For individual developers, it's -straightforward to just compile your code and so a build system might seem -excessive. But at a larger scale, having a build system helps with managing -shared dependencies, such as relying on another part of the code base, or an -external resource, such as a library. Build systems help to make sure that you -have everything you need to build your code before it starts building. Build -systems also increase velocity when they're set up to help engineers share -resources and results. +Note: Much of this section's content comes from the *Build Systems and Build Philosophy* chapter of the [*Software Engineering at Google* book](https://abseil.io/resources/swe-book/html/ch18.html). Thank you to the original author, Erik Kuefler, for allowing its reuse and modification here! -This section covers some history and basics of building and build systems, -including design decisions that went into making Bazel. If you're -familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you -can skip this section, but it's a helpful overview to understand why -artifact-based build systems are excellent at enabling scale. +- **[Why a Build System?](/basics/build-systems)** -Note: Much of this section's content comes from the _Build Systems and -Build Philosophy_ chapter of the -[_Software Engineering at Google_ book](https://abseil.io/resources/swe-book/html/ch18.html). -Thank you to the original author, Erik Kuefler, for allowing its reuse and -modification here! + If you haven't used a build system before, start here. This page covers why you should use a build system, and why compilers and build scripts aren't the best choice once your organization starts to scale beyond a few developers. -* **[Why a Build System?](/basics/build-systems)** +- **[Task-Based Build Systems](/basics/task-based-builds)** - If you haven't used a build system before, start here. This page covers why - you should use a build system, and why compilers and build scripts aren't - the best choice once your organization starts to scale beyond a few - developers. + This page discusses task-based build systems (such as Make, Maven, and Gradle) and some of their challenges. -* **[Task-Based Build Systems](/basics/task-based-builds)** +- **[Artifact-Based Build Systems](/basics/artifact-based-builds)** - This page discusses task-based build systems (such as Make, Maven, and - Gradle) and some of their challenges. + This page discusses artifact-based build systems in response to the pain points of task-based build systems. -* **[Artifact-Based Build Systems](/basics/artifact-based-builds)** +- **[Distributed Builds](/basics/distributed-builds)** - This page discusses artifact-based build systems in response to the pain - points of task-based build systems. + This page covers distributed builds, or builds that are executed outside of your local machine. This requires more robust infrastructure to share resources and build results (and is where the true wizardry happens!) -* **[Distributed Builds](/basics/distributed-builds)** +- **[Dependency Management](/basics/dependencies)** - This page covers distributed builds, or builds that are executed outside of - your local machine. This requires more robust infrastructure to share - resources and build results (and is where the true wizardry happens!) - -* **[Dependency Management](/basics/dependencies)** - - This page covers some complications of dependencies at a large scale and - strategies to counteract those complications. + This page covers some complications of dependencies at a large scale and strategies to counteract those complications. diff --git a/basics/task-based-builds.mdx b/basics/task-based-builds.mdx index 9dd3f8c0..daef39b3 100644 --- a/basics/task-based-builds.mdx +++ b/basics/task-based-builds.mdx @@ -2,94 +2,68 @@ title: 'Task-Based Build Systems' --- - - -This page covers task-based build systems, how they work and some of the -complications that can occur with task-based systems. After shell scripts, -task-based build systems are the next logical evolution of building. - +This page covers task-based build systems, how they work and some of the complications that can occur with task-based systems. After shell scripts, task-based build systems are the next logical evolution of building. ## Understanding task-based build systems -In a task-based build system, the fundamental unit of work is the task. Each -task is a script that can execute any sort of logic, and tasks specify other -tasks as dependencies that must run before them. Most major build systems in use -today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of -shell scripts, most modern build systems require engineers to create build files -that describe how to perform the build. +In a task-based build system, the fundamental unit of work is the task. Each task is a script that can execute any sort of logic, and tasks specify other tasks as dependencies that must run before them. Most major build systems in use today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of shell scripts, most modern build systems require engineers to create build files that describe how to perform the build. -Take this example from the -[Ant manual](https://ant.apache.org/manual/using.html): +Take this example from the [Ant manual](https://ant.apache.org/manual/using.html): ```xml - - +<project name="MyProject" default="dist" basedir="."> + <description> simple example build file - - - - - - - - - - - - - - - - - - - - - - - - - - - - + </description> + + <property name="src" location="src"/> + <property name="build" location="build"/> + <property name="dist" location="dist"/> + + <target name="init"> + + <tstamp/> + + <mkdir dir="${build}"/> + </target> + <target name="compile" depends="init" + description="compile the source"> + + <javac srcdir="${src}" destdir="${build}"/> + </target> + <target name="dist" depends="compile" + description="generate the distribution"> + + <mkdir dir="${dist}/lib"/> + + <jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar" basedir="${build}"/> + </target> + <target name="clean" + description="clean up"> + + <delete dir="${build}"/> + <delete dir="${dist}"/> + </target> +</project> ``` -The buildfile is written in XML and defines some simple metadata about the build -along with a list of tasks (the `` tags in the XML). (Ant uses the word -_target_ to represent a _task_, and it uses the word _task_ to refer to -_commands_.) Each task executes a list of possible commands defined by Ant, -which here include creating and deleting directories, running `javac`, and -creating a JAR file. This set of commands can be extended by user-provided -plug-ins to cover any sort of logic. Each task can also define the tasks it -depends on via the depends attribute. These dependencies form an acyclic graph, -as seen in Figure 1. +The buildfile is written in XML and defines some simple metadata about the build along with a list of tasks (the `<target>` tags in the XML). (Ant uses the word *target* to represent a *task*, and it uses the word *task* to refer to *commands*.) Each task executes a list of possible commands defined by Ant, which here include creating and deleting directories, running `javac`, and creating a JAR file. This set of commands can be extended by user-provided plug-ins to cover any sort of logic. Each task can also define the tasks it depends on via the depends attribute. These dependencies form an acyclic graph, as seen in Figure 1. [![Acrylic graph showing dependencies](/images/task-dependencies.png)](/images/task-dependencies.png) Figure 1. An acyclic graph showing dependencies -Users perform builds by providing tasks to Ant’s command-line tool. For example, -when a user types `ant dist`, Ant takes the following steps: - -1. Loads a file named `build.xml` in the current directory and parses it to - create the graph structure shown in Figure 1. -1. Looks for the task named `dist` that was provided on the command line and - discovers that it has a dependency on the task named `compile`. -1. Looks for the task named `compile` and discovers that it has a dependency on - the task named `init`. -1. Looks for the task named `init` and discovers that it has no dependencies. -1. Executes the commands defined in the `init` task. -1. Executes the commands defined in the `compile` task given that all of that - task’s dependencies have been run. -1. Executes the commands defined in the `dist` task given that all of that - task’s dependencies have been run. - -In the end, the code executed by Ant when running the `dist` task is equivalent -to the following shell script: +Users perform builds by providing tasks to Ant’s command-line tool. For example, when a user types `ant dist`, Ant takes the following steps: + +1. Loads a file named `build.xml` in the current directory and parses it to create the graph structure shown in Figure 1. +2. Looks for the task named `dist` that was provided on the command line and discovers that it has a dependency on the task named `compile`. +3. Looks for the task named `compile` and discovers that it has a dependency on the task named `init`. +4. Looks for the task named `init` and discovers that it has no dependencies. +5. Executes the commands defined in the `init` task. +6. Executes the commands defined in the `compile` task given that all of that task’s dependencies have been run. +7. Executes the commands defined in the `dist` task given that all of that task’s dependencies have been run. + +In the end, the code executed by Ant when running the `dist` task is equivalent to the following shell script: ```posix-terminal ./createTimestamp.sh @@ -103,114 +77,32 @@ mkdir -p dist/lib/ jar cf dist/lib/MyProject-$(date --iso-8601).jar build/* ``` -When the syntax is stripped away, the buildfile and the build script actually -aren’t too different. But we’ve already gained a lot by doing this. We can -create new buildfiles in other directories and link them together. We can easily -add new tasks that depend on existing tasks in arbitrary and complex ways. We -need only pass the name of a single task to the `ant` command-line tool, and it -determines everything that needs to be run. - -Ant is an old piece of software, originally released in 2000. Other tools like -Maven and Gradle have improved on Ant in the intervening years and essentially -replaced it by adding features like automatic management of external -dependencies and a cleaner syntax without any XML. But the nature of these newer -systems remains the same: they allow engineers to write build scripts in a -principled and modular way as tasks and provide tools for executing those tasks -and managing dependencies among them. +When the syntax is stripped away, the buildfile and the build script actually aren’t too different. But we’ve already gained a lot by doing this. We can create new buildfiles in other directories and link them together. We can easily add new tasks that depend on existing tasks in arbitrary and complex ways. We need only pass the name of a single task to the `ant` command-line tool, and it determines everything that needs to be run. + +Ant is an old piece of software, originally released in 2000. Other tools like Maven and Gradle have improved on Ant in the intervening years and essentially replaced it by adding features like automatic management of external dependencies and a cleaner syntax without any XML. But the nature of these newer systems remains the same: they allow engineers to write build scripts in a principled and modular way as tasks and provide tools for executing those tasks and managing dependencies among them. ## The dark side of task-based build systems -Because these tools essentially let engineers define any script as a task, they -are extremely powerful, allowing you to do pretty much anything you can imagine -with them. But that power comes with drawbacks, and task-based build systems can -become difficult to work with as their build scripts grow more complex. The -problem with such systems is that they actually end up giving _too much power to -engineers and not enough power to the system_. Because the system has no idea -what the scripts are doing, performance suffers, as it must be very conservative -in how it schedules and executes build steps. And there’s no way for the system -to confirm that each script is doing what it should, so scripts tend to grow in -complexity and end up being another thing that needs debugging. +Because these tools essentially let engineers define any script as a task, they are extremely powerful, allowing you to do pretty much anything you can imagine with them. But that power comes with drawbacks, and task-based build systems can become difficult to work with as their build scripts grow more complex. The problem with such systems is that they actually end up giving *too much power to engineers and not enough power to the system*. Because the system has no idea what the scripts are doing, performance suffers, as it must be very conservative in how it schedules and executes build steps. And there’s no way for the system to confirm that each script is doing what it should, so scripts tend to grow in complexity and end up being another thing that needs debugging. ### Difficulty of parallelizing build steps -Modern development workstations are quite powerful, with multiple cores that are -capable of executing several build steps in parallel. But task-based systems are -often unable to parallelize task execution even when it seems like they should -be able to. Suppose that task A depends on tasks B and C. Because tasks B and C -have no dependency on each other, is it safe to run them at the same time so -that the system can more quickly get to task A? Maybe, if they don’t touch any -of the same resources. But maybe not—perhaps both use the same file to track -their statuses and running them at the same time causes a conflict. There’s no -way in general for the system to know, so either it has to risk these conflicts -(leading to rare but very difficult-to-debug build problems), or it has to -restrict the entire build to running on a single thread in a single process. -This can be a huge waste of a powerful developer machine, and it completely -rules out the possibility of distributing the build across multiple machines. +Modern development workstations are quite powerful, with multiple cores that are capable of executing several build steps in parallel. But task-based systems are often unable to parallelize task execution even when it seems like they should be able to. Suppose that task A depends on tasks B and C. Because tasks B and C have no dependency on each other, is it safe to run them at the same time so that the system can more quickly get to task A? Maybe, if they don’t touch any of the same resources. But maybe not—perhaps both use the same file to track their statuses and running them at the same time causes a conflict. There’s no way in general for the system to know, so either it has to risk these conflicts (leading to rare but very difficult-to-debug build problems), or it has to restrict the entire build to running on a single thread in a single process. This can be a huge waste of a powerful developer machine, and it completely rules out the possibility of distributing the build across multiple machines. ### Difficulty performing incremental builds -A good build system allows engineers to perform reliable incremental builds such -that a small change doesn’t require the entire codebase to be rebuilt from -scratch. This is especially important if the build system is slow and unable to -parallelize build steps for the aforementioned reasons. But unfortunately, -task-based build systems struggle here, too. Because tasks can do anything, -there’s no way in general to check whether they’ve already been done. Many tasks -simply take a set of source files and run a compiler to create a set of -binaries; thus, they don’t need to be rerun if the underlying source files -haven’t changed. But without additional information, the system can’t say this -for sure—maybe the task downloads a file that could have changed, or maybe it -writes a timestamp that could be different on each run. To guarantee -correctness, the system typically must rerun every task during each build. Some -build systems try to enable incremental builds by letting engineers specify the -conditions under which a task needs to be rerun. Sometimes this is feasible, but -often it’s a much trickier problem than it appears. For example, in languages -like C++ that allow files to be included directly by other files, it’s -impossible to determine the entire set of files that must be watched for changes -without parsing the input sources. Engineers often end up taking shortcuts, and -these shortcuts can lead to rare and frustrating problems where a task result is -reused even when it shouldn’t be. When this happens frequently, engineers get -into the habit of running clean before every build to get a fresh state, -completely defeating the purpose of having an incremental build in the first -place. Figuring out when a task needs to be rerun is surprisingly subtle, and is -a job better handled by machines than humans. +A good build system allows engineers to perform reliable incremental builds such that a small change doesn’t require the entire codebase to be rebuilt from scratch. This is especially important if the build system is slow and unable to parallelize build steps for the aforementioned reasons. But unfortunately, task-based build systems struggle here, too. Because tasks can do anything, there’s no way in general to check whether they’ve already been done. Many tasks simply take a set of source files and run a compiler to create a set of binaries; thus, they don’t need to be rerun if the underlying source files haven’t changed. But without additional information, the system can’t say this for sure—maybe the task downloads a file that could have changed, or maybe it writes a timestamp that could be different on each run. To guarantee correctness, the system typically must rerun every task during each build. Some build systems try to enable incremental builds by letting engineers specify the conditions under which a task needs to be rerun. Sometimes this is feasible, but often it’s a much trickier problem than it appears. For example, in languages like C++ that allow files to be included directly by other files, it’s impossible to determine the entire set of files that must be watched for changes without parsing the input sources. Engineers often end up taking shortcuts, and these shortcuts can lead to rare and frustrating problems where a task result is reused even when it shouldn’t be. When this happens frequently, engineers get into the habit of running clean before every build to get a fresh state, completely defeating the purpose of having an incremental build in the first place. Figuring out when a task needs to be rerun is surprisingly subtle, and is a job better handled by machines than humans. ### Difficulty maintaining and debugging scripts -Finally, the build scripts imposed by task-based build systems are often just -difficult to work with. Though they often receive less scrutiny, build scripts -are code just like the system being built, and are easy places for bugs to hide. -Here are some examples of bugs that are very common when working with a -task-based build system: - -* Task A depends on task B to produce a particular file as output. The owner - of task B doesn’t realize that other tasks rely on it, so they change it to - produce output in a different location. This can’t be detected until someone - tries to run task A and finds that it fails. -* Task A depends on task B, which depends on task C, which is producing a - particular file as output that’s needed by task A. The owner of task B - decides that it doesn’t need to depend on task C any more, which causes task - A to fail even though task B doesn’t care about task C at all! -* The developer of a new task accidentally makes an assumption about the - machine running the task, such as the location of a tool or the value of - particular environment variables. The task works on their machine, but fails - whenever another developer tries it. -* A task contains a nondeterministic component, such as downloading a file - from the internet or adding a timestamp to a build. Now, people get - potentially different results each time they run the build, meaning that - engineers won’t always be able to reproduce and fix one another’s failures - or failures that occur on an automated build system. -* Tasks with multiple dependencies can create race conditions. If task A - depends on both task B and task C, and task B and C both modify the same - file, task A gets a different result depending on which one of tasks B and C - finishes first. - -There’s no general-purpose way to solve these performance, correctness, or -maintainability problems within the task-based framework laid out here. So long -as engineers can write arbitrary code that runs during the build, the system -can’t have enough information to always be able to run builds quickly and -correctly. To solve the problem, we need to take some power out of the hands of -engineers and put it back in the hands of the system and reconceptualize the -role of the system not as running tasks, but as producing artifacts. - -This approach led to the creation of artifact-based build systems, like Blaze -and Bazel. +Finally, the build scripts imposed by task-based build systems are often just difficult to work with. Though they often receive less scrutiny, build scripts are code just like the system being built, and are easy places for bugs to hide. Here are some examples of bugs that are very common when working with a task-based build system: + +- Task A depends on task B to produce a particular file as output. The owner of task B doesn’t realize that other tasks rely on it, so they change it to produce output in a different location. This can’t be detected until someone tries to run task A and finds that it fails. +- Task A depends on task B, which depends on task C, which is producing a particular file as output that’s needed by task A. The owner of task B decides that it doesn’t need to depend on task C any more, which causes task A to fail even though task B doesn’t care about task C at all! +- The developer of a new task accidentally makes an assumption about the machine running the task, such as the location of a tool or the value of particular environment variables. The task works on their machine, but fails whenever another developer tries it. +- A task contains a nondeterministic component, such as downloading a file from the internet or adding a timestamp to a build. Now, people get potentially different results each time they run the build, meaning that engineers won’t always be able to reproduce and fix one another’s failures or failures that occur on an automated build system. +- Tasks with multiple dependencies can create race conditions. If task A depends on both task B and task C, and task B and C both modify the same file, task A gets a different result depending on which one of tasks B and C finishes first. + +There’s no general-purpose way to solve these performance, correctness, or maintainability problems within the task-based framework laid out here. So long as engineers can write arbitrary code that runs during the build, the system can’t have enough information to always be able to run builds quickly and correctly. To solve the problem, we need to take some power out of the hands of engineers and put it back in the hands of the system and reconceptualize the role of the system not as running tasks, but as producing artifacts. + +This approach led to the creation of artifact-based build systems, like Blaze and Bazel. diff --git a/brand/index.mdx b/brand/index.mdx index 2a21cd43..f6b1bd31 100644 --- a/brand/index.mdx +++ b/brand/index.mdx @@ -2,87 +2,57 @@ title: 'Bazel Brand Guidelines' --- - - -The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and -are treated separately from the copyright or patent license grants contained in -the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel -Trademarks other than those permitted in these guidelines must be approved in -advance. +The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and are treated separately from the copyright or patent license grants contained in the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel Trademarks other than those permitted in these guidelines must be approved in advance. ## Purpose of the Brand Guidelines -These guidelines exist to ensure that the Bazel project can share its technology -under open source licenses while making sure that the "Bazel" brand is protected -as a meaningful source identifier in a way that's consistent with trademark law. -By adhering to these guidelines, you help to promote the freedom to use and -develop high-quality Bazel technology. +These guidelines exist to ensure that the Bazel project can share its technology under open source licenses while making sure that the "Bazel" brand is protected as a meaningful source identifier in a way that's consistent with trademark law. By adhering to these guidelines, you help to promote the freedom to use and develop high-quality Bazel technology. ## Acceptable Uses -Given the open nature of Bazel, you may use the Bazel trademark to refer to the -project without prior written permission. Examples of these approved references -include the following: +Given the open nature of Bazel, you may use the Bazel trademark to refer to the project without prior written permission. Examples of these approved references include the following: -* To refer to the Bazel Project itself; -* To link to bazel.build; -* To refer to unmodified source code or other files shared by the Bazel - repositories on GitHub; -* In blog posts, news articles, or educational materials about Bazel; -* To accurately identify that your design or implementation is based on, is - for use with, or is compatible with Bazel technology. +- To refer to the Bazel Project itself; +- To link to bazel.build; +- To refer to unmodified source code or other files shared by the Bazel repositories on GitHub; +- In blog posts, news articles, or educational materials about Bazel; +- To accurately identify that your design or implementation is based on, is for use with, or is compatible with Bazel technology. Examples: -* \[Your Product\] for Bazel -* \[Your Product\] is compatible with Bazel -* \[XYZ\] Conference for Bazel Users +- \[Your Product] for Bazel +- \[Your Product] is compatible with Bazel +- \[XYZ] Conference for Bazel Users ## General Guidelines -* The Bazel name may never be used or registered in a manner that would cause - confusion as to Google's sponsorship, affiliation, or endorsement. -* Don't use the Bazel name as part of your company name, product name, domain - name, or social media profile. -* Other than as permitted by these guidelines, the Bazel name should not be - combined with other trademarks, terms, or source identifiers. -* Don't remove, distort or alter any element of the Bazel Trademarks. That - includes modifying the Bazel Trademark, for example, through hyphenation, - combination or abbreviation. Do not shorten, abbreviate, or create acronyms - out of the Bazel Trademarks. -* Don't display the word Bazel using any different stylization, color, or font - from the surrounding text. -* Don't use the term Bazel as a verb or use it in possessive form. -* Don't use the Bazel logo on any website, product UI, or promotional - materials without prior written permission from - [product@bazel.build](mailto:product@bazel.build). +- The Bazel name may never be used or registered in a manner that would cause confusion as to Google's sponsorship, affiliation, or endorsement. +- Don't use the Bazel name as part of your company name, product name, domain name, or social media profile. +- Other than as permitted by these guidelines, the Bazel name should not be combined with other trademarks, terms, or source identifiers. +- Don't remove, distort or alter any element of the Bazel Trademarks. That includes modifying the Bazel Trademark, for example, through hyphenation, combination or abbreviation. Do not shorten, abbreviate, or create acronyms out of the Bazel Trademarks. +- Don't display the word Bazel using any different stylization, color, or font from the surrounding text. +- Don't use the term Bazel as a verb or use it in possessive form. +- Don't use the Bazel logo on any website, product UI, or promotional materials without prior written permission from [product@bazel.build](mailto:product@bazel.build). ## Usage for Events and Community Groups -The Bazel word mark may be used referentially in events, community groups, or -other gatherings related to the Bazel build system, but it may not be used in a -manner that implies official status or endorsement. +The Bazel word mark may be used referentially in events, community groups, or other gatherings related to the Bazel build system, but it may not be used in a manner that implies official status or endorsement. Examples of appropriate naming conventions are: -* \[XYZ\] Bazel User Group -* Bazel Community Day at \[XYZ\] -* \[XYZ\] Conference for Bazel Users +- \[XYZ] Bazel User Group +- Bazel Community Day at \[XYZ] +- \[XYZ] Conference for Bazel Users -where \[XYZ\] represents the location and optionally other wordings. +where \[XYZ] represents the location and optionally other wordings. -Any naming convention that may imply official status or endorsement requires -review for approval from [product@bazel.build](mailto:product@bazel.build). +Any naming convention that may imply official status or endorsement requires review for approval from [product@bazel.build](mailto:product@bazel.build). Examples of naming conventions that require prior written permission: -* BazelCon -* Bazel Conference +- BazelCon +- Bazel Conference ## Contact Us -Please do not hesitate to contact us at -[product@bazel.build](mailto:product@bazel.build) if you are unsure whether your -intended use of the Bazel Trademarks is in compliance with these guidelines, or -to ask for permission to use the Bazel Trademarks, clearly describing the -intended usage and duration. +Please do not hesitate to contact us at [product@bazel.build](mailto:product@bazel.build) if you are unsure whether your intended use of the Bazel Trademarks is in compliance with these guidelines, or to ask for permission to use the Bazel Trademarks, clearly describing the intended usage and duration. diff --git a/build/share-variables.mdx b/build/share-variables.mdx index b248034e..3bf6683b 100644 --- a/build/share-variables.mdx +++ b/build/share-variables.mdx @@ -2,13 +2,9 @@ title: 'Sharing Variables' --- +`BUILD` files are intended to be simple and declarative. They will typically consist of a series of target declarations. As your code base and your `BUILD` files get larger, you will probably notice some duplication, such as: - -`BUILD` files are intended to be simple and declarative. They will typically -consist of a series of target declarations. As your code base and your `BUILD` -files get larger, you will probably notice some duplication, such as: - -``` python +```python cc_library( name = "foo", copts = ["-DVERSION=5"], @@ -23,17 +19,11 @@ cc_library( ) ``` -Code duplication in `BUILD` files is usually fine. This can make the file more -readable: each declaration can be read and understood without any context. This -is important, not only for humans, but also for external tools. For example, a -tool might be able to read and update `BUILD` files to add missing dependencies. -Code refactoring and code reuse might prevent this kind of automated -modification. +Code duplication in `BUILD` files is usually fine. This can make the file more readable: each declaration can be read and understood without any context. This is important, not only for humans, but also for external tools. For example, a tool might be able to read and update `BUILD` files to add missing dependencies. Code refactoring and code reuse might prevent this kind of automated modification. -If it is useful to share values (for example, if values must be kept in sync), -you can introduce a variable: +If it is useful to share values (for example, if values must be kept in sync), you can introduce a variable: -``` python +```python COPTS = ["-DVERSION=5"] cc_library( @@ -50,24 +40,21 @@ cc_library( ) ``` -Multiple declarations now use the value `COPTS`. By convention, use uppercase -letters to name global constants. +Multiple declarations now use the value `COPTS`. By convention, use uppercase letters to name global constants. ## Sharing variables across multiple BUILD files -If you need to share a value across multiple `BUILD` files, you have to put it -in a `.bzl` file. `.bzl` files contain definitions (variables and functions) -that can be used in `BUILD` files. +If you need to share a value across multiple `BUILD` files, you have to put it in a `.bzl` file. `.bzl` files contain definitions (variables and functions) that can be used in `BUILD` files. In `path/to/variables.bzl`, write: -``` python +```python COPTS = ["-DVERSION=5"] ``` Then, you can update your `BUILD` files to access the variable: -``` python +```python load("//path/to:variables.bzl", "COPTS") cc_library( diff --git a/build/style-guide.mdx b/build/style-guide.mdx index cfc3cf86..d0a01fe0 100644 --- a/build/style-guide.mdx +++ b/build/style-guide.mdx @@ -2,31 +2,17 @@ title: 'BUILD Style Guide' --- - - ## Prefer DAMP BUILD files over DRY -The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by -introducing abstractions such as variables and functions to avoid redundancy in -code. +The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by introducing abstractions such as variables and functions to avoid redundancy in code. -In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — -encourages readability over uniqueness to make files easier to understand and -maintain. +In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — encourages readability over uniqueness to make files easier to understand and maintain. -`BUILD` files aren't code, they are configurations. They aren't tested like -code, but do need to be maintained by people and tools. That makes DAMP better -for them than DRY. +`BUILD` files aren't code, they are configurations. They aren't tested like code, but do need to be maintained by people and tools. That makes DAMP better for them than DRY. ## BUILD.bazel file formatting -`BUILD` file formatting follows the same approach as Go, where a standardized -tool takes care of most formatting issues. -[Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and -emits the source code in a standard style. Every `BUILD` file is therefore -formatted in the same automated way, which makes formatting a non-issue during -code reviews. It also makes it easier for tools to understand, edit, and -generate `BUILD` files. +`BUILD` file formatting follows the same approach as Go, where a standardized tool takes care of most formatting issues. [Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and emits the source code in a standard style. Every `BUILD` file is therefore formatted in the same automated way, which makes formatting a non-issue during code reviews. It also makes it easier for tools to understand, edit, and generate `BUILD` files. `BUILD` file formatting must match the output of `buildifier`. @@ -59,18 +45,15 @@ py_test( **Recommendation**: Use the following order (every element is optional): -* Package description (a comment) +- Package description (a comment) -* All `load()` statements +- All `load()` statements -* The `package()` function. +- The `package()` function. -* Calls to rules and macros +- Calls to rules and macros -Buildifier makes a distinction between a standalone comment and a comment -attached to an element. If a comment is not attached to a specific element, use -an empty line after it. The distinction is important when doing automated -changes (for example, to keep or remove a comment when deleting a rule). +Buildifier makes a distinction between a standalone comment and a comment attached to an element. If a comment is not attached to a specific element, use an empty line after it. The distinction is important when doing automated changes (for example, to keep or remove a comment when deleting a rule). ```python # Standalone comment (such as to make a section in a file) @@ -81,11 +64,7 @@ cc_library(name = "cc") ## References to targets in the current package -Files should be referred to by their paths relative to the package directory -(without ever using up-references, such as `..`). Generated files should be -prefixed with "`:`" to indicate that they are not sources. Source files -should not be prefixed with `:`. Rules should be prefixed with `:`. For -example, assuming `x.cc` is a source file: +Files should be referred to by their paths relative to the package directory (without ever using up-references, such as `..`). Generated files should be prefixed with "`:`" to indicate that they are not sources. Source files should not be prefixed with `:`. Rules should be prefixed with `:`. For example, assuming `x.cc` is a source file: ```python cc_library( @@ -98,100 +77,70 @@ genrule( name = "gen_header", srcs = [], outs = ["x.h"], - cmd = "echo 'int x();' > $@", + cmd = "echo 'int x();' > $@", ) ``` ## Target naming -Target names should be descriptive. If a target contains one source file, -the target should generally have a name derived from that source (for example, a -`cc_library` for `chat.cc` could be named `chat`, or a `java_library` for -`DirectMessage.java` could be named `direct_message`). - -The eponymous target for a package (the target with the same name as the -containing directory) should provide the functionality described by the -directory name. If there is no such target, do not create an eponymous -target. - -Prefer using the short name when referring to an eponymous target (`//x` -instead of `//x:x`). If you are in the same package, prefer the local -reference (`:x` instead of `//x`). - -Avoid using "reserved" target names which have special meaning. This includes -`all`, `__pkg__`, and `__subpackages__`, these names have special -semantics and can cause confusion and unexpected behaviors when they are used. - -In the absence of a prevailing team convention these are some non-binding -recommendations that are broadly used at Google: - -* In general, use ["snake_case"](https://en.wikipedia.org/wiki/Snake_case) - * For a `java_library` with one `src` this means using a name that is not - the same as the filename without the extension - * For Java `*_binary` and `*_test` rules, use - ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). - This allows for the target name to match one of the `src`s. For - `java_test`, this makes it possible for the `test_class` attribute to be - inferred from the name of the target. -* If there are multiple variants of a particular target then add a suffix to - disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) -* Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` -* Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to - avoid conflicts between a `_library` target and its corresponding `_binary`) -* For proto related targets: - * `proto_library` targets should have names ending in `_proto` - * Languages specific `*_proto_library` rules should match the underlying - proto but replace `_proto` with a language specific suffix such as: - * **`cc_proto_library`**: `_cc_proto` - * **`java_proto_library`**: `_java_proto` - * **`java_lite_proto_library`**: `_java_proto_lite` +Target names should be descriptive. If a target contains one source file, the target should generally have a name derived from that source (for example, a `cc_library` for `chat.cc` could be named `chat`, or a `java_library` for `DirectMessage.java` could be named `direct_message`). + +The eponymous target for a package (the target with the same name as the containing directory) should provide the functionality described by the directory name. If there is no such target, do not create an eponymous target. + +Prefer using the short name when referring to an eponymous target (`//x` instead of `//x:x`). If you are in the same package, prefer the local reference (`:x` instead of `//x`). + +Avoid using "reserved" target names which have special meaning. This includes `all`, `__pkg__`, and `__subpackages__`, these names have special semantics and can cause confusion and unexpected behaviors when they are used. + +In the absence of a prevailing team convention these are some non-binding recommendations that are broadly used at Google: + +- In general, use ["snake\_case"](https://en.wikipedia.org/wiki/Snake_case) + + - For a `java_library` with one `src` this means using a name that is not the same as the filename without the extension + - For Java `*_binary` and `*_test` rules, use ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). This allows for the target name to match one of the `src`s. For `java_test`, this makes it possible for the `test_class` attribute to be inferred from the name of the target. + +- If there are multiple variants of a particular target then add a suffix to disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) + +- Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` + +- Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to avoid conflicts between a `_library` target and its corresponding `_binary`) + +- For proto related targets: + + - `proto_library` targets should have names ending in `_proto` + + - Languages specific `*_proto_library` rules should match the underlying proto but replace `_proto` with a language specific suffix such as: + + - **`cc_proto_library`**: `_cc_proto` + - **`java_proto_library`**: `_java_proto` + - **`java_lite_proto_library`**: `_java_proto_lite` ## Visibility -Visibility should be scoped as tightly as possible, while still allowing access -by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as -appropriate. +Visibility should be scoped as tightly as possible, while still allowing access by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as appropriate. -Avoid setting package `default_visibility` to `//visibility:public`. -`//visibility:public` should be individually set only for targets in the -project's public API. These could be libraries that are designed to be depended -on by external projects or binaries that could be used by an external project's -build process. +Avoid setting package `default_visibility` to `//visibility:public`. `//visibility:public` should be individually set only for targets in the project's public API. These could be libraries that are designed to be depended on by external projects or binaries that could be used by an external project's build process. ## Dependencies -Dependencies should be restricted to direct dependencies (dependencies -needed by the sources listed in the rule). Do not list transitive dependencies. +Dependencies should be restricted to direct dependencies (dependencies needed by the sources listed in the rule). Do not list transitive dependencies. -Package-local dependencies should be listed first and referred to in a way -compatible with the -[References to targets in the current package](#targets-current-package) -section above (not by their absolute package name). +Package-local dependencies should be listed first and referred to in a way compatible with the [References to targets in the current package](#targets-current-package) section above (not by their absolute package name). -Prefer to list dependencies directly, as a single list. Putting the "common" -dependencies of several targets into a variable reduces maintainability, makes -it impossible for tools to change the dependencies of a target, and can lead to -unused dependencies. +Prefer to list dependencies directly, as a single list. Putting the "common" dependencies of several targets into a variable reduces maintainability, makes it impossible for tools to change the dependencies of a target, and can lead to unused dependencies. ## Globs -Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it -is more error-prone and less obvious than an empty list. +Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it is more error-prone and less obvious than an empty list. ### Recursive -Do not use recursive globs to match source files (for example, -`glob(["**/*.java"])`). +Do not use recursive globs to match source files (for example, `glob(["**/*.java"])`). -Recursive globs make `BUILD` files difficult to reason about because they skip -subdirectories containing `BUILD` files. +Recursive globs make `BUILD` files difficult to reason about because they skip subdirectories containing `BUILD` files. -Recursive globs are generally less efficient than having a `BUILD` file per -directory with a dependency graph defined between them as this enables better -remote caching and parallelism. +Recursive globs are generally less efficient than having a `BUILD` file per directory with a dependency graph defined between them as this enables better remote caching and parallelism. -It is good practice to author a `BUILD` file in each directory and define a -dependency graph between them. +It is good practice to author a `BUILD` file in each directory and define a dependency graph between them. ### Non-recursive @@ -199,21 +148,16 @@ Non-recursive globs are generally acceptable. ## Avoid list comprehensions -Avoid using list comprehensions at the top level of a `BUILD.bazel` file. -Automate repetitive calls by creating each named target with a separate -top-level rule or macro call. Give each a short `name` parameter for clarity. +Avoid using list comprehensions at the top level of a `BUILD.bazel` file. Automate repetitive calls by creating each named target with a separate top-level rule or macro call. Give each a short `name` parameter for clarity. List comprehension reduces the following: -* Maintainability. It's difficult or impossible for human maintainers and - large scale automated changes to update list comprehensions correctly. -* Discoverability. Since the pattern doesn't have `name` parameters, - it's hard to find the rule by name. +- Maintainability. It's difficult or impossible for human maintainers and large scale automated changes to update list comprehensions correctly. +- Discoverability. Since the pattern doesn't have `name` parameters, it's hard to find the rule by name. -A common application of the list comprehension pattern is to generate tests. For -example: +A common application of the list comprehension pattern is to generate tests. For example: -```build {.bad} +```build [[java_test( name = "test_%s_%s" % (backend, count), srcs = [ ... ], @@ -228,8 +172,7 @@ example: ]] ``` -We recommend using simpler alternatives. For example, define a macro that -generates one test and invoke it for each top-level `name`: +We recommend using simpler alternatives. For example, define a macro that generates one test and invoke it for each top-level `name`: ```build my_java_test(name = "test_fake_1", @@ -243,7 +186,7 @@ my_java_test(name = "test_fake_10", Don't use list variables to encapsulate common dependencies: -```build {.bad} +```build COMMON_DEPS = [ "//d:e", "//x/y:z", @@ -260,12 +203,11 @@ cc_library(name = "b", ) ``` -Similarly, don't use a library target with -[`exports`](/reference/be/java#java_library.exports) to group dependencies. +Similarly, don't use a library target with [`exports`](/reference/be/java#java_library.exports) to group dependencies. Instead, list the dependencies separately for each target: -```build {.good} +```build cc_library(name = "a", srcs = ["a.cc"], deps = [ @@ -285,38 +227,25 @@ cc_library(name = "b", ) ``` -Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools -maintain them. There will be repetition, but you won't have to think about how -to manage the dependencies. +Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools maintain them. There will be repetition, but you won't have to think about how to manage the dependencies. ## Prefer literal strings -Although Starlark provides string operators for concatenation (`+`) and -formatting (`%`), use them with caution. It is tempting to factor out common -string parts to make expressions more concise or break long lines. However, +Although Starlark provides string operators for concatenation (`+`) and formatting (`%`), use them with caution. It is tempting to factor out common string parts to make expressions more concise or break long lines. However, -* It is harder to read broken-up string values at a glance. +- It is harder to read broken-up string values at a glance. -* Automated tools such as - [buildozer][buildozer] and Code Search have trouble finding values and - updating them correctly when the values broken up. +- Automated tools such as [buildozer](https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md) and Code Search have trouble finding values and updating them correctly when the values broken up. -* In `BUILD` files, readability is more important than avoiding repetition - (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). +- In `BUILD` files, readability is more important than avoiding repetition (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). -* This Style Guide - [warns against splitting label-valued strings](#other-conventions) - and - [explicitly permits long lines](#differences-python-style-guide). +- This Style Guide [warns against splitting label-valued strings](#other-conventions) and [explicitly permits long lines](#differences-python-style-guide). -* Buildifier automatically fuses concatenated strings when it detects that - they are labels. +- Buildifier automatically fuses concatenated strings when it detects that they are labels. -Therefore, prefer explicit, literal strings over concatenated or formatted -strings, especially in label-type attributes such as `name` and `deps`. For -example, this `BUILD` fragment: +Therefore, prefer explicit, literal strings over concatenated or formatted strings, especially in label-type attributes such as `name` and `deps`. For example, this `BUILD` fragment: -```build {.bad} +```build NAME = "foo" PACKAGE = "//a/b" @@ -330,7 +259,7 @@ proto_library( would be better rewritten as -```build {.good} +```build proto_library( name = "foo_proto", deps = ["//a/b:other_proto"], @@ -338,66 +267,32 @@ proto_library( ) ``` -[buildozer]: https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md - ## Limit the symbols exported by each `.bzl` file -Minimize the number of symbols (rules, macros, constants, functions) exported by -each public `.bzl` (Starlark) file. We recommend that a file should export -multiple symbols only if they are certain to be used together. Otherwise, split -it into multiple `.bzl` files, each with its own [bzl_library][bzl_library]. +Minimize the number of symbols (rules, macros, constants, functions) exported by each public `.bzl` (Starlark) file. We recommend that a file should export multiple symbols only if they are certain to be used together. Otherwise, split it into multiple `.bzl` files, each with its own [bzl\_library](https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library). -Excessive symbols can cause `.bzl` files to grow into broad "libraries" of -symbols, causing changes to single files to force Bazel to rebuild many targets. - -[bzl_library]: https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library +Excessive symbols can cause `.bzl` files to grow into broad "libraries" of symbols, causing changes to single files to force Bazel to rebuild many targets. ## Other conventions - * Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), - use lowercase and underscores to declare variables (such as `my_variable`). +- Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), use lowercase and underscores to declare variables (such as `my_variable`). - * Labels should never be split, even if they are longer than 79 characters. - Labels should be string literals whenever possible. *Rationale*: It makes - find and replace easy. It also improves readability. +- Labels should never be split, even if they are longer than 79 characters. Labels should be string literals whenever possible. *Rationale*: It makes find and replace easy. It also improves readability. - * The value of the name attribute should be a literal constant string (except - in macros). *Rationale*: External tools use the name attribute to refer a - rule. They need to find rules without having to interpret code. +- The value of the name attribute should be a literal constant string (except in macros). *Rationale*: External tools use the name attribute to refer a rule. They need to find rules without having to interpret code. - * When setting boolean-type attributes, use boolean values, not integer values. - For legacy reasons, rules still convert integers to booleans as needed, - but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying - "deflake this target by rerunning it once". `flaky = True` unambiguously says - "this test is flaky". +- When setting boolean-type attributes, use boolean values, not integer values. For legacy reasons, rules still convert integers to booleans as needed, but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying "deflake this target by rerunning it once". `flaky = True` unambiguously says "this test is flaky". ## Differences with Python style guide -Although compatibility with -[Python style guide](https://www.python.org/dev/peps/pep-0008/) -is a goal, there are a few differences: - - * No strict line length limit. Long comments and long strings are often split - to 79 columns, but it is not required. It should not be enforced in code - reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this - limit. It is common for `BUILD` files to be generated or edited by tools, - which does not go well with a line length limit. - - * Implicit string concatenation is not supported. Use the `+` operator. - *Rationale*: `BUILD` files contain many string lists. It is easy to forget a - comma, which leads to a complete different result. This has created many bugs - in the past. [See also this discussion.](https://lwn.net/Articles/551438/) - - * Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: - Named arguments are much more frequent than in Python and are always on a - separate line. Spaces improve readability. This convention has been around - for a long time, and it is not worth modifying all existing `BUILD` files. - - * By default, use double quotation marks for strings. *Rationale*: This is not - specified in the Python style guide, but it recommends consistency. So we - decided to use only double-quoted strings. Many languages use double-quotes - for string literals. - - * Use a single blank line between two top-level definitions. *Rationale*: The - structure of a `BUILD` file is not like a typical Python file. It has only - top-level statements. Using a single-blank line makes `BUILD` files shorter. +Although compatibility with [Python style guide](https://www.python.org/dev/peps/pep-0008/) is a goal, there are a few differences: + +- No strict line length limit. Long comments and long strings are often split to 79 columns, but it is not required. It should not be enforced in code reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this limit. It is common for `BUILD` files to be generated or edited by tools, which does not go well with a line length limit. + +- Implicit string concatenation is not supported. Use the `+` operator. *Rationale*: `BUILD` files contain many string lists. It is easy to forget a comma, which leads to a complete different result. This has created many bugs in the past. [See also this discussion.](https://lwn.net/Articles/551438/) + +- Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: Named arguments are much more frequent than in Python and are always on a separate line. Spaces improve readability. This convention has been around for a long time, and it is not worth modifying all existing `BUILD` files. + +- By default, use double quotation marks for strings. *Rationale*: This is not specified in the Python style guide, but it recommends consistency. So we decided to use only double-quoted strings. Many languages use double-quotes for string literals. + +- Use a single blank line between two top-level definitions. *Rationale*: The structure of a `BUILD` file is not like a typical Python file. It has only top-level statements. Using a single-blank line makes `BUILD` files shorter. diff --git a/community/recommended-rules.mdx b/community/recommended-rules.mdx index 86daa056..426d8784 100644 --- a/community/recommended-rules.mdx +++ b/community/recommended-rules.mdx @@ -2,53 +2,33 @@ title: 'Recommended Rules' --- +In the documentation, we provide a list of [recommended rules](/rules). - -In the documentation, we provide a list of -[recommended rules](/rules). - -This is a set of high quality rules, which will provide a good experience to our -users. We make a distinction between the supported rules, and the hundreds of -rules you can find on the Internet. +This is a set of high quality rules, which will provide a good experience to our users. We make a distinction between the supported rules, and the hundreds of rules you can find on the Internet. ## Nomination -If a ruleset meets the requirements below, a rule maintainer can nominate it -to be part of the _recommended rules_ by filing a -[GitHub issue](https://github.com/bazelbuild/bazel/). +If a ruleset meets the requirements below, a rule maintainer can nominate it to be part of the *recommended rules* by filing a [GitHub issue](https://github.com/bazelbuild/bazel/). -After a review by the [Bazel core team](/contribute/policy), it -will be recommended on the Bazel website. +After a review by the [Bazel core team](/contribute/policy), it will be recommended on the Bazel website. ## Requirements for the rule maintainers -* The ruleset provides an important feature, useful to a large number of Bazel - users (for example, support for a widely popular language). -* The ruleset is well maintained. There must be at least two active maintainers. -* The ruleset is well documented, with examples, and easy to use. -* The ruleset follows the best practices and is performant (see - [the performance guide](/rules/performance)). -* The ruleset has sufficient test coverage. -* The ruleset is tested on - [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) - with the latest version of Bazel. Tests should always pass (when used as a - presubmit check). -* The ruleset is also tested with the upcoming incompatible changes. Breakages - should be fixed within two weeks. Migration issues should be reported to the - Bazel team quickly. +- The ruleset provides an important feature, useful to a large number of Bazel users (for example, support for a widely popular language). +- The ruleset is well maintained. There must be at least two active maintainers. +- The ruleset is well documented, with examples, and easy to use. +- The ruleset follows the best practices and is performant (see [the performance guide](/rules/performance)). +- The ruleset has sufficient test coverage. +- The ruleset is tested on [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) with the latest version of Bazel. Tests should always pass (when used as a presubmit check). +- The ruleset is also tested with the upcoming incompatible changes. Breakages should be fixed within two weeks. Migration issues should be reported to the Bazel team quickly. ## Requirements for Bazel developers -* Recommended rules are frequently tested with Bazel at head (at least once a - day). -* No change in Bazel may break a recommended rule (with the default set of - flags). If it happens, the change should be fixed or rolled back. +- Recommended rules are frequently tested with Bazel at head (at least once a day). +- No change in Bazel may break a recommended rule (with the default set of flags). If it happens, the change should be fixed or rolled back. ## Demotion -If there is a concern that a particular ruleset is no longer meeting the -requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be -filed. +If there is a concern that a particular ruleset is no longer meeting the requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be filed. -Rule maintainers will be contacted and need to respond in 2 weeks. Based on the -outcome, Bazel core team might make a decision to demote the rule set. +Rule maintainers will be contacted and need to respond in 2 weeks. Based on the outcome, Bazel core team might make a decision to demote the rule set. diff --git a/community/remote-execution-services.mdx b/community/remote-execution-services.mdx index 6dee80fa..7a971881 100644 --- a/community/remote-execution-services.mdx +++ b/community/remote-execution-services.mdx @@ -2,28 +2,23 @@ title: 'Remote Execution Services' --- - - Use the following services to run Bazel with remote execution: -* Manual +- Manual - * Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) - directly to create your own remote execution service. + - Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) directly to create your own remote execution service. -* Self-service +- Self-service - * [Buildbarn](https://github.com/buildbarn) - * [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - * [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - * [NativeLink](https://github.com/TraceMachina/nativelink) + - [Buildbarn](https://github.com/buildbarn) + - [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + - [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + - [NativeLink](https://github.com/TraceMachina/nativelink) -* Commercial +- Commercial - * [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. - * [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. - * [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, - caching, and results UI. - * [EngFlow Remote Execution](https://www.engflow.com) - Remote execution - and remote caching service with Build and Test UI. Can be self-hosted or hosted. - * [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. + - [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. + - [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. + - [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, caching, and results UI. + - [EngFlow Remote Execution](https://www.engflow.com) - Remote execution and remote caching service with Build and Test UI. Can be self-hosted or hosted. + - [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. diff --git a/community/sig.mdx b/community/sig.mdx index ae5f9189..33ef6b15 100644 --- a/community/sig.mdx +++ b/community/sig.mdx @@ -2,42 +2,23 @@ title: 'Bazel Special Interest Groups' --- +Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular areas and to support communication and coordination between [Bazel owners, maintainers, and contributors](/contribute/policy). This policy applies to [`bazelbuild`](http://github.com/bazelbuild). +SIGs do their work in public. The ideal scope for a SIG covers a well-defined domain, where the majority of participation is from the community. SIGs may focus on community maintained repositories in `bazelbuild` (such as language rules) or focus on areas of code in the Bazel repository (such as Remote Execution). -Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular -areas and to support communication and coordination between [Bazel owners, -maintainers, and contributors](/contribute/policy). This policy -applies to [`bazelbuild`](http://github.com/bazelbuild). +While not all SIGs will have the same level of energy, breadth of scope, or governance models, there should be sufficient evidence that there are community members willing to engage and contribute should the interest group be established. Before joining, review the group's work, and then get in touch with the SIG leader. Membership policies vary on a per-SIG basis. -SIGs do their work in public. The ideal scope for a SIG covers a well-defined -domain, where the majority of participation is from the community. SIGs may -focus on community maintained repositories in `bazelbuild` (such as language -rules) or focus on areas of code in the Bazel repository (such as Remote -Execution). - -While not all SIGs will have the same level of energy, breadth of scope, or -governance models, there should be sufficient evidence that there are community -members willing to engage and contribute should the interest group be -established. Before joining, review the group's work, and then get in touch -with the SIG leader. Membership policies vary on a per-SIG basis. - -See the complete list of -[Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). +See the complete list of [Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). ### Non-goals: What a SIG is not -SIGs are intended to facilitate collaboration on shared work. A SIG is -therefore: +SIGs are intended to facilitate collaboration on shared work. A SIG is therefore: -- *Not a support forum:* a mailing list and a SIG is not the same thing -- *Not immediately required:* early on in a project's life, you may not know - if you have shared work or collaborators -- *Not free labor:* energy is required to grow and coordinate the work - collaboratively +- *Not a support forum:* a mailing list and a SIG is not the same thing +- *Not immediately required:* early on in a project's life, you may not know if you have shared work or collaborators +- *Not free labor:* energy is required to grow and coordinate the work collaboratively -Bazel Owners take a conservative approach to SIG creation—thanks to the ease of -starting projects on GitHub, there are many avenues where collaboration can -happen without the need for a SIG. +Bazel Owners take a conservative approach to SIG creation—thanks to the ease of starting projects on GitHub, there are many avenues where collaboration can happen without the need for a SIG. ## SIG lifecycle @@ -45,114 +26,65 @@ This section covers how to create a SIG. ### Research and consultation -To propose a new SIG group, first gather evidence for approval, as specified -below. Some possible avenues to consider are: +To propose a new SIG group, first gather evidence for approval, as specified below. Some possible avenues to consider are: -- A well-defined problem or set of problems the group would solve -- Consultation with community members who would benefit, assessing both the - benefit and their willingness to commit -- For existing projects, evidence from issues and PRs that contributors care - about the topic -- Potential goals for the group to achieve -- Resource requirements of running the group +- A well-defined problem or set of problems the group would solve +- Consultation with community members who would benefit, assessing both the benefit and their willingness to commit +- For existing projects, evidence from issues and PRs that contributors care about the topic +- Potential goals for the group to achieve +- Resource requirements of running the group -Even if the need for a SIG seems self-evident, the research and consultation is -still important to the success of the group. +Even if the need for a SIG seems self-evident, the research and consultation is still important to the success of the group. ### Create the new group -The new group should follow the below process for chartering. In particular, it -must demonstrate: - -- A clear purpose and benefit to Bazel (either around a sub-project or - application area) -- Two or more contributors willing to act as group leads, existence of other - contributors, and evidence of demand for the group -- Each group needs to use at least one publicly accessible mailing list. A SIG - may reuse one of the public lists, such as - [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list - for @bazel.build, or create their own list -- Resources the SIG initially requires (usually, mailing list and regular - video call.) -- SIGs can serve documents and files from their directory in - [`bazelbuild/community`](https://github.com/bazelbuild/community) - or from their own repository in the - [`bazelbuild`](https://github.com/bazelbuild) GitHub - organization. SIGs may link to external resources if they choose to organize - their work outside of the `bazelbuild` GitHub organization -- Bazel Owners approve or reject SIG applications and consult other - stakeholders as necessary - -Before entering the formal parts of the process, you should consult with -the Bazel product team, at product@bazel.build. Most SIGs require conversation -and iteration before approval. - -The formal request for the new group is done by submitting a charter as a PR to -[`bazelbuild/community`](https://github.com/bazelbuild/community), -and including the request in the comments on the PR following the template -below. On approval, the PR for the group is merged and the required resources -created. +The new group should follow the below process for chartering. In particular, it must demonstrate: + +- A clear purpose and benefit to Bazel (either around a sub-project or application area) +- Two or more contributors willing to act as group leads, existence of other contributors, and evidence of demand for the group +- Each group needs to use at least one publicly accessible mailing list. A SIG may reuse one of the public lists, such as [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list for @bazel.build, or create their own list +- Resources the SIG initially requires (usually, mailing list and regular video call.) +- SIGs can serve documents and files from their directory in [`bazelbuild/community`](https://github.com/bazelbuild/community) or from their own repository in the [`bazelbuild`](https://github.com/bazelbuild) GitHub organization. SIGs may link to external resources if they choose to organize their work outside of the `bazelbuild` GitHub organization +- Bazel Owners approve or reject SIG applications and consult other stakeholders as necessary + +Before entering the formal parts of the process, you should consult with the Bazel product team, at [product@bazel.build](mailto:product@bazel.build). Most SIGs require conversation and iteration before approval. + +The formal request for the new group is done by submitting a charter as a PR to [`bazelbuild/community`](https://github.com/bazelbuild/community), and including the request in the comments on the PR following the template below. On approval, the PR for the group is merged and the required resources created. ### Template Request for New SIG -To request a new SIG, use the template in the community repo: -[SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). +To request a new SIG, use the template in the community repo: [SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). ### Chartering -To establish a group, you need a charter and must follow the Bazel -[code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). -Archives of the group will be public. Membership may either be open to all -without approval, or available on request, pending approval of the group -administrator. +To establish a group, you need a charter and must follow the Bazel [code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). Archives of the group will be public. Membership may either be open to all without approval, or available on request, pending approval of the group administrator. -The charter must nominate an administrator. As well as an administrator, the -group must include at least one person as lead (these may be the same person), -who serves as point of contact for coordination as required with the Bazel -product team. +The charter must nominate an administrator. As well as an administrator, the group must include at least one person as lead (these may be the same person), who serves as point of contact for coordination as required with the Bazel product team. -Group creators must post their charter to the group mailing list. The community -repository in the Bazel GitHub organization archives such documents and -policies. As groups evolve their practices and conventions, they should update -their charters within the relevant part of the community repository. +Group creators must post their charter to the group mailing list. The community repository in the Bazel GitHub organization archives such documents and policies. As groups evolve their practices and conventions, they should update their charters within the relevant part of the community repository. ### Collaboration and inclusion -While not mandated, the group should choose to make use of collaboration -via scheduled conference calls or chat channels to conduct meetings. Any such -meetings should be advertised on the mailing list, and notes posted to the -mailing list afterwards. Regular meetings help drive accountability and progress -in a SIG. +While not mandated, the group should choose to make use of collaboration via scheduled conference calls or chat channels to conduct meetings. Any such meetings should be advertised on the mailing list, and notes posted to the mailing list afterwards. Regular meetings help drive accountability and progress in a SIG. -Bazel product team members may proactively monitor and encourage the group to -discussion and action as appropriate. +Bazel product team members may proactively monitor and encourage the group to discussion and action as appropriate. ### Launch a SIG Required activities: -- Notify Bazel general discussion groups - ([bazel-discuss](https://groups.google.com/g/bazel-discuss), - [bazel-dev](https://groups.google.com/g/bazel-dev)). +- Notify Bazel general discussion groups ([bazel-discuss](https://groups.google.com/g/bazel-discuss), [bazel-dev](https://groups.google.com/g/bazel-dev)). Optional activities: -- Create a blog post for the Bazel blog +- Create a blog post for the Bazel blog ### Health and termination of SIGs -The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners -occasionally request the SIG lead to report on the SIG's work, to inform the -broader Bazel community of the group's activity. +The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners occasionally request the SIG lead to report on the SIG's work, to inform the broader Bazel community of the group's activity. -If a SIG no longer has a useful purpose or interested community, it may be -archived and cease operation. The Bazel product team reserves the right to -archive such inactive SIGs to maintain the overall health of the project, -though it is a less preferable outcome. A SIG may also opt to disband if -it recognizes it has reached the end of its useful life. +If a SIG no longer has a useful purpose or interested community, it may be archived and cease operation. The Bazel product team reserves the right to archive such inactive SIGs to maintain the overall health of the project, though it is a less preferable outcome. A SIG may also opt to disband if it recognizes it has reached the end of its useful life. ## Note -*This content has been adopted from Tensorflow’s -[SIG playbook](https://www.tensorflow.org/community/sig_playbook) -with modifications.* +*This content has been adopted from Tensorflow’s [SIG playbook](https://www.tensorflow.org/community/sig_playbook) with modifications.* diff --git a/community/users.mdx b/community/users.mdx index 91e26c47..b7cf01fc 100644 --- a/community/users.mdx +++ b/community/users.mdx @@ -2,276 +2,189 @@ title: 'Who''s Using Bazel' --- +Note: Using Bazel? You can add your company on [StackShare](https://stackshare.io/bazel). To add yourself to this page, contact [product@bazel.build](mailto:product@bazel.build). - -Note: Using Bazel? You can add your company on -[StackShare](https://stackshare.io/bazel). To add yourself to this page, -contact [product@bazel.build](mailto:product@bazel.build). - -This page lists companies and OSS projects that are known to use Bazel. -This does not constitute an endorsement. +This page lists companies and OSS projects that are known to use Bazel. This does not constitute an endorsement. ## Companies using Bazel ### [acqio](https://acqio.com.br) - +![](/community/images/acqio_logo.svg) -Acqio is a Fintech that provides payment products and services for small and -medium merchants. Acqio has a handful of monorepos and uses Bazel along with -Kubernetes to deliver fast and reliable microservices. +Acqio is a Fintech that provides payment products and services for small and medium merchants. Acqio has a handful of monorepos and uses Bazel along with Kubernetes to deliver fast and reliable microservices. ### [Adobe](https://www.adobe.com/) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Adobe_logo_and_wordmark_%282017%29.svg/440px-Adobe_logo_and_wordmark_%282017%29.svg.png) -Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for -continuous, GitOps driven Kubernetes deployments. +Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for continuous, GitOps driven Kubernetes deployments. ### [Asana](https://asana.com) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Asana_logo.svg/256px-Asana_logo.svg.png) -Asana is a web and mobile application designed to help teams track their work. -In their own words: +Asana is a web and mobile application designed to help teams track their work. In their own words: -> Bazel has increased reliability, stability, and speed for all of builds/tests -at Asana. We no longer need to clean because of incorrect caches. +> Bazel has increased reliability, stability, and speed for all of builds/tests at Asana. We no longer need to clean because of incorrect caches. ### [Ascend.io](https://ascend.io) -Ascend is a Palo Alto startup that offers solutions for large data sets -analysis. Their motto is _Big data is hard. We make it easy_. +Ascend is a Palo Alto startup that offers solutions for large data sets analysis. Their motto is *Big data is hard. We make it easy*. ### [ASML](https://asml.com) - +![](https://upload.wikimedia.org/wikipedia/en/6/6c/ASML_Holding_N.V._logo.svg) -ASML is an innovation leader in the semiconductor industry. We provide chipmakers -with everything they need – hardware, software and services – to mass produce -patterns on silicon through lithography. +ASML is an innovation leader in the semiconductor industry. We provide chipmakers with everything they need – hardware, software and services – to mass produce patterns on silicon through lithography. ### [Beeswax](https://www.beeswax.com/) -> Beeswax is a New York based startup that provides real time bidding as -service. Bazel powers their Jenkins based continuous integration and deployment -framework. Beeswax loves Bazel because it is blazingly fast, correct and well -supported across many languages and platforms. +> Beeswax is a New York based startup that provides real time bidding as service. Bazel powers their Jenkins based continuous integration and deployment framework. Beeswax loves Bazel because it is blazingly fast, correct and well supported across many languages and platforms. ### [Braintree](https://www.braintreepayments.com) - +![](https://upload.wikimedia.org/wikipedia/commons/0/00/Braintree-logo1.png) -Braintree, a PayPal subsidiary, develops payment solutions for websites and -applications. They use Bazel for parts of their internal build and Paul Gross -even posted a -[nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). +Braintree, a PayPal subsidiary, develops payment solutions for websites and applications. They use Bazel for parts of their internal build and Paul Gross even posted a [nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). ### [Canva](https://www.canva.com/) - -Canva leverages Bazel to manage its large polyglot codebase, which includes -Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered -significant developer and compute infrastructure efficiencies, for example 5-6x -decreases in average CI build times, and it continues to become the foundation -of fast, reproducible, and standardised software builds at the company. +![](https://upload.wikimedia.org/wikipedia/commons/b/bb/Canva_Logo.svg) + +Canva leverages Bazel to manage its large polyglot codebase, which includes Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered significant developer and compute infrastructure efficiencies, for example 5-6x decreases in average CI build times, and it continues to become the foundation of fast, reproducible, and standardised software builds at the company. ### [CarGurus](https://www.cargurus.com) - -CarGurus is on a mission to build the world's most trusted and transparent -automotive marketplace and uses Bazel to build their polyglot monorepo. +![](https://www.cargurus.com/gfx/reskin/logos/logo_CarGurus.svg) + +CarGurus is on a mission to build the world's most trusted and transparent automotive marketplace and uses Bazel to build their polyglot monorepo. ### [Compass](https://www.compass.com) -Compass is a tech-driven real estate platform. With an elite team of real -estate, technology and business professionals, we aim to be the best and most -trusted source for home seekers. +Compass is a tech-driven real estate platform. With an elite team of real estate, technology and business professionals, we aim to be the best and most trusted source for home seekers. ### [Databricks](https://databricks.com) - -Databricks provides cloud-based integrated workspaces based on Apache Spark™. +![](https://databricks.com/wp-content/uploads/2021/10/db-nav-logo.svg) Databricks provides cloud-based integrated workspaces based on Apache Spark™. -> The Databricks codebase is a Monorepo, containing the Scala code that powers -most of our services, Javascript for front-end UI, Python for scripting, -Jsonnet to configure our infrastructure, and much more [...] Even though our -monorepo contains a million lines of Scala, working with code within is fast -and snappy. -([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) +> The Databricks codebase is a Monorepo, containing the Scala code that powers most of our services, Javascript for front-end UI, Python for scripting, Jsonnet to configure our infrastructure, and much more \[...] Even though our monorepo contains a million lines of Scala, working with code within is fast and snappy. ([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) ### [Dataform](https://dataform.co) -Dataform provides scalable analytics for data teams. They maintain a handful of -NPM packages and a documentation site in one single monorepo and they do it all -with Bazel. +Dataform provides scalable analytics for data teams. They maintain a handful of NPM packages and a documentation site in one single monorepo and they do it all with Bazel. -After the migration to Bazel, they -[reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), -including: +After the migration to Bazel, they [reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), including: -> * Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). -> * Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes -> * Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop +> - Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). +> - Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes +> - Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop ### [Deep Silver FISHLABS](https://www.dsfishlabs.com) -Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with -C++/Python/Go/C as a base for their internal build tooling and especially for -baking and deploying all their 3D Assets. + +Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with C++/Python/Go/C as a base for their internal build tooling and especially for baking and deploying all their 3D Assets. ### [Dropbox](https://www.dropbox.com/) - -At Dropbox, Bazel is a key component to our distributed build and test -environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable -production releases. + +![](/community/images/dropbox.png) At Dropbox, Bazel is a key component to our distributed build and test environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable production releases. ### [Engel & Völkers](https://www.engelvoelkers.com) -Engel & Völkers AG is a privately owned German company that, via a series of -franchised offices, provides services related to real estate transactions. +Engel & Völkers AG is a privately owned German company that, via a series of franchised offices, provides services related to real estate transactions. -> One of our internal project has seen a decrease of compilation time from 11 -minutes to roughly 1 minute, this was an impressive achievement and we are -currently working on bringing Bazel to more projects. -([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) +> One of our internal project has seen a decrease of compilation time from 11 minutes to roughly 1 minute, this was an impressive achievement and we are currently working on bringing Bazel to more projects. ([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) ### [Etsy](https://www.etsy.com/) - -Etsy is an e-commerce website focused on handmade or vintage items and supplies, -as well as unique factory-manufactured items. +![](https://upload.wikimedia.org/wikipedia/commons/a/aa/Etsy_logo_lg_rgb.png) -They use Bazel to build and test its Java-based search platform. Bazel produces -both packages for bare metal servers and repeatable Docker images. +Etsy is an e-commerce website focused on handmade or vintage items and supplies, as well as unique factory-manufactured items. + +They use Bazel to build and test its Java-based search platform. Bazel produces both packages for bare metal servers and repeatable Docker images. ### [Evertz.io](https://www.evertz.io/) -Evertz.io is a multi-tenant, serverless SaaS platform for offering cost -effective, multi-regional services worldwide to the Broadcast Media Industry, -created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). +Evertz.io is a multi-tenant, serverless SaaS platform for offering cost effective, multi-regional services worldwide to the Broadcast Media Industry, created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). -The website is fully built and deployed with an Angular and Bazel workflow -([source](https://twitter.com/MattMackay/status/1113947685508341762)). +The website is fully built and deployed with an Angular and Bazel workflow ([source](https://twitter.com/MattMackay/status/1113947685508341762)). ### [FINDMINE](http://www.findmine.com) - -FINDMINE is a automation technology for the retail industry that uses machine -learning to scale the currently manual and tedious process of product curation. -We use Bazel to mechanize our entire python package building, testing, and -deployment process. +![](https://www.findmine.com/static/assets/landpage/findmine-color-logo.png) + +FINDMINE is a automation technology for the retail industry that uses machine learning to scale the currently manual and tedious process of product curation. We use Bazel to mechanize our entire python package building, testing, and deployment process. ### [Flexport](https://www.flexport.com/) -Flexport is a tech-enabled global freight forwarder; our mission is to make -global trade easier for everyone. At Flexport, we use Bazel to build/test our -Java/JavaScript services and client libraries and to generate Java and Ruby -code from protobuf definitions. -[Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) +Flexport is a tech-enabled global freight forwarder; our mission is to make global trade easier for everyone. At Flexport, we use Bazel to build/test our Java/JavaScript services and client libraries and to generate Java and Ruby code from protobuf definitions. [Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) ### [Foursquare](https://foursquare.com) - -Foursquare's mission is to create technology that constructs meaningful -bridges between digital spaces and physical places. We manage millions of -lines of primarily Scala and Python code powering data-intensive -applications, including complex codegen and container build processes, with -Bazel. +![](https://upload.wikimedia.org/wikipedia/commons/9/99/FSQ_logo.png) + +Foursquare's mission is to create technology that constructs meaningful bridges between digital spaces and physical places. We manage millions of lines of primarily Scala and Python code powering data-intensive applications, including complex codegen and container build processes, with Bazel. ### [GermanTechJobs](https://germantechjobs.de) - -Bazel has simplified our workflows 10-fold and enabled shipping features at -scale. +![](https://upload.wikimedia.org/wikipedia/commons/9/98/GermanTechJobs_Logo.png) + +Bazel has simplified our workflows 10-fold and enabled shipping features at scale. ### [Google](https://google.com) - -Bazel was designed to be able to scale to Google's needs and meet Google's -requirements of reproducibility and platform/language support. All software at -Google is built using Bazel. Google uses Bazel and its rules for millions of -builds every day. +![](https://upload.wikimedia.org/wikipedia/commons/2/2f/Google_2015_logo.svg) + +Bazel was designed to be able to scale to Google's needs and meet Google's requirements of reproducibility and platform/language support. All software at Google is built using Bazel. Google uses Bazel and its rules for millions of builds every day. ### [Huawei](http://www.huawei.com/) -> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go -projects, except for Go projects, others originally were built by Maven. We -write a simple tool to translate a Maven-built project into Bazel-built one. -More and more projects will use Bazel in recent future. +> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go projects, except for Go projects, others originally were built by Maven. We write a simple tool to translate a Maven-built project into Bazel-built one. More and more projects will use Bazel in recent future. ### [IMC Trading](https://imc.com) - -> IMC is a global proprietary trading firm and market maker headquarted in -Amsterdam. We are using Bazel to continuously build and test our -Java/C++/Python/SystemVerilog projects. +![](https://upload.wikimedia.org/wikipedia/commons/1/17/IMC_Logo.svg) + +> IMC is a global proprietary trading firm and market maker headquarted in Amsterdam. We are using Bazel to continuously build and test our Java/C++/Python/SystemVerilog projects. ### [Improbable.io](https://improbable.io/) -Improbable.io develops SpatialOS, a distributed operating system that enables -creating huge simulations inhabited by millions of complex entities. +Improbable.io develops SpatialOS, a distributed operating system that enables creating huge simulations inhabited by millions of complex entities. ### [Interaxon](https://www.choosemuse.com/) -InteraXon is a thought-controlled computing firm that creates hardware and -software platforms to convert brainwaves into digital signals. +InteraXon is a thought-controlled computing firm that creates hardware and software platforms to convert brainwaves into digital signals. ### [Jupiter](https://jupiter.co/) -Jupiter is a company that provides delivery of groceries and household -essentials every week. +Jupiter is a company that provides delivery of groceries and household essentials every week. -They use Bazel in their backend code, specifically to compile protos and Kotlin -to JVM binaries, using remote caching. -([source](https://starship.jupiter.co/jupiter-stack/)) +They use Bazel in their backend code, specifically to compile protos and Kotlin to JVM binaries, using remote caching. ([source](https://starship.jupiter.co/jupiter-stack/)) ### [Just](https://gojust.com/) -Just is an enterprise financial technology company, headquartered in Norway, -creating software solutions to transform how global corporate treasurers manage -risk and liquidity. Their entire application stack is built with Bazel. +Just is an enterprise financial technology company, headquartered in Norway, creating software solutions to transform how global corporate treasurers manage risk and liquidity. Their entire application stack is built with Bazel. ### [Line](https://line.me/) -Line provides an app for instant communications, which is the most popular -messaging application in Japan. -They use Bazel on their codebase consisting of about 60% Swift and 40% -C/C++/Objective-C/Objective-C++ -([source](https://twitter.com/thi_dt/status/1253334262020886532)). +Line provides an app for instant communications, which is the most popular messaging application in Japan. They use Bazel on their codebase consisting of about 60% Swift and 40% C/C++/Objective-C/Objective-C++ ([source](https://twitter.com/thi_dt/status/1253334262020886532)). -> After switching to Bazel, we were able to achieve a huge improvement in the -build times. This brought a significant improvement in the turn-around time -during a QA period. Distributing a new build to our testers no longer means -another hour waiting for building and testing. -([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) +> After switching to Bazel, we were able to achieve a huge improvement in the build times. This brought a significant improvement in the turn-around time during a QA period. Distributing a new build to our testers no longer means another hour waiting for building and testing. ([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) ### [LingoChamp](https://www.liulishuo.com/en) - -LingoChamp provides professional solutions to English learners. We use Bazel -for our go, java and python projects. +![](/community/images/liulishuo.png) LingoChamp provides professional solutions to English learners. We use Bazel for our go, java and python projects. ### [LinkedIn](https://linkedin.com/) - -LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social -network. LinkedIn uses Bazel for building its iOS Apps. +![](/community/images/Linkedin-Logo.png) LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social network. LinkedIn uses Bazel for building its iOS Apps. ### [Lucid Software](https://lucid.co/) - +![](/community/images/Lucid_Software-logo.svg) -Lucid Software is a leader in visual collaboration, helping teams see and build the -future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), -[Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams -can align around a shared vision, clarify complexity, and collaborate visually, no -matter where they’re located. +Lucid Software is a leader in visual collaboration, helping teams see and build the future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), [Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams can align around a shared vision, clarify complexity, and collaborate visually, no matter where they’re located. -Lucid uses Bazel to build millions of lines of Scala and TypeScript. -Migrating to Bazel has tremendously sped up its builds, reduced external -dependencies on the build environment, and simplified developers' experience -with the build system. Bazel has improved developer productivity at Lucid and -unlocked further growth. +Lucid uses Bazel to build millions of lines of Scala and TypeScript. Migrating to Bazel has tremendously sped up its builds, reduced external dependencies on the build environment, and simplified developers' experience with the build system. Bazel has improved developer productivity at Lucid and unlocked further growth. ### [Lyft](https://www.lyft.com/) @@ -279,131 +192,81 @@ Lyft is using Bazel for their iOS ([source](https://twitter.com/SmileyKeith/stat ### [Meetup](http://www.meetup.com/) -Meetup is an online social networking portal that facilitates offline group -meetings. -The Meetup engineering team contributes to -[rules_scala](https://github.com/bazelbuild/rules_scala) and is the -maintainer of [rules_avro](https://github.com/meetup/rules_avro) -and [rules_openapi](https://github.com/meetup/rules_openapi). - +Meetup is an online social networking portal that facilitates offline group meetings. The Meetup engineering team contributes to [rules\_scala](https://github.com/bazelbuild/rules_scala) and is the maintainer of [rules\_avro](https://github.com/meetup/rules_avro) and [rules\_openapi](https://github.com/meetup/rules_openapi). ### [Nvidia](https://www.nvidia.com/) -> At Nvidia we have been using dazel(docker bazel) for python to work around -some of bazel's python short comings. Everything else runs in normal bazel -(Mostly Go / Scala/ C++/ Cuda) -([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) - +> At Nvidia we have been using dazel(docker bazel) for python to work around some of bazel's python short comings. Everything else runs in normal bazel (Mostly Go / Scala/ C++/ Cuda) ([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) ### [Peloton Technology](http://www.peloton-tech.com) -Peloton Technology is an automated vehicle technology company that tackles truck -accidents and fuel use. They use Bazel to _enable reliable builds for automotive -safety systems_. +Peloton Technology is an automated vehicle technology company that tackles truck accidents and fuel use. They use Bazel to *enable reliable builds for automotive safety systems*. ### [Pigweed](https://pigweed.dev) - +![](https://pigweed.dev/_static/pw_logo.svg) -Pigweed is an open-source solution for sustained, robust, and rapid embedded -product development for large teams. Pigweed has shipped in millions of -devices, including Google's suite of Pixel devices, Nest thermostats, -[satellites](https://www.spinlaunch.com/), and [autonomous aerial -drones](https://www.flyzipline.com/). +Pigweed is an open-source solution for sustained, robust, and rapid embedded product development for large teams. Pigweed has shipped in millions of devices, including Google's suite of Pixel devices, Nest thermostats, [satellites](https://www.spinlaunch.com/), and [autonomous aerial drones](https://www.flyzipline.com/). -Pigweed [uses Bazel as its primary build -system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for -Embedded][pw-bazel-great] blog post discusses why we think it's a great build -system for embedded projects! - -[pw-bazel-great]: https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded +Pigweed [uses Bazel as its primary build system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for Embedded](https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded) blog post discusses why we think it's a great build system for embedded projects! ### [Pinterest](https://www.pinterest.com/) - +![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Pinterest_Logo.svg/200px-Pinterest_Logo.svg.png) -Pinterest is the world’s catalog of ideas. They use Bazel to build various -backend services (Java/C++) and the iOS application (Objective-C/C++). +Pinterest is the world’s catalog of ideas. They use Bazel to build various backend services (Java/C++) and the iOS application (Objective-C/C++). -> We identified Bazel was the best fit for our goals to build a foundation for -an order of magnitude improvement in performance, eliminate variability in -build environments and adopt incrementally. As a result, we’re now shipping all -our iOS releases using Bazel. -[Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) +> We identified Bazel was the best fit for our goals to build a foundation for an order of magnitude improvement in performance, eliminate variability in build environments and adopt incrementally. As a result, we’re now shipping all our iOS releases using Bazel. [Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) ### [PubRef](https://github.com/pubref) -PubRef is an emerging scientific publishing platform. They use Bazel with -[rules_closure](https://github.com/bazelbuild/rules_closure) to build the -frontend, native java rules to build the main backend, -[rules_go](https://github.com/bazelbuild/rules_go), -[rules_node](https://github.com/pubref/rules_node), and -[rules_kotlin](https://github.com/pubref/rules_kotlin) to build assorted -backend services. [rules_protobuf](https://github.com/pubref/rules_protobuf) is -used to assist with gRPC-based communication between backend services. -PubRef.org is based in Boulder, CO. +PubRef is an emerging scientific publishing platform. They use Bazel with [rules\_closure](https://github.com/bazelbuild/rules_closure) to build the frontend, native java rules to build the main backend, [rules\_go](https://github.com/bazelbuild/rules_go), [rules\_node](https://github.com/pubref/rules_node), and [rules\_kotlin](https://github.com/pubref/rules_kotlin) to build assorted backend services. [rules\_protobuf](https://github.com/pubref/rules_protobuf) is used to assist with gRPC-based communication between backend services. PubRef.org is based in Boulder, CO. ### [Redfin](https://redfin.com/) -Redfin is a next-generation real estate brokerage with full-service local -agents. They use Bazel to build and deploy the website and various backend -services. - -> With the conversion mostly behind us, things are greatly improved! Our CI -builds are faster (*way* faster: they used to take 40–90 minutes, and now dev -builds average 5–6 minutes). Reliability is far higher, too. This is harder to -quantify, but the shift from unexplained build failures being something that -“just happens” to being viewed as real problems to be solved has put us on a -virtuous cycle of ever-increasing reliability. -([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) + +Redfin is a next-generation real estate brokerage with full-service local agents. They use Bazel to build and deploy the website and various backend services. + +> With the conversion mostly behind us, things are greatly improved! Our CI builds are faster (*way* faster: they used to take 40–90 minutes, and now dev builds average 5–6 minutes). Reliability is far higher, too. This is harder to quantify, but the shift from unexplained build failures being something that “just happens” to being viewed as real problems to be solved has put us on a virtuous cycle of ever-increasing reliability. ([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) ### [Ritual](https://ritual.co) - -Ritual is a mobile pick up app, connecting restaurants with customers to offer -a simple, time-saving tool to get the food and beverages they want, without the -wait. Ritual uses Bazel for their backend services. +![](https://lh3.googleusercontent.com/7Ir6j25ROnsXhtQXveOzup33cizxLf-TiifSC1cI6op0bQVB-WePmPjJOfXUBQ0L3KpkheObAiS28e-TS8hZtDzxOIc) + +Ritual is a mobile pick up app, connecting restaurants with customers to offer a simple, time-saving tool to get the food and beverages they want, without the wait. Ritual uses Bazel for their backend services. ### [Snap](https://www.snap.com/en-US/) -Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel -in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more -details about their process, see their [engineering blog](https://eng.snap.com/blog/). +Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more details about their process, see their [engineering blog](https://eng.snap.com/blog/). ### [Stripe](https://stripe.com) - + +![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Stripe_Logo%2C_revised_2016.svg/320px-Stripe_Logo%2C_revised_2016.svg.png) Stripe provides mobile payment solutions. They use Bazel in their build and test pipelines, as detailed in their [engineering blog](https://stripe.com/blog/fast-secure-builds-choose-two). ### [Tinder](https://tinder.com) - -Tinder migrated its iOS app from CocoaPods to Bazel -in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). +![](https://policies.tinder.com/static/b0327365f4c0a31c4337157c10e9fadf/c1b63/tinder_full_color_watermark.png) + +Tinder migrated its iOS app from CocoaPods to Bazel in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). ### [Tink](https://tink.com/) - -Tink is a european fintech, building the best way to connect to banks across -Europe. +![](https://cdn.tink.se/tink-logos/LOW/Tink_Black.png) -They are using Bazel to build their backend services from a polyglot monorepo. -Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) -meetup group. +Tink is a european fintech, building the best way to connect to banks across Europe. + +They are using Bazel to build their backend services from a polyglot monorepo. Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) meetup group. ### [Tokopedia](https://www.tokopedia.com/) -Tokopedia is an Indonesian technology company specializing in e-commerce, with -over 90 million monthly active users and over 7 million merchants on the -platform. +Tokopedia is an Indonesian technology company specializing in e-commerce, with over 90 million monthly active users and over 7 million merchants on the platform. -They wrote the article -[How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), -where they explain how Bazel sped up their builds. The build duration went from -55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote -caching. +They wrote the article [How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), where they explain how Bazel sped up their builds. The build duration went from 55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote caching. ### [Trunk.io](https://trunk.io/merge/trunk-merge-and-bazel) - + +![](/community/images/trunk-logo-dark.svg) Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initialized Capital. Trunk offers a powerful pull request merge service with first-class support for the Bazel build system. By leveraging Bazel's understanding of dependencies within a codebase, Trunk's merge service intelligently creates parallel merge lanes, allowing independent changes to be tested and merged simultaneously. @@ -411,71 +274,49 @@ Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initial ### [Twitter](https://twitter.com/) -Twitter has made the decision to migrate from Pants to Bazel as their primary -build tool -([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). +Twitter has made the decision to migrate from Pants to Bazel as their primary build tool ([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). ### [Two Sigma](https://www.twosigma.com/) - -Two Sigma is a New York-headquartered technology company dedicated to finding -value in the world’s data. +![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Two_Sigma_logo.svg/2880px-Two_Sigma_logo.svg.png) + +Two Sigma is a New York-headquartered technology company dedicated to finding value in the world’s data. ### [TypeDB](https://typedb.com) -TypeDB Logo -TypeDB is a database technology that can be used to intuitively model -interconnected data. Through its type-theoretic and polymorphic query language, -TypeQL, the data can be accessed with simple, human-readable queries that run at -lightspeed. +![TypeDB Logo](/community/images/typedb.png) + +TypeDB is a database technology that can be used to intuitively model interconnected data. Through its type-theoretic and polymorphic query language, TypeQL, the data can be accessed with simple, human-readable queries that run at lightspeed. -Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution -pipeline that manages many repositories in a wide variety of languages, and -deploys to numerous platforms seamlessly. The TypeDB team has also released -Bazel rules for assembling and deploying software distributions. +Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution pipeline that manages many repositories in a wide variety of languages, and deploys to numerous platforms seamlessly. The TypeDB team has also released Bazel rules for assembling and deploying software distributions. ### [Uber](https://www.uber.com) -Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo -is likely one of the largest Go repositories using Bazel. See the article -[Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) -to learn more about their experience. +Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo is likely one of the largest Go repositories using Bazel. See the article [Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) to learn more about their experience. ### [Uber Advanced Technologies Group](https://www.uber.com/info/atg/) - -Uber Advanced Technologies Group is focused on autonomous vehicle efforts at -Uber, including trucking/freight and autonomous ride sharing. The organization -uses Bazel as its primary build system. +![](https://upload.wikimedia.org/wikipedia/commons/thumb/6/62/Uber_logo.svg/220px-Uber_logo.svg.png) + +Uber Advanced Technologies Group is focused on autonomous vehicle efforts at Uber, including trucking/freight and autonomous ride sharing. The organization uses Bazel as its primary build system. ### [Vistar Media](http://vistarmedia.com) -Vistar Media is an advertising platform that enables brands to reach consumers -based on their behavior in the physical world. Their engineering team is -primarily based out of Philadelphia and is using Bazel for builds, deploys, to -speed up testing, and to consolidate repositories written with a variety of -different technologies. + +Vistar Media is an advertising platform that enables brands to reach consumers based on their behavior in the physical world. Their engineering team is primarily based out of Philadelphia and is using Bazel for builds, deploys, to speed up testing, and to consolidate repositories written with a variety of different technologies. ### [VMware](https://www.vmware.com/) -VMware uses Bazel to produce deterministic, reliable builds while developing -innovative products for their customers. + +VMware uses Bazel to produce deterministic, reliable builds while developing innovative products for their customers. ### [Wix](https://www.wix.com/) -Wix is a cloud-based web development platform. Their backend uses Java and Scala -code. They use remote execution with Google Cloud Build. +Wix is a cloud-based web development platform. Their backend uses Java and Scala code. They use remote execution with Google Cloud Build. -> We have seen about 5 times faster clean builds when running with bazel remote -execution which utilizes bazel’s great build/test parallelism capabilities when -it dispatches build/test actions to a worker farm. Average build times are more -than 10 times faster due to the utilization of bazel’s aggressive caching -mechanism. -([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) +> We have seen about 5 times faster clean builds when running with bazel remote execution which utilizes bazel’s great build/test parallelism capabilities when it dispatches build/test actions to a worker farm. Average build times are more than 10 times faster due to the utilization of bazel’s aggressive caching mechanism. ([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) ### [Zenly](https://zen.ly/) -Zenly is a live map of your friends and family. It’s the most fun way to meet up -— or just see what’s up! — so you can feel together, even when you're apart. - +Zenly is a live map of your friends and family. It’s the most fun way to meet up — or just see what’s up! — so you can feel together, even when you're apart. *** @@ -483,44 +324,33 @@ Zenly is a live map of your friends and family. It’s the most fun way to meet ### [Abseil](https://abseil.io/) -Abseil is an open-source collection of C++ code (compliant to C++11) designed -to augment the C++ standard library. +Abseil is an open-source collection of C++ code (compliant to C++11) designed to augment the C++ standard library. ### [Angular](https://angular.io) - +![](https://upload.wikimedia.org/wikipedia/commons/c/cf/Angular_full_color_logo.svg) -Angular is a popular web framework. -Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). +Angular is a popular web framework. Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). ### [Apollo](https://github.com/ApolloAuto/apollo) -Apollo is a high performance, flexible architecture which accelerates the -development, testing, and deployment of Autonomous Vehicles. +Apollo is a high performance, flexible architecture which accelerates the development, testing, and deployment of Autonomous Vehicles. ### [brpc](https://github.com/brpc/brpc) -An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ -instances(not counting clients) and thousands kinds of services, called -"baidu-rpc" inside Baidu. +An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ instances(not counting clients) and thousands kinds of services, called "baidu-rpc" inside Baidu. ### [cert-manager](https://github.com/jetstack/cert-manager) -cert-manager is a Kubernetes add-on to automate the management and issuance of -TLS certificates from various issuing sources. It will ensure certificates are -valid and up to date periodically, and attempt to renew certificates at an -appropriate time before expiry. +cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry. ### [CallBuilder](https://github.com/google/CallBuilder) -A Java code generator that allows you to create a builder by writing one -function. +A Java code generator that allows you to create a builder by writing one function. ### [CPPItertools](https://github.com/ryanhaining/cppitertools) -C++ library providing range-based for loop add-ons inspired by the Python -builtins and itertools library. Like itertools and the Python3 builtins, this -library uses lazy evaluation wherever possible. +C++ library providing range-based for loop add-ons inspired by the Python builtins and itertools library. Like itertools and the Python3 builtins, this library uses lazy evaluation wherever possible. ### [Copybara](https://github.com/google/copybara) @@ -528,13 +358,11 @@ Copybara is a tool for transforming and moving code between repositories. ### [Dagger](https://google.github.io/dagger/) -Dagger is a fully static, compile-time dependency injection framework for both -Java and Android. +Dagger is a fully static, compile-time dependency injection framework for both Java and Android. ### [DAML](https://github.com/digital-asset/daml) -DAML is a smart contract language for building future-proof distributed -applications on a safe, privacy-aware runtime. +DAML is a smart contract language for building future-proof distributed applications on a safe, privacy-aware runtime. ### [DeepMind Lab](https://github.com/deepmind/lab) @@ -542,10 +370,7 @@ A customisable 3D platform for agent-based AI research. ### [Drake](https://github.com/RobotLocomotion/drake) -Drake is a C++ toolbox started at MIT and now led by the Toyota Research -Institute. It is a collection of tools for analyzing the dynamics of our robots -and building control systems for them, with a heavy emphasis on -optimization-based design/analysis. +Drake is a C++ toolbox started at MIT and now led by the Toyota Research Institute. It is a collection of tools for analyzing the dynamics of our robots and building control systems for them, with a heavy emphasis on optimization-based design/analysis. ### [Envoy](https://github.com/lyft/envoy) @@ -553,19 +378,15 @@ C++ L7 proxy and communication bus ### [Error Prone](https://github.com/google/error-prone) -Catches common Java mistakes as compile-time errors. (Migration to Bazel is in -progress.) +Catches common Java mistakes as compile-time errors. (Migration to Bazel is in progress.) ### [Extensible Service Proxy](https://github.com/cloudendpoints/esp) -Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management -capabilities for JSON/REST or gRPC API services. The current implementation is -based on an NGINX HTTP reverse proxy server. +Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management capabilities for JSON/REST or gRPC API services. The current implementation is based on an NGINX HTTP reverse proxy server. ### [FFruit](https://gitlab.com/perezd/ffruit/) -FFruit is a free & open source Android application to the popular service -[Falling Fruit](https://fallingfruit.org). +FFruit is a free & open source Android application to the popular service [Falling Fruit](https://fallingfruit.org). ### [Gerrit Code Review](https://gerritcodereview.com) @@ -577,61 +398,51 @@ Gitiles is a simple repository browser for Git repositories, built on JGit. ### [Grakn](https://github.com/graknlabs/grakn) -Grakn (https://grakn.ai/) is the knowledge graph engine to organise complex -networks of data and make it queryable. +Grakn ([https://grakn.ai/](https://grakn.ai/)) is the knowledge graph engine to organise complex networks of data and make it queryable. ### [GRPC](http://www.grpc.io) -A language-and-platform-neutral remote procedure call system. -(Bazel is a supported, although not primary, build system.) + +A language-and-platform-neutral remote procedure call system. (Bazel is a supported, although not primary, build system.) ### [gVisor](https://github.com/google/gvisor) + gVisor is a container runtime sandbox. ### [Guetzli](https://github.com/google/guetzli/) -Guetzli is a JPEG encoder that aims for excellent compression density at high -visual quality. +Guetzli is a JPEG encoder that aims for excellent compression density at high visual quality. ### [Gulava](http://www.github.com/google/gulava/) -A Java code generator that lets you write Prolog-style predicates and use them -seamlessly from normal Java code. +A Java code generator that lets you write Prolog-style predicates and use them seamlessly from normal Java code. ### [Heron](https://github.com/apache/incubator-heron) -Heron is a realtime, distributed, fault-tolerant stream processing engine from -Twitter. +Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter. ### [Internet Computer Protocol](https://internetcomputer.org/) - +![](https://internetcomputer.org/img/IC_logo_horizontal_white.svg) -The Internet Computer Protocol is a publicly available blockchain network that -enables replicated execution of general-purpose computation, serving hundreds -of thousands of applications and their users. +The Internet Computer Protocol is a publicly available blockchain network that enables replicated execution of general-purpose computation, serving hundreds of thousands of applications and their users. ### [Jazzer](https://github.com/CodeIntelligenceTesting/jazzer) - +![](https://www.code-intelligence.com/hubfs/Logos/CI%20Logos/Jazzer_einfach.png) Jazzer is a fuzzer for Java and other JVM-based languages that integrates with JUnit 5. ### [JGit](https://eclipse.org/jgit/) -JGit is a lightweight, pure Java library implementing the Git version control -system. +JGit is a lightweight, pure Java library implementing the Git version control system. ### [Jsonnet](https://jsonnet.org/) -An elegant, formally-specified config generation language for JSON. -(Bazel is a supported build system.) +An elegant, formally-specified config generation language for JSON. (Bazel is a supported build system.) ### [Kubernetes](https://github.com/kubernetes/kubernetes) - -Kubernetes is an open source system for managing containerized applications -across multiple hosts, providing basic mechanisms for deployment, maintenance, -and scaling of applications. +![](https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.png) Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. ### [Kythe](https://github.com/google/kythe) @@ -639,10 +450,9 @@ An ecosystem for building tools that work with code. ### [ls-lint](https://github.com/loeffel-io/ls-lint) - +![](https://raw.githubusercontent.com/loeffel-io/ls-lint/master/assets/logo/ls-lint.png) -An extremely fast directory and filename linter - Bring some structure to your -project file system. +An extremely fast directory and filename linter - Bring some structure to your project file system. ### [Nomulus](https://github.com/google/nomulus) @@ -650,19 +460,11 @@ Top-level domain name registry service on Google App Engine. ### [ONOS : Open Network Operating System](https://github.com/opennetworkinglab/onos) - -ONOS is the only SDN controller platform that supports the transition from -legacy “brown field” networks to SDN “green field” networks. This enables -exciting new capabilities, and disruptive deployment and operational cost points -for network operators. +![](https://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Logo_for_the_ONOS_open_source_project.png/175px-Logo_for_the_ONOS_open_source_project.png) ONOS is the only SDN controller platform that supports the transition from legacy “brown field” networks to SDN “green field” networks. This enables exciting new capabilities, and disruptive deployment and operational cost points for network operators. ### [PetitParser for Java](https://github.com/petitparser/java-petitparser) -Grammars for programming languages are traditionally specified statically. -They are hard to compose and reuse due to ambiguities that inevitably arise. -PetitParser combines ideas from scannnerless parsing, parser combinators, -parsing expression grammars and packrat parsers to model grammars and parsers -as objects that can be reconfigured dynamically. +Grammars for programming languages are traditionally specified statically. They are hard to compose and reuse due to ambiguities that inevitably arise. PetitParser combines ideas from scannnerless parsing, parser combinators, parsing expression grammars and packrat parsers to model grammars and parsers as objects that can be reconfigured dynamically. ### [PlaidML](https://github.com/plaidml/plaidml) @@ -670,14 +472,11 @@ PlaidML is a framework for making deep learning work everywhere. ### [Project V](https://www.v2ray.com/) - -Project V is a set of tools to help you build your own privacy network over -internet. +![](https://www.v2ray.com/resources/v2ray_1024.png) Project V is a set of tools to help you build your own privacy network over internet. ### [Prysmatic Labs Ethereum 2.0 Implementation](https://github.com/prysmaticlabs/prysm) -Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed -computing platform. +Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed computing platform. ### [Ray](https://github.com/ray-project/ray) @@ -685,8 +484,7 @@ Ray is a flexible, high-performance distributed execution framework. ### [Resty](https://github.com/go-resty/resty) -Resty is a Simple HTTP and REST client library for Go (inspired by Ruby -rest-client). +Resty is a Simple HTTP and REST client library for Go (inspired by Ruby rest-client). ### [Roughtime](https://roughtime.googlesource.com/roughtime) @@ -698,9 +496,7 @@ Selenium is a portable framework for testing web applications. ### [Semantic](https://github.com/github/semantic) -Semantic is a Haskell library and command line tool for parsing, analyzing, and -comparing source code. It is developed by GitHub (and used for example for the -code navigation). +Semantic is a Haskell library and command line tool for parsing, analyzing, and comparing source code. It is developed by GitHub (and used for example for the code navigation). ### [Served](https://github.com/meltwater/served) @@ -708,13 +504,11 @@ Served is a C++ library for building high performance RESTful web servers. ### [Sonnet](https://github.com/deepmind/sonnet) -Sonnet is a library built on top of TensorFlow for building complex neural -networks. +Sonnet is a library built on top of TensorFlow for building complex neural networks. ### [Sorbet](https://github.com/sorbet/sorbet) -Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to -codebases with millions of lines of code and can be adopted incrementally. +Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to codebases with millions of lines of code and can be adopted incrementally. ### [Spotify](https://spotify.com) @@ -722,12 +516,11 @@ Spotify is using Bazel to build their iOS and Android Apps ([source](https://twi ### [Tink](https://github.com/google/tink) -Tink is a multi-language, cross-platform, open source library that provides -cryptographic APIs that are secure, easy to use correctly, and hard(er) to -misuse. +Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. ### [TensorFlow](http://tensorflow.org) - + +![](https://upload.wikimedia.org/wikipedia/commons/a/a4/TensorFlowLogo.png) An open source software library for machine intelligence. @@ -741,10 +534,8 @@ Project Wycheproof tests crypto libraries against known attacks. ### [XIOSim](https://github.com/s-kanev/XIOSim) -XIOSim is a detailed user-mode microarchitectural simulator for the x86 -architecture. +XIOSim is a detailed user-mode microarchitectural simulator for the x86 architecture. ### [ZhihuDailyPurify](https://github.com/izzyleung/ZhihuDailyPurify) -ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese -question-and-answer webs. +ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese question-and-answer webs. diff --git a/concepts/build-files.mdx b/concepts/build-files.mdx new file mode 100644 index 00000000..87f9d167 --- /dev/null +++ b/concepts/build-files.mdx @@ -0,0 +1,73 @@ +--- +title: 'BUILD files' +--- + +The previous sections described packages, targets and labels, and the build dependency graph abstractly. This section describes the concrete syntax used to define a package. + +By definition, every package contains a `BUILD` file, which is a short program. + +Note: The `BUILD` file can be named either `BUILD` or `BUILD.bazel`. If both files exist, `BUILD.bazel` takes precedence over `BUILD`. For simplicity's sake, the documentation refers to these files simply as `BUILD` files. + +`BUILD` files are evaluated using an imperative language, [Starlark](https://github.com/bazelbuild/starlark/). + +They are interpreted as a sequential list of statements. + +In general, order does matter: variables must be defined before they are used, for example. However, most `BUILD` files consist only of declarations of build rules, and the relative order of these statements is immaterial; all that matters is *which* rules were declared, and with what values, by the time package evaluation completes. + +When a build rule function, such as `cc_library`, is executed, it creates a new target in the graph. This target can later be referred using a label. + +In simple `BUILD` files, rule declarations can be re-ordered freely without changing the behavior. + +To encourage a clean separation between code and data, `BUILD` files cannot contain function definitions, `for` statements or `if` statements (but list comprehensions and `if` expressions are allowed). Functions can be declared in `.bzl` files instead. Additionally, `*args` and `**kwargs` arguments are not allowed in `BUILD` files; instead list all the arguments explicitly. + +Crucially, programs in Starlark can't perform arbitrary I/O. This invariant makes the interpretation of `BUILD` files hermetic — dependent only on a known set of inputs, which is essential for ensuring that builds are reproducible. For more details, see [Hermeticity](/basics/hermeticity). + +Because `BUILD` files need to be updated whenever the dependencies of the underlying code change, they are typically maintained by multiple people on a team. `BUILD` file authors should comment liberally to document the role of each build target, whether or not it is intended for public use, and to document the role of the package itself. + +## Loading an extension + +Bazel extensions are files ending in `.bzl`. Use the `load` statement to import a symbol from an extension. + +``` +load("//foo/bar:file.bzl", "some_library") +``` + +This code loads the file `foo/bar/file.bzl` and adds the `some_library` symbol to the environment. This can be used to load new rules, functions, or constants (for example, a string or a list). Multiple symbols can be imported by using additional arguments to the call to `load`. Arguments must be string literals (no variable) and `load` statements must appear at top-level — they cannot be in a function body. + +The first argument of `load` is a [label](/concepts/labels) identifying a `.bzl` file. If it's a relative label, it is resolved with respect to the package (not directory) containing the current `bzl` file. Relative labels in `load` statements should use a leading `:`. + +`load` also supports aliases, therefore, you can assign different names to the imported symbols. + +``` +load("//foo/bar:file.bzl", library_alias = "some_library") +``` + +You can define multiple aliases within one `load` statement. Moreover, the argument list can contain both aliases and regular symbol names. The following example is perfectly legal (please note when to use quotation marks). + +``` +load(":my_rules.bzl", "some_rule", nice_alias = "some_other_rule") +``` + +In a `.bzl` file, symbols starting with `_` are not exported and cannot be loaded from another file. + +You can use [load visibility](/concepts/visibility#load-visibility) to restrict who may load a `.bzl` file. + +## Types of build rules + +The majority of build rules come in families, grouped together by language. For example, `cc_binary`, `cc_library` and `cc_test` are the build rules for C++ binaries, libraries, and tests, respectively. Other languages use the same naming scheme, with a different prefix, such as `java_*` for Java. Some of these functions are documented in the [Build Encyclopedia](/reference/be/overview), but it is possible for anyone to create new rules. + +- `*_binary` rules build executable programs in a given language. After a build, the executable will reside in the build tool's binary output tree at the corresponding name for the rule's label, so `//my:program` would appear at (for example) `$(BINDIR)/my/program`. + + In some languages, such rules also create a runfiles directory containing all the files mentioned in a `data` attribute belonging to the rule, or any rule in its transitive closure of dependencies; this set of files is gathered together in one place for ease of deployment to production. + +- `*_test` rules are a specialization of a `*_binary` rule, used for automated testing. Tests are simply programs that return zero on success. + + Like binaries, tests also have runfiles trees, and the files beneath it are the only files that a test may legitimately open at runtime. For example, a program `cc_test(name='x', data=['//foo:bar'])` may open and read `$TEST_SRCDIR/workspace/foo/bar` during execution. (Each programming language has its own utility function for accessing the value of `$TEST_SRCDIR`, but they are all equivalent to using the environment variable directly.) Failure to observe the rule will cause the test to fail when it is executed on a remote testing host. + +- `*_library` rules specify separately-compiled modules in the given programming language. Libraries can depend on other libraries, and binaries and tests can depend on libraries, with the expected separate-compilation behavior. + +[← Labels](/concepts/labels) · [Dependencies →](/concepts/dependencies) + +## File encoding + +`BUILD` and `.bzl` files should be encoded in UTF-8, of which ASCII is a valid subset. Arbitrary byte sequences are currently allowed, but may stop being supported in the future. diff --git a/concepts/build-ref.mdx b/concepts/build-ref.mdx index e8839d40..33c1e76c 100644 --- a/concepts/build-ref.mdx +++ b/concepts/build-ref.mdx @@ -2,51 +2,27 @@ title: 'Repositories, workspaces, packages, and targets' --- - - -Bazel builds software from source code organized in directory trees called -repositories. A defined set of repositories comprises the workspace. Source -files in repositories are organized in a nested hierarchy of packages, where -each package is a directory that contains a set of related source files and one -`BUILD` file. The `BUILD` file specifies what software outputs can be built from -the source. +Bazel builds software from source code organized in directory trees called repositories. A defined set of repositories comprises the workspace. Source files in repositories are organized in a nested hierarchy of packages, where each package is a directory that contains a set of related source files and one `BUILD` file. The `BUILD` file specifies what software outputs can be built from the source. ### Repositories -Source files used in a Bazel build are organized in _repositories_ (often -shortened to _repos_). A repo is a directory tree with a boundary marker file at -its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or -in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. +Source files used in a Bazel build are organized in *repositories* (often shortened to *repos*). A repo is a directory tree with a boundary marker file at its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. -The repo in which the current Bazel command is being run is called the _main -repo_. Other, (external) repos are defined by _repo rules_; see [external -dependencies overview](/external/overview) for more information. +The repo in which the current Bazel command is being run is called the *main repo*. Other, (external) repos are defined by *repo rules*; see [external dependencies overview](/external/overview) for more information. ## Workspace -A _workspace_ is the environment shared by all Bazel commands run from the same -main repo. It encompasses the main repo and the set of all defined external -repos. +A *workspace* is the environment shared by all Bazel commands run from the same main repo. It encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". ## Packages -The primary unit of code organization in a repository is the _package_. A -package is a collection of related files and a specification of how they can be -used to produce output artifacts. +The primary unit of code organization in a repository is the *package*. A package is a collection of related files and a specification of how they can be used to produce output artifacts. -A package is defined as a directory containing a -[`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A -package includes all files in its directory, plus all subdirectories beneath it, -except those which themselves contain a `BUILD` file. From this definition, no -file or directory may be a part of two different packages. +A package is defined as a directory containing a [`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A package includes all files in its directory, plus all subdirectories beneath it, except those which themselves contain a `BUILD` file. From this definition, no file or directory may be a part of two different packages. -For example, in the following directory tree there are two packages, `my/app`, -and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but -a directory belonging to package `my/app`. +For example, in the following directory tree there are two packages, `my/app`, and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but a directory belonging to package `my/app`. ``` src/my/app/BUILD @@ -58,48 +34,18 @@ src/my/app/tests/test.cc ## Targets -A package is a container of _targets_, which are defined in the package's -`BUILD` file. Most targets are one of two principal kinds, _files_ and _rules_. - -Files are further divided into two kinds. _Source files_ are usually written by -the efforts of people, and checked in to the repository. _Generated files_, -sometimes called derived files or output files, are not checked in, but are -generated from source files. - -The second kind of target is declared with a _rule_. Each rule instance -specifies the relationship between a set of input and a set of output files. The -inputs to a rule may be source files, but they also may be the outputs of other -rules. - -Whether the input to a rule is a source file or a generated file is in most -cases immaterial; what matters is only the contents of that file. This fact -makes it easy to replace a complex source file with a generated file produced by -a rule, such as happens when the burden of manually maintaining a highly -structured file becomes too tiresome, and someone writes a program to derive it. -No change is required to the consumers of that file. Conversely, a generated -file may easily be replaced by a source file with only local changes. - -The inputs to a rule may also include _other rules_. The precise meaning of such -relationships is often quite complex and language- or rule-dependent, but -intuitively it is simple: a C++ library rule A might have another C++ library -rule B for an input. The effect of this dependency is that B's header files are -available to A during compilation, B's symbols are available to A during -linking, and B's runtime data is available to A during execution. - -An invariant of all rules is that the files generated by a rule always belong to -the same package as the rule itself; it is not possible to generate files into -another package. It is not uncommon for a rule's inputs to come from another -package, though. - -Package groups are sets of packages whose purpose is to limit accessibility of -certain rules. Package groups are defined by the `package_group` function. They -have three properties: the list of packages they contain, their name, and other -package groups they include. The only allowed ways to refer to them are from the -`visibility` attribute of rules or from the `default_visibility` attribute of -the `package` function; they do not generate or consume files. For more -information, refer to the [`package_group` -documentation](/reference/be/functions#package_group). - - - Labels - +A package is a container of *targets*, which are defined in the package's `BUILD` file. Most targets are one of two principal kinds, *files* and *rules*. + +Files are further divided into two kinds. *Source files* are usually written by the efforts of people, and checked in to the repository. *Generated files*, sometimes called derived files or output files, are not checked in, but are generated from source files. + +The second kind of target is declared with a *rule*. Each rule instance specifies the relationship between a set of input and a set of output files. The inputs to a rule may be source files, but they also may be the outputs of other rules. + +Whether the input to a rule is a source file or a generated file is in most cases immaterial; what matters is only the contents of that file. This fact makes it easy to replace a complex source file with a generated file produced by a rule, such as happens when the burden of manually maintaining a highly structured file becomes too tiresome, and someone writes a program to derive it. No change is required to the consumers of that file. Conversely, a generated file may easily be replaced by a source file with only local changes. + +The inputs to a rule may also include *other rules*. The precise meaning of such relationships is often quite complex and language- or rule-dependent, but intuitively it is simple: a C++ library rule A might have another C++ library rule B for an input. The effect of this dependency is that B's header files are available to A during compilation, B's symbols are available to A during linking, and B's runtime data is available to A during execution. + +An invariant of all rules is that the files generated by a rule always belong to the same package as the rule itself; it is not possible to generate files into another package. It is not uncommon for a rule's inputs to come from another package, though. + +Package groups are sets of packages whose purpose is to limit accessibility of certain rules. Package groups are defined by the `package_group` function. They have three properties: the list of packages they contain, their name, and other package groups they include. The only allowed ways to refer to them are from the `visibility` attribute of rules or from the `default_visibility` attribute of the `package` function; they do not generate or consume files. For more information, refer to the [`package_group` documentation](/reference/be/functions#package_group). + +[Labels→](/concepts/labels) diff --git a/concepts/dependencies.mdx b/concepts/dependencies.mdx new file mode 100644 index 00000000..6d8c88cf --- /dev/null +++ b/concepts/dependencies.mdx @@ -0,0 +1,192 @@ +--- +title: 'Dependencies' +--- + +A target `A` *depends upon* a target `B` if `B` is needed by `A` at build or execution time. The *depends upon* relation induces a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) over targets, and it is called a *dependency graph*. + +A target's *direct* dependencies are those other targets reachable by a path of length 1 in the dependency graph. A target's *transitive* dependencies are those targets upon which it depends via a path of any length through the graph. + +In fact, in the context of builds, there are two dependency graphs, the graph of *actual dependencies* and the graph of *declared dependencies*. Most of the time, the two graphs are so similar that this distinction need not be made, but it is useful for the discussion below. + +## Actual and declared dependencies + +A target `X` is *actually dependent* on target `Y` if `Y` must be present, built, and up-to-date in order for `X` to be built correctly. *Built* could mean generated, processed, compiled, linked, archived, compressed, executed, or any of the other kinds of tasks that routinely occur during a build. + +A target `X` has a *declared dependency* on target `Y` if there is a dependency edge from `X` to `Y` in the package of `X`. + +For correct builds, the graph of actual dependencies *A* must be a subgraph of the graph of declared dependencies *D*. That is, every pair of directly-connected nodes `x --> y` in *A* must also be directly connected in *D*. It can be said that *D* is an *overapproximation* of *A*. + +Important: *D* should not be too much of an overapproximation of *A* because redundant declared dependencies can make builds slower and binaries larger. + +`BUILD` file writers must explicitly declare all of the actual direct dependencies for every rule to the build system, and no more. + +Failure to observe this principle causes undefined behavior: the build may fail, but worse, the build may depend on some prior operations, or upon transitive declared dependencies the target happens to have. Bazel checks for missing dependencies and report errors, but it's not possible for this checking to be complete in all cases. + +You need not (and should not) attempt to list everything indirectly imported, even if it is *needed* by `A` at execution time. + +During a build of target `X`, the build tool inspects the entire transitive closure of dependencies of `X` to ensure that any changes in those targets are reflected in the final result, rebuilding intermediates as needed. + +The transitive nature of dependencies leads to a common mistake. Sometimes, code in one file may use code provided by an *indirect* dependency — a transitive but not direct edge in the declared dependency graph. Indirect dependencies don't appear in the `BUILD` file. Because the rule doesn't directly depend on the provider, there is no way to track changes, as shown in the following example timeline: + +### 1. Declared dependencies match actual dependencies + +At first, everything works. The code in package `a` uses code in package `b`. The code in package `b` uses code in package `c`, and thus `a` transitively depends on `c`. + +| `a/BUILD` | `b/BUILD` | +| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ``` +rule( + name = "a", + srcs = "a.in", + deps = "//b:b", +) + +``` | ``` +rule( + name = "b", + srcs = "b.in", + deps = "//c:c", +) + +``` | +| `a / a.in` | `b / b.in` | +| ``` +import b; +b.foo(); + +``` | ``` +import c; +function foo() { + c.bar(); +} + +``` | +| ![Declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Declared** dependency graph | ![Actual dependency graph that matches the declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Actual** dependency graph | + +The declared dependencies overapproximate the actual dependencies. All is well. + +### 2. Adding an undeclared dependency + +A latent hazard is introduced when someone adds code to `a` that creates a direct *actual* dependency on `c`, but forgets to declare it in the build file `a/BUILD`. + +| `a / a.in` |   | +| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| ``` + import b; + import c; + b.foo(); + c.garply(); + +``` |   | +| ![Declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Declared** dependency graph | ![Actual dependency graph with arrows connecting a, b, and c. An arrow now connects A to C as well. This does not match the declared dependency graph](/docs/images/a_b_c_ac.svg)**Actual** dependency graph | + +The declared dependencies no longer overapproximate the actual dependencies. This may build ok, because the transitive closures of the two graphs are equal, but masks a problem: `a` has an actual but undeclared dependency on `c`. + +### 3. Divergence between declared and actual dependency graphs + +The hazard is revealed when someone refactors `b` so that it no longer depends on `c`, inadvertently breaking `a` through no fault of their own. + +|   | `b/BUILD` | +| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +|   | ``` +rule( + name = "b", + srcs = "b.in", + deps = "//d:d", +) + +``` | +|   | `b / b.in` | +|   | ``` + import d; + function foo() { + d.baz(); + } + +``` | +| ![Declared dependency graph with arrows connecting a and b. b no longer connects to c, which breaks a's connection to c](/docs/images/ab_c.svg)**Declared** dependency graph | ![Actual dependency graph that shows a connecting to b and c, but b no longer connects to c](/docs/images/a_b_a_c.svg)**Actual** dependency graph | + +The declared dependency graph is now an underapproximation of the actual dependencies, even when transitively closed; the build is likely to fail. + +The problem could have been averted by ensuring that the actual dependency from `a` to `c` introduced in Step 2 was properly declared in the `BUILD` file. + +## Types of dependencies + +Most build rules have three attributes for specifying different kinds of generic dependencies: `srcs`, `deps` and `data`. These are explained below. For more details, see [Attributes common to all rules](/reference/be/common-definitions). + +Many rules also have additional attributes for rule-specific kinds of dependencies, for example, `compiler` or `resources`. These are detailed in the [Build Encyclopedia](/reference/be/). + +### `srcs` dependencies + +Files consumed directly by the rule or rules that output source files. + +### `deps` dependencies + +Rule pointing to separately-compiled modules providing header files, symbols, libraries, data, etc. + +### `data` dependencies + +A build target might need some data files to run correctly. These data files aren't source code: they don't affect how the target is built. For example, a unit test might compare a function's output to the contents of a file. When you build the unit test you don't need the file, but you do need it when you run the test. The same applies to tools that are launched during execution. + +The build system runs tests in an isolated directory where only files listed as `data` are available. Thus, if a binary/library/test needs some files to run, specify them (or a build rule containing them) in `data`. For example: + +``` +# I need a config file from a directory named env: +java_binary( + name = "setenv", + ... + data = [":env/default_env.txt"], +) + +# I need test data from another directory +sh_test( + name = "regtest", + srcs = ["regtest.sh"], + data = [ + "//data:file1.txt", + "//data:file2.txt", + ... + ], +) +``` + +These files are available using the relative path `path/to/data/file`. In tests, you can refer to these files by joining the paths of the test's source directory and the workspace-relative path, for example, `${TEST_SRCDIR}/workspace/path/to/data/file`. + +## Using labels to reference directories + +As you look over our `BUILD` files, you might notice that some `data` labels refer to directories. These labels end with `/.` or `/` like these examples, which you should not use: + + +\`data = \["//data/regression:unittest/."]\` + +\`data = \["testdata/."]\` + +\`data = \["testdata/"]\` + + +This seems convenient, particularly for tests because it allows a test to use all the data files in the directory. + +But try not to do this. In order to ensure correct incremental rebuilds (and re-execution of tests) after a change, the build system must be aware of the complete set of files that are inputs to the build (or test). When you specify a directory, the build system performs a rebuild only when the directory itself changes (due to addition or deletion of files), but won't be able to detect edits to individual files as those changes don't affect the enclosing directory. Rather than specifying directories as inputs to the build system, you should enumerate the set of files contained within them, either explicitly or using the [`glob()`](/reference/be/functions#glob) function. (Use `**` to force the `glob()` to be recursive.) + + +\`data = glob(\["testdata/\*\*"])\` + + +Unfortunately, there are some scenarios where directory labels must be used. For example, if the `testdata` directory contains files whose names don't conform to the [label syntax](/concepts/labels#labels-lexical-specification), then explicit enumeration of files, or use of the [`glob()`](/reference/be/functions#glob) function produces an invalid labels error. You must use directory labels in this case, but beware of the associated risk of incorrect rebuilds described above. + +If you must use directory labels, keep in mind that you can't refer to the parent package with a relative `../` path; instead, use an absolute path like `//data/regression:unittest/.`. + +Note: Directory labels are only valid for data dependencies. If you try to use a directory as a label in an argument other than `data`, it will fail and you will get a (probably cryptic) error message. + +Any external rule, such as a test, that needs to use multiple files must explicitly declare its dependence on all of them. You can use `filegroup()` to group files together in the `BUILD` file: + +``` +filegroup( + name = 'my_data', + srcs = glob(['my_unittest_data/*']) +) +``` + +You can then reference the label `my_data` as the data dependency in your test. + +[← BUILD files](/concepts/build-files) · [Visibility →](/concepts/visibility) diff --git a/concepts/labels.mdx b/concepts/labels.mdx new file mode 100644 index 00000000..bbef34cb --- /dev/null +++ b/concepts/labels.mdx @@ -0,0 +1,148 @@ +--- +title: 'Labels' +--- + +A **label** is an identifier for a target. A typical label in its full canonical form looks like: + +```none +@@myrepo//my/app/main:app_binary +``` + +The first part of the label is the repository name, `@@myrepo`. The double-`@` syntax signifies that this is a [*canonical* repo name](/external/overview#canonical-repo-name), which is unique within the workspace. Labels with canonical repo names unambiguously identify a target no matter which context they appear in. + +Often the canonical repo name is an arcane string that looks like `@@rules_java++toolchains+local_jdk`. What is much more commonly seen is labels with an [*apparent* repo name](/external/overview#apparent-repo-name), which looks like: + +``` +@myrepo//my/app/main:app_binary +``` + +The only difference is the repo name being prefixed with one `@` instead of two. This refers to a repo with the apparent name `myrepo`, which could be different based on the context this label appears in. + +In the typical case that a label refers to the same repository from which it is used, the repo name part may be omitted. So, inside `@@myrepo` the first label is usually written as + +``` +//my/app/main:app_binary +``` + +The second part of the label is the un-qualified package name `my/app/main`, the path to the package relative to the repository root. Together, the repository name and the un-qualified package name form the fully-qualified package name `@@myrepo//my/app/main`. When the label refers to the same package it is used in, the package name (and optionally, the colon) may be omitted. So, inside `@@myrepo//my/app/main`, this label may be written either of the following ways: + +``` +app_binary +:app_binary +``` + +It is a matter of convention that the colon is omitted for files, but retained for rules, but it is not otherwise significant. + +The part of the label after the colon, `app_binary` is the un-qualified target name. When it matches the last component of the package path, it, and the colon, may be omitted. So, these two labels are equivalent: + +``` +//my/app/lib +//my/app/lib:lib +``` + +The name of a file target in a subdirectory of the package is the file's path relative to the package root (the directory containing the `BUILD` file). So, this file is in the `my/app/main/testdata` subdirectory of the repository: + +``` +//my/app/main:testdata/input.txt +``` + +Strings like `//my/app` and `@@some_repo//my/app` have two meanings depending on the context in which they are used: when Bazel expects a label, they mean `//my/app:app` and `@@some_repo//my/app:app`, respectively. But, when Bazel expects a package (e.g. in `package_group` specifications), they reference the package that contains that label. + +A common mistake in `BUILD` files is using `//my/app` to refer to a package, or to *all* targets in a package--it does not. Remember, it is equivalent to `//my/app:app`, so it names the `app` target in the `my/app` package of the current repository. + +However, the use of `//my/app` to refer to a package is encouraged in the specification of a `package_group` or in `.bzl` files, because it clearly communicates that the package name is absolute and rooted in the top-level directory of the workspace. + +Relative labels cannot be used to refer to targets in other packages; the repository identifier and package name must always be specified in this case. For example, if the source tree contains both the package `my/app` and the package `my/app/testdata` (each of these two directories has its own `BUILD` file), the latter package contains a file named `testdepot.zip`. Here are two ways (one wrong, one correct) to refer to this file within `//my/app:BUILD`: + + +\`testdata\` is a different package, so you can't use a relative path + + +``` +testdata/testdepot.zip +``` + + +refer to \`testdata\` with its full path + + +``` +//my/app/testdata:testdepot.zip +``` + +Labels starting with `@@//` are references to the main repository, which will still work even from external repositories. Therefore `@@//a/b/c` is different from `//a/b/c` when referenced from an external repository. The former refers back to the main repository, while the latter looks for `//a/b/c` in the external repository itself. This is especially relevant when writing rules in the main repository that refer to targets in the main repository, and will be used from external repositories. + +For information about the different ways you can refer to targets, see [target patterns](/run/build#specifying-build-targets). + +### Lexical specification of a label + +Label syntax discourages use of metacharacters that have special meaning to the shell. This helps to avoid inadvertent quoting problems, and makes it easier to construct tools and scripts that manipulate labels, such as the [Bazel Query Language](/query/language). + +The precise details of allowed target names are below. + +### Target names — `<var>package-name</var>:target-name` + +`target-name` is the name of the target within the package. The name of a rule is the value of the `name` attribute in the rule's declaration in a `BUILD` file; the name of a file is its pathname relative to the directory containing the `BUILD` file. + +Target names must be composed entirely of characters drawn from the set `a`–`z`, `A`–`Z`, `0`–`9`, and the punctuation symbols `!%-@^_"#$&'()*-+,;<=>?[]{|}~/.`. + +Filenames must be relative pathnames in normal form, which means they must neither start nor end with a slash (for example, `/foo` and `foo/` are forbidden) nor contain multiple consecutive slashes as path separators (for example, `foo//bar`). Similarly, up-level references (`..`) and current-directory references (`./`) are forbidden. + + +Do not use \`..\` to refer to files in other packages + +Use \`//package-name:filename\` + + +While it is common to use `/` in the name of a file target, avoid the use of `/` in the names of rules. Especially when the shorthand form of a label is used, it may confuse the reader. The label `//foo/bar/wiz` is always a shorthand for `//foo/bar/wiz:wiz`, even if there is no such package `foo/bar/wiz`; it never refers to `//foo:bar/wiz`, even if that target exists. + +However, there are some situations where use of a slash is convenient, or sometimes even necessary. For example, the name of certain rules must match their principal source file, which may reside in a subdirectory of the package. + +### Package names — `//package-name:<var>target-name</var>` + +The name of a package is the name of the directory containing its `BUILD` file, relative to the top-level directory of the containing repository. For example: `my/app`. + +On a technical level, Bazel enforces the following: + +- Allowed characters in package names are the lowercase letters `a` through `z`, the uppercase letters `A` through `Z`, the digits `0` through `9`, the characters ``! \"#$%&'()*+,-.;<=>?@[]^_`{|}`` (yes, there's a space character in there!), and of course forward slash `/` (since it's the directory separator). +- Package names may not start or end with a forward slash character `/`. +- Package names may not contain the substring `//`. This wouldn't make sense---what would the corresponding directory path be? +- Package names may not contain the substring `/./` or `/../` or `/.../` etc. This enforcement is done to avoid confusion when translating between a logical package name and a physical directory name, given the semantic meaning of the dot character in path strings. + +On a practical level: + +- For a language with a directory structure that is significant to its module system (for example, Java), it's important to choose directory names that are valid identifiers in the language. For example, don't start with a leading digit and avoid special characters, especially underscores and hyphens. +- Although Bazel supports targets in the workspace's root package (for example, `//:foo`), it's best to leave that package empty so all meaningful packages have descriptive names. + +## Rules + +A rule specifies the relationship between inputs and outputs, and the steps to build the outputs. Rules can be of one of many different kinds (sometimes called the *rule class*), which produce compiled executables and libraries, test executables and other supported outputs as described in the [Build Encyclopedia](/reference/be/overview). + +`BUILD` files declare *targets* by invoking *rules*. + +In the example below, we see the declaration of the target `my_app` using the `cc_binary` rule. + +```python +cc_binary( + name = "my_app", + srcs = ["my_app.cc"], + deps = [ + "//absl/base", + "//absl/strings", + ], +) +``` + +Every rule invocation has a `name` attribute (which must be a valid [target name](#target-names)), that declares a target within the package of the `BUILD` file. + +Every rule has a set of *attributes*; the applicable attributes for a given rule, and the significance and semantics of each attribute are a function of the rule's kind; see the [Build Encyclopedia](/reference/be/overview) for a list of rules and their corresponding attributes. Each attribute has a name and a type. Some of the common types an attribute can have are integer, label, list of labels, string, list of strings, output label, list of output labels. Not all attributes need to be specified in every rule. Attributes thus form a dictionary from keys (names) to optional, typed values. + +The `srcs` attribute present in many rules has type "list of labels"; its value, if present, is a list of labels, each being the name of a target that is an input to this rule. + +In some cases, the name of the rule kind is somewhat arbitrary, and more interesting are the names of the files generated by the rule, and this is true of genrules. For more information, see [General Rules: genrule](/reference/be/general#genrule). + +In other cases, the name is significant: for `*_binary` and `*_test` rules, for example, the rule name determines the name of the executable produced by the build. + +This directed acyclic graph over targets is called the *target graph* or *build dependency graph*, and is the domain over which the [Bazel Query tool](/query/guide) operates. + +[← Targets](/concepts/build-ref) · [BUILD files →](/concepts/build-files) diff --git a/concepts/platforms.mdx b/concepts/platforms.mdx index e560ea4d..e2ecbde8 100644 --- a/concepts/platforms.mdx +++ b/concepts/platforms.mdx @@ -2,30 +2,23 @@ title: 'Migrating to Platforms' --- - - -Bazel has sophisticated [support](#background) for modeling -[platforms][Platforms] and [toolchains][Toolchains] for multi-architecture and -cross-compiled builds. +Bazel has sophisticated [support](#background) for modeling [platforms](/extending/platforms) and [toolchains](/extending/toolchains) for multi-architecture and cross-compiled builds. This page summarizes the state of this support. -Key Point: Bazel's platform and toolchain APIs are available today. Not all -languages support them. Use these APIs with your project if you can. Bazel is -migrating all major languages so eventually all builds will be platform-based. +Key Point: Bazel's platform and toolchain APIs are available today. Not all languages support them. Use these APIs with your project if you can. Bazel is migrating all major languages so eventually all builds will be platform-based. See also: -* [Platforms][Platforms] -* [Toolchains][Toolchains] -* [Background][Background] +- [Platforms](/extending/platforms) +- [Toolchains](/extending/toolchains) +- [Background](#background) ## Status ### C++ -C++ rules use platforms to select toolchains when -`--incompatible_enable_cc_toolchain_resolution` is set. +C++ rules use platforms to select toolchains when `--incompatible_enable_cc_toolchain_resolution` is set. This means you can configure a C++ project with: @@ -41,23 +34,19 @@ bazel build //:my_cpp_project` --cpu=... --crosstool_top=... --compiler=... This will be enabled by default in Bazel 7.0 ([#7260](https://github.com/bazelbuild/bazel/issues/7260)). -To test your C++ project with platforms, see -[Migrating Your Project](#migrating-your-project) and -[Configuring C++ toolchains]. +To test your C++ project with platforms, see [Migrating Your Project](#migrating-your-project) and [Configuring C++ toolchains](/tutorials/ccp-toolchain-config). ### Java Java rules use platforms to select toolchains. -This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, -`--javabase`, and `--host_javabase`. +This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, `--javabase`, and `--host_javabase`. See [Java and Bazel](/docs/bazel-and-java) for details. ### Android -Android rules use platforms to select toolchains when -`--incompatible_enable_android_toolchain_resolution` is set. +Android rules use platforms to select toolchains when `--incompatible_enable_android_toolchain_resolution` is set. This means you can configure an Android project with: @@ -65,63 +54,42 @@ This means you can configure an Android project with: bazel build //:my_android_project --android_platforms=//:my_android_platform ``` -instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, -and `--fat_apk_cpu`. +instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, and `--fat_apk_cpu`. This will be enabled by default in Bazel 7.0 ([#16285](https://github.com/bazelbuild/bazel/issues/16285)). -To test your Android project with platforms, see -[Migrating Your Project](#migrating-your-project). +To test your Android project with platforms, see [Migrating Your Project](#migrating-your-project). ### Apple -[Apple rules] do not support platforms and are not yet scheduled -for support. +[Apple rules](https://github.com/bazelbuild/rules_apple) do not support platforms and are not yet scheduled for support. -You can still use platform APIs with Apple builds (for example, when building -with a mixture of Apple rules and pure C++) with [platform -mappings](#platform-mappings). +You can still use platform APIs with Apple builds (for example, when building with a mixture of Apple rules and pure C++) with [platform mappings](#platform-mappings). ### Other languages -* [Go rules] fully support platforms -* [Rust rules] fully support platforms. +- [Go rules](https://github.com/bazelbuild/rules_go) fully support platforms +- [Rust rules](https://github.com/bazelbuild/rules_rust) fully support platforms. -If you own a language rule set, see [Migrating your rule set] for adding -support. +If you own a language rule set, see [Migrating your rule set](#migrating-your-rule-set) for adding support. ## Background -*Platforms* and *toolchains* were introduced to standardize how software -projects target different architectures and cross-compile. +*Platforms* and *toolchains* were introduced to standardize how software projects target different architectures and cross-compile. -This was -[inspired][Inspiration] -by the observation that language maintainers were already doing this in ad -hoc, incompatible ways. For example, C++ rules used `--cpu` and - `--crosstool_top` to declare a target CPU and toolchain. Neither of these -correctly models a "platform". This produced awkward and incorrect builds. +This was [inspired](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) by the observation that language maintainers were already doing this in ad hoc, incompatible ways. For example, C++ rules used `--cpu` and `--crosstool_top` to declare a target CPU and toolchain. Neither of these correctly models a "platform". This produced awkward and incorrect builds. -Java, Android, and other languages evolved their own flags for similar purposes, -none of which interoperated with each other. This made cross-language builds -confusing and complicated. +Java, Android, and other languages evolved their own flags for similar purposes, none of which interoperated with each other. This made cross-language builds confusing and complicated. -Bazel is intended for large, multi-language, multi-platform projects. This -demands more principled support for these concepts, including a clear -standard API. +Bazel is intended for large, multi-language, multi-platform projects. This demands more principled support for these concepts, including a clear standard API. ### Need for migration -Upgrading to the new API requires two efforts: releasing the API and upgrading -rule logic to use it. +Upgrading to the new API requires two efforts: releasing the API and upgrading rule logic to use it. -The first is done but the second is ongoing. This consists of ensuring -language-specific platforms and toolchains are defined, language logic reads -toolchains through the new API instead of old flags like `--crosstool_top`, and -`config_setting`s select on the new API instead of old flags. +The first is done but the second is ongoing. This consists of ensuring language-specific platforms and toolchains are defined, language logic reads toolchains through the new API instead of old flags like `--crosstool_top`, and `config_setting`s select on the new API instead of old flags. -This work is straightforward but requires a distinct effort for each language, -plus fair warning for project owners to test against upcoming changes. +This work is straightforward but requires a distinct effort for each language, plus fair warning for project owners to test against upcoming changes. This is why this is an ongoing migration. @@ -136,25 +104,18 @@ bazel build //:myproject --platforms=//:myplatform This implies: 1. Your project's rules choose the right toolchains for `//:myplatform`. -1. Your project's dependencies choose the right toolchains for `//:myplatform`. -1. `//:myplatform` references -[common declarations][Common Platform Declarations] -of `CPU`, `OS`, and other generic, language-independent properties -1. All relevant [`select()`s][select()] properly match `//:myplatform`. -1. `//:myplatform` is defined in a clear, accessible place: in your project's -repo if the platform is unique to your project, or some common place all -consuming projects can find it - -Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be -deprecated and removed as soon as it's safe to do so. +2. Your project's dependencies choose the right toolchains for `//:myplatform`. +3. `//:myplatform` references [common declarations](https://github.com/bazelbuild/platforms) of `CPU`, `OS`, and other generic, language-independent properties +4. All relevant [`select()`s](/docs/configurable-attributes) properly match `//:myplatform`. +5. `//:myplatform` is defined in a clear, accessible place: in your project's repo if the platform is unique to your project, or some common place all consuming projects can find it -Ultimately, this will be the *sole* way to configure architectures. +Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be deprecated and removed as soon as it's safe to do so. +Ultimately, this will be the *sole* way to configure architectures. ## Migrating your project -If you build with languages that support platforms, your build should already -work with an invocation like: +If you build with languages that support platforms, your build should already work with an invocation like: ```posix-terminal bazel build //:myproject --platforms=//:myplatform @@ -162,49 +123,29 @@ bazel build //:myproject --platforms=//:myplatform See [Status](#status) and your language's documentation for precise details. -If a language requires a flag to enable platform support, you also need to set -that flag. See [Status](#status) for details. +If a language requires a flag to enable platform support, you also need to set that flag. See [Status](#status) for details. For your project to build, you need to check the following: -1. `//:myplatform` must exist. It's generally the project owner's responsibility - to define platforms because different projects target different machines. - See [Default platforms](#default-platforms). +1. `//:myplatform` must exist. It's generally the project owner's responsibility to define platforms because different projects target different machines. See [Default platforms](#default-platforms). -1. The toolchains you want to use must exist. If using stock toolchains, the - language owners should include instructions for how to register them. If - writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your - `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). +2. The toolchains you want to use must exist. If using stock toolchains, the language owners should include instructions for how to register them. If writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). -1. `select()`s and [configuration transitions][Starlark transitions] must - resolve properly. See [select()](#select) and [Transitions](#transitions). +3. `select()`s and [configuration transitions](/extending/config#user-defined-transitions) must resolve properly. See [select()](#select) and [Transitions](#transitions). -1. If your build mixes languages that do and don't support platforms, you may - need platform mappings to help the legacy languages work with the new API. - See [Platform mappings](#platform-mappings) for details. +4. If your build mixes languages that do and don't support platforms, you may need platform mappings to help the legacy languages work with the new API. See [Platform mappings](#platform-mappings) for details. If you still have problems, [reach out](#questions) for support. ### Default platforms -Project owners should define explicit -[platforms][Defining Constraints and Platforms] to describe the architectures -they want to build for. These are then triggered with `--platforms`. +Project owners should define explicit [platforms](/extending/platforms#constraints-platforms) to describe the architectures they want to build for. These are then triggered with `--platforms`. -When `--platforms` isn't set, Bazel defaults to a `platform` representing the -local build machine. This is auto-generated at `@platforms//host` (aliased as -`@bazel_tools//tools:host_platform`) -so there's no need to explicitly define it. It maps the local machine's `OS` -and `CPU` with `constraint_value`s declared in -[`@platforms`](https://github.com/bazelbuild/platforms). +When `--platforms` isn't set, Bazel defaults to a `platform` representing the local build machine. This is auto-generated at `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`) so there's no need to explicitly define it. It maps the local machine's `OS` and `CPU` with `constraint_value`s declared in [`@platforms`](https://github.com/bazelbuild/platforms). ### `select()` -Projects can [`select()`][select()] on -[`constraint_value` targets][constraint_value Rule] but not complete -platforms. This is intentional so `select()` supports as wide a variety of -machines as possible. A library with `ARM`-specific sources should support *all* -`ARM`-powered machines unless there's reason to be more specific. +Projects can [`select()`](/docs/configurable-attributes) on [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value) but not complete platforms. This is intentional so `select()` supports as wide a variety of machines as possible. A library with `ARM`-specific sources should support *all* `ARM`-powered machines unless there's reason to be more specific. To select on one or more `constraint_value`s, use: @@ -228,75 +169,47 @@ config_setting( ) ``` -More details [here][select() Platforms]. +More details [here](/docs/configurable-attributes#platforms). -`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. -When migrating your project to platforms, you must either convert them to -`constraint_values` or use [platform mappings](#platform-mappings) to support -both styles during migration. +`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. When migrating your project to platforms, you must either convert them to `constraint_values` or use [platform mappings](#platform-mappings) to support both styles during migration. ### Transitions -[Starlark transitions][Starlark transitions] change -flags down parts of your build graph. If your project uses a transition that -sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read -`--platforms` won't see these changes. +[Starlark transitions](/extending/config#user-defined-transitions) change flags down parts of your build graph. If your project uses a transition that sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read `--platforms` won't see these changes. -When migrating your project to platforms, you must either convert changes like -`return { "//command_line_option:cpu": "arm" }` to `return { -"//command_line_option:platforms": "//:my_arm_platform" }` or use [platform -mappings](#platform-mappings) to support both styles during migration. -window. +When migrating your project to platforms, you must either convert changes like `return { "//command_line_option:cpu": "arm" }` to `return { "//command_line_option:platforms": "//:my_arm_platform" }` or use [platform mappings](#platform-mappings) to support both styles during migration. window. ## Migrating your rule set If you own a rule set and want to support platforms, you need to: -1. Have rule logic resolve toolchains with the toolchain API. See - [toolchain API][Toolchains] (`ctx.toolchains`). +1. Have rule logic resolve toolchains with the toolchain API. See [toolchain API](/extending/toolchains) (`ctx.toolchains`). -1. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so - rule logic alternately resolves toolchains through the new API or old flags - like `--crosstool_top` during migration testing. +2. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so rule logic alternately resolves toolchains through the new API or old flags like `--crosstool_top` during migration testing. -1. Define the relevant properties that make up platform components. See - [Common platform properties](#common-platform-properties) +3. Define the relevant properties that make up platform components. See [Common platform properties](#common-platform-properties) -1. Define standard toolchains and make them accessible to users through your - rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) +4. Define standard toolchains and make them accessible to users through your rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) -1. Ensure [`select()`s](#select) and - [configuration transitions](#transitions) support platforms. This is the - biggest challenge. It's particularly challenging for multi-language projects - (which may fail if *all* languages can't read `--platforms`). +5. Ensure [`select()`s](#select) and [configuration transitions](#transitions) support platforms. This is the biggest challenge. It's particularly challenging for multi-language projects (which may fail if *all* languages can't read `--platforms`). -If you need to mix with rules that don't support platforms, you may need -[platform mappings](#platform-mappings) to bridge the gap. +If you need to mix with rules that don't support platforms, you may need [platform mappings](#platform-mappings) to bridge the gap. ### Common platform properties -Common, cross-language platform properties like `OS` and `CPU` should be -declared in [`@platforms`](https://github.com/bazelbuild/platforms). -This encourages sharing, standardization, and cross-language compatibility. +Common, cross-language platform properties like `OS` and `CPU` should be declared in [`@platforms`](https://github.com/bazelbuild/platforms). This encourages sharing, standardization, and cross-language compatibility. -Properties unique to your rules should be declared in your rule's repo. This -lets you maintain clear ownership over the specific concepts your rules are -responsible for. +Properties unique to your rules should be declared in your rule's repo. This lets you maintain clear ownership over the specific concepts your rules are responsible for. -If your rules use custom-purpose OSes or CPUs, these should be declared in your -rule's repo vs. -[`@platforms`](https://github.com/bazelbuild/platforms). +If your rules use custom-purpose OSes or CPUs, these should be declared in your rule's repo vs. [`@platforms`](https://github.com/bazelbuild/platforms). ## Platform mappings -*Platform mappings* is a temporary API that lets platform-aware logic mix with -legacy logic in the same build. This is a blunt tool that's only intended to -smooth incompatibilities with different migration timeframes. +*Platform mappings* is a temporary API that lets platform-aware logic mix with legacy logic in the same build. This is a blunt tool that's only intended to smooth incompatibilities with different migration timeframes. -Caution: Only use this if necessary, and expect to eventually eliminate it. +Caution: Only use this if necessary, and expect to eventually eliminate it. -A platform mapping is a map of either a `platform()` to a -corresponding set of legacy flags or the reverse. For example: +A platform mapping is a map of either a `platform()` to a corresponding set of legacy flags or the reverse. For example: ```python platforms: @@ -317,20 +230,15 @@ flags: //platforms:macos ``` -Bazel uses this to guarantee all settings, both platform-based and -legacy, are consistently applied throughout the build, including through -[transitions](#transitions). +Bazel uses this to guarantee all settings, both platform-based and legacy, are consistently applied throughout the build, including through [transitions](#transitions). -By default Bazel reads mappings from the `platform_mappings` file in your -workspace root. You can also set -`--platform_mappings=//:my_custom_mapping`. +By default Bazel reads mappings from the `platform_mappings` file in your workspace root. You can also set `--platform_mappings=//:my_custom_mapping`. -See the [platform mappings design] for details. +See the [platform mappings design](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit) for details. ## API review -A [`platform`][platform Rule] is a collection of -[`constraint_value` targets][constraint_value Rule]: +A [`platform`](/reference/be/platforms-and-toolchains#platform) is a collection of [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value): ```python platform( @@ -342,9 +250,7 @@ platform( ) ``` -A [`constraint_value`][constraint_value Rule] is a machine -property. Values of the same "kind" are grouped under a common -[`constraint_setting`][constraint_setting Rule]: +A [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) is a machine property. Values of the same "kind" are grouped under a common [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting): ```python constraint_setting(name = "os") @@ -358,72 +264,27 @@ constraint_value( ) ``` -A [`toolchain`][Toolchains] is a [Starlark rule][Starlark rule]. Its -attributes declare a language's tools (like `compiler = -"//mytoolchain:custom_gcc"`). Its [providers][Starlark Provider] pass -this information to rules that need to build with these tools. +A [`toolchain`](/extending/toolchains) is a [Starlark rule](/extending/rules). Its attributes declare a language's tools (like `compiler = "//mytoolchain:custom_gcc"`). Its [providers](/extending/rules#providers) pass this information to rules that need to build with these tools. -Toolchains declare the `constraint_value`s of machines they can -[target][target_compatible_with Attribute] -(`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can -[run on][exec_compatible_with Attribute] -(`exec_compatible_with = ["@platforms//os:mac"]`). +Toolchains declare the `constraint_value`s of machines they can [target](/reference/be/platforms-and-toolchains#toolchain.target_compatible_with) (`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can [run on](/reference/be/platforms-and-toolchains#toolchain.exec_compatible_with) (`exec_compatible_with = ["@platforms//os:mac"]`). -When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel -automatically selects a toolchain that can run on the build machine and -build binaries for `//:myplatform`. This is known as *toolchain resolution*. +When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel automatically selects a toolchain that can run on the build machine and build binaries for `//:myplatform`. This is known as *toolchain resolution*. -The set of available toolchains can be registered in the `MODULE.bazel` file -with [`register_toolchains`][register_toolchains Function] or at the -command line with [`--extra_toolchains`][extra_toolchains Flag]. +The set of available toolchains can be registered in the `MODULE.bazel` file with [`register_toolchains`](/rules/lib/globals/module#register_toolchains) or at the command line with [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). -For more information see [here][Toolchains]. +For more information see [here](/extending/toolchains). ## Questions -For general support and questions about the migration timeline, contact -[bazel-discuss] or the owners of the appropriate rules. +For general support and questions about the migration timeline, contact [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) or the owners of the appropriate rules. -For discussions on the design and evolution of the platform/toolchain APIs, -contact [bazel-dev]. +For discussions on the design and evolution of the platform/toolchain APIs, contact [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev). ## See also -* [Configurable Builds - Part 1] -* [Platforms] -* [Toolchains] -* [Bazel Platforms Cookbook] -* [Platforms examples] -* [Example C++ toolchain] - -[Android Rules]: /docs/bazel-and-android -[Apple Rules]: https://github.com/bazelbuild/rules_apple -[Background]: #background -[Bazel platforms Cookbook]: https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/ -[bazel-dev]: https://groups.google.com/forum/#!forum/bazel-dev -[bazel-discuss]: https://groups.google.com/forum/#!forum/bazel-discuss -[Common Platform Declarations]: https://github.com/bazelbuild/platforms -[constraint_setting Rule]: /reference/be/platforms-and-toolchains#constraint_setting -[constraint_value Rule]: /reference/be/platforms-and-toolchains#constraint_value -[Configurable Builds - Part 1]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html -[Configuring C++ toolchains]: /tutorials/ccp-toolchain-config -[Defining Constraints and Platforms]: /extending/platforms#constraints-platforms -[Example C++ toolchain]: https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms -[exec_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.exec_compatible_with -[extra_toolchains Flag]: /reference/command-line-reference#flag--extra_toolchains -[Go Rules]: https://github.com/bazelbuild/rules_go -[Inspiration]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html -[Migrating your rule set]: #migrating-your-rule-set -[Platforms]: /extending/platforms -[Platforms examples]: https://github.com/hlopko/bazel_platforms_examples -[platform mappings design]: https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit -[platform Rule]: /reference/be/platforms-and-toolchains#platform -[register_toolchains Function]: /rules/lib/globals/module#register_toolchains -[Rust rules]: https://github.com/bazelbuild/rules_rust -[select()]: /docs/configurable-attributes -[select() Platforms]: /docs/configurable-attributes#platforms -[Starlark provider]: /extending/rules#providers -[Starlark rule]: /extending/rules -[Starlark transitions]: /extending/config#user-defined-transitions -[target_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.target_compatible_with -[Toolchains]: /extending/toolchains +- [Configurable Builds - Part 1](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) +- [Platforms](/extending/platforms) +- [Toolchains](/extending/toolchains) +- [Bazel Platforms Cookbook](https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/) +- [Platforms examples](https://github.com/hlopko/bazel_platforms_examples) +- [Example C++ toolchain](https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms) diff --git a/concepts/visibility.mdx b/concepts/visibility.mdx index 982f0a0e..10fd9eec 100644 --- a/concepts/visibility.mdx +++ b/concepts/visibility.mdx @@ -2,97 +2,49 @@ title: 'Visibility' --- +This page covers Bazel's two visibility systems: [target visibility](#target-visibility) and [load visibility](#load-visibility). - -This page covers Bazel's two visibility systems: -[target visibility](#target-visibility) and [load visibility](#load-visibility). - -Both types of visibility help other developers distinguish between your -library's public API and its implementation details, and help enforce structure -as your workspace grows. You can also use visibility when deprecating a public -API to allow current users while denying new ones. +Both types of visibility help other developers distinguish between your library's public API and its implementation details, and help enforce structure as your workspace grows. You can also use visibility when deprecating a public API to allow current users while denying new ones. ## Target visibility -**Target visibility** controls who may depend on your target — that is, who may -use your target's label inside an attribute such as `deps`. A target will fail -to build during the [analysis](/reference/glossary#analysis-phase) phase if it -violates the visibility of one of its dependencies. +**Target visibility** controls who may depend on your target — that is, who may use your target's label inside an attribute such as `deps`. A target will fail to build during the [analysis](/reference/glossary#analysis-phase) phase if it violates the visibility of one of its dependencies. -Generally, a target `A` is visible to a target `B` if they are in the same -location, or if `A` grants visibility to `B`'s location. In the absence of -[symbolic macros](/extending/macros), the term "location" can be simplified -to just "package"; see [below](#symbolic-macros) for more on symbolic macros. +Generally, a target `A` is visible to a target `B` if they are in the same location, or if `A` grants visibility to `B`'s location. In the absence of [symbolic macros](/extending/macros), the term "location" can be simplified to just "package"; see [below](#symbolic-macros) for more on symbolic macros. -Visibility is specified by listing allowed packages. Allowing a package does not -necessarily mean that its subpackages are also allowed. For more details on -packages and subpackages, see [Concepts and terminology](/concepts/build-ref). +Visibility is specified by listing allowed packages. Allowing a package does not necessarily mean that its subpackages are also allowed. For more details on packages and subpackages, see [Concepts and terminology](/concepts/build-ref). -For prototyping, you can disable target visibility enforcement by setting the -flag `--check_visibility=false`. This shouldn't be done for production usage in -submitted code. +For prototyping, you can disable target visibility enforcement by setting the flag `--check_visibility=false`. This shouldn't be done for production usage in submitted code. -The primary way to control visibility is with a rule's -[`visibility`](/reference/be/common-definitions#common.visibility) attribute. -The following subsections describe the attribute's format, how to apply it to -various kinds of targets, and the interaction between the visibility system and -symbolic macros. +The primary way to control visibility is with a rule's [`visibility`](/reference/be/common-definitions#common.visibility) attribute. The following subsections describe the attribute's format, how to apply it to various kinds of targets, and the interaction between the visibility system and symbolic macros. ### Visibility specifications -All rule targets have a `visibility` attribute that takes a list of labels. Each -label has one of the following forms. With the exception of the last form, these -are just syntactic placeholders that don't correspond to any actual target. +All rule targets have a `visibility` attribute that takes a list of labels. Each label has one of the following forms. With the exception of the last form, these are just syntactic placeholders that don't correspond to any actual target. -* `"//visibility:public"`: Grants access to all packages. +- `"//visibility:public"`: Grants access to all packages. -* `"//visibility:private"`: Does not grant any additional access; only targets - in this location's package can use this target. +- `"//visibility:private"`: Does not grant any additional access; only targets in this location's package can use this target. -* `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its - subpackages). +- `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its subpackages). -* `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its - direct and indirect subpackages. +- `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its direct and indirect subpackages. -* `"//some_pkg:my_package_group"`: Grants access to all of the packages that - are part of the given [`package_group`](/reference/be/functions#package_group). +- `"//some_pkg:my_package_group"`: Grants access to all of the packages that are part of the given [`package_group`](/reference/be/functions#package_group). - * Package groups use a - [different syntax](/reference/be/functions#package_group.packages) for - specifying packages. Within a package group, the forms - `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively - replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, - `"//visibility:public"` and `"//visibility:private"` are just `"public"` - and `"private"`. + - Package groups use a [different syntax](/reference/be/functions#package_group.packages) for specifying packages. Within a package group, the forms `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, `"//visibility:public"` and `"//visibility:private"` are just `"public"` and `"private"`. -For example, if `//some/package:mytarget` has its `visibility` set to -`[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target -that is part of the `//some/package/...` source tree, as well as targets -declared in `//tests/BUILD`, but not by targets defined in -`//tests/integration/BUILD`. +For example, if `//some/package:mytarget` has its `visibility` set to `[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target that is part of the `//some/package/...` source tree, as well as targets declared in `//tests/BUILD`, but not by targets defined in `//tests/integration/BUILD`. -**Best practice:** To make several targets visible to the same set -of packages, use a `package_group` instead of repeating the list in each -target's `visibility` attribute. This increases readability and prevents the -lists from getting out of sync. +**Best practice:** To make several targets visible to the same set of packages, use a `package_group` instead of repeating the list in each target's `visibility` attribute. This increases readability and prevents the lists from getting out of sync. -**Best practice:** When granting visibility to another team's project, prefer -`__subpackages__` over `__pkg__` to avoid needless visibility churn as that -project evolves and adds new subpackages. +**Best practice:** When granting visibility to another team's project, prefer `__subpackages__` over `__pkg__` to avoid needless visibility churn as that project evolves and adds new subpackages. -Note: The `visibility` attribute may not specify non-`package_group` targets. -Doing so triggers a "Label does not refer to a package group" or "Cycle in -dependency graph" error. +Note: The `visibility` attribute may not specify non-`package_group` targets. Doing so triggers a "Label does not refer to a package group" or "Cycle in dependency graph" error. ### Rule target visibility -A rule target's visibility is determined by taking its `visibility` attribute --- or a suitable default if not given -- and appending the location where the -target was declared. For targets not declared in a symbolic macro, if the -package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), -this default is used; for all other packages and for targets declared in a -symbolic macro, the default is just `["//visibility:private"]`. +A rule target's visibility is determined by taking its `visibility` attribute -- or a suitable default if not given -- and appending the location where the target was declared. For targets not declared in a symbolic macro, if the package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), this default is used; for all other packages and for targets declared in a symbolic macro, the default is just `["//visibility:private"]`. ```starlark # //mypkg/BUILD @@ -130,15 +82,11 @@ package_group( ) ``` -**Best practice:** Avoid setting `default_visibility` to public. It may be -convenient for prototyping or in small codebases, but the risk of inadvertently -creating public targets increases as the codebase grows. It's better to be -explicit about which targets are part of a package's public interface. +**Best practice:** Avoid setting `default_visibility` to public. It may be convenient for prototyping or in small codebases, but the risk of inadvertently creating public targets increases as the codebase grows. It's better to be explicit about which targets are part of a package's public interface. ### Generated file target visibility -A generated file target has the same visibility as the rule target that -generates it. +A generated file target has the same visibility as the rule target that generates it. ```starlark # //mypkg/BUILD @@ -168,36 +116,21 @@ some_rule( ### Source file target visibility -Source file targets can either be explicitly declared using -[`exports_files`](/reference/be/functions#exports_files), or implicitly created -by referring to their filename in a label attribute of a rule (outside of a -symbolic macro). As with rule targets, the location of the call to -`exports_files`, or the BUILD file that referred to the input file, is always -automatically appended to the file's visibility. +Source file targets can either be explicitly declared using [`exports_files`](/reference/be/functions#exports_files), or implicitly created by referring to their filename in a label attribute of a rule (outside of a symbolic macro). As with rule targets, the location of the call to `exports_files`, or the BUILD file that referred to the input file, is always automatically appended to the file's visibility. -Files declared by `exports_files` can have their visibility set by the -`visibility` parameter to that function. If this parameter is not given, the visibility is public. +Files declared by `exports_files` can have their visibility set by the `visibility` parameter to that function. If this parameter is not given, the visibility is public. -Note: `exports_files` may not be used to override the visibility of a generated -file. +Note: `exports_files` may not be used to override the visibility of a generated file. -For files that do not appear in a call to `exports_files`, the visibility -depends on the value of the flag -[`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): +For files that do not appear in a call to `exports_files`, the visibility depends on the value of the flag [`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): -* If the flag is true, the visibility is private. +- If the flag is true, the visibility is private. -* Else, the legacy behavior applies: The visibility is the same as the - `BUILD` file's `default_visibility`, or private if a default visibility is - not specified. +- Else, the legacy behavior applies: The visibility is the same as the `BUILD` file's `default_visibility`, or private if a default visibility is not specified. -Avoid relying on the legacy behavior. Always write an `exports_files` -declaration whenever a source file target needs non-private visibility. +Avoid relying on the legacy behavior. Always write an `exports_files` declaration whenever a source file target needs non-private visibility. -**Best practice:** When possible, prefer to expose a rule target rather than a -source file. For example, instead of calling `exports_files` on a `.java` file, -wrap the file in a non-private `java_library` target. Generally, rule targets -should only directly reference source files that live in the same package. +**Best practice:** When possible, prefer to expose a rule target rather than a source file. For example, instead of calling `exports_files` on a `.java` file, wrap the file in a non-private `java_library` target. Generally, rule targets should only directly reference source files that live in the same package. #### Example @@ -218,98 +151,45 @@ cc_binary( ### Config setting visibility -Historically, Bazel has not enforced visibility for -[`config_setting`](/reference/be/general#config_setting) targets that are -referenced in the keys of a [`select()`](/reference/be/functions#select). There -are two flags to remove this legacy behavior: +Historically, Bazel has not enforced visibility for [`config_setting`](/reference/be/general#config_setting) targets that are referenced in the keys of a [`select()`](/reference/be/functions#select). There are two flags to remove this legacy behavior: -* [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) - enables visibility checking for these targets. To assist with migration, it - also causes any `config_setting` that does not specify a `visibility` to be - considered public (regardless of package-level `default_visibility`). +- [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) enables visibility checking for these targets. To assist with migration, it also causes any `config_setting` that does not specify a `visibility` to be considered public (regardless of package-level `default_visibility`). -* [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) - causes `config_setting`s that do not specify a `visibility` to respect the - package's `default_visibility` and to fallback on private visibility, just - like any other rule target. It is a no-op if - `--incompatible_enforce_config_setting_visibility` is not set. +- [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) causes `config_setting`s that do not specify a `visibility` to respect the package's `default_visibility` and to fallback on private visibility, just like any other rule target. It is a no-op if `--incompatible_enforce_config_setting_visibility` is not set. -Avoid relying on the legacy behavior. Any `config_setting` that is intended to -be used outside the current package should have an explicit `visibility`, if the -package does not already specify a suitable `default_visibility`. +Avoid relying on the legacy behavior. Any `config_setting` that is intended to be used outside the current package should have an explicit `visibility`, if the package does not already specify a suitable `default_visibility`. ### Package group target visibility -`package_group` targets do not have a `visibility` attribute. They are always -publicly visible. +`package_group` targets do not have a `visibility` attribute. They are always publicly visible. ### Visibility of implicit dependencies -Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — -dependencies that are not spelled out in a `BUILD` file but are inherent to -every instance of that rule. For example, a `cc_library` rule might create an -implicit dependency from each of its rule targets to an executable target -representing a C++ compiler. +Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — dependencies that are not spelled out in a `BUILD` file but are inherent to every instance of that rule. For example, a `cc_library` rule might create an implicit dependency from each of its rule targets to an executable target representing a C++ compiler. -The visibility of such an implicit dependency is checked with respect to the -package containing the `.bzl` file in which the rule (or aspect) is defined. In -our example, the C++ compiler could be private so long as it lives in the same -package as the definition of the `cc_library` rule. As a fallback, if the -implicit dependency is not visible from the definition, it is checked with -respect to the `cc_library` target. +The visibility of such an implicit dependency is checked with respect to the package containing the `.bzl` file in which the rule (or aspect) is defined. In our example, the C++ compiler could be private so long as it lives in the same package as the definition of the `cc_library` rule. As a fallback, if the implicit dependency is not visible from the definition, it is checked with respect to the `cc_library` target. -If you want to restrict the usage of a rule to certain packages, use -[load visibility](#load-visibility) instead. +If you want to restrict the usage of a rule to certain packages, use [load visibility](#load-visibility) instead. ### Visibility and symbolic macros -This section describes how the visibility system interacts with -[symbolic macros](/extending/macros). +This section describes how the visibility system interacts with [symbolic macros](/extending/macros). #### Locations within symbolic macros -A key detail of the visibility system is how we determine the location of a -declaration. For targets that are not declared in a symbolic macro, the location -is just the package where the target lives -- the package of the `BUILD` file. -But for targets created in a symbolic macro, the location is the package -containing the `.bzl` file where the macro's definition (the -`my_macro = macro(...)` statement) appears. When a target is created inside -multiple nested targets, it is always the innermost symbolic macro's definition -that is used. - -The same system is used to determine what location to check against a given -dependency's visibility. If the consuming target was created inside a macro, we -look at the innermost macro's definition rather than the package the consuming -target lives in. - -This means that all macros whose code is defined in the same package are -automatically "friends" with one another. Any target directly created by a macro -defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, -regardless of what packages the macros are actually instantiated in. Likewise, -they can see, and can be seen by, targets declared directly in `//lib/BUILD` and -its legacy macros. Conversely, targets that live in the same package cannot -necessarily see one another if at least one of them is created by a symbolic -macro. - -Within a symbolic macro's implementation function, the `visibility` parameter -has the effective value of the macro's `visibility` attribute after appending -the location where the macro was called. The standard way for a macro to export -one of its targets to its caller is to forward this value along to the target's -declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit -this attribute won't be visible to the caller of the macro unless the caller -happens to be in the same package as the macro definition. This behavior -composes, in the sense that a chain of nested calls to submacros may each pass -`visibility = visibility`, re-exporting the inner macro's exported targets to -the caller at each level, without exposing any of the macros' implementation -details. +A key detail of the visibility system is how we determine the location of a declaration. For targets that are not declared in a symbolic macro, the location is just the package where the target lives -- the package of the `BUILD` file. But for targets created in a symbolic macro, the location is the package containing the `.bzl` file where the macro's definition (the `my_macro = macro(...)` statement) appears. When a target is created inside multiple nested targets, it is always the innermost symbolic macro's definition that is used. + +The same system is used to determine what location to check against a given dependency's visibility. If the consuming target was created inside a macro, we look at the innermost macro's definition rather than the package the consuming target lives in. + +This means that all macros whose code is defined in the same package are automatically "friends" with one another. Any target directly created by a macro defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, regardless of what packages the macros are actually instantiated in. Likewise, they can see, and can be seen by, targets declared directly in `//lib/BUILD` and its legacy macros. Conversely, targets that live in the same package cannot necessarily see one another if at least one of them is created by a symbolic macro. + +Within a symbolic macro's implementation function, the `visibility` parameter has the effective value of the macro's `visibility` attribute after appending the location where the macro was called. The standard way for a macro to export one of its targets to its caller is to forward this value along to the target's declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit this attribute won't be visible to the caller of the macro unless the caller happens to be in the same package as the macro definition. This behavior composes, in the sense that a chain of nested calls to submacros may each pass `visibility = visibility`, re-exporting the inner macro's exported targets to the caller at each level, without exposing any of the macros' implementation details. #### Delegating privileges to a submacro -The visibility model has a special feature to allow a macro to delegate its -permissions to a submacro. This is important for factoring and composing macros. +The visibility model has a special feature to allow a macro to delegate its permissions to a submacro. This is important for factoring and composing macros. -Suppose you have a macro `my_macro` that creates a dependency edge using a rule -`some_library` from another package: +Suppose you have a macro `my_macro` that creates a dependency edge using a rule `some_library` from another package: ```starlark # //macro/defs.bzl @@ -338,10 +218,7 @@ load("//macro:defs.bzl", "my_macro") my_macro(name = "foo", ...) ``` -The `//pkg:foo_dependency` target has no `visibility` specified, so it is only -visible within `//macro`, which works fine for the consuming target. Now, what -happens if the author of `//lib` refactors `some_library` to instead be -implemented using a macro? +The `//pkg:foo_dependency` target has no `visibility` specified, so it is only visible within `//macro`, which works fine for the consuming target. Now, what happens if the author of `//lib` refactors `some_library` to instead be implemented using a macro? ```starlark # //lib:defs.bzl @@ -357,88 +234,43 @@ def _impl(name, visibility, deps, ...): some_library = macro(implementation = _impl, ...) ``` -With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than -`//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's -visibility. The author of `my_macro` can't be expected to pass -`visibility = ["//lib"]` to the declaration of the dependency just to work -around this implementation detail. +With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than `//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's visibility. The author of `my_macro` can't be expected to pass `visibility = ["//lib"]` to the declaration of the dependency just to work around this implementation detail. -For this reason, when a dependency of a target is also an attribute value of the -macro that declared the target, we check the dependency's visibility against the -location of the macro instead of the location of the consuming target. +For this reason, when a dependency of a target is also an attribute value of the macro that declared the target, we check the dependency's visibility against the location of the macro instead of the location of the consuming target. -In this example, to validate whether `//pkg:foo_consumer` can see -`//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an -input to the call to `some_library` inside of `my_macro`, and instead check the -dependency's visibility against the location of this call, `//macro`. +In this example, to validate whether `//pkg:foo_consumer` can see `//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an input to the call to `some_library` inside of `my_macro`, and instead check the dependency's visibility against the location of this call, `//macro`. -This process can repeat recursively, as long as a target or macro declaration is -inside of another symbolic macro taking the dependency's label in one of its -label-typed attributes. +This process can repeat recursively, as long as a target or macro declaration is inside of another symbolic macro taking the dependency's label in one of its label-typed attributes. -Note: Visibility delegation does not work for labels that were not passed into -the macro, such as labels derived by string manipulation. +Note: Visibility delegation does not work for labels that were not passed into the macro, such as labels derived by string manipulation. #### Finalizers -Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), -in addition to seeing targets following the usual symbolic macro visibility -rules, can *also* see all targets which are visible to the finalizer target's -package. +Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. -In other words, if you migrate a `native.existing_rules()`-based legacy macro to -a finalizer, the targets declared by the finalizer will still be able to see -their old dependencies. +In other words, if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. -It is possible to define targets that a finalizer can introspect using -`native.existing_rules()`, but which it cannot use as dependencies under the -visibility system. For example, if a macro-defined target is not visible to its -own package or to the finalizer macro's definition, and is not delegated to the -finalizer, the finalizer cannot see such a target. Note, however, that a -`native.existing_rules()`-based legacy macro will also be unable to see such a -target. +It is possible to define targets that a finalizer can introspect using `native.existing_rules()`, but which it cannot use as dependencies under the visibility system. For example, if a macro-defined target is not visible to its own package or to the finalizer macro's definition, and is not delegated to the finalizer, the finalizer cannot see such a target. Note, however, that a `native.existing_rules()`-based legacy macro will also be unable to see such a target. ## Load visibility -**Load visibility** controls whether a `.bzl` file may be loaded from other -`BUILD` or `.bzl` files outside the current package. +**Load visibility** controls whether a `.bzl` file may be loaded from other `BUILD` or `.bzl` files outside the current package. -In the same way that target visibility protects source code that is encapsulated -by targets, load visibility protects build logic that is encapsulated by `.bzl` -files. For instance, a `BUILD` file author might wish to factor some repetitive -target declarations into a macro in a `.bzl` file. Without the protection of -load visibility, they might find their macro reused by other collaborators in -the same workspace, so that modifying the macro breaks other teams' builds. +In the same way that target visibility protects source code that is encapsulated by targets, load visibility protects build logic that is encapsulated by `.bzl` files. For instance, a `BUILD` file author might wish to factor some repetitive target declarations into a macro in a `.bzl` file. Without the protection of load visibility, they might find their macro reused by other collaborators in the same workspace, so that modifying the macro breaks other teams' builds. -Note that a `.bzl` file may or may not have a corresponding source file target. -If it does, there is no guarantee that the load visibility and the target -visibility coincide. That is, the same `BUILD` file might be able to load the -`.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), -or vice versa. This can sometimes cause problems for rules that wish to consume -`.bzl` files as source code, such as for documentation generation or testing. +Note that a `.bzl` file may or may not have a corresponding source file target. If it does, there is no guarantee that the load visibility and the target visibility coincide. That is, the same `BUILD` file might be able to load the `.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), or vice versa. This can sometimes cause problems for rules that wish to consume `.bzl` files as source code, such as for documentation generation or testing. -For prototyping, you may disable load visibility enforcement by setting -`--check_bzl_visibility=false`. As with `--check_visibility=false`, this should -not be done for submitted code. +For prototyping, you may disable load visibility enforcement by setting `--check_bzl_visibility=false`. As with `--check_visibility=false`, this should not be done for submitted code. Load visibility is available as of Bazel 6.0. ### Declaring load visibility -To set the load visibility of a `.bzl` file, call the -[`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. -The argument to `visibility()` is a list of package specifications, just like -the [`packages`](/reference/be/functions#package_group.packages) attribute of -`package_group`. However, `visibility()` does not accept negative package -specifications. +To set the load visibility of a `.bzl` file, call the [`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. The argument to `visibility()` is a list of package specifications, just like the [`packages`](/reference/be/functions#package_group.packages) attribute of `package_group`. However, `visibility()` does not accept negative package specifications. -The call to `visibility()` must only occur once per file, at the top level (not -inside a function), and ideally immediately following the `load()` statements. +The call to `visibility()` must only occur once per file, at the top level (not inside a function), and ideally immediately following the `load()` statements. -Unlike target visibility, the default load visibility is always public. Files -that do not call `visibility()` are always loadable from anywhere in the -workspace. It is a good idea to add `visibility("private")` to the top of any -new `.bzl` file that is not specifically intended for use outside the package. +Unlike target visibility, the default load visibility is always public. Files that do not call `visibility()` are always loadable from anywhere in the workspace. It is a good idea to add `visibility("private")` to the top of any new `.bzl` file that is not specifically intended for use outside the package. ### Example @@ -480,8 +312,7 @@ This section describes tips for managing load visibility declarations. #### Factoring visibilities -When multiple `.bzl` files should have the same visibility, it can be helpful to -factor their package specifications into a common list. For example: +When multiple `.bzl` files should have the same visibility, it can be helpful to factor their package specifications into a common list. For example: ```starlark # //mylib/internal_defs.bzl @@ -513,18 +344,13 @@ visibility(clients) ... ``` -This helps prevent accidental skew between the various `.bzl` files' -visibilities. It also is more readable when the `clients` list is large. +This helps prevent accidental skew between the various `.bzl` files' visibilities. It also is more readable when the `clients` list is large. #### Composing visibilities -Sometimes a `.bzl` file might need to be visible to an allowlist that is -composed of multiple smaller allowlists. This is analogous to how a -`package_group` can incorporate other `package_group`s via its -[`includes`](/reference/be/functions#package_group.includes) attribute. +Sometimes a `.bzl` file might need to be visible to an allowlist that is composed of multiple smaller allowlists. This is analogous to how a `package_group` can incorporate other `package_group`s via its [`includes`](/reference/be/functions#package_group.includes) attribute. -Suppose you are deprecating a widely used macro. You want it to be visible only -to existing users and to the packages owned by your own team. You might write: +Suppose you are deprecating a widely used macro. You want it to be visible only to existing users and to the packages owned by your own team. You might write: ```starlark # //mylib/macros.bzl @@ -538,12 +364,7 @@ visibility(our_packages + their_remaining_uses) #### Deduplicating with package groups -Unlike target visibility, you cannot define a load visibility in terms of a -`package_group`. If you want to reuse the same allowlist for both target -visibility and load visibility, it's best to move the list of package -specifications into a .bzl file, where both kinds of declarations may refer to -it. Building off the example in [Factoring visibilities](#factoring-visibilities) -above, you might write: +Unlike target visibility, you cannot define a load visibility in terms of a `package_group`. If you want to reuse the same allowlist for both target visibility and load visibility, it's best to move the list of package specifications into a .bzl file, where both kinds of declarations may refer to it. Building off the example in [Factoring visibilities](#factoring-visibilities) above, you might write: ```starlark # //mylib/BUILD @@ -556,17 +377,11 @@ package_group( ) ``` -This only works if the list does not contain any negative package -specifications. +This only works if the list does not contain any negative package specifications. #### Protecting individual symbols -Any Starlark symbol whose name begins with an underscore cannot be loaded from -another file. This makes it easy to create private symbols, but does not allow -you to share these symbols with a limited set of trusted files. On the other -hand, load visibility gives you control over what other packages may see your -`.bzl file`, but does not allow you to prevent any non-underscored symbol from -being loaded. +Any Starlark symbol whose name begins with an underscore cannot be loaded from another file. This makes it easy to create private symbols, but does not allow you to share these symbols with a limited set of trusted files. On the other hand, load visibility gives you control over what other packages may see your `.bzl file`, but does not allow you to prevent any non-underscored symbol from being loaded. Luckily, you can combine these two features to get fine-grained control. @@ -603,8 +418,4 @@ public_util = _public_util #### bzl-visibility Buildifier lint -There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) -that provides a warning if users load a file from a directory named `internal` -or `private`, when the user's file is not itself underneath the parent of that -directory. This lint predates the load visibility feature and is unnecessary in -workspaces where `.bzl` files declare visibilities. +There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) that provides a warning if users load a file from a directory named `internal` or `private`, when the user's file is not itself underneath the parent of that directory. This lint predates the load visibility feature and is unnecessary in workspaces where `.bzl` files declare visibilities. diff --git a/configure/attributes.mdx b/configure/attributes.mdx index 7bc3f41e..df76c4ba 100644 --- a/configure/attributes.mdx +++ b/configure/attributes.mdx @@ -2,15 +2,9 @@ title: 'Configurable Build Attributes' --- +***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. - -**_Configurable attributes_**, commonly known as [`select()`]( -/reference/be/functions#select), is a Bazel feature that lets users toggle the values -of build rule attributes at the command line. - -This can be used, for example, for a multiplatform library that automatically -chooses the appropriate implementation for the architecture, or for a -feature-configurable binary that can be customized at build time. +This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. ## Example @@ -41,59 +35,30 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the -command line. Specifically, `deps` becomes: - -
and tags at end of lines (common in tables) -/]*>[[:space:]]*[^<[:space:]]+[[:space:]]*$/ { - # Only add closing tag if there isn't already one - if ($0 !~ /<\/td>$/) { - $0 = $0 "" - } -} - -# Be more careful with - don't match
- - - - - - - - - - - - - - - - - - - - -
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
- -`select()` serves as a placeholder for a value that will be chosen based on -*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) -targets. By using `select()` in a configurable attribute, the attribute -effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: + +| | | +| ----------------------------------------------- | ------------------ | +| Command | deps = | +| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | +| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | + +`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either -* They all resolve to the same value. For example, when running on linux x86, this is unambiguous - `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` - is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when -nothing else does. +- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. + +The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, -`resources`, `cmd`, and most other attributes. Only a small number of attributes -are *non-configurable*, and these are clearly annotated. For example, -`config_setting`'s own -[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies -under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of -the machine running Bazel (which, thanks to cross-compilation, may be different -than the CPU the target is built for). This is known as a -[configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). Given @@ -137,30 +102,19 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and -`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s -build parameters, which include `--cpu=arm`. The `tools` attribute changes -`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on -`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a -[`config_setting`](/reference/be/general#config_setting) or -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of -expected command line flag settings. By encapsulating these in a target, it's -easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands -them for all builds in all projects. These are specified with -[`config_setting`](/reference/be/general#config_setting)'s -[`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -173,31 +127,21 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` -is the expected value for that flag. `:meaningful_condition_name` matches if -*every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, -[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because -it only affects how Bazel reports progress to the user. Targets can't use that -flag to construct their results. The exact set of supported flags isn't -documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with -[Starlark build settings][BuildSettings]. Unlike built-in flags, these are -defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s -[`flag_values`](/reference/be/general#config_setting.flag_values) -attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: ```python config_setting( @@ -210,29 +154,17 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) -for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. -[`--define`](/reference/command-line-reference#flag--define) -is an alternative legacy syntax for custom flags (for example -`--define foo=bar`). This can be expressed either in the -[values](/reference/be/general#config_setting.values) attribute -(`values = {"define": "foo=bar"}`) or the -[define_values](/reference/be/general#config_setting.define_values) attribute -(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards -compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The -`config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition -matches. +The built-in condition `//conditions:default` matches when no other condition matches. -Because of the "exactly one match" rule, a configurable attribute with no match -and no default condition emits a `"no matching conditions"` error. This can -protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -258,16 +190,11 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s -[`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides -flexibility, it can also be burdensome to individually set each one every time -you want to build a target. - [Platforms](/extending/platforms) -let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -325,12 +252,9 @@ platform( ) ``` -The platform can be specified on the command line. It activates the -`config_setting`s that contain a subset of the platform's `constraint_values`, -allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, -you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -357,11 +281,9 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to -check against single values. +This saves the need for boilerplate `config_setting`s when you only need to check against single values. -Platforms are still under development. See the -[documentation](/concepts/platforms) for details. +Platforms are still under development. See the [documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -383,12 +305,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: - - Duplicate labels can appear in different paths of the same `select`. - - Duplicate labels can *not* appear within the same path of a `select`. - - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -`select` cannot appear inside another `select`. If you need to nest `selects` -and your attribute takes other targets as values, use an intermediate target: +- Duplicate labels can appear in different paths of the same `select`. +- Duplicate labels can *not* appear within the same path of a `select`. +- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) + +`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -409,8 +331,7 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND -chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). ## OR chaining @@ -429,9 +350,7 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and -maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple -times. +Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. One option is to predefine the value as a BUILD variable: @@ -450,18 +369,13 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary -duplication. +This makes it easier to manage the dependency. But it still causes unnecessary duplication. For more direct support, use one of the following: ### `selects.with_or` -The -[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing conditions directly inside a `select`: +The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -480,18 +394,12 @@ sh_binary( ### `selects.config_setting_group` - -The -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing multiple `config_setting`s: +The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` - ```python config_setting( name = "config1", @@ -515,17 +423,13 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across -different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. -It's an error for multiple conditions to match unless one is an unambiguous -"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the -[Skylib](https://github.com/bazelbuild/bazel-skylib) macro -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -550,13 +454,11 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed -inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to -fails with the error: +By default, when no condition matches, the target the `select()` is attached to fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -566,8 +468,7 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) -attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: ```python cc_library( @@ -590,8 +491,7 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable -attributes. For example, given: +Rule implementations receive the *resolved values* of configurable attributes. For example, given: ```python # myapp/BUILD @@ -611,9 +511,7 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native -rules. But *they cannot directly manipulate them*. For example, there's no way -for a macro to convert +Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert ```python select({"foo": "val"}, ...) @@ -627,32 +525,22 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* -because macros are evaluated in Bazel's [loading phase](/run/build#loading), -which occurs before flag values are known. -This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while -technically feasible, lack a coherent UI. Further design is necessary to change -this. +Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's -[loading phase](/reference/glossary#loading-phase). -This means it doesn't know what command line flags a target uses since those -flags aren't evaluated until later in the build (in the -[analysis phase](/reference/glossary#analysis-phase)). -So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has -all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` + ```python # myapp/BUILD @@ -701,14 +589,9 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for -details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. -The key issue this question usually means is that select() doesn't work in -*macros*. These are different than *rules*. See the -documentation on [rules](/extending/rules) and [macros](/extending/macros) -to understand the difference. -Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: Define a rule and macro: @@ -788,9 +671,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before -Bazel reads the build's command line flags. That means there isn't enough -information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -813,9 +694,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition -[can't evaluate `select()`s](#faq-select-macro), any attempt to do so -usually produces an error: +Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -828,8 +707,7 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly -vigilant with them: +Booleans are a special case that fail silently, so you should be particularly vigilant with them: ```sh $ cat myapp/defs.bzl @@ -851,21 +729,13 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. -So what they're really evaluting is the `select()` object itself. According to -[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design -standards, all objects aside from a very small number of exceptions -automatically return true. +This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before -Bazel knows what the build's command line parameters are. Can they at least read -the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but it isn't yet a Bazel feature. -What you *can* do today is prepare a straight dictionary, then feed it into a -`select()`: +Conceptually this is possible, but it isn't yet a Bazel feature. What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: ```sh $ cat myapp/defs.bzl @@ -877,7 +747,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -909,7 +779,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -917,17 +787,11 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo -rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in -the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't -actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in -the `actual` attribute, to perform this type of run-time determination. This -works correctly, since `alias()` is a BUILD rule, and is evaluated with a -specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. ```sh $ cat WORKSPACE @@ -953,37 +817,31 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target -that depends on either `//:ssl` or `//external:ssl` will see the alternative -located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, -use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo +$ bazel cquery //myapp:foo <desired build flags> //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on -//myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' +$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the -configuration that resolves `//myapp:foo`'s `select()`. You can inspect its -values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -1002,18 +860,13 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the -[cquery docs](/query/cquery) for guidance on using `somepath` to get the right -one. +`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the -same command line flags as the `bazel cquery`. The `config` command relies on -the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform -is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. For example: @@ -1036,15 +889,11 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the -`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the -`:x86_linux_platform` defined here? The author of the `BUILD` file and the user -who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with -these constraints: +Instead, define a `config_setting` that matches **any** platform with these constraints: ```py config_setting( @@ -1065,13 +914,11 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what -platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you -can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -1091,7 +938,4 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and -confuses users when the expected condition does not match. - -[BuildSettings]: /extending/config#user-defined-build-settings +The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. diff --git a/configure/best-practices.mdx b/configure/best-practices.mdx index deecf9dd..90eec9fc 100644 --- a/configure/best-practices.mdx +++ b/configure/best-practices.mdx @@ -2,10 +2,7 @@ title: 'Best Practices' --- - - -This page assumes you are familiar with Bazel and provides guidelines and -advice on structuring your projects to take full advantage of Bazel's features. +This page assumes you are familiar with Bazel and provides guidelines and advice on structuring your projects to take full advantage of Bazel's features. The overall goals are: @@ -14,81 +11,45 @@ The overall goals are: - To make code well-structured and testable. - To create a build configuration that is easy to understand and maintain. -These guidelines are not requirements: few projects will be able to adhere to -all of them. As the man page for lint says, "A special reward will be presented -to the first person to produce a real program that produces no errors with -strict checking." However, incorporating as many of these principles as possible -should make a project more readable, less error-prone, and faster to build. +These guidelines are not requirements: few projects will be able to adhere to all of them. As the man page for lint says, "A special reward will be presented to the first person to produce a real program that produces no errors with strict checking." However, incorporating as many of these principles as possible should make a project more readable, less error-prone, and faster to build. -This page uses the requirement levels described in -[this RFC](https://www.ietf.org/rfc/rfc2119.txt). +This page uses the requirement levels described in [this RFC](https://www.ietf.org/rfc/rfc2119.txt). ## Running builds and tests -A project should always be able to run `bazel build //...` and -`bazel test //...` successfully on its stable branch. Targets that are necessary -but do not build under certain circumstances (such as,require specific build -flags, don't build on a certain platform, require license agreements) should be -tagged as specifically as possible (for example, "`requires-osx`"). This -tagging allows targets to be filtered at a more fine-grained level than the -"manual" tag and allows someone inspecting the `BUILD` file to understand what -a target's restrictions are. +A project should always be able to run `bazel build //...` and `bazel test //...` successfully on its stable branch. Targets that are necessary but do not build under certain circumstances (such as,require specific build flags, don't build on a certain platform, require license agreements) should be tagged as specifically as possible (for example, "`requires-osx`"). This tagging allows targets to be filtered at a more fine-grained level than the "manual" tag and allows someone inspecting the `BUILD` file to understand what a target's restrictions are. ## Third-party dependencies You may declare third-party dependencies: -* Either declare them as remote repositories in the `MODULE.bazel` file. -* Or put them in a directory called `third_party/` under your workspace directory. +- Either declare them as remote repositories in the `MODULE.bazel` file. +- Or put them in a directory called `third_party/` under your workspace directory. ## Depending on binaries -Everything should be built from source whenever possible. Generally this means -that, instead of depending on a library `some-library.so`, you'd create a -`BUILD` file and build `some-library.so` from its sources, then depend on that -target. +Everything should be built from source whenever possible. Generally this means that, instead of depending on a library `some-library.so`, you'd create a `BUILD` file and build `some-library.so` from its sources, then depend on that target. -Always building from source ensures that a build is not using a library that -was built with incompatible flags or a different architecture. There are also -some features like coverage, static analysis, or dynamic analysis that only -work on the source. +Always building from source ensures that a build is not using a library that was built with incompatible flags or a different architecture. There are also some features like coverage, static analysis, or dynamic analysis that only work on the source. ## Versioning -Prefer building all code from head whenever possible. When versions must be -used, avoid including the version in the target name (for example, `//guava`, -not `//guava-20.0`). This naming makes the library easier to update (only one -target needs to be updated). It's also more resilient to diamond dependency -issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, -you could end up with a library that tries to depend on two different versions. -If you created a misleading alias to point both targets to one `guava` library, -then the `BUILD` files are misleading. +Prefer building all code from head whenever possible. When versions must be used, avoid including the version in the target name (for example, `//guava`, not `//guava-20.0`). This naming makes the library easier to update (only one target needs to be updated). It's also more resilient to diamond dependency issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, you could end up with a library that tries to depend on two different versions. If you created a misleading alias to point both targets to one `guava` library, then the `BUILD` files are misleading. ## Using the `.bazelrc` file -For project-specific options, use the configuration file your -`workspace/.bazelrc` (see [bazelrc format](/run/bazelrc)). +For project-specific options, use the configuration file your `<var>workspace</var>/.bazelrc` (see [bazelrc format](/run/bazelrc)). -If you want to support per-user options for your project that you **do not** -want to check into source control, include the line: +If you want to support per-user options for your project that you **do not** want to check into source control, include the line: ``` try-import %workspace%/user.bazelrc ``` -(or any other file name) in your `workspace/.bazelrc` -and add `user.bazelrc` to your `.gitignore`. -The open-source -[bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) +(or any other file name) in your `<var>workspace</var>/.bazelrc` and add `user.bazelrc` to your `.gitignore`. -generates a custom bazelrc file that matches your Bazel version and provides a -preset of recommended flags. +The open-source [bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) generates a custom bazelrc file that matches your Bazel version and provides a preset of recommended flags. ## Packages -Every directory that contains buildable files should be a package. If a `BUILD` -file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's -a sign that a `BUILD` file should be added to that subdirectory. The longer -this structure exists, the more likely circular dependencies will be -inadvertently created, a target's scope will creep, and an increasing number -of reverse dependencies will have to be updated. +Every directory that contains buildable files should be a package. If a `BUILD` file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's a sign that a `BUILD` file should be added to that subdirectory. The longer this structure exists, the more likely circular dependencies will be inadvertently created, a target's scope will creep, and an increasing number of reverse dependencies will have to be updated. diff --git a/configure/coverage.mdx b/configure/coverage.mdx index 03a8ab9b..72cde9c5 100644 --- a/configure/coverage.mdx +++ b/configure/coverage.mdx @@ -2,129 +2,67 @@ title: 'Code coverage with Bazel' --- +Bazel features a `coverage` sub-command to produce code coverage reports on repositories that can be tested with `bazel coverage`. Due to the idiosyncrasies of the various language ecosystems, it is not always trivial to make this work for a given project. +This page documents the general process for creating and viewing coverage reports, and also features some language-specific notes for languages whose configuration is well-known. It is best read by first reading [the general section](#creating-a-coverage-report), and then reading about the requirements for a specific language. Note also the [remote execution section](#remote-execution), which requires some additional considerations. -Bazel features a `coverage` sub-command to produce code coverage -reports on repositories that can be tested with `bazel coverage`. Due -to the idiosyncrasies of the various language ecosystems, it is not -always trivial to make this work for a given project. - -This page documents the general process for creating and viewing -coverage reports, and also features some language-specific notes for -languages whose configuration is well-known. It is best read by first -reading [the general section](#creating-a-coverage-report), and then -reading about the requirements for a specific language. Note also the -[remote execution section](#remote-execution), which requires some -additional considerations. - -While a lot of customization is possible, this document focuses on -producing and consuming [`lcov`][lcov] reports, which is currently the -most well-supported route. +While a lot of customization is possible, this document focuses on producing and consuming [`lcov`](https://github.com/linux-test-project/lcov) reports, which is currently the most well-supported route. ## Creating a coverage report ### Preparation -The basic workflow for creating coverage reports requires the -following: +The basic workflow for creating coverage reports requires the following: - A basic repository with test targets - A toolchain with the language-specific code coverage tools installed - A correct "instrumentation" configuration -The former two are language-specific and mostly straightforward, -however the latter can be more difficult for complex projects. +The former two are language-specific and mostly straightforward, however the latter can be more difficult for complex projects. -"Instrumentation" in this case refers to the coverage tools that are -used for a specific target. Bazel allows turning this on for a -specific subset of files using the -[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) -flag, which specifies a filter for targets that are tested with the -instrumentation enabled. To enable instrumentation for tests, the -[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) -flag is required. +"Instrumentation" in this case refers to the coverage tools that are used for a specific target. Bazel allows turning this on for a specific subset of files using the [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) flag, which specifies a filter for targets that are tested with the instrumentation enabled. To enable instrumentation for tests, the [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) flag is required. -By default, bazel tries to match the target package(s), and prints the -relevant filter as an `INFO` message. +By default, bazel tries to match the target package(s), and prints the relevant filter as an `INFO` message. ### Running coverage -To produce a coverage report, use [`bazel coverage ---combined_report=lcov -[target]`](/reference/command-line-reference#coverage). This runs the -tests for the target, generating coverage reports in the lcov format -for each file. +To produce a coverage report, use [`bazel coverage --combined_report=lcov [target]`](/reference/command-line-reference#coverage). This runs the tests for the target, generating coverage reports in the lcov format for each file. -Once finished, bazel runs an action that collects all the produced -coverage files, and merges them into one, which is then finally -created under `$(bazel info -output_path)/_coverage/_coverage_report.dat`. +Once finished, bazel runs an action that collects all the produced coverage files, and merges them into one, which is then finally created under `$(bazel info output_path)/_coverage/_coverage_report.dat`. -Coverage reports are also produced if tests fail, though note that -this does not extend to the failed tests - only passing tests are -reported. +Coverage reports are also produced if tests fail, though note that this does not extend to the failed tests - only passing tests are reported. ### Viewing coverage -The coverage report is only output in the non-human-readable `lcov` -format. From this, we can use the `genhtml` utility (part of [the lcov -project][lcov]) to produce a report that can be viewed in a web -browser: +The coverage report is only output in the non-human-readable `lcov` format. From this, we can use the `genhtml` utility (part of [the lcov project](https://github.com/linux-test-project/lcov)) to produce a report that can be viewed in a web browser: ```console genhtml --branch-coverage --output genhtml "$(bazel info output_path)/_coverage/_coverage_report.dat" ``` -Note that `genhtml` reads the source code as well, to annotate missing -coverage in these files. For this to work, it is expected that -`genhtml` is executed in the root of the bazel project. +Note that `genhtml` reads the source code as well, to annotate missing coverage in these files. For this to work, it is expected that `genhtml` is executed in the root of the bazel project. -To view the result, simply open the `index.html` file produced in the -`genhtml` directory in any web browser. +To view the result, simply open the `index.html` file produced in the `genhtml` directory in any web browser. -For further help and information around the `genhtml` tool, or the -`lcov` coverage format, see [the lcov project][lcov]. +For further help and information around the `genhtml` tool, or the `lcov` coverage format, see [the lcov project](https://github.com/linux-test-project/lcov). ## Remote execution Running with remote test execution currently has a few caveats: -- The report combination action cannot yet run remotely. This is - because Bazel does not consider the coverage output files as part of - its graph (see [this issue][remote_report_issue]), and can therefore - not correctly treat them as inputs to the combination action. To - work around this, use `--strategy=CoverageReport=local`. - - Note: It may be necessary to specify something like - `--strategy=CoverageReport=local,remote` instead, if Bazel is set - up to try `local,remote`, due to how Bazel resolves strategies. -- `--remote_download_minimal` and similar flags can also not be used - as a consequence of the former. -- Bazel will currently fail to create coverage information if tests - have been cached previously. To work around this, - `--nocache_test_results` can be set specifically for coverage runs, - although this of course incurs a heavy cost in terms of test times. -- `--experimental_split_coverage_postprocessing` and - `--experimental_fetch_all_coverage_outputs` - - Usually coverage is run as part of the test action, and so by - default, we don't get all coverage back as outputs of the remote - execution by default. These flags override the default and obtain - the coverage data. See [this issue][split_coverage_issue] for more - details. +- The report combination action cannot yet run remotely. This is because Bazel does not consider the coverage output files as part of its graph (see [this issue](https://github.com/bazelbuild/bazel/issues/4685)), and can therefore not correctly treat them as inputs to the combination action. To work around this, use `--strategy=CoverageReport=local`. + - Note: It may be necessary to specify something like `--strategy=CoverageReport=local,remote` instead, if Bazel is set up to try `local,remote`, due to how Bazel resolves strategies. +- `--remote_download_minimal` and similar flags can also not be used as a consequence of the former. +- Bazel will currently fail to create coverage information if tests have been cached previously. To work around this, `--nocache_test_results` can be set specifically for coverage runs, although this of course incurs a heavy cost in terms of test times. +- `--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs` + - Usually coverage is run as part of the test action, and so by default, we don't get all coverage back as outputs of the remote execution by default. These flags override the default and obtain the coverage data. See [this issue](https://github.com/bazelbuild/bazel/issues/4685) for more details. ## Language-specific configuration ### Java -Java should work out-of-the-box with the default configuration. The -[bazel toolchains][bazel_toolchains] contain everything necessary for -remote execution, as well, including JUnit. +Java should work out-of-the-box with the default configuration. The [bazel toolchains](https://github.com/bazelbuild/bazel-toolchains) contain everything necessary for remote execution, as well, including JUnit. ### Python -See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) -for additional steps needed to enable coverage support in Python. - -[lcov]: https://github.com/linux-test-project/lcov -[bazel_toolchains]: https://github.com/bazelbuild/bazel-toolchains -[remote_report_issue]: https://github.com/bazelbuild/bazel/issues/4685 -[split_coverage_issue]: https://github.com/bazelbuild/bazel/issues/4685 +See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) for additional steps needed to enable coverage support in Python. diff --git a/configure/integrate-cpp.mdx b/configure/integrate-cpp.mdx new file mode 100644 index 00000000..1b9aac1f --- /dev/null +++ b/configure/integrate-cpp.mdx @@ -0,0 +1,37 @@ +--- +title: 'Integrating with C++ Rules' +--- + +This page describes how to integrate with C++ rules on various levels. + +## Accessing the C++ toolchain + +You should use the helper functions available at [`@rules_cc//cc:find_cc_toolchain.bzl`](https://github.com/bazelbuild/rules_cc/blob/main/cc/find_cc_toolchain.bzl) to depend on a CC toolchain from a Starlark rule. + +To depend on a C++ toolchain in your rule, set the `toolchains` parameter to `use_cc_toolchain()`. Then, in the rule implementation, use `find_cpp_toolchain(ctx)` to get the [`CcToolchainInfo`](/rules/lib/providers/CcToolchainInfo). A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/write_cc_toolchain_cpu/write_cc_toolchain_cpu.bzl). + +## Generating command lines and environment variables using the C++ toolchain + +Typically, you would integrate with the C++ toolchain to have the same command line flags as C++ rules do, but without using C++ actions directly. This is because when writing our own actions, they must behave consistently with the C++ toolchain - for example, passing C++ command line flags to a tool that invokes the C++ compiler behind the scenes. + +C++ rules use a special way of constructing command lines based on [feature configuration](/docs/cc-toolchain-config-reference). To construct a command line, you need the following: + +- `features` and `action_configs` - these come from the `CcToolchainConfigInfo` and encapsulated in `CcToolchainInfo` +- `FeatureConfiguration` - returned by [cc\_common.configure\_features](/rules/lib/toplevel/cc_common#configure_features) +- cc toolchain config variables - returned by [cc\_common.create\_compile\_variables](/rules/lib/toplevel/cc_common#create_compile_variables) or [cc\_common.create\_link\_variables](/rules/lib/toplevel/cc_common#create_link_variables). + +There still are tool-specific getters, such as [compiler\_executable](/rules/lib/providers/CcToolchainInfo#compiler_executable). Prefer `get_tool_for_action` over these, as tool-specific getters will eventually be removed. + +A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/my_c_compile/my_c_compile.bzl). + +## Implementing Starlark rules that depend on C++ rules and/or that C++ rules can depend on + +Most C++ rules provide [`CcInfo`](/rules/lib/providers/CcInfo), a provider containing [`CompilationContext`](/rules/lib/builtins/CompilationContext) and [`LinkingContext`](/rules/lib/builtins/LinkingContext). Through these it is possible to access information such as all transitive headers or libraries to link. From `CcInfo` and from the `CcToolchainInfo` custom Starlark rules should be able to get all the information they need. + +If a custom Starlark rule provides `CcInfo`, it's a signal to the C++ rules that they can also depend on it. Be careful, however - if you only need to propagate `CcInfo` through the graph to the binary rule that then makes use of it, wrap `CcInfo` in a different provider. For example, if `java_library` rule wanted to propagate native dependencies up to the `java_binary`, it shouldn't provide `CcInfo` directly (`cc_binary` depending on `java_library` doesn't make sense), it should wrap it in, for example, `JavaCcInfo`. + +A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/my_c_archive/my_c_archive.bzl). + +## Reusing logic and actions of C++ rules + +*Not stable yet; This section will be updated once the API stabilizes. Follow [#4570](https://github.com/bazelbuild/bazel/issues/4570) for up-to-date information.* diff --git a/configure/windows.mdx b/configure/windows.mdx index 2dbb90ee..566c425d 100644 --- a/configure/windows.mdx +++ b/configure/windows.mdx @@ -2,24 +2,17 @@ title: 'Using Bazel on Windows' --- - - -This page covers Best Practices for using Bazel on Windows. For installation -instructions, see [Install Bazel on Windows](/install/windows). +This page covers Best Practices for using Bazel on Windows. For installation instructions, see [Install Bazel on Windows](/install/windows). ## Known issues -Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. -[GitHub-Windows]. - -[GitHub-Windows]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows +Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. [GitHub-Windows](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows). ## Best practices ### Avoid long path issues -Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. -To avoid hitting this issue, you can specify a short output directory for Bazel by the [\-\-output_user_root](/reference/command-line-reference#flag--output_user_root) flag. +Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. To avoid hitting this issue, you can specify a short output directory for Bazel by the [--output\_user\_root](/reference/command-line-reference#flag--output_user_root) flag. For example, add the following line to your bazelrc file: @@ -29,14 +22,10 @@ startup --output_user_root=C:/tmp ### Enable symlink support -Some features require Bazel to be able to create file symlinks on Windows, -either by enabling -[Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) -(on Windows 10 version 1703 or newer), or by running Bazel as an administrator. -This enables the following features: +Some features require Bazel to be able to create file symlinks on Windows, either by enabling [Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) (on Windows 10 version 1703 or newer), or by running Bazel as an administrator. This enables the following features: -* [\-\-windows_enable_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) -* [\-\-enable_runfiles](/reference/command-line-reference#flag--enable_runfiles) +- [--windows\_enable\_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) +- [--enable\_runfiles](/reference/command-line-reference#flag--enable_runfiles) To make it easier, add the following lines to your bazelrc file: @@ -48,23 +37,11 @@ build --enable_runfiles **Note**: Creating symlinks on Windows is an expensive operation. The `--enable_runfiles` flag can potentially create a large amount of file symlinks. Only enable this feature when you need it. -{/* TODO(pcloudy): https://github.com/bazelbuild/bazel/issues/6402 - Write a doc about runfiles library and add a link to it here */} - ### Running Bazel: MSYS2 shell vs. command prompt vs. PowerShell -**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from -PowerShell. +**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from PowerShell. -As of 2020-01-15, **do not** run Bazel from `bash` -- either -from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel -may work for most use cases, some things are broken, like -[interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). -Also, if you choose to run under MSYS2, you need to disable MSYS2's -automatic path conversion, otherwise MSYS will convert command line arguments -that _look like_ Unix paths (such as `//foo:bar`) into Windows paths. See -[this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) -for details. +As of 2020-01-15, **do not** run Bazel from `bash` -- either from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel may work for most use cases, some things are broken, like [interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). Also, if you choose to run under MSYS2, you need to disable MSYS2's automatic path conversion, otherwise MSYS will convert command line arguments that *look like* Unix paths (such as `//foo:bar`) into Windows paths. See [this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) for details. ### Using Bazel without Bash (MSYS2) @@ -78,13 +55,7 @@ Starting with Bazel 1.0, you can build any rule without Bash unless it is a: - `sh_binary` or `sh_test` rule, because these inherently need Bash - Starlark rule that uses `ctx.actions.run_shell()` or `ctx.resolve_command()` -However, `genrule` is often used for simple tasks like -[copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) -or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). -Instead of using `genrule` (and depending on Bash) you may find a suitable rule -in the -[bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). -When built on Windows, **these rules do not require Bash**. +However, `genrule` is often used for simple tasks like [copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). Instead of using `genrule` (and depending on Bash) you may find a suitable rule in the [bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). When built on Windows, **these rules do not require Bash**. #### Using bazel test without Bash @@ -104,24 +75,15 @@ Starting with Bazel 1.0, you can run any rule without Bash, except when: - you use `--run_under` or `--script_path` - the test rule itself requires Bash (because its executable is a shell script) -#### Using sh\_binary and sh\_* rules, and ctx.actions.run_shell() without Bash +#### Using sh\_binary and sh\_\* rules, and ctx.actions.run\_shell() without Bash -You need Bash to build and test `sh_*` rules, and to build and test Starlark -rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This -applies not only to rules in your project, but to rules in any of the external -repositories your project depends on (even transitively). +You need Bash to build and test `sh_*` rules, and to build and test Starlark rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This applies not only to rules in your project, but to rules in any of the external repositories your project depends on (even transitively). -In the future, there may be an option to use Windows Subsystem for -Linux (WSL) to build these rules, but currently it is not a priority for -the Bazel-on-Windows subteam. +In the future, there may be an option to use Windows Subsystem for Linux (WSL) to build these rules, but currently it is not a priority for the Bazel-on-Windows subteam. ### Setting environment variables -Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only -set in that command prompt session. If you start a new `cmd.exe`, you need to -set the variables again. To always set the variables when `cmd.exe` starts, you -can add them to the User variables or System variables in the `Control Panel > -System Properties > Advanced > Environment Variables...` dialog box. +Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only set in that command prompt session. If you start a new `cmd.exe`, you need to set the variables again. To always set the variables when `cmd.exe` starts, you can add them to the User variables or System variables in the `Control Panel > System Properties > Advanced > Environment Variables...` dialog box. ## Build on Windows @@ -129,71 +91,54 @@ System Properties > Advanced > Environment Variables...` dialog box. To build C++ targets with MSVC, you need: -* [The Visual C++ compiler](/install/windows#install-vc). +- [The Visual C++ compiler](/install/windows#install-vc). -* (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. +- (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. - Bazel automatically detects the Visual C++ compiler on your system. - To tell Bazel to use a specific VC installation, you can set the - following environment variables: + Bazel automatically detects the Visual C++ compiler on your system. To tell Bazel to use a specific VC installation, you can set the following environment variables: - For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. + For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. - * `BAZEL_VC` the Visual C++ Build Tools installation directory + - `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC + ``` - * `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version - number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools - version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel - will choose the latest version. + - `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel will choose the latest version. - ``` - set BAZEL_VC_FULL_VERSION=14.16.27023 - ``` + ``` + set BAZEL_VC_FULL_VERSION=14.16.27023 + ``` - For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) + For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) - * `BAZEL_VC` the Visual C++ Build Tools installation directory + - `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC + ``` -* The [Windows - SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). +- The [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). - The Windows SDK contains header files and libraries you need when building - Windows applications, including Bazel itself. By default, the latest Windows SDK installed will - be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You - can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 - SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified - Windows SDK installed. + The Windows SDK contains header files and libraries you need when building Windows applications, including Bazel itself. By default, the latest Windows SDK installed will be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified Windows SDK installed. - **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't - support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise - `BAZEL_WINSDK_FULL_VERSION` will be ignored. + **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise `BAZEL_WINSDK_FULL_VERSION` will be ignored. - ``` - set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 - ``` + ``` + set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 + ``` If everything is set up, you can build a C++ target now! -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` -bazel build //examples/cpp:hello-world -bazel-bin\examples\cpp\hello-world.exe +bazel build //examples/cpp:hello-world +bazel-bin\examples\cpp\hello-world.exe ``` -By default, the built binaries target x64 architecture. To build for ARM64 -architecture, use +By default, the built binaries target x64 architecture. To build for ARM64 architecture, use ```none --platforms=//:windows_arm64 --extra_toolchains=@local_config_cc//:cc-toolchain-arm64_windows @@ -207,111 +152,88 @@ cc_configure = use_extension("@rules_cc//cc:extensions.bzl", "cc_configure_exten use_repo(cc_configure, "local_config_cc") ``` -To build and use Dynamically Linked Libraries (DLL files), see [this -example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). +To build and use Dynamically Linked Libraries (DLL files), see [this example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). -**Command Line Length Limit**: To prevent the -[Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), -enable the compiler parameter file feature via `--features=compiler_param_file`. +**Command Line Length Limit**: To prevent the [Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), enable the compiler parameter file feature via `--features=compiler_param_file`. ### Build C++ with Clang From 0.29.0, Bazel supports building with LLVM's MSVC-compatible compiler driver (`clang-cl.exe`). -**Requirement**: To build with Clang, you have to install **both** -[LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, -because although you use `clang-cl.exe` as compiler, you still need to link to -Visual C++ libraries. +**Requirement**: To build with Clang, you have to install **both** [LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, because although you use `clang-cl.exe` as compiler, you still need to link to Visual C++ libraries. -Bazel can automatically detect LLVM installation on your system, or you can explicitly tell -Bazel where LLVM is installed by `BAZEL_LLVM`. +Bazel can automatically detect LLVM installation on your system, or you can explicitly tell Bazel where LLVM is installed by `BAZEL_LLVM`. -* `BAZEL_LLVM` the LLVM installation directory +- `BAZEL_LLVM` the LLVM installation directory - ```posix-terminal - set BAZEL_LLVM=C:\Program Files\LLVM - ``` + ```posix-terminal + set BAZEL_LLVM=C:\Program Files\LLVM + ``` To enable the Clang toolchain for building C++, there are several situations. -* In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the - top level `BUILD` file): +- In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the top level `BUILD` file): - ``` - platform( - name = "x64_windows-clang-cl", - constraint_values = [ - "@platforms//cpu:x86_64", - "@platforms//os:windows", - "@bazel_tools//tools/cpp:clang-cl", - ], - ) - ``` + ``` + platform( + name = "x64_windows-clang-cl", + constraint_values = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@bazel_tools//tools/cpp:clang-cl", + ], + ) + ``` - Then enable the Clang toolchain by specifying the following build flags: + Then enable the Clang toolchain by specifying the following build flags: - ``` - --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl - ``` + ``` + --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl + ``` -* In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by - a build flag `--compiler=clang-cl`. +- In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by a build flag `--compiler=clang-cl`. - If your build sets the flag - [\-\-incompatible_enable_cc_toolchain_resolution] - (https://github.com/bazelbuild/bazel/issues/7260) - to `true`, then use the approach for Bazel 7.0.0. + If your build sets the flag [--incompatible\_enable\_cc\_toolchain\_resolution](https://github.com/bazelbuild/bazel/issues/7260) to `true`, then use the approach for Bazel 7.0.0. -* In Bazel 0.28 and older: Clang is not supported. +- In Bazel 0.28 and older: Clang is not supported. ### Build Java To build Java targets, you need: -* [The Java SE Development Kit](/install/windows#install-jdk) +- [The Java SE Development Kit](/install/windows#install-jdk) On Windows, Bazel builds two output files for `java_binary` rules: -* a `.jar` file -* a `.exe` file that can set up the environment for the JVM and run the binary +- a `.jar` file +- a `.exe` file that can set up the environment for the JVM and run the binary -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world - bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe + bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world + bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe ``` ### Build Python To build Python targets, you need: -* The [Python interpreter](/install/windows#install-python) +- The [Python interpreter](/install/windows#install-python) On Windows, Bazel builds two output files for `py_binary` rules: -* a self-extracting zip file -* an executable file that can launch the Python interpreter with the - self-extracting zip file as the argument +- a self-extracting zip file +- an executable file that can launch the Python interpreter with the self-extracting zip file as the argument -You can either run the executable file (it has a `.exe` extension) or you can run -Python with the self-extracting zip file as the argument. +You can either run the executable file (it has a `.exe` extension) or you can run Python with the self-extracting zip file as the argument. -Try building a target from one of our [sample -projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/py_native:bin - bazel-bin\examples\py_native\bin.exe - python bazel-bin\examples\py_native\bin.zip + bazel build //examples/py_native:bin + bazel-bin\examples\py_native\bin.exe + python bazel-bin\examples\py_native\bin.zip ``` -If you are interested in details about how Bazel builds Python targets on -Windows, check out this [design -doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). +If you are interested in details about how Bazel builds Python targets on Windows, check out this [design doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). diff --git a/contribute/breaking-changes.mdx b/contribute/breaking-changes.mdx index 5dda1b9d..c30b163a 100644 --- a/contribute/breaking-changes.mdx +++ b/contribute/breaking-changes.mdx @@ -2,65 +2,41 @@ title: 'Guide for rolling out breaking changes' --- - - -It is inevitable that we will make breaking changes to Bazel. We will have to -change our designs and fix the things that do not quite work. However, we need -to make sure that community and Bazel ecosystem can follow along. To that end, -Bazel project has adopted a -[backward compatibility policy](/release/backward-compatibility). -This document describes the process for Bazel contributors to make a breaking -change in Bazel to adhere to this policy. +It is inevitable that we will make breaking changes to Bazel. We will have to change our designs and fix the things that do not quite work. However, we need to make sure that community and Bazel ecosystem can follow along. To that end, Bazel project has adopted a [backward compatibility policy](/release/backward-compatibility). This document describes the process for Bazel contributors to make a breaking change in Bazel to adhere to this policy. 1. Follow the [design document policy](/contribute/design-documents). -1. [File a GitHub issue.](#github-issue) +2. [File a GitHub issue.](#github-issue) -1. [Implement the change.](#implementation) +3. [Implement the change.](#implementation) -1. [Update labels.](#labels) +4. [Update labels.](#labels) -1. [Update repositories.](#update-repos) +5. [Update repositories.](#update-repos) -1. [Flip the incompatible flag.](#flip-flag) +6. [Flip the incompatible flag.](#flip-flag) ## GitHub issue -[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) -in the Bazel repository. -[See example.](https://github.com/bazelbuild/bazel/issues/6611) +[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) in the Bazel repository. [See example.](https://github.com/bazelbuild/bazel/issues/6611) We recommend that: -* The title starts with the name of the flag (the flag name will start with - `incompatible_`). +- The title starts with the name of the flag (the flag name will start with `incompatible_`). -* You add the label - [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). +- You add the label [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). -* The description contains a description of the change and a link to relevant - design documents. +- The description contains a description of the change and a link to relevant design documents. -* The description contains a migration recipe, to explain users how they should - update their code. Ideally, when the change is mechanical, include a link to a - migration tool. +- The description contains a migration recipe, to explain users how they should update their code. Ideally, when the change is mechanical, include a link to a migration tool. -* The description includes an example of the error message users will get if - they don't migrate. This will make the GitHub issue more discoverable from - search engines. Make sure that the error message is helpful and actionable. - When possible, the error message should include the name of the incompatible - flag. +- The description includes an example of the error message users will get if they don't migrate. This will make the GitHub issue more discoverable from search engines. Make sure that the error message is helpful and actionable. When possible, the error message should include the name of the incompatible flag. -For the migration tool, consider contributing to -[Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). -It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. -It may also report warnings. +For the migration tool, consider contributing to [Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. It may also report warnings. ## Implementation -Create a new flag in Bazel. The default value must be false. The help text -should contain the URL of the GitHub issue. As the flag name starts with -`incompatible_`, it needs metadata tags: +Create a new flag in Bazel. The default value must be false. The help text should contain the URL of the GitHub issue. As the flag name starts with `incompatible_`, it needs metadata tags: ```java metadataTags = { @@ -68,80 +44,60 @@ should contain the URL of the GitHub issue. As the flag name starts with }, ``` -In the commit description, add a brief summary of the flag. -Also add [`RELNOTES:`](release-notes.md) in the following form: -`RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` +In the commit description, add a brief summary of the flag. Also add [`RELNOTES:`](release-notes.md) in the following form: `RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` -The commit should also update the relevant documentation, so that there is no -window of commits in which the code is inconsistent with the docs. Since our -documentation is versioned, changes to the docs will not be inadvertently -released prematurely. +The commit should also update the relevant documentation, so that there is no window of commits in which the code is inconsistent with the docs. Since our documentation is versioned, changes to the docs will not be inadvertently released prematurely. ## Labels -Once the commit is merged and the incompatible change is ready to be adopted, add the label -[`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) -to the GitHub issue. +Once the commit is merged and the incompatible change is ready to be adopted, add the label [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) to the GitHub issue. -If a problem is found with the flag and users are not expected to migrate yet: -remove the flags `migration-ready`. +If a problem is found with the flag and users are not expected to migrate yet: remove the flags `migration-ready`. -If you plan to flip the flag in the next major release, add label `breaking-change-X.0" to the issue. +If you plan to flip the flag in the next major release, add label \`breaking-change-X.0" to the issue. ## Updating repositories -Bazel CI tests a list of important projects at -[Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often -dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). -Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). +Bazel CI tests a list of important projects at [Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). Our dev support team monitors the [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) label. Once you add this label to the GitHub issue, they will handle the following: 1. Create a comment in the GitHub issue to track the list of failures and downstream projects that need to be migrated ([see example](https://github.com/bazelbuild/bazel/issues/17032#issuecomment-1353077469)) -1. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) +2. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) -1. Follow up to make sure all issues are addressed before the target release date +3. Follow up to make sure all issues are addressed before the target release date Migrating projects in the downstream pipeline is NOT entirely the responsibility of the incompatible change author, but you can do the following to accelerate the migration and make life easier for both Bazel users and the Bazel Green Team. 1. Send PRs to fix downstream projects. -1. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). +2. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). ## Flipping the flag Before flipping the default value of the flag to true, please make sure that: -* Core repositories in the ecosystem are migrated. +- Core repositories in the ecosystem are migrated. - On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), - the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. + On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. -* All issues in the checklist are marked as fixed/closed. +- All issues in the checklist are marked as fixed/closed. -* User concerns and questions have been resolved. +- User concerns and questions have been resolved. When the flag is ready to flip in Bazel, but blocked on internal migration at Google, please consider setting the flag value to false in the internal `blazerc` file to unblock the flag flip. By doing this, we can ensure Bazel users depend on the new behaviour by default as early as possible. When changing the flag default to true, please: -* Use `RELNOTES[INC]` in the commit description, with the - following format: - `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for - details` - You can include additional information in the rest of the commit description. -* Use `Fixes #xyz` in the description, so that the GitHub issue gets closed - when the commit is merged. -* Review and update documentation if needed. -* File a new issue `#abc` to track the removal of the flag. +- Use `RELNOTES[INC]` in the commit description, with the following format: `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for details` You can include additional information in the rest of the commit description. +- Use `Fixes #xyz` in the description, so that the GitHub issue gets closed when the commit is merged. +- Review and update documentation if needed. +- File a new issue `#abc` to track the removal of the flag. ## Removing the flag -After the flag is flipped at HEAD, it should be removed from Bazel eventually. -When you plan to remove the incompatible flag: +After the flag is flipped at HEAD, it should be removed from Bazel eventually. When you plan to remove the incompatible flag: -* Consider leaving more time for users to migrate if it's a major incompatible change. - Ideally, the flag should be available in at least one major release. -* For the commit that removes the flag, use `Fixes #abc` in the description - so that the GitHub issue gets closed when the commit is merged. +- Consider leaving more time for users to migrate if it's a major incompatible change. Ideally, the flag should be available in at least one major release. +- For the commit that removes the flag, use `Fixes #abc` in the description so that the GitHub issue gets closed when the commit is merged. diff --git a/contribute/codebase.mdx b/contribute/codebase.mdx index 44e0150d..99109430 100644 --- a/contribute/codebase.mdx +++ b/contribute/codebase.mdx @@ -2,1652 +2,773 @@ title: 'The Bazel codebase' --- - - -This document is a description of the codebase and how Bazel is structured. It -is intended for people willing to contribute to Bazel, not for end-users. +This document is a description of the codebase and how Bazel is structured. It is intended for people willing to contribute to Bazel, not for end-users. ## Introduction -The codebase of Bazel is large (~350KLOC production code and ~260 KLOC test -code) and no one is familiar with the whole landscape: everyone knows their -particular valley very well, but few know what lies over the hills in every -direction. +The codebase of Bazel is large (\~350KLOC production code and \~260 KLOC test code) and no one is familiar with the whole landscape: everyone knows their particular valley very well, but few know what lies over the hills in every direction. -In order for people midway upon the journey not to find themselves within a -forest dark with the straightforward pathway being lost, this document tries to -give an overview of the codebase so that it's easier to get started with -working on it. +In order for people midway upon the journey not to find themselves within a forest dark with the straightforward pathway being lost, this document tries to give an overview of the codebase so that it's easier to get started with working on it. -The public version of the source code of Bazel lives on GitHub at -[github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not -the "source of truth"; it's derived from a Google-internal source tree that -contains additional functionality that is not useful outside Google. The -long-term goal is to make GitHub the source of truth. +The public version of the source code of Bazel lives on GitHub at [github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not the "source of truth"; it's derived from a Google-internal source tree that contains additional functionality that is not useful outside Google. The long-term goal is to make GitHub the source of truth. -Contributions are accepted through the regular GitHub pull request mechanism, -and manually imported by a Googler into the internal source tree, then -re-exported back out to GitHub. +Contributions are accepted through the regular GitHub pull request mechanism, and manually imported by a Googler into the internal source tree, then re-exported back out to GitHub. ## Client/server architecture -The bulk of Bazel resides in a server process that stays in RAM between builds. -This allows Bazel to maintain state between builds. +The bulk of Bazel resides in a server process that stays in RAM between builds. This allows Bazel to maintain state between builds. -This is why the Bazel command line has two kinds of options: startup and -command. In a command line like this: +This is why the Bazel command line has two kinds of options: startup and command. In a command line like this: ``` bazel --host_jvm_args=-Xmx8G build -c opt //foo:bar ``` -Some options (`--host_jvm_args=`) are before the name of the command to be run -and some are after (`-c opt`); the former kind is called a "startup option" and -affects the server process as a whole, whereas the latter kind, the "command -option", only affects a single command. - -Each server instance has a single associated workspace (collection of source -trees known as "repositories") and each workspace usually has a single active -server instance. This can be circumvented by specifying a custom output base -(see the "Directory layout" section for more information). - -Bazel is distributed as a single ELF executable that is also a valid .zip file. -When you type `bazel`, the above ELF executable implemented in C++ (the -"client") gets control. It sets up an appropriate server process using the -following steps: - -1. Checks whether it has already extracted itself. If not, it does that. This - is where the implementation of the server comes from. -2. Checks whether there is an active server instance that works: it is running, - it has the right startup options and uses the right workspace directory. It - finds the running server by looking at the directory `$OUTPUT_BASE/server` - where there is a lock file with the port the server is listening on. -3. If needed, kills the old server process -4. If needed, starts up a new server process - -After a suitable server process is ready, the command that needs to be run is -communicated to it over a gRPC interface, then the output of Bazel is piped back -to the terminal. Only one command can be running at the same time. This is -implemented using an elaborate locking mechanism with parts in C++ and parts in -Java. There is some infrastructure for running multiple commands in parallel, -since the inability to run `bazel version` in parallel with another command -is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s -and some state in `BlazeRuntime`. - -At the end of a command, the Bazel server transmits the exit code the client -should return. An interesting wrinkle is the implementation of `bazel run`: the -job of this command is to run something Bazel just built, but it can't do that -from the server process because it doesn't have a terminal. So instead it tells -the client what binary it should `exec()` and with what arguments. - -When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC -connection, which tries to terminate the command as soon as possible. After the -third Ctrl-C, the client sends a SIGKILL to the server instead. - -The source code of the client is under `src/main/cpp` and the protocol used to -communicate with the server is in `src/main/protobuf/command_server.proto` . - -The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls -from the client are handled by `GrpcServerImpl.run()`. +Some options (`--host_jvm_args=`) are before the name of the command to be run and some are after (`-c opt`); the former kind is called a "startup option" and affects the server process as a whole, whereas the latter kind, the "command option", only affects a single command. + +Each server instance has a single associated workspace (collection of source trees known as "repositories") and each workspace usually has a single active server instance. This can be circumvented by specifying a custom output base (see the "Directory layout" section for more information). + +Bazel is distributed as a single ELF executable that is also a valid .zip file. When you type `bazel`, the above ELF executable implemented in C++ (the "client") gets control. It sets up an appropriate server process using the following steps: + +1. Checks whether it has already extracted itself. If not, it does that. This is where the implementation of the server comes from. +2. Checks whether there is an active server instance that works: it is running, it has the right startup options and uses the right workspace directory. It finds the running server by looking at the directory `$OUTPUT_BASE/server` where there is a lock file with the port the server is listening on. +3. If needed, kills the old server process +4. If needed, starts up a new server process + +After a suitable server process is ready, the command that needs to be run is communicated to it over a gRPC interface, then the output of Bazel is piped back to the terminal. Only one command can be running at the same time. This is implemented using an elaborate locking mechanism with parts in C++ and parts in Java. There is some infrastructure for running multiple commands in parallel, since the inability to run `bazel version` in parallel with another command is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s and some state in `BlazeRuntime`. + +At the end of a command, the Bazel server transmits the exit code the client should return. An interesting wrinkle is the implementation of `bazel run`: the job of this command is to run something Bazel just built, but it can't do that from the server process because it doesn't have a terminal. So instead it tells the client what binary it should `exec()` and with what arguments. + +When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC connection, which tries to terminate the command as soon as possible. After the third Ctrl-C, the client sends a SIGKILL to the server instead. + +The source code of the client is under `src/main/cpp` and the protocol used to communicate with the server is in `src/main/protobuf/command_server.proto` . + +The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls from the client are handled by `GrpcServerImpl.run()`. ## Directory layout -Bazel creates a somewhat complicated set of directories during a build. A full -description is available in [Output directory layout](/remote/output-directories). +Bazel creates a somewhat complicated set of directories during a build. A full description is available in [Output directory layout](/remote/output-directories). -The "main repo" is the source tree Bazel is run in. It usually corresponds to -something you checked out from source control. The root of this directory is -known as the "workspace root". +The "main repo" is the source tree Bazel is run in. It usually corresponds to something you checked out from source control. The root of this directory is known as the "workspace root". -Bazel puts all of its data under the "output user root". This is usually -`$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the -`--output_user_root` startup option. +Bazel puts all of its data under the "output user root". This is usually `$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the `--output_user_root` startup option. -The "install base" is where Bazel is extracted to. This is done automatically -and each Bazel version gets a subdirectory based on its checksum under the -install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed -using the `--install_base` command line option. +The "install base" is where Bazel is extracted to. This is done automatically and each Bazel version gets a subdirectory based on its checksum under the install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed using the `--install_base` command line option. -The "output base" is the place where the Bazel instance attached to a specific -workspace writes to. Each output base has at most one Bazel server instance -running at any time. It's usually at `$OUTPUT_USER_ROOT/`. It can be changed using the `--output_base` startup option, -which is, among other things, useful for getting around the limitation that only -one Bazel instance can be running in any workspace at any given time. +The "output base" is the place where the Bazel instance attached to a specific workspace writes to. Each output base has at most one Bazel server instance running at any time. It's usually at `$OUTPUT_USER_ROOT/<checksum of the path to the workspace>`. It can be changed using the `--output_base` startup option, which is, among other things, useful for getting around the limitation that only one Bazel instance can be running in any workspace at any given time. The output directory contains, among other things: -* The fetched external repositories at `$OUTPUT_BASE/external`. -* The exec root, a directory that contains symlinks to all the source - code for the current build. It's located at `$OUTPUT_BASE/execroot`. During - the build, the working directory is `$EXECROOT/`. We are planning to change this to `$EXECROOT`, although it's a - long term plan because it's a very incompatible change. -* Files built during the build. +- The fetched external repositories at `$OUTPUT_BASE/external`. +- The exec root, a directory that contains symlinks to all the source code for the current build. It's located at `$OUTPUT_BASE/execroot`. During the build, the working directory is `$EXECROOT/<name of main repository>`. We are planning to change this to `$EXECROOT`, although it's a long term plan because it's a very incompatible change. +- Files built during the build. ## The process of executing a command -Once the Bazel server gets control and is informed about a command it needs to -execute, the following sequence of events happens: +Once the Bazel server gets control and is informed about a command it needs to execute, the following sequence of events happens: -1. `BlazeCommandDispatcher` is informed about the new request. It decides - whether the command needs a workspace to run in (almost every command except - for ones that don't have anything to do with source code, such as version or - help) and whether another command is running. +1. `BlazeCommandDispatcher` is informed about the new request. It decides whether the command needs a workspace to run in (almost every command except for ones that don't have anything to do with source code, such as version or help) and whether another command is running. -2. The right command is found. Each command must implement the interface - `BlazeCommand` and must have the `@Command` annotation (this is a bit of an - antipattern, it would be nice if all the metadata a command needs was - described by methods on `BlazeCommand`) +2. The right command is found. Each command must implement the interface `BlazeCommand` and must have the `@Command` annotation (this is a bit of an antipattern, it would be nice if all the metadata a command needs was described by methods on `BlazeCommand`) -3. The command line options are parsed. Each command has different command line - options, which are described in the `@Command` annotation. +3. The command line options are parsed. Each command has different command line options, which are described in the `@Command` annotation. -4. An event bus is created. The event bus is a stream for events that happen - during the build. Some of these are exported to outside of Bazel under the - aegis of the Build Event Protocol in order to tell the world how the build - goes. +4. An event bus is created. The event bus is a stream for events that happen during the build. Some of these are exported to outside of Bazel under the aegis of the Build Event Protocol in order to tell the world how the build goes. -5. The command gets control. The most interesting commands are those that run a - build: build, test, run, coverage and so on: this functionality is - implemented by `BuildTool`. +5. The command gets control. The most interesting commands are those that run a build: build, test, run, coverage and so on: this functionality is implemented by `BuildTool`. -6. The set of target patterns on the command line is parsed and wildcards like - `//pkg:all` and `//pkg/...` are resolved. This is implemented in - `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as - `TargetPatternPhaseValue`. +6. The set of target patterns on the command line is parsed and wildcards like `//pkg:all` and `//pkg/...` are resolved. This is implemented in `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as `TargetPatternPhaseValue`. -7. The loading/analysis phase is run to produce the action graph (a directed - acyclic graph of commands that need to be executed for the build). +7. The loading/analysis phase is run to produce the action graph (a directed acyclic graph of commands that need to be executed for the build). -8. The execution phase is run. This means running every action required to - build the top-level targets that are requested are run. +8. The execution phase is run. This means running every action required to build the top-level targets that are requested are run. ## Command line options -The command line options for a Bazel invocation are described in an -`OptionsParsingResult` object, which in turn contains a map from "option -classes" to the values of the options. An "option class" is a subclass of -`OptionsBase` and groups command line options together that are related to each -other. For example: - -1. Options related to a programming language (`CppOptions` or `JavaOptions`). - These should be a subclass of `FragmentOptions` and are eventually wrapped - into a `BuildOptions` object. -2. Options related to the way Bazel executes actions (`ExecutionOptions`) - -These options are designed to be consumed in the analysis phase and (either -through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). -Some of them (for example, whether to do C++ include scanning or not) are read -in the execution phase, but that always requires explicit plumbing since -`BuildConfiguration` is not available then. For more information, see the -section "Configurations". - -**WARNING:** We like to pretend that `OptionsBase` instances are immutable and -use them that way (such as a part of `SkyKeys`). This is not the case and -modifying them is a really good way to break Bazel in subtle ways that are hard -to debug. Unfortunately, making them actually immutable is a large endeavor. -(Modifying a `FragmentOptions` immediately after construction before anyone else -gets a chance to keep a reference to it and before `equals()` or `hashCode()` is -called on it is okay.) +The command line options for a Bazel invocation are described in an `OptionsParsingResult` object, which in turn contains a map from "option classes" to the values of the options. An "option class" is a subclass of `OptionsBase` and groups command line options together that are related to each other. For example: + +1. Options related to a programming language (`CppOptions` or `JavaOptions`). These should be a subclass of `FragmentOptions` and are eventually wrapped into a `BuildOptions` object. +2. Options related to the way Bazel executes actions (`ExecutionOptions`) + +These options are designed to be consumed in the analysis phase and (either through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). Some of them (for example, whether to do C++ include scanning or not) are read in the execution phase, but that always requires explicit plumbing since `BuildConfiguration` is not available then. For more information, see the section "Configurations". + +**WARNING:** We like to pretend that `OptionsBase` instances are immutable and use them that way (such as a part of `SkyKeys`). This is not the case and modifying them is a really good way to break Bazel in subtle ways that are hard to debug. Unfortunately, making them actually immutable is a large endeavor. (Modifying a `FragmentOptions` immediately after construction before anyone else gets a chance to keep a reference to it and before `equals()` or `hashCode()` is called on it is okay.) Bazel learns about option classes in the following ways: -1. Some are hard-wired into Bazel (`CommonCommandOptions`) -2. From the `@Command` annotation on each Bazel command -3. From `ConfiguredRuleClassProvider` (these are command line options related - to individual programming languages) -4. Starlark rules can also define their own options (see - [here](/extending/config)) +1. Some are hard-wired into Bazel (`CommonCommandOptions`) +2. From the `@Command` annotation on each Bazel command +3. From `ConfiguredRuleClassProvider` (these are command line options related to individual programming languages) +4. Starlark rules can also define their own options (see [here](/extending/config)) -Each option (excluding Starlark-defined options) is a member variable of a -`FragmentOptions` subclass that has the `@Option` annotation, which specifies -the name and the type of the command line option along with some help text. +Each option (excluding Starlark-defined options) is a member variable of a `FragmentOptions` subclass that has the `@Option` annotation, which specifies the name and the type of the command line option along with some help text. -The Java type of the value of a command line option is usually something simple -(a string, an integer, a Boolean, a label, etc.). However, we also support -options of more complicated types; in this case, the job of converting from the -command line string to the data type falls to an implementation of -`com.google.devtools.common.options.Converter`. +The Java type of the value of a command line option is usually something simple (a string, an integer, a Boolean, a label, etc.). However, we also support options of more complicated types; in this case, the job of converting from the command line string to the data type falls to an implementation of `com.google.devtools.common.options.Converter`. ## The source tree, as seen by Bazel -Bazel is in the business of building software, which happens by reading and -interpreting the source code. The totality of the source code Bazel operates on -is called "the workspace" and it is structured into repositories, packages and -rules. +Bazel is in the business of building software, which happens by reading and interpreting the source code. The totality of the source code Bazel operates on is called "the workspace" and it is structured into repositories, packages and rules. ### Repositories -A "repository" is a source tree on which a developer works; it usually -represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, -that is, a single source tree that contains all source code used to run the build. -Bazel, in contrast, supports projects whose source code spans multiple -repositories. The repository from which Bazel is invoked is called the "main -repository", the others are called "external repositories". +A "repository" is a source tree on which a developer works; it usually represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, that is, a single source tree that contains all source code used to run the build. Bazel, in contrast, supports projects whose source code spans multiple repositories. The repository from which Bazel is invoked is called the "main repository", the others are called "external repositories". -A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or -in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The -main repo is the source tree where you're invoking Bazel from. External repos -are defined in various ways; see [external dependencies -overview](/external/overview) for more information. +A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The main repo is the source tree where you're invoking Bazel from. External repos are defined in various ways; see [external dependencies overview](/external/overview) for more information. -Code of external repositories is symlinked or downloaded under -`$OUTPUT_BASE/external`. +Code of external repositories is symlinked or downloaded under `$OUTPUT_BASE/external`. -When running the build, the whole source tree needs to be pieced together; this -is done by `SymlinkForest`, which symlinks every package in the main repository -to `$EXECROOT` and every external repository to either `$EXECROOT/external` or -`$EXECROOT/..`. +When running the build, the whole source tree needs to be pieced together; this is done by `SymlinkForest`, which symlinks every package in the main repository to `$EXECROOT` and every external repository to either `$EXECROOT/external` or `$EXECROOT/..`. ### Packages -Every repository is composed of packages, a collection of related files and -a specification of the dependencies. These are specified by a file called -`BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason -why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this -file name. However, it turned out to be a commonly used path segment, especially -on Windows, where file names are case-insensitive. - -Packages are independent of each other: changes to the `BUILD` file of a package -cannot cause other packages to change. The addition or removal of `BUILD` files -_can _change other packages, since recursive globs stop at package boundaries -and thus the presence of a `BUILD` file stops the recursion. - -The evaluation of a `BUILD` file is called "package loading". It's implemented -in the class `PackageFactory`, works by calling the Starlark interpreter and -requires knowledge of the set of available rule classes. The result of package -loading is a `Package` object. It's mostly a map from a string (the name of a -target) to the target itself. - -A large chunk of complexity during package loading is globbing: Bazel does not -require every source file to be explicitly listed and instead can run globs -(such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that -descend into subdirectories (but not into subpackages). This requires access to -the file system and since that can be slow, we implement all sorts of tricks to -make it run in parallel and as efficiently as possible. +Every repository is composed of packages, a collection of related files and a specification of the dependencies. These are specified by a file called `BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this file name. However, it turned out to be a commonly used path segment, especially on Windows, where file names are case-insensitive. + +Packages are independent of each other: changes to the `BUILD` file of a package cannot cause other packages to change. The addition or removal of `BUILD` files \_can \_change other packages, since recursive globs stop at package boundaries and thus the presence of a `BUILD` file stops the recursion. + +The evaluation of a `BUILD` file is called "package loading". It's implemented in the class `PackageFactory`, works by calling the Starlark interpreter and requires knowledge of the set of available rule classes. The result of package loading is a `Package` object. It's mostly a map from a string (the name of a target) to the target itself. + +A large chunk of complexity during package loading is globbing: Bazel does not require every source file to be explicitly listed and instead can run globs (such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that descend into subdirectories (but not into subpackages). This requires access to the file system and since that can be slow, we implement all sorts of tricks to make it run in parallel and as efficiently as possible. Globbing is implemented in the following classes: -* `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber -* `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to - the legacy globber in order to avoid "Skyframe restarts" (described below) +- `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber +- `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to the legacy globber in order to avoid "Skyframe restarts" (described below) -The `Package` class itself contains some members that are exclusively used to -parse the "external" package (related to external dependencies) and which do not -make sense for real packages. This is -a design flaw because objects describing regular packages should not contain -fields that describe something else. These include: +The `Package` class itself contains some members that are exclusively used to parse the "external" package (related to external dependencies) and which do not make sense for real packages. This is a design flaw because objects describing regular packages should not contain fields that describe something else. These include: -* The repository mappings -* The registered toolchains -* The registered execution platforms +- The repository mappings +- The registered toolchains +- The registered execution platforms -Ideally, there would be more separation between parsing the "external" package -from parsing regular packages so that `Package` does not need to cater for the -needs of both. This is unfortunately difficult to do because the two are -intertwined quite deeply. +Ideally, there would be more separation between parsing the "external" package from parsing regular packages so that `Package` does not need to cater for the needs of both. This is unfortunately difficult to do because the two are intertwined quite deeply. ### Labels, Targets, and Rules Packages are composed of targets, which have the following types: -1. **Files:** things that are either the input or the output of the build. In - Bazel parlance, we call them _artifacts_ (discussed elsewhere). Not all - files created during the build are targets; it's common for an output of - Bazel not to have an associated label. -2. **Rules:** these describe steps to derive its outputs from its inputs. They - are generally associated with a programming language (such as `cc_library`, - `java_library` or `py_library`), but there are some language-agnostic ones - (such as `genrule` or `filegroup`) -3. **Package groups:** discussed in the [Visibility](#visibility) section. - -The name of a target is called a _Label_. The syntax of labels is -`@repo//pac/kage:name`, where `repo` is the name of the repository the Label is -in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of -the file (if the label refers to a source file) relative to the directory of the -package. When referring to a target on the command line, some parts of the label -can be omitted: - -1. If the repository is omitted, the label is taken to be in the main - repository. -2. If the package part is omitted (such as `name` or `:name`), the label is taken - to be in the package of the current working directory (relative paths - containing uplevel references (..) are not allowed) - -A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may -be implemented either in Starlark (the `rule()` function) or in Java (so called -"native rules", type `RuleClass`). In the long term, every language-specific -rule will be implemented in Starlark, but some legacy rule families (such as Java -or C++) are still in Java for the time being. - -Starlark rule classes need to be imported at the beginning of `BUILD` files -using the `load()` statement, whereas Java rule classes are "innately" known by -Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. +1. **Files:** things that are either the input or the output of the build. In Bazel parlance, we call them *artifacts* (discussed elsewhere). Not all files created during the build are targets; it's common for an output of Bazel not to have an associated label. +2. **Rules:** these describe steps to derive its outputs from its inputs. They are generally associated with a programming language (such as `cc_library`, `java_library` or `py_library`), but there are some language-agnostic ones (such as `genrule` or `filegroup`) +3. **Package groups:** discussed in the [Visibility](#visibility) section. + +The name of a target is called a *Label*. The syntax of labels is `@repo//pac/kage:name`, where `repo` is the name of the repository the Label is in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of the file (if the label refers to a source file) relative to the directory of the package. When referring to a target on the command line, some parts of the label can be omitted: + +1. If the repository is omitted, the label is taken to be in the main repository. +2. If the package part is omitted (such as `name` or `:name`), the label is taken to be in the package of the current working directory (relative paths containing uplevel references (..) are not allowed) + +A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may be implemented either in Starlark (the `rule()` function) or in Java (so called "native rules", type `RuleClass`). In the long term, every language-specific rule will be implemented in Starlark, but some legacy rule families (such as Java or C++) are still in Java for the time being. + +Starlark rule classes need to be imported at the beginning of `BUILD` files using the `load()` statement, whereas Java rule classes are "innately" known by Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. Rule classes contain information such as: -1. Its attributes (such as `srcs`, `deps`): their types, default values, - constraints, etc. -2. The configuration transitions and aspects attached to each attribute, if any -3. The implementation of the rule -4. The transitive info providers the rule "usually" creates +1. Its attributes (such as `srcs`, `deps`): their types, default values, constraints, etc. +2. The configuration transitions and aspects attached to each attribute, if any +3. The implementation of the rule +4. The transitive info providers the rule "usually" creates -**Terminology note:** In the codebase, we often use "Rule" to mean the target -created by a rule class. But in Starlark and in user-facing documentation, -"Rule" should be used exclusively to refer to the rule class itself; the target -is just a "target". Also note that despite `RuleClass` having "class" in its -name, there is no Java inheritance relationship between a rule class and targets -of that type. +**Terminology note:** In the codebase, we often use "Rule" to mean the target created by a rule class. But in Starlark and in user-facing documentation, "Rule" should be used exclusively to refer to the rule class itself; the target is just a "target". Also note that despite `RuleClass` having "class" in its name, there is no Java inheritance relationship between a rule class and targets of that type. ## Skyframe -The evaluation framework underlying Bazel is called Skyframe. Its model is that -everything that needs to be built during a build is organized into a directed -acyclic graph with edges pointing from any pieces of data to its dependencies, -that is, other pieces of data that need to be known to construct it. - -The nodes in the graph are called `SkyValue`s and their names are called -`SkyKey`s. Both are deeply immutable; only immutable objects should be -reachable from them. This invariant almost always holds, and in case it doesn't -(such as for the individual options classes `BuildOptions`, which is a member of -`BuildConfigurationValue` and its `SkyKey`) we try really hard not to change -them or to change them in only ways that are not observable from the outside. -From this it follows that everything that is computed within Skyframe (such as -configured targets) must also be immutable. - -The most convenient way to observe the Skyframe graph is to run `bazel dump ---skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best -to do it for tiny builds, since it can get pretty large. - -Skyframe lives in the `com.google.devtools.build.skyframe` package. The -similarly-named package `com.google.devtools.build.lib.skyframe` contains the -implementation of Bazel on top of Skyframe. More information about Skyframe is -available [here](/reference/skyframe). - -To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the -`SkyFunction` corresponding to the type of the key. During the function's -evaluation, it may request other dependencies from Skyframe by calling the -various overloads of `SkyFunction.Environment.getValue()`. This has the -side-effect of registering those dependencies into Skyframe's internal graph, so -that Skyframe will know to re-evaluate the function when any of its dependencies -change. In other words, Skyframe's caching and incremental computation work at -the granularity of `SkyFunction`s and `SkyValue`s. - -Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` -will return null. The function should then yield control back to Skyframe by -itself returning null. At some later point, Skyframe will evaluate the -unavailable dependency, then restart the function from the beginning — only this -time the `getValue()` call will succeed with a non-null result. - -A consequence of this is that any computation performed inside the `SkyFunction` -prior to the restart must be repeated. But this does not include work done to -evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work -around this issue by: - -1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to - limit the number of restarts. -2. Breaking up a `SkyValue` into separate pieces computed by different - `SkyFunction`s, so that they can be computed and cached independently. This - should be done strategically, since it has the potential to increases memory - usage. -3. Storing state between restarts, either using - `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache - "behind the back of Skyframe". With complex SkyFunctions, state management - between restarts can get tricky, so - [`StateMachine`s](/contribute/statemachine-guide) were introduced for a - structured approach to logical concurrency, including hooks to suspend and - resume hierarchical computations within a `SkyFunction`. Example: - [`DependencyResolver#computeDependencies`][statemachine_example] - uses a `StateMachine` with `getState()` to compute the potentially huge set - of direct dependencies of a configured target, which otherwise can result in - expensive restarts. - -[statemachine_example]: https://developers.google.com/devsite/reference/markdown/links#reference_links - -Fundamentally, Bazel need these types of workarounds because hundreds of -thousands of in-flight Skyframe nodes is common, and Java's support of -lightweight threads [does not outperform][virtual_threads] the -`StateMachine` implementation as of 2023. - -[virtual_threads]: /contribute/statemachine-guide#epilogue_eventually_removing_callbacks +The evaluation framework underlying Bazel is called Skyframe. Its model is that everything that needs to be built during a build is organized into a directed acyclic graph with edges pointing from any pieces of data to its dependencies, that is, other pieces of data that need to be known to construct it. + +The nodes in the graph are called `SkyValue`s and their names are called `SkyKey`s. Both are deeply immutable; only immutable objects should be reachable from them. This invariant almost always holds, and in case it doesn't (such as for the individual options classes `BuildOptions`, which is a member of `BuildConfigurationValue` and its `SkyKey`) we try really hard not to change them or to change them in only ways that are not observable from the outside. From this it follows that everything that is computed within Skyframe (such as configured targets) must also be immutable. + +The most convenient way to observe the Skyframe graph is to run `bazel dump --skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best to do it for tiny builds, since it can get pretty large. + +Skyframe lives in the `com.google.devtools.build.skyframe` package. The similarly-named package `com.google.devtools.build.lib.skyframe` contains the implementation of Bazel on top of Skyframe. More information about Skyframe is available [here](/reference/skyframe). + +To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the `SkyFunction` corresponding to the type of the key. During the function's evaluation, it may request other dependencies from Skyframe by calling the various overloads of `SkyFunction.Environment.getValue()`. This has the side-effect of registering those dependencies into Skyframe's internal graph, so that Skyframe will know to re-evaluate the function when any of its dependencies change. In other words, Skyframe's caching and incremental computation work at the granularity of `SkyFunction`s and `SkyValue`s. + +Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` will return null. The function should then yield control back to Skyframe by itself returning null. At some later point, Skyframe will evaluate the unavailable dependency, then restart the function from the beginning — only this time the `getValue()` call will succeed with a non-null result. + +A consequence of this is that any computation performed inside the `SkyFunction` prior to the restart must be repeated. But this does not include work done to evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work around this issue by: + +1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to limit the number of restarts. +2. Breaking up a `SkyValue` into separate pieces computed by different `SkyFunction`s, so that they can be computed and cached independently. This should be done strategically, since it has the potential to increases memory usage. +3. Storing state between restarts, either using `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache "behind the back of Skyframe". With complex SkyFunctions, state management between restarts can get tricky, so [`StateMachine`s](/contribute/statemachine-guide) were introduced for a structured approach to logical concurrency, including hooks to suspend and resume hierarchical computations within a `SkyFunction`. Example: [`DependencyResolver#computeDependencies`](https://developers.google.com/devsite/reference/markdown/links#reference_links) uses a `StateMachine` with `getState()` to compute the potentially huge set of direct dependencies of a configured target, which otherwise can result in expensive restarts. + +Fundamentally, Bazel need these types of workarounds because hundreds of thousands of in-flight Skyframe nodes is common, and Java's support of lightweight threads [does not outperform](/contribute/statemachine-guide#epilogue_eventually_removing_callbacks) the `StateMachine` implementation as of 2023. ## Starlark -Starlark is the domain-specific language people use to configure and extend -Bazel. It's conceived as a restricted subset of Python that has far fewer types, -more restrictions on control flow, and most importantly, strong immutability -guarantees to enable concurrent reads. It is not Turing-complete, which -discourages some (but not all) users from trying to accomplish general -programming tasks within the language. +Starlark is the domain-specific language people use to configure and extend Bazel. It's conceived as a restricted subset of Python that has far fewer types, more restrictions on control flow, and most importantly, strong immutability guarantees to enable concurrent reads. It is not Turing-complete, which discourages some (but not all) users from trying to accomplish general programming tasks within the language. -Starlark is implemented in the `net.starlark.java` package. -It also has an independent Go implementation -[here](https://github.com/google/starlark-go). The Java -implementation used in Bazel is currently an interpreter. +Starlark is implemented in the `net.starlark.java` package. It also has an independent Go implementation [here](https://github.com/google/starlark-go). The Java implementation used in Bazel is currently an interpreter. Starlark is used in several contexts, including: -1. **`BUILD` files.** This is where new build targets are defined. Starlark - code running in this context only has access to the contents of the `BUILD` - file itself and `.bzl` files loaded by it. -2. **The `MODULE.bazel` file.** This is where external dependencies are - defined. Starlark code running in this context only has very limited access - to a few predefined directives. -3. **`.bzl` files.** This is where new build rules, repo rules, module - extensions are defined. Starlark code here can define new functions and load - from other `.bzl` files. +1. **`BUILD` files.** This is where new build targets are defined. Starlark code running in this context only has access to the contents of the `BUILD` file itself and `.bzl` files loaded by it. +2. **The `MODULE.bazel` file.** This is where external dependencies are defined. Starlark code running in this context only has very limited access to a few predefined directives. +3. **`.bzl` files.** This is where new build rules, repo rules, module extensions are defined. Starlark code here can define new functions and load from other `.bzl` files. -The dialects available for `BUILD` and `.bzl` files are slightly different -because they express different things. A list of differences is available -[here](/rules/language#differences-between-build-and-bzl-files). +The dialects available for `BUILD` and `.bzl` files are slightly different because they express different things. A list of differences is available [here](/rules/language#differences-between-build-and-bzl-files). More information about Starlark is available [here](/rules/language). ## The loading/analysis phase -The loading/analysis phase is where Bazel determines what actions are needed to -build a particular rule. Its basic unit is a "configured target", which is, -quite sensibly, a (target, configuration) pair. - -It's called the "loading/analysis phase" because it can be split into two -distinct parts, which used to be serialized, but they can now overlap in time: - -1. Loading packages, that is, turning `BUILD` files into the `Package` objects - that represent them -2. Analyzing configured targets, that is, running the implementation of the - rules to produce the action graph - -Each configured target in the transitive closure of the configured targets -requested on the command line must be analyzed bottom-up; that is, leaf nodes -first, then up to the ones on the command line. The inputs to the analysis of -a single configured target are: - -1. **The configuration.** ("how" to build that rule; for example, the target - platform but also things like command line options the user wants to be - passed to the C++ compiler) -2. **The direct dependencies.** Their transitive info providers are available - to the rule being analyzed. They are called like that because they provide a - "roll-up" of the information in the transitive closure of the configured - target, such as all the .jar files on the classpath or all the .o files that - need to be linked into a C++ binary) -3. **The target itself**. This is the result of loading the package the target - is in. For rules, this includes its attributes, which is usually what - matters. -4. **The implementation of the configured target.** For rules, this can either - be in Starlark or in Java. All non-rule configured targets are implemented - in Java. +The loading/analysis phase is where Bazel determines what actions are needed to build a particular rule. Its basic unit is a "configured target", which is, quite sensibly, a (target, configuration) pair. + +It's called the "loading/analysis phase" because it can be split into two distinct parts, which used to be serialized, but they can now overlap in time: + +1. Loading packages, that is, turning `BUILD` files into the `Package` objects that represent them +2. Analyzing configured targets, that is, running the implementation of the rules to produce the action graph + +Each configured target in the transitive closure of the configured targets requested on the command line must be analyzed bottom-up; that is, leaf nodes first, then up to the ones on the command line. The inputs to the analysis of a single configured target are: + +1. **The configuration.** ("how" to build that rule; for example, the target platform but also things like command line options the user wants to be passed to the C++ compiler) +2. **The direct dependencies.** Their transitive info providers are available to the rule being analyzed. They are called like that because they provide a "roll-up" of the information in the transitive closure of the configured target, such as all the .jar files on the classpath or all the .o files that need to be linked into a C++ binary) +3. **The target itself**. This is the result of loading the package the target is in. For rules, this includes its attributes, which is usually what matters. +4. **The implementation of the configured target.** For rules, this can either be in Starlark or in Java. All non-rule configured targets are implemented in Java. The output of analyzing a configured target is: -1. The transitive info providers that configured targets that depend on it can - access -2. The artifacts it can create and the actions that produce them. +1. The transitive info providers that configured targets that depend on it can access +2. The artifacts it can create and the actions that produce them. -The API offered to Java rules is `RuleContext`, which is the equivalent of the -`ctx` argument of Starlark rules. Its API is more powerful, but at the same -time, it's easier to do Bad Things™, for example to write code whose time or -space complexity is quadratic (or worse), to make the Bazel server crash with a -Java exception or to violate invariants (such as by inadvertently modifying an -`Options` instance or by making a configured target mutable) +The API offered to Java rules is `RuleContext`, which is the equivalent of the `ctx` argument of Starlark rules. Its API is more powerful, but at the same time, it's easier to do Bad Things™, for example to write code whose time or space complexity is quadratic (or worse), to make the Bazel server crash with a Java exception or to violate invariants (such as by inadvertently modifying an `Options` instance or by making a configured target mutable) -The algorithm that determines the direct dependencies of a configured target -lives in `DependencyResolver.dependentNodeMap()`. +The algorithm that determines the direct dependencies of a configured target lives in `DependencyResolver.dependentNodeMap()`. ### Configurations -Configurations are the "how" of building a target: for what platform, with what -command line options, etc. - -The same target can be built for multiple configurations in the same build. This -is useful, for example, when the same code is used for a tool that's run during -the build and for the target code and we are cross-compiling or when we are -building a fat Android app (one that contains native code for multiple CPU -architectures) - -Conceptually, the configuration is a `BuildOptions` instance. However, in -practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides -additional sundry pieces of functionality. It propagates from the top of the -dependency graph to the bottom. If it changes, the build needs to be -re-analyzed. - -This results in anomalies like having to re-analyze the whole build if, for -example, the number of requested test runs changes, even though that only -affects test targets (we have plans to "trim" configurations so that this is -not the case, but it's not ready yet). - -When a rule implementation needs part of the configuration, it needs to declare -it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` -. This is both to avoid mistakes (such as Python rules using the Java fragment) and -to facilitate configuration trimming so that such as if Python options change, C++ -targets don't need to be re-analyzed. - -The configuration of a rule is not necessarily the same as that of its "parent" -rule. The process of changing the configuration in a dependency edge is called a -"configuration transition". It can happen in two places: - -1. On a dependency edge. These transitions are specified in - `Attribute.Builder.cfg()` and are functions from a `Rule` (where the - transition happens) and a `BuildOptions` (the original configuration) to one - or more `BuildOptions` (the output configuration). -2. On any incoming edge to a configured target. These are specified in - `RuleClass.Builder.cfg()`. +Configurations are the "how" of building a target: for what platform, with what command line options, etc. + +The same target can be built for multiple configurations in the same build. This is useful, for example, when the same code is used for a tool that's run during the build and for the target code and we are cross-compiling or when we are building a fat Android app (one that contains native code for multiple CPU architectures) + +Conceptually, the configuration is a `BuildOptions` instance. However, in practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides additional sundry pieces of functionality. It propagates from the top of the dependency graph to the bottom. If it changes, the build needs to be re-analyzed. + +This results in anomalies like having to re-analyze the whole build if, for example, the number of requested test runs changes, even though that only affects test targets (we have plans to "trim" configurations so that this is not the case, but it's not ready yet). + +When a rule implementation needs part of the configuration, it needs to declare it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` . This is both to avoid mistakes (such as Python rules using the Java fragment) and to facilitate configuration trimming so that such as if Python options change, C++ targets don't need to be re-analyzed. + +The configuration of a rule is not necessarily the same as that of its "parent" rule. The process of changing the configuration in a dependency edge is called a "configuration transition". It can happen in two places: + +1. On a dependency edge. These transitions are specified in `Attribute.Builder.cfg()` and are functions from a `Rule` (where the transition happens) and a `BuildOptions` (the original configuration) to one or more `BuildOptions` (the output configuration). +2. On any incoming edge to a configured target. These are specified in `RuleClass.Builder.cfg()`. The relevant classes are `TransitionFactory` and `ConfigurationTransition`. Configuration transitions are used, for example: -1. To declare that a particular dependency is used during the build and it - should thus be built in the execution architecture -2. To declare that a particular dependency must be built for multiple - architectures (such as for native code in fat Android APKs) +1. To declare that a particular dependency is used during the build and it should thus be built in the execution architecture +2. To declare that a particular dependency must be built for multiple architectures (such as for native code in fat Android APKs) -If a configuration transition results in multiple configurations, it's called a -_split transition._ +If a configuration transition results in multiple configurations, it's called a *split transition.* -Configuration transitions can also be implemented in Starlark (documentation -[here](/extending/config)) +Configuration transitions can also be implemented in Starlark (documentation [here](/extending/config)) ### Transitive info providers -Transitive info providers are a way (and the _only _way) for configured targets -to learn things about other configured targets that they depend on, and the only -way to tell things about themselves to other configured targets that depend on -them. The reason why "transitive" is in their name is that this is usually some -sort of roll-up of the transitive closure of a configured target. - -There is generally a 1:1 correspondence between Java transitive info providers -and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of -`FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was -deemed to be more Starlark-ish than a direct transliteration of the Java one). -Their key is one of the following things: - -1. A Java Class object. This is only available for providers that are not - accessible from Starlark. These providers are a subclass of - `TransitiveInfoProvider`. -2. A string. This is legacy and heavily discouraged since it's susceptible to - name clashes. Such transitive info providers are direct subclasses of - `build.lib.packages.Info` . -3. A provider symbol. This can be created from Starlark using the `provider()` - function and is the recommended way to create new providers. The symbol is - represented by a `Provider.Key` instance in Java. - -New providers implemented in Java should be implemented using `BuiltinProvider`. -`NativeProvider` is deprecated (we haven't had time to remove it yet) and -`TransitiveInfoProvider` subclasses cannot be accessed from Starlark. +Transitive info providers are a way (and the \_only \_way) for configured targets to learn things about other configured targets that they depend on, and the only way to tell things about themselves to other configured targets that depend on them. The reason why "transitive" is in their name is that this is usually some sort of roll-up of the transitive closure of a configured target. + +There is generally a 1:1 correspondence between Java transitive info providers and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of `FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was deemed to be more Starlark-ish than a direct transliteration of the Java one). Their key is one of the following things: + +1. A Java Class object. This is only available for providers that are not accessible from Starlark. These providers are a subclass of `TransitiveInfoProvider`. +2. A string. This is legacy and heavily discouraged since it's susceptible to name clashes. Such transitive info providers are direct subclasses of `build.lib.packages.Info` . +3. A provider symbol. This can be created from Starlark using the `provider()` function and is the recommended way to create new providers. The symbol is represented by a `Provider.Key` instance in Java. + +New providers implemented in Java should be implemented using `BuiltinProvider`. `NativeProvider` is deprecated (we haven't had time to remove it yet) and `TransitiveInfoProvider` subclasses cannot be accessed from Starlark. ### Configured targets -Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a -subclass for each rule class implemented in Java. Starlark configured targets -are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . +Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a subclass for each rule class implemented in Java. Starlark configured targets are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . -Configured target factories should use `RuleConfiguredTargetBuilder` to -construct their return value. It consists of the following things: +Configured target factories should use `RuleConfiguredTargetBuilder` to construct their return value. It consists of the following things: -1. Their `filesToBuild`, the hazy concept of "the set of files this rule - represents." These are the files that get built when the configured target - is on the command line or in the srcs of a genrule. -2. Their runfiles, regular and data. -3. Their output groups. These are various "other sets of files" the rule can - build. They can be accessed using the output\_group attribute of the - filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. +1. Their `filesToBuild`, the hazy concept of "the set of files this rule represents." These are the files that get built when the configured target is on the command line or in the srcs of a genrule. +2. Their runfiles, regular and data. +3. Their output groups. These are various "other sets of files" the rule can build. They can be accessed using the output\_group attribute of the filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. ### Runfiles -Some binaries need data files to run. A prominent example is tests that need -input files. This is represented in Bazel by the concept of "runfiles". A -"runfiles tree" is a directory tree of the data files for a particular binary. -It is created in the file system as a symlink tree with individual symlinks -pointing to the files in the source or output trees. - -A set of runfiles is represented as a `Runfiles` instance. It is conceptually a -map from the path of a file in the runfiles tree to the `Artifact` instance that -represents it. It's a little more complicated than a single `Map` for two -reasons: - -* Most of the time, the runfiles path of a file is the same as its execpath. - We use this to save some RAM. -* There are various legacy kinds of entries in runfiles trees, which also need - to be represented. - -Runfiles are collected using `RunfilesProvider`: an instance of this class -represents the runfiles a configured target (such as a library) and its transitive -closure needs and they are gathered like a nested set (in fact, they are -implemented using nested sets under the cover): each target unions the runfiles -of its dependencies, adds some of its own, then sends the resulting set upwards -in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` -instances, one for when the rule is depended on through the "data" attribute and -one for every other kind of incoming dependency. This is because a target -sometimes presents different runfiles when depended on through a data attribute -than otherwise. This is undesired legacy behavior that we haven't gotten around -removing yet. - -Runfiles of binaries are represented as an instance of `RunfilesSupport`. This -is different from `Runfiles` because `RunfilesSupport` has the capability of -actually being built (unlike `Runfiles`, which is just a mapping). This -necessitates the following additional components: - -* **The input runfiles manifest.** This is a serialized description of the - runfiles tree. It is used as a proxy for the contents of the runfiles tree - and Bazel assumes that the runfiles tree changes if and only if the contents - of the manifest change. -* **The output runfiles manifest.** This is used by runtime libraries that - handle runfiles trees, notably on Windows, which sometimes doesn't support - symbolic links. -* **Command line arguments** for running the binary whose runfiles the - `RunfilesSupport` object represents. +Some binaries need data files to run. A prominent example is tests that need input files. This is represented in Bazel by the concept of "runfiles". A "runfiles tree" is a directory tree of the data files for a particular binary. It is created in the file system as a symlink tree with individual symlinks pointing to the files in the source or output trees. + +A set of runfiles is represented as a `Runfiles` instance. It is conceptually a map from the path of a file in the runfiles tree to the `Artifact` instance that represents it. It's a little more complicated than a single `Map` for two reasons: + +- Most of the time, the runfiles path of a file is the same as its execpath. We use this to save some RAM. +- There are various legacy kinds of entries in runfiles trees, which also need to be represented. + +Runfiles are collected using `RunfilesProvider`: an instance of this class represents the runfiles a configured target (such as a library) and its transitive closure needs and they are gathered like a nested set (in fact, they are implemented using nested sets under the cover): each target unions the runfiles of its dependencies, adds some of its own, then sends the resulting set upwards in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` instances, one for when the rule is depended on through the "data" attribute and one for every other kind of incoming dependency. This is because a target sometimes presents different runfiles when depended on through a data attribute than otherwise. This is undesired legacy behavior that we haven't gotten around removing yet. + +Runfiles of binaries are represented as an instance of `RunfilesSupport`. This is different from `Runfiles` because `RunfilesSupport` has the capability of actually being built (unlike `Runfiles`, which is just a mapping). This necessitates the following additional components: + +- **The input runfiles manifest.** This is a serialized description of the runfiles tree. It is used as a proxy for the contents of the runfiles tree and Bazel assumes that the runfiles tree changes if and only if the contents of the manifest change. +- **The output runfiles manifest.** This is used by runtime libraries that handle runfiles trees, notably on Windows, which sometimes doesn't support symbolic links. +- **Command line arguments** for running the binary whose runfiles the `RunfilesSupport` object represents. ### Aspects -Aspects are a way to "propagate computation down the dependency graph". They are -described for users of Bazel -[here](/extending/aspects). A good -motivating example is protocol buffers: a `proto_library` rule should not know -about any particular language, but building the implementation of a protocol -buffer message (the "basic unit" of protocol buffers) in any programming -language should be coupled to the `proto_library` rule so that if two targets in -the same language depend on the same protocol buffer, it gets built only once. - -Just like configured targets, they are represented in Skyframe as a `SkyValue` -and the way they are constructed is very similar to how configured targets are -built: they have a factory class called `ConfiguredAspectFactory` that has -access to a `RuleContext`, but unlike configured target factories, it also knows -about the configured target it is attached to and its providers. - -The set of aspects propagated down the dependency graph is specified for each -attribute using the `Attribute.Builder.aspects()` function. There are a few -confusingly-named classes that participate in the process: - -1. `AspectClass` is the implementation of the aspect. It can be either in Java - (in which case it's a subclass) or in Starlark (in which case it's an - instance of `StarlarkAspectClass`). It's analogous to - `RuleConfiguredTargetFactory`. -2. `AspectDefinition` is the definition of the aspect; it includes the - providers it requires, the providers it provides and contains a reference to - its implementation, such as the appropriate `AspectClass` instance. It's - analogous to `RuleClass`. -3. `AspectParameters` is a way to parametrize an aspect that is propagated down - the dependency graph. It's currently a string to string map. A good example - of why it's useful is protocol buffers: if a language has multiple APIs, the - information as to which API the protocol buffers should be built for should - be propagated down the dependency graph. -4. `Aspect` represents all the data that's needed to compute an aspect that - propagates down the dependency graph. It consists of the aspect class, its - definition and its parameters. -5. `RuleAspect` is the function that determines which aspects a particular rule - should propagate. It's a `Rule` -> `Aspect` function. - -A somewhat unexpected complication is that aspects can attach to other aspects; -for example, an aspect collecting the classpath for a Java IDE will probably -want to know about all the .jar files on the classpath, but some of them are -protocol buffers. In that case, the IDE aspect will want to attach to the -(`proto_library` rule + Java proto aspect) pair. - -The complexity of aspects on aspects is captured in the class -`AspectCollection`. +Aspects are a way to "propagate computation down the dependency graph". They are described for users of Bazel [here](/extending/aspects). A good motivating example is protocol buffers: a `proto_library` rule should not know about any particular language, but building the implementation of a protocol buffer message (the "basic unit" of protocol buffers) in any programming language should be coupled to the `proto_library` rule so that if two targets in the same language depend on the same protocol buffer, it gets built only once. + +Just like configured targets, they are represented in Skyframe as a `SkyValue` and the way they are constructed is very similar to how configured targets are built: they have a factory class called `ConfiguredAspectFactory` that has access to a `RuleContext`, but unlike configured target factories, it also knows about the configured target it is attached to and its providers. + +The set of aspects propagated down the dependency graph is specified for each attribute using the `Attribute.Builder.aspects()` function. There are a few confusingly-named classes that participate in the process: + +1. `AspectClass` is the implementation of the aspect. It can be either in Java (in which case it's a subclass) or in Starlark (in which case it's an instance of `StarlarkAspectClass`). It's analogous to `RuleConfiguredTargetFactory`. +2. `AspectDefinition` is the definition of the aspect; it includes the providers it requires, the providers it provides and contains a reference to its implementation, such as the appropriate `AspectClass` instance. It's analogous to `RuleClass`. +3. `AspectParameters` is a way to parametrize an aspect that is propagated down the dependency graph. It's currently a string to string map. A good example of why it's useful is protocol buffers: if a language has multiple APIs, the information as to which API the protocol buffers should be built for should be propagated down the dependency graph. +4. `Aspect` represents all the data that's needed to compute an aspect that propagates down the dependency graph. It consists of the aspect class, its definition and its parameters. +5. `RuleAspect` is the function that determines which aspects a particular rule should propagate. It's a `Rule` -> `Aspect` function. + +A somewhat unexpected complication is that aspects can attach to other aspects; for example, an aspect collecting the classpath for a Java IDE will probably want to know about all the .jar files on the classpath, but some of them are protocol buffers. In that case, the IDE aspect will want to attach to the (`proto_library` rule + Java proto aspect) pair. + +The complexity of aspects on aspects is captured in the class `AspectCollection`. ### Platforms and toolchains -Bazel supports multi-platform builds, that is, builds where there may be -multiple architectures where build actions run and multiple architectures for -which code is built. These architectures are referred to as _platforms_ in Bazel -parlance (full documentation -[here](/extending/platforms)) - -A platform is described by a key-value mapping from _constraint settings_ (such as -the concept of "CPU architecture") to _constraint values_ (such as a particular CPU -like x86\_64). We have a "dictionary" of the most commonly used constraint -settings and values in the `@platforms` repository. - -The concept of _toolchain_ comes from the fact that depending on what platforms -the build is running on and what platforms are targeted, one may need to use -different compilers; for example, a particular C++ toolchain may run on a -specific OS and be able to target some other OSes. Bazel must determine the C++ -compiler that is used based on the set execution and target platform -(documentation for toolchains -[here](/extending/toolchains)). - -In order to do this, toolchains are annotated with the set of execution and -target platform constraints they support. In order to do this, the definition of -a toolchain are split into two parts: - -1. A `toolchain()` rule that describes the set of execution and target - constraints a toolchain supports and tells what kind (such as C++ or Java) of - toolchain it is (the latter is represented by the `toolchain_type()` rule) -2. A language-specific rule that describes the actual toolchain (such as - `cc_toolchain()`) - -This is done in this way because we need to know the constraints for every -toolchain in order to do toolchain resolution and language-specific -`*_toolchain()` rules contain much more information than that, so they take more -time to load. +Bazel supports multi-platform builds, that is, builds where there may be multiple architectures where build actions run and multiple architectures for which code is built. These architectures are referred to as *platforms* in Bazel parlance (full documentation [here](/extending/platforms)) + +A platform is described by a key-value mapping from *constraint settings* (such as the concept of "CPU architecture") to *constraint values* (such as a particular CPU like x86\_64). We have a "dictionary" of the most commonly used constraint settings and values in the `@platforms` repository. + +The concept of *toolchain* comes from the fact that depending on what platforms the build is running on and what platforms are targeted, one may need to use different compilers; for example, a particular C++ toolchain may run on a specific OS and be able to target some other OSes. Bazel must determine the C++ compiler that is used based on the set execution and target platform (documentation for toolchains [here](/extending/toolchains)). + +In order to do this, toolchains are annotated with the set of execution and target platform constraints they support. In order to do this, the definition of a toolchain are split into two parts: + +1. A `toolchain()` rule that describes the set of execution and target constraints a toolchain supports and tells what kind (such as C++ or Java) of toolchain it is (the latter is represented by the `toolchain_type()` rule) +2. A language-specific rule that describes the actual toolchain (such as `cc_toolchain()`) + +This is done in this way because we need to know the constraints for every toolchain in order to do toolchain resolution and language-specific `*_toolchain()` rules contain much more information than that, so they take more time to load. Execution platforms are specified in one of the following ways: -1. In the MODULE.bazel file using the `register_execution_platforms()` function -2. On the command line using the --extra\_execution\_platforms command line - option - -The set of available execution platforms is computed in -`RegisteredExecutionPlatformsFunction` . - -The target platform for a configured target is determined by -`PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we -eventually want to support multiple target platforms, but it's not implemented -yet. - -The set of toolchains to be used for a configured target is determined by -`ToolchainResolutionFunction`. It is a function of: - -* The set of registered toolchains (in the MODULE.bazel file and the - configuration) -* The desired execution and target platforms (in the configuration) -* The set of toolchain types that are required by the configured target (in - `UnloadedToolchainContextKey)` -* The set of execution platform constraints of the configured target (the - `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` - -Its result is an `UnloadedToolchainContext`, which is essentially a map from -toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of -the selected toolchain. It's called "unloaded" because it does not contain the -toolchains themselves, only their labels. - -Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` -and used by the implementation of the configured target that requested them. - -We also have a legacy system that relies on there being one single "host" -configuration and target configurations being represented by various -configuration flags, such as `--cpu` . We are gradually transitioning to the above -system. In order to handle cases where people rely on the legacy configuration -values, we have implemented -[platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) -to translate between the legacy flags and the new-style platform constraints. -Their code is in `PlatformMappingFunction` and uses a non-Starlark "little -language". +1. In the MODULE.bazel file using the `register_execution_platforms()` function +2. On the command line using the --extra\_execution\_platforms command line option + +The set of available execution platforms is computed in `RegisteredExecutionPlatformsFunction` . + +The target platform for a configured target is determined by `PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we eventually want to support multiple target platforms, but it's not implemented yet. + +The set of toolchains to be used for a configured target is determined by `ToolchainResolutionFunction`. It is a function of: + +- The set of registered toolchains (in the MODULE.bazel file and the configuration) +- The desired execution and target platforms (in the configuration) +- The set of toolchain types that are required by the configured target (in `UnloadedToolchainContextKey)` +- The set of execution platform constraints of the configured target (the `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` + +Its result is an `UnloadedToolchainContext`, which is essentially a map from toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of the selected toolchain. It's called "unloaded" because it does not contain the toolchains themselves, only their labels. + +Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` and used by the implementation of the configured target that requested them. + +We also have a legacy system that relies on there being one single "host" configuration and target configurations being represented by various configuration flags, such as `--cpu` . We are gradually transitioning to the above system. In order to handle cases where people rely on the legacy configuration values, we have implemented [platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) to translate between the legacy flags and the new-style platform constraints. Their code is in `PlatformMappingFunction` and uses a non-Starlark "little language". ### Constraints -Sometimes one wants to designate a target as being compatible with only a few -platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: +Sometimes one wants to designate a target as being compatible with only a few platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: -* Rule-specific constraints -* `environment_group()` / `environment()` -* Platform constraints +- Rule-specific constraints +- `environment_group()` / `environment()` +- Platform constraints -Rule-specific constraints are mostly used within Google for Java rules; they are -on their way out and they are not available in Bazel, but the source code may -contain references to it. The attribute that governs this is called -`constraints=` . +Rule-specific constraints are mostly used within Google for Java rules; they are on their way out and they are not available in Bazel, but the source code may contain references to it. The attribute that governs this is called `constraints=` . -#### environment_group() and environment() +#### environment\_group() and environment() These rules are a legacy mechanism and are not widely used. -All build rules can declare which "environments" they can be built for, where an -"environment" is an instance of the `environment()` rule. +All build rules can declare which "environments" they can be built for, where an "environment" is an instance of the `environment()` rule. There are various ways supported environments can be specified for a rule: -1. Through the `restricted_to=` attribute. This is the most direct form of - specification; it declares the exact set of environments the rule supports. -2. Through the `compatible_with=` attribute. This declares environments a rule - supports in addition to "standard" environments that are supported by - default. -3. Through the package-level attributes `default_restricted_to=` and - `default_compatible_with=`. -4. Through default specifications in `environment_group()` rules. Every - environment belongs to a group of thematically related peers (such as "CPU - architectures", "JDK versions" or "mobile operating systems"). The - definition of an environment group includes which of these environments - should be supported by "default" if not otherwise specified by the - `restricted_to=` / `environment()` attributes. A rule with no such - attributes inherits all defaults. -5. Through a rule class default. This overrides global defaults for all - instances of the given rule class. This can be used, for example, to make - all `*_test` rules testable without each instance having to explicitly - declare this capability. - -`environment()` is implemented as a regular rule whereas `environment_group()` -is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a -function that is available by default from Starlark -(`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous -target. This is to avoid a cyclic dependency that would arise because each -environment needs to declare the environment group it belongs to and each -environment group needs to declare its default environments. - -A build can be restricted to a certain environment with the -`--target_environment` command line option. - -The implementation of the constraint check is in -`RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. +1. Through the `restricted_to=` attribute. This is the most direct form of specification; it declares the exact set of environments the rule supports. +2. Through the `compatible_with=` attribute. This declares environments a rule supports in addition to "standard" environments that are supported by default. +3. Through the package-level attributes `default_restricted_to=` and `default_compatible_with=`. +4. Through default specifications in `environment_group()` rules. Every environment belongs to a group of thematically related peers (such as "CPU architectures", "JDK versions" or "mobile operating systems"). The definition of an environment group includes which of these environments should be supported by "default" if not otherwise specified by the `restricted_to=` / `environment()` attributes. A rule with no such attributes inherits all defaults. +5. Through a rule class default. This overrides global defaults for all instances of the given rule class. This can be used, for example, to make all `*_test` rules testable without each instance having to explicitly declare this capability. + +`environment()` is implemented as a regular rule whereas `environment_group()` is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a function that is available by default from Starlark (`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous target. This is to avoid a cyclic dependency that would arise because each environment needs to declare the environment group it belongs to and each environment group needs to declare its default environments. + +A build can be restricted to a certain environment with the `--target_environment` command line option. + +The implementation of the constraint check is in `RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. #### Platform constraints -The current "official" way to describe what platforms a target is compatible -with is by using the same constraints used to describe toolchains and platforms. -It was implemented in pull request -[#10945](https://github.com/bazelbuild/bazel/pull/10945). +The current "official" way to describe what platforms a target is compatible with is by using the same constraints used to describe toolchains and platforms. It was implemented in pull request [#10945](https://github.com/bazelbuild/bazel/pull/10945). ### Visibility -If you work on a large codebase with a lot of developers (like at Google), you -want to take care to prevent everyone else from arbitrarily depending on your -code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), -people _will_ come to rely on behaviors that you considered to be implementation -details. +If you work on a large codebase with a lot of developers (like at Google), you want to take care to prevent everyone else from arbitrarily depending on your code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), people *will* come to rely on behaviors that you considered to be implementation details. -Bazel supports this by the mechanism called _visibility_: you can limit which -targets can depend on a particular target using the -[visibility](/reference/be/common-definitions#common-attributes) attribute. This -attribute is a little special because, although it holds a list of labels, these -labels may encode a pattern over package names rather than a pointer to any -particular target. (Yes, this is a design flaw.) +Bazel supports this by the mechanism called *visibility*: you can limit which targets can depend on a particular target using the [visibility](/reference/be/common-definitions#common-attributes) attribute. This attribute is a little special because, although it holds a list of labels, these labels may encode a pattern over package names rather than a pointer to any particular target. (Yes, this is a design flaw.) This is implemented in the following places: -* The `RuleVisibility` interface represents a visibility declaration. It can - be either a constant (fully public or fully private) or a list of labels. -* Labels can refer to either package groups (predefined list of packages), to - packages directly (`//pkg:__pkg__`) or subtrees of packages - (`//pkg:__subpackages__`). This is different from the command line syntax, - which uses `//pkg:*` or `//pkg/...`. -* Package groups are implemented as their own target (`PackageGroup`) and - configured target (`PackageGroupConfiguredTarget`). We could probably - replace these with simple rules if we wanted to. Their logic is implemented - with the help of: `PackageSpecification`, which corresponds to a - single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds - to a single `package_group`'s `packages` attribute; and - `PackageSpecificationProvider`, which aggregates over a `package_group` and - its transitive `includes`. -* The conversion from visibility label lists to dependencies is done in - `DependencyResolver.visitTargetVisibility` and a few other miscellaneous - places. -* The actual check is done in - `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` +- The `RuleVisibility` interface represents a visibility declaration. It can be either a constant (fully public or fully private) or a list of labels. +- Labels can refer to either package groups (predefined list of packages), to packages directly (`//pkg:__pkg__`) or subtrees of packages (`//pkg:__subpackages__`). This is different from the command line syntax, which uses `//pkg:*` or `//pkg/...`. +- Package groups are implemented as their own target (`PackageGroup`) and configured target (`PackageGroupConfiguredTarget`). We could probably replace these with simple rules if we wanted to. Their logic is implemented with the help of: `PackageSpecification`, which corresponds to a single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds to a single `package_group`'s `packages` attribute; and `PackageSpecificationProvider`, which aggregates over a `package_group` and its transitive `includes`. +- The conversion from visibility label lists to dependencies is done in `DependencyResolver.visitTargetVisibility` and a few other miscellaneous places. +- The actual check is done in `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` ### Nested sets -Oftentimes, a configured target aggregates a set of files from its dependencies, -adds its own, and wraps the aggregate set into a transitive info provider so -that configured targets that depend on it can do the same. Examples: +Oftentimes, a configured target aggregates a set of files from its dependencies, adds its own, and wraps the aggregate set into a transitive info provider so that configured targets that depend on it can do the same. Examples: -* The C++ header files used for a build -* The object files that represent the transitive closure of a `cc_library` -* The set of .jar files that need to be on the classpath for a Java rule to - compile or run -* The set of Python files in the transitive closure of a Python rule +- The C++ header files used for a build +- The object files that represent the transitive closure of a `cc_library` +- The set of .jar files that need to be on the classpath for a Java rule to compile or run +- The set of Python files in the transitive closure of a Python rule -If we did this the naive way by using, for example, `List` or `Set`, we'd end up with -quadratic memory usage: if there is a chain of N rules and each rule adds a -file, we'd have 1+2+...+N collection members. +If we did this the naive way by using, for example, `List` or `Set`, we'd end up with quadratic memory usage: if there is a chain of N rules and each rule adds a file, we'd have 1+2+...+N collection members. -In order to get around this problem, we came up with the concept of a -`NestedSet`. It's a data structure that is composed of other `NestedSet` -instances and some members of its own, thereby forming a directed acyclic graph -of sets. They are immutable and their members can be iterated over. We define -multiple iteration order (`NestedSet.Order`): preorder, postorder, topological -(a node always comes after its ancestors) and "don't care, but it should be the -same each time". +In order to get around this problem, we came up with the concept of a `NestedSet`. It's a data structure that is composed of other `NestedSet` instances and some members of its own, thereby forming a directed acyclic graph of sets. They are immutable and their members can be iterated over. We define multiple iteration order (`NestedSet.Order`): preorder, postorder, topological (a node always comes after its ancestors) and "don't care, but it should be the same each time". The same data structure is called `depset` in Starlark. ### Artifacts and Actions -The actual build consists of a set of commands that need to be run to produce -the output the user wants. The commands are represented as instances of the -class `Action` and the files are represented as instances of the class -`Artifact`. They are arranged in a bipartite, directed, acyclic graph called the -"action graph". - -Artifacts come in two kinds: source artifacts (ones that are available -before Bazel starts executing) and derived artifacts (ones that need to be -built). Derived artifacts can themselves be multiple kinds: - -1. **Regular artifacts.** These are checked for up-to-dateness by computing - their checksum, with mtime as a shortcut; we don't checksum the file if its - ctime hasn't changed. -2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by - calling readlink(). Unlike regular artifacts, these can be dangling - symlinks. Usually used in cases where one then packs up some files into an - archive of some sort. -3. **Tree artifacts.** These are not single files, but directory trees. They - are checked for up-to-dateness by checking the set of files in it and their - contents. They are represented as a `TreeArtifact`. -4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a - rebuild. This is used exclusively for build stamp information: we don't want - to do a rebuild just because the current time changed. - -There is no fundamental reason why source artifacts cannot be tree artifacts or -unresolved symlink artifacts; it's just that we haven't implemented it yet. At -the moment, source symlinks are always resolved, and while source directories -are supported, their contents are entirely opaque to build rules and thus don't -support the same kind of lazy command line expansion as tree artifacts do. - -Actions are best understood as a command that needs to be run, the environment -it needs and the set of outputs it produces. The following things are the main -components of the description of an action: - -* The command line that needs to be run -* The input artifacts it needs -* The environment variables that need to be set -* Annotations that describe the environment (such as platform) it needs to run in - \ - -There are also a few other special cases, like writing a file whose content is -known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are -a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be -separate classes), although Java and C++ have their own action types -(`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). - -We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is -pretty close, but C++ is a bit of a special-case due to .d file parsing and -include scanning. - -The action graph is mostly "embedded" into the Skyframe graph: conceptually, the -execution of an action is represented as an invocation of -`ActionExecutionFunction`. The mapping from an action graph dependency edge to a -Skyframe dependency edge is described in -`ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few -optimizations in order to keep the number of Skyframe edges low: - -* Derived artifacts do not have their own `SkyValue`s. Instead, - `Artifact.getGeneratingActionKey()` is used to find out the key for the - action that generates it -* Nested sets have their own Skyframe key. +The actual build consists of a set of commands that need to be run to produce the output the user wants. The commands are represented as instances of the class `Action` and the files are represented as instances of the class `Artifact`. They are arranged in a bipartite, directed, acyclic graph called the "action graph". + +Artifacts come in two kinds: source artifacts (ones that are available before Bazel starts executing) and derived artifacts (ones that need to be built). Derived artifacts can themselves be multiple kinds: + +1. **Regular artifacts.** These are checked for up-to-dateness by computing their checksum, with mtime as a shortcut; we don't checksum the file if its ctime hasn't changed. +2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by calling readlink(). Unlike regular artifacts, these can be dangling symlinks. Usually used in cases where one then packs up some files into an archive of some sort. +3. **Tree artifacts.** These are not single files, but directory trees. They are checked for up-to-dateness by checking the set of files in it and their contents. They are represented as a `TreeArtifact`. +4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a rebuild. This is used exclusively for build stamp information: we don't want to do a rebuild just because the current time changed. + +There is no fundamental reason why source artifacts cannot be tree artifacts or unresolved symlink artifacts; it's just that we haven't implemented it yet. At the moment, source symlinks are always resolved, and while source directories are supported, their contents are entirely opaque to build rules and thus don't support the same kind of lazy command line expansion as tree artifacts do. + +Actions are best understood as a command that needs to be run, the environment it needs and the set of outputs it produces. The following things are the main components of the description of an action: + +- The command line that needs to be run +- The input artifacts it needs +- The environment variables that need to be set +- Annotations that describe the environment (such as platform) it needs to run in \\ + +There are also a few other special cases, like writing a file whose content is known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be separate classes), although Java and C++ have their own action types (`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). + +We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is pretty close, but C++ is a bit of a special-case due to .d file parsing and include scanning. + +The action graph is mostly "embedded" into the Skyframe graph: conceptually, the execution of an action is represented as an invocation of `ActionExecutionFunction`. The mapping from an action graph dependency edge to a Skyframe dependency edge is described in `ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few optimizations in order to keep the number of Skyframe edges low: + +- Derived artifacts do not have their own `SkyValue`s. Instead, `Artifact.getGeneratingActionKey()` is used to find out the key for the action that generates it +- Nested sets have their own Skyframe key. ### Shared actions -Some actions are generated by multiple configured targets; Starlark rules are -more limited since they are only allowed to put their derived actions into a -directory determined by their configuration and their package (but even so, -rules in the same package can conflict), but rules implemented in Java can put -derived artifacts anywhere. - -This is considered to be a misfeature, but getting rid of it is really hard -because it produces significant savings in execution time when, for example, a -source file needs to be processed somehow and that file is referenced by -multiple rules (handwave-handwave). This comes at the cost of some RAM: each -instance of a shared action needs to be stored in memory separately. - -If two actions generate the same output file, they must be exactly the same: -have the same inputs, the same outputs and run the same command line. This -equivalence relation is implemented in `Actions.canBeShared()` and it is -verified between the analysis and execution phases by looking at every Action. -This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` -and is one of the few places in Bazel that requires a "global" view of the -build. +Some actions are generated by multiple configured targets; Starlark rules are more limited since they are only allowed to put their derived actions into a directory determined by their configuration and their package (but even so, rules in the same package can conflict), but rules implemented in Java can put derived artifacts anywhere. + +This is considered to be a misfeature, but getting rid of it is really hard because it produces significant savings in execution time when, for example, a source file needs to be processed somehow and that file is referenced by multiple rules (handwave-handwave). This comes at the cost of some RAM: each instance of a shared action needs to be stored in memory separately. + +If two actions generate the same output file, they must be exactly the same: have the same inputs, the same outputs and run the same command line. This equivalence relation is implemented in `Actions.canBeShared()` and it is verified between the analysis and execution phases by looking at every Action. This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` and is one of the few places in Bazel that requires a "global" view of the build. ## The execution phase -This is when Bazel actually starts running build actions, such as commands that -produce outputs. - -The first thing Bazel does after the analysis phase is to determine what -Artifacts need to be built. The logic for this is encoded in -`TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the -configured targets on the command line and the contents of a special output -group for the explicit purpose of expressing "if this target is on the command -line, build these artifacts". - -The next step is creating the execution root. Since Bazel has the option to read -source packages from different locations in the file system (`--package_path`), -it needs to provide locally executed actions with a full source tree. This is -handled by the class `SymlinkForest` and works by taking note of every target -used in the analysis phase and building up a single directory tree that symlinks -every package with a used target from its actual location. An alternative would -be to pass the correct paths to commands (taking `--package_path` into account). -This is undesirable because: - -* It changes action command lines when a package is moved from a package path - entry to another (used to be a common occurrence) -* It results in different command lines if an action is run remotely than if - it's run locally -* It requires a command line transformation specific to the tool in use - (consider the difference between such as Java classpaths and C++ include paths) -* Changing the command line of an action invalidates its action cache entry -* `--package_path` is slowly and steadily being deprecated - -Then, Bazel starts traversing the action graph (the bipartite, directed graph -composed of actions and their input and output artifacts) and running actions. -The execution of each action is represented by an instance of the `SkyValue` -class `ActionExecutionValue`. - -Since running an action is expensive, we have a few layers of caching that can -be hit behind Skyframe: - -* `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts - of `ActionExecutionFunction` cheap -* The local action cache contains data about the state of the file system -* Remote execution systems usually also contain their own cache +This is when Bazel actually starts running build actions, such as commands that produce outputs. + +The first thing Bazel does after the analysis phase is to determine what Artifacts need to be built. The logic for this is encoded in `TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the configured targets on the command line and the contents of a special output group for the explicit purpose of expressing "if this target is on the command line, build these artifacts". + +The next step is creating the execution root. Since Bazel has the option to read source packages from different locations in the file system (`--package_path`), it needs to provide locally executed actions with a full source tree. This is handled by the class `SymlinkForest` and works by taking note of every target used in the analysis phase and building up a single directory tree that symlinks every package with a used target from its actual location. An alternative would be to pass the correct paths to commands (taking `--package_path` into account). This is undesirable because: + +- It changes action command lines when a package is moved from a package path entry to another (used to be a common occurrence) +- It results in different command lines if an action is run remotely than if it's run locally +- It requires a command line transformation specific to the tool in use (consider the difference between such as Java classpaths and C++ include paths) +- Changing the command line of an action invalidates its action cache entry +- `--package_path` is slowly and steadily being deprecated + +Then, Bazel starts traversing the action graph (the bipartite, directed graph composed of actions and their input and output artifacts) and running actions. The execution of each action is represented by an instance of the `SkyValue` class `ActionExecutionValue`. + +Since running an action is expensive, we have a few layers of caching that can be hit behind Skyframe: + +- `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts of `ActionExecutionFunction` cheap +- The local action cache contains data about the state of the file system +- Remote execution systems usually also contain their own cache ### The local action cache -This cache is another layer that sits behind Skyframe; even if an action is -re-executed in Skyframe, it can still be a hit in the local action cache. It -represents the state of the local file system and it's serialized to disk which -means that when one starts up a new Bazel server, one can get local action cache -hits even though the Skyframe graph is empty. +This cache is another layer that sits behind Skyframe; even if an action is re-executed in Skyframe, it can still be a hit in the local action cache. It represents the state of the local file system and it's serialized to disk which means that when one starts up a new Bazel server, one can get local action cache hits even though the Skyframe graph is empty. -This cache is checked for hits using the method -`ActionCacheChecker.getTokenIfNeedToExecute()` . +This cache is checked for hits using the method `ActionCacheChecker.getTokenIfNeedToExecute()` . -Contrary to its name, it's a map from the path of a derived artifact to the -action that emitted it. The action is described as: +Contrary to its name, it's a map from the path of a derived artifact to the action that emitted it. The action is described as: -1. The set of its input and output files and their checksum -2. Its "action key", which is usually the command line that was executed, but - in general, represents everything that's not captured by the checksum of the - input files (such as for `FileWriteAction`, it's the checksum of the data - that's written) +1. The set of its input and output files and their checksum +2. Its "action key", which is usually the command line that was executed, but in general, represents everything that's not captured by the checksum of the input files (such as for `FileWriteAction`, it's the checksum of the data that's written) -There is also a highly experimental "top-down action cache" that is still under -development, which uses transitive hashes to avoid going to the cache as many -times. +There is also a highly experimental "top-down action cache" that is still under development, which uses transitive hashes to avoid going to the cache as many times. ### Input discovery and input pruning -Some actions are more complicated than just having a set of inputs. Changes to -the set of inputs of an action come in two forms: - -* An action may discover new inputs before its execution or decide that some - of its inputs are not actually necessary. The canonical example is C++, - where it's better to make an educated guess about what header files a C++ - file uses from its transitive closure so that we don't heed to send every - file to remote executors; therefore, we have an option not to register every - header file as an "input", but scan the source file for transitively - included headers and only mark those header files as inputs that are - mentioned in `#include` statements (we overestimate so that we don't need to - implement a full C preprocessor) This option is currently hard-wired to - "false" in Bazel and is only used at Google. -* An action may realize that some files were not used during its execution. In - C++, this is called ".d files": the compiler tells which header files were - used after the fact, and in order to avoid the embarrassment of having worse - incrementality than Make, Bazel makes use of this fact. This offers a better - estimate than the include scanner because it relies on the compiler. +Some actions are more complicated than just having a set of inputs. Changes to the set of inputs of an action come in two forms: + +- An action may discover new inputs before its execution or decide that some of its inputs are not actually necessary. The canonical example is C++, where it's better to make an educated guess about what header files a C++ file uses from its transitive closure so that we don't heed to send every file to remote executors; therefore, we have an option not to register every header file as an "input", but scan the source file for transitively included headers and only mark those header files as inputs that are mentioned in `#include` statements (we overestimate so that we don't need to implement a full C preprocessor) This option is currently hard-wired to "false" in Bazel and is only used at Google. +- An action may realize that some files were not used during its execution. In C++, this is called ".d files": the compiler tells which header files were used after the fact, and in order to avoid the embarrassment of having worse incrementality than Make, Bazel makes use of this fact. This offers a better estimate than the include scanner because it relies on the compiler. These are implemented using methods on Action: -1. `Action.discoverInputs()` is called. It should return a nested set of - Artifacts that are determined to be required. These must be source artifacts - so that there are no dependency edges in the action graph that don't have an - equivalent in the configured target graph. -2. The action is executed by calling `Action.execute()`. -3. At the end of `Action.execute()`, the action can call - `Action.updateInputs()` to tell Bazel that not all of its inputs were - needed. This can result in incorrect incremental builds if a used input is - reported as unused. +1. `Action.discoverInputs()` is called. It should return a nested set of Artifacts that are determined to be required. These must be source artifacts so that there are no dependency edges in the action graph that don't have an equivalent in the configured target graph. +2. The action is executed by calling `Action.execute()`. +3. At the end of `Action.execute()`, the action can call `Action.updateInputs()` to tell Bazel that not all of its inputs were needed. This can result in incorrect incremental builds if a used input is reported as unused. -When an action cache returns a hit on a fresh Action instance (such as created -after a server restart), Bazel calls `updateInputs()` itself so that the set of -inputs reflects the result of input discovery and pruning done before. +When an action cache returns a hit on a fresh Action instance (such as created after a server restart), Bazel calls `updateInputs()` itself so that the set of inputs reflects the result of input discovery and pruning done before. -Starlark actions can make use of the facility to declare some inputs as unused -using the `unused_inputs_list=` argument of -`ctx.actions.run()`. +Starlark actions can make use of the facility to declare some inputs as unused using the `unused_inputs_list=` argument of `ctx.actions.run()`. ### Various ways to run actions: Strategies/ActionContexts -Some actions can be run in different ways. For example, a command line can be -executed locally, locally but in various kinds of sandboxes, or remotely. The -concept that embodies this is called an `ActionContext` (or `Strategy`, since we -successfully went only halfway with a rename...) +Some actions can be run in different ways. For example, a command line can be executed locally, locally but in various kinds of sandboxes, or remotely. The concept that embodies this is called an `ActionContext` (or `Strategy`, since we successfully went only halfway with a rename...) The life cycle of an action context is as follows: -1. When the execution phase is started, `BlazeModule` instances are asked what - action contexts they have. This happens in the constructor of - `ExecutionTool`. Action context types are identified by a Java `Class` - instance that refers to a sub-interface of `ActionContext` and which - interface the action context must implement. -2. The appropriate action context is selected from the available ones and is - forwarded to `ActionExecutionContext` and `BlazeExecutor` . -3. Actions request contexts using `ActionExecutionContext.getContext()` and - `BlazeExecutor.getStrategy()` (there should really be only one way to do - it…) - -Strategies are free to call other strategies to do their jobs; this is used, for -example, in the dynamic strategy that starts actions both locally and remotely, -then uses whichever finishes first. - -One notable strategy is the one that implements persistent worker processes -(`WorkerSpawnStrategy`). The idea is that some tools have a long startup time -and should therefore be reused between actions instead of starting one anew for -every action (This does represent a potential correctness issue, since Bazel -relies on the promise of the worker process that it doesn't carry observable -state between individual requests) - -If the tool changes, the worker process needs to be restarted. Whether a worker -can be reused is determined by computing a checksum for the tool used using -`WorkerFilesHash`. It relies on knowing which inputs of the action represent -part of the tool and which represent inputs; this is determined by the creator -of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are -counted as parts of the tool. +1. When the execution phase is started, `BlazeModule` instances are asked what action contexts they have. This happens in the constructor of `ExecutionTool`. Action context types are identified by a Java `Class` instance that refers to a sub-interface of `ActionContext` and which interface the action context must implement. +2. The appropriate action context is selected from the available ones and is forwarded to `ActionExecutionContext` and `BlazeExecutor` . +3. Actions request contexts using `ActionExecutionContext.getContext()` and `BlazeExecutor.getStrategy()` (there should really be only one way to do it…) + +Strategies are free to call other strategies to do their jobs; this is used, for example, in the dynamic strategy that starts actions both locally and remotely, then uses whichever finishes first. + +One notable strategy is the one that implements persistent worker processes (`WorkerSpawnStrategy`). The idea is that some tools have a long startup time and should therefore be reused between actions instead of starting one anew for every action (This does represent a potential correctness issue, since Bazel relies on the promise of the worker process that it doesn't carry observable state between individual requests) + +If the tool changes, the worker process needs to be restarted. Whether a worker can be reused is determined by computing a checksum for the tool used using `WorkerFilesHash`. It relies on knowing which inputs of the action represent part of the tool and which represent inputs; this is determined by the creator of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are counted as parts of the tool. More information about strategies (or action contexts!): -* Information about various strategies for running actions is available - [here](https://jmmv.dev/2019/12/bazel-strategies.html). -* Information about the dynamic strategy, one where we run an action both - locally and remotely to see whichever finishes first is available - [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). -* Information about the intricacies of executing actions locally is available - [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). +- Information about various strategies for running actions is available [here](https://jmmv.dev/2019/12/bazel-strategies.html). +- Information about the dynamic strategy, one where we run an action both locally and remotely to see whichever finishes first is available [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). +- Information about the intricacies of executing actions locally is available [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). ### The local resource manager -Bazel _can_ run many actions in parallel. The number of local actions that -_should_ be run in parallel differs from action to action: the more resources an -action requires, the less instances should be running at the same time to avoid -overloading the local machine. +Bazel *can* run many actions in parallel. The number of local actions that *should* be run in parallel differs from action to action: the more resources an action requires, the less instances should be running at the same time to avoid overloading the local machine. -This is implemented in the class `ResourceManager`: each action has to be -annotated with an estimate of the local resources it requires in the form of a -`ResourceSet` instance (CPU and RAM). Then when action contexts do something -that requires local resources, they call `ResourceManager.acquireResources()` -and are blocked until the required resources are available. +This is implemented in the class `ResourceManager`: each action has to be annotated with an estimate of the local resources it requires in the form of a `ResourceSet` instance (CPU and RAM). Then when action contexts do something that requires local resources, they call `ResourceManager.acquireResources()` and are blocked until the required resources are available. -A more detailed description of local resource management is available -[here](https://jmmv.dev/2019/12/bazel-local-resources.html). +A more detailed description of local resource management is available [here](https://jmmv.dev/2019/12/bazel-local-resources.html). ### The structure of the output directory -Each action requires a separate place in the output directory where it places -its outputs. The location of derived artifacts is usually as follows: +Each action requires a separate place in the output directory where it places its outputs. The location of derived artifacts is usually as follows: ``` -$EXECROOT/bazel-out//bin// +$EXECROOT/bazel-out/<configuration>/bin/<package>/<artifact name> ``` -How is the name of the directory that is associated with a particular -configuration determined? There are two conflicting desirable properties: - -1. If two configurations can occur in the same build, they should have - different directories so that both can have their own version of the same - action; otherwise, if the two configurations disagree about such as the command - line of an action producing the same output file, Bazel doesn't know which - action to choose (an "action conflict") -2. If two configurations represent "roughly" the same thing, they should have - the same name so that actions executed in one can be reused for the other if - the command lines match: for example, changes to the command line options to - the Java compiler should not result in C++ compile actions being re-run. - -So far, we have not come up with a principled way of solving this problem, which -has similarities to the problem of configuration trimming. A longer discussion -of options is available -[here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). -The main problematic areas are Starlark rules (whose authors usually aren't -intimately familiar with Bazel) and aspects, which add another dimension to the -space of things that can produce the "same" output file. - -The current approach is that the path segment for the configuration is -`-` with various suffixes added so that configuration -transitions implemented in Java don't result in action conflicts. In addition, a -checksum of the set of Starlark configuration transitions is added so that users -can't cause action conflicts. It is far from perfect. This is implemented in -`OutputDirectories.buildMnemonic()` and relies on each configuration fragment -adding its own part to the name of the output directory. +How is the name of the directory that is associated with a particular configuration determined? There are two conflicting desirable properties: + +1. If two configurations can occur in the same build, they should have different directories so that both can have their own version of the same action; otherwise, if the two configurations disagree about such as the command line of an action producing the same output file, Bazel doesn't know which action to choose (an "action conflict") +2. If two configurations represent "roughly" the same thing, they should have the same name so that actions executed in one can be reused for the other if the command lines match: for example, changes to the command line options to the Java compiler should not result in C++ compile actions being re-run. + +So far, we have not come up with a principled way of solving this problem, which has similarities to the problem of configuration trimming. A longer discussion of options is available [here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). The main problematic areas are Starlark rules (whose authors usually aren't intimately familiar with Bazel) and aspects, which add another dimension to the space of things that can produce the "same" output file. + +The current approach is that the path segment for the configuration is `<CPU>-<compilation mode>` with various suffixes added so that configuration transitions implemented in Java don't result in action conflicts. In addition, a checksum of the set of Starlark configuration transitions is added so that users can't cause action conflicts. It is far from perfect. This is implemented in `OutputDirectories.buildMnemonic()` and relies on each configuration fragment adding its own part to the name of the output directory. ## Tests Bazel has rich support for running tests. It supports: -* Running tests remotely (if a remote execution backend is available) -* Running tests multiple times in parallel (for deflaking or gathering timing - data) -* Sharding tests (splitting test cases in same test over multiple processes - for speed) -* Re-running flaky tests -* Grouping tests into test suites +- Running tests remotely (if a remote execution backend is available) +- Running tests multiple times in parallel (for deflaking or gathering timing data) +- Sharding tests (splitting test cases in same test over multiple processes for speed) +- Re-running flaky tests +- Grouping tests into test suites -Tests are regular configured targets that have a TestProvider, which describes -how the test should be run: +Tests are regular configured targets that have a TestProvider, which describes how the test should be run: -* The artifacts whose building result in the test being run. This is a "cache - status" file that contains a serialized `TestResultData` message -* The number of times the test should be run -* The number of shards the test should be split into -* Some parameters about how the test should be run (such as the test timeout) +- The artifacts whose building result in the test being run. This is a "cache status" file that contains a serialized `TestResultData` message +- The number of times the test should be run +- The number of shards the test should be split into +- Some parameters about how the test should be run (such as the test timeout) ### Determining which tests to run Determining which tests are run is an elaborate process. -First, during target pattern parsing, test suites are recursively expanded. The -expansion is implemented in `TestsForTargetPatternFunction`. A somewhat -surprising wrinkle is that if a test suite declares no tests, it refers to -_every_ test in its package. This is implemented in `Package.beforeBuild()` by -adding an implicit attribute called `$implicit_tests` to test suite rules. - -Then, tests are filtered for size, tags, timeout and language according to the -command line options. This is implemented in `TestFilter` and is called from -`TargetPatternPhaseFunction.determineTests()` during target parsing and the -result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason -why rule attributes which can be filtered for are not configurable is that this -happens before the analysis phase, therefore, the configuration is not -available. - -This is then processed further in `BuildView.createResult()`: targets whose -analysis failed are filtered out and tests are split into exclusive and -non-exclusive tests. It's then put into `AnalysisResult`, which is how -`ExecutionTool` knows which tests to run. - -In order to lend some transparency to this elaborate process, the `tests()` -query operator (implemented in `TestsFunction`) is available to tell which tests -are run when a particular target is specified on the command line. It's -unfortunately a reimplementation, so it probably deviates from the above in -multiple subtle ways. +First, during target pattern parsing, test suites are recursively expanded. The expansion is implemented in `TestsForTargetPatternFunction`. A somewhat surprising wrinkle is that if a test suite declares no tests, it refers to *every* test in its package. This is implemented in `Package.beforeBuild()` by adding an implicit attribute called `$implicit_tests` to test suite rules. + +Then, tests are filtered for size, tags, timeout and language according to the command line options. This is implemented in `TestFilter` and is called from `TargetPatternPhaseFunction.determineTests()` during target parsing and the result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason why rule attributes which can be filtered for are not configurable is that this happens before the analysis phase, therefore, the configuration is not available. + +This is then processed further in `BuildView.createResult()`: targets whose analysis failed are filtered out and tests are split into exclusive and non-exclusive tests. It's then put into `AnalysisResult`, which is how `ExecutionTool` knows which tests to run. + +In order to lend some transparency to this elaborate process, the `tests()` query operator (implemented in `TestsFunction`) is available to tell which tests are run when a particular target is specified on the command line. It's unfortunately a reimplementation, so it probably deviates from the above in multiple subtle ways. ### Running tests -The way the tests are run is by requesting cache status artifacts. This then -results in the execution of a `TestRunnerAction`, which eventually calls the -`TestActionContext` chosen by the `--test_strategy` command line option that -runs the test in the requested way. - -Tests are run according to an elaborate protocol that uses environment variables -to tell tests what's expected from them. A detailed description of what Bazel -expects from tests and what tests can expect from Bazel is available -[here](/reference/test-encyclopedia). At the -simplest, an exit code of 0 means success, anything else means failure. - -In addition to the cache status file, each test process emits a number of other -files. They are put in the "test log directory" which is the subdirectory called -`testlogs` of the output directory of the target configuration: - -* `test.xml`, a JUnit-style XML file detailing the individual test cases in - the test shard -* `test.log`, the console output of the test. stdout and stderr are not - separated. -* `test.outputs`, the "undeclared outputs directory"; this is used by tests - that want to output files in addition to what they print to the terminal. - -There are two things that can happen during test execution that cannot during -building regular targets: exclusive test execution and output streaming. - -Some tests need to be executed in exclusive mode, for example not in parallel with -other tests. This can be elicited either by adding `tags=["exclusive"]` to the -test rule or running the test with `--test_strategy=exclusive` . Each exclusive -test is run by a separate Skyframe invocation requesting the execution of the -test after the "main" build. This is implemented in -`SkyframeExecutor.runExclusiveTest()`. - -Unlike regular actions, whose terminal output is dumped when the action -finishes, the user can request the output of tests to be streamed so that they -get informed about the progress of a long-running test. This is specified by the -`--test_output=streamed` command line option and implies exclusive test -execution so that outputs of different tests are not interspersed. - -This is implemented in the aptly-named `StreamedTestOutput` class and works by -polling changes to the `test.log` file of the test in question and dumping new -bytes to the terminal where Bazel rules. - -Results of the executed tests are available on the event bus by observing -various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). -They are dumped to the Build Event Protocol and they are emitted to the console -by `AggregatingTestListener`. +The way the tests are run is by requesting cache status artifacts. This then results in the execution of a `TestRunnerAction`, which eventually calls the `TestActionContext` chosen by the `--test_strategy` command line option that runs the test in the requested way. + +Tests are run according to an elaborate protocol that uses environment variables to tell tests what's expected from them. A detailed description of what Bazel expects from tests and what tests can expect from Bazel is available [here](/reference/test-encyclopedia). At the simplest, an exit code of 0 means success, anything else means failure. + +In addition to the cache status file, each test process emits a number of other files. They are put in the "test log directory" which is the subdirectory called `testlogs` of the output directory of the target configuration: + +- `test.xml`, a JUnit-style XML file detailing the individual test cases in the test shard +- `test.log`, the console output of the test. stdout and stderr are not separated. +- `test.outputs`, the "undeclared outputs directory"; this is used by tests that want to output files in addition to what they print to the terminal. + +There are two things that can happen during test execution that cannot during building regular targets: exclusive test execution and output streaming. + +Some tests need to be executed in exclusive mode, for example not in parallel with other tests. This can be elicited either by adding `tags=["exclusive"]` to the test rule or running the test with `--test_strategy=exclusive` . Each exclusive test is run by a separate Skyframe invocation requesting the execution of the test after the "main" build. This is implemented in `SkyframeExecutor.runExclusiveTest()`. + +Unlike regular actions, whose terminal output is dumped when the action finishes, the user can request the output of tests to be streamed so that they get informed about the progress of a long-running test. This is specified by the `--test_output=streamed` command line option and implies exclusive test execution so that outputs of different tests are not interspersed. + +This is implemented in the aptly-named `StreamedTestOutput` class and works by polling changes to the `test.log` file of the test in question and dumping new bytes to the terminal where Bazel rules. + +Results of the executed tests are available on the event bus by observing various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). They are dumped to the Build Event Protocol and they are emitted to the console by `AggregatingTestListener`. ### Coverage collection -Coverage is reported by the tests in LCOV format in the files -`bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . - -To collect coverage, each test execution is wrapped in a script called -`collect_coverage.sh` . - -This script sets up the environment of the test to enable coverage collection -and determine where the coverage files are written by the coverage runtime(s). -It then runs the test. A test may itself run multiple subprocesses and consist -of parts written in multiple different programming languages (with separate -coverage collection runtimes). The wrapper script is responsible for converting -the resulting files to LCOV format if necessary, and merges them into a single -file. - -The interposition of `collect_coverage.sh` is done by the test strategies and -requires `collect_coverage.sh` to be on the inputs of the test. This is -accomplished by the implicit attribute `:coverage_support` which is resolved to -the value of the configuration flag `--coverage_support` (see -`TestConfiguration.TestOptions.coverageSupport`) - -Some languages do offline instrumentation, meaning that the coverage -instrumentation is added at compile time (such as C++) and others do online -instrumentation, meaning that coverage instrumentation is added at execution -time. - -Another core concept is _baseline coverage_. This is the coverage of a library, -binary, or test if no code in it was run. The problem it solves is that if you -want to compute the test coverage for a binary, it is not enough to merge the -coverage of all of the tests because there may be code in the binary that is not -linked into any test. Therefore, what we do is to emit a coverage file for every -binary which contains only the files we collect coverage for with no covered -lines. The default baseline coverage file for a target is at -`bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are -encouraged to generate their own baseline coverage files with more meaningful -content than just the names of the source files. - -We track two groups of files for coverage collection for each rule: the set of -instrumented files and the set of instrumentation metadata files. - -The set of instrumented files is just that, a set of files to instrument. For -online coverage runtimes, this can be used at runtime to decide which files to -instrument. It is also used to implement baseline coverage. - -The set of instrumentation metadata files is the set of extra files a test needs -to generate the LCOV files Bazel requires from it. In practice, this consists of -runtime-specific files; for example, gcc emits .gcno files during compilation. -These are added to the set of inputs of test actions if coverage mode is -enabled. - -Whether or not coverage is being collected is stored in the -`BuildConfiguration`. This is handy because it is an easy way to change the test -action and the action graph depending on this bit, but it also means that if -this bit is flipped, all targets need to be re-analyzed (some languages, such as -C++ require different compiler options to emit code that can collect coverage, -which mitigates this issue somewhat, since then a re-analysis is needed anyway). - -The coverage support files are depended on through labels in an implicit -dependency so that they can be overridden by the invocation policy, which allows -them to differ between the different versions of Bazel. Ideally, these -differences would be removed, and we standardized on one of them. - -We also generate a "coverage report" which merges the coverage collected for -every test in a Bazel invocation. This is handled by -`CoverageReportActionFactory` and is called from `BuildView.createResult()` . It -gets access to the tools it needs by looking at the `:coverage_report_generator` -attribute of the first test that is executed. +Coverage is reported by the tests in LCOV format in the files `bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . + +To collect coverage, each test execution is wrapped in a script called `collect_coverage.sh` . + +This script sets up the environment of the test to enable coverage collection and determine where the coverage files are written by the coverage runtime(s). It then runs the test. A test may itself run multiple subprocesses and consist of parts written in multiple different programming languages (with separate coverage collection runtimes). The wrapper script is responsible for converting the resulting files to LCOV format if necessary, and merges them into a single file. + +The interposition of `collect_coverage.sh` is done by the test strategies and requires `collect_coverage.sh` to be on the inputs of the test. This is accomplished by the implicit attribute `:coverage_support` which is resolved to the value of the configuration flag `--coverage_support` (see `TestConfiguration.TestOptions.coverageSupport`) + +Some languages do offline instrumentation, meaning that the coverage instrumentation is added at compile time (such as C++) and others do online instrumentation, meaning that coverage instrumentation is added at execution time. + +Another core concept is *baseline coverage*. This is the coverage of a library, binary, or test if no code in it was run. The problem it solves is that if you want to compute the test coverage for a binary, it is not enough to merge the coverage of all of the tests because there may be code in the binary that is not linked into any test. Therefore, what we do is to emit a coverage file for every binary which contains only the files we collect coverage for with no covered lines. The default baseline coverage file for a target is at `bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are encouraged to generate their own baseline coverage files with more meaningful content than just the names of the source files. + +We track two groups of files for coverage collection for each rule: the set of instrumented files and the set of instrumentation metadata files. + +The set of instrumented files is just that, a set of files to instrument. For online coverage runtimes, this can be used at runtime to decide which files to instrument. It is also used to implement baseline coverage. + +The set of instrumentation metadata files is the set of extra files a test needs to generate the LCOV files Bazel requires from it. In practice, this consists of runtime-specific files; for example, gcc emits .gcno files during compilation. These are added to the set of inputs of test actions if coverage mode is enabled. + +Whether or not coverage is being collected is stored in the `BuildConfiguration`. This is handy because it is an easy way to change the test action and the action graph depending on this bit, but it also means that if this bit is flipped, all targets need to be re-analyzed (some languages, such as C++ require different compiler options to emit code that can collect coverage, which mitigates this issue somewhat, since then a re-analysis is needed anyway). + +The coverage support files are depended on through labels in an implicit dependency so that they can be overridden by the invocation policy, which allows them to differ between the different versions of Bazel. Ideally, these differences would be removed, and we standardized on one of them. + +We also generate a "coverage report" which merges the coverage collected for every test in a Bazel invocation. This is handled by `CoverageReportActionFactory` and is called from `BuildView.createResult()` . It gets access to the tools it needs by looking at the `:coverage_report_generator` attribute of the first test that is executed. ## The query engine -Bazel has a -[little language](/query/guide) -used to ask it various things about various graphs. The following query kinds -are provided: - -* `bazel query` is used to investigate the target graph -* `bazel cquery` is used to investigate the configured target graph -* `bazel aquery` is used to investigate the action graph - -Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. -Additional additional query functions can be done by subclassing `QueryFunction` -. In order to allow streaming query results, instead of collecting them to some -data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which -calls it for results it wants to return. - -The result of a query can be emitted in various ways: labels, labels and rule -classes, XML, protobuf and so on. These are implemented as subclasses of -`OutputFormatter`. - -A subtle requirement of some query output formats (proto, definitely) is that -Bazel needs to emit _all _the information that package loading provides so that -one can diff the output and determine whether a particular target has changed. -As a consequence, attribute values need to be serializable, which is why there -are only so few attribute types without any attributes having complex Starlark -values. The usual workaround is to use a label, and attach the complex -information to the rule with that label. It's not a very satisfying workaround -and it would be very nice to lift this requirement. +Bazel has a [little language](/query/guide) used to ask it various things about various graphs. The following query kinds are provided: + +- `bazel query` is used to investigate the target graph +- `bazel cquery` is used to investigate the configured target graph +- `bazel aquery` is used to investigate the action graph + +Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. Additional additional query functions can be done by subclassing `QueryFunction` . In order to allow streaming query results, instead of collecting them to some data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which calls it for results it wants to return. + +The result of a query can be emitted in various ways: labels, labels and rule classes, XML, protobuf and so on. These are implemented as subclasses of `OutputFormatter`. + +A subtle requirement of some query output formats (proto, definitely) is that Bazel needs to emit \_all \_the information that package loading provides so that one can diff the output and determine whether a particular target has changed. As a consequence, attribute values need to be serializable, which is why there are only so few attribute types without any attributes having complex Starlark values. The usual workaround is to use a label, and attach the complex information to the rule with that label. It's not a very satisfying workaround and it would be very nice to lift this requirement. ## The module system -Bazel can be extended by adding modules to it. Each module must subclass -`BlazeModule` (the name is a relic of the history of Bazel when it used to be -called Blaze) and gets information about various events during the execution of -a command. +Bazel can be extended by adding modules to it. Each module must subclass `BlazeModule` (the name is a relic of the history of Bazel when it used to be called Blaze) and gets information about various events during the execution of a command. -They are mostly used to implement various pieces of "non-core" functionality -that only some versions of Bazel (such as the one we use at Google) need: +They are mostly used to implement various pieces of "non-core" functionality that only some versions of Bazel (such as the one we use at Google) need: -* Interfaces to remote execution systems -* New commands +- Interfaces to remote execution systems +- New commands -The set of extension points `BlazeModule` offers is somewhat haphazard. Don't -use it as an example of good design principles. +The set of extension points `BlazeModule` offers is somewhat haphazard. Don't use it as an example of good design principles. ## The event bus -The main way BlazeModules communicate with the rest of Bazel is by an event bus -(`EventBus`): a new instance is created for every build, various parts of Bazel -can post events to it and modules can register listeners for the events they are -interested in. For example, the following things are represented as events: +The main way BlazeModules communicate with the rest of Bazel is by an event bus (`EventBus`): a new instance is created for every build, various parts of Bazel can post events to it and modules can register listeners for the events they are interested in. For example, the following things are represented as events: -* The list of build targets to be built has been determined - (`TargetParsingCompleteEvent`) -* The top-level configurations have been determined - (`BuildConfigurationEvent`) -* A target was built, successfully or not (`TargetCompleteEvent`) -* A test was run (`TestAttempt`, `TestSummary`) +- The list of build targets to be built has been determined (`TargetParsingCompleteEvent`) +- The top-level configurations have been determined (`BuildConfigurationEvent`) +- A target was built, successfully or not (`TargetCompleteEvent`) +- A test was run (`TestAttempt`, `TestSummary`) -Some of these events are represented outside of Bazel in the -[Build Event Protocol](/remote/bep) -(they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things -outside the Bazel process to observe the build. They are accessible either as a -file that contains protocol messages or Bazel can connect to a server (called -the Build Event Service) to stream events. +Some of these events are represented outside of Bazel in the [Build Event Protocol](/remote/bep) (they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things outside the Bazel process to observe the build. They are accessible either as a file that contains protocol messages or Bazel can connect to a server (called the Build Event Service) to stream events. -This is implemented in the `build.lib.buildeventservice` and -`build.lib.buildeventstream` Java packages. +This is implemented in the `build.lib.buildeventservice` and `build.lib.buildeventstream` Java packages. ## External repositories -Note: The information in this section is out of date, as code in this area has -undergone extensive change in the past couple of years. Please refer to -[external dependencies overview](/external/overview) for more up-to-date -information. +Note: The information in this section is out of date, as code in this area has undergone extensive change in the past couple of years. Please refer to [external dependencies overview](/external/overview) for more up-to-date information. -Whereas Bazel was originally designed to be used in a monorepo (a single source -tree containing everything one needs to build), Bazel lives in a world where -this is not necessarily true. "External repositories" are an abstraction used to -bridge these two worlds: they represent code that is necessary for the build but -is not in the main source tree. +Whereas Bazel was originally designed to be used in a monorepo (a single source tree containing everything one needs to build), Bazel lives in a world where this is not necessarily true. "External repositories" are an abstraction used to bridge these two worlds: they represent code that is necessary for the build but is not in the main source tree. ### The WORKSPACE file -The set of external repositories is determined by parsing the WORKSPACE file. -For example, a declaration like this: +The set of external repositories is determined by parsing the WORKSPACE file. For example, a declaration like this: ``` local_repository(name="foo", path="/foo/bar") ``` -Results in the repository called `@foo` being available. Where this gets -complicated is that one can define new repository rules in Starlark files, which -can then be used to load new Starlark code, which can be used to define new -repository rules and so on… +Results in the repository called `@foo` being available. Where this gets complicated is that one can define new repository rules in Starlark files, which can then be used to load new Starlark code, which can be used to define new repository rules and so on… -To handle this case, the parsing of the WORKSPACE file (in -`WorkspaceFileFunction`) is split up into chunks delineated by `load()` -statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and -computing `WorkspaceFileFunction` until index X means evaluating it until the -Xth `load()` statement. +To handle this case, the parsing of the WORKSPACE file (in `WorkspaceFileFunction`) is split up into chunks delineated by `load()` statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and computing `WorkspaceFileFunction` until index X means evaluating it until the Xth `load()` statement. ### Fetching repositories -Before the code of the repository is available to Bazel, it needs to be -_fetched_. This results in Bazel creating a directory under -`$OUTPUT_BASE/external/`. +Before the code of the repository is available to Bazel, it needs to be *fetched*. This results in Bazel creating a directory under `$OUTPUT_BASE/external/<repository name>`. Fetching the repository happens in the following steps: -1. `PackageLookupFunction` realizes that it needs a repository and creates a - `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` -2. `RepositoryLoaderFunction` forwards the request to - `RepositoryDelegatorFunction` for unclear reasons (the code says it's to - avoid re-downloading things in case of Skyframe restarts, but it's not a - very solid reasoning) -3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to - fetch by iterating over the chunks of the WORKSPACE file until the requested - repository is found -4. The appropriate `RepositoryFunction` is found that implements the repository - fetching; it's either the Starlark implementation of the repository or a - hard-coded map for repositories that are implemented in Java. - -There are various layers of caching since fetching a repository can be very -expensive: - -1. There is a cache for downloaded files that is keyed by their checksum - (`RepositoryCache`). This requires the checksum to be available in the - WORKSPACE file, but that's good for hermeticity anyway. This is shared by - every Bazel server instance on the same workstation, regardless of which - workspace or output base they are running in. -2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` - that contains a checksum of the rule that was used to fetch it. If the Bazel - server restarts but the checksum does not change, it's not re-fetched. This - is implemented in `RepositoryDelegatorFunction.DigestWriter` . -3. The `--distdir` command line option designates another cache that is used to - look up artifacts to be downloaded. This is useful in enterprise settings - where Bazel should not fetch random things from the Internet. This is - implemented by `DownloadManager` . - -Once a repository is downloaded, the artifacts in it are treated as source -artifacts. This poses a problem because Bazel usually checks for up-to-dateness -of source artifacts by calling stat() on them, and these artifacts are also -invalidated when the definition of the repository they are in changes. Thus, -`FileStateValue`s for an artifact in an external repository need to depend on -their external repository. This is handled by `ExternalFilesHelper`. +1. `PackageLookupFunction` realizes that it needs a repository and creates a `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` +2. `RepositoryLoaderFunction` forwards the request to `RepositoryDelegatorFunction` for unclear reasons (the code says it's to avoid re-downloading things in case of Skyframe restarts, but it's not a very solid reasoning) +3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to fetch by iterating over the chunks of the WORKSPACE file until the requested repository is found +4. The appropriate `RepositoryFunction` is found that implements the repository fetching; it's either the Starlark implementation of the repository or a hard-coded map for repositories that are implemented in Java. + +There are various layers of caching since fetching a repository can be very expensive: + +1. There is a cache for downloaded files that is keyed by their checksum (`RepositoryCache`). This requires the checksum to be available in the WORKSPACE file, but that's good for hermeticity anyway. This is shared by every Bazel server instance on the same workstation, regardless of which workspace or output base they are running in. +2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` that contains a checksum of the rule that was used to fetch it. If the Bazel server restarts but the checksum does not change, it's not re-fetched. This is implemented in `RepositoryDelegatorFunction.DigestWriter` . +3. The `--distdir` command line option designates another cache that is used to look up artifacts to be downloaded. This is useful in enterprise settings where Bazel should not fetch random things from the Internet. This is implemented by `DownloadManager` . + +Once a repository is downloaded, the artifacts in it are treated as source artifacts. This poses a problem because Bazel usually checks for up-to-dateness of source artifacts by calling stat() on them, and these artifacts are also invalidated when the definition of the repository they are in changes. Thus, `FileStateValue`s for an artifact in an external repository need to depend on their external repository. This is handled by `ExternalFilesHelper`. ### Repository mappings -It can happen that multiple repositories want to depend on the same repository, -but in different versions (this is an instance of the "diamond dependency -problem"). For example, if two binaries in separate repositories in the build -want to depend on Guava, they will presumably both refer to Guava with labels -starting `@guava//` and expect that to mean different versions of it. - -Therefore, Bazel allows one to re-map external repository labels so that the -string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the -repository of one binary and another Guava repository (such as `@guava2//`) the -repository of the other. - -Alternatively, this can also be used to **join** diamonds. If a repository -depends on `@guava1//`, and another depends on `@guava2//`, repository mapping -allows one to re-map both repositories to use a canonical `@guava//` repository. - -The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute -of individual repository definitions. It then appears in Skyframe as a member of -`WorkspaceFileValue`, where it is plumbed to: - -* `Package.Builder.repositoryMapping` which is used to transform label-valued - attributes of rules in the package by - `RuleClass.populateRuleAttributeValues()` -* `Package.repositoryMapping` which is used in the analysis phase (for - resolving things like `$(location)` which are not parsed in the loading - phase) -* `BzlLoadFunction` for resolving labels in load() statements +It can happen that multiple repositories want to depend on the same repository, but in different versions (this is an instance of the "diamond dependency problem"). For example, if two binaries in separate repositories in the build want to depend on Guava, they will presumably both refer to Guava with labels starting `@guava//` and expect that to mean different versions of it. + +Therefore, Bazel allows one to re-map external repository labels so that the string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the repository of one binary and another Guava repository (such as `@guava2//`) the repository of the other. + +Alternatively, this can also be used to **join** diamonds. If a repository depends on `@guava1//`, and another depends on `@guava2//`, repository mapping allows one to re-map both repositories to use a canonical `@guava//` repository. + +The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute of individual repository definitions. It then appears in Skyframe as a member of `WorkspaceFileValue`, where it is plumbed to: + +- `Package.Builder.repositoryMapping` which is used to transform label-valued attributes of rules in the package by `RuleClass.populateRuleAttributeValues()` +- `Package.repositoryMapping` which is used in the analysis phase (for resolving things like `$(location)` which are not parsed in the loading phase) +- `BzlLoadFunction` for resolving labels in load() statements ## JNI bits -The server of Bazel is _mostly_ written in Java. The exception is the parts that -Java cannot do by itself or couldn't do by itself when we implemented it. This -is mostly limited to interaction with the file system, process control and -various other low-level things. +The server of Bazel is *mostly* written in Java. The exception is the parts that Java cannot do by itself or couldn't do by itself when we implemented it. This is mostly limited to interaction with the file system, process control and various other low-level things. -The C++ code lives under src/main/native and the Java classes with native -methods are: +The C++ code lives under src/main/native and the Java classes with native methods are: -* `NativePosixFiles` and `NativePosixFileSystem` -* `ProcessUtils` -* `WindowsFileOperations` and `WindowsFileProcesses` -* `com.google.devtools.build.lib.platform` +- `NativePosixFiles` and `NativePosixFileSystem` +- `ProcessUtils` +- `WindowsFileOperations` and `WindowsFileProcesses` +- `com.google.devtools.build.lib.platform` ## Console output -Emitting console output seems like a simple thing, but the confluence of running -multiple processes (sometimes remotely), fine-grained caching, the desire to -have a nice and colorful terminal output and having a long-running server makes -it non-trivial. - -Right after the RPC call comes in from the client, two `RpcOutputStream` -instances are created (for stdout and stderr) that forward the data printed into -them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) -pair). Anything that needs to be printed on the console goes through these -streams. Then these streams are handed over to -`BlazeCommandDispatcher.execExclusively()`. - -Output is by default printed with ANSI escape sequences. When these are not -desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In -addition, `System.out` and `System.err` are redirected to these output streams. -This is so that debugging information can be printed using -`System.err.println()` and still end up in the terminal output of the client -(which is different from that of the server). Care is taken that if a process -produces binary output (such as `bazel query --output=proto`), no munging of stdout -takes place. - -Short messages (errors, warnings and the like) are expressed through the -`EventHandler` interface. Notably, these are different from what one posts to -the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, -warning, info, and a few others) and they may have a `Location` (the place in -the source code that caused the event to happen). - -Some `EventHandler` implementations store the events they received. This is used -to replay information to the UI caused by various kinds of cached processing, -for example, the warnings emitted by a cached configured target. - -Some `EventHandler`s also allow posting events that eventually find their way to -the event bus (regular `Event`s do _not _appear there). These are -implementations of `ExtendedEventHandler` and their main use is to replay cached -`EventBus` events. These `EventBus` events all implement `Postable`, but not -everything that is posted to `EventBus` necessarily implements this interface; -only those that are cached by an `ExtendedEventHandler` (it would be nice and -most of the things do; it's not enforced, though) - -Terminal output is _mostly_ emitted through `UiEventHandler`, which is -responsible for all the fancy output formatting and progress reporting Bazel -does. It has two inputs: - -* The event bus -* The event stream piped into it through Reporter - -The only direct connection the command execution machinery (for example the rest of -Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, -which allows direct access to these streams. It's only used when a command needs -to dump large amounts of possible binary data (such as `bazel query`). +Emitting console output seems like a simple thing, but the confluence of running multiple processes (sometimes remotely), fine-grained caching, the desire to have a nice and colorful terminal output and having a long-running server makes it non-trivial. + +Right after the RPC call comes in from the client, two `RpcOutputStream` instances are created (for stdout and stderr) that forward the data printed into them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) pair). Anything that needs to be printed on the console goes through these streams. Then these streams are handed over to `BlazeCommandDispatcher.execExclusively()`. + +Output is by default printed with ANSI escape sequences. When these are not desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In addition, `System.out` and `System.err` are redirected to these output streams. This is so that debugging information can be printed using `System.err.println()` and still end up in the terminal output of the client (which is different from that of the server). Care is taken that if a process produces binary output (such as `bazel query --output=proto`), no munging of stdout takes place. + +Short messages (errors, warnings and the like) are expressed through the `EventHandler` interface. Notably, these are different from what one posts to the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, warning, info, and a few others) and they may have a `Location` (the place in the source code that caused the event to happen). + +Some `EventHandler` implementations store the events they received. This is used to replay information to the UI caused by various kinds of cached processing, for example, the warnings emitted by a cached configured target. + +Some `EventHandler`s also allow posting events that eventually find their way to the event bus (regular `Event`s do \_not \_appear there). These are implementations of `ExtendedEventHandler` and their main use is to replay cached `EventBus` events. These `EventBus` events all implement `Postable`, but not everything that is posted to `EventBus` necessarily implements this interface; only those that are cached by an `ExtendedEventHandler` (it would be nice and most of the things do; it's not enforced, though) + +Terminal output is *mostly* emitted through `UiEventHandler`, which is responsible for all the fancy output formatting and progress reporting Bazel does. It has two inputs: + +- The event bus +- The event stream piped into it through Reporter + +The only direct connection the command execution machinery (for example the rest of Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, which allows direct access to these streams. It's only used when a command needs to dump large amounts of possible binary data (such as `bazel query`). ## Profiling Bazel -Bazel is fast. Bazel is also slow, because builds tend to grow until just the -edge of what's bearable. For this reason, Bazel includes a profiler which can be -used to profile builds and Bazel itself. It's implemented in a class that's -aptly named `Profiler`. It's turned on by default, although it records only -abridged data so that its overhead is tolerable; The command line -`--record_full_profiler_data` makes it record everything it can. - -It emits a profile in the Chrome profiler format; it's best viewed in Chrome. -It's data model is that of task stacks: one can start tasks and end tasks and -they are supposed to be neatly nested within each other. Each Java thread gets -its own task stack. **TODO:** How does this work with actions and -continuation-passing style? - -The profiler is started and stopped in `BlazeRuntime.initProfiler()` and -`BlazeRuntime.afterCommand()` respectively and attempts to be live for as long -as possible so that we can profile everything. To add something to the profile, -call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure -represents the end of the task. It's best used with try-with-resources -statements. - -We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on -and it mostly records maximum heap sizes and GC behavior. +Bazel is fast. Bazel is also slow, because builds tend to grow until just the edge of what's bearable. For this reason, Bazel includes a profiler which can be used to profile builds and Bazel itself. It's implemented in a class that's aptly named `Profiler`. It's turned on by default, although it records only abridged data so that its overhead is tolerable; The command line `--record_full_profiler_data` makes it record everything it can. + +It emits a profile in the Chrome profiler format; it's best viewed in Chrome. It's data model is that of task stacks: one can start tasks and end tasks and they are supposed to be neatly nested within each other. Each Java thread gets its own task stack. **TODO:** How does this work with actions and continuation-passing style? + +The profiler is started and stopped in `BlazeRuntime.initProfiler()` and `BlazeRuntime.afterCommand()` respectively and attempts to be live for as long as possible so that we can profile everything. To add something to the profile, call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure represents the end of the task. It's best used with try-with-resources statements. + +We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on and it mostly records maximum heap sizes and GC behavior. ## Testing Bazel -Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and -ones that only run the analysis phase. We call the former "integration tests" -and the latter "unit tests", although they are more like integration tests that -are, well, less integrated. We also have some actual unit tests, where they are -necessary. +Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and ones that only run the analysis phase. We call the former "integration tests" and the latter "unit tests", although they are more like integration tests that are, well, less integrated. We also have some actual unit tests, where they are necessary. Of integration tests, we have two kinds: -1. Ones implemented using a very elaborate bash test framework under - `src/test/shell` -2. Ones implemented in Java. These are implemented as subclasses of - `BuildIntegrationTestCase` - -`BuildIntegrationTestCase` is the preferred integration testing framework as it -is well-equipped for most testing scenarios. As it is a Java framework, it -provides debuggability and seamless integration with many common development -tools. There are many examples of `BuildIntegrationTestCase` classes in the -Bazel repository. - -Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a -scratch file system you can use to write `BUILD` files, then various helper -methods can request configured targets, change the configuration and assert -various things about the result of the analysis. +1. Ones implemented using a very elaborate bash test framework under `src/test/shell` +2. Ones implemented in Java. These are implemented as subclasses of `BuildIntegrationTestCase` + +`BuildIntegrationTestCase` is the preferred integration testing framework as it is well-equipped for most testing scenarios. As it is a Java framework, it provides debuggability and seamless integration with many common development tools. There are many examples of `BuildIntegrationTestCase` classes in the Bazel repository. + +Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a scratch file system you can use to write `BUILD` files, then various helper methods can request configured targets, change the configuration and assert various things about the result of the analysis. diff --git a/contribute/design-documents.mdx b/contribute/design-documents.mdx index 1fe70b9d..199af600 100644 --- a/contribute/design-documents.mdx +++ b/contribute/design-documents.mdx @@ -2,145 +2,94 @@ title: 'Design Documents' --- - - -If you're planning to add, change, or remove a user-facing feature, or make a -*significant architectural change* to Bazel, you **must** write a design -document and have it reviewed before you can submit the change. +If you're planning to add, change, or remove a user-facing feature, or make a *significant architectural change* to Bazel, you **must** write a design document and have it reviewed before you can submit the change. Here are some examples of significant changes: -* Addition or deletion of native build rules -* Breaking-changes to native rules -* Changes to a native build rule semantics that affect the behavior of more - than a single rule -* Changes to Bazel's rule definition API -* Changes to the APIs that Bazel uses to connect to other systems -* Changes to the Starlark language, semantics, or APIs -* Changes that could have a pervasive effect on Bazel performance or memory - usage (for better or for worse) -* Changes to widely used internal APIs -* Changes to flags and command-line interface. +- Addition or deletion of native build rules +- Breaking-changes to native rules +- Changes to a native build rule semantics that affect the behavior of more than a single rule +- Changes to Bazel's rule definition API +- Changes to the APIs that Bazel uses to connect to other systems +- Changes to the Starlark language, semantics, or APIs +- Changes that could have a pervasive effect on Bazel performance or memory usage (for better or for worse) +- Changes to widely used internal APIs +- Changes to flags and command-line interface. ## Reasons for design reviews -When you write a design document, you can coordinate with other Bazel developers -and seek guidance from Bazel's core team. For example, when a proposal adds, -removes, or modifies any function or object available in BUILD, MODULE.bazel, or -bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. -Design documents are reviewed before submission because: - -* Bazel is a very complex system; seemingly innocuous local changes can have - significant global consequences. -* The team gets many feature requests from users; such requests need to be - evaluated not only for technical feasibility but importance with regards to - other feature requests. -* Bazel features are frequently implemented by people outside the core team; - such contributors have widely varying levels of Bazel expertise. -* The Bazel team itself has varying levels of expertise; no single team member - has a complete understanding of every corner of Bazel. -* Changes to Bazel must account for backward compatibility and avoid breaking - changes. +When you write a design document, you can coordinate with other Bazel developers and seek guidance from Bazel's core team. For example, when a proposal adds, removes, or modifies any function or object available in BUILD, MODULE.bazel, or bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. Design documents are reviewed before submission because: + +- Bazel is a very complex system; seemingly innocuous local changes can have significant global consequences. +- The team gets many feature requests from users; such requests need to be evaluated not only for technical feasibility but importance with regards to other feature requests. +- Bazel features are frequently implemented by people outside the core team; such contributors have widely varying levels of Bazel expertise. +- The Bazel team itself has varying levels of expertise; no single team member has a complete understanding of every corner of Bazel. +- Changes to Bazel must account for backward compatibility and avoid breaking changes. Bazel's design review policy helps to maximize the likelihood that: -* all feature requests get a baseline level of scrutiny. -* the right people will weigh in on designs before we've invested in an - implementation that may not work. +- all feature requests get a baseline level of scrutiny. +- the right people will weigh in on designs before we've invested in an implementation that may not work. -To help you get started, take a look at the design documents in the -[Bazel Proposals Repository](https://github.com/bazelbuild/proposals). -Designs are works in progress, so implementation details can change over time -and with feedback. The published design documents capture the initial design, -and *not* the ongoing changes as designs are implemented. Always go to the -documentation for descriptions of current Bazel functionality. +To help you get started, take a look at the design documents in the [Bazel Proposals Repository](https://github.com/bazelbuild/proposals). Designs are works in progress, so implementation details can change over time and with feedback. The published design documents capture the initial design, and *not* the ongoing changes as designs are implemented. Always go to the documentation for descriptions of current Bazel functionality. ## Contributor Workflow -As a contributor, you can write a design document, send pull requests and -request reviewers for your proposal. +As a contributor, you can write a design document, send pull requests and request reviewers for your proposal. ### Write the design document All design documents must have a header that includes: -* author -* date of last major change -* list of reviewers, including one (and only one) - [lead reviewer](#lead-reviewer) -* current status (_draft_, _in review_, _approved_, _rejected_, - _being implemented_, _implemented_) -* link to discussion thread (_to be added after the announcement_) +- author +- date of last major change +- list of reviewers, including one (and only one) [lead reviewer](#lead-reviewer) +- current status (*draft*, *in review*, *approved*, *rejected*, *being implemented*, *implemented*) +- link to discussion thread (*to be added after the announcement*) -The document can be written either [as a world-readable Google Doc](#gdocs) -or [using Markdown](#markdown). Read below about for a -[Markdown / Google Docs comparison](#markdown-versus-gdocs). +The document can be written either [as a world-readable Google Doc](#gdocs) or [using Markdown](#markdown). Read below about for a [Markdown / Google Docs comparison](#markdown-versus-gdocs). -Proposals that have a user-visible impact must have a section documenting the -impact on backward compatibility (and a rollout plan if needed). +Proposals that have a user-visible impact must have a section documenting the impact on backward compatibility (and a rollout plan if needed). ### Create a Pull Request -Share your design doc by creating a pull request (PR) to add the document to -[the design index](https://github.com/bazelbuild/proposals). Add -your markdown file or a document link to your PR. +Share your design doc by creating a pull request (PR) to add the document to [the design index](https://github.com/bazelbuild/proposals). Add your markdown file or a document link to your PR. -When possible, [choose a lead reviewer](#lead-reviewer). -and cc other reviewers. If you don't choose a lead reviewer, a Bazel -maintainer will assign one to your PR. +When possible, [choose a lead reviewer](#lead-reviewer). and cc other reviewers. If you don't choose a lead reviewer, a Bazel maintainer will assign one to your PR. -After you create your PR, reviewers can make preliminary comments during the -code review. For example, the lead reviewer can suggest extra reviewers, or -point out missing information. The lead reviewer approves the PR when they -believe the review process can start. This doesn't mean the proposal is perfect -or will be approved; it means that the proposal contains enough information to -start the discussion. +After you create your PR, reviewers can make preliminary comments during the code review. For example, the lead reviewer can suggest extra reviewers, or point out missing information. The lead reviewer approves the PR when they believe the review process can start. This doesn't mean the proposal is perfect or will be approved; it means that the proposal contains enough information to start the discussion. ### Announce the new proposal -Send an announcement to -[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when -the PR is submitted. +Send an announcement to [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when the PR is submitted. -You may copy other groups (for example, -[bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), -to get feedback from Bazel end-users). +You may copy other groups (for example, [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), to get feedback from Bazel end-users). ### Iterate with reviewers -Anyone interested can comment on your proposal. Try to answer questions, -clarify the proposal, and address concerns. +Anyone interested can comment on your proposal. Try to answer questions, clarify the proposal, and address concerns. -Discussion should happen on the announcement thread. If the proposal is in a -Google Doc, comments may be used instead (Note that anonymous comments are -allowed). +Discussion should happen on the announcement thread. If the proposal is in a Google Doc, comments may be used instead (Note that anonymous comments are allowed). ### Update the status -Create a new PR to update the status of the proposal, when iteration is -complete. Send the PR to the same lead reviewer and cc the other reviewers. +Create a new PR to update the status of the proposal, when iteration is complete. Send the PR to the same lead reviewer and cc the other reviewers. -To officially accept the proposal, the lead reviewer approves the PR after -ensuring that the other reviewers agree with the decision. +To officially accept the proposal, the lead reviewer approves the PR after ensuring that the other reviewers agree with the decision. -There must be at least 1 week between the first announcement and the approval of -a proposal. This ensures that users had enough time to read the document and -share their concerns. +There must be at least 1 week between the first announcement and the approval of a proposal. This ensures that users had enough time to read the document and share their concerns. -Implementation can begin before the proposal is accepted, for example as a -proof-of-concept or an experimentation. However, you cannot submit the change -before the review is complete. +Implementation can begin before the proposal is accepted, for example as a proof-of-concept or an experimentation. However, you cannot submit the change before the review is complete. ### Choosing a lead reviewer A lead reviewer should be a domain expert who is: -* Knowledgeable of the relevant subsystems -* Objective and capable of providing constructive feedback -* Available for the entire review period to lead the process +- Knowledgeable of the relevant subsystems +- Objective and capable of providing constructive feedback +- Available for the entire review period to lead the process -Consider checking the contacts for various [team -labels](/contribute/maintainers-guide#team-labels). +Consider checking the contacts for various [team labels](/contribute/maintainers-guide#team-labels). ## Markdown vs Google Docs @@ -148,49 +97,34 @@ Decide what works best for you, since both are accepted. Benefits of using Google Docs: -* Effective for brainstorming, since it is easy to get started with. -* Collaborative editing. -* Quick iteration. -* Easy way to suggest edits. +- Effective for brainstorming, since it is easy to get started with. +- Collaborative editing. +- Quick iteration. +- Easy way to suggest edits. Benefits of using Markdown files: -* Clean URLs for linking. -* Explicit record of revisions. -* No forgetting to set up access rights before publicizing a link. -* Easily searchable with search engines. -* Future-proof: Plain text is not at the mercy of any specific tool - and doesn't require an Internet connection. -* It is possible to update them even if the author is not around anymore. -* They can be processed automatically (update/detect dead links, fetch - list of authors, etc.). +- Clean URLs for linking. +- Explicit record of revisions. +- No forgetting to set up access rights before publicizing a link. +- Easily searchable with search engines. +- Future-proof: Plain text is not at the mercy of any specific tool and doesn't require an Internet connection. +- It is possible to update them even if the author is not around anymore. +- They can be processed automatically (update/detect dead links, fetch list of authors, etc.). -You can choose to first iterate on a Google Doc, and then convert it to -Markdown for posterity. +You can choose to first iterate on a Google Doc, and then convert it to Markdown for posterity. ### Using Google Docs -For consistency, use the [Bazel design doc template]( -https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). -It includes the necessary header and creates visual -consistency with other Bazel related documents. To do that, click on **File** > -**Make a copy** or click this link to [make a copy of the design doc -template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). +For consistency, use the [Bazel design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). It includes the necessary header and creates visual consistency with other Bazel related documents. To do that, click on **File** > **Make a copy** or click this link to [make a copy of the design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). -To make your document readable to the world, click on -**Share** > **Advanced** > **Change…**, and -choose "On - Anyone with the link". If you allow comments on the document, -anyone can comment anonymously, even without a Google account. +To make your document readable to the world, click on **Share** > **Advanced** > **Change…**, and choose "On - Anyone with the link". If you allow comments on the document, anyone can comment anonymously, even without a Google account. ### Using Markdown -Documents are stored on GitHub and use the -[GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) -([Specification](https://github.github.com/gfm/)). +Documents are stored on GitHub and use the [GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) ([Specification](https://github.github.com/gfm/)). -Create a PR to update an existing document. Significant changes should be -reviewed by the document reviewers. Trivial changes (such as typos, formatting) -can be approved by anyone. +Create a PR to update an existing document. Significant changes should be reviewed by the document reviewers. Trivial changes (such as typos, formatting) can be approved by anyone. ## Reviewer workflow @@ -198,57 +132,41 @@ A reviewer comments, reviews and approves design documents. ### General reviewer responsibilities -You're responsible for reviewing design documents, asking for additional -information if needed, and approving a design that passes the review process. +You're responsible for reviewing design documents, asking for additional information if needed, and approving a design that passes the review process. #### When you receive a new proposal -1. Take a quick look at the document. -1. Comment if critical information is missing, or if the design doesn't fit - with the goals of the project. -1. Suggest additional reviewers. -1. Approve the PR when it is ready for review. +1. Take a quick look at the document. +2. Comment if critical information is missing, or if the design doesn't fit with the goals of the project. +3. Suggest additional reviewers. +4. Approve the PR when it is ready for review. #### During the review process -1. Engage in a dialogue with the design author about issues that are problematic - or require clarification. -1. If appropriate, invite comments from non-reviewers who should be aware of - the design. -1. Decide which comments must be addressed by the author as a prerequisite to - approval. -1. Write "LGTM" (_Looks Good To Me_) in the discussion thread when you are - happy with the current state of the proposal. +1. Engage in a dialogue with the design author about issues that are problematic or require clarification. +2. If appropriate, invite comments from non-reviewers who should be aware of the design. +3. Decide which comments must be addressed by the author as a prerequisite to approval. +4. Write "LGTM" (*Looks Good To Me*) in the discussion thread when you are happy with the current state of the proposal. -Follow this process for all design review requests. Do not approve designs -affecting Bazel if they are not in the -[design index](https://github.com/bazelbuild/proposals). +Follow this process for all design review requests. Do not approve designs affecting Bazel if they are not in the [design index](https://github.com/bazelbuild/proposals). ### Lead reviewer responsibilities -You're responsible for making the go / no-go decision on implementation -of a pending design. If you're not able to do this, you should identify a -suitable delegate (reassign the PR to the delegate), or reassign the bug to a -Bazel manager for further disposition. +You're responsible for making the go / no-go decision on implementation of a pending design. If you're not able to do this, you should identify a suitable delegate (reassign the PR to the delegate), or reassign the bug to a Bazel manager for further disposition. #### During the review process -1. Ensure that the comment and design iteration process moves forward - constructively. -1. Prior to approval, ensure that concerns from other reviewers have been - resolved. +1. Ensure that the comment and design iteration process moves forward constructively. +2. Prior to approval, ensure that concerns from other reviewers have been resolved. #### After approval by all reviewers -1. Make sure there has been at least 1 week since the announcement on the - mailing list. -1. Make sure the PR updates the status. -1. Approve the PR sent by the proposal author. +1. Make sure there has been at least 1 week since the announcement on the mailing list. +2. Make sure the PR updates the status. +3. Approve the PR sent by the proposal author. #### Rejecting designs -1. Make sure the PR author sends a PR; or send them a PR. -1. The PR updates the status of the document. -1. Add a comment to the document explaining why the design can't be approved in - its current state, and outlining next steps, if any (such as "revisit invalid - assumptions and resubmit"). +1. Make sure the PR author sends a PR; or send them a PR. +2. The PR updates the status of the document. +3. Add a comment to the document explaining why the design can't be approved in its current state, and outlining next steps, if any (such as "revisit invalid assumptions and resubmit"). diff --git a/contribute/docs-style-guide.mdx b/contribute/docs-style-guide.mdx new file mode 100644 index 00000000..a2e60e67 --- /dev/null +++ b/contribute/docs-style-guide.mdx @@ -0,0 +1,176 @@ +--- +title: 'Bazel docs style guide' +--- + +Thank you for contributing to Bazel's documentation. This serves as a quick documentation style guide to get you started. For any style questions not answered by this guide, follow the [Google developer documentation style guide](https://developers.google.com/style). + +## Defining principles + +Bazel docs should uphold these principles: + +- **Concise.** Use as few words as possible. +- **Clear.** Use plain language. Write without jargon for a fifth-grade reading level. +- **Consistent.** Use the same words or phrases for repeated concepts throughout the docs. +- **Correct.** Write in a way where the content stays correct for as long as possible by avoiding time-based information and promises for the future. + +## Writing + +This section contains basic writing tips. + +### Headings + +- Page-level headings start at H2. (H1 headings are used as page titles.) + +- Make headers as short as is sensible. This way, they fit in the TOC without wrapping. + + - ✅ Yes: : Permissions + - ⚠️ No: : A brief note on permissions + +- Use sentence case for headings + + - ✅ Yes: : Set up your workspace + - ⚠️ No: : Set Up Your Workspace + +- Try to make headings task-based or actionable. If headings are conceptual, it may be based around understanding, but write to what the user does. + + - ✅ Yes: : Preserving graph order + - ⚠️ No: : On the preservation of graph order + +### Names + +- Capitalize proper nouns, such as Bazel and Starlark. + + - ✅ Yes: : At the end of the build, Bazel prints the requested targets. + - ⚠️ No: : At the end of the build, bazel prints the requested targets. + +- Keep it consistent. Don't introduce new names for existing concepts. Where applicable, use the term defined in the [Glossary](/reference/glossary). + + - For example, if you're writing about issuing commands on a terminal, don't use both terminal and command line on the page. + +### Page scope + +- Each page should have one purpose and that should be defined at the beginning. This helps readers find what they need quicker. + + - ✅ Yes: : This page covers how to install Bazel on Windows. + - ⚠️ No: : (No introductory sentence.) + +- At the end of the page, tell the reader what to do next. For pages where there is no clear action, you can include links to similar concepts, examples, or other avenues for exploration. + +### Subject + +In Bazel documentation, the audience should primarily be users—the people using Bazel to build their software. + +- Address your reader as "you". (If for some reason you can't use "you", use gender-neutral language, such as they.) + + - ✅ Yes: : To build Java code using Bazel, you must install a JDK. + - **MAYBE:** For users to build Java code with Bazel, they must install a JDK. + - ⚠️ No: : For a user to build Java code with Bazel, he or she must install a JDK. + +- If your audience is NOT general Bazel users, define the audience at the beginning of the page or in the section. Other audiences can include maintainers, contributors, migrators, or other roles. + +- Avoid "we". In user docs, there is no author; just tell people what's possible. + + - ✅ Yes: : As Bazel evolves, you should update your code base to maintain compatibility. + - ⚠️ No: : Bazel is evolving, and we will make changes to Bazel that at times will be incompatible and require some changes from Bazel users. + +### Temporal + +Where possible, avoid terms that orient things in time, such as referencing specific dates (Q2 2022) or saying "now", "currently", or "soon." These go stale quickly and could be incorrect if it's a future projection. Instead, specify a version level instead, such as "Bazel X.x and higher supports \ or a GitHub issue link. + +- ✅ Yes: : Bazel 0.10.0 or later supports remote caching. +- ⚠️ No: : Bazel will soon support remote caching, likely in October 2017. + +### Tense + +- Use present tense. Avoid past or future tense unless absolutely necessary for clarity. + + - ✅ Yes: : Bazel issues an error when it finds dependencies that don't conform to this rule. + - ⚠️ No: : If Bazel finds a dependency that does not conform to this rule, Bazel will issue an error. + +- Where possible, use active voice (where a subject acts upon an object) not passive voice (where an object is acted upon by a subject). Generally, active voice makes sentences clearer because it shows who is responsible. If using active voice detracts from clarity, use passive voice. + + - ✅ Yes: : Bazel initiates X and uses the output to build Y. + - ⚠️ No: : X is initiated by Bazel and then afterward Y will be built with the output. + +### Tone + +Write with a business friendly tone. + +- Avoid colloquial language. It's harder to translate phrases that are specific to English. + + - ✅ Yes: : Good rulesets + - ⚠️ No: : So what is a good ruleset? + +- Avoid overly formal language. Write as though you're explaining the concept to someone who is curious about tech, but doesn't know the details. + +## Formatting + +### File type + +For readability, wrap lines at 80 characters. Long links or code snippets may be longer, but should start on a new line. For example: + +Note: Where possible, use Markdown instead of HTML in your files. Follow the [GitHub Markdown Syntax Guide](https://guides.github.com/features/mastering-markdown/#syntax) for recommended Markdown style. + +### Links + +- Use descriptive link text instead of "here" or "below". This practice makes it easier to scan a doc and is better for screen readers. + + - ✅ Yes: : For more details, see \[Installing Bazel]. + - ⚠️ No: : For more details, see \[here]. + +- End the sentence with the link, if possible. + + - ✅ Yes: : For more details, see \[link]. + - ⚠️ No: : See \[link] for more information. + +### Lists + +- Use an ordered list to describe how to accomplish a task with steps + +- Use an unordered list to list things that aren't task based. (There should still be an order of sorts, such as alphabetical, importance, etc.) + +- Write with parallel structure. For example: + + 1. Make all the list items sentences. + 2. Start with verbs that are the same tense. + 3. Use an ordered list if there are steps to follow. + +### Placeholders + +- Use angle brackets to denote a variable that users should change. In Markdown, escape the angle brackets with a back slash: `\<example\>`. + + - ✅ Yes: : `bazel help <command>`: Prints help and options for `<command>` + - ⚠️ No: : bazel help *command*: Prints help and options for "command" + +- Especially for complicated code samples, use placeholders that make sense in context. + +### Table of contents + +Use the auto-generated TOC supported by the site. Don't add a manual TOC. + +## Code + +Code samples are developers' best friends. You probably know how to write these already, but here are a few tips. + +If you're referencing a small snippet of code, you can embed it in a sentence. If you want the reader to use the code, such as copying a command, use a code block. + +### Code blocks + +- Keep it short. Eliminate all redundant or unnecessary text from a code sample. +- In Markdown, specify the type of code block by adding the sample's language. + +```` +```shell +... +```` + +- Separate commands and output into different code blocks. + +### Inline code formatting + +- Use code style for filenames, directories, paths, and small bits of code. + +- Use inline code styling instead of *italics*, "quotes," or **bolding**. + + - ✅ Yes: : `bazel help <command>`: Prints help and options for `<command>` + - ⚠️ No: : bazel help *command*: Prints help and options for "command" diff --git a/contribute/docs.mdx b/contribute/docs.mdx index cc240cc4..940b20e2 100644 --- a/contribute/docs.mdx +++ b/contribute/docs.mdx @@ -2,42 +2,26 @@ title: 'Contribute to Bazel documentation' --- - - -Thank you for contributing to Bazel's documentation! There are a few ways to -help create better docs for our community. +Thank you for contributing to Bazel's documentation! There are a few ways to help create better docs for our community. ## Documentation types This site includes a few types of content. - - *Narrative documentation*, which is written by technical writers and - engineers. Most of this site is narrative documentation that covers - conceptual and task-based guides. - - *Reference documentation*, which is generated documentation from code comments. - You can't make changes to the reference doc pages directly, but instead need - to change their source. +- *Narrative documentation*, which is written by technical writers and engineers. Most of this site is narrative documentation that covers conceptual and task-based guides. +- *Reference documentation*, which is generated documentation from code comments. You can't make changes to the reference doc pages directly, but instead need to change their source. ## Documentation infrastructure -Bazel documentation is served from Google and the source files are mirrored in -Bazel's GitHub repository. You can make changes to the source files in GitHub. -If approved, you can merge the changes and a Bazel maintainer will update the -website source to publish your updates. - +Bazel documentation is served from Google and the source files are mirrored in Bazel's GitHub repository. You can make changes to the source files in GitHub. If approved, you can merge the changes and a Bazel maintainer will update the website source to publish your updates. ## Small changes -You can approach small changes, such as fixing errors or typos, in a couple of -ways. +You can approach small changes, such as fixing errors or typos, in a couple of ways. - - **Pull request**. You can create a pull request in GitHub with the - [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. - - **Bug**. You can file a bug with details and suggested changes and the Bazel - documentation owners will make the update. +- **Pull request**. You can create a pull request in GitHub with the [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. +- **Bug**. You can file a bug with details and suggested changes and the Bazel documentation owners will make the update. ## Large changes -If you want to make substantial changes to existing documentation or propose -new documentation, you can either create a pull request or start with a Google -doc and contact the Bazel Owners to collaborate. +If you want to make substantial changes to existing documentation or propose new documentation, you can either create a pull request or start with a Google doc and contact the Bazel Owners to collaborate. diff --git a/contribute/index.mdx b/contribute/index.mdx index ee667729..18b12854 100644 --- a/contribute/index.mdx +++ b/contribute/index.mdx @@ -2,57 +2,41 @@ title: 'Contributing to Bazel' --- - - There are many ways to help the Bazel project and ecosystem. ## Provide feedback -As you use Bazel, you may find things that can be improved. -You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) -when: +As you use Bazel, you may find things that can be improved. You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) when: - - Bazel crashes or you encounter a bug that can [only be resolved using `bazel - clean`](/run/build#correct-incremental-rebuilds). - - The documentation is incomplete or unclear. You can also report issues - from the page you are viewing by using the "Create issue" - link at the top right corner of the page. - - An error message could be improved. +- Bazel crashes or you encounter a bug that can [only be resolved using `bazel clean`](/run/build#correct-incremental-rebuilds). +- The documentation is incomplete or unclear. You can also report issues from the page you are viewing by using the "Create issue" link at the top right corner of the page. +- An error message could be improved. ## Participate in the community You can engage with the Bazel community by: - - Answering questions [on Stack Overflow]( - https://stackoverflow.com/questions/tagged/bazel). - - Helping other users [on Slack](https://slack.bazel.build). - - Improving documentation or [contributing examples]( - https://github.com/bazelbuild/examples). - - Sharing your experience or your tips, for example, on a blog or social media. +- Answering questions [on Stack Overflow](https://stackoverflow.com/questions/tagged/bazel). +- Helping other users [on Slack](https://slack.bazel.build). +- Improving documentation or [contributing examples](https://github.com/bazelbuild/examples). +- Sharing your experience or your tips, for example, on a blog or social media. ## Contribute code -Bazel is a large project and making a change to the Bazel source code -can be difficult. +Bazel is a large project and making a change to the Bazel source code can be difficult. You can contribute to the Bazel ecosystem by: - - Helping rules maintainers by contributing pull requests. - - Creating new rules and open-sourcing them. - - Contributing to Bazel-related tools, for example, migration tools. - - Improving Bazel integration with other IDEs and tools. +- Helping rules maintainers by contributing pull requests. +- Creating new rules and open-sourcing them. +- Contributing to Bazel-related tools, for example, migration tools. +- Improving Bazel integration with other IDEs and tools. -Before making a change, [create a GitHub -issue](http://github.com/bazelbuild/bazel/issues) -or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). +Before making a change, [create a GitHub issue](http://github.com/bazelbuild/bazel/issues) or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). -The most helpful contributions fix bugs or add features (as opposed -to stylistic, refactoring, or "cleanup" changes). Your change should -include tests and documentation, keeping in mind backward-compatibility, -portability, and the impact on memory usage and performance. +The most helpful contributions fix bugs or add features (as opposed to stylistic, refactoring, or "cleanup" changes). Your change should include tests and documentation, keeping in mind backward-compatibility, portability, and the impact on memory usage and performance. -To learn about how to submit a change, see the -[patch acceptance process](/contribute/patch-acceptance). +To learn about how to submit a change, see the [patch acceptance process](/contribute/patch-acceptance). ## Bazel's code description @@ -60,23 +44,19 @@ Bazel has a large codebase with code in multiple locations. See the [codebase gu Bazel is organized as follows: -* Client code is in `src/main/cpp` and provides the command-line interface. -* Protocol buffers are in `src/main/protobuf`. -* Server code is in `src/main/java` and `src/test/java`. - * Core code which is mostly composed of [SkyFrame](/reference/skyframe) - and some utilities. - * Built-in rules are in `com.google.devtools.build.lib.rules` and in - `com.google.devtools.build.lib.bazel.rules`. You might want to read about - the [Challenges of Writing Rules](/rules/challenges) first. -* Java native interfaces are in `src/main/native`. -* Various tooling for language support are described in the list in the - [compiling Bazel](/install/compile-source) section. +- Client code is in `src/main/cpp` and provides the command-line interface. + +- Protocol buffers are in `src/main/protobuf`. + +- Server code is in `src/main/java` and `src/test/java`. + + - Core code which is mostly composed of [SkyFrame](/reference/skyframe) and some utilities. + - Built-in rules are in `com.google.devtools.build.lib.rules` and in `com.google.devtools.build.lib.bazel.rules`. You might want to read about the [Challenges of Writing Rules](/rules/challenges) first. + +- Java native interfaces are in `src/main/native`. +- Various tooling for language support are described in the list in the [compiling Bazel](/install/compile-source) section. ### Searching Bazel's source code -To quickly search through Bazel's source code, use -[Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's -repositories, branches, and files. You can also view history, diffs, and blame -information. To learn more, see the -[Bazel Code Search User Guide](/contribute/search). +To quickly search through Bazel's source code, use [Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's repositories, branches, and files. You can also view history, diffs, and blame information. To learn more, see the [Bazel Code Search User Guide](/contribute/search). diff --git a/contribute/maintainers-guide.mdx b/contribute/maintainers-guide.mdx index 9d745afb..b98a408e 100644 --- a/contribute/maintainers-guide.mdx +++ b/contribute/maintainers-guide.mdx @@ -2,205 +2,141 @@ title: 'Guide for Bazel Maintainers' --- - - This is a guide for the maintainers of the Bazel open source project. -If you are looking to contribute to Bazel, please read [Contributing to -Bazel](/contribute) instead. +If you are looking to contribute to Bazel, please read [Contributing to Bazel](/contribute) instead. The objectives of this page are to: -1. Serve as the maintainers' source of truth for the project’s contribution - process. -1. Set expectations between the community contributors and the project - maintainers. +1. Serve as the maintainers' source of truth for the project’s contribution process. +2. Set expectations between the community contributors and the project maintainers. -Bazel's [core group of contributors](/contribute/policy) has dedicated -subteams to manage aspects of the open source project. These are: +Bazel's [core group of contributors](/contribute/policy) has dedicated subteams to manage aspects of the open source project. These are: -* **Release Process**: Manage Bazel's release process. -* **Green Team**: Grow a healthy ecosystem of rules and tools. -* **Developer Experience Gardeners**: Encourage external contributions, review - issues and pull requests, and make our development workflow more open. +- **Release Process**: Manage Bazel's release process. +- **Green Team**: Grow a healthy ecosystem of rules and tools. +- **Developer Experience Gardeners**: Encourage external contributions, review issues and pull requests, and make our development workflow more open. ## Releases -* [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) -* [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) +- [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) +- [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) ## Continuous Integration -Read the Green team's guide to Bazel's CI infrastructure on the -[bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) -repository. +Read the Green team's guide to Bazel's CI infrastructure on the [bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) repository. ## Lifecycle of an Issue -1. A user creates an issue by choosing one of the -[issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) - and it enters the pool of [unreviewed open - issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). -1. A member on the Developer Experience (DevEx) subteam rotation reviews the - issue. - 1. If the issue is **not a bug** or a **feature request**, the DevEx member - will usually close the issue and redirect the user to - [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and - [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for - higher visibility on the question. - 1. If the issue belongs in one of the rules repositories owned by the - community, like [rules_apple](https://github.com.bazelbuild/rules_apple), - the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) - to the correct repository. - 1. If the issue is vague or has missing information, the DevEx member will - assign the issue back to the user to request for more information before - continuing. This usually occurs when the user does not choose the right - [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) - or provides incomplete information. -1. After reviewing the issue, the DevEx member decides if the issue requires - immediate attention. If it does, they will assign the **P0** - [priority](#priority) label and an owner from the list of team leads. -1. The DevEx member assigns the `untriaged` label and exactly one [team - label](#team-labels) for routing. -1. The DevEx member also assigns exactly one `type:` label, such as `type: bug` - or `type: feature request`, according to the type of the issue. -1. For platform-specific issues, the DevEx member assigns one `platform:` label, - such as `platform:apple` for Mac-specific issues. -1. If the issue is low priority and can be worked on by a new community - contributor, the DevEx member assigns the `good first issue` label. -At this stage, the issue enters the pool of [untriaged open -issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). - -Each Bazel subteam will triage all issues under labels they own, preferably on a -weekly basis. The subteam will review and evaluate the issue and provide a -resolution, if possible. If you are an owner of a team label, see [this section -](#label-own) for more information. +1. A user creates an issue by choosing one of the [issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) and it enters the pool of [unreviewed open issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93\&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). + +2. A member on the Developer Experience (DevEx) subteam rotation reviews the issue. + + 1. If the issue is **not a bug** or a **feature request**, the DevEx member will usually close the issue and redirect the user to [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for higher visibility on the question. + 2. If the issue belongs in one of the rules repositories owned by the community, like [rules\_apple](https://github.com.bazelbuild/rules_apple), the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) to the correct repository. + 3. If the issue is vague or has missing information, the DevEx member will assign the issue back to the user to request for more information before continuing. This usually occurs when the user does not choose the right [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) or provides incomplete information. + +3. After reviewing the issue, the DevEx member decides if the issue requires immediate attention. If it does, they will assign the **P0** [priority](#priority) label and an owner from the list of team leads. + +4. The DevEx member assigns the `untriaged` label and exactly one [team label](#team-labels) for routing. + +5. The DevEx member also assigns exactly one `type:` label, such as `type: bug` or `type: feature request`, according to the type of the issue. + +6. For platform-specific issues, the DevEx member assigns one `platform:` label, such as `platform:apple` for Mac-specific issues. + +7. If the issue is low priority and can be worked on by a new community contributor, the DevEx member assigns the `good first issue` label. At this stage, the issue enters the pool of [untriaged open issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). + +Each Bazel subteam will triage all issues under labels they own, preferably on a weekly basis. The subteam will review and evaluate the issue and provide a resolution, if possible. If you are an owner of a team label, see [this section ](#label-own)for more information. When an issue is resolved, it can be closed. ## Lifecycle of a Pull Request 1. A user creates a pull request. -1. If you a member of a Bazel team and sending a PR against your own area, - you are responsible for assigning your team label and finding the best - reviewer. -1. Otherwise, during daily triage, a DevEx member assigns one - [team label](#team-labels) and the team's technical lead (TL) for routing. +2. If you a member of a Bazel team and sending a PR against your own area, you are responsible for assigning your team label and finding the best reviewer. +3. Otherwise, during daily triage, a DevEx member assigns one [team label](#team-labels) and the team's technical lead (TL) for routing. 1. The TL may optionally assign someone else to review the PR. -1. The assigned reviewer reviews the PR and works with the author until it is - approved or dropped. -1. If approved, the reviewer **imports** the PR's commit(s) into Google's - internal version control system for further tests. As Bazel is the same build - system used internally at Google, we need to test all PR commits against the - internal test suite. This is the reason why we do not merge PRs directly. -1. If the imported commit passes all internal tests, the commit will be squashed - and exported back out to GitHub. -1. When the commit merges into master, GitHub automatically closes the PR. - +4. The assigned reviewer reviews the PR and works with the author until it is approved or dropped. +5. If approved, the reviewer **imports** the PR's commit(s) into Google's internal version control system for further tests. As Bazel is the same build system used internally at Google, we need to test all PR commits against the internal test suite. This is the reason why we do not merge PRs directly. +6. If the imported commit passes all internal tests, the commit will be squashed and exported back out to GitHub. +7. When the commit merges into master, GitHub automatically closes the PR. ## My team owns a label. What should I do? -Subteams need to triage all issues in the [labels they own](#team-labels), -preferably on a weekly basis. +Subteams need to triage all issues in the [labels they own](#team-labels), preferably on a weekly basis. ### Issues 1. Filter the list of issues by your team label **and** the `untriaged` label. -1. Review the issue. -1. Identify a [priority level](#priority) and assign the label. - 1. The issue may have already been prioritized by the DevEx subteam if it's a - P0. Re-prioritize if needed. - 1. Each issue needs to have exactly one [priority label](#priority). If an - issue is either P0 or P1 we assume that is actively worked on. -1. Remove the `untriaged` label. +2. Review the issue. +3. Identify a [priority level](#priority) and assign the label. +4. The issue may have already been prioritized by the DevEx subteam if it's a P0. Re-prioritize if needed. +5. Each issue needs to have exactly one [priority label](#priority). If an issue is either P0 or P1 we assume that is actively worked on. +6. Remove the `untriaged` label. -Note that you need to be in the [bazelbuild -organization](https://github.com/bazelbuild) to be able to add or remove labels. +Note that you need to be in the [bazelbuild organization](https://github.com/bazelbuild) to be able to add or remove labels. ### Pull Requests 1. Filter the list of pull requests by your team label. -1. Review open pull requests. - 1. **Optional**: If you are assigned for the review but is not the right fit - for it, re-assign the appropriate reviewer to perform a code review. -1. Work with the pull request creator to complete a code review. -1. Approve the PR. -1. Ensure that all tests pass. -1. Import the patch to the internal version control system and run the internal - presubmits. -1. Submit the internal patch. If the patch submits and exports successfully, the - PR will be closed automatically by GitHub. +2. Review open pull requests. +3. **Optional**: If you are assigned for the review but is not the right fit for it, re-assign the appropriate reviewer to perform a code review. +4. Work with the pull request creator to complete a code review. +5. Approve the PR. +6. Ensure that all tests pass. +7. Import the patch to the internal version control system and run the internal presubmits. +8. Submit the internal patch. If the patch submits and exports successfully, the PR will be closed automatically by GitHub. ## Priority -The following definitions for priority will be used by the maintainers to triage -issues. - -* [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken - functionality that causes a Bazel release (minus release candidates) to be - unusable, or a downed service that severely impacts development of the Bazel - project. This includes regressions introduced in a new release that blocks a - significant number of users, or an incompatible breaking change that was not - compliant to the [Breaking - Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) - policy. No practical workaround exists. -* [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or - feature which should be addressed in the next release, or a serious issue that - impacts many users (including the development of the Bazel project), but a - practical workaround exists. Typically does not require immediate action. In - high demand and planned in the current quarter's roadmap. -* [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature - that should be addressed but we don't currently work on. Moderate live issue - in a released Bazel version that is inconvenient for a user that needs to be - addressed in an future release and/or an easy workaround exists. -* [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug - fix or enhancement with small impact. Not prioritized into Bazel roadmaps or - any imminent release, however community contributions are encouraged. -* [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect - or feature request that is unlikely to get closed. Can also be kept open for a - potential re-prioritization if more users are impacted. +The following definitions for priority will be used by the maintainers to triage issues. + +- [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken functionality that causes a Bazel release (minus release candidates) to be unusable, or a downed service that severely impacts development of the Bazel project. This includes regressions introduced in a new release that blocks a significant number of users, or an incompatible breaking change that was not compliant to the [Breaking Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) policy. No practical workaround exists. +- [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or feature which should be addressed in the next release, or a serious issue that impacts many users (including the development of the Bazel project), but a practical workaround exists. Typically does not require immediate action. In high demand and planned in the current quarter's roadmap. +- [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature that should be addressed but we don't currently work on. Moderate live issue in a released Bazel version that is inconvenient for a user that needs to be addressed in an future release and/or an easy workaround exists. +- [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug fix or enhancement with small impact. Not prioritized into Bazel roadmaps or any imminent release, however community contributions are encouraged. +- [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect or feature request that is unlikely to get closed. Can also be kept open for a potential re-prioritization if more users are impacted. ## Team labels -* [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team - * Contact: [ahumesky](https://github.com/ahumesky) -* [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues - * Contact: [meisterT](https://github.com/meisterT) -* [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI - * Contact: [meisterT](https://github.com/meisterT) -* [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags - * Contact: [gregestren](https://github.com/gregestren) -* [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc - * Contact: [haxorz](https://github.com/haxorz) -* [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team -* [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file - * Contact: [meteorcloudy](https://github.com/meteorcloudy) -* [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob - * Contact: [brandjon](https://github.com/brandjon) -* [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team - * Contact: [meisterT](https://github.com/meisterT) -* [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure - * Contact: [meteorcloudy](https://github.com/meteorcloudy) -* [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team - * Contact: [meisterT](https://github.com/meisterT) -* [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team - * Contact: [coeuvre](https://github.com/coeuvre) -* [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts - * Contact: [comius](https://github.com/comius) -* [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic - * Contact: [pzembrod](https://github.com/pzembrod) -* [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules - * Contact: [hvadehra](https://github.com/hvadehra) -* [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules - * Contact: [rickeylev](https://github.com/rickeylev) -* [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel - * Contact: [comius](https://github.com/comius) -* [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. - * Contact: [brandjon](https://github.com/brandjon) -* [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. - * Contact: [brandjon](https://github.com/brandjon) - -For new issues, we deprecated the `category: *` labels in favor of the team -labels. +- [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team + - Contact: [ahumesky](https://github.com/ahumesky) +- [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues + - Contact: [meisterT](https://github.com/meisterT) +- [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI + - Contact: [meisterT](https://github.com/meisterT) +- [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags + - Contact: [gregestren](https://github.com/gregestren) +- [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc + - Contact: [haxorz](https://github.com/haxorz) +- [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team +- [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file + - Contact: [meteorcloudy](https://github.com/meteorcloudy) +- [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob + - Contact: [brandjon](https://github.com/brandjon) +- [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team + - Contact: [meisterT](https://github.com/meisterT) +- [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure + - Contact: [meteorcloudy](https://github.com/meteorcloudy) +- [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team + - Contact: [meisterT](https://github.com/meisterT) +- [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team + - Contact: [coeuvre](https://github.com/coeuvre) +- [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts + - Contact: [comius](https://github.com/comius) +- [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic + - Contact: [pzembrod](https://github.com/pzembrod) +- [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules + - Contact: [hvadehra](https://github.com/hvadehra) +- [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules + - Contact: [rickeylev](https://github.com/rickeylev) +- [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel + - Contact: [comius](https://github.com/comius) +- [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. + - Contact: [brandjon](https://github.com/brandjon) +- [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. + - Contact: [brandjon](https://github.com/brandjon) + +For new issues, we deprecated the `category: *` labels in favor of the team labels. See the full list of labels [here](https://github.com/bazelbuild/bazel/labels). diff --git a/contribute/naming.mdx b/contribute/naming.mdx index 144b08af..806f7cd2 100644 --- a/contribute/naming.mdx +++ b/contribute/naming.mdx @@ -2,72 +2,42 @@ title: 'Naming a Bazel related project' --- +First, thank you for contributing to the Bazel ecosystem! Please reach out to the Bazel community on the [bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss) to share your project and its suggested name. - -First, thank you for contributing to the Bazel ecosystem! Please reach out to -the Bazel community on the -[bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss -) to share your project and its suggested name. - -If you are building a Bazel related tool or sharing your Skylark rules, -we recommend following these guidelines for the name of your project: +If you are building a Bazel related tool or sharing your Skylark rules, we recommend following these guidelines for the name of your project: ## Naming Starlark rules -See [Deploying new Starlark rules](/rules/deploying) -in the docs. +See [Deploying new Starlark rules](/rules/deploying) in the docs. ## Naming other Bazel related tools -This section applies if you are building a tool to enrich the Bazel ecosystem. -For example, a new IDE plugin or a new build system migrator. +This section applies if you are building a tool to enrich the Bazel ecosystem. For example, a new IDE plugin or a new build system migrator. -Picking a good name for your tool can be hard. If we’re not careful and use too -many codenames, the Bazel ecosystem could become very difficult to understand -for newcomers. +Picking a good name for your tool can be hard. If we’re not careful and use too many codenames, the Bazel ecosystem could become very difficult to understand for newcomers. Follow these guidelines for naming Bazel tools: -1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand -for our users, we should avoid confusing them with too many new names. - -2. Prefer **using a name that includes "Bazel"**: This helps to express that it -is a Bazel related tool, it also helps people find it with a search engine. +1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand for our users, we should avoid confusing them with too many new names. -3. Prefer **using names that are descriptive about what the tool is doing**: -Ideally, the name should not need a subtitle for users to have a first good -guess at what the tool does. Using english words separated by spaces is a good -way to achieve this. +2. Prefer **using a name that includes "Bazel"**: This helps to express that it is a Bazel related tool, it also helps people find it with a search engine. -4. **It is not a requirement to use a floral or food theme**: Bazel evokes -[basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to -look for a name that is a plant, food or that relates to "basil." +3. Prefer **using names that are descriptive about what the tool is doing**: Ideally, the name should not need a subtitle for users to have a first good guess at what the tool does. Using english words separated by spaces is a good way to achieve this. -5. **If your tool relates to another third party brand, use it only as a -descriptor**: For example, use "Bazel migrator for Cmake" instead of -"Cmake Bazel migrator". +4. **It is not a requirement to use a floral or food theme**: Bazel evokes [basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to look for a name that is a plant, food or that relates to "basil." -These guidelines also apply to the GitHub repository URL. Reading the repository -URL should help people understand what the tool does. Of course, the repository -name can be shorter and must use dashes instead of spaces and lower case letters. +5. **If your tool relates to another third party brand, use it only as a descriptor**: For example, use "Bazel migrator for Cmake" instead of "Cmake Bazel migrator". +These guidelines also apply to the GitHub repository URL. Reading the repository URL should help people understand what the tool does. Of course, the repository name can be shorter and must use dashes instead of spaces and lower case letters. Examples of good names: -* *Bazel for Eclipse*: Users will understand that if they want to use Bazel - with Eclipse, this is where they should be looking. It uses a third party brand - as a descriptor. -* *Bazel buildfarm*: A "buildfarm" is a - [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users - will understand that this project relates to building on servers. +- *Bazel for Eclipse*: Users will understand that if they want to use Bazel with Eclipse, this is where they should be looking. It uses a third party brand as a descriptor. +- *Bazel buildfarm*: A "buildfarm" is a [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users will understand that this project relates to building on servers. Examples of names to avoid: -* *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) - does not relate enough to the Bazel project. -* *Bazelizer*: The tool behind this name could do a lot of things, this name is - not descriptive enough. +- *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) does not relate enough to the Bazel project. +- *Bazelizer*: The tool behind this name could do a lot of things, this name is not descriptive enough. -Note that these recommendations are aligned with the -[guidelines](https://opensource.google.com/docs/releasing/preparing/#name) -Google uses when open sourcing a project. +Note that these recommendations are aligned with the [guidelines](https://opensource.google.com/docs/releasing/preparing/#name) Google uses when open sourcing a project. diff --git a/contribute/patch-acceptance.mdx b/contribute/patch-acceptance.mdx index 87376afd..c877c554 100644 --- a/contribute/patch-acceptance.mdx +++ b/contribute/patch-acceptance.mdx @@ -2,51 +2,26 @@ title: 'Patch Acceptance Process' --- +This page outlines how contributors can propose and make changes to the Bazel code base. +1. Read the [Bazel Contribution policy](/contribute/policy). -This page outlines how contributors can propose and make changes to the Bazel -code base. +2. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to discuss your plan and design. Pull requests that change or add behavior need a corresponding issue for tracking. -1. Read the [Bazel Contribution policy](/contribute/policy). -1. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to - discuss your plan and design. Pull requests that change or add behavior - need a corresponding issue for tracking. -1. If you're proposing significant changes, write a - [design document](/contribute/design-documents). -1. Ensure you've signed a [Contributor License - Agreement](https://cla.developers.google.com). -1. Prepare a git commit that implements the feature. Don't forget to add tests - and update the documentation. If your change has user-visible effects, please - [add release notes](/contribute/release-notes). If it is an incompatible change, - read the [guide for rolling out breaking changes](/contribute/breaking-changes). -1. Create a pull request on - [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, - read [about pull - requests](https://help.github.com/articles/about-pull-requests/). Note that - we restrict permissions to create branches on the main Bazel repository, so - you will need to push your commit to [your own fork of the - repository](https://help.github.com/articles/working-with-forks/). -1. A Bazel maintainer should assign you a reviewer within two business days - (excluding holidays in the USA and Germany). If you aren't assigned a - reviewer in that time, you can request one by emailing - [bazel-discuss@googlegroups.com] - (mailto:bazel-discuss@googlegroups.com). -1. Work with the reviewer to complete a code review. For each change, create a - new commit and push it to make changes to your pull request. If the review - takes too long (for instance, if the reviewer is unresponsive), send an email to - [bazel-discuss@googlegroups.com] - (mailto:bazel-discuss@googlegroups.com). -1. After your review is complete, a Bazel maintainer applies your patch to - Google's internal version control system. - - This triggers internal presubmit checks - that may suggest more changes. If you haven't expressed a preference, the - maintainer submitting your change adds "trivial" changes (such as - [linting](https://en.wikipedia.org/wiki/Lint_(software))) that don't affect - design. If deeper changes are required or you'd prefer to apply - changes directly, you and the reviewer should communicate preferences - clearly in review comments. - - After internal submission, the patch is exported as a Git commit, - at which point the GitHub pull request is closed. All final changes - are attributed to you. +3. If you're proposing significant changes, write a [design document](/contribute/design-documents). + +4. Ensure you've signed a [Contributor License Agreement](https://cla.developers.google.com). + +5. Prepare a git commit that implements the feature. Don't forget to add tests and update the documentation. If your change has user-visible effects, please [add release notes](/contribute/release-notes). If it is an incompatible change, read the [guide for rolling out breaking changes](/contribute/breaking-changes). + +6. Create a pull request on [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, read [about pull requests](https://help.github.com/articles/about-pull-requests/). Note that we restrict permissions to create branches on the main Bazel repository, so you will need to push your commit to [your own fork of the repository](https://help.github.com/articles/working-with-forks/). + +7. A Bazel maintainer should assign you a reviewer within two business days (excluding holidays in the USA and Germany). If you aren't assigned a reviewer in that time, you can request one by emailing \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). + +8. Work with the reviewer to complete a code review. For each change, create a new commit and push it to make changes to your pull request. If the review takes too long (for instance, if the reviewer is unresponsive), send an email to \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). + +9. After your review is complete, a Bazel maintainer applies your patch to Google's internal version control system. + + This triggers internal presubmit checks that may suggest more changes. If you haven't expressed a preference, the maintainer submitting your change adds "trivial" changes (such as [linting](https://en.wikipedia.org/wiki/Lint_\(software\))) that don't affect design. If deeper changes are required or you'd prefer to apply changes directly, you and the reviewer should communicate preferences clearly in review comments. + + After internal submission, the patch is exported as a Git commit, at which point the GitHub pull request is closed. All final changes are attributed to you. diff --git a/contribute/policy.mdx b/contribute/policy.mdx index 1bf00290..407f4c68 100644 --- a/contribute/policy.mdx +++ b/contribute/policy.mdx @@ -1,78 +1,59 @@ -translation: human -page_type: lcat --- title: 'Contribution policy' --- - - This page covers Bazel's governance model and contribution policy. ## Governance model -The [Bazel project](https://github.com/bazelbuild) is led and managed by Google -and has a large community of contributors outside of Google. Some Bazel -components (such as specific rules repositories under the -[bazelbuild](https://github.com/bazelbuild) organization) are led, -maintained, and managed by members of the community. The Google Bazel team -reviews suggestions to add community-owned repositories (such as rules) to the -[bazelbuild](https://github.com/bazelbuild) GitHub organization. +The [Bazel project](https://github.com/bazelbuild) is led and managed by Google and has a large community of contributors outside of Google. Some Bazel components (such as specific rules repositories under the [bazelbuild](https://github.com/bazelbuild) organization) are led, maintained, and managed by members of the community. The Google Bazel team reviews suggestions to add community-owned repositories (such as rules) to the [bazelbuild](https://github.com/bazelbuild) GitHub organization. ### Contributor roles -Here are outlines of the roles in the Bazel project, including their -responsibilities: - -* **Owners**: The Google Bazel team. Owners are responsible for: - * Strategy, maintenance, and leadership of the Bazel project. - * Building and maintaining Bazel's core functionality. - * Appointing Maintainers and approving new repositories. -* **Maintainers**: The Google Bazel team and designated GitHub users. - Maintainers are responsible for: - * Building and maintaining the primary functionality of their repository. - * Reviewing and approving contributions to areas of the Bazel code base. - * Supporting users and contributors with timely and transparent issue - management, PR review, and documentation. - * Releasing, testing and collaborating with Bazel Owners. -* **Contributors**: All users who contribute code or documentation to the - Bazel project. - * Creating well-written PRs to contribute to Bazel's codebase and - documentation. - * Using standard channels, such as GitHub Issues, to propose changes and - report issues. +Here are outlines of the roles in the Bazel project, including their responsibilities: + +- **Owners**: The Google Bazel team. Owners are responsible for: + + - Strategy, maintenance, and leadership of the Bazel project. + - Building and maintaining Bazel's core functionality. + - Appointing Maintainers and approving new repositories. + +- **Maintainers**: The Google Bazel team and designated GitHub users. Maintainers are responsible for: + + - Building and maintaining the primary functionality of their repository. + - Reviewing and approving contributions to areas of the Bazel code base. + - Supporting users and contributors with timely and transparent issue management, PR review, and documentation. + - Releasing, testing and collaborating with Bazel Owners. + +- **Contributors**: All users who contribute code or documentation to the Bazel project. + + - Creating well-written PRs to contribute to Bazel's codebase and documentation. + - Using standard channels, such as GitHub Issues, to propose changes and report issues. ### Becoming a Maintainer -Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as -rule sets. Contributors with a record of consistent, responsible past -contributions who are planning major contributions in the future could be -considered to become qualified Maintainers. +Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as rule sets. Contributors with a record of consistent, responsible past contributions who are planning major contributions in the future could be considered to become qualified Maintainers. ## Contribution policy -The Bazel project accepts contributions from external contributors. Here are the -contribution policies for Google-managed and Community-managed areas of code. - -* **Licensing**. All Maintainers and Contributors must sign the - [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). -* **Contributions**. Owners and Maintainers should make every effort to accept - worthwhile contributions. All contributions must be: - * Well written and well tested - * Discussed and approved by the Maintainers of the relevant area of code. - Discussions and approvals happen on GitHub Issues and in GitHub PRs. - Larger contributions require a - [design review](/contribute/design-documents). - * Added to Bazel's Continuous Integration system if not already present. - * Supportable and aligned with Bazel product direction -* **Code review**. All changes in all `bazelbuild` repositories require - review: - * All PRs must be approved by an Owner or Maintainer. - * Only Owners and Maintainers can merge PRs. -* **Compatibility**. Owners may need to reject or request modifications to PRs - in the unlikely event that the change requires substantial modifications to - internal Google systems. -* **Documentation**. Where relevant, feature contributions should include - documentation updates. - -For more details on contributing to Bazel, see our -[contribution guidelines](/contribute/). +The Bazel project accepts contributions from external contributors. Here are the contribution policies for Google-managed and Community-managed areas of code. + +- **Licensing**. All Maintainers and Contributors must sign the [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). + +- **Contributions**. Owners and Maintainers should make every effort to accept worthwhile contributions. All contributions must be: + + - Well written and well tested + - Discussed and approved by the Maintainers of the relevant area of code. Discussions and approvals happen on GitHub Issues and in GitHub PRs. Larger contributions require a [design review](/contribute/design-documents). + - Added to Bazel's Continuous Integration system if not already present. + - Supportable and aligned with Bazel product direction + +- **Code review**. All changes in all `bazelbuild` repositories require review: + + - All PRs must be approved by an Owner or Maintainer. + - Only Owners and Maintainers can merge PRs. + +- **Compatibility**. Owners may need to reject or request modifications to PRs in the unlikely event that the change requires substantial modifications to internal Google systems. + +- **Documentation**. Where relevant, feature contributions should include documentation updates. + +For more details on contributing to Bazel, see our [contribution guidelines](/contribute/). diff --git a/contribute/release-notes.mdx b/contribute/release-notes.mdx index 83e1d75b..bd0d9467 100644 --- a/contribute/release-notes.mdx +++ b/contribute/release-notes.mdx @@ -2,77 +2,44 @@ title: 'Writing release notes' --- - - This document is targeted at Bazel contributors. -Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release -note. This is used by the Bazel team to track changes in each release and write -the release announcement. +Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release note. This is used by the Bazel team to track changes in each release and write the release announcement. ## Overview -* Is your change a bugfix? In that case, you don't need a release note. Please - include a reference to the GitHub issue. +- Is your change a bugfix? In that case, you don't need a release note. Please include a reference to the GitHub issue. -* If the change adds / removes / changes Bazel in a user-visible way, then it - may be advantageous to mention it. +- If the change adds / removes / changes Bazel in a user-visible way, then it may be advantageous to mention it. -If the change is significant, follow the [design document -policy](/contribute/design-documents) first. +If the change is significant, follow the [design document policy](/contribute/design-documents) first. ## Guidelines -The release notes will be read by our users, so it should be short (ideally one -sentence), avoid jargon (Bazel-internal terminology), should focus on what the -change is about. +The release notes will be read by our users, so it should be short (ideally one sentence), avoid jargon (Bazel-internal terminology), should focus on what the change is about. -* Include a link to the relevant documentation. Almost any release note should - contain a link. If the description mentions a flag, a feature, a command name, - users will probably want to know more about it. +- Include a link to the relevant documentation. Almost any release note should contain a link. If the description mentions a flag, a feature, a command name, users will probably want to know more about it. -* Use backquotes around code, symbols, flags, or any word containing an - underscore. +- Use backquotes around code, symbols, flags, or any word containing an underscore. -* Do not just copy and paste bug descriptions. They are often cryptic and only - make sense to us and leave the user scratching their head. Release notes are - meant to explain what has changed and why in user-understandable language. +- Do not just copy and paste bug descriptions. They are often cryptic and only make sense to us and leave the user scratching their head. Release notes are meant to explain what has changed and why in user-understandable language. -* Always use present tense and the format "Bazel now supports Y" or "X now does - Z." We don't want our release notes to sound like bug entries. All release - note entries should be informative and use a consistent style and language. +- Always use present tense and the format "Bazel now supports Y" or "X now does Z." We don't want our release notes to sound like bug entries. All release note entries should be informative and use a consistent style and language. -* If something has been deprecated or removed, use "X has been deprecated" or "X - has been removed." Not "is removed" or "was removed." +- If something has been deprecated or removed, use "X has been deprecated" or "X has been removed." Not "is removed" or "was removed." -* If Bazel now does something differently, use "X now $newBehavior instead of - $oldBehavior" in present tense. This lets the user know in detail what to - expect when they use the new release. +- If Bazel now does something differently, use "X now $newBehavior instead of $oldBehavior" in present tense. This lets the user know in detail what to expect when they use the new release. -* If Bazel now supports or no longer supports something, use "Bazel now supports - / no longer supports X". +- If Bazel now supports or no longer supports something, use "Bazel now supports / no longer supports X". -* Explain why something has been removed / deprecated / changed. One sentence is - enough but we want the user to be able to evaluate impact on their builds. +- Explain why something has been removed / deprecated / changed. One sentence is enough but we want the user to be able to evaluate impact on their builds. -* Do NOT make any promises about future functionality. Avoid "this flag will be - removed" or "this will be changed." It introduces uncertainty. The first thing - the user will wonder is "when?" and we don't want them to start worrying about - their current builds breaking at some unknown time. +- Do NOT make any promises about future functionality. Avoid "this flag will be removed" or "this will be changed." It introduces uncertainty. The first thing the user will wonder is "when?" and we don't want them to start worrying about their current builds breaking at some unknown time. ## Process -As part of the [release -process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), -we collect the `RELNOTES` tags of every commit. We copy everything in a [Google -Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) -where we review, edit, and organize the notes. +As part of the [release process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), we collect the `RELNOTES` tags of every commit. We copy everything in a [Google Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) where we review, edit, and organize the notes. -The release manager sends an email to the -[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. -Bazel contributors are invited to contribute to the document and make sure -their changes are correctly reflected in the announcement. +The release manager sends an email to the [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. Bazel contributors are invited to contribute to the document and make sure their changes are correctly reflected in the announcement. -Later, the announcement will be submitted to the [Bazel -blog](https://blog.bazel.build/), using the [bazel-blog -repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). +Later, the announcement will be submitted to the [Bazel blog](https://blog.bazel.build/), using the [bazel-blog repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). diff --git a/contribute/search.mdx b/contribute/search.mdx new file mode 100644 index 00000000..63f2526c --- /dev/null +++ b/contribute/search.mdx @@ -0,0 +1,167 @@ +--- +title: 'Searching the codebase' +--- + +## Product overview + +Bazel's [code search and source browsing interface](https://source.bazel.build) is a web-based tool for browsing Bazel source code repositories. You can use these features to navigate among different repositories, branches, and files. You can also view history, diffs, and blame information. + +## Getting started + +Note: For the best experience, use the latest version of Chrome, Safari, or Firefox. + +To access the code search and source browsing interface, open [https://source.bazel.build](https://source.bazel.build) in your web browser. + +The main screen appears. This screen contains the following components: + +1. The Breadcrumb toolbar. This toolbar displays your current location in the repository and allows you to move quickly to another location such as another repository, or another location within a repository, such as a file, branch, or commit. + +2. A list of repositories that you can browse. + +At the top of the screen is a search box. You can use this box to search for specific files and code. + +## Working with repositories + +### Opening a repository + +To open a repository, click its name from the main screen. + +Alternatively, you can use the Breadcrumb toolbar to browse for a specificrepository. This toolbar displays your current location in the repository and allows you to move quickly to another location such as another repository, or another location within a repository, such as a file, branch, or commit. + +### Switch repositories + +To switch to a different repository, select the repository from the Breadcrumb toolbar. + +### View a repository at a specific commit + +To view a repository at a specific commit: + +1. From the view of the repository, select the file. +2. From the Breadcrumb toolbar, open the **Branch** menu. +3. In the submenu that appears, click **Commit**. +4. Select the commit you want to view. + +The interface now shows the repository as it existed at that commit. + +### Open a branch, commit, or tag + +By default, the code search and source browsing interface opens a repository to the default branch. To open a different branch, from the Breadcrumb toolbar, click the **Branch/Commit/Tag** menu. A submenu opens, allowing you to select a branch using a branch name, a tag name, or through a search box. + +- To select a branch using a branch name, select **Branch** and then click the name of the branch. +- To select a branch using a tag name, select **Tag** and then click the tag name. +- To select a branch using a commit id, select **Commit** and then click the commit id. +- To search for a branch, commit, or tag, select the corresponding item and type a search term in the search box. + +## Working with files + +When you select a repository from the main screen, the screen changes to display a view of that repository. If a README file exists, its contents appear in the file pane, located on the right side of the screen. Otherwise, a list of repository's files and folders appear. On the left side of the screen is a tree view of the repository's files and folders. You can use this tree to browse and open specific files. + +Notice that, when you are viewing a repository, the Breadcrumb toolbar now has three components: + +- A **Repository** menu, from which you can select different repositories +- A **Branch/Commit/Tag** menu, from which you can select specific branches, tags, or commits +- A **File path** box, which displays the name of the current file or folder and its corresponding path + +### Open a file + +You can open a file by browsing to its directory and selecting it. The view of the repository updates to show the contents of the file in the file pane, and its location in the repository in the tree pane. + +### View file changes + +To view file changes: + +1. From the view of the repository, select the file. +2. Click **BLAME**, located in the upper-right corner. + +The file pane updates to display who made changes to the file and when. + +### View change history + +To view the change history of a file: + +1. From the view of the repository, select the file. +2. Click **HISTORY**, located in the upper-right corner. The **Change history** pane appears, showing the commits for this file. + +### View code reviews + +For Gerrit code reviews, you can open the tool directly from the Change History pane. + +To view the code review for a file: + +1. From the view of the repository, select the file. +2. Click **HISTORY**, located in the upper-right corner. The Change History pane appears, showing the commits for this file. +3. Hover over a commit. A **More** button (three vertical dots) appears. +4. Click the **More** button. +5. Select **View code review**. + +The Gerrit Code Review tool opens in a new browser window. + +### Open a file at a specific commit + +To open a file at a specific commit: + +1. From the view of the repository, select the file. +2. Click **HISTORY**, located in the upper-right corner. The Change History pane appears, showing the commits for this file. +3. Hover over a commit. A **VIEW** button appears. +4. Click the **VIEW** button. + +### Compare a file to a different commit + +To compare a file at a different commit: + +1. From the view of the repository, select the file. To compare from two different commits, first open the file at that commit. +2. Hover over a commit. A **DIFF** button appears. +3. Click the **DIFF** button. + +The file pane updates to display a side-by-side comparison between the two files. The oldest of the two commits is always on the left. + +In the Change History pane, both commits are highlighted, and a label indicates if the commit is displayed on the left or the right. + +To change either file, hover over the commit in the Change History pane. Then, click either the **Left** or **Right** button to have the open the commit on the left or right side of the diff. + +### Browsing cross references + +Another way to browse source repositories is through the use of cross references. These references appear automatically as hyperlinks within a given source file. + +To make cross references easier to identify, click **Cross References**, located in the upper-right corner. This option displays an underline below all cross references in a file. + +**Note:** If **Cross References** is grayed out, it indicates that cross references are not available for that file. + +Click a cross reference to open the Cross Reference pane. This pane contains two sections: + +- A **Definition** section, which lists the file or files that define the reference +- A **References** section, which lists the files in which the reference also appears + +Both sections display the name of the file, as well as the line or lines that contains the reference. To open a file from the Cross Reference pane, click the line number entry. The file appears in a new section of the pane, allowing you to continue to browse the file while keeping the original file in view. + +You can continue to browse cross references using the Cross Reference pane, just as you can in the File pane. When you do, the pane displays a breadcrumb trail, which you can use to navigate between different cross references. + +## Searching for code + +You can search for specific files or code snippets using the search box located at the top of the screen. Searches are always against the default branch. + +All searches use [RE2 regular expressions](https://github.com/google/re2/wiki/Syntax) by default. If you do not want to use regular expressions, enclose your search in double quotes ( " ). + +**Note:** To quickly search for a specific file, either add a backslash in front of the period, or enclose the entire file name in quotes. + +``` +foo\.java +"foo.java" +``` + +You can refine your search using the following filters. + +| **Filter** | **Other options** | **Description** | **Example** | +| ---------- | ------------------ | ---------------------------------------------------------------------------- | -------------------- | +| lang: | language: | Perform an exact match by file language. | lang:java test | +| file: | filepath: path: f: | | | +| case:yes | | Make the search case sensitive. By default, searches are not case-sensitive. | case:yes Hello World | +| class: | | Search for a class name. | class:MainClass | +| function: | func: | Search for a function name. | function:print | +| - | | Negates the term from the search. | hello -world | +| \\ | | Escapes special characters, such as ., \\, or (. | run\\(\\) | +| "\[term]" | | Perform a literal search. | "class:main" | + +## Additional Support + +To report an issue, click the **Feedback** button that appears in the top right-hand corner of the screen and enter your feedback in the provided form. diff --git a/contribute/statemachine-guide.mdx b/contribute/statemachine-guide.mdx index e98a96e8..46993892 100644 --- a/contribute/statemachine-guide.mdx +++ b/contribute/statemachine-guide.mdx @@ -1,64 +1,28 @@ --- -title: 'A Guide to Skyframe `StateMachine`s' +title: 'A Guide to Skyframe StateMachines' --- - - ## Overview -A Skyframe `StateMachine` is a *deconstructed* function-object that resides on -the heap. It supports flexible and evaluation without redundancy[^1] when -required values are not immediately available but computed asynchronously. The -`StateMachine` cannot tie up a thread resource while waiting, but instead has to -be suspended and resumed. The deconstruction thus exposes explicit re-entry -points so that prior computations can be skipped. +A Skyframe `StateMachine` is a *deconstructed* function-object that resides on the heap. It supports flexible and evaluation without redundancy[1](#user-content-fn-1) when required values are not immediately available but computed asynchronously. The `StateMachine` cannot tie up a thread resource while waiting, but instead has to be suspended and resumed. The deconstruction thus exposes explicit re-entry points so that prior computations can be skipped. -`StateMachine`s can be used to express sequences, branching, structured logical -concurrency and are tailored specifically for Skyframe interaction. -`StateMachine`s can be composed into larger `StateMachine`s and share -sub-`StateMachine`s. Concurrency is always hierarchical by construction and -purely logical. Every concurrent subtask runs in the single shared parent -SkyFunction thread. +`StateMachine`s can be used to express sequences, branching, structured logical concurrency and are tailored specifically for Skyframe interaction. `StateMachine`s can be composed into larger `StateMachine`s and share sub-`StateMachine`s. Concurrency is always hierarchical by construction and purely logical. Every concurrent subtask runs in the single shared parent SkyFunction thread. ## Introduction -This section briefly motivates and introduces `StateMachine`s, found in the -[`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) -package. +This section briefly motivates and introduces `StateMachine`s, found in the [`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) package. ### A brief introduction to Skyframe restarts -Skyframe is a framework that performs parallel evaluation of dependency graphs. -Each node in the graph corresponds with the evaluation of a SkyFunction with a -SkyKey specifying its parameters and SkyValue specifying its result. The -computational model is such that a SkyFunction may lookup SkyValues by SkyKey, -triggering recursive, parallel evaluation of additional SkyFunctions. Instead of -blocking, which would tie up a thread, when a requested SkyValue is not yet -ready because some subgraph of computation is incomplete, the requesting -SkyFunction observes a `null` `getValue` response and should return `null` -instead of a SkyValue, signaling that it is incomplete due to missing inputs. -Skyframe *restarts* the SkyFunctions when all previously requested SkyValues -become available. - -Before the introduction of `SkyKeyComputeState`, the traditional way of handling -a restart was to fully rerun the computation. Although this has quadratic -complexity, functions written this way eventually complete because each rerun, -fewer lookups return `null`. With `SkyKeyComputeState` it is possible to -associate hand-specified check-point data with a SkyFunction, saving significant -recomputation. - -`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate -virtually all recomputation when a SkyFunction restarts (assuming that -`SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume -execution hooks. +Skyframe is a framework that performs parallel evaluation of dependency graphs. Each node in the graph corresponds with the evaluation of a SkyFunction with a SkyKey specifying its parameters and SkyValue specifying its result. The computational model is such that a SkyFunction may lookup SkyValues by SkyKey, triggering recursive, parallel evaluation of additional SkyFunctions. Instead of blocking, which would tie up a thread, when a requested SkyValue is not yet ready because some subgraph of computation is incomplete, the requesting SkyFunction observes a `null` `getValue` response and should return `null` instead of a SkyValue, signaling that it is incomplete due to missing inputs. Skyframe *restarts* the SkyFunctions when all previously requested SkyValues become available. + +Before the introduction of `SkyKeyComputeState`, the traditional way of handling a restart was to fully rerun the computation. Although this has quadratic complexity, functions written this way eventually complete because each rerun, fewer lookups return `null`. With `SkyKeyComputeState` it is possible to associate hand-specified check-point data with a SkyFunction, saving significant recomputation. + +`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate virtually all recomputation when a SkyFunction restarts (assuming that `SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume execution hooks. ### Stateful computations inside `SkyKeyComputeState` -From an object-oriented design standpoint, it makes sense to consider storing -computational objects inside `SkyKeyComputeState` instead of pure data values. -In *Java*, the bare minimum description of a behavior carrying object is a -*functional interface* and it turns out to be sufficient. A `StateMachine` has -the following, curiously recursive, definition[^2]. +From an object-oriented design standpoint, it makes sense to consider storing computational objects inside `SkyKeyComputeState` instead of pure data values. In *Java*, the bare minimum description of a behavior carrying object is a *functional interface* and it turns out to be sufficient. A `StateMachine` has the following, curiously recursive, definition[2](#user-content-fn-2). ``` @FunctionalInterface @@ -67,12 +31,9 @@ public interface StateMachine { } ``` -The `Tasks` interface is analogous to `SkyFunction.Environment` but it is -designed for asynchrony and adds support for logically concurrent subtasks[^3]. +The `Tasks` interface is analogous to `SkyFunction.Environment` but it is designed for asynchrony and adds support for logically concurrent subtasks[3](#user-content-fn-3). -The return value of `step` is another `StateMachine`, allowing the specification -of a sequence of steps, inductively. `step` returns `DONE` when the -`StateMachine` is done. For example: +The return value of `step` is another `StateMachine`, allowing the specification of a sequence of steps, inductively. `step` returns `DONE` when the `StateMachine` is done. For example: ``` class HelloWorld implements StateMachine { @@ -98,98 +59,66 @@ hello world ``` -Note that the method reference `this::step2` is also a `StateMachine` due to -`step2` satisfying `StateMachine`'s functional interface definition. Method -references are the most common way to specify the next state in a -`StateMachine`. +Note that the method reference `this::step2` is also a `StateMachine` due to `step2` satisfying `StateMachine`'s functional interface definition. Method references are the most common way to specify the next state in a `StateMachine`. ![Suspending and resuming](/contribute/images/suspend-resume.svg) -Intuitively, breaking a computation down into `StateMachine` steps, instead of a -monolithic function, provides the hooks needed to *suspend* and *resume* a -computation. When `StateMachine.step` returns, there is an explicit *suspension* -point. The continuation specified by the returned `StateMachine` value is an -explicit *resume* point. Recomputation can thus be avoided because the -computation can be picked up exactly where it left off. +Intuitively, breaking a computation down into `StateMachine` steps, instead of a monolithic function, provides the hooks needed to *suspend* and *resume* a computation. When `StateMachine.step` returns, there is an explicit *suspension* point. The continuation specified by the returned `StateMachine` value is an explicit *resume* point. Recomputation can thus be avoided because the computation can be picked up exactly where it left off. ### Callbacks, continuations and asynchronous computation -In technical terms, a `StateMachine` serves as a *continuation*, determining the -subsequent computation to be executed. Instead of blocking, a `StateMachine` can -voluntarily *suspend* by returning from the `step` function, which transfers -control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can -then switch to a ready `StateMachine` or relinquish control back to Skyframe. +In technical terms, a `StateMachine` serves as a *continuation*, determining the subsequent computation to be executed. Instead of blocking, a `StateMachine` can voluntarily *suspend* by returning from the `step` function, which transfers control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can then switch to a ready `StateMachine` or relinquish control back to Skyframe. -Traditionally, *callbacks* and *continuations* are conflated into one concept. -However, `StateMachine`s maintain a distinction between the two. +Traditionally, *callbacks* and *continuations* are conflated into one concept. However, `StateMachine`s maintain a distinction between the two. -* *Callback* - describes where to store the result of an asynchronous - computation. -* *Continuation* - specifies the next execution state. +- *Callback* - describes where to store the result of an asynchronous computation. +- *Continuation* - specifies the next execution state. -Callbacks are required when invoking an asynchronous operation, which means that -the actual operation doesn't occur immediately upon calling the method, as in -the case of a SkyValue lookup. Callbacks should be kept as simple as possible. +Callbacks are required when invoking an asynchronous operation, which means that the actual operation doesn't occur immediately upon calling the method, as in the case of a SkyValue lookup. Callbacks should be kept as simple as possible. -Caution: A common pitfall of callbacks is that the asynchronous computation must -ensure the callback is called by the end of every reachable path. It's possible -to overlook some branches and the compiler doesn't give warnings about this. +Caution: A common pitfall of callbacks is that the asynchronous computation must ensure the callback is called by the end of every reachable path. It's possible to overlook some branches and the compiler doesn't give warnings about this. -*Continuations* are the `StateMachine` return values of `StateMachine`s and -encapsulate the complex execution that follows once all asynchronous -computations resolve. This structured approach helps to keep the complexity of -callbacks manageable. +*Continuations* are the `StateMachine` return values of `StateMachine`s and encapsulate the complex execution that follows once all asynchronous computations resolve. This structured approach helps to keep the complexity of callbacks manageable. ## Tasks -The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues -by SkyKey and to schedule concurrent subtasks. +The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues by SkyKey and to schedule concurrent subtasks. ``` interface Tasks { void enqueue(StateMachine subtask); - void lookUp(SkyKey key, Consumer sink); + void lookUp(SkyKey key, Consumer<SkyValue> sink); - - void lookUp(SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); + <E extends Exception> + void lookUp(SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); // lookUp overloads for 2 and 3 exception types exist, but are elided here. } ``` -Tip: When any state uses the `Tasks` interface to perform lookups or create -subtasks, those lookups and subtasks will complete before the next state begins. +Tip: When any state uses the `Tasks` interface to perform lookups or create subtasks, those lookups and subtasks will complete before the next state begins. -Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create -subtasks, they all *transitively* complete before the next state begins. +Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create subtasks, they all *transitively* complete before the next state begins. ### SkyValue lookups -`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are -analogous to `SkyFunction.Environment.getValue` and -`SkyFunction.Environment.getValueOrThrow` and have similar exception handling -semantics. The implementation does not immediately perform the lookup, but -instead, batches[^4] as many lookups as possible before doing so. The value -might not be immediately available, for example, requiring a Skyframe restart, -so the caller specifies what to do with the resulting value using a callback. +`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are analogous to `SkyFunction.Environment.getValue` and `SkyFunction.Environment.getValueOrThrow` and have similar exception handling semantics. The implementation does not immediately perform the lookup, but instead, batches[4](#user-content-fn-4) as many lookups as possible before doing so. The value might not be immediately available, for example, requiring a Skyframe restart, so the caller specifies what to do with the resulting value using a callback. -The `StateMachine` processor ([`Driver`s and bridging to -SkyFrame](#drivers-and-bridging)) guarantees that the value is available before -the next state begins. An example follows. +The `StateMachine` processor ([`Driver`s and bridging to SkyFrame](#drivers-and-bridging)) guarantees that the value is available before the next state begins. An example follows. ``` -class DoesLookup implements StateMachine, Consumer { +class DoesLookup implements StateMachine, Consumer<SkyValue> { private Value value; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new Key(), (Consumer) this); + tasks.lookUp(new Key(), (Consumer<SkyValue>) this); return this::processValue; } // The `lookUp` call in `step` causes this to be called before `processValue`. - @Override // Implementation of Consumer. + @Override // Implementation of Consumer<SkyValue>. public void accept(SkyValue value) { this.value = (Value)value; } @@ -201,25 +130,15 @@ class DoesLookup implements StateMachine, Consumer { } ``` -In the above example, the first step does a lookup for `new Key()`, passing -`this` as the consumer. That is possible because `DoesLookup` implements -`Consumer`. +In the above example, the first step does a lookup for `new Key()`, passing `this` as the consumer. That is possible because `DoesLookup` implements `Consumer<SkyValue>`. -Tip: When passing `this` as a value sink, it's helpful to readers to upcast it -to the receiver type to narrow down the purpose of passing `this`. The example -passes `(Consumer) this`. +Tip: When passing `this` as a value sink, it's helpful to readers to upcast it to the receiver type to narrow down the purpose of passing `this`. The example passes `(Consumer<SkyValue>) this`. -By contract, before the next state `DoesLookup.processValue` begins, all the -lookups of `DoesLookup.step` are complete. Therefore `value` is available when -it is accessed in `processValue`. +By contract, before the next state `DoesLookup.processValue` begins, all the lookups of `DoesLookup.step` are complete. Therefore `value` is available when it is accessed in `processValue`. ### Subtasks -`Tasks.enqueue` requests the execution of logically concurrent subtasks. -Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s -can do, including recursively creating more subtasks or looking up SkyValues. -Much like `lookUp`, the state machine driver ensures that all subtasks are -complete before proceeding to the next step. An example follows. +`Tasks.enqueue` requests the execution of logically concurrent subtasks. Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s can do, including recursively creating more subtasks or looking up SkyValues. Much like `lookUp`, the state machine driver ensures that all subtasks are complete before proceeding to the next step. An example follows. ``` class Subtasks implements StateMachine { @@ -257,22 +176,15 @@ class Subtasks implements StateMachine { } ``` -Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a -single thread so the "concurrent" update of `i` does not need any -synchronization. +Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a single thread so the "concurrent" update of `i` does not need any synchronization. ### Structured concurrency -Since every `lookUp` and `enqueue` must resolve before advancing to the next -state, it means that concurrency is naturally limited to tree-structures. It's -possible to create hierarchical[^5] concurrency as shown in the following -example. +Since every `lookUp` and `enqueue` must resolve before advancing to the next state, it means that concurrency is naturally limited to tree-structures. It's possible to create hierarchical[5](#user-content-fn-5) concurrency as shown in the following example. ![Structured Concurrency](/contribute/images/structured-concurrency.svg) -It's hard to tell from the *UML* that the concurrency structure forms a tree. -There's an [alternate view](#concurrency-tree-diagram) that better shows the -tree structure. +It's hard to tell from the *UML* that the concurrency structure forms a tree. There's an [alternate view](#concurrency-tree-diagram) that better shows the tree structure. ![Unstructured Concurrency](/contribute/images/unstructured-concurrency.svg) @@ -280,19 +192,15 @@ Structured concurrency is much easier to reason about. ## Composition and control flow patterns -This section presents examples for how multiple `StateMachine`s can be composed -and solutions to certain control flow problems. +This section presents examples for how multiple `StateMachine`s can be composed and solutions to certain control flow problems. ### Sequential states -This is the most common and straightforward control flow pattern. An example of -this is shown in [Stateful computations inside -`SkyKeyComputeState`](#stateful-computations). +This is the most common and straightforward control flow pattern. An example of this is shown in [Stateful computations inside `SkyKeyComputeState`](#stateful-computations). ### Branching -Branching states in `StateMachine`s can be achieved by returning different -values using regular *Java* control flow, as shown in the following example. +Branching states in `StateMachine`s can be achieved by returning different values using regular *Java* control flow, as shown in the following example. ``` class Branch implements StateMachine { @@ -312,18 +220,11 @@ It’s very common for certain branches to return `DONE`, for early completion. ### Advanced sequential composition -Since the `StateMachine` control structure is memoryless, sharing `StateMachine` -definitions as subtasks can sometimes be awkward. Let *M1* and -*M2* be `StateMachine` instances that share a `StateMachine`, *S*, -with *M1* and *M2* being the sequences *<A, S, B>* and -*<X, S, Y>* respectively. The problem is that *S* doesn’t know whether to -continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a -call stack. This section reviews some techniques for achieving this. +Since the `StateMachine` control structure is memoryless, sharing `StateMachine` definitions as subtasks can sometimes be awkward. Let *M1* and *M2* be `StateMachine` instances that share a `StateMachine`, *S*, with *M1* and *M2* being the sequences *\* and *\* respectively. The problem is that *S* doesn’t know whether to continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a call stack. This section reviews some techniques for achieving this. #### `StateMachine` as terminal sequence element -This doesn’t solve the initial problem posed. It only demonstrates sequential -composition when the shared `StateMachine` is terminal in the sequence. +This doesn’t solve the initial problem posed. It only demonstrates sequential composition when the shared `StateMachine` is terminal in the sequence. ``` // S is the shared state machine. @@ -350,8 +251,7 @@ This works even if *S* is itself a complex state machine. #### Subtask for sequential composition -Since enqueued subtasks are guaranteed to complete before the next state, it’s -sometimes possible to slightly abuse[^6] the subtask mechanism. +Since enqueued subtasks are guaranteed to complete before the next state, it’s sometimes possible to slightly abuse[6](#user-content-fn-6) the subtask mechanism. ``` class M1 implements StateMachine { @@ -359,7 +259,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // S starts after `step` returns and by contract must complete before `doB` - // begins. It is effectively sequential, inducing the sequence < A, S, B >. + // begins. It is effectively sequential, inducing the sequence < A, S, B >. tasks.enqueue(new S()); return this::doB; } @@ -374,7 +274,7 @@ class M2 implements StateMachine { @Override public StateMachine step(Tasks tasks) { performX(); - // Similarly, this induces the sequence < X, S, Y>. + // Similarly, this induces the sequence < X, S, Y>. tasks.enqueue(new S()); return this::doY; } @@ -388,10 +288,7 @@ class M2 implements StateMachine { #### `runAfter` injection -Sometimes, abusing `Tasks.enqueue` is impossible because there are other -parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* -executes. In this case, injecting a `runAfter` parameter into *S* can be used to -inform *S* of what to do next. +Sometimes, abusing `Tasks.enqueue` is impossible because there are other parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* executes. In this case, injecting a `runAfter` parameter into *S* can be used to inform *S* of what to do next. ``` class S implements StateMachine { @@ -418,7 +315,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // Passes `this::doB` as the `runAfter` parameter of S, resulting in the - // sequence < A, S, B >. + // sequence < A, S, B >. return new S(/* runAfter= */ this::doB); } @@ -433,7 +330,7 @@ class M2 implements StateMachine { public StateMachine step(Tasks tasks) { performX(); // Passes `this::doY` as the `runAfter` parameter of S, resulting in the - // sequence < X, S, Y >. + // sequence < X, S, Y >. return new S(/* runAfter= */ this::doY); } @@ -444,10 +341,7 @@ class M2 implements StateMachine { } ``` -This approach is cleaner than abusing subtasks. However, applying this too -liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is -the road to [Callback Hell](#callback-hell). It’s better to break up sequential -`runAfter`s with ordinary sequential states instead. +This approach is cleaner than abusing subtasks. However, applying this too liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is the road to [Callback Hell](#callback-hell). It’s better to break up sequential `runAfter`s with ordinary sequential states instead. ``` return new S(/* runAfter= */ new T(/* runAfter= */ this::nextStep)) @@ -466,37 +360,21 @@ can be replaced with the following. } ``` -Note: It's possible to pass `DONE` as the `runAfter` parameter when there's -nothing to run afterwards. +Note: It's possible to pass `DONE` as the `runAfter` parameter when there's nothing to run afterwards. -Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` -to let the reader know the meaning at the callsite. +Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` to let the reader know the meaning at the callsite. #### *Forbidden* alternative: `runAfterUnlessError` -In an earlier draft, we had considered a `runAfterUnlessError` that would abort -early on errors. This was motivated by the fact that errors often end up getting -checked twice, once by the `StateMachine` that has a `runAfter` reference and -once by the `runAfter` machine itself. +In an earlier draft, we had considered a `runAfterUnlessError` that would abort early on errors. This was motivated by the fact that errors often end up getting checked twice, once by the `StateMachine` that has a `runAfter` reference and once by the `runAfter` machine itself. -After some deliberation, we decided that uniformity of the code is more -important than deduplicating the error checking. It would be confusing if the -`runAfter` mechanism did not work in a consistent manner with the -`tasks.enqueue` mechanism, which always requires error checking. +After some deliberation, we decided that uniformity of the code is more important than deduplicating the error checking. It would be confusing if the `runAfter` mechanism did not work in a consistent manner with the `tasks.enqueue` mechanism, which always requires error checking. -Warning: When using `runAfter`, the machine that has the injected `runAfter` -should invoke it unconditionally at completion, even on error, for consistency. +Warning: When using `runAfter`, the machine that has the injected `runAfter` should invoke it unconditionally at completion, even on error, for consistency. ### Direct delegation -Each time there is a formal state transition, the main `Driver` loop advances. -As per contract, advancing states means that all previously enqueued SkyValue -lookups and subtasks resolve before the next state executes. Sometimes the logic -of a delegate `StateMachine` makes a phase advance unnecessary or -counterproductive. For example, if the first `step` of the delegate performs -SkyKey lookups that could be parallelized with lookups of the delegating state -then a phase advance would make them sequential. It could make more sense to -perform direct delegation, as shown in the example below. +Each time there is a formal state transition, the main `Driver` loop advances. As per contract, advancing states means that all previously enqueued SkyValue lookups and subtasks resolve before the next state executes. Sometimes the logic of a delegate `StateMachine` makes a phase advance unnecessary or counterproductive. For example, if the first `step` of the delegate performs SkyKey lookups that could be parallelized with lookups of the delegating state then a phase advance would make them sequential. It could make more sense to perform direct delegation, as shown in the example below. ``` class Parent implements StateMachine { @@ -542,49 +420,37 @@ class Delegate implements StateMachine { ## Data flow -The focus of the previous discussion has been on managing control flow. This -section describes the propagation of data values. +The focus of the previous discussion has been on managing control flow. This section describes the propagation of data values. ### Implementing `Tasks.lookUp` callbacks -There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue -lookups](#skyvalue-lookups). This section provides rationale and suggests -approaches for handling multiple SkyValues. +There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue lookups](#skyvalue-lookups). This section provides rationale and suggests approaches for handling multiple SkyValues. #### `Tasks.lookUp` callbacks The `Tasks.lookUp` method takes a callback, `sink`, as a parameter. ``` - void lookUp(SkyKey key, Consumer sink); + void lookUp(SkyKey key, Consumer<SkyValue> sink); ``` The idiomatic approach would be to use a *Java* lambda to implement this: ``` - tasks.lookUp(key, value -> myValue = (MyValueClass)value); + tasks.lookUp(key, value -> myValue = (MyValueClass)value); ``` -with `myValue` being a member variable of the `StateMachine` instance doing the -lookup. However, the lambda requires an extra memory allocation compared to -implementing the `Consumer` interface in the `StateMachine` -implementation. The lambda is still useful when there are multiple lookups that -would be ambiguous. +with `myValue` being a member variable of the `StateMachine` instance doing the lookup. However, the lambda requires an extra memory allocation compared to implementing the `Consumer<SkyValue>` interface in the `StateMachine` implementation. The lambda is still useful when there are multiple lookups that would be ambiguous. -Note: Bikeshed warning. There is a noticeable difference of approximately 1% -end-to-end CPU usage when implementing callbacks systematically in -`StateMachine` implementations compared to using lambdas, which makes this -recommendation debatable. To avoid unnecessary debates, it is advised to leave -the decision up to the individual implementing the solution. +Note: Bikeshed warning. There is a noticeable difference of approximately 1% end-to-end CPU usage when implementing callbacks systematically in `StateMachine` implementations compared to using lambdas, which makes this recommendation debatable. To avoid unnecessary debates, it is advised to leave the decision up to the individual implementing the solution. -There are also error handling overloads of `Tasks.lookUp`, that are analogous to -`SkyFunction.Environment.getValueOrThrow`. +There are also error handling overloads of `Tasks.lookUp`, that are analogous to `SkyFunction.Environment.getValueOrThrow`. ``` - void lookUp( - SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); + <E extends Exception> void lookUp( + SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); - interface ValueOrExceptionSink { + interface ValueOrExceptionSink<E extends Exception> { void acceptValueOrException(@Nullable SkyValue value, @Nullable E exception); } ``` @@ -592,13 +458,13 @@ There are also error handling overloads of `Tasks.lookUp`, that are analogous to An example implementation is shown below. ``` -class PerformLookupWithError extends StateMachine, ValueOrExceptionSink { +class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyException> { private MyValue value; private MyException error; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink) this); + tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink<MyException>) this); return this::processResult; } @@ -627,33 +493,29 @@ class PerformLookupWithError extends StateMachine, ValueOrExceptionSink) this); + tasks.lookUp(configurationKey, (Consumer<SkyValue>) this); } var packageId = configuredTarget.getLabel().getPackageIdentifier(); - tasks.lookUp(PackageValue.key(packageId), (Consumer) this); + tasks.lookUp(PackageValue.key(packageId), (Consumer<SkyValue>) this); return this::constructResult; } - @Override // Implementation of `Consumer`. + @Override // Implementation of `Consumer<SkyValue>`. public void accept(SkyValue value) { if (value instanceof BuildConfigurationValue) { this.configurationValue = (BuildConfigurationValue) value; @@ -667,18 +529,11 @@ been simplified from prototype production code. } ``` -The `Consumer` callback implementation can be shared unambiguously -because the value types are different. When that’s not the case, falling back to -lambda-based implementations or full inner-class instances that implement the -appropriate callbacks is viable. +The `Consumer<SkyValue>` callback implementation can be shared unambiguously because the value types are different. When that’s not the case, falling back to lambda-based implementations or full inner-class instances that implement the appropriate callbacks is viable. ### Propagating values between `StateMachine`s -So far, this document has only explained how to arrange work in a subtask, but -subtasks also need to report a values back to the caller. Since subtasks are -logically asynchronous, their results are communicated back to the caller using -a *callback*. To make this work, the subtask defines a sink interface that is -injected via its constructor. +So far, this document has only explained how to arrange work in a subtask, but subtasks also need to report a values back to the caller. Since subtasks are logically asynchronous, their results are communicated back to the caller using a *callback*. To make this work, the subtask defines a sink interface that is injected via its constructor. ``` class BarProducer implements StateMachine { @@ -709,16 +564,9 @@ class BarProducer implements StateMachine { } ``` -Tip: It would be tempting to use the more concise signature void `accept(Bar -value)` rather than the stuttery `void acceptBarValue(Bar value)` above. -However, `Consumer` is a common overload of `void accept(Bar value)`, -so doing this often leads to violations of the [Overloads: never -split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) -style-guide rule. +Tip: It would be tempting to use the more concise signature void `accept(Bar value)` rather than the stuttery `void acceptBarValue(Bar value)` above. However, `Consumer<SkyValue>` is a common overload of `void accept(Bar value)`, so doing this often leads to violations of the [Overloads: never split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) style-guide rule. -Tip: Using a custom `ResultSink` type instead of a generic one from -`java.util.function` makes it easy to find implementations in the code base, -improving readability. +Tip: Using a custom `ResultSink` type instead of a generic one from `java.util.function` makes it easy to find implementations in the code base, improving readability. A caller `StateMachine` would then look like the following. @@ -767,70 +615,37 @@ class Caller implements StateMachine, BarProducer.ResultSink { } ``` -The preceding example demonstrates a few things. `Caller` has to propagate its -results back and defines its own `Caller.ResultSink`. `Caller` implements the -`BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if -`value` is null to determine if an error occurred. This is a common behavior -pattern after accepting output from either a subtask or SkyValue lookup. +The preceding example demonstrates a few things. `Caller` has to propagate its results back and defines its own `Caller.ResultSink`. `Caller` implements the `BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if `value` is null to determine if an error occurred. This is a common behavior pattern after accepting output from either a subtask or SkyValue lookup. -Note that the implementation of `acceptBarError` eagerly forwards the result to -the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). +Note that the implementation of `acceptBarError` eagerly forwards the result to the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). -Alternatives for top-level `StateMachine`s are described in [`Driver`s and -bridging to SkyFunctions](#drivers-and-bridging). +Alternatives for top-level `StateMachine`s are described in [`Driver`s and bridging to SkyFunctions](#drivers-and-bridging). ### Error handling -There's a couple of examples of error handling already in [`Tasks.lookUp` -callbacks](#tasks-lookup-callbacks) and [Propagating values between -`StateMachines`](#propagating-values). Exceptions, other than -`InterruptedException` are not thrown, but instead passed around through -callbacks as values. Such callbacks often have exclusive-or semantics, with -exactly one of a value or error being passed. +There's a couple of examples of error handling already in [`Tasks.lookUp` callbacks](#tasks-lookup-callbacks) and [Propagating values between `StateMachines`](#propagating-values). Exceptions, other than `InterruptedException` are not thrown, but instead passed around through callbacks as values. Such callbacks often have exclusive-or semantics, with exactly one of a value or error being passed. -The next section describes a a subtle, but important interaction with Skyframe -error handling. +The next section describes a a subtle, but important interaction with Skyframe error handling. #### Error bubbling (--nokeep\_going) -Warning: Errors need to be eagerly propagated all the way back to the -SkyFunction for error bubbling to function correctly. +Warning: Errors need to be eagerly propagated all the way back to the SkyFunction for error bubbling to function correctly. -During error bubbling, a SkyFunction may be restarted even if not all requested -SkyValues are available. In such cases, the subsequent state will never be -reached due to the `Tasks` API contract. However, the `StateMachine` should -still propagate the exception. +During error bubbling, a SkyFunction may be restarted even if not all requested SkyValues are available. In such cases, the subsequent state will never be reached due to the `Tasks` API contract. However, the `StateMachine` should still propagate the exception. -Since propagation must occur regardless of whether the next state is reached, -the error handling callback must perform this task. For an inner `StateMachine`, -this is achieved by invoking the parent callback. +Since propagation must occur regardless of whether the next state is reached, the error handling callback must perform this task. For an inner `StateMachine`, this is achieved by invoking the parent callback. -At the top-level `StateMachine`, which interfaces with the SkyFunction, this can -be done by calling the `setException` method of `ValueOrExceptionProducer`. -`ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even -if there are missing SkyValues. +At the top-level `StateMachine`, which interfaces with the SkyFunction, this can be done by calling the `setException` method of `ValueOrExceptionProducer`. `ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even if there are missing SkyValues. -If a `Driver` is being utilized directly, it is essential to check for -propagated errors from the SkyFunction, even if the machine has not finished -processing. +If a `Driver` is being utilized directly, it is essential to check for propagated errors from the SkyFunction, even if the machine has not finished processing. ### Event Handling -For SkyFunctions that need to emit events, a `StoredEventHandler` is injected -into SkyKeyComputeState and further injected into `StateMachine`s that require -them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping -certain events unless they are replayed but this was subsequently fixed. -`StoredEventHandler` injection is preserved because it simplifies the -implementation of events emitted from error handling callbacks. +For SkyFunctions that need to emit events, a `StoredEventHandler` is injected into SkyKeyComputeState and further injected into `StateMachine`s that require them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping certain events unless they are replayed but this was subsequently fixed. `StoredEventHandler` injection is preserved because it simplifies the implementation of events emitted from error handling callbacks. ## `Driver`s and bridging to SkyFunctions -A `Driver` is responsible for managing the execution of `StateMachine`s, -beginning with a specified root `StateMachine`. As `StateMachine`s can -recursively enqueue subtask `StateMachine`s, a single `Driver` can manage -numerous subtasks. These subtasks create a tree structure, a result of -[Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue -lookups across subtasks for improved efficiency. +A `Driver` is responsible for managing the execution of `StateMachine`s, beginning with a specified root `StateMachine`. As `StateMachine`s can recursively enqueue subtask `StateMachine`s, a single `Driver` can manage numerous subtasks. These subtasks create a tree structure, a result of [Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue lookups across subtasks for improved efficiency. There are a number of classes built around the `Driver`, with the following API. @@ -841,24 +656,15 @@ public final class Driver { } ``` -`Driver` takes a single root `StateMachine` as a parameter. Calling -`Driver.drive` executes the `StateMachine` as far as it can go without a -Skyframe restart. It returns true when the `StateMachine` completes and false -otherwise, indicating that not all values were available. +`Driver` takes a single root `StateMachine` as a parameter. Calling `Driver.drive` executes the `StateMachine` as far as it can go without a Skyframe restart. It returns true when the `StateMachine` completes and false otherwise, indicating that not all values were available. -`Driver` maintains the concurrent state of the `StateMachine` and it is well -suited for embedding in `SkyKeyComputeState`. +`Driver` maintains the concurrent state of the `StateMachine` and it is well suited for embedding in `SkyKeyComputeState`. ### Directly instantiating `Driver` -`StateMachine` implementations conventionally communicate their results via -callbacks. It's possible to directly instantiate a `Driver` as shown in the -following example. +`StateMachine` implementations conventionally communicate their results via callbacks. It's possible to directly instantiate a `Driver` as shown in the following example. -The `Driver` is embedded in the `SkyKeyComputeState` implementation along with -an implementation of the corresponding `ResultSink` to be defined a bit further -down. At the top level, the `State` object is an appropriate receiver for the -result of the computation as it is guaranteed to outlive `Driver`. +The `Driver` is embedded in the `SkyKeyComputeState` implementation along with an implementation of the corresponding `ResultSink` to be defined a bit further down. At the top level, the `State` object is an appropriate receiver for the result of the computation as it is guaranteed to outlive `Driver`. ``` class State implements SkyKeyComputeState, ResultProducer.ResultSink { @@ -940,8 +746,7 @@ private Result computeResult(State state, Skyfunction.Environment env) ### Embedding `Driver` -If the `StateMachine` produces a value and raises no exceptions, embedding -`Driver` is another possible implementation, as shown in the following example. +If the `StateMachine` produces a value and raises no exceptions, embedding `Driver` is another possible implementation, as shown in the following example. ``` class ResultProducer implements StateMachine { @@ -970,8 +775,7 @@ class ResultProducer implements StateMachine { } ``` -The SkyFunction may have code that looks like the following (where `State` is -the function specific type of `SkyKeyComputeState`). +The SkyFunction may have code that looks like the following (where `State` is the function specific type of `SkyKeyComputeState`). ``` @Nullable // Null when a Skyframe restart is needed. @@ -992,19 +796,16 @@ Result computeResult(SkyFunction.Environment env, State state) } ``` -Embedding `Driver` in the `StateMachine` implementation is a better fit for -Skyframe's synchronous coding style. +Embedding `Driver` in the `StateMachine` implementation is a better fit for Skyframe's synchronous coding style. ### StateMachines that may produce exceptions -Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` -and `ValueOrException2Producer` classes that have synchronous APIs to match -synchronous SkyFunction code. +Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` and `ValueOrException2Producer` classes that have synchronous APIs to match synchronous SkyFunction code. The `ValueOrExceptionProducer` abstract class includes the following methods. ``` -public abstract class ValueOrExceptionProducer +public abstract class ValueOrExceptionProducer<V, E extends Exception> implements StateMachine { @Nullable public final V tryProduceValue(Environment env) @@ -1017,65 +818,34 @@ public abstract class ValueOrExceptionProducer } ``` -It includes an embedded `Driver` instance and closely resembles the -`ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces -with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, -implementations call `setValue` or `setException` when either of those occur. -When both occur, the exception takes priority. The `tryProduceValue` method -bridges the asynchronous callback code to synchronous code and throws an -exception when one is set. +It includes an embedded `Driver` instance and closely resembles the `ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, implementations call `setValue` or `setException` when either of those occur. When both occur, the exception takes priority. The `tryProduceValue` method bridges the asynchronous callback code to synchronous code and throws an exception when one is set. -As previously noted, during error bubbling, it's possible for an error to occur -even if the machine is not yet done because not all inputs are available. To -accommodate this, `tryProduceValue` throws any set exceptions, even before the -machine is done. +As previously noted, during error bubbling, it's possible for an error to occur even if the machine is not yet done because not all inputs are available. To accommodate this, `tryProduceValue` throws any set exceptions, even before the machine is done. ## Epilogue: Eventually removing callbacks -`StateMachine`s are a highly efficient, but boilerplate intensive way to perform -asynchronous computation. Continuations (particularly in the form of `Runnable`s -passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, -but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and -there are no efficient asynchronous APIs for disk I/O. Eventually, it would be -good to optimize away callbacks as they have a learning curve and impede -readability. - -One of the most promising alternatives is *Java* virtual threads. Instead of -having to write callbacks, everything is replaced with synchronous, blocking -calls. This is possible because tying up a virtual thread resource, unlike a -platform thread, is supposed to be cheap. However, even with virtual threads, -replacing simple synchronous operations with thread creation and synchronization -primitives is too expensive. We performed a migration from `StateMachine`s to -*Java* virtual threads and they were orders of magnitude slower, leading to -almost a 3x increase in end-to-end analysis latency. Since virtual threads are -still a preview feature, it's possible that this migration can be performed at a -later date when performance improves. - -Another approach to consider is waiting for *Loom* coroutines, if they ever -become available. The advantage here is that it might be possible to reduce -synchronization overhead by using cooperative multitasking. - -If all else fails, low-level bytecode rewriting could also be a viable -alternative. With enough optimization, it might be possible to achieve -performance that approaches hand-written callback code. +`StateMachine`s are a highly efficient, but boilerplate intensive way to perform asynchronous computation. Continuations (particularly in the form of `Runnable`s passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and there are no efficient asynchronous APIs for disk I/O. Eventually, it would be good to optimize away callbacks as they have a learning curve and impede readability. + +One of the most promising alternatives is *Java* virtual threads. Instead of having to write callbacks, everything is replaced with synchronous, blocking calls. This is possible because tying up a virtual thread resource, unlike a platform thread, is supposed to be cheap. However, even with virtual threads, replacing simple synchronous operations with thread creation and synchronization primitives is too expensive. We performed a migration from `StateMachine`s to *Java* virtual threads and they were orders of magnitude slower, leading to almost a 3x increase in end-to-end analysis latency. Since virtual threads are still a preview feature, it's possible that this migration can be performed at a later date when performance improves. + +Another approach to consider is waiting for *Loom* coroutines, if they ever become available. The advantage here is that it might be possible to reduce synchronization overhead by using cooperative multitasking. + +If all else fails, low-level bytecode rewriting could also be a viable alternative. With enough optimization, it might be possible to achieve performance that approaches hand-written callback code. ## Appendix ### Callback Hell -Callback hell is an infamous problem in asynchronous code that uses callbacks. -It stems from the fact that the continuation for a subsequent step is nested -within the previous step. If there are many steps, this nesting can be extremely -deep. If coupled with control flow the code becomes unmanageable. +Callback hell is an infamous problem in asynchronous code that uses callbacks. It stems from the fact that the continuation for a subsequent step is nested within the previous step. If there are many steps, this nesting can be extremely deep. If coupled with control flow the code becomes unmanageable. ``` class CallbackHell implements StateMachine { @Override public StateMachine step(Tasks task) { doA(); - return (t, l) -> { + return (t, l) -> { doB(); - return (t1, l2) -> { + return (t1, l2) -> { doC(); return DONE; }; @@ -1084,11 +854,7 @@ class CallbackHell implements StateMachine { } ``` -One of the advantages of nested implementations is that the stack frame of the -outer step can be preserved. In *Java*, captured lambda variables must be -effectively final so using such variables can be cumbersome. Deep nesting is -avoided by returning method references as continuations instead of lambdas as -shown as follows. +One of the advantages of nested implementations is that the stack frame of the outer step can be preserved. In *Java*, captured lambda variables must be effectively final so using such variables can be cumbersome. Deep nesting is avoided by returning method references as continuations instead of lambdas as shown as follows. ``` class CallbackHellAvoided implements StateMachine { @@ -1110,23 +876,18 @@ class CallbackHellAvoided implements StateMachine { } ``` -Callback hell may also occur if the [`runAfter` injection](#runafter-injection) -pattern is used too densely, but this can be avoided by interspersing injections -with sequential steps. +Callback hell may also occur if the [`runAfter` injection](#runafter-injection) pattern is used too densely, but this can be avoided by interspersing injections with sequential steps. #### Example: Chained SkyValue lookups -It is often the case that the application logic requires dependent chains of -SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. -Thinking about this naively, this would result in a complex, deeply nested -callback structure. +It is often the case that the application logic requires dependent chains of SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. Thinking about this naively, this would result in a complex, deeply nested callback structure. ``` private ValueType1 value1; private ValueType2 value2; private StateMachine step1(...) { - tasks.lookUp(key1, (Consumer) this); // key1 has type KeyType1. + tasks.lookUp(key1, (Consumer<SkyValue>) this); // key1 has type KeyType1. return this::step2; } @@ -1146,91 +907,48 @@ private void acceptValueType2(SkyValue value) { } ``` -However, since continuations are specified as method references, the code looks -procedural across state transitions: `step2` follows `step1`. Note that here, a -lambda is used to assign `value2`. This makes the ordering of the code match the -ordering of the computation from top-to-bottom. +However, since continuations are specified as method references, the code looks procedural across state transitions: `step2` follows `step1`. Note that here, a lambda is used to assign `value2`. This makes the ordering of the code match the ordering of the computation from top-to-bottom. ### Miscellaneous Tips #### Readability: Execution Ordering -To improve readability, strive to keep the `StateMachine.step` implementations -in execution order and callback implementations immediately following where they -are passed in the code. This isn't always possible where the control flow -branches. Additional comments might be helpful in such cases. +To improve readability, strive to keep the `StateMachine.step` implementations in execution order and callback implementations immediately following where they are passed in the code. This isn't always possible where the control flow branches. Additional comments might be helpful in such cases. -In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an -intermediate method reference is created to achieve this. This trades a small -amount of performance for readability, which is likely worthwhile here. +In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an intermediate method reference is created to achieve this. This trades a small amount of performance for readability, which is likely worthwhile here. #### Generational Hypothesis -Medium-lived *Java* objects break the generational hypothesis of the *Java* -garbage collector, which is designed to handle objects that live for a very -short time or objects that live forever. By definition, objects in -`SkyKeyComputeState` violate this hypothesis. Such objects, containing the -constructed tree of all still-running `StateMachine`s, rooted at `Driver` have -an intermediate lifespan as they suspend, waiting for asynchronous computations -to complete. - -It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes -possible to observe an increase in GC time, even with dramatic decreases in -actual garbage generated. Since `StateMachine`s have an intermediate lifespan -they could be promoted to old gen, causing it to fill up more quickly, thus -necessitating more expensive major or full GCs to clean up. - -The initial precaution is to minimize the use of `StateMachine` variables, but -it is not always feasible, for example, if a value is needed across multiple -states. Where it is possible, local stack `step` variables are young generation -variables and efficiently GC'd. - -For `StateMachine` variables, breaking things down into subtasks and following -the recommended pattern for [Propagating values between -`StateMachine`s](#propagating-values) is also helpful. Observe that when -following the pattern, only child `StateMachine`s have references to parent -`StateMachine`s and not vice versa. This means that as children complete and -update the parents using result callbacks, the children naturally fall out of -scope and become eligible for GC. - -Finally, in some cases, a `StateMachine` variable is needed in earlier states -but not in later states. It can be beneficial to null out references of large -objects once it is known that they are no longer needed. +Medium-lived *Java* objects break the generational hypothesis of the *Java* garbage collector, which is designed to handle objects that live for a very short time or objects that live forever. By definition, objects in `SkyKeyComputeState` violate this hypothesis. Such objects, containing the constructed tree of all still-running `StateMachine`s, rooted at `Driver` have an intermediate lifespan as they suspend, waiting for asynchronous computations to complete. + +It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes possible to observe an increase in GC time, even with dramatic decreases in actual garbage generated. Since `StateMachine`s have an intermediate lifespan they could be promoted to old gen, causing it to fill up more quickly, thus necessitating more expensive major or full GCs to clean up. + +The initial precaution is to minimize the use of `StateMachine` variables, but it is not always feasible, for example, if a value is needed across multiple states. Where it is possible, local stack `step` variables are young generation variables and efficiently GC'd. + +For `StateMachine` variables, breaking things down into subtasks and following the recommended pattern for [Propagating values between `StateMachine`s](#propagating-values) is also helpful. Observe that when following the pattern, only child `StateMachine`s have references to parent `StateMachine`s and not vice versa. This means that as children complete and update the parents using result callbacks, the children naturally fall out of scope and become eligible for GC. + +Finally, in some cases, a `StateMachine` variable is needed in earlier states but not in later states. It can be beneficial to null out references of large objects once it is known that they are no longer needed. #### Naming states -When naming a method, it's usually possible to name a method for the behavior -that happens within that method. It's less clear how to do this in -`StateMachine`s because there is no stack. For example, suppose method `foo` -calls a sub-method `bar`. In a `StateMachine`, this could be translated into the -state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior -`bar`. As a result, method names for states tend to be narrower in scope, -potentially reflecting local behavior. +When naming a method, it's usually possible to name a method for the behavior that happens within that method. It's less clear how to do this in `StateMachine`s because there is no stack. For example, suppose method `foo` calls a sub-method `bar`. In a `StateMachine`, this could be translated into the state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior `bar`. As a result, method names for states tend to be narrower in scope, potentially reflecting local behavior. ### Concurrency tree diagram -The following is an alternative view of the diagram in [Structured -concurrency](#structured-concurrency) that better depicts the tree structure. -The blocks form a small tree. +The following is an alternative view of the diagram in [Structured concurrency](#structured-concurrency) that better depicts the tree structure. The blocks form a small tree. ![Structured Concurrency 3D](/contribute/images/structured-concurrency-3d.svg) -[^1]: In contrast to Skyframe's convention of restarting from the beginning when - values are not available. -[^2]: Note that `step` is permitted to throw `InterruptedException`, but the - examples omit this. There are a few low methods in *Bazel* code that throw - this exception and it propagates up to the `Driver`, to be described later, - that runs the `StateMachine`. It's fine to not declare it to be thrown when - unneeded. -[^3]: Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which - performs *independent* work for each dependency. Instead of manipulating - complex data structures that process all the dependencies at once, - introducing inefficiencies, each dependency has its own independent - `StateMachine`. -[^4]: Multiple `tasks.lookUp` calls within a single step are batched together. - Additional batching can be created by lookups occurring within concurrent - subtasks. -[^5]: This is conceptually similar to Java’s structured concurrency - [jeps/428](https://openjdk.org/jeps/428). -[^6]: Doing this is similar to spawning a thread and joining it to achieve - sequential composition. +## Footnotes + +1. In contrast to Skyframe's convention of restarting from the beginning when values are not available. [↩](#user-content-fnref-1) + +2. Note that `step` is permitted to throw `InterruptedException`, but the examples omit this. There are a few low methods in *Bazel* code that throw this exception and it propagates up to the `Driver`, to be described later, that runs the `StateMachine`. It's fine to not declare it to be thrown when unneeded. [↩](#user-content-fnref-2) + +3. Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which performs *independent* work for each dependency. Instead of manipulating complex data structures that process all the dependencies at once, introducing inefficiencies, each dependency has its own independent `StateMachine`. [↩](#user-content-fnref-3) + +4. Multiple `tasks.lookUp` calls within a single step are batched together. Additional batching can be created by lookups occurring within concurrent subtasks. [↩](#user-content-fnref-4) + +5. This is conceptually similar to Java’s structured concurrency [jeps/428](https://openjdk.org/jeps/428). [↩](#user-content-fnref-5) + +6. Doing this is similar to spawning a thread and joining it to achieve sequential composition. [↩](#user-content-fnref-6) diff --git a/contribute/windows-chocolatey-maintenance.mdx b/contribute/windows-chocolatey-maintenance.mdx index c6aee8fb..a570547d 100644 --- a/contribute/windows-chocolatey-maintenance.mdx +++ b/contribute/windows-chocolatey-maintenance.mdx @@ -2,22 +2,16 @@ title: 'Maintaining Bazel Chocolatey package on Windows' --- - - -Note: The Chocolatey package is experimental; please provide feedback -(`@petemounce` in issue tracker). +Note: The Chocolatey package is experimental; please provide feedback (`@petemounce` in issue tracker). ## Prerequisites You need: -* [chocolatey package manager](https://chocolatey.org) installed -* (to publish) a chocolatey API key granting you permission to publish the - `bazel` package - * [@petemounce](https://github.com/petemounce) currently - maintains this unofficial package. -* (to publish) to have set up that API key for the chocolatey source locally - via `choco apikey -k -s https://chocolatey.org/` +- [chocolatey package manager](https://chocolatey.org) installed +- (to publish) a chocolatey API key granting you permission to publish the `bazel` package + - [@petemounce](https://github.com/petemounce) currently maintains this unofficial package. +- (to publish) to have set up that API key for the chocolatey source locally via `choco apikey -k <your key here> -s https://chocolatey.org/` ## Build @@ -29,44 +23,38 @@ pushd scripts/packages/chocolatey popd ``` -Should result in `scripts/packages/chocolatey/bazel..nupkg` being -created. +Should result in `scripts/packages/chocolatey/bazel.<version>.nupkg` being created. The `build.ps1` script supports `mode` values `local`, `rc` and `release`. ## Test -0. Build the package (with `-mode local`) +1. Build the package (with `-mode local`) - * run a webserver (`python -m SimpleHTTPServer` in - `scripts/packages/chocolatey` is convenient and starts one on - `http://localhost:8000`) + - run a webserver (`python -m SimpleHTTPServer` in `scripts/packages/chocolatey` is convenient and starts one on `http://localhost:8000`) -0. Test the install +2. Test the install - The `test.ps1` should install the package cleanly (and error if it did not - install cleanly), then tell you what to do next. + The `test.ps1` should install the package cleanly (and error if it did not install cleanly), then tell you what to do next. -0. Test the uninstall +3. Test the uninstall - ```sh - choco uninstall bazel - # should remove bazel from the system - ``` + ```sh + choco uninstall bazel + # should remove bazel from the system + ``` Chocolatey's moderation process automates checks here as well. ## Release -Modify `tools/parameters.json` for the new release's URI and checksum once the -release has been published to github releases. +Modify `tools/parameters.json` for the new release's URI and checksum once the release has been published to github releases. ```powershell -./build.ps1 -version -isRelease -./test.ps1 -version +./build.ps1 -version <version> -isRelease +./test.ps1 -version <version> # if the test.ps1 passes choco push bazel.x.y.z.nupkg --source https://chocolatey.org/ ``` -Chocolatey.org will then run automated checks and respond to the push via email -to the maintainers. +Chocolatey.org will then run automated checks and respond to the push via email to the maintainers. diff --git a/contribute/windows-scoop-maintenance.mdx b/contribute/windows-scoop-maintenance.mdx index 58e2a6c4..dc4aadd9 100644 --- a/contribute/windows-scoop-maintenance.mdx +++ b/contribute/windows-scoop-maintenance.mdx @@ -2,38 +2,26 @@ title: 'Maintaining Bazel Scoop package on Windows' --- - - -Note: The Scoop package is experimental. To provide feedback, go to -`@excitoon` in issue tracker. +Note: The Scoop package is experimental. To provide feedback, go to `@excitoon` in issue tracker. ## Prerequisites You need: -* [Scoop package manager](https://scoop.sh/) installed -* GitHub account in order to publish and create pull requests to - [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) - * [@excitoon](https://github.com/excitoon) currently maintains this - unofficial package. Feel free to ask questions by - [e-mail](mailto:vladimir.chebotarev@gmail.com) or - [Telegram](http://telegram.me/excitoon). +- [Scoop package manager](https://scoop.sh/) installed +- GitHub account in order to publish and create pull requests to [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) + - [@excitoon](https://github.com/excitoon) currently maintains this unofficial package. Feel free to ask questions by [e-mail](mailto:vladimir.chebotarev@gmail.com) or [Telegram](http://telegram.me/excitoon). ## Release process -Scoop packages are very easy to maintain. Once you have the URL of released -Bazel, you need to make appropriate changes in -[this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): +Scoop packages are very easy to maintain. Once you have the URL of released Bazel, you need to make appropriate changes in [this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): - update version - update dependencies if needed - update URL - update hash (`sha256` by default) -In your filesystem, `bazel.json` is located in the directory -`%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to -your clone of a Git repository -[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). +In your filesystem, `bazel.json` is located in the directory `%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to your clone of a Git repository [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). Test the result: @@ -44,9 +32,7 @@ bazel version bazel something_else ``` -The first time, make a fork of -[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and -specify it as your own remote for `%UserProfile%/scoop/buckets/main`: +The first time, make a fork of [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and specify it as your own remote for `%UserProfile%/scoop/buckets/main`: ``` git remote add mine FORK_URL diff --git a/docs/android-build-performance.mdx b/docs/android-build-performance.mdx index 0d5edc77..a96a199d 100644 --- a/docs/android-build-performance.mdx +++ b/docs/android-build-performance.mdx @@ -2,53 +2,39 @@ title: 'Android Build Performance' --- - - -This page contains information on optimizing build performance for Android -apps specifically. For general build performance optimization with Bazel, see -[Optimizing Performance](/rules/performance). +This page contains information on optimizing build performance for Android apps specifically. For general build performance optimization with Bazel, see [Optimizing Performance](/rules/performance). ## Recommended flags -The flags are in the -[`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so -they can be pasted directly into a `bazelrc` file and invoked with -`--config=` on the command line. +The flags are in the [`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so they can be pasted directly into a `bazelrc` file and invoked with `--config=<configuration_name>` on the command line. **Profiling performance** -Bazel writes a JSON trace profile by default to a file called -`command.profile.gz` in Bazel's output base. -See the [JSON Profile documentation](/rules/performance#performance-profiling) for -how to read and interact with the profile. +Bazel writes a JSON trace profile by default to a file called `command.profile.gz` in Bazel's output base. See the [JSON Profile documentation](/rules/performance#performance-profiling) for how to read and interact with the profile. **Persistent workers for Android build actions**. -A subset of Android build actions has support for -[persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). +A subset of Android build actions has support for [persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). These actions' mnemonics are: -* DexBuilder -* Javac -* Desugar -* AaptPackage -* AndroidResourceParser -* AndroidResourceValidator -* AndroidResourceCompiler -* RClassGenerator -* AndroidResourceLink -* AndroidAapt2 -* AndroidAssetMerger -* AndroidResourceMerger -* AndroidCompiledResourceMerger - -Enabling workers can result in better build performance by saving on JVM -startup costs from invoking each of these tools, but at the cost of increased -memory usage on the system by persisting them. - -To enable workers for these actions, apply these flags with -`--config=android_workers` on the command line: +- DexBuilder +- Javac +- Desugar +- AaptPackage +- AndroidResourceParser +- AndroidResourceValidator +- AndroidResourceCompiler +- RClassGenerator +- AndroidResourceLink +- AndroidAapt2 +- AndroidAssetMerger +- AndroidResourceMerger +- AndroidCompiledResourceMerger + +Enabling workers can result in better build performance by saving on JVM startup costs from invoking each of these tools, but at the cost of increased memory usage on the system by persisting them. + +To enable workers for these actions, apply these flags with `--config=android_workers` on the command line: ``` build:android_workers --strategy=DexBuilder=worker @@ -68,11 +54,7 @@ build:android_workers --strategy=Desugar=worker build:android_workers --persistent_android_resource_processor ``` -The default number of persistent workers created per action is `4`. We have -[measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) -by capping the number of instances for each action to `1` or `2`, although this -may vary depending on the system Bazel is running on, and the project being -built. +The default number of persistent workers created per action is `4`. We have [measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) by capping the number of instances for each action to `1` or `2`, although this may vary depending on the system Bazel is running on, and the project being built. To cap the number of instances for an action, apply these flags: @@ -86,12 +68,8 @@ build:android_workers --worker_max_instances=AaptPackage=2 **Using AAPT2** -[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved -performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the -`--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on -`android_binary` and `android_local_test`. +[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the `--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on `android_binary` and `android_local_test`. **SSD optimizations** -The `--experimental_multi_threaded_digest` flag is useful for optimizing digest -computation on SSDs. +The `--experimental_multi_threaded_digest` flag is useful for optimizing digest computation on SSDs. diff --git a/docs/android-instrumentation-test.mdx b/docs/android-instrumentation-test.mdx index fca7b577..e5b23f88 100644 --- a/docs/android-instrumentation-test.mdx +++ b/docs/android-instrumentation-test.mdx @@ -2,34 +2,23 @@ title: 'Android Instrumentation Tests' --- - - -_If you're new to Bazel, start with the [Building Android with -Bazel](/start/android-app ) tutorial._ +*If you're new to Bazel, start with the [Building Android with Bazel](/start/android-app) tutorial.* ![Running Android instrumentation tests in parallel](/docs/images/android_test.gif "Android instrumentation test") **Figure 1.** Running parallel Android instrumentation tests. -[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) -allows developers to test their apps on Android emulators and devices. -It utilizes real Android framework APIs and the Android Test Library. +[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) allows developers to test their apps on Android emulators and devices. It utilizes real Android framework APIs and the Android Test Library. -For hermeticity and reproducibility, Bazel creates and launches Android -emulators in a sandbox, ensuring that tests always run from a clean state. Each -test gets an isolated emulator instance, allowing tests to run in parallel -without passing states between them. +For hermeticity and reproducibility, Bazel creates and launches Android emulators in a sandbox, ensuring that tests always run from a clean state. Each test gets an isolated emulator instance, allowing tests to run in parallel without passing states between them. -For more information on Android instrumentation tests, check out the [Android -developer -documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). +For more information on Android instrumentation tests, check out the [Android developer documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). Please file issues in the [GitHub issue tracker](https://github.com/bazelbuild/bazel/issues). ## How it works -When you run `bazel test` on an `android_instrumentation_test` target for the -first time, Bazel performs the following steps: +When you run `bazel test` on an `android_instrumentation_test` target for the first time, Bazel performs the following steps: 1. Builds the test APK, APK under test, and their transitive dependencies 2. Creates, boots, and caches clean emulator states @@ -39,9 +28,7 @@ first time, Bazel performs the following steps: 6. Shuts down the emulator 7. Reports the results -In subsequent test runs, Bazel boots the emulator from the clean, cached state -created in step 2, so there are no leftover states from previous runs. Caching -emulator state also speeds up test runs. +In subsequent test runs, Bazel boots the emulator from the clean, cached state created in step 2, so there are no leftover states from previous runs. Caching emulator state also speeds up test runs. ## Prerequisites @@ -54,17 +41,14 @@ Ensure your environment satisfies the following prerequisites: ```posix-terminal bazel info release ``` + This results in output similar to the following: -```none {:.devsite-disable-click-to-copy} +```none release 4.1.0 ``` -- **KVM**. Bazel requires emulators to have [hardware - acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) - with KVM on Linux. You can follow these - [installation instructions](https://help.ubuntu.com/community/KVM/Installation) - for Ubuntu. +- **KVM**. Bazel requires emulators to have [hardware acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) with KVM on Linux. You can follow these [installation instructions](https://help.ubuntu.com/community/KVM/Installation) for Ubuntu. To verify that KVM has the correct configuration, run: @@ -74,34 +58,32 @@ apt-get install cpu-checker && kvm-ok If it prints the following message, you have the correct configuration: -```none {:.devsite-disable-click-to-copy} +```none INFO: /dev/kvm exists KVM acceleration can be used ``` -- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires - the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). +- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). To install it, run: ```posix-terminal apt-get install xvfb ``` -Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` -by running: + +Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` by running: ```posix-terminal which Xvfb ``` + The output is the following: ```{:.devsite-disable-click-to-copy} /usr/bin/Xvfb ``` -- **32-bit Libraries**. Some of the binaries used by the test infrastructure are - 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For - Ubuntu, install these 32-bit libraries: +- **32-bit Libraries**. Some of the binaries used by the test infrastructure are 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For Ubuntu, install these 32-bit libraries: ```posix-terminal sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 libbz2-1.0:i386 @@ -115,7 +97,6 @@ Here is a typical target dependency graph of an `android_instrumentation_test`: **Figure 2.** Target dependency graph of an `android_instrumentation_test`. - ### BUILD file The graph translates into a `BUILD` file like this: @@ -169,59 +150,45 @@ android_library( The main attributes of the rule `android_instrumentation_test` are: -- `test_app`: An `android_binary` target. This target contains test code and - dependencies like Espresso and UIAutomator. The selected `android_binary` - target is required to specify an `instruments` attribute pointing to another - `android_binary`, which is the app under test. +- `test_app`: An `android_binary` target. This target contains test code and dependencies like Espresso and UIAutomator. The selected `android_binary` target is required to specify an `instruments` attribute pointing to another `android_binary`, which is the app under test. -- `target_device`: An `android_device` target. This target describes the - specifications of the Android emulator which Bazel uses to create, launch and - run the tests. See the [section on choosing an Android - device](#android-device-target) for more information. +- `target_device`: An `android_device` target. This target describes the specifications of the Android emulator which Bazel uses to create, launch and run the tests. See the [section on choosing an Android device](#android-device-target) for more information. -The test app's `AndroidManifest.xml` must include [an `` -tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). -This tag must specify the attributes for the **package of the target app** and -the **fully qualified class name of the instrumentation test runner**, -`androidx.test.runner.AndroidJUnitRunner`. +The test app's `AndroidManifest.xml` must include [an `<instrumentation>` tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). This tag must specify the attributes for the **package of the target app** and the **fully qualified class name of the instrumentation test runner**, `androidx.test.runner.AndroidJUnitRunner`. Here is an example `AndroidTestManifest.xml` for the test app: ```xml - - + android:versionName="1.0"> - + android:targetPackage="com.example.android.app" /> - + android:targetSdkVersion="27" /> - - - - + <application > + + </application> +</manifest> ``` ### WORKSPACE dependencies -In order to use this rule, your project needs to depend on these external -repositories: +In order to use this rule, your project needs to depend on these external repositories: - `@androidsdk`: The Android SDK. Download this through Android Studio. -- `@android_test_support`: Hosts the test runner, emulator launcher, and - `android_device` targets. You can find the [latest release - here](https://github.com/android/android-test/releases). +- `@android_test_support`: Hosts the test runner, emulator launcher, and `android_device` targets. You can find the [latest release here](https://github.com/android/android-test/releases). -Enable these dependencies by adding the following lines to your `WORKSPACE` -file: +Enable these dependencies by adding the following lines to your `WORKSPACE` file: ```python # Android SDK @@ -243,25 +210,20 @@ android_test_repositories() ## Maven dependencies -For managing dependencies on Maven artifacts from repositories, such as [Google -Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), -you should use a Maven resolver, such as -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). +For managing dependencies on Maven artifacts from repositories, such as [Google Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), you should use a Maven resolver, such as [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). -The rest of this page shows how to use `rules_jvm_external` to -resolve and fetch dependencies from Maven repositories. +The rest of this page shows how to use `rules_jvm_external` to resolve and fetch dependencies from Maven repositories. -## Choosing an android_device target +## Choosing an android\_device target -`android_instrumentation_test.target_device` specifies which Android device to -run the tests on. These `android_device` targets are defined in -[`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). +`android_instrumentation_test.target_device` specifies which Android device to run the tests on. These `android_device` targets are defined in [`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). For example, you can query for the sources for a particular target by running: ```posix-terminal bazel query --output=build @android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86 ``` + Which results in output that looks similar to: ```python @@ -287,29 +249,22 @@ android_device( The device target names use this template: ``` -@android_test_support//tools/android/emulated_devices/{{ "" }}device_type{{ "" }}:{{ "" }}system{{ "" }}_{{ "" }}api_level{{ "" }}_x86_qemu2 +@android_test_support//tools/android/emulated_devices/<var>device_type</var>:<var>system</var>_<var>api_level</var>_x86_qemu2 ``` -In order to launch an `android_device`, the `system_image` for the selected API -level is required. To download the system image, use Android SDK's -`tools/bin/sdkmanager`. For example, to download the system image for -`generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager -"system-images;android-23;default;x86"`. +In order to launch an `android_device`, the `system_image` for the selected API level is required. To download the system image, use Android SDK's `tools/bin/sdkmanager`. For example, to download the system image for `generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager "system-images;android-23;default;x86"`. -To see the full list of supported `android_device` targets in -`@android_test_support`, run the following command: +To see the full list of supported `android_device` targets in `@android_test_support`, run the following command: ```posix-terminal bazel query 'filter("x86_qemu2$", kind(android_device, @android_test_support//tools/android/emulated_devices/...:*))' ``` -Bazel currently supports x86-based emulators only. For better performance, use -`QEMU2` `android_device` targets instead of `QEMU` ones. +Bazel currently supports x86-based emulators only. For better performance, use `QEMU2` `android_device` targets instead of `QEMU` ones. ## Running tests -To run tests, add these lines to your project's -`project root:/.bazelrc` file. +To run tests, add these lines to your project's `<var>project root</var>:<var>/.bazelrc` file. ``` # Configurations for testing with Bazel @@ -339,13 +294,11 @@ Then, use one of the configurations to run tests: - `bazel test //my/test:target --config=headless` - `bazel test //my/test:target --config=local_device` -Use __only one configuration__ or tests will fail. +Use **only one configuration** or tests will fail. ### Headless testing -With `Xvfb`, it is possible to test with emulators without the graphical -interface, also known as headless testing. To disable the graphical interface -when running tests, pass the test argument `--enable_display=false` to Bazel: +With `Xvfb`, it is possible to test with emulators without the graphical interface, also known as headless testing. To disable the graphical interface when running tests, pass the test argument `--enable_display=false` to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=false @@ -353,9 +306,7 @@ bazel test //my/test:target --test_arg=--enable_display=false ### GUI testing -If the `$DISPLAY` environment variable is set, it's possible to enable the -graphical interface of the emulator while the test is running. To do this, pass -these test arguments to Bazel: +If the `$DISPLAY` environment variable is set, it's possible to enable the graphical interface of the emulator while the test is running. To do this, pass these test arguments to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY @@ -363,26 +314,15 @@ bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY ### Testing with a local emulator or device -Bazel also supports testing directly on a locally launched emulator or connected -device. Pass the flags -`--test_strategy=exclusive` and -`--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. -If there is more than one connected device, pass the flag -`--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of -the device/emulator listed in `adb devices`. +Bazel also supports testing directly on a locally launched emulator or connected device. Pass the flags `--test_strategy=exclusive` and `--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. If there is more than one connected device, pass the flag `--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of the device/emulator listed in `adb devices`. ## Sample projects -If you are looking for canonical project samples, see the [Android testing -samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) -for projects using Espresso and UIAutomator. +If you are looking for canonical project samples, see the [Android testing samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) for projects using Espresso and UIAutomator. ## Espresso setup -If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) -(`androidx.test.espresso`), you can use the following snippets to set up your -Bazel workspace with the list of commonly used Espresso artifacts and their -dependencies: +If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) (`androidx.test.espresso`), you can use the following snippets to set up your Bazel workspace with the list of commonly used Espresso artifacts and their dependencies: ``` androidx.test.espresso:espresso-core @@ -393,8 +333,7 @@ org.hamcrest:java-hamcrest junit:junit ``` -One way to organize these dependencies is to create a `//:test_deps` shared -library in your `project root/BUILD.bazel` file: +One way to organize these dependencies is to create a `//:test_deps` shared library in your `<var>project root</var>/BUILD.bazel` file: ```python java_library( @@ -411,7 +350,7 @@ java_library( ) ``` -Then, add the required dependencies in `project root/WORKSPACE`: +Then, add the required dependencies in `<var>project root</var>/WORKSPACE`: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -444,8 +383,7 @@ maven_install( ) ``` -Finally, in your test `android_binary` target, add the `//:test_deps` -dependency: +Finally, in your test `android_binary` target, add the `//:test_deps` dependency: ```python android_binary( @@ -463,17 +401,14 @@ android_binary( ### Reading test logs -Use `--test_output=errors` to print logs for failing tests, or -`--test_output=all` to print all test output. If you're looking for an -individual test log, go to -`$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. +Use `--test_output=errors` to print logs for failing tests, or `--test_output=all` to print all test output. If you're looking for an individual test log, go to `$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. -For example, the test logs for `BasicSample` canonical project are in -`bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: +For example, the test logs for `BasicSample` canonical project are in `bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: ```posix-terminal tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ``` + This results in the following output: ```none @@ -531,9 +466,7 @@ $ tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ### Reading emulator logs -The emulator logs for `android_device` targets are stored in the `/tmp/` -directory with the name `emulator_xxxxx.log`, where `xxxxx` is a -randomly-generated sequence of characters. +The emulator logs for `android_device` targets are stored in the `/tmp/` directory with the name `emulator_xxxxx.log`, where `xxxxx` is a randomly-generated sequence of characters. Use this command to find the latest emulator log: @@ -543,8 +476,7 @@ ls -1t /tmp/emulator_*.log | head -n 1 ### Testing against multiple API levels -If you would like to test against multiple API levels, you can use a list -comprehension to create test targets for each API level. For example: +If you would like to test against multiple API levels, you can use a list comprehension to create test targets for each API level. For example: ```python API_LEVELS = [ @@ -563,14 +495,10 @@ API_LEVELS = [ ## Known issues -- [Forked adb server processes are not terminated after - tests](https://github.com/bazelbuild/bazel/issues/4853) -- While APK building works on all platforms (Linux, macOS, Windows), testing - only works on Linux. -- Even with `--config=local_adb`, users still need to specify - `android_instrumentation_test.target_device`. -- If using a local device or emulator, Bazel does not uninstall the APKs after - the test. Clean the packages by running this command: +- [Forked adb server processes are not terminated after tests](https://github.com/bazelbuild/bazel/issues/4853) +- While APK building works on all platforms (Linux, macOS, Windows), testing only works on Linux. +- Even with `--config=local_adb`, users still need to specify `android_instrumentation_test.target_device`. +- If using a local device or emulator, Bazel does not uninstall the APKs after the test. Clean the packages by running this command: ```posix-terminal adb shell pm list diff --git a/docs/android-ndk.mdx b/docs/android-ndk.mdx index b434797d..292231dc 100644 --- a/docs/android-ndk.mdx +++ b/docs/android-ndk.mdx @@ -2,25 +2,13 @@ title: 'Using the Android Native Development Kit with Bazel' --- - - -_If you're new to Bazel, please start with the [Building Android with -Bazel](/start/android-app ) tutorial._ +*If you're new to Bazel, please start with the [Building Android with Bazel](/start/android-app) tutorial.* ## Overview -Bazel can run in many different build configurations, including several that use -the Android Native Development Kit (NDK) toolchain. This means that normal -`cc_library` and `cc_binary` rules can be compiled for Android directly within -Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository -rule and its related bzlmod extension. +Bazel can run in many different build configurations, including several that use the Android Native Development Kit (NDK) toolchain. This means that normal `cc_library` and `cc_binary` rules can be compiled for Android directly within Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository rule and its related bzlmod extension. -For general Android -compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). -This tutorial demonstrates how to integrate C++ library dependencies into -Android apps and uses -[`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK -toolchain discovery and registration. +For general Android compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). This tutorial demonstrates how to integrate C++ library dependencies into Android apps and uses [`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK toolchain discovery and registration. ## Prerequisites @@ -28,12 +16,7 @@ Please ensure that you have installed the Android SDK and NDK. ### NDK and SDK setup -External repository setup varies depending on whether you are using WORKSPACE -or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* -Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of -each other. -If you are using one dependency management solution, you don't need to add -the boilerplate for the other. +External repository setup varies depending on whether you are using WORKSPACE or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of each other. If you are using one dependency management solution, you don't need to add the boilerplate for the other. #### Bzlmod MODULE.bazel setup @@ -77,23 +60,18 @@ android_ndk_repository( Compatibility notes for WORKSPACE: -* Both `rules_android` and `rules_android_ndk` rules require extra - boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date - and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) - file of `rules_android_ndk`'s basic example app. +- Both `rules_android` and `rules_android_ndk` rules require extra boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) file of `rules_android_ndk`'s basic example app. -For more information about the `android_ndk_repository` rule, see its -[docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). +For more information about the `android_ndk_repository` rule, see its [docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). ## Quick start -To build C++ for Android, simply add `cc_library` dependencies to your -`android_binary` or `android_library` rules. +To build C++ for Android, simply add `cc_library` dependencies to your `android_binary` or `android_library` rules. For example, given the following `BUILD` file for an Android app: ```python -# In /app/src/main/BUILD.bazel +# In <project>/app/src/main/BUILD.bazel load("@rules_cc//cc:cc_library.bzl", "cc_library") load("@rules_android//rules:rules.bzl", "android_binary", "android_library") @@ -122,19 +100,17 @@ This `BUILD` file results in the following target graph: ![Example results](/docs/images/android_ndk.png "Build graph results") -**Figure 1.** Build graph of Android project with cc_library dependencies. +**Figure 1.** Build graph of Android project with cc\_library dependencies. To build the app, simply run: ```posix-terminal -bazel build //app/src/main:app --android_platforms= +bazel build //app/src/main:app --android_platforms=<your platform> ``` -Note that if you don't specify `--android_platforms`, your build will fail with -errors about missing JNI headers. +Note that if you don't specify `--android_platforms`, your build will fail with errors about missing JNI headers. -The `bazel build` command compiles the Java files, Android resource files, and -`cc_library` rules, and packages everything into an APK: +The `bazel build` command compiles the Java files, Android resource files, and `cc_library` rules, and packages everything into an APK: ```posix-terminal $ zipinfo -1 bazel-bin/app/src/main/app.apk @@ -150,27 +126,21 @@ META-INF/CERT.RSA META-INF/MANIFEST.MF ``` -Bazel compiles all of the cc_libraries into a single shared object (`.so`) file, -targeted the architectures specified by `--android_platforms`. -See the section on [configuring the target ABI](#configuring-target-abi) for -more details. +Bazel compiles all of the cc\_libraries into a single shared object (`.so`) file, targeted the architectures specified by `--android_platforms`. See the section on [configuring the target ABI](#configuring-target-abi) for more details. ## Example setup -This example is available in the [Bazel examples -repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). +This example is available in the [Bazel examples repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). -In the `BUILD.bazel` file, three targets are defined with the `android_binary`, -`android_library`, and `cc_library` rules. +In the `BUILD.bazel` file, three targets are defined with the `android_binary`, `android_library`, and `cc_library` rules. The `android_binary` top-level target builds the APK. -The `cc_library` target contains a single C++ source file with a JNI function -implementation: +The `cc_library` target contains a single C++ source file with a JNI function implementation: ```c++ -#include -#include +#include <jni.h> +#include <string> extern "C" JNIEXPORT jstring @@ -180,14 +150,11 @@ Java_com_example_android_bazel_MainActivity_stringFromJNI( JNIEnv *env, jobject /* this */) { std::string hello = "Hello from C++"; - return env->NewStringUTF(hello.c_str()); + return env->NewStringUTF(hello.c_str()); } ``` -The `android_library` target specifies the Java sources, resource files, and the -dependency on a `cc_library` target. For this example, `MainActivity.java` loads -the shared object file `libapp.so`, and defines the method signature for the JNI -function: +The `android_library` target specifies the Java sources, resource files, and the dependency on a `cc_library` target. For this example, `MainActivity.java` loads the shared object file `libapp.so`, and defines the method signature for the JNI function: ```java public class MainActivity extends AppCompatActivity { @@ -206,23 +173,19 @@ public class MainActivity extends AppCompatActivity { } ``` -Note: The name of the native library is derived from the name of the top -level `android_binary` target. In this example, it is `app`. +Note: The name of the native library is derived from the name of the top level `android_binary` target. In this example, it is `app`. ## Configuring the target ABI To configure the target ABI, use the `--android_platforms` flag as follows: ```posix-terminal -bazel build //:app --android_platforms={{ "" }}comma-separated list of platforms{{ "" }} +bazel build //:app --android_platforms=<var>comma-separated list of platforms</var> ``` -Just like the `--platforms` flag, the values passed to `--android_platforms` are -the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) -targets, using standard constraint values to describe your device. +Just like the `--platforms` flag, the values passed to `--android_platforms` are the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) targets, using standard constraint values to describe your device. -For example, for an Android device with a 64-bit ARM processor, you'd define -your platform like this: +For example, for an Android device with a 64-bit ARM processor, you'd define your platform like this: ```py platform( @@ -234,43 +197,36 @@ platform( ) ``` -Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) -OS constraint. To migrate the CPU constraint, check this chart: +Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) OS constraint. To migrate the CPU constraint, check this chart: -CPU Value | Platform -------------- | ------------------------------------------ -`armeabi-v7a` | `@platforms//cpu:armv7` -`arm64-v8a` | `@platforms//cpu:arm64` -`x86` | `@platforms//cpu:x86_32` -`x86_64` | `@platforms//cpu:x86_64` +| CPU Value | Platform | +| ------------- | ------------------------ | +| `armeabi-v7a` | `@platforms//cpu:armv7` | +| `arm64-v8a` | `@platforms//cpu:arm64` | +| `x86` | `@platforms//cpu:x86_32` | +| `x86_64` | `@platforms//cpu:x86_64` | -And, of course, for a multi-architecture APK, you pass multiple labels, for -example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in -your top-level `BUILD.bazel` file). +And, of course, for a multi-architecture APK, you pass multiple labels, for example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in your top-level `BUILD.bazel` file). -Bazel is unable to select a default Android platform, so one must be defined and -specified with `--android_platforms`. +Bazel is unable to select a default Android platform, so one must be defined and specified with `--android_platforms`. -Depending on the NDK revision and Android API level, the following ABIs are -available: +Depending on the NDK revision and Android API level, the following ABIs are available: | NDK revision | ABIs | -|--------------|-------------------------------------------------------------| +| ------------ | ----------------------------------------------------------- | | 16 and lower | armeabi, armeabi-v7a, arm64-v8a, mips, mips64, x86, x86\_64 | | 17 and above | armeabi-v7a, arm64-v8a, x86, x86\_64 | -See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) -for more information on these ABIs. +See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) for more information on these ABIs. -Multi-ABI Fat APKs are not recommended for release builds since they increase -the size of the APK, but can be useful for development and QA builds. +Multi-ABI Fat APKs are not recommended for release builds since they increase the size of the APK, but can be useful for development and QA builds. ## Selecting a C++ standard Use the following flags to build according to a C++ standard: | C++ Standard | Flag | -|--------------|-------------------------| +| ------------ | ----------------------- | | C++98 | Default, no flag needed | | C++11 | `--cxxopt=-std=c++11` | | C++14 | `--cxxopt=-std=c++14` | @@ -282,11 +238,9 @@ For example: bazel build //:app --cxxopt=-std=c++11 ``` -Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and -`--linkopt` in the [User Manual](/docs/user-manual#cxxopt). +Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and `--linkopt` in the [User Manual](/docs/user-manual#cxxopt). -Compiler and linker flags can also be specified as attributes in `cc_library` -using `copts` and `linkopts`. For example: +Compiler and linker flags can also be specified as attributes in `cc_library` using `copts` and `linkopts`. For example: ```python cc_library( @@ -299,11 +253,9 @@ cc_library( ## Building a `cc_library` for Android without using `android_binary` -To build a standalone `cc_binary` or `cc_library` for Android without using an -`android_binary`, use the `--platforms` flag. +To build a standalone `cc_binary` or `cc_library` for Android without using an `android_binary`, use the `--platforms` flag. -For example, assuming you have defined Android platforms in -`my/platforms/BUILD`: +For example, assuming you have defined Android platforms in `my/platforms/BUILD`: ```posix-terminal bazel build //my/cc/jni:target \ @@ -312,13 +264,9 @@ bazel build //my/cc/jni:target \ With this approach, the entire build tree is affected. -Note: All of the targets on the command line must be compatible with -building for Android when specifying these flags, which may make it difficult to -use [Bazel wild-cards](/run/build#specifying-build-targets) like -`/...` and `:all`. +Note: All of the targets on the command line must be compatible with building for Android when specifying these flags, which may make it difficult to use [Bazel wild-cards](/run/build#specifying-build-targets) like `/...` and `:all`. -These flags can be put into a `bazelrc` config (one for each ABI), in -`project/.bazelrc`: +These flags can be put into a `bazelrc` config (one for each ABI), in `<var>project</var>/.bazelrc`: ``` common:android_x86 --platforms=//my/platforms:x86 @@ -326,7 +274,7 @@ common:android_x86 --platforms=//my/platforms:x86 common:android_armeabi-v7a --platforms=//my/platforms:armeabi-v7a # In general -common:android_ --platforms=//my/platforms: +common:android_<abi> --platforms=//my/platforms:<abi> ``` Then, to build a `cc_library` for `x86` for example, run: @@ -335,7 +283,4 @@ Then, to build a `cc_library` for `x86` for example, run: bazel build //my/cc/jni:target --config=android_x86 ``` -In general, use this method for low-level targets (like `cc_library`) or when -you know exactly what you're building; rely on the automatic configuration -transitions from `android_binary` for high-level targets where you're expecting -to build a lot of targets you don't control. +In general, use this method for low-level targets (like `cc_library`) or when you know exactly what you're building; rely on the automatic configuration transitions from `android_binary` for high-level targets where you're expecting to build a lot of targets you don't control. diff --git a/docs/bazel-and-android.mdx b/docs/bazel-and-android.mdx index bf3625c9..b043a8bc 100644 --- a/docs/bazel-and-android.mdx +++ b/docs/bazel-and-android.mdx @@ -2,44 +2,27 @@ title: 'Android and Bazel' --- - - -This page contains resources that help you use Bazel with Android projects. It -links to a tutorial, build rules, and other information specific to building -Android projects with Bazel. +This page contains resources that help you use Bazel with Android projects. It links to a tutorial, build rules, and other information specific to building Android projects with Bazel. ## Getting started The following resources will help you work with Bazel on Android projects: -* [Tutorial: Building an Android app](/start/android-app ). This - tutorial is a good place to start learning about Bazel commands and concepts, - and how to build Android apps with Bazel. -* [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). - This codelab explains how to build Android apps with Bazel. +- [Tutorial: Building an Android app](/start/android-app). This tutorial is a good place to start learning about Bazel commands and concepts, and how to build Android apps with Bazel. +- [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). This codelab explains how to build Android apps with Bazel. ## Features -Bazel has Android rules for building and testing Android apps, integrating with -the SDK/NDK, and creating emulator images. There are also Bazel plugins for -Android Studio and IntelliJ. - -* [Android rules](/reference/be/android). The Build Encyclopedia describes the rules - for building and testing Android apps with Bazel. -* [Integration with Android Studio](/install/ide). Bazel is compatible with - Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) - plugin. -* [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` - feature provides automated build-and-deploy functionality for building and - testing Android apps directly on Android devices and emulators. -* [Android instrumentation testing](/docs/android-instrumentation-test) on - emulators and devices. -* [Android NDK integration](/docs/android-ndk). Bazel supports compiling to - native code through direct NDK integration and the C++ rules. -* [Android build performance](/docs/android-build-performance). This page - provides information on optimizing build performance for Android apps. +Bazel has Android rules for building and testing Android apps, integrating with the SDK/NDK, and creating emulator images. There are also Bazel plugins for Android Studio and IntelliJ. + +- [Android rules](/reference/be/android). The Build Encyclopedia describes the rules for building and testing Android apps with Bazel. +- [Integration with Android Studio](/install/ide). Bazel is compatible with Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) plugin. +- [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` feature provides automated build-and-deploy functionality for building and testing Android apps directly on Android devices and emulators. +- [Android instrumentation testing](/docs/android-instrumentation-test) on emulators and devices. +- [Android NDK integration](/docs/android-ndk). Bazel supports compiling to native code through direct NDK integration and the C++ rules. +- [Android build performance](/docs/android-build-performance). This page provides information on optimizing build performance for Android apps. ## Further reading -* Integrating with dependencies from Google Maven and Maven Central with [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external). -* Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). +- Integrating with dependencies from Google Maven and Maven Central with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external). +- Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). diff --git a/docs/bazel-and-apple.mdx b/docs/bazel-and-apple.mdx index 6e4a06fe..430c0f1f 100644 --- a/docs/bazel-and-apple.mdx +++ b/docs/bazel-and-apple.mdx @@ -2,85 +2,52 @@ title: 'Apple Apps and Bazel' --- - - -This page contains resources that help you use Bazel to build macOS and iOS -projects. It links to a tutorial, build rules, and other information specific to -using Bazel to build and test for those platforms. +This page contains resources that help you use Bazel to build macOS and iOS projects. It links to a tutorial, build rules, and other information specific to using Bazel to build and test for those platforms. ## Working with Bazel The following resources will help you work with Bazel on macOS and iOS projects: -* [Tutorial: Building an iOS app](/start/ios-app) -* [Objective-C build rules](/reference/be/objective-c) -* [General Apple rules](https://github.com/bazelbuild/rules_apple) -* [Integration with Xcode](/install/ide) +- [Tutorial: Building an iOS app](/start/ios-app) +- [Objective-C build rules](/reference/be/objective-c) +- [General Apple rules](https://github.com/bazelbuild/rules_apple) +- [Integration with Xcode](/install/ide) ## Migrating to Bazel -If you currently build your macOS and iOS projects with Xcode, follow the steps -in the migration guide to start building them with Bazel: +If you currently build your macOS and iOS projects with Xcode, follow the steps in the migration guide to start building them with Bazel: -* [Migrating from Xcode to Bazel](/migrate/xcode) +- [Migrating from Xcode to Bazel](/migrate/xcode) ## Apple apps and new rules -**Note**: Creating new rules is for advanced build and test scenarios. -You do not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. -The following modules, configuration fragments, and providers will help you -[extend Bazel's capabilities](/extending/concepts) -when building your macOS and iOS projects: +The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your macOS and iOS projects: -* Modules: +- Modules: - * [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) - * [`apple_common`](/rules/lib/toplevel/apple_common) - * [`apple_platform`](/rules/lib/builtins/apple_platform) - * [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) - * [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) + - [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) + - [`apple_common`](/rules/lib/toplevel/apple_common) + - [`apple_platform`](/rules/lib/builtins/apple_platform) + - [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) + - [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) -* Configuration fragments: +- Configuration fragments: - * [`apple`](/rules/lib/fragments/apple) + - [`apple`](/rules/lib/fragments/apple) -* Providers: +- Providers: - * [`ObjcProvider`](/rules/lib/providers/ObjcProvider) - * [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) + - [`ObjcProvider`](/rules/lib/providers/ObjcProvider) + - [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) ## Xcode selection -If your build requires Xcode, Bazel will select an appropriate version based on -the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes -the set of available Xcode versions and sets a default version if -`--xcode_version` is not passed. This default is overridden by the -`--xcode_version` flag, as long as it is set to an Xcode version that is -represented in the `--xcode_config` target. - -If you do not pass `--xcode_config`, Bazel will use the autogenerated -[`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the -Xcode versions available on your host machine. The default version is -the newest available Xcode version. This is appropriate for local execution. - -If you are performing remote builds, you should set `--xcode_config` to an -[`xcode_config`](/reference/be/objective-c#xcode_config) -target whose `versions` attribute is a list of remotely available -[`xcode_version`](/reference/be/objective-c#xcode_version) -targets, and whose `default` attribute is one of these -[`xcode_versions`](/reference/be/objective-c#xcode_version). - -If you are using dynamic execution, you should set `--xcode_config` to an -[`xcode_config`](/reference/be/objective-c#xcode_config) -target whose `remote_versions` attribute is an -[`available_xcodes`](/reference/be/workspace#available_xcodes) -target containing the remotely available Xcode versions, and whose -`local_versions` attribute is an -[`available_xcodes`](/reference/be/workspace#available_xcodes) -target containing the locally available Xcode versions. For `local_versions`, -you probably want to use the autogenerated -`@local_config_xcode//:host_available_xcodes`. The default Xcode version is the -newest mutually available version, if there is one, otherwise the default of the -`local_versions` target. If you prefer to use the `local_versions` default -as the default, you can pass `--experimental_prefer_mutual_default=false`. +If your build requires Xcode, Bazel will select an appropriate version based on the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes the set of available Xcode versions and sets a default version if `--xcode_version` is not passed. This default is overridden by the `--xcode_version` flag, as long as it is set to an Xcode version that is represented in the `--xcode_config` target. + +If you do not pass `--xcode_config`, Bazel will use the autogenerated [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the Xcode versions available on your host machine. The default version is the newest available Xcode version. This is appropriate for local execution. + +If you are performing remote builds, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `versions` attribute is a list of remotely available [`xcode_version`](/reference/be/objective-c#xcode_version) targets, and whose `default` attribute is one of these [`xcode_versions`](/reference/be/objective-c#xcode_version). + +If you are using dynamic execution, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `remote_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the remotely available Xcode versions, and whose `local_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the locally available Xcode versions. For `local_versions`, you probably want to use the autogenerated `@local_config_xcode//:host_available_xcodes`. The default Xcode version is the newest mutually available version, if there is one, otherwise the default of the `local_versions` target. If you prefer to use the `local_versions` default as the default, you can pass `--experimental_prefer_mutual_default=false`. diff --git a/docs/bazel-and-cpp.mdx b/docs/bazel-and-cpp.mdx index 73e502cb..17ecb7c8 100644 --- a/docs/bazel-and-cpp.mdx +++ b/docs/bazel-and-cpp.mdx @@ -2,101 +2,78 @@ title: 'C++ and Bazel' --- - - -This page contains resources that help you use Bazel with C++ projects. It links -to a tutorial, build rules, and other information specific to building C++ -projects with Bazel. +This page contains resources that help you use Bazel with C++ projects. It links to a tutorial, build rules, and other information specific to building C++ projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on C++ projects: -* [Tutorial: Building a C++ project](/start/cpp) -* [C++ common use cases](/tutorials/cpp-use-cases) -* [C/C++ rules](/reference/be/c-cpp) -* Essential Libraries - - [Abseil](https://abseil.io/docs/cpp/quickstart) - - [Boost](https://github.com/nelhage/rules_boost) - - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) -* [C++ toolchain configuration](/docs/cc-toolchain-config-reference) -* [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) -* [Integrating with C++ rules](/configure/integrate-cpp) +- [Tutorial: Building a C++ project](/start/cpp) + +- [C++ common use cases](/tutorials/cpp-use-cases) + +- [C/C++ rules](/reference/be/c-cpp) + +- Essential Libraries + + - [Abseil](https://abseil.io/docs/cpp/quickstart) + - [Boost](https://github.com/nelhage/rules_boost) + - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) + +- [C++ toolchain configuration](/docs/cc-toolchain-config-reference) + +- [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) + +- [Integrating with C++ rules](/configure/integrate-cpp) ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are -best practices specific to C++ projects. +In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to C++ projects. ### BUILD files Follow the guidelines below when creating your BUILD files: -* Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) - rule target per compilation unit in the directory. - -* You should granularize your C++ libraries as much as - possible to maximize incrementality and parallelize the build. - -* If there is a single source file in `srcs`, name the library the same as - that C++ file's name. This library should contain C++ file(s), any matching - header file(s), and the library's direct dependencies. For example: - - ```python - cc_library( - name = "mylib", - srcs = ["mylib.cc"], - hdrs = ["mylib.h"], - deps = [":lower-level-lib"] - ) - ``` - -* Use one `cc_test` rule target per `cc_library` target in the file. Name the - target `[library-name]_test` and the source file `[library-name]_test.cc`. - For example, a test target for the `mylib` library target shown above would - look like this: - - ```python - cc_test( - name = "mylib_test", - srcs = ["mylib_test.cc"], - deps = [":mylib"] - ) - ``` +- Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) rule target per compilation unit in the directory. + +- You should granularize your C++ libraries as much as possible to maximize incrementality and parallelize the build. + +- If there is a single source file in `srcs`, name the library the same as that C++ file's name. This library should contain C++ file(s), any matching header file(s), and the library's direct dependencies. For example: + + ```python + cc_library( + name = "mylib", + srcs = ["mylib.cc"], + hdrs = ["mylib.h"], + deps = [":lower-level-lib"] + ) + ``` + +- Use one `cc_test` rule target per `cc_library` target in the file. Name the target `[library-name]_test` and the source file `[library-name]_test.cc`. For example, a test target for the `mylib` library target shown above would look like this: + + ```python + cc_test( + name = "mylib_test", + srcs = ["mylib_test.cc"], + deps = [":mylib"] + ) + ``` ### Include paths Follow these guidelines for include paths: -* Make all include paths relative to the workspace directory. +- Make all include paths relative to the workspace directory. -* Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not - angle-brackets (`#include <foo/bar/baz.h>`). +- Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not angle-brackets (`#include <foo/bar/baz.h>`). -* Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` - (parent directory). +- Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` (parent directory). -* For legacy or `third_party` code that requires includes pointing outside the - project repository, such as external repository includes requiring a prefix, - use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and - [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) - arguments on the `cc_library` rule target. +- For legacy or `third_party` code that requires includes pointing outside the project repository, such as external repository includes requiring a prefix, use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) arguments on the `cc_library` rule target. ### Toolchain features -The following optional [features](/docs/cc-toolchain-config-reference#features) -can improve the hygiene of a C++ project. They can be enabled using the -`--features` command-line flag or the `features` attribute of -[`repo`](/external/overview#repo.bazel), -[`package`](/reference/be/functions#package) or `cc_*` rules: - -* The `parse_headers` feature makes it so that the C++ compiler is used to parse - (but not compile) all header files in the built targets and their dependencies - when using the - [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) - flag. This can help catch issues in header-only libraries and ensure that - headers are self-contained and independent of the order in which they are - included. -* The `layering_check` feature enforces that targets only include headers - provided by their direct dependencies. The default toolchain supports this - feature on Linux with `clang` as the compiler. +The following optional [features](/docs/cc-toolchain-config-reference#features) can improve the hygiene of a C++ project. They can be enabled using the `--features` command-line flag or the `features` attribute of [`repo`](/external/overview#repo.bazel), [`package`](/reference/be/functions#package) or `cc_*` rules: + +- The `parse_headers` feature makes it so that the C++ compiler is used to parse (but not compile) all header files in the built targets and their dependencies when using the [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) flag. This can help catch issues in header-only libraries and ensure that headers are self-contained and independent of the order in which they are included. +- The `layering_check` feature enforces that targets only include headers provided by their direct dependencies. The default toolchain supports this feature on Linux with `clang` as the compiler. diff --git a/docs/bazel-and-java.mdx b/docs/bazel-and-java.mdx index bd5af010..463fa6dd 100644 --- a/docs/bazel-and-java.mdx +++ b/docs/bazel-and-java.mdx @@ -2,164 +2,120 @@ title: 'Java and Bazel' --- - - -This page contains resources that help you use Bazel with Java projects. It -links to a tutorial, build rules, and other information specific to building -Java projects with Bazel. +This page contains resources that help you use Bazel with Java projects. It links to a tutorial, build rules, and other information specific to building Java projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on Java projects: -* [Tutorial: Building a Java Project](/start/java) -* [Java rules](/reference/be/java) +- [Tutorial: Building a Java Project](/start/java) +- [Java rules](/reference/be/java) ## Migrating to Bazel -If you currently build your Java projects with Maven, follow the steps in the -migration guide to start building your Maven projects with Bazel: +If you currently build your Java projects with Maven, follow the steps in the migration guide to start building your Maven projects with Bazel: -* [Migrating from Maven to Bazel](/migrate/maven) +- [Migrating from Maven to Bazel](/migrate/maven) ## Java versions There are two relevant versions of Java that are set with configuration flags: -* the version of the source files in the repository -* the version of the Java runtime that is used to execute the code and to test - it +- the version of the source files in the repository +- the version of the Java runtime that is used to execute the code and to test it ### Configuring the version of the source code in your repository -Without an additional configuration, Bazel assumes all Java source files in the -repository are written in a single Java version. To specify the version of the -sources in the repository add `build --java_language_version={ver}` to -`.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners -should set this flag so that Bazel and its users can reference the source code's -Java version number. For more details, see -[Java language version flag](/docs/user-manual#java-language-version). +Without an additional configuration, Bazel assumes all Java source files in the repository are written in a single Java version. To specify the version of the sources in the repository add `build --java_language_version={ver}` to `.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners should set this flag so that Bazel and its users can reference the source code's Java version number. For more details, see [Java language version flag](/docs/user-manual#java-language-version). ### Configuring the JVM used to execute and test the code Bazel uses one JDK for compilation and another JVM to execute and test the code. -By default Bazel compiles the code using a JDK it downloads and it executes and -tests the code with the JVM installed on the local machine. Bazel searches for -the JVM using `JAVA_HOME` or path. +By default Bazel compiles the code using a JDK it downloads and it executes and tests the code with the JVM installed on the local machine. Bazel searches for the JVM using `JAVA_HOME` or path. -The resulting binaries are compatible with locally installed JVM in system -libraries, which means the resulting binaries depend on what is installed on the -machine. +The resulting binaries are compatible with locally installed JVM in system libraries, which means the resulting binaries depend on what is installed on the machine. -To configure the JVM used for execution and testing use `--java_runtime_version` -flag. The default value is `local_jdk`. +To configure the JVM used for execution and testing use `--java_runtime_version` flag. The default value is `local_jdk`. ### Hermetic testing and compilation -To create a hermetic compile, you can use command line flag -`--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and -tested on the JVM downloaded from a remote repository. For more details, see -[Java runtime version flag](/docs/user-manual#java_runtime_version). +To create a hermetic compile, you can use command line flag `--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and tested on the JVM downloaded from a remote repository. For more details, see [Java runtime version flag](/docs/user-manual#java_runtime_version). ### Configuring compilation and execution of build tools in Java -There is a second pair of JDK and JVM used to build and execute tools, which are -used in the build process, but are not in the build results. That JDK and JVM -are controlled using `--tool_java_language_version` and -`--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, -respectively. +There is a second pair of JDK and JVM used to build and execute tools, which are used in the build process, but are not in the build results. That JDK and JVM are controlled using `--tool_java_language_version` and `--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, respectively. #### Compiling using locally installed JDK -Bazel by default compiles using remote JDK, because it is overriding JDK's -internals. The compilation toolchains using locally installed JDK are configured, -however not used. +Bazel by default compiles using remote JDK, because it is overriding JDK's internals. The compilation toolchains using locally installed JDK are configured, however not used. -To compile using locally installed JDK, that is use the compilation toolchains -for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, -however, mind that this may not work on JDK of arbitrary vendors. +To compile using locally installed JDK, that is use the compilation toolchains for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, however, mind that this may not work on JDK of arbitrary vendors. -For more details, see -[configuring Java toolchains](#config-java-toolchains). +For more details, see [configuring Java toolchains](#config-java-toolchains). ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are -best practices specific to Java projects. +In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to Java projects. ### Directory structure -Prefer Maven's standard directory layout (sources under `src/main/java`, tests -under `src/test/java`). +Prefer Maven's standard directory layout (sources under `src/main/java`, tests under `src/test/java`). ### BUILD files Follow these guidelines when creating your `BUILD` files: -* Use one `BUILD` file per directory containing Java sources, because this - improves build performance. +- Use one `BUILD` file per directory containing Java sources, because this improves build performance. -* Every `BUILD` file should contain one `java_library` rule that looks like - this: +- Every `BUILD` file should contain one `java_library` rule that looks like this: - ```python - java_library( - name = "directory-name", - srcs = glob(["*.java"]), - deps = [...], - ) - ``` + ```python + java_library( + name = "directory-name", + srcs = glob(["*.java"]), + deps = [...], + ) + ``` -* The name of the library should be the name of the directory containing the - `BUILD` file. This makes the label of the library shorter, that is use - `"//package"` instead of `"//package:package"`. +- The name of the library should be the name of the directory containing the `BUILD` file. This makes the label of the library shorter, that is use `"//package"` instead of `"//package:package"`. -* The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of - all Java files in the directory. +- The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of all Java files in the directory. -* Tests should be in a matching directory under `src/test` and depend on this - library. +- Tests should be in a matching directory under `src/test` and depend on this library. ## Creating new rules for advanced Java builds -**Note**: Creating new rules is for advanced build and test scenarios. You do -not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. + +The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your Java projects: + +- Main Java module: [`java_common`](/rules/lib/toplevel/java_common) + +- Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) -The following modules, configuration fragments, and providers will help you -[extend Bazel's capabilities](/extending/concepts) when building your Java -projects: +- Configuration fragment: [`java`](/rules/lib/fragments/java) -* Main Java module: [`java_common`](/rules/lib/toplevel/java_common) -* Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) -* Configuration fragment: [`java`](/rules/lib/fragments/java) -* Other modules: +- Other modules: - * [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) - * [`java_compilation_info`](/rules/lib/providers/java_compilation_info) - * [`java_output_jars`](/rules/lib/providers/java_output_jars) - * [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) - * [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) + - [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) + - [`java_compilation_info`](/rules/lib/providers/java_compilation_info) + - [`java_output_jars`](/rules/lib/providers/java_output_jars) + - [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) + - [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) ## Configuring the Java toolchains Bazel uses two types of Java toolchains: -* execution, used to execute and test Java binaries, controlled with the - `--java_runtime_version` flag -* compilation, used to compile Java sources, controlled with the - `--java_language_version` flag +- execution, used to execute and test Java binaries, controlled with the `--java_runtime_version` flag +- compilation, used to compile Java sources, controlled with the `--java_language_version` flag ### Configuring additional execution toolchains -Execution toolchain is the JVM, either local or from a repository, with some -additional information about its version, operating system, and CPU -architecture. +Execution toolchain is the JVM, either local or from a repository, with some additional information about its version, operating system, and CPU architecture. -Java execution toolchains may added using the `local_java_repository` or -`remote_java_repository` repo rules in a module extension. Adding the rule makes -the JVM available using a flag. When multiple definitions for the same operating -system and CPU architecture are given, the first one is used. +Java execution toolchains may added using the `local_java_repository` or `remote_java_repository` repo rules in a module extension. Adding the rule makes the JVM available using a flag. When multiple definitions for the same operating system and CPU architecture are given, the first one is used. Example configuration of local JVM in MODULE.bazel: @@ -199,51 +155,29 @@ register_toolchains("@openjdk_canary_linux_arm_toolchain_config_repo//:all") ### Configuring additional compilation toolchains -Compilation toolchain is composed of JDK and multiple tools that Bazel uses -during the compilation and that provides additional features, such as: Error -Prone, strict Java dependencies, header compilation, Android desugaring, -coverage instrumentation, and genclass handling for IDEs. +Compilation toolchain is composed of JDK and multiple tools that Bazel uses during the compilation and that provides additional features, such as: Error Prone, strict Java dependencies, header compilation, Android desugaring, coverage instrumentation, and genclass handling for IDEs. -JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the -aforementioned features. Actual compilation is executed using the internal -compiler by the JDK. The JDK used for compilation is specified by `java_runtime` -attribute of the toolchain. +JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the aforementioned features. Actual compilation is executed using the internal compiler by the JDK. The JDK used for compilation is specified by `java_runtime` attribute of the toolchain. -Bazel overrides some JDK internals. In case of JDK version > 9, -`java.compiler` and `jdk.compiler` modules are patched using JDK's flag -`--patch_module`. In case of JDK version 8, the Java compiler is patched using -`-Xbootclasspath` flag. +Bazel overrides some JDK internals. In case of JDK version > 9, `java.compiler` and `jdk.compiler` modules are patched using JDK's flag `--patch_module`. In case of JDK version 8, the Java compiler is patched using `-Xbootclasspath` flag. -VanillaJavaBuilder is a second implementation of JavaBuilder, -which does not modify JDK's internal compiler and does not have any of the -additional features. VanillaJavaBuilder is not used by any of the built-in -toolchains. +VanillaJavaBuilder is a second implementation of JavaBuilder, which does not modify JDK's internal compiler and does not have any of the additional features. VanillaJavaBuilder is not used by any of the built-in toolchains. In addition to JavaBuilder, Bazel uses several other tools during compilation. -The `ijar` tool processes `jar` files to remove everything except call -signatures. Resulting jars are called header jars. They are used to improve the -compilation incrementality by only recompiling downstream dependents when the -body of a function changes. +The `ijar` tool processes `jar` files to remove everything except call signatures. Resulting jars are called header jars. They are used to improve the compilation incrementality by only recompiling downstream dependents when the body of a function changes. The `singlejar` tool packs together multiple `jar` files into a single one. -The `genclass` tool post-processes the output of a Java compilation, and produces -a `jar` containing only the class files for sources that were generated by -annotation processors. +The `genclass` tool post-processes the output of a Java compilation, and produces a `jar` containing only the class files for sources that were generated by annotation processors. -The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in -LCOV format. +The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in LCOV format. The `TestRunner` tool executes JUnit 4 tests in a controlled environment. -You can reconfigure the compilation by adding `default_java_toolchain` macro to -a `BUILD` file and registering it either by adding `register_toolchains` rule to -the `MODULE.bazel` file or by using -[`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. +You can reconfigure the compilation by adding `default_java_toolchain` macro to a `BUILD` file and registering it either by adding `register_toolchains` rule to the `MODULE.bazel` file or by using [`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. -The toolchain is only used when the `source_version` attribute matches the -value specified by `--java_language_version` flag. +The toolchain is only used when the `source_version` attribute matches the value specified by `--java_language_version` flag. Example toolchain configuration: @@ -264,37 +198,26 @@ default_java_toolchain( ) ``` -which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` -or by adding `register_toolchains("//:repository_default_toolchain_definition")` -to the workpace. +which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` or by adding `register_toolchains("//:repository_default_toolchain_definition")` to the workpace. Predefined configurations: -- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 -- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of - arbitrary vendors. -- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt - tools (`ijar`, `singlejar`) -- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are - built from sources (this may be useful on operating system with different - libc) +- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 +- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of arbitrary vendors. +- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt tools (`ijar`, `singlejar`) +- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are built from sources (this may be useful on operating system with different libc) #### Configuring JVM and Java compiler flags -You may configure JVM and javac flags either with flags or with - `default_java_toolchain` attributes. +You may configure JVM and javac flags either with flags or with `default_java_toolchain` attributes. -The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and -`--host_javacopt`. +The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and `--host_javacopt`. -The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, -`javabuilder_jvm_opts`, and `turbine_jvm_opts`. +The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, `javabuilder_jvm_opts`, and `turbine_jvm_opts`. #### Package specific Java compiler flags configuration -You can configure different Java compiler flags for specific source -files using `package_configuration` attribute of `default_java_toolchain`. -Please refer to the example below. +You can configure different Java compiler flags for specific source files using `package_configuration` attribute of `default_java_toolchain`. Please refer to the example below. ```python load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolchain") @@ -329,14 +252,11 @@ package_group( #### Multiple versions of Java source code in a single repository -Bazel only supports compiling a single version of Java sources in a build. -build. This means that when building a Java test or an application, all - dependencies are built against the same Java version. +Bazel only supports compiling a single version of Java sources in a build. build. This means that when building a Java test or an application, all dependencies are built against the same Java version. However, separate builds may be executed using different flags. -To make the task of using different flags easier, sets of flags for a specific -version may be grouped with `.bazelrc` configs": +To make the task of using different flags easier, sets of flags for a specific version may be grouped with `.bazelrc` configs": ```python build:java8 --java_language_version=8 @@ -345,5 +265,4 @@ build:java11 --java_language_version=11 build:java11 --java_runtime_version=remotejdk_11 ``` -These configs can be used with the `--config` flag, for example -`bazel test --config=java11 //:java11_test`. +These configs can be used with the `--config` flag, for example `bazel test --config=java11 //:java11_test`. diff --git a/docs/bazel-and-javascript.mdx b/docs/bazel-and-javascript.mdx index 63d80189..4a2c176a 100644 --- a/docs/bazel-and-javascript.mdx +++ b/docs/bazel-and-javascript.mdx @@ -2,23 +2,19 @@ title: 'JavaScript and Bazel' --- - - -This page contains resources that help you use Bazel with JavaScript projects. -It links to build rules and other information specific to building JavaScript -with Bazel. +This page contains resources that help you use Bazel with JavaScript projects. It links to build rules and other information specific to building JavaScript with Bazel. The following resources will help you work with Bazel on JavaScript projects: -* [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) -* [rules_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs -* [rules_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler -* [rules_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier -* [rules_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) -* [rules_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) -* [rules_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) -* [rules_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler -* [rules_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) -* [rules_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) -* [rules_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) -* [rules_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) +- [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) +- [rules\_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs +- [rules\_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler +- [rules\_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier +- [rules\_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) +- [rules\_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) +- [rules\_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) +- [rules\_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler +- [rules\_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) +- [rules\_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) +- [rules\_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) +- [rules\_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) diff --git a/docs/cc-toolchain-config-reference.mdx b/docs/cc-toolchain-config-reference.mdx new file mode 100644 index 00000000..d000204c --- /dev/null +++ b/docs/cc-toolchain-config-reference.mdx @@ -0,0 +1,488 @@ +--- +title: 'C++ Toolchain Configuration' +--- + +## Overview + +To invoke the compiler with the right options, Bazel needs some knowledge about the compiler internals, such as include directories and important flags. In other words, Bazel needs a simplified model of the compiler to understand its workings. + +Bazel needs to know the following: + +- Whether the compiler supports thinLTO, modules, dynamic linking, or PIC (position independent code). +- Paths to the required tools such as gcc, ld, ar, objcopy, and so on. +- The built-in system include directories. Bazel needs these to validate that all headers that were included in the source file were properly declared in the `BUILD` file. +- The default sysroot. +- Which flags to use for compilation, linking, archiving. +- Which flags to use for the supported compilation modes (opt, dbg, fastbuild). +- Make variables specifically required by the compiler. + +If the compiler has support for multiple architectures, Bazel needs to configure them separately. + +[`CcToolchainConfigInfo`](/rules/lib/providers/CcToolchainConfigInfo) is a provider that provides the necessary level of granularity for configuring the behavior of Bazel's C++ rules. By default, Bazel automatically configures `CcToolchainConfigInfo` for your build, but you have the option to configure it manually. For that, you need a Starlark rule that provides the `CcToolchainConfigInfo` and you need to point the [`toolchain_config`](/reference/be/c-cpp#cc_toolchain.toolchain_config) attribute of the [`cc_toolchain`](/reference/be/c-cpp#cc_toolchain) to your rule. You can create the `CcToolchainConfigInfo` by calling [`cc_common.create_cc_toolchain_config_info()`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info). You can find Starlark constructors for all structs you'll need in the process in [`@rules_cc//cc:cc_toolchain_config_lib.bzl`](https://github.com/bazelbuild/rules_cc/blob/master/cc/cc_toolchain_config_lib.bzl). + +When a C++ target enters the analysis phase, Bazel selects the appropriate `cc_toolchain` target based on the `BUILD` file, and obtains the `CcToolchainConfigInfo` provider from the target specified in the `cc_toolchain.toolchain_config` attribute. The `cc_toolchain` target passes this information to the C++ target through a `CcToolchainProvider`. + +For example, a compile or link action, instantiated by a rule such as `cc_binary` or `cc_library`, needs the following information: + +- The compiler or linker to use +- Command-line flags for the compiler/linker +- Configuration flags passed through the `--copt/--linkopt` options +- Environment variables +- Artifacts needed in the sandbox in which the action executes + +All of the above information except the artifacts required in the sandbox is specified in the Starlark target that the `cc_toolchain` points to. + +The artifacts to be shipped to the sandbox are declared in the `cc_toolchain` target. For example, with the `cc_toolchain.linker_files` attribute you can specify the linker binary and toolchain libraries to ship into the sandbox. + +## Toolchain selection + +The toolchain selection logic operates as follows: + +1. User specifies a `cc_toolchain_suite` target in the `BUILD` file and points Bazel to the target using the [`--crosstool_top` option](/docs/user-manual#flag--crosstool_top). + +2. The `cc_toolchain_suite` target references multiple toolchains. The values of the `--cpu` and `--compiler` flags determine which of those toolchains is selected, either based only on the `--cpu` flag value, or based on a joint `--cpu | --compiler` value. The selection process is as follows: + +- If the `--compiler` option is specified, Bazel selects the corresponding entry from the `cc_toolchain_suite.toolchains` attribute with `--cpu | --compiler`. If Bazel does not find a corresponding entry, it throws an error. + +- If the `--compiler` option is not specified, Bazel selects the corresponding entry from the `cc_toolchain_suite.toolchains` attribute with just `--cpu`. + +- If no flags are specified, Bazel inspects the host system and selects a `--cpu` value based on its findings. See the [inspection mechanism code](https://source.bazel.build/bazel/+/1b73bc37e184e71651eb631223dcce321ba16211:src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java). + +Once a toolchain has been selected, corresponding `feature` and `action_config` objects in the Starlark rule govern the configuration of the build (that is, items described later). These messages allow the implementation of fully fledged C++ features in Bazel without modifying the Bazel binary. C++ rules support multiple unique actions documented in detail [in the Bazel source code](https://source.bazel.build/bazel/+/4f547a7ea86df80e4c76145ffdbb0c8b75ba3afa:tools/build_defs/cc/action_names.bzl). + +## Features + +A feature is an entity that requires command-line flags, actions, constraints on the execution environment, or dependency alterations. A feature can be something as simple as allowing `BUILD` files to select configurations of flags, such as `treat_warnings_as_errors`, or interact with the C++ rules and include new compile actions and inputs to the compilation, such as `header_modules` or `thin_lto`. + +Ideally, `CcToolchainConfigInfo` contains a list of features, where each feature consists of one or more flag groups, each defining a list of flags that apply to specific Bazel actions. + +A feature is specified by name, which allows full decoupling of the Starlark rule configuration from Bazel releases. In other words, a Bazel release does not affect the behavior of `CcToolchainConfigInfo` configurations as long as those configurations do not require the use of new features. + +A feature is enabled in one of the following ways: + +- The feature's `enabled` field is set to `true`. +- Bazel or the rule owner explicitly enable it. +- The user enables it through the `--feature` Bazel option or `features` rule attribute. + +Features can have interdependencies, depend on command line flags, `BUILD` file settings, and other variables. + +### Feature relationships + +Dependencies are typically managed directly with Bazel, which simply enforces the requirements and manages conflicts intrinsic to the nature of the features defined in the build. The toolchain specification allows for more granular constraints for use directly within the Starlark rule that govern feature support and expansion. These are: + +| | | +| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Constraint** | **Description** | +| ``` +requires = [ + feature_set (features = [ + 'feature-name-1', + 'feature-name-2' + ]), +] +``` | Feature-level. The feature is supported only if the specified required features are enabled. For example, when a feature is only supported in certain build modes (`opt`, `dbg`, or `fastbuild`). If \`requires\` contains multiple \`feature\_set\`s the feature is supported if any of the \`feature\_set\`s is satisfied (when all specified features are enabled). | +| ``` +implies = ['feature'] +``` | Feature-level. This feature implies the specified feature(s). Enabling a feature also implicitly enables all features implied by it (that is, it functions recursively).Also provides the ability to factor common subsets of functionality out of a set of features, such as the common parts of sanitizers. Implied features cannot be disabled. | +| ``` +provides = ['feature'] +``` | Feature-level. Indicates that this feature is one of several mutually exclusive alternate features. For example, all of the sanitizers could specify `provides = ["sanitizer"]`.This improves error handling by listing the alternatives if the user asks for two or more mutually exclusive features at once. | +| ``` +with_features = [ + with_feature_set( + features = ['feature-1'], + not_features = ['feature-2'], + ), +] +``` | Flag set-level. A feature can specify multiple flag sets with multiple. When `with_features` is specified, the flag set will only expand to the build command if there is at least one `with_feature_set` for which all of the features in the specified `features` set are enabled, and all the features specified in `not_features` set are disabled. If `with_features` is not specified, the flag set will be applied unconditionally for every action specified. | + +## Actions + +Actions provide the flexibility to modify the circumstances under which an action executes without assuming how the action will be run. An `action_config` specifies the tool binary that an action invokes, while a `feature` specifies the configuration (flags) that determine how that tool behaves when the action is invoked. + +[Features](#features) reference actions to signal which Bazel actions they affect since actions can modify the Bazel action graph. The `CcToolchainConfigInfo` provider contains actions that have flags and tools associated with them, such as `c++-compile`. Flags are assigned to each action by associating them with a feature. + +Each action name represents a single type of action performed by Bazel, such as compiling or linking. There is, however, a many-to-one relationship between actions and Bazel action types, where a Bazel action type refers to a Java class that implements an action (such as `CppCompileAction`). In particular, the "assembler actions" and "compiler actions" in the table below are `CppCompileAction`, while the link actions are `CppLinkAction`. + +### Assembler actions + +| | | +| --------------------- | --------------------------------------------------------- | +| **Action** | **Description** | +| `preprocess-assemble` | Assemble with preprocessing. Typically for `.S` files. | +| `assemble` | Assemble without preprocessing. Typically for `.s` files. | + +### Compiler actions + +| | | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Action** | **Description** | +| `cc-flags-make-variable` | Propagates `CC_FLAGS` to genrules. | +| `c-compile` | Compile as C. | +| `c++-compile` | Compile as C++. | +| `c++-header-parsing` | Run the compiler's parser on a header file to ensure that the header is self-contained, as it will otherwise produce compilation errors. Applies only to toolchains that support modules. | + +### Link actions + +| | | +| --------------------------------- | ----------------------------------------------------------- | +| **Action** | **Description** | +| `c++-link-dynamic-library` | Link a shared library containing all of its dependencies. | +| `c++-link-nodeps-dynamic-library` | Link a shared library only containing `cc_library` sources. | +| `c++-link-executable` | Link a final ready-to-run library. | + +### AR actions + +AR actions assemble object files into archive libraries (`.a` files) via `ar` and encode some semantics into the name. + +| | | +| ------------------------- | ---------------------------------- | +| **Action** | **Description** | +| `c++-link-static-library` | Create a static library (archive). | + +### LTO actions + +| | | +| ------------- | ------------------------------------------------------ | +| **Action** | **Description** | +| `lto-backend` | ThinLTO action compiling bitcodes into native objects. | +| `lto-index` | ThinLTO action generating global index. | + +## Using action\_config + +The `action_config` is a Starlark struct that describes a Bazel action by specifying the tool (binary) to invoke during the action and sets of flags, defined by features. These flags apply constraints to the action's execution. + +The `action_config()` constructor has the following parameters: + +| | | +| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Attribute** | **Description** | +| `action_name` | The Bazel action to which this action corresponds. Bazel uses this attribute to discover per-action tool and execution requirements. | +| `tools` | The executable to invoke. The tool applied to the action will be the first tool in the list with a feature set that matches the feature configuration. Default value must be provided. | +| `flag_sets` | A list of flags that applies to a group of actions. Same as for a feature. | +| `env_sets` | A list of environment constraints that applies to a group of actions. Same as for a feature. | + +An `action_config` can require and imply other features and `action_config`s as dictated by the [feature relationships](#feature-relationships) described earlier. This behavior is similar to that of a feature. + +The last two attributes are redundant against the corresponding attributes on features and are included because some Bazel actions require certain flags or environment variables and the goal is to avoid unnecessary `action_config`+`feature` pairs. Typically, sharing a single feature across multiple `action_config`s is preferred. + +You can not define more than one `action_config` with the same `action_name` within the same toolchain. This prevents ambiguity in tool paths and enforces the intention behind `action_config` - that an action's properties are clearly described in a single place in the toolchain. + +### Using tool constructor + +An`action_config` can specify a set of tools via its `tools` parameter. The `tool()` constructor takes in the following parameters: + +| | | +| --------------- | ------------------------------------------------------------------------------------------ | +| **Field** | **Description** | +| `path` | Path to the tool in question (relative to the current location). | +| `with_features` | A list of feature sets out of which at least one must be satisfied for this tool to apply. | + +For a given `action_config`, only a single `tool` applies its tool path and execution requirements to the Bazel action. A tool is selected by iterating through the `tools` attribute on an `action_config` until a tool with a `with_feature` set matching the feature configuration is found (see [Feature relationships](#feature-relationships) earlier on this page for more information). You should end your tool lists with a default tool that corresponds to an empty feature configuration. + +### Example usage + +Features and actions can be used together to implement Bazel actions with diverse cross-platform semantics. For example, debug symbol generation on macOS requires generating symbols in the compile action, then invoking a specialized tool during the link action to create compressed dsym archive, and then decompressing that archive to produce the application bundle and `.plist` files consumable by Xcode. + +With Bazel, this process can instead be implemented as follows, with `unbundle-debuginfo` being a Bazel action: + +``` +load("@rules_cc//cc:defs.bzl", "ACTION_NAMES") + +action_configs = [ + action_config ( + action_name = ACTION_NAMES.cpp_link_executable, + tools = [ + tool( + with_features = [ + with_feature(features=["generate-debug-symbols"]), + ], + path = "toolchain/mac/ld-with-dsym-packaging", + ), + tool (path = "toolchain/mac/ld"), + ], + ), +] + +features = [ + feature( + name = "generate-debug-symbols", + flag_sets = [ + flag_set ( + actions = [ + ACTION_NAMES.c_compile, + ACTION_NAMES.cpp_compile + ], + flag_groups = [ + flag_group( + flags = ["-g"], + ), + ], + ) + ], + implies = ["unbundle-debuginfo"], + ), +] +``` + +This same feature can be implemented entirely differently for Linux, which uses `fission`, or for Windows, which produces `.pdb` files. For example, the implementation for `fission`-based debug symbol generation might look as follows: + +``` +load("@rules_cc//cc:defs.bzl", "ACTION_NAMES") + +action_configs = [ + action_config ( + name = ACTION_NAMES.cpp_compile, + tools = [ + tool( + path = "toolchain/bin/gcc", + ), + ], + ), +] + +features = [ + feature ( + name = "generate-debug-symbols", + requires = [with_feature_set(features = ["dbg"])], + flag_sets = [ + flag_set( + actions = [ACTION_NAMES.cpp_compile], + flag_groups = [ + flag_group( + flags = ["-gsplit-dwarf"], + ), + ], + ), + flag_set( + actions = [ACTION_NAMES.cpp_link_executable], + flag_groups = [ + flag_group( + flags = ["-Wl", "--gdb-index"], + ), + ], + ), + ], + ), +] +``` + +### Flag groups + +`CcToolchainConfigInfo` allows you to bundle flags into groups that serve a specific purpose. You can specify a flag within using pre-defined variables within the flag value, which the compiler expands when adding the flag to the build command. For example: + +``` +flag_group ( + flags = ["%{output_execpath}"], +) +``` + +In this case, the contents of the flag will be replaced by the output file path of the action. + +Flag groups are expanded to the build command in the order in which they appear in the list, top-to-bottom, left-to-right. + +For flags that need to repeat with different values when added to the build command, the flag group can iterate variables of type `list`. For example, the variable `include_path` of type `list`: + +``` +flag_group ( + iterate_over = "include_paths", + flags = ["-I%{include_paths}"], +) +``` + +expands to `-I<path>` for each path element in the `include_paths` list. All flags (or `flag_group`s) in the body of a flag group declaration are expanded as a unit. For example: + +``` +flag_group ( + iterate_over = "include_paths", + flags = ["-I", "%{include_paths}"], +) +``` + +expands to `-I <path>` for each path element in the `include_paths` list. + +A variable can repeat multiple times. For example: + +``` +flag_group ( + iterate_over = "include_paths", + flags = ["-iprefix=%{include_paths}", "-isystem=%{include_paths}"], +) +``` + +expands to: + +``` +-iprefix=<inc0> -isystem=<inc0> -iprefix=<inc1> -isystem=<inc1> +``` + +Variables can correspond to structures accessible using dot-notation. For example: + +``` +flag_group ( + flags = ["-l%{libraries_to_link.name}"], +) +``` + +Structures can be nested and may also contain sequences. To prevent name clashes and to be explicit, you must specify the full path through the fields. For example: + +``` +flag_group ( + iterate_over = "libraries_to_link", + flag_groups = [ + flag_group ( + iterate_over = "libraries_to_link.shared_libraries", + flags = ["-l%{libraries_to_link.shared_libraries.name}"], + ), + ], +) +``` + +### Conditional expansion + +Flag groups support conditional expansion based on the presence of a particular variable or its field using the `expand_if_available`, `expand_if_not_available`, `expand_if_true`, `expand_if_false`, or `expand_if_equal` attributes. For example: + +``` +flag_group ( + iterate_over = "libraries_to_link", + flag_groups = [ + flag_group ( + iterate_over = "libraries_to_link.shared_libraries", + flag_groups = [ + flag_group ( + expand_if_available = "libraries_to_link.shared_libraries.is_whole_archive", + flags = ["--whole_archive"], + ), + flag_group ( + flags = ["-l%{libraries_to_link.shared_libraries.name}"], + ), + flag_group ( + expand_if_available = "libraries_to_link.shared_libraries.is_whole_archive", + flags = ["--no_whole_archive"], + ), + ], + ), + ], +) +``` + +Note: The `--whole_archive` and `--no_whole_archive` options are added to the build command only when a currently iterated library has an `is_whole_archive` field. + +## CcToolchainConfigInfo reference + +This section provides a reference of build variables, features, and other information required to successfully configure C++ rules. + +### CcToolchainConfigInfo build variables + +The following is a reference of `CcToolchainConfigInfo` build variables. + +Note: The **Action** column indicates the relevant action type, if applicable. + +| | | | +| ---------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Variable** | **Action** | **Description** | +| **`source_file`** | compile | Source file to compile. | +| **`input_file`** | strip | Artifact to strip. | +| **`output_file`** | compile, strip | Compilation output. | +| **`output_assembly_file`** | compile | Emitted assembly file. Applies only when the `compile` action emits assembly text, typically when using the `--save_temps` flag. The contents are the same as for `output_file`. | +| **`output_preprocess_file`** | compile | Preprocessed output. Applies only to compile actions that only preprocess the source files, typically when using the `--save_temps` flag. The contents are the same as for `output_file`. | +| **`includes`** | compile | Sequence of files the compiler must unconditionally include in the compiled source. | +| **`include_paths`** | compile | Sequence directories in which the compiler searches for headers included using `#include<foo.h>` and `#include "foo.h"`. | +| **`quote_include_paths`** | compile | Sequence of `-iquote` includes - directories in which the compiler searches for headers included using `#include "foo.h"`. | +| **`system_include_paths`** | compile | Sequence of `-isystem` includes - directories in which the compiler searches for headers included using `#include <foo.h>`. | +| **`dependency_file`** | compile | The `.d` dependency file generated by the compiler. | +| **`preprocessor_defines`** | compile | Sequence of `defines`, such as `--DDEBUG`. | +| **`pic`** | compile | Compiles the output as position-independent code. | +| **`gcov_gcno_file`** | compile | The `gcov` coverage file. | +| **`per_object_debug_info_file`** | compile | The per-object debug info (`.dwp`) file. | +| **`stripopts`** | strip | Sequence of `stripopts`. | +| **`legacy_compile_flags`** | compile | Sequence of flags from legacy `CROSSTOOL` fields such as `compiler_flag`, `optional_compiler_flag`, `cxx_flag`, and `optional_cxx_flag`. | +| **`user_compile_flags`** | compile | Sequence of flags from either the `copt` rule attribute or the `--copt`, `--cxxopt`, and `--conlyopt` flags. | +| **`unfiltered_compile_flags`** | compile | Sequence of flags from the `unfiltered_cxx_flag` legacy `CROSSTOOL` field or the `unfiltered_compile_flags` feature. These are not filtered by the `nocopts` rule attribute. | +| **`sysroot`** | | The `sysroot`. | +| **`runtime_library_search_directories`** | link | Entries in the linker runtime search path (usually set with the `-rpath` flag). | +| **`library_search_directories`** | link | Entries in the linker search path (usually set with the `-L` flag). | +| **`libraries_to_link`** | link | Flags providing files to link as inputs in the linker invocation. | +| **`def_file_path`** | link | Location of def file used on Windows with MSVC. | +| **`linker_param_file`** | link | Location of linker param file created by bazel to overcome command line length limit. | +| **`output_execpath`** | link | Execpath of the output of the linker. | +| **`generate_interface_library`** | link | `"yes"` or `"no"` depending on whether interface library should be generated. | +| **`interface_library_builder_path`** | link | Path to the interface library builder tool. | +| **`interface_library_input_path`** | link | Input for the interface library `ifso` builder tool. | +| **`interface_library_output_path`** | link | Path where to generate interface library using the `ifso` builder tool. | +| **`legacy_link_flags`** | link | Linker flags coming from the legacy `CROSSTOOL` fields. | +| **`user_link_flags`** | link | Linker flags coming from the `--linkopt` or `linkopts` attribute. | +| **`linkstamp_paths`** | link | A build variable giving linkstamp paths. | +| **`force_pic`** | link | Presence of this variable indicates that PIC/PIE code should be generated (Bazel option \`--force\_pic\` was passed). | +| **`strip_debug_symbols`** | link | Presence of this variable indicates that the debug symbols should be stripped. | +| **`is_cc_test`** | link | Truthy when current action is a `cc_test` linking action, false otherwise. | +| **`is_using_fission`** | compile, link | Presence of this variable indicates that fission (per-object debug info) is activated. Debug info will be in `.dwo` files instead of `.o` files and the compiler and linker need to know this. | +| **`fdo_instrument_path`** | compile, link | Path to the directory that stores FDO instrumentation profile. | +| **`fdo_profile_path`** | compile | Path to FDO profile. | +| **`fdo_prefetch_hints_path`** | compile | Path to the cache prefetch profile. | +| **`cs_fdo_instrument_path`** | compile, link | Path to the directory that stores context sensitive FDO instrumentation profile. | + +### Well-known features + +The following is a reference of features and their activation conditions. + +| | | +| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Feature** | **Documentation** | +| **`opt \| dbg \| fastbuild`** | Enabled by default based on compilation mode. | +| **`static_linking_mode \| dynamic_linking_mode`** | Enabled by default based on linking mode. | +| **`per_object_debug_info`** | Enabled if the `supports_fission` feature is specified and enabled and the current compilation mode is specified in the `--fission` flag. | +| **`supports_start_end_lib`** | If enabled (and the option `--start_end_lib` is set), Bazel will not link against static libraries but instead use the `--start-lib/--end-lib` linker options to link against objects directly. This speeds up the build since Bazel doesn't have to build static libraries. | +| **`supports_interface_shared_libraries`** | If enabled (and the option `--interface_shared_objects` is set), Bazel will link targets that have `linkstatic` set to False (`cc_test`s by default) against interface shared libraries. This makes incremental relinking faster. | +| **`supports_dynamic_linker`** | If enabled, C++ rules will know the toolchain can produce shared libraries. | +| **`static_link_cpp_runtimes`** | If enabled, Bazel will link the C++ runtime statically in static linking mode and dynamically in dynamic linking mode. Artifacts specified in the `cc_toolchain.static_runtime_lib` or `cc_toolchain.dynamic_runtime_lib` attribute (depending on the linking mode) will be added to the linking actions. | +| **`supports_pic`** | If enabled, toolchain will know to use PIC objects for dynamic libraries. The \`pic\` variable is present whenever PIC compilation is needed. If not enabled by default, and \`--force\_pic\` is passed, Bazel will request \`supports\_pic\` and validate that the feature is enabled. If the feature is missing, or couldn't be enabled, \`--force\_pic\` cannot be used. | +| **`static_linking_mode \| dynamic_linking_mode`** | Enabled by default based on linking mode. | +| **`no_legacy_features`** | Prevents Bazel from adding legacy features to the C++ configuration when present. See the complete list of features below. | +| **`shorten_virtual_includes`** | If enabled, virtual include header files are linked under `bin/_virtual_includes/<hash of target path>` instead of `bin/<target package path>/_virtual_includes/<target name>`. Useful on Windows to avoid long path issue with MSVC. | + +#### Legacy features patching logic + +Bazel applies the following changes to the toolchain's features for backwards compatibility: + +- Moves `legacy_compile_flags` feature to the top of the toolchain +- Moves `default_compile_flags` feature to the top of the toolchain +- Adds `dependency_file` (if not present) feature to the top of the toolchain +- Adds `pic` (if not present) feature to the top of the toolchain +- Adds `per_object_debug_info` (if not present) feature to the top of the toolchain +- Adds `preprocessor_defines` (if not present) feature to the top of the toolchain +- Adds `includes` (if not present) feature to the top of the toolchain +- Adds `include_paths` (if not present) feature to the top of the toolchain +- Adds `fdo_instrument` (if not present) feature to the top of the toolchain +- Adds `fdo_optimize` (if not present) feature to the top of the toolchain +- Adds `cs_fdo_instrument` (if not present) feature to the top of the toolchain +- Adds `cs_fdo_optimize` (if not present) feature to the top of the toolchain +- Adds `fdo_prefetch_hints` (if not present) feature to the top of the toolchain +- Adds `autofdo` (if not present) feature to the top of the toolchain +- Adds `build_interface_libraries` (if not present) feature to the top of the toolchain +- Adds `dynamic_library_linker_tool` (if not present) feature to the top of the toolchain +- Adds `shared_flag` (if not present) feature to the top of the toolchain +- Adds `linkstamps` (if not present) feature to the top of the toolchain +- Adds `output_execpath_flags` (if not present) feature to the top of the toolchain +- Adds `runtime_library_search_directories` (if not present) feature to the top of the toolchain +- Adds `library_search_directories` (if not present) feature to the top of the toolchain +- Adds `archiver_flags` (if not present) feature to the top of the toolchain +- Adds `libraries_to_link` (if not present) feature to the top of the toolchain +- Adds `force_pic_flags` (if not present) feature to the top of the toolchain +- Adds `user_link_flags` (if not present) feature to the top of the toolchain +- Adds `legacy_link_flags` (if not present) feature to the top of the toolchain +- Adds `static_libgcc` (if not present) feature to the top of the toolchain +- Adds `fission_support` (if not present) feature to the top of the toolchain +- Adds `strip_debug_symbols` (if not present) feature to the top of the toolchain +- Adds `coverage` (if not present) feature to the top of the toolchain +- Adds `llvm_coverage_map_format` (if not present) feature to the top of the toolchain +- Adds `gcc_coverage_map_format` (if not present) feature to the top of the toolchain +- Adds `fully_static_link` (if not present) feature to the bottom of the toolchain +- Adds `user_compile_flags` (if not present) feature to the bottom of the toolchain +- Adds `sysroot` (if not present) feature to the bottom of the toolchain +- Adds `unfiltered_compile_flags` (if not present) feature to the bottom of the toolchain +- Adds `linker_param_file` (if not present) feature to the bottom of the toolchain +- Adds `compiler_input_flags` (if not present) feature to the bottom of the toolchain +- Adds `compiler_output_flags` (if not present) feature to the bottom of the toolchain + +This is a long list of features. The plan is to get rid of them once [Crosstool in Starlark](https://github.com/bazelbuild/bazel/issues/5380) is done. For the curious reader see the implementation in [CppActionConfigs](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java?q=cppactionconfigs\&ss=bazel), and for production toolchains consider adding `no_legacy_features` to make the toolchain more standalone. diff --git a/docs/configurable-attributes.mdx b/docs/configurable-attributes.mdx index 958341f3..6ec62150 100644 --- a/docs/configurable-attributes.mdx +++ b/docs/configurable-attributes.mdx @@ -2,15 +2,9 @@ title: 'Configurable Build Attributes' --- +***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. - -**_Configurable attributes_**, commonly known as [`select()`]( -/reference/be/functions#select), is a Bazel feature that lets users toggle the values -of build rule attributes at the command line. - -This can be used, for example, for a multiplatform library that automatically -chooses the appropriate implementation for the architecture, or for a -feature-configurable binary that can be customized at build time. +This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. ## Example @@ -41,60 +35,30 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the -command line. Specifically, `deps` becomes: - - - - - - - - - - - - - - - - - - - - - - -
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
- -`select()` serves as a placeholder for a value that will be chosen based on -*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) -targets. By using `select()` in a configurable attribute, the attribute -effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: + +| | | +| ----------------------------------------------- | ------------------ | +| Command | deps = | +| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | +| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | +| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | + +`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either: -* They all resolve to the same value. For example, when running on linux x86, this is unambiguous - `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` - is an unambiguous specialization of `values = {"cpu": "x86"}`. +- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when -nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, -`resources`, `cmd`, and most other attributes. Only a small number of attributes -are *non-configurable*, and these are clearly annotated. For example, -`config_setting`'s own -[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies -under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of -the machine running Bazel (which, thanks to cross-compilation, may be different -than the CPU the target is built for). This is known as a -[configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). Given @@ -138,30 +102,19 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and -`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s -build parameters, which include `--cpu=arm`. The `tools` attribute changes -`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on -`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a -[`config_setting`](/reference/be/general#config_setting) or -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of -expected command line flag settings. By encapsulating these in a target, it's -easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands -them for all builds in all projects. These are specified with -[`config_setting`](/reference/be/general#config_setting)'s -[`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -174,31 +127,21 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` -is the expected value for that flag. `:meaningful_condition_name` matches if -*every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, -[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because -it only affects how Bazel reports progress to the user. Targets can't use that -flag to construct their results. The exact set of supported flags isn't -documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with -[Starlark build settings][BuildSettings]. Unlike built-in flags, these are -defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s -[`flag_values`](/reference/be/general#config_setting.flag_values) -attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: ```python config_setting( @@ -211,29 +154,17 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) -for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. -[`--define`](/reference/command-line-reference#flag--define) -is an alternative legacy syntax for custom flags (for example -`--define foo=bar`). This can be expressed either in the -[values](/reference/be/general#config_setting.values) attribute -(`values = {"define": "foo=bar"}`) or the -[define_values](/reference/be/general#config_setting.define_values) attribute -(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards -compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The -`config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition -matches. +The built-in condition `//conditions:default` matches when no other condition matches. -Because of the "exactly one match" rule, a configurable attribute with no match -and no default condition emits a `"no matching conditions"` error. This can -protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -259,16 +190,11 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s -[`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides -flexibility, it can also be burdensome to individually set each one every time -you want to build a target. - [Platforms](/extending/platforms) -let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -326,12 +252,9 @@ platform( ) ``` -The platform can be specified on the command line. It activates the -`config_setting`s that contain a subset of the platform's `constraint_values`, -allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, -you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -358,11 +281,9 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to -check against single values. +This saves the need for boilerplate `config_setting`s when you only need to check against single values. -Platforms are still under development. See the -[documentation](/concepts/platforms) for details. +Platforms are still under development. See the [documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -384,12 +305,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: - - Duplicate labels can appear in different paths of the same `select`. - - Duplicate labels can *not* appear within the same path of a `select`. - - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -`select` cannot appear inside another `select`. If you need to nest `selects` -and your attribute takes other targets as values, use an intermediate target: +- Duplicate labels can appear in different paths of the same `select`. +- Duplicate labels can *not* appear within the same path of a `select`. +- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) + +`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -410,8 +331,7 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND -chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). ## OR chaining @@ -430,9 +350,7 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and -maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple -times. +Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. One option is to predefine the value as a BUILD variable: @@ -451,18 +369,13 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary -duplication. +This makes it easier to manage the dependency. But it still causes unnecessary duplication. For more direct support, use one of the following: ### `selects.with_or` -The -[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing conditions directly inside a `select`: +The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -481,18 +394,12 @@ sh_binary( ### `selects.config_setting_group` - -The -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) -macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) -module supports `OR`ing multiple `config_setting`s: +The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` - ```python config_setting( name = "config1", @@ -516,17 +423,13 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across -different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. -It's an error for multiple conditions to match unless one is an unambiguous -"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the -[Skylib](https://github.com/bazelbuild/bazel-skylib) macro -[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -551,13 +454,11 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed -inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to -fails with the error: +By default, when no condition matches, the target the `select()` is attached to fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -567,8 +468,7 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) -attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: ```python cc_library( @@ -591,8 +491,7 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable -attributes. For example, given: +Rule implementations receive the *resolved values* of configurable attributes. For example, given: ```python # myapp/BUILD @@ -612,9 +511,7 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native -rules. But *they cannot directly manipulate them*. For example, there's no way -for a macro to convert +Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert ```python select({"foo": "val"}, ...) @@ -628,32 +525,22 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* -because macros are evaluated in Bazel's [loading phase](/run/build#loading), -which occurs before flag values are known. -This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while -technically feasible, lack a coherent UI. Further design is necessary to change -this. +Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's -[loading phase](/reference/glossary#loading-phase). -This means it doesn't know what command line flags a target uses since those -flags aren't evaluated until later in the build (in the -[analysis phase](/reference/glossary#analysis-phase)). -So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has -all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` + ```python # myapp/BUILD @@ -702,14 +589,9 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for -details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. -The key issue this question usually means is that select() doesn't work in -*macros*. These are different than *rules*. See the -documentation on [rules](/extending/rules) and [macros](/extending/macros) -to understand the difference. -Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: Define a rule and macro: @@ -789,9 +671,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before -Bazel reads the build's command line flags. That means there isn't enough -information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -814,9 +694,7 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition -[can't evaluate `select()`s](#faq-select-macro), any attempt to do so -usually produces an error: +Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -829,8 +707,7 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly -vigilant with them: +Booleans are a special case that fail silently, so you should be particularly vigilant with them: ```sh $ cat myapp/defs.bzl @@ -852,21 +729,13 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. -So what they're really evaluting is the `select()` object itself. According to -[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design -standards, all objects aside from a very small number of exceptions -automatically return true. +This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before -Bazel knows what the build's command line parameters are. Can they at least read -the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). -What you *can* do today is prepare a straight dictionary, then feed it into a -`select()`: +Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: ```sh $ cat myapp/defs.bzl @@ -878,7 +747,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -910,7 +779,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -918,17 +787,11 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo -rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in -the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't -actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in -the `actual` attribute, to perform this type of run-time determination. This -works correctly, since `alias()` is a BUILD rule, and is evaluated with a -specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. You can even have a `bind()` target point to an `alias()`, if needed. @@ -956,37 +819,31 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target -that depends on either `//:ssl` or `//external:ssl` will see the alternative -located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, -use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo +$ bazel cquery //myapp:foo <desired build flags> //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on -//myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' +$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the -configuration that resolves `//myapp:foo`'s `select()`. You can inspect its -values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -1005,18 +862,13 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the -[cquery docs](/query/cquery) for guidance on using `somepath` to get the right -one. +`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the -same command line flags as the `bazel cquery`. The `config` command relies on -the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform -is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. For example: @@ -1039,15 +891,11 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the -`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the -`:x86_linux_platform` defined here? The author of the `BUILD` file and the user -who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with -these constraints: +Instead, define a `config_setting` that matches **any** platform with these constraints: ```py config_setting( @@ -1068,13 +916,11 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what -platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you -can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -1094,7 +940,4 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and -confuses users when the expected condition does not match. - -[BuildSettings]: /extending/config#user-defined-build-settings +The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. diff --git a/docs/mobile-install.mdx b/docs/mobile-install.mdx index 8bc4a679..1d08dcd0 100644 --- a/docs/mobile-install.mdx +++ b/docs/mobile-install.mdx @@ -2,203 +2,100 @@ title: 'bazel mobile-install' --- +Fast iterative development for Android - - -

Fast iterative development for Android

- -This page describes how `bazel mobile-install` makes iterative development -for Android much faster. It describes the benefits of this approach versus the -drawbacks of separate build and install steps. +This page describes how `bazel mobile-install` makes iterative development for Android much faster. It describes the benefits of this approach versus the drawbacks of separate build and install steps. ## Summary To install small changes to an Android app very quickly, do the following: - 1. Find the `android_binary` rule of the app you want to install. - 2. Connect your device to `adb`. - 3. Run `bazel mobile-install :your_target`. App startup will be a little - slower than usual. - 4. Edit the code or Android resources. - 5. Run `bazel mobile-install :your_target`. - 6. Enjoy a fast and minimal incremental installation! +1. Find the `android_binary` rule of the app you want to install. +2. Connect your device to `adb`. +3. Run `bazel mobile-install :your_target`. App startup will be a little slower than usual. +4. Edit the code or Android resources. +5. Run `bazel mobile-install :your_target`. +6. Enjoy a fast and minimal incremental installation! Some command line options to Bazel that may be useful: - - `--adb` tells Bazel which adb binary to use - - `--adb_arg` can be used to add extra arguments to the command line of `adb`. - One useful application of this is to select which device you want to install - to if you have multiple devices connected to your workstation: - `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=` +- `--adb` tells Bazel which adb binary to use +- `--adb_arg` can be used to add extra arguments to the command line of `adb`. One useful application of this is to select which device you want to install to if you have multiple devices connected to your workstation: `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=<SERIAL>` -When in doubt, look at the -[example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), -contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), -or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) +When in doubt, look at the [example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) ## Introduction -One of the most important attributes of a developer's toolchain is speed: there -is a world of difference between changing the code and seeing it run within a -second and having to wait minutes, sometimes hours, before you get any feedback -on whether your changes do what you expect them to. +One of the most important attributes of a developer's toolchain is speed: there is a world of difference between changing the code and seeing it run within a second and having to wait minutes, sometimes hours, before you get any feedback on whether your changes do what you expect them to. -Unfortunately, the traditional Android toolchain for building an .apk entails -many monolithic, sequential steps and all of these have to be done in order to -build an Android app. At Google, waiting five minutes to build a single-line -change was not unusual on larger projects like Google Maps. +Unfortunately, the traditional Android toolchain for building an .apk entails many monolithic, sequential steps and all of these have to be done in order to build an Android app. At Google, waiting five minutes to build a single-line change was not unusual on larger projects like Google Maps. -`bazel mobile-install` makes iterative development for Android much faster by -using a combination of change pruning, work sharding, and clever manipulation of -Android internals, all without changing any of your app's code. +`bazel mobile-install` makes iterative development for Android much faster by using a combination of change pruning, work sharding, and clever manipulation of Android internals, all without changing any of your app's code. ## Problems with traditional app installation Building an Android app has some issues, including: -- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) -is invoked exactly once in the build and it does not know how to reuse work from -previous builds: it dexes every method again, even though only one method was -changed. +- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) is invoked exactly once in the build and it does not know how to reuse work from previous builds: it dexes every method again, even though only one method was changed. -- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 -connection, and larger apps can take a lot of time to upload. The entire app is -uploaded, even if only small parts have changed, for example, a resource or a -single method, so this can be a major bottleneck. +- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 connection, and larger apps can take a lot of time to upload. The entire app is uploaded, even if only small parts have changed, for example, a resource or a single method, so this can be a major bottleneck. -- Compilation to native code. Android L introduced ART, a new Android runtime, -which compiles apps ahead-of-time rather than compiling them just-in-time like -Dalvik. This makes apps much faster at the cost of longer installation -time. This is a good tradeoff for users because they typically install an app -once and use it many times, but results in slower development where an app is -installed many times and each version is run at most a handful of times. +- Compilation to native code. Android L introduced ART, a new Android runtime, which compiles apps ahead-of-time rather than compiling them just-in-time like Dalvik. This makes apps much faster at the cost of longer installation time. This is a good tradeoff for users because they typically install an app once and use it many times, but results in slower development where an app is installed many times and each version is run at most a handful of times. ## The approach of `bazel mobile-install` `bazel mobile-install `makes the following improvements: - - Sharded desugaring and dexing. After building the app's Java code, Bazel - shards the class files into approximately equal-sized parts and invokes `d8` - separately on them. `d8` is not invoked on shards that did not change since - the last build. These shards are then compiled into separate sharded APKs. +- Sharded desugaring and dexing. After building the app's Java code, Bazel shards the class files into approximately equal-sized parts and invokes `d8` separately on them. `d8` is not invoked on shards that did not change since the last build. These shards are then compiled into separate sharded APKs. - - Incremental file transfer. Android resources, .dex files, and native - libraries are removed from the main .apk and are stored in under a separate - mobile-install directory. This makes it possible to update code and Android - resources independently without reinstalling the whole app. Thus, - transferring the files takes less time and only the .dex files that have - changed are recompiled on-device. +- Incremental file transfer. Android resources, .dex files, and native libraries are removed from the main .apk and are stored in under a separate mobile-install directory. This makes it possible to update code and Android resources independently without reinstalling the whole app. Thus, transferring the files takes less time and only the .dex files that have changed are recompiled on-device. - - Sharded installation. Mobile-install uses Android Studio's - [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) - tool to combine sharded APKs on the connected device and provide a cohesive - experience. +- Sharded installation. Mobile-install uses Android Studio's [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) tool to combine sharded APKs on the connected device and provide a cohesive experience. ### Sharded Dexing -Sharded dexing is reasonably straightforward: once the .jar files are built, a -[tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) -shards them into separate .jar files of approximately equal size, then invokes -`d8` on those that were changed since the previous build. The logic that -determines which shards to dex is not specific to Android: it just uses the -general change pruning algorithm of Bazel. - -The first version of the sharding algorithm simply ordered the .class files -alphabetically, then cut the list up into equal-sized parts, but this proved to -be suboptimal: if a class was added or removed (even a nested or an anonymous -one), it would cause all the classes alphabetically after it to shift by one, -resulting in dexing those shards again. Thus, it was decided to shard Java -packages rather than individual classes. Of course, this still results in -dexing many shards if a new package is added or removed, but that is much less -frequent than adding or removing a single class. - -The number of shards is controlled by command-line configuration, using the -`--define=num_dex_shards=N` flag. In an ideal world, Bazel would -automatically determine how many shards are best, but Bazel currently must know -the set of actions (for example, commands to be executed during the build) before -executing any of them, so it cannot determine the optimal number of shards -because it doesn't know how many Java classes there will eventually be in the -app. Generally speaking, the more shards, the faster the build and the -installation will be, but the slower app startup becomes, because the dynamic -linker has to do more work. The sweet spot is usually between 10 and 50 shards. +Sharded dexing is reasonably straightforward: once the .jar files are built, a [tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) shards them into separate .jar files of approximately equal size, then invokes `d8` on those that were changed since the previous build. The logic that determines which shards to dex is not specific to Android: it just uses the general change pruning algorithm of Bazel. + +The first version of the sharding algorithm simply ordered the .class files alphabetically, then cut the list up into equal-sized parts, but this proved to be suboptimal: if a class was added or removed (even a nested or an anonymous one), it would cause all the classes alphabetically after it to shift by one, resulting in dexing those shards again. Thus, it was decided to shard Java packages rather than individual classes. Of course, this still results in dexing many shards if a new package is added or removed, but that is much less frequent than adding or removing a single class. + +The number of shards is controlled by command-line configuration, using the `--define=num_dex_shards=N` flag. In an ideal world, Bazel would automatically determine how many shards are best, but Bazel currently must know the set of actions (for example, commands to be executed during the build) before executing any of them, so it cannot determine the optimal number of shards because it doesn't know how many Java classes there will eventually be in the app. Generally speaking, the more shards, the faster the build and the installation will be, but the slower app startup becomes, because the dynamic linker has to do more work. The sweet spot is usually between 10 and 50 shards. ### Incremental deployment -Incremental APK shard transfer and installation is now handled by the -`apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). -Whereas earlier (native) versions of mobile-install required manually tracking -first-time installations and selectively apply the `--incremental` -flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) -has been greatly simplified. The same mobile-install -invocation can be used regardless of how many times the app has been installed -or reinstalled. - -At a high level, the `apkdeployer` tool is a wrapper around various `adb` -sub-commands. The main entrypoint logic can be found in the -[`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) -class, with other utility classes colocated in the same package. -The `Deployer` class ingests, among other things, a list of paths to split -APKs and a protobuf with information about the installation, and leverages -deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) -in order to create an install session and incrementally deploy app splits. -See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) -and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) -classes for implementation details. +Incremental APK shard transfer and installation is now handled by the `apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). Whereas earlier (native) versions of mobile-install required manually tracking first-time installations and selectively apply the `--incremental` flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) has been greatly simplified. The same mobile-install invocation can be used regardless of how many times the app has been installed or reinstalled. + +At a high level, the `apkdeployer` tool is a wrapper around various `adb` sub-commands. The main entrypoint logic can be found in the [`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) class, with other utility classes colocated in the same package. The `Deployer` class ingests, among other things, a list of paths to split APKs and a protobuf with information about the installation, and leverages deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) in order to create an install session and incrementally deploy app splits. See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) classes for implementation details. ## Results ### Performance -In general, `bazel mobile-install` results in a 4x to 10x speedup of building -and installing large apps after a small change. +In general, `bazel mobile-install` results in a 4x to 10x speedup of building and installing large apps after a small change. The following numbers were computed for a few Google products: - +![](/docs/images/mobile-install-performance.svg) -This, of course, depends on the nature of the change: recompilation after -changing a base library takes more time. +This, of course, depends on the nature of the change: recompilation after changing a base library takes more time. ### Limitations -The tricks the stub application plays don't work in every case. -The following cases highlight where it does not work as expected: +The tricks the stub application plays don't work in every case. The following cases highlight where it does not work as expected: - - Mobile-install is only supported via the Starlark rules of `rules_android`. - See the ["brief history of mobile-install"](#mobile-install-history) for - more detail. +- Mobile-install is only supported via the Starlark rules of `rules_android`. See the ["brief history of mobile-install"](#mobile-install-history) for more detail. - - Only devices running ART are supported. Mobile-install uses API and runtime features - that only exist on devices running ART, not Dalvik. Any Android runtime more - recent than Android L (API 21+) should be compatible. +- Only devices running ART are supported. Mobile-install uses API and runtime features that only exist on devices running ART, not Dalvik. Any Android runtime more recent than Android L (API 21+) should be compatible. - - Bazel itself must be run with a tool Java runtime _and_ language version - of 17 or higher. +- Bazel itself must be run with a tool Java runtime *and* language version of 17 or higher. - - Bazel versions prior to 8.4.0 must specify some additional flags for - mobile-install. See [the Bazel Android tutorial](/start/android-app). These - flags inform Bazel where the Starlark mobile-install aspect is and which - rules are supported. +- Bazel versions prior to 8.4.0 must specify some additional flags for mobile-install. See [the Bazel Android tutorial](/start/android-app). These flags inform Bazel where the Starlark mobile-install aspect is and which rules are supported. ### A brief history of mobile-install -Earlier Bazel versions _natively_ included built-in build and test rules for -popular languages and ecosystems such as C++, Java, and Android. These rules -were therefore referred to as _native_ rules. Bazel 8 (released in 2024) removed -support for these rules because many of them had been migrated to the -[Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) -for more details. - -The legacy native Android rules also supported a legacy _native_ version of -mobile-install functionality. This is referred to as "mobile-install v1" or -"native mobile-install" now. This functionality was deleted in Bazel 8, along -with the built-in Android rules. -Now, all mobile-install functionality, as well as all Android build and test -rules, are implemented in Starlark and reside in the `rules_android` GitHub -repository. The latest version is known as "mobile-install v3" or "MIv3". +Earlier Bazel versions *natively* included built-in build and test rules for popular languages and ecosystems such as C++, Java, and Android. These rules were therefore referred to as *native* rules. Bazel 8 (released in 2024) removed support for these rules because many of them had been migrated to the [Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) for more details. -_Naming note_: There was a "mobile-install **v2**" available only internally -at Google at one point, but this was never published externally, and only v3 -continues to be used for both Google-internal and OSS rules_android deployment. +The legacy native Android rules also supported a legacy *native* version of mobile-install functionality. This is referred to as "mobile-install v1" or "native mobile-install" now. This functionality was deleted in Bazel 8, along with the built-in Android rules. +Now, all mobile-install functionality, as well as all Android build and test rules, are implemented in Starlark and reside in the `rules_android` GitHub repository. The latest version is known as "mobile-install v3" or "MIv3". +*Naming note*: There was a "mobile-install **v2**" available only internally at Google at one point, but this was never published externally, and only v3 continues to be used for both Google-internal and OSS rules\_android deployment. diff --git a/docs/sandboxing.mdx b/docs/sandboxing.mdx index 68697953..b12027ec 100644 --- a/docs/sandboxing.mdx +++ b/docs/sandboxing.mdx @@ -2,119 +2,49 @@ title: 'Sandboxing' --- +This article covers sandboxing in Bazel and debugging your sandboxing environment. +*Sandboxing* is a permission restricting strategy that isolates processes from each other or from resources in a system. For Bazel, this means restricting file system access. -This article covers sandboxing in Bazel and debugging your sandboxing -environment. +Bazel's file system sandbox runs processes in a working directory that only contains known inputs, such that compilers and other tools don't see source files they should not access, unless they know the absolute paths to them. -*Sandboxing* is a permission restricting strategy that isolates processes from -each other or from resources in a system. For Bazel, this means restricting file -system access. +Sandboxing doesn't hide the host environment in any way. Processes can freely access all files on the file system. However, on platforms that support user namespaces, processes can't modify any files outside their working directory. This ensures that the build graph doesn't have hidden dependencies that could affect the reproducibility of the build. -Bazel's file system sandbox runs processes in a working directory that only -contains known inputs, such that compilers and other tools don't see source -files they should not access, unless they know the absolute paths to them. - -Sandboxing doesn't hide the host environment in any way. Processes can freely -access all files on the file system. However, on platforms that support user -namespaces, processes can't modify any files outside their working directory. -This ensures that the build graph doesn't have hidden dependencies that could -affect the reproducibility of the build. - -More specifically, Bazel constructs an `execroot/` directory for each action, -which acts as the action's work directory at execution time. `execroot/` -contains all input files to the action and serves as the container for any -generated outputs. Bazel then uses an operating-system-provided technique, -containers on Linux and `sandbox-exec` on macOS, to constrain the action within -`execroot/`. +More specifically, Bazel constructs an `execroot/` directory for each action, which acts as the action's work directory at execution time. `execroot/` contains all input files to the action and serves as the container for any generated outputs. Bazel then uses an operating-system-provided technique, containers on Linux and `sandbox-exec` on macOS, to constrain the action within `execroot/`. ## Reasons for sandboxing -- Without action sandboxing, Bazel doesn't know if a tool uses undeclared - input files (files that are not explicitly listed in the dependencies of an - action). When one of the undeclared input files changes, Bazel still - believes that the build is up-to-date and won’t rebuild the action. This can - result in an incorrect incremental build. +- Without action sandboxing, Bazel doesn't know if a tool uses undeclared input files (files that are not explicitly listed in the dependencies of an action). When one of the undeclared input files changes, Bazel still believes that the build is up-to-date and won’t rebuild the action. This can result in an incorrect incremental build. -- Incorrect reuse of cache entries creates problems during remote caching. A - bad cache entry in a shared cache affects every developer on the project, - and wiping the entire remote cache is not a feasible solution. +- Incorrect reuse of cache entries creates problems during remote caching. A bad cache entry in a shared cache affects every developer on the project, and wiping the entire remote cache is not a feasible solution. -- Sandboxing mimics the behavior of remote execution — if a build works well - with sandboxing, it will likely also work with remote execution. By making - remote execution upload all necessary files (including local tools), you can - significantly reduce maintenance costs for compile clusters compared to - having to install the tools on every machine in the cluster every time you - want to try out a new compiler or make a change to an existing tool. +- Sandboxing mimics the behavior of remote execution — if a build works well with sandboxing, it will likely also work with remote execution. By making remote execution upload all necessary files (including local tools), you can significantly reduce maintenance costs for compile clusters compared to having to install the tools on every machine in the cluster every time you want to try out a new compiler or make a change to an existing tool. ## What sandbox strategy to use -You can choose which kind of sandboxing to use, if any, with the -[strategy flags](user-manual.html#strategy-options). Using the `sandboxed` -strategy makes Bazel pick one of the sandbox implementations listed below, -preferring an OS-specific sandbox to the less hermetic generic one. -[Persistent workers](/remote/persistent) run in a generic sandbox if you pass -the `--worker_sandboxing` flag. - -The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. -It simply executes the action's command line with the working directory set to -the execroot of your workspace. - -`processwrapper-sandbox` is a sandboxing strategy that does not require any -"advanced" features - it should work on any POSIX system out of the box. It -builds a sandbox directory consisting of symlinks that point to the original -source files, executes the action's command line with the working directory set -to this directory instead of the execroot, then moves the known output artifacts -out of the sandbox into the execroot and deletes the sandbox. This prevents the -action from accidentally using any input files that are not declared and from -littering the execroot with unknown output files. - -`linux-sandbox` goes one step further and builds on top of the -`processwrapper-sandbox`. Similar to what Docker does under the hood, it uses -Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the -action from the host. That is, it makes the entire filesystem read-only except -for the sandbox directory, so the action cannot accidentally modify anything on -the host filesystem. This prevents situations like a buggy test accidentally rm --rf'ing your $HOME directory. Optionally, you can also prevent the action from -accessing the network. `linux-sandbox` uses PID namespaces to prevent the action -from seeing any other processes and to reliably kill all processes (even daemons -spawned by the action) at the end. - -`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool -to achieve roughly the same as the Linux sandbox. - -Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" -scenario due to restrictions in the mechanisms provided by the operating -systems. Because Docker also uses Linux namespaces for its container magic, you -cannot easily run `linux-sandbox` inside a Docker container, unless you use -`docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a -process that's already being sandboxed. Thus, in these cases, Bazel -automatically falls back to using `processwrapper-sandbox`. - -If you would rather get a build error — such as to not accidentally build with a -less strict execution strategy — explicitly modify the list of execution -strategies that Bazel tries to use (for example, `bazel build ---spawn_strategy=worker,linux-sandbox`). - -Dynamic execution usually requires sandboxing for local execution. To opt out, -pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently -sandboxes [persistent workers](/remote/persistent). +You can choose which kind of sandboxing to use, if any, with the [strategy flags](user-manual.html#strategy-options). Using the `sandboxed` strategy makes Bazel pick one of the sandbox implementations listed below, preferring an OS-specific sandbox to the less hermetic generic one. [Persistent workers](/remote/persistent) run in a generic sandbox if you pass the `--worker_sandboxing` flag. + +The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. It simply executes the action's command line with the working directory set to the execroot of your workspace. + +`processwrapper-sandbox` is a sandboxing strategy that does not require any "advanced" features - it should work on any POSIX system out of the box. It builds a sandbox directory consisting of symlinks that point to the original source files, executes the action's command line with the working directory set to this directory instead of the execroot, then moves the known output artifacts out of the sandbox into the execroot and deletes the sandbox. This prevents the action from accidentally using any input files that are not declared and from littering the execroot with unknown output files. + +`linux-sandbox` goes one step further and builds on top of the `processwrapper-sandbox`. Similar to what Docker does under the hood, it uses Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the action from the host. That is, it makes the entire filesystem read-only except for the sandbox directory, so the action cannot accidentally modify anything on the host filesystem. This prevents situations like a buggy test accidentally rm -rf'ing your $HOME directory. Optionally, you can also prevent the action from accessing the network. `linux-sandbox` uses PID namespaces to prevent the action from seeing any other processes and to reliably kill all processes (even daemons spawned by the action) at the end. + +`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool to achieve roughly the same as the Linux sandbox. + +Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" scenario due to restrictions in the mechanisms provided by the operating systems. Because Docker also uses Linux namespaces for its container magic, you cannot easily run `linux-sandbox` inside a Docker container, unless you use `docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a process that's already being sandboxed. Thus, in these cases, Bazel automatically falls back to using `processwrapper-sandbox`. + +If you would rather get a build error — such as to not accidentally build with a less strict execution strategy — explicitly modify the list of execution strategies that Bazel tries to use (for example, `bazel build --spawn_strategy=worker,linux-sandbox`). + +Dynamic execution usually requires sandboxing for local execution. To opt out, pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently sandboxes [persistent workers](/remote/persistent). ## Downsides to sandboxing -- Sandboxing incurs extra setup and teardown cost. How big this cost is - depends on many factors, including the shape of the build and the - performance of the host OS. For Linux, sandboxed builds are rarely more than - a few percent slower. Setting `--reuse_sandbox_directories` can - mitigate the setup and teardown cost. +- Sandboxing incurs extra setup and teardown cost. How big this cost is depends on many factors, including the shape of the build and the performance of the host OS. For Linux, sandboxed builds are rarely more than a few percent slower. Setting `--reuse_sandbox_directories` can mitigate the setup and teardown cost. -- Sandboxing effectively disables any cache the tool may have. You can - mitigate this by using [persistent workers](/remote/persistent), at - the cost of weaker sandbox guarantees. +- Sandboxing effectively disables any cache the tool may have. You can mitigate this by using [persistent workers](/remote/persistent), at the cost of weaker sandbox guarantees. -- [Multiplex workers](/remote/multiplex) require explicit worker support - to be sandboxed. Workers that do not support multiplex sandboxing run as - singleplex workers under dynamic execution, which can cost extra memory. +- [Multiplex workers](/remote/multiplex) require explicit worker support to be sandboxed. Workers that do not support multiplex sandboxing run as singleplex workers under dynamic execution, which can cost extra memory. ## Debugging @@ -122,11 +52,7 @@ Follow the strategies below to debug issues with sandboxing. ### Deactivated namespaces -On some platforms, such as -[Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) -cluster nodes or Debian, user namespaces are deactivated by default due to -security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file -exists and contains a 0, you can activate user namespaces by running: +On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file exists and contains a 0, you can activate user namespaces by running: ```posix-terminal sudo sysctl kernel.unprivileged_userns_clone=1 @@ -134,16 +60,11 @@ exists and contains a 0, you can activate user namespaces by running: ### Rule execution failures -The sandbox may fail to execute rules because of the system setup. If you see a -message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or -directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for -genrules, and `--spawn_strategy=local` for other rules. +The sandbox may fail to execute rules because of the system setup. If you see a message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for genrules, and `--spawn_strategy=local` for other rules. ### Detailed debugging for build failures -If your build failed, use `--verbose_failures` and `--sandbox_debug` to make -Bazel show the exact command it ran when your build failed, including the part -that sets up the sandbox. +If your build failed, use `--verbose_failures` and `--sandbox_debug` to make Bazel show the exact command it ran when your build failed, including the part that sets up the sandbox. Example error message: @@ -166,9 +87,6 @@ namespace-sandbox failed: error executing command /some/path/to/your/some-compiler --some-params some-target) ``` -You can now inspect the generated sandbox directory and see which files Bazel -created and run the command again to see how it behaves. +You can now inspect the generated sandbox directory and see which files Bazel created and run the command again to see how it behaves. -Note that Bazel does not delete the sandbox directory when you use -`--sandbox_debug`. Unless you are actively debugging, you should disable -`--sandbox_debug` because it fills up your disk over time. +Note that Bazel does not delete the sandbox directory when you use `--sandbox_debug`. Unless you are actively debugging, you should disable `--sandbox_debug` because it fills up your disk over time. diff --git a/docs/user-manual.mdx b/docs/user-manual.mdx new file mode 100644 index 00000000..cbe48b01 --- /dev/null +++ b/docs/user-manual.mdx @@ -0,0 +1,1443 @@ +--- +title: 'Commands and Options' +--- + +This page covers the options that are available with various Bazel commands, such as `bazel build`, `bazel run`, and `bazel test`. This page is a companion to the list of Bazel's commands in [Build with Bazel](/run/build). + +## Target syntax + +Some commands, like `build` or `test`, can operate on a list of targets. They use a syntax more flexible than labels, which is documented in [Specifying targets to build](/run/build#specifying-build-targets). + +## Options + +The following sections describe the options available during a build. When `--long` is used on a help command, the on-line help messages provide summary information about the meaning, type and default value for each option. + +Most options can only be specified once. When specified multiple times, the last instance wins. Options that can be specified multiple times are identified in the on-line help with the text 'may be used multiple times'. + +### Package location + +#### `--package_path` + +**WARNING:** The `--package_path` option is deprecated. Bazel prefers packages in the main repository to be under the workspace root. + +This option specifies the set of directories that are searched to find the BUILD file for a given package. + +Bazel finds its packages by searching the package path. This is a colon separated ordered list of bazel directories, each being the root of a partial source tree. + +*To specify a custom package path* using the `--package_path` option: + +``` + % bazel build --package_path %workspace%:/some/other/root +``` + +Package path elements may be specified in three formats: + +1. If the first character is `/`, the path is absolute. +2. If the path starts with `%workspace%`, the path is taken relative to the nearest enclosing bazel directory. For instance, if your working directory is `/home/bob/clients/bob_client/bazel/foo`, then the string `%workspace%` in the package-path is expanded to `/home/bob/clients/bob_client/bazel`. +3. Anything else is taken relative to the working directory. This is usually not what you mean to do, and may behave unexpectedly if you use Bazel from directories below the bazel workspace. For instance, if you use the package-path element `.`, and then cd into the directory `/home/bob/clients/bob_client/bazel/foo`, packages will be resolved from the `/home/bob/clients/bob_client/bazel/foo` directory. + +If you use a non-default package path, specify it in your [Bazel configuration file](/run/bazelrc) for convenience. + +*Bazel doesn't require any packages to be in the current directory*, so you can do a build from an empty bazel workspace if all the necessary packages can be found somewhere else on the package path. + +Example: Building from an empty client + +``` + % mkdir -p foo/bazel + % cd foo/bazel + % touch MODULE.bazel + % bazel build --package_path /some/other/path //foo +``` + +#### `--deleted_packages` + +This option specifies a comma-separated list of packages which Bazel should consider deleted, and not attempt to load from any directory on the package path. This can be used to simulate the deletion of packages without actually deleting them. This option can be passed multiple times, in which case the individual lists are concatenated. + +### Error checking + +These options control Bazel's error-checking and/or warnings. + +#### `--[no]check_visibility` + +If this option is set to false, visibility checks are demoted to warnings. The default value of this option is true, so that by default, visibility checking is done. + +#### `--output_filter=<var>regex</var>` + +The `--output_filter` option will only show build and compilation warnings for targets that match the regular expression. If a target does not match the given regular expression and its execution succeeds, its standard output and standard error are thrown away. + +Here are some typical values for this option: + +| | | +| ------------------------------------------------------------------------------ | --------------------------------------------- | +| \`--output\_filter='^//(first/project\|second/project):'\` | Show the output for the specified packages. | +| \`--output\_filter='^//((?!(first/bad\_project\|second/bad\_project):).)\*$'\` | Don't show output for the specified packages. | +| \`--output\_filter=\` | Show everything. | +| \`--output\_filter=DONT\_MATCH\_ANYTHING\` | Show nothing. | + +### Tool flags + +These options control which options Bazel will pass to other tools. + +#### `--copt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler. The argument will be passed to the compiler whenever it is invoked for preprocessing, compiling, and/or assembling C, C++, or assembler code. It will not be passed when linking. + +This option can be used multiple times. For example: + +``` + % bazel build --copt="-g0" --copt="-fpic" //foo +``` + +will compile the `foo` library without debug tables, generating position-independent code. + +Note: Changing `--copt` settings will force a recompilation of all affected object files. Also note that copts values listed in specific cc\_library or cc\_binary build rules will be placed on the compiler command line *after* these options. + +Warning: C++-specific options (such as `-fno-implicit-templates`) should be specified in `--cxxopt`, not in `--copt`. Likewise, C-specific options (such as -Wstrict-prototypes) should be specified in `--conlyopt`, not in `copt`. Similarly, compiler options that only have an effect at link time (such as `-l`) should be specified in `--linkopt`, not in `--copt`. + +#### `--host_copt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler for source files that are compiled in the exec configuration. This is analogous to the [`--copt`](#copt) option, but applies only to the exec configuration. + +#### `--host_conlyopt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler for C source files that are compiled in the exec configuration. This is analogous to the [`--conlyopt`](#cconlyopt) option, but applies only to the exec configuration. + +#### `--host_cxxopt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler for C++ source files that are compiled in the exec configuration. This is analogous to the [`--cxxopt`](#cxxopt) option, but applies only to the exec configuration. + +#### `--host_linkopt=<var>linker-option</var>` + +This option takes an argument which is to be passed to the linker for source files that are compiled in the exec configuration. This is analogous to the [`--linkopt`](#linkopt) option, but applies only to the exec configuration. + +#### `--conlyopt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler when compiling C source files. + +This is similar to `--copt`, but only applies to C compilation, not to C++ compilation or linking. So you can pass C-specific options (such as `-Wno-pointer-sign`) using `--conlyopt`. + +Note: copts parameters listed in specific cc\_library or cc\_binary build rules are placed on the compiler command line *after* these options. + +#### `--cxxopt=<var>cc-option</var>` + +This option takes an argument which is to be passed to the compiler when compiling C++ source files. + +This is similar to `--copt`, but only applies to C++ compilation, not to C compilation or linking. So you can pass C++-specific options (such as `-fpermissive` or `-fno-implicit-templates`) using `--cxxopt`. + +For example: + +``` + % bazel build --cxxopt="-fpermissive" --cxxopt="-Wno-error" //foo/cruddy_code +``` + +Note: copts parameters listed in specific cc\_library or cc\_binary build rules are placed on the compiler command line *after* these options. + +#### `--linkopt=<var>linker-option</var>` + +This option takes an argument which is to be passed to the compiler when linking. + +This is similar to `--copt`, but only applies to linking, not to compilation. So you can pass compiler options that only make sense at link time (such as `-lssp` or `-Wl,--wrap,abort`) using `--linkopt`. For example: + +``` + % bazel build --copt="-fmudflap" --linkopt="-lmudflap" //foo/buggy_code +``` + +Build rules can also specify link options in their attributes. This option's settings always take precedence. Also see [cc\_library.linkopts](/reference/be/c-cpp#cc_library.linkopts). + +#### `--strip (always|never|sometimes)` + +This option determines whether Bazel will strip debugging information from all binaries and shared libraries, by invoking the linker with the `-Wl,--strip-debug` option. `--strip=always` means always strip debugging information. `--strip=never` means never strip debugging information. The default value of `--strip=sometimes` means strip if the `--compilation_mode` is `fastbuild`. + +``` + % bazel build --strip=always //foo:bar +``` + +will compile the target while stripping debugging information from all generated binaries. + +Note: If you want debugging information, it's not enough to disable stripping; you also need to make sure that the debugging information was generated by the compiler, which you can do by using either `-c dbg` or `--copt -g`. + +Bazel's `--strip` option corresponds with ld's `--strip-debug` option: it only strips debugging information. If for some reason you want to strip *all* symbols, not just *debug* symbols, you would need to use ld's `--strip-all` option, which you can do by passing `--linkopt=-Wl,--strip-all` to Bazel. Also be aware that setting Bazel's `--strip` flag will override `--linkopt=-Wl,--strip-all`, so you should only set one or the other. + +If you are only building a single binary and want all symbols stripped, you could also pass `--stripopt=--strip-all` and explicitly build the `//foo:bar.stripped` version of the target. As described in the section on `--stripopt`, this applies a strip action after the final binary is linked rather than including stripping in all of the build's link actions. + +#### `--stripopt=<var>strip-option</var>` + +This is an additional option to pass to the `strip` command when generating a [`*.stripped` binary](/reference/be/c-cpp#cc_binary_implicit_outputs). The default is `-S -p`. This option can be used multiple times. + +Note: `--stripopt` does not apply to the stripping of the main binary with `[--strip](#flag--strip)=(always|sometimes)`. + +#### `--fdo_instrument=<var>profile-output-dir</var>` + +The `--fdo_instrument` option enables the generation of FDO (feedback directed optimization) profile output when the built C/C++ binary is executed. For GCC, the argument provided is used as a directory prefix for a per-object file directory tree of .gcda files containing profile information for each .o file. + +Once the profile data tree has been generated, the profile tree should be zipped up, and provided to the `--fdo_optimize=<var>profile-zip</var>` Bazel option to enable the FDO-optimized compilation. + +For the LLVM compiler the argument is also the directory under which the raw LLVM profile data file(s) is dumped. For example: `--fdo_instrument=<var>/path/to/rawprof/dir/</var>`. + +The options `--fdo_instrument` and `--fdo_optimize` cannot be used at the same time. + +#### `--fdo_optimize=<var>profile-zip</var>` + +The `--fdo_optimize` option enables the use of the per-object file profile information to perform FDO (feedback directed optimization) optimizations when compiling. For GCC, the argument provided is the zip file containing the previously-generated file tree of .gcda files containing profile information for each .o file. + +Alternatively, the argument provided can point to an auto profile identified by the extension .afdo. + +Note: This option also accepts labels that resolve to source files. You may need to add an `exports_files` directive to the corresponding package to make the file visible to Bazel. + +For the LLVM compiler the argument provided should point to the indexed LLVM profile output file prepared by the llvm-profdata tool, and should have a .profdata extension. + +The options `--fdo_instrument` and `--fdo_optimize` cannot be used at the same time. + +#### `--java_language_version=<var>version</var>` + +This option specifies the version of Java sources. For example: + +``` + % bazel build --java_language_version=8 java/com/example/common/foo:all +``` + +compiles and allows only constructs compatible with Java 8 specification. Default value is 11. --> Possible values are: 8, 9, 10, 11, 17, and 21 and may be extended by registering custom Java toolchains using `default_java_toolchain`. + +#### `--tool_java_language_version=<var>version</var>` + +The Java language version used to build tools that are executed during a build. Default value is 11. + +#### `--java_runtime_version=<var>version</var>` + +This option specifies the version of JVM to use to execute the code and run the tests. For example: + +``` + % bazel run --java_runtime_version=remotejdk_11 java/com/example/common/foo:java_application +``` + +downloads JDK 11 from a remote repository and run the Java application using it. + +Default value is `local_jdk`. Possible values are: `local_jdk`, `local_jdk_<var>version</var>`, `remotejdk_11`, `remotejdk_17`, and `remotejdk_21`. You can extend the values by registering custom JVM using either `local_java_repository` or `remote_java_repository` repository rules. + +#### `--tool_java_runtime_version=<var>version</var>` + +The version of JVM used to execute tools that are needed during a build. Default value is `remotejdk_11`. + +#### `--jvmopt=<var>jvm-option</var>` + +This option allows option arguments to be passed to the Java VM. It can be used with one big argument, or multiple times with individual arguments. For example: + +``` + % bazel build --jvmopt="-server -Xms256m" java/com/example/common/foo:all +``` + +will use the server VM for launching all Java binaries and set the startup heap size for the VM to 256 MB. + +#### `--javacopt=<var>javac-option</var>` + +This option allows option arguments to be passed to javac. It can be used with one big argument, or multiple times with individual arguments. For example: + +``` + % bazel build --javacopt="-g:source,lines" //myprojects:prog +``` + +will rebuild a java\_binary with the javac default debug info (instead of the bazel default). + +The option is passed to javac after the Bazel built-in default options for javac and before the per-rule options. The last specification of any option to javac wins. The default options for javac are: + +``` + -source 8 -target 8 -encoding UTF-8 +``` + +Note: Changing `--javacopt` settings will force a recompilation of all affected classes. Also note that javacopts parameters listed in specific java\_library or java\_binary build rules will be placed on the javac command line *after* these options. + +#### `--strict_java_deps (default|strict|off|warn|error)` + +This option controls whether javac checks for missing direct dependencies. Java targets must explicitly declare all directly used targets as dependencies. This flag instructs javac to determine the jars actually used for type checking each java file, and warn/error if they are not the output of a direct dependency of the current target. + +- `off` means checking is disabled. +- `warn` means javac will generate standard java warnings of type `[strict]` for each missing direct dependency. +- `default`, `strict` and `error` all mean javac will generate errors instead of warnings, causing the current target to fail to build if any missing direct dependencies are found. This is also the default behavior when the flag is unspecified. + +### Build semantics + +These options affect the build commands and/or the output file contents. + +#### `--compilation_mode (fastbuild|opt|dbg)` (-c) + +The `--compilation_mode` option (often shortened to `-c`, especially `-c opt`) takes an argument of `fastbuild`, `dbg` or `opt`, and affects various C/C++ code-generation options, such as the level of optimization and the completeness of debug tables. Bazel uses a different output directory for each different compilation mode, so you can switch between modes without needing to do a full rebuild *every* time. + +- `fastbuild` means build as fast as possible: generate minimal debugging information (`-gmlt -Wl,-S`), and don't optimize. This is the default. Note: `-DNDEBUG` will **not** be set. +- `dbg` means build with debugging enabled (`-g`), so that you can use gdb (or another debugger). +- `opt` means build with optimization enabled and with `assert()` calls disabled (`-O2 -DNDEBUG`). Debugging information will not be generated in `opt` mode unless you also pass `--copt -g`. + +#### `--cpu=<var>cpu</var>` + +This option specifies the target CPU architecture to be used for the compilation of binaries during the build. + +Note: A particular combination of crosstool version, compiler version, and target CPU is allowed only if it has been specified in the currently used CROSSTOOL file. + +#### `--action_env=<var>VAR=VALUE</var>` + +Specifies the set of environment variables available during the execution of all actions. Variables can be either specified by name, in which case the value will be taken from the invocation environment, or by the `name=value` pair which sets the value independent of the invocation environment. + +This `--action_env` flag can be specified multiple times. If a value is assigned to the same variable across multiple `--action_env` flags, the latest assignment wins. + +#### `--experimental_action_listener=<var>label</var>` + +Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. + +The `experimental_action_listener` option instructs Bazel to use details from the [`action_listener`](/reference/be/extra-actions#action_listener) rule specified by \{\{ "`" }}label{{ "`" \}\} to insert [`extra_actions`](/reference/be/extra-actions#extra_action) into the build graph. + +#### `--[no]experimental_extra_action_top_level_only` + +Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. + +If this option is set to true, extra actions specified by the [`--experimental_action_listener`](#experimental-action-listener) command line option will only be scheduled for top level targets. + +#### `--experimental_extra_action_filter=<var>regex</var>` + +Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. + +The `experimental_extra_action_filter` option instructs Bazel to filter the set of targets to schedule `extra_actions` for. + +This flag is only applicable in combination with the [`--experimental_action_listener`](#experimental-action-listener) flag. + +By default all `extra_actions` in the transitive closure of the requested targets-to-build get scheduled for execution. `--experimental_extra_action_filter` will restrict scheduling to `extra_actions` of which the owner's label matches the specified regular expression. + +The following example will limit scheduling of `extra_actions` to only apply to actions of which the owner's label contains '/bar/': + +``` +% bazel build --experimental_action_listener=//test:al //foo/... \ + --experimental_extra_action_filter=.*/bar/.* +``` + +#### `--host_cpu=<var>cpu</var>` + +This option specifies the name of the CPU architecture that should be used to build host tools. + +#### `--android_platforms=<var>platform[,platform]*</var>` + +The platforms to build the transitive `deps` of `android_binary` rules (specifically for native dependencies like C++). For example, if a `cc_library` appears in the transitive `deps` of an `android_binary` rule it is be built once for each platform specified with `--android_platforms` for the `android_binary` rule, and included in the final output. + +There is no default value for this flag: a custom Android platform must be defined and used. + +One `.so` file is created and packaged in the APK for each platform specified with `--android_platforms`. The `.so` file's name prefixes the name of the `android_binary` rule with "lib". For example, if the name of the `android_binary` is "foo", then the file is `libfoo.so`. + +#### `--per_file_copt=<var>[+-]regex[,[+-]regex]...@option[,option]...</var>` + +When present, any C++ file with a label or an execution path matching one of the inclusion regex expressions and not matching any of the exclusion expressions will be built with the given options. The label matching uses the canonical form of the label (i.e //`package`:`label_name`). + +The execution path is the relative path to your workspace directory including the base name (including extension) of the C++ file. It also includes any platform dependent prefixes. + +Note: If only one of the label or the execution path matches the options will be used. + +To match the generated files (such as genrule outputs) Bazel can only use the execution path. In this case the regexp shouldn't start with '//' since that doesn't match any execution paths. Package names can be used like this: `--per_file_copt=base/.*\.pb\.cc@-g0`. This will match every `.pb.cc` file under a directory called `base`. + +This option can be used multiple times. + +The option is applied regardless of the compilation mode used. For example, it is possible to compile with `--compilation_mode=opt` and selectively compile some files with stronger optimization turned on, or with optimization disabled. + +**Caveat**: If some files are selectively compiled with debug symbols the symbols might be stripped during linking. This can be prevented by setting `--strip=never`. + +**Syntax**: `[+-]regex[,[+-]regex]...@option[,option]...` Where `regex` stands for a regular expression that can be prefixed with a `+` to identify include patterns and with `-` to identify exclude patterns. `option` stands for an arbitrary option that is passed to the C++ compiler. If an option contains a `,` it has to be quoted like so `\,`. Options can also contain `@`, since only the first `@` is used to separate regular expressions from options. + +**Example**: `--per_file_copt=//foo:.*\.cc,-//foo:file\.cc@-O0,-fprofile-arcs` adds the `-O0` and the `-fprofile-arcs` options to the command line of the C++ compiler for all `.cc` files in `//foo/` except `file.cc`. + +#### `--dynamic_mode=<var>mode</var>` + +Determines whether C++ binaries will be linked dynamically, interacting with the [linkstatic attribute](/reference/be/c-cpp#cc_binary.linkstatic) on build rules. + +Modes: + +- `default`: Allows bazel to choose whether to link dynamically. See [linkstatic](/reference/be/c-cpp#cc_binary.linkstatic) for more information. +- `fully`: Links all targets dynamically. This will speed up linking time, and reduce the size of the resulting binaries. +- `off`: Links all targets in [mostly static](/reference/be/c-cpp#cc_binary.linkstatic) mode. If `-static` is set in linkopts, targets will change to fully static. + +#### `--fission (yes|no|[dbg][,opt][,fastbuild])` + +Enables [Fission](https://gcc.gnu.org/wiki/DebugFission), which writes C++ debug information to dedicated .dwo files instead of .o files, where it would otherwise go. This substantially reduces the input size to links and can reduce link times. + +When set to `[dbg][,opt][,fastbuild]` (example: `--fission=dbg,fastbuild`), Fission is enabled only for the specified set of compilation modes. This is useful for bazelrc settings. When set to `yes`, Fission is enabled universally. When set to `no`, Fission is disabled universally. Default is `no`. + +#### `--force_ignore_dash_static` + +If this flag is set, any `-static` options in linkopts of `cc_*` rules BUILD files are ignored. This is only intended as a workaround for C++ hardening builds. + +#### `--[no]force_pic` + +If enabled, all C++ compilations produce position-independent code ("-fPIC"), links prefer PIC pre-built libraries over non-PIC libraries, and links produce position-independent executables ("-pie"). Default is disabled. + +Note: Dynamically linked binaries (for example `--dynamic_mode fully`) generate PIC code regardless of this flag's setting. So this flag is for cases where users want PIC code explicitly generated for static links. + +#### `--android_resource_shrinking` + +Selects whether to perform resource shrinking for android\_binary rules. Sets the default for the [shrink\_resources attribute](/reference/be/android#android_binary.shrink_resources) on android\_binary rules; see the documentation for that rule for further details. Defaults to off. + +#### `--custom_malloc=<var>malloc-library-target</var>` + +When specified, always use the given malloc implementation, overriding all `malloc="target"` attributes, including in those targets that use the default (by not specifying any `malloc`). + +#### `--crosstool_top=<var>label</var>` + +This option specifies the location of the crosstool compiler suite to be used for all C++ compilation during a build. Bazel will look in that location for a CROSSTOOL file and uses that to automatically determine settings for `--compiler`. + +#### `--host_crosstool_top=<var>label</var>` + +If not specified, Bazel uses the value of `--crosstool_top` to compile code in the exec configuration, such as tools run during the build. The main purpose of this flag is to enable cross-compilation. + +#### `--apple_crosstool_top=<var>label</var>` + +The crosstool to use for compiling C/C++ rules in the transitive `deps` of objc\_*, ios\_\_*, and apple\_\* rules. For those targets, this flag overwrites `--crosstool_top`. + +#### `--compiler=<var>version</var>` + +This option specifies the C/C++ compiler version (such as `gcc-4.1.0`) to be used for the compilation of binaries during the build. If you want to build with a custom crosstool, you should use a CROSSTOOL file instead of specifying this flag. + +Note: Only certain combinations of crosstool version, compiler version, and target CPU are allowed. + +#### `--android_sdk=<var>label</var>` + +Deprecated. This shouldn't be directly specified. + +This option specifies the Android SDK/platform toolchain and Android runtime library that will be used to build any Android-related rule. + +The Android SDK will be automatically selected if an `android_sdk_repository` rule is defined in the WORKSPACE file. + +#### `--java_toolchain=<var>label</var>` + +No-op. Kept only for backwards compatibility. + +#### `--host_java_toolchain=<var>label</var>` + +No-op. Kept only for backwards compatibility. + +#### `--javabase=(<var>label</var>)` + +No-op. Kept only for backwards compatibility. + +#### `--host_javabase=<var>label</var>` + +No-op. Kept only for backwards compatibility. + +### Execution strategy + +These options affect how Bazel will execute the build. They should not have any significant effect on the output files generated by the build. Typically their main effect is on the speed of the build. + +#### `--spawn_strategy=<var>strategy</var>` + +This option controls where and how commands are executed. + +- `standalone` causes commands to be executed as local subprocesses. This value is deprecated. Please use `local` instead. +- `sandboxed` causes commands to be executed inside a sandbox on the local machine. This requires that all input files, data dependencies and tools are listed as direct dependencies in the `srcs`, `data` and `tools` attributes. Bazel enables local sandboxing by default, on systems that support sandboxed execution. +- `local` causes commands to be executed as local subprocesses. +- `worker` causes commands to be executed using a persistent worker, if available. +- `docker` causes commands to be executed inside a docker sandbox on the local machine. This requires that docker is installed. +- `remote` causes commands to be executed remotely; this is only available if a remote executor has been configured separately. + +#### `--strategy <var>mnemonic</var>=<var>strategy</var>` + +This option controls where and how commands are executed, overriding the [--spawn\_strategy](#spawn-strategy) (and [--genrule\_strategy](#genrule-strategy) with mnemonic Genrule) on a per-mnemonic basis. See [--spawn\_strategy](#spawn-strategy) for the supported strategies and their effects. + +#### `--strategy_regexp=<var><filter,filter,...>=<strategy></var>` + +This option specifies which strategy should be used to execute commands that have descriptions matching a certain `regex_filter`. See [--per\_file\_copt](#per-file-copt) for details on regex\_filter matching. See [--spawn\_strategy](#spawn-strategy) for the supported strategies and their effects. + +The last `regex_filter` that matches the description is used. This option overrides other flags for specifying strategy. + +- Example: `--strategy_regexp=//foo.*\\.cc,-//foo/bar=local` means to run actions using `local` strategy if their descriptions match //foo.\*.cc but not //foo/bar. +- Example: `--strategy_regexp='Compiling.*/bar=local' --strategy_regexp=Compiling=sandboxed` runs 'Compiling //foo/bar/baz' with the `sandboxed` strategy, but reversing the order runs it with `local`. +- Example: `--strategy_regexp='Compiling.*/bar=local,sandboxed'` runs 'Compiling //foo/bar/baz' with the `local` strategy and falls back to `sandboxed` if it fails. + +#### `--genrule_strategy=<var>strategy</var>` + +This is a deprecated short-hand for `--strategy=Genrule=<var>strategy</var>`. + +#### `--jobs=<var>n</var>` (-j) + +This option, which takes an integer argument, specifies a limit on the number of jobs that should be executed concurrently during the execution phase of the build. + +Note : The number of concurrent jobs that Bazel will run is determined not only by the `--jobs` setting, but also by Bazel's scheduler, which tries to avoid running concurrent jobs that will use up more resources (RAM or CPU) than are available, based on some (very crude) estimates of the resource consumption of each job. The behavior of the scheduler can be controlled by the `--local_resources` option. + +#### `--progress_report_interval=<var>n</var>` + +Bazel periodically prints a progress report on jobs that are not finished yet (such as long running tests). This option sets the reporting frequency, progress will be printed every `n` seconds. + +The default is 0, that means an incremental algorithm: the first report will be printed after 10 seconds, then 30 seconds and after that progress is reported once every minute. + +When bazel is using cursor control, as specified by [`--curses`](#curses), progress is reported every second. + +#### `--local_resources <var>resources or resource expression</var>` + +These options specify the amount of local resources (RAM in MB and number of CPU logical cores) that Bazel can take into consideration when scheduling build and test activities to run locally. They take an float, or a keyword (HOST\_RAM or HOST\_CPUS) optionally followed by `[-|*`float`]` (for example, `--local_resources=cpu=2`, `--local_resources=memory=HOST_RAM*.5`, `--local_resources=cpu=HOST_CPUS-1`). The flags are independent; one or both may be set. By default, Bazel estimates the amount of RAM and number of CPU cores directly from the local system's configuration. + +#### `--[no]build_runfile_links` + +This option, which is enabled by default, specifies whether the runfiles symlinks for tests and binaries should be built in the output directory. Using `--nobuild_runfile_links` can be useful to validate if all targets compile without incurring the overhead for building the runfiles trees. + +When tests (or applications) are executed, their run-time data dependencies are gathered together in one place. Within Bazel's output tree, this "runfiles" tree is typically rooted as a sibling of the corresponding binary or test. During test execution, runfiles may be accessed using paths of the form `$TEST_SRCDIR/<var>canonical_repo_name</var>/<var>packagename</var>/<var>filename</var>`. The runfiles tree ensures that tests have access to all the files upon which they have a declared dependence, and nothing more. By default, the runfiles tree is implemented by constructing a set of symbolic links to the required files. As the set of links grows, so does the cost of this operation, and for some large builds it can contribute significantly to overall build time, particularly because each individual test (or application) requires its own runfiles tree. + +#### `--[no]build_runfile_manifests` + +This option, which is enabled by default, specifies whether runfiles manifests should be written to the output tree. Disabling it implies `--nobuild_runfile_links`. + +It can be disabled when executing tests remotely, as runfiles trees will be created remotely from in-memory manifests. + +#### `--[no]discard_analysis_cache` + +When this option is enabled, Bazel will discard the analysis cache right before execution starts, thus freeing up additional memory (around 10%) for the [execution phase](/run/build#execution). The drawback is that further incremental builds will be slower. See also [memory-saving mode](/configure/memory). + +#### `--[no]keep_going` (-k) + +As in GNU Make, the execution phase of a build stops when the first error is encountered. Sometimes it is useful to try to build as much as possible even in the face of errors. This option enables that behavior, and when it is specified, the build will attempt to build every target whose prerequisites were successfully built, but will ignore errors. + +While this option is usually associated with the execution phase of a build, it also affects the analysis phase: if several targets are specified in a build command, but only some of them can be successfully analyzed, the build will stop with an error unless `--keep_going` is specified, in which case the build will proceed to the execution phase, but only for the targets that were successfully analyzed. + +#### `--[no]use_ijars` + +This option changes the way `java_library` targets are compiled by Bazel. Instead of using the output of a `java_library` for compiling dependent `java_library` targets, Bazel will create interface jars that contain only the signatures of non-private members (public, protected, and default (package) access methods and fields) and use the interface jars to compile the dependent targets. This makes it possible to avoid recompilation when changes are only made to method bodies or private members of a class. + +Note: Using `--use_ijars` might give you a different error message when you are accidentally referring to a non visible member of another class: Instead of getting an error that the member is not visible you will get an error that the member does not exist. Changing the `--use_ijars` setting will force a recompilation of all affected classes. + +#### `--[no]interface_shared_objects` + +This option enables *interface shared objects*, which makes binaries and other shared libraries depend on the *interface* of a shared object, rather than its implementation. When only the implementation changes, Bazel can avoid rebuilding targets that depend on the changed shared library unnecessarily. + +### Output selection + +These options determine what to build or test. + +#### `--[no]build` + +This option causes the execution phase of the build to occur; it is on by default. When it is switched off, the execution phase is skipped, and only the first two phases, loading and analysis, occur. + +This option can be useful for validating BUILD files and detecting errors in the inputs, without actually building anything. + +#### `--[no]build_tests_only` + +If specified, Bazel will build only what is necessary to run the `*_test` and `test_suite` rules that were not filtered due to their [size](#test-size-filters), [timeout](#test-timeout-filters), [tag](#test-tag-filters), or [language](#test-lang-filters). If specified, Bazel will ignore other targets specified on the command line. By default, this option is disabled and Bazel will build everything requested, including `*_test` and `test_suite` rules that are filtered out from testing. This is useful because running `bazel test --build_tests_only foo/...` may not detect all build breakages in the `foo` tree. + +#### `--[no]check_up_to_date` + +This option causes Bazel not to perform a build, but merely check whether all specified targets are up-to-date. If so, the build completes successfully, as usual. However, if any files are out of date, instead of being built, an error is reported and the build fails. This option may be useful to determine whether a build has been performed more recently than a source edit (for example, for pre-submit checks) without incurring the cost of a build. + +See also [`--check_tests_up_to_date`](#check-tests-up-to-date). + +#### `--[no]compile_one_dependency` + +Compile a single dependency of the argument files. This is useful for syntax checking source files in IDEs, for example, by rebuilding a single target that depends on the source file to detect errors as early as possible in the edit/build/test cycle. This argument affects the way all non-flag arguments are interpreted: each argument must be a file target label or a plain filename relative to the current working directory, and one rule that depends on each source filename is built. For C++ and Java sources, rules in the same language space are preferentially chosen. For multiple rules with the same preference, the one that appears first in the BUILD file is chosen. An explicitly named target pattern which does not reference a source file results in an error. + +#### `--save_temps` + +The `--save_temps` option causes temporary outputs from the compiler to be saved. These include .s files (assembler code), .i (preprocessed C) and .ii (preprocessed C++) files. These outputs are often useful for debugging. Temps will only be generated for the set of targets specified on the command line. + +Note: The implementation of `--save_temps` does not use the compiler's `-save-temps` flag. Instead, there are two passes, one with `-S` and one with `-E`. A consequence of this is that if your build fails, Bazel may not yet have produced the ".i" or ".ii" and ".s" files. If you're trying to use `--save_temps` to debug a failed compilation, you may need to also use `--keep_going` so that Bazel will still try to produce the preprocessed files after the compilation fails. + +The `--save_temps` flag currently works only for cc\_\* rules. + +To ensure that Bazel prints the location of the additional output files, check that your [`--show_result <var>n</var>`](#show-result) setting is high enough. + +#### `--build_tag_filters=<var>tag[,tag]*</var>` + +If specified, Bazel will build only targets that have at least one required tag (if any of them are specified) and does not have any excluded tags. Build tag filter is specified as comma delimited list of tag keywords, optionally preceded with '-' sign used to denote excluded tags. Required tags may also have a preceding '+' sign. + +When running tests, Bazel ignores `--build_tag_filters` for test targets, which are built and run even if they do not match this filter. To avoid building them, filter test targets using `--test_tag_filters` or by explicitly excluding them. + +#### `--test_size_filters=<var>size[,size]*</var>` + +If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets with the given size. Test size filter is specified as comma delimited list of allowed test size values (small, medium, large or enormous), optionally preceded with '-' sign used to denote excluded test sizes. For example, + +``` + % bazel test --test_size_filters=small,medium //foo:all +``` + +and + +``` + % bazel test --test_size_filters=-large,-enormous //foo:all +``` + +will test only small and medium tests inside //foo. + +By default, test size filtering is not applied. + +#### `--test_timeout_filters=<var>timeout[,timeout]*</var>` + +If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets with the given timeout. Test timeout filter is specified as comma delimited list of allowed test timeout values (short, moderate, long or eternal), optionally preceded with '-' sign used to denote excluded test timeouts. See [--test\_size\_filters](#test-size-filters) for example syntax. + +By default, test timeout filtering is not applied. + +#### `--test_tag_filters=<var>tag[,tag]*</var>` + +If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets that have at least one required tag (if any of them are specified) and does not have any excluded tags. Test tag filter is specified as comma delimited list of tag keywords, optionally preceded with '-' sign used to denote excluded tags. Required tags may also have a preceding '+' sign. + +For example, + +``` + % bazel test --test_tag_filters=performance,stress,-flaky //myproject:all +``` + +will test targets that are tagged with either `performance` or `stress` tag but are **not** tagged with the `flaky` tag. + +By default, test tag filtering is not applied. Note that you can also filter on test's `size` and `local` tags in this manner. + +#### `--test_lang_filters=<var>string[,string]*</var>` + +Specifies a comma-separated list of strings referring to names of test rule classes. To refer to the rule class `foo_test`, use the string "foo". Bazel will test (or build if `--build_tests_only` is also specified) only targets of the referenced rule classes. To instead exclude those targets, use the string "-foo". For example, + +``` + % bazel test --test_lang_filters=foo,bar //baz/... +``` + +will test only targets that are instances of \`foo\_test\` or \`bar\_test\` in \`//baz/...\`, while + +``` + % bazel test --test_lang_filters=-foo,-bar //baz/... +``` + +will test all the targets in \`//baz/...\` except for the \`foo\_test\` and \`bar\_test\` instances. + +Tip: You can use `bazel query --output=label_kind "//p:t"` to learn the rule class name of the target `//p:t`. And you can look at the pair of instantiation stacks in the output of `bazel query --output=build "//p:t"` to learn why that target is an instance of that rule class. + +Warning: The option name "--test\_lang\_filter" is vestigal and is therefore unfortunately misleading; don't make assumptions about the semantics based on the name. + +#### `--test_filter=<var>filter-expression</var>` + +Specifies a filter that the test runner may use to pick a subset of tests for running. All targets specified in the invocation are built, but depending on the expression only some of them may be executed; in some cases, only certain test methods are run. + +The particular interpretation of \{\{ "`" }}filter-expression{{ "`" \}\} is up to the test framework responsible for running the test. It may be a glob, substring, or regexp. `--test_filter` is a convenience over passing different `--test_arg` filter arguments, but not all frameworks support it. + +### Verbosity + +These options control the verbosity of Bazel's output, either to the terminal, or to additional log files. + +#### `--explain=<var>logfile</var>` + +This option, which requires a filename argument, causes the dependency checker in `bazel build`'s execution phase to explain, for each build step, either why it is being executed, or that it is up-to-date. The explanation is written to *logfile*. + +If you are encountering unexpected rebuilds, this option can help to understand the reason. Add it to your `.bazelrc` so that logging occurs for all subsequent builds, and then inspect the log when you see an execution step executed unexpectedly. This option may carry a small performance penalty, so you might want to remove it when it is no longer needed. + +#### `--verbose_explanations` + +This option increases the verbosity of the explanations generated when the [--explain](#explain) option is enabled. + +In particular, if verbose explanations are enabled, and an output file is rebuilt because the command used to build it has changed, then the output in the explanation file will include the full details of the new command (at least for most commands). + +Using this option may significantly increase the length of the generated explanation file and the performance penalty of using `--explain`. + +If `--explain` is not enabled, then `--verbose_explanations` has no effect. + +#### `--profile=<var>file</var>` + +This option, which takes a filename argument, causes Bazel to write profiling data into a file. The data then can be analyzed or parsed using the `bazel analyze-profile` command. The Build profile can be useful in understanding where Bazel's `build` command is spending its time. + +#### `--[no]show_loading_progress` + +This option causes Bazel to output package-loading progress messages. If it is disabled, the messages won't be shown. + +#### `--[no]show_progress` + +This option causes progress messages to be displayed; it is on by default. When disabled, progress messages are suppressed. + +#### `--show_progress_rate_limit=<var>n</var>` + +This option causes bazel to display at most one progress message per `n` seconds, where \{\{ "`" }}n{{ "`" \}\} is a real number. The default value for this option is 0.02, meaning bazel will limit the progress messages to one per every 0.02 seconds. + +#### `--show_result=<var>n</var>` + +This option controls the printing of result information at the end of a `bazel build` command. By default, if a single build target was specified, Bazel prints a message stating whether or not the target was successfully brought up-to-date, and if so, the list of output files that the target created. If multiple targets were specified, result information is not displayed. + +While the result information may be useful for builds of a single target or a few targets, for large builds (such as an entire top-level project tree), this information can be overwhelming and distracting; this option allows it to be controlled. `--show_result` takes an integer argument, which is the maximum number of targets for which full result information should be printed. By default, the value is 1. Above this threshold, no result information is shown for individual targets. Thus zero causes the result information to be suppressed always, and a very large value causes the result to be printed always. + +Users may wish to choose a value in-between if they regularly alternate between building a small group of targets (for example, during the compile-edit-test cycle) and a large group of targets (for example, when establishing a new workspace or running regression tests). In the former case, the result information is very useful whereas in the latter case it is less so. As with all options, this can be specified implicitly via the [`.bazelrc`](/run/bazelrc) file. + +The files are printed so as to make it easy to copy and paste the filename to the shell, to run built executables. The "up-to-date" or "failed" messages for each target can be easily parsed by scripts which drive a build. + +#### `--sandbox_debug` + +This option causes Bazel to print extra debugging information when using sandboxing for action execution. This option also preserves sandbox directories, so that the files visible to actions during execution can be examined. + +#### `--subcommands` (`-s`) + +This option causes Bazel's execution phase to print the full command line for each command prior to executing it. + +``` + >>>>> # //examples/cpp:hello-world [action 'Linking examples/cpp/hello-world'] + (cd /home/johndoe/.cache/bazel/_bazel_johndoe/4c084335afceb392cfbe7c31afee3a9f/bazel && \ + exec env - \ + /usr/bin/gcc -o bazel-out/local-fastbuild/bin/examples/cpp/hello-world -B/usr/bin/ -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes -Wl,-S -Wl,@bazel-out/local_linux-fastbuild/bin/examples/cpp/hello-world-2.params) +``` + +Where possible, commands are printed in a Bourne shell compatible syntax, so that they can be easily copied and pasted to a shell command prompt. (The surrounding parentheses are provided to protect your shell from the `cd` and `exec` calls; be sure to copy them!) However some commands are implemented internally within Bazel, such as creating symlink trees. For these there's no command line to display. + +`--subcommands=pretty_print` may be passed to print the arguments of the command as a list rather than as a single line. This may help make long command lines more readable. + +See also [--verbose\_failures](#verbose-failures), below. + +For logging subcommands to a file in a tool-friendly format, see [--execution\_log\_json\_file](/reference/command-line-reference#flag--execution_log_json_file) and [--execution\_log\_binary\_file](/reference/command-line-reference#flag--execution_log_binary_file). + +#### `--verbose_failures` + +This option causes Bazel's execution phase to print the full command line for commands that failed. This can be invaluable for debugging a failing build. + +Failing commands are printed in a Bourne shell compatible syntax, suitable for copying and pasting to a shell prompt. + +### Workspace status + +Use these options to "stamp" Bazel-built binaries: to embed additional information into the binaries, such as the source control revision or other workspace-related information. You can use this mechanism with rules that support the `stamp` attribute, such as `genrule`, `cc_binary`, and more. + +#### `--workspace_status_command=<var>program</var>` + +This flag lets you specify a binary that Bazel runs before each build. The program can report information about the status of the workspace, such as the current source control revision. + +The flag's value must be a path to a native program. On Linux/macOS this may be any executable. On Windows this must be a native binary, typically an ".exe", ".bat", or a ".cmd" file. + +The program should print zero or more key/value pairs to standard output, one entry on each line, then exit with zero (otherwise the build fails). The key names can be anything but they may only use upper case letters and underscores. The first space after the key name separates it from the value. The value is the rest of the line (including additional whitespaces). Neither the key nor the value may span multiple lines. Keys must not be duplicated. + +Bazel partitions the keys into two buckets: "stable" and "volatile". (The names "stable" and "volatile" are a bit counter-intuitive, so don't think much about them.) + +Bazel then writes the key-value pairs into two files: + +- `bazel-out/stable-status.txt` contains all keys and values where the key's name starts with `STABLE_` +- `bazel-out/volatile-status.txt` contains the rest of the keys and their values + +The contract is: + +- "stable" keys' values should change rarely, if possible. If the contents of `bazel-out/stable-status.txt` change, Bazel invalidates the actions that depend on them. In other words, if a stable key's value changes, Bazel will rerun stamped actions. Therefore the stable status should not contain things like timestamps, because they change all the time, and would make Bazel rerun stamped actions with each build. + + Bazel always outputs the following stable keys: + + - `BUILD_EMBED_LABEL`: value of `--embed_label` + - `BUILD_HOST`: the name of the host machine that Bazel is running on + - `BUILD_USER`: the name of the user that Bazel is running as + +- "volatile" keys' values may change often. Bazel expects them to change all the time, like timestamps do, and duly updates the `bazel-out/volatile-status.txt` file. In order to avoid rerunning stamped actions all the time though, **Bazel pretends that the volatile file never changes**. In other words, if the volatile status file is the only file whose contents has changed, Bazel will not invalidate actions that depend on it. If other inputs of the actions have changed, then Bazel reruns that action, and the action will see the updated volatile status, but just the volatile status changing alone will not invalidate the action. + + Bazel always outputs the following volatile keys: + + - `BUILD_TIMESTAMP`: time of the build in seconds since the Unix Epoch (the value of `System.currentTimeMillis()` divided by a thousand) + - `FORMATTED_DATE`: time of the build Formatted as `yyyy MMM d HH mm ss EEE`(for example 2023 Jun 2 01 44 29 Fri) in UTC. + +On Linux/macOS you can pass `--workspace_status_command=/bin/true` to disable retrieving workspace status, because `true` does nothing, successfully (exits with zero) and prints no output. On Windows you can pass the path of MSYS's `true.exe` for the same effect. + +If the workspace status command fails (exits non-zero) for any reason, the build will fail. + +Example program on Linux using Git: + +``` +#!/bin/bash +echo "CURRENT_TIME $(date +%s)" +echo "RANDOM_HASH $(cat /proc/sys/kernel/random/uuid)" +echo "STABLE_GIT_COMMIT $(git rev-parse HEAD)" +echo "STABLE_USER_NAME $USER" +``` + +Pass this program's path with `--workspace_status_command`, and the stable status file will include the STABLE lines and the volatile status file will include the rest of the lines. + +#### `--[no]stamp` + +This option, in conjunction with the `stamp` rule attribute, controls whether to embed build information in binaries. + +Stamping can be enabled or disabled explicitly on a per-rule basis using the `stamp` attribute. Please refer to the Build Encyclopedia for details. When a rule sets `stamp = -1` (the default for `*_binary` rules), this option determines whether stamping is enabled. + +Bazel never stamps binaries that are built for the exec configuration, regardless of this option or the `stamp` attribute. For rules that set `stamp = 0` (the default for `*_test` rules), stamping is disabled regardless of `--[no]stamp`. Specifying `--stamp` does not force targets to be rebuilt if their dependencies have not changed. + +Setting `--nostamp` is generally desireable for build performance, as it reduces input volatility and maximizes build caching. + +### Platform + +Use these options to control the host and target platforms that configure how builds work, and to control what execution platforms and toolchains are available to Bazel rules. + +Please see background information on [Platforms](/extending/platforms) and [Toolchains](/extending/toolchains). + +#### `--platforms=<var>labels</var>` + +The labels of the platform rules describing the target platforms for the current command. + +#### `--host_platform=<var>label</var>` + +The label of a platform rule that describes the host system. + +#### `--extra_execution_platforms=<var>labels</var>` + +The platforms that are available as execution platforms to run actions. Platforms can be specified by exact target, or as a target pattern. These platforms will be considered before those declared in MODULE.bazel files by [register\_execution\_platforms()](/rules/lib/globals/module#register_execution_platforms). This option accepts a comma-separated list of platforms in order of priority. If the flag is passed multiple times, the most recent overrides. + +#### `--extra_toolchains=<var>labels</var>` + +The toolchain rules to be considered during toolchain resolution. Toolchains can be specified by exact target, or as a target pattern. These toolchains will be considered before those declared in MODULE.bazel files by [register\_toolchains()](/rules/lib/globals/module#register_toolchains). + +#### `--toolchain_resolution_debug=<var>regex</var>` + +Print debug information while finding toolchains if the toolchain type matches the regex. Multiple regexes can be separated by commas. The regex can be negated by using a `-` at the beginning. This might help developers of Bazel or Starlark rules with debugging failures due to missing toolchains. + +### Miscellaneous + +#### `--flag_alias=<var>alias_name=target_path</var>` + +A convenience flag used to bind longer Starlark build settings to a shorter name. For more details, see the [Starlark Configurations](/extending/config#using-build-setting-aliases). + +#### `--symlink_prefix=<var>string</var>` + +Changes the prefix of the generated convenience symlinks. The default value for the symlink prefix is `bazel-` which will create the symlinks `bazel-bin`, `bazel-testlogs`, and `bazel-genfiles`. + +If the symbolic links cannot be created for any reason, a warning is issued but the build is still considered a success. In particular, this allows you to build in a read-only directory or one that you have no permission to write into. Any paths printed in informational messages at the conclusion of a build will only use the symlink-relative short form if the symlinks point to the expected location; in other words, you can rely on the correctness of those paths, even if you cannot rely on the symlinks being created. + +Some common values of this option: + +- **Suppress symlink creation:** `--symlink_prefix=/` will cause Bazel to not create or update any symlinks, including the `bazel-out` and `bazel-<workspace>` symlinks. Use this option to suppress symlink creation entirely. + +- **Reduce clutter:** `--symlink_prefix=.bazel/` will cause Bazel to create symlinks called `bin` (etc) inside a hidden directory `.bazel`. + +#### `--platform_suffix=<var>string</var>` + +Adds a suffix to the configuration short name, which is used to determine the output directory. Setting this option to different values puts the files into different directories, for example to improve cache hit rates for builds that otherwise clobber each others output files, or to keep the output files around for comparisons. + +#### `--default_visibility=<var>(private|public)</var>` + +Temporary flag for testing bazel default visibility changes. Not intended for general use but documented for completeness' sake. + +#### `--starlark_cpu_profile=_file_` + +This flag, whose value is the name of a file, causes Bazel to gather statistics about CPU usage by all Starlark threads, and write the profile, in [pprof](https://github.com/google/pprof) format, to the named file. + +Use this option to help identify Starlark functions that make loading and analysis slow due to excessive computation. For example: + +``` +$ bazel build --nobuild --starlark_cpu_profile=/tmp/pprof.gz my/project/... +$ pprof /tmp/pprof.gz +(pprof) top +Type: CPU +Time: Feb 6, 2020 at 12:06pm (PST) +Duration: 5.26s, Total samples = 3.34s (63.55%) +Showing nodes accounting for 3.34s, 100% of 3.34s total + flat flat% sum% cum cum% + 1.86s 55.69% 55.69% 1.86s 55.69% sort_source_files + 1.02s 30.54% 86.23% 1.02s 30.54% expand_all_combinations + 0.44s 13.17% 99.40% 0.44s 13.17% range + 0.02s 0.6% 100% 3.34s 100% sorted + 0 0% 100% 1.38s 41.32% my/project/main/BUILD + 0 0% 100% 1.96s 58.68% my/project/library.bzl + 0 0% 100% 3.34s 100% main +``` + +For different views of the same data, try the `pprof` commands `svg`, `web`, and `list`. + +## Using Bazel for releases + +Bazel is used both by software engineers during the development cycle, and by release engineers when preparing binaries for deployment to production. This section provides a list of tips for release engineers using Bazel. + +### Significant options + +When using Bazel for release builds, the same issues arise as for other scripts that perform a build. For more details, see [Call Bazel from scripts](/run/scripts). In particular, the following options are strongly recommended: + +- [`--bazelrc=/dev/null`](/run/bazelrc) +- [`--nokeep_state_after_build`](/reference/command-line-reference#common_options-flag--keep_state_after_build) + +These options are also important: + +- [`--package_path`](#package-path) +- [`--symlink_prefix`](#symlink-prefix): for managing builds for multiple configurations, it may be convenient to distinguish each build with a distinct identifier, such as "64bit" vs. "32bit". This option differentiates the `bazel-bin` (etc.) symlinks. + +## Running tests + +To build and run tests with bazel, type `bazel test` followed by the name of the test targets. + +By default, this command performs simultaneous build and test activity, building all specified targets (including any non-test targets specified on the command line) and testing `*_test` and `test_suite` targets as soon as their prerequisites are built, meaning that test execution is interleaved with building. Doing so usually results in significant speed gains. + +### Options for `bazel test` + +#### `--cache_test_results=(yes|no|auto)` (`-t`) + +If this option is set to 'auto' (the default) then Bazel will only rerun a test if any of the following conditions applies: + +- Bazel detects changes in the test or its dependencies +- the test is marked as `external` +- multiple test runs were requested with `--runs_per_test` +- the test failed. + +If 'no', all tests will be executed unconditionally. + +If 'yes', the caching behavior will be the same as auto except that it may cache test failures and test runs with `--runs_per_test`. + +Note: Test results are *always* saved in Bazel's output tree, regardless of whether this option is enabled, so you needn't have used `--cache_test_results` on the prior run(s) of `bazel test` in order to get cache hits. The option only affects whether Bazel will *use* previously saved results, not whether it will save results of the current run. + +Users who have enabled this option by default in their `.bazelrc` file may find the abbreviations `-t` (on) or `-t-` (off) convenient for overriding the default on a particular run. + +#### `--check_tests_up_to_date` + +This option tells Bazel not to run the tests, but to merely check and report the cached test results. If there are any tests which have not been previously built and run, or whose tests results are out-of-date (for example, because the source code or the build options have changed), then Bazel will report an error message ("test result is not up-to-date"), will record the test's status as "NO STATUS" (in red, if color output is enabled), and will return a non-zero exit code. + +This option also implies [`--check_up_to_date`](#check-up-to-date) behavior. + +This option may be useful for pre-submit checks. + +#### `--test_verbose_timeout_warnings` + +This option tells Bazel to explicitly warn the user if a test's timeout is significantly longer than the test's actual execution time. While a test's timeout should be set such that it is not flaky, a test that has a highly over-generous timeout can hide real problems that crop up unexpectedly. + +For instance, a test that normally executes in a minute or two should not have a timeout of ETERNAL or LONG as these are much, much too generous. + +This option is useful to help users decide on a good timeout value or sanity check existing timeout values. + +Note: Each test shard is allotted the timeout of the entire `XX_test` target. Using this option does not affect a test's timeout value, merely warns if Bazel thinks the timeout could be restricted further. + +#### `--[no]test_keep_going` + +By default, all tests are run to completion. If this flag is disabled, however, the build is aborted on any non-passing test. Subsequent build steps and test invocations are not run, and in-flight invocations are canceled. Do not specify both `--notest_keep_going` and `--keep_going`. + +#### `--flaky_test_attempts=<var>attempts</var>` + +This option specifies the maximum number of times a test should be attempted if it fails for any reason. A test that initially fails but eventually succeeds is reported as `FLAKY` on the test summary. It is, however, considered to be passed when it comes to identifying Bazel exit code or total number of passed tests. Tests that fail all allowed attempts are considered to be failed. + +By default (when this option is not specified, or when it is set to default), only a single attempt is allowed for regular tests, and 3 for test rules with the `flaky` attribute set. You can specify an integer value to override the maximum limit of test attempts. Bazel allows a maximum of 10 test attempts in order to prevent abuse of the system. + +#### `--runs_per_test=<var>[regex@]number</var>` + +This option specifies the number of times each test should be executed. All test executions are treated as separate tests (fallback functionality will apply to each of them independently). + +The status of a target with failing runs depends on the value of the `--runs_per_test_detects_flakes` flag: + +- If absent, any failing run causes the entire test to fail. +- If present and two runs from the same shard return PASS and FAIL, the test will receive a status of flaky (unless other failing runs cause it to fail). + +If a single number is specified, all tests will run that many times. Alternatively, a regular expression may be specified using the syntax regex\@number. This constrains the effect of `--runs_per_test` to targets which match the regex (`--runs_per_test=^//pizza:.*@4` runs all tests under `//pizza/` 4 times). This form of `--runs_per_test` may be specified more than once. + +#### `--[no]runs_per_test_detects_flakes` + +If this option is specified (by default it is not), Bazel will detect flaky test shards through `--runs_per_test`. If one or more runs for a single shard fail and one or more runs for the same shard pass, the target will be considered flaky with the flag. If unspecified, the target will report a failing status. + +#### `--test_summary=<var>output_style</var>` + +Specifies how the test result summary should be displayed. + +- `short` prints the results of each test along with the name of the file containing the test output if the test failed. This is the default value. +- `terse` like `short`, but even shorter: only print information about tests which did not pass. +- `detailed` prints each individual test case that failed, not only each test. The names of test output files are omitted. +- `none` does not print test summary. + +#### `--test_output=<var>output_style</var>` + +Specifies how test output should be displayed: + +- `summary` shows a summary of whether each test passed or failed. Also shows the output log file name for failed tests. The summary will be printed at the end of the build (during the build, one would see just simple progress messages when tests start, pass or fail). This is the default behavior. +- `errors` sends combined stdout/stderr output from failed tests only into the stdout immediately after test is completed, ensuring that test output from simultaneous tests is not interleaved with each other. Prints a summary at the build as per summary output above. +- `all` is similar to `errors` but prints output for all tests, including those which passed. +- `streamed` streams stdout/stderr output from each test in real-time. + +#### `--java_debug` + +This option causes the Java virtual machine of a java test to wait for a connection from a JDWP-compliant debugger before starting the test. This option implies `--test_output=streamed`. + +#### `--[no]verbose_test_summary` + +By default this option is enabled, causing test times and other additional information (such as test attempts) to be printed to the test summary. If `--noverbose_test_summary` is specified, test summary will include only test name, test status and cached test indicator and will be formatted to stay within 80 characters when possible. + +#### `--test_tmpdir=<var>path</var>` + +Specifies temporary directory for tests executed locally. Each test will be executed in a separate subdirectory inside this directory. The directory will be cleaned at the beginning of the each `bazel test` command. By default, bazel will place this directory under Bazel output base directory. + +Note: This is a directory for running tests, not storing test results (those are always stored under the `bazel-out` directory). + +#### `--test_timeout=<var>seconds</var>` OR `--test_timeout=<var>seconds</var>,<var>seconds</var>,<var>seconds</var>,<var>seconds</var>` + +Overrides the timeout value for all tests by using specified number of seconds as a new timeout value. If only one value is provided, then it will be used for all test timeout categories. + +Alternatively, four comma-separated values may be provided, specifying individual timeouts for short, moderate, long and eternal tests (in that order). In either form, zero or a negative value for any of the test sizes will be substituted by the default timeout for the given timeout categories as defined by the page [Writing Tests](/reference/test-encyclopedia). By default, Bazel will use these timeouts for all tests by inferring the timeout limit from the test's size whether the size is implicitly or explicitly set. + +Tests which explicitly state their timeout category as distinct from their size will receive the same value as if that timeout had been implicitly set by the size tag. So a test of size 'small' which declares a 'long' timeout will have the same effective timeout that a 'large' tests has with no explicit timeout. + +#### `--test_arg=<var>arg</var>` + +Passes command-line options/flags/arguments to each test process. This option can be used multiple times to pass several arguments. For example, `--test_arg=--logtostderr --test_arg=--v=3`. + +Note that, unlike the `bazel run` command, you can't pass test arguments directly as in `bazel test -- target --logtostderr --v=3`. That's because extraneous arguments passed to `bazel test` are interpreted as additional test targets. That is, `--logtostderr` and `--v=3` would each be interpreted as a test target. This ambiguity doesn't exist for a `bazel run` command, which only accepts one target. + +`--test_arg` can be passed to a `bazel run` command, but it's ignored unless the target being run is a test target. (As with any other flag, if it's passed in a `bazel run` command after a `--` token, it's not processed by Bazel but forwarded verbatim to the executed target.) + +#### `--test_env=<var>variable</var>=_value_` OR `--test_env=<var>variable</var>` + +Specifies additional variables that must be injected into the test environment for each test. If \{\{ "`" }}value{{ "`" \}\} is not specified it will be inherited from the shell environment used to start the `bazel test` command. + +The environment can be accessed from within a test by using `System.getenv("var")` (Java), `getenv("var")` (C or C++), + +#### `--run_under=<var>command-prefix</var>` + +This specifies a prefix that the test runner will insert in front of the test command before running it. The \{\{ "`" }}command-prefix{{ "`" \}\} is split into words using Bourne shell tokenization rules, and then the list of words is prepended to the command that will be executed. + +If the first word is a fully-qualified label (starts with `//`) it is built. Then the label is substituted by the corresponding executable location that is prepended to the command that will be executed along with the other words. + +Some caveats apply: + +- The PATH used for running tests may be different than the PATH in your environment, so you may need to use an **absolute path** for the `--run_under` command (the first word in \{\{ "`" }}command-prefix{{ "`" \}\}). +- **`stdin` is not connected**, so `--run_under` can't be used for interactive commands. + +Examples: + +``` + --run_under=/usr/bin/strace + --run_under='/usr/bin/strace -c' + --run_under=/usr/bin/valgrind + --run_under='/usr/bin/valgrind --quiet --num-callers=20' +``` + +#### Test selection + +As documented under [Output selection options](#output-selection), you can filter tests by [size](#test-size-filters), [timeout](#test-timeout-filters), [tag](#test-tag-filters), or [language](#test-lang-filters). A convenience [general name filter](#test-filter) can forward particular filter args to the test runner. + +#### Other options for `bazel test` + +The syntax and the remaining options are exactly like [`bazel build`](/run/build). + +## Running executables + +The `bazel run` command is similar to `bazel build`, except it is used to build *and run* a single target. Here is a typical session (`//java/myapp:myapp` says hello and prints out its args): + +``` + % bazel run java/myapp:myapp -- --arg1 --arg2 + INFO: Analyzed target //java/myapp:myapp (13 packages loaded, 27 targets configured). + INFO: Found 1 target... + Target //java/myapp:myapp up-to-date: + bazel-bin/java/myapp/myapp + INFO: Elapsed time: 14.290s, Critical Path: 5.54s, ... + INFO: Build completed successfully, 4 total actions + INFO: Running command line: bazel-bin/java/myapp/myapp <args omitted> + Hello there + $EXEC_ROOT/java/myapp/myapp + --arg1 + --arg2 +``` + +Note: `--` is needed so that Bazel does not interpret `--arg1` and `--arg2` as Bazel options, but rather as part of the command line for running the binary. Additionally, Bazel will avoid logging these arguments to the console in case they contain sensitive information. + +`bazel run` is similar, but not identical, to directly invoking the binary built by Bazel and its behavior is different depending on whether the binary to be invoked is a test or not. + +When the binary is not a test, the current working directory will be the runfiles tree of the binary. + +When the binary is a test, the current working directory will be the exec root and a good-faith attempt is made to replicate the environment tests are usually run in. The emulation is not perfect, though, and tests that have multiple shards cannot be run this way (the `--test_sharding_strategy=disabled` command line option can be used to work around this) + +The following extra environment variables are also available to the binary: + +- `BUILD_WORKSPACE_DIRECTORY`: the root of the workspace where the build was run. +- `BUILD_WORKING_DIRECTORY`: the current working directory where Bazel was run from. +- `BUILD_ID`: the build ID of the `bazel run` invocation. This is usually unique, except if Bazel was run with `--script_path` and the resulting script is re-used. +- `BUILD_EXECROOT`: the execution root of the `bazel run` invocation. + +These can be used, for example, to interpret file names on the command line in a user-friendly way. + +### Options for `bazel run` + +#### `--run_under=<var>command-prefix</var>` + +This has the same effect as the `--run_under` option for `bazel test` ([see above](#test-run-under)), except that it applies to the command being run by `bazel run` rather than to the tests being run by `bazel test` and cannot run under label. + +#### Filtering logging outputs from Bazel + +When invoking a binary with `bazel run`, Bazel prints logging output from Bazel itself and the binary under invocation. To make the logs less noisy, you can suppress the outputs from Bazel itself with the `--ui_event_filters` and `--noshow_progress` flags. + +For example: `bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress //java/myapp:myapp` + +### Executing tests + +`bazel run` can also execute test binaries, which has the effect of running the test in a close approximation of the environment described at [Writing Tests](/reference/test-encyclopedia). Note that none of the `--test_*` arguments have an effect when running a test in this manner except `--test_arg` . + +## Cleaning build outputs + +### The `clean` command + +Bazel has a `clean` command, analogous to that of Make. It deletes the output directories for all build configurations performed by this Bazel instance, or the entire working tree created by this Bazel instance, and resets internal caches. If executed without any command-line options, then the output directory for all configurations will be cleaned. + +Recall that each Bazel instance is associated with a single workspace, thus the `clean` command will delete all outputs from all builds you've done with that Bazel instance in that workspace. + +To completely remove the entire working tree created by a Bazel instance, you can specify the `--expunge` option. When executed with `--expunge`, the clean command simply removes the entire output base tree which, in addition to the build output, contains all temp files created by Bazel. It also stops the Bazel server after the clean, equivalent to the [`shutdown`](#shutdown) command. For example, to clean up all disk and memory traces of a Bazel instance, you could specify: + +``` + % bazel clean --expunge +``` + +Alternatively, you can expunge in the background by using `--expunge_async`. It is safe to invoke a Bazel command in the same client while the asynchronous expunge continues to run. + +Note: This may introduce IO contention. + +The `clean` command is provided primarily as a means of reclaiming disk space for workspaces that are no longer needed. Bazel's incremental rebuilds may not be perfect so `clean` can be used to recover a consistent state when problems arise. + +Bazel's design is such that these problems are fixable and these bugs are a high priority to be fixed. If you ever find an incorrect incremental build, file a bug report, and report bugs in the tools rather than using `clean`. + +## Querying the dependency graph + +Bazel includes a query language for asking questions about the dependency graph used during the build. The query language is used by two commands: query and cquery. The major difference between the two commands is that query runs after the [loading phase](/run/build#loading) and cquery runs after the [analysis phase](/run/build#analysis). These tools are an invaluable aid to many software engineering tasks. + +The query language is based on the idea of algebraic operations over graphs; it is documented in detail in + +[Bazel Query Reference](/query/language). Please refer to that document for reference, for examples, and for query-specific command-line options. + +The query tool accepts several command-line option. `--output` selects the output format. `--[no]keep_going` (disabled by default) causes the query tool to continue to make progress upon errors; this behavior may be disabled if an incomplete result is not acceptable in case of errors. + +The `--[no]tool_deps` option, enabled by default, causes dependencies in non-target configurations to be included in the dependency graph over which the query operates. + +The `--[no]implicit_deps` option, enabled by default, causes implicit dependencies to be included in the dependency graph over which the query operates. An implicit dependency is one that is not explicitly specified in the BUILD file but added by bazel. + +Example: "Show the locations of the definitions (in BUILD files) of all genrules required to build all the tests in the PEBL tree." + +``` + bazel query --output location 'kind(genrule, deps(kind(".*_test rule", foo/bar/pebl/...)))' +``` + +## Querying the action graph + +Caution: The aquery command is still experimental and its API will change. + +The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis configured target graph and exposes information about actions, artifacts and their relationships. + +The tool accepts several command-line options. `--output` selects the output format. The default output format (`text`) is human-readable, use `proto` or `textproto` for machine-readable format. Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. + +It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. + +For more details, see [Action Graph Query](/query/aquery). + +## Miscellaneous commands and options + +### `help` + +The `help` command provides on-line help. By default, it shows a summary of available commands and help topics, as shown in [Building with Bazel](/run/build#quickstart). Specifying an argument displays detailed help for a particular topic. Most topics are Bazel commands, such as `build` or `query`, but there are some additional help topics that do not correspond to commands. + +#### `--[no]long` (`-l`) + +By default, `bazel help [<var>topic</var>]` prints only a summary of the relevant options for a topic. If the `--long` option is specified, the type, default value and full description of each option is also printed. + +### `shutdown` + +Bazel server processes may be stopped by using the `shutdown` command. This command causes the Bazel server to exit as soon as it becomes idle (for example, after the completion of any builds or other commands that are currently in progress). For more details, see [Client/server implementation](/run/client-server). + +Bazel servers stop themselves after an idle timeout, so this command is rarely necessary; however, it can be useful in scripts when it is known that no further builds will occur in a given workspace. + +`shutdown` accepts one option, `--iff_heap_size_greater_than _n_`, which requires an integer argument (in MB). If specified, this makes the shutdown conditional on the amount of memory already consumed. This is useful for scripts that initiate a lot of builds, as any memory leaks in the Bazel server could cause it to crash spuriously on occasion; performing a conditional restart preempts this condition. + +### `info` + +The `info` command prints various values associated with the Bazel server instance, or with a specific build configuration. (These may be used by scripts that drive a build.) + +The `info` command also permits a single (optional) argument, which is the name of one of the keys in the list below. In this case, `bazel info <var>key</var>` will print only the value for that one key. (This is especially convenient when scripting Bazel, as it avoids the need to pipe the result through `sed -ne /key:/s/key://p`: + +#### Configuration-independent data + +- `release`: the release label for this Bazel instance, or "development version" if this is not a released binary. + +- `workspace` the absolute path to the base workspace directory. + +- `install_base`: the absolute path to the installation directory used by this Bazel instance for the current user. Bazel installs its internally required executables below this directory. + +- `output_base`: the absolute path to the base output directory used by this Bazel instance for the current user and workspace combination. Bazel puts all of its scratch and build output below this directory. + +- `execution_root`: the absolute path to the execution root directory under output\_base. This directory is the root for all files accessible to commands executed during the build, and is the working directory for those commands. If the workspace directory is writable, a symlink named `bazel-<workspace>` is placed there pointing to this directory. + +- `output_path`: the absolute path to the output directory beneath the execution root used for all files actually generated as a result of build commands. If the workspace directory is writable, a symlink named `bazel-out` is placed there pointing to this directory. + +- `server_pid`: the process ID of the Bazel server process. + +- `server_log`: the absolute path to the Bazel server's debug log file. This file contains debugging information for all commands over the lifetime of the Bazel server, and is intended for human consumption by Bazel developers and power users. + +- `command_log`: the absolute path to the command log file; this contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. + +- `used-heap-size`, `committed-heap-size`, `max-heap-size`: reports various JVM heap size parameters. Respectively: memory currently used, memory currently guaranteed to be available to the JVM from the system, maximum possible allocation. + +- `gc-count`, `gc-time`: The cumulative count of garbage collections since the start of this Bazel server and the time spent to perform them. Note that these values are not reset at the start of every build. + +- `package_path`: A colon-separated list of paths which would be searched for packages by bazel. Has the same format as the `--package_path` build command line argument. + +Example: the process ID of the Bazel server. + +``` +% bazel info server_pid +1285 +``` + +#### Configuration-specific data + +These data may be affected by the configuration options passed to `bazel info`, for example `--cpu`, `--compilation_mode`, etc. The `info` command accepts all the options that control dependency analysis, since some of these determine the location of the output directory of a build, the choice of compiler, etc. + +- `bazel-bin`, `bazel-testlogs`, `bazel-genfiles`: reports the absolute path to the `bazel-*` directories in which programs generated by the build are located. This is usually, though not always, the same as the `bazel-*` symlinks created in the base workspace directory after a successful build. However, if the workspace directory is read-only, no `bazel-*` symlinks can be created. Scripts that use the value reported by `bazel info`, instead of assuming the existence of the symlink, will be more robust. +- The complete ["Make" environment](/reference/be/make-variables). If the `--show_make_env` flag is specified, all variables in the current configuration's "Make" environment are also displayed (such as `CC`, `GLIBC_VERSION`, etc). These are the variables accessed using the `$(CC)` or `varref("CC")` syntax inside BUILD files. + +Example: the C++ compiler for the current configuration. This is the `$(CC)` variable in the "Make" environment, so the `--show_make_env` flag is needed. + +``` + % bazel info --show_make_env -c opt COMPILATION_MODE + opt +``` + +Example: the `bazel-bin` output directory for the current configuration. This is guaranteed to be correct even in cases where the `bazel-bin` symlink cannot be created for some reason (such as if you are building from a read-only directory). + +``` +% bazel info --cpu=piii bazel-bin +/var/tmp/_bazel_johndoe/fbd0e8a34f61ce5d491e3da69d959fe6/execroot/io_bazel/bazel-out/piii-opt/bin +% bazel info --cpu=k8 bazel-bin +/var/tmp/_bazel_johndoe/fbd0e8a34f61ce5d491e3da69d959fe6/execroot/io_bazel/bazel-out/k8-opt/bin +``` + +### `version` and `--version` + +The version command prints version details about the built Bazel binary, including the changelist at which it was built and the date. These are particularly useful in determining if you have the latest Bazel, or if you are reporting bugs. Some of the interesting values are: + +- `changelist`: the changelist at which this version of Bazel was released. +- `label`: the release label for this Bazel instance, or "development version" if this is not a released binary. Very useful when reporting bugs. + +`bazel --version`, with no other args, will emit the same output as `bazel version --gnu_format`, except without the side-effect of potentially starting a Bazel server or unpacking the server archive. `bazel --version` can be run from anywhere - it does not require a workspace directory. + +### `mobile-install` + +The `mobile-install` command installs apps to mobile devices. Currently only Android devices running ART are supported. + +See [bazel mobile-install](/docs/mobile-install) for more information. + +Note: This command does not install the same thing that `bazel build` produces: Bazel tweaks the app so that it can be built, installed and re-installed quickly. This should, however, be mostly transparent to the app. + +The following options are supported: + +#### `--incremental` + +If set, Bazel tries to install the app incrementally, that is, only those parts that have changed since the last build. This cannot update resources referenced from `AndroidManifest.xml`, native code or Java resources (such as those referenced by `Class.getResource()`). If these things change, this option must be omitted. Contrary to the spirit of Bazel and due to limitations of the Android platform, it is the **responsibility of the user** to know when this command is good enough and when a full install is needed. + +If you are using a device with Marshmallow or later, consider the [`--split_apks`](#split-apks) flag. + +#### `--split_apks` + +Whether to use split apks to install and update the application on the device. Works only with devices with Marshmallow or later. Note that the [`--incremental`](#incremental) flag is not necessary when using `--split_apks`. + +#### `--start_app` + +Starts the app in a clean state after installing. Equivalent to `--start=COLD`. + +#### `--debug_app` + +Waits for debugger to be attached before starting the app in a clean state after installing. Equivalent to `--start=DEBUG`. + +#### `--start=_start_type_` + +How the app should be started after installing it. Supported \_start\_type\_s are: + +- `NO` Does not start the app. This is the default. +- `COLD` Starts the app from a clean state after install. +- `WARM` Preserves and restores the application state on incremental installs. +- `DEBUG` Waits for the debugger before starting the app in a clean state after install. + +Note: If more than one of `--start=_start_type_`, `--start_app` or `--debug_app` is set, the last value is used. + +#### `--adb=<var>path</var>` + +Indicates the `adb` binary to be used. + +The default is to use the adb in the Android SDK specified by [`--android_sdk`](#android-sdk). + +#### `--adb_arg=<var>serial</var>` + +Extra arguments to `adb`. These come before the subcommand in the command line and are typically used to specify which device to install to. For example, to select the Android device or emulator to use: + +``` +% bazel mobile-install --adb_arg=-s --adb_arg=deadbeef +``` + +invokes `adb` as + +``` +adb -s deadbeef install ... +``` + +#### `--incremental_install_verbosity=<var>number</var>` + +The verbosity for incremental install. Set to 1 for debug logging to be printed to the console. + +### `dump` + +The `dump` command prints to stdout a dump of the internal state of the Bazel server. This command is intended primarily for use by Bazel developers, so the output of this command is not specified, and is subject to change. + +By default, command will just print help message outlining possible options to dump specific areas of the Bazel state. In order to dump internal state, at least one of the options must be specified. + +Following options are supported: + +- `--action_cache` dumps action cache content. +- `--packages` dumps package cache content. +- `--skyframe` dumps state of internal Bazel dependency graph. +- `--rules` dumps rule summary for each rule and aspect class, including counts and action counts. This includes both native and Starlark rules. If memory tracking is enabled, then the rules' memory consumption is also printed. +- `--skylark_memory` dumps a [pprof](https://github.com/google/pprof) compatible .gz file to the specified path. You must enable memory tracking for this to work. + +#### Memory tracking + +Some `dump` commands require memory tracking. To turn this on, you have to pass startup flags to Bazel: + +- `--host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar` +- `--host_jvm_args=-DRULE_MEMORY_TRACKER=1` + +The java-agent is checked into Bazel at `third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar`, so make sure you adjust `$BAZEL` for where you keep your Bazel repository. + +Do not forget to keep passing these options to Bazel for every command or the server will restart. + +Example: + +``` + % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ + --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ + build --nobuild <targets> + + # Dump rules + % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ + --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ + dump --rules + + # Dump Starlark heap and analyze it with pprof + % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ + --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ + dump --skylark_memory=$HOME/prof.gz + % pprof -flame $HOME/prof.gz +``` + +### `analyze-profile` + +The `analyze-profile` command analyzes a [JSON trace profile](/advanced/performance/json-trace-profile) previously gathered during a Bazel invocation. + +### `canonicalize-flags` + +The [`canonicalize-flags`](/reference/command-line-reference#canonicalize-flags-options) command, which takes a list of options for a Bazel command and returns a list of options that has the same effect. The new list of options is canonical. For example, two lists of options with the same effect are canonicalized to the same new list. + +The `--for_command` option can be used to select between different commands. At this time, only `build` and `test` are supported. Options that the given command does not support cause an error. + +Note: A small number of options cannot be reordered, because Bazel cannot ensure that the effect is identical. Also note that this command *does not* expand flags from `--config`. + +As an example: + +``` + % bazel canonicalize-flags -- --config=any_name --test_tag_filters="-lint" + --config=any_name + --test_tag_filters=-lint +``` + +### Startup options + +The options described in this section affect the startup of the Java virtual machine used by Bazel server process, and they apply to all subsequent commands handled by that server. If there is an already running Bazel server and the startup options do not match, it will be restarted. + +All of the options described in this section must be specified using the `--key=value` or `--key value` syntax. Also, these options must appear *before* the name of the Bazel command. Use `startup --key=value` to list these in a `.bazelrc` file. + +#### `--output_base=<var>dir</var>` + +This option requires a path argument, which must specify a writable directory. Bazel will use this location to write all its output. The output base is also the key by which the client locates the Bazel server. By changing the output base, you change the server which will handle the command. + +By default, the output base is derived from the user's login name, and the name of the workspace directory (actually, its MD5 digest), so a typical value looks like: `/var/tmp/google/_bazel_johndoe/d41d8cd98f00b204e9800998ecf8427e`. + +Note: The client uses the output base to find the Bazel server instance, so if you specify a different output base in a Bazel command, a different server will be found (or started) to handle the request. It's possible to perform two concurrent builds in the same workspace directory by varying the output base. + +For example: + +``` + OUTPUT_BASE=/var/tmp/google/_bazel_johndoe/custom_output_base +% bazel --output_base ${OUTPUT_BASE}1 build //foo & bazel --output_base ${OUTPUT_BASE}2 build //bar +``` + +In this command, the two Bazel commands run concurrently (because of the shell `&` operator), each using a different Bazel server instance (because of the different output bases). In contrast, if the default output base was used in both commands, then both requests would be sent to the same server, which would handle them sequentially: building `//foo` first, followed by an incremental build of `//bar`. + +Note: We recommend you do not use an NFS or similar networked file system for the root directory, as the higher access latency will cause noticeably slower builds. + +#### `--output_user_root=<var>dir</var>` + +Points to the root directory where output and install bases are created. The directory must either not exist or be owned by the calling user. In the past, this was allowed to point to a directory shared among various users but it's not allowed any longer. This may be allowed once [issue #11100](https://github.com/bazelbuild/bazel/issues/11100) is addressed. + +If the `--output_base` option is specified, it overrides using `--output_user_root` to calculate the output base. + +The install base location is calculated based on `--output_user_root`, plus the MD5 identity of the Bazel embedded binaries. + +You can use the `--output_user_root` option to choose an alternate base location for all of Bazel's output (install base and output base) if there is a better location in your filesystem layout. + +Note: We recommend you do not use an NFS or similar networked file system for the root directory, as the higher access latency will cause noticeably slower builds. + +#### `--server_javabase=<var>dir</var>` + +Specifies the Java virtual machine in which *Bazel itself* runs. The value must be a path to the directory containing a JDK or JRE. It should not be a label. This option should appear before any Bazel command, for example: + +``` + % bazel --server_javabase=/usr/local/buildtools/java/jdk build //foo +``` + +This flag does *not* affect the JVMs used by Bazel subprocesses such as applications, tests, tools, and so on. Use build options [--javabase](#javabase) or [--host\_javabase](#host-javabase) instead. + +This flag was previously named `--host_javabase` (sometimes referred to as the 'left-hand side' `--host_javabase`), but was renamed to avoid confusion with the build flag [--host\_javabase](#host-javabase) (sometimes referred to as the 'right-hand side' `--host_javabase`). + +#### `--host_jvm_args=<var>string</var>` + +Specifies a startup option to be passed to the Java virtual machine in which *Bazel itself* runs. This can be used to set the stack size, for example: + +``` + % bazel --host_jvm_args="-Xss256K" build //foo +``` + +This option can be used multiple times with individual arguments. Note that setting this flag should rarely be needed. You can also pass a space-separated list of strings, each of which will be interpreted as a separate JVM argument, but this feature will soon be deprecated. + +That this does *not* affect any JVMs used by subprocesses of Bazel: applications, tests, tools, and so on. To pass JVM options to executable Java programs, whether run by `bazel run` or on the command-line, you should use the `--jvm_flags` argument which all `java_binary` and `java_test` programs support. Alternatively for tests, use `bazel test --test_arg=--jvm_flags=foo ...`. + +#### `--host_jvm_debug` + +This option causes the Java virtual machine to wait for a connection from a JDWP-compliant debugger before calling the main method of *Bazel itself*. This is primarily intended for use by Bazel developers. + +Note: This does *not* affect any JVMs used by subprocesses of Bazel: applications, tests, tools, etc. + +#### `--autodetect_server_javabase` + +This option causes Bazel to automatically search for an installed JDK on startup, and to fall back to the installed JRE if the embedded JRE isn't available. `--explicit_server_javabase` can be used to pick an explicit JRE to run Bazel with. + +#### `--batch` + +Batch mode causes Bazel to not use the [standard client/server mode](/run/client-server), but instead runs a bazel java process for a single command, which has been used for more predictable semantics with respect to signal handling, job control, and environment variable inheritance, and is necessary for running bazel in a chroot jail. + +Batch mode retains proper queueing semantics within the same output\_base. That is, simultaneous invocations will be processed in order, without overlap. If a batch mode Bazel is run on a client with a running server, it first kills the server before processing the command. + +Bazel will run slower in batch mode, or with the alternatives described above. This is because, among other things, the build file cache is memory-resident, so it is not preserved between sequential batch invocations. Therefore, using batch mode often makes more sense in cases where performance is less critical, such as continuous builds. + +Warning: `--batch` is sufficiently slower than standard client/server mode. Additionally it might not support all of the features and optimizations which are made possible by a persistent Bazel server. If you're using `--batch` for the purpose of build isolation, you should use the command option `--nokeep_state_after_build`, which guarantees that no incremental in-memory state is kept between builds. In order to restart the Bazel server and JVM after a build, please explicitly do so using the "shutdown" command. + +#### `--max_idle_secs=<var>n</var>` + +This option specifies how long, in seconds, the Bazel server process should wait after the last client request, before it exits. The default value is 10800 (3 hours). `--max_idle_secs=0` will cause the Bazel server process to persist indefinitely. + +Note: this flag is only read if Bazel needs to start a new server. Changing this option will not cause the server to restart. + +Note: system sleep time where a build is not running is counted as idle time. + +This option may be used by scripts that invoke Bazel to ensure that they do not leave Bazel server processes on a user's machine when they would not be running otherwise. For example, a presubmit script might wish to invoke `bazel query` to ensure that a user's pending change does not introduce unwanted dependencies. However, if the user has not done a recent build in that workspace, it would be undesirable for the presubmit script to start a Bazel server just for it to remain idle for the rest of the day. By specifying a small value of `--max_idle_secs` in the query request, the script can ensure that *if* it caused a new server to start, that server will exit promptly, but if instead there was already a server running, that server will continue to run until it has been idle for the usual time. Of course, the existing server's idle timer will be reset. + +#### `--[no]shutdown_on_low_sys_mem` + +If enabled and `--max_idle_secs` is set to a positive duration, after the build server has been idle for a while, shut down the server when the system is low on memory. Linux only. + +In addition to running an idle check corresponding to max\_idle\_secs, the build server will starts monitoring available system memory after the server has been idle for some time. If the available system memory becomes critically low, the server will exit. + +#### `--[no]block_for_lock` + +If enabled, Bazel will wait for other Bazel commands holding the server lock to complete before progressing. If disabled, Bazel will exit in error if it cannot immediately acquire the lock and proceed. + +Developers might use this in presubmit checks to avoid long waits caused by another Bazel command in the same client. + +#### `--io_nice_level=<var>n</var>` + +Sets a level from 0-7 for best-effort IO scheduling. 0 is highest priority, 7 is lowest. The anticipatory scheduler may only honor up to priority 4. Negative values are ignored. + +#### `--batch_cpu_scheduling` + +Use `batch` CPU scheduling for Bazel. This policy is useful for workloads that are non-interactive, but do not want to lower their nice value. See 'man 2 sched\_setscheduler'. This policy may provide for better system interactivity at the expense of Bazel throughput. + +### Miscellaneous options + +#### `--[no]announce_rc` + +Controls whether Bazel announces startup options and command options read from the bazelrc files when starting up. + +#### `--color (yes|no|auto)` + +This option determines whether Bazel will use colors to highlight its output on the screen. + +If this option is set to `yes`, color output is enabled. If this option is set to `auto`, Bazel will use color output only if the output is being sent to a terminal and the TERM environment variable is set to a value other than `dumb`, `emacs`, or `xterm-mono`. If this option is set to `no`, color output is disabled, regardless of whether the output is going to a terminal and regardless of the setting of the TERM environment variable. + +#### `--config=<var>name</var>` + +Selects additional config section from [the rc files](/run/bazelrc#bazelrc-file-locations); for the current `command`, it also pulls in the options from `command:name` if such a section exists. Can be specified multiple times to add flags from several config sections. Expansions can refer to other definitions (for example, expansions can be chained). + +#### `--curses (yes|no|auto)` + +This option determines whether Bazel will use cursor controls in its screen output. This results in less scrolling data, and a more compact, easy-to-read stream of output from Bazel. This works well with `--color`. + +If this option is set to `yes`, use of cursor controls is enabled. If this option is set to `no`, use of cursor controls is disabled. If this option is set to `auto`, use of cursor controls will be enabled under the same conditions as for `--color=auto`. + +#### `--[no]show_timestamps` + +If specified, a timestamp is added to each message generated by Bazel specifying the time at which the message was displayed. diff --git a/extending/aspects.mdx b/extending/aspects.mdx index ac9a0273..c956a8ef 100644 --- a/extending/aspects.mdx +++ b/extending/aspects.mdx @@ -2,32 +2,16 @@ title: 'Aspects' --- +This page explains the basics and benefits of using [aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced examples. +Aspects allow augmenting build dependency graphs with additional information and actions. Some typical scenarios when aspects can be useful: -This page explains the basics and benefits of using -[aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced -examples. - -Aspects allow augmenting build dependency graphs with additional information -and actions. Some typical scenarios when aspects can be useful: - -* IDEs that integrate Bazel can use aspects to collect information about the - project. -* Code generation tools can leverage aspects to execute on their inputs in - *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy - of [protobuf](https://developers.google.com/protocol-buffers/) library - definitions, and language-specific rules can use aspects to attach - actions generating protobuf support code for a particular language. +- IDEs that integrate Bazel can use aspects to collect information about the project. +- Code generation tools can leverage aspects to execute on their inputs in *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy of [protobuf](https://developers.google.com/protocol-buffers/) library definitions, and language-specific rules can use aspects to attach actions generating protobuf support code for a particular language. ## Aspect basics -`BUILD` files provide a description of a project’s source code: what source -files are part of the project, what artifacts (_targets_) should be built from -those files, what the dependencies between those files are, etc. Bazel uses -this information to perform a build, that is, it figures out the set of actions -needed to produce the artifacts (such as running compiler or linker) and -executes those actions. Bazel accomplishes this by constructing a _dependency -graph_ between targets and visiting this graph to collect those actions. +`BUILD` files provide a description of a project’s source code: what source files are part of the project, what artifacts (*targets*) should be built from those files, what the dependencies between those files are, etc. Bazel uses this information to perform a build, that is, it figures out the set of actions needed to produce the artifacts (such as running compiler or linker) and executes those actions. Bazel accomplishes this by constructing a *dependency graph* between targets and visiting this graph to collect those actions. Consider the following `BUILD` file: @@ -46,41 +30,21 @@ This `BUILD` file defines a dependency graph shown in the following figure: **Figure 1.** `BUILD` file dependency graph. -Bazel analyzes this dependency graph by calling an implementation function of -the corresponding [rule](/extending/rules) (in this case "java_library") for every -target in the above example. Rule implementation functions generate actions that -build artifacts, such as `.jar` files, and pass information, such as locations -and names of those artifacts, to the reverse dependencies of those targets in -[providers](/extending/rules#providers). - -Aspects are similar to rules in that they have an implementation function that -generates actions and returns providers. However, their power comes from -the way the dependency graph is built for them. An aspect has an implementation -and a list of all attributes it propagates along. Consider an aspect A that -propagates along attributes named "deps". This aspect can be applied to -a target X, yielding an aspect application node A(X). During its application, -aspect A is applied recursively to all targets that X refers to in its "deps" -attribute (all attributes in A's propagation list). - -Thus a single act of applying aspect A to a target X yields a "shadow graph" of -the original dependency graph of targets shown in the following figure: +Bazel analyzes this dependency graph by calling an implementation function of the corresponding [rule](/extending/rules) (in this case "java\_library") for every target in the above example. Rule implementation functions generate actions that build artifacts, such as `.jar` files, and pass information, such as locations and names of those artifacts, to the reverse dependencies of those targets in [providers](/extending/rules#providers). + +Aspects are similar to rules in that they have an implementation function that generates actions and returns providers. However, their power comes from the way the dependency graph is built for them. An aspect has an implementation and a list of all attributes it propagates along. Consider an aspect A that propagates along attributes named "deps". This aspect can be applied to a target X, yielding an aspect application node A(X). During its application, aspect A is applied recursively to all targets that X refers to in its "deps" attribute (all attributes in A's propagation list). + +Thus a single act of applying aspect A to a target X yields a "shadow graph" of the original dependency graph of targets shown in the following figure: ![Build Graph with Aspect](/rules/build-graph-aspects.png "Build graph with aspects") **Figure 2.** Build graph with aspects. -The only edges that are shadowed are the edges along the attributes in -the propagation set, thus the `runtime_deps` edge is not shadowed in this -example. An aspect implementation function is then invoked on all nodes in -the shadow graph similar to how rule implementations are invoked on the nodes -of the original graph. +The only edges that are shadowed are the edges along the attributes in the propagation set, thus the `runtime_deps` edge is not shadowed in this example. An aspect implementation function is then invoked on all nodes in the shadow graph similar to how rule implementations are invoked on the nodes of the original graph. ## Simple example -This example demonstrates how to recursively print the source files for a -rule and all of its dependencies that have a `deps` attribute. It shows -an aspect implementation, an aspect definition, and how to invoke the aspect -from the Bazel command line. +This example demonstrates how to recursively print the source files for a rule and all of its dependencies that have a `deps` attribute. It shows an aspect implementation, an aspect definition, and how to invoke the aspect from the Bazel command line. ```python def _print_aspect_impl(target, ctx): @@ -111,25 +75,16 @@ print_aspect = aspect( required_providers = [CcInfo], ) ``` -Aspect definitions are similar to rule definitions, and defined using -the [`aspect`](/rules/lib/globals/bzl#aspect) function. -Just like a rule, an aspect has an implementation function which in this case is -``_print_aspect_impl``. +Aspect definitions are similar to rule definitions, and defined using the [`aspect`](/rules/lib/globals/bzl#aspect) function. + +Just like a rule, an aspect has an implementation function which in this case is `_print_aspect_impl`. -``attr_aspects`` is a list of rule attributes along which the aspect propagates. -In this case, the aspect will propagate along the ``deps`` attribute of the -rules that it is applied to. +`attr_aspects` is a list of rule attributes along which the aspect propagates. In this case, the aspect will propagate along the `deps` attribute of the rules that it is applied to. -Another common argument for `attr_aspects` is `['*']` which would propagate the -aspect to all attributes of a rule. +Another common argument for `attr_aspects` is `['*']` which would propagate the aspect to all attributes of a rule. -``required_providers`` is a list of providers that allows the aspect to limit -its propagation to only the targets whose rules advertise its required -providers. For more details consult -[the documentation of the aspect function](/rules/lib/globals/bzl#aspect). -In this case, the aspect will only apply on targets that declare `CcInfo` -provider. +`required_providers` is a list of providers that allows the aspect to limit its propagation to only the targets whose rules advertise its required providers. For more details consult [the documentation of the aspect function](/rules/lib/globals/bzl#aspect). In this case, the aspect will only apply on targets that declare `CcInfo` provider. ### Aspect implementation @@ -145,48 +100,32 @@ def _print_aspect_impl(target, ctx): return [] ``` -Aspect implementation functions are similar to the rule implementation -functions. They return [providers](/extending/rules#providers), can generate -[actions](/extending/rules#actions), and take two arguments: +Aspect implementation functions are similar to the rule implementation functions. They return [providers](/extending/rules#providers), can generate [actions](/extending/rules#actions), and take two arguments: -* `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. -* `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes - and generate outputs and actions. +- `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. +- `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes and generate outputs and actions. -The implementation function can access the attributes of the target rule via -[`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are -provided by the target to which it is applied (via the `target` argument). +The implementation function can access the attributes of the target rule via [`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are provided by the target to which it is applied (via the `target` argument). -Aspects are required to return a list of providers. In this example, the aspect -does not provide anything, so it returns an empty list. +Aspects are required to return a list of providers. In this example, the aspect does not provide anything, so it returns an empty list. ### Invoking the aspect using the command line -The simplest way to apply an aspect is from the command line using the -[`--aspects`](/reference/command-line-reference#flag--aspects) -argument. Assuming the aspect above were defined in a file named `print.bzl` -this: +The simplest way to apply an aspect is from the command line using the [`--aspects`](/reference/command-line-reference#flag--aspects) argument. Assuming the aspect above were defined in a file named `print.bzl` this: ```bash bazel build //MyExample:example --aspects print.bzl%print_aspect ``` -would apply the `print_aspect` to the target `example` and all of the -target rules that are accessible recursively via the `deps` attribute. +would apply the `print_aspect` to the target `example` and all of the target rules that are accessible recursively via the `deps` attribute. -The `--aspects` flag takes one argument, which is a specification of the aspect -in the format `%`. +The `--aspects` flag takes one argument, which is a specification of the aspect in the format `<extension file label>%<aspect top-level name>`. ## Advanced example -The following example demonstrates using an aspect from a target rule -that counts files in targets, potentially filtering them by extension. -It shows how to use a provider to return values, how to use parameters to pass -an argument into an aspect implementation, and how to invoke an aspect from a rule. +The following example demonstrates using an aspect from a target rule that counts files in targets, potentially filtering them by extension. It shows how to use a provider to return values, how to use parameters to pass an argument into an aspect implementation, and how to invoke an aspect from a rule. -Note: Aspects added in rules' attributes are called *rule-propagated aspects* as -opposed to *command-line aspects* that are specified using the ``--aspects`` -flag. +Note: Aspects added in rules' attributes are called *rule-propagated aspects* as opposed to *command-line aspects* that are specified using the `--aspects` flag. `file_count.bzl` file: @@ -274,28 +213,15 @@ file_count_aspect = aspect( ) ``` -This example shows how the aspect propagates through the ``deps`` attribute. +This example shows how the aspect propagates through the `deps` attribute. -``attrs`` defines a set of attributes for an aspect. Public aspect attributes -define parameters and can only be of types ``bool``, ``int`` or ``string``. -For rule-propagated aspects, ``int`` and ``string`` parameters must have -``values`` specified on them. This example has a parameter called ``extension`` -that is allowed to have '``*``', '``h``', or '``cc``' as a value. +`attrs` defines a set of attributes for an aspect. Public aspect attributes define parameters and can only be of types `bool`, `int` or `string`. For rule-propagated aspects, `int` and `string` parameters must have `values` specified on them. This example has a parameter called `extension` that is allowed to have '`*`', '`h`', or '`cc`' as a value. -For rule-propagated aspects, parameter values are taken from the rule requesting -the aspect, using the attribute of the rule that has the same name and type. -(see the definition of ``file_count_rule``). +For rule-propagated aspects, parameter values are taken from the rule requesting the aspect, using the attribute of the rule that has the same name and type. (see the definition of `file_count_rule`). -For command-line aspects, the parameters values can be passed using -[``--aspects_parameters``](/reference/command-line-reference#flag--aspects_parameters) -flag. The ``values`` restriction of ``int`` and ``string`` parameters may be -omitted. +For command-line aspects, the parameters values can be passed using [`--aspects_parameters`](/reference/command-line-reference#flag--aspects_parameters) flag. The `values` restriction of `int` and `string` parameters may be omitted. -Aspects are also allowed to have private attributes of types ``label`` or -``label_list``. Private label attributes can be used to specify dependencies on -tools or libraries that are needed for actions generated by aspects. There is not -a private attribute defined in this example, but the following code snippet -demonstrates how you could pass in a tool to an aspect: +Aspects are also allowed to have private attributes of types `label` or `label_list`. Private label attributes can be used to specify dependencies on tools or libraries that are needed for actions generated by aspects. There is not a private attribute defined in this example, but the following code snippet demonstrates how you could pass in a tool to an aspect: ```python ... @@ -333,40 +259,17 @@ def _file_count_aspect_impl(target, ctx): return [FileCountInfo(count = count)] ``` -Just like a rule implementation function, an aspect implementation function -returns a struct of providers that are accessible to its dependencies. - -In this example, the ``FileCountInfo`` is defined as a provider that has one -field ``count``. It is best practice to explicitly define the fields of a -provider using the ``fields`` attribute. - -The set of providers for an aspect application A(X) is the union of providers -that come from the implementation of a rule for target X and from the -implementation of aspect A. The providers that a rule implementation propagates -are created and frozen before aspects are applied and cannot be modified from an -aspect. It is an error if a target and an aspect that is applied to it each -provide a provider with the same type, with the exceptions of -[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) -(which is merged, so long as the -rule and aspect specify different output groups) and -[`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) -(which is taken from the aspect). This means that aspect implementations may -never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). - -The parameters and private attributes are passed in the attributes of the -``ctx``. This example references the ``extension`` parameter and determines -what files to count. - -For returning providers, the values of attributes along which -the aspect is propagated (from the `attr_aspects` list) are replaced with -the results of an application of the aspect to them. For example, if target -X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be [A(Y), A(Z)]. -In this example, ``ctx.rule.attr.deps`` are Target objects that are the -results of applying the aspect to the 'deps' of the original target to which -the aspect has been applied. - -In the example, the aspect accesses the ``FileCountInfo`` provider from the -target's dependencies to accumulate the total transitive number of files. +Just like a rule implementation function, an aspect implementation function returns a struct of providers that are accessible to its dependencies. + +In this example, the `FileCountInfo` is defined as a provider that has one field `count`. It is best practice to explicitly define the fields of a provider using the `fields` attribute. + +The set of providers for an aspect application A(X) is the union of providers that come from the implementation of a rule for target X and from the implementation of aspect A. The providers that a rule implementation propagates are created and frozen before aspects are applied and cannot be modified from an aspect. It is an error if a target and an aspect that is applied to it each provide a provider with the same type, with the exceptions of [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) (which is merged, so long as the rule and aspect specify different output groups) and [`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) (which is taken from the aspect). This means that aspect implementations may never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). + +The parameters and private attributes are passed in the attributes of the `ctx`. This example references the `extension` parameter and determines what files to count. + +For returning providers, the values of attributes along which the aspect is propagated (from the `attr_aspects` list) are replaced with the results of an application of the aspect to them. For example, if target X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be \[A(Y), A(Z)]. In this example, `ctx.rule.attr.deps` are Target objects that are the results of applying the aspect to the 'deps' of the original target to which the aspect has been applied. + +In the example, the aspect accesses the `FileCountInfo` provider from the target's dependencies to accumulate the total transitive number of files. ### Invoking the aspect from a rule @@ -384,13 +287,9 @@ file_count_rule = rule( ) ``` -The rule implementation demonstrates how to access the ``FileCountInfo`` -via the ``ctx.attr.deps``. +The rule implementation demonstrates how to access the `FileCountInfo` via the `ctx.attr.deps`. -The rule definition demonstrates how to define a parameter (``extension``) -and give it a default value (``*``). Note that having a default value that -was not one of '``cc``', '``h``', or '``*``' would be an error due to the -restrictions placed on the parameter in the aspect definition. +The rule definition demonstrates how to define a parameter (`extension`) and give it a default value (`*`). Note that having a default value that was not one of '`cc`', '`h`', or '`*`' would be an error due to the restrictions placed on the parameter in the aspect definition. ### Invoking an aspect through a target rule @@ -409,13 +308,10 @@ file_count_rule( ) ``` -This demonstrates how to pass the ``extension`` parameter into the aspect -via the rule. Since the ``extension`` parameter has a default value in the -rule implementation, ``extension`` would be considered an optional parameter. +This demonstrates how to pass the `extension` parameter into the aspect via the rule. Since the `extension` parameter has a default value in the rule implementation, `extension` would be considered an optional parameter. -When the ``file_count`` target is built, our aspect will be evaluated for -itself, and all of the targets accessible recursively via ``deps``. +When the `file_count` target is built, our aspect will be evaluated for itself, and all of the targets accessible recursively via `deps`. ## References -* [`aspect` API reference](/rules/lib/globals/bzl#aspect) +- [`aspect` API reference](/rules/lib/globals/bzl#aspect) diff --git a/extending/auto-exec-groups.mdx b/extending/auto-exec-groups.mdx index 9fee9783..72af2c3a 100644 --- a/extending/auto-exec-groups.mdx +++ b/extending/auto-exec-groups.mdx @@ -2,18 +2,11 @@ title: 'Automatic Execution Groups (AEGs)' --- - - -Automatic execution groups select an [execution platform][exec_platform] -for each toolchain type. In other words, one target can have multiple -execution platforms without defining execution groups. +Automatic execution groups select an [execution platform](https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs.) for each toolchain type. In other words, one target can have multiple execution platforms without defining execution groups. ## Quick summary -Automatic execution groups are closely connected to toolchains. If you are using -toolchains, you need to set them on the affected actions (actions which use an -executable or a tool from a toolchain) by adding `toolchain` parameter. For -example: +Automatic execution groups are closely connected to toolchains. If you are using toolchains, you need to set them on the affected actions (actions which use an executable or a tool from a toolchain) by adding `toolchain` parameter. For example: ```python ctx.actions.run( @@ -23,15 +16,10 @@ ctx.actions.run( toolchain = '@bazel_tools//tools/jdk:toolchain_type', ) ``` -If the action does not use a tool or executable from a toolchain, and Blaze -doesn't detect that ([the error](#first-error-message) is raised), you can set -`toolchain = None`. -If you need to use multiple toolchains on a single execution platform (an action -uses executable or tools from two or more toolchains), you need to manually -define [exec_groups][exec_groups] (check -[When should I use a custom exec_group?][multiple_toolchains_exec_groups] -section). +If the action does not use a tool or executable from a toolchain, and Blaze doesn't detect that ([the error](#first-error-message) is raised), you can set `toolchain = None`. + +If you need to use multiple toolchains on a single execution platform (an action uses executable or tools from two or more toolchains), you need to manually define [exec\_groups](https://bazel.build/extending/exec-groups) (check [When should I use a custom exec\_group?](/extending/auto-exec-groups#when-should-use-exec-groups) section). ## History @@ -44,20 +32,11 @@ my_rule = rule( ) ``` -Rule `my_rule` registers two toolchain types. This means that the [Toolchain -Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used -to find an execution platform which supports both toolchain types. The selected -execution platform was used for each registered action inside the rule, unless -specified differently with [exec_groups][exec_groups]. -In other words, all actions inside the rule used to have a single execution -platform even if they used tools from different toolchains (execution platform -is selected for each target). This resulted in failures when there was no -execution platform supporting all toolchains. +Rule `my_rule` registers two toolchain types. This means that the [Toolchain Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used to find an execution platform which supports both toolchain types. The selected execution platform was used for each registered action inside the rule, unless specified differently with [exec\_groups](https://bazel.build/extending/exec-groups). In other words, all actions inside the rule used to have a single execution platform even if they used tools from different toolchains (execution platform is selected for each target). This resulted in failures when there was no execution platform supporting all toolchains. ## Current state -With AEGs, the execution platform is selected for each toolchain type. The -implementation function of the earlier example, `my_rule`, would look like: +With AEGs, the execution platform is selected for each toolchain type. The implementation function of the earlier example, `my_rule`, would look like: ```python def _impl(ctx): @@ -74,29 +53,17 @@ def _impl(ctx): ) ``` -This rule creates two actions, the `First action` which uses executable from a -`//tools:toolchain_type_1` and the `Second action` which uses executable from a -`//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed -on a single execution platform which supports both toolchain types. With AEGs, -by adding the `toolchain` parameter inside the actions, each action executes on -the execution platform that provides the toolchain. The actions may be executed -on different execution platforms. +This rule creates two actions, the `First action` which uses executable from a `//tools:toolchain_type_1` and the `Second action` which uses executable from a `//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed on a single execution platform which supports both toolchain types. With AEGs, by adding the `toolchain` parameter inside the actions, each action executes on the execution platform that provides the toolchain. The actions may be executed on different execution platforms. -The same is effective with [ctx.actions.run_shell][run_shell] where `toolchain` -parameter should be added when `tools` are from a toolchain. +The same is effective with [ctx.actions.run\_shell](https://bazel.build/rules/lib/builtins/actions#run_shell) where `toolchain` parameter should be added when `tools` are from a toolchain. ## Difference between custom exec groups and automatic exec groups -As the name suggests, AEGs are exec groups created automatically for each -toolchain type registered on a rule. There is no need to manually specify them, -unlike the "classic" exec groups. Moreover, name of AEG is automatically set to -its toolchain type (e.g. `//tools:toolchain_type_1`). +As the name suggests, AEGs are exec groups created automatically for each toolchain type registered on a rule. There is no need to manually specify them, unlike the "classic" exec groups. Moreover, name of AEG is automatically set to its toolchain type (e.g. `//tools:toolchain_type_1`). -### When should I use a custom exec_group? +### When should I use a custom exec\_group? -Custom exec_groups are needed only in case where multiple toolchains need to -execute on a single execution platform. In all other cases there's no need to -define custom exec_groups. For example: +Custom exec\_groups are needed only in case where multiple toolchains need to execute on a single execution platform. In all other cases there's no need to define custom exec\_groups. For example: ```python def _impl(ctx): @@ -121,9 +88,7 @@ my_rule = rule( ## Migration of AEGs -Internally in google3, Blaze is already using AEGs. -Externally for Bazel, migration is in the process. Some rules are already using -this feature (e.g. Java and C++ rules). +Internally in google3, Blaze is already using AEGs. Externally for Bazel, migration is in the process. Some rules are already using this feature (e.g. Java and C++ rules). ### Which Bazel versions support this migration? @@ -131,8 +96,7 @@ AEGs are fully supported from Bazel 7. ### How to enable AEGs? -Set `--incompatible_auto_exec_groups` to true. More information about the flag -on [the GitHub issue][github_flag]. +Set `--incompatible_auto_exec_groups` to true. More information about the flag on [the GitHub issue](https://github.com/bazelbuild/bazel/issues/17134). ### How to enable AEGs inside a particular rule? @@ -146,38 +110,23 @@ my_rule = rule( } ) ``` -This enables AEGs only in `my_rule` and its actions start using the new logic -when selecting the execution platform. Incompatible flag is overridden with this -attribute. + +This enables AEGs only in `my_rule` and its actions start using the new logic when selecting the execution platform. Incompatible flag is overridden with this attribute. ### How to disable AEGs in case of an error? -Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in -your project ([flag's GitHub issue][github_flag]), or disable a particular rule -by setting `_use_auto_exec_groups` attribute to `False` -([more details about the attribute](#how-enable-particular-rule)). +Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in your project ([flag's GitHub issue](https://github.com/bazelbuild/bazel/issues/17134)), or disable a particular rule by setting `_use_auto_exec_groups` attribute to `False` ([more details about the attribute](#how-enable-particular-rule)). ### Error messages while migrating to AEGs #### Couldn't identify if tools are from implicit dependencies or a toolchain. Please set the toolchain parameter. If you're not using a toolchain, set it to 'None'. - * In this case you get a stack of calls before the error happened and you can - clearly see which exact action needs the toolchain parameter. Check which - toolchain is used for the action and set it with the toolchain param. If no - toolchain is used inside the action for tools or executable, set it to - `None`. -#### Action declared for non-existent toolchain '[toolchain_type]'. - * This means that you've set the toolchain parameter on the action but didn't -register it on the rule. Register the toolchain or set `None` inside the action. +- In this case you get a stack of calls before the error happened and you can clearly see which exact action needs the toolchain parameter. Check which toolchain is used for the action and set it with the toolchain param. If no toolchain is used inside the action for tools or executable, set it to `None`. -## Additional material +#### Action declared for non-existent toolchain '\[toolchain\_type]'. + +- This means that you've set the toolchain parameter on the action but didn't register it on the rule. Register the toolchain or set `None` inside the action. -For more information, check design document: -[Automatic exec groups for toolchains][aegs_design_doc]. +## Additional material -[exec_platform]: https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs. -[exec_groups]: https://bazel.build/extending/exec-groups -[github_flag]: https://github.com/bazelbuild/bazel/issues/17134 -[aegs_design_doc]: https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch -[run_shell]: https://bazel.build/rules/lib/builtins/actions#run_shell -[multiple_toolchains_exec_groups]: /extending/auto-exec-groups#when-should-use-exec-groups +For more information, check design document: [Automatic exec groups for toolchains](https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch). diff --git a/extending/concepts.mdx b/extending/concepts.mdx index e634c611..da9fbd1d 100644 --- a/extending/concepts.mdx +++ b/extending/concepts.mdx @@ -2,111 +2,62 @@ title: 'Extension Overview' --- +This page describes how to extend the BUILD language using macros and rules. - -{/* [TOC] */} - -This page describes how to extend the BUILD language using macros -and rules. - -Bazel extensions are files ending in `.bzl`. Use a -[load statement](/concepts/build-files#load) to import a symbol from an extension. +Bazel extensions are files ending in `.bzl`. Use a [load statement](/concepts/build-files#load) to import a symbol from an extension. Before learning the more advanced concepts, first: -* Read about the [Starlark language](/rules/language), used in both the - `BUILD` and `.bzl` files. +- Read about the [Starlark language](/rules/language), used in both the `BUILD` and `.bzl` files. -* Learn how you can [share variables](/build/share-variables) - between two `BUILD` files. +- Learn how you can [share variables](/build/share-variables) between two `BUILD` files. ## Macros and rules -A macro is a function that instantiates rules. Macros come in two flavors: -[symbolic macros](/extending/macros) (new in Bazel 8) and [legacy -macros](/extending/legacy-macros). The two flavors of macros are defined -differently, but behave almost the same from the point of view of a user. A -macro is useful when a `BUILD` file is getting too repetitive or too complex, as -it lets you reuse some code. The function is evaluated as soon as the `BUILD` -file is read. After the evaluation of the `BUILD` file, Bazel has little -information about macros. If your macro generates a `genrule`, Bazel will -behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one -exception is that targets declared in a symbolic macro have [special visibility -semantics](/extending/macros#visibility): a symbolic macro can hide its internal -targets from the rest of the package.) - -A [rule](/extending/rules) is more powerful than a macro. It can access Bazel -internals and have full control over what is going on. It may for example pass -information to other rules. - -If you want to reuse simple logic, start with a macro; we recommend a symbolic -macro, unless you need to support older Bazel versions. If a macro becomes -complex, it is often a good idea to make it a rule. Support for a new language -is typically done with a rule. Rules are for advanced users, and most users will -never have to write one; they will only load and call existing rules. +A macro is a function that instantiates rules. Macros come in two flavors: [symbolic macros](/extending/macros) (new in Bazel 8) and [legacy macros](/extending/legacy-macros). The two flavors of macros are defined differently, but behave almost the same from the point of view of a user. A macro is useful when a `BUILD` file is getting too repetitive or too complex, as it lets you reuse some code. The function is evaluated as soon as the `BUILD` file is read. After the evaluation of the `BUILD` file, Bazel has little information about macros. If your macro generates a `genrule`, Bazel will behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one exception is that targets declared in a symbolic macro have [special visibility semantics](/extending/macros#visibility): a symbolic macro can hide its internal targets from the rest of the package.) + +A [rule](/extending/rules) is more powerful than a macro. It can access Bazel internals and have full control over what is going on. It may for example pass information to other rules. + +If you want to reuse simple logic, start with a macro; we recommend a symbolic macro, unless you need to support older Bazel versions. If a macro becomes complex, it is often a good idea to make it a rule. Support for a new language is typically done with a rule. Rules are for advanced users, and most users will never have to write one; they will only load and call existing rules. ## Evaluation model A build consists of three phases. -* **Loading phase**. First, load and evaluate all extensions and all `BUILD` - files that are needed for the build. The execution of the `BUILD` files simply - instantiates rules (each time a rule is called, it gets added to a graph). - This is where macros are evaluated. - -* **Analysis phase**. The code of the rules is executed (their `implementation` - function), and actions are instantiated. An action describes how to generate - a set of outputs from a set of inputs, such as "run gcc on hello.c and get - hello.o". You must list explicitly which files will be generated before - executing the actual commands. In other words, the analysis phase takes - the graph generated by the loading phase and generates an action graph. - -* **Execution phase**. Actions are executed, when at least one of their outputs is - required. If a file is missing or if a command fails to generate one output, - the build fails. Tests are also run during this phase. - -Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` -files. A file is read at most once per build and the result of the evaluation is -cached and reused. A file is evaluated only once all its dependencies (`load()` -statements) have been resolved. By design, loading a `.bzl` file has no visible -side-effect, it only defines values and functions. - -Bazel tries to be clever: it uses dependency analysis to know which files must -be loaded, which rules must be analyzed, and which actions must be executed. For -example, if a rule generates actions that you don't need for the current build, -they will not be executed. +- **Loading phase**. First, load and evaluate all extensions and all `BUILD` files that are needed for the build. The execution of the `BUILD` files simply instantiates rules (each time a rule is called, it gets added to a graph). This is where macros are evaluated. + +- **Analysis phase**. The code of the rules is executed (their `implementation` function), and actions are instantiated. An action describes how to generate a set of outputs from a set of inputs, such as "run gcc on hello.c and get hello.o". You must list explicitly which files will be generated before executing the actual commands. In other words, the analysis phase takes the graph generated by the loading phase and generates an action graph. + +- **Execution phase**. Actions are executed, when at least one of their outputs is required. If a file is missing or if a command fails to generate one output, the build fails. Tests are also run during this phase. + +Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` files. A file is read at most once per build and the result of the evaluation is cached and reused. A file is evaluated only once all its dependencies (`load()` statements) have been resolved. By design, loading a `.bzl` file has no visible side-effect, it only defines values and functions. + +Bazel tries to be clever: it uses dependency analysis to know which files must be loaded, which rules must be analyzed, and which actions must be executed. For example, if a rule generates actions that you don't need for the current build, they will not be executed. ## Creating extensions -* [Create your first macro](/rules/macro-tutorial) in order to reuse some code. - Then [learn more about macros](/extending/macros) and [using them to create - "custom verbs"](/rules/verbs-tutorial). +- [Create your first macro](/rules/macro-tutorial) in order to reuse some code. Then [learn more about macros](/extending/macros) and [using them to create "custom verbs"](/rules/verbs-tutorial). -* [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. - Next, you can read more about the [rules concepts](/extending/rules). +- [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. Next, you can read more about the [rules concepts](/extending/rules). -The two links below will be very useful when writing your own extensions. Keep -them within reach: +The two links below will be very useful when writing your own extensions. Keep them within reach: -* The [API reference](/rules/lib) +- The [API reference](/rules/lib) -* [Examples](https://github.com/bazelbuild/examples/tree/master/rules) +- [Examples](https://github.com/bazelbuild/examples/tree/master/rules) ## Going further -In addition to [macros](/extending/macros) and [rules](/extending/rules), you -may want to write [aspects](/extending/aspects) and [repository -rules](/external/repo). +In addition to [macros](/extending/macros) and [rules](/extending/rules), you may want to write [aspects](/extending/aspects) and [repository rules](/external/repo). -* Use [Buildifier](https://github.com/bazelbuild/buildtools) - consistently to format and lint your code. +- Use [Buildifier](https://github.com/bazelbuild/buildtools) consistently to format and lint your code. -* Follow the [`.bzl` style guide](/rules/bzl-style). +- Follow the [`.bzl` style guide](/rules/bzl-style). -* [Test](/rules/testing) your code. +- [Test](/rules/testing) your code. -* [Generate documentation](https://skydoc.bazel.build/) to help your users. +- [Generate documentation](https://skydoc.bazel.build/) to help your users. -* [Optimize the performance](/rules/performance) of your code. +- [Optimize the performance](/rules/performance) of your code. -* [Deploy](/rules/deploying) your extensions to other people. +- [Deploy](/rules/deploying) your extensions to other people. diff --git a/extending/config.mdx b/extending/config.mdx new file mode 100644 index 00000000..9b1ddc23 --- /dev/null +++ b/extending/config.mdx @@ -0,0 +1,631 @@ +--- +title: 'Configurations' +--- + +This page covers the benefits and basic usage of Starlark configurations, Bazel's API for customizing how your project builds. It includes how to define build settings and provides examples. + +This makes it possible to: + +- define custom flags for your project, obsoleting the need for [`--define`](/docs/configurable-attributes#custom-keys) +- write [transitions](/rules/lib/builtins/transition#transition) to configure deps in different configurations than their parents (such as `--compilation_mode=opt` or `--cpu=arm`) +- bake better defaults into rules (such as automatically build `//my:android_app` with a specified SDK) + +and more, all completely from .bzl files (no Bazel release required). See the `bazelbuild/examples` repo for [examples](https://github.com/bazelbuild/examples/tree/HEAD/configurations). + +## User-defined build settings + +A build setting is a single piece of [configuration](/extending/rules#configurations) information. Think of a configuration as a key/value map. Setting `--cpu=ppc` and `--copt="-DFoo"` produces a configuration that looks like `{cpu: ppc, copt: "-DFoo"}`. Each entry is a build setting. + +Traditional flags like `cpu` and `copt` are native settings — their keys are defined and their values are set inside native bazel java code. Bazel users can only read and write them via the command line and other APIs maintained natively. Changing native flags, and the APIs that expose them, requires a bazel release. User-defined build settings are defined in `.bzl` files (and thus, don't need a bazel release to register changes). They also can be set via the command line (if they're designated as `flags`, see more below), but can also be set via [user-defined transitions](#user-defined-transitions). + +### Defining build settings + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/basic_build_setting) + +#### The `build_setting` `rule()` parameter + +Build settings are rules like any other rule and are differentiated using the Starlark `rule()` function's `build_setting` [attribute](/rules/lib/globals/bzl#rule.build_setting). + +```python +# example/buildsettings/build_settings.bzl +string_flag = rule( + implementation = _impl, + build_setting = config.string(flag = True) +) +``` + +The `build_setting` attribute takes a function that designates the type of the build setting. The type is limited to a set of basic Starlark types like `bool` and `string`. See the `config` module [documentation](/rules/lib/toplevel/config) for details. More complicated typing can be done in the rule's implementation function. More on this below. + +The `config` module's functions takes an optional boolean parameter, `flag`, which is set to false by default. if `flag` is set to true, the build setting can be set on the command line by users as well as internally by rule writers via default values and [transitions](/rules/lib/builtins/transition#transition). Not all settings should be settable by users. For example, if you as a rule writer have some debug mode that you'd like to turn on inside test rules, you don't want to give users the ability to indiscriminately turn on that feature inside other non-test rules. + +#### Using ctx.build\_setting\_value + +Like all rules, build setting rules have [implementation functions](/extending/rules#implementation-function). The basic Starlark-type value of the build settings can be accessed via the `ctx.build_setting_value` method. This method is only available to [`ctx`](/rules/lib/builtins/ctx) objects of build setting rules. These implementation methods can directly forward the build settings value or do additional work on it, like type checking or more complex struct creation. Here's how you would implement an `enum`-typed build setting: + +```python +# example/buildsettings/build_settings.bzl +TemperatureProvider = provider(fields = ['type']) + +temperatures = ["HOT", "LUKEWARM", "ICED"] + +def _impl(ctx): + raw_temperature = ctx.build_setting_value + if raw_temperature not in temperatures: + fail(str(ctx.label) + " build setting allowed to take values {" + + ", ".join(temperatures) + "} but was set to unallowed value " + + raw_temperature) + return TemperatureProvider(type = raw_temperature) + +temperature = rule( + implementation = _impl, + build_setting = config.string(flag = True) +) +``` + +Note: if a rule depends on a build setting, it will receive whatever providers the build setting implementation function returns, like any other dependency. But all other references to the value of the build setting (such as in transitions) will see its basic Starlark-typed value, not this post implementation function value. + +#### Defining multi-set string flags + +String settings have an additional `allow_multiple` parameter which allows the flag to be set multiple times on the command line or in bazelrcs. Their default value is still set with a string-typed attribute: + +```python +# example/buildsettings/build_settings.bzl +allow_multiple_flag = rule( + implementation = _impl, + build_setting = config.string(flag = True, allow_multiple = True) +) +``` + +```python +# example/BUILD +load("//example/buildsettings:build_settings.bzl", "allow_multiple_flag") +allow_multiple_flag( + name = "roasts", + build_setting_default = "medium" +) +``` + +Each setting of the flag is treated as a single value: + +```shell +$ bazel build //my/target --//example:roasts=blonde \ + --//example:roasts=medium,dark +``` + +The above is parsed to `{"//example:roasts": ["blonde", "medium,dark"]}` and `ctx.build_setting_value` returns the list `["blonde", "medium,dark"]`. + +#### Instantiating build settings + +Rules defined with the `build_setting` parameter have an implicit mandatory `build_setting_default` attribute. This attribute takes on the same type as declared by the `build_setting` param. + +```python +# example/buildsettings/build_settings.bzl +FlavorProvider = provider(fields = ['type']) + +def _impl(ctx): + return FlavorProvider(type = ctx.build_setting_value) + +flavor = rule( + implementation = _impl, + build_setting = config.string(flag = True) +) +``` + +```python +# example/BUILD +load("//example/buildsettings:build_settings.bzl", "flavor") +flavor( + name = "favorite_flavor", + build_setting_default = "APPLE" +) +``` + +### Predefined settings + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/use_skylib_build_setting) + +The [Skylib](https://github.com/bazelbuild/bazel-skylib) library includes a set of predefined settings you can instantiate without having to write custom Starlark. + +For example, to define a setting that accepts a limited set of string values: + +```python +# example/BUILD +load("@bazel_skylib//rules:common_settings.bzl", "string_flag") +string_flag( + name = "myflag", + values = ["a", "b", "c"], + build_setting_default = "a", +) +``` + +For a complete list, see [Common build setting rules](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/common_settings.bzl). + +### Using build settings + +#### Depending on build settings + +If a target would like to read a piece of configuration information, it can directly depend on the build setting via a regular attribute dependency. + +```python +# example/rules.bzl +load("//example/buildsettings:build_settings.bzl", "FlavorProvider") +def _rule_impl(ctx): + if ctx.attr.flavor[FlavorProvider].type == "ORANGE": + ... + +drink_rule = rule( + implementation = _rule_impl, + attrs = { + "flavor": attr.label() + } +) +``` + +```python +# example/BUILD +load("//example:rules.bzl", "drink_rule") +load("//example/buildsettings:build_settings.bzl", "flavor") +flavor( + name = "favorite_flavor", + build_setting_default = "APPLE" +) +drink_rule( + name = "my_drink", + flavor = ":favorite_flavor", +) +``` + +Languages may wish to create a canonical set of build settings which all rules for that language depend on. Though the native concept of `fragments` no longer exists as a hardcoded object in Starlark configuration world, one way to translate this concept would be to use sets of common implicit attributes. For example: + +```python +# kotlin/rules.bzl +_KOTLIN_CONFIG = { + "_compiler": attr.label(default = "//kotlin/config:compiler-flag"), + "_mode": attr.label(default = "//kotlin/config:mode-flag"), + ... +} + +... + +kotlin_library = rule( + implementation = _rule_impl, + attrs = dicts.add({ + "library-attr": attr.string() + }, _KOTLIN_CONFIG) +) + +kotlin_binary = rule( + implementation = _binary_impl, + attrs = dicts.add({ + "binary-attr": attr.label() + }, _KOTLIN_CONFIG) +``` + +#### Using build settings on the command line + +Similar to most native flags, you can use the command line to set build settings [that are marked as flags](#rule-parameter). The build setting's name is its full target path using `name=value` syntax: + +```shell +$ bazel build //my/target --//example:string_flag=some-value # allowed +$ bazel build //my/target --//example:string_flag some-value # not allowed +``` + +Special boolean syntax is supported: + +```shell +$ bazel build //my/target --//example:boolean_flag +$ bazel build //my/target --no//example:boolean_flag +``` + +#### Using build setting aliases + +You can set an alias for your build setting target path to make it easier to read on the command line. Aliases function similarly to native flags and also make use of the double-dash option syntax. + +Set an alias by adding `--flag_alias=ALIAS_NAME=TARGET_PATH` to your `.bazelrc` . For example, to set an alias to `coffee`: + +```shell +# .bazelrc +build --flag_alias=coffee=//experimental/user/starlark_configurations/basic_build_setting:coffee-temp +``` + +Best Practice: Setting an alias multiple times results in the most recent one taking precedence. Use unique alias names to avoid unintended parsing results. + +To make use of the alias, type it in place of the build setting target path. With the above example of `coffee` set in the user's `.bazelrc`: + +```shell +$ bazel build //my/target --coffee=ICED +``` + +instead of + +```shell +$ bazel build //my/target --//experimental/user/starlark_configurations/basic_build_setting:coffee-temp=ICED +``` + +Best Practice: While it possible to set aliases on the command line, leaving them in a `.bazelrc` reduces command line clutter. + +### Label-typed build settings + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/label_typed_build_setting) + +Unlike other build settings, label-typed settings cannot be defined using the `build_setting` rule parameter. Instead, bazel has two built-in rules: `label_flag` and `label_setting`. These rules forward the providers of the actual target to which the build setting is set. `label_flag` and `label_setting` can be read/written by transitions and `label_flag` can be set by the user like other `build_setting` rules can. Their only difference is they can't customely defined. + +Label-typed settings will eventually replace the functionality of late-bound defaults. Late-bound default attributes are Label-typed attributes whose final values can be affected by configuration. In Starlark, this will replace the [`configuration_field`](/rules/lib/globals/bzl#configuration_field) API. + +```python +# example/rules.bzl +MyProvider = provider(fields = ["my_field"]) + +def _dep_impl(ctx): + return MyProvider(my_field = "yeehaw") + +dep_rule = rule( + implementation = _dep_impl +) + +def _parent_impl(ctx): + if ctx.attr.my_field_provider[MyProvider].my_field == "cowabunga": + ... + +parent_rule = rule( + implementation = _parent_impl, + attrs = { "my_field_provider": attr.label() } +) +``` + +```python +# example/BUILD +load("//example:rules.bzl", "dep_rule", "parent_rule") + +dep_rule(name = "dep") + +parent_rule(name = "parent", my_field_provider = ":my_field_provider") + +label_flag( + name = "my_field_provider", + build_setting_default = ":dep" +) +``` + +### Build settings and select() + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) + +Users can configure attributes on build settings by using [`select()`](/reference/be/functions#select). Build setting targets can be passed to the `flag_values` attribute of `config_setting`. The value to match to the configuration is passed as a `String` then parsed to the type of the build setting for matching. + +```python +config_setting( + name = "my_config", + flag_values = { + "//example:favorite_flavor": "MANGO" + } +) +``` + +## User-defined transitions + +A configuration [transition](/rules/lib/builtins/transition#transition) maps the transformation from one configured target to another within the build graph. + +Important: Transitions have [memory and performance impact](#memory-performance-considerations). + +### Defining + +Transitions define configuration changes between rules. For example, a request like "compile my dependency for a different CPU than its parent" is handled by a transition. + +Formally, a transition is a function from an input configuration to one or more output configurations. Most transitions are 1:1 such as "override the input configuration with `--cpu=ppc`". 1:2+ transitions can also exist but come with special restrictions. + +In Starlark, transitions are defined much like rules, with a defining `transition()` [function](/rules/lib/builtins/transition#transition) and an implementation function. + +```python +# example/transitions/transitions.bzl +def _impl(settings, attr): + _ignore = (settings, attr) + return {"//example:favorite_flavor" : "MINT"} + +hot_chocolate_transition = transition( + implementation = _impl, + inputs = [], + outputs = ["//example:favorite_flavor"] +) +``` + +The `transition()` function takes in an implementation function, a set of build settings to read(`inputs`), and a set of build settings to write (`outputs`). The implementation function has two parameters, `settings` and `attr`. `settings` is a dictionary \{`String`:`Object`\} of all settings declared in the `inputs` parameter to `transition()`. + +`attr` is a dictionary of attributes and values of the rule to which the transition is attached. When attached as an [outgoing edge transition](#outgoing-edge-transitions), the values of these attributes are all configured post-select() resolution. When attached as an [incoming edge transition](#incoming-edge-transitions), `attr` does not include any attributes that use a selector to resolve their value. If an incoming edge transition on `--foo` reads attribute `bar` and then also selects on `--foo` to set attribute `bar`, then there's a chance for the incoming edge transition to read the wrong value of `bar` in the transition. + +Note: Since transitions are attached to rule definitions and `select()`s are attached to rule instantiations (such as targets), errors related to `select()`s on read attributes will pop up when users create targets rather than when rules are written. It may be worth taking extra care to communicate to rule users which attributes they should be wary of selecting on or taking other precautions. + +The implementation function must return a dictionary (or list of dictionaries, in the case of transitions with multiple output configurations) of new build settings values to apply. The returned dictionary keyset(s) must contain exactly the set of build settings passed to the `outputs` parameter of the transition function. This is true even if a build setting is not actually changed over the course of the transition - its original value must be explicitly passed through in the returned dictionary. + +### Defining 1:2+ transitions + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/multi_arch_binary) + +[Outgoing edge transition](#outgoing-edge-transitions) can map a single input configuration to two or more output configurations. This is useful for defining rules that bundle multi-architecture code. + +1:2+ transitions are defined by returning a list of dictionaries in the transition implementation function. + +```python +# example/transitions/transitions.bzl +def _impl(settings, attr): + _ignore = (settings, attr) + return [ + {"//example:favorite_flavor" : "LATTE"}, + {"//example:favorite_flavor" : "MOCHA"}, + ] + +coffee_transition = transition( + implementation = _impl, + inputs = [], + outputs = ["//example:favorite_flavor"] +) +``` + +They can also set custom keys that the rule implementation function can use to read individual dependencies: + +```python +# example/transitions/transitions.bzl +def _impl(settings, attr): + _ignore = (settings, attr) + return { + "Apple deps": {"//command_line_option:cpu": "ppc"}, + "Linux deps": {"//command_line_option:cpu": "x86"}, + } + +multi_arch_transition = transition( + implementation = _impl, + inputs = [], + outputs = ["//command_line_option:cpu"] +) +``` + +### Attaching transitions + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/attaching_transitions_to_rules) + +Transitions can be attached in two places: incoming edges and outgoing edges. Effectively this means rules can transition their own configuration (incoming edge transition) and transition their dependencies' configurations (outgoing edge transition). + +NOTE: There is currently no way to attach Starlark transitions to native rules. If you need to do this, contact [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com) for help with figuring out workarounds. + +### Incoming edge transitions + +Incoming edge transitions are activated by attaching a `transition` object (created by `transition()`) to `rule()`'s `cfg` parameter: + +```python +# example/rules.bzl +load("example/transitions:transitions.bzl", "hot_chocolate_transition") +drink_rule = rule( + implementation = _impl, + cfg = hot_chocolate_transition, + ... +``` + +Incoming edge transitions must be 1:1 transitions. + +### Outgoing edge transitions + +Outgoing edge transitions are activated by attaching a `transition` object (created by `transition()`) to an attribute's `cfg` parameter: + +```python +# example/rules.bzl +load("example/transitions:transitions.bzl", "coffee_transition") +drink_rule = rule( + implementation = _impl, + attrs = { "dep": attr.label(cfg = coffee_transition)} + ... +``` + +Outgoing edge transitions can be 1:1 or 1:2+. + +See [Accessing attributes with transitions](#accessing-attributes-with-transitions) for how to read these keys. + +### Transitions on native options + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/transition_on_native_flag) + +Starlark transitions can also declare reads and writes on native build configuration options via a special prefix to the option name. + +```python +# example/transitions/transitions.bzl +def _impl(settings, attr): + _ignore = (settings, attr) + return {"//command_line_option:cpu": "k8"} + +cpu_transition = transition( + implementation = _impl, + inputs = [], + outputs = ["//command_line_option:cpu"] +``` + +#### Unsupported native options + +Bazel doesn't support transitioning on `--define` with `"//command_line_option:define"`. Instead, use a custom [build setting](#user-defined-build-settings). In general, new usages of `--define` are discouraged in favor of build settings. + +Bazel doesn't support transitioning on `--config`. This is because `--config` is an "expansion" flag that expands to other flags. + +Crucially, `--config` may include flags that don't affect build configuration, such as [`--spawn_strategy`](/docs/user-manual#spawn-strategy) . Bazel, by design, can't bind such flags to individual targets. This means there's no coherent way to apply them in transitions. + +As a workaround, you can explicitly itemize the flags that *are* part of the configuration in your transition. This requires maintaining the `--config`'s expansion in two places, which is a known UI blemish. + +### Transitions on allow multiple build settings + +When setting build settings that [allow multiple values](#defining-multi-set-string-flags), the value of the setting must be set with a list. + +```python +# example/buildsettings/build_settings.bzl +string_flag = rule( + implementation = _impl, + build_setting = config.string(flag = True, allow_multiple = True) +) +``` + +```python +# example/BUILD +load("//example/buildsettings:build_settings.bzl", "string_flag") +string_flag(name = "roasts", build_setting_default = "medium") +``` + +```python +# example/transitions/rules.bzl +def _transition_impl(settings, attr): + # Using a value of just "dark" here will throw an error + return {"//example:roasts" : ["dark"]}, + +coffee_transition = transition( + implementation = _transition_impl, + inputs = [], + outputs = ["//example:roasts"] +) +``` + +### No-op transitions + +If a transition returns `{}`, `[]`, or `None`, this is shorthand for keeping all settings at their original values. This can be more convenient than explicitly setting each output to itself. + +```python +# example/transitions/transitions.bzl +def _impl(settings, attr): + _ignore = (attr) + if settings["//example:already_chosen"] is True: + return {} + return { + "//example:favorite_flavor": "dark chocolate", + "//example:include_marshmallows": "yes", + "//example:desired_temperature": "38C", + } + +hot_chocolate_transition = transition( + implementation = _impl, + inputs = ["//example:already_chosen"], + outputs = [ + "//example:favorite_flavor", + "//example:include_marshmallows", + "//example:desired_temperature", + ] +) +``` + +### Accessing attributes with transitions + +[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/read_attr_in_transition) + +When [attaching a transition to an outgoing edge](#outgoing-edge-transitions) (regardless of whether the transition is a 1:1 or 1:2+ transition), `ctx.attr` is forced to be a list if it isn't already. The order of elements in this list is unspecified. + +```python +# example/transitions/rules.bzl +def _transition_impl(settings, attr): + return {"//example:favorite_flavor" : "LATTE"}, + +coffee_transition = transition( + implementation = _transition_impl, + inputs = [], + outputs = ["//example:favorite_flavor"] +) + +def _rule_impl(ctx): + # Note: List access even though "dep" is not declared as list + transitioned_dep = ctx.attr.dep[0] + + # Note: Access doesn't change, other_deps was already a list + for other_dep in ctx.attr.other_deps: + # ... + +coffee_rule = rule( + implementation = _rule_impl, + attrs = { + "dep": attr.label(cfg = coffee_transition) + "other_deps": attr.label_list(cfg = coffee_transition) + }) +``` + +If the transition is `1:2+` and sets custom keys, `ctx.split_attr` can be used to read individual deps for each key: + +```python +# example/transitions/rules.bzl +def _impl(settings, attr): + _ignore = (settings, attr) + return { + "Apple deps": {"//command_line_option:cpu": "ppc"}, + "Linux deps": {"//command_line_option:cpu": "x86"}, + } + +multi_arch_transition = transition( + implementation = _impl, + inputs = [], + outputs = ["//command_line_option:cpu"] +) + +def _rule_impl(ctx): + apple_dep = ctx.split_attr.dep["Apple deps"] + linux_dep = ctx.split_attr.dep["Linux deps"] + # ctx.attr has a list of all deps for all keys. Order is not guaranteed. + all_deps = ctx.attr.dep + +multi_arch_rule = rule( + implementation = _rule_impl, + attrs = { + "dep": attr.label(cfg = multi_arch_transition) + }) +``` + +See [complete example](https://github.com/bazelbuild/examples/tree/main/configurations/multi_arch_binary) here. + +## Integration with platforms and toolchains + +Many native flags today, like `--cpu` and `--crosstool_top` are related to toolchain resolution. In the future, explicit transitions on these types of flags will likely be replaced by transitioning on the [target platform](/extending/platforms). + +## Memory and performance considerations + +Adding transitions, and therefore new configurations, to your build comes at a cost: larger build graphs, less comprehensible build graphs, and slower builds. It's worth considering these costs when considering using transitions in your build rules. Below is an example of how a transition might create exponential growth of your build graph. + +### Badly behaved builds: a case study + +![Scalability graph](/rules/scalability-graph.png "Scalability graph") + +**Figure 1.** Scalability graph showing a top level target and its dependencies. + +This graph shows a top level target, `//pkg:app`, which depends on two targets, a `//pkg:1_0` and `//pkg:1_1`. Both these targets depend on two targets, `//pkg:2_0` and `//pkg:2_1`. Both these targets depend on two targets, `//pkg:3_0` and `//pkg:3_1`. This continues on until `//pkg:n_0` and `//pkg:n_1`, which both depend on a single target, `//pkg:dep`. + +Building `//pkg:app` requires \\(2n+2\\) targets: + +- `//pkg:app` +- `//pkg:dep` +- `//pkg:i_0` and `//pkg:i_1` for \\(i\\) in \\(\[1..n]\\) + +Imagine you [implement](#user-defined-build-settings) a flag `--//foo:owner=<STRING>` and `//pkg:i_b` applies + +``` +depConfig = myConfig + depConfig.owner="$(myConfig.owner)$(b)" +``` + +In other words, `//pkg:i_b` appends `b` to the old value of `--owner` for all its deps. + +This produces the following [configured targets](/reference/glossary#configured-target): + +``` +//pkg:app //foo:owner="" +//pkg:1_0 //foo:owner="" +//pkg:1_1 //foo:owner="" +//pkg:2_0 (via //pkg:1_0) //foo:owner="0" +//pkg:2_0 (via //pkg:1_1) //foo:owner="1" +//pkg:2_1 (via //pkg:1_0) //foo:owner="0" +//pkg:2_1 (via //pkg:1_1) //foo:owner="1" +//pkg:3_0 (via //pkg:1_0 → //pkg:2_0) //foo:owner="00" +//pkg:3_0 (via //pkg:1_0 → //pkg:2_1) //foo:owner="01" +//pkg:3_0 (via //pkg:1_1 → //pkg:2_0) //foo:owner="10" +//pkg:3_0 (via //pkg:1_1 → //pkg:2_1) //foo:owner="11" +... +``` + +`//pkg:dep` produces \\(2^n\\) configured targets: `config.owner=` "\\(b\_0b\_1...b\_n\\)" for all \\(b\_i\\) in \\(\{0,1\}\\). + +This makes the build graph exponentially larger than the target graph, with corresponding memory and performance consequences. + +TODO: Add strategies for measurement and mitigation of these issues. + +## Further reading + +For more details on modifying build configurations, see: + +- [Starlark Build Configuration](https://docs.google.com/document/d/1vc8v-kXjvgZOdQdnxPTaV0rrLxtP2XwnD2tAZlYJOqw/edit?usp=sharing) +- Full [set](https://github.com/bazelbuild/examples/tree/HEAD/configurations) of end to end examples diff --git a/extending/depsets.mdx b/extending/depsets.mdx index 4e84d3f2..1f0d18c4 100644 --- a/extending/depsets.mdx +++ b/extending/depsets.mdx @@ -2,40 +2,21 @@ title: 'Depsets' --- +[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently collecting data across a target’s transitive dependencies. They are an essential element of rule processing. - -[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently -collecting data across a target’s transitive dependencies. They are an essential -element of rule processing. - -The defining feature of depset is its time- and space-efficient union operation. -The depset constructor accepts a list of elements ("direct") and a list of other -depsets ("transitive"), and returns a depset representing a set containing all the -direct elements and the union of all the transitive sets. Conceptually, the -constructor creates a new graph node that has the direct and transitive nodes -as its successors. Depsets have a well-defined ordering semantics, based on -traversal of this graph. +The defining feature of depset is its time- and space-efficient union operation. The depset constructor accepts a list of elements ("direct") and a list of other depsets ("transitive"), and returns a depset representing a set containing all the direct elements and the union of all the transitive sets. Conceptually, the constructor creates a new graph node that has the direct and transitive nodes as its successors. Depsets have a well-defined ordering semantics, based on traversal of this graph. Example uses of depsets include: -* Storing the paths of all object files for a program’s libraries, which can - then be passed to a linker action through a provider. +- Storing the paths of all object files for a program’s libraries, which can then be passed to a linker action through a provider. -* For an interpreted language, storing the transitive source files that are - included in an executable's runfiles. +- For an interpreted language, storing the transitive source files that are included in an executable's runfiles. ## Description and operations -Conceptually, a depset is a directed acyclic graph (DAG) that typically looks -similar to the target graph. It is constructed from the leaves up to the root. -Each target in a dependency chain can add its own contents on top of the -previous without having to read or copy them. +Conceptually, a depset is a directed acyclic graph (DAG) that typically looks similar to the target graph. It is constructed from the leaves up to the root. Each target in a dependency chain can add its own contents on top of the previous without having to read or copy them. -Each node in the DAG holds a list of direct elements and a list of child nodes. -The contents of the depset are the transitive elements, such as the direct elements -of all the nodes. A new depset can be created using the -[depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct -elements and another list of child nodes. +Each node in the DAG holds a list of direct elements and a list of child nodes. The contents of the depset are the transitive elements, such as the direct elements of all the nodes. A new depset can be created using the [depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct elements and another list of child nodes. ```python s = depset(["a", "b", "c"]) @@ -45,11 +26,7 @@ print(s) # depset(["a", "b", "c"]) print(t) # depset(["d", "e", "a", "b", "c"]) ``` -To retrieve the contents of a depset, use the -[to_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive -elements, not including duplicates. There is no way to directly inspect the -precise structure of the DAG, although this structure does affect the order in -which the elements are returned. +To retrieve the contents of a depset, use the [to\_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive elements, not including duplicates. There is no way to directly inspect the precise structure of the DAG, although this structure does affect the order in which the elements are returned. ```python s = depset(["a", "b", "c"]) @@ -58,11 +35,9 @@ print("c" in s.to_list()) # True print(s.to_list() == ["a", "b", "c"]) # True ``` -The allowed items in a depset are restricted, just as the allowed keys in -dictionaries are restricted. In particular, depset contents may not be mutable. +The allowed items in a depset are restricted, just as the allowed keys in dictionaries are restricted. In particular, depset contents may not be mutable. -Depsets use reference equality: a depset is equal to itself, but unequal to any -other depset, even if they have the same contents and same internal structure. +Depsets use reference equality: a depset is equal to itself, but unequal to any other depset, even if they have the same contents and same internal structure. ```python s = depset(["a", "b", "c"]) @@ -86,9 +61,7 @@ t = depset(["c", "b", "a"]) print(sorted(s.to_list()) == sorted(t.to_list())) # True ``` -There is no ability to remove elements from a depset. If this is needed, you -must read out the entire contents of the depset, filter the elements you want to -remove, and reconstruct a new depset. This is not particularly efficient. +There is no ability to remove elements from a depset. If this is needed, you must read out the entire contents of the depset, filter the elements you want to remove, and reconstruct a new depset. This is not particularly efficient. ```python s = depset(["a", "b", "c"]) @@ -105,24 +78,9 @@ print(s) # depset(["a"]) ### Order -The `to_list` operation performs a traversal over the DAG. The kind of traversal -depends on the *order* that was specified at the time the depset was -constructed. It is useful for Bazel to support multiple orders because sometimes -tools care about the order of their inputs. For example, a linker action may -need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the -linker’s command line. Other tools might have the opposite requirement. - -Three traversal orders are supported: `postorder`, `preorder`, and -`topological`. The first two work exactly like [tree -traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) -except that they operate on DAGs and skip already visited nodes. The third order -works as a topological sort from root to leaves, essentially the same as -preorder except that shared children are listed only after all of their parents. -Preorder and postorder operate as left-to-right traversals, but note that within -each node direct elements have no order relative to children. For topological -order, there is no left-to-right guarantee, and even the -all-parents-before-child guarantee does not apply in the case that there are -duplicate elements in different nodes of the DAG. +The `to_list` operation performs a traversal over the DAG. The kind of traversal depends on the *order* that was specified at the time the depset was constructed. It is useful for Bazel to support multiple orders because sometimes tools care about the order of their inputs. For example, a linker action may need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the linker’s command line. Other tools might have the opposite requirement. + +Three traversal orders are supported: `postorder`, `preorder`, and `topological`. The first two work exactly like [tree traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) except that they operate on DAGs and skip already visited nodes. The third order works as a topological sort from root to leaves, essentially the same as preorder except that shared children are listed only after all of their parents. Preorder and postorder operate as left-to-right traversals, but note that within each node direct elements have no order relative to children. For topological order, there is no left-to-right guarantee, and even the all-parents-before-child guarantee does not apply in the case that there are duplicate elements in different nodes of the DAG. ```python # This demonstrates different traversal orders. @@ -151,20 +109,13 @@ print(create("preorder").to_list()) # ["d", "b", "a", "c"] print(create("topological").to_list()) # ["d", "b", "c", "a"] ``` -Due to how traversals are implemented, the order must be specified at the time -the depset is created with the constructor’s `order` keyword argument. If this -argument is omitted, the depset has the special `default` order, in which case -there are no guarantees about the order of any of its elements (except that it -is deterministic). +Due to how traversals are implemented, the order must be specified at the time the depset is created with the constructor’s `order` keyword argument. If this argument is omitted, the depset has the special `default` order, in which case there are no guarantees about the order of any of its elements (except that it is deterministic). ## Full example -This example is available at -[https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). +This example is available at [https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). -Suppose there is a hypothetical interpreted language Foo. In order to build -each `foo_binary` you need to know all the `*.foo` files that it directly or -indirectly depends on. +Suppose there is a hypothetical interpreted language Foo. In order to build each `foo_binary` you need to know all the `*.foo` files that it directly or indirectly depends on. ```python # //depsets:BUILD @@ -208,21 +159,14 @@ foo_binary( import sys if __name__ == "__main__": - assert len(sys.argv) >= 1 + assert len(sys.argv) >= 1 output = open(sys.argv[1], "wt") for path in sys.argv[2:]: input = open(path, "rt") output.write(input.read()) ``` -Here, the transitive sources of the binary `d` are all of the `*.foo` files in -the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` -target to know about any file besides `d.foo`, the `foo_library` targets need to -pass them along in a provider. Each library receives the providers from its own -dependencies, adds its own immediate sources, and passes on a new provider with -the augmented contents. The `foo_binary` rule does the same, except that instead -of returning a provider, it uses the complete list of sources to construct a -command line for an action. +Here, the transitive sources of the binary `d` are all of the `*.foo` files in the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` target to know about any file besides `d.foo`, the `foo_library` targets need to pass them along in a provider. Each library receives the providers from its own dependencies, adds its own immediate sources, and passes on a new provider with the augmented contents. The `foo_binary` rule does the same, except that instead of returning a provider, it uses the complete list of sources to construct a command line for an action. Here’s a complete implementation of the `foo_library` and `foo_binary` rules. @@ -279,15 +223,11 @@ foo_binary = rule( ) ``` -You can test this by copying these files into a fresh package, renaming the -labels appropriately, creating the source `*.foo` files with dummy content, and -building the `d` target. - +You can test this by copying these files into a fresh package, renaming the labels appropriately, creating the source `*.foo` files with dummy content, and building the `d` target. ## Performance -To see the motivation for using depsets, consider what would happen if -`get_transitive_srcs()` collected its sources in a list. +To see the motivation for using depsets, consider what would happen if `get_transitive_srcs()` collected its sources in a list. ```python def get_transitive_srcs(srcs, deps): @@ -298,12 +238,9 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This does not take into account duplicates, so the source files for `a` -will appear twice on the command line and twice in the contents of the output -file. +This does not take into account duplicates, so the source files for `a` will appear twice on the command line and twice in the contents of the output file. -An alternative is using a general set, which can be simulated by a -dictionary where the keys are the elements and all the keys map to `True`. +An alternative is using a general set, which can be simulated by a dictionary where the keys are the elements and all the keys map to `True`. ```python def get_transitive_srcs(srcs, deps): @@ -316,31 +253,16 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This gets rid of the duplicates, but it makes the order of the command line -arguments (and therefore the contents of the files) unspecified, although still -deterministic. +This gets rid of the duplicates, but it makes the order of the command line arguments (and therefore the contents of the files) unspecified, although still deterministic. -Moreover, both approaches are asymptotically worse than the depset-based -approach. Consider the case where there is a long chain of dependencies on -Foo libraries. Processing every rule requires copying all of the transitive -sources that came before it into a new data structure. This means that the -time and space cost for analyzing an individual library or binary target -is proportional to its own height in the chain. For a chain of length n, -foolib_1 ← foolib_2 ← … ← foolib_n, the overall cost is effectively O(n^2). +Moreover, both approaches are asymptotically worse than the depset-based approach. Consider the case where there is a long chain of dependencies on Foo libraries. Processing every rule requires copying all of the transitive sources that came before it into a new data structure. This means that the time and space cost for analyzing an individual library or binary target is proportional to its own height in the chain. For a chain of length n, foolib\_1 ← foolib\_2 ← … ← foolib\_n, the overall cost is effectively O(n^2). -Generally speaking, depsets should be used whenever you are accumulating -information through your transitive dependencies. This helps ensure that -your build scales well as your target graph grows deeper. +Generally speaking, depsets should be used whenever you are accumulating information through your transitive dependencies. This helps ensure that your build scales well as your target graph grows deeper. -Finally, it’s important to not retrieve the contents of the depset -unnecessarily in rule implementations. One call to `to_list()` -at the end in a binary rule is fine, since the overall cost is just O(n). It’s -when many non-terminal targets try to call `to_list()` that quadratic behavior -occurs. +Finally, it’s important to not retrieve the contents of the depset unnecessarily in rule implementations. One call to `to_list()` at the end in a binary rule is fine, since the overall cost is just O(n). It’s when many non-terminal targets try to call `to_list()` that quadratic behavior occurs. For more information about using depsets efficiently, see the [performance](/rules/performance) page. ## API Reference Please see [here](/rules/lib/builtins/depset) for more details. - diff --git a/extending/exec-groups.mdx b/extending/exec-groups.mdx index a8b45b92..6f7e7198 100644 --- a/extending/exec-groups.mdx +++ b/extending/exec-groups.mdx @@ -2,40 +2,21 @@ title: 'Execution Groups' --- - - -Execution groups allow for multiple execution platforms within a single target. -Each execution group has its own [toolchain](/extending/toolchains) dependencies and -performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). +Execution groups allow for multiple execution platforms within a single target. Each execution group has its own [toolchain](/extending/toolchains) dependencies and performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). ## Current status -Execution groups for certain natively declared actions, like `CppLink`, can be -used inside `exec_properties` to set per-action, per-target execution -requirements. For more details, see the -[Default execution groups](#exec-groups-for-native-rules) section. +Execution groups for certain natively declared actions, like `CppLink`, can be used inside `exec_properties` to set per-action, per-target execution requirements. For more details, see the [Default execution groups](#exec-groups-for-native-rules) section. ## Background -Execution groups allow the rule author to define sets of actions, each with a -potentially different execution platform. Multiple execution platforms can allow -actions to execution differently, for example compiling an iOS app on a remote -(linux) worker and then linking/code signing on a local mac worker. +Execution groups allow the rule author to define sets of actions, each with a potentially different execution platform. Multiple execution platforms can allow actions to execution differently, for example compiling an iOS app on a remote (linux) worker and then linking/code signing on a local mac worker. -Being able to define groups of actions also helps alleviate the usage of action -mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be -unique and can only reference a single action. This is especially helpful in -allocating extra resources to specific memory and processing intensive actions -like linking in C++ builds without over-allocating to less demanding tasks. +Being able to define groups of actions also helps alleviate the usage of action mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be unique and can only reference a single action. This is especially helpful in allocating extra resources to specific memory and processing intensive actions like linking in C++ builds without over-allocating to less demanding tasks. ## Defining execution groups -During rule definition, rule authors can -[declare](/rules/lib/globals/bzl#exec_group) -a set of execution groups. On each execution group, the rule author can specify -everything needed to select an execution platform for that execution group, -namely any constraints via `exec_compatible_with` and toolchain types via -`toolchain`. +During rule definition, rule authors can [declare](/rules/lib/globals/bzl#exec_group) a set of execution groups. On each execution group, the rule author can specify everything needed to select an execution platform for that execution group, namely any constraints via `exec_compatible_with` and toolchain types via `toolchain`. ```python # foo.bzl @@ -56,25 +37,13 @@ my_rule = rule( ) ``` -In the code snippet above, you can see that tool dependencies can also specify -transition for an exec group using the -[`cfg`](/rules/lib/toplevel/attr#label) -attribute param and the -[`config`](/rules/lib/toplevel/config) -module. The module exposes an `exec` function which takes a single string -parameter which is the name of the exec group for which the dependency should be -built. +In the code snippet above, you can see that tool dependencies can also specify transition for an exec group using the [`cfg`](/rules/lib/toplevel/attr#label) attribute param and the [`config`](/rules/lib/toplevel/config) module. The module exposes an `exec` function which takes a single string parameter which is the name of the exec group for which the dependency should be built. -As on native rules, the `test` execution group is present by default on Starlark -test rules. +As on native rules, the `test` execution group is present by default on Starlark test rules. ## Accessing execution groups -In the rule implementation, you can declare that actions should be run on the -execution platform of an execution group. You can do this by using the `exec_group` -param of action generating methods, specifically [`ctx.actions.run`] -(/rules/lib/builtins/actions#run) and -[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). +In the rule implementation, you can declare that actions should be run on the execution platform of an execution group. You can do this by using the `exec_group` param of action generating methods, specifically \[`ctx.actions.run`] (/rules/lib/builtins/actions#run) and [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). ```python # foo.bzl @@ -86,9 +55,7 @@ def _impl(ctx): ) ``` -Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) -of execution groups, similarly to how you -can access the resolved toolchain of a target: +Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) of execution groups, similarly to how you can access the resolved toolchain of a target: ```python # foo.bzl @@ -101,28 +68,18 @@ def _impl(ctx): ) ``` -Note: If an action uses a toolchain from an execution group, but doesn't specify -that execution group in the action declaration, that may potentially cause -issues. A mismatch like this may not immediately cause failures, but is a latent -problem. +Note: If an action uses a toolchain from an execution group, but doesn't specify that execution group in the action declaration, that may potentially cause issues. A mismatch like this may not immediately cause failures, but is a latent problem. ### Default execution groups The following execution groups are predefined: -* `test`: Test runner actions (for more details, see - the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). -* `cpp_link`: C++ linking actions. +- `test`: Test runner actions (for more details, see the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). +- `cpp_link`: C++ linking actions. ## Using execution groups to set execution properties -Execution groups are integrated with the -[`exec_properties`](/reference/be/common-definitions#common-attributes) -attribute that exists on every rule and allows the target writer to specify a -string dict of properties that is then passed to the execution machinery. For -example, if you wanted to set some property, say memory, for the target and give -certain actions a higher memory allocation, you would write an `exec_properties` -entry with an execution-group-augmented key, such as: +Execution groups are integrated with the [`exec_properties`](/reference/be/common-definitions#common-attributes) attribute that exists on every rule and allows the target writer to specify a string dict of properties that is then passed to the execution machinery. For example, if you wanted to set some property, say memory, for the target and give certain actions a higher memory allocation, you would write an `exec_properties` entry with an execution-group-augmented key, such as: ```python # BUILD @@ -136,24 +93,13 @@ my_rule( ) ``` -All actions with `exec_group = "link"` would see the exec properties -dictionary as `{"mem": "16g"}`. As you see here, execution-group-level -settings override target-level settings. +All actions with `exec_group = "link"` would see the exec properties dictionary as `{"mem": "16g"}`. As you see here, execution-group-level settings override target-level settings. ## Using execution groups to set platform constraints -Execution groups are also integrated with the -[`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and -[`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) -attributes that exist on every rule and allow the target writer to specify -additional constraints that must be satisfied by the execution platforms -selected for the target's actions. +Execution groups are also integrated with the [`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and [`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) attributes that exist on every rule and allow the target writer to specify additional constraints that must be satisfied by the execution platforms selected for the target's actions. -For example, if the rule `my_test` defines the `link` execution group in -addition to the default and the `test` execution group, then the following -usage of these attributes would run actions in the default execution group on -a platform with a high number of CPUs, the test action on Linux, and the link -action on the default execution platform: +For example, if the rule `my_test` defines the `link` execution group in addition to the default and the `test` execution group, then the following usage of these attributes would run actions in the default execution group on a platform with a high number of CPUs, the test action on Linux, and the link action on the default execution platform: ```python # BUILD @@ -180,23 +126,16 @@ my_test( ### Execution groups for native rules -The following execution groups are available for actions defined by native -rules: +The following execution groups are available for actions defined by native rules: -* `test`: Test runner actions. -* `cpp_link`: C++ linking actions. +- `test`: Test runner actions. +- `cpp_link`: C++ linking actions. ### Execution groups and platform execution properties -It is possible to define `exec_properties` for arbitrary execution groups on -platform targets (unlike `exec_properties` set directly on a target, where -properties for unknown execution groups are rejected). Targets then inherit the -execution platform's `exec_properties` that affect the default execution group -and any other relevant execution groups. +It is possible to define `exec_properties` for arbitrary execution groups on platform targets (unlike `exec_properties` set directly on a target, where properties for unknown execution groups are rejected). Targets then inherit the execution platform's `exec_properties` that affect the default execution group and any other relevant execution groups. -For example, suppose running tests on the exec platform requires some resource -to be available, but it isn't required for compiling and linking; this can be -modelled as follows: +For example, suppose running tests on the exec platform requires some resource to be available, but it isn't required for compiling and linking; this can be modelled as follows: ```python constraint_setting(name = "resource") @@ -217,5 +156,4 @@ cc_test( ) ``` -`exec_properties` defined directly on targets take precedence over those that -are inherited from the execution platform. +`exec_properties` defined directly on targets take precedence over those that are inherited from the execution platform. diff --git a/extending/legacy-macros.mdx b/extending/legacy-macros.mdx index 9c2b8e28..910cc46a 100644 --- a/extending/legacy-macros.mdx +++ b/extending/legacy-macros.mdx @@ -2,12 +2,7 @@ title: 'Legacy Macros' --- - - -Legacy macros are unstructured functions called from `BUILD` files that can -create targets. By the end of the -[loading phase](/extending/concepts#evaluation-model), legacy macros don't exist -anymore, and Bazel sees only the concrete set of instantiated rules. +Legacy macros are unstructured functions called from `BUILD` files that can create targets. By the end of the [loading phase](/extending/concepts#evaluation-model), legacy macros don't exist anymore, and Bazel sees only the concrete set of instantiated rules. ## Why you shouldn't use legacy macros (and should use Symbolic macros instead) @@ -15,38 +10,30 @@ Where possible you should use [symbolic macros](macros.md#macros). Symbolic macros -* Prevent action at a distance -* Make it possible to hide implementation details through granular visibility -* Take typed attributes, which in turn means automatic label and select - conversion. -* Are more readable -* Will soon have [lazy evaluation](macros.md#laziness) +- Prevent action at a distance +- Make it possible to hide implementation details through granular visibility +- Take typed attributes, which in turn means automatic label and select conversion. +- Are more readable +- Will soon have [lazy evaluation](macros.md#laziness) ## Usage The typical use case for a macro is when you want to reuse a rule. -For example, genrule in a `BUILD` file generates a file using `//:generator` -with a `some_arg` argument hardcoded in the command: +For example, genrule in a `BUILD` file generates a file using `//:generator` with a `some_arg` argument hardcoded in the command: ```python genrule( name = "file", outs = ["file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", tools = ["//:generator"], ) ``` -Note: `$@` is a -[Make variable](/reference/be/make-variables#predefined_genrule_variables) that -refers to the execution-time locations of the files in the `outs` attribute -list. It is equivalent to `$(locations :file.txt)`. +Note: `$@` is a [Make variable](/reference/be/make-variables#predefined_genrule_variables) that refers to the execution-time locations of the files in the `outs` attribute list. It is equivalent to `$(locations :file.txt)`. -If you want to generate more files with different arguments, you may want to -extract this code to a macro function. To create a macro called -`file_generator`, which has `name` and `arg` parameters, we can replace the -genrule with the following: +If you want to generate more files with different arguments, you may want to extract this code to a macro function. To create a macro called `file_generator`, which has `name` and `arg` parameters, we can replace the genrule with the following: ```python load("//path:generator.bzl", "file_generator") @@ -67,27 +54,22 @@ file_generator( ) ``` -Here, you load the `file_generator` symbol from a `.bzl` file located in the -`//path` package. By putting macro function definitions in a separate `.bzl` -file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be -loaded from any package in the workspace. +Here, you load the `file_generator` symbol from a `.bzl` file located in the `//path` package. By putting macro function definitions in a separate `.bzl` file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be loaded from any package in the workspace. -Finally, in `path/generator.bzl`, write the definition of the macro to -encapsulate and parameterize the original genrule definition: +Finally, in `path/generator.bzl`, write the definition of the macro to encapsulate and parameterize the original genrule definition: ```python def file_generator(name, arg, visibility=None): native.genrule( name = name, outs = [name + ".txt"], - cmd = "$(location //:generator) %s > $@" % arg, + cmd = "$(location //:generator) %s > $@" % arg, tools = ["//:generator"], visibility = visibility, ) ``` -You can also use macros to chain rules together. This example shows chained -genrules, where a genrule uses the outputs of a previous genrule as inputs: +You can also use macros to chain rules together. This example shows chained genrules, where a genrule uses the outputs of a previous genrule as inputs: ```python def chained_genrules(name, visibility=None): @@ -103,23 +85,19 @@ def chained_genrules(name, visibility=None): name = name + "-two", srcs = [name + ".one"], outs = [name + ".two"], - cmd = "$(location :tool-two) $< $@", + cmd = "$(location :tool-two) $< $@", tools = [":tool-two"], visibility = visibility, ) ``` -The example only assigns a visibility value to the second genrule. This allows -macro authors to hide the outputs of intermediate rules from being depended upon -by other targets in the workspace. +The example only assigns a visibility value to the second genrule. This allows macro authors to hide the outputs of intermediate rules from being depended upon by other targets in the workspace. -Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the -`srcs` attribute list. +Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the `srcs` attribute list. ## Expanding macros -When you want to investigate what a macro does, use the `query` command with -`--output=build` to see the expanded form: +When you want to investigate what a macro does, use the `query` command with `--output=build` to see the expanded form: ```none $ bazel query --output=build :file @@ -128,14 +106,13 @@ genrule( name = "file", tools = ["//:generator"], outs = ["//test:file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", ) ``` ## Instantiating native rules -Native rules (rules that don't need a `load()` statement) can be instantiated -from the [native](/rules/lib/toplevel/native) module: +Native rules (rules that don't need a `load()` statement) can be instantiated from the [native](/rules/lib/toplevel/native) module: ```python def my_macro(name, visibility=None): @@ -146,23 +123,13 @@ def my_macro(name, visibility=None): ) ``` -If you need to know the package name (for example, which `BUILD` file is calling -the macro), use the function -[native.package_name()](/rules/lib/toplevel/native#package_name). Note that -`native` can only be used in `.bzl` files, and not in `BUILD` files. +If you need to know the package name (for example, which `BUILD` file is calling the macro), use the function [native.package\_name()](/rules/lib/toplevel/native#package_name). Note that `native` can only be used in `.bzl` files, and not in `BUILD` files. ## Label resolution in macros -Since legacy macros are evaluated in the -[loading phase](concepts.md#evaluation-model), label strings such as -`"//foo:bar"` that occur in a legacy macro are interpreted relative to the -`BUILD` file in which the macro is used rather than relative to the `.bzl` file -in which it is defined. This behavior is generally undesirable for macros that -are meant to be used in other repositories, such as because they are part of a -published Starlark ruleset. +Since legacy macros are evaluated in the [loading phase](concepts.md#evaluation-model), label strings such as `"//foo:bar"` that occur in a legacy macro are interpreted relative to the `BUILD` file in which the macro is used rather than relative to the `.bzl` file in which it is defined. This behavior is generally undesirable for macros that are meant to be used in other repositories, such as because they are part of a published Starlark ruleset. -To get the same behavior as for Starlark rules, wrap the label strings with the -[`Label`](/rules/lib/builtins/Label#Label) constructor: +To get the same behavior as for Starlark rules, wrap the label strings with the [`Label`](/rules/lib/builtins/Label#Label) constructor: ```python # @my_ruleset//rules:defs.bzl @@ -184,71 +151,39 @@ def my_cc_wrapper(name, deps = [], **kwargs): ) ``` -With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys -that are label strings will be automatically resolved to `Label` objects -relative to the package of the file that contains the `select` call. If this is -not chosen, wrap the label string with -[native.package_relative_label()](/rules/lib/toplevel/native#package_relative_label). +With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys that are label strings will be automatically resolved to `Label` objects relative to the package of the file that contains the `select` call. If this is not chosen, wrap the label string with [native.package\_relative\_label()](/rules/lib/toplevel/native#package_relative_label). ## Debugging -* `bazel query --output=build //my/path:all` will show you how the `BUILD` - file looks after evaluation. All legacy macros, globs, loops are expanded. - Known limitation: `select` expressions are not shown in the output. +- `bazel query --output=build //my/path:all` will show you how the `BUILD` file looks after evaluation. All legacy macros, globs, loops are expanded. Known limitation: `select` expressions are not shown in the output. -* You may filter the output based on `generator_function` (which function - generated the rules) or `generator_name` (the name attribute of the macro): - `bash $ bazel query --output=build 'attr(generator_function, my_macro, - //my/path:all)'` +- You may filter the output based on `generator_function` (which function generated the rules) or `generator_name` (the name attribute of the macro): `bash $ bazel query --output=build 'attr(generator_function, my_macro, //my/path:all)'` -* To find out where exactly the rule `foo` is generated in a `BUILD` file, you - can try the following trick. Insert this line near the top of the `BUILD` - file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when - the rule `foo` is created (due to a name conflict), which will show you the - full stack trace. +- To find out where exactly the rule `foo` is generated in a `BUILD` file, you can try the following trick. Insert this line near the top of the `BUILD` file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when the rule `foo` is created (due to a name conflict), which will show you the full stack trace. -* You can also use [print](/rules/lib/globals/all#print) for debugging. It - displays the message as a `DEBUG` log line during the loading phase. Except - in rare cases, either remove `print` calls, or make them conditional under a - `debugging` parameter that defaults to `False` before submitting the code to - the depot. +- You can also use [print](/rules/lib/globals/all#print) for debugging. It displays the message as a `DEBUG` log line during the loading phase. Except in rare cases, either remove `print` calls, or make them conditional under a `debugging` parameter that defaults to `False` before submitting the code to the depot. ## Errors -If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) -function. Explain clearly to the user what went wrong and how to fix their -`BUILD` file. It is not possible to catch an error. +If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) function. Explain clearly to the user what went wrong and how to fix their `BUILD` file. It is not possible to catch an error. ```python def my_macro(name, deps, visibility=None): - if len(deps) < 2: + if len(deps) < 2: fail("Expected at least two values in deps") # ... ``` ## Conventions -* All public functions (functions that don't start with underscore) that - instantiate rules must have a `name` argument. This argument should not be - optional (don't give a default value). +- All public functions (functions that don't start with underscore) that instantiate rules must have a `name` argument. This argument should not be optional (don't give a default value). -* Public functions should use a docstring following - [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). +- Public functions should use a docstring following [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). -* In `BUILD` files, the `name` argument of the macros must be a keyword - argument (not a positional argument). +- In `BUILD` files, the `name` argument of the macros must be a keyword argument (not a positional argument). -* The `name` attribute of rules generated by a macro should include the name - argument as a prefix. For example, `macro(name = "foo")` can generate a - `cc_library` `foo` and a genrule `foo_gen`. +- The `name` attribute of rules generated by a macro should include the name argument as a prefix. For example, `macro(name = "foo")` can generate a `cc_library` `foo` and a genrule `foo_gen`. -* In most cases, optional parameters should have a default value of `None`. - `None` can be passed directly to native rules, which treat it the same as if - you had not passed in any argument. Thus, there is no need to replace it - with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer - to the rules it creates, as their defaults may be complex or may change over - time. Additionally, a parameter that is explicitly set to its default value - looks different than one that is never set (or set to `None`) when accessed - through the query language or build-system internals. +- In most cases, optional parameters should have a default value of `None`. `None` can be passed directly to native rules, which treat it the same as if you had not passed in any argument. Thus, there is no need to replace it with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer to the rules it creates, as their defaults may be complex or may change over time. Additionally, a parameter that is explicitly set to its default value looks different than one that is never set (or set to `None`) when accessed through the query language or build-system internals. -* Macros should have an optional `visibility` argument. +- Macros should have an optional `visibility` argument. diff --git a/extending/macros.mdx b/extending/macros.mdx index 136665d0..1e4ac1e3 100644 --- a/extending/macros.mdx +++ b/extending/macros.mdx @@ -2,42 +2,23 @@ title: 'Macros' --- +This page covers the basics of using macros and includes typical use cases, debugging, and conventions. +A macro is a function called from the `BUILD` file that can instantiate rules. Macros are mainly used for encapsulation and code reuse of existing rules and other macros. -This page covers the basics of using macros and includes typical use cases, -debugging, and conventions. +Macros come in two flavors: symbolic macros, which are described on this page, and [legacy macros](legacy-macros.md). Where possible, we recommend using symbolic macros for code clarity. -A macro is a function called from the `BUILD` file that can instantiate rules. -Macros are mainly used for encapsulation and code reuse of existing rules and -other macros. +Symbolic macros offer typed arguments (string to label conversion, relative to where the macro was called) and the ability to restrict and specify the visibility of targets created. They are designed to be amenable to lazy evaluation (which will be added in a future Bazel release). Symbolic macros are available by default in Bazel 8. Where this document mentions `macros`, it's referring to **symbolic macros**. -Macros come in two flavors: symbolic macros, which are described on this page, -and [legacy macros](legacy-macros.md). Where possible, we recommend using -symbolic macros for code clarity. - -Symbolic macros offer typed arguments (string to label conversion, relative to -where the macro was called) and the ability to restrict and specify the -visibility of targets created. They are designed to be amenable to lazy -evaluation (which will be added in a future Bazel release). Symbolic macros are -available by default in Bazel 8. Where this document mentions `macros`, it's -referring to **symbolic macros**. - -An executable example of symbolic macros can be found in the -[examples repository](https://github.com/bazelbuild/examples/tree/main/macros). +An executable example of symbolic macros can be found in the [examples repository](https://github.com/bazelbuild/examples/tree/main/macros). ## Usage -Macros are defined in `.bzl` files by calling the -[`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with -two required parameters: `attrs` and `implementation`. +Macros are defined in `.bzl` files by calling the [`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with two required parameters: `attrs` and `implementation`. ### Attributes -`attrs` accepts a dictionary of attribute name to [attribute -types](https://bazel.build/rules/lib/toplevel/attr#members), which represents -the arguments to the macro. Two common attributes – `name` and `visibility` – -are implicitly added to all macros and are not included in the dictionary passed -to `attrs`. +`attrs` accepts a dictionary of attribute name to [attribute types](https://bazel.build/rules/lib/toplevel/attr#members), which represents the arguments to the macro. Two common attributes – `name` and `visibility` – are implicitly added to all macros and are not included in the dictionary passed to `attrs`. ```starlark # macro/macro.bzl @@ -50,31 +31,13 @@ my_macro = macro( ) ``` -Attribute type declarations accept the -[parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), -`mandatory`, `default`, and `doc`. Most attribute types also accept the -`configurable` parameter, which determines whether the attribute accepts -`select`s. If an attribute is `configurable`, it will parse non-`select` values -as an unconfigurable `select` – `"foo"` will become -`select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). +Attribute type declarations accept the [parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), `mandatory`, `default`, and `doc`. Most attribute types also accept the `configurable` parameter, which determines whether the attribute accepts `select`s. If an attribute is `configurable`, it will parse non-`select` values as an unconfigurable `select` – `"foo"` will become `select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). #### Attribute inheritance -Macros are often intended to wrap a rule (or another macro), and the macro's -author often wants to forward the bulk of the wrapped symbol's attributes -unchanged, using `**kwargs`, to the macro's main target (or main inner macro). - -To support this pattern, a macro can *inherit attributes* from a rule or another -macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or -[macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s -`inherit_attrs` argument. (You can also use the special string `"common"` -instead of a rule or macro symbol to inherit the [common attributes defined for -all Starlark build -rules](https://bazel.build/reference/be/common-definitions#common-attributes).) -Only public attributes get inherited, and the attributes in the macro's own -`attrs` dictionary override inherited attributes with the same name. You can -also *remove* inherited attributes by using `None` as a value in the `attrs` -dictionary: +Macros are often intended to wrap a rule (or another macro), and the macro's author often wants to forward the bulk of the wrapped symbol's attributes unchanged, using `**kwargs`, to the macro's main target (or main inner macro). + +To support this pattern, a macro can *inherit attributes* from a rule or another macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or [macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s `inherit_attrs` argument. (You can also use the special string `"common"` instead of a rule or macro symbol to inherit the [common attributes defined for all Starlark build rules](https://bazel.build/reference/be/common-definitions#common-attributes).) Only public attributes get inherited, and the attributes in the macro's own `attrs` dictionary override inherited attributes with the same name. You can also *remove* inherited attributes by using `None` as a value in the `attrs` dictionary: ```starlark # macro/macro.bzl @@ -90,11 +53,7 @@ my_macro = macro( ) ``` -The default value of non-mandatory inherited attributes is always overridden to -be `None`, regardless of the original attribute definition's default value. If -you need to examine or modify an inherited non-mandatory attribute – for -example, if you want to add a tag to an inherited `tags` attribute – you must -make sure to handle the `None` case in your macro's implementation function: +The default value of non-mandatory inherited attributes is always overridden to be `None`, regardless of the original attribute definition's default value. If you need to examine or modify an inherited non-mandatory attribute – for example, if you want to add a tag to an inherited `tags` attribute – you must make sure to handle the `None` case in your macro's implementation function: ```starlark # macro/macro.bzl @@ -112,15 +71,9 @@ def _my_macro_impl(name, visibility, tags, **kwargs): ### Implementation -`implementation` accepts a function which contains the logic of the macro. -Implementation functions often create targets by calling one or more rules, and -they are usually private (named with a leading underscore). Conventionally, -they are named the same as their macro, but prefixed with `_` and suffixed with -`_impl`. +`implementation` accepts a function which contains the logic of the macro. Implementation functions often create targets by calling one or more rules, and they are usually private (named with a leading underscore). Conventionally, they are named the same as their macro, but prefixed with `_` and suffixed with `_impl`. -Unlike rule implementation functions, which take a single argument (`ctx`) that -contains a reference to the attributes, macro implementation functions accept a -parameter for each argument. +Unlike rule implementation functions, which take a single argument (`ctx`) that contains a reference to the attributes, macro implementation functions accept a parameter for each argument. ```starlark # macro/macro.bzl @@ -138,11 +91,7 @@ def _my_macro_impl(name, visibility, deps, create_test): ) ``` -If a macro inherits attributes, its implementation function *must* have a -`**kwargs` residual keyword parameter, which can be forwarded to the call that -invokes the inherited rule or submacro. (This helps ensure that your macro won't -be broken if the rule or macro which from which you are inheriting adds a new -attribute.) +If a macro inherits attributes, its implementation function *must* have a `**kwargs` residual keyword parameter, which can be forwarded to the call that invokes the inherited rule or submacro. (This helps ensure that your macro won't be broken if the rule or macro which from which you are inheriting adds a new attribute.) ### Declaration @@ -164,12 +113,9 @@ my_macro( ) ``` -This would create targets -`//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. +This would create targets `//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. -Just like in rule calls, if an attribute value in a macro call is set to `None`, -that attribute is treated as if it was omitted by the macro's caller. For -example, the following two macro calls are equivalent: +Just like in rule calls, if an attribute value in a macro call is set to `None`, that attribute is treated as if it was omitted by the macro's caller. For example, the following two macro calls are equivalent: ```starlark # pkg/BUILD @@ -177,27 +123,17 @@ my_macro(name = "abc", srcs = ["src.cc"], deps = None) my_macro(name = "abc", srcs = ["src.cc"]) ``` -This is generally not useful in `BUILD` files, but is helpful when -programmatically wrapping a macro inside another macro. +This is generally not useful in `BUILD` files, but is helpful when programmatically wrapping a macro inside another macro. ## Details ### Naming conventions for targets created -The names of any targets or submacros created by a symbolic macro must -either match the macro's `name` parameter or must be prefixed by `name` followed -by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only -create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, -for example, `foo_bar`. +The names of any targets or submacros created by a symbolic macro must either match the macro's `name` parameter or must be prefixed by `name` followed by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, for example, `foo_bar`. -Targets or files that violate macro naming convention can be declared, but -cannot be built and cannot be used as dependencies. +Targets or files that violate macro naming convention can be declared, but cannot be built and cannot be used as dependencies. -Non-macro files and targets within the same package as a macro instance should -*not* have names that conflict with potential macro target names, though this -exclusivity is not enforced. We are in the progress of implementing -[lazy evaluation](#laziness) as a performance improvement for Symbolic macros, -which will be impaired in packages that violate the naming schema. +Non-macro files and targets within the same package as a macro instance should *not* have names that conflict with potential macro target names, though this exclusivity is not enforced. We are in the progress of implementing [lazy evaluation](#laziness) as a performance improvement for Symbolic macros, which will be impaired in packages that violate the naming schema. ### Restrictions @@ -205,54 +141,35 @@ Symbolic macros have some additional restrictions compared to legacy macros. Symbolic macros -* must take a `name` argument and a `visibility` argument -* must have an `implementation` function -* may not return values -* may not mutate their arguments -* may not call `native.existing_rules()` unless they are special `finalizer` - macros -* may not call `native.package()` -* may not call `glob()` -* may not call `native.environment_group()` -* must create targets whose names adhere to the [naming schema](#naming) -* can't refer to input files that weren't declared or passed in as an argument -* can't refer to private targets of their callers (see - [visibility and macros](#visibility) for more details). +- must take a `name` argument and a `visibility` argument +- must have an `implementation` function +- may not return values +- may not mutate their arguments +- may not call `native.existing_rules()` unless they are special `finalizer` macros +- may not call `native.package()` +- may not call `glob()` +- may not call `native.environment_group()` +- must create targets whose names adhere to the [naming schema](#naming) +- can't refer to input files that weren't declared or passed in as an argument +- can't refer to private targets of their callers (see [visibility and macros](#visibility) for more details). ### Visibility and macros -The [visibility](/concepts/visibility) system helps protect the implementation -details of both (symbolic) macros and their callers. +The [visibility](/concepts/visibility) system helps protect the implementation details of both (symbolic) macros and their callers. -By default, targets created in a symbolic macro are visible within the macro -itself, but not necessarily to the macro's caller. The macro can "export" a -target as a public API by forwarding the value of its own `visibility` -attribute, as in `some_rule(..., visibility = visibility)`. +By default, targets created in a symbolic macro are visible within the macro itself, but not necessarily to the macro's caller. The macro can "export" a target as a public API by forwarding the value of its own `visibility` attribute, as in `some_rule(..., visibility = visibility)`. The key ideas of macro visibility are: -1. Visibility is checked based on what macro declared the target, not what - package called the macro. +1. Visibility is checked based on what macro declared the target, not what package called the macro. - * In other words, being in the same package does not by itself make one - target visible to another. This protects the macro's internal targets - from becoming dependencies of other macros or top-level targets in the - package. + - In other words, being in the same package does not by itself make one target visible to another. This protects the macro's internal targets from becoming dependencies of other macros or top-level targets in the package. -1. All `visibility` attributes, on both rules and macros, automatically - include the place where the rule or macro was called. +2. All `visibility` attributes, on both rules and macros, automatically include the place where the rule or macro was called. - * Thus, a target is unconditionally visible to other targets declared in the - same macro (or the `BUILD` file, if not in a macro). + - Thus, a target is unconditionally visible to other targets declared in the same macro (or the `BUILD` file, if not in a macro). -In practice, this means that when a macro declares a target without setting its -`visibility`, the target defaults to being internal to the macro. (The package's -[default visibility](/reference/be/functions#package.default_visibility) does -not apply within a macro.) Exporting the target means that the target is visible -to whatever the macro's caller specified in the macro's `visibility` attribute, -plus the package of the macro's caller itself, as well as the macro's own code. -Another way of thinking of it is that the visibility of a macro determines who -(aside from the macro itself) can see the macro's exported targets. +In practice, this means that when a macro declares a target without setting its `visibility`, the target defaults to being internal to the macro. (The package's [default visibility](/reference/be/functions#package.default_visibility) does not apply within a macro.) Exporting the target means that the target is visible to whatever the macro's caller specified in the macro's `visibility` attribute, plus the package of the macro's caller itself, as well as the macro's own code. Another way of thinking of it is that the visibility of a macro determines who (aside from the macro itself) can see the macro's exported targets. ```starlark # tool/BUILD @@ -314,51 +231,25 @@ some_rule( ) ``` -If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if -the `//pkg` package had set its `default_visibility` to that value, then -`//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a -macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain -protected. - -A macro can declare that a target is visible to a friend package by passing -`visibility = ["//some_friend:__pkg__"]` (for an internal target) or -`visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). -Note that it is an antipattern for a macro to declare a target with public -visibility (`visibility = ["//visibility:public"]`). This is because it makes -the target unconditionally visible to every package, even if the caller -specified a more restricted visibility. - -All visibility checking is done with respect to the innermost currently running -symbolic macro. However, there is a visibility delegation mechanism: If a macro -passes a label as an attribute value to an inner macro, any usages of the label -in the inner macro are checked with respect to the outer macro. See the -[visibility page](/concepts/visibility#symbolic-macros) for more details. - -Remember that legacy macros are entirely transparent to the visibility system, -and behave as though their location is whatever BUILD file or symbolic macro -they were called from. +If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if the `//pkg` package had set its `default_visibility` to that value, then `//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain protected. + +A macro can declare that a target is visible to a friend package by passing `visibility = ["//some_friend:__pkg__"]` (for an internal target) or `visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). Note that it is an antipattern for a macro to declare a target with public visibility (`visibility = ["//visibility:public"]`). This is because it makes the target unconditionally visible to every package, even if the caller specified a more restricted visibility. + +All visibility checking is done with respect to the innermost currently running symbolic macro. However, there is a visibility delegation mechanism: If a macro passes a label as an attribute value to an inner macro, any usages of the label in the inner macro are checked with respect to the outer macro. See the [visibility page](/concepts/visibility#symbolic-macros) for more details. + +Remember that legacy macros are entirely transparent to the visibility system, and behave as though their location is whatever BUILD file or symbolic macro they were called from. #### Finalizers and visibility -Targets declared in a rule finalizer, in addition to seeing targets following -the usual symbolic macro visibility rules, can *also* see all targets which are -visible to the finalizer target's package. +Targets declared in a rule finalizer, in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. -This means that if you migrate a `native.existing_rules()`-based legacy macro to -a finalizer, the targets declared by the finalizer will still be able to see -their old dependencies. +This means that if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. -However, note that it's possible to declare a target in a symbolic macro such -that a finalizer's targets cannot see it under the visibility system – even -though the finalizer can *introspect* its attributes using -`native.existing_rules()`. +However, note that it's possible to declare a target in a symbolic macro such that a finalizer's targets cannot see it under the visibility system – even though the finalizer can *introspect* its attributes using `native.existing_rules()`. ### Selects -If an attribute is `configurable` (the default) and its value is not `None`, -then the macro implementation function will see the attribute value as wrapped -in a trivial `select`. This makes it easier for the macro author to catch bugs -where they did not anticipate that the attribute value could be a `select`. +If an attribute is `configurable` (the default) and its value is not `None`, then the macro implementation function will see the attribute value as wrapped in a trivial `select`. This makes it easier for the macro author to catch bugs where they did not anticipate that the attribute value could be a `select`. For example, consider the following macro: @@ -369,38 +260,15 @@ my_macro = macro( ) ``` -If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` -to be invoked with its `deps` parameter set to `select({"//conditions:default": -["//a"]})`. If this causes the implementation function to fail (say, because the -code tried to index into the value as in `deps[0]`, which is not allowed for -`select`s), the macro author can then make a choice: either they can rewrite -their macro to only use operations compatible with `select`, or they can mark -the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The -latter ensures that users are not permitted to pass a `select` value in. - -Rule targets reverse this transformation, and store trivial `select`s as their -unconditional values; in the above example, if `_my_macro_impl` declares a rule -target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as -`["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` -values to be stored in all targets instantiated by macros. - -If the value of a configurable attribute is `None`, it does not get wrapped in a -`select`. This ensures that tests like `my_attr == None` still work, and that -when the attribute is forwarded to a rule with a computed default, the rule -behaves properly (that is, as if the attribute were not passed in at all). It is -not always possible for an attribute to take on a `None` value, but it can -happen for the `attr.label()` type, and for any inherited non-mandatory -attribute. +If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` to be invoked with its `deps` parameter set to `select({"//conditions:default": ["//a"]})`. If this causes the implementation function to fail (say, because the code tried to index into the value as in `deps[0]`, which is not allowed for `select`s), the macro author can then make a choice: either they can rewrite their macro to only use operations compatible with `select`, or they can mark the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The latter ensures that users are not permitted to pass a `select` value in. + +Rule targets reverse this transformation, and store trivial `select`s as their unconditional values; in the above example, if `_my_macro_impl` declares a rule target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as `["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` values to be stored in all targets instantiated by macros. + +If the value of a configurable attribute is `None`, it does not get wrapped in a `select`. This ensures that tests like `my_attr == None` still work, and that when the attribute is forwarded to a rule with a computed default, the rule behaves properly (that is, as if the attribute were not passed in at all). It is not always possible for an attribute to take on a `None` value, but it can happen for the `attr.label()` type, and for any inherited non-mandatory attribute. ## Finalizers -A rule finalizer is a special symbolic macro which – regardless of its lexical -position in a BUILD file – is evaluated in the final stage of loading a package, -after all non-finalizer targets have been defined. Unlike ordinary symbolic -macros, a finalizer can call `native.existing_rules()`, where it behaves -slightly differently than in legacy macros: it only returns the set of -non-finalizer rule targets. The finalizer may assert on the state of that set or -define new targets. +A rule finalizer is a special symbolic macro which – regardless of its lexical position in a BUILD file – is evaluated in the final stage of loading a package, after all non-finalizer targets have been defined. Unlike ordinary symbolic macros, a finalizer can call `native.existing_rules()`, where it behaves slightly differently than in legacy macros: it only returns the set of non-finalizer rule targets. The finalizer may assert on the state of that set or define new targets. To declare a finalizer, call `macro()` with `finalizer = True`: @@ -426,24 +294,17 @@ my_finalizer = macro( ## Laziness -IMPORTANT: We are in the process of implementing lazy macro expansion and -evaluation. This feature is not available yet. +IMPORTANT: We are in the process of implementing lazy macro expansion and evaluation. This feature is not available yet. -Currently, all macros are evaluated as soon as the BUILD file is loaded, which -can negatively impact performance for targets in packages that also have costly -unrelated macros. In the future, non-finalizer symbolic macros will only be -evaluated if they're required for the build. The prefix naming schema helps -Bazel determine which macro to expand given a requested target. +Currently, all macros are evaluated as soon as the BUILD file is loaded, which can negatively impact performance for targets in packages that also have costly unrelated macros. In the future, non-finalizer symbolic macros will only be evaluated if they're required for the build. The prefix naming schema helps Bazel determine which macro to expand given a requested target. ## Migration troubleshooting Here are some common migration headaches and how to fix them. -* Legacy macro calls `glob()` +- Legacy macro calls `glob()` -Move the `glob()` call to your BUILD file (or to a legacy macro called from the -BUILD file), and pass the `glob()` value to the symbolic macro using a -label-list attribute: +Move the `glob()` call to your BUILD file (or to a legacy macro called from the BUILD file), and pass the `glob()` value to the symbolic macro using a label-list attribute: ```starlark # BUILD file @@ -453,13 +314,10 @@ my_macro( ) ``` -* Legacy macro has a parameter that isn't a valid starlark `attr` type. - -Pull as much logic as possible into a nested symbolic macro, but keep the -top level macro a legacy macro. +- Legacy macro has a parameter that isn't a valid starlark `attr` type. -* Legacy macro calls a rule that creates a target that breaks the naming schema +Pull as much logic as possible into a nested symbolic macro, but keep the top level macro a legacy macro. -That's okay, just don't depend on the "offending" target. The naming check will -be quietly ignored. +- Legacy macro calls a rule that creates a target that breaks the naming schema +That's okay, just don't depend on the "offending" target. The naming check will be quietly ignored. diff --git a/extending/platforms.mdx b/extending/platforms.mdx index 94e6290f..e6e349fd 100644 --- a/extending/platforms.mdx +++ b/extending/platforms.mdx @@ -2,60 +2,29 @@ title: 'Platforms' --- +Bazel can build and test code on a variety of hardware, operating systems, and system configurations, using many different versions of build tools such as linkers and compilers. To help manage this complexity, Bazel has a concept of *constraints* and *platforms*. A constraint is a dimension in which build or production environments may differ, such as CPU architecture, the presence or absence of a GPU, or the version of a system-installed compiler. A platform is a named collection of choices for these constraints, representing the particular resources that are available in some environment. - -Bazel can build and test code on a variety of hardware, operating systems, and -system configurations, using many different versions of build tools such as -linkers and compilers. To help manage this complexity, Bazel has a concept of -*constraints* and *platforms*. A constraint is a dimension in which build or -production environments may differ, such as CPU architecture, the presence or -absence of a GPU, or the version of a system-installed compiler. A platform is a -named collection of choices for these constraints, representing the particular -resources that are available in some environment. - -Modeling the environment as a platform helps Bazel to automatically select the -appropriate -[toolchains](/extending/toolchains) -for build actions. Platforms can also be used in combination with the -[config_setting](/reference/be/general#config_setting) -rule to write [configurable attributes](/docs/configurable-attributes). +Modeling the environment as a platform helps Bazel to automatically select the appropriate [toolchains](/extending/toolchains) for build actions. Platforms can also be used in combination with the [config\_setting](/reference/be/general#config_setting) rule to write [configurable attributes](/docs/configurable-attributes). Bazel recognizes three roles that a platform may serve: -* **Host** - the platform on which Bazel itself runs. -* **Execution** - a platform on which build tools execute build actions to - produce intermediate and final outputs. -* **Target** - a platform on which a final output resides and executes. +- **Host** - the platform on which Bazel itself runs. +- **Execution** - a platform on which build tools execute build actions to produce intermediate and final outputs. +- **Target** - a platform on which a final output resides and executes. Bazel supports the following build scenarios regarding platforms: -* **Single-platform builds** (default) - host, execution, and target platforms - are the same. For example, building a Linux executable on Ubuntu running on - an Intel x64 CPU. +- **Single-platform builds** (default) - host, execution, and target platforms are the same. For example, building a Linux executable on Ubuntu running on an Intel x64 CPU. -* **Cross-compilation builds** - host and execution platforms are the same, but - the target platform is different. For example, building an iOS app on macOS - running on a MacBook Pro. +- **Cross-compilation builds** - host and execution platforms are the same, but the target platform is different. For example, building an iOS app on macOS running on a MacBook Pro. -* **Multi-platform builds** - host, execution, and target platforms are all - different. +- **Multi-platform builds** - host, execution, and target platforms are all different. -Tip: for detailed instructions on migrating your project to platforms, see -[Migrating to Platforms](/concepts/platforms). +Tip: for detailed instructions on migrating your project to platforms, see [Migrating to Platforms](/concepts/platforms). ## Defining constraints and platforms -The space of possible choices for platforms is defined by using the -[`constraint_setting`][constraint_setting] and -[`constraint_value`][constraint_value] rules within `BUILD` files. -`constraint_setting` creates a new dimension, while -`constraint_value` creates a new value for a given dimension; together they -effectively define an enum and its possible values. For example, the following -snippet of a `BUILD` file introduces a constraint for the system's glibc version -with two possible values. - -[constraint_setting]: /reference/be/platforms-and-toolchains#constraint_setting -[constraint_value]: /reference/be/platforms-and-toolchains#constraint_value +The space of possible choices for platforms is defined by using the [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) and [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) rules within `BUILD` files. `constraint_setting` creates a new dimension, while `constraint_value` creates a new value for a given dimension; together they effectively define an enum and its possible values. For example, the following snippet of a `BUILD` file introduces a constraint for the system's glibc version with two possible values. ```python constraint_setting(name = "glibc_version") @@ -71,16 +40,9 @@ constraint_value( ) ``` -Constraints and their values may be defined across different packages in the -workspace. They are referenced by label and subject to the usual visibility -controls. If visibility allows, you can extend an existing constraint setting by -defining your own value for it. +Constraints and their values may be defined across different packages in the workspace. They are referenced by label and subject to the usual visibility controls. If visibility allows, you can extend an existing constraint setting by defining your own value for it. -The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with -certain choices of constraint values. The -following creates a platform named `linux_x86`, and says that it describes any -environment that runs a Linux operating system on an x86_64 architecture with a -glibc version of 2.25. (See below for more on Bazel's built-in constraints.) +The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with certain choices of constraint values. The following creates a platform named `linux_x86`, and says that it describes any environment that runs a Linux operating system on an x86\_64 architecture with a glibc version of 2.25. (See below for more on Bazel's built-in constraints.) ```python platform( @@ -93,52 +55,33 @@ platform( ) ``` -Note: It is an error for a platform to specify more than one value of the -same constraint setting, such as `@platforms//cpu:x86_64` and -`@platforms//cpu:arm` for `@platforms//cpu:cpu`. +Note: It is an error for a platform to specify more than one value of the same constraint setting, such as `@platforms//cpu:x86_64` and `@platforms//cpu:arm` for `@platforms//cpu:cpu`. ## Generally useful constraints and platforms -To keep the ecosystem consistent, Bazel team maintains a repository with -constraint definitions for the most popular CPU architectures and operating -systems. These are all located in -[https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). +To keep the ecosystem consistent, Bazel team maintains a repository with constraint definitions for the most popular CPU architectures and operating systems. These are all located in [https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). -Bazel ships with the following special platform definition: -`@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the -autodetected host platform value - -represents autodetected platform for the system Bazel is running on. +Bazel ships with the following special platform definition: `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the autodetected host platform value - represents autodetected platform for the system Bazel is running on. ## Specifying a platform for a build -You can specify the host and target platforms for a build using the following -command-line flags: - -* `--host_platform` - defaults to `@bazel_tools//tools:host_platform` - * This target is aliased to `@platforms//host`, which is backed by a repo - rule that detects the host OS and CPU and writes the platform target. - * There's also `@platforms//host:constraints.bzl`, which exposes - an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and - Starlark files. -* `--platforms` - defaults to the host platform - * This means that when no other flags are set, - `@platforms//host` is the target platform. - * If `--host_platform` is set and not `--platforms`, the value of - `--host_platform` is both the host and target platform. +You can specify the host and target platforms for a build using the following command-line flags: + +- `--host_platform` - defaults to `@bazel_tools//tools:host_platform` + + - This target is aliased to `@platforms//host`, which is backed by a repo rule that detects the host OS and CPU and writes the platform target. + - There's also `@platforms//host:constraints.bzl`, which exposes an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and Starlark files. + +- `--platforms` - defaults to the host platform + + - This means that when no other flags are set, `@platforms//host` is the target platform. + - If `--host_platform` is set and not `--platforms`, the value of `--host_platform` is both the host and target platform. ## Skipping incompatible targets -When building for a specific target platform it is often desirable to skip -targets that will never work on that platform. For example, your Windows device -driver is likely going to generate lots of compiler errors when building on a -Linux machine with `//...`. Use the -[`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) -attribute to tell Bazel what target platform constraints your code has. +When building for a specific target platform it is often desirable to skip targets that will never work on that platform. For example, your Windows device driver is likely going to generate lots of compiler errors when building on a Linux machine with `//...`. Use the [`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) attribute to tell Bazel what target platform constraints your code has. -The simplest use of this attribute restricts a target to a single platform. -The target will not be built for any platform that doesn't satisfy all of the -constraints. The following example restricts `win_driver_lib.cc` to 64-bit -Windows. +The simplest use of this attribute restricts a target to a single platform. The target will not be built for any platform that doesn't satisfy all of the constraints. The following example restricts `win_driver_lib.cc` to 64-bit Windows. ```python cc_library( @@ -151,16 +94,11 @@ cc_library( ) ``` -`:win_driver_lib` is *only* compatible for building with 64-bit Windows and -incompatible with all else. Incompatibility is transitive. Any targets -that transitively depend on an incompatible target are themselves considered -incompatible. +`:win_driver_lib` is *only* compatible for building with 64-bit Windows and incompatible with all else. Incompatibility is transitive. Any targets that transitively depend on an incompatible target are themselves considered incompatible. ### When are targets skipped? -Targets are skipped when they are considered incompatible and included in the -build as part of a target pattern expansion. For example, the following two -invocations skip any incompatible targets found in a target pattern expansion. +Targets are skipped when they are considered incompatible and included in the build as part of a target pattern expansion. For example, the following two invocations skip any incompatible targets found in a target pattern expansion. ```console $ bazel build --platforms=//:myplatform //... @@ -170,15 +108,9 @@ $ bazel build --platforms=//:myplatform //... $ bazel build --platforms=//:myplatform //:all ``` -Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are -similarly skipped if the `test_suite` is specified on the command line with -[`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). -In other words, `test_suite` targets on the command line behave like `:all` and -`...`. Using `--noexpand_test_suites` prevents expansion and causes -`test_suite` targets with incompatible tests to also be incompatible. +Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are similarly skipped if the `test_suite` is specified on the command line with [`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). In other words, `test_suite` targets on the command line behave like `:all` and `...`. Using `--noexpand_test_suites` prevents expansion and causes `test_suite` targets with incompatible tests to also be incompatible. -Explicitly specifying an incompatible target on the command line results in an -error message and a failed build. +Explicitly specifying an incompatible target on the command line results in an error message and a failed build. ```console $ bazel build --platforms=//:myplatform //:target_incompatible_with_myplatform @@ -188,20 +120,13 @@ ERROR: Target //:target_incompatible_with_myplatform is incompatible and cannot FAILED: Build did NOT complete successfully ``` -Incompatible explicit targets are silently skipped if -`--skip_incompatible_explicit_targets` is enabled. +Incompatible explicit targets are silently skipped if `--skip_incompatible_explicit_targets` is enabled. ### More expressive constraints -For more flexibility in expressing constraints, use the -`@platforms//:incompatible` -[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) -that no platform satisfies. +For more flexibility in expressing constraints, use the `@platforms//:incompatible` [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) that no platform satisfies. -Use [`select()`](/reference/be/functions#select) in combination with -`@platforms//:incompatible` to express more complicated restrictions. For -example, use it to implement basic OR logic. The following marks a library -compatible with macOS and Linux, but no other platforms. +Use [`select()`](/reference/be/functions#select) in combination with `@platforms//:incompatible` to express more complicated restrictions. For example, use it to implement basic OR logic. The following marks a library compatible with macOS and Linux, but no other platforms. Note: An empty constraints list is equivalent to "compatible with everything". @@ -221,16 +146,11 @@ The above can be interpreted as follows: 1. When targeting macOS, the target has no constraints. 2. When targeting Linux, the target has no constraints. -3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because - `@platforms//:incompatible` is not part of any platform, the target is - deemed incompatible. +3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because `@platforms//:incompatible` is not part of any platform, the target is deemed incompatible. -To make your constraints more readable, use -[skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). +To make your constraints more readable, use [skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). -You can express inverse compatibility in a similar way. The following example -describes a library that is compatible with everything _except_ for ARM. +You can express inverse compatibility in a similar way. The following example describes a library that is compatible with everything *except* for ARM. ```python cc_library( @@ -245,15 +165,9 @@ cc_library( ### Detecting incompatible targets using `bazel cquery` -You can use the -[`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) -in `bazel cquery`'s [Starlark output -format](/query/cquery#output-format-definition) to distinguish -incompatible targets from compatible ones. +You can use the [`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) in `bazel cquery`'s [Starlark output format](/query/cquery#output-format-definition) to distinguish incompatible targets from compatible ones. -This can be used to filter out incompatible targets. The example below will -only print the labels for targets that are compatible. Incompatible targets are -not printed. +This can be used to filter out incompatible targets. The example below will only print the labels for targets that are compatible. Incompatible targets are not printed. ```console $ cat example.cquery @@ -263,11 +177,9 @@ def format(target): return target.label return "" - $ bazel cquery //... --output=starlark --starlark:file=example.cquery ``` ### Known Issues -Incompatible targets [ignore visibility -restrictions](https://github.com/bazelbuild/bazel/issues/16044). +Incompatible targets [ignore visibility restrictions](https://github.com/bazelbuild/bazel/issues/16044). diff --git a/extending/rules.mdx b/extending/rules.mdx index 248ee328..b7afa71c 100644 --- a/extending/rules.mdx +++ b/extending/rules.mdx @@ -2,57 +2,28 @@ title: 'Rules' --- +A **rule** defines a series of [**actions**](#actions) that Bazel performs on inputs to produce a set of outputs, which are referenced in [**providers**](#providers) returned by the rule's [**implementation function**](#implementation_function). For example, a C++ binary rule might: +1. Take a set of `.cpp` source files (inputs). +2. Run `g++` on the source files (action). +3. Return the `DefaultInfo` provider with the executable output and other files to make available at runtime. +4. Return the `CcInfo` provider with C++-specific information gathered from the target and its dependencies. -A **rule** defines a series of [**actions**](#actions) that Bazel performs on -inputs to produce a set of outputs, which are referenced in -[**providers**](#providers) returned by the rule's -[**implementation function**](#implementation_function). For example, a C++ -binary rule might: - -1. Take a set of `.cpp` source files (inputs). -2. Run `g++` on the source files (action). -3. Return the `DefaultInfo` provider with the executable output and other files - to make available at runtime. -4. Return the `CcInfo` provider with C++-specific information gathered from the - target and its dependencies. - -From Bazel's perspective, `g++` and the standard C++ libraries are also inputs -to this rule. As a rule writer, you must consider not only the user-provided -inputs to a rule, but also all of the tools and libraries required to execute -the actions. - -Before creating or modifying any rule, ensure you are familiar with Bazel's -[build phases](/extending/concepts). It is important to understand the three -phases of a build (loading, analysis, and execution). It is also useful to -learn about [macros](/extending/macros) to understand the difference between rules and -macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). -Then, use this page as a reference. - -A few rules are built into Bazel itself. These *native rules*, such as -`genrule` and `filegroup`, provide some core support. -By defining your own rules, you can add support for languages and tools -that Bazel doesn't support natively. - -Bazel provides an extensibility model for writing rules using the -[Starlark](/rules/language) language. These rules are written in `.bzl` files, which -can be loaded directly from `BUILD` files. - -When defining your own rule, you get to decide what attributes it supports and -how it generates its outputs. - -The rule's `implementation` function defines its exact behavior during the -[analysis phase](/extending/concepts#evaluation-model). This function doesn't run any -external commands. Rather, it registers [actions](#actions) that will be used -later during the execution phase to build the rule's outputs, if they are -needed. +From Bazel's perspective, `g++` and the standard C++ libraries are also inputs to this rule. As a rule writer, you must consider not only the user-provided inputs to a rule, but also all of the tools and libraries required to execute the actions. + +Before creating or modifying any rule, ensure you are familiar with Bazel's [build phases](/extending/concepts). It is important to understand the three phases of a build (loading, analysis, and execution). It is also useful to learn about [macros](/extending/macros) to understand the difference between rules and macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). Then, use this page as a reference. + +A few rules are built into Bazel itself. These *native rules*, such as `genrule` and `filegroup`, provide some core support. By defining your own rules, you can add support for languages and tools that Bazel doesn't support natively. + +Bazel provides an extensibility model for writing rules using the [Starlark](/rules/language) language. These rules are written in `.bzl` files, which can be loaded directly from `BUILD` files. + +When defining your own rule, you get to decide what attributes it supports and how it generates its outputs. + +The rule's `implementation` function defines its exact behavior during the [analysis phase](/extending/concepts#evaluation-model). This function doesn't run any external commands. Rather, it registers [actions](#actions) that will be used later during the execution phase to build the rule's outputs, if they are needed. ## Rule creation -In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new -rule, and store the result in a global variable. The call to `rule` specifies -[attributes](#attributes) and an -[implementation function](#implementation_function): +In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new rule, and store the result in a global variable. The call to `rule` specifies [attributes](#attributes) and an [implementation function](#implementation_function): ```python example_library = rule( @@ -66,10 +37,7 @@ example_library = rule( This defines a [rule kind](/query/language#kind) named `example_library`. -The call to `rule` also must specify if the rule creates an -[executable](#executable-rules) output (with `executable = True`), or specifically -a test executable (with `test = True`). If the latter, the rule is a *test rule*, -and the name of the rule must end in `_test`. +The call to `rule` also must specify if the rule creates an [executable](#executable-rules) output (with `executable = True`), or specifically a test executable (with `test = True`). If the latter, the rule is a *test rule*, and the name of the rule must end in `_test`. ## Target instantiation @@ -85,48 +53,23 @@ example_library( ) ``` -Each call to a build rule returns no value, but has the side effect of defining -a target. This is called *instantiating* the rule. This specifies a name for the -new target and values for the target's [attributes](#attributes). +Each call to a build rule returns no value, but has the side effect of defining a target. This is called *instantiating* the rule. This specifies a name for the new target and values for the target's [attributes](#attributes). -Rules can also be called from Starlark functions and loaded in `.bzl` files. -Starlark functions that call rules are called [Starlark macros](/extending/macros). -Starlark macros must ultimately be called from `BUILD` files, and can only be -called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` -files are evaluated to instantiate targets. +Rules can also be called from Starlark functions and loaded in `.bzl` files. Starlark functions that call rules are called [Starlark macros](/extending/macros). Starlark macros must ultimately be called from `BUILD` files, and can only be called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` files are evaluated to instantiate targets. ## Attributes -An *attribute* is a rule argument. Attributes can provide specific values to a -target's [implementation](#implementation_function), or they can refer to other -targets, creating a graph of dependencies. +An *attribute* is a rule argument. Attributes can provide specific values to a target's [implementation](#implementation_function), or they can refer to other targets, creating a graph of dependencies. -Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map -from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) -module) to the `attrs` parameter of `rule`. -[Common attributes](/reference/be/common-definitions#common-attributes), such as -`name` and `visibility`, are implicitly added to all rules. Additional -attributes are implicitly added to -[executable and test rules](#executable-rules) specifically. Attributes which -are implicitly added to a rule can't be included in the dictionary passed to -`attrs`. +Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) module) to the `attrs` parameter of `rule`. [Common attributes](/reference/be/common-definitions#common-attributes), such as `name` and `visibility`, are implicitly added to all rules. Additional attributes are implicitly added to [executable and test rules](#executable-rules) specifically. Attributes which are implicitly added to a rule can't be included in the dictionary passed to `attrs`. ### Dependency attributes -Rules that process source code usually define the following attributes to handle -various [types of dependencies](/concepts/dependencies#types_of_dependencies): - -* `srcs` specifies source files processed by a target's actions. Often, the - attribute schema specifies which file extensions are expected for the sort - of source file the rule processes. Rules for languages with header files - generally specify a separate `hdrs` attribute for headers processed by a - target and its consumers. -* `deps` specifies code dependencies for a target. The attribute schema should - specify which [providers](#providers) those dependencies must provide. (For - example, `cc_library` provides `CcInfo`.) -* `data` specifies files to be made available at runtime to any executable - which depends on a target. That should allow arbitrary files to be - specified. +Rules that process source code usually define the following attributes to handle various [types of dependencies](/concepts/dependencies#types_of_dependencies): + +- `srcs` specifies source files processed by a target's actions. Often, the attribute schema specifies which file extensions are expected for the sort of source file the rule processes. Rules for languages with header files generally specify a separate `hdrs` attribute for headers processed by a target and its consumers. +- `deps` specifies code dependencies for a target. The attribute schema should specify which [providers](#providers) those dependencies must provide. (For example, `cc_library` provides `CcInfo`.) +- `data` specifies files to be made available at runtime to any executable which depends on a target. That should allow arbitrary files to be specified. ```python example_library = rule( @@ -141,16 +84,7 @@ example_library = rule( ) ``` -These are examples of *dependency attributes*. Any attribute that specifies -an input label (those defined with -[`attr.label_list`](/rules/lib/toplevel/attr#label_list), -[`attr.label`](/rules/lib/toplevel/attr#label), or -[`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) -specifies dependencies of a certain type -between a target and the targets whose labels (or the corresponding -[`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target -is defined. The repository, and possibly the path, for these labels is resolved -relative to the defined target. +These are examples of *dependency attributes*. Any attribute that specifies an input label (those defined with [`attr.label_list`](/rules/lib/toplevel/attr#label_list), [`attr.label`](/rules/lib/toplevel/attr#label), or [`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) specifies dependencies of a certain type between a target and the targets whose labels (or the corresponding [`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target is defined. The repository, and possibly the path, for these labels is resolved relative to the defined target. ```python example_library( @@ -164,27 +98,15 @@ example_library( ) ``` -In this example, `other_target` is a dependency of `my_target`, and therefore -`other_target` is analyzed first. It is an error if there is a cycle in the -dependency graph of targets. +In this example, `other_target` is a dependency of `my_target`, and therefore `other_target` is analyzed first. It is an error if there is a cycle in the dependency graph of targets. - +[]() ### Private attributes and implicit dependencies -A dependency attribute with a default value creates an *implicit dependency*. It -is implicit because it's a part of the target graph that the user doesn't -specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a -relationship between a rule and a *tool* (a build-time dependency, such as a -compiler), since most of the time a user is not interested in specifying what -tool the rule uses. Inside the rule's implementation function, this is treated -the same as other dependencies. +A dependency attribute with a default value creates an *implicit dependency*. It is implicit because it's a part of the target graph that the user doesn't specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a relationship between a rule and a *tool* (a build-time dependency, such as a compiler), since most of the time a user is not interested in specifying what tool the rule uses. Inside the rule's implementation function, this is treated the same as other dependencies. -If you want to provide an implicit dependency without allowing the user to -override that value, you can make the attribute *private* by giving it a name -that begins with an underscore (`_`). Private attributes must have default -values. It generally only makes sense to use private attributes for implicit -dependencies. +If you want to provide an implicit dependency without allowing the user to override that value, you can make the attribute *private* by giving it a name that begins with an underscore (`_`). Private attributes must have default values. It generally only makes sense to use private attributes for implicit dependencies. ```python example_library = rule( @@ -201,67 +123,34 @@ example_library = rule( ) ``` -In this example, every target of type `example_library` has an implicit -dependency on the compiler `//tools:example_compiler`. This allows -`example_library`'s implementation function to generate actions that invoke the -compiler, even though the user did not pass its label as an input. Since -`_compiler` is a private attribute, it follows that `ctx.attr._compiler` -will always point to `//tools:example_compiler` in all targets of this rule -type. Alternatively, you can name the attribute `compiler` without the -underscore and keep the default value. This allows users to substitute a -different compiler if necessary, but it requires no awareness of the compiler's -label. - -Implicit dependencies are generally used for tools that reside in the same -repository as the rule implementation. If the tool comes from the -[execution platform](/extending/platforms) or a different repository instead, the -rule should obtain that tool from a [toolchain](/extending/toolchains). +In this example, every target of type `example_library` has an implicit dependency on the compiler `//tools:example_compiler`. This allows `example_library`'s implementation function to generate actions that invoke the compiler, even though the user did not pass its label as an input. Since `_compiler` is a private attribute, it follows that `ctx.attr._compiler` will always point to `//tools:example_compiler` in all targets of this rule type. Alternatively, you can name the attribute `compiler` without the underscore and keep the default value. This allows users to substitute a different compiler if necessary, but it requires no awareness of the compiler's label. + +Implicit dependencies are generally used for tools that reside in the same repository as the rule implementation. If the tool comes from the [execution platform](/extending/platforms) or a different repository instead, the rule should obtain that tool from a [toolchain](/extending/toolchains). ### Output attributes -*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and -[`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the -target generates. These differ from dependency attributes in two ways: +*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and [`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the target generates. These differ from dependency attributes in two ways: -* They define output file targets instead of referring to targets defined - elsewhere. -* The output file targets depend on the instantiated rule target, instead of - the other way around. +- They define output file targets instead of referring to targets defined elsewhere. +- The output file targets depend on the instantiated rule target, instead of the other way around. -Typically, output attributes are only used when a rule needs to create outputs -with user-defined names which can't be based on the target name. If a rule has -one output attribute, it is typically named `out` or `outs`. +Typically, output attributes are only used when a rule needs to create outputs with user-defined names which can't be based on the target name. If a rule has one output attribute, it is typically named `out` or `outs`. -Output attributes are the preferred way of creating *predeclared outputs*, which -can be specifically depended upon or -[requested at the command line](#requesting_output_files). +Output attributes are the preferred way of creating *predeclared outputs*, which can be specifically depended upon or [requested at the command line](#requesting_output_files). ## Implementation function -Every rule requires an `implementation` function. These functions are executed -strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the -graph of targets generated in the loading phase into a graph of -[actions](#actions) to be performed during the execution phase. As such, -implementation functions can't actually read or write files. +Every rule requires an `implementation` function. These functions are executed strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the graph of targets generated in the loading phase into a graph of [actions](#actions) to be performed during the execution phase. As such, implementation functions can't actually read or write files. -Rule implementation functions are usually private (named with a leading -underscore). Conventionally, they are named the same as their rule, but suffixed -with `_impl`. +Rule implementation functions are usually private (named with a leading underscore). Conventionally, they are named the same as their rule, but suffixed with `_impl`. -Implementation functions take exactly one parameter: a -[rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of -[providers](#providers). +Implementation functions take exactly one parameter: a [rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of [providers](#providers). ### Targets -Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) -objects. These objects contain the [providers](#providers) generated when the -target's implementation function was executed. +Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) objects. These objects contain the [providers](#providers) generated when the target's implementation function was executed. -[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each -dependency attribute, containing `Target` objects representing each direct -dependency using that attribute. For `label_list` attributes, this is a list of -`Targets`. For `label` attributes, this is a single `Target` or `None`. +[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each dependency attribute, containing `Target` objects representing each direct dependency using that attribute. For `label_list` attributes, this is a list of `Targets`. For `label` attributes, this is a single `Target` or `None`. A list of provider objects are returned by a target's implementation function: @@ -269,14 +158,9 @@ A list of provider objects are returned by a target's implementation function: return [ExampleInfo(headers = depset(...))] ``` -Those can be accessed using index notation (`[]`), with the type of provider as -a key. These can be [custom providers](#custom_providers) defined in Starlark or -[providers for native rules](/rules/lib/providers) available as Starlark -global variables. +Those can be accessed using index notation (`[]`), with the type of provider as a key. These can be [custom providers](#custom_providers) defined in Starlark or [providers for native rules](/rules/lib/providers) available as Starlark global variables. -For example, if a rule takes header files using a `hdrs` attribute and provides -them to the compilation actions of the target and its consumers, it could -collect them like so: +For example, if a rule takes header files using a `hdrs` attribute and provides them to the compilation actions of the target and its consumers, it could collect them like so: ```python def _example_library_impl(ctx): @@ -284,24 +168,15 @@ def _example_library_impl(ctx): transitive_headers = [hdr[ExampleInfo].headers for hdr in ctx.attr.hdrs] ``` -There's a legacy struct style, which is strongly discouraged and rules should be -[migrated away from it](#migrating_from_legacy_providers). +There's a legacy struct style, which is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). ### Files -Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't -perform file I/O during the analysis phase, these objects can't be used to -directly read or write file content. Rather, they are passed to action-emitting -functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the -action graph. +Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't perform file I/O during the analysis phase, these objects can't be used to directly read or write file content. Rather, they are passed to action-emitting functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the action graph. -A `File` can either be a source file or a generated file. Each generated file -must be an output of exactly one action. Source files can't be the output of -any action. +A `File` can either be a source file or a generated file. Each generated file must be an output of exactly one action. Source files can't be the output of any action. -For each dependency attribute, the corresponding field of -[`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all -dependencies using that attribute: +For each dependency attribute, the corresponding field of [`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all dependencies using that attribute: ```python def _example_library_impl(ctx): @@ -311,20 +186,11 @@ def _example_library_impl(ctx): ... ``` -[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for -dependency attributes whose specs set `allow_single_file = True`. -[`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only -contains fields for dependency attributes whose specs set `executable = True`. +[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for dependency attributes whose specs set `allow_single_file = True`. [`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only contains fields for dependency attributes whose specs set `executable = True`. ### Declaring outputs -During the analysis phase, a rule's implementation function can create outputs. -Since all labels have to be known during the loading phase, these additional -outputs have no labels. `File` objects for outputs can be created using -[`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and -[`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). -Often, the names of outputs are based on the target's name, -[`ctx.label.name`](/rules/lib/builtins/ctx#label): +During the analysis phase, a rule's implementation function can create outputs. Since all labels have to be known during the loading phase, these additional outputs have no labels. `File` objects for outputs can be created using [`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and [`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). Often, the names of outputs are based on the target's name, [`ctx.label.name`](/rules/lib/builtins/ctx#label): ```python def _example_library_impl(ctx): @@ -333,31 +199,20 @@ def _example_library_impl(ctx): ... ``` -For *predeclared outputs*, like those created for -[output attributes](#output_attributes), `File` objects instead can be retrieved -from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). +For *predeclared outputs*, like those created for [output attributes](#output_attributes), `File` objects instead can be retrieved from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). ### Actions -An action describes how to generate a set of outputs from a set of inputs, for -example "run gcc on hello.c and get hello.o". When an action is created, Bazel -doesn't run the command immediately. It registers it in a graph of dependencies, -because an action can depend on the output of another action. For example, in C, -the linker must be called after the compiler. +An action describes how to generate a set of outputs from a set of inputs, for example "run gcc on hello.c and get hello.o". When an action is created, Bazel doesn't run the command immediately. It registers it in a graph of dependencies, because an action can depend on the output of another action. For example, in C, the linker must be called after the compiler. -General-purpose functions that create actions are defined in -[`ctx.actions`](/rules/lib/builtins/actions): +General-purpose functions that create actions are defined in [`ctx.actions`](/rules/lib/builtins/actions): -* [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. -* [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell - command. -* [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. -* [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to - generate a file from a template. +- [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. +- [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell command. +- [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. +- [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to generate a file from a template. -[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently -accumulate the arguments for actions. It avoids flattening depsets until -execution time: +[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently accumulate the arguments for actions. It avoids flattening depsets until execution time: ```python def _example_library_impl(ctx): @@ -384,61 +239,31 @@ def _example_library_impl(ctx): ... ``` -Actions take a list or depset of input files and generate a (non-empty) list of -output files. The set of input and output files must be known during the -[analysis phase](/extending/concepts#evaluation-model). It might depend on the value of -attributes, including providers from dependencies, but it can't depend on the -result of the execution. For example, if your action runs the unzip command, you -must specify which files you expect to be inflated (before running unzip). -Actions which create a variable number of files internally can wrap those in a -single file (such as a zip, tar, or other archive format). +Actions take a list or depset of input files and generate a (non-empty) list of output files. The set of input and output files must be known during the [analysis phase](/extending/concepts#evaluation-model). It might depend on the value of attributes, including providers from dependencies, but it can't depend on the result of the execution. For example, if your action runs the unzip command, you must specify which files you expect to be inflated (before running unzip). Actions which create a variable number of files internally can wrap those in a single file (such as a zip, tar, or other archive format). -Actions must list all of their inputs. Listing inputs that are not used is -permitted, but inefficient. +Actions must list all of their inputs. Listing inputs that are not used is permitted, but inefficient. -Actions must create all of their outputs. They may write other files, but -anything not in outputs won't be available to consumers. All declared outputs -must be written by some action. +Actions must create all of their outputs. They may write other files, but anything not in outputs won't be available to consumers. All declared outputs must be written by some action. -Actions are comparable to pure functions: They should depend only on the -provided inputs, and avoid accessing computer information, username, clock, -network, or I/O devices (except for reading inputs and writing outputs). This is -important because the output will be cached and reused. +Actions are comparable to pure functions: They should depend only on the provided inputs, and avoid accessing computer information, username, clock, network, or I/O devices (except for reading inputs and writing outputs). This is important because the output will be cached and reused. -Dependencies are resolved by Bazel, which decides which actions to -execute. It is an error if there is a cycle in the dependency graph. Creating -an action doesn't guarantee that it will be executed, that depends on whether -its outputs are needed for the build. +Dependencies are resolved by Bazel, which decides which actions to execute. It is an error if there is a cycle in the dependency graph. Creating an action doesn't guarantee that it will be executed, that depends on whether its outputs are needed for the build. ### Providers -Providers are pieces of information that a rule exposes to other rules that -depend on it. This data can include output files, libraries, parameters to pass -on a tool's command line, or anything else a target's consumers should know -about. +Providers are pieces of information that a rule exposes to other rules that depend on it. This data can include output files, libraries, parameters to pass on a tool's command line, or anything else a target's consumers should know about. -Since a rule's implementation function can only read providers from the -instantiated target's immediate dependencies, rules need to forward any -information from a target's dependencies that needs to be known by a target's -consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). +Since a rule's implementation function can only read providers from the instantiated target's immediate dependencies, rules need to forward any information from a target's dependencies that needs to be known by a target's consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). -A target's providers are specified by a list of provider objects returned by -the implementation function. +A target's providers are specified by a list of provider objects returned by the implementation function. -Old implementation functions can also be written in a legacy style where the -implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of -provider objects. This style is strongly discouraged and rules should be -[migrated away from it](#migrating_from_legacy_providers). +Old implementation functions can also be written in a legacy style where the implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of provider objects. This style is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). #### Default outputs -A target's *default outputs* are the outputs that are requested by default when -the target is requested for build at the command line. For example, a -`java_library` target `//pkg:foo` has `foo.jar` as a default output, so that -will be built by the command `bazel build //pkg:foo`. +A target's *default outputs* are the outputs that are requested by default when the target is requested for build at the command line. For example, a `java_library` target `//pkg:foo` has `foo.jar` as a default output, so that will be built by the command `bazel build //pkg:foo`. -Default outputs are specified by the `files` parameter of -[`DefaultInfo`](/rules/lib/providers/DefaultInfo): +Default outputs are specified by the `files` parameter of [`DefaultInfo`](/rules/lib/providers/DefaultInfo): ```python def _example_library_impl(ctx): @@ -449,37 +274,17 @@ def _example_library_impl(ctx): ] ``` -If `DefaultInfo` is not returned by a rule implementation or the `files` -parameter is not specified, `DefaultInfo.files` defaults to all -*predeclared outputs* (generally, those created by [output -attributes](#output_attributes)). +If `DefaultInfo` is not returned by a rule implementation or the `files` parameter is not specified, `DefaultInfo.files` defaults to all *predeclared outputs* (generally, those created by [output attributes](#output_attributes)). -Rules that perform actions should provide default outputs, even if those outputs -are not expected to be directly used. Actions that are not in the graph of the -requested outputs are pruned. If an output is only used by a target's consumers, -those actions won't be performed when the target is built in isolation. This -makes debugging more difficult because rebuilding just the failing target won't -reproduce the failure. +Rules that perform actions should provide default outputs, even if those outputs are not expected to be directly used. Actions that are not in the graph of the requested outputs are pruned. If an output is only used by a target's consumers, those actions won't be performed when the target is built in isolation. This makes debugging more difficult because rebuilding just the failing target won't reproduce the failure. #### Runfiles -Runfiles are a set of files used by a target at runtime (as opposed to build -time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates -a directory tree containing symlinks pointing to the runfiles. This stages the -environment for the binary so it can access the runfiles during runtime. +Runfiles are a set of files used by a target at runtime (as opposed to build time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates a directory tree containing symlinks pointing to the runfiles. This stages the environment for the binary so it can access the runfiles during runtime. -Runfiles can be added manually during rule creation. -[`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method -on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the -`runfiles` parameter on `DefaultInfo`. The executable output of -[executable rules](#executable-rules) is implicitly added to the runfiles. +Runfiles can be added manually during rule creation. [`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the `runfiles` parameter on `DefaultInfo`. The executable output of [executable rules](#executable-rules) is implicitly added to the runfiles. -Some rules specify attributes, generally named -[`data`](/reference/be/common-definitions#common.data), whose outputs are added to -a targets' runfiles. Runfiles should also be merged in from `data`, as well as -from any attributes which might provide code for eventual execution, generally -`srcs` (which might contain `filegroup` targets with associated `data`) and -`deps`. +Some rules specify attributes, generally named [`data`](/reference/be/common-definitions#common.data), whose outputs are added to a targets' runfiles. Runfiles should also be merged in from `data`, as well as from any attributes which might provide code for eventual execution, generally `srcs` (which might contain `filegroup` targets with associated `data`) and `deps`. ```python def _example_library_impl(ctx): @@ -503,8 +308,7 @@ def _example_library_impl(ctx): #### Custom providers -Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) -function to convey rule-specific information: +Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) function to convey rule-specific information: ```python ExampleInfo = provider( @@ -537,23 +341,11 @@ def _example_library_impl(ctx): ##### Custom initialization of providers -It's possible to guard the instantiation of a provider with custom -preprocessing and validation logic. This can be used to ensure that all -provider instances satisfy certain invariants, or to give users a cleaner API for -obtaining an instance. +It's possible to guard the instantiation of a provider with custom preprocessing and validation logic. This can be used to ensure that all provider instances satisfy certain invariants, or to give users a cleaner API for obtaining an instance. -This is done by passing an `init` callback to the -[`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the -return type of `provider()` changes to be a tuple of two values: the provider -symbol that is the ordinary return value when `init` is not used, and a "raw -constructor". +This is done by passing an `init` callback to the [`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the return type of `provider()` changes to be a tuple of two values: the provider symbol that is the ordinary return value when `init` is not used, and a "raw constructor". -In this case, when the provider symbol is called, instead of directly returning -a new instance, it will forward the arguments along to the `init` callback. The -callback's return value must be a dict mapping field names (strings) to values; -this is used to initialize the fields of the new instance. Note that the -callback may have any signature, and if the arguments don't match the signature -an error is reported as if the callback were invoked directly. +In this case, when the provider symbol is called, instead of directly returning a new instance, it will forward the arguments along to the `init` callback. The callback's return value must be a dict mapping field names (strings) to values; this is used to initialize the fields of the new instance. Note that the callback may have any signature, and if the arguments don't match the signature an error is reported as if the callback were invoked directly. The raw constructor, by contrast, will bypass the `init` callback. @@ -586,9 +378,7 @@ ExampleInfo( ) ``` -The raw constructor can be used to define alternative public factory functions -that don't go through the `init` logic. For example, exampleinfo.bzl -could define: +The raw constructor can be used to define alternative public factory functions that don't go through the `init` logic. For example, exampleinfo.bzl could define: ```python def make_barebones_exampleinfo(headers): @@ -596,12 +386,9 @@ def make_barebones_exampleinfo(headers): return _new_exampleinfo(files_to_link = depset(), headers = all_headers) ``` -Typically, the raw constructor is bound to a variable whose name begins with an -underscore (`_new_exampleinfo` above), so that user code can't load it and -generate arbitrary provider instances. +Typically, the raw constructor is bound to a variable whose name begins with an underscore (`_new_exampleinfo` above), so that user code can't load it and generate arbitrary provider instances. -Another use for `init` is to prevent the user from calling the provider -symbol altogether, and force them to use a factory function instead: +Another use for `init` is to prevent the user from calling the provider symbol altogether, and force them to use a factory function instead: ```python def _exampleinfo_init_banned(*args, **kwargs): @@ -616,15 +403,11 @@ def make_exampleinfo(...): return _new_exampleinfo(...) ``` - +[]() ## Executable rules and test rules -Executable rules define targets that can be invoked by a `bazel run` command. -Test rules are a special kind of executable rule whose targets can also be -invoked by a `bazel test` command. Executable and test rules are created by -setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or -[`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: +Executable rules define targets that can be invoked by a `bazel run` command. Test rules are a special kind of executable rule whose targets can also be invoked by a `bazel test` command. Executable and test rules are created by setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or [`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: ```python example_binary = rule( @@ -640,17 +423,9 @@ example_test = rule( ) ``` -Test rules must have names that end in `_test`. (Test *target* names also often -end in `_test` by convention, but this is not required.) Non-test rules must not -have this suffix. +Test rules must have names that end in `_test`. (Test *target* names also often end in `_test` by convention, but this is not required.) Non-test rules must not have this suffix. -Both kinds of rules must produce an executable output file (which may or may not -be predeclared) that will be invoked by the `run` or `test` commands. To tell -Bazel which of a rule's outputs to use as this executable, pass it as the -`executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) -provider. That `executable` is added to the default outputs of the rule (so you -don't need to pass that to both `executable` and `files`). It's also implicitly -added to the [runfiles](#runfiles): +Both kinds of rules must produce an executable output file (which may or may not be predeclared) that will be invoked by the `run` or `test` commands. To tell Bazel which of a rule's outputs to use as this executable, pass it as the `executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) provider. That `executable` is added to the default outputs of the rule (so you don't need to pass that to both `executable` and `files`). It's also implicitly added to the [runfiles](#runfiles): ```python def _example_binary_impl(ctx): @@ -662,30 +437,13 @@ def _example_binary_impl(ctx): ] ``` -The action that generates this file must set the executable bit on the file. For -a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or -[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done -by the underlying tool that is invoked by the action. For a -[`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. - -As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a -special `ctx.outputs.executable` predeclared output. This file serves as the -default executable if you don't specify one using `DefaultInfo`; it must not be -used otherwise. This output mechanism is deprecated because it doesn't support -customizing the executable file's name at analysis time. - -See examples of an -[executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) -and a -[test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). - -[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and -[test rules](/reference/be/common-definitions#common-attributes-tests) have additional -attributes implicitly defined, in addition to those added for -[all rules](/reference/be/common-definitions#common-attributes). The defaults of -implicitly-added attributes can't be changed, though this can be worked around -by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the -default: +The action that generates this file must set the executable bit on the file. For a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done by the underlying tool that is invoked by the action. For a [`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. + +As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a special `ctx.outputs.executable` predeclared output. This file serves as the default executable if you don't specify one using `DefaultInfo`; it must not be used otherwise. This output mechanism is deprecated because it doesn't support customizing the executable file's name at analysis time. + +See examples of an [executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) and a [test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). + +[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and [test rules](/reference/be/common-definitions#common-attributes-tests) have additional attributes implicitly defined, in addition to those added for [all rules](/reference/be/common-definitions#common-attributes). The defaults of implicitly-added attributes can't be changed, though this can be worked around by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the default: ```python def example_test(size = "small", **kwargs): @@ -698,8 +456,7 @@ _example_test = rule( ### Runfiles location -When an executable target is run with `bazel run` (or `test`), the root of the -runfiles directory is adjacent to the executable. The paths relate as follows: +When an executable target is run with `bazel run` (or `test`), the root of the runfiles directory is adjacent to the executable. The paths relate as follows: ```python # Given launcher_path and runfile_file: @@ -710,50 +467,21 @@ execution_root_relative_path = "%s/%s/%s" % ( runfiles_root, workspace_name, runfile_path) ``` -The path to a `File` under the runfiles directory corresponds to -[`File.short_path`](/rules/lib/builtins/File#short_path). +The path to a `File` under the runfiles directory corresponds to [`File.short_path`](/rules/lib/builtins/File#short_path). -The binary executed directly by `bazel` is adjacent to the root of the -`runfiles` directory. However, binaries called *from* the runfiles can't make -the same assumption. To mitigate this, each binary should provide a way to -accept its runfiles root as a parameter using an environment, or command line -argument or flag. This allows binaries to pass the correct canonical runfiles root -to the binaries it calls. If that's not set, a binary can guess that it was the -first binary called and look for an adjacent runfiles directory. +The binary executed directly by `bazel` is adjacent to the root of the `runfiles` directory. However, binaries called *from* the runfiles can't make the same assumption. To mitigate this, each binary should provide a way to accept its runfiles root as a parameter using an environment, or command line argument or flag. This allows binaries to pass the correct canonical runfiles root to the binaries it calls. If that's not set, a binary can guess that it was the first binary called and look for an adjacent runfiles directory. ## Advanced topics ### Requesting output files -A single target can have several output files. When a `bazel build` command is -run, some of the outputs of the targets given to the command are considered to -be *requested*. Bazel only builds these requested files and the files that they -directly or indirectly depend on. (In terms of the action graph, Bazel only -executes the actions that are reachable as transitive dependencies of the -requested files.) - -In addition to [default outputs](#default_outputs), any *predeclared output* can -be explicitly requested on the command line. Rules can specify predeclared -outputs using [output attributes](#output_attributes). In that case, the user -explicitly chooses labels for outputs when they instantiate the rule. To obtain -[`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding -attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can -[implicitly define predeclared outputs](#deprecated_predeclared_outputs) based -on the target name as well, but this feature is deprecated. - -In addition to default outputs, there are *output groups*, which are collections -of output files that may be requested together. These can be requested with -[`--output_groups`](/reference/command-line-reference#flag--output_groups). For -example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` -output group, these files can be built by running `bazel build //pkg:mytarget ---output_groups=debug_files`. Since non-predeclared outputs don't have labels, -they can only be requested by appearing in the default outputs or an output -group. - -Output groups can be specified with the -[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many -built-in providers, `OutputGroupInfo` can take parameters with arbitrary names -to define output groups with that name: +A single target can have several output files. When a `bazel build` command is run, some of the outputs of the targets given to the command are considered to be *requested*. Bazel only builds these requested files and the files that they directly or indirectly depend on. (In terms of the action graph, Bazel only executes the actions that are reachable as transitive dependencies of the requested files.) + +In addition to [default outputs](#default_outputs), any *predeclared output* can be explicitly requested on the command line. Rules can specify predeclared outputs using [output attributes](#output_attributes). In that case, the user explicitly chooses labels for outputs when they instantiate the rule. To obtain [`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can [implicitly define predeclared outputs](#deprecated_predeclared_outputs) based on the target name as well, but this feature is deprecated. + +In addition to default outputs, there are *output groups*, which are collections of output files that may be requested together. These can be requested with [`--output_groups`](/reference/command-line-reference#flag--output_groups). For example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` output group, these files can be built by running `bazel build //pkg:mytarget --output_groups=debug_files`. Since non-predeclared outputs don't have labels, they can only be requested by appearing in the default outputs or an output group. + +Output groups can be specified with the [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many built-in providers, `OutputGroupInfo` can take parameters with arbitrary names to define output groups with that name: ```python def _example_library_impl(ctx): @@ -770,82 +498,37 @@ def _example_library_impl(ctx): ] ``` -Also unlike most providers, `OutputGroupInfo` can be returned by both an -[aspect](/extending/aspects) and the rule target to which that aspect is applied, as -long as they don't define the same output groups. In that case, the resulting -providers are merged. +Also unlike most providers, `OutputGroupInfo` can be returned by both an [aspect](/extending/aspects) and the rule target to which that aspect is applied, as long as they don't define the same output groups. In that case, the resulting providers are merged. -Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts -of files from a target to the actions of its consumers. Define -[rule-specific providers](#custom_providers) for that instead. +Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts of files from a target to the actions of its consumers. Define [rule-specific providers](#custom_providers) for that instead. ### Configurations -Imagine that you want to build a C++ binary for a different architecture. The -build can be complex and involve multiple steps. Some of the intermediate -binaries, like compilers and code generators, have to run on -[the execution platform](/extending/platforms#overview) (which could be your host, -or a remote executor). Some binaries like the final output must be built for the -target architecture. - -For this reason, Bazel has a concept of "configurations" and transitions. The -topmost targets (the ones requested on the command line) are built-in the -"target" configuration, while tools that should run on the execution platform -are built-in an "exec" configuration. Rules may generate different actions based -on the configuration, for instance to change the cpu architecture that is passed -to the compiler. In some cases, the same library may be needed for different -configurations. If this happens, it will be analyzed and potentially built -multiple times. - -By default, Bazel builds a target's dependencies in the same configuration as -the target itself, in other words without transitions. When a dependency is a -tool that's needed to help build the target, the corresponding attribute should -specify a transition to an exec configuration. This causes the tool and all its -dependencies to build for the execution platform. - -For each dependency attribute, you can use `cfg` to decide if dependencies -should build in the same configuration or transition to an exec configuration. -If a dependency attribute has the flag `executable = True`, `cfg` must be set -explicitly. This is to guard against accidentally building a tool for the wrong -configuration. -[See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) - -In general, sources, dependent libraries, and executables that will be needed at -runtime can use the same configuration. - -Tools that are executed as part of the build (such as compilers or code generators) -should be built for an exec configuration. In this case, specify `cfg = "exec"` in -the attribute. - -Otherwise, executables that are used at runtime (such as as part of a test) should -be built for the target configuration. In this case, specify `cfg = "target"` in -the attribute. - -`cfg = "target"` doesn't actually do anything: it's purely a convenience value to -help rule designers be explicit about their intentions. When `executable = False`, -which means `cfg` is optional, only set this when it truly helps readability. - -You can also use `cfg = my_transition` to use -[user-defined transitions](/extending/config#user-defined-transitions), which allow -rule authors a great deal of flexibility in changing configurations, with the -drawback of -[making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). - -**Note**: Historically, Bazel didn't have the concept of execution platforms, -and instead all build actions were considered to run on the host machine. Bazel -versions before 6.0 created a distinct "host" configuration to represent this. -If you see references to "host" in code or old documentation, that's what this -refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual -overhead. - - +Imagine that you want to build a C++ binary for a different architecture. The build can be complex and involve multiple steps. Some of the intermediate binaries, like compilers and code generators, have to run on [the execution platform](/extending/platforms#overview) (which could be your host, or a remote executor). Some binaries like the final output must be built for the target architecture. + +For this reason, Bazel has a concept of "configurations" and transitions. The topmost targets (the ones requested on the command line) are built-in the "target" configuration, while tools that should run on the execution platform are built-in an "exec" configuration. Rules may generate different actions based on the configuration, for instance to change the cpu architecture that is passed to the compiler. In some cases, the same library may be needed for different configurations. If this happens, it will be analyzed and potentially built multiple times. + +By default, Bazel builds a target's dependencies in the same configuration as the target itself, in other words without transitions. When a dependency is a tool that's needed to help build the target, the corresponding attribute should specify a transition to an exec configuration. This causes the tool and all its dependencies to build for the execution platform. + +For each dependency attribute, you can use `cfg` to decide if dependencies should build in the same configuration or transition to an exec configuration. If a dependency attribute has the flag `executable = True`, `cfg` must be set explicitly. This is to guard against accidentally building a tool for the wrong configuration. [See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) + +In general, sources, dependent libraries, and executables that will be needed at runtime can use the same configuration. + +Tools that are executed as part of the build (such as compilers or code generators) should be built for an exec configuration. In this case, specify `cfg = "exec"` in the attribute. + +Otherwise, executables that are used at runtime (such as as part of a test) should be built for the target configuration. In this case, specify `cfg = "target"` in the attribute. + +`cfg = "target"` doesn't actually do anything: it's purely a convenience value to help rule designers be explicit about their intentions. When `executable = False`, which means `cfg` is optional, only set this when it truly helps readability. + +You can also use `cfg = my_transition` to use [user-defined transitions](/extending/config#user-defined-transitions), which allow rule authors a great deal of flexibility in changing configurations, with the drawback of [making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). + +**Note**: Historically, Bazel didn't have the concept of execution platforms, and instead all build actions were considered to run on the host machine. Bazel versions before 6.0 created a distinct "host" configuration to represent this. If you see references to "host" in code or old documentation, that's what this refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual overhead. + +[]() ### Configuration fragments -Rules may access -[configuration fragments](/rules/lib/fragments) such as -`cpp` and `java`. However, all required fragments must be declared in -order to avoid access errors: +Rules may access [configuration fragments](/rules/lib/fragments) such as `cpp` and `java`. However, all required fragments must be declared in order to avoid access errors: ```python def _impl(ctx): @@ -862,14 +545,7 @@ my_rule = rule( ### Runfiles symlinks -Normally, the relative path of a file in the runfiles tree is the same as the -relative path of that file in the source tree or generated output tree. If these -need to be different for some reason, you can specify the `root_symlinks` or -`symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to -files, where the paths are relative to the root of the runfiles directory. The -`symlinks` dictionary is the same, but paths are implicitly prefixed with the -name of the main workspace (*not* the name of the repository containing the -current target). +Normally, the relative path of a file in the runfiles tree is the same as the relative path of that file in the source tree or generated output tree. If these need to be different for some reason, you can specify the `root_symlinks` or `symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to files, where the paths are relative to the root of the runfiles directory. The `symlinks` dictionary is the same, but paths are implicitly prefixed with the name of the main workspace (*not* the name of the repository containing the current target). ```python ... @@ -881,37 +557,20 @@ current target). # sometarget.runfiles/ # some/ # path/ - # here.foo -> some_data_file2 - # / + # here.foo -> some_data_file2 + # <workspace_name>/ # some/ # path/ - # here.bar -> some_data_file3 + # here.bar -> some_data_file3 ``` -If `symlinks` or `root_symlinks` is used, be careful not to map two different -files to the same path in the runfiles tree. This will cause the build to fail -with an error describing the conflict. To fix, you will need to modify your -`ctx.runfiles` arguments to remove the collision. This checking will be done for -any targets using your rule, as well as targets of any kind that depend on those -targets. This is especially risky if your tool is likely to be used transitively -by another tool; symlink names must be unique across the runfiles of a tool and -all of its dependencies. +If `symlinks` or `root_symlinks` is used, be careful not to map two different files to the same path in the runfiles tree. This will cause the build to fail with an error describing the conflict. To fix, you will need to modify your `ctx.runfiles` arguments to remove the collision. This checking will be done for any targets using your rule, as well as targets of any kind that depend on those targets. This is especially risky if your tool is likely to be used transitively by another tool; symlink names must be unique across the runfiles of a tool and all of its dependencies. ### Code coverage -When the [`coverage`](/reference/command-line-reference#coverage) command is run, -the build may need to add coverage instrumentation for certain targets. The -build also gathers the list of source files that are instrumented. The subset of -targets that are considered is controlled by the flag -[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). -Test targets are excluded, unless -[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) -is specified. +When the [`coverage`](/reference/command-line-reference#coverage) command is run, the build may need to add coverage instrumentation for certain targets. The build also gathers the list of source files that are instrumented. The subset of targets that are considered is controlled by the flag [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). Test targets are excluded, unless [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) is specified. -If a rule implementation adds coverage instrumentation at build time, it needs -to account for that in its implementation function. -[ctx.coverage_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns -`True` in coverage mode if a target's sources should be instrumented: +If a rule implementation adds coverage instrumentation at build time, it needs to account for that in its implementation function. [ctx.coverage\_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns `True` in coverage mode if a target's sources should be instrumented: ```python # Are this rule's sources instrumented? @@ -919,13 +578,9 @@ if ctx.coverage_instrumented(): # Do something to turn on coverage for this compile action ``` -Logic that always needs to be on in coverage mode (whether a target's sources -specifically are instrumented or not) can be conditioned on -[ctx.configuration.coverage_enabled](/rules/lib/builtins/configuration#coverage_enabled). +Logic that always needs to be on in coverage mode (whether a target's sources specifically are instrumented or not) can be conditioned on [ctx.configuration.coverage\_enabled](/rules/lib/builtins/configuration#coverage_enabled). -If the rule directly includes sources from its dependencies before compilation -(such as header files), it may also need to turn on compile-time instrumentation if -the dependencies' sources should be instrumented: +If the rule directly includes sources from its dependencies before compilation (such as header files), it may also need to turn on compile-time instrumentation if the dependencies' sources should be instrumented: ```python # Are this rule's sources or any of the sources for its direct dependencies @@ -934,13 +589,7 @@ if ctx.coverage_instrumented() or any([ctx.coverage_instrumented(dep) for dep in # Do something to turn on coverage for this compile action ``` -Rules also should provide information about which attributes are relevant for -coverage with the `InstrumentedFilesInfo` provider, constructed using -[`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). -The `dependency_attributes` parameter of `instrumented_files_info` should list -all runtime dependency attributes, including code dependencies like `deps` and -data dependencies like `data`. The `source_attributes` parameter should list the -rule's source files attributes if coverage instrumentation might be added: +Rules also should provide information about which attributes are relevant for coverage with the `InstrumentedFilesInfo` provider, constructed using [`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). The `dependency_attributes` parameter of `instrumented_files_info` should list all runtime dependency attributes, including code dependencies like `deps` and data dependencies like `data`. The `source_attributes` parameter should list the rule's source files attributes if coverage instrumentation might be added: ```python def _example_library_impl(ctx): @@ -957,18 +606,11 @@ def _example_library_impl(ctx): ] ``` -If `InstrumentedFilesInfo` is not returned, a default one is created with each -non-tool [dependency attribute](#dependency_attributes) that doesn't set -[`cfg`](#configuration) to `"exec"` in the attribute schema. in -`dependency_attributes`. (This isn't ideal behavior, since it puts attributes -like `srcs` in `dependency_attributes` instead of `source_attributes`, but it -avoids the need for explicit coverage configuration for all rules in the -dependency chain.) +If `InstrumentedFilesInfo` is not returned, a default one is created with each non-tool [dependency attribute](#dependency_attributes) that doesn't set [`cfg`](#configuration) to `"exec"` in the attribute schema. in `dependency_attributes`. (This isn't ideal behavior, since it puts attributes like `srcs` in `dependency_attributes` instead of `source_attributes`, but it avoids the need for explicit coverage configuration for all rules in the dependency chain.) #### Test rules -Test rules require additional setup to generate coverage reports. The rule -itself has to add the following implicit attributes: +Test rules require additional setup to generate coverage reports. The rule itself has to add the following implicit attributes: ```python my_test = rule( @@ -991,115 +633,59 @@ my_test = rule( ) ``` -By using `configuration_field`, the dependency on the Java LCOV merger tool can -be avoided as long as coverage is not requested. +By using `configuration_field`, the dependency on the Java LCOV merger tool can be avoided as long as coverage is not requested. -When the test is run, it should emit coverage information in the form of one or -more [LCOV files] -(https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT) -with unique names into the directory specified by the `COVERAGE_DIR` environment -variable. Bazel will then merge these files into a single LCOV file using the -`_lcov_merger` tool. If present, it will also collect C/C++ coverage using the -`_collect_cc_coverage` tool. +When the test is run, it should emit coverage information in the form of one or more \[LCOV files] ([https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT](https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT)) with unique names into the directory specified by the `COVERAGE_DIR` environment variable. Bazel will then merge these files into a single LCOV file using the `_lcov_merger` tool. If present, it will also collect C/C++ coverage using the `_collect_cc_coverage` tool. ### Baseline coverage -Since coverage is only collected for code that ends up in the dependency tree of -a test, coverage reports can be misleading as they don't necessarily cover all -the code matched by the `--instrumentation_filter` flag. +Since coverage is only collected for code that ends up in the dependency tree of a test, coverage reports can be misleading as they don't necessarily cover all the code matched by the `--instrumentation_filter` flag. -For this reason, Bazel allows rules to specify baseline coverage files using the -`baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These -files must be generated in LCOV format by a user-defined action and are supposed -to list all lines, branches, functions and/or blocks in the target's source -files (according to the `sources_attributes` and `extensions` parameters). For -source files in targets that are instrumented for coverage, Bazel merges their -baseline coverage into the combined coverage report generated with -`--combined_report` and thus ensures that untested files still show up as -uncovered. +For this reason, Bazel allows rules to specify baseline coverage files using the `baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These files must be generated in LCOV format by a user-defined action and are supposed to list all lines, branches, functions and/or blocks in the target's source files (according to the `sources_attributes` and `extensions` parameters). For source files in targets that are instrumented for coverage, Bazel merges their baseline coverage into the combined coverage report generated with `--combined_report` and thus ensures that untested files still show up as uncovered. -If a rule doesn't provide any baseline coverage files, Bazel generates synthetic -coverage information that only mentions the source file paths, but doesn't -contain any information about their contents. +If a rule doesn't provide any baseline coverage files, Bazel generates synthetic coverage information that only mentions the source file paths, but doesn't contain any information about their contents. ### Validation Actions -Sometimes you need to validate something about the build, and the -information required to do that validation is available only in artifacts -(source files or generated files). Because this information is in artifacts, -rules can't do this validation at analysis time because rules can't read -files. Instead, actions must do this validation at execution time. When -validation fails, the action will fail, and hence so will the build. - -Examples of validations that might be run are static analysis, linting, -dependency and consistency checks, and style checks. - -Validation actions can also help to improve build performance by moving parts -of actions that are not required for building artifacts into separate actions. -For example, if a single action that does compilation and linting can be -separated into a compilation action and a linting action, then the linting -action can be run as a validation action and run in parallel with other actions. - -These "validation actions" often don't produce anything that is used elsewhere -in the build, since they only need to assert things about their inputs. This -presents a problem though: If a validation action doesn't produce anything that -is used elsewhere in the build, how does a rule get the action to run? -Historically, the approach was to have the validation action output an empty -file, and artificially add that output to the inputs of some other important -action in the build: - - - -This works, because Bazel will always run the validation action when the compile -action is run, but this has significant drawbacks: - -1. The validation action is in the critical path of the build. Because Bazel -thinks the empty output is required to run the compile action, it will run the -validation action first, even though the compile action will ignore the input. -This reduces parallelism and slows down builds. - -2. If other actions in the build might run instead of the -compile action, then the empty outputs of validation actions need to be added to -those actions as well (`java_library`'s source jar output, for example). This is -also a problem if new actions that might run instead of the compile action are -added later, and the empty validation output is accidentally left off. +Sometimes you need to validate something about the build, and the information required to do that validation is available only in artifacts (source files or generated files). Because this information is in artifacts, rules can't do this validation at analysis time because rules can't read files. Instead, actions must do this validation at execution time. When validation fails, the action will fail, and hence so will the build. + +Examples of validations that might be run are static analysis, linting, dependency and consistency checks, and style checks. + +Validation actions can also help to improve build performance by moving parts of actions that are not required for building artifacts into separate actions. For example, if a single action that does compilation and linting can be separated into a compilation action and a linting action, then the linting action can be run as a validation action and run in parallel with other actions. + +These "validation actions" often don't produce anything that is used elsewhere in the build, since they only need to assert things about their inputs. This presents a problem though: If a validation action doesn't produce anything that is used elsewhere in the build, how does a rule get the action to run? Historically, the approach was to have the validation action output an empty file, and artificially add that output to the inputs of some other important action in the build: + +![](/rules/validation_action_historical.svg) + +This works, because Bazel will always run the validation action when the compile action is run, but this has significant drawbacks: + +1. The validation action is in the critical path of the build. Because Bazel thinks the empty output is required to run the compile action, it will run the validation action first, even though the compile action will ignore the input. This reduces parallelism and slows down builds. + +2. If other actions in the build might run instead of the compile action, then the empty outputs of validation actions need to be added to those actions as well (`java_library`'s source jar output, for example). This is also a problem if new actions that might run instead of the compile action are added later, and the empty validation output is accidentally left off. The solution to these problems is to use the Validations Output Group. #### Validations Output Group -The Validations Output Group is an output group designed to hold the otherwise -unused outputs of validation actions, so that they don't need to be artificially -added to the inputs of other actions. +The Validations Output Group is an output group designed to hold the otherwise unused outputs of validation actions, so that they don't need to be artificially added to the inputs of other actions. -This group is special in that its outputs are always requested, regardless of -the value of the `--output_groups` flag, and regardless of how the target is -depended upon (for example, on the command line, as a dependency, or through -implicit outputs of the target). Note that normal caching and incrementality -still apply: if the inputs to the validation action have not changed and the -validation action previously succeeded, then the validation action won't be -run. +This group is special in that its outputs are always requested, regardless of the value of the `--output_groups` flag, and regardless of how the target is depended upon (for example, on the command line, as a dependency, or through implicit outputs of the target). Note that normal caching and incrementality still apply: if the inputs to the validation action have not changed and the validation action previously succeeded, then the validation action won't be run. - +![](/rules/validation_action.svg) -Using this output group still requires that validation actions output some file, -even an empty one. This might require wrapping some tools that normally don't -create outputs so that a file is created. +Using this output group still requires that validation actions output some file, even an empty one. This might require wrapping some tools that normally don't create outputs so that a file is created. A target's validation actions are not run in three cases: -* When the target is depended upon as a tool -* When the target is depended upon as an implicit dependency (for example, an - attribute that starts with "_") -* When the target is built in the exec configuration. +- When the target is depended upon as a tool +- When the target is depended upon as an implicit dependency (for example, an attribute that starts with "\_") +- When the target is built in the exec configuration. -It is assumed that these targets have their own -separate builds and tests that would uncover any validation failures. +It is assumed that these targets have their own separate builds and tests that would uncover any validation failures. #### Using the Validations Output Group -The Validations Output Group is named `_validation` and is used like any other -output group: +The Validations Output Group is named `_validation` and is used like any other output group: ```python def _rule_with_validation_impl(ctx): @@ -1119,7 +705,6 @@ def _rule_with_validation_impl(ctx): OutputGroupInfo(_validation = depset([validation_output])), ] - rule_with_validation = rule( implementation = _rule_with_validation_impl, outputs = { @@ -1136,17 +721,9 @@ rule_with_validation = rule( ) ``` -Notice that the validation output file is not added to the `DefaultInfo` or the -inputs to any other action. The validation action for a target of this rule kind -will still run if the target is depended upon by label, or any of the target's -implicit outputs are directly or indirectly depended upon. +Notice that the validation output file is not added to the `DefaultInfo` or the inputs to any other action. The validation action for a target of this rule kind will still run if the target is depended upon by label, or any of the target's implicit outputs are directly or indirectly depended upon. -It is usually important that the outputs of validation actions only go into the -validation output group, and are not added to the inputs of other actions, as -this could defeat parallelism gains. Note however that Bazel doesn't -have any special checking to enforce this. Therefore, you should test -that validation action outputs are not added to the inputs of any actions in the -tests for Starlark rules. For example: +It is usually important that the outputs of validation actions only go into the validation output group, and are not added to the inputs of other actions, as this could defeat parallelism gains. Note however that Bazel doesn't have any special checking to enforce this. Therefore, you should test that validation action outputs are not added to the inputs of any actions in the tests for Starlark rules. For example: ```python load("@bazel_skylib//lib:unittest.bzl", "analysistest") @@ -1171,8 +748,7 @@ validation_outputs_test = analysistest.make(_validation_outputs_test_impl) #### Validation Actions Flag -Running validation actions is controlled by the `--run_validations` command line -flag, which defaults to true. +Running validation actions is controlled by the `--run_validations` command line flag, which defaults to true. ## Deprecated features @@ -1180,52 +756,23 @@ flag, which defaults to true. There are two **deprecated** ways of using predeclared outputs: -* The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies - a mapping between output attribute names and string templates for generating - predeclared output labels. Prefer using non-predeclared outputs and - explicitly adding outputs to `DefaultInfo.files`. Use the rule target's - label as input for rules which consume the output instead of a predeclared - output's label. +- The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies a mapping between output attribute names and string templates for generating predeclared output labels. Prefer using non-predeclared outputs and explicitly adding outputs to `DefaultInfo.files`. Use the rule target's label as input for rules which consume the output instead of a predeclared output's label. -* For [executable rules](#executable-rules), `ctx.outputs.executable` refers - to a predeclared executable output with the same name as the rule target. - Prefer declaring the output explicitly, for example with - `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that - generates the executable sets its permissions to allow execution. Explicitly - pass the executable output to the `executable` parameter of `DefaultInfo`. +- For [executable rules](#executable-rules), `ctx.outputs.executable` refers to a predeclared executable output with the same name as the rule target. Prefer declaring the output explicitly, for example with `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that generates the executable sets its permissions to allow execution. Explicitly pass the executable output to the `executable` parameter of `DefaultInfo`. ### Runfiles features to avoid -[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) -type have a complex set of features, many of which are kept for legacy reasons. -The following recommendations help reduce complexity: - -* **Avoid** use of the `collect_data` and `collect_default` modes of - [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect - runfiles across certain hardcoded dependency edges in confusing ways. - Instead, add files using the `files` or `transitive_files` parameters of - `ctx.runfiles`, or by merging in runfiles from dependencies with - `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. - -* **Avoid** use of the `data_runfiles` and `default_runfiles` of the - `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. - The distinction between "default" and "data" runfiles is maintained for - legacy reasons. For example, some rules put their default outputs in - `data_runfiles`, but not `default_runfiles`. Instead of using - `data_runfiles`, rules should *both* include default outputs and merge in - `default_runfiles` from attributes which provide runfiles (often - [`data`](/reference/be/common-definitions#common-attributes.data)). - -* When retrieving `runfiles` from `DefaultInfo` (generally only for merging - runfiles between the current rule and its dependencies), use - `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. +[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) type have a complex set of features, many of which are kept for legacy reasons. The following recommendations help reduce complexity: + +- **Avoid** use of the `collect_data` and `collect_default` modes of [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect runfiles across certain hardcoded dependency edges in confusing ways. Instead, add files using the `files` or `transitive_files` parameters of `ctx.runfiles`, or by merging in runfiles from dependencies with `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. + +- **Avoid** use of the `data_runfiles` and `default_runfiles` of the `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. The distinction between "default" and "data" runfiles is maintained for legacy reasons. For example, some rules put their default outputs in `data_runfiles`, but not `default_runfiles`. Instead of using `data_runfiles`, rules should *both* include default outputs and merge in `default_runfiles` from attributes which provide runfiles (often [`data`](/reference/be/common-definitions#common-attributes.data)). + +- When retrieving `runfiles` from `DefaultInfo` (generally only for merging runfiles between the current rule and its dependencies), use `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. ### Migrating from legacy providers -Historically, Bazel providers were simple fields on the `Target` object. They -were accessed using the dot operator, and they were created by putting the field -in a [`struct`](/rules/lib/builtins/struct) returned by the rule's -implementation function instead of a list of provider objects: +Historically, Bazel providers were simple fields on the `Target` object. They were accessed using the dot operator, and they were created by putting the field in a [`struct`](/rules/lib/builtins/struct) returned by the rule's implementation function instead of a list of provider objects: ```python return struct(example_info = struct(headers = depset(...))) @@ -1237,13 +784,9 @@ Such providers can be retrieved from the corresponding field of the `Target` obj transitive_headers = [hdr.example_info.headers for hdr in ctx.attr.hdrs] ``` -*This style is deprecated and should not be used in new code;* see following for -information that may help you migrate. The new provider mechanism avoids name -clashes. It also supports data hiding, by requiring any code accessing a -provider instance to retrieve it using the provider symbol. +*This style is deprecated and should not be used in new code;* see following for information that may help you migrate. The new provider mechanism avoids name clashes. It also supports data hiding, by requiring any code accessing a provider instance to retrieve it using the provider symbol. -For the moment, legacy providers are still supported. A rule can return both -legacy and modern providers as follows: +For the moment, legacy providers are still supported. A rule can return both legacy and modern providers as follows: ```python def _old_rule_impl(ctx): @@ -1260,40 +803,18 @@ def _old_rule_impl(ctx): providers = [modern_data, ...]) ``` -If `dep` is the resulting `Target` object for an instance of this rule, the -providers and their contents can be retrieved as `dep.legacy_info.x` and -`dep[MyInfo].y`. - -In addition to `providers`, the returned struct can also take several other -fields that have special meaning (and thus don't create a corresponding legacy -provider): - -* The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and - `executable` correspond to the same-named fields of - [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of - these fields while also returning a `DefaultInfo` provider. - -* The field `output_groups` takes a struct value and corresponds to an - [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). - -In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in -[`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency -attributes, legacy providers are passed in as strings and modern providers are -passed in by their `Info` symbol. Be sure to change from strings to symbols -when migrating. For complex or large rule sets where it is difficult to update -all rules atomically, you may have an easier time if you follow this sequence of -steps: - -1. Modify the rules that produce the legacy provider to produce both the legacy - and modern providers, using the preceding syntax. For rules that declare they - return the legacy provider, update that declaration to include both the - legacy and modern providers. - -2. Modify the rules that consume the legacy provider to instead consume the - modern provider. If any attribute declarations require the legacy provider, - also update them to instead require the modern provider. Optionally, you can - interleave this work with step 1 by having consumers accept or require either - provider: Test for the presence of the legacy provider using - `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. - -3. Fully remove the legacy provider from all rules. +If `dep` is the resulting `Target` object for an instance of this rule, the providers and their contents can be retrieved as `dep.legacy_info.x` and `dep[MyInfo].y`. + +In addition to `providers`, the returned struct can also take several other fields that have special meaning (and thus don't create a corresponding legacy provider): + +- The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and `executable` correspond to the same-named fields of [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of these fields while also returning a `DefaultInfo` provider. + +- The field `output_groups` takes a struct value and corresponds to an [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). + +In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in [`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency attributes, legacy providers are passed in as strings and modern providers are passed in by their `Info` symbol. Be sure to change from strings to symbols when migrating. For complex or large rule sets where it is difficult to update all rules atomically, you may have an easier time if you follow this sequence of steps: + +1. Modify the rules that produce the legacy provider to produce both the legacy and modern providers, using the preceding syntax. For rules that declare they return the legacy provider, update that declaration to include both the legacy and modern providers. + +2. Modify the rules that consume the legacy provider to instead consume the modern provider. If any attribute declarations require the legacy provider, also update them to instead require the modern provider. Optionally, you can interleave this work with step 1 by having consumers accept or require either provider: Test for the presence of the legacy provider using `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. + +3. Fully remove the legacy provider from all rules. diff --git a/extending/toolchains.mdx b/extending/toolchains.mdx index 0d92d6e1..ef07d948 100644 --- a/extending/toolchains.mdx +++ b/extending/toolchains.mdx @@ -2,23 +2,11 @@ title: 'Toolchains' --- - - -This page describes the toolchain framework, which is a way for rule authors to -decouple their rule logic from platform-based selection of tools. It is -recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) -pages before continuing. This page covers why toolchains are needed, how to -define and use them, and how Bazel selects an appropriate toolchain based on -platform constraints. +This page describes the toolchain framework, which is a way for rule authors to decouple their rule logic from platform-based selection of tools. It is recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) pages before continuing. This page covers why toolchains are needed, how to define and use them, and how Bazel selects an appropriate toolchain based on platform constraints. ## Motivation -Let's first look at the problem toolchains are designed to solve. Suppose you -are writing rules to support the "bar" programming language. Your `bar_binary` -rule would compile `*.bar` files using the `barc` compiler, a tool that itself -is built as another target in your workspace. Since users who write `bar_binary` -targets shouldn't have to specify a dependency on the compiler, you make it an -implicit dependency by adding it to the rule definition as a private attribute. +Let's first look at the problem toolchains are designed to solve. Suppose you are writing rules to support the "bar" programming language. Your `bar_binary` rule would compile `*.bar` files using the `barc` compiler, a tool that itself is built as another target in your workspace. Since users who write `bar_binary` targets shouldn't have to specify a dependency on the compiler, you make it an implicit dependency by adding it to the rule definition as a private attribute. ```python bar_binary = rule( @@ -34,9 +22,7 @@ bar_binary = rule( ) ``` -`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so -it'll be built before any `bar_binary` target. It can be accessed by the rule's -implementation function just like any other attribute: +`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so it'll be built before any `bar_binary` target. It can be accessed by the rule's implementation function just like any other attribute: ```python BarcInfo = provider( @@ -58,16 +44,9 @@ def _bar_binary_impl(ctx): ... ``` -The issue here is that the compiler's label is hardcoded into `bar_binary`, yet -different targets may need different compilers depending on what platform they -are being built for and what platform they are being built on -- called the -*target platform* and *execution platform*, respectively. Furthermore, the rule -author does not necessarily even know all the available tools and platforms, so -it is not feasible to hardcode them in the rule's definition. +The issue here is that the compiler's label is hardcoded into `bar_binary`, yet different targets may need different compilers depending on what platform they are being built for and what platform they are being built on -- called the *target platform* and *execution platform*, respectively. Furthermore, the rule author does not necessarily even know all the available tools and platforms, so it is not feasible to hardcode them in the rule's definition. -A less-than-ideal solution would be to shift the burden onto users, by making -the `_compiler` attribute non-private. Then individual targets could be -hardcoded to build for one platform or another. +A less-than-ideal solution would be to shift the burden onto users, by making the `_compiler` attribute non-private. Then individual targets could be hardcoded to build for one platform or another. ```python bar_binary( @@ -83,8 +62,7 @@ bar_binary( ) ``` -You can improve on this solution by using `select` to choose the `compiler` -[based on the platform](/docs/configurable-attributes): +You can improve on this solution by using `select` to choose the `compiler` [based on the platform](/docs/configurable-attributes): ```python config_setting( @@ -111,26 +89,13 @@ bar_binary( ) ``` -But this is tedious and a bit much to ask of every single `bar_binary` user. -If this style is not used consistently throughout the workspace, it leads to -builds that work fine on a single platform but fail when extended to -multi-platform scenarios. It also does not address the problem of adding support -for new platforms and compilers without modifying existing rules or targets. +But this is tedious and a bit much to ask of every single `bar_binary` user. If this style is not used consistently throughout the workspace, it leads to builds that work fine on a single platform but fail when extended to multi-platform scenarios. It also does not address the problem of adding support for new platforms and compilers without modifying existing rules or targets. -The toolchain framework solves this problem by adding an extra level of -indirection. Essentially, you declare that your rule has an abstract dependency -on *some* member of a family of targets (a toolchain type), and Bazel -automatically resolves this to a particular target (a toolchain) based on the -applicable platform constraints. Neither the rule author nor the target author -need know the complete set of available platforms and toolchains. +The toolchain framework solves this problem by adding an extra level of indirection. Essentially, you declare that your rule has an abstract dependency on *some* member of a family of targets (a toolchain type), and Bazel automatically resolves this to a particular target (a toolchain) based on the applicable platform constraints. Neither the rule author nor the target author need know the complete set of available platforms and toolchains. ## Writing rules that use toolchains -Under the toolchain framework, instead of having rules depend directly on tools, -they instead depend on *toolchain types*. A toolchain type is a simple target -that represents a class of tools that serve the same role for different -platforms. For instance, you can declare a type that represents the bar -compiler: +Under the toolchain framework, instead of having rules depend directly on tools, they instead depend on *toolchain types*. A toolchain type is a simple target that represents a class of tools that serve the same role for different platforms. For instance, you can declare a type that represents the bar compiler: ```python # By convention, toolchain_type targets are named "toolchain_type" and @@ -139,9 +104,7 @@ compiler: toolchain_type(name = "toolchain_type") ``` -The rule definition in the previous section is modified so that instead of -taking in the compiler as an attribute, it declares that it consumes a -`//bar_tools:toolchain_type` toolchain. +The rule definition in the previous section is modified so that instead of taking in the compiler as an attribute, it declares that it consumes a `//bar_tools:toolchain_type` toolchain. ```python bar_binary = rule( @@ -155,8 +118,7 @@ bar_binary = rule( ) ``` -The implementation function now accesses this dependency under `ctx.toolchains` -instead of `ctx.attr`, using the toolchain type as the key. +The implementation function now accesses this dependency under `ctx.toolchains` instead of `ctx.attr`, using the toolchain type as the key. ```python def _bar_binary_impl(ctx): @@ -171,28 +133,15 @@ def _bar_binary_impl(ctx): ... ``` -`ctx.toolchains["//bar_tools:toolchain_type"]` returns the -[`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) -of whatever target Bazel resolved the toolchain dependency to. The fields of the -`ToolchainInfo` object are set by the underlying tool's rule; in the next -section, this rule is defined such that there is a `barcinfo` field that wraps -a `BarcInfo` object. +`ctx.toolchains["//bar_tools:toolchain_type"]` returns the [`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) of whatever target Bazel resolved the toolchain dependency to. The fields of the `ToolchainInfo` object are set by the underlying tool's rule; in the next section, this rule is defined such that there is a `barcinfo` field that wraps a `BarcInfo` object. -Bazel's procedure for resolving toolchains to targets is described -[below](#toolchain-resolution). Only the resolved toolchain target is actually -made a dependency of the `bar_binary` target, not the whole space of candidate -toolchains. +Bazel's procedure for resolving toolchains to targets is described [below](#toolchain-resolution). Only the resolved toolchain target is actually made a dependency of the `bar_binary` target, not the whole space of candidate toolchains. ### Mandatory and Optional Toolchains -By default, when a rule expresses a toolchain type dependency using a bare label -(as shown above), the toolchain type is considered to be **mandatory**. If Bazel -is unable to find a matching toolchain (see -[Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain -type, this is an error and analysis halts. +By default, when a rule expresses a toolchain type dependency using a bare label (as shown above), the toolchain type is considered to be **mandatory**. If Bazel is unable to find a matching toolchain (see [Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain type, this is an error and analysis halts. -It is possible instead to declare an **optional** toolchain type dependency, as -follows: +It is possible instead to declare an **optional** toolchain type dependency, as follows: ```python bar_binary = rule( @@ -203,20 +152,20 @@ bar_binary = rule( ) ``` -When an optional toolchain type cannot be resolved, analysis continues, and the -result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. +When an optional toolchain type cannot be resolved, analysis continues, and the result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. -The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) -function defaults to mandatory. +The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) function defaults to mandatory. The following forms can be used: -- Mandatory toolchain types: - - `toolchains = ["//bar_tools:toolchain_type"]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` +- Mandatory toolchain types: + + - `toolchains = ["//bar_tools:toolchain_type"]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` + - Optional toolchain types: - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` ```python bar_binary = rule( @@ -228,15 +177,11 @@ bar_binary = rule( ) ``` -You can mix and match forms in the same rule, also. However, if the same -toolchain type is listed multiple times, it will take the most strict version, -where mandatory is more strict than optional. +You can mix and match forms in the same rule, also. However, if the same toolchain type is listed multiple times, it will take the most strict version, where mandatory is more strict than optional. ### Writing aspects that use toolchains -Aspects have access to the same toolchain API as rules: you can define required -toolchain types, access toolchains via the context, and use them to generate new -actions using the toolchain. +Aspects have access to the same toolchain API as rules: you can define required toolchain types, access toolchains via the context, and use them to generate new actions using the toolchain. ```py bar_aspect = aspect( @@ -255,28 +200,15 @@ def _bar_aspect_impl(target, ctx): To define some toolchains for a given toolchain type, you need three things: -1. A language-specific rule representing the kind of tool or tool suite. By - convention this rule's name is suffixed with "\_toolchain". +1. A language-specific rule representing the kind of tool or tool suite. By convention this rule's name is suffixed with "\_toolchain". - 1. **Note:** The `\_toolchain` rule cannot create any build actions. - Rather, it collects artifacts from other rules and forwards them to the - rule that uses the toolchain. That rule is responsible for creating all - build actions. + 1. **Note:** The `\_toolchain` rule cannot create any build actions. Rather, it collects artifacts from other rules and forwards them to the rule that uses the toolchain. That rule is responsible for creating all build actions. -2. Several targets of this rule type, representing versions of the tool or tool - suite for different platforms. +2. Several targets of this rule type, representing versions of the tool or tool suite for different platforms. -3. For each such target, an associated target of the generic - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - rule, to provide metadata used by the toolchain framework. This `toolchain` - target also refers to the `toolchain_type` associated with this toolchain. - This means that a given `_toolchain` rule could be associated with any - `toolchain_type`, and that only in a `toolchain` instance that uses - this `_toolchain` rule that the rule is associated with a `toolchain_type`. +3. For each such target, an associated target of the generic [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) rule, to provide metadata used by the toolchain framework. This `toolchain` target also refers to the `toolchain_type` associated with this toolchain. This means that a given `_toolchain` rule could be associated with any `toolchain_type`, and that only in a `toolchain` instance that uses this `_toolchain` rule that the rule is associated with a `toolchain_type`. -For our running example, here's a definition for a `bar_toolchain` rule. Our -example has only a compiler, but other tools such as a linker could also be -grouped underneath it. +For our running example, here's a definition for a `bar_toolchain` rule. Our example has only a compiler, but other tools such as a linker could also be grouped underneath it. ```python def _bar_toolchain_impl(ctx): @@ -299,13 +231,7 @@ bar_toolchain = rule( ) ``` -The rule must return a `ToolchainInfo` provider, which becomes the object that -the consuming rule retrieves using `ctx.toolchains` and the label of the -toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value -pairs. The specification of exactly what fields are added to the `ToolchainInfo` -should be clearly documented at the toolchain type. In this example, the values -return wrapped in a `BarcInfo` object to reuse the schema defined above; this -style may be useful for validation and code reuse. +The rule must return a `ToolchainInfo` provider, which becomes the object that the consuming rule retrieves using `ctx.toolchains` and the label of the toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value pairs. The specification of exactly what fields are added to the `ToolchainInfo` should be clearly documented at the toolchain type. In this example, the values return wrapped in a `BarcInfo` object to reuse the schema defined above; this style may be useful for validation and code reuse. Now you can define targets for specific `barc` compilers. @@ -331,10 +257,7 @@ bar_toolchain( ) ``` -Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. -These definitions link the language-specific targets to the toolchain type and -provide the constraint information that tells Bazel when the toolchain is -appropriate for a given platform. +Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. These definitions link the language-specific targets to the toolchain type and provide the constraint information that tells Bazel when the toolchain is appropriate for a given platform. ```python toolchain( @@ -366,21 +289,13 @@ toolchain( ) ``` -The use of relative path syntax above suggests these definitions are all in the -same package, but there's no reason the toolchain type, language-specific -toolchain targets, and `toolchain` definition targets can't all be in separate -packages. +The use of relative path syntax above suggests these definitions are all in the same package, but there's no reason the toolchain type, language-specific toolchain targets, and `toolchain` definition targets can't all be in separate packages. -See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) -for a real-world example. +See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) for a real-world example. ### Toolchains and configurations -An important question for rule authors is, when a `bar_toolchain` target is -analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions -should be used for dependencies? The example above uses string attributes, but -what would happen for a more complicated toolchain that depends on other targets -in the Bazel repository? +An important question for rule authors is, when a `bar_toolchain` target is analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions should be used for dependencies? The example above uses string attributes, but what would happen for a more complicated toolchain that depends on other targets in the Bazel repository? Let's see a more complex version of `bar_toolchain`: @@ -406,32 +321,13 @@ bar_toolchain = rule( ) ``` -The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, -but the meaning of the `cfg` parameter is slightly different. - -The dependency from a target (called the "parent") to a toolchain via toolchain -resolution uses a special configuration transition called the "toolchain -transition". The toolchain transition keeps the configuration the same, except -that it forces the execution platform to be the same for the toolchain as for -the parent (otherwise, toolchain resolution for the toolchain could pick any -execution platform, and wouldn't necessarily be the same as for parent). This -allows any `exec` dependencies of the toolchain to also be executable for the -parent's build actions. Any of the toolchain's dependencies which use `cfg = -"target"` (or which don't specify `cfg`, since "target" is the default) are -built for the same target platform as the parent. This allows toolchain rules to -contribute both libraries (the `system_lib` attribute above) and tools (the -`compiler` attribute) to the build rules which need them. The system libraries -are linked into the final artifact, and so need to be built for the same -platform, whereas the compiler is a tool invoked during the build, and needs to -be able to run on the execution platform. +The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, but the meaning of the `cfg` parameter is slightly different. + +The dependency from a target (called the "parent") to a toolchain via toolchain resolution uses a special configuration transition called the "toolchain transition". The toolchain transition keeps the configuration the same, except that it forces the execution platform to be the same for the toolchain as for the parent (otherwise, toolchain resolution for the toolchain could pick any execution platform, and wouldn't necessarily be the same as for parent). This allows any `exec` dependencies of the toolchain to also be executable for the parent's build actions. Any of the toolchain's dependencies which use `cfg = "target"` (or which don't specify `cfg`, since "target" is the default) are built for the same target platform as the parent. This allows toolchain rules to contribute both libraries (the `system_lib` attribute above) and tools (the `compiler` attribute) to the build rules which need them. The system libraries are linked into the final artifact, and so need to be built for the same platform, whereas the compiler is a tool invoked during the build, and needs to be able to run on the execution platform. ## Registering and building with toolchains -At this point all the building blocks are assembled, and you just need to make -the toolchains available to Bazel's resolution procedure. This is done by -registering the toolchain, either in a `MODULE.bazel` file using -`register_toolchains()`, or by passing the toolchains' labels on the command -line using the `--extra_toolchains` flag. +At this point all the building blocks are assembled, and you just need to make the toolchains available to Bazel's resolution procedure. This is done by registering the toolchain, either in a `MODULE.bazel` file using `register_toolchains()`, or by passing the toolchains' labels on the command line using the `--extra_toolchains` flag. ```python register_toolchains( @@ -444,16 +340,12 @@ register_toolchains( ) ``` -When using target patterns to register toolchains, the order in which the -individual toolchains are registered is determined by the following rules: +When using target patterns to register toolchains, the order in which the individual toolchains are registered is determined by the following rules: -* The toolchains defined in a subpackage of a package are registered before the - toolchains defined in the package itself. -* Within a package, toolchains are registered in the lexicographical order of - their names. +- The toolchains defined in a subpackage of a package are registered before the toolchains defined in the package itself. +- Within a package, toolchains are registered in the lexicographical order of their names. -Now when you build a target that depends on a toolchain type, an appropriate -toolchain will be selected based on the target and execution platforms. +Now when you build a target that depends on a toolchain type, an appropriate toolchain will be selected based on the target and execution platforms. ```python # my_pkg/BUILD @@ -475,109 +367,52 @@ bar_binary( bazel build //my_pkg:my_bar_binary --platforms=//my_pkg:my_target_platform ``` -Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that -has `@platforms//os:linux` and therefore resolve the -`//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. -This will end up building `//bar_tools:barc_linux` but not -`//bar_tools:barc_windows`. +Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that has `@platforms//os:linux` and therefore resolve the `//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. This will end up building `//bar_tools:barc_linux` but not `//bar_tools:barc_windows`. ## Toolchain resolution -Note: [Some Bazel rules](/concepts/platforms#status) do not yet support -toolchain resolution. - -For each target that uses toolchains, Bazel's toolchain resolution procedure -determines the target's concrete toolchain dependencies. The procedure takes as -input a set of required toolchain types, the target platform, the list of -available execution platforms, and the list of available toolchains. Its outputs -are a selected toolchain for each toolchain type as well as a selected execution -platform for the current target. - -The available execution platforms and toolchains are gathered from the -external dependency graph via -[`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) -and -[`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in -`MODULE.bazel` files. -Additional execution platforms and toolchains may also be specified on the -command line via -[`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) -and -[`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). -The host platform is automatically included as an available execution platform. -Available platforms and toolchains are tracked as ordered lists for determinism, -with preference given to earlier items in the list. - -The set of available toolchains, in priority order, is created from -`--extra_toolchains` and `register_toolchains`: - -1. Toolchains registered using `--extra_toolchains` are added first. (Within - these, the **last** toolchain has highest priority.) -2. Toolchains registered using `register_toolchains` in the transitive external - dependency graph, in the following order: (Within these, the **first** - mentioned toolchain has highest priority.) - 1. Toolchains registered by the root module (as in, the `MODULE.bazel` at the - workspace root); - 2. Toolchains registered in the user's `WORKSPACE` file, including in any - macros invoked from there; - 3. Toolchains registered by non-root modules (as in, dependencies specified by - the root module, and their dependencies, and so forth); - 4. Toolchains registered in the "WORKSPACE suffix"; this is only used by - certain native rules bundled with the Bazel installation. - -**NOTE:** [Pseudo-targets like `:all`, `:*`, and -`/...`](/run/build#specifying-build-targets) are ordered by Bazel's package -loading mechanism, which uses a lexicographic ordering. +Note: [Some Bazel rules](/concepts/platforms#status) do not yet support toolchain resolution. + +For each target that uses toolchains, Bazel's toolchain resolution procedure determines the target's concrete toolchain dependencies. The procedure takes as input a set of required toolchain types, the target platform, the list of available execution platforms, and the list of available toolchains. Its outputs are a selected toolchain for each toolchain type as well as a selected execution platform for the current target. + +The available execution platforms and toolchains are gathered from the external dependency graph via [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) and [`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in `MODULE.bazel` files. Additional execution platforms and toolchains may also be specified on the command line via [`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) and [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). The host platform is automatically included as an available execution platform. Available platforms and toolchains are tracked as ordered lists for determinism, with preference given to earlier items in the list. + +The set of available toolchains, in priority order, is created from `--extra_toolchains` and `register_toolchains`: + +1. Toolchains registered using `--extra_toolchains` are added first. (Within these, the **last** toolchain has highest priority.) +2. Toolchains registered using `register_toolchains` in the transitive external dependency graph, in the following order: (Within these, the **first** mentioned toolchain has highest priority.) +3. Toolchains registered by the root module (as in, the `MODULE.bazel` at the workspace root); +4. Toolchains registered in the user's `WORKSPACE` file, including in any macros invoked from there; +5. Toolchains registered by non-root modules (as in, dependencies specified by the root module, and their dependencies, and so forth); +6. Toolchains registered in the "WORKSPACE suffix"; this is only used by certain native rules bundled with the Bazel installation. + +**NOTE:** [Pseudo-targets like `:all`, `:*`, and `/...`](/run/build#specifying-build-targets) are ordered by Bazel's package loading mechanism, which uses a lexicographic ordering. The resolution steps are as follows. -1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a - platform if, for each `constraint_value` in its list, the platform also has - that `constraint_value` (either explicitly or as a default). +1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a platform if, for each `constraint_value` in its list, the platform also has that `constraint_value` (either explicitly or as a default). - If the platform has `constraint_value`s from `constraint_setting`s not - referenced by the clause, these do not affect matching. + If the platform has `constraint_value`s from `constraint_setting`s not referenced by the clause, these do not affect matching. -1. If the target being built specifies the - [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) - (or its rule definition specifies the - [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), - the list of available execution platforms is filtered to remove - any that do not match the execution constraints. +2. If the target being built specifies the [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) (or its rule definition specifies the [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), the list of available execution platforms is filtered to remove any that do not match the execution constraints. -1. The list of available toolchains is filtered to remove any toolchains - specifying `target_settings` that don't match the current configuration. +3. The list of available toolchains is filtered to remove any toolchains specifying `target_settings` that don't match the current configuration. -1. For each available execution platform, you associate each toolchain type with - the first available toolchain, if any, that is compatible with this execution - platform and the target platform. +4. For each available execution platform, you associate each toolchain type with the first available toolchain, if any, that is compatible with this execution platform and the target platform. -1. Any execution platform that failed to find a compatible mandatory toolchain - for one of its toolchain types is ruled out. Of the remaining platforms, the - first one becomes the current target's execution platform, and its associated - toolchains (if any) become dependencies of the target. +5. Any execution platform that failed to find a compatible mandatory toolchain for one of its toolchain types is ruled out. Of the remaining platforms, the first one becomes the current target's execution platform, and its associated toolchains (if any) become dependencies of the target. -The chosen execution platform is used to run all actions that the target -generates. +The chosen execution platform is used to run all actions that the target generates. -In cases where the same target can be built in multiple configurations (such as -for different CPUs) within the same build, the resolution procedure is applied -independently to each version of the target. +In cases where the same target can be built in multiple configurations (such as for different CPUs) within the same build, the resolution procedure is applied independently to each version of the target. -If the rule uses [execution groups](/extending/exec-groups), each execution -group performs toolchain resolution separately, and each has its own execution -platform and toolchains. +If the rule uses [execution groups](/extending/exec-groups), each execution group performs toolchain resolution separately, and each has its own execution platform and toolchains. ## Debugging toolchains -If you are adding toolchain support to an existing rule, use the -`--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag -provides verbose output for toolchain types or target names that match the regex variable. You -can use `.*` to output all information. Bazel will output names of toolchains it -checks and skips during the resolution process. +If you are adding toolchain support to an existing rule, use the `--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag provides verbose output for toolchain types or target names that match the regex variable. You can use `.*` to output all information. Bazel will output names of toolchains it checks and skips during the resolution process. -For example, to debug toolchain selection for all actions created directly by -`//my:target`: +For example, to debug toolchain selection for all actions created directly by `//my:target`: ```sh $ bazel build //my:all --toolchain_resolution_debug=//my:target @@ -589,8 +424,7 @@ To debug toolchain selection for all actions over all build targets: $ bazel build //my:all --toolchain_resolution_debug=.* ``` -If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain -resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: +If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: ``` # Find all direct dependencies of //cc:my_cc_lib. This includes explicitly @@ -609,5 +443,5 @@ $ bazel cquery 'deps(//cc:my_cc_lib, 1)' # Which of these are from toolchain resolution? $ bazel cquery 'deps(//cc:my_cc_lib, 1)' --transitions=lite | grep "toolchain dependency" - [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 + [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 ``` diff --git a/external/extension.mdx b/external/extension.mdx index ababc7fe..679096e2 100644 --- a/external/extension.mdx +++ b/external/extension.mdx @@ -2,42 +2,20 @@ title: 'Module extensions' --- +Module extensions allow users to extend the module system by reading input data from modules across the dependency graph, performing necessary logic to resolve dependencies, and finally creating repos by calling [repo rules](/external/repo). These extensions have capabilities similar to repo rules, which enables them to perform file I/O, send network requests, and so on. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. - -Module extensions allow users to extend the module system by reading input data -from modules across the dependency graph, performing necessary logic to resolve -dependencies, and finally creating repos by calling [repo -rules](/external/repo). These extensions have capabilities similar to repo -rules, which enables them to perform file I/O, send network requests, and so on. -Among other things, they allow Bazel to interact with other package management -systems while also respecting the dependency graph built out of Bazel modules. - -You can define module extensions in `.bzl` files, just like repo rules. They're -not invoked directly; rather, each module specifies pieces of data called *tags* -for extensions to read. Bazel runs module resolution before evaluating any -extensions. The extension reads all the tags belonging to it across the entire -dependency graph. +You can define module extensions in `.bzl` files, just like repo rules. They're not invoked directly; rather, each module specifies pieces of data called *tags* for extensions to read. Bazel runs module resolution before evaluating any extensions. The extension reads all the tags belonging to it across the entire dependency graph. ## Extension usage -Extensions are hosted in Bazel modules themselves. To use an extension in a -module, first add a `bazel_dep` on the module hosting the extension, and then -call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function -to bring it into scope. Consider the following example — a snippet from a -`MODULE.bazel` file to use the "maven" extension defined in the -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) -module: +Extensions are hosted in Bazel modules themselves. To use an extension in a module, first add a `bazel_dep` on the module hosting the extension, and then call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function to bring it into scope. Consider the following example — a snippet from a `MODULE.bazel` file to use the "maven" extension defined in the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) module: ```python bazel_dep(name = "rules_jvm_external", version = "4.5") maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -This binds the return value of `use_extension` to a variable, which allows the -user to use dot-syntax to specify tags for the extension. The tags must follow -the schema defined by the corresponding *tag classes* specified in the -[extension definition](#extension_definition). For an example specifying some -`maven.install` and `maven.artifact` tags: +This binds the return value of `use_extension` to a variable, which allows the user to use dot-syntax to specify tags for the extension. The tags must follow the schema defined by the corresponding *tag classes* specified in the [extension definition](#extension_definition). For an example specifying some `maven.install` and `maven.artifact` tags: ```python maven.install(artifacts = ["org.junit:junit:4.13.2"]) @@ -47,33 +25,19 @@ maven.artifact(group = "com.google.guava", exclusions = ["com.google.j2objc:j2objc-annotations"]) ``` -Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos -generated by the extension into the scope of the current module. +Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos generated by the extension into the scope of the current module. ```python use_repo(maven, "maven") ``` -Repos generated by an extension are part of its API. In this example, the -"maven" module extension promises to generate a repo called `maven`. With the -declaration above, the extension properly resolves labels such as -`@maven//:org_junit_junit` to point to the repo generated by the "maven" -extension. +Repos generated by an extension are part of its API. In this example, the "maven" module extension promises to generate a repo called `maven`. With the declaration above, the extension properly resolves labels such as `@maven//:org_junit_junit` to point to the repo generated by the "maven" extension. -Note: Module extensions are evaluated lazily. This means that an extension will -typically not be evaluated unless some module brings one of its repositories -into scope using `use_repo` and that repository is referenced in a build. While -testing a module extension, `bazel mod deps` can be useful as it -unconditionally evaluates all module extensions. +Note: Module extensions are evaluated lazily. This means that an extension will typically not be evaluated unless some module brings one of its repositories into scope using `use_repo` and that repository is referenced in a build. While testing a module extension, `bazel mod deps` can be useful as it unconditionally evaluates all module extensions. ## Extension definition -You can define module extensions similarly to [repo rules](/external/repo), -using the [`module_extension`](/rules/lib/globals/bzl#module_extension) -function. However, while repo rules have a number of attributes, module -extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which -has a number of attributes. The tag classes define schemas for tags used by this -extension. For example, the "maven" extension above might be defined like this: +You can define module extensions similarly to [repo rules](/external/repo), using the [`module_extension`](/rules/lib/globals/bzl#module_extension) function. However, while repo rules have a number of attributes, module extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which has a number of attributes. The tag classes define schemas for tags used by this extension. For example, the "maven" extension above might be defined like this: ```python # @rules_jvm_external//:extensions.bzl @@ -86,13 +50,9 @@ maven = module_extension( ) ``` -These declarations show that `maven.install` and `maven.artifact` tags can be -specified using the specified attribute schema. +These declarations show that `maven.install` and `maven.artifact` tags can be specified using the specified attribute schema. -The implementation function of module extensions are similar to those of repo -rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, -which grants access to all modules using the extension and all pertinent tags. -The implementation function then calls repo rules to generate repos. +The implementation function of module extensions are similar to those of repo rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, which grants access to all modules using the extension and all pertinent tags. The implementation function then calls repo rules to generate repos. ```python # @rules_jvm_external//:extensions.bzl @@ -120,65 +80,33 @@ def _maven_impl(ctx): ### Extension identity -Module extensions are identified by the name and the `.bzl` file that appears -in the call to `use_extension`. In the following example, the extension `maven` -is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the -name `maven`: +Module extensions are identified by the name and the `.bzl` file that appears in the call to `use_extension`. In the following example, the extension `maven` is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the name `maven`: ```python maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -Re-exporting an extension from a different `.bzl` file gives it a new identity -and if both versions of the extension are used in the transitive module graph, -then they will be evaluated separately and will only see the tags associated -with that particular identity. +Re-exporting an extension from a different `.bzl` file gives it a new identity and if both versions of the extension are used in the transitive module graph, then they will be evaluated separately and will only see the tags associated with that particular identity. -As an extension author you should make sure that users will only use your -module extension from one single `.bzl` file. +As an extension author you should make sure that users will only use your module extension from one single `.bzl` file. ## Repository names and visibility -Repos generated by extensions have canonical names in the form of `{{ "" -}}module_repo_canonical_name+extension_name{{ -"" }}+repo_name`. Note that the canonical name -format is not an API you should depend on — it's subject to change at any time. - -This naming policy means that each extension has its own "repo namespace"; two -distinct extensions can each define a repo with the same name without risking -any clashes. It also means that `repository_ctx.name` reports the canonical name -of the repo, which is *not* the same as the name specified in the repo rule -call. - -Taking repos generated by module extensions into consideration, there are -several repo visibility rules: - -* A Bazel module repo can see all repos introduced in its `MODULE.bazel` file - via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and - [`use_repo`](/rules/lib/globals/module#use_repo). -* A repo generated by a module extension can see all repos visible to the - module that hosts the extension, *plus* all other repos generated by the - same module extension (using the names specified in the repo rule calls as - their apparent names). - * This might result in a conflict. If the module repo can see a repo with - the apparent name `foo`, and the extension generates a repo with the - specified name `foo`, then for all repos generated by that extension - `foo` refers to the former. -* Similarly, in a module extension's implementation function, repos created - by the extension can refer to each other by their apparent names in - attributes, regardless of the order in which they are created. - * In case of a conflict with a repository visible to the module, labels - passed to repository rule attributes can be wrapped in a call to - [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to - the repo visible to the module instead of the extension-generated repo - of the same name. +Repos generated by extensions have canonical names in the form of `<var>module_repo_canonical_name</var>+<var>extension_name</var>+<var>repo_name</var>`. Note that the canonical name format is not an API you should depend on — it's subject to change at any time. + +This naming policy means that each extension has its own "repo namespace"; two distinct extensions can each define a repo with the same name without risking any clashes. It also means that `repository_ctx.name` reports the canonical name of the repo, which is *not* the same as the name specified in the repo rule call. + +Taking repos generated by module extensions into consideration, there are several repo visibility rules: + +- A Bazel module repo can see all repos introduced in its `MODULE.bazel` file via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_repo`](/rules/lib/globals/module#use_repo). +- A repo generated by a module extension can see all repos visible to the module that hosts the extension, *plus* all other repos generated by the same module extension (using the names specified in the repo rule calls as their apparent names). + - This might result in a conflict. If the module repo can see a repo with the apparent name `foo`, and the extension generates a repo with the specified name `foo`, then for all repos generated by that extension `foo` refers to the former. +- Similarly, in a module extension's implementation function, repos created by the extension can refer to each other by their apparent names in attributes, regardless of the order in which they are created. + - In case of a conflict with a repository visible to the module, labels passed to repository rule attributes can be wrapped in a call to [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to the repo visible to the module instead of the extension-generated repo of the same name. ### Overriding and injecting module extension repos -The root module can use -[`override_repo`](/rules/lib/globals/module#override_repo) and -[`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject -module extension repos. +The root module can use [`override_repo`](/rules/lib/globals/module#override_repo) and [`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject module extension repos. #### Example: Replacing `rules_java`'s `java_tools` with a vendored copy @@ -232,80 +160,32 @@ inject_repo(go_deps, "zlib") ## Best practices -This section describes best practices when writing extensions so they are -straightforward to use, maintainable, and adapt well to changes over time. +This section describes best practices when writing extensions so they are straightforward to use, maintainable, and adapt well to changes over time. ### Put each extension in a separate file -When extensions are in a different files, it allows one extension to load -repositories generated by another extension. Even if you don't use this -functionality, it's best to put them in separate files in case you need it -later. This is because the extension's identify is based on its file, so moving -the extension into another file later changes your public API and is a backwards -incompatible change for your users. +When extensions are in a different files, it allows one extension to load repositories generated by another extension. Even if you don't use this functionality, it's best to put them in separate files in case you need it later. This is because the extension's identify is based on its file, so moving the extension into another file later changes your public API and is a backwards incompatible change for your users. ### Specify reproducibility and use facts -If your extension always defines the same repositories given the same inputs -(extension tags, files it reads, etc.) and in particular doesn't rely on -any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by -a checksum, consider returning -[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with -`reproducible = True`. This allows Bazel to skip this extension when writing to -the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces -the chance of merge conflicts. Note that Bazel still caches the results of -reproducible extensions in a way that persists across server restarts, so even -a long-running extension can be marked as reproducible without a performance -penalty. - -If your extension relies on effectively immutable data obtained from outside -the build, most commonly from the network, but you don't have a checksum -available to guard the download, consider using the `facts` parameter of -[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to -persistently record such data and thus allow your extension to become -reproducible. `facts` is expected to be a dictionary with string keys and -arbitrary JSON-like Starlark values that is always persisted in the lockfile and -available to future evaluations of the extension via the -[`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. - -`facts` are not invalidated even when the code of your module extension changes, -so be prepared to handle the case where the structure of `facts` changes. -Bazel also assumes that two different `facts` dicts produced by two different -evaluations of the same extension can be shallowly merged (i.e., as if by using -the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` -not supporting enumeration of its entries, just lookups by key. - -An example of using `facts` would be to record a mapping from version numbers of -some SDK to the an object containing the download URL and checksum of that -version. The first time the extension is evaluated, it can fetch this mapping -from the network, but on later evaluations it can use the mapping from `facts` -to avoid the network requests. +If your extension always defines the same repositories given the same inputs (extension tags, files it reads, etc.) and in particular doesn't rely on any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by a checksum, consider returning [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with `reproducible = True`. This allows Bazel to skip this extension when writing to the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces the chance of merge conflicts. Note that Bazel still caches the results of reproducible extensions in a way that persists across server restarts, so even a long-running extension can be marked as reproducible without a performance penalty. + +If your extension relies on effectively immutable data obtained from outside the build, most commonly from the network, but you don't have a checksum available to guard the download, consider using the `facts` parameter of [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to persistently record such data and thus allow your extension to become reproducible. `facts` is expected to be a dictionary with string keys and arbitrary JSON-like Starlark values that is always persisted in the lockfile and available to future evaluations of the extension via the [`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. + +`facts` are not invalidated even when the code of your module extension changes, so be prepared to handle the case where the structure of `facts` changes. Bazel also assumes that two different `facts` dicts produced by two different evaluations of the same extension can be shallowly merged (i.e., as if by using the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` not supporting enumeration of its entries, just lookups by key. + +An example of using `facts` would be to record a mapping from version numbers of some SDK to the an object containing the download URL and checksum of that version. The first time the extension is evaluated, it can fetch this mapping from the network, but on later evaluations it can use the mapping from `facts` to avoid the network requests. ### Specify dependence on operating system and architecture -If your extension relies on the operating system or its architecture type, -ensure to indicate this in the extension definition using the `os_dependent` -and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the -need for re-evaluation if there are changes to either of them. +If your extension relies on the operating system or its architecture type, ensure to indicate this in the extension definition using the `os_dependent` and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the need for re-evaluation if there are changes to either of them. -Since this kind of dependence on the host makes it more difficult to maintain -the lockfile entry for this extension, consider -[marking the extension reproducible](#specify_reproducibility) if possible. +Since this kind of dependence on the host makes it more difficult to maintain the lockfile entry for this extension, consider [marking the extension reproducible](#specify_reproducibility) if possible. ### Only the root module should directly affect repository names -Remember that when an extension creates repositories, they are created within -the namespace of the extension. This means collisions can occur if different -modules use the same extension and end up creating a repository with the same -name. This often manifests as a module extension's `tag_class` having a `name` -argument that is passed as a repository rule's `name` value. - -For example, say the root module, `A`, depends on module `B`. Both modules -depend on module `mylang`. If both `A` and `B` call -`mylang.toolchain(name="foo")`, they will both try to create a repository named -`foo` within the `mylang` module and an error will occur. - -To avoid this, either remove the ability to set the repository name directly, -or only allow the root module to do so. It's OK to allow the root module this -ability because nothing will depend on it, so it doesn't have to worry about -another module creating a conflicting name. +Remember that when an extension creates repositories, they are created within the namespace of the extension. This means collisions can occur if different modules use the same extension and end up creating a repository with the same name. This often manifests as a module extension's `tag_class` having a `name` argument that is passed as a repository rule's `name` value. + +For example, say the root module, `A`, depends on module `B`. Both modules depend on module `mylang`. If both `A` and `B` call `mylang.toolchain(name="foo")`, they will both try to create a repository named `foo` within the `mylang` module and an error will occur. + +To avoid this, either remove the ability to set the repository name directly, or only allow the root module to do so. It's OK to allow the root module this ability because nothing will depend on it, so it doesn't have to worry about another module creating a conflicting name. diff --git a/external/faq.mdx b/external/faq.mdx index a0239cb5..58d6c521 100644 --- a/external/faq.mdx +++ b/external/faq.mdx @@ -2,145 +2,63 @@ title: 'Frequently asked questions' --- +This page answers some frequently asked questions about external dependencies in Bazel. +## MODULE.bazel +### How should I version a Bazel module? +Setting `version` with the [`module`](/rules/lib/globals/module#module) directive in the source archive `MODULE.bazel` can have several downsides and unintended side effects if not managed carefully: -This page answers some frequently asked questions about external dependencies in -Bazel. +- Duplication: releasing a new version of a module typically involves both incrementing the version in `MODULE.bazel` and tagging the release, two separate steps that can fall out of sync. While automation can reduce this risk, it's simpler and safer to avoid it altogether. -## MODULE.bazel +- Inconsistency: users overriding a module with a specific commit using a [non-registry override](module.md#non-registry_overrides) will see an incorrect version. for example, if the `MODULE.bazel` in the source archive sets `version = "0.3.0"` but additional commits have been made since that release, a user overriding with one of those commits would still see `0.3.0`. In reality, the version should reflect that it's ahead of the release, for example `0.3.1-rc1`. -### How should I version a Bazel module? +- Non-registry override issues: using placeholder values can cause issues when users override a module with a non-registry override. For example, `0.0.0` doesn't sort as the highest version, which is usually the expected behavior users want when doing a non-registry override. + +Thus, it's best to avoid setting the version in the source archive `MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry (e.g., the [Bazel Central Registry](https://registry.bazel.build/)), which is the actual source of truth for the module version during Bazel's external dependency resolution (see [Bazel registries](https://bazel.build/external/registry)). + +This is usually automated, for example the [`rules-template`](https://github.com/bazel-contrib/rules-template) example rule repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml) to publish the release to the BCR. The action [generates a patch for the source archive `MODULE.bazel`](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216) with the release version. This patch is stored in the registry and is applied when the module is fetched during Bazel's external dependency resolution. -Setting `version` with the [`module`] directive in the source archive -`MODULE.bazel` can have several downsides and unintended side effects if not -managed carefully: - -* Duplication: releasing a new version of a module typically involves both - incrementing the version in `MODULE.bazel` and tagging the release, two - separate steps that can fall out of sync. While automation can - reduce this risk, it's simpler and safer to avoid it altogether. - -* Inconsistency: users overriding a module with a specific commit using a - [non-registry override] will see an incorrect version. for example, if the - `MODULE.bazel` in the source archive sets `version = "0.3.0"` but - additional commits have been made since that release, a user overriding - with one of those commits would still see `0.3.0`. In reality, the version - should reflect that it's ahead of the release, for example `0.3.1-rc1`. - -* Non-registry override issues: using placeholder values can cause issues - when users override a module with a non-registry override. For example, - `0.0.0` doesn't sort as the highest version, which is usually the expected - behavior users want when doing a non-registry override. - -Thus, it's best to avoid setting the version in the source archive -`MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry -(e.g., the [Bazel Central Registry]), which is the actual source of truth for -the module version during Bazel's external dependency resolution (see [Bazel -registries]). - -This is usually automated, for example the [`rules-template`] example rule -repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action] to -publish the release to the BCR. The action [generates a patch for the source -archive `MODULE.bazel`] with the release version. This patch is stored in the -registry and is applied when the module is fetched during Bazel's external -dependency resolution. - -This way, the version in the releases in the registry will be correctly set to -the released version and thus, `bazel_dep`, `single_version_override` and -`multiple_version_override` will work as expected, while avoiding potential -issues when doing a non-registry override because the version in the source -archive will be the default value (`''`), which will always be handled -correctly (it's the default version value after all) and will behave as -expected when sorting (the empty string is treated as the highest version). - -[Bazel Central Registry]: https://registry.bazel.build/ -[Bazel registries]: https://bazel.build/external/registry -[bazel-contrib/publish-to-bcr publish.yaml GitHub Action]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml -[generates a patch for the source archive `MODULE.bazel`]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216 -[`module`]: /rules/lib/globals/module#module -[non-registry override]: module.md#non-registry_overrides -[`rules-template`]: https://github.com/bazel-contrib/rules-template +This way, the version in the releases in the registry will be correctly set to the released version and thus, `bazel_dep`, `single_version_override` and `multiple_version_override` will work as expected, while avoiding potential issues when doing a non-registry override because the version in the source archive will be the default value (`''`), which will always be handled correctly (it's the default version value after all) and will behave as expected when sorting (the empty string is treated as the highest version). ### When should I increment the compatibility level? -The [`compatibility_level`](module.md#compatibility_level) of a Bazel module -should be incremented _in the same commit_ that introduces a backwards -incompatible ("breaking") change. +The [`compatibility_level`](module.md#compatibility_level) of a Bazel module should be incremented *in the same commit* that introduces a backwards incompatible ("breaking") change. -However, Bazel can throw an error if it detects that versions of the _same -module_ with _different compatibility levels_ exist in the resolved dependency -graph. This can happen when for example' two modules depend on versions of a -third module with different compatibility levels. +However, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. This can happen when for example' two modules depend on versions of a third module with different compatibility levels. -Thus, incrementing `compatibility_level` too frequently can be very disruptive -and is discouraged. To avoid this situation, the `compatibility_level` should be -incremented _only_ when the breaking change affects most use cases and isn't -easy to migrate and/or work-around. +Thus, incrementing `compatibility_level` too frequently can be very disruptive and is discouraged. To avoid this situation, the `compatibility_level` should be incremented *only* when the breaking change affects most use cases and isn't easy to migrate and/or work-around. ### Why does MODULE.bazel not support `load`s? -During dependency resolution, the MODULE.bazel file of all referenced external -dependencies are fetched from registries. At this stage, the source archives of -the dependencies are not fetched yet; so if the MODULE.bazel file `load`s -another file, there is no way for Bazel to actually fetch that file without -fetching the entire source archive. Note the MODULE.bazel file itself is -special, as it's directly hosted on the registry. - -There are a few use cases that people asking for `load`s in MODULE.bazel are -generally interested in, and they can be solved without `load`s: - -* Ensuring that the version listed in MODULE.bazel is consistent with build - metadata stored elsewhere, for example in a .bzl file: This can be achieved - by using the - [`native.module_version`](/rules/lib/toplevel/native#module_version) method - in a .bzl file loaded from a BUILD file. -* Splitting up a very large MODULE.bazel file into manageable sections, - particularly for monorepos: The root module can use the - [`include`](/rules/lib/globals/module#include) directive to split its - MODULE.bazel file into multiple segments. For the same reason we don't allow - `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. -* Users of the old WORKSPACE system might remember declaring a repo, and then - immediately `load`ing from that repo to perform complex logic. This - capability has been replaced by [module extensions](extension). +During dependency resolution, the MODULE.bazel file of all referenced external dependencies are fetched from registries. At this stage, the source archives of the dependencies are not fetched yet; so if the MODULE.bazel file `load`s another file, there is no way for Bazel to actually fetch that file without fetching the entire source archive. Note the MODULE.bazel file itself is special, as it's directly hosted on the registry. + +There are a few use cases that people asking for `load`s in MODULE.bazel are generally interested in, and they can be solved without `load`s: + +- Ensuring that the version listed in MODULE.bazel is consistent with build metadata stored elsewhere, for example in a .bzl file: This can be achieved by using the [`native.module_version`](/rules/lib/toplevel/native#module_version) method in a .bzl file loaded from a BUILD file. +- Splitting up a very large MODULE.bazel file into manageable sections, particularly for monorepos: The root module can use the [`include`](/rules/lib/globals/module#include) directive to split its MODULE.bazel file into multiple segments. For the same reason we don't allow `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. +- Users of the old WORKSPACE system might remember declaring a repo, and then immediately `load`ing from that repo to perform complex logic. This capability has been replaced by [module extensions](extension). ### Can I specify a SemVer range for a `bazel_dep`? -No. Some other package managers like [npm][npm-semver] and [Cargo][cargo-semver] -support version ranges (implicitly or explicitly), and this often requires a -constraint solver (making the output harder to predict for users) and makes -version resolution nonreproducible without a lockfile. +No. Some other package managers like [npm](https://docs.npmjs.com/about-semantic-versioning) and [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax) support version ranges (implicitly or explicitly), and this often requires a constraint solver (making the output harder to predict for users) and makes version resolution nonreproducible without a lockfile. -Bazel instead uses [Minimal Version Selection](module#version-selection) like -Go, which in contrast makes the output easy to predict and guarantees -reproducibility. This is a tradeoff that matches Bazel's design goals. +Bazel instead uses [Minimal Version Selection](module#version-selection) like Go, which in contrast makes the output easy to predict and guarantees reproducibility. This is a tradeoff that matches Bazel's design goals. -Furthermore, Bazel module versions are [a superset of -SemVer](module#version-format), so what makes sense in a strict SemVer -environment doesn't always carry over to Bazel module versions. +Furthermore, Bazel module versions are [a superset of SemVer](module#version-format), so what makes sense in a strict SemVer environment doesn't always carry over to Bazel module versions. ### Can I automatically get the latest version for a `bazel_dep`? -Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", -version = "latest")` to automatically get the latest version of a dep. This is -similar to [the question about SemVer -ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also -no. +Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", version = "latest")` to automatically get the latest version of a dep. This is similar to [the question about SemVer ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also no. -The recommended solution here is to have automation take care of this. For -example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports -Bazel modules. +The recommended solution here is to have automation take care of this. For example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports Bazel modules. -Sometimes, users asking this question are really looking for a way to quickly -iterate during local development. This can be achieved by using a -[`local_path_override`](/rules/lib/globals/module#local_path_override). +Sometimes, users asking this question are really looking for a way to quickly iterate during local development. This can be achieved by using a [`local_path_override`](/rules/lib/globals/module#local_path_override). ### Why all these `use_repo`s? -Module extension usages in MODULE.bazel files sometimes come with a big -`use_repo` directive. For example, a typical usage of the -[`go_deps` extension][go_deps] from `gazelle` might look like: +Module extension usages in MODULE.bazel files sometimes come with a big `use_repo` directive. For example, a typical usage of the [`go_deps` extension](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies) from `gazelle` might look like: ```python go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") @@ -155,175 +73,82 @@ use_repo( ) ``` -The long `use_repo` directive may seem redundant, since the information is -arguably already in the referenced `go.mod` file. +The long `use_repo` directive may seem redundant, since the information is arguably already in the referenced `go.mod` file. -The reason Bazel needs this `use_repo` directive is that it runs module -extensions lazily. That is, a module extension is only run if its result is -observed. Since a module extension's "output" is repo definitions, this means -that we only run a module extension if a repo it defines is requested (for -instance, if the target `@org_golang_x_net//:foo` is built, in the example -above). However, we don't know which repos a module extension would define until -after we run it. This is where the `use_repo` directive comes in; the user can -tell Bazel which repos they expect the extension to generate, and Bazel would -then only run the extension when these specific repos are used. +The reason Bazel needs this `use_repo` directive is that it runs module extensions lazily. That is, a module extension is only run if its result is observed. Since a module extension's "output" is repo definitions, this means that we only run a module extension if a repo it defines is requested (for instance, if the target `@org_golang_x_net//:foo` is built, in the example above). However, we don't know which repos a module extension would define until after we run it. This is where the `use_repo` directive comes in; the user can tell Bazel which repos they expect the extension to generate, and Bazel would then only run the extension when these specific repos are used. -To help the maintain this `use_repo` directive, a module extension can return -an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) -object from its implementation function. The user can run the `bazel mod tidy` -command to update the `use_repo` directives for these module extensions. +To help the maintain this `use_repo` directive, a module extension can return an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) object from its implementation function. The user can run the `bazel mod tidy` command to update the `use_repo` directives for these module extensions. ## Bzlmod migration ### Which is evaluated first, MODULE.bazel or WORKSPACE? -When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to -wonder which system is consulted first. The short answer is that MODULE.bazel -(Bzlmod) is evaluated first. - -The long answer is that "which evaluates first" is not the right question to -ask; rather, the right question to ask is: in the context of the repo with -[canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent -repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what -is the repo mapping of `@@base`? - -Labels with apparent repo names (a single leading `@`) can refer to different -things based on the context they're resolved from. When you see a label -`@bar//:baz` and wonder what it actually points to, you need to first find out -what the context repo is: for example, if the label is in a BUILD file located -in the repo `@@foo`, then the context repo is `@@foo`. - -Then, depending on what the context repo is, the ["repository -visibility" table](migration#repository-visibility) in the migration guide can -be used to find out which repo an apparent name actually resolves to. - -* If the context repo is the main repo (`@@`): - 1. If `bar` is an apparent repo name introduced by the root module's - MODULE.bazel file (through any of - [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), - [`use_repo`](/rules/lib/globals/module#use_repo), - [`module`](/rules/lib/globals/module#module.repo_name), - [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` - resolves to what that MODULE.bazel file claims. - 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its - canonical name is `@@bar`), then `@bar` resolves to `@@bar`. - 3. Otherwise, `@bar` resolves to something like - `@@[unknown repo 'bar' requested from @@]`, and this will ultimately - result in an error. -* If the context repo is a Bzlmod-world repo (that is, it corresponds to a - non-root Bazel module, or is generated by a module extension), then it - will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. - * Notably, this includes any repos introduced in a `non_module_deps`-like - module extension in the root module, or `use_repo_rule` instantiations - in the root module. -* If the context repo is defined in WORKSPACE: - 1. First, check if the context repo definition has the magical - `repo_mapping` attribute. If so, go through the mapping first (so for a - repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking - at `@baz` below). - 2. If `bar` is an apparent repo name introduced by the root module's - MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file - claims. (This is the same as item 1 in the main repo case.) - 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a - repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel - will throw an error. +When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to wonder which system is consulted first. The short answer is that MODULE.bazel (Bzlmod) is evaluated first. + +The long answer is that "which evaluates first" is not the right question to ask; rather, the right question to ask is: in the context of the repo with [canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what is the repo mapping of `@@base`? + +Labels with apparent repo names (a single leading `@`) can refer to different things based on the context they're resolved from. When you see a label `@bar//:baz` and wonder what it actually points to, you need to first find out what the context repo is: for example, if the label is in a BUILD file located in the repo `@@foo`, then the context repo is `@@foo`. + +Then, depending on what the context repo is, the ["repository visibility" table](migration#repository-visibility) in the migration guide can be used to find out which repo an apparent name actually resolves to. + +- If the context repo is the main repo (`@@`): + + 1. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file (through any of [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), [`use_repo`](/rules/lib/globals/module#use_repo), [`module`](/rules/lib/globals/module#module.repo_name), [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` resolves to what that MODULE.bazel file claims. + 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its canonical name is `@@bar`), then `@bar` resolves to `@@bar`. + 3. Otherwise, `@bar` resolves to something like `@@[unknown repo 'bar' requested from @@]`, and this will ultimately result in an error. + +- If the context repo is a Bzlmod-world repo (that is, it corresponds to a non-root Bazel module, or is generated by a module extension), then it will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. + - Notably, this includes any repos introduced in a `non_module_deps`-like module extension in the root module, or `use_repo_rule` instantiations in the root module. + +- If the context repo is defined in WORKSPACE: + + 1. First, check if the context repo definition has the magical `repo_mapping` attribute. If so, go through the mapping first (so for a repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking at `@baz` below). + 2. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file claims. (This is the same as item 1 in the main repo case.) + 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel will throw an error. For a more succinct version: -* Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world - repos. -* WORKSPACE-world repos (including the main repo) will first see what the root - module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world - repos. +- Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world repos. +- WORKSPACE-world repos (including the main repo) will first see what the root module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world repos. -Of note, labels in the Bazel command line (including Starlark flags, label-typed -flag values, and build/test target patterns) are treated as having the main repo -as the context repo. +Of note, labels in the Bazel command line (including Starlark flags, label-typed flag values, and build/test target patterns) are treated as having the main repo as the context repo. ## Other ### How do I prepare and run an offline build? -Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag -(like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the -context of the main repo, see [question -above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target -pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of -`@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). +Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag (like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the context of the main repo, see [question above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of `@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). -The make sure no fetches happen during a build, use `--nofetch`. More precisely, -this makes any attempt to run a non-local repository rule fail. +The make sure no fetches happen during a build, use `--nofetch`. More precisely, this makes any attempt to run a non-local repository rule fail. -If you want to fetch repos _and_ modify them to test locally, consider using -the [`bazel vendor`](vendor) command. +If you want to fetch repos *and* modify them to test locally, consider using the [`bazel vendor`](vendor) command. ### How do I use HTTP proxies? -Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly -accepted by other programs, such as -[curl](https://everything.curl.dev/usingcurl/proxies/env.html). +Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly accepted by other programs, such as [curl](https://everything.curl.dev/usingcurl/proxies/env.html). ### How do I make Bazel prefer IPv6 in dual-stack IPv4/IPv6 setups? -On IPv6-only machines, Bazel can download dependencies with no changes. However, -on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, -preferring IPv4 if enabled. In some situations, for example when the IPv4 -network cannot resolve/reach external addresses, this can cause `Network -unreachable` exceptions and build failures. In these cases, you can override -Bazel's behavior to prefer IPv6 by using the -[`java.net.preferIPv6Addresses=true` system -property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). -Specifically: - -* Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup - option](/docs/user-manual#startup-options), for example by adding the - following line in your [`.bazelrc` file](/run/bazelrc): - - `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` - -* When running Java build targets that need to connect to the internet (such - as for integration tests), use the - `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool - flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` - file](/run/bazelrc): - - `build --jvmopt=-Djava.net.preferIPv6Addresses` - -* If you are using - [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for - dependency version resolution, also add - `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment - variable to [provide JVM options for - Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). +On IPv6-only machines, Bazel can download dependencies with no changes. However, on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, preferring IPv4 if enabled. In some situations, for example when the IPv4 network cannot resolve/reach external addresses, this can cause `Network unreachable` exceptions and build failures. In these cases, you can override Bazel's behavior to prefer IPv6 by using the [`java.net.preferIPv6Addresses=true` system property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). Specifically: -### Can repo rules be run remotely with remote execution? +- Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup option](/docs/user-manual#startup-options), for example by adding the following line in your [`.bazelrc` file](/run/bazelrc): + + `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` + +- When running Java build targets that need to connect to the internet (such as for integration tests), use the `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` file](/run/bazelrc): -No; or at least, not yet. Users employing remote execution services to speed up -their builds may notice that repo rules are still run locally. For example, an -`http_archive` would be first downloaded onto the local machine (using any local -download cache if applicable), extracted, and then each source file would be -uploaded to the remote execution service as an input file. It's natural to ask -why the remote execution service doesn't just download and extract that archive, -saving a useless roundtrip. + `build --jvmopt=-Djava.net.preferIPv6Addresses` -Part of the reason is that repo rules (and module extensions) are akin to -"scripts" that are run by Bazel itself. A remote executor doesn't necessarily -even have a Bazel installed. +- If you are using [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for dependency version resolution, also add `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment variable to [provide JVM options for Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). + +### Can repo rules be run remotely with remote execution? -Another reason is that Bazel often needs the BUILD files in the downloaded and -extracted archives to perform loading and analysis, which _are_ performed -locally. +No; or at least, not yet. Users employing remote execution services to speed up their builds may notice that repo rules are still run locally. For example, an `http_archive` would be first downloaded onto the local machine (using any local download cache if applicable), extracted, and then each source file would be uploaded to the remote execution service as an input file. It's natural to ask why the remote execution service doesn't just download and extract that archive, saving a useless roundtrip. -There are preliminary ideas to solve this problem by re-imagining repo rules as -build rules, which would naturally allow them to be run remotely, but conversely -raise new architectural concerns (for example, the `query` commands would -potentially need to run actions, complicating their design). +Part of the reason is that repo rules (and module extensions) are akin to "scripts" that are run by Bazel itself. A remote executor doesn't necessarily even have a Bazel installed. -For more previous discussion on this topic, see [A way to support repositories -that need Bazel for being -fetched](https://github.com/bazelbuild/bazel/discussions/20464). +Another reason is that Bazel often needs the BUILD files in the downloaded and extracted archives to perform loading and analysis, which *are* performed locally. -[npm-semver]: https://docs.npmjs.com/about-semantic-versioning -[cargo-semver]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax -[go_deps]: https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies +There are preliminary ideas to solve this problem by re-imagining repo rules as build rules, which would naturally allow them to be run remotely, but conversely raise new architectural concerns (for example, the `query` commands would potentially need to run actions, complicating their design). +For more previous discussion on this topic, see [A way to support repositories that need Bazel for being fetched](https://github.com/bazelbuild/bazel/discussions/20464). diff --git a/external/lockfile.mdx b/external/lockfile.mdx index 9f312615..a20423c3 100644 --- a/external/lockfile.mdx +++ b/external/lockfile.mdx @@ -1,98 +1,46 @@ -keywords: product:Bazel,lockfile,Bzlmod --- title: 'Bazel Lockfile' --- - - -The lockfile feature in Bazel enables the recording of specific versions or -dependencies of software libraries or packages required by a project. It -achieves this by storing the result of module resolution and extension -evaluation. The lockfile promotes reproducible builds, ensuring consistent -development environments. Additionally, it enhances build efficiency by allowing -Bazel to skip the parts of the resolution process that are unaffected by changes -in project dependencies. Furthermore, the lockfile improves stability by -preventing unexpected updates or breaking changes in external libraries, thereby -reducing the risk of introducing bugs. +The lockfile feature in Bazel enables the recording of specific versions or dependencies of software libraries or packages required by a project. It achieves this by storing the result of module resolution and extension evaluation. The lockfile promotes reproducible builds, ensuring consistent development environments. Additionally, it enhances build efficiency by allowing Bazel to skip the parts of the resolution process that are unaffected by changes in project dependencies. Furthermore, the lockfile improves stability by preventing unexpected updates or breaking changes in external libraries, thereby reducing the risk of introducing bugs. ## Lockfile Generation -The lockfile is generated under the workspace root with the name -`MODULE.bazel.lock`. It is created or updated during the build process, -specifically after module resolution and extension evaluation. Importantly, it -only includes dependencies that are included in the current invocation of the -build. +The lockfile is generated under the workspace root with the name `MODULE.bazel.lock`. It is created or updated during the build process, specifically after module resolution and extension evaluation. Importantly, it only includes dependencies that are included in the current invocation of the build. -When changes occur in the project that affect its dependencies, the lockfile is -automatically updated to reflect the new state. This ensures that the lockfile -remains focused on the specific set of dependencies required for the current -build, providing an accurate representation of the project's resolved -dependencies. +When changes occur in the project that affect its dependencies, the lockfile is automatically updated to reflect the new state. This ensures that the lockfile remains focused on the specific set of dependencies required for the current build, providing an accurate representation of the project's resolved dependencies. ## Lockfile Usage -The lockfile can be controlled by the flag -[`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to -customize the behavior of Bazel when the project state differs from the -lockfile. The available modes are: - -* `update` (Default): Use the information that is present in the lockfile to - skip downloads of known registry files and to avoid re-evaluating extensions - whose results are still up-to-date. If information is missing, it will - be added to the lockfile. In this mode, Bazel also avoids refreshing - mutable information, such as yanked versions, for dependencies that haven't - changed. -* `refresh`: Like `update`, but mutable information is always refreshed when - switching to this mode and roughly every hour while in this mode. -* `error`: Like `update`, but if any information is missing or out-of-date, - Bazel will fail with an error. This mode never changes the lockfile or - performs network requests during resolution. Module extensions that marked - themselves as `reproducible` may still perform network requests, but are - expected to always produce the same result. -* `off`: The lockfile is neither checked nor updated. +The lockfile can be controlled by the flag [`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to customize the behavior of Bazel when the project state differs from the lockfile. The available modes are: + +- `update` (Default): Use the information that is present in the lockfile to skip downloads of known registry files and to avoid re-evaluating extensions whose results are still up-to-date. If information is missing, it will be added to the lockfile. In this mode, Bazel also avoids refreshing mutable information, such as yanked versions, for dependencies that haven't changed. +- `refresh`: Like `update`, but mutable information is always refreshed when switching to this mode and roughly every hour while in this mode. +- `error`: Like `update`, but if any information is missing or out-of-date, Bazel will fail with an error. This mode never changes the lockfile or performs network requests during resolution. Module extensions that marked themselves as `reproducible` may still perform network requests, but are expected to always produce the same result. +- `off`: The lockfile is neither checked nor updated. ## Lockfile Benefits The lockfile offers several benefits and can be utilized in various ways: -- **Reproducible builds.** By capturing the specific versions or dependencies - of software libraries, the lockfile ensures that builds are reproducible - across different environments and over time. Developers can rely on - consistent and predictable results when building their projects. +- **Reproducible builds.** By capturing the specific versions or dependencies of software libraries, the lockfile ensures that builds are reproducible across different environments and over time. Developers can rely on consistent and predictable results when building their projects. -- **Fast incremental resolutions.** The lockfile enables Bazel to avoid - downloading registry files that were already used in a previous build. - This significantly improves build efficiency, especially in scenarios where - resolution can be time-consuming. +- **Fast incremental resolutions.** The lockfile enables Bazel to avoid downloading registry files that were already used in a previous build. This significantly improves build efficiency, especially in scenarios where resolution can be time-consuming. -- **Stability and risk reduction.** The lockfile helps maintain stability by - preventing unexpected updates or breaking changes in external libraries. By - locking the dependencies to specific versions, the risk of introducing bugs - due to incompatible or untested updates is reduced. +- **Stability and risk reduction.** The lockfile helps maintain stability by preventing unexpected updates or breaking changes in external libraries. By locking the dependencies to specific versions, the risk of introducing bugs due to incompatible or untested updates is reduced. ### Hidden lockfile -Bazel also maintains another lockfile at -`"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this -lockfile are explicitly unspecified. It is only used as a performance -optimization. While it can be deleted together with the output base via -`bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a -module extension. +Bazel also maintains another lockfile at `"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this lockfile are explicitly unspecified. It is only used as a performance optimization. While it can be deleted together with the output base via `bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a module extension. ## Lockfile Contents -The lockfile contains all the necessary information to determine whether the -project state has changed. It also includes the result of building the project -in the current state. The lockfile consists of two main parts: +The lockfile contains all the necessary information to determine whether the project state has changed. It also includes the result of building the project in the current state. The lockfile consists of two main parts: -1. Hashes of all remote files that are inputs to module resolution. -2. For each module extension, the lockfile includes inputs that affect it, - represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as - well as the output of running that extension, referred to as - `generatedRepoSpecs` +1. Hashes of all remote files that are inputs to module resolution. +2. For each module extension, the lockfile includes inputs that affect it, represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as well as the output of running that extension, referred to as `generatedRepoSpecs` -Here is an example that demonstrates the structure of the lockfile, along with -explanations for each section: +Here is an example that demonstrates the structure of the lockfile, along with explanations for each section: ```json { @@ -152,106 +100,53 @@ explanations for each section: ### Registry File Hashes -The `registryFileHashes` section contains the hashes of all files from -remote registries accessed during module resolution. Since the resolution -algorithm is fully deterministic when given the same inputs and all remote -inputs are hashed, this ensures a fully reproducible resolution result while -avoiding excessive duplication of remote information in the lockfile. Note that -this also requires recording when a particular registry didn't contain a certain -module, but a registry with lower precedence did (see the "not found" entry in -the example). This inherently mutable information can be updated via -`bazel mod deps --lockfile_mode=refresh`. +The `registryFileHashes` section contains the hashes of all files from remote registries accessed during module resolution. Since the resolution algorithm is fully deterministic when given the same inputs and all remote inputs are hashed, this ensures a fully reproducible resolution result while avoiding excessive duplication of remote information in the lockfile. Note that this also requires recording when a particular registry didn't contain a certain module, but a registry with lower precedence did (see the "not found" entry in the example). This inherently mutable information can be updated via `bazel mod deps --lockfile_mode=refresh`. -Bazel uses the hashes from the lockfile to look up registry files in the -repository cache before downloading them, which speeds up subsequent -resolutions. +Bazel uses the hashes from the lockfile to look up registry files in the repository cache before downloading them, which speeds up subsequent resolutions. ### Selected Yanked Versions -The `selectedYankedVersions` section contains the yanked versions of modules -that were selected by module resolution. Since this usually result in an error -when trying to build, this section is only non-empty when yanked versions are -explicitly allowed via `--allow_yanked_versions` or -`BZLMOD_ALLOW_YANKED_VERSIONS`. +The `selectedYankedVersions` section contains the yanked versions of modules that were selected by module resolution. Since this usually result in an error when trying to build, this section is only non-empty when yanked versions are explicitly allowed via `--allow_yanked_versions` or `BZLMOD_ALLOW_YANKED_VERSIONS`. -This field is needed since, compared to module files, yanked version information -is inherently mutable and thus can't be referenced by a hash. This information -can be updated via `bazel mod deps --lockfile_mode=refresh`. +This field is needed since, compared to module files, yanked version information is inherently mutable and thus can't be referenced by a hash. This information can be updated via `bazel mod deps --lockfile_mode=refresh`. ### Module Extensions -The `moduleExtensions` section is a map that includes only the extensions used -in the current invocation or previously invoked, while excluding any extensions -that are no longer utilized. In other words, if an extension is not being used -anymore across the dependency graph, it is removed from the `moduleExtensions` -map. - -If an extension is independent of the operating system or architecture type, -this section features only a single "general" entry. Otherwise, multiple -entries are included, named after the OS, architecture, or both, with each -corresponding to the result of evaluating the extension on those specifics. - -Each entry in the extension map corresponds to a used extension and is -identified by its containing file and name. The corresponding value for each -entry contains the relevant information associated with that extension: - -1. The `bzlTransitiveDigest` is the digest of the extension implementation - and the .bzl files transitively loaded by it. -2. The `usagesDigest` is the digest of the _usages_ of the extension in the - dependency graph, which includes all tags. -3. Further unspecified fields that track other inputs to the extension, - such as contents of files or directories it reads or environment - variables it uses. -4. The `generatedRepoSpecs` encode the repositories created by the - extension with the current input. -5. The optional `moduleExtensionMetadata` field contains metadata provided by - the extension such as whether certain repositories it created should be - imported via `use_repo` by the root module. This information powers the - `bazel mod tidy` command. - -Module extensions can opt out of being included in the lockfile by setting the -returning metadata with `reproducible = True`. By doing so, they promise that -they will always create the same repositories when given the same inputs. +The `moduleExtensions` section is a map that includes only the extensions used in the current invocation or previously invoked, while excluding any extensions that are no longer utilized. In other words, if an extension is not being used anymore across the dependency graph, it is removed from the `moduleExtensions` map. + +If an extension is independent of the operating system or architecture type, this section features only a single "general" entry. Otherwise, multiple entries are included, named after the OS, architecture, or both, with each corresponding to the result of evaluating the extension on those specifics. + +Each entry in the extension map corresponds to a used extension and is identified by its containing file and name. The corresponding value for each entry contains the relevant information associated with that extension: + +1. The `bzlTransitiveDigest` is the digest of the extension implementation and the .bzl files transitively loaded by it. +2. The `usagesDigest` is the digest of the *usages* of the extension in the dependency graph, which includes all tags. +3. Further unspecified fields that track other inputs to the extension, such as contents of files or directories it reads or environment variables it uses. +4. The `generatedRepoSpecs` encode the repositories created by the extension with the current input. +5. The optional `moduleExtensionMetadata` field contains metadata provided by the extension such as whether certain repositories it created should be imported via `use_repo` by the root module. This information powers the `bazel mod tidy` command. + +Module extensions can opt out of being included in the lockfile by setting the returning metadata with `reproducible = True`. By doing so, they promise that they will always create the same repositories when given the same inputs. ## Best Practices -To maximize the benefits of the lockfile feature, consider the following best -practices: +To maximize the benefits of the lockfile feature, consider the following best practices: -* Regularly update the lockfile to reflect changes in project dependencies or - configuration. This ensures that subsequent builds are based on the most - up-to-date and accurate set of dependencies. To lock down all extensions - at once, run `bazel mod deps --lockfile_mode=update`. +- Regularly update the lockfile to reflect changes in project dependencies or configuration. This ensures that subsequent builds are based on the most up-to-date and accurate set of dependencies. To lock down all extensions at once, run `bazel mod deps --lockfile_mode=update`. -* Include the lockfile in version control to facilitate collaboration and - ensure that all team members have access to the same lockfile, promoting - consistent development environments across the project. +- Include the lockfile in version control to facilitate collaboration and ensure that all team members have access to the same lockfile, promoting consistent development environments across the project. -* Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a - `.bazelversion` file in version control that specifies the Bazel version - corresponding to the lockfile. Because Bazel itself is a dependency of - your build, the lockfile is specific to the Bazel version, and will - change even between [backwards compatible](/release/backward-compatibility) - Bazel releases. Using `bazelisk` ensures that all developers are using - a Bazel version that matches the lockfile. +- Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a `.bazelversion` file in version control that specifies the Bazel version corresponding to the lockfile. Because Bazel itself is a dependency of your build, the lockfile is specific to the Bazel version, and will change even between [backwards compatible](/release/backward-compatibility) Bazel releases. Using `bazelisk` ensures that all developers are using a Bazel version that matches the lockfile. -By following these best practices, you can effectively utilize the lockfile -feature in Bazel, leading to more efficient, reliable, and collaborative -software development workflows. +By following these best practices, you can effectively utilize the lockfile feature in Bazel, leading to more efficient, reliable, and collaborative software development workflows. ## Merge Conflicts -The lockfile format is designed to minimize merge conflicts, but they can still -happen. +The lockfile format is designed to minimize merge conflicts, but they can still happen. ### Automatic Resolution -Bazel provides a custom -[git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) -to help resolve these conflicts automatically. +Bazel provides a custom [git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) to help resolve these conflicts automatically. -Set up the driver by adding this line to a `.gitattributes` file in the root of -your git repository: +Set up the driver by adding this line to a `.gitattributes` file in the root of your git repository: ```gitattributes # A custom merge driver for the Bazel lockfile. @@ -259,8 +154,7 @@ your git repository: MODULE.bazel.lock merge=bazel-lockfile-merge ``` -Then each developer who wants to use the driver has to register it once by -following these steps: +Then each developer who wants to use the driver has to register it once by following these steps: 1. Install [jq](https://jqlang.github.io/jq/download/) (1.5 or higher). 2. Run the following commands: @@ -269,18 +163,15 @@ following these steps: jq_script=$(curl https://raw.githubusercontent.com/bazelbuild/bazel/master/scripts/bazel-lockfile-merge.jq) printf '%s\n' "${jq_script}" | less # to optionally inspect the jq script git config --global merge.bazel-lockfile-merge.name "Merge driver for the Bazel lockfile (MODULE.bazel.lock)" -git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" +git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" ``` ### Manual Resolution -Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` -fields can be safely resolved by keeping all the entries from both sides of the -conflict. +Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` fields can be safely resolved by keeping all the entries from both sides of the conflict. Other types of merge conflicts should not be resolved manually. Instead: -1. Restore the previous state of the lockfile - via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. +1. Restore the previous state of the lockfile via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. 2. Resolve any conflicts in the `MODULE.bazel` file. 3. Run `bazel mod deps` to update the lockfile. diff --git a/external/migration.mdx b/external/migration.mdx index 8e70b1f3..49ba462f 100644 --- a/external/migration.mdx +++ b/external/migration.mdx @@ -1,267 +1,200 @@ -keywords: bzlmod --- title: 'Bzlmod Migration Guide' --- - - -Due to the [shortcomings of -WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the -legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late -2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate -your project to Bzlmod and drop WORKSPACE for managing external dependencies. +Due to the [shortcomings of WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate your project to Bzlmod and drop WORKSPACE for managing external dependencies. ## Why migrate to Bzlmod? -* There are many [advantages](overview#benefits) compared to the legacy - WORKSPACE system, which helps to ensure a healthy growth of the Bazel - ecosystem. +- There are many [advantages](overview#benefits) compared to the legacy WORKSPACE system, which helps to ensure a healthy growth of the Bazel ecosystem. -* If your project is a dependency of other projects, migrating to Bzlmod will - unblock their migration and make it easier for them to depend on your - project. +- If your project is a dependency of other projects, migrating to Bzlmod will unblock their migration and make it easier for them to depend on your project. -* Migration to Bzlmod is a necessary step in order to use future Bazel - versions (mandatory in Bazel 9). +- Migration to Bzlmod is a necessary step in order to use future Bazel versions (mandatory in Bazel 9). ## How to migrate to Bzlmod? Recommended migration process: -1. Use [migration tool](/external/migration_tool) as a helper tool to - streamline the migration process as much as possible. -2. If there are errors left after using the migration tool, resolve them - manually. For understanding the main differences between concepts inside - `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus - Bzlmod](#workspace-vs-bzlmod) section. +1. Use [migration tool](/external/migration_tool) as a helper tool to streamline the migration process as much as possible. +2. If there are errors left after using the migration tool, resolve them manually. For understanding the main differences between concepts inside `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus Bzlmod](#workspace-vs-bzlmod) section. ## WORKSPACE vs Bzlmod -Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This -section explains how to migrate from specific WORKSPACE functionalities to -Bzlmod. +Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This section explains how to migrate from specific WORKSPACE functionalities to Bzlmod. ### Define the root of a Bazel workspace -The WORKSPACE file marks the source root of a Bazel project, this responsibility -is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions -prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at -your workspace root, maybe with comments like: +The WORKSPACE file marks the source root of a Bazel project, this responsibility is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at your workspace root, maybe with comments like: -* **WORKSPACE** +- **WORKSPACE** - ```python - # This file marks the root of the Bazel workspace. - # See MODULE.bazel for external dependencies setup. - ``` + ```python + # This file marks the root of the Bazel workspace. + # See MODULE.bazel for external dependencies setup. + ``` ### Enable Bzlmod in your bazelrc -`.bazelrc` lets you set flags that apply every time your run Bazel. To enable -Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so -it applies to every command: +`.bazelrc` lets you set flags that apply every time your run Bazel. To enable Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so it applies to every command: -* **.bazelrc** +- **.bazelrc** - ``` - # Enable Bzlmod for every Bazel command - common --enable_bzlmod - ``` + ``` + # Enable Bzlmod for every Bazel command + common --enable_bzlmod + ``` ### Specify repository name for your workspace -* **WORKSPACE** +- **WORKSPACE** - The [`workspace`](/rules/lib/globals/workspace#workspace) function is used - to specify a repository name for your workspace. This allows a target - `//foo:bar` in the workspace to be referenced as `@//foo:bar`. If not specified, the default repository name for your - workspace is `__main__`. + The [`workspace`](/rules/lib/globals/workspace#workspace) function is used to specify a repository name for your workspace. This allows a target `//foo:bar` in the workspace to be referenced as `@<workspace name>//foo:bar`. If not specified, the default repository name for your workspace is `__main__`. - ```python - ## WORKSPACE - workspace(name = "com_foo_bar") - ``` + ```python + ## WORKSPACE + workspace(name = "com_foo_bar") + ``` -* **Bzlmod** +- **Bzlmod** - It's recommended to reference targets in the same workspace with the - `//foo:bar` syntax without `@`. But if you do need the old syntax - , you can use the module name specified by the - [`module`](/rules/lib/globals/module#module) function as the repository - name. If the module name is different from the needed repository name, you - can use `repo_name` attribute of the - [`module`](/rules/lib/globals/module#module) function to override the - repository name. + It's recommended to reference targets in the same workspace with the `//foo:bar` syntax without `@<repo name>`. But if you do need the old syntax , you can use the module name specified by the [`module`](/rules/lib/globals/module#module) function as the repository name. If the module name is different from the needed repository name, you can use `repo_name` attribute of the [`module`](/rules/lib/globals/module#module) function to override the repository name. - ```python - ## MODULE.bazel - module( - name = "bar", - repo_name = "com_foo_bar", - ) - ``` + ```python + ## MODULE.bazel + module( + name = "bar", + repo_name = "com_foo_bar", + ) + ``` ### Fetch external dependencies as Bazel modules -If your dependency is a Bazel project, you should be able to depend on it as a -Bazel module when it also adopts Bzlmod. +If your dependency is a Bazel project, you should be able to depend on it as a Bazel module when it also adopts Bzlmod. -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, it's common to use the - [`http_archive`](/rules/lib/repo/http#http_archive) or - [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to - download the sources of the Bazel project. + With WORKSPACE, it's common to use the [`http_archive`](/rules/lib/repo/http#http_archive) or [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to download the sources of the Bazel project. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "bazel_skylib", - urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], - sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", - ) - load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") - bazel_skylib_workspace() + http_archive( + name = "bazel_skylib", + urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], + sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", + ) + load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") + bazel_skylib_workspace() - http_archive( - name = "rules_java", - urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], - sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", - ) - load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") - rules_java_dependencies() - rules_java_toolchains() - ``` + http_archive( + name = "rules_java", + urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], + sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", + ) + load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") + rules_java_dependencies() + rules_java_toolchains() + ``` - As you can see, it's a common pattern that users need to load transitive - dependencies from a macro of the dependency. Assume both `bazel_skylib` and - `rules_java` depends on `platform`, the exact version of the `platform` - dependency is determined by the order of the macros. + As you can see, it's a common pattern that users need to load transitive dependencies from a macro of the dependency. Assume both `bazel_skylib` and `rules_java` depends on `platform`, the exact version of the `platform` dependency is determined by the order of the macros. -* **Bzlmod** +- **Bzlmod** - With Bzlmod, as long as your dependency is available in [Bazel Central - Registry](https://registry.bazel.build) or your custom [Bazel - registry](/external/registry), you can simply depend on it with a - [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. + With Bzlmod, as long as your dependency is available in [Bazel Central Registry](https://registry.bazel.build) or your custom [Bazel registry](/external/registry), you can simply depend on it with a [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. - ```python - ## MODULE.bazel - bazel_dep(name = "bazel_skylib", version = "1.4.2") - bazel_dep(name = "rules_java", version = "6.1.1") - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "bazel_skylib", version = "1.4.2") + bazel_dep(name = "rules_java", version = "6.1.1") + ``` - Bzlmod resolves Bazel module dependencies transitively using the - [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal - required version of `platform` is selected automatically. + Bzlmod resolves Bazel module dependencies transitively using the [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal required version of `platform` is selected automatically. ### Override a dependency as a Bazel module -As the root module, you can override Bazel module dependencies in different -ways. +As the root module, you can override Bazel module dependencies in different ways. -Please read the [overrides](/external/module#overrides) section for more -information. +Please read the [overrides](/external/module#overrides) section for more information. -You can find some example usages in the -[examples][override-examples] -repository. - -[override-examples]: https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module +You can find some example usages in the [examples](https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module) repository. ### Fetch external dependencies with module extensions -If your dependency is not a Bazel project or not yet available in any Bazel -registry, you can introduce it using -[`use_repo_rule`](/external/module#use_repo_rule) or [module -extensions](/external/extension). +If your dependency is not a Bazel project or not yet available in any Bazel registry, you can introduce it using [`use_repo_rule`](/external/module#use_repo_rule) or [module extensions](/external/extension). -* **WORKSPACE** +- **WORKSPACE** - Download a file using the [`http_file`](/rules/lib/repo/http#http_file) - repository rule. + Download a file using the [`http_file`](/rules/lib/repo/http#http_file) repository rule. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` -* **Bzlmod** +- **Bzlmod** - With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel - file to directly instantiate repos: + With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel file to directly instantiate repos: - ```python - ## MODULE.bazel - http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## MODULE.bazel + http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Under the hood, this is implemented using a module extension. If you need to - perform more complex logic than simply invoking a repo rule, you could also - implement a module extension yourself. You'll need to move the definition - into a `.bzl` file, which also lets you share the definition between - WORKSPACE and Bzlmod during the migration period. + Under the hood, this is implemented using a module extension. If you need to perform more complex logic than simply invoking a repo rule, you could also implement a module extension yourself. You'll need to move the definition into a `.bzl` file, which also lets you share the definition between WORKSPACE and Bzlmod during the migration period. - ```python - ## repositories.bzl - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - def my_data_dependency(): - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## repositories.bzl + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + def my_data_dependency(): + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Implement a module extension to load the dependencies macro. You can define - it in the same `.bzl` file of the macro, but to keep compatibility with - older Bazel versions, it's better to define it in a separate `.bzl` file. + Implement a module extension to load the dependencies macro. You can define it in the same `.bzl` file of the macro, but to keep compatibility with older Bazel versions, it's better to define it in a separate `.bzl` file. - ```python - ## extensions.bzl - load("//:repositories.bzl", "my_data_dependency") - def _non_module_dependencies_impl(_ctx): - my_data_dependency() + ```python + ## extensions.bzl + load("//:repositories.bzl", "my_data_dependency") + def _non_module_dependencies_impl(_ctx): + my_data_dependency() - non_module_dependencies = module_extension( - implementation = _non_module_dependencies_impl, - ) - ``` + non_module_dependencies = module_extension( + implementation = _non_module_dependencies_impl, + ) + ``` - To make the repository visible to the root project, you should declare the - usages of the module extension and the repository in the MODULE.bazel file. + To make the repository visible to the root project, you should declare the usages of the module extension and the repository in the MODULE.bazel file. - ```python - ## MODULE.bazel - non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") - use_repo(non_module_dependencies, "data_file") - ``` + ```python + ## MODULE.bazel + non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") + use_repo(non_module_dependencies, "data_file") + ``` ### Resolve conflict external dependencies with module extension -A project can provide a macro that introduces external repositories based on -inputs from its callers. But what if there are multiple callers in the -dependency graph and they cause a conflict? +A project can provide a macro that introduces external repositories based on inputs from its callers. But what if there are multiple callers in the dependency graph and they cause a conflict? -Assume the project `foo` provides the following macro which takes `version` as -an argument. +Assume the project `foo` provides the following macro which takes `version` as an argument. ```python -## repositories.bzl in foo {:#repositories.bzl-foo} +## repositories.bzl in foo load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") def data_deps(version = "1.0"): http_file( @@ -271,297 +204,239 @@ def data_deps(version = "1.0"): ) ``` -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, you can load the macro from `@foo` and specify the version - of the data dependency you need. Assume you have another dependency `@bar`, - which also depends on `@foo` but requires a different version of the data - dependency. + With WORKSPACE, you can load the macro from `@foo` and specify the version of the data dependency you need. Assume you have another dependency `@bar`, which also depends on `@foo` but requires a different version of the data dependency. - ```python - ## WORKSPACE + ```python + ## WORKSPACE - # Introduce @foo and @bar. - ... + # Introduce @foo and @bar. + ... - load("@foo//:repositories.bzl", "data_deps") - data_deps(version = "2.0") + load("@foo//:repositories.bzl", "data_deps") + data_deps(version = "2.0") - load("@bar//:repositories.bzl", "bar_deps") - bar_deps() # -> which calls data_deps(version = "3.0") - ``` + load("@bar//:repositories.bzl", "bar_deps") + bar_deps() # -> which calls data_deps(version = "3.0") + ``` - In this case, the end user must carefully adjust the order of macros in the - WORKSPACE to get the version they need. This is one of the biggest pain - points with WORKSPACE since it doesn't really provide a sensible way to - resolve dependencies. + In this case, the end user must carefully adjust the order of macros in the WORKSPACE to get the version they need. This is one of the biggest pain points with WORKSPACE since it doesn't really provide a sensible way to resolve dependencies. -* **Bzlmod** +- **Bzlmod** - With Bzlmod, the author of project `foo` can use module extension to resolve - conflicts. For example, let's assume it makes sense to always select the - maximal required version of the data dependency among all Bazel modules. + With Bzlmod, the author of project `foo` can use module extension to resolve conflicts. For example, let's assume it makes sense to always select the maximal required version of the data dependency among all Bazel modules. - ```python - ## extensions.bzl in foo - load("//:repositories.bzl", "data_deps") - - data = tag_class(attrs={"version": attr.string()}) - - def _data_deps_extension_impl(module_ctx): - # Select the maximal required version in the dependency graph. - version = "1.0" - for mod in module_ctx.modules: - for data in mod.tags.data: - version = max(version, data.version) - data_deps(version) - - data_deps_extension = module_extension( - implementation = _data_deps_extension_impl, - tag_classes = {"data": data}, - ) - ``` + ```python + ## extensions.bzl in foo + load("//:repositories.bzl", "data_deps") - ```python - ## MODULE.bazel in bar - bazel_dep(name = "foo", version = "1.0") + data = tag_class(attrs={"version": attr.string()}) - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "3.0") - use_repo(foo_data_deps, "data_file") - ``` + def _data_deps_extension_impl(module_ctx): + # Select the maximal required version in the dependency graph. + version = "1.0" + for mod in module_ctx.modules: + for data in mod.tags.data: + version = max(version, data.version) + data_deps(version) - ```python - ## MODULE.bazel in root module - bazel_dep(name = "foo", version = "1.0") - bazel_dep(name = "bar", version = "1.0") + data_deps_extension = module_extension( + implementation = _data_deps_extension_impl, + tag_classes = {"data": data}, + ) + ``` - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "2.0") - use_repo(foo_data_deps, "data_file") - ``` + ```python + ## MODULE.bazel in bar + bazel_dep(name = "foo", version = "1.0") - In this case, the root module requires data version `2.0`, while its - dependency `bar` requires `3.0`. The module extension in `foo` can correctly - resolve this conflict and automatically select version `3.0` for the data - dependency. + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "3.0") + use_repo(foo_data_deps, "data_file") + ``` -### Integrate third party package manager + ```python + ## MODULE.bazel in root module + bazel_dep(name = "foo", version = "1.0") + bazel_dep(name = "bar", version = "1.0") + + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "2.0") + use_repo(foo_data_deps, "data_file") + ``` + + In this case, the root module requires data version `2.0`, while its dependency `bar` requires `3.0`. The module extension in `foo` can correctly resolve this conflict and automatically select version `3.0` for the data dependency. -Following the last section, since module extension provides a way to collect -information from the dependency graph, perform custom logic to resolve -dependencies and call repository rules to introduce external repositories, this -provides a great way for rules authors to enhance the rulesets that integrate -package managers for specific languages. +### Integrate third party package manager -Please read the [module extensions](/external/extension) page to learn more -about how to use module extensions. +Following the last section, since module extension provides a way to collect information from the dependency graph, perform custom logic to resolve dependencies and call repository rules to introduce external repositories, this provides a great way for rules authors to enhance the rulesets that integrate package managers for specific languages. -Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies -from different package managers: +Please read the [module extensions](/external/extension) page to learn more about how to use module extensions. -- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) -- [rules_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) -- [rules_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) +Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies from different package managers: -A minimal example that integrates a pseudo package manager is available at the -[examples][pkg-mgr-example] -repository. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) +- [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) +- [rules\_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) -[pkg-mgr-example]: https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager +A minimal example that integrates a pseudo package manager is available at the [examples](https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager) repository. ### Detect toolchains on the host machine -When Bazel build rules need to detect what toolchains are available on your host -machine, they use repository rules to inspect the host machine and generate -toolchain info as external repositories. +When Bazel build rules need to detect what toolchains are available on your host machine, they use repository rules to inspect the host machine and generate toolchain info as external repositories. -* **WORKSPACE** +- **WORKSPACE** - Given the following repository rule to detect a shell toolchain. + Given the following repository rule to detect a shell toolchain. - ```python - ## local_config_sh.bzl - def _sh_config_rule_impl(repository_ctx): - sh_path = get_sh_path_from_env("SH_BIN_PATH") - - if not sh_path: - sh_path = detect_sh_from_path() - - if not sh_path: - sh_path = "/shell/binary/not/found" - - repository_ctx.file("BUILD", """ - load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") - sh_toolchain( - name = "local_sh", - path = "{sh_path}", - visibility = ["//visibility:public"], - ) - toolchain( - name = "local_sh_toolchain", - toolchain = ":local_sh", - toolchain_type = "@bazel_tools//tools/sh:toolchain_type", - ) - """.format(sh_path = sh_path)) + ```python + ## local_config_sh.bzl + def _sh_config_rule_impl(repository_ctx): + sh_path = get_sh_path_from_env("SH_BIN_PATH") - sh_config_rule = repository_rule( - environ = ["SH_BIN_PATH"], - local = True, - implementation = _sh_config_rule_impl, - ) - ``` + if not sh_path: + sh_path = detect_sh_from_path() - You can load the repository rule in WORKSPACE. + if not sh_path: + sh_path = "/shell/binary/not/found" - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - ``` + repository_ctx.file("BUILD", """ + load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") + sh_toolchain( + name = "local_sh", + path = "{sh_path}", + visibility = ["//visibility:public"], + ) + toolchain( + name = "local_sh_toolchain", + toolchain = ":local_sh", + toolchain_type = "@bazel_tools//tools/sh:toolchain_type", + ) + """.format(sh_path = sh_path)) -* **Bzlmod** + sh_config_rule = repository_rule( + environ = ["SH_BIN_PATH"], + local = True, + implementation = _sh_config_rule_impl, + ) + ``` - With Bzlmod, you can introduce the same repository using a module extension, - which is similar to introducing the `@data_file` repository in the last - section. + You can load the repository rule in WORKSPACE. - ``` - ## local_config_sh_extension.bzl - load("//:local_config_sh.bzl", "sh_config_rule") + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + ``` - sh_config_extension = module_extension( - implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), - ) - ``` +- **Bzlmod** - Then use the extension in the MODULE.bazel file. + With Bzlmod, you can introduce the same repository using a module extension, which is similar to introducing the `@data_file` repository in the last section. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - ``` + ``` + ## local_config_sh_extension.bzl + load("//:local_config_sh.bzl", "sh_config_rule") -### Register toolchains & execution platforms + sh_config_extension = module_extension( + implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), + ) + ``` -Following the last section, after introducing a repository hosting toolchain -information (e.g. `local_config_sh`), you probably want to register the -toolchain. + Then use the extension in the MODULE.bazel file. -* **WORKSPACE** + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + ``` - With WORKSPACE, you can register the toolchain in the following ways. +### Register toolchains & execution platforms - 1. You can register the toolchain the `.bzl` file and load the macro in the - WORKSPACE file. +Following the last section, after introducing a repository hosting toolchain information (e.g. `local_config_sh`), you probably want to register the toolchain. - ```python - ## local_config_sh.bzl - def sh_configure(): - sh_config_rule(name = "local_config_sh") - native.register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +- **WORKSPACE** - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_configure") - sh_configure() - ``` + With WORKSPACE, you can register the toolchain in the following ways. - 2. Or register the toolchain in the WORKSPACE file directly. + 1. You can register the toolchain the `.bzl` file and load the macro in the WORKSPACE file. - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## local_config_sh.bzl + def sh_configure(): + sh_config_rule(name = "local_config_sh") + native.register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` -* **Bzlmod** + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_configure") + sh_configure() + ``` - With Bzlmod, the - [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and - [`register_execution_platforms`][register_execution_platforms] - APIs are only available in the MODULE.bazel file. You cannot call - `native.register_toolchains` in a module extension. + 2. Or register the toolchain in the WORKSPACE file directly. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` + +- **Bzlmod** + + With Bzlmod, the [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) APIs are only available in the MODULE.bazel file. You cannot call `native.register_toolchains` in a module extension. -The toolchains and execution platforms registered in `WORKSPACE`, -`WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this -order of precedence during toolchain selection (from highest to lowest): + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` -1. toolchains and execution platforms registered in the root module's - `MODULE.bazel` file. -2. toolchains and execution platforms registered in the `WORKSPACE` or - `WORKSPACE.bzlmod` file. -3. toolchains and execution platforms registered by modules that are - (transitive) dependencies of the root module. -4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` - [suffix](/external/migration#builtin-default-deps). +The toolchains and execution platforms registered in `WORKSPACE`, `WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this order of precedence during toolchain selection (from highest to lowest): -[register_execution_platforms]: /rules/lib/globals/module#register_execution_platforms +1. toolchains and execution platforms registered in the root module's `MODULE.bazel` file. +2. toolchains and execution platforms registered in the `WORKSPACE` or `WORKSPACE.bzlmod` file. +3. toolchains and execution platforms registered by modules that are (transitive) dependencies of the root module. +4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` [suffix](/external/migration#builtin-default-deps). ### Introduce local repositories -You may need to introduce a dependency as a local repository when you need a -local version of the dependency for debugging or you want to incorporate a -directory in your workspace as external repository. +You may need to introduce a dependency as a local repository when you need a local version of the dependency for debugging or you want to incorporate a directory in your workspace as external repository. -* **WORKSPACE** +- **WORKSPACE** - With WORKSPACE, this is achieved by two native repository rules, - [`local_repository`](/reference/be/workspace#local_repository) and - [`new_local_repository`](/reference/be/workspace#new_local_repository). + With WORKSPACE, this is achieved by two native repository rules, [`local_repository`](/reference/be/workspace#local_repository) and [`new_local_repository`](/reference/be/workspace#new_local_repository). - ```python - ## WORKSPACE - local_repository( - name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## WORKSPACE + local_repository( + name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` -* **Bzlmod** +- **Bzlmod** - With Bzlmod, you can use - [`local_path_override`](/rules/lib/globals/module#local_path_override) to - override a module with a local path. + With Bzlmod, you can use [`local_path_override`](/rules/lib/globals/module#local_path_override) to override a module with a local path. - ```python - ## MODULE.bazel - bazel_dep(name = "rules_java") - local_path_override( - module_name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "rules_java") + local_path_override( + module_name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` - Note: With `local_path_override`, you can only introduce a local directory - as a Bazel module, which means it should have a MODULE.bazel file and its - transitive dependencies are taken into consideration during dependency - resolution. In addition, all module override directives can only be used by - the root module. + Note: With `local_path_override`, you can only introduce a local directory as a Bazel module, which means it should have a MODULE.bazel file and its transitive dependencies are taken into consideration during dependency resolution. In addition, all module override directives can only be used by the root module. - It is also possible to introduce a local repository with module extension. - However, you cannot call `native.local_repository` in module extension, - there is ongoing effort on starlarkifying all native repository rules (check - [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). - Then you can call the corresponding starlark `local_repository` in a module - extension. It's also trivial to implement a custom version of - `local_repository` repository rule if this is a blocking issue for you. + It is also possible to introduce a local repository with module extension. However, you cannot call `native.local_repository` in module extension, there is ongoing effort on starlarkifying all native repository rules (check [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). Then you can call the corresponding starlark `local_repository` in a module extension. It's also trivial to implement a custom version of `local_repository` repository rule if this is a blocking issue for you. ### Bind targets -The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and -not supported in Bzlmod. It was introduced to give a target an alias in the -special `//external` package. All users depending on this should migrate away. +The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and not supported in Bzlmod. It was introduced to give a target an alias in the special `//external` package. All users depending on this should migrate away. For example, if you have @@ -573,290 +448,180 @@ bind( ) ``` -This allows other targets to depend on `//external:openssl`. You can migrate -away from this by: +This allows other targets to depend on `//external:openssl`. You can migrate away from this by: -* Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. - * Tip: Use `bazel query --output=build --noenable_bzlmod - --enable_workspace [target]` command to find relevant info - about the target. +- Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. -* Or use the [`alias`](/reference/be/general#alias) build rule - * Define the following target in a package (e.g. `//third_party`) + - Tip: Use `bazel query --output=build --noenable_bzlmod --enable_workspace [target]` command to find relevant info about the target. - ```python - ## third_party/BUILD - alias( - name = "openssl", - actual = "@my-ssl//src:openssl-lib", - ) - ``` +- Or use the [`alias`](/reference/be/general#alias) build rule + + - Define the following target in a package (e.g. `//third_party`) + + ```python + ## third_party/BUILD + alias( + name = "openssl", + actual = "@my-ssl//src:openssl-lib", + ) + ``` - * Replace all usages of `//external:openssl` with `//third_party:openssl`. + - Replace all usages of `//external:openssl` with `//third_party:openssl`. ### Fetch versus Sync -Fetch and sync commands are used to download external repos locally and keep -them updated. Sometimes also to allow building offline using the `--nofetch` -flag after fetching all repos needed for a build. +Fetch and sync commands are used to download external repos locally and keep them updated. Sometimes also to allow building offline using the `--nofetch` flag after fetching all repos needed for a build. -* **WORKSPACE** +- **WORKSPACE** - Sync performs a force fetch for all repositories, or for a specific - configured set of repos, while fetch is _only_ used to fetch for a specific - target. + Sync performs a force fetch for all repositories, or for a specific configured set of repos, while fetch is *only* used to fetch for a specific target. -* **Bzlmod** +- **Bzlmod** - The sync command is no longer applicable, but fetch offers - [various options](/reference/command-line-reference#fetch-options). - You can fetch a target, a repository, a set of configured repos or all - repositories involved in your dependency resolution and module extensions. - The fetch result is cached and to force a fetch you must include the - `--force` option during the fetch process. + The sync command is no longer applicable, but fetch offers [various options](/reference/command-line-reference#fetch-options). You can fetch a target, a repository, a set of configured repos or all repositories involved in your dependency resolution and module extensions. The fetch result is cached and to force a fetch you must include the `--force` option during the fetch process. ## Manual migration -This section provides useful information and guidance for your **manual** Bzlmod -migration process. For more automatized migration process, check [recommended -migration process](#how-migrate-to-bzlmod) section. +This section provides useful information and guidance for your **manual** Bzlmod migration process. For more automatized migration process, check [recommended migration process](#how-migrate-to-bzlmod) section. ### Know your dependencies in WORKSPACE -The first step of migration is to understand what dependencies you have. It -could be hard to figure out what exact dependencies are introduced in the -WORKSPACE file because transitive dependencies are often loaded with `*_deps` -macros. +The first step of migration is to understand what dependencies you have. It could be hard to figure out what exact dependencies are introduced in the WORKSPACE file because transitive dependencies are often loaded with `*_deps` macros. #### Inspect external dependency with workspace resolved file -Fortunately, the flag -[`--experimental_repository_resolved_file`][resolved_file_flag] -can help. This flag essentially generates a "lock file" of all fetched external -dependencies in your last Bazel command. You can find more details in this [blog -post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). - -[resolved_file_flag]: /reference/command-line-reference#flag--experimental_repository_resolved_file +Fortunately, the flag [`--experimental_repository_resolved_file`](/reference/command-line-reference#flag--experimental_repository_resolved_file) can help. This flag essentially generates a "lock file" of all fetched external dependencies in your last Bazel command. You can find more details in this [blog post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). It can be used in two ways: -1. To fetch info of external dependencies needed for building certain targets. +1. To fetch info of external dependencies needed for building certain targets. - ```shell - bazel clean --expunge - bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar - ``` + ```shell + bazel clean --expunge + bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar + ``` -2. To fetch info of all external dependencies defined in the WORKSPACE file. +2. To fetch info of all external dependencies defined in the WORKSPACE file. - ```shell - bazel clean --expunge - bazel sync --experimental_repository_resolved_file=resolved.bzl - ``` + ```shell + bazel clean --expunge + bazel sync --experimental_repository_resolved_file=resolved.bzl + ``` - With the `bazel sync` command, you can fetch all dependencies defined in the - WORKSPACE file, which include: + With the `bazel sync` command, you can fetch all dependencies defined in the WORKSPACE file, which include: - * `bind` usages - * `register_toolchains` & `register_execution_platforms` usages + - `bind` usages + - `register_toolchains` & `register_execution_platforms` usages - However, if your project is cross platforms, bazel sync may break on certain - platforms because some repository rules may only run correctly on supported - platforms. + However, if your project is cross platforms, bazel sync may break on certain platforms because some repository rules may only run correctly on supported platforms. -After running the command, you should have information of your external -dependencies in the `resolved.bzl` file. +After running the command, you should have information of your external dependencies in the `resolved.bzl` file. #### Inspect external dependency with `bazel query` You may also know `bazel query` can be used for inspecting repository rules with ```shell -bazel query --output=build //external: +bazel query --output=build //external:<repo name> ``` -While it is more convenient and much faster, [bazel query can lie about -external dependency version](https://github.com/bazelbuild/bazel/issues/12947), -so be careful using it! Querying and inspecting external -dependencies with Bzlmod is going to achieved by a [new -subcommand](https://github.com/bazelbuild/bazel/issues/15365). +While it is more convenient and much faster, [bazel query can lie about external dependency version](https://github.com/bazelbuild/bazel/issues/12947), so be careful using it! Querying and inspecting external dependencies with Bzlmod is going to achieved by a [new subcommand](https://github.com/bazelbuild/bazel/issues/15365). #### Built-in default dependencies -If you check the file generated by `--experimental_repository_resolved_file`, -you are going to find many dependencies that are not defined in your WORKSPACE. -This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE -file content to inject some default dependencies, which are usually required by -native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With -Bzlmod, those dependencies are introduced with a built-in module -[`bazel_tools`][bazel_tools] , which is a default dependency for every other -Bazel module. - -[bazel_tools]: https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools +If you check the file generated by `--experimental_repository_resolved_file`, you are going to find many dependencies that are not defined in your WORKSPACE. This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE file content to inject some default dependencies, which are usually required by native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With Bzlmod, those dependencies are introduced with a built-in module [`bazel_tools`](https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools) , which is a default dependency for every other Bazel module. ### Hybrid mode for gradual migration -Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies -from the WORKSPACE file to Bzlmod to be a gradual process. +Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies from the WORKSPACE file to Bzlmod to be a gradual process. -Note: In practice, loading "*_deps" macros in WORKSPACE often causes confusions -with Bzlmod dependencies, therefore we recommend starting with a -WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. +Note: In practice, loading "\*\_deps" macros in WORKSPACE often causes confusions with Bzlmod dependencies, therefore we recommend starting with a WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. #### WORKSPACE.bzlmod -During the migration, Bazel users may need to switch between builds with and -without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the -process smoother. +During the migration, Bazel users may need to switch between builds with and without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the process smoother. -WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, -if a WORKSPACE.bzlmod file also exists at the workspace root: +WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, if a WORKSPACE.bzlmod file also exists at the workspace root: -* `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. -* No [prefixes or suffixes](/external/migration#builtin-default-deps) are - added to the WORKSPACE.bzlmod file. +- `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. +- No [prefixes or suffixes](/external/migration#builtin-default-deps) are added to the WORKSPACE.bzlmod file. Using the WORKSPACE.bzlmod file can make the migration easier because: -* When Bzlmod is disabled, you fall back to fetching dependencies from the - original WORKSPACE file. -* When Bzlmod is enabled, you can better track what dependencies are left to - migrate with WORKSPACE.bzlmod. +- When Bzlmod is disabled, you fall back to fetching dependencies from the original WORKSPACE file. +- When Bzlmod is enabled, you can better track what dependencies are left to migrate with WORKSPACE.bzlmod. #### Repository visibility -Bzlmod is able to control which other repositories are visible from a given -repository, check [repository names and strict -deps](/external/module#repository_names_and_strict_deps) for more details. +Bzlmod is able to control which other repositories are visible from a given repository, check [repository names and strict deps](/external/module#repository_names_and_strict_deps) for more details. -Here is a summary of repository visibilities from different types of -repositories when also taking WORKSPACE into consideration. +Here is a summary of repository visibilities from different types of repositories when also taking WORKSPACE into consideration. -| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | -|----------------|--------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------|----------------------| -| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | -| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | -| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | -| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | +| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | +| ---------------------- | ------------------ | ----------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------ | +| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | +| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | +| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | +| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | -Note: For the root module, if a repository `@foo` is defined in WORKSPACE and -`@foo` is also used as an [apparent repository -name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` -refers to the one introduced in MODULE.bazel. +Note: For the root module, if a repository `@foo` is defined in WORKSPACE and `@foo` is also used as an [apparent repository name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` refers to the one introduced in MODULE.bazel. -Note: For a module extension generated repository `@bar`, if `@foo` is used as -an [apparent repository name](/external/overview#apparent-repo-name) of -another repository generated by the same module extension and direct -dependencies of the module hosting the module extension, then for repository -`@bar`, `@foo` refers to the latter. +Note: For a module extension generated repository `@bar`, if `@foo` is used as an [apparent repository name](/external/overview#apparent-repo-name) of another repository generated by the same module extension and direct dependencies of the module hosting the module extension, then for repository `@bar`, `@foo` refers to the latter. ### Manual migration process A typical Bzlmod migration process can look like this: -1. Understand what dependencies you have in WORKSPACE. -1. Add an empty MODULE.bazel file at your project root. -1. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. -1. Build your targets with Bzlmod enabled and check which repository is - missing. -1. Check the definition of the missing repository in the resolved dependency - file. -1. Introduce the missing dependency as a Bazel module, through a module - extension, or leave it in the WORKSPACE.bzlmod for later migration. -1. Go back to 4 and repeat until all dependencies are available. +1. Understand what dependencies you have in WORKSPACE. +2. Add an empty MODULE.bazel file at your project root. +3. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. +4. Build your targets with Bzlmod enabled and check which repository is missing. +5. Check the definition of the missing repository in the resolved dependency file. +6. Introduce the missing dependency as a Bazel module, through a module extension, or leave it in the WORKSPACE.bzlmod for later migration. +7. Go back to 4 and repeat until all dependencies are available. ## Publish Bazel modules -If your Bazel project is a dependency for other projects, you can publish your -project in the [Bazel Central Registry](https://registry.bazel.build/). - -To be able to check in your project in the BCR, you need a source archive URL of -the project. Take note of a few things when creating the source archive: +If your Bazel project is a dependency for other projects, you can publish your project in the [Bazel Central Registry](https://registry.bazel.build/). -* **Make sure the archive is pointing to a specific version.** +To be able to check in your project in the BCR, you need a source archive URL of the project. Take note of a few things when creating the source archive: - The BCR can only accept versioned source archives because Bzlmod needs to - conduct version comparison during dependency resolution. +- **Make sure the archive is pointing to a specific version.** -* **Make sure the archive URL is stable.** + The BCR can only accept versioned source archives because Bzlmod needs to conduct version comparison during dependency resolution. - Bazel verifies the content of the archive by a hash value, so you should - make sure the checksum of the downloaded file never changes. If the URL is - from GitHub, please create and upload a release archive in the release page. - GitHub isn't going to guarantee the checksum of source archives generated on - demand. In short, URLs in the form of - `https://github.com///releases/download/...` is considered stable - while `https://github.com///archive/...` is not. Check [GitHub - Archive Checksum - Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) - for more context. +- **Make sure the archive URL is stable.** -* **Make sure the source tree follows the layout of the original repository.** + Bazel verifies the content of the archive by a hash value, so you should make sure the checksum of the downloaded file never changes. If the URL is from GitHub, please create and upload a release archive in the release page. GitHub isn't going to guarantee the checksum of source archives generated on demand. In short, URLs in the form of `https://github.com/<org>/<repo>/releases/download/...` is considered stable while `https://github.com/<org>/<repo>/archive/...` is not. Check [GitHub Archive Checksum Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) for more context. - In case your repository is very large and you want to create a distribution - archive with reduced size by stripping out unnecessary sources, please make - sure the stripped source tree is a subset of the original source tree. This - makes it easier for end users to override the module to a non-release - version by [`archive_override`](/rules/lib/globals/module#archive_override) - and [`git_override`](/rules/lib/globals/module#git_override). +- **Make sure the source tree follows the layout of the original repository.** -* **Include a test module in a subdirectory that tests your most common - APIs.** + In case your repository is very large and you want to create a distribution archive with reduced size by stripping out unnecessary sources, please make sure the stripped source tree is a subset of the original source tree. This makes it easier for end users to override the module to a non-release version by [`archive_override`](/rules/lib/globals/module#archive_override) and [`git_override`](/rules/lib/globals/module#git_override). - A test module is a Bazel project with its own WORKSPACE and MODULE.bazel - file located in a subdirectory of the source archive which depends on the - actual module to be published. It should contain examples or some - integration tests that cover your most common APIs. Check - [test module][test_module] to learn how to set it up. +- **Include a test module in a subdirectory that tests your most common APIs.** -[test_module]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module + A test module is a Bazel project with its own WORKSPACE and MODULE.bazel file located in a subdirectory of the source archive which depends on the actual module to be published. It should contain examples or some integration tests that cover your most common APIs. Check [test module](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module) to learn how to set it up. -When you have your source archive URL ready, follow the [BCR contribution -guidelines][bcr_contrib_guide] to submit your module to the BCR with a GitHub -Pull Request. +When you have your source archive URL ready, follow the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module) to submit your module to the BCR with a GitHub Pull Request. -[bcr_contrib_guide]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module - -It is **highly recommended** to set up the [Publish to -BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your -repository to automate the process of submitting your module to the BCR. +It is **highly recommended** to set up the [Publish to BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your repository to automate the process of submitting your module to the BCR. ## Best practices -This section documents a few best practices you should follow for better -managing your external dependencies. +This section documents a few best practices you should follow for better managing your external dependencies. #### Split targets into different packages to avoid fetching unnecessary dependencies. -Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev -dependencies for tests are forced to be fetched unnecessarily for building -targets that don't need them. This is actually not Bzlmod specific, but -following this practices makes it easier to specify dev dependencies correctly. +Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev dependencies for tests are forced to be fetched unnecessarily for building targets that don't need them. This is actually not Bzlmod specific, but following this practices makes it easier to specify dev dependencies correctly. #### Specify dev dependencies -You can set the `dev_dependency` attribute to true for -[`bazel_dep`](/rules/lib/globals/module#bazel_dep) and -[`use_extension`](/rules/lib/globals/module#use_extension) directives so that -they don't propagate to dependent projects. As the root module, you can use the -[`--ignore_dev_dependency`][ignore_dev_dep_flag] flag to verify if your targets -still build without dev dependencies and overrides. - -[ignore_dev_dep_flag]: /reference/command-line-reference#flag--ignore_dev_dependency - - +You can set the `dev_dependency` attribute to true for [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_extension`](/rules/lib/globals/module#use_extension) directives so that they don't propagate to dependent projects. As the root module, you can use the [`--ignore_dev_dependency`](/reference/command-line-reference#flag--ignore_dev_dependency) flag to verify if your targets still build without dev dependencies and overrides. ## Community migration progress -You can check the [Bazel Central Registry](https://registry.bazel.build) to find -out if your dependencies are already available. Otherwise feel free to join this -[GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to -upvote or post the dependencies that are blocking your migration. +You can check the [Bazel Central Registry](https://registry.bazel.build) to find out if your dependencies are already available. Otherwise feel free to join this [GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to upvote or post the dependencies that are blocking your migration. ## Report issues -Please check the [Bazel GitHub issue list][bzlmod_github_issue] for known Bzlmod -issues. Feel free to file new issues or feature requests that can help unblock -your migration! - -[bzlmod_github_issue]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod +Please check the [Bazel GitHub issue list](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod) for known Bzlmod issues. Feel free to file new issues or feature requests that can help unblock your migration! diff --git a/external/migration_tool.mdx b/external/migration_tool.mdx new file mode 100644 index 00000000..74a2b475 --- /dev/null +++ b/external/migration_tool.mdx @@ -0,0 +1,642 @@ +--- +title: 'Bzlmod Migration Tool' +--- + +To simplify the often complex process of moving from `WORKSPACE` to Bzlmod, it's highly recommended to use the [migration script](https://github.com/bazelbuild/bazel-central-registry/blob/main/tools/migrate_to_bzlmod.py). This helper tool automates many of the steps involved in migrating your external dependency management system. + +**Note**: If you want to try out the AI driven Bzlmod migration, check [Bzlmod Migration Agent Setup](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools/code-agent). + +## Core Functionality + +The script's primary functions are: + +- **Collecting dependency information:** Analyzing your project's `WORKSPACE` file to identify external repositories used by specified build targets, using Bazel's [experimental\_repository\_resolved\_file](https://bazel.build/versions/8.2.0/reference/command-line-reference#flag--experimental_repository_resolved_file) flag to generate a resolved dependencies file containing this information. + +- **Identifying direct dependencies:** Using `bazel query` to determine which repositories are direct dependencies for the specified targets. + +- **Migrating to Bzlmod:** Translating relevant `WORKSPACE` dependencies into their Bzlmod equivalents. This is a two-step process: + + 1. Introduce all identified direct dependencies to the `MODULE.bazel` file. + 2. Build specified targets with Bzlmod enabled, then iteratively identify and fix recognizable errors. This step is needed since some dependencies might be missing in the first step. + +- **Generating a migration report:** Creating a `migration_info.md` file that documents the migration process. This report includes a list of direct dependencies, the generated Bzlmod declarations, and any manual steps that may be required to complete the migration. + +The migration tool supports: + +- Dependencies available in the Bazel Central Registry + +- User-defined custom repository rules + +- Package manager dependencies + + - Maven + - Go + - Python + +**Important Notes**: + +- The migration tool is a best-effort utility. Always double-check its recommendations for correctness. +- Use the migration tool with Bazel 7 (not supported with Bazel 8). + +## How to Use the Migration Tool + +Before you begin: + +- Upgrade to the latest Bazel 7 release, which provides robust support for both WORKSPACE and Bzlmod. + +- Verify the following command runs successfully for your project's main build targets: + + ```shell + bazel build --nobuild --enable_workspace --noenable_bzlmod <targets> + ``` + +### Command for running the script + +Once the prerequisites are met, run the following commands to use the migration tool: + +``` +# Clone the Bazel Central Registry repository +git clone https://github.com/bazelbuild/bazel-central-registry.git +cd bazel-central-registry + +# Build the migration tool +bazel build //tools:migrate_to_bzlmod + +# Create a convenient alias for the tool +alias migrate2bzlmod=$(realpath ./bazel-bin/tools/migrate_to_bzlmod) + +# Navigate to your project's root directory and run the tool +cd <your project root> +migrate2bzlmod -t <targets> +``` + +### Files generated by this script + +- `MODULE.bazel` - The central manifest file for Bzlmod, which declares the project's metadata and its direct dependencies on other Bazel modules. +- `migration_info.md` - A file providing step-by-step instructions on how the migration tool was executed, designed to assist in the manual completion of the migration process, if necessary. +- `resolved_deps.py` - Contains a comprehensive list of the project's external dependencies, generated by analyzing the project's `WORKSPACE` file, serving as a reference during the transition. +- `query_direct_deps` - Contains migration-relevant information regarding the utilized targets, obtained by invoking Bazel with `--output=build` on the project's `WORKSPACE` file. This file is primarily consumed by the migration script. +- `extension_for_XXX` - A file containing a module extension definition. The migration tool generates these files for dependencies that are not standard Bazel modules but can be managed using Bzlmod's [module extensions](/external/extension). + +### Flags + +Flags available in this migration scripts are: + +- `--t`/`--target`: Targets to migrate. This flag is repeatable, and the targets are accumulated. +- `--i`/`--initial`: Deletes `MODULE.bazel`, `resolved_deps.py`, `migration_info.md` files and starts from scratch - Detect direct dependencies, introduce them in MODULE.bazel and rerun generation of resolved dependencies. + +### Post-migration cleanup + +- Delete `migration_info.md`, `resolved_deps.py` and `query_direct_deps`. +- Clean up comments from `MODULE.bazel` file which were used for the migration, such as `# -- bazel_dep definitions -- #`. + +## Migration Example + +To see the migration script in action, consider the following scenario when Python, Maven and Go dependencies are declared in `WORKSPACE` file. + +Click here to see \`WORKSPACE\` file + +```python +workspace(name="example") + +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") +load(":my_custom_macro.bzl", "my_custom_macro") + +http_archive( + name = "rules_cc", + sha256 = "b8b918a85f9144c01f6cfe0f45e4f2838c7413961a8ff23bc0c6cdf8bb07a3b6", + strip_prefix = "rules_cc-0.1.5", + urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.1.5/rules_cc-0.1.5.tar.gz"], +) + +# Module dependency +# ------------------- +http_archive( + name = "rules_shell", + sha256 = "3e114424a5c7e4fd43e0133cc6ecdfe54e45ae8affa14fadd839f29901424043", + strip_prefix = "rules_shell-0.4.0", + url = "https://github.com/bazelbuild/rules_shell/releases/download/v0.4.0/rules_shell-v0.4.0.tar.gz", +) + +# Repo rule +# ------------------- +http_archive( + name = "com_github_cockroachdb_cockroach", + sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", + strip_prefix = "cockroach-22.1.6", + url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", +) + +# Module extension +# ------------------- +# Macro which invokes repository_rule +my_custom_macro( + name = "my_custom_repo", +) + +# Go dependencies +# ------------------- +http_archive( + name = "io_bazel_rules_go", + integrity = "sha256-M6zErg9wUC20uJPJ/B3Xqb+ZjCPn/yxFF3QdQEmpdvg=", + urls = [ + "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", + "https://github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", + ], +) + +http_archive( + name = "bazel_gazelle", + integrity = "sha256-12v3pg/YsFBEQJDfooN6Tq+YKeEWVhjuNdzspcvfWNU=", + urls = [ + "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", + "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", + ], +) + +load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies") +load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository") + +go_rules_dependencies() +go_register_toolchains(version = "1.23.1") +gazelle_dependencies() + +go_repository( + name = "org_golang_x_net", + importpath = "golang.org/x/net", + sum = "h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=", + version = "v0.0.0-20190311183353-d8887717615a", + build_file_proto_mode = "disable", + build_naming_convention = "import", +) + +# Python dependencies +# ------------------- +http_archive( + name = "rules_python", + integrity = "sha256-qDdnnxOC8mlowe5vg5x9r5B5qlMSgGmh8oFd7KpjcwQ=", + strip_prefix = "rules_python-1.4.0", + url = "https://github.com/bazelbuild/rules_python/releases/download/1.4.0/rules_python-1.4.0.tar.gz", +) + +load("@rules_python//python:repositories.bzl", "py_repositories") +py_repositories() + +load("@rules_python//python:pip.bzl", "pip_parse") +pip_parse( + name = "my_python_deps", + requirements_lock = "@example//:requirements_lock.txt", +) + +load("@my_python_deps//:requirements.bzl", "install_deps") +install_deps() + +load("@rules_python//python:repositories.bzl", "python_register_toolchains") +python_register_toolchains( + name = "python_3_11", + python_version = "3.11", +) + +# Maven dependencies +# __________________ + +RULES_JVM_EXTERNAL_TAG = "4.5" +RULES_JVM_EXTERNAL_SHA = "b17d7388feb9bfa7f2fa09031b32707df529f26c91ab9e5d909eb1676badd9a6" + +http_archive( + name = "rules_jvm_external", + strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG, + sha256 = RULES_JVM_EXTERNAL_SHA, + url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG, +) + +load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps") +rules_jvm_external_deps() +load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup") +rules_jvm_external_setup() + +load("@rules_jvm_external//:defs.bzl", "maven_install") +maven_install( + name = "px_deps", + artifacts = [ + "org.antlr:antlr4:4.11.1", + ], + repositories = [ + "https://repo1.maven.org/maven2", + ], +) +``` + +Moreover, to demonstrate usage of module extension, custom macro is invoked from `WORKSPACE` and it is defined in `my_custom_macro.bzl`. + +Click here to see \`my\_custom\_macro.bzl\` file + +```python +"""Repo rule and macro used for testing""" + +def _test_repo_rule_impl(repository_ctx): + repository_ctx.file( + "BUILD", + content = """ +genrule( + name = "foo", + outs = ["rule_name.out"], + cmd = "touch $@", + visibility = ["//visibility:public"], +) +""" + ) + +_test_repo_rule = repository_rule( + implementation = _test_repo_rule_impl, +) + +def my_custom_macro(name): + _test_repo_rule(name = name) +``` + +The end goal is to have `MODULE.bazel` file and delete the `WORKSPACE` file, without impacting the user experience. + +The first step is to follow [How to Use the Migration Tool](#migration-tool-how-to-use), which mostly is checking the bazel version (it must be Bazel 7) and adding an alias to the migration script. + +Then, running `migrate2bzlmod -t=//...` outputs: + +``` + bazel 7.6.1 + + Generating ./resolved_deps.py file - It might take a while... + + RESOLVED: rules_java has been introduced as a Bazel module. + RESOLVED: bazel_gazelle has been introduced as a Bazel module. + RESOLVED: io_bazel_rules_go has been introduced as a Bazel module. + RESOLVED: rules_python has been introduced as a Bazel module. + IMPORTANT: 3.11 is used as a default python version. If you need a different version, please change it manually and then rerun the migration tool. + RESOLVED: my_python_deps has been introduced as python extension. + RESOLVED: org_golang_x_net has been introduced as go extension. + RESOLVED: rules_jvm_external has been introduced as a Bazel module. + RESOLVED: org.antlr has been introduced as maven extension. + RESOLVED: rules_shell has been introduced as a Bazel module. + + Congratulations! All external repositories needed for building //... are available with Bzlmod! + IMPORTANT: Fix potential build time issues by running the following command: + bazel build --enable_bzlmod --noenable_workspace //... + + IMPORTANT: For details about the migration process, check `migration_info.md` file. +``` + +which gives the following important information: + +- Generates `./resolved_deps.py` file, which contains info about all external repositories declared and loaded using your `WORKSPACE` file. +- `RESOLVED` keyword describes all dependencies which are resolved by the tool and added to the `MODULE.bazel` file. +- `IMPORTANT` keyword describes significant information worth investing time. +- All dependencies have been resolved in this example, at least with `--nobuild` flag. +- It is important to run the full build (command specified) and manually fix potential errors (e.g. toolchain not registered correctly). +- `migration_info.md` file contains details about the migration. Check details [at this section](#migration-tool-report-generation). + +### Transformations + +This section illustrates the migration of code from the `WORKSPACE` file to `MODULE.bazel`. + +**WORKSPACE - Bazel Module** + +``` +http_archive( + name = "rules_shell", + sha256 = "3e114424a5c7e4fd43e0133cc6ecdfe54e45ae8affa14fadd839f29901424043", + strip_prefix = "rules_shell-0.4.0", + url = "https://github.com/bazelbuild/rules_shell/releases/download/v0.4.0/rules_shell-v0.4.0.tar.gz", +) +``` + +**MODULE.bazel - Bazel Module** + +``` +bazel_dep(name = "rules_shell", version = "0.6.1") +``` + +*** + +**WORKSPACE - Go Extension** + +``` +http_archive( + name = "io_bazel_rules_go", + integrity = "sha256-M6zErg9wUC20uJPJ/B3Xqb+ZjCPn/yxFF3QdQEmpdvg=", + urls = [ + "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", + "https://github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", + ], +) +http_archive( + name = "bazel_gazelle", + integrity = "sha256-12v3pg/YsFBEQJDfooN6Tq+YKeEWVhjuNdzspcvfWNU=", + urls = [ + "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", + "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", + ], +) + +load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies") +load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository") + +go_rules_dependencies() +go_register_toolchains(version = "1.23.1") +gazelle_dependencies() + +go_repository( +name = "org_golang_x_net", +importpath = "golang.org/x/net", +sum = "h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=", +version = "v0.0.0-20190311183353-d8887717615a", +build_file_proto_mode = "disable", +build_naming_convention = "import", +) +``` + +**MODULE.bazel - Go Extension** + +``` +go_deps = use_extension("@bazel_gazelle//:extensions.bzl", "go_deps") +go_sdk = use_extension("@io_bazel_rules_go//go:extensions.bzl", "go_sdk") + +go_deps.from_file(go_mod = "//:go.mod") +use_repo(go_deps, "org_golang_x_net") +go_sdk.from_file(go_mod = "//:go.mod") + +go_deps.gazelle_override( +path = "golang.org/x/net", +directives = [ +"gazelle:proto disable", +"gazelle:go_naming_convention import", +], +) +``` + +*** + +**WORKSPACE - Python Extension** + +``` +http_archive( + name = "rules_python", + integrity = "sha256-qDdnnxOC8mlowe5vg5x9r5B5qlMSgGmh8oFd7KpjcwQ=", + strip_prefix = "rules_python-1.4.0", + url = "https://github.com/bazelbuild/rules_python/releases/download/1.4.0/rules_python-1.4.0.tar.gz", +) + +load("@rules_python//python:repositories.bzl", "py_repositories") +py_repositories() + +load("@rules_python//python:pip.bzl", "pip_parse") +pip_parse( +name = "my_python_deps", +requirements_lock = "@example//:requirements_lock.txt", +) + +load("@my_python_deps//:requirements.bzl", "install_deps") +install_deps() + +load("@rules_python//python:repositories.bzl", "python_register_toolchains") +python_register_toolchains( +name = "python_3_11", +python_version = "3.11", +) +``` + +**MODULE.bazel - Python Extension** + +``` +pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") +pip.parse( + hub_name = "my_python_deps", + python_version = "3.11", + requirements_lock = "//:requirements_lock.txt", +) +use_repo(pip, "my_python_deps") + +python = use_extension("@rules_python//python/extensions:python.bzl", "python") +python.defaults(python_version = "3.11") +python.toolchain(python_version = "3.11") +``` + +*** + +**WORKSPACE - Maven Extension** + +``` + +RULES_JVM_EXTERNAL_TAG = "4.5" +RULES_JVM_EXTERNAL_SHA = "b17d7388feb9bfa7f2fa09031b32707df529f26c91ab9e5d909eb1676badd9a6" + +http_archive( +name = "rules_jvm_external", +strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG, +sha256 = RULES_JVM_EXTERNAL_SHA, +url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG, +) + +load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps") +rules_jvm_external_deps() +load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup") +rules_jvm_external_setup() + +load("@rules_jvm_external//:defs.bzl", "maven_install") +maven_install( +name = "px_deps", +artifacts = [ +"org.antlr:antlr4:4.11.1", +], +repositories = [ +"https://repo1.maven.org/maven2", +], +) +``` + +**MODULE.bazel - Maven Extension** + +``` +bazel_dep(name = "rules_jvm_external", version = "6.8") + +maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") +use_repo(maven, "px_deps") + +maven.artifact( +name = "px_deps", +group = "org.antlr", +artifact = "antlr4", +version = "4.11.1" +) +``` + +*** + +**WORKSPACE - Repo rule** + +``` +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + +http_archive( +name = "com_github_cockroachdb_cockroach", +sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", +strip_prefix = "cockroach-22.1.6", +url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", +) +``` + +**MODULE.bazel - Repo rule** + +``` +http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + +http_archive( +name = "com_github_cockroachdb_cockroach", +url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", +sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", +strip_prefix = "cockroach-22.1.6", +) +``` + +*** + +**WORKSPACE - Module extension** + +``` +load(":my_custom_macro.bzl", "my_custom_macro") + +my_custom_macro( +name = "my_custom_repo", +) +``` + +**MODULE.bazel - Module extension** + +``` +extension_for_my_custom_macro = use_extension("//:extension_for_my_custom_macro.bzl", "extension_for_my_custom_macro") +use_repo(extension_for_my_custom_macro, "my_custom_repo") +``` + +**extension\_for\_my\_custom\_macro.bzl** + +``` +load("//:my_custom_macro.bzl", "my_custom_macro") + +def _extension_for_my_custom_macro_impl(ctx): +my_custom_macro( +name = "my_custom_repo", +) + +extension_for_my_custom_macro = module_extension(implementation = _extension_for_my_custom_macro_impl) +``` + +*** + +## Tips with debugging + +This section provides useful commands and information to help debug issues that may arise during the Bzlmod migration. + +### Useful tips + +- Override version - Not rarely it happens that upgrading the version of a dependency causes troubles. Bzlmod could change the version of the dependency due to the [MVS algorithm](/external/module#version-selection). In order to use the same or similar version as it was in the WORKSPACE, override it with [single\_version\_override](/rules/lib/globals/module#single_version_override). Note that this is useful for debugging differences between WORKSPACE and Bzlmod, but you shouldn't rely on this feature in the long term. + + `single_version_override(module_name = "{dep_name}", version = "{version}")` + +- Use [bazel mod](/external/mod-command#syntax) command. + + - Check the version of a specified repo with `show_repo` command. For example: + + `bazel mod show_repo @rules_python` + + - Check information about a module extension with the `show_extension` command. For example: + + `bazel mod show_extension @rules_python//python/extensions:pip.bzl%pip` + +- Use [vendor mode](/external/vendor) to create a local copy of a repo when you want to monitor or control the source of the repo. For example: + + `bazel vendor --enable_bzlmod --vendor_dir=vendor_src --repo=@protobuf` + +### Migration Report Generation + +This file is updated with each run of the migration script or it's generated from scratch if it's the first run or if the [`--i` flag](#migration-script-flags) is used. The report contains: + +- Command for local testing. + +- List of direct dependencies (at least the ones which are directly used in the project). + +- For each dependency, a drop-down menu for checking where the repository was declared in the `WORKSPACE` file, which is particularly useful for the debugging. You can see it as: + + ``` + > Click here to see where and how the repo was declared in the WORKSPACE + file + ``` + +- For each dependency, how it was implemented in `MODULE.bazel` file. From the earlier [Migration Example](#migration-tool-example), that would look as: + + 1. Bazel module Dependency - `Migration of rules_python` + + ``` + Found perfect name match in BCR: rules_python + Found partially name matches in BCR: rules_python_gazelle_plugin + + It has been introduced as a Bazel module: + `bazel_dep(name = "rules_python", version = "1.6.1")` + ``` + + - The script will automatically use the `perfect name match` if it finds it. In case of an error, you can double check if the name was correctly added. + + 2. Python extension - `Migration of my_python_deps` + + ``` + pip.parse( + hub_name = "my_python_deps", + requirements_lock = "//:requirements_lock.txt", + python_version = "3.11", + ) + use_repo(pip, "my_python_deps") + ``` + + 3. Maven extension - `Migration of org.antlr (px_deps):` + + ``` + maven.artifact( + name = "px_deps", + group = "org.antlr", + artifact = "antlr4", + version = "4.11.1" + ) + ``` + + 4. Go extension - `Migration of org_golang_x_net` + + ``` + go_deps.from_file(go_mod = "//:go.mod") + go_sdk.from_file(go_mod = "//:go.mod") + + go_deps.gazelle_override( + path = "golang.org/x/net", + directives = [ + "gazelle:proto disable", + "gazelle:go_naming_convention import", + ], + ) + ``` + + - It has been introduced as a go module with the help of `go.mod`. If `go.mod` and `go.sum` are not available, go module is added directly to the `MODULE.bazel` file. + - `gazelle_override` is used for adding specific directives. + +## Useful links + +- Official pages for the external extensions + + - [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) + - [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) + - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi/download.html) + +- Community posts and videos + + - [Migrating to Bazel Modules](https://blog.engflow.com/2025/01/16/migrating-to-bazel-modules-aka-bzlmod---module-extensions/index.html#migrating-to-bazel-modules-aka-bzlmod-module-extensions). + - [Moving to Bzlmod](https://www.youtube.com/watch?v=W9uXRYLVHUk). + - [How Uber Manages Go Dependencies with Bzlmod](https://www.youtube.com/watch?v=hIqzkUE_pSY). + +## Feedback + +If you would like to contribute, do so by creating an Issue or PR at [bazel-central-registry](https://github.com/bazelbuild/bazel-central-registry). diff --git a/external/mod-command.mdx b/external/mod-command.mdx new file mode 100644 index 00000000..197bb3c6 --- /dev/null +++ b/external/mod-command.mdx @@ -0,0 +1,418 @@ +--- +title: 'mod Command' +--- + +The `mod` command provides a range of tools to help the user understand their external dependency graph. It lets you visualize the dependency graph, find out why a certain module or a version of a module is present in the graph, view the repo definitions backing modules, inspect usages of module extensions and repos they generate, among other functions. + +## Syntax + +```sh +bazel mod <subcommand> [<options>] [<arg> [<arg>...]] +``` + +The available subcommands and their respective required arguments are: + +- `graph`: Displays the full dependency graph of the project, starting from the root module. If one or more modules are specified in `--from`, these modules are shown directly under the root, and the graph is only expanded starting from them (see [example](#mod-example1)). + +- `deps <arg>...`: Displays the resolved direct dependencies of each of the specified modules, similarly to `graph`. + +- `all_paths <arg>...`: Displays all dependency paths from the --from modules to the target modules. To simplify the output, only the first shortest path is shown when multiple paths share the same suffix. For example, A -> B -> X would be shown, but the longer A -> C -> B -> X would be omitted. In other words, for every module Y that directly depends on the target module X, the output contains only the shortest path going through Y to reach X. + +- `path <arg>...`: Has the same semantics as `all_paths`, but only display a single path from one of the `--from` modules to one of the argument modules. + +- `explain <arg>...`: Shows all the places where the specified modules appear in the dependency graph, along with the modules that directly depend on them. The output of the `explain` command is essentially a pruned version of the `all_paths` command, containing 1) the root module; 2) the root module's direct dependencies that lead to the argument modules; 3) the argument modules' direct dependents; and 4) the argument modules themselves (see [example](#mod-example5)). + +- `show_repo <arg>...`: Displays the definition of the specified repos (see [example](#mod-example6)). Use `--all_repos` to show definitions of all repos in the entire dependency graph, or `--all_visible_repos` to show definitions of all repos visible from the `--base_module`. + +- `show_extension <extension>...`: Displays information about each of the specified extensions: a list of the generated repos along with the modules that import them using `use_repo`, and a list of the usages of that extension in each of the modules where it is used, containing the specified tags and the `use_repo` calls (see [example](#mod-example8)). + +`<arg>` refers to one or more modules or repos. It can be one of: + +- The literal string `<root>`: The root module representing your current project. + +- `<name>@<version>`: The module `<name>` at version `<version>`. For a module with a non-registry override, use an underscore (`_`) as the `<version>`. + +- `<name>`: All present versions of the module `<name>`. + +- `@<repo_name>`: The repo with the given [apparent name](overview#apparent-repo-name) in the context of the `--base_module`. + +- `@@<repo_name>`: The repo with the given [canonical name](overview#canonical-repo-name). + +In a context requiring specifying modules, `<arg>`s referring to repos that correspond to modules (as opposed to extension-generated repos) can also be used. Conversely, in a context requiring specifying repos, `<arg>`s referring to modules can stand in for the corresponding repos. + +`<extension>` must be of the form `<arg><label_to_bzl_file>%<extension_name>`. The `<label_to_bzl_file>` part must be a repo-relative label (for example, `//pkg/path:file.bzl`). + +The following options only affect the subcommands that print graphs (`graph`, `deps`, `all_paths`, `path`, and `explain`): + +- `--from <arg>[,<arg>[,...]]` *default: `<root>`*: The module(s) from which the graph is expanded in `graph`, `all_paths`, `path`, and `explain`. Check the subcommands' descriptions for more details. + +- `--verbose` *default: "false"*: Include in the output graph extra information about the version resolution of each module. If the module version changed during resolution, show either which version replaced it or what was the original version, the reason it was replaced, and which modules requested the new version if the reason was [Minimal Version Selection](module#version-selection). + +- `--include_unused` *default: "false"*: Include in the output graph the modules which were originally present in the dependency graph, but became unused after module resolution. + +- `--extension_info <mode>`: Include information about the module extension usages as part of the output graph (see [example](#mod-example7)). `<mode>` can be one of: + + - `hidden` *(default)*: Don't show anything about extensions. + + - `usages`: Show extensions under each module where they are used. They are printed in the form of `$<extension>`. + + - `repos`: In addition to `usages`, show the repo imported using `use_repo` under each extension usage. + + - `all`: In addition to `usages` and `repos`, also show extension-generated repos that are not imported by any module. These extra repos are shown under the first occurrence of their generating extension in the output, and are connected with a dotted edge. + +- `--extension_filter <extension>[,<extension>[,...]]`: If specified, the output graph only includes modules that use the specified extensions, and the paths that lead to those modules. Specifying an empty extension list (as in `--extension_filter=`) is equivalent to specifying *all* extensions used by any module in the dependency graph. + +- `--depth <N>`: The depth of the output graph. A depth of 1 only displays the root and its direct dependencies. Defaults to 1 for `explain`, 2 for `deps` and infinity for the others. + +- `--cycles` *default: "false"*: Include cycle edges in the output graph. + +- `--include_builtin` *default: "false"*: Include built-in modules (such as `@bazel_tools`) in the output graph. This flag is disabled by default, as built-in modules are implicitly depended on by every other module, which greatly clutters the output. + +- `--charset <charset>` *default: utf8*: Specify the charset to use for text output. Valid values are `"utf8"` and `"ascii"`. The only significant difference is in the special characters used to draw the graph in the `"text"` output format, which don't exist in the `"ascii"` charset. Therefore, the `"ascii"` charset is present to also support the usage on legacy platforms which cannot use Unicode. + +- `--output <mode>`: Include information about the module extension usages as part of the output graph. `<mode`> can be one of: + + - `text` *(default)*: A human-readable representation of the output graph (flattened as a tree). + + - `json`: Outputs the graph in the form of a JSON object (flattened as a tree). + + - `graph`: Outputs the graph in the Graphviz *dot* representation. + + Tip: Use the following command to pipe the output through the *dot* engine and export the graph representation as an SVG image. + + ```sh + bazel mod graph --output graph | dot -Tsvg > /tmp/graph.svg + ``` + +Other options include: + +- `--base_module <arg>` *default: `<root>`*: Specify a module relative to which apparent repo names in arguments are interpreted. Note that this argument itself can be in the form of `@<repo_name>`; this is always interpreted relative to the root module. + +- `--extension_usages <arg>[,<arg>[,...]]`: Filters `show_extension` to only display extension usages from the specified modules. + +## Examples + +Some possible usages of the `mod` command on a real Bazel project are showcased below to give you a general idea on how you can use it to inspect your project's external dependencies. + +`MODULE.bazel` file: + +```python +module( + name = "my_project", + version = "1.0", +) + +bazel_dep(name = "bazel_skylib", version = "1.1.1", repo_name = "skylib1") +bazel_dep(name = "bazel_skylib", version = "1.2.0", repo_name = "skylib2") +multiple_version_override(module_name = "bazel_skylib", versions = ["1.1.1", "1.2.0"]) + +bazel_dep(name = "stardoc", version = "0.5.0") +bazel_dep(name = "rules_java", version = "5.0.0") + +toolchains = use_extension("@rules_java//java:extensions.bzl", "toolchains") +use_repo(toolchains, my_jdk="remotejdk17_linux") +``` + +| | | +| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| ![Graph Before Resolution](images/mod_exampleBefore.svg)Graph Before Resolution | ![Graph After Resolution](images/mod_exampleResolved.svg)Graph After Resolution | + +1. []()Display the whole dependency graph of your project. + + ```sh + bazel mod graph + ``` + + ```none + <root> (my_project@1.0) + ├───bazel_skylib@1.1.1 + │ └───platforms@0.0.4 + ├───bazel_skylib@1.2.0 + │ └───platforms@0.0.4 ... + ├───rules_java@5.0.0 + │ ├───platforms@0.0.4 ... + │ ├───rules_cc@0.0.1 + │ │ ├───bazel_skylib@1.1.1 ... + │ │ └───platforms@0.0.4 ... + │ └───rules_proto@4.0.0 + │ ├───bazel_skylib@1.1.1 ... + │ └───rules_cc@0.0.1 ... + └───stardoc@0.5.0 + ├───bazel_skylib@1.1.1 ... + └───rules_java@5.0.0 ... + ``` + + Note: The `...` symbol indicates that the node has already been expanded somewhere else and was not expanded again to reduce noise. + +2. []()Display the whole dependency graph (including unused modules and with extra information about version resolution). + + ```sh + bazel mod graph --include_unused --verbose + ``` + + ```none + <root> (my_project@1.0) + ├───bazel_skylib@1.1.1 + │ └───platforms@0.0.4 + ├───bazel_skylib@1.2.0 + │ └───platforms@0.0.4 ... + ├───rules_java@5.0.0 + │ ├───platforms@0.0.4 ... + │ ├───rules_cc@0.0.1 + │ │ ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) + │ │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + │ │ └───platforms@0.0.4 ... + │ └───rules_proto@4.0.0 + │ ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) + │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + │ └───rules_cc@0.0.1 ... + └───stardoc@0.5.0 + ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + ├───rules_java@5.0.0 ... (was 4.0.0, cause <root>, bazel_tools@_) + ├───bazel_skylib@1.0.3 (to 1.1.1, cause multiple_version_override) + │ └───platforms@0.0.4 ... + └───rules_java@4.0.0 (to 5.0.0, cause <root>, bazel_tools@_) + ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) + └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + ``` + +3. []()Display the dependency graph expanded from some specific modules. + + ```sh + bazel mod graph --from rules_java --include_unused + ``` + + ```none + <root> (my_project@1.0) + ├───rules_java@5.0.0 + │ ├───platforms@0.0.4 + │ ├───rules_cc@0.0.1 + │ │ ├───bazel_skylib@1.0.3 ... (unused) + │ │ ├───bazel_skylib@1.1.1 ... + │ │ └───platforms@0.0.4 ... + │ └───rules_proto@4.0.0 + │ ├───bazel_skylib@1.0.3 ... (unused) + │ ├───bazel_skylib@1.1.1 ... + │ └───rules_cc@0.0.1 ... + └╌╌rules_java@4.0.0 (unused) + ├───bazel_skylib@1.0.3 (unused) + │ └───platforms@0.0.4 ... + └───bazel_skylib@1.1.1 + └───platforms@0.0.4 ... + ``` + + Note: The dotted line is used to indicate an *indirect* (transitive) dependency edge between two nodes. + +4. []()Display all paths between two of your modules. + + ```sh + bazel mod all_paths bazel_skylib@1.1.1 --from rules_proto + ``` + + ```none + <root> (my_project@1.0) + └╌╌rules_proto@4.0.0 + ├───bazel_skylib@1.1.1 + └───rules_cc@0.0.1 + └───bazel_skylib@1.1.1 ... + ``` + +5. []()See why and how your project depends on some module(s). + + ```sh + bazel mod explain @skylib1 --verbose --include_unused + ``` + + ```none + <root> (my_project@1.0) + ├───bazel_skylib@1.1.1 + ├───rules_java@5.0.0 + │ ├───rules_cc@0.0.1 + │ │ └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + │ └───rules_proto@4.0.0 + │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + │ └───rules_cc@0.0.1 ... + └───stardoc@0.5.0 + ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + ├╌╌rules_cc@0.0.1 + │ └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + └╌╌rules_proto@4.0.0 + ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) + └───rules_cc@0.0.1 ... + ``` + +6. []()See the underlying rule of your modules' repos. + + ```sh + bazel mod show_repo rules_cc stardoc + ``` + + ```none + ## rules_cc@0.0.1: + load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + http_archive( + name = "rules_cc+", + urls = ["https://bcr.bazel.build/test-mirror/github.com/bazelbuild/rules_cc/releases/download/0.0.1/rules_cc-0.0.1.tar.gz", "https://github.com/bazelbuild/rules_cc/releases/download/0.0.1/rules_cc-0.0.1.tar.gz"], + integrity = "sha256-Tcy/0iwN7xZMj0dFi9UODHFI89kgAs20WcKpamhJgkE=", + strip_prefix = "", + remote_patches = {"https://bcr.bazel.build/modules/rules_cc/0.0.1/patches/add_module_extension.patch": "sha256-g3+zmGs0YT2HKOVevZpN0Jet89Ylw90Cp9XsIAY8QqU="}, + remote_patch_strip = 1, + ) + + ## stardoc: + load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + http_archive( + name = "stardoc+", + urls = ["https://bcr.bazel.build/test-mirror/github.com/bazelbuild/stardoc/releases/download/0.5.0/stardoc-0.5.0.tar.gz", "https://github.com/bazelbuild/stardoc/releases/download/0.5.0/stardoc-0.5.0.tar.gz"], + integrity = "sha256-yXlNzIAmow/2fPfPkeviRcopSyCwcYRdEsGSr+JDrXI=", + strip_prefix = "", + remote_patches = {}, + remote_patch_strip = 0, + ) + ``` + + `bazel mod show_repo` also works with repos imported by `use_repo` and repos created with `use_repo_rule`. If `show_repo` is invoked with an apparent repository name or `--all_visible_repos`, then the apparent repository name is shown on a line prefixed with `##`. + + ```sh + bazel mod show_repo @jq_linux_arm64 + bazel mod show_repo --all_visible_repos + ``` + + ```none + ## @jq_linux_arm64: + load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + http_file( + name = "+http_file+jq_linux_arm64", + executable = True, + integrity = "sha256-TdLYoGYd8LIvG7mh+YMPBrbzuPfZEhGh7118TwaotKU=", + urls = ["https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-arm64"], + ) + ``` + +7. []()See what module extensions are used in your dependency graph. + + ```sh + bazel mod graph --extension_info=usages + ``` + + ```none + <root> (my_project@1.0) + ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains + ├───rules_java@5.0.0 # + │ ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains + │ ├───rules_cc@0.0.1 # + │ │ └───$@@rules_cc.0.0.1//bzlmod:extensions.bzl%cc_configure + │ └───rules_proto@4.0.0 + │ └───rules_cc@0.0.1 ... + └───stardoc@0.5.0 + └───rules_java@5.0.0 ... + ``` + +8. []()See what repositories are generated and imported from some specific extension as part of the dependency graph. + + ```sh + bazel mod show_extension @@rules_java+5.0.0//java:extensions.bzl%toolchains + ``` + + ```none + <root> (my_project@1.0) + ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains + │ ├───remotejdk17_linux + │ ├╌╌remotejdk11_linux + │ ├╌╌remotejdk11_linux_aarch64 + │ ├╌╌remotejdk11_linux_ppc64le + │ ├╌╌remotejdk11_linux_s390x + ...(some lines omitted)... + ├───rules_java@5.0.0 # + │ └───$@@rules_java.5.0.0//java:extensions.bzl%toolchains ... + │ ├───local_jdk + │ ├───remote_java_tools + │ ├───remote_java_tools_darwin + │ ├───remote_java_tools_linux + │ ├───remote_java_tools_windows + │ ├───remotejdk11_linux_aarch64_toolchain_config_repo + │ ├───remotejdk11_linux_ppc64le_toolchain_config_repo + ...(some lines omitted)... + └───stardoc@0.5.0 + └───rules_java@5.0.0 ... + ``` + +9. []()See the list of generated repositories of an extension and how that extension is used in each module. + + ```sh + bazel mod graph --extension_info=all --extension_filter=@rules_java//java:extensions.bzl%toolchains + ``` + + ```none + ## @@rules_java.5.0.0//java:extensions.bzl%toolchains: + + Fetched repositories: + - local_jdk (imported by bazel_tools@_, rules_java@5.0.0) + - remote_java_tools (imported by bazel_tools@_, rules_java@5.0.0) + - remote_java_tools_darwin (imported by bazel_tools@_, rules_java@5.0.0) + - remote_java_tools_linux (imported by bazel_tools@_, rules_java@5.0.0) + - remote_java_tools_windows (imported by bazel_tools@_, rules_java@5.0.0) + - remotejdk11_linux_aarch64_toolchain_config_repo (imported by rules_java@5.0.0) + - remotejdk11_linux_ppc64le_toolchain_config_repo (imported by rules_java@5.0.0) + ...(some lines omitted)... + - remotejdk17_linux (imported by <root>) + - remotejdk11_linux + - remotejdk11_linux_aarch64 + - remotejdk11_linux_ppc64le + - remotejdk11_linux_s390x + - remotejdk11_macos + ...(some lines omitted)... + + # Usage in <root> at <root>/MODULE.bazel:14:27 with the specified attributes: + use_repo( + toolchains, + my_jdk="remotejdk17_linux", + ) + + # Usage in bazel_tools@_ at bazel_tools@_/MODULE.bazel:23:32 with the specified attributes: + use_repo( + toolchains, + "local_jdk", + "remote_java_tools", + "remote_java_tools_linux", + "remote_java_tools_windows", + "remote_java_tools_darwin", + ) + + # Usage in rules_java@5.0.0 at rules_java@5.0.0/MODULE.bazel:30:27 with the specified attributes: + use_repo( + toolchains, + "remote_java_tools", + "remote_java_tools_linux", + "remote_java_tools_windows", + "remote_java_tools_darwin", + "local_jdk", + "remotejdk11_linux_toolchain_config_repo", + "remotejdk11_macos_toolchain_config_repo", + "remotejdk11_macos_aarch64_toolchain_config_repo", + ...(some lines omitted)... + ) + ``` + +10. []()See the underlying rule of some extension-generated repositories. + +```` +```sh +```` + +```` +bazel mod show_repo --base_module=rules_java @remote_java_tools +``` + +```none +## @remote_java_tools: +# <builtin> +http_archive( + name = "rules_java++toolchains+remote_java_tools", + urls = ["https://mirror.bazel.build/bazel_java_tools/releases/java/v11.5/java_tools-v11.5.zip", "https://github.com/bazelbuild/java_tools/releases/download/java_v11.5/java_tools-v11.5.zip"], + sha256 = "b763ee80e5754e593fd6d5be6d7343f905bc8b73d661d36d842b024ca11b6793", +) +# Rule http_archive defined at (most recent call last): +# /home/user/.cache/bazel/_bazel_user/6e893e0f5a92cc4cf5909a6e4b2770f9/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31 in <toplevel> +``` +```` diff --git a/external/module.mdx b/external/module.mdx index 0499c4cd..9ac802a4 100644 --- a/external/module.mdx +++ b/external/module.mdx @@ -2,16 +2,9 @@ title: 'Bazel modules' --- +A Bazel **module** is a Bazel project that can have multiple versions, each of which publishes metadata about other modules that it depends on. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. - -A Bazel **module** is a Bazel project that can have multiple versions, each of -which publishes metadata about other modules that it depends on. This is -analogous to familiar concepts in other dependency management systems, such as a -Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. - -A module must have a `MODULE.bazel` file at its repo root. This file is the -module's manifest, declaring its name, version, list of direct dependencies, and -other information. For a basic example: +A module must have a `MODULE.bazel` file at its repo root. This file is the module's manifest, declaring its name, version, list of direct dependencies, and other information. For a basic example: ```python module(name = "my-module", version = "1.0") @@ -20,50 +13,29 @@ bazel_dep(name = "rules_cc", version = "0.0.1") bazel_dep(name = "protobuf", version = "3.19.0") ``` -See the [full list](/rules/lib/globals/module) of directives available in -`MODULE.bazel` files. +See the [full list](/rules/lib/globals/module) of directives available in `MODULE.bazel` files. -To perform module resolution, Bazel starts by reading the root module's -`MODULE.bazel` file, and then repeatedly requests any dependency's -`MODULE.bazel` file from a [Bazel registry](/external/registry) until it -discovers the entire dependency graph. +To perform module resolution, Bazel starts by reading the root module's `MODULE.bazel` file, and then repeatedly requests any dependency's `MODULE.bazel` file from a [Bazel registry](/external/registry) until it discovers the entire dependency graph. -By default, Bazel then [selects](#version-selection) one version of each module -to use. Bazel represents each module with a repo, and consults the registry -again to learn how to define each of the repos. +By default, Bazel then [selects](#version-selection) one version of each module to use. Bazel represents each module with a repo, and consults the registry again to learn how to define each of the repos. ## Version format -Bazel has a diverse ecosystem and projects use various versioning schemes. The -most popular by far is [SemVer](https://semver.org), but there are -also prominent projects using different schemes such as -[Abseil](https://github.com/abseil/abseil-cpp/releases), whose -versions are date-based, for example `20210324.2`). +Bazel has a diverse ecosystem and projects use various versioning schemes. The most popular by far is [SemVer](https://semver.org), but there are also prominent projects using different schemes such as [Abseil](https://github.com/abseil/abseil-cpp/releases), whose versions are date-based, for example `20210324.2`). -For this reason, Bazel adopts a more relaxed version of the SemVer spec. The -differences include: +For this reason, Bazel adopts a more relaxed version of the SemVer spec. The differences include: -* SemVer prescribes that the "release" part of the version must consist of 3 - segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so - that any number of segments is allowed. -* In SemVer, each of the segments in the "release" part must be digits only. - In Bazel, this is loosened to allow letters too, and the comparison - semantics match the "identifiers" in the "prerelease" part. -* Additionally, the semantics of major, minor, and patch version increases are - not enforced. However, see [compatibility level](#compatibility_level) for - details on how we denote backwards compatibility. +- SemVer prescribes that the "release" part of the version must consist of 3 segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so that any number of segments is allowed. +- In SemVer, each of the segments in the "release" part must be digits only. In Bazel, this is loosened to allow letters too, and the comparison semantics match the "identifiers" in the "prerelease" part. +- Additionally, the semantics of major, minor, and patch version increases are not enforced. However, see [compatibility level](#compatibility_level) for details on how we denote backwards compatibility. -Any valid SemVer version is a valid Bazel module version. Additionally, two -SemVer versions `a` and `b` compare `a < b` if and only if the same holds when -they're compared as Bazel module versions. +Any valid SemVer version is a valid Bazel module version. Additionally, two SemVer versions `a` and `b` compare `a < b` if and only if the same holds when they're compared as Bazel module versions. -Finally, to learn more about module versioning, [see the `MODULE.bazel` -FAQ](faq#module-versioning-best-practices). +Finally, to learn more about module versioning, [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). ## Version selection -Consider the diamond dependency problem, a staple in the versioned dependency -management space. Suppose you have the dependency graph: +Consider the diamond dependency problem, a staple in the versioned dependency management space. Suppose you have the dependency graph: ``` A 1.0 @@ -73,164 +45,78 @@ management space. Suppose you have the dependency graph: D 1.0 D 1.1 ``` -Which version of `D` should be used? To resolve this question, Bazel uses the -[Minimal Version Selection](https://research.swtch.com/vgo-mvs) -(MVS) algorithm introduced in the Go module system. MVS assumes that all new -versions of a module are backwards compatible, and so picks the highest version -specified by any dependent (`D 1.1` in our example). It's called "minimal" -because `D 1.1` is the earliest version that could satisfy our requirements — -even if `D 1.2` or newer exists, we don't select them. Using MVS creates a -version selection process that is *high-fidelity* and *reproducible*. +Which version of `D` should be used? To resolve this question, Bazel uses the [Minimal Version Selection](https://research.swtch.com/vgo-mvs) (MVS) algorithm introduced in the Go module system. MVS assumes that all new versions of a module are backwards compatible, and so picks the highest version specified by any dependent (`D 1.1` in our example). It's called "minimal" because `D 1.1` is the earliest version that could satisfy our requirements — even if `D 1.2` or newer exists, we don't select them. Using MVS creates a version selection process that is *high-fidelity* and *reproducible*. ### Yanked versions -The registry can declare certain versions as *yanked* if they should be avoided -(such as for security vulnerabilities). Bazel throws an error when selecting a -yanked version of a module. To fix this error, either upgrade to a newer, -non-yanked version, or use the -[`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) -flag to explicitly allow the yanked version. +The registry can declare certain versions as *yanked* if they should be avoided (such as for security vulnerabilities). Bazel throws an error when selecting a yanked version of a module. To fix this error, either upgrade to a newer, non-yanked version, or use the [`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) flag to explicitly allow the yanked version. ## Compatibility level -In Go, MVS's assumption about backwards compatibility works because it treats -backwards incompatible versions of a module as a separate module. In terms of -SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can -coexist in the resolved dependency graph. This is, in turn, made possible by -encoding the major version in the package path in Go, so there aren't any -compile-time or linking-time conflicts. Bazel, however, cannot provide such -guarantees because it follows [a relaxed version of SemVer](#version-format). - -Thus, Bazel needs the equivalent of the SemVer major version number to detect -backwards incompatible ("breaking") versions. This number is called the -*compatibility level*, and is specified by each module version in its -[`module()`](/rule/lib/globals/module#module) directive. With this information, -Bazel can throw an error if it detects that versions of the _same module_ with -_different compatibility levels_ exist in the resolved dependency graph. - -Finally, incrementing the compatibility level can be disruptive to the users. -To learn more about when and how to increment it, [check the `MODULE.bazel` -FAQ](faq#incrementing-compatibility-level). +In Go, MVS's assumption about backwards compatibility works because it treats backwards incompatible versions of a module as a separate module. In terms of SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can coexist in the resolved dependency graph. This is, in turn, made possible by encoding the major version in the package path in Go, so there aren't any compile-time or linking-time conflicts. Bazel, however, cannot provide such guarantees because it follows [a relaxed version of SemVer](#version-format). + +Thus, Bazel needs the equivalent of the SemVer major version number to detect backwards incompatible ("breaking") versions. This number is called the *compatibility level*, and is specified by each module version in its [`module()`](/rule/lib/globals/module#module) directive. With this information, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. + +Finally, incrementing the compatibility level can be disruptive to the users. To learn more about when and how to increment it, [check the `MODULE.bazel` FAQ](faq#incrementing-compatibility-level). ## Overrides -Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel -module resolution. Only the root module's overrides take effect — if a module is -used as a dependency, its overrides are ignored. +Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel module resolution. Only the root module's overrides take effect — if a module is used as a dependency, its overrides are ignored. -Each override is specified for a certain module name, affecting all of its -versions in the dependency graph. Although only the root module's overrides take -effect, they can be for transitive dependencies that the root module does not -directly depend on. +Each override is specified for a certain module name, affecting all of its versions in the dependency graph. Although only the root module's overrides take effect, they can be for transitive dependencies that the root module does not directly depend on. ### Single-version override -The [`single_version_override`](/rules/lib/globals/module#single_version_override) -serves multiple purposes: +The [`single_version_override`](/rules/lib/globals/module#single_version_override) serves multiple purposes: -* With the `version` attribute, you can pin a dependency to a specific - version, regardless of which versions of the dependency are requested in the - dependency graph. -* With the `registry` attribute, you can force this dependency to come from a - specific registry, instead of following the normal [registry - selection](/external/registry#selecting_registries) process. -* With the `patch*` attributes, you can specify a set of patches to apply to - the downloaded module. +- With the `version` attribute, you can pin a dependency to a specific version, regardless of which versions of the dependency are requested in the dependency graph. +- With the `registry` attribute, you can force this dependency to come from a specific registry, instead of following the normal [registry selection](/external/registry#selecting_registries) process. +- With the `patch*` attributes, you can specify a set of patches to apply to the downloaded module. These attributes are all optional and can be mixed and matched with each other. ### Multiple-version override -A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) -can be specified to allow multiple versions of the same module to coexist in the -resolved dependency graph. - -You can specify an explicit list of allowed versions for the module, which must -all be present in the dependency graph before resolution — there must exist -*some* transitive dependency depending on each allowed version. After -resolution, only the allowed versions of the module remain, while Bazel upgrades -other versions of the module to the nearest higher allowed version at the same -compatibility level. If no higher allowed version at the same compatibility -level exists, Bazel throws an error. - -For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the -dependency graph before resolution and the major version is the compatibility -level: - -* A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in - `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other - versions remaining the same. -* A multiple-version override allowing `1.5` and `2.0` results in an error, as - `1.7` has no higher version at the same compatibility level to upgrade to. -* A multiple-version override allowing `1.9` and `2.0` results in an error, as - `1.9` is not present in the dependency graph before resolution. - -Additionally, users can also override the registry using the `registry` -attribute, similarly to single-version overrides. +A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) can be specified to allow multiple versions of the same module to coexist in the resolved dependency graph. + +You can specify an explicit list of allowed versions for the module, which must all be present in the dependency graph before resolution — there must exist *some* transitive dependency depending on each allowed version. After resolution, only the allowed versions of the module remain, while Bazel upgrades other versions of the module to the nearest higher allowed version at the same compatibility level. If no higher allowed version at the same compatibility level exists, Bazel throws an error. + +For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the dependency graph before resolution and the major version is the compatibility level: + +- A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other versions remaining the same. +- A multiple-version override allowing `1.5` and `2.0` results in an error, as `1.7` has no higher version at the same compatibility level to upgrade to. +- A multiple-version override allowing `1.9` and `2.0` results in an error, as `1.9` is not present in the dependency graph before resolution. + +Additionally, users can also override the registry using the `registry` attribute, similarly to single-version overrides. ### Non-registry overrides -Non-registry overrides completely remove a module from version resolution. Bazel -does not request these `MODULE.bazel` files from a registry, but instead from -the repo itself. +Non-registry overrides completely remove a module from version resolution. Bazel does not request these `MODULE.bazel` files from a registry, but instead from the repo itself. Bazel supports the following non-registry overrides: -* [`archive_override`](/rules/lib/globals/module#archive_override) -* [`git_override`](/rules/lib/globals/module#git_override) -* [`local_path_override`](/rules/lib/globals/module#local_path_override) +- [`archive_override`](/rules/lib/globals/module#archive_override) +- [`git_override`](/rules/lib/globals/module#git_override) +- [`local_path_override`](/rules/lib/globals/module#local_path_override) -Note that setting a version value in the source archive `MODULE.bazel` can have -downsides when the module is being overridden with a non-registry override. To -learn more about this [see the `MODULE.bazel` -FAQ](faq#module-versioning-best-practices). +Note that setting a version value in the source archive `MODULE.bazel` can have downsides when the module is being overridden with a non-registry override. To learn more about this [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). ## Define repos that don't represent Bazel modules -With `bazel_dep`, you can define repos that represent other Bazel modules. -Sometimes there is a need to define a repo that does _not_ represent a Bazel -module; for example, one that contains a plain JSON file to be read as data. +With `bazel_dep`, you can define repos that represent other Bazel modules. Sometimes there is a need to define a repo that does *not* represent a Bazel module; for example, one that contains a plain JSON file to be read as data. -In this case, you could use the [`use_repo_rule` -directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo -by invoking a repo rule. This repo will only be visible to the module it's -defined in. +In this case, you could use the [`use_repo_rule` directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo by invoking a repo rule. This repo will only be visible to the module it's defined in. -Under the hood, this is implemented using the same mechanism as [module -extensions](/external/extension), which lets you define repos with more -flexibility. +Under the hood, this is implemented using the same mechanism as [module extensions](/external/extension), which lets you define repos with more flexibility. ## Repository names and strict deps -The [apparent name](/external/overview#apparent-repo-name) of a repo backing a -module to its direct dependents defaults to its module name, unless the -`repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) -directive says otherwise. Note that this means a module can only find its direct -dependencies. This helps prevent accidental breakages due to changes in -transitive dependencies. - -The [canonical name](/external/overview#canonical-repo-name) of a repo backing a -module is either `module_name+version{{ -"" }}` (for example, `bazel_skylib+1.0.3`) or `module_name{{ -"" }}+` (for example, `bazel_features+`), depending on whether there are -multiple versions of the module in the entire dependency graph (see -[`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). -Note that **the canonical name format** is not an API you should depend on and -**is subject to change at any time**. Instead of hard-coding the canonical name, -use a supported way to get it directly from Bazel: - -* In BUILD and `.bzl` files, use - [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance - constructed from a label string given by the apparent name of the repo, e.g., - `Label("@bazel_skylib").repo_name`. -* When looking up runfiles, use - [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) - or one of the runfiles libraries in - `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, - in `@rules_foo//foo/runfiles`. -* When interacting with Bazel from an external tool such as an IDE or language - server, use the `bazel mod dump_repo_mapping` command to get the mapping from - apparent names to canonical names for a given set of repositories. - -[Module extensions](/external/extension) can also introduce additional repos -into the visible scope of a module. +The [apparent name](/external/overview#apparent-repo-name) of a repo backing a module to its direct dependents defaults to its module name, unless the `repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive says otherwise. Note that this means a module can only find its direct dependencies. This helps prevent accidental breakages due to changes in transitive dependencies. + +The [canonical name](/external/overview#canonical-repo-name) of a repo backing a module is either `<var>module_name</var>+<var>version</var>` (for example, `bazel_skylib+1.0.3`) or `<var>module_name</var>+` (for example, `bazel_features+`), depending on whether there are multiple versions of the module in the entire dependency graph (see [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). Note that **the canonical name format** is not an API you should depend on and **is subject to change at any time**. Instead of hard-coding the canonical name, use a supported way to get it directly from Bazel: + +- In BUILD and `.bzl` files, use [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance constructed from a label string given by the apparent name of the repo, e.g., `Label("@bazel_skylib").repo_name`. +- When looking up runfiles, use [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) or one of the runfiles libraries in `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, in `@rules_foo//foo/runfiles`. +- When interacting with Bazel from an external tool such as an IDE or language server, use the `bazel mod dump_repo_mapping` command to get the mapping from apparent names to canonical names for a given set of repositories. + +[Module extensions](/external/extension) can also introduce additional repos into the visible scope of a module. diff --git a/external/overview.mdx b/external/overview.mdx index 6e55b579..84593001 100644 --- a/external/overview.mdx +++ b/external/overview.mdx @@ -2,28 +2,15 @@ title: 'External dependencies overview' --- +Bazel supports *external dependencies*, source files (both text and binary) used in your build that are not from your workspace. For example, they could be a ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local machine outside your current workspace. - - -Bazel supports *external dependencies*, source files (both text and binary) used -in your build that are not from your workspace. For example, they could be a -ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local -machine outside your current workspace. - -This document gives an overview of the system before examining some of the -concepts in more detail. +This document gives an overview of the system before examining some of the concepts in more detail. ## Overview of the system -Bazel's external dependency system works on the basis of [*Bazel -modules*](#module), each of which is a versioned Bazel project, and -[*repositories*](#repository) (or repos), which are directory trees containing -source files. +Bazel's external dependency system works on the basis of [*Bazel modules*](#module), each of which is a versioned Bazel project, and [*repositories*](#repository) (or repos), which are directory trees containing source files. -Bazel starts from the root module -- that is, the project you're working on. -Like all modules, it needs to have a `MODULE.bazel` file at its directory root, -declaring its basic metadata and direct dependencies. The following is a basic -example: +Bazel starts from the root module -- that is, the project you're working on. Like all modules, it needs to have a `MODULE.bazel` file at its directory root, declaring its basic metadata and direct dependencies. The following is a basic example: ```python module(name = "my-module", version = "1.0") @@ -32,29 +19,13 @@ bazel_dep(name = "rules_cc", version = "0.1.1") bazel_dep(name = "platforms", version = "0.0.11") ``` -From there, Bazel looks up all transitive dependency modules in a -[*Bazel registry*](registry) — by default, the [Bazel Central -Registry](https://bcr.bazel.build/). The registry provides the -dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire -transitive dependency graph before performing version resolution. - -After version resolution, in which one version is selected for each module, -Bazel consults the registry again to learn how to define a repo for each module --- that is, how the sources for each dependency module should be fetched. Most -of the time, these are just archives downloaded from the internet and extracted. - -Modules can also specify customized pieces of data called *tags*, which are -consumed by [*module extensions*](extension) after module resolution -to define additional repos. These extensions can perform actions like file I/O -and sending network requests. Among other things, they allow Bazel to interact -with other package management systems while also respecting the dependency graph -built out of Bazel modules. - -The three kinds of repos -- the main repo (which is the source tree you're -working in), the repos representing transitive dependency modules, and the repos -created by module extensions -- form the [*workspace*](#workspace) together. -External repos (non-main repos) are fetched on demand, for example when they're -referred to by labels (like `@repo//pkg:target`) in BUILD files. +From there, Bazel looks up all transitive dependency modules in a [*Bazel registry*](registry) — by default, the [Bazel Central Registry](https://bcr.bazel.build/). The registry provides the dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire transitive dependency graph before performing version resolution. + +After version resolution, in which one version is selected for each module, Bazel consults the registry again to learn how to define a repo for each module -- that is, how the sources for each dependency module should be fetched. Most of the time, these are just archives downloaded from the internet and extracted. + +Modules can also specify customized pieces of data called *tags*, which are consumed by [*module extensions*](extension) after module resolution to define additional repos. These extensions can perform actions like file I/O and sending network requests. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. + +The three kinds of repos -- the main repo (which is the source tree you're working in), the repos representing transitive dependency modules, and the repos created by module extensions -- form the [*workspace*](#workspace) together. External repos (non-main repos) are fetched on demand, for example when they're referred to by labels (like `@repo//pkg:target`) in BUILD files. ## Benefits @@ -62,54 +33,30 @@ Bazel's external dependency system offers a wide range of benefits. ### Automatic Dependency Resolution -- **Deterministic Version Resolution**: Bazel adopts the deterministic - [MVS](module#version-selection) version resolution algorithm, - minimizing conflicts and addressing diamond dependency issues. -- **Simplified Dependency Management**: `MODULE.bazel` declares only direct - dependencies, while transitive dependencies are automatically resolved, - providing a clearer overview of the project's dependencies. -- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: - Only direct dependencies are visible, ensuring correctness and - predictability. +- **Deterministic Version Resolution**: Bazel adopts the deterministic [MVS](module#version-selection) version resolution algorithm, minimizing conflicts and addressing diamond dependency issues. +- **Simplified Dependency Management**: `MODULE.bazel` declares only direct dependencies, while transitive dependencies are automatically resolved, providing a clearer overview of the project's dependencies. +- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: Only direct dependencies are visible, ensuring correctness and predictability. ### Ecosystem Integration -- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized - repository for discovering and managing common dependencies as Bazel - modules. -- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ - library) is adapted for Bazel and made available in BCR, it streamlines its - integration for the whole community and eliminates duplicated effort and - conflicts of custom BUILD files. -- **Unified Integration with Language-Specific Package Managers**: Rulesets - streamline integration with external package managers for non-Bazel - dependencies, including: - * [rules_jvm_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) - for Maven, - * [rules_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) - for PyPi, - * [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) - for Go Modules, - * [rules_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) - for Cargo. +- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized repository for discovering and managing common dependencies as Bazel modules. + +- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ library) is adapted for Bazel and made available in BCR, it streamlines its integration for the whole community and eliminates duplicated effort and conflicts of custom BUILD files. + +- **Unified Integration with Language-Specific Package Managers**: Rulesets streamline integration with external package managers for non-Bazel dependencies, including: + + - [rules\_jvm\_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) for Maven, + - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) for PyPi, + - [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) for Go Modules, + - [rules\_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) for Cargo. ### Advanced Features -- **[Module Extensions](extension)**: The - [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module - extension features allow flexible use of custom repository rules and - resolution logic to introduce any non-Bazel dependencies. -- **[`bazel mod` Command](mod-command)**: The sub-command offers - powerful ways to inspect external dependencies. You know exactly how an - external dependency is defined and where it comes from. -- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you - need to facilitate offline builds. -- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and - accelerates dependency resolution. -- **(Upcoming) [BCR Provenance - Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: - Strengthen supply chain security by ensuring verified provenance of - dependencies. +- **[Module Extensions](extension)**: The [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module extension features allow flexible use of custom repository rules and resolution logic to introduce any non-Bazel dependencies. +- **[`bazel mod` Command](mod-command)**: The sub-command offers powerful ways to inspect external dependencies. You know exactly how an external dependency is defined and where it comes from. +- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you need to facilitate offline builds. +- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and accelerates dependency resolution. +- **(Upcoming) [BCR Provenance Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: Strengthen supply chain security by ensuring verified provenance of dependencies. ## Concepts @@ -117,8 +64,7 @@ This section gives more detail on concepts related to external dependencies. ### Module -A Bazel project that can have multiple versions, each of which can have -dependencies on other modules. +A Bazel project that can have multiple versions, each of which can have dependencies on other modules. In a local Bazel workspace, a module is represented by a repository. @@ -126,118 +72,71 @@ For more details, see [Bazel modules](module). ### Repository -A directory tree with a boundary marker file at its root, containing source -files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo -represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in -legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file -will signify the boundary of a repo; multiple such files can coexist in a -directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. ### Main repository The repository in which the current Bazel command is being run. -The root of the main repository is also known as the -**workspace root**. +The root of the main repository is also known as the [](). ### Workspace -The environment shared by all Bazel commands run in the same main repository. It -encompasses the main repo and the set of all defined external repos. +The environment shared by all Bazel commands run in the same main repository. It encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". ### Canonical repository name -The name by which a repository is always addressable. Within the context of a -workspace, each repository has a single canonical name. A target inside a repo -whose canonical name is `canonical_name` can be addressed by the label -`@@canonical_name//package:target` (note the double `@`). +The name by which a repository is always addressable. Within the context of a workspace, each repository has a single canonical name. A target inside a repo whose canonical name is `canonical_name` can be addressed by the label `@@canonical_name//package:target` (note the double `@`). The main repository always has the empty string as the canonical name. ### Apparent repository name -The name by which a repository is addressable in the context of a certain other -repo. This can be thought of as a repo's "nickname": The repo with the canonical -name `michael` might have the apparent name `mike` in the context of the repo -`alice`, but might have the apparent name `mickey` in the context of the repo -`bob`. In this case, a target inside `michael` can be addressed by the label -`@mike//package:target` in the context of `alice` (note the single `@`). +The name by which a repository is addressable in the context of a certain other repo. This can be thought of as a repo's "nickname": The repo with the canonical name `michael` might have the apparent name `mike` in the context of the repo `alice`, but might have the apparent name `mickey` in the context of the repo `bob`. In this case, a target inside `michael` can be addressed by the label `@mike//package:target` in the context of `alice` (note the single `@`). -Conversely, this can be understood as a **repository mapping**: each repo -maintains a mapping from "apparent repo name" to a "canonical repo name". +Conversely, this can be understood as a **repository mapping**: each repo maintains a mapping from "apparent repo name" to a "canonical repo name". ### Repository rule -A schema for repository definitions that tells Bazel how to materialize a -repository. For example, it could be "download a zip archive from a certain URL -and extract it", or "fetch a certain Maven artifact and make it available as a -`java_import` target", or simply "symlink a local directory". Every repo is -**defined** by calling a repo rule with an appropriate number of arguments. +A schema for repository definitions that tells Bazel how to materialize a repository. For example, it could be "download a zip archive from a certain URL and extract it", or "fetch a certain Maven artifact and make it available as a `java_import` target", or simply "symlink a local directory". Every repo is **defined** by calling a repo rule with an appropriate number of arguments. -See [Repository rules](repo) for more information about how to write -your own repository rules. +See [Repository rules](repo) for more information about how to write your own repository rules. -The most common repo rules by far are -[`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive -from a URL and extracts it, and -[`local_repository`](/reference/be/workspace#local_repository), which symlinks a -local directory that is already a Bazel repository. +The most common repo rules by far are [`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive from a URL and extracts it, and [`local_repository`](/reference/be/workspace#local_repository), which symlinks a local directory that is already a Bazel repository. ### Fetch a repository -The action of making a repo available on local disk by running its associated -repo rule. The repos defined in a workspace are not available on local disk -before they are fetched. +The action of making a repo available on local disk by running its associated repo rule. The repos defined in a workspace are not available on local disk before they are fetched. -Normally, Bazel only fetches a repo when it needs something from the repo, -and the repo hasn't already been fetched. If the repo has already been fetched -before, Bazel only re-fetches it if its definition has changed. +Normally, Bazel only fetches a repo when it needs something from the repo, and the repo hasn't already been fetched. If the repo has already been fetched before, Bazel only re-fetches it if its definition has changed. -The `fetch` command can be used to initiate a pre-fetch for a repository, -target, or all necessary repositories to perform any build. This capability -enables offline builds using the `--nofetch` option. +The `fetch` command can be used to initiate a pre-fetch for a repository, target, or all necessary repositories to perform any build. This capability enables offline builds using the `--nofetch` option. -The `--fetch` option serves to manage network access. Its default value is true. -However, when set to false (`--nofetch`), the command will utilize any cached -version of the dependency, and if none exists, the command will result in -failure. +The `--fetch` option serves to manage network access. Its default value is true. However, when set to false (`--nofetch`), the command will utilize any cached version of the dependency, and if none exists, the command will result in failure. -See [fetch options](/reference/command-line-reference#fetch-options) for more -information about controlling fetch. +See [fetch options](/reference/command-line-reference#fetch-options) for more information about controlling fetch. ### Directory layout -After being fetched, the repo can be found in the subdirectory `external` in the -[output base](/remote/output-directories), under its canonical name. +After being fetched, the repo can be found in the subdirectory `external` in the [output base](/remote/output-directories), under its canonical name. -You can run the following command to see the contents of the repo with the -canonical name `canonical_name`: +You can run the following command to see the contents of the repo with the canonical name `canonical_name`: ```posix-terminal -ls $(bazel info output_base)/external/{{ '' }} canonical_name {{ '' }} +ls $(bazel info output_base)/external/<var> canonical_name </var> ``` ### REPO.bazel file -The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost -boundary of the directory tree that constitutes a repo. It doesn't need to -contain anything to serve as a repo boundary file; however, it can also be used -to specify some common attributes for all build targets inside the repo. +The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost boundary of the directory tree that constitutes a repo. It doesn't need to contain anything to serve as a repo boundary file; however, it can also be used to specify some common attributes for all build targets inside the repo. -The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no -`load` statements are supported. The `repo()` function takes the same arguments as the [`package()` -function](/reference/be/functions#package) in `BUILD` files; whereas `package()` -specifies common attributes for all build targets inside the package, `repo()` -analogously does so for all build targets inside the repo. +The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no `load` statements are supported. The `repo()` function takes the same arguments as the [`package()` function](/reference/be/functions#package) in `BUILD` files; whereas `package()` specifies common attributes for all build targets inside the package, `repo()` analogously does so for all build targets inside the repo. -For example, you can specify a common license for all targets in your repo by -having the following `REPO.bazel` file: +For example, you can specify a common license for all targets in your repo by having the following `REPO.bazel` file: ```python repo( @@ -247,12 +146,9 @@ repo( ## The legacy WORKSPACE system -In older Bazel versions (before 9.0), external dependencies were introduced by -defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a -similar syntax to `BUILD` files, employing repo rules instead of build rules. +In older Bazel versions (before 9.0), external dependencies were introduced by defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a similar syntax to `BUILD` files, employing repo rules instead of build rules. -The following snippet is an example to use the `http_archive` repo rule in the -`WORKSPACE` file: +The following snippet is an example to use the `http_archive` repo rule in the `WORKSPACE` file: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -263,43 +159,26 @@ http_archive( ) ``` -The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` -system, by default, the canonical name of a repo is also its apparent name to -all other repos. +The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` system, by default, the canonical name of a repo is also its apparent name to all other repos. -See the [full list](/rules/lib/globals/workspace) of functions available in -`WORKSPACE` files. +See the [full list](/rules/lib/globals/workspace) of functions available in `WORKSPACE` files. ### Shortcomings of the `WORKSPACE` system -In the years after the `WORKSPACE` system was introduced, users reported many -pain points, including: - -* Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all - transitive dependencies must be defined in the `WORKSPACE` file of the main - repo, in addition to direct dependencies. -* To work around this, projects have adopted the "deps.bzl" pattern, in which - they define a macro which in turn defines multiple repos, and ask users to - call this macro in their `WORKSPACE` files. - * This has its own problems: macros cannot `load` other `.bzl` files, so - these projects have to define their transitive dependencies in this - "deps" macro, or work around this issue by having the user call multiple - layered "deps" macros. - * Bazel evaluates the `WORKSPACE` file sequentially. Additionally, - dependencies are specified using `http_archive` with URLs, without any - version information. This means that there is no reliable way to perform - version resolution in the case of diamond dependencies (`A` depends on - `B` and `C`; `B` and `C` both depend on different versions of `D`). - -Due to the shortcomings of WORKSPACE, the new module-based system (codenamed -"Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. -Read the [Bzlmod migration guide](migration) on how to migrate -to Bzlmod. +In the years after the `WORKSPACE` system was introduced, users reported many pain points, including: + +- Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all transitive dependencies must be defined in the `WORKSPACE` file of the main repo, in addition to direct dependencies. + +- To work around this, projects have adopted the "deps.bzl" pattern, in which they define a macro which in turn defines multiple repos, and ask users to call this macro in their `WORKSPACE` files. + + - This has its own problems: macros cannot `load` other `.bzl` files, so these projects have to define their transitive dependencies in this "deps" macro, or work around this issue by having the user call multiple layered "deps" macros. + - Bazel evaluates the `WORKSPACE` file sequentially. Additionally, dependencies are specified using `http_archive` with URLs, without any version information. This means that there is no reliable way to perform version resolution in the case of diamond dependencies (`A` depends on `B` and `C`; `B` and `C` both depend on different versions of `D`). + +Due to the shortcomings of WORKSPACE, the new module-based system (codenamed "Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. Read the [Bzlmod migration guide](migration) on how to migrate to Bzlmod. ### External links on Bzlmod -* [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) -* [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) - (original Bzlmod design doc) -* [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) -* [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) +- [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) +- [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) (original Bzlmod design doc) +- [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) +- [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) diff --git a/external/registry.mdx b/external/registry.mdx new file mode 100644 index 00000000..48c7cf62 --- /dev/null +++ b/external/registry.mdx @@ -0,0 +1,83 @@ +--- +title: 'Bazel registries' +--- + +Bazel discovers dependencies by requesting their information from Bazel *registries*: databases of Bazel modules. Bazel only supports one type of registries — [*index registries*](#index_registry) — local directories or static HTTP servers following a specific format. + +## Index registry + +An index registry is a local directory or a static HTTP server containing information about a list of modules — including their homepage, maintainers, the `MODULE.bazel` file of each version, and how to fetch the source of each version. Notably, it does *not* need to serve the source archives itself. + +An index registry must have the following format: + +- [`/bazel_registry.json`](#bazel-registry-json): An optional JSON file containing metadata for the registry. + +- `/modules`: A directory containing a subdirectory for each module in this registry + +- `/modules/$MODULE`: A directory containing a subdirectory for each version of the module named `$MODULE`, as well as the [`metadata.json` file](#metadata-json) containing metadata for this module. + +- `/modules/$MODULE/$VERSION`: A directory containing the following files: + + - `MODULE.bazel`: The `MODULE.bazel` file of this module version. Note that this is the `MODULE.bazel` file read during Bazel's external dependency resolution, *not* the one from the source archive (unless there's a [non-registry override](/external/module#non-registry_overrides)). Also note that it's best to use this file to set the version of a release and avoid doing so in the source archive `MODULE.bazel` file. To learn more about module versioning, [see the FAQ](faq.md#module-versioning-best-practices). + - [`source.json`](#source-json): A JSON file containing information on how to fetch the source of this module version + - `patches/`: An optional directory containing patch files, only used when `source.json` has "archive" type + - `overlay/`: An optional directory containing overlay files, only used when `source.json` has "archive" type + +### `bazel_registry.json` + +`bazel_registry.json` is an optional file that specifies metadata applying to the entire registry. It can contain the following fields: + +- `mirrors`: an array of strings, specifying the list of mirrors to use for source archives. + - The mirrored URL is a concatenation of the mirror itself, and the source URL of the module specified by its `source.json` file sans the protocol. For example, if a module's source URL is `https://foo.com/bar/baz`, and `mirrors` contains `["https://mirror1.com/", "https://example.com/mirror2/"]`, then the URLs Bazel will try in order are `https://mirror1.com/foo.com/bar/baz`, `https://example.com/mirror2/foo.com/bar/baz`, and finally the original source URL itself `https://foo.com/bar/baz`. +- `module_base_path`: a string, specifying the base path for modules with `local_path` type in the `source.json` file + +### `metadata.json` + +`metadata.json` is an optional JSON file containing information about the module, with the following fields: + +- `versions`: An array of strings, each denoting a version of the module available in this registry. This array should match the children of the module directory. +- `yanked_versions`: A JSON object specifying the [*yanked* versions](/external/module#yanked_versions) of this module. The keys should be versions to yank, and the values should be descriptions of why the version is yanked, ideally containing a link to more information. + +Note that the BCR requires more information in the `metadata.json` file. + +### `source.json` + +`source.json` is a required JSON file containing information about how to fetch a specific version of a module. The schema of this file depends on its `type` field, which defaults to `archive`. + +- If `type` is `archive` (the default), this module version is backed by an [`http_archive`](/rules/lib/repo/http#http_archive) repo rule; it's fetched by downloading an archive from a given URL and extracting its contents. It supports the following fields: + + - `url`: A string, the URL of the source archive + - `mirror_urls`: A list of string, the mirror URLs of the source archive. The URLs are tried in order after `url` as backups. + - `integrity`: A string, the [Subresource Integrity](https://w3c.github.io/webappsec-subresource-integrity/#integrity-metadata-description) checksum of the archive + - `strip_prefix`: A string, the directory prefix to strip when extracting the source archive + - `overlay`: A JSON object containing overlay files to layer on top of the extracted archive. The patch files are located under the `/modules/$MODULE/$VERSION/overlay` directory. The keys are the overlay file names, and the values are the integrity checksum of the overlay files. The overlays are applied before the patch files. + - `patches`: A JSON object containing patch files to apply to the extracted archive. The patch files are located under the `/modules/$MODULE/$VERSION/patches` directory. The keys are the patch file names, and the values are the integrity checksum of the patch files. The patches are applied after the overlay files and in the order they appear in `patches`. + - `patch_strip`: A number; the same as the `--strip` argument of Unix `patch`. + - `archive_type`: A string, the archive type of the downloaded file (Same as [`type` on `http_archive`](/rules/lib/repo/http#http_archive-type)). + +- If `type` is `git_repository`, this module version is backed by a [`git_repository`](/rules/lib/repo/git#git_repository) repo rule; it's fetched by cloning a Git repository. + - The following fields are supported, and are directly forwarded to the underlying `git_repository` repo rule: `remote`, `commit`, `shallow_since`, `tag`, `init_submodules`, `verbose`, and `strip_prefix`. + +- If `type` is `local_path`, this module version is backed by a [`local_repository`](/rules/lib/repo/local#local_repository) repo rule; it's symlinked to a directory on local disk. It supports the following field: + + - `path`: The local path to the repo, calculated as following: + + - If `path` is an absolute path, it stays as it is + - If `path` is a relative path and `module_base_path` is an absolute path, it resolves to `<module_base_path>/<path>` + - If `path` and `module_base_path` are both relative paths, it resolves to `<registry_path>/<module_base_path>/<path>`. Registry must be hosted locally and used by `--registry=file://<registry_path>`. Otherwise, Bazel will throw an error + +## Bazel Central Registry + +The Bazel Central Registry (BCR) at [https://bcr.bazel.build/](https://bcr.bazel.build/) is an index registry with contents backed by the GitHub repo [`bazelbuild/bazel-central-registry`](https://github.com/bazelbuild/bazel-central-registry). You can browse its contents using the web frontend at [https://registry.bazel.build/](https://registry.bazel.build/). + +The Bazel community maintains the BCR, and contributors are welcome to submit pull requests. See the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/blob/main/docs/README.md). + +In addition to following the format of a normal index registry, the BCR requires a `presubmit.yml` file for each module version (`/modules/$MODULE/$VERSION/presubmit.yml`). This file specifies a few essential build and test targets that you can use to check the validity of this module version. The BCR's CI pipelines also uses this to ensure interoperability between modules. + +## Selecting registries + +The repeatable Bazel flag `--registry` can be used to specify the list of registries to request modules from, so you can set up your project to fetch dependencies from a third-party or internal registry. Earlier registries take precedence. For convenience, you can put a list of `--registry` flags in the `.bazelrc` file of your project. + +If your registry is hosted on GitHub (for example, as a fork of `bazelbuild/bazel-central-registry`) then your `--registry` value needs a raw GitHub address under `raw.githubusercontent.com`. For example, on the `main` branch of the `my-org` fork, you would set `--registry=https://raw.githubusercontent.com/my-org/bazel-central-registry/main/`. + +Using the `--registry` flag stops the Bazel Central Registry from being used by default, but you can add it back by adding `--registry=https://bcr.bazel.build`. diff --git a/external/repo.mdx b/external/repo.mdx index b878f030..bbf37aee 100644 --- a/external/repo.mdx +++ b/external/repo.mdx @@ -2,39 +2,19 @@ title: 'Repository Rules' --- +This page covers how to define repository rules and provides examples for more details. - -This page covers how to define repository rules and provides examples for more -details. - -An [external repository](/external/overview#repository) is a directory tree, -containing source files usable in a Bazel build, which is generated on demand by -running its corresponding **repo rule**. Repos can be defined in a multitude of -ways, but ultimately, each repo is defined by invoking a repo rule, just as -build targets are defined by invoking build rules. They can be used to depend on -third-party libraries (such as Maven packaged libraries) but also to generate -`BUILD` files specific to the host Bazel is running on. +An [external repository](/external/overview#repository) is a directory tree, containing source files usable in a Bazel build, which is generated on demand by running its corresponding **repo rule**. Repos can be defined in a multitude of ways, but ultimately, each repo is defined by invoking a repo rule, just as build targets are defined by invoking build rules. They can be used to depend on third-party libraries (such as Maven packaged libraries) but also to generate `BUILD` files specific to the host Bazel is running on. ## Repository rule definition -In a `.bzl` file, use the -[repository_rule](/rules/lib/globals/bzl#repository_rule) function to define a -new repo rule and store it in a global variable. After a repo rule is defined, -it can be invoked as a function to define repos. This invocation is usually -performed from inside a [module extension](/external/extension) implementation -function. +In a `.bzl` file, use the [repository\_rule](/rules/lib/globals/bzl#repository_rule) function to define a new repo rule and store it in a global variable. After a repo rule is defined, it can be invoked as a function to define repos. This invocation is usually performed from inside a [module extension](/external/extension) implementation function. -The two major components of a repo rule definition are its attribute schema and -implementation function. The attribute schema determines the names and types of -attributes passed to a repo rule invocation, and the implementation function is -run when the repo needs to be fetched. +The two major components of a repo rule definition are its attribute schema and implementation function. The attribute schema determines the names and types of attributes passed to a repo rule invocation, and the implementation function is run when the repo needs to be fetched. ## Attributes -Attributes are arguments passed to the repo rule invocation. The schema of -attributes accepted by a repo rule is specified using the `attrs` argument when -the repo rule is defined with a call to `repository_rule`. An example defining -`url` and `sha256` attributes as strings: +Attributes are arguments passed to the repo rule invocation. The schema of attributes accepted by a repo rule is specified using the `attrs` argument when the repo rule is defined with a call to `repository_rule`. An example defining `url` and `sha256` attributes as strings: ```python http_archive = repository_rule( @@ -46,8 +26,7 @@ http_archive = repository_rule( ) ``` -To access an attribute within the implementation function, use -`repository_ctx.attr.`: +To access an attribute within the implementation function, use `repository_ctx.attr.<attribute_name>`: ```python def _impl(repository_ctx): @@ -55,29 +34,15 @@ def _impl(repository_ctx): checksum = repository_ctx.attr.sha256 ``` -All `repository_rule`s have the implicitly defined attribute `name`. This is a -string attribute that behaves somewhat magically: when specified as an input to -a repo rule invocation, it takes an apparent repo name; but when read from the -repo rule's implementation function using `repository_ctx.attr.name`, it returns -the canonical repo name. +All `repository_rule`s have the implicitly defined attribute `name`. This is a string attribute that behaves somewhat magically: when specified as an input to a repo rule invocation, it takes an apparent repo name; but when read from the repo rule's implementation function using `repository_ctx.attr.name`, it returns the canonical repo name. ## Implementation function -Every repo rule requires an `implementation` function. It contains the actual -logic of the rule and is executed strictly in the Loading Phase. +Every repo rule requires an `implementation` function. It contains the actual logic of the rule and is executed strictly in the Loading Phase. -The function has exactly one input parameter, `repository_ctx`. The function -returns either `None` to signify that the rule is reproducible given the -specified parameters, or a dict with a set of parameters for that rule that -would turn that rule into a reproducible one generating the same repo. For -example, for a rule tracking a git repository that would mean returning a -specific commit identifier instead of a floating branch that was originally -specified. +The function has exactly one input parameter, `repository_ctx`. The function returns either `None` to signify that the rule is reproducible given the specified parameters, or a dict with a set of parameters for that rule that would turn that rule into a reproducible one generating the same repo. For example, for a rule tracking a git repository that would mean returning a specific commit identifier instead of a floating branch that was originally specified. -The input parameter `repository_ctx` can be used to access attribute values, and -non-hermetic functions (finding a binary, executing a binary, creating a file in -the repository or downloading a file from the Internet). See [the API -docs](/rules/lib/builtins/repository_ctx) for more context. Example: +The input parameter `repository_ctx` can be used to access attribute values, and non-hermetic functions (finding a binary, executing a binary, creating a file in the repository or downloading a file from the Internet). See [the API docs](/rules/lib/builtins/repository_ctx) for more context. Example: ```python def _impl(repository_ctx): @@ -90,72 +55,38 @@ local_repository = repository_rule( ## When is the implementation function executed? -The implementation function of a repo rule is executed when Bazel needs a target -from that repository, for example when another target (in another repo) depends -on it or if it is mentioned on the command line. The implementation function is -then expected to create the repo in the file system. This is called "fetching" -the repo. - -In contrast to regular targets, repos are not necessarily re-fetched when -something changes that would cause the repo to be different. This is because -there are things that Bazel either cannot detect changes to or it would cause -too much overhead on every build (for example, things that are fetched from the -network). Therefore, repos are re-fetched only if one of the following things -changes: - -* The attributes passed to the repo rule invocation. -* The Starlark code comprising the implementation of the repo rule. -* The value of any environment variable passed to `repository_ctx`'s - `getenv()` method or declared with the `environ` attribute of the - [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of - these environment variables can be hard-wired on the command line with the - [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. -* The existence, contents, and type of any paths being - [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation - function of the repo rule. - * Certain other methods of `repository_ctx` with a `watch` parameter, such - as `read()`, `execute()`, and `extract()`, can also cause paths to be - watched. - * Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) - and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths - to be watched in other ways. -* When `bazel fetch --force` is executed. - -There are two parameters of `repository_rule` that control when the repositories -are re-fetched: - -* If the `configure` flag is set, the repository is re-fetched on `bazel - fetch --force --configure` (non-`configure` repositories are not - re-fetched). -* If the `local` flag is set, in addition to the above cases, the repo is also - re-fetched when the Bazel server restarts. +The implementation function of a repo rule is executed when Bazel needs a target from that repository, for example when another target (in another repo) depends on it or if it is mentioned on the command line. The implementation function is then expected to create the repo in the file system. This is called "fetching" the repo. + +In contrast to regular targets, repos are not necessarily re-fetched when something changes that would cause the repo to be different. This is because there are things that Bazel either cannot detect changes to or it would cause too much overhead on every build (for example, things that are fetched from the network). Therefore, repos are re-fetched only if one of the following things changes: + +- The attributes passed to the repo rule invocation. + +- The Starlark code comprising the implementation of the repo rule. + +- The value of any environment variable passed to `repository_ctx`'s `getenv()` method or declared with the `environ` attribute of the [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of these environment variables can be hard-wired on the command line with the [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. + +- The existence, contents, and type of any paths being [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation function of the repo rule. + + - Certain other methods of `repository_ctx` with a `watch` parameter, such as `read()`, `execute()`, and `extract()`, can also cause paths to be watched. + - Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths to be watched in other ways. + +- When `bazel fetch --force` is executed. + +There are two parameters of `repository_rule` that control when the repositories are re-fetched: + +- If the `configure` flag is set, the repository is re-fetched on `bazel fetch --force --configure` (non-`configure` repositories are not re-fetched). +- If the `local` flag is set, in addition to the above cases, the repo is also re-fetched when the Bazel server restarts. ## Forcing refetch of external repos -Sometimes, an external repo can become outdated without any change to its -definition or dependencies. For example, a repo fetching sources might follow a -particular branch of a third-party repository, and new commits are available on -that branch. In this case, you can ask bazel to refetch all external repos -unconditionally by calling `bazel fetch --force --all`. +Sometimes, an external repo can become outdated without any change to its definition or dependencies. For example, a repo fetching sources might follow a particular branch of a third-party repository, and new commits are available on that branch. In this case, you can ask bazel to refetch all external repos unconditionally by calling `bazel fetch --force --all`. -Moreover, some repo rules inspect the local machine and might become outdated if -the local machine was upgraded. Here you can ask Bazel to only refetch those -external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) -definition has the `configure` attribute set, use `bazel fetch --force ---configure`. +Moreover, some repo rules inspect the local machine and might become outdated if the local machine was upgraded. Here you can ask Bazel to only refetch those external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) definition has the `configure` attribute set, use `bazel fetch --force --configure`. ## Examples -- [C++ auto-configured - toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): - it uses a repo rule to automatically create the C++ configuration files for - Bazel by looking for the local C++ compiler, the environment and the flags - the C++ compiler supports. +- [C++ auto-configured toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): it uses a repo rule to automatically create the C++ configuration files for Bazel by looking for the local C++ compiler, the environment and the flags the C++ compiler supports. -- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) - uses several `repository_rule` to defines the list of dependencies needed to - use the Go rules. +- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) uses several `repository_rule` to defines the list of dependencies needed to use the Go rules. -- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) - creates an external repository called `@maven` by default that generates - build targets for every Maven artifact in the transitive dependency tree. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) creates an external repository called `@maven` by default that generates build targets for every Maven artifact in the transitive dependency tree. diff --git a/external/vendor.mdx b/external/vendor.mdx index 653c1942..a2e79e8d 100644 --- a/external/vendor.mdx +++ b/external/vendor.mdx @@ -1,13 +1,8 @@ -keywords: product:Bazel,Bzlmod,vendor --- title: 'Vendor Mode' --- - - -Vendor mode is a feature that lets you create a local copy of -external dependencies. This is useful for offline builds, or when you want to -control the source of an external dependency. +Vendor mode is a feature that lets you create a local copy of external dependencies. This is useful for offline builds, or when you want to control the source of an external dependency. ## Enable vendor mode @@ -16,19 +11,15 @@ You can enable vendor mode by specifying `--vendor_dir` flag. For example, by adding it to your `.bazelrc` file: ```none -# Enable vendor mode with vendor directory under /vendor_src +# Enable vendor mode with vendor directory under <workspace>/vendor_src common --vendor_dir=vendor_src ``` -The vendor directory can be either a relative path to your workspace root or an -absolute path. +The vendor directory can be either a relative path to your workspace root or an absolute path. ## Vendor a specific external repository -You can use the `vendor` command with the `--repo` flag to specify which repo -to vendor, it accepts both [canonical repo -name](/external/overview#canonical-repo-name) and [apparent repo -name](/external/overview#apparent-repo-name). +You can use the `vendor` command with the `--repo` flag to specify which repo to vendor, it accepts both [canonical repo name](/external/overview#canonical-repo-name) and [apparent repo name](/external/overview#apparent-repo-name). For example, running: @@ -42,13 +33,11 @@ or bazel vendor --vendor_dir=vendor_src --repo=@@rules_cc+ ``` -will both get rules_cc to be vendored under -`/vendor_src/rules_cc+`. +will both get rules\_cc to be vendored under `<workspace root>/vendor_src/rules_cc+`. ## Vendor external dependencies for given targets -To vendor all external dependencies required for building given target patterns, -you can run `bazel vendor `. +To vendor all external dependencies required for building given target patterns, you can run `bazel vendor <target patterns>`. For example @@ -56,12 +45,9 @@ For example bazel vendor --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -will vendor all repos required for building the `//src/main:hello-world` target -and all targets under `//src/test/...` with the current configuration. +will vendor all repos required for building the `//src/main:hello-world` target and all targets under `//src/test/...` with the current configuration. -Under the hood, it's doing a `bazel build --nobuild` command to analyze the -target patterns, therefore build flags could be applied to this command and -affect the result. +Under the hood, it's doing a `bazel build --nobuild` command to analyze the target patterns, therefore build flags could be applied to this command and affect the result. ### Build the target offline @@ -71,20 +57,15 @@ With the external dependencies vendored, you can build the target offline by bazel build --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -The build should work in a clean build environment without network access and -repository cache. +The build should work in a clean build environment without network access and repository cache. -Therefore, you should be able to check in the vendored source and build the same -targets offline on another machine. +Therefore, you should be able to check in the vendored source and build the same targets offline on another machine. -Note: If you make changes to the targets to build, the external dependencies, -the build configuration, or the Bazel version, you may need to re-vendor to make -sure offline build still works. +Note: If you make changes to the targets to build, the external dependencies, the build configuration, or the Bazel version, you may need to re-vendor to make sure offline build still works. ## Vendor all external dependencies -To vendor all repos in your transitive external dependencies graph, you can -run: +To vendor all repos in your transitive external dependencies graph, you can run: ```none bazel vendor --vendor_dir=vendor_src @@ -92,25 +73,20 @@ bazel vendor --vendor_dir=vendor_src Note that vendoring all dependencies has a few **disadvantages**: -- Fetching all repos, including those introduced transitively, can be time-consuming. -- The vendor directory can become very large. -- Some repos may fail to fetch if they are not compatible with the current platform or environment. +- Fetching all repos, including those introduced transitively, can be time-consuming. +- The vendor directory can become very large. +- Some repos may fail to fetch if they are not compatible with the current platform or environment. Therefore, consider vendoring for specific targets first. ## Configure vendor mode with VENDOR.bazel -You can control how given repos are handled with the VENDOR.bazel file located -under the vendor directory. +You can control how given repos are handled with the VENDOR.bazel file located under the vendor directory. -There are two directives available, both accepting a list of -[canonical repo names](/external/overview#canonical-repo-name) as arguments: +There are two directives available, both accepting a list of [canonical repo names](/external/overview#canonical-repo-name) as arguments: - `ignore()`: to completely ignore a repository from vendor mode. -- `pin()`: to pin a repository to its current vendored source as if there is a - `--override_repository` flag for this repo. Bazel will NOT update the vendored - source for this repo while running the vendor command unless it's unpinned. - The user can modify and maintain the vendored source for this repo manually. +- `pin()`: to pin a repository to its current vendored source as if there is a `--override_repository` flag for this repo. Bazel will NOT update the vendored source for this repo while running the vendor command unless it's unpinned. The user can modify and maintain the vendored source for this repo manually. For example @@ -121,93 +97,59 @@ pin("@@bazel_skylib+") With this configuration -- Both repos will be excluded from subsequent vendor commands. -- Repo `bazel_skylib` will be overridden to the source located under the - vendor directory. -- The user can safely modify the vendored source of `bazel_skylib`. -- To re-vendor `bazel_skylib`, the user has to disable the pin statement - first. +- Both repos will be excluded from subsequent vendor commands. +- Repo `bazel_skylib` will be overridden to the source located under the vendor directory. +- The user can safely modify the vendored source of `bazel_skylib`. +- To re-vendor `bazel_skylib`, the user has to disable the pin statement first. -Note: Repository rules with -[`local`](/rules/lib/globals/bzl#repository_rule.local) or -[`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are -always excluded from vendoring. +Note: Repository rules with [`local`](/rules/lib/globals/bzl#repository_rule.local) or [`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are always excluded from vendoring. ## Understand how vendor mode works -Bazel fetches external dependencies of a project under `$(bazel info -output_base)/external`. Vendoring external dependencies means moving out -relevant files and directories to the given vendor directory and use the -vendored source for later builds. +Bazel fetches external dependencies of a project under `$(bazel info output_base)/external`. Vendoring external dependencies means moving out relevant files and directories to the given vendor directory and use the vendored source for later builds. The content being vendored includes: -- The repo directory -- The repo marker file +- The repo directory +- The repo marker file -During a build, if the vendored marker file is up-to-date or the repo is -pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating -a symlink to it under `$(bazel info output_base)/external` instead of actually -running the repository rule. Otherwise, a warning is printed and Bazel will -fallback to fetching the latest version of the repo. +During a build, if the vendored marker file is up-to-date or the repo is pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating a symlink to it under `$(bazel info output_base)/external` instead of actually running the repository rule. Otherwise, a warning is printed and Bazel will fallback to fetching the latest version of the repo. -Note: Bazel assumes the vendored source is not changed by users unless the repo -is pinned in the VENDOR.bazel file. If a user does change the vendored source -without pinning the repo, the changed vendored source will be used, but it will -be overwritten if its existing marker file is -outdated and the repo is vendored again. +Note: Bazel assumes the vendored source is not changed by users unless the repo is pinned in the VENDOR.bazel file. If a user does change the vendored source without pinning the repo, the changed vendored source will be used, but it will be overwritten if its existing marker file is outdated and the repo is vendored again. ### Vendor registry files -Bazel has to perform the Bazel module resolution in order to fetch external -dependencies, which may require accessing registry files through internet. To -achieve offline build, Bazel vendors all registry files fetched from -network under the `/_registries` directory. +Bazel has to perform the Bazel module resolution in order to fetch external dependencies, which may require accessing registry files through internet. To achieve offline build, Bazel vendors all registry files fetched from network under the `<vendor_dir>/_registries` directory. ### Vendor symlinks -External repositories may contain symlinks pointing to other files or -directories. To make sure symlinks work correctly, Bazel uses the following -strategy to rewrite symlinks in the vendored source: +External repositories may contain symlinks pointing to other files or directories. To make sure symlinks work correctly, Bazel uses the following strategy to rewrite symlinks in the vendored source: -- Create a symlink `/bazel-external` that points to `$(bazel info - output_base)/external`. It is refreshed by every Bazel command - automatically. -- For the vendored source, rewrite all symlinks that originally point to a - path under `$(bazel info output_base)/external` to a relative path under - `/bazel-external`. +- Create a symlink `<vendor_dir>/bazel-external` that points to `$(bazel info output_base)/external`. It is refreshed by every Bazel command automatically. +- For the vendored source, rewrite all symlinks that originally point to a path under `$(bazel info output_base)/external` to a relative path under `<vendor_dir>/bazel-external`. For example, if the original symlink is ```none -/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file +<vendor_dir>/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file ``` It will be rewritten to ```none -/repo_foo+/link => ../../bazel-external/repo_bar+/file +<vendor_dir>/repo_foo+/link => ../../bazel-external/repo_bar+/file ``` where ```none -/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed +<vendor_dir>/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed ``` -Since `/bazel-external` is generated by Bazel automatically, it's -recommended to add it to `.gitignore` or equivalent to avoid checking it in. - -With this strategy, symlinks in the vendored source should work correctly even -after the vendored source is moved to another location or the bazel output base -is changed. +Since `<vendor_dir>/bazel-external` is generated by Bazel automatically, it's recommended to add it to `.gitignore` or equivalent to avoid checking it in. -Note: symlinks that point to an absolute path outside of $(bazel info -output_base)/external are not rewritten. Therefore, it could still break -cross-machine compatibility. +With this strategy, symlinks in the vendored source should work correctly even after the vendored source is moved to another location or the bazel output base is changed. -Note: On Windows, vendoring symlinks only works with -[`--windows_enable_symlinks`][windows_enable_symlinks] -flag enabled. +Note: symlinks that point to an absolute path outside of $(bazel info output\_base)/external are not rewritten. Therefore, it could still break cross-machine compatibility. -[windows_enable_symlinks]: /reference/command-line-reference#flag--windows_enable_symlinks +Note: On Windows, vendoring symlinks only works with [`--windows_enable_symlinks`](/reference/command-line-reference#flag--windows_enable_symlinks) flag enabled. diff --git a/help.mdx b/help.mdx index 754b4bc2..af11fb80 100644 --- a/help.mdx +++ b/help.mdx @@ -2,53 +2,49 @@ title: 'Getting Help' --- - - -This page lists Bazel resources beyond the documentation and covers how to get -support from the Bazel team and community. +This page lists Bazel resources beyond the documentation and covers how to get support from the Bazel team and community. ## Search existing material In addition to the documentation, you can find helpful information by searching: -* [Bazel user group](https://groups.google.com/g/bazel-discuss) -* [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) -* [Bazel blog](https://blog.bazel.build/) -* [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -* [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) +- [Bazel user group](https://groups.google.com/g/bazel-discuss) +- [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) +- [Bazel blog](https://blog.bazel.build/) +- [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +- [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) ## Watch videos There are recordings of Bazel talks at various conferences, such as: -* Bazel’s annual conference, BazelCon: - * [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) - * [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) - * [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) - * [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) - * [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) - * [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) - * [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) - * [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) -* Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) +- Bazel’s annual conference, BazelCon: + + - [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) + - [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) + - [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) + - [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) + - [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) + - [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) + - [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) + - [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +- Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) ## Ask the Bazel community If there are no existing answers, you can ask the community by: -* Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) -* Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) -* Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -* Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) -* Consulting a [Bazel community expert](/community/experts) +- Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) +- Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) +- Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +- Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) +- Consulting a [Bazel community expert](/community/experts) ## Understand Bazel's support level -Please read the [release page](/release) to understand Bazel's release model and -what level of support Bazel provides. +Please read the [release page](/release) to understand Bazel's release model and what level of support Bazel provides. ## File a bug -If you encounter a bug or want to request a feature, file a [GitHub -Issue](https://github.com/bazelbuild/bazel/issues). +If you encounter a bug or want to request a feature, file a [GitHub Issue](https://github.com/bazelbuild/bazel/issues). diff --git a/install/bazelisk.mdx b/install/bazelisk.mdx index a3189cb7..4bc9f3e5 100644 --- a/install/bazelisk.mdx +++ b/install/bazelisk.mdx @@ -2,58 +2,39 @@ title: 'Installing / Updating Bazel using Bazelisk' --- - - ## Installing Bazel -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the -recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically -downloads and installs the appropriate version of Bazel. Use Bazelisk if you -need to switch between different versions of Bazel depending on the current -working directory, or to always keep Bazel updated to the latest release. +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically downloads and installs the appropriate version of Bazel. Use Bazelisk if you need to switch between different versions of Bazel depending on the current working directory, or to always keep Bazel updated to the latest release. -For more details, see -[the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). +For more details, see [the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). ## Updating Bazel -Bazel has a [backward compatibility policy](/release/backward-compatibility) -(see [guidance for rolling out incompatible -changes](/contribute/breaking-changes) if you -are the author of one). That page summarizes best practices on how to test and -migrate your project with upcoming incompatible changes and how to provide -feedback to the incompatible change authors. +Bazel has a [backward compatibility policy](/release/backward-compatibility) (see [guidance for rolling out incompatible changes](/contribute/breaking-changes) if you are the author of one). That page summarizes best practices on how to test and migrate your project with upcoming incompatible changes and how to provide feedback to the incompatible change authors. ### Managing Bazel versions with Bazelisk -[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage -Bazel versions. +[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage Bazel versions. Bazelisk can: -* Auto-update Bazel to the latest LTS or rolling release. -* Build the project with a Bazel version specified in the .bazelversion - file. Check in that file into your version control to ensure reproducibility - of your builds. -* Help migrate your project for incompatible changes (see above) -* Easily try release candidates +- Auto-update Bazel to the latest LTS or rolling release. +- Build the project with a Bazel version specified in the .bazelversion file. Check in that file into your version control to ensure reproducibility of your builds. +- Help migrate your project for incompatible changes (see above) +- Easily try release candidates ### Recommended migration process -Within minor updates to any LTS release, any -project can be prepared for the next release without breaking -compatibility with the current release. However, there may be -backward-incompatible changes between major LTS versions. +Within minor updates to any LTS release, any project can be prepared for the next release without breaking compatibility with the current release. However, there may be backward-incompatible changes between major LTS versions. Follow this process to migrate from one major version to another: 1. Read the release notes to get advice on how to migrate to the next version. -1. Major incompatible changes should have an associated `--incompatible_*` flag - and a corresponding GitHub issue: - * Migration guidance is available in the associated GitHub issue. - * Tooling is available for some of incompatible changes migration. For - example, [buildifier](https://github.com/bazelbuild/buildtools/releases). - * Report migration problems by commenting on the associated GitHub issue. - -After migration, you can continue to build your projects without worrying about -backward-compatibility until the next major release. + +2. Major incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue: + + - Migration guidance is available in the associated GitHub issue. + - Tooling is available for some of incompatible changes migration. For example, [buildifier](https://github.com/bazelbuild/buildtools/releases). + - Report migration problems by commenting on the associated GitHub issue. + +After migration, you can continue to build your projects without worrying about backward-compatibility until the next major release. diff --git a/install/compile-source.mdx b/install/compile-source.mdx index 3b87883d..8fc9f63c 100644 --- a/install/compile-source.mdx +++ b/install/compile-source.mdx @@ -2,91 +2,67 @@ title: 'Compiling Bazel from Source' --- - - -This page describes how to install Bazel from source and provides -troubleshooting tips for common issues. +This page describes how to install Bazel from source and provides troubleshooting tips for common issues. To build Bazel from source, you can do one of the following: -* Build it [using an existing Bazel binary](#build-bazel-using-bazel) +- Build it [using an existing Bazel binary](#build-bazel-using-bazel) -* Build it [without an existing Bazel binary](#bootstrap-bazel) which is known - as _bootstrapping_. +- Build it [without an existing Bazel binary](#bootstrap-bazel) which is known as *bootstrapping*. ## Build Bazel using Bazel ### Summary -1. Get the latest Bazel release from the - [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with - [Bazelisk](https://github.com/bazelbuild/bazelisk). +1. Get the latest Bazel release from the [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with [Bazelisk](https://github.com/bazelbuild/bazelisk). -2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) - and extract somewhere. - Alternatively you can git clone the source tree from https://github.com/bazelbuild/bazel +2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) and extract somewhere. Alternatively you can git clone the source tree from [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) -3. Install the same prerequisites as for bootstrapping (see - [for Unix-like systems](#bootstrap-unix-prereq) or - [for Windows](#bootstrap-windows-prereq)) +3. Install the same prerequisites as for bootstrapping (see [for Unix-like systems](#bootstrap-unix-prereq) or [for Windows](#bootstrap-windows-prereq)) -4. Build a development build of Bazel using Bazel: - `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on - Windows) +4. Build a development build of Bazel using Bazel: `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on Windows) -5. The resulting binary is at `bazel-bin/src/bazel-dev` - (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you - like and use immediately without further installation. +5. The resulting binary is at `bazel-bin/src/bazel-dev` (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you like and use immediately without further installation. Detailed instructions follow below. ### Step 1: Get the latest Bazel release -**Goal**: Install or download a release version of Bazel. Make sure you can run -it by typing `bazel` in a terminal. +**Goal**: Install or download a release version of Bazel. Make sure you can run it by typing `bazel` in a terminal. -**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing -Bazel binary. You can install one from a package manager or download one from -GitHub. See [Installing Bazel](/install). (Or you can [build from -scratch (bootstrap)](#bootstrap-bazel).) +**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing Bazel binary. You can install one from a package manager or download one from GitHub. See [Installing Bazel](/install). (Or you can [build from scratch (bootstrap)](#bootstrap-bazel).) **Troubleshooting**: -* If you cannot run Bazel by typing `bazel` in a terminal: +- If you cannot run Bazel by typing `bazel` in a terminal: - * Maybe your Bazel binary's directory is not on the PATH. + - Maybe your Bazel binary's directory is not on the PATH. - This is not a big problem. Instead of typing `bazel`, you will need to - type the full path. + This is not a big problem. Instead of typing `bazel`, you will need to type the full path. - * Maybe the Bazel binary itself is not called `bazel` (on Unixes) or - `bazel.exe` (on Windows). + - Maybe the Bazel binary itself is not called `bazel` (on Unixes) or `bazel.exe` (on Windows). - This is not a big problem. You can either rename the binary, or type the - binary's name instead of `bazel`. + This is not a big problem. You can either rename the binary, or type the binary's name instead of `bazel`. - * Maybe the binary is not executable (on Unixes). + - Maybe the binary is not executable (on Unixes). - You must make the binary executable by running `chmod +x /path/to/bazel`. + You must make the binary executable by running `chmod +x /path/to/bazel`. ### Step 2: Download Bazel's sources from GitHub -If you are familiar with Git, then just git clone https://github.com/bazelbuild/bazel +If you are familiar with Git, then just git clone [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) Otherwise: -1. Download the - [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). +1. Download the [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). -2. Extract the contents somewhere. +2. Extract the contents somewhere. - For example create a `bazel-src` directory under your home directory and - extract there. + For example create a `bazel-src` directory under your home directory and extract there. ### Step 3: Install prerequisites -Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ -compiler, MSYS2 (if you are building on Windows), etc. +Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ compiler, MSYS2 (if you are building on Windows), etc. ### Step 4a: Build Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -96,66 +72,63 @@ For instructions for Windows, see [Build Bazel on Windows](#build-bazel-on-windo **Instructions**: -1. Start a Bash terminal +1. Start a Bash terminal -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - cd ~/bazel-src + ``` + cd ~/bazel-src + ``` -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev + ``` + bazel build //src:bazel-dev + ``` - Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` - to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel - version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). +4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). ### Step 4b: Build Bazel on Windows -For instructions for Unix-like systems, see -[Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). +For instructions for Unix-like systems, see [Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). -**Goal**: Run Bazel to build a custom Bazel binary -(`bazel-bin\src\bazel-dev.exe`). +**Goal**: Run Bazel to build a custom Bazel binary (`bazel-bin\src\bazel-dev.exe`). **Instructions**: -1. Start Command Prompt (Start Menu > Run > "cmd.exe") +1. Start Command Prompt (Start Menu > Run > "cmd.exe") -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - cd %USERPROFILE%\bazel-src + ``` + cd %USERPROFILE%\bazel-src + ``` -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev.exe + bazel build //src:bazel-dev.exe - Alternatively you can run `bazel build //src:bazel.exe - --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel.exe --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel - version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin\src\bazel-dev.exe` (or - `bazel-bin\src\bazel.exe`). +4. The output will be at `bazel-bin\src\bazel-dev.exe` (or `bazel-bin\src\bazel.exe`). ### Step 5: Install the built binary Actually, there's nothing to install. -The output of the previous step is a self-contained Bazel binary. You can copy -it to any directory and use immediately. (It's useful if that directory is on -your PATH so that you can run "bazel" everywhere.) +The output of the previous step is a self-contained Bazel binary. You can copy it to any directory and use immediately. (It's useful if that directory is on your PATH so that you can run "bazel" everywhere.) ---- +*** ## Build Bazel from scratch (bootstrapping) @@ -165,24 +138,16 @@ You can also build Bazel from scratch, without using an existing Bazel binary. (This step is the same for all platforms.) -1. Download `bazel--dist.zip` from - [GitHub](https://github.com/bazelbuild/bazel/releases), for example - `bazel-0.28.1-dist.zip`. +1. Download `bazel-<version>-dist.zip` from [GitHub](https://github.com/bazelbuild/bazel/releases), for example `bazel-0.28.1-dist.zip`. - **Attention**: + **Attention**: - - There is a **single, architecture-independent** distribution archive. - There are no architecture-specific or OS-specific distribution archives. - - These sources are **not the same as the GitHub source tree**. You - have to use the distribution archive to bootstrap Bazel. You cannot - use a source tree cloned from GitHub. (The distribution archive contains - generated source files that are required for bootstrapping and are not part - of the normal Git source tree.) + - There is a **single, architecture-independent** distribution archive. There are no architecture-specific or OS-specific distribution archives. + - These sources are **not the same as the GitHub source tree**. You have to use the distribution archive to bootstrap Bazel. You cannot use a source tree cloned from GitHub. (The distribution archive contains generated source files that are required for bootstrapping and are not part of the normal Git source tree.) -2. Unpack the distribution archive somewhere on disk. +2. Unpack the distribution archive somewhere on disk. - You should verify the signature made by Bazel's - [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. + You should verify the signature made by Bazel's [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. ### Step 2a: Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -190,18 +155,17 @@ For instructions for Windows, see [Bootstrap Bazel on Windows](#bootstrap-window #### 2.1. Install the prerequisites -* **Bash** +- **Bash** -* **zip, unzip** +- **zip, unzip** -* **C++ build toolchain** +- **C++ build toolchain** -* **JDK.** Version 21 is required. +- **JDK.** Version 21 is required. -* **Python**. Version 3 is required. +- **Python**. Version 3 is required. -For example on Ubuntu Linux you can install these requirements using the -following command: +For example on Ubuntu Linux you can install these requirements using the following command: ```sh sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip @@ -209,90 +173,76 @@ sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip #### 2.2. Bootstrap Bazel on Unix -1. Open a shell or Terminal window. +1. Open a shell or Terminal window. -3. `cd` to the directory where you unpacked the distribution archive. +2. `cd` to the directory where you unpacked the distribution archive. -3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. +3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. -The compiled output is placed into `output/bazel`. This is a self-contained -Bazel binary, without an embedded JDK. You can copy it anywhere or use it -in-place. For convenience, copy this binary to a directory that's on your -`PATH` (such as `/usr/local/bin` on Linux). +The compiled output is placed into `output/bazel`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH` (such as `/usr/local/bin` on Linux). -To build the `bazel` binary in a reproducible way, also set -[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) -in the "Run the compilation script" step. +To build the `bazel` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. ### Step 2b: Bootstrap Bazel on Windows -For instructions for Unix-like systems, see -[Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). +For instructions for Unix-like systems, see [Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). #### 2.1. Install the prerequisites -* [MSYS2 shell](https://msys2.github.io/) +- [MSYS2 shell](https://msys2.github.io/) -* **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: +- **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: - ``` - pacman -S zip unzip patch - ``` + ``` + pacman -S zip unzip patch + ``` -* **The Visual C++ compiler.** Install the Visual C++ compiler either as part - of Visual Studio 2015 or newer, or by installing the latest [Build Tools - for Visual Studio 2017](https://aka.ms/BuildTools). +- **The Visual C++ compiler.** Install the Visual C++ compiler either as part of Visual Studio 2015 or newer, or by installing the latest [Build Tools for Visual Studio 2017](https://aka.ms/BuildTools). -* **JDK.** Version 21 is required. +- **JDK.** Version 21 is required. -* **Python**. Versions 2 and 3 are supported, installing one of them is - enough. You need the Windows-native version (downloadable from - [https://www.python.org](https://www.python.org)). Versions installed via - pacman in MSYS2 will not work. +- **Python**. Versions 2 and 3 are supported, installing one of them is enough. You need the Windows-native version (downloadable from [https://www.python.org](https://www.python.org)). Versions installed via pacman in MSYS2 will not work. #### 2.2. Bootstrap Bazel on Windows -1. Open the MSYS2 shell. +1. Open the MSYS2 shell. + +2. Set the following environment variables: + + - Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the path to the Visual Studio directory (BAZEL\_V**S**) or to the Visual C++ directory (BAZEL\_V**C**). Setting one of them is enough. + + - `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the examples below. + + Do not set this to `C:\Windows\System32\bash.exe`. (You have that file if you installed Windows Subsystem for Linux.) Bazel does not support this version of `bash.exe`. -2. Set the following environment variables: - * Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the - path to the Visual Studio directory (BAZEL\_VS) or to the Visual - C++ directory (BAZEL\_VC). Setting one of them is enough. - * `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the - examples below. + - `PATH`: Add the Python directory. - Do not set this to `C:\Windows\System32\bash.exe`. (You have that file - if you installed Windows Subsystem for Linux.) Bazel does not support - this version of `bash.exe`. - * `PATH`: Add the Python directory. - * `JAVA_HOME`: Set to the JDK directory. + - `JAVA_HOME`: Set to the JDK directory. - **Example** (using BAZEL\_VS): + **Example** (using BAZEL\_V**S**): - export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` + export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` - or (using BAZEL\_VC): + or (using BAZEL\_V**C**): - export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` + export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" + ``` -3. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` +4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` -The compiled output is placed into `output/bazel.exe`. This is a self-contained -Bazel binary, without an embedded JDK. You can copy it anywhere or use it -in-place. For convenience, copy this binary to a directory that's on -your `PATH`. +The compiled output is placed into `output/bazel.exe`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH`. -To build the `bazel.exe` binary in a reproducible way, also set -[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) -in the "Run the compilation script" step. +To build the `bazel.exe` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. -You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the -Command Prompt (`cmd.exe`) or PowerShell. +You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the Command Prompt (`cmd.exe`) or PowerShell. diff --git a/install/completion.mdx b/install/completion.mdx index 1d1d1b76..309ce3f8 100644 --- a/install/completion.mdx +++ b/install/completion.mdx @@ -2,11 +2,7 @@ title: 'Command-Line Completion' --- - - -You can enable command-line completion (also known as tab-completion) in Bash -and Zsh. This lets you tab-complete command names, flags names and flag values, -and target names. +You can enable command-line completion (also known as tab-completion) in Bash and Zsh. This lets you tab-complete command names, flags names and flag values, and target names. ## Bash @@ -14,55 +10,51 @@ Bazel comes with a Bash completion script. If you installed Bazel: -* From the APT repository, then you're done -- the Bash completion script is - already installed in `/etc/bash_completion.d`. +- From the APT repository, then you're done -- the Bash completion script is already installed in `/etc/bash_completion.d`. + +- From Homebrew, then you're done -- the Bash completion script is already installed in `$(brew --prefix)/etc/bash_completion.d`. + +- From the installer downloaded from GitHub, then: + + 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. + + 2. Do one of the following: + + - Either copy this file to your completion directory (if you have one). + + Example: on Ubuntu this is the `/etc/bash_completion.d` directory. + + - Or source the completion file from Bash's RC file. -* From Homebrew, then you're done -- the Bash completion script is - already installed in `$(brew --prefix)/etc/bash_completion.d`. + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: -* From the installer downloaded from GitHub, then: - 1. Locate the absolute path of the completion file. The installer copied it - to the `bin` directory. + ``` + source /path/to/bazel-complete.bash + ``` - Example: if you ran the installer with `--user`, this will be - `$HOME/.bazel/bin`. If you ran the installer as root, this will be - `/usr/local/lib/bazel/bin`. - 2. Do one of the following: - * Either copy this file to your completion directory (if you have - one). +- Via [bootstrapping](/install/compile-source), then: - Example: on Ubuntu this is the `/etc/bash_completion.d` directory. - * Or source the completion file from Bash's RC file. + 1. Emit the completion script into a file: - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) - or `~/.bash_profile` (on macOS), using the path to your completion - file's absolute path: + ``` + bazel help completion bash > bazel-complete.bash + ``` - ``` - source /path/to/bazel-complete.bash - ``` + 2. Do one of the following: -* Via [bootstrapping](/install/compile-source), then: - 1. Emit the completion script into a file: + - Copy this file to your completion directory, if you have one. - ``` - bazel help completion bash > bazel-complete.bash - ``` - 2. Do one of the following: - * Copy this file to your completion directory, if you have - one. + Example: on Ubuntu this is the `/etc/bash_completion.d` directory - Example: on Ubuntu this is the `/etc/bash_completion.d` directory - * Copy it somewhere on your local disk, such as to `$HOME`, and - source the completion file from Bash's RC file. + - Copy it somewhere on your local disk, such as to `$HOME`, and source the completion file from Bash's RC file. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) - or `~/.bash_profile` (on macOS), using the path to your completion - file's absolute path: + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: - ``` - source /path/to/bazel-complete.bash - ``` + ``` + source /path/to/bazel-complete.bash + ``` ## Zsh @@ -70,57 +62,48 @@ Bazel comes with a Zsh completion script. If you installed Bazel: -* From the APT repository, then you're done -- the Zsh completion script is - already installed in `/usr/share/zsh/vendor-completions`. - - > If you have a heavily customized `.zshrc` and the autocomplete - > does not function, try one of the following solutions: - > - > Add the following to your `.zshrc`: - > - > ``` - > zstyle :compinstall filename '/home/tradical/.zshrc' - > - > autoload -Uz compinit - > compinit - > ``` - > - > or - > - > Follow the instructions - > [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) - > - > If you are using `oh-my-zsh`, you may want to install and enable - > the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the - > solutions described above. - -* From Homebrew, then you're done -- the Zsh completion script is - already installed in `$(brew --prefix)/share/zsh/site-functions`. - -* From the installer downloaded from GitHub, then: - 1. Locate the absolute path of the completion file. The installer copied it - to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be - `$HOME/.bazel/bin`. If you ran the installer as root, this will be - `/usr/local/lib/bazel/bin`. - - 2. Add this script to a directory on your `$fpath`: - - ``` - fpath[1,0]=~/.zsh/completion/ - mkdir -p ~/.zsh/completion/ - cp /path/from/above/step/_bazel ~/.zsh/completion - ``` - - You may have to call `rm -f ~/.zcompdump; compinit` - the first time to make it work. - - 3. Optionally, add the following to your .zshrc. - - ``` - # This way the completion script does not have to parse Bazel's options - # repeatedly. The directory in cache-path must be created manually. - zstyle ':completion:*' use-cache on - zstyle ':completion:*' cache-path ~/.zsh/cache - ``` +- From the APT repository, then you're done -- the Zsh completion script is already installed in `/usr/share/zsh/vendor-completions`. + + > If you have a heavily customized `.zshrc` and the autocomplete does not function, try one of the following solutions: + > + > Add the following to your `.zshrc`: + > + > ``` + > zstyle :compinstall filename '/home/tradical/.zshrc' + > + > autoload -Uz compinit + > compinit + > ``` + > + > or + > + > Follow the instructions [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) + > + > If you are using `oh-my-zsh`, you may want to install and enable the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the solutions described above. + +- From Homebrew, then you're done -- the Zsh completion script is already installed in `$(brew --prefix)/share/zsh/site-functions`. + +- From the installer downloaded from GitHub, then: + + 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. + + 2. Add this script to a directory on your `$fpath`: + + ``` + fpath[1,0]=~/.zsh/completion/ + mkdir -p ~/.zsh/completion/ + cp /path/from/above/step/_bazel ~/.zsh/completion + ``` + + You may have to call `rm -f ~/.zcompdump; compinit` the first time to make it work. + + 3. Optionally, add the following to your .zshrc. + + ``` + # This way the completion script does not have to parse Bazel's options + # repeatedly. The directory in cache-path must be created manually. + zstyle ':completion:*' use-cache on + zstyle ':completion:*' cache-path ~/.zsh/cache + ``` diff --git a/install/docker-container.mdx b/install/docker-container.mdx index b1efb76c..2e97dc2b 100644 --- a/install/docker-container.mdx +++ b/install/docker-container.mdx @@ -2,19 +2,11 @@ title: 'Getting Started with Bazel Docker Container' --- - - -This page provides details on the contents of the Bazel container, how to build -the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel -inside the Bazel container, and how to build this project directly -from the host machine using the Bazel container with directory mounting. +This page provides details on the contents of the Bazel container, how to build the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel inside the Bazel container, and how to build this project directly from the host machine using the Bazel container with directory mounting. ## Build Abseil project from your host machine with directory mounting -The instructions in this section allow you to build using the Bazel container -with the sources checked out in your host environment. A container is started up -for each build command you execute. Build results are cached in your host -environment so they can be reused across builds. +The instructions in this section allow you to build using the Bazel container with the sources checked out in your host environment. A container is started up for each build command you execute. Build results are cached in your host environment so they can be reused across builds. Clone the project to a directory in your host machine. @@ -28,8 +20,7 @@ Create a folder that will have cached results to be shared across builds. mkdir -p /tmp/build_output/ ``` -Use the Bazel container to build the project and make the build -outputs available in the output folder in your host machine. +Use the Bazel container to build the project and make the build outputs available in the output folder in your host machine. ```posix-terminal docker run \ @@ -43,9 +34,7 @@ docker run \ build //absl/... ``` -Build the project with sanitizers by adding the `--config=asan|tsan|msan` build -flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or -MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. ```posix-terminal docker run \ @@ -61,10 +50,7 @@ docker run \ ## Build Abseil project from inside the container -The instructions in this section allow you to build using the Bazel container -with the sources inside the container. By starting a container at the beginning -of your development workflow and doing changes in the worskpace within the -container, build results will be cached. +The instructions in this section allow you to build using the Bazel container with the sources inside the container. By starting a container at the beginning of your development workflow and doing changes in the worskpace within the container, build results will be cached. Start a shell in the Bazel container: @@ -86,9 +72,7 @@ Do a regular build. ubuntu@5a99103747c6:~/abseil-cpp$ bazel build //absl/... ``` -Build the project with sanitizers by adding the `--config=asan|tsan|msan` -build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or -MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. ```posix-terminal ubuntu@5a99103747c6:~/abseil-cpp$ bazel build --config={asan | tsan | msan} -- //absl/... -//absl/types:variant_test diff --git a/install/ide.mdx b/install/ide.mdx index aa6210b3..5c6252da 100644 --- a/install/ide.mdx +++ b/install/ide.mdx @@ -2,56 +2,39 @@ title: 'Integrating Bazel with IDEs' --- +This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android Studio, and CLion (or build your own IDE plugin). It also includes links to installation and plugin details. +IDEs integrate with Bazel in a variety of ways, from features that allow Bazel executions from within the IDE, to awareness of Bazel structures such as syntax highlighting of the `BUILD` files. -This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android -Studio, and CLion (or build your own IDE plugin). It also includes links to -installation and plugin details. - -IDEs integrate with Bazel in a variety of ways, from features that allow Bazel -executions from within the IDE, to awareness of Bazel structures such as syntax -highlighting of the `BUILD` files. - -If you are interested in developing an editor or IDE plugin for Bazel, please -join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start -a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). +If you are interested in developing an editor or IDE plugin for Bazel, please join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). ## IDEs and editors ### IntelliJ, Android Studio, and CLion -[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and -CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). +[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). This is the open source version of the plugin used internally at Google. Features: -* Interop with language-specific plugins. Supported languages include Java, - Scala, and Python. -* Import `BUILD` files into the IDE with semantic awareness of Bazel targets. -* Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and - `.bzl`files -* Build, test, and execute binaries directly from the IDE -* Create configurations for debugging and running binaries. +- Interop with language-specific plugins. Supported languages include Java, Scala, and Python. +- Import `BUILD` files into the IDE with semantic awareness of Bazel targets. +- Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and `.bzl`files +- Build, test, and execute binaries directly from the IDE +- Create configurations for debugging and running binaries. To install, go to the IDE's plugin browser and search for `Bazel`. -To manually install older versions, download the zip files from JetBrains' -Plugin Repository and install the zip file from the IDE's plugin browser: +To manually install older versions, download the zip files from JetBrains' Plugin Repository and install the zip file from the IDE's plugin browser: -* [Android Studio - plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) -* [IntelliJ - plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) -* [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) +- [Android Studio plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) +- [IntelliJ plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) +- [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) ### Xcode -[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), -[Tulsi](https://tulsi.bazel.build), and -[XCHammer](https://github.com/pinterest/xchammer) generate Xcode -projects from Bazel `BUILD` files. +[rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), [Tulsi](https://tulsi.bazel.build), and [XCHammer](https://github.com/pinterest/xchammer) generate Xcode projects from Bazel `BUILD` files. ### Visual Studio Code @@ -59,21 +42,16 @@ Official plugin for VS Code. Features: -* Bazel Build Targets tree -* Starlark debugger for `.bzl` files during a build (set breakpoints, step - through code, inspect variables, and so on) +- Bazel Build Targets tree +- Starlark debugger for `.bzl` files during a build (set breakpoints, step through code, inspect variables, and so on) -Find [the plugin on the Visual Studio -marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) -or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). -The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). +Find [the plugin on the Visual Studio marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Atom -Find the [`language-bazel` package](https://atom.io/packages/language-bazel) -on the Atom package manager. +Find the [`language-bazel` package](https://atom.io/packages/language-bazel) on the Atom package manager. See also: [Autocomplete for Source Code](#autocomplete-for-source-code) @@ -85,31 +63,23 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Emacs -See [`bazelbuild/bazel-emacs-mode` on -GitHub](https://github.com/bazelbuild/emacs-bazel-mode) +See [`bazelbuild/bazel-emacs-mode` on GitHub](https://github.com/bazelbuild/emacs-bazel-mode) See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Visual Studio -[Lavender](https://github.com/tmandry/lavender) is an experimental project for -generating Visual Studio projects that use Bazel for building. +[Lavender](https://github.com/tmandry/lavender) is an experimental project for generating Visual Studio projects that use Bazel for building. ### Eclipse -[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) -is a set of plugins for importing Bazel packages into an Eclipse workspace as -Eclipse projects. +[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) is a set of plugins for importing Bazel packages into an Eclipse workspace as Eclipse projects. ## Autocomplete for Source Code ### C Language Family (C++, C, Objective-C, Objective-C++, and CUDA) -[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) -run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. -The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to -provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and -consumes the Protobuf output of Bazel to extract the compile commands. +[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and consumes the Protobuf output of Bazel to extract the compile commands. [`hedronvision/bazel-compile-commands-extractor`](https://github.com/hedronvision/bazel-compile-commands-extractor) enables autocomplete, smart navigation, quick fixes, and more in a wide variety of extensible editors, including VSCode, Vim, Emacs, Atom, and Sublime. It lets language servers, like clangd and ccls, and other types of tooling, draw upon Bazel's understanding of how `cc` and `objc` code will be compiled, including how it configures cross-compilation for other platforms. @@ -119,11 +89,8 @@ consumes the Protobuf output of Bazel to extract the compile commands. ## Automatically run build and test on file change -[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a -tool for building Bazel targets when source files change. +[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a tool for building Bazel targets when source files change. ## Building your own IDE plugin -Read the [**IDE support** blog -post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about -the Bazel APIs to use when building an IDE plugin. +Read the [**IDE support** blog post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about the Bazel APIs to use when building an IDE plugin. diff --git a/install/index.mdx b/install/index.mdx index 29b46972..898cd417 100644 --- a/install/index.mdx +++ b/install/index.mdx @@ -2,10 +2,7 @@ title: 'Installing Bazel' --- - - -This page describes the various platforms supported by Bazel and links -to the packages for more details. +This page describes the various platforms supported by Bazel and links to the packages for more details. [Bazelisk](/install/bazelisk) is the recommended way to install Bazel on [Ubuntu Linux](/install/ubuntu), [macOS](/install/os-x), and [Windows](/install/windows). @@ -13,24 +10,21 @@ You can find available Bazel releases on our [release page](/release). ## Community-supported packages -Bazel community members maintain these packages. The Bazel team doesn't -officially support them. Contact the package maintainers for support. +Bazel community members maintain these packages. The Bazel team doesn't officially support them. Contact the package maintainers for support. -* [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*&branch=edge&repo=&arch=&origin=&flagged=&maintainer=) -* [Arch Linux][arch] -* [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) -* [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) -* [FreeBSD](https://www.freshports.org/devel/bazel) -* [Homebrew](https://formulae.brew.sh/formula/bazel) -* [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) -* [openSUSE](/install/suse) -* [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) -* [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) +- [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*\&branch=edge\&repo=\&arch=\&origin=\&flagged=\&maintainer=) +- [Arch Linux](https://archlinux.org/packages/extra/x86_64/bazel/) +- [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) +- [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) +- [FreeBSD](https://www.freshports.org/devel/bazel) +- [Homebrew](https://formulae.brew.sh/formula/bazel) +- [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) +- [openSUSE](/install/suse) +- [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) +- [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) ## Community-supported architectures -* [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) +- [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) For other platforms, you can try to [compile from source](/install/compile-source). - -[arch]: https://archlinux.org/packages/extra/x86_64/bazel/ diff --git a/install/os-x.mdx b/install/os-x.mdx index b8d5bca2..eb8c34e1 100644 --- a/install/os-x.mdx +++ b/install/os-x.mdx @@ -2,23 +2,21 @@ title: 'Installing Bazel on macOS' --- - - This page describes how to install Bazel on macOS and set up your environment. You can install Bazel on macOS using one of the following methods: -* *Recommended*: [Use Bazelisk](/install/bazelisk) -* [Use Homebrew](#install-on-mac-os-x-homebrew) -* [Use the binary installer](#install-with-installer-mac-os-x) -* [Compile Bazel from source](/install/compile-source) +- *Recommended*: [Use Bazelisk](/install/bazelisk) +- [Use Homebrew](#install-on-mac-os-x-homebrew) +- [Use the binary installer](#install-with-installer-mac-os-x) +- [Compile Bazel from source](/install/compile-source) Bazel comes with two completion scripts. After installing Bazel, you can: -* Access the [bash completion script](/install/completion#bash) -* Install the [zsh completion script](/install/completion#zsh) +- Access the [bash completion script](/install/completion#bash) +- Install the [zsh completion script](/install/completion#zsh) -

Installing using Homebrew

+## Installing using Homebrew ### Step 1: Install Homebrew on macOS @@ -36,46 +34,37 @@ Install the Bazel package via Homebrew as follows: brew install bazel ``` -All set! You can confirm Bazel is installed successfully by running the -following command: +All set! You can confirm Bazel is installed successfully by running the following command: ```posix-terminal bazel --version ``` -Once installed, you can upgrade to a newer version of Bazel using the -following command: +Once installed, you can upgrade to a newer version of Bazel using the following command: ```posix-terminal brew upgrade bazel ``` -

Installing using the binary installer

+## Installing using the binary installer -The binary installers are on Bazel's -[GitHub releases page](https://github.com/bazelbuild/bazel/releases). +The binary installers are on Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary. Some additional libraries -must also be installed for Bazel to work. +The installer contains the Bazel binary. Some additional libraries must also be installed for Bazel to work. ### Step 1: Install Xcode command line tools -If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode -command line tools package by using `xcode-select`: +If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode command line tools package by using `xcode-select`: ```posix-terminal xcode-select --install ``` -Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS -SDK 8.1 installed on your system. +Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS SDK 8.1 installed on your system. -Download Xcode from the -[App Store](https://apps.apple.com/us/app/xcode/id497799835) or the -[Apple Developer site](https://developer.apple.com/download/more/?=xcode). +Download Xcode from the [App Store](https://apps.apple.com/us/app/xcode/id497799835) or the [Apple Developer site](https://developer.apple.com/download/more/?=xcode). -Once Xcode is installed, accept the license agreement for all users with the -following command: +Once Xcode is installed, accept the license agreement for all users with the following command: ```posix-terminal sudo xcodebuild -license accept @@ -83,59 +72,46 @@ sudo xcodebuild -license accept ### Step 2: Download the Bazel installer -Next, download the Bazel binary installer named -`bazel--installer-darwin-x86_64.sh` from the -[Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-<version>-installer-darwin-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). -**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, -you need to download the installer from the terminal using `curl`, replacing -the version variable with the Bazel version you want to download: +**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, you need to download the installer from the terminal using `curl`, replacing the version variable with the Bazel version you want to download: ```posix-terminal export BAZEL_VERSION=5.2.0 -curl -fLO "https://github.com/bazelbuild/bazel/releases/download/{{ '' }}$BAZEL_VERSION{{ '' }}/bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" +curl -fLO "https://github.com/bazelbuild/bazel/releases/download/<var>$BAZEL_VERSION</var>/bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" ``` -This is a temporary workaround until the macOS release flow supports -signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). +This is a temporary workaround until the macOS release flow supports signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). ### Step 3: Run the installer Run the Bazel installer as follows: ```posix-terminal -chmod +x "bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" +chmod +x "bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" -./bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh --user +./bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and -sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see -additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. -If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that _**“bazel-real” cannot be -opened because the developer cannot be verified**_, you need to re-download -the installer from the terminal using `curl` as a workaround; see Step 2 above. +If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that ***“bazel-real” cannot be opened because the developer cannot be verified***, you need to re-download the installer from the terminal using `curl` as a workaround; see Step 2 above. ### Step 4: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel -executable is installed in your `HOME/bin` directory. -It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `<var>HOME</var>/bin` directory. It's a good idea to add this directory to your default paths, as follows: ```posix-terminal -export PATH="{{ '' }}PATH{{ '' }}:{{ '' }}HOME{{ '' }}/bin" +export PATH="<var>PATH</var>:<var>HOME</var>/bin" ``` -You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` -file. +You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` file. -All set! You can confirm Bazel is installed successfully by running the -following command: +All set! You can confirm Bazel is installed successfully by running the following command: ```posix-terminal bazel --version ``` -To update to a newer release of Bazel, download and install the desired version. +To update to a newer release of Bazel, download and install the desired version. diff --git a/install/suse.mdx b/install/suse.mdx index 741b108f..73debc99 100644 --- a/install/suse.mdx +++ b/install/suse.mdx @@ -2,23 +2,17 @@ title: 'Installing Bazel on openSUSE Tumbleweed & Leap' --- - - This page describes how to install Bazel on openSUSE Tumbleweed and Leap. -`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues -using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). +`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). -Packages are provided for openSUSE Tumbleweed and Leap. You can find all -available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93&baseproject=ALL&q=bazel). +Packages are provided for openSUSE Tumbleweed and Leap. You can find all available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93\&baseproject=ALL\&q=bazel). The commands below must be run either via `sudo` or while logged in as `root`. ## Installing Bazel on openSUSE -Run the following commands to install the package. If you need a specific -version, you can install it via the specific `bazelXXX` package, otherwise, -just `bazel` is enough: +Run the following commands to install the package. If you need a specific version, you can install it via the specific `bazelXXX` package, otherwise, just `bazel` is enough: To install the latest version of Bazel, run: @@ -26,9 +20,7 @@ To install the latest version of Bazel, run: zypper install bazel ``` -You can also install a specific version of Bazel by specifying the package -version with `bazelversion`. For example, to install -Bazel 4.2, run: +You can also install a specific version of Bazel by specifying the package version with `bazel<var>version</var>`. For example, to install Bazel 4.2, run: ```posix-terminal zypper install bazel4.2 diff --git a/install/ubuntu.mdx b/install/ubuntu.mdx index 73d42751..b498f42f 100644 --- a/install/ubuntu.mdx +++ b/install/ubuntu.mdx @@ -2,37 +2,30 @@ title: 'Installing Bazel on Ubuntu' --- - - -This page describes the options for installing Bazel on Ubuntu. -It also provides links to the Bazel completion scripts and the binary installer, -if needed as a backup option (for example, if you don't have admin access). +This page describes the options for installing Bazel on Ubuntu. It also provides links to the Bazel completion scripts and the binary installer, if needed as a backup option (for example, if you don't have admin access). Supported Ubuntu Linux platforms: -* 22.04 (LTS) -* 20.04 (LTS) -* 18.04 (LTS) +- 22.04 (LTS) +- 20.04 (LTS) +- 18.04 (LTS) -Bazel should be compatible with other Ubuntu releases and Debian -"stretch" and above, but is untested and not guaranteed to work. +Bazel should be compatible with other Ubuntu releases and Debian "stretch" and above, but is untested and not guaranteed to work. Install Bazel on Ubuntu using one of the following methods: -* *Recommended*: [Use Bazelisk](/install/bazelisk) -* [Use our custom APT repository](#install-on-ubuntu) -* [Use the binary installer](#binary-installer) -* [Use the Bazel Docker container](#docker-container) -* [Compile Bazel from source](/install/compile-source) +- *Recommended*: [Use Bazelisk](/install/bazelisk) +- [Use our custom APT repository](#install-on-ubuntu) +- [Use the binary installer](#binary-installer) +- [Use the Bazel Docker container](#docker-container) +- [Compile Bazel from source](/install/compile-source) -**Note:** For Arm-based systems, the APT repository does not contain an `arm64` -release, and there is no binary installer available. Either use Bazelisk or -compile from source. +**Note:** For Arm-based systems, the APT repository does not contain an `arm64` release, and there is no binary installer available. Either use Bazelisk or compile from source. Bazel comes with two completion scripts. After installing Bazel, you can: -* Access the [bash completion script](/install/completion#bash) -* Install the [zsh completion script](/install/completion#zsh) +- Access the [bash completion script](/install/completion#bash) +- Install the [zsh completion script](/install/completion#zsh) ## Using Bazel's apt repository @@ -43,16 +36,14 @@ Bazel comes with two completion scripts. After installing Bazel, you can: ```posix-terminal sudo apt install apt-transport-https curl gnupg -y -curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg +curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg sudo mv bazel-archive-keyring.gpg /usr/share/keyrings echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list ``` -The component name "jdk1.8" is kept only for legacy reasons and doesn't relate -to supported or included JDK versions. Bazel releases are Java-version agnostic. -Changing the "jdk1.8" component name would break existing users of the repo. +The component name "jdk1.8" is kept only for legacy reasons and doesn't relate to supported or included JDK versions. Bazel releases are Java-version agnostic. Changing the "jdk1.8" component name would break existing users of the repo. ### Step 2: Install and update Bazel @@ -66,18 +57,13 @@ Once installed, you can upgrade to a newer version of Bazel as part of your norm sudo apt update && sudo apt full-upgrade ``` -The `bazel` package always installs the latest stable version of Bazel. You -can install specific, older versions of Bazel in addition to the latest one, -such as this: +The `bazel` package always installs the latest stable version of Bazel. You can install specific, older versions of Bazel in addition to the latest one, such as this: ```posix-terminal sudo apt install bazel-1.0.0 ``` -This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This -can be useful if you need a specific version of Bazel to build a project, for -example because it uses a `.bazelversion` file to explicitly state with which -Bazel version it should be built. +This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This can be useful if you need a specific version of Bazel to build a project, for example because it uses a `.bazelversion` file to explicitly state with which Bazel version it should be built. Optionally, you can set `bazel` to a specific version by creating a symlink: @@ -89,8 +75,7 @@ bazel --version # 1.0.0 ### Step 3: Install a JDK (optional) -Bazel includes a private, bundled JRE as its runtime and doesn't require you to -install any specific version of Java. +Bazel includes a private, bundled JRE as its runtime and doesn't require you to install any specific version of Java. However, if you want to build Java code using Bazel, you have to install a JDK. @@ -100,14 +85,11 @@ sudo apt install default-jdk ## Using the binary installer -Generally, you should use the apt repository, but the binary installer -can be useful if you don't have admin permissions on your machine or -can't add custom repositories. +Generally, you should use the apt repository, but the binary installer can be useful if you don't have admin permissions on your machine or can't add custom repositories. The binary installers can be downloaded from Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary and extracts it into your `$HOME/bin` -folder. Some additional libraries must be installed manually for Bazel to work. +The installer contains the Bazel binary and extracts it into your `$HOME/bin` folder. Some additional libraries must be installed manually for Bazel to work. ### Step 1: Install required packages @@ -125,42 +107,34 @@ sudo apt-get install default-jdk ### Step 2: Run the installer -Next, download the Bazel binary installer named `bazel-version-installer-linux-x86_64.sh` -from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-<var>version</var>-installer-linux-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). Run it as follows: ```posix-terminal -chmod +x bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh +chmod +x bazel-<var>version</var>-installer-linux-x86_64.sh -./bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh --user +./bazel-<var>version</var>-installer-linux-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and -sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see -additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. ### Step 3: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel -executable is installed in your `$HOME/bin` directory. -It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `$HOME/bin` directory. It's a good idea to add this directory to your default paths, as follows: ```posix-terminal export PATH="$PATH:$HOME/bin" ``` -You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it -permanent. +You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it permanent. ## Using the Bazel Docker container -We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. -You can use the Docker container as follows: +We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. You can use the Docker container as follows: ``` -$ docker pull gcr.io/bazel-public/bazel: +$ docker pull gcr.io/bazel-public/bazel:<bazel version> ``` The Docker container is built by [these steps](https://github.com/bazelbuild/continuous-integration/tree/master/bazel/oci). - diff --git a/install/windows.mdx b/install/windows.mdx index b38c0986..3572118b 100644 --- a/install/windows.mdx +++ b/install/windows.mdx @@ -2,16 +2,11 @@ title: 'Installing Bazel on Windows' --- - - -This page describes the requirements and steps to install Bazel on Windows. -It also includes troubleshooting and other ways to install Bazel, such as -using Chocolatey or Scoop. +This page describes the requirements and steps to install Bazel on Windows. It also includes troubleshooting and other ways to install Bazel, such as using Chocolatey or Scoop. ## Installing Bazel -This section covers the prerequisites, environment setup, and detailed -steps during installation on Windows. +This section covers the prerequisites, environment setup, and detailed steps during installation on Windows. ### Check your system @@ -19,33 +14,31 @@ Recommended: 64 bit Windows 10, version 1703 (Creators Update) or newer To check your Windows version: -* Click the Start button. -* Type `winver` in the search box and press Enter. -* You should see the About Windows box with your Windows version information. +- Click the Start button. +- Type `winver` in the search box and press Enter. +- You should see the About Windows box with your Windows version information. ### Install the prerequisites -* [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) +- [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) ### Download Bazel *Recommended*: [Use Bazelisk](/install/bazelisk) - Alternatively you can: -* [Download the Bazel binary (`bazel-version-windows-x86_64.exe`) from - GitHub](https://github.com/bazelbuild/bazel/releases). -* [Install Bazel from Chocolatey](#chocolately) -* [Install Bazel from Scoop](#scoop) -* [Build Bazel from source](/install/compile-source) +- [Download the Bazel binary (`bazel-<var>version</var>-windows-x86_64.exe`) from GitHub](https://github.com/bazelbuild/bazel/releases). +- [Install Bazel from Chocolatey](#chocolately) +- [Install Bazel from Scoop](#scoop) +- [Build Bazel from source](/install/compile-source) ### Set up your environment To make Bazel easily accessible from command prompts or PowerShell by default, you can rename the Bazel binary to `bazel.exe` and add it to your default paths. ```posix-terminal -set PATH=%PATH%;{{ '' }}path to the Bazel binary{{ '' }} +set PATH=%PATH%;<var>path to the Bazel binary</var> ``` You can also change your system `PATH` environment variable to make it permanent. Check out how to [set environment variables](/configure/windows#set-environment-variables). @@ -57,64 +50,57 @@ You can also change your system `PATH` environment variable to make it permanent To check the installation is correct, try to run: ```posix-terminal -bazel {{ '' }}version{{ '' }} +bazel <var>version</var> ``` Next, you can check out more tips and guidance here: -* [Installing compilers and language runtimes](#install-compilers) -* [Troubleshooting](#troubleshooting) -* [Best practices on Windows](/configure/windows#best-practices) -* [Tutorials](/start/#tutorials) +- [Installing compilers and language runtimes](#install-compilers) +- [Troubleshooting](#troubleshooting) +- [Best practices on Windows](/configure/windows#best-practices) +- [Tutorials](/start/#tutorials) ## Installing compilers and language runtimes Depending on which languages you want to build, you will need: -* [MSYS2 x86_64](https://www.msys2.org/) +- [MSYS2 x86\_64](https://www.msys2.org/) - MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix - tools (like `grep`, `tar`, `git`). + MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix tools (like `grep`, `tar`, `git`). - You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are - `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an - error if a build target needs Bash but Bazel could not locate it. + You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an error if a build target needs Bash but Bazel could not locate it. -* Common MSYS2 packages +- Common MSYS2 packages - You will likely need these to build and run targets that depend on Bash. MSYS2 does not install - these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). + You will likely need these to build and run targets that depend on Bash. MSYS2 does not install these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). - Open the MSYS2 terminal and run this command: + Open the MSYS2 terminal and run this command: - ```posix-terminal - pacman -S zip unzip patch diffutils git - ``` + ```posix-terminal + pacman -S zip unzip patch diffutils git + ``` - Optional: If you want to use Bazel from CMD or Powershell and still be able - to use Bash tools, make sure to add - `MSYS2_INSTALL_PATH/usr/bin` to your - `PATH` environment variable. + Optional: If you want to use Bazel from CMD or Powershell and still be able to use Bash tools, make sure to add `<var>MSYS2_INSTALL_PATH</var>/usr/bin` to your `PATH` environment variable. -* [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) +- [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) - You will need this to build C++ code on Windows. + You will need this to build C++ code on Windows. - Also supported: + Also supported: - * Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK + - Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK -* [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) +- [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) - You will need this to build Java code on Windows. + You will need this to build Java code on Windows. - Also supported: Java 8, 9, and 10 + Also supported: Java 8, 9, and 10 -* [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) +- [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) - You will need this to build Python code on Windows. + You will need this to build Python code on Windows. - Also supported: Python 2.7 or newer for Windows x86-64 + Also supported: Python 2.7 or newer for Windows x86-64 ## Troubleshooting @@ -122,11 +108,11 @@ Depending on which languages you want to build, you will need: **Possible reasons**: -* you installed MSYS2 not under the default install path +- you installed MSYS2 not under the default install path -* you installed MSYS2 i686 instead of MSYS2 x86\_64 +- you installed MSYS2 i686 instead of MSYS2 x86\_64 -* you installed MSYS instead of MSYS2 +- you installed MSYS instead of MSYS2 **Solution**: @@ -134,104 +120,95 @@ Ensure you installed MSYS2 x86\_64. If that doesn't help: -1. Go to Start Menu > Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for <username>"), and click the "New..." - button below it. +3. Look at the list on the top ("User variables for \"), and click the "New\..." button below it. -4. For "Variable name", enter `BAZEL_SH` +4. For "Variable name", enter `BAZEL_SH` -5. Click "Browse File..." +5. Click "Browse File..." -6. Navigate to the MSYS2 directory, then to `usr\bin` below it. +6. Navigate to the MSYS2 directory, then to `usr\bin` below it. - For example, this might be `C:\msys64\usr\bin` on your system. + For example, this might be `C:\msys64\usr\bin` on your system. -7. Select the `bash.exe` or `bash` file and click OK +7. Select the `bash.exe` or `bash` file and click OK -8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. +8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. ### Bazel does not find Visual Studio or Visual C++ **Possible reasons**: -* you installed multiple versions of Visual Studio +- you installed multiple versions of Visual Studio -* you installed and removed various versions of Visual Studio +- you installed and removed various versions of Visual Studio -* you installed various versions of the Windows SDK +- you installed various versions of the Windows SDK -* you installed Visual Studio not under the default install path +- you installed Visual Studio not under the default install path **Solution**: -1. Go to Start Menu > Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for <username>"), and click the "New..." - button below it. +3. Look at the list on the top ("User variables for \"), and click the "New\..." button below it. -4. For "Variable name", enter `BAZEL_VC` +4. For "Variable name", enter `BAZEL_VC` -5. Click "Browse Directory..." +5. Click "Browse Directory..." -6. Navigate to the `VC` directory of Visual Studio. +6. Navigate to the `VC` directory of Visual Studio. - For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` - on your system. + For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` on your system. -7. Select the `VC` folder and click OK +7. Select the `VC` folder and click OK -8. The "Variable value" field now has the path to `VC`. Click OK to close the window. +8. The "Variable value" field now has the path to `VC`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. ## Other ways to install Bazel ### Using Chocolatey -1. Install the [Chocolatey](https://chocolatey.org) package manager +1. Install the [Chocolatey](https://chocolatey.org) package manager -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - choco install bazel - ``` + ```posix-terminal + choco install bazel + ``` - This command will install the latest available version of Bazel and - its dependencies, such as the MSYS2 shell. This will not install Visual C++ - though. + This command will install the latest available version of Bazel and its dependencies, such as the MSYS2 shell. This will not install Visual C++ though. -See [Chocolatey installation and package maintenance -guide](/contribute/windows-chocolatey-maintenance) for more -information about the Chocolatey package. +See [Chocolatey installation and package maintenance guide](/contribute/windows-chocolatey-maintenance) for more information about the Chocolatey package. ### Using Scoop -1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: +1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: - ```posix-terminal - iex (new-object net.webclient).downloadstring('https://get.scoop.sh') - ``` + ```posix-terminal + iex (new-object net.webclient).downloadstring('https://get.scoop.sh') + ``` -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - scoop install bazel - ``` + ```posix-terminal + scoop install bazel + ``` -See [Scoop installation and package maintenance -guide](/contribute/windows-scoop-maintenance) for more -information about the Scoop package. +See [Scoop installation and package maintenance guide](/contribute/windows-scoop-maintenance) for more information about the Scoop package. ### Build from source diff --git a/migrate/index.mdx b/migrate/index.mdx index 2a574067..8683aedb 100644 --- a/migrate/index.mdx +++ b/migrate/index.mdx @@ -2,9 +2,7 @@ title: 'Migrating to Bazel' --- - - This page links to migration guides for Bazel. -* [Maven](/migrate/maven) -* [Xcode](/migrate/xcode) +- [Maven](/migrate/maven) +- [Xcode](/migrate/xcode) diff --git a/migrate/maven.mdx b/migrate/maven.mdx index 38aaffc0..0673ef53 100644 --- a/migrate/maven.mdx +++ b/migrate/maven.mdx @@ -2,55 +2,34 @@ title: 'Migrating from Maven to Bazel' --- +This page describes how to migrate from Maven to Bazel, including the prerequisites and installation steps. It describes the differences between Maven and Bazel, and provides a migration example using the Guava project. +When migrating from any build tool to Bazel, it's best to have both build tools running in parallel until you have fully migrated your development team, CI system, and any other relevant systems. You can run Maven and Bazel in the same repository. -This page describes how to migrate from Maven to Bazel, including the -prerequisites and installation steps. It describes the differences between Maven -and Bazel, and provides a migration example using the Guava project. - -When migrating from any build tool to Bazel, it's best to have both build tools -running in parallel until you have fully migrated your development team, CI -system, and any other relevant systems. You can run Maven and Bazel in the same -repository. - -Note: While Bazel supports downloading and publishing Maven artifacts with -[rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) -, it does not directly support Maven-based plugins. Maven plugins can't be -directly run by Bazel since there's no Maven compatibility layer. +Note: While Bazel supports downloading and publishing Maven artifacts with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) , it does not directly support Maven-based plugins. Maven plugins can't be directly run by Bazel since there's no Maven compatibility layer. ## Before you begin -* [Install Bazel](/install) if it's not yet installed. -* If you're new to Bazel, go through the tutorial [Introduction to Bazel: - Build Java](/start/java) before you start migrating. The tutorial explains - Bazel's concepts, structure, and label syntax. +- [Install Bazel](/install) if it's not yet installed. +- If you're new to Bazel, go through the tutorial [Introduction to Bazel: Build Java](/start/java) before you start migrating. The tutorial explains Bazel's concepts, structure, and label syntax. ## Differences between Maven and Bazel -* Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files - and multiple targets per `BUILD` file, allowing for builds that are more - incremental than Maven's. -* Maven takes charge of steps for the deployment process. Bazel does not - automate deployment. -* Bazel enables you to express dependencies between languages. -* As you add new sections to the project, with Bazel you may need to add new - `BUILD` files. Best practice is to add a `BUILD` file to each new Java - package. +- Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files and multiple targets per `BUILD` file, allowing for builds that are more incremental than Maven's. +- Maven takes charge of steps for the deployment process. Bazel does not automate deployment. +- Bazel enables you to express dependencies between languages. +- As you add new sections to the project, with Bazel you may need to add new `BUILD` files. Best practice is to add a `BUILD` file to each new Java package. ## Migrate from Maven to Bazel The steps below describe how to migrate your project to Bazel: -1. [Create the MODULE.bazel file](#1-build) -2. [Create one BUILD file](#2-build) -3. [Create more BUILD files](#3-build) -4. [Build using Bazel](#4-build) +1. [Create the MODULE.bazel file](#1-build) +2. [Create one BUILD file](#2-build) +3. [Create more BUILD files](#3-build) +4. [Build using Bazel](#4-build) -Examples below come from a migration of the [Guava -project](https://github.com/google/guava) from Maven to Bazel. The -Guava project used is release `v31.1`. The examples using Guava do not walk -through each step in the migration, but they do show the files and contents that -are generated or added manually for the migration. +Examples below come from a migration of the [Guava project](https://github.com/google/guava) from Maven to Bazel. The Guava project used is release `v31.1`. The examples using Guava do not walk through each step in the migration, but they do show the files and contents that are generated or added manually for the migration. ``` $ git clone https://github.com/google/guava.git && cd guava @@ -59,22 +38,13 @@ $ git checkout v31.1 ### 1. Create the MODULE.bazel file -Create a file named `MODULE.bazel` at the root of your project. If your project -has no external dependencies, this file can be empty. +Create a file named `MODULE.bazel` at the root of your project. If your project has no external dependencies, this file can be empty. -If your project depends on files or packages that are not in one of the -project's directories, specify these external dependencies in the MODULE.bazel -file. You can use `rules_jvm_external` to manage dependencies from Maven. For -instructions about using this ruleset, see [the -README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) -. +If your project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the MODULE.bazel file. You can use `rules_jvm_external` to manage dependencies from Maven. For instructions about using this ruleset, see [the README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) . #### Guava project example: external dependencies -You can list the external dependencies of the [Guava -project](https://github.com/google/guava) with the -[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) -ruleset. +You can list the external dependencies of the [Guava project](https://github.com/google/guava) with the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) ruleset. Add the following snippet to the `MODULE.bazel` file: @@ -98,89 +68,74 @@ use_repo(maven, "maven") ### 2. Create one BUILD file -Now that you have your workspace defined and external dependencies (if -applicable) listed, you need to create `BUILD` files to describe how your -project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use -many `BUILD` files to build a project. These files specify multiple build -targets, which allow Bazel to produce incremental builds. - -Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of -your project and using it to do an initial build using Bazel. Then, you refine -your build by adding more `BUILD` files with more granular targets. - -1. In the same directory as your `MODULE.bazel` file, create a text file and - name it `BUILD`. - -2. In this `BUILD` file, use the appropriate rule to create one target to build - your project. Here are some tips: - - * Use the appropriate rule: - * To build projects with a single Maven module, use the - `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - ) - ``` - - * To build projects with multiple Maven modules, use the - `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob([ - "Module1/src/main/java/**/*.java", - "Module2/src/main/java/**/*.java", - ... - ]), - resources = glob([ - "Module1/src/main/resources/**", - "Module2/src/main/resources/**", - ... - ]), - deps = ["//:all-external-targets"], - ) - ``` - - * To build binaries, use the `java_binary` rule: - - ```python - java_binary( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - main_class = "com.example.Main" - ) - ``` - - * Specify the attributes: - * `name`: Give the target a meaningful name. In the examples - above, the target is called "everything." - * `srcs`: Use globbing to list all .java files in your project. - * `resources`: Use globbing to list all resources in your project. - * `deps`: You need to determine which external dependencies your - project needs. - * Take a look at the [example below of this top-level BUILD - file](#guava-2) from the migration of the Guava project. - -3. Now that you have a `BUILD` file at the root of your project, build your - project to ensure that it works. On the command line, from your workspace - directory, use `bazel build //:everything` to build your project with Bazel. - - The project has now been successfully built with Bazel. You will need to add - more `BUILD` files to allow incremental builds of the project. +Now that you have your workspace defined and external dependencies (if applicable) listed, you need to create `BUILD` files to describe how your project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use many `BUILD` files to build a project. These files specify multiple build targets, which allow Bazel to produce incremental builds. + +Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of your project and using it to do an initial build using Bazel. Then, you refine your build by adding more `BUILD` files with more granular targets. + +1. In the same directory as your `MODULE.bazel` file, create a text file and name it `BUILD`. + +2. In this `BUILD` file, use the appropriate rule to create one target to build your project. Here are some tips: + + - Use the appropriate rule: + + - To build projects with a single Maven module, use the `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + ) + ``` + + - To build projects with multiple Maven modules, use the `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob([ + "Module1/src/main/java/**/*.java", + "Module2/src/main/java/**/*.java", + ... + ]), + resources = glob([ + "Module1/src/main/resources/**", + "Module2/src/main/resources/**", + ... + ]), + deps = ["//:all-external-targets"], + ) + ``` + + - To build binaries, use the `java_binary` rule: + + ```python + java_binary( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + main_class = "com.example.Main" + ) + ``` + + - Specify the attributes: + + - `name`: Give the target a meaningful name. In the examples above, the target is called "everything." + - `srcs`: Use globbing to list all .java files in your project. + - `resources`: Use globbing to list all resources in your project. + - `deps`: You need to determine which external dependencies your project needs. + + - Take a look at the [example below of this top-level BUILD file](#guava-2) from the migration of the Guava project. + +3. Now that you have a `BUILD` file at the root of your project, build your project to ensure that it works. On the command line, from your workspace directory, use `bazel build //:everything` to build your project with Bazel. + + The project has now been successfully built with Bazel. You will need to add more `BUILD` files to allow incremental builds of the project. #### Guava project example: start with one BUILD file -When migrating the Guava project to Bazel, initially one `BUILD` file is used to -build the entire project. Here are the contents of this initial `BUILD` file in -the workspace directory: +When migrating the Guava project to Bazel, initially one `BUILD` file is used to build the entire project. Here are the contents of this initial `BUILD` file in the workspace directory: ```python java_library( @@ -202,40 +157,25 @@ java_library( ### 3. Create more BUILD files (optional) -Bazel does work with just one `BUILD file`, as you saw after completing your -first build. You should still consider breaking the build into smaller chunks by -adding more `BUILD` files with granular targets. +Bazel does work with just one `BUILD file`, as you saw after completing your first build. You should still consider breaking the build into smaller chunks by adding more `BUILD` files with granular targets. -Multiple `BUILD` files with multiple targets will give the build increased -granularity, allowing: +Multiple `BUILD` files with multiple targets will give the build increased granularity, allowing: -* increased incremental builds of the project, -* increased parallel execution of the build, -* better maintainability of the build for future users, and -* control over visibility of targets between packages, which can prevent - issues such as libraries containing implementation details leaking into - public APIs. +- increased incremental builds of the project, +- increased parallel execution of the build, +- better maintainability of the build for future users, and +- control over visibility of targets between packages, which can prevent issues such as libraries containing implementation details leaking into public APIs. Tips for adding more `BUILD` files: -* You can start by adding a `BUILD` file to each Java package. Start with Java - packages that have the fewest dependencies and work you way up to packages - with the most dependencies. -* As you add `BUILD` files and specify targets, add these new targets to the - `deps` sections of targets that depend on them. Note that the `glob()` - function does not cross package boundaries, so as the number of packages - grows the files matched by `glob()` will shrink. -* Any time you add a `BUILD` file to a `main` directory, ensure that you add a - `BUILD` file to the corresponding `test` directory. -* Take care to limit visibility properly between packages. -* To simplify troubleshooting errors in your setup of `BUILD` files, ensure - that the project continues to build with Bazel as you add each build file. - Run `bazel build //...` to ensure all of your targets still build. +- You can start by adding a `BUILD` file to each Java package. Start with Java packages that have the fewest dependencies and work you way up to packages with the most dependencies. +- As you add `BUILD` files and specify targets, add these new targets to the `deps` sections of targets that depend on them. Note that the `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. +- Any time you add a `BUILD` file to a `main` directory, ensure that you add a `BUILD` file to the corresponding `test` directory. +- Take care to limit visibility properly between packages. +- To simplify troubleshooting errors in your setup of `BUILD` files, ensure that the project continues to build with Bazel as you add each build file. Run `bazel build //...` to ensure all of your targets still build. ### 4. Build using Bazel -You've been building using Bazel as you add `BUILD` files to validate the setup -of the build. +You've been building using Bazel as you add `BUILD` files to validate the setup of the build. -When you have `BUILD` files at the desired granularity, you can use Bazel to -produce all of your builds. +When you have `BUILD` files at the desired granularity, you can use Bazel to produce all of your builds. diff --git a/migrate/xcode.mdx b/migrate/xcode.mdx index 986cd115..4cd229c9 100644 --- a/migrate/xcode.mdx +++ b/migrate/xcode.mdx @@ -2,243 +2,163 @@ title: 'Migrating from Xcode to Bazel' --- - - -This page describes how to build or test an Xcode project with Bazel. It -describes the differences between Xcode and Bazel, and provides the steps for -converting an Xcode project to a Bazel project. It also provides troubleshooting -solutions to address common errors. +This page describes how to build or test an Xcode project with Bazel. It describes the differences between Xcode and Bazel, and provides the steps for converting an Xcode project to a Bazel project. It also provides troubleshooting solutions to address common errors. ## Differences between Xcode and Bazel -* Bazel requires you to explicitly specify every build target and its - dependencies, plus the corresponding build settings via build rules. +- Bazel requires you to explicitly specify every build target and its dependencies, plus the corresponding build settings via build rules. -* Bazel requires all files on which the project depends to be present within - the workspace directory or specified as dependencies in the `MODULE.bazel` - file. +- Bazel requires all files on which the project depends to be present within the workspace directory or specified as dependencies in the `MODULE.bazel` file. -* When building Xcode projects with Bazel, the `BUILD` file(s) become the - source of truth. If you work on the project in Xcode, you must generate a - new version of the Xcode project that matches the `BUILD` files using - [rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) - whenever you update the `BUILD` files. Certain changes to the `BUILD` files - such as adding dependencies to a target don't require regenerating the - project which can speed up development. If you're not using Xcode, the - `bazel build` and `bazel test` commands provide build and test capabilities - with certain limitations described later in this guide. +- When building Xcode projects with Bazel, the `BUILD` file(s) become the source of truth. If you work on the project in Xcode, you must generate a new version of the Xcode project that matches the `BUILD` files using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) whenever you update the `BUILD` files. Certain changes to the `BUILD` files such as adding dependencies to a target don't require regenerating the project which can speed up development. If you're not using Xcode, the `bazel build` and `bazel test` commands provide build and test capabilities with certain limitations described later in this guide. ## Before you begin Before you begin, do the following: -1. [Install Bazel](/install) if you have not already done so. +1. [Install Bazel](/install) if you have not already done so. -2. If you're not familiar with Bazel and its concepts, complete the [iOS app - tutorial](/start/ios-app)). You should understand the Bazel workspace, - including the `MODULE.bazel` and `BUILD` files, as well as the concepts of - targets, build rules, and Bazel packages. +2. If you're not familiar with Bazel and its concepts, complete the [iOS app tutorial](/start/ios-app)). You should understand the Bazel workspace, including the `MODULE.bazel` and `BUILD` files, as well as the concepts of targets, build rules, and Bazel packages. -3. Analyze and understand the project's dependencies. +3. Analyze and understand the project's dependencies. ### Analyze project dependencies -Unlike Xcode, Bazel requires you to explicitly declare all dependencies for -every target in the `BUILD` file. +Unlike Xcode, Bazel requires you to explicitly declare all dependencies for every target in the `BUILD` file. -For more information on external dependencies, see [Working with external -dependencies](/docs/external). +For more information on external dependencies, see [Working with external dependencies](/docs/external). ## Build or test an Xcode project with Bazel To build or test an Xcode project with Bazel, do the following: -1. [Create the `MODULE.bazel` file](#create-workspace) +1. [Create the `MODULE.bazel` file](#create-workspace) -2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) +2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) -3. [Create a `BUILD` file:](#create-build-file) +3. [Create a `BUILD` file:](#create-build-file) - a. [Add the application target](#add-app-target) + a. [Add the application target](#add-app-target) - b. [(Optional) Add the test target(s)](#add-test-target) + b. [(Optional) Add the test target(s)](#add-test-target) - c. [Add the library target(s)](#add-library-target) + c. [Add the library target(s)](#add-library-target) -4. [(Optional) Granularize the build](#granularize-build) +4. [(Optional) Granularize the build](#granularize-build) -5. [Run the build](#run-build) +5. [Run the build](#run-build) -6. [Generate the Xcode project with rules_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) +6. [Generate the Xcode project with rules\_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) ### Step 1: Create the `MODULE.bazel` file -Create a `MODULE.bazel` file in a new directory. This directory becomes the -Bazel workspace root. If the project uses no external dependencies, this file -can be empty. If the project depends on files or packages that are not in one of -the project's directories, specify these external dependencies in the -`MODULE.bazel` file. +Create a `MODULE.bazel` file in a new directory. This directory becomes the Bazel workspace root. If the project uses no external dependencies, this file can be empty. If the project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the `MODULE.bazel` file. -Note: Place the project source code within the directory tree containing the -`MODULE.bazel` file. +Note: Place the project source code within the directory tree containing the `MODULE.bazel` file. ### Step 2: (Experimental) Integrate SwiftPM dependencies -To integrate SwiftPM dependencies into the Bazel workspace with -[swift_bazel](https://github.com/cgrindel/swift_bazel), you must -convert them into Bazel packages as described in the [following -tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) -. +To integrate SwiftPM dependencies into the Bazel workspace with [swift\_bazel](https://github.com/cgrindel/swift_bazel), you must convert them into Bazel packages as described in the [following tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) . -Note: SwiftPM support is a manual process with many variables. SwiftPM -integration with Bazel has not been fully verified and is not officially -supported. +Note: SwiftPM support is a manual process with many variables. SwiftPM integration with Bazel has not been fully verified and is not officially supported. ### Step 3: Create a `BUILD` file -Once you have defined the workspace and external dependencies, you need to -create a `BUILD` file that tells Bazel how the project is structured. Create the -`BUILD` file at the root of the Bazel workspace and configure it to do an -initial build of the project as follows: +Once you have defined the workspace and external dependencies, you need to create a `BUILD` file that tells Bazel how the project is structured. Create the `BUILD` file at the root of the Bazel workspace and configure it to do an initial build of the project as follows: -* [Step 3a: Add the application target](#step-3a-add-the-application-target) -* [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) -* [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) +- [Step 3a: Add the application target](#step-3a-add-the-application-target) +- [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) +- [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) -**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, -packages, and targets](/concepts/build-ref). +**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, packages, and targets](/concepts/build-ref). #### Step 3a: Add the application target -Add a -[`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) -or an -[`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) -rule target. This target builds a macOS or iOS application bundle, respectively. -In the target, specify the following at the minimum: +Add a [`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) or an [`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) rule target. This target builds a macOS or iOS application bundle, respectively. In the target, specify the following at the minimum: -* `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the - binary. +- `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the binary. -* `provisioning_profile` - provisioning profile from your Apple Developer - account (if building for an iOS device device). +- `provisioning_profile` - provisioning profile from your Apple Developer account (if building for an iOS device device). -* `families` (iOS only) - whether to build the application for iPhone, iPad, - or both. +- `families` (iOS only) - whether to build the application for iPhone, iPad, or both. -* `infoplists` - list of .plist files to merge into the final Info.plist file. +- `infoplists` - list of .plist files to merge into the final Info.plist file. -* `minimum_os_version` - the minimum version of macOS or iOS that the - application supports. This ensures Bazel builds the application with the - correct API levels. +- `minimum_os_version` - the minimum version of macOS or iOS that the application supports. This ensures Bazel builds the application with the correct API levels. #### Step 3b: (Optional) Add the test target(s) -Bazel's [Apple build -rules](https://github.com/bazelbuild/rules_apple) support running -unit and UI tests on all Apple platforms. Add test targets as follows: +Bazel's [Apple build rules](https://github.com/bazelbuild/rules_apple) support running unit and UI tests on all Apple platforms. Add test targets as follows: -* [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) - to run library-based and application-based unit tests on a macOS. +- [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) to run library-based and application-based unit tests on a macOS. -* [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) - to build and run library-based unit tests on iOS. +- [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) to build and run library-based unit tests on iOS. -* [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) - to build and run user interface tests in the iOS simulator. +- [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) to build and run user interface tests in the iOS simulator. -* Similar test rules exist for - [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), - [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) - and - [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). +- Similar test rules exist for [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) and [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). -At the minimum, specify a value for the `minimum_os_version` attribute. While -other packaging attributes, such as `bundle_identifier` and `infoplists`, -default to most commonly used values, ensure that those defaults are compatible -with the project and adjust them as necessary. For tests that require the iOS -simulator, also specify the `ios_application` target name as the value of the -`test_host` attribute. +At the minimum, specify a value for the `minimum_os_version` attribute. While other packaging attributes, such as `bundle_identifier` and `infoplists`, default to most commonly used values, ensure that those defaults are compatible with the project and adjust them as necessary. For tests that require the iOS simulator, also specify the `ios_application` target name as the value of the `test_host` attribute. #### Step 3c: Add the library target(s) -Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each -Objective-C library and a -[`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) -target for each Swift library on which the application and/or tests depend. +Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each Objective-C library and a [`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) target for each Swift library on which the application and/or tests depend. Add the library targets as follows: -* Add the application library targets as dependencies to the application - targets. +- Add the application library targets as dependencies to the application targets. -* Add the test library targets as dependencies to the test targets. +- Add the test library targets as dependencies to the test targets. -* List the implementation sources in the `srcs` attribute. +- List the implementation sources in the `srcs` attribute. -* List the headers in the `hdrs` attribute. +- List the headers in the `hdrs` attribute. -Note: You can use the [`glob`](/reference/be/functions#glob) function to include -all sources and/or headers of a certain type. Use it carefully as it might -include files you do not want Bazel to build. +Note: You can use the [`glob`](/reference/be/functions#glob) function to include all sources and/or headers of a certain type. Use it carefully as it might include files you do not want Bazel to build. -You can browse existing examples for various types of applications directly in -the [rules_apple examples -directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For -example: +You can browse existing examples for various types of applications directly in the [rules\_apple examples directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For example: -* [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) +- [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) -* [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) +- [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) -* [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) +- [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) -For more information on build rules, see [Apple Rules for -Bazel](https://github.com/bazelbuild/rules_apple). +For more information on build rules, see [Apple Rules for Bazel](https://github.com/bazelbuild/rules_apple). At this point, it is a good idea to test the build: -`bazel build //:` +`bazel build //:<application_target>` ### Step 4: (Optional) Granularize the build -If the project is large, or as it grows, consider chunking it into multiple -Bazel packages. This increased granularity provides: +If the project is large, or as it grows, consider chunking it into multiple Bazel packages. This increased granularity provides: -* Increased incrementality of builds, +- Increased incrementality of builds, -* Increased parallelization of build tasks, +- Increased parallelization of build tasks, -* Better maintainability for future users, +- Better maintainability for future users, -* Better control over source code visibility across targets and packages. This - prevents issues such as libraries containing implementation details leaking - into public APIs. +- Better control over source code visibility across targets and packages. This prevents issues such as libraries containing implementation details leaking into public APIs. Tips for granularizing the project: -* Put each library in its own Bazel package. Start with those requiring the - fewest dependencies and work your way up the dependency tree. +- Put each library in its own Bazel package. Start with those requiring the fewest dependencies and work your way up the dependency tree. -* As you add `BUILD` files and specify targets, add these new targets to the - `deps` attributes of targets that depend on them. +- As you add `BUILD` files and specify targets, add these new targets to the `deps` attributes of targets that depend on them. -* The `glob()` function does not cross package boundaries, so as the number of - packages grows the files matched by `glob()` will shrink. +- The `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. -* When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to - the corresponding `test` directory. +- When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to the corresponding `test` directory. -* Enforce healthy visibility limits across packages. +- Enforce healthy visibility limits across packages. -* Build the project after each major change to the `BUILD` files and fix build - errors as you encounter them. +- Build the project after each major change to the `BUILD` files and fix build errors as you encounter them. ### Step 5: Run the build -Run the fully migrated build to ensure it completes with no errors or warnings. -Run every application and test target individually to more easily find sources -of any errors that occur. +Run the fully migrated build to ensure it completes with no errors or warnings. Run every application and test target individually to more easily find sources of any errors that occur. For example: @@ -246,25 +166,17 @@ For example: bazel build //:my-target ``` -### Step 6: Generate the Xcode project with rules_xcodeproj +### Step 6: Generate the Xcode project with rules\_xcodeproj -When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source -of truth about the build. To make Xcode aware of this, you must generate a -Bazel-compatible Xcode project using -[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) -. +When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source of truth about the build. To make Xcode aware of this, you must generate a Bazel-compatible Xcode project using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) . ### Troubleshooting -Bazel errors can arise when it gets out of sync with the selected Xcode version, -like when you apply an update. Here are some things to try if you're -experiencing errors with Xcode, for example "Xcode version must be specified to -use an Apple CROSSTOOL". +Bazel errors can arise when it gets out of sync with the selected Xcode version, like when you apply an update. Here are some things to try if you're experiencing errors with Xcode, for example "Xcode version must be specified to use an Apple CROSSTOOL". -* Manually run Xcode and accept any terms and conditions. +- Manually run Xcode and accept any terms and conditions. -* Use Xcode select to indicate the correct version, accept the license, and - clear Bazel's state. +- Use Xcode select to indicate the correct version, accept the license, and clear Bazel's state. ```posix-terminal sudo xcode-select -s /Applications/Xcode.app/Contents/Developer @@ -274,7 +186,6 @@ use an Apple CROSSTOOL". bazel sync --configure ``` -* If this does not work, you may also try running `bazel clean --expunge`. +- If this does not work, you may also try running `bazel clean --expunge`. -Note: If you've saved your Xcode to a different path, you can use `xcode-select --s` to point to that path. +Note: If you've saved your Xcode to a different path, you can use `xcode-select -s` to point to that path. diff --git a/query/aquery.mdx b/query/aquery.mdx index 151565b7..446855d1 100644 --- a/query/aquery.mdx +++ b/query/aquery.mdx @@ -2,23 +2,13 @@ title: 'Action Graph Query (aquery)' --- +The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis Configured Target Graph and exposes information about **Actions, Artifacts and their relationships.** +`aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs/outputs/mnemonics. -The `aquery` command allows you to query for actions in your build graph. -It operates on the post-analysis Configured Target Graph and exposes -information about **Actions, Artifacts and their relationships.** +The tool accepts several command-line [options](#command-options). Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. -`aquery` is useful when you are interested in the properties of the Actions/Artifacts -generated from the Configured Target Graph. For example, the actual commands run -and their inputs/outputs/mnemonics. - -The tool accepts several command-line [options](#command-options). -Notably, the aquery command runs on top of a regular Bazel build and inherits -the set of options available during a build. - -It supports the same set of functions that is also available to traditional -`query` but `siblings`, `buildfiles` and -`tests`. +It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. An example `aquery` output (without specific details): @@ -41,11 +31,9 @@ A simple example of the syntax for `aquery` is as follows: The query expression (in quotes) consists of the following: -* `aquery_function(...)`: functions specific to `aquery`. - More details [below](#using-aquery-functions). -* `function(...)`: the standard [functions](/query/language#functions) - as traditional `query`. -* `//target` is the label to the interested target. +- `aquery_function(...)`: functions specific to `aquery`. More details [below](#using-aquery-functions). +- `function(...)`: the standard [functions](/query/language#functions) as traditional `query`. +- `//target` is the label to the interested target. ``` # aquery examples: @@ -64,14 +52,13 @@ $ bazel aquery 'inputs(".*cpp", deps(//src/target_a))' There are three `aquery` functions: -* `inputs`: filter actions by inputs. -* `outputs`: filter actions by outputs -* `mnemonic`: filter actions by mnemonic +- `inputs`: filter actions by inputs. +- `outputs`: filter actions by outputs +- `mnemonic`: filter actions by mnemonic `expr ::= inputs(word, expr)` - The `inputs` operator returns the actions generated from building `expr`, - whose input filenames match the regex provided by `word`. +The `inputs` operator returns the actions generated from building `expr`, whose input filenames match the regex provided by `word`. `$ bazel aquery 'inputs(".*cpp", deps(//src/target_a))'` @@ -83,13 +70,9 @@ You can also combine functions to achieve the AND operation. For example: $ bazel aquery 'mnemonic("Cpp.*", (inputs(".*cpp", inputs("foo.*", //src/target_a))))' ``` - The above command would find all actions involved in building `//src/target_a`, - whose mnemonics match `"Cpp.*"` and inputs match the patterns - `".*cpp"` and `"foo.*"`. +The above command would find all actions involved in building `//src/target_a`, whose mnemonics match `"Cpp.*"` and inputs match the patterns `".*cpp"` and `"foo.*"`. -Important: aquery functions can't be nested inside non-aquery functions. -Conceptually, this makes sense since the output of aquery functions is Actions, -not Configured Targets. +Important: aquery functions can't be nested inside non-aquery functions. Conceptually, this makes sense since the output of aquery functions is Actions, not Configured Targets. An example of the syntax error produced: @@ -104,23 +87,17 @@ An example of the syntax error produced: ### Build options -`aquery` runs on top of a regular Bazel build and thus inherits the set of -[options](/reference/command-line-reference#build-options) -available during a build. +`aquery` runs on top of a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. ### Aquery options #### `--output=(text|summary|commands|proto|jsonproto|textproto), default=text` -The default output format (`text`) is human-readable, -use `proto`, `textproto`, or `jsonproto` for machine-readable format. -The proto message is `analysis.ActionGraphContainer`. +The default output format (`text`) is human-readable, use `proto`, `textproto`, or `jsonproto` for machine-readable format. The proto message is `analysis.ActionGraphContainer`. -The `commands` output format prints a list of build commands with -one command per line. +The `commands` output format prints a list of build commands with one command per line. -In general, do not depend on the order of output. For more information, -see the [core query ordering contract](/query/language#graph-order). +In general, do not depend on the order of output. For more information, see the [core query ordering contract](/query/language#graph-order). #### `--include_commandline, default=true` @@ -142,45 +119,35 @@ Warning: Enabling this flag will automatically enable the `--include_commandline #### `--include_file_write_contents, default=false` -Include file contents for the `actions.write()` action and the contents of the -manifest file for the `SourceSymlinkManifest` action The file contents is -returned in the `file_contents` field with `--output=`xxx`proto`. -With `--output=text`, the output has +Include file contents for the `actions.write()` action and the contents of the manifest file for the `SourceSymlinkManifest` action The file contents is returned in the `file_contents` field with `--output=`xxx`proto`. With `--output=text`, the output has + ``` -FileWriteContents: [] +FileWriteContents: [<base64-encoded file contents>] ``` + line #### `--skyframe_state, default=false` Without performing extra analysis, dump the Action Graph from Skyframe. -Note: Specifying a target with `--skyframe_state` is currently not supported. -This flag is only available with `--output=proto` or `--output=textproto`. +Note: Specifying a target with `--skyframe_state` is currently not supported. This flag is only available with `--output=proto` or `--output=textproto`. ## Other tools and features ### Querying against the state of Skyframe -[Skyframe](/reference/skyframe) is the evaluation and -incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph -constructed from the previous runs of the [Analysis phase](/run/build#analysis). +[Skyframe](/reference/skyframe) is the evaluation and incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph constructed from the previous runs of the [Analysis phase](/run/build#analysis). -In some cases, it is useful to query the Action Graph on Skyframe. -An example use case would be: +In some cases, it is useful to query the Action Graph on Skyframe. An example use case would be: -1. Run `bazel build //target_a` -2. Run `bazel build //target_b` -3. File `foo.out` was generated. +1. Run `bazel build //target_a` +2. Run `bazel build //target_b` +3. File `foo.out` was generated. -_As a Bazel user, I want to determine if `foo.out` was generated from building -`//target_a` or `//target_b`_. +*As a Bazel user, I want to determine if `foo.out` was generated from building `//target_a` or `//target_b`*. -One could run `bazel aquery 'outputs("foo.out", //target_a)'` and -`bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible -for creating `foo.out`, and in turn the target. However, the number of different -targets previously built can be larger than 2, which makes running multiple `aquery` -commands a hassle. +One could run `bazel aquery 'outputs("foo.out", //target_a)'` and `bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible for creating `foo.out`, and in turn the target. However, the number of different targets previously built can be larger than 2, which makes running multiple `aquery` commands a hassle. As an alternative, the `--skyframe_state` flag can be used: @@ -194,22 +161,17 @@ As an alternative, the `--skyframe_state` flag can be used: $ bazel aquery --output=proto --skyframe_state 'outputs("foo.out")' ``` -With `--skyframe_state` mode, `aquery` takes the content of the Action Graph -that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and -outputs the content, without re-running the analysis phase. +With `--skyframe_state` mode, `aquery` takes the content of the Action Graph that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and outputs the content, without re-running the analysis phase. #### Special considerations ##### Output format -`--skyframe_state` is currently only available for `--output=proto` -and `--output=textproto` +`--skyframe_state` is currently only available for `--output=proto` and `--output=textproto` ##### Non-inclusion of target labels in the query expression -Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, -regardless of the targets. Having the target label specified in the query together with -`--skyframe_state` is considered a syntax error: +Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, regardless of the targets. Having the target label specified in the query together with `--skyframe_state` is considered a syntax error: ``` # WRONG: Target Included @@ -227,12 +189,9 @@ regardless of the targets. Having the target label specified in the query togeth ### Comparing aquery outputs -You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. -For instance: when you make some changes to your rule definition and want to verify that the -command lines being run did not change. `aquery_differ` is the tool for that. +You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. For instance: when you make some changes to your rule definition and want to verify that the command lines being run did not change. `aquery_differ` is the tool for that. -The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. -To use it, clone the repository to your local machine. An example usage: +The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. To use it, clone the repository to your local machine. An example usage: ``` $ bazel run //tools/aquery_differ -- \ @@ -243,9 +202,7 @@ To use it, clone the repository to your local machine. An example usage: --attrs=inputs ``` -The above command returns the difference between the `before` and `after` aquery outputs: -which actions were present in one but not the other, which actions have different -command line/inputs in each aquery output, ...). The result of running the above command would be: +The above command returns the difference between the `before` and `after` aquery outputs: which actions were present in one but not the other, which actions have different command line/inputs in each aquery output, ...). The result of running the above command would be: ``` Aquery output 'after' change contains an action that generates the following outputs that aquery output 'before' change doesn't: @@ -268,36 +225,29 @@ command line/inputs in each aquery output, ...). The result of running the above `--before, --after`: The aquery output files to be compared -`--input_type=(proto|text_proto), default=proto`: the format of the input -files. Support is provided for `proto` and `textproto` aquery output. +`--input_type=(proto|text_proto), default=proto`: the format of the input files. Support is provided for `proto` and `textproto` aquery output. -`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions -to be compared. +`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions to be compared. ### Aspect-on-aspect -It is possible for [Aspects](/extending/aspects) -to be applied on top of each other. The aquery output of the action generated by -these Aspects would then include the _Aspect path_, which is the sequence of -Aspects applied to the target which generated the action. +It is possible for [Aspects](/extending/aspects) to be applied on top of each other. The aquery output of the action generated by these Aspects would then include the *Aspect path*, which is the sequence of Aspects applied to the target which generated the action. An example of Aspect-on-Aspect: ``` t0 ^ - | <- a1 + | <- a1 t1 ^ - | <- a2 + | <- a2 t2 ``` -Let ti be a target of rule ri, which applies an Aspect ai -to its dependencies. +Let ti be a target of rule ri, which applies an Aspect ai to its dependencies. -Assume that a2 generates an action X when applied to target t0. The text output of -`bazel aquery --include_aspects 'deps(//t2)'` for action X would be: +Assume that a2 generates an action X when applied to target t0. The text output of `bazel aquery --include_aspects 'deps(//t2)'` for action X would be: ``` action ... @@ -305,13 +255,11 @@ Assume that a2 generates an action X when applied to target t0. The text output Target: //my_pkg:t0 Configuration: ... AspectDescriptors: [//my_pkg:rule.bzl%**a2**(foo=...) - -> //my_pkg:rule.bzl%**a1**(bar=...)] + -> //my_pkg:rule.bzl%**a1**(bar=...)] ... ``` -This means that action `X` was generated by Aspect `a2` applied onto -`a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied -onto target `t0`. +This means that action `X` was generated by Aspect `a2` applied onto `a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied onto target `t0`. Each `AspectDescriptor` has the following format: @@ -319,49 +267,31 @@ Each `AspectDescriptor` has the following format: AspectClass([param=value,...]) ``` -`AspectClass` could be the name of the Aspect class (for native Aspects) or -`bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are -sorted in topological order of the -[dependency graph](/extending/aspects#aspect_basics). +`AspectClass` could be the name of the Aspect class (for native Aspects) or `bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are sorted in topological order of the [dependency graph](/extending/aspects#aspect_basics). ### Linking with the JSON profile -While aquery provides information about the actions being run in a build (why they're being run, -their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) -tells us the timing and duration of their execution. -It is possible to combine these 2 sets of information via a common denominator: an action's primary output. +While aquery provides information about the actions being run in a build (why they're being run, their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) tells us the timing and duration of their execution. It is possible to combine these 2 sets of information via a common denominator: an action's primary output. -To include actions' outputs in the JSON profile, generate the profile with -`--experimental_include_primary_output --noslim_profile`. -Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output -is included by default by aquery. +To include actions' outputs in the JSON profile, generate the profile with `--experimental_include_primary_output --noslim_profile`. Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output is included by default by aquery. -We don't currently provide a canonical tool to combine these 2 data sources, but you should be -able to build your own script with the above information. +We don't currently provide a canonical tool to combine these 2 data sources, but you should be able to build your own script with the above information. ## Known issues ### Handling shared actions -Sometimes actions are -[shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) -between configured targets. +Sometimes actions are [shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) between configured targets. -In the execution phase, those shared actions are -[simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. -However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these -like separate actions whose output Artifacts have the exact same `execPath`. As a result, -equivalent Artifacts appear duplicated. +In the execution phase, those shared actions are [simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these like separate actions whose output Artifacts have the exact same `execPath`. As a result, equivalent Artifacts appear duplicated. -The list of aquery issues/planned features can be found on -[GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). +The list of aquery issues/planned features can be found on [GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). ## FAQs ### The ActionKey remains the same even though the content of an input file changed. -In the context of aquery, the `ActionKey` refers to the `String` gotten from -[ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): +In the context of aquery, the `ActionKey` refers to the `String` gotten from [ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): ``` Returns a string encoding all of the significant behaviour of this Action that might affect the @@ -383,8 +313,7 @@ In the context of aquery, the `ActionKey` refers to the `String` gotten from input names change or else action validation may falsely validate. ``` -This excludes the changes to the content of the input files, and is not to be confused with -[RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). +This excludes the changes to the content of the input files, and is not to be confused with [RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). ## Updates diff --git a/query/cquery.mdx b/query/cquery.mdx index d35ea829..6a073058 100644 --- a/query/cquery.mdx +++ b/query/cquery.mdx @@ -2,21 +2,14 @@ title: 'Configurable Query (cquery)' --- +`cquery` is a variant of [`query`](/query/language) that correctly handles [`select()`](/docs/configurable-attributes) and build options' effects on the build graph. - -`cquery` is a variant of [`query`](/query/language) that correctly handles -[`select()`](/docs/configurable-attributes) and build options' effects on the -build graph. - -It achieves this by running over the results of Bazel's [analysis -phase](/extending/concepts#evaluation-model), -which integrates these effects. `query`, by contrast, runs over the results of -Bazel's loading phase, before options are evaluated. +It achieves this by running over the results of Bazel's [analysis phase](/extending/concepts#evaluation-model), which integrates these effects. `query`, by contrast, runs over the results of Bazel's loading phase, before options are evaluated. For example: ``` -$ cat > tree/BUILD <<EOF +$ cat > tree/BUILD <<EOF sh_library( name = "ash", deps = select({ @@ -30,11 +23,11 @@ sh_library(name = "white-ash") sh_library(name = "common-ash") config_setting( name = "excelsior", - values = \{"define": "species=excelsior"\}, + values = {"define": "species=excelsior"}, ) config_setting( name = "americana", - values = \{"define": "species=americana"\}, + values = {"define": "species=americana"}, ) EOF ``` @@ -59,13 +52,9 @@ $ bazel cquery "deps(//tree:ash)" --define species=excelsior --noimplicit_deps //tree:excelsior (9f87702) ``` -Each result includes a [unique identifier](#configurations) `(9f87702)` of -the [configuration](/reference/glossary#configuration) the -target is built with. +Each result includes a [unique identifier](#configurations) `(9f87702)` of the [configuration](/reference/glossary#configuration) the target is built with. -Since `cquery` runs over the configured target graph. it doesn't have insight -into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) -rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). +Since `cquery` runs over the configured target graph. it doesn't have insight into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). ## Basic syntax @@ -75,19 +64,10 @@ A simple `cquery` call looks like: The query expression `"function(//target)"` consists of the following: -* **`function(...)`** is the function to run on the target. `cquery` - supports most - of `query`'s [functions](/query/language#functions), plus a - few new ones. -* **`//target`** is the expression fed to the function. In this example, the - expression is a simple target. But the query language also allows nesting of functions. - See the [Query guide](/query/guide) for examples. - +- **`function(...)`** is the function to run on the target. `cquery` supports most of `query`'s [functions](/query/language#functions), plus a few new ones. +- **`//target`** is the expression fed to the function. In this example, the expression is a simple target. But the query language also allows nesting of functions. See the [Query guide](/query/guide) for examples. -`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) -phases. Unless otherwise specified, `cquery` parses the target(s) listed in the -query expression. See [`--universe_scope`](#universe-scope) -for querying dependencies of top-level build targets. +`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) phases. Unless otherwise specified, `cquery` parses the target(s) listed in the query expression. See [`--universe_scope`](#universe-scope) for querying dependencies of top-level build targets. ## Configurations @@ -97,9 +77,7 @@ The line: //tree:ash (9f87702) ``` -means `//tree:ash` was built in a configuration with ID `9f87702`. For most -targets, this is an opaque hash of the build option values defining the -configuration. +means `//tree:ash` was built in a configuration with ID `9f87702`. For most targets, this is an opaque hash of the build option values defining the configuration. To see the configuration's complete contents, run: @@ -107,25 +85,15 @@ To see the configuration's complete contents, run: $ bazel config 9f87702 ``` -`9f87702` is a prefix of the complete ID. This is because complete IDs are -SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid -prefix of a complete ID, similar to -[Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). - To see complete IDs, run `$ bazel config`. +`9f87702` is a prefix of the complete ID. This is because complete IDs are SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid prefix of a complete ID, similar to [Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). To see complete IDs, run `$ bazel config`. ## Target pattern evaluation -`//foo` has a different meaning for `cquery` than for `query`. This is because -`cquery` evaluates _configured_ targets and the build graph may have multiple -configured versions of `//foo`. +`//foo` has a different meaning for `cquery` than for `query`. This is because `cquery` evaluates *configured* targets and the build graph may have multiple configured versions of `//foo`. -For `cquery`, a target pattern in the query expression evaluates -to every configured target with a label that matches that pattern. Output is -deterministic, but `cquery` makes no ordering guarantee beyond the -[core query ordering contract](/query/language#graph-order). +For `cquery`, a target pattern in the query expression evaluates to every configured target with a label that matches that pattern. Output is deterministic, but `cquery` makes no ordering guarantee beyond the [core query ordering contract](/query/language#graph-order). -This produces subtler results for query expressions than with `query`. -For example, the following can produce multiple results: +This produces subtler results for query expressions than with `query`. For example, the following can produce multiple results: ``` # Analyzes //foo in the target configuration, but also analyzes @@ -137,22 +105,13 @@ $ bazel cquery //foo --universe_scope=//foo,//genrule_with_foo_as_tool //foo (exec) ``` -If you want to precisely declare which instance to query over, use -the [`config`](#config) function. +If you want to precisely declare which instance to query over, use the [`config`](#config) function. -See `query`'s [target pattern -documentation](/query/language#target-patterns) for more information on target -patterns. +See `query`'s [target pattern documentation](/query/language#target-patterns) for more information on target patterns. ## Functions -Of the [set of functions](/query/language#functions "list of query functions") -supported by `query`, `cquery` supports all but -[`allrdeps`](/query/language#allrdeps), -[`buildfiles`](/query/language#buildfiles), -[`rbuildfiles`](/query/language#rbuildfiles), -[`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and -[`visible`](/query/language#visible). +Of the [set of functions](/query/language#functions "list of query functions") supported by `query`, `cquery` supports all but [`allrdeps`](/query/language#allrdeps), [`buildfiles`](/query/language#buildfiles), [`rbuildfiles`](/query/language#rbuildfiles), [`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and [`visible`](/query/language#visible). `cquery` also introduces the following new functions: @@ -160,13 +119,9 @@ supported by `query`, `cquery` supports all but `expr ::= config(expr, word)` -The `config` operator attempts to find the configured target for -the label denoted by the first argument and configuration specified by the -second argument. +The `config` operator attempts to find the configured target for the label denoted by the first argument and configuration specified by the second argument. -Valid values for the second argument are `null` or a -[custom configuration hash](#configurations). Hashes can be retrieved from `$ -bazel config` or a previous `cquery`'s output. +Valid values for the second argument are `null` or a [custom configuration hash](#configurations). Hashes can be retrieved from `$ bazel config` or a previous `cquery`'s output. Examples: @@ -182,27 +137,19 @@ $ bazel cquery "deps(//foo)" $ bazel cquery "config(//baz, 3732cc8)" ``` -If not all results of the first argument can be found in the specified -configuration, only those that can be found are returned. If no results -can be found in the specified configuration, the query fails. +If not all results of the first argument can be found in the specified configuration, only those that can be found are returned. If no results can be found in the specified configuration, the query fails. ## Options ### Build options -`cquery` runs over a regular Bazel build and thus inherits the set of -[options](/reference/command-line-reference#build-options) available during a -build. +`cquery` runs over a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. -### Using cquery options +### Using cquery options #### `--universe_scope` (comma-separated list) -Often, the dependencies of configured targets go through -[transitions](/extending/rules#configurations), -which causes their configuration to differ from their dependent. This flag -allows you to query a target as if it were built as a dependency or a transitive -dependency of another target. For example: +Often, the dependencies of configured targets go through [transitions](/extending/rules#configurations), which causes their configuration to differ from their dependent. This flag allows you to query a target as if it were built as a dependency or a transitive dependency of another target. For example: ``` # x/BUILD @@ -210,7 +157,7 @@ genrule( name = "my_gen", srcs = ["x.in"], outs = ["x.cc"], - cmd = "$(locations :tool) $< >$@", + cmd = "$(locations :tool) $< >$@", tools = [":tool"], ) cc_binary( @@ -219,75 +166,34 @@ cc_binary( ) ``` -Genrules configure their tools in the -[exec configuration](/extending/rules#configurations) -so the following queries would produce the following outputs: - - - - - - - - - - - - - - - - - - - - - -
QueryTarget BuiltOutput
bazel cquery "//x:tool"//x:tool//x:tool(targetconfig)
bazel cquery "//x:tool" --universe_scope="//x:my_gen"//x:my_gen//x:tool(execconfig)
- -If this flag is set, its contents are built. _If it's not set, all targets -mentioned in the query expression are built_ instead. The transitive closure of -the built targets are used as the universe of the query. Either way, the targets -to be built must be buildable at the top level (that is, compatible with -top-level options). `cquery` returns results in the transitive closure of these -top-level targets. - -Even if it's possible to build all targets in a query expression at the top -level, it may be beneficial to not do so. For example, explicitly setting -`--universe_scope` could prevent building targets multiple times in -configurations you don't care about. It could also help specify which -configuration version of a target you're looking for. You should set this flag -if your query expression is more complex than `deps(//foo)`. +Genrules configure their tools in the [exec configuration](/extending/rules#configurations) so the following queries would produce the following outputs: + +| Query | Target Built | Output | +| ------------------------------------------------------- | ------------ | ---------------------- | +| bazel cquery "//x:tool" | //x:tool | //x:tool(targetconfig) | +| bazel cquery "//x:tool" --universe\_scope="//x:my\_gen" | //x:my\_gen | //x:tool(execconfig) | + +If this flag is set, its contents are built. *If it's not set, all targets mentioned in the query expression are built* instead. The transitive closure of the built targets are used as the universe of the query. Either way, the targets to be built must be buildable at the top level (that is, compatible with top-level options). `cquery` returns results in the transitive closure of these top-level targets. + +Even if it's possible to build all targets in a query expression at the top level, it may be beneficial to not do so. For example, explicitly setting `--universe_scope` could prevent building targets multiple times in configurations you don't care about. It could also help specify which configuration version of a target you're looking for. You should set this flag if your query expression is more complex than `deps(//foo)`. #### `--implicit_deps` (boolean, default=True) -Setting this flag to false filters out all results that aren't explicitly set in -the BUILD file and instead set elsewhere by Bazel. This includes filtering -resolved toolchains. +Setting this flag to false filters out all results that aren't explicitly set in the BUILD file and instead set elsewhere by Bazel. This includes filtering resolved toolchains. #### `--tool_deps` (boolean, default=True) -Setting this flag to false filters out all configured targets for which the -path from the queried target to them crosses a transition between the target -configuration and the -[non-target configurations](/extending/rules#configurations). If the queried -target is in the target configuration, setting `--notool_deps` will only return -targets that also are in the target configuration. If the queried target is in a -non-target configuration, setting `--notool_deps` will only return targets also -in non-target configurations. This setting generally does not affect filtering -of resolved toolchains. +Setting this flag to false filters out all configured targets for which the path from the queried target to them crosses a transition between the target configuration and the [non-target configurations](/extending/rules#configurations). If the queried target is in the target configuration, setting `--notool_deps` will only return targets that also are in the target configuration. If the queried target is in a non-target configuration, setting `--notool_deps` will only return targets also in non-target configurations. This setting generally does not affect filtering of resolved toolchains. #### `--include_aspects` (boolean, default=True) Include dependencies added by [aspects](/extending/aspects). -If this flag is disabled, `cquery somepath(X, Y)` and -`cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. +If this flag is disabled, `cquery somepath(X, Y)` and `cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. ## Output formats -By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. -There are other options for exposing the results as well. +By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. There are other options for exposing the results as well. ### Transitions @@ -296,22 +202,11 @@ There are other options for exposing the results as well. --transitions=full ``` -Configuration [transitions](/extending/rules#configurations) -are used to build targets underneath the top level targets in different -configurations than the top level targets. +Configuration [transitions](/extending/rules#configurations) are used to build targets underneath the top level targets in different configurations than the top level targets. -For example, a target might impose a transition to the exec configuration on all -dependencies in its `tools` attribute. These are known as attribute -transitions. Rules can also impose transitions on their own configurations, -known as rule class transitions. This output format outputs information about -these transitions such as what type they are and the effect they have on build -options. +For example, a target might impose a transition to the exec configuration on all dependencies in its `tools` attribute. These are known as attribute transitions. Rules can also impose transitions on their own configurations, known as rule class transitions. This output format outputs information about these transitions such as what type they are and the effect they have on build options. -This output format is triggered by the `--transitions` flag which by default is -set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs -information about rule class transitions and attribute transitions including a -detailed diff of the options before and after the transition. `LITE` mode -outputs the same information without the options diff. +This output format is triggered by the `--transitions` flag which by default is set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs information about rule class transitions and attribute transitions including a detailed diff of the options before and after the transition. `LITE` mode outputs the same information without the options diff. ### Protocol message output @@ -319,27 +214,17 @@ outputs the same information without the options diff. --output=proto ``` -This option causes the resulting targets to be printed in a binary protocol -buffer form. The definition of the protocol buffer can be found at -[src/main/protobuf/analysis_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). +This option causes the resulting targets to be printed in a binary protocol buffer form. The definition of the protocol buffer can be found at [src/main/protobuf/analysis\_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). -`CqueryResult` is the top level message containing the results of the cquery. It -has a list of `ConfiguredTarget` messages and a list of `Configuration` -messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal -to that of the `id` field from the corresponding `Configuration` message. +`CqueryResult` is the top level message containing the results of the cquery. It has a list of `ConfiguredTarget` messages and a list of `Configuration` messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal to that of the `id` field from the corresponding `Configuration` message. -#### --[no]proto:include_configurations +#### --\[no]proto:include\_configurations -By default, cquery results return configuration information as part of each -configured target. If you'd like to omit this information and get proto output -that is formatted exactly like query's proto output, set this flag to false. +By default, cquery results return configuration information as part of each configured target. If you'd like to omit this information and get proto output that is formatted exactly like query's proto output, set this flag to false. -See [query's proto output documentation](/query/language#output-formats) -for more proto output-related options. +See [query's proto output documentation](/query/language#output-formats) for more proto output-related options. -Note: While selects are resolved both at the top level of returned -targets and within attributes, all possible inputs for selects are still -included as `rule_input` fields. +Note: While selects are resolved both at the top level of returned targets and within attributes, all possible inputs for selects are still included as `rule_input` fields. ### Graph output @@ -347,10 +232,7 @@ included as `rule_input` fields. --output=graph ``` -This option generates output as a Graphviz-compatible .dot file. See `query`'s -[graph output documentation](/query/language#display-result-graph) for details. `cquery` -also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and -[`--graph:factored`](/query/language#graph-factored). +This option generates output as a Graphviz-compatible .dot file. See `query`'s [graph output documentation](/query/language#display-result-graph) for details. `cquery` also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and [`--graph:factored`](/query/language#graph-factored). ### Files output @@ -358,23 +240,11 @@ also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and --output=files ``` -This option prints a list of the output files produced by each target matched -by the query similar to the list printed at the end of a `bazel build` -invocation. The output contains only the files advertised in the requested -output groups as determined by the -[`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. -It does include source files. +This option prints a list of the output files produced by each target matched by the query similar to the list printed at the end of a `bazel build` invocation. The output contains only the files advertised in the requested output groups as determined by the [`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. It does include source files. -All paths emitted by this output format are relative to the -[execroot](https://bazel.build/remote/output-directories), which can be obtained -via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, -paths to files in the main repository also resolve relative to the workspace -directory. +All paths emitted by this output format are relative to the [execroot](https://bazel.build/remote/output-directories), which can be obtained via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, paths to files in the main repository also resolve relative to the workspace directory. -Note: The output of `bazel cquery --output=files //pkg:foo` contains the output -files of `//pkg:foo` in *all* configurations that occur in the build (also see -the [section on target pattern evaluation](#target-pattern-evaluation)). If that -is not desired, wrap you query in [`config(..., target)`](#config). +Note: The output of `bazel cquery --output=files //pkg:foo` contains the output files of `//pkg:foo` in *all* configurations that occur in the build (also see the [section on target pattern evaluation](#target-pattern-evaluation)). If that is not desired, wrap you query in [`config(..., target)`](#config). ### Defining the output format using Starlark @@ -382,39 +252,21 @@ is not desired, wrap you query in [`config(..., target)`](#config). --output=starlark ``` -This output format calls a [Starlark](/rules/language) -function for each configured target in the query result, and prints the value -returned by the call. The `--starlark:file` flag specifies the location of a -Starlark file that defines a function named `format` with a single parameter, -`target`. This function is called for each [Target](/rules/lib/builtins/Target) -in the query result. Alternatively, for convenience, you may specify just the -body of a function declared as `def format(target): return expr` by using the -`--starlark:expr` flag. +This output format calls a [Starlark](/rules/language) function for each configured target in the query result, and prints the value returned by the call. The `--starlark:file` flag specifies the location of a Starlark file that defines a function named `format` with a single parameter, `target`. This function is called for each [Target](/rules/lib/builtins/Target) in the query result. Alternatively, for convenience, you may specify just the body of a function declared as `def format(target): return expr` by using the `--starlark:expr` flag. #### 'cquery' Starlark dialect -The cquery Starlark environment differs from a BUILD or .bzl file. It includes -all core Starlark -[built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), -plus a few cquery-specific ones described below, but not (for example) `glob`, -`native`, or `rule`, and it does not support load statements. +The cquery Starlark environment differs from a BUILD or .bzl file. It includes all core Starlark [built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), plus a few cquery-specific ones described below, but not (for example) `glob`, `native`, or `rule`, and it does not support load statements. -##### build_options(target) +##### build\_options(target) -`build_options(target)` returns a map whose keys are build option identifiers -(see [Configurations](/extending/config)) and whose values are their Starlark -values. Build options whose values are not legal Starlark values are omitted -from this map. +`build_options(target)` returns a map whose keys are build option identifiers (see [Configurations](/extending/config)) and whose values are their Starlark values. Build options whose values are not legal Starlark values are omitted from this map. -If the target is an input file, `build_options(target)` returns None, as input -file targets have a null configuration. +If the target is an input file, `build_options(target)` returns None, as input file targets have a null configuration. ##### providers(target) -`providers(target)` returns a map whose keys are names of -[providers](/extending/rules#providers) -(for example, `"DefaultInfo"`) and whose values are their Starlark values. -Providers whose values are not legal Starlark values are omitted from this map. +`providers(target)` returns a map whose keys are names of [providers](/extending/rules#providers) (for example, `"DefaultInfo"`) and whose values are their Starlark values. Providers whose values are not legal Starlark values are omitted from this map. #### Examples @@ -425,8 +277,7 @@ Print a space-separated list of the base names of all files produced by `//foo`: --starlark:expr="' '.join([f.basename for f in providers(target)['DefaultInfo'].files.to_list()])" ``` -Print a space-separated list of the paths of all files produced by **rule** targets in -`//bar` and its subpackages: +Print a space-separated list of the paths of all files produced by **rule** targets in `//bar` and its subpackages: ``` bazel cquery 'kind(rule, //bar/...)' --output=starlark \ @@ -454,8 +305,7 @@ Print the value of the command line option `--javacopt` when building `//foo`. --starlark:expr="build_options(target)['//command_line_option:javacopt']" ``` -Print the label of each target with exactly one output. This example uses -Starlark functions defined in a file. +Print the label of each target with exactly one output. This example uses Starlark functions defined in a file. ``` $ cat example.cquery @@ -472,8 +322,7 @@ Starlark functions defined in a file. $ bazel cquery //baz --output=starlark --starlark:file=example.cquery ``` -Print the label of each target which is strictly Python 3. This example uses -Starlark functions defined in a file. +Print the label of each target which is strictly Python 3. This example uses Starlark functions defined in a file. ``` $ cat example.cquery @@ -494,7 +343,7 @@ Extract a value from a user defined Provider. ``` $ cat some_package/my_rule.bzl - MyRuleInfo = provider(fields=\{"color": "the name of a color"\}) + MyRuleInfo = provider(fields={"color": "the name of a color"}) def _my_rule_impl(ctx): ... @@ -519,96 +368,51 @@ Extract a value from a user defined Provider. ## cquery vs. query -`cquery` and `query` complement each other and excel in -different niches. Consider the following to decide which is right for you: - -* `cquery` follows specific `select()` branches to - model the exact graph you build. `query` doesn't know which - branch the build chooses, so overapproximates by including all branches. -* `cquery`'s precision requires building more of the graph than - `query` does. Specifically, `cquery` - evaluates _configured targets_ while `query` only - evaluates _targets_. This takes more time and uses more memory. -* `cquery`'s interpretation of - the [query language](/query/language) introduces ambiguity - that `query` avoids. For example, - if `"//foo"` exists in two configurations, which one - should `cquery "deps(//foo)"` use? - The [`config`](#config) function can help with this. +`cquery` and `query` complement each other and excel in different niches. Consider the following to decide which is right for you: + +- `cquery` follows specific `select()` branches to model the exact graph you build. `query` doesn't know which branch the build chooses, so overapproximates by including all branches. +- `cquery`'s precision requires building more of the graph than `query` does. Specifically, `cquery` evaluates *configured targets* while `query` only evaluates *targets*. This takes more time and uses more memory. +- `cquery`'s interpretation of the [query language](/query/language) introduces ambiguity that `query` avoids. For example, if `"//foo"` exists in two configurations, which one should `cquery "deps(//foo)"` use? The [`config`](#config) function can help with this. ## Non-deterministic output -`cquery` does not automatically wipe the build graph from previous commands. -It's therefore prone to picking up results from past queries. +`cquery` does not automatically wipe the build graph from previous commands. It's therefore prone to picking up results from past queries. -For example, `genrule` exerts an exec transition on its `tools` attribute - -that is, it configures its tools in the [exec configuration] -(https://bazel.build/rules/rules#configurations). +For example, `genrule` exerts an exec transition on its `tools` attribute - that is, it configures its tools in the \[exec configuration] ([https://bazel.build/rules/rules#configurations](https://bazel.build/rules/rules#configurations)). You can see the lingering effects of that transition below. ``` -$ cat > foo/BUILD << /tmp/deps.svg +$ bazel query "allpaths(//foo, third_party/...)" --notool_deps --output graph | dot -Tsvg > /tmp/deps.svg ``` -Note: `dot` supports other image formats, just replace `svg` with the -format identifier, for example, `png`. +Note: `dot` supports other image formats, just replace `svg` with the format identifier, for example, `png`. When a dependency graph is big and complicated, it can be helpful start with a single path: @@ -62,8 +46,7 @@ $ bazel query "somepath(//foo:foo, third_party/zlib:zlibonly)" //third_party/zlib:zlibonly ``` -If you do not specify `--output graph` with `allpaths`, -you will get a flattened list of the dependency graph. +If you do not specify `--output graph` with `allpaths`, you will get a flattened list of the dependency graph. ``` $ bazel query "allpaths(//foo, third_party/...)" @@ -93,27 +76,15 @@ $ bazel query "allpaths(//foo, third_party/...)" ### Aside: implicit dependencies -The BUILD file for `//foo` never references -`//translations/tools:aggregator`. So, where's the direct dependency? +The BUILD file for `//foo` never references `//translations/tools:aggregator`. So, where's the direct dependency? -Certain rules include implicit dependencies on additional libraries or tools. -For example, to build a `genproto` rule, you need first to build the Protocol -Compiler, so every `genproto` rule carries an implicit dependency on the -protocol compiler. These dependencies are not mentioned in the build file, -but added in by the build tool. The full set of implicit dependencies is - currently undocumented. Using `--noimplicit_deps` allows you to filter out - these deps from your query results. For cquery, this will include resolved toolchains. +Certain rules include implicit dependencies on additional libraries or tools. For example, to build a `genproto` rule, you need first to build the Protocol Compiler, so every `genproto` rule carries an implicit dependency on the protocol compiler. These dependencies are not mentioned in the build file, but added in by the build tool. The full set of implicit dependencies is currently undocumented. Using `--noimplicit_deps` allows you to filter out these deps from your query results. For cquery, this will include resolved toolchains. ## Reverse dependencies -You might want to know the set of targets that depends on some target. For instance, -if you're going to change some code, you might want to know what other code -you're about to break. You can use `rdeps(u, x)` to find the reverse -dependencies of the targets in `x` within the transitive closure of `u`. +You might want to know the set of targets that depends on some target. For instance, if you're going to change some code, you might want to know what other code you're about to break. You can use `rdeps(u, x)` to find the reverse dependencies of the targets in `x` within the transitive closure of `u`. -Bazel's [Sky Query](/query/language#sky-query) -supports the `allrdeps` function which allows you to query reverse dependencies -in a universe you specify. +Bazel's [Sky Query](/query/language#sky-query) supports the `allrdeps` function which allows you to query reverse dependencies in a universe you specify. ## Miscellaneous uses @@ -123,33 +94,47 @@ You can use `bazel query` to analyze many dependency relationships. #### What packages exist beneath `foo`? -```bazel query 'foo/...' --output package``` +``` +bazel query 'foo/...' --output package +``` #### What rules are defined in the `foo` package? -```bazel query 'kind(rule, foo:*)' --output label_kind``` +``` +bazel query 'kind(rule, foo:*)' --output label_kind +``` #### What files are generated by rules in the `foo` package? -```bazel query 'kind("generated file", //foo:*)'``` +``` +bazel query 'kind("generated file", //foo:*)' +``` #### What targets are generated by starlark macro `foo`? -```bazel query 'attr(generator_function, foo, //path/to/search/...)'``` +``` +bazel query 'attr(generator_function, foo, //path/to/search/...)' +``` #### What's the set of BUILD files needed to build `//foo`? -```bazel query 'buildfiles(deps(//foo))' | cut -f1 -d:``` +``` +bazel query 'buildfiles(deps(//foo))' | cut -f1 -d: +``` #### What are the individual tests that a `test_suite` expands to? -```bazel query 'tests(//foo:smoke_tests)'``` +``` +bazel query 'tests(//foo:smoke_tests)' +``` #### Which of those are C++ tests? -```bazel query 'kind(cc_.*, tests(//foo:smoke_tests))'``` +``` +bazel query 'kind(cc_.*, tests(//foo:smoke_tests))' +``` -#### Which of those are small? Medium? Large? +#### Which of those are small? Medium? Large? ``` bazel query 'attr(size, small, tests(//foo:smoke_tests))' @@ -161,19 +146,27 @@ bazel query 'attr(size, large, tests(//foo:smoke_tests))' #### What are the tests beneath `foo` that match a pattern? -```bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))'``` +``` +bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))' +``` The pattern is a regex and is applied to the full name of the rule. It's similar to doing -```bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t'``` +``` +bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t' +``` #### What package contains file `path/to/file/bar.java`? -``` bazel query path/to/file/bar.java --output=package``` +``` + bazel query path/to/file/bar.java --output=package +``` #### What is the build label for `path/to/file/bar.java?` -```bazel query path/to/file/bar.java``` +``` +bazel query path/to/file/bar.java +``` #### What rule target(s) contain file `path/to/file/bar.java` as a source? @@ -186,28 +179,36 @@ bazel query "attr('srcs', $fullname, ${fullname//:*/}:*)" #### What packages does `foo` depend on? (What do I need to check out to build `foo`) -```bazel query 'buildfiles(deps(//foo:foo))' --output package``` +``` +bazel query 'buildfiles(deps(//foo:foo))' --output package +``` -Note: `buildfiles` is required in order to correctly obtain all files -referenced by `subinclude`; see the reference manual for details. +Note: `buildfiles` is required in order to correctly obtain all files referenced by `subinclude`; see the reference manual for details. #### What packages does the `foo` tree depend on, excluding `foo/contrib`? -```bazel query 'deps(foo/... except foo/contrib/...)' --output package``` +``` +bazel query 'deps(foo/... except foo/contrib/...)' --output package +``` ### What rule dependencies exist ... #### What genproto rules does bar depend upon? -```bazel query 'kind(genproto, deps(bar/...))'``` +``` +bazel query 'kind(genproto, deps(bar/...))' +``` #### Find the definition of some JNI (C++) library that is transitively depended upon by a Java binary rule in the servlet tree. -```bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location``` +``` +bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location +``` ##### ...Now find the definitions of all the Java binaries that depend on them -```bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in +``` +bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in let cls = kind(cc_.*library, deps($jbs)) in $jbs intersect allpaths($jbs, $cls)' ``` @@ -218,54 +219,67 @@ referenced by `subinclude`; see the reference manual for details. Source files: -```bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$``` +``` +bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$ +``` Generated files: -```bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$``` +``` +bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$ +``` -#### What is the complete set of Java source files required to build QUX's tests? +#### What is the complete set of Java source files required to build QUX's tests? Source files: -```bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` +``` +bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ +``` Generated files: -```bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` +``` +bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ +``` ### What differences in dependencies between X and Y exist ... #### What targets does `//foo` depend on that `//foo:foolib` does not? -```bazel query 'deps(//foo) except deps(//foo:foolib)'``` +``` +bazel query 'deps(//foo) except deps(//foo:foolib)' +``` -#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does _not_ depend on? +#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does *not* depend on? -```bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))'``` +``` +bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))' +``` ### Why does this dependency exist ... #### Why does `bar` depend on `groups2`? -```bazel query 'somepath(bar/...,groups2/...:*)'``` +``` +bazel query 'somepath(bar/...,groups2/...:*)' +``` -Once you have the results of this query, you will often find that a single -target stands out as being an unexpected or egregious and undesirable -dependency of `bar`. The query can then be further refined to: +Once you have the results of this query, you will often find that a single target stands out as being an unexpected or egregious and undesirable dependency of `bar`. The query can then be further refined to: #### Show me a path from `docker/updater:updater_systest` (a `py_test`) to some `cc_library` that it depends upon: -```bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in - somepath(docker/updater:updater_systest, $cc)'``` +``` +bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in + somepath(docker/updater:updater_systest, $cc)' +``` #### Why does library `//photos/frontend:lib` depend on two variants of the same library `//third_party/jpeglib` and `//third_party/jpeg`? -This query boils down to: "show me the subgraph of `//photos/frontend:lib` that -depends on both libraries". When shown in topological order, the last element -of the result is the most likely culprit. +This query boils down to: "show me the subgraph of `//photos/frontend:lib` that depends on both libraries". When shown in topological order, the last element of the result is the most likely culprit. -```bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) +``` +bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) intersect allpaths(//photos/frontend:lib, //third_party/jpeg)' //photos/frontend:lib @@ -277,41 +291,41 @@ of the result is the most likely culprit. //third_party/jpeg/img:renderer ``` -### What depends on ... +### What depends on ... #### What rules under bar depend on Y? -```bazel query 'bar/... intersect allpaths(bar/..., Y)'``` +``` +bazel query 'bar/... intersect allpaths(bar/..., Y)' +``` -Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X -depend on Y?" If expression X is non-trivial, it may be convenient to bind a -name to it using `let` to avoid duplication. +Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X depend on Y?" If expression X is non-trivial, it may be convenient to bind a name to it using `let` to avoid duplication. #### What targets directly depend on T, in T's package? -```bazel query 'same_pkg_direct_rdeps(T)'``` +``` +bazel query 'same_pkg_direct_rdeps(T)' +``` ### How do I break a dependency ... -{/* TODO find a convincing value of X to plug in here */} - #### What dependency paths do I have to break to make `bar` no longer depend on X? To output the graph to a `svg` file: -```bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg``` +``` +bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg +``` ### Misc #### How many sequential steps are there in the `//foo-tests` build? -Unfortunately, the query language can't currently give you the longest path -from x to y, but it can find the (or rather _a_) most distant node from the -starting point, or show you the _lengths_ of the longest path from x to every -y that it depends on. Use `maxrank`: +Unfortunately, the query language can't currently give you the longest path from x to y, but it can find the (or rather *a*) most distant node from the starting point, or show you the *lengths* of the longest path from x to every y that it depends on. Use `maxrank`: -```bazel query 'deps(//foo-tests)' --output maxrank | tail -1 -85 //third_party/zlib:zutil.c``` +``` +bazel query 'deps(//foo-tests)' --output maxrank | tail -1 +85 //third_party/zlib:zutil.c +``` -The result indicates that there exist paths of length 85 that must occur in -order in this build. +The result indicates that there exist paths of length 85 that must occur in order in this build. diff --git a/query/language.mdx b/query/language.mdx new file mode 100644 index 00000000..8f00fa6f --- /dev/null +++ b/query/language.mdx @@ -0,0 +1,872 @@ +--- +title: 'The Bazel Query Reference' +--- + +This page is the reference manual for the *Bazel Query Language* used when you use `bazel query` to analyze build dependencies. It also describes the output formats `bazel query` supports. + +For practical use cases, see the [Bazel Query How-To](/query/guide). + +## Additional query reference + +In addition to `query`, which runs on the post-loading phase target graph, Bazel includes *action graph query* and *configurable query*. + +### Action graph query + +The action graph query (`aquery`) operates on the post-analysis Configured Target Graph and exposes information about **Actions**, **Artifacts**, and their relationships. `aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs, outputs, and mnemonics. + +For more details, see the [aquery reference](/query/aquery). + +### Configurable query + +Traditional Bazel query runs on the post-loading phase target graph and therefore has no concept of configurations and their related concepts. Notably, it doesn't correctly resolve [select statements](/reference/be/functions#select) and instead returns all possible resolutions of selects. However, the configurable query environment, `cquery`, properly handles configurations but doesn't provide all of the functionality of this original query. + +For more details, see the [cquery reference](/query/cquery). + +## Examples + +How do people use `bazel query`? Here are typical examples: + +Why does the `//foo` tree depend on `//bar/baz`? Show a path: + +``` +somepath(foo/..., //bar/baz:all) +``` + +What C++ libraries do all the `foo` tests depend on that the `foo_bin` target does not? + +``` +kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo:foo_bin)) +``` + +## Tokens: The lexical syntax + +Expressions in the query language are composed of the following tokens: + +- **Keywords**, such as `let`. Keywords are the reserved words of the language, and each of them is described below. The complete set of keywords is: + + - [`except`](#set-operations) + + - [`in`](#variables) + + - [`intersect`](#set-operations) + + - [`let`](#variables) + + - [`set`](#set) + + - [`union`](#set-operations) + +- **Words**, such as "`foo/...`" or "`.*test rule`" or "`//bar/baz:all`". If a character sequence is "quoted" (begins and ends with a single-quote ' or begins and ends with a double-quote "), it is a word. If a character sequence is not quoted, it may still be parsed as a word. Unquoted words are sequences of characters drawn from the alphabet characters A-Za-z, the numerals 0-9, and the special characters `*/@.-_:$~[]` (asterisk, forward slash, at, period, hyphen, underscore, colon, dollar sign, tilde, left square brace, right square brace). However, unquoted words may not start with a hyphen `-` or asterisk `*` even though relative [target names](/concepts/labels#target-names) may start with those characters. As a special rule meant to simplify the handling of labels referring to external repositories, unquoted words that start with `@@` may contain `+` characters. + + Unquoted words also may not include the characters plus sign `+` or equals sign `=`, even though those characters are permitted in target names. When writing code that generates query expressions, target names should be quoted. + + Quoting *is* necessary when writing scripts that construct Bazel query expressions from user-supplied values. + + ``` + //foo:bar+wiz # WRONG: scanned as //foo:bar + wiz. + //foo:bar=wiz # WRONG: scanned as //foo:bar = wiz. + "//foo:bar+wiz" # OK. + "//foo:bar=wiz" # OK. + ``` + + Note that this quoting is in addition to any quoting that may be required by your shell, such as: + + ```posix-terminal + bazel query ' "//foo:bar=wiz" ' # single-quotes for shell, double-quotes for Bazel. + ``` + + Keywords and operators, when quoted, are treated as ordinary words. For example, `some` is a keyword but "some" is a word. Both `foo` and "foo" are words. + + However, be careful when using single or double quotes in target names. When quoting one or more target names, use only one type of quotes (either all single or all double quotes). + + The following are examples of what the Java query string will be: + + ``` + 'a"'a' # WRONG: Error message: unclosed quotation. + "a'"a" # WRONG: Error message: unclosed quotation. + '"a" + 'a'' # WRONG: Error message: unexpected token 'a' after query expression '"a" + ' + "'a' + "a"" # WRONG: Error message: unexpected token 'a' after query expression ''a' + ' + "a'a" # OK. + 'a"a' # OK. + '"a" + "a"' # OK + "'a' + 'a'" # OK + ``` + + We chose this syntax so that quote marks aren't needed in most cases. The (unusual) `".*test rule"` example needs quotes: it starts with a period and contains a space. Quoting `"cc_library"` is unnecessary but harmless. + +- **Punctuation**, such as parens `()`, period `.` and comma `,`. Words containing punctuation (other than the exceptions listed above) must be quoted. + +Whitespace characters outside of a quoted word are ignored. + +## Bazel query language concepts + +The Bazel query language is a language of expressions. Every expression evaluates to a **partially-ordered set** of targets, or equivalently, a **graph** (DAG) of targets. This is the only datatype. + +Set and graph refer to the same datatype, but emphasize different aspects of it, for example: + +- **Set:** The partial order of the targets is not interesting. +- **Graph:** The partial order of targets is significant. + +### Cycles in the dependency graph + +Build dependency graphs should be acyclic. + +The algorithms used by the query language are robust against cycles, and will not report cycles as errors. + +Note that the post-loading phase unconfigured target graph that `bazel query` operates over may contain cycles that do not exist in the configured target graph. Cycles in the configured target graph are detected and reported as errors by [`bazel cquery`](/query/cquery) and [`bazel aquery`](/query/aquery). + +### Implicit dependencies + +In addition to build dependencies that are defined explicitly in `BUILD` files, Bazel adds additional *implicit* dependencies to rules. Implicit dependencies may be defined by: + +- [Private attributes](/extending/rules#private_attributes_and_implicit_dependencies) +- [Toolchain requirements](/extending/toolchains#writing-rules-toolchains) + +By default, `bazel query` takes implicit dependencies into account when computing the query result. This behavior can be changed with the `--[no]implicit_deps` option. + +Note that, as query does not consider configurations, potential toolchain **implementations** are not considered dependencies, only the required toolchain types. See [toolchain documentation](/extending/toolchains#writing-rules-toolchains). + +### Soundness + +Bazel query language expressions operate over the build dependency graph, which is the graph implicitly defined by all rule declarations in all `BUILD` files. It is important to understand that this graph is somewhat abstract, and does not constitute a complete description of how to perform all the steps of a build. In order to perform a build, a *configuration* is required too; see the [configurations](/docs/user-manual#configurations) section of the User's Guide for more detail. + +The result of evaluating an expression in the Bazel query language is true *for all configurations*, which means that it may be a conservative over-approximation, and not exactly precise. If you use the query tool to compute the set of all source files needed during a build, it may report more than are actually necessary because, for example, the query tool will include all the files needed to support message translation, even though you don't intend to use that feature in your build. + +### On the preservation of graph order + +Operations preserve any ordering constraints inherited from their subexpressions. You can think of this as "the law of conservation of partial order". Consider an example: if you issue a query to determine the transitive closure of dependencies of a particular target, the resulting set is ordered according to the dependency graph. If you filter that set to include only the targets of `file` kind, the same *transitive* partial ordering relation holds between every pair of targets in the resulting subset - even though none of these pairs is actually directly connected in the original graph. (There are no file-file edges in the build dependency graph). + +However, while all operators *preserve* order, some operations, such as the [set operations](#set-operations) don't *introduce* any ordering constraints of their own. Consider this expression: + +``` +deps(x) union y +``` + +The order of the final result set is guaranteed to preserve all the ordering constraints of its subexpressions, namely, that all the transitive dependencies of `x` are correctly ordered with respect to each other. However, the query guarantees nothing about the ordering of the targets in `y`, nor about the ordering of the targets in `deps(x)` relative to those in `y` (except for those targets in `y` that also happen to be in `deps(x)`). + +Operators that introduce ordering constraints include: `allpaths`, `deps`, `rdeps`, `somepath`, and the target pattern wildcards `package:*`, `dir/...`, etc. + +### Sky query + +*Sky Query* is a mode of query that operates over a specified *universe scope*. + +#### Special functions available only in SkyQuery + +Sky Query mode has the additional query functions `allrdeps` and `rbuildfiles`. These functions operate over the entire universe scope (which is why they don't make sense for normal Query). + +#### Specifying a universe scope + +Sky Query mode is activated by passing the following two flags: (`--universe_scope` or `--infer_universe_scope`) and `--order_output=no`. `--universe_scope=<target_pattern1>,...,<target_patternN>` tells query to preload the transitive closure of the target pattern specified by the target patterns, which can be both additive and subtractive. All queries are then evaluated in this "scope". In particular, the [`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles) operators only return results from this scope. `--infer_universe_scope` tells Bazel to infer a value for `--universe_scope` from the query expression. This inferred value is the list of unique target patterns in the query expression, but this might not be what you want. For example: + +```posix-terminal +bazel query --infer_universe_scope --order_output=no "allrdeps(//my:target)" +``` + +The list of unique target patterns in this query expression is `["//my:target"]`, so Bazel treats this the same as the invocation: + +```posix-terminal +bazel query --universe_scope=//my:target --order_output=no "allrdeps(//my:target)" +``` + +But the result of that query with `--universe_scope` is only `//my:target`; none of the reverse dependencies of `//my:target` are in the universe, by construction! On the other hand, consider: + +```posix-terminal +bazel query --infer_universe_scope --order_output=no "tests(//a/... + b/...) intersect allrdeps(siblings(rbuildfiles(my/starlark/file.bzl)))" +``` + +This is a meaningful query invocation that is trying to compute the test targets in the [`tests`](#tests) expansion of the targets under some directories that transitively depend on targets whose definition uses a certain `.bzl` file. Here, `--infer_universe_scope` is a convenience, especially in the case where the choice of `--universe_scope` would otherwise require you to parse the query expression yourself. + +So, for query expressions that use universe-scoped operators like [`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles) be sure to use `--infer_universe_scope` only if its behavior is what you want. + +Sky Query has some advantages and disadvantages compared to the default query. The main disadvantage is that it cannot order its output according to graph order, and thus certain [output formats](#output-formats) are forbidden. Its advantages are that it provides two operators ([`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles)) that are not available in the default query. As well, Sky Query does its work by introspecting the [Skyframe](/reference/skyframe) graph, rather than creating a new graph, which is what the default implementation does. Thus, there are some circumstances in which it is faster and uses less memory. + +## Expressions: Syntax and semantics of the grammar + +This is the grammar of the Bazel query language, expressed in EBNF notation: + +```none +expr ::= <var>word</var> + | let <var>name</var> = <var>expr</var> in <var>expr</var> + | (<var>expr</var>) + | <var>expr</var> intersect <var>expr</var> + | <var>expr</var> ^ <var>expr</var> + | <var>expr</var> union <var>expr</var> + | <var>expr</var> + <var>expr</var> + | <var>expr</var> except <var>expr</var> + | <var>expr</var> - <var>expr</var> + | set(<var>word</var> *) + | <var>word</var> '(' <var>int</var> | <var>word</var> | <var>expr</var> ... ')' +``` + +The following sections describe each of the productions of this grammar in order. + +### Target patterns + +``` +expr ::= <var>word</var> +``` + +Syntactically, a *target pattern* is just a word. It's interpreted as an (unordered) set of targets. The simplest target pattern is a label, which identifies a single target (file or rule). For example, the target pattern `//foo:bar` evaluates to a set containing one element, the target, the `bar` rule. + +Target patterns generalize labels to include wildcards over packages and targets. For example, `foo/...:all` (or just `foo/...`) is a target pattern that evaluates to a set containing all *rules* in every package recursively beneath the `foo` directory; `bar/baz:all` is a target pattern that evaluates to a set containing all the rules in the `bar/baz` package, but not its subpackages. + +Similarly, `foo/...:*` is a target pattern that evaluates to a set containing all *targets* (rules *and* files) in every package recursively beneath the `foo` directory; `bar/baz:*` evaluates to a set containing all the targets in the `bar/baz` package, but not its subpackages. + +Because the `:*` wildcard matches files as well as rules, it's often more useful than `:all` for queries. Conversely, the `:all` wildcard (implicit in target patterns like `foo/...`) is typically more useful for builds. + +`bazel query` target patterns work the same as `bazel build` build targets do. For more details, see [Target Patterns](/docs/user-manual#target-patterns), or type `bazel help target-syntax`. + +Target patterns may evaluate to a singleton set (in the case of a label), to a set containing many elements (as in the case of `foo/...`, which has thousands of elements) or to the empty set, if the target pattern matches no targets. + +All nodes in the result of a target pattern expression are correctly ordered relative to each other according to the dependency relation. So, the result of `foo:*` is not just the set of targets in package `foo`, it is also the *graph* over those targets. (No guarantees are made about the relative ordering of the result nodes against other nodes.) For more details, see the [graph order](#graph-order) section. + +### Variables + +```none +expr ::= let <var>name</var> = <var>expr</var><sub>1</sub> in <var>expr</var><sub>2</sub> + | <var>$name</var> +``` + +The Bazel query language allows definitions of and references to variables. The result of evaluation of a `let` expression is the same as that of \{\{ '`' }}expr{{ '`' \}\}2, with all free occurrences of variable \{\{ '`' }}name{{ '`' \}\} replaced by the value of \{\{ '`' }}expr{{ '`' \}\}1. + +For example, `let v = foo/... in allpaths($v, //common) intersect $v` is equivalent to the `allpaths(foo/...,//common) intersect foo/...`. + +An occurrence of a variable reference `name` other than in an enclosing `let <var>name</var> = ...` expression is an error. In other words, top-level query expressions cannot have free variables. + +In the above grammar productions, `name` is like *word*, but with the additional constraint that it be a legal identifier in the C programming language. References to the variable must be prepended with the "$" character. + +Each `let` expression defines only a single variable, but you can nest them. + +Both [target patterns](#target-patterns) and variable references consist of just a single token, a word, creating a syntactic ambiguity. However, there is no semantic ambiguity, because the subset of words that are legal variable names is disjoint from the subset of words that are legal target patterns. + +Technically speaking, `let` expressions do not increase the expressiveness of the query language: any query expressible in the language can also be expressed without them. However, they improve the conciseness of many queries, and may also lead to more efficient query evaluation. + +### Parenthesized expressions + +```none +expr ::= (<var>expr</var>) +``` + +Parentheses associate subexpressions to force an order of evaluation. A parenthesized expression evaluates to the value of its argument. + +### Algebraic set operations: intersection, union, set difference + +```none +expr ::= <var>expr</var> intersect <var>expr</var> + | <var>expr</var> ^ <var>expr</var> + | <var>expr</var> union <var>expr</var> + | <var>expr</var> + <var>expr</var> + | <var>expr</var> except <var>expr</var> + | <var>expr</var> - <var>expr</var> +``` + +These three operators compute the usual set operations over their arguments. Each operator has two forms, a nominal form, such as `intersect`, and a symbolic form, such as `^`. Both forms are equivalent; the symbolic forms are quicker to type. (For clarity, the rest of this page uses the nominal forms.) + +For example, + +``` +foo/... except foo/bar/... +``` + +evaluates to the set of targets that match `foo/...` but not `foo/bar/...`. + +You can write the same query as: + +``` +foo/... - foo/bar/... +``` + +The `intersect` (`^`) and `union` (`+`) operations are commutative (symmetric); `except` (`-`) is asymmetric. The parser treats all three operators as left-associative and of equal precedence, so you might want parentheses. For example, the first two of these expressions are equivalent, but the third is not: + +``` +x intersect y union z +(x intersect y) union z +x intersect (y union z) +``` + +Important: Use parentheses where there is any danger of ambiguity in reading a query expression. + +### Read targets from an external source: set + +```none +expr ::= set(<var>word</var> *) +``` + +The `set(<var>a</var> <var>b</var> <var>c</var> ...)` operator computes the union of a set of zero or more [target patterns](#target-patterns), separated by whitespace (no commas). + +In conjunction with the Bourne shell's `$(...)` feature, `set()` provides a means of saving the results of one query in a regular text file, manipulating that text file using other programs (such as standard UNIX shell tools), and then introducing the result back into the query tool as a value for further processing. For example: + +```posix-terminal +bazel query deps(//my:target) --output=label | grep ... | sed ... | awk ... > foo + +bazel query "kind(cc_binary, set($(<foo)))" +``` + +In the next example,`kind(cc_library, deps(//some_dir/foo:main, 5))` is computed by filtering on the `maxrank` values using an `awk` program. + +```posix-terminal +bazel query 'deps(//some_dir/foo:main)' --output maxrank | awk '($1 < 5) { print $2;} ' > foo + +bazel query "kind(cc_library, set($(<foo)))" +``` + +In these examples, `$(<foo)` is a shorthand for `$(cat foo)`, but shell commands other than `cat` may be used too—such as the previous `awk` command. + +Note: `set()` introduces no graph ordering constraints, so path information may be lost when saving and reloading sets of nodes using it. For more details, see the [graph order](#graph-order) section below. + +## Functions + +```none +expr ::= <var>word</var> '(' <var>int</var> | <var>word</var> | <var>expr</var> ... ')' +``` + +The query language defines several functions. The name of the function determines the number and type of arguments it requires. The following functions are available: + +- [`allpaths`](#somepath-allpaths) +- [`attr`](#attr) +- [`buildfiles`](#buildfiles) +- [`rbuildfiles`](#rbuildfiles) +- [`deps`](#deps) +- [`filter`](#filter) +- [`kind`](#kind) +- [`labels`](#labels) +- [`loadfiles`](#loadfiles) +- [`rdeps`](#rdeps) +- [`allrdeps`](#allrdeps) +- [`same_pkg_direct_rdeps`](#same_pkg_direct_rdeps) +- [`siblings`](#siblings) +- [`some`](#some) +- [`somepath`](#somepath-allpaths) +- [`tests`](#tests) +- [`visible`](#visible) + +### Transitive closure of dependencies: deps + +```none +expr ::= deps(<var>expr</var>) + | deps(<var>expr</var>, <var>depth</var>) +``` + +The `deps(<var>x</var>)` operator evaluates to the graph formed by the transitive closure of dependencies of its argument set \{\{ '`' }}x{{ '`' \}\}. For example, the value of `deps(//foo)` is the dependency graph rooted at the single node `foo`, including all its dependencies. The value of `deps(foo/...)` is the dependency graphs whose roots are all rules in every package beneath the `foo` directory. In this context, 'dependencies' means only rule and file targets, therefore the `BUILD` and Starlark files needed to create these targets are not included here. For that you should use the [`buildfiles`](#buildfiles) operator. + +The resulting graph is ordered according to the dependency relation. For more details, see the section on [graph order](#graph-order). + +The `deps` operator accepts an optional second argument, which is an integer literal specifying an upper bound on the depth of the search. So `deps(foo:*, 0)` returns all targets in the `foo` package, while `deps(foo:*, 1)` further includes the direct prerequisites of any target in the `foo` package, and `deps(foo:*, 2)` further includes the nodes directly reachable from the nodes in `deps(foo:*, 1)`, and so on. (These numbers correspond to the ranks shown in the [`minrank`](#output-ranked) output format.) If the \{\{ '`' }}depth{{ '`' \}\} parameter is omitted, the search is unbounded: it computes the reflexive transitive closure of prerequisites. + +### Transitive closure of reverse dependencies: rdeps + +```none +expr ::= rdeps(<var>expr</var>, <var>expr</var>) + | rdeps(<var>expr</var>, <var>expr</var>, <var>depth</var>) +``` + +The `rdeps(<var>u</var>, <var>x</var>)` operator evaluates to the reverse dependencies of the argument set \{\{ '`' }}x{{ '`' \}\} within the transitive closure of the universe set \{\{ '`' }}u{{ '`' \}\}. + +The resulting graph is ordered according to the dependency relation. See the section on [graph order](#graph-order) for more details. + +The `rdeps` operator accepts an optional third argument, which is an integer literal specifying an upper bound on the depth of the search. The resulting graph only includes nodes within a distance of the specified depth from any node in the argument set. So `rdeps(//foo, //common, 1)` evaluates to all nodes in the transitive closure of `//foo` that directly depend on `//common`. (These numbers correspond to the ranks shown in the [`minrank`](#output-ranked) output format.) If the \{\{ '`' }}depth{{ '`' \}\} parameter is omitted, the search is unbounded. + +### Transitive closure of all reverse dependencies: allrdeps + +``` +expr ::= allrdeps(<var>expr</var>) + | allrdeps(<var>expr</var>, <var>depth</var>) +``` + +Note: Only available with [Sky Query](#sky-query) + +The `allrdeps` operator behaves just like the [`rdeps`](#rdeps) operator, except that the "universe set" is whatever the `--universe_scope` flag evaluated to, instead of being separately specified. Thus, if `--universe_scope=//foo/...` was passed, then `allrdeps(//bar)` is equivalent to `rdeps(//foo/..., //bar)`. + +### Direct reverse dependencies in the same package: same\_pkg\_direct\_rdeps + +``` +expr ::= same_pkg_direct_rdeps(<var>expr</var>) +``` + +The `same_pkg_direct_rdeps(<var>x</var>)` operator evaluates to the full set of targets that are in the same package as a target in the argument set, and which directly depend on it. + +### Dealing with a target's package: siblings + +``` +expr ::= siblings(<var>expr</var>) +``` + +The `siblings(<var>x</var>)` operator evaluates to the full set of targets that are in the same package as a target in the argument set. + +### Arbitrary choice: some + +``` +expr ::= some(<var>expr</var>) + | some(<var>expr</var>, <var>count</var>) +``` + +The `some(<var>x</var>, <var>k</var>)` operator selects at most \{\{ '`' }}k{{ '`' \}\} targets arbitrarily from its argument set \{\{ '`' }}x{{ '`' \}\}, and evaluates to a set containing only those targets. Parameter \{\{ '`' }}k{{ '`' \}\} is optional; if missing, the result will be a singleton set containing only one target arbitrarily selected. If the size of argument set \{\{ '`' }}x{{ '`' \}\} is smaller than \{\{ '`' }}k{{ '`' \}\}, the whole argument set \{\{ '`' }}x{{ '`' \}\} will be returned. + +For example, the expression `some(//foo:main union //bar:baz)` evaluates to a singleton set containing either `//foo:main` or `//bar:baz`—though which one is not defined. The expression `some(//foo:main union //bar:baz, 2)` or `some(//foo:main union //bar:baz, 3)` returns both `//foo:main` and `//bar:baz`. + +If the argument is a singleton, then `some` computes the identity function: `some(//foo:main)` is equivalent to `//foo:main`. + +It is an error if the specified argument set is empty, as in the expression `some(//foo:main intersect //bar:baz)`. + +### Path operators: somepath, allpaths + +``` +expr ::= somepath(<var>expr</var>, <var>expr</var>) + | allpaths(<var>expr</var>, <var>expr</var>) +``` + +The `somepath(<var>S</var>, <var>E</var>)` and `allpaths(<var>S</var>, <var>E</var>)` operators compute paths between two sets of targets. Both queries accept two arguments, a set \{\{ '`' }}S{{ '`' \}\} of starting points and a set \{\{ '`' }}E{{ '`' \}\} of ending points. `somepath` returns the graph of nodes on *some* arbitrary path from a target in \{\{ '`' }}S{{ '`' \}\} to a target in \{\{ '`' }}E{{ '`' \}\}; `allpaths` returns the graph of nodes on *all* paths from any target in \{\{ '`' }}S{{ '`' \}\} to any target in \{\{ '`' }}E{{ '`' \}\}. + +The resulting graphs are ordered according to the dependency relation. See the section on [graph order](#graph-order) for more details. + +| | | | +| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| ![Somepath](/docs/images/somepath1.svg)`somepath(S1 + S2, E)`, one possible result. | ![Somepath](/docs/images/somepath2.svg)`somepath(S1 + S2, E)`, another possible result. | ![Allpaths](/docs/images/allpaths.svg)`allpaths(S1 + S2, E)` | + +### Target kind filtering: kind + +``` +expr ::= kind(<var>word</var>, <var>expr</var>) +``` + +The `kind(<var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards those targets that are not of the expected kind. The \{\{ '`' }}pattern{{ '`' \}\} parameter specifies what kind of target to match. + +For example, the kinds for the four targets defined by the `BUILD` file (for package `p`) shown below are illustrated in the table: + +| Code | Target | Kind | +| ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -------------- | +| ``` + genrule( + name = "a", + srcs = ["a.in"], + outs = ["a.out"], + cmd = "...", + ) + +``` | `//p:a` | genrule rule | +| | `//p:a.in` | source file | +| | `//p:a.out` | generated file | +| | `//p:BUILD` | source file | + +Thus, `kind("cc_.* rule", foo/...)` evaluates to the set of all `cc_library`, `cc_binary`, etc, rule targets beneath `foo`, and `kind("source file", deps(//foo))` evaluates to the set of all source files in the transitive closure of dependencies of the `//foo` target. + +Quotation of the \{\{ '`' }}pattern{{ '`' \}\} argument is often required because without it, many [regular expressions](#regex), such as `source file` and `.*_test`, are not considered words by the parser. + +When matching for `package group`, targets ending in `:all` may not yield any results. Use `:all-targets` instead. + +### Target name filtering: filter + +``` +expr ::= filter(<var>word</var>, <var>expr</var>) +``` + +The `filter(<var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets whose labels (in absolute form) do not match the pattern; it evaluates to a subset of its input. + +The first argument, \{\{ '`' }}pattern{{ '`' \}\} is a word containing a [regular expression](#regex) over target names. A `filter` expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\} and the label (in absolute form, such as `//foo:bar`) of \{\{ '`' }}x{{ '`' \}\} contains an (unanchored) match for the regular expression \{\{ '`' }}pattern{{ '`' \}\}. Since all target names start with `//`, it may be used as an alternative to the `^` regular expression anchor. + +This operator often provides a much faster and more robust alternative to the `intersect` operator. For example, in order to see all `bar` dependencies of the `//foo:foo` target, one could evaluate + +``` +deps(//foo) intersect //bar/... +``` + +This statement, however, will require parsing of all `BUILD` files in the `bar` tree, which will be slow and prone to errors in irrelevant `BUILD` files. An alternative would be: + +``` +filter(//bar, deps(//foo)) +``` + +which would first calculate the set of `//foo` dependencies and then would filter only targets matching the provided pattern—in other words, targets with names containing `//bar` as a substring. + +Another common use of the `filter(<var>pattern</var>, <var>expr</var>)` operator is to filter specific files by their name or extension. For example, + +``` +filter("\.cc$", deps(//foo)) +``` + +will provide a list of all `.cc` files used to build `//foo`. + +### Rule attribute filtering: attr + +``` +expr ::= attr(<var>word</var>, <var>word</var>, <var>expr</var>) +``` + +The `attr(<var>name</var>, <var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets that aren't rules, rule targets that do not have attribute \{\{ '`' }}name{{ '`' \}\} defined or rule targets where the attribute value does not match the provided [regular expression](#regex) \{\{ '`' }}pattern{{ '`' \}\}; it evaluates to a subset of its input. + +The first argument, \{\{ '`' }}name{{ '`' \}\} is the name of the rule attribute that should be matched against the provided [regular expression](#regex) pattern. The second argument, \{\{ '`' }}pattern{{ '`' \}\} is a regular expression over the attribute values. An `attr` expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\}, is a rule with the defined attribute \{\{ '`' }}name{{ '`' \}\} and the attribute value contains an (unanchored) match for the regular expression \{\{ '`' }}pattern{{ '`' \}\}. If \{\{ '`' }}name{{ '`' \}\} is an optional attribute and rule does not specify it explicitly then default attribute value will be used for comparison. For example, + +``` +attr(linkshared, 0, deps(//foo)) +``` + +will select all `//foo` dependencies that are allowed to have a linkshared attribute (such as, `cc_binary` rule) and have it either explicitly set to 0 or do not set it at all but default value is 0 (such as for `cc_binary` rules). + +List-type attributes (such as `srcs`, `data`, etc) are converted to strings of the form `[value<sub>1</sub>, ..., value<sub>n</sub>]`, starting with a `[` bracket, ending with a `]` bracket and using "`, `" (comma, space) to delimit multiple values. Labels are converted to strings by using the absolute form of the label. For example, an attribute `deps=[":foo", "//otherpkg:bar", "wiz"]` would be converted to the string `[//thispkg:foo, //otherpkg:bar, //thispkg:wiz]`. Brackets are always present, so the empty list would use string value `[]` for matching purposes. For example, + +``` +attr("srcs", "\[\]", deps(//foo)) +``` + +will select all rules among `//foo` dependencies that have an empty `srcs` attribute, while + +``` +attr("data", ".{3,}", deps(//foo)) +``` + +will select all rules among `//foo` dependencies that specify at least one value in the `data` attribute (every label is at least 3 characters long due to the `//` and `:`). + +To select all rules among `//foo` dependencies with a particular `value` in a list-type attribute, use + +``` +attr("tags", "[\[ ]value[,\]]", deps(//foo)) +``` + +This works because the character before `value` will be `[` or a space and the character after `value` will be a comma or `]`. + +To select all rules among `//foo` dependencies with a particular `key` and `value` in a dict-type attribute, use + +``` +attr("some_dict_attribute", "[{ ]key=value[,}]", deps(//foo)) +``` + +This would select `//foo` if `//foo` is defined as + +``` +some_rule( + name = "foo", + some_dict_attribute = { + "key": "value", + }, +) +``` + +This works because the character before `key=value` will be `{` or a space and the character after `key=value` will be a comma or `}`. + +### Rule visibility filtering: visible + +``` +expr ::= visible(<var>expr</var>, <var>expr</var>) +``` + +The `visible(<var>predicate</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets without the required visibility. + +The first argument, \{\{ '`' }}predicate{{ '`' \}\}, is a set of targets that all targets in the output must be visible to. A \{\{ '`' }}visible{{ '`' \}\} expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\}, and for all targets \{\{ '`' }}y{{ '`' \}\} in \{\{ '`' }}predicate{{ '`' \}\} \{\{ '`' }}x{{ '`' \}\} is visible to \{\{ '`' }}y{{ '`' \}\}. For example: + +``` +visible(//foo, //bar:*) +``` + +will select all targets in the package `//bar` that `//foo` can depend on without violating visibility restrictions. + +### Evaluation of rule attributes of type label: labels + +``` +expr ::= labels(<var>word</var>, <var>expr</var>) +``` + +The `labels(<var>attr_name</var>, <var>inputs</var>)` operator returns the set of targets specified in the attribute \{\{ '`' }}attr_name{{ '`' \}\} of type "label" or "list of label" in some rule in set \{\{ '`' }}inputs{{ '`' \}\}. + +For example, `labels(srcs, //foo)` returns the set of targets appearing in the `srcs` attribute of the `//foo` rule. If there are multiple rules with `srcs` attributes in the \{\{ '`' }}inputs{{ '`' \}\} set, the union of their `srcs` is returned. + +### Expand and filter test\_suites: tests + +``` +expr ::= tests(<var>expr</var>) +``` + +The `tests(<var>x</var>)` operator returns the set of all test rules in set \{\{ '`' }}x{{ '`' \}\}, expanding any `test_suite` rules into the set of individual tests that they refer to, and applying filtering by `tag` and `size`. + +By default, query evaluation ignores any non-test targets in all `test_suite` rules. This can be changed to errors with the `--strict_test_suite` option. + +For example, the query `kind(test, foo:*)` lists all the `*_test` and `test_suite` rules in the `foo` package. All the results are (by definition) members of the `foo` package. In contrast, the query `tests(foo:*)` will return all of the individual tests that would be executed by `bazel test foo:*`: this may include tests belonging to other packages, that are referenced directly or indirectly via `test_suite` rules. + +### Package definition files: buildfiles + +``` +expr ::= buildfiles(<var>expr</var>) +``` + +The `buildfiles(<var>x</var>)` operator returns the set of files that define the packages of each target in set \{\{ '`' }}x{{ '`' \}\}; in other words, for each package, its `BUILD` file, plus any .bzl files it references via `load`. Note that this also returns the `BUILD` files of the packages containing these `load`ed files. + +This operator is typically used when determining what files or packages are required to build a specified target, often in conjunction with the [`--output package`](#output-package) option, below). For example, + +```posix-terminal +bazel query 'buildfiles(deps(//foo))' --output package +``` + +returns the set of all packages on which `//foo` transitively depends. + +Note: A naive attempt at the above query would omit the `buildfiles` operator and use only `deps`, but this yields an incorrect result: while the result contains the majority of needed packages, those packages that contain only files that are `load()`'ed will be missing. + +Warning: Bazel pretends each `.bzl` file produced by `buildfiles` has a corresponding target (for example, file `a/b.bzl` => target `//a:b.bzl`), but this isn't necessarily the case. Therefore, `buildfiles` doesn't compose well with other query operators and its results can be misleading when formatted in a structured way, such as [`--output=xml`](#xml). + +### Package definition files: rbuildfiles + +``` +expr ::= rbuildfiles(<var>word</var>, ...) +``` + +Note: Only available with [Sky Query](#sky-query). + +The `rbuildfiles` operator takes a comma-separated list of path fragments and returns the set of `BUILD` files that transitively depend on these path fragments. For instance, if `//foo` is a package, then `rbuildfiles(foo/BUILD)` will return the `//foo:BUILD` target. If the `foo/BUILD` file has `load('//bar:file.bzl'...` in it, then `rbuildfiles(bar/file.bzl)` will return the `//foo:BUILD` target, as well as the targets for any other `BUILD` files that load `//bar:file.bzl` + +The scope of the rbuildfiles operator is the universe specified by the `--universe_scope` flag. Files that do not correspond directly to `BUILD` files and `.bzl` files do not affect the results. For instance, source files (like `foo.cc`) are ignored, even if they are explicitly mentioned in the `BUILD` file. Symlinks, however, are respected, so that if `foo/BUILD` is a symlink to `bar/BUILD`, then `rbuildfiles(bar/BUILD)` will include `//foo:BUILD` in its results. + +The `rbuildfiles` operator is almost morally the inverse of the [`buildfiles`](#buildfiles) operator. However, this moral inversion holds more strongly in one direction: the outputs of `rbuildfiles` are just like the inputs of `buildfiles`; the former will only contain `BUILD` file targets in packages, and the latter may contain such targets. In the other direction, the correspondence is weaker. The outputs of the `buildfiles` operator are targets corresponding to all packages and .`bzl` files needed by a given input. However, the inputs of the `rbuildfiles` operator are not those targets, but rather the path fragments that correspond to those targets. + +### Package definition files: loadfiles + +``` +expr ::= loadfiles(<var>expr</var>) +``` + +The `loadfiles(<var>x</var>)` operator returns the set of Starlark files that are needed to load the packages of each target in set \{\{ '`' }}x{{ '`' \}\}. In other words, for each package, it returns the .bzl files that are referenced from its `BUILD` files. + +Warning: Bazel pretends each of these .bzl files has a corresponding target (for example, file `a/b.bzl` => target `//a:b.bzl`), but this isn't necessarily the case. Therefore, `loadfiles` doesn't compose well with other query operators and its results can be misleading when formatted in a structured way, such as [`--output=xml`](#xml). + +## Output formats + +`bazel query` generates a graph. You specify the content, format, and ordering by which `bazel query` presents this graph by means of the `--output` command-line option. + +When running with [Sky Query](#sky-query), only output formats that are compatible with unordered output are allowed. Specifically, `graph`, `minrank`, and `maxrank` output formats are forbidden. + +Some of the output formats accept additional options. The name of each output option is prefixed with the output format to which it applies, so `--graph:factored` applies only when `--output=graph` is being used; it has no effect if an output format other than `graph` is used. Similarly, `--xml:line_numbers` applies only when `--output=xml` is being used. + +### On the ordering of results + +Although query expressions always follow the "[law of conservation of graph order](#graph-order)", *presenting* the results may be done in either a dependency-ordered or unordered manner. This does **not** influence the targets in the result set or how the query is computed. It only affects how the results are printed to stdout. Moreover, nodes that are equivalent in the dependency order may or may not be ordered alphabetically. The `--order_output` flag can be used to control this behavior. (The `--[no]order_results` flag has a subset of the functionality of the `--order_output` flag and is deprecated.) + +The default value of this flag is `auto`, which prints results in **lexicographical order**. However, when `somepath(a,b)` is used, the results will be printed in `deps` order instead. + +When this flag is `no` and `--output` is one of `build`, `label`, `label_kind`, `location`, `package`, `proto`, or `xml`, the outputs will be printed in arbitrary order. **This is generally the fastest option**. It is not supported though when `--output` is one of `graph`, `minrank` or `maxrank`: with these formats, Bazel always prints results ordered by the dependency order or rank. + +When this flag is `deps`, Bazel prints results in some topological order—that is, dependents first and dependencies after. However, nodes that are unordered by the dependency order (because there is no path from either one to the other) may be printed in any order. + +When this flag is `full`, Bazel prints nodes in a fully deterministic (total) order. First, all nodes are sorted alphabetically. Then, each node in the list is used as the start of a post-order depth-first search in which outgoing edges to unvisited nodes are traversed in alphabetical order of the successor nodes. Finally, nodes are printed in the reverse of the order in which they were visited. + +Printing nodes in this order may be slower, so it should be used only when determinism is important. + +### Print the source form of targets as they would appear in BUILD + +``` +--output build +``` + +With this option, the representation of each target is as if it were hand-written in the BUILD language. All variables and function calls (such as glob, macros) are expanded, which is useful for seeing the effect of Starlark macros. Additionally, each effective rule reports a `generator_name` and/or `generator_function`) value, giving the name of the macro that was evaluated to produce the effective rule. + +Although the output uses the same syntax as `BUILD` files, it is not guaranteed to produce a valid `BUILD` file. + +### Print the label of each target + +``` +--output label +``` + +With this option, the set of names (or *labels*) of each target in the resulting graph is printed, one label per line, in topological order (unless `--noorder_results` is specified, see [notes on the ordering of results](#result-order)). (A topological ordering is one in which a graph node appears earlier than all of its successors.) Of course there are many possible topological orderings of a graph (*reverse postorder* is just one); which one is chosen is not specified. + +When printing the output of a `somepath` query, the order in which the nodes are printed is the order of the path. + +Caveat: in some corner cases, there may be two distinct targets with the same label; for example, a `sh_binary` rule and its sole (implicit) `srcs` file may both be called `foo.sh`. If the result of a query contains both of these targets, the output (in `label` format) will appear to contain a duplicate. When using the `label_kind` (see below) format, the distinction becomes clear: the two targets have the same name, but one has kind `sh_binary rule` and the other kind `source file`. + +### Print the label and kind of each target + +``` +--output label_kind +``` + +Like `label`, this output format prints the labels of each target in the resulting graph, in topological order, but it additionally precedes the label by the [*kind*](#kind) of the target. + +### Print targets in protocol buffer format + +``` +--output proto +``` + +Prints the query output as a [`QueryResult`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffer. + +### Print targets in length-delimited protocol buffer format + +``` +--output streamed_proto +``` + +Prints a [length-delimited](https://protobuf.dev/programming-guides/encoding/#size-limit) stream of [`Target`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffers. This is useful to *(i)* get around [size limitations](https://protobuf.dev/programming-guides/encoding/#size-limit) of protocol buffers when there are too many targets to fit in a single `QueryResult` or *(ii)* to start processing while Bazel is still outputting. + +### Print targets in text proto format + +``` +--output textproto +``` + +Similar to `--output proto`, prints the [`QueryResult`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffer but in [text format](https://protobuf.dev/reference/protobuf/textformat-spec/). + +### Print targets in ndjson format + +``` +--output streamed_jsonproto +``` + +Similar to `--output streamed_proto`, prints a stream of [`Target`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffers but in [ndjson](https://github.com/ndjson/ndjson-spec) format. + +### Print the label of each target, in rank order + +``` +--output minrank --output maxrank +``` + +Like `label`, the `minrank` and `maxrank` output formats print the labels of each target in the resulting graph, but instead of appearing in topological order, they appear in rank order, preceded by their rank number. These are unaffected by the result ordering `--[no]order_results` flag (see [notes on the ordering of results](#result-order)). + +There are two variants of this format: `minrank` ranks each node by the length of the shortest path from a root node to it. "Root" nodes (those which have no incoming edges) are of rank 0, their successors are of rank 1, etc. (As always, edges point from a target to its prerequisites: the targets it depends upon.) + +`maxrank` ranks each node by the length of the longest path from a root node to it. Again, "roots" have rank 0, all other nodes have a rank which is one greater than the maximum rank of all their predecessors. + +All nodes in a cycle are considered of equal rank. (Most graphs are acyclic, but cycles do occur simply because `BUILD` files contain erroneous cycles.) + +These output formats are useful for discovering how deep a graph is. If used for the result of a `deps(x)`, `rdeps(x)`, or `allpaths` query, then the rank number is equal to the length of the shortest (with `minrank`) or longest (with `maxrank`) path from `x` to a node in that rank. `maxrank` can be used to determine the longest sequence of build steps required to build a target. + +Note: The ranked output of a `somepath` query is basically meaningless because `somepath` doesn't guarantee to return either a shortest or a longest path, and it may include "transitive" edges from one path node to another that are not direct edges in original graph. + +For example, the graph on the left yields the outputs on the right when `--output minrank` and `--output maxrank` are specified, respectively. + +| | | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ![Out ranked](/docs/images/out-ranked.svg) | ``` + minrank + + 0 //c:c + 1 //b:b + 1 //a:a + 2 //b:b.cc + 2 //a:a.cc + </pre> +</td> +<td> + <pre> + maxrank + + 0 //c:c + 1 //b:b + 2 //a:a + 2 //b:b.cc + 3 //a:a.cc + </pre> +</td> + + +``` | + +### Print the location of each target + +``` +--output location +``` + +Like `label_kind`, this option prints out, for each target in the result, the target's kind and label, but it is prefixed by a string describing the location of that target, as a filename and line number. The format resembles the output of `grep`. Thus, tools that can parse the latter (such as Emacs or vi) can also use the query output to step through a series of matches, allowing the Bazel query tool to be used as a dependency-graph-aware "grep for BUILD files". + +The location information varies by target kind (see the [kind](#kind) operator). For rules, the location of the rule's declaration within the `BUILD` file is printed. For source files, the location of line 1 of the actual file is printed. For a generated file, the location of the rule that generates it is printed. (The query tool does not have sufficient information to find the actual location of the generated file, and in any case, it might not exist if a build has not yet been performed.) + +### Print the set of packages + +`--output package` + +This option prints the name of all packages to which some target in the result set belongs. The names are printed in lexicographical order; duplicates are excluded. Formally, this is a *projection* from the set of labels (package, target) onto packages. + +Packages in external repositories are formatted as `@repo//foo/bar` while packages in the main repository are formatted as `foo/bar`. + +In conjunction with the `deps(...)` query, this output option can be used to find the set of packages that must be checked out in order to build a given set of targets. + +### Display a graph of the result + +`--output graph` + +This option causes the query result to be printed as a directed graph in the popular AT\&T GraphViz format. Typically the result is saved to a file, such as `.png` or `.svg`. (If the `dot` program is not installed on your workstation, you can install it using the command `sudo apt-get install graphviz`.) See the example section below for a sample invocation. + +This output format is particularly useful for `allpaths`, `deps`, or `rdeps` queries, where the result includes a *set of paths* that cannot be easily visualized when rendered in a linear form, such as with `--output label`. + +By default, the graph is rendered in a *factored* form. That is, topologically-equivalent nodes are merged together into a single node with multiple labels. This makes the graph more compact and readable, because typical result graphs contain highly repetitive patterns. For example, a `java_library` rule may depend on hundreds of Java source files all generated by the same `genrule`; in the factored graph, all these files are represented by a single node. This behavior may be disabled with the `--nograph:factored` option. + +#### `--graph:node_limit <var>n</var>` + +The option specifies the maximum length of the label string for a graph node in the output. Longer labels will be truncated; -1 disables truncation. Due to the factored form in which graphs are usually printed, the node labels may be very long. GraphViz cannot handle labels exceeding 1024 characters, which is the default value of this option. This option has no effect unless `--output=graph` is being used. + +#### `--[no]graph:factored` + +By default, graphs are displayed in factored form, as explained [above](#output-graph). When `--nograph:factored` is specified, graphs are printed without factoring. This makes visualization using GraphViz impractical, but the simpler format may ease processing by other tools (such as grep). This option has no effect unless `--output=graph` is being used. + +### XML + +`--output xml` + +This option causes the resulting targets to be printed in an XML form. The output starts with an XML header such as this + +``` + <?xml version="1.0" encoding="UTF-8"?> + <query version="2"> +``` + +and then continues with an XML element for each target in the result graph, in topological order (unless [unordered results](#result-order) are requested), and then finishes with a terminating + +``` +</query> +``` + +Simple entries are emitted for targets of `file` kind: + +``` + <source-file name='//foo:foo_main.cc' .../> + <generated-file name='//foo:libfoo.so' .../> +``` + +But for rules, the XML is structured and contains definitions of all the attributes of the rule, including those whose value was not explicitly specified in the rule's `BUILD` file. + +Additionally, the result includes `rule-input` and `rule-output` elements so that the topology of the dependency graph can be reconstructed without having to know that, for example, the elements of the `srcs` attribute are forward dependencies (prerequisites) and the contents of the `outs` attribute are backward dependencies (consumers). + +`rule-input` elements for [implicit dependencies](#implicit_deps) are suppressed if `--noimplicit_deps` is specified. + +``` + <rule class='cc_binary rule' name='//foo:foo' ...> + <list name='srcs'> + <label value='//foo:foo_main.cc'/> + <label value='//foo:bar.cc'/> + ... + </list> + <list name='deps'> + <label value='//common:common'/> + <label value='//collections:collections'/> + ... + </list> + <list name='data'> + ... + </list> + <int name='linkstatic' value='0'/> + <int name='linkshared' value='0'/> + <list name='licenses'/> + <list name='distribs'> + <distribution value="INTERNAL" /> + </list> + <rule-input name="//common:common" /> + <rule-input name="//collections:collections" /> + <rule-input name="//foo:foo_main.cc" /> + <rule-input name="//foo:bar.cc" /> + ... + </rule> +``` + +Every XML element for a target contains a `name` attribute, whose value is the target's label, and a `location` attribute, whose value is the target's location as printed by the [`--output location`](#print-target-location). + +#### `--[no]xml:line_numbers` + +By default, the locations displayed in the XML output contain line numbers. When `--noxml:line_numbers` is specified, line numbers are not printed. + +#### `--[no]xml:default_values` + +By default, XML output does not include rule attribute whose value is the default value for that kind of attribute (for example, if it were not specified in the `BUILD` file, or the default value was provided explicitly). This option causes such attribute values to be included in the XML output. + +### Regular expressions + +Regular expressions in the query language use the Java regex library, so you can use the full syntax for [`java.util.regex.Pattern`](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html). + +### Querying with external repositories + +If the build depends on rules from [external repositories](/external/overview) then query results will include these dependencies. For example, if `//foo:bar` depends on `@other-repo//baz:lib`, then `bazel query 'deps(//foo:bar)'` will list `@other-repo//baz:lib` as a dependency. diff --git a/query/quickstart.mdx b/query/quickstart.mdx new file mode 100644 index 00000000..a17220fb --- /dev/null +++ b/query/quickstart.mdx @@ -0,0 +1,500 @@ +--- +title: 'Query quickstart' +--- + +This tutorial covers how to work with Bazel to trace dependencies in your code using a premade Bazel project. + +For language and `--output` flag details, see the [Bazel query reference](/query/language) and [Bazel cquery reference](/query/cquery) manuals. Get help in your IDE by typing `bazel help query` or `bazel help cquery` on the command line. + +## Objective + +This guide runs you through a set of basic queries you can use to learn more about your project's file dependencies. It is intended for new Bazel developers with a basic knowledge of how Bazel and `BUILD` files work. + +## Prerequisites + +Start by installing [Bazel](https://bazel.build/install), if you haven’t already. This tutorial uses Git for source control, so for best results, install [Git](https://github.com/git-guides/install-git) as well. + +To visualize dependency graphs, the tool called Graphviz is used, which you can [download](https://graphviz.org/download/) in order to follow along. + +### Get the sample project + +Next, retrieve the sample app from [Bazel's Examples repository](https://github.com/bazelbuild/examples) by running the following in your command-line tool of choice: + +```posix-terminal +git clone https://github.com/bazelbuild/examples.git +``` + +The sample project for this tutorial is in the `examples/query-quickstart` directory. + +## Getting started + +### What are Bazel queries? + +Queries help you to learn about a Bazel codebase by analyzing the relationships between `BUILD` files and examining the resulting output for useful information. This guide previews some basic query functions, but for more options see the [query guide](https://bazel.build/query/guide). Queries help you learn about dependencies in large scale projects without manually navigating through `BUILD` files. + +To run a query, open your command line terminal and enter: + +```posix-terminal +bazel query 'query_function' +``` + +### Scenario + +Imagine a scenario that delves into the relationship between Cafe Bazel and its respective chef. This Cafe exclusively sells pizza and mac & cheese. Take a look below at how the project is structured: + +``` +bazelqueryguide +├── BUILD +├── src +│ └── main +│ └── java +│ └── com +│ └── example +│ ├── customers +│ │ ├── Jenny.java +│ │ ├── Amir.java +│ │ └── BUILD +│ ├── dishes +│ │ ├── Pizza.java +│ │ ├── MacAndCheese.java +│ │ └── BUILD +│ ├── ingredients +│ │ ├── Cheese.java +│ │ ├── Tomatoes.java +│ │ ├── Dough.java +│ │ ├── Macaroni.java +│ │ └── BUILD +│ ├── restaurant +│ │ ├── Cafe.java +│ │ ├── Chef.java +│ │ └── BUILD +│ ├── reviews +│ │ ├── Review.java +│ │ └── BUILD +│ └── Runner.java +└── MODULE.bazel +``` + +Throughout this tutorial, unless directed otherwise, try not to look in the `BUILD` files to find the information you need and instead solely use the query function. + +A project consists of different packages that make up a Cafe. They are separated into: `restaurant`, `ingredients`, `dishes`, `customers`, and `reviews`. Rules within these packages define different components of the Cafe with various tags and dependencies. + +### Running a build + +This project contains a main method inside of `Runner.java` that you can execute to print out a menu of the Cafe. Build the project using Bazel with the command `bazel build` and use `:` to signal that the target is named `runner`. See [target names](https://bazel.build/concepts/labels#target-names) to learn how to reference targets. + +To build this project, paste this command into a terminal: + +```posix-terminal +bazel build :runner +``` + +Your output should look something like this if the build is successful. + +```bash +INFO: Analyzed target //:runner (49 packages loaded, 784 targets configured). +INFO: Found 1 target... +Target //:runner up-to-date: + bazel-bin/runner.jar + bazel-bin/runner +INFO: Elapsed time: 16.593s, Critical Path: 4.32s +INFO: 23 processes: 4 internal, 10 darwin-sandbox, 9 worker. +INFO: Build completed successfully, 23 total actions +``` + +After it has built successfully, run the application by pasting this command: + +```posix-terminal +bazel-bin/runner +``` + +```bash +--------------------- MENU ------------------------- + +Pizza - Cheesy Delicious Goodness +Macaroni & Cheese - Kid-approved Dinner + +---------------------------------------------------- +``` + +This leaves you with a list of the menu items given along with a short description. + +## Exploring targets + +The project lists ingredients and dishes in their own packages. To use a query to view the rules of a package, run the command `bazel query package/…` + +In this case, you can use this to look through the ingredients and dishes that this Cafe has by running: + +```posix-terminal +bazel query //src/main/java/com/example/dishes/... +``` + +```posix-terminal +bazel query //src/main/java/com/example/ingredients/... +``` + +If you query for the targets of the ingredients package, the output should look like: + +```bash +//src/main/java/com/example/ingredients:cheese +//src/main/java/com/example/ingredients:dough +//src/main/java/com/example/ingredients:macaroni +//src/main/java/com/example/ingredients:tomato +``` + +## Finding dependencies + +What targets does your runner rely on to run? + +Say you want to dive deeper into the structure of your project without prodding into the filesystem (which may be untenable for large projects). What rules does Cafe Bazel use? + +If, like in this example, the target for your runner is `runner`, discover the underlying dependencies of the target by running the command: + +```posix-terminal +bazel query --noimplicit_deps "deps(target)" +``` + +```posix-terminal +bazel query --noimplicit_deps "deps(:runner)" +``` + +```bash +//:runner +//:src/main/java/com/example/Runner.java +//src/main/java/com/example/dishes:MacAndCheese.java +//src/main/java/com/example/dishes:Pizza.java +//src/main/java/com/example/dishes:macAndCheese +//src/main/java/com/example/dishes:pizza +//src/main/java/com/example/ingredients:Cheese.java +//src/main/java/com/example/ingredients:Dough.java +//src/main/java/com/example/ingredients:Macaroni.java +//src/main/java/com/example/ingredients:Tomato.java +//src/main/java/com/example/ingredients:cheese +//src/main/java/com/example/ingredients:dough +//src/main/java/com/example/ingredients:macaroni +//src/main/java/com/example/ingredients:tomato +//src/main/java/com/example/restaurant:Cafe.java +//src/main/java/com/example/restaurant:Chef.java +//src/main/java/com/example/restaurant:cafe +//src/main/java/com/example/restaurant:chef +``` + +Note: Adding the flag `--noimplicit_deps` removes configurations and potential toolchains to simplify the list. When you omit this flag, Bazel returns implicit dependencies not specified in the `BUILD` file and clutters the output. + +In most cases, use the query function `deps()` to see individual output dependencies of a specific target. + +## Visualizing the dependency graph (optional) + +Note: This section uses Graphviz, so make sure to [download Graphviz](https://graphviz.org/download/) to follow along. + +The section describes how you can visualize the dependency paths for a specific query. [Graphviz](https://graphviz.org/) helps to see the path as a directed acyclic graph image as opposed to a flattened list. You can alter the display of the Bazel query graph by using various `--output` command line options. See [Output Formats](https://bazel.build/query/language#output-formats) for options. + +Start by running your desired query and add the flag `--noimplicit_deps` to remove excessive tool dependencies. Then, follow the query with the output flag and store the graph into a file called `graph.in` to create a text representation of the graph. + +To search for all dependencies of the target `:runner` and format the output as a graph: + +```posix-terminal +bazel query --noimplicit_deps 'deps(:runner)' --output graph > graph.in +``` + +This creates a file called `graph.in`, which is a text representation of the build graph. Graphviz uses `dot `– a tool that processes text into a visualization — to create a png: + +```posix-terminal +dot -Tpng < graph.in > graph.png +``` + +If you open up `graph.png`, you should see something like this. The graph below has been simplified to make the essential path details clearer in this guide. + +![Diagram showing a relationship from cafe to chef to the dishes: pizza and mac and cheese which diverges into the separate ingredients: cheese, tomatoes, dough, and macaroni.](images/query_graph1.png "Dependency graph") + +This helps when you want to see the outputs of the different query functions throughout this guide. + +## Finding reverse dependencies + +If instead you have a target you’d like to analyze what other targets use it, you can use a query to examine what targets depend on a certain rule. This is called a “reverse dependency”. Using `rdeps()` can be useful when editing a file in a codebase that you’re unfamiliar with, and can save you from unknowingly breaking other files which depended on it. + +For instance, you want to make some edits to the ingredient `cheese`. To avoid causing an issue for Cafe Bazel, you need to check what dishes rely on `cheese`. + +Caution: Since `ingredients` is its own package, you must use a different naming convention for the target `cheese` in the form of `//package:target`. Read more about referencing targets, or [Labels](https://bazel.build/concepts/labels). + +To see what targets depend on a particular target/package, you can use `rdeps(universe_scope, target)`. The `rdeps()` query function takes in at least two arguments: a `universe_scope` — the relevant directory — and a `target`. Bazel searches for the target’s reverse dependencies within the `universe_scope` provided. The `rdeps()` operator accepts an optional third argument: an integer literal specifying the upper bound on the depth of the search. + +Tip: To search within the whole scope of the project, set the `universe_scope` to `//...` + +To look for reverse dependencies of the target `cheese` within the scope of the entire project ‘//…’ run the command: + +```posix-terminal +bazel query "rdeps(universe_scope, target)" +``` + +``` +ex) bazel query "rdeps(//... , //src/main/java/com/example/ingredients:cheese)" +``` + +```bash +//:runner +//src/main/java/com/example/dishes:macAndCheese +//src/main/java/com/example/dishes:pizza +//src/main/java/com/example/ingredients:cheese +//src/main/java/com/example/restaurant:cafe +//src/main/java/com/example/restaurant:chef +``` + +The query return shows that cheese is relied on by both pizza and macAndCheese. What a surprise! + +## Finding targets based on tags + +Two customers walk into Bazel Cafe: Amir and Jenny. There is nothing known about them except for their names. Luckily, they have their orders tagged in the 'customers' `BUILD` file. How can you access this tag? + +Developers can tag Bazel targets with different identifiers, often for testing purposes. For instance, tags on tests can annotate a test's role in your debug and release process, especially for C++ and Python tests, which lack any runtime annotation ability. Using tags and size elements gives flexibility in assembling suites of tests based around a codebase’s check-in policy. + +In this example, the tags are either one of `pizza` or `macAndCheese` to represent the menu items. This command queries for targets that have tags matching your identifier within a certain package. + +``` +bazel query 'attr(tags, "pizza", //src/main/java/com/example/customers/...)' +``` + +This query returns all of the targets in the 'customers' package that have a tag of "pizza". + +### Test yourself + +Use this query to learn what Jenny wants to order. + +#### Answer + +Mac and Cheese + +## Adding a new dependency + +Cafe Bazel has expanded its menu — customers can now order a Smoothie! This specific smoothie consists of the ingredients `Strawberry` and `Banana`. + +First, add the ingredients that the smoothie depends on: `Strawberry.java` and `Banana.java`. Add the empty Java classes. + +**`src/main/java/com/example/ingredients/Strawberry.java`** + +```java +package com.example.ingredients; + +public class Strawberry { + +} +``` + +**`src/main/java/com/example/ingredients/Banana.java`** + +```java +package com.example.ingredients; + +public class Banana { + +} +``` + +Next, add `Smoothie.java` to the appropriate directory: `dishes`. + +**`src/main/java/com/example/dishes/Smoothie.java`** + +```java +package com.example.dishes; + +public class Smoothie { + public static final String DISH_NAME = "Smoothie"; + public static final String DESCRIPTION = "Yummy and Refreshing"; +} +``` + +Lastly, add these files as rules in the appropriate `BUILD` files. Create a new java library for each new ingredient, including its name, public visibility, and its newly created 'src' file. You should wind up with this updated `BUILD` file: + +**`src/main/java/com/example/ingredients/BUILD`** + +``` +java_library( + name = "cheese", + visibility = ["//visibility:public"], + srcs = ["Cheese.java"], +) + +java_library( + name = "dough", + visibility = ["//visibility:public"], + srcs = ["Dough.java"], +) + +java_library( + name = "macaroni", + visibility = ["//visibility:public"], + srcs = ["Macaroni.java"], +) + +java_library( + name = "tomato", + visibility = ["//visibility:public"], + srcs = ["Tomato.java"], +) + +java_library( + name = "strawberry", + visibility = ["//visibility:public"], + srcs = ["Strawberry.java"], +) + +java_library( + name = "banana", + visibility = ["//visibility:public"], + srcs = ["Banana.java"], +) +``` + +In the `BUILD` file for dishes, you want to add a new rule for `Smoothie`. Doing so includes the Java file created for `Smoothie` as a 'src' file, and the new rules you made for each ingredient of the smoothie. + +**`src/main/java/com/example/dishes/BUILD`** + +``` +java_library( + name = "macAndCheese", + visibility = ["//visibility:public"], + srcs = ["MacAndCheese.java"], + deps = [ + "//src/main/java/com/example/ingredients:cheese", + "//src/main/java/com/example/ingredients:macaroni", + ], +) + +java_library( + name = "pizza", + visibility = ["//visibility:public"], + srcs = ["Pizza.java"], + deps = [ + "//src/main/java/com/example/ingredients:cheese", + "//src/main/java/com/example/ingredients:dough", + "//src/main/java/com/example/ingredients:tomato", + ], +) + +java_library( + name = "smoothie", + visibility = ["//visibility:public"], + srcs = ["Smoothie.java"], + deps = [ + "//src/main/java/com/example/ingredients:strawberry", + "//src/main/java/com/example/ingredients:banana", + ], +) +``` + +Lastly, you want to include the smoothie as a dependency in the Chef’s `BUILD` file. + +**`src/main/java/com/example/restaurant/BUILD`** + +``` +java_library( + name = "chef", + visibility = ["//visibility:public"], + srcs = [ + "Chef.java", + ], + + deps = [ + "//src/main/java/com/example/dishes:macAndCheese", + "//src/main/java/com/example/dishes:pizza", + "//src/main/java/com/example/dishes:smoothie", + ], +) + +java_library( + name = "cafe", + visibility = ["//visibility:public"], + srcs = [ + "Cafe.java", + ], + deps = [ + ":chef", + ], +) +``` + +Build `cafe` again to confirm that there are no errors. If it builds successfully, congratulations! You’ve added a new dependency for the 'Cafe'. If not, look out for spelling mistakes and package naming. For more information about writing `BUILD` files see [BUILD Style Guide](https://bazel.build/build/style-guide). + +Now, visualize the new dependency graph with the addition of the `Smoothie` to compare with the previous one. For clarity, name the graph input as `graph2.in` and `graph2.png`. + +```posix-terminal +bazel query --noimplicit_deps 'deps(:runner)' --output graph > graph2.in +``` + +```posix-terminal +dot -Tpng < graph2.in > graph2.png +``` + +[![The same graph as the first one except now there is a spoke stemming from the chef target with smoothie which leads to banana and strawberry](images/query_graph2.png "Updated dependency graph")](images/query_graph2.png) + +Looking at `graph2.png`, you can see that `Smoothie` has no shared dependencies with other dishes but is just another target that the `Chef` relies on. + +## somepath() and allpaths() + +What if you want to query why one package depends on another package? Displaying a dependency path between the two provides the answer. + +Two functions can help you find dependency paths: `somepath()` and `allpaths()`. Given a starting target S and an end point E, find a path between S and E by using `somepath(S,E)`. + +Explore the differences between these two functions by looking at the relationships between the 'Chef' and 'Cheese' targets. There are different possible paths to get from one target to the other: + +- Chef → MacAndCheese → Cheese +- Chef → Pizza → Cheese + +`somepath()` gives you a single path out of the two options, whereas 'allpaths()' outputs every possible path. + +Using Cafe Bazel as an example, run the following: + +```posix-terminal +bazel query "somepath(//src/main/java/com/example/restaurant/..., //src/main/java/com/example/ingredients:cheese)" +``` + +```bash +//src/main/java/com/example/restaurant:cafe +//src/main/java/com/example/restaurant:chef +//src/main/java/com/example/dishes:macAndCheese +//src/main/java/com/example/ingredients:cheese +``` + +The output follows the first path of Cafe → Chef → MacAndCheese → Cheese. If instead you use `allpaths()`, you get: + +```posix-terminal +bazel query "allpaths(//src/main/java/com/example/restaurant/..., //src/main/java/com/example/ingredients:cheese)" +``` + +```bash +//src/main/java/com/example/dishes:macAndCheese +//src/main/java/com/example/dishes:pizza +//src/main/java/com/example/ingredients:cheese +//src/main/java/com/example/restaurant:cafe +//src/main/java/com/example/restaurant:chef +``` + +![Output path of cafe to chef to pizza,mac and cheese to cheese](images/query_graph3.png "Output path for dependency") + +The output of `allpaths()` is a little harder to read as it is a flattened list of the dependencies. Visualizing this graph using Graphviz makes the relationship clearer to understand. + +## Test yourself + +One of Cafe Bazel’s customers gave the restaurant's first review! Unfortunately, the review is missing some details such as the identity of the reviewer and what dish it’s referencing. Luckily, you can access this information with Bazel. The `reviews` package contains a program that prints a review from a mystery customer. Build and run it with: + +```posix-terminal +bazel build //src/main/java/com/example/reviews:review +``` + +```posix-terminal +bazel-bin/src/main/java/com/example/reviews/review +``` + +Going off Bazel queries only, try to find out who wrote the review, and what dish they were describing. + +#### Hint + +Check the tags and dependencies for useful information. + +#### Answer + +This review was describing the Pizza and Amir was the reviewer. If you look at what dependencies that this rule had using `bazel query --noimplicit\_deps 'deps(//src/main/java/com/example/reviews:review)'` The result of this command reveals that Amir is the reviewer! Next, since you know the reviewer is Amir, you can use the query function to seek which tag Amir has in the \`BUILD\` file to see what dish is there. The command `bazel query 'attr(tags, "pizza", //src/main/java/com/example/customers/...)'` output that Amir is the only customer that ordered a pizza and is the reviewer which gives us the answer. + +## Wrapping up + +Congratulations! You have now run several basic queries, which you can try out on own projects. To learn more about the query language syntax, refer to the [Query reference page](https://bazel.build/query/language). Want more advanced queries? The [Query guide](https://bazel.build/query/guide) showcases an in-depth list of more use cases than are covered in this guide. diff --git a/reference/flag-cheatsheet.mdx b/reference/flag-cheatsheet.mdx new file mode 100644 index 00000000..5caf5628 --- /dev/null +++ b/reference/flag-cheatsheet.mdx @@ -0,0 +1,91 @@ +--- +title: 'Bazel flag cheat sheet' +--- + +Navigating Bazel's extensive list of command line flags can be a challenge. This page focuses on the most crucial flags you'll need to know. + +**Tip:** Select the flag name in table to navigate to its entry in the command line reference. + +## Useful general options + +The following flags are meant to be set explicitly on the command line. + +| Flag | Description | +| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ### `--config` | You can organize flags in a **.bazelrc** file into configurations, like ones for debugging or release builds. Additional configuration groups can be selected with `--config=<group>`. | +| ### `--keep_going` | Bazel should try as much as possible to continue with build and test execution. By default, Bazel fails eagerly.``` +</td> +``` | +| ### `--remote_download_outputs` | When using remote execution or caching (both disk and remote), you can signal to Bazel that you want to download **all** (intermediate) build artifacts as follows:``` +--remote_download_outputs=all +```By default, Bazel only downloads top-level artifacts, such as the final binary, and intermediate artifacts that are necessary for local actions. | +| ### `--stamp` | Adds build info (user, timestamp) to binaries.**Note:** Because this increases build time, it's only intended for release builds. | + +## Uncover Build & Test Issues + +The following flags can help you better understand Bazel build or test errors. + +| Flag | Description | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| ### `--announce_rc` | Shows which flags are implicitly set through user-defined, machine-defined, or project-defined **.bazelrc** files. | +| ### `--auto_output_filter` | By default, Bazel tries to prevent log spam and does only print compiler warnings and Starlark debug output for packages and subpackages requested on the command line. To disable all filtering, set `--auto_output_filter=none`. | +| ### `--sandbox_debug` | Lets you drill into sandboxing errors. For details on why Bazel sandboxes builds by default and what gets sandboxed, see our [sandboxing documentation](https://bazel.build/docs/sandboxing).**Tip:** If you think the error might be caused by sandboxing, try turning sandboxing off temporarily.To do this, add `--spawn_strategy=local` to your command. | +| ### `--subcommands (-s)` | Displays a comprehensive list of every command that Bazel runs during a build, regardless of whether it succeeds or fails | + +## Startup + +Caution: Startup flags need to be passed before the command and cause a server restart. Toggle these flags with caution. + +| Flag | Description | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ### `--bazelrc` | You can specify default Bazel options in **.bazelrc** files. If multiple **.bazelrc** files exist, you can select which **.bazelrc** file is used by adding `--bazelrc=<path to the .bazelrc file>`.**Tip:** `--bazelrc=dev/null` disables the search for **.bazelrc** files.This is ideal for scenarios where you want to ensure a clean build environment, such as release builds, and prevent any unintended configuration changes from **.bazelrc** files | +| ### `--host_jvm_args` | Limits the amount of RAM the Bazel server uses.For example, the following limits the Bazel heap size to **3**GB:``` +--host_jvm_args=-Xmx3g +```**Note:** `-Xmx` is used to set the maximum heap size for the Java Virtual Machine (JVM). The heap is the area of memory where objects are allocated. The correct format for this option is `-Xmx<size>` , where `<size>` is the maximum heap size, specified with a unit such as:* m for megabytes +* g for gigabytes +* k for kilobytes | +| ### `--output_base` | Controls Bazel's output tree. Bazel doesn't store build outputs, including logs, within the source tree itself. Instead, it uses a distinct output tree for this purpose.**Tip:** Using multiple output bases in one Bazel workspace lets you run multiple Bazel servers concurrently. This can be useful when trying to avoid analysis thrashing. For more information, see [Choosing the output base](https://bazel.build/run/scripts#output-base-option). | + +## Bazel tests + +The following flags are related to Bazel test + +| Flag | Description | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| ### `--java_debug` | Causes Java tests to wait for a debugger connection before being executed. | +| ### `--runs_per_test` | The number of times to run tests. For example, to run tests N times, add `--runs_per_test=N`. This can be useful to debug flaky tests and see whether a fix causes a test to pass consistently. | +| ### `--test_filter` | This flag is particularly useful when iterating on a single test method, such as when a change you made breaks a test. Instead of re-running all the test methods in the test suite, you can focus solely on the specific test(s) that failed. This allows for faster feedback and more efficient debugging. This flag is often used in conjunction with `--test_output=streamed` for real-time test output. | +| ### `--test_output` | Specifies the output mode. By default, Bazel captures test output in local log files. When iterating on a broken test, you typically want to use `--test_output=streamed` to see the test output in real time. | + +## Bazel run + +The following flags are related to Bazel run. + +| Flag | Description | +| ----------------- | ---------------------------------------------------------------------------------------------------------- | +| ### `--run_under` | Changes how executables are invoked. For example `--run_under="strace -c"` is commonly used for debugging. | + +## User-specific bazelrc options + +The following flags are related to user-specific **.bazelrc** options. + +| Flag | Description | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ### `--disk_cache` | A path to a directory where Bazel can read and write actions and action outputs. If the directory doesn't exist, it will be created.You can share build artifacts between multiple branches or workspaces and speed up Bazel builds by adding `--disk_cache=<path>` to your command. | +| ### `--jobs` | The number of concurrent jobs to run.This is typically only required when using remote execution where a remote build cluster executes more jobs than you have cores locally. | +| ### `--local_resources` | Limits how much CPU or RAM is consumed by locally running actions.**Note:** This has no impact on the amount of CPU or RAM that the Bazel server itself consumes for tasks like analysis and build orchestration. | +| ### `--sandbox_base` | Lets the sandbox create its sandbox directories underneath this path. By default, Bazel executes local actions sandboxed which adds some overhead to the build.**Tip:** Specify a path on tmpfs, for example `/run/shm`, to possibly improve performance a lot when your build or tests have many input files. | + +## Project-specific bazelrc options + +The following flags are related to project-specific **.bazelrc** options. + +| Flag | Description | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| ### `--flaky_test_attempts` | Retry each test up to the specified number of times in case of any test failure. This is especially useful on Continuous Integration. Tests that require more than one attempt to pass are marked as **FLAKY** in the test summary. | +| ### `--remote_cache` | A URI of a caching endpoint. Setting up remote caching can be a great way to speed up Bazel builds. It can be combined with a local disk cache. | +| ### `--remote_download_regex` | Force remote build outputs whose path matches this pattern to be downloaded, irrespective of the `--remote_download_outputs` setting. Multiple patterns may be specified by repeating this flag. | +| ### `--remote_executor` | `HOST` or `HOST:PORT` of a remote execution endpoint. Pass this if you are using a remote execution service. You'll often need to Add `--remote_instance_name=<name>`. | +| ### `--remote_instance_name` | The value to pass as `instance_name` in the remote execution API. | +| ### `--show_timestamps` | If specified, a timestamp is added to each message generated by Bazel specifying the time at which the message was displayed. This is useful on CI systems to quickly understand what step took how long. | +| ### `--spawn_strategy` | Even with remote execution, running some build actions locally might be faster. This depends on factors like your build cluster's capacity, network speed, and network delays.**Tip:** To run actions both locally and remotely and accept the faster result add `--spawn_strategy=dynamic` to your build command. | diff --git a/reference/glossary.mdx b/reference/glossary.mdx index da7478c3..f045f9e6 100644 --- a/reference/glossary.mdx +++ b/reference/glossary.mdx @@ -2,289 +2,155 @@ title: 'Bazel Glossary' --- - - ### Action -A command to run during the build, for example, a call to a compiler that takes -[artifacts](#artifact) as inputs and produces other artifacts as outputs. -Includes metadata like the command line arguments, action key, environment -variables, and declared input/output artifacts. +A command to run during the build, for example, a call to a compiler that takes [artifacts](#artifact) as inputs and produces other artifacts as outputs. Includes metadata like the command line arguments, action key, environment variables, and declared input/output artifacts. **See also:** [Rules documentation](/extending/rules#actions) ### Action cache -An on-disk cache that stores a mapping of executed [actions](#action) to the -outputs they created. The cache key is known as the [action key](#action-key). A -core component for Bazel's incrementality model. The cache is stored in the -output base directory and thus survives Bazel server restarts. +An on-disk cache that stores a mapping of executed [actions](#action) to the outputs they created. The cache key is known as the [action key](#action-key). A core component for Bazel's incrementality model. The cache is stored in the output base directory and thus survives Bazel server restarts. ### Action graph -An in-memory graph of [actions](#action) and the [artifacts](#artifact) that -these actions read and generate. The graph might include artifacts that exist as -source files (for example, in the file system) as well as generated -intermediate/final artifacts that are not mentioned in `BUILD` files. Produced -during the [analysis phase](#analysis-phase) and used during the [execution -phase](#execution-phase). +An in-memory graph of [actions](#action) and the [artifacts](#artifact) that these actions read and generate. The graph might include artifacts that exist as source files (for example, in the file system) as well as generated intermediate/final artifacts that are not mentioned in `BUILD` files. Produced during the [analysis phase](#analysis-phase) and used during the [execution phase](#execution-phase). ### Action graph query (aquery) -A [query](#query-concept) tool that can query over build [actions](#action). -This provides the ability to analyze how [build rules](#rule) translate into the -actual work builds do. +A [query](#query-concept) tool that can query over build [actions](#action). This provides the ability to analyze how [build rules](#rule) translate into the actual work builds do. ### Action key -The cache key of an [action](#action). Computed based on action metadata, which -might include the command to be executed in the action, compiler flags, library -locations, or system headers, depending on the action. Enables Bazel to cache or -invalidate individual actions deterministically. +The cache key of an [action](#action). Computed based on action metadata, which might include the command to be executed in the action, compiler flags, library locations, or system headers, depending on the action. Enables Bazel to cache or invalidate individual actions deterministically. ### Analysis phase -The second phase of a build. Processes the [target graph](#target-graph) -specified in [`BUILD` files](#build-file) to produce an in-memory [action -graph](#action-graph) that determines the order of actions to run during the -[execution phase](#execution-phase). This is the phase in which rule -implementations are evaluated. +The second phase of a build. Processes the [target graph](#target-graph) specified in [`BUILD` files](#build-file) to produce an in-memory [action graph](#action-graph) that determines the order of actions to run during the [execution phase](#execution-phase). This is the phase in which rule implementations are evaluated. ### Artifact -A source file or a generated file. Can also be a directory of files, known as -[tree artifacts](#tree-artifact). +A source file or a generated file. Can also be a directory of files, known as [tree artifacts](#tree-artifact). -An artifact may be an input to multiple actions, but must only be generated by -at most one action. +An artifact may be an input to multiple actions, but must only be generated by at most one action. -An artifact that corresponds to a [file target](#target) can be addressed by a -label. +An artifact that corresponds to a [file target](#target) can be addressed by a label. ### Aspect -A mechanism for rules to create additional [actions](#action) in their -dependencies. For example, if target A depends on B, one can apply an aspect on -A that traverses *up* a dependency edge to B, and runs additional actions in B -to generate and collect additional output files. These additional actions are -cached and reused between targets requiring the same aspect. Created with the -`aspect()` Starlark Build API function. Can be used, for example, to generate -metadata for IDEs, and create actions for linting. +A mechanism for rules to create additional [actions](#action) in their dependencies. For example, if target A depends on B, one can apply an aspect on A that traverses *up* a dependency edge to B, and runs additional actions in B to generate and collect additional output files. These additional actions are cached and reused between targets requiring the same aspect. Created with the `aspect()` Starlark Build API function. Can be used, for example, to generate metadata for IDEs, and create actions for linting. **See also:** [Aspects documentation](/extending/aspects) ### Aspect-on-aspect -A composition mechanism whereby aspects can be applied to the results -of other aspects. For example, an aspect that generates information for use by -IDEs can be applied on top of an aspect that generates `.java` files from a -proto. +A composition mechanism whereby aspects can be applied to the results of other aspects. For example, an aspect that generates information for use by IDEs can be applied on top of an aspect that generates `.java` files from a proto. -For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that -`B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute -must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) -attribute. +For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that `B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) attribute. ### Attribute -A parameter to a [rule](#rule), used to express per-target build information. -Examples include `srcs`, `deps`, and `copts`, which respectively declare a -target's source files, dependencies, and custom compiler options. The particular -attributes available for a given target depend on its rule type. +A parameter to a [rule](#rule), used to express per-target build information. Examples include `srcs`, `deps`, and `copts`, which respectively declare a target's source files, dependencies, and custom compiler options. The particular attributes available for a given target depend on its rule type. ### .bazelrc -Bazel’s configuration file used to change the default values for [startup -flags](#startup-flags) and [command flags](#command-flags), and to define common -groups of options that can then be set together on the Bazel command line using -a `--config` flag. Bazel can combine settings from multiple bazelrc files -(systemwide, per-workspace, per-user, or from a custom location), and a -`bazelrc` file may also import settings from other `bazelrc` files. +Bazel’s configuration file used to change the default values for [startup flags](#startup-flags) and [command flags](#command-flags), and to define common groups of options that can then be set together on the Bazel command line using a `--config` flag. Bazel can combine settings from multiple bazelrc files (systemwide, per-workspace, per-user, or from a custom location), and a `bazelrc` file may also import settings from other `bazelrc` files. ### Blaze -The Google-internal version of Bazel. Google’s main build system for its -mono-repository. +The Google-internal version of Bazel. Google’s main build system for its mono-repository. ### BUILD File -A `BUILD` file is the main configuration file that tells Bazel what software -outputs to build, what their dependencies are, and how to build them. Bazel -takes a `BUILD` file as input and uses the file to create a graph of dependencies -and to derive the actions that must be completed to build intermediate and final -software outputs. A `BUILD` file marks a directory and any sub-directories not -containing a `BUILD` file as a [package](#package), and can contain -[targets](#target) created by [rules](#rule). The file can also be named -`BUILD.bazel`. +A `BUILD` file is the main configuration file that tells Bazel what software outputs to build, what their dependencies are, and how to build them. Bazel takes a `BUILD` file as input and uses the file to create a graph of dependencies and to derive the actions that must be completed to build intermediate and final software outputs. A `BUILD` file marks a directory and any sub-directories not containing a `BUILD` file as a [package](#package), and can contain [targets](#target) created by [rules](#rule). The file can also be named `BUILD.bazel`. ### BUILD.bazel File -See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same -directory. +See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same directory. ### .bzl File -A file that defines rules, [macros](#macro), and constants written in -[Starlark](#starlark). These can then be imported into [`BUILD` -files](#build-file) using the `load()` function. - -{/* TODO: ### Build event protocol */} - -{/* TODO: ### Build flag */} +A file that defines rules, [macros](#macro), and constants written in [Starlark](#starlark). These can then be imported into [`BUILD` files](#build-file) using the `load()` function. ### Build graph -The dependency graph that Bazel constructs and traverses to perform a build. -Includes nodes like [targets](#target), [configured -targets](#configured-target), [actions](#action), and [artifacts](#artifact). A -build is considered complete when all [artifacts](#artifact) on which a set of -requested targets depend are verified as up-to-date. +The dependency graph that Bazel constructs and traverses to perform a build. Includes nodes like [targets](#target), [configured targets](#configured-target), [actions](#action), and [artifacts](#artifact). A build is considered complete when all [artifacts](#artifact) on which a set of requested targets depend are verified as up-to-date. ### Build setting -A Starlark-defined piece of [configuration](#configuration). -[Transitions](#transition) can set build settings to change a subgraph's -configuration. If exposed to the user as a [command-line flag](#command-flags), -also known as a build flag. +A Starlark-defined piece of [configuration](#configuration). [Transitions](#transition) can set build settings to change a subgraph's configuration. If exposed to the user as a [command-line flag](#command-flags), also known as a build flag. ### Clean build -A build that doesn't use the results of earlier builds. This is generally slower -than an [incremental build](#incremental-build) but commonly considered to be -more [correct](#correctness). Bazel guarantees both clean and incremental builds -are always correct. +A build that doesn't use the results of earlier builds. This is generally slower than an [incremental build](#incremental-build) but commonly considered to be more [correct](#correctness). Bazel guarantees both clean and incremental builds are always correct. ### Client-server model -The `bazel` command-line client automatically starts a background server on the -local machine to execute Bazel [commands](#command). The server persists across -commands but automatically stops after a period of inactivity (or explicitly via -bazel shutdown). Splitting Bazel into a server and client helps amortize JVM -startup time and supports faster [incremental builds](#incremental-build) -because the [action graph](#action-graph) remains in memory across commands. +The `bazel` command-line client automatically starts a background server on the local machine to execute Bazel [commands](#command). The server persists across commands but automatically stops after a period of inactivity (or explicitly via bazel shutdown). Splitting Bazel into a server and client helps amortize JVM startup time and supports faster [incremental builds](#incremental-build) because the [action graph](#action-graph) remains in memory across commands. ### Command -Used on the command line to invoke different Bazel functions, like `bazel -build`, `bazel test`, `bazel run`, and `bazel query`. +Used on the command line to invoke different Bazel functions, like `bazel build`, `bazel test`, `bazel run`, and `bazel query`. ### Command flags -A set of flags specific to a [command](#command). Command flags are specified -*after* the command (`bazel build `). Flags can be applicable to -one or more commands. For example, `--configure` is a flag exclusively for the -`bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, -`test` and more. Flags are often used for [configuration](#configuration) -purposes, so changes in flag values can cause Bazel to invalidate in-memory -graphs and restart the [analysis phase](#analysis-phase). +A set of flags specific to a [command](#command). Command flags are specified *after* the command (`bazel build <command flags>`). Flags can be applicable to one or more commands. For example, `--configure` is a flag exclusively for the `bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, `test` and more. Flags are often used for [configuration](#configuration) purposes, so changes in flag values can cause Bazel to invalidate in-memory graphs and restart the [analysis phase](#analysis-phase). ### Configuration -Information outside of [rule](#rule) definitions that impacts how rules generate -[actions](#action). Every build has at least one configuration specifying the -target platform, action environment variables, and command-line [build -flags](#command-flags). [Transitions](#transition) may create additional -configurations, such as for host tools or cross-compilation. +Information outside of [rule](#rule) definitions that impacts how rules generate [actions](#action). Every build has at least one configuration specifying the target platform, action environment variables, and command-line [build flags](#command-flags). [Transitions](#transition) may create additional configurations, such as for host tools or cross-compilation. **See also:** [Configurations](/extending/rules#configurations) -{/* TODO: ### Configuration fragment */} - ### Configuration trimming -The process of only including the pieces of [configuration](#configuration) a -target actually needs. For example, if you build Java binary `//:j` with C++ -dependency `//:c`, it's wasteful to include the value of `--javacopt` in the -configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ -build cacheability. +The process of only including the pieces of [configuration](#configuration) a target actually needs. For example, if you build Java binary `//:j` with C++ dependency `//:c`, it's wasteful to include the value of `--javacopt` in the configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ build cacheability. ### Configured query (cquery) -A [query](#query-concept) tool that queries over [configured -targets](#configured-target) (after the [analysis phase](#analysis-phase) -completes). This means `select()` and [build flags](#command-flags) (such as -`--platforms`) are accurately reflected in the results. +A [query](#query-concept) tool that queries over [configured targets](#configured-target) (after the [analysis phase](#analysis-phase) completes). This means `select()` and [build flags](#command-flags) (such as `--platforms`) are accurately reflected in the results. **See also:** [cquery documentation](/query/cquery) ### Configured target -The result of evaluating a [target](#target) with a -[configuration](#configuration). The [analysis phase](#analysis-phase) produces -this by combining the build's options with the targets that need to be built. -For example, if `//:foo` builds for two different architectures in the same -build, it has two configured targets: `` and ``. +The result of evaluating a [target](#target) with a [configuration](#configuration). The [analysis phase](#analysis-phase) produces this by combining the build's options with the targets that need to be built. For example, if `//:foo` builds for two different architectures in the same build, it has two configured targets: `<//:foo, x86>` and `<//:foo, arm>`. ### Correctness -A build is correct when its output faithfully reflects the state of its -transitive inputs. To achieve correct builds, Bazel strives to be -[hermetic](#hermeticity), reproducible, and making [build -analysis](#analysis-phase) and [action execution](#execution-phase) -deterministic. +A build is correct when its output faithfully reflects the state of its transitive inputs. To achieve correct builds, Bazel strives to be [hermetic](#hermeticity), reproducible, and making [build analysis](#analysis-phase) and [action execution](#execution-phase) deterministic. ### Dependency -A directed edge between two [targets](#target). A target `//:foo` has a *target -dependency* on target `//:bar` if `//:foo`'s attribute values contain a -reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an -action in `//:foo` depends on an input [artifact](#artifact) created by an -action in `//:bar`. +A directed edge between two [targets](#target). A target `//:foo` has a *target dependency* on target `//:bar` if `//:foo`'s attribute values contain a reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an action in `//:foo` depends on an input [artifact](#artifact) created by an action in `//:bar`. -In certain contexts, it could also refer to an _external dependency_; see -[modules](#module). +In certain contexts, it could also refer to an *external dependency*; see [modules](#module). ### Depset -A data structure for collecting data on transitive dependencies. Optimized so -that merging depsets is time and space efficient, because it’s common to have -very large depsets (hundreds of thousands of files). Implemented to -recursively refer to other depsets for space efficiency reasons. [Rule](#rule) -implementations should not "flatten" depsets by converting them to lists unless -the rule is at the top level of the build graph. Flattening large depsets incurs -huge memory consumption. Also known as *nested sets* in Bazel's internal -implementation. +A data structure for collecting data on transitive dependencies. Optimized so that merging depsets is time and space efficient, because it’s common to have very large depsets (hundreds of thousands of files). Implemented to recursively refer to other depsets for space efficiency reasons. [Rule](#rule) implementations should not "flatten" depsets by converting them to lists unless the rule is at the top level of the build graph. Flattening large depsets incurs huge memory consumption. Also known as *nested sets* in Bazel's internal implementation. **See also:** [Depset documentation](/extending/depsets) ### Disk cache -A local on-disk blob store for the remote caching feature. Can be used in -conjunction with an actual remote blob store. +A local on-disk blob store for the remote caching feature. Can be used in conjunction with an actual remote blob store. ### Distdir -A read-only directory containing files that Bazel would otherwise fetch from the -internet using repository rules. Enables builds to run fully offline. +A read-only directory containing files that Bazel would otherwise fetch from the internet using repository rules. Enables builds to run fully offline. ### Dynamic execution -An execution strategy that selects between local and remote execution based on -various heuristics, and uses the execution results of the faster successful -method. Certain [actions](#action) are executed faster locally (for example, -linking) and others are faster remotely (for example, highly parallelizable -compilation). A dynamic execution strategy can provide the best possible -incremental and clean build times. +An execution strategy that selects between local and remote execution based on various heuristics, and uses the execution results of the faster successful method. Certain [actions](#action) are executed faster locally (for example, linking) and others are faster remotely (for example, highly parallelizable compilation). A dynamic execution strategy can provide the best possible incremental and clean build times. ### Execution phase -The third phase of a build. Executes the [actions](#action) in the [action -graph](#action-graph) created during the [analysis phase](#analysis-phase). -These actions invoke executables (compilers, scripts) to read and write -[artifacts](#artifact). *Spawn strategies* control how these actions are -executed: locally, remotely, dynamically, sandboxed, docker, and so on. +The third phase of a build. Executes the [actions](#action) in the [action graph](#action-graph) created during the [analysis phase](#analysis-phase). These actions invoke executables (compilers, scripts) to read and write [artifacts](#artifact). *Spawn strategies* control how these actions are executed: locally, remotely, dynamically, sandboxed, docker, and so on. ### Execution root -A directory in the [workspace](#workspace)’s [output base](#output-base) -directory where local [actions](#action) are executed in -non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks -of input [artifacts](#artifact) from the workspace. The execution root also -contains symlinks to external repositories as other inputs and the `bazel-out` -directory to store outputs. Prepared during the [loading phase](#loading-phase) -by creating a *symlink forest* of the directories that represent the transitive -closure of packages on which a build depends. Accessible with `bazel info -execution_root` on the command line. +A directory in the [workspace](#workspace)’s [output base](#output-base) directory where local [actions](#action) are executed in non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks of input [artifacts](#artifact) from the workspace. The execution root also contains symlinks to external repositories as other inputs and the `bazel-out` directory to store outputs. Prepared during the [loading phase](#loading-phase) by creating a *symlink forest* of the directories that represent the transitive closure of packages on which a build depends. Accessible with `bazel info execution_root` on the command line. ### File @@ -292,52 +158,27 @@ See [Artifact](#artifact). ### Hermeticity -A build is hermetic if there are no external influences on its build and test -operations, which helps to make sure that results are deterministic and -[correct](#correctness). For example, hermetic builds typically disallow network -access to actions, restrict access to declared inputs, use fixed timestamps and -timezones, restrict access to environment variables, and use fixed seeds for -random number generators +A build is hermetic if there are no external influences on its build and test operations, which helps to make sure that results are deterministic and [correct](#correctness). For example, hermetic builds typically disallow network access to actions, restrict access to declared inputs, use fixed timestamps and timezones, restrict access to environment variables, and use fixed seeds for random number generators ### Incremental build -An incremental build reuses the results of earlier builds to reduce build time -and resource usage. Dependency checking and caching aim to produce correct -results for this type of build. An incremental build is the opposite of a clean -build. - -{/* TODO: ### Install base */} +An incremental build reuses the results of earlier builds to reduce build time and resource usage. Dependency checking and caching aim to produce correct results for this type of build. An incremental build is the opposite of a clean build. ### Label -An identifier for a [target](#target). Generally has the form -`@repo//path/to/package:target`, where `repo` is the (apparent) name of the -[repository](#repository) containing the target, `path/to/package` is the path -to the directory that contains the [`BUILD` file](#build-file) declaring the -target (this directory is also known as the [package](#package)), and `target` -is the name of the target itself. Depending on the situation, parts of this -syntax may be omitted. +An identifier for a [target](#target). Generally has the form `@repo//path/to/package:target`, where `repo` is the (apparent) name of the [repository](#repository) containing the target, `path/to/package` is the path to the directory that contains the [`BUILD` file](#build-file) declaring the target (this directory is also known as the [package](#package)), and `target` is the name of the target itself. Depending on the situation, parts of this syntax may be omitted. **See also**: [Labels](/concepts/labels) ### Loading phase -The first phase of a build where Bazel executes [`BUILD` files](#build-file) to -create [packages](#package). [Macros](#macro) and certain functions like -`glob()` are evaluated in this phase. Interleaved with the second phase of the -build, the [analysis phase](#analysis-phase), to build up a [target -graph](#target-graph). +The first phase of a build where Bazel executes [`BUILD` files](#build-file) to create [packages](#package). [Macros](#macro) and certain functions like `glob()` are evaluated in this phase. Interleaved with the second phase of the build, the [analysis phase](#analysis-phase), to build up a [target graph](#target-graph). ### Legacy macro -A flavor of [macro](#macro) which is declared as an ordinary -[Starlark](#starlark) function, and which runs as a side effect of executing a -`BUILD` file. +A flavor of [macro](#macro) which is declared as an ordinary [Starlark](#starlark) function, and which runs as a side effect of executing a `BUILD` file. -Legacy macros can do anything a function can. This means they can be convenient, -but they can also be harder to read, write, and use. A legacy macro might -unexpectedly mutate its arguments or fail when given a `select()` or ill-typed -argument. +Legacy macros can do anything a function can. This means they can be convenient, but they can also be harder to read, write, and use. A legacy macro might unexpectedly mutate its arguments or fail when given a `select()` or ill-typed argument. Contrast with [symbolic macros](#symbolic-macro). @@ -345,34 +186,19 @@ Contrast with [symbolic macros](#symbolic-macro). ### Macro -A mechanism to compose multiple [rule](#rule) target declarations together under -a single [Starlark](#starlark) callable. Enables reusing common rule declaration -patterns across `BUILD` files. Expanded to the underlying rule target -declarations during the [loading phase](#loading-phase). +A mechanism to compose multiple [rule](#rule) target declarations together under a single [Starlark](#starlark) callable. Enables reusing common rule declaration patterns across `BUILD` files. Expanded to the underlying rule target declarations during the [loading phase](#loading-phase). -Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and -[legacy macros](#legacy-macro). +Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and [legacy macros](#legacy-macro). ### Mnemonic -A short, human-readable string selected by a rule author to quickly understand -what an [action](#action) in the rule is doing. Mnemonics can be used as -identifiers for *spawn strategy* selections. Some examples of action mnemonics -are `Javac` from Java rules, `CppCompile` from C++ rules, and -`AndroidManifestMerger` from Android rules. +A short, human-readable string selected by a rule author to quickly understand what an [action](#action) in the rule is doing. Mnemonics can be used as identifiers for *spawn strategy* selections. Some examples of action mnemonics are `Javac` from Java rules, `CppCompile` from C++ rules, and `AndroidManifestMerger` from Android rules. ### Module -A Bazel project that can have multiple versions, each of which can have -dependencies on other modules. This is analogous to familiar concepts in other -dependency management systems, such as a Maven _artifact_, an npm _package_, a -Go _module_, or a Cargo _crate_. Modules form the backbone of Bazel's external -dependency management system. +A Bazel project that can have multiple versions, each of which can have dependencies on other modules. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. Modules form the backbone of Bazel's external dependency management system. -Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its -root. This file contains metadata about the module itself (such as its name and -version), its direct dependencies, and various other data including toolchain -registrations and [module extension](#module-extension) input. +Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its root. This file contains metadata about the module itself (such as its name and version), its direct dependencies, and various other data including toolchain registrations and [module extension](#module-extension) input. Module metadata is hosted in Bazel registries. @@ -380,336 +206,180 @@ Module metadata is hosted in Bazel registries. ### Module Extension -A piece of logic that can be run to generate [repos](#repository) by reading -inputs from across the [module](#module) dependency graph and invoking [repo -rules](#repository-rule). Module extensions have capabilities similar to repo -rules, allowing them to access the internet, perform file I/O, and so on. +A piece of logic that can be run to generate [repos](#repository) by reading inputs from across the [module](#module) dependency graph and invoking [repo rules](#repository-rule). Module extensions have capabilities similar to repo rules, allowing them to access the internet, perform file I/O, and so on. **See also:** [Module extensions](/external/extension) ### Native rules -[Rules](#rule) that are built into Bazel and implemented in Java. Such rules -appear in [`.bzl` files](#bzl-file) as functions in the native module (for -example, `native.cc_library` or `native.java_library`). User-defined rules -(non-native) are created using [Starlark](#starlark). +[Rules](#rule) that are built into Bazel and implemented in Java. Such rules appear in [`.bzl` files](#bzl-file) as functions in the native module (for example, `native.cc_library` or `native.java_library`). User-defined rules (non-native) are created using [Starlark](#starlark). ### Output base -A [workspace](#workspace)-specific directory to store Bazel output files. Used -to separate outputs from the *workspace*'s source tree (the [main -repo](#repository)). Located in the [output user root](#output-user-root). +A [workspace](#workspace)-specific directory to store Bazel output files. Used to separate outputs from the *workspace*'s source tree (the [main repo](#repository)). Located in the [output user root](#output-user-root). ### Output groups -A group of files that is expected to be built when Bazel finishes building a -target. [Rules](#rule) put their usual outputs in the "default output group" -(e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` -targets). The default output group is the output group whose -[artifacts](#artifact) are built when a target is requested on the command line. -Rules can define more named output groups that can be explicitly specified in -[`BUILD` files](#build-file) (`filegroup` rule) or the command line -(`--output_groups` flag). +A group of files that is expected to be built when Bazel finishes building a target. [Rules](#rule) put their usual outputs in the "default output group" (e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` targets). The default output group is the output group whose [artifacts](#artifact) are built when a target is requested on the command line. Rules can define more named output groups that can be explicitly specified in [`BUILD` files](#build-file) (`filegroup` rule) or the command line (`--output_groups` flag). ### Output user root -A user-specific directory to store Bazel's outputs. The directory name is -derived from the user's system username. Prevents output file collisions if -multiple users are building the same project on the system at the same time. -Contains subdirectories corresponding to build outputs of individual workspaces, -also known as [output bases](#output-base). +A user-specific directory to store Bazel's outputs. The directory name is derived from the user's system username. Prevents output file collisions if multiple users are building the same project on the system at the same time. Contains subdirectories corresponding to build outputs of individual workspaces, also known as [output bases](#output-base). ### Package -The set of [targets](#target) defined by a [`BUILD` file](#build-file). A -package's name is the `BUILD` file's path relative to the [repo](#repository) -root. A package can contain subpackages, or subdirectories containing `BUILD` -files, thus forming a package hierarchy. +The set of [targets](#target) defined by a [`BUILD` file](#build-file). A package's name is the `BUILD` file's path relative to the [repo](#repository) root. A package can contain subpackages, or subdirectories containing `BUILD` files, thus forming a package hierarchy. ### Package group -A [target](#target) representing a set of packages. Often used in `visibility` -attribute values. +A [target](#target) representing a set of packages. Often used in `visibility` attribute values. ### Platform -A "machine type" involved in a build. This includes the machine Bazel runs on -(the "host" platform), the machines build tools execute on ("exec" platforms), -and the machines targets are built for ("target platforms"). +A "machine type" involved in a build. This includes the machine Bazel runs on (the "host" platform), the machines build tools execute on ("exec" platforms), and the machines targets are built for ("target platforms"). ### Provider -A schema describing a unit of information to pass between -[rule targets](#rule-target) along dependency relationships. Typically this -contains information like compiler options, transitive source or output files, -and build metadata. Frequently used in conjunction with [depsets](#depset) to -efficiently store accumulated transitive data. An example of a built-in provider -is `DefaultInfo`. +A schema describing a unit of information to pass between [rule targets](#rule-target) along dependency relationships. Typically this contains information like compiler options, transitive source or output files, and build metadata. Frequently used in conjunction with [depsets](#depset) to efficiently store accumulated transitive data. An example of a built-in provider is `DefaultInfo`. -Note: The object holding specific data for a given rule target is -referred to as a "provider instance", although sometimes this is conflated with -"provider". +Note: The object holding specific data for a given rule target is referred to as a "provider instance", although sometimes this is conflated with "provider". **See also:** [Provider documentation](/extending/rules#providers) ### Query (concept) -The process of analyzing a [build graph](#build-graph) to understand -[target](#target) properties and dependency structures. Bazel supports three -query variants: [query](#query-command), [cquery](#configured-query), and -[aquery](#action-graph-query). +The process of analyzing a [build graph](#build-graph) to understand [target](#target) properties and dependency structures. Bazel supports three query variants: [query](#query-command), [cquery](#configured-query), and [aquery](#action-graph-query). ### query (command) -A [query](#query-concept) tool that operates over the build's post-[loading -phase](#loading-phase) [target graph](#target-graph). This is relatively fast, -but can't analyze the effects of `select()`, [build flags](#command-flags), -[artifacts](#artifact), or build [actions](#action). +A [query](#query-concept) tool that operates over the build's post-[loading phase](#loading-phase) [target graph](#target-graph). This is relatively fast, but can't analyze the effects of `select()`, [build flags](#command-flags), [artifacts](#artifact), or build [actions](#action). **See also:** [Query how-to](/query/guide), [Query reference](/query/language) ### Repository -A directory tree with a boundary marker file at its root, containing source -files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo -represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or -`WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a -repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. The *main repo* is the repo in which the current Bazel command is being run. -*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` -files, or invoking [repo rules](#repository-rule) in [module -extensions](#module-extension). They can be fetched on demand to a predetermined -"magical" location on disk. +*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` files, or invoking [repo rules](#repository-rule) in [module extensions](#module-extension). They can be fetched on demand to a predetermined "magical" location on disk. -Each repo has a unique, constant *canonical* name, and potentially different -*apparent* names when viewed from other repos. +Each repo has a unique, constant *canonical* name, and potentially different *apparent* names when viewed from other repos. **See also**: [External dependencies overview](/external/overview) ### Repository cache -A shared content-addressable cache of files downloaded by Bazel for builds, -shareable across [workspaces](#workspace). Enables offline builds after the -initial download. Commonly used to cache files downloaded through [repository -rules](#repository-rule) like `http_archive` and repository rule APIs like -`repository_ctx.download`. Files are cached only if their SHA-256 checksums are -specified for the download. +A shared content-addressable cache of files downloaded by Bazel for builds, shareable across [workspaces](#workspace). Enables offline builds after the initial download. Commonly used to cache files downloaded through [repository rules](#repository-rule) like `http_archive` and repository rule APIs like `repository_ctx.download`. Files are cached only if their SHA-256 checksums are specified for the download. ### Repository rule -A schema for repository definitions that tells Bazel how to materialize (or -"fetch") a [repository](#repository). Often shortened to just **repo rule**. -Repo rules are invoked by Bazel internally to define repos backed by -[modules](#module), or can be invoked by [module extensions](#module-extension). -Repo rules can access the internet or perform file I/O; the most common repo -rule is `http_archive` to download an archive containing source files from the -internet. +A schema for repository definitions that tells Bazel how to materialize (or "fetch") a [repository](#repository). Often shortened to just **repo rule**. Repo rules are invoked by Bazel internally to define repos backed by [modules](#module), or can be invoked by [module extensions](#module-extension). Repo rules can access the internet or perform file I/O; the most common repo rule is `http_archive` to download an archive containing source files from the internet. **See also:** [Repo rule documentation](/external/repo) ### Reproducibility -The property of a build or test that a set of inputs to the build or test will -always produce the same set of outputs every time, regardless of time, method, -or environment. Note that this does not necessarily imply that the outputs are -[correct](#correctness) or the desired outputs. +The property of a build or test that a set of inputs to the build or test will always produce the same set of outputs every time, regardless of time, method, or environment. Note that this does not necessarily imply that the outputs are [correct](#correctness) or the desired outputs. ### Rule -A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as -`cc_library`. From the perspective of a `BUILD` file author, a rule consists of -a set of [attributes](#attributes) and black box logic. The logic tells the -rule target how to produce output [artifacts](#artifact) and pass information to -other rule targets. From the perspective of `.bzl` authors, rules are the -primary way to extend Bazel to support new programming languages and -environments. - -Rules are instantiated to produce rule targets in the -[loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule -targets communicate information to their downstream dependencies in the form of -[providers](#provider), and register [actions](#action) describing how to -generate their output artifacts. These actions are run in the [execution -phase](#execution-phase). - -Note: Historically the term "rule" has been used to refer to a rule target. -This usage was inherited from tools like Make, but causes confusion and should -be avoided for Bazel. +A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as `cc_library`. From the perspective of a `BUILD` file author, a rule consists of a set of [attributes](#attributes) and black box logic. The logic tells the rule target how to produce output [artifacts](#artifact) and pass information to other rule targets. From the perspective of `.bzl` authors, rules are the primary way to extend Bazel to support new programming languages and environments. + +Rules are instantiated to produce rule targets in the [loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule targets communicate information to their downstream dependencies in the form of [providers](#provider), and register [actions](#action) describing how to generate their output artifacts. These actions are run in the [execution phase](#execution-phase). + +Note: Historically the term "rule" has been used to refer to a rule target. This usage was inherited from tools like Make, but causes confusion and should be avoided for Bazel. **See also:** [Rules documentation](/extending/rules) ### Rule target -A [target](#target) that is an instance of a rule. Contrasts with file targets -and package groups. Not to be confused with [rule](#rule). +A [target](#target) that is an instance of a rule. Contrasts with file targets and package groups. Not to be confused with [rule](#rule). ### Runfiles -The runtime dependencies of an executable [target](#target). Most commonly, the -executable is the executable output of a test rule, and the runfiles are runtime -data dependencies of the test. Before the invocation of the executable (during -bazel test), Bazel prepares the tree of runfiles alongside the test executable -according to their source directory structure. +The runtime dependencies of an executable [target](#target). Most commonly, the executable is the executable output of a test rule, and the runfiles are runtime data dependencies of the test. Before the invocation of the executable (during bazel test), Bazel prepares the tree of runfiles alongside the test executable according to their source directory structure. **See also:** [Runfiles documentation](/extending/rules#runfiles) ### Sandboxing -A technique to isolate a running [action](#action) inside a restricted and -temporary [execution root](#execution-root), helping to ensure that it doesn’t -read undeclared inputs or write undeclared outputs. Sandboxing greatly improves -[hermeticity](#hermeticity), but usually has a performance cost, and requires -support from the operating system. The performance cost depends on the platform. -On Linux, it's not significant, but on macOS it can make sandboxing unusable. +A technique to isolate a running [action](#action) inside a restricted and temporary [execution root](#execution-root), helping to ensure that it doesn’t read undeclared inputs or write undeclared outputs. Sandboxing greatly improves [hermeticity](#hermeticity), but usually has a performance cost, and requires support from the operating system. The performance cost depends on the platform. On Linux, it's not significant, but on macOS it can make sandboxing unusable. ### Skyframe [Skyframe](/reference/skyframe) is the core parallel, functional, and incremental evaluation framework of Bazel. -{/* TODO: ### Spawn strategy */} - ### Stamping -A feature to embed additional information into Bazel-built -[artifacts](#artifact). For example, this can be used for source control, build -time and other workspace or environment-related information for release builds. -Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that -support the stamp attribute. +A feature to embed additional information into Bazel-built [artifacts](#artifact). For example, this can be used for source control, build time and other workspace or environment-related information for release builds. Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that support the stamp attribute. ### Starlark -The extension language for writing [rules](/extending/rules) and [macros](#macro). A -restricted subset of Python (syntactically and grammatically) aimed for the -purpose of configuration, and for better performance. Uses the [`.bzl` -file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more -restricted version of Starlark (such as no `def` function definitions), formerly -known as Skylark. +The extension language for writing [rules](/extending/rules) and [macros](#macro). A restricted subset of Python (syntactically and grammatically) aimed for the purpose of configuration, and for better performance. Uses the [`.bzl` file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more restricted version of Starlark (such as no `def` function definitions), formerly known as Skylark. **See also:** [Starlark language documentation](/rules/language) -{/* TODO: ### Starlark rules */} - -{/* TODO: ### Starlark rule sandwich */} - ### Startup flags -The set of flags specified between `bazel` and the [command](#query-command), -for example, bazel `--host_jvm_debug` build. These flags modify the -[configuration](#configuration) of the Bazel server, so any modification to -startup flags causes a server restart. Startup flags are not specific to any -command. +The set of flags specified between `bazel` and the [command](#query-command), for example, bazel `--host_jvm_debug` build. These flags modify the [configuration](#configuration) of the Bazel server, so any modification to startup flags causes a server restart. Startup flags are not specific to any command. ### Symbolic macro -A flavor of [macro](#macro) which is declared with a [rule](#rule)-like -[attribute](#attribute) schema, allows hiding internal declared -[targets](#target) from their own package, and enforces a predictable naming -pattern on the targets that the macro declares. Designed to avoid some of the -problems seen in large [legacy macro](#legacy-macro) codebases. +A flavor of [macro](#macro) which is declared with a [rule](#rule)-like [attribute](#attribute) schema, allows hiding internal declared [targets](#target) from their own package, and enforces a predictable naming pattern on the targets that the macro declares. Designed to avoid some of the problems seen in large [legacy macro](#legacy-macro) codebases. **See also:** [Symbolic macro documentation](/extending/macros) ### Target -An object that is defined in a [`BUILD` file](#build-file) and identified by a -[label](#label). Targets represent the buildable units of a workspace from -the perspective of the end user. +An object that is defined in a [`BUILD` file](#build-file) and identified by a [label](#label). Targets represent the buildable units of a workspace from the perspective of the end user. -A target that is declared by instantiating a [rule](#rule) is called a [rule -target](#rule-target). Depending on the rule, these may be runnable (like -`cc_binary`) or testable (like `cc_test`). Rule targets typically depend on -other targets via their [attributes](#attribute) (such as `deps`); these -dependencies form the basis of the [target graph](#target-graph). +A target that is declared by instantiating a [rule](#rule) is called a [rule target](#rule-target). Depending on the rule, these may be runnable (like `cc_binary`) or testable (like `cc_test`). Rule targets typically depend on other targets via their [attributes](#attribute) (such as `deps`); these dependencies form the basis of the [target graph](#target-graph). -Aside from rule targets, there are also file targets and [package group](#package-group) -targets. File targets correspond to [artifacts](#artifact) that are referenced -within a `BUILD` file. As a special case, the `BUILD` file of any package is -always considered a source file target in that package. +Aside from rule targets, there are also file targets and [package group](#package-group) targets. File targets correspond to [artifacts](#artifact) that are referenced within a `BUILD` file. As a special case, the `BUILD` file of any package is always considered a source file target in that package. -Targets are discovered during the [loading phase](#loading-phase). During the -[analysis phase](#analysis-phase), targets are associated with [build -configurations](#configuration) to form [configured -targets](#configured-target). +Targets are discovered during the [loading phase](#loading-phase). During the [analysis phase](#analysis-phase), targets are associated with [build configurations](#configuration) to form [configured targets](#configured-target). ### Target graph -An in-memory graph of [targets](#target) and their dependencies. Produced during -the [loading phase](#loading-phase) and used as an input to the [analysis -phase](#analysis-phase). +An in-memory graph of [targets](#target) and their dependencies. Produced during the [loading phase](#loading-phase) and used as an input to the [analysis phase](#analysis-phase). ### Target pattern -A way to specify a group of [targets](#target) on the command line. Commonly -used patterns are `:all` (all rule targets), `:*` (all rule + file targets), -`...` (current [package](#package) and all subpackages recursively). Can be used -in combination, for example, `//...:*` means all rule and file targets in all -packages recursively from the root of the [workspace](#workspace). +A way to specify a group of [targets](#target) on the command line. Commonly used patterns are `:all` (all rule targets), `:*` (all rule + file targets), `...` (current [package](#package) and all subpackages recursively). Can be used in combination, for example, `//...:*` means all rule and file targets in all packages recursively from the root of the [workspace](#workspace). ### Tests -Rule [targets](#target) instantiated from test rules, and therefore contains a -test executable. A return code of zero from the completion of the executable -indicates test success. The exact contract between Bazel and tests (such as test -environment variables, test result collection methods) is specified in the [Test -Encyclopedia](/reference/test-encyclopedia). +Rule [targets](#target) instantiated from test rules, and therefore contains a test executable. A return code of zero from the completion of the executable indicates test success. The exact contract between Bazel and tests (such as test environment variables, test result collection methods) is specified in the [Test Encyclopedia](/reference/test-encyclopedia). ### Toolchain -A set of tools to build outputs for a language. Typically, a toolchain includes -compilers, linkers, interpreters or/and linters. A toolchain can also vary by -platform, that is, a Unix compiler toolchain's components may differ for the -Windows variant, even though the toolchain is for the same language. Selecting -the right toolchain for the platform is known as toolchain resolution. +A set of tools to build outputs for a language. Typically, a toolchain includes compilers, linkers, interpreters or/and linters. A toolchain can also vary by platform, that is, a Unix compiler toolchain's components may differ for the Windows variant, even though the toolchain is for the same language. Selecting the right toolchain for the platform is known as toolchain resolution. ### Top-level target -A build [target](#target) is top-level if it’s requested on the Bazel command -line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is -called, then for this build, `//:foo` is top-level, and `//:bar` isn’t -top-level, although both targets will need to be built. An important difference -between top-level and non-top-level targets is that [command -flags](#command-flags) set on the Bazel command line (or via -[.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level -targets, but might be modified by a [transition](#transition) for non-top-level -targets. +A build [target](#target) is top-level if it’s requested on the Bazel command line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is called, then for this build, `//:foo` is top-level, and `//:bar` isn’t top-level, although both targets will need to be built. An important difference between top-level and non-top-level targets is that [command flags](#command-flags) set on the Bazel command line (or via [.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level targets, but might be modified by a [transition](#transition) for non-top-level targets. ### Transition -A mapping of [configuration](#configuration) state from one value to another. -Enables [targets](#target) in the [build graph](#build-graph) to have different -configurations, even if they were instantiated from the same [rule](#rule). A -common usage of transitions is with *split* transitions, where certain parts of -the [target graph](#target-graph) is forked with distinct configurations for -each fork. For example, one can build an Android APK with native binaries -compiled for ARM and x86 using split transitions in a single build. +A mapping of [configuration](#configuration) state from one value to another. Enables [targets](#target) in the [build graph](#build-graph) to have different configurations, even if they were instantiated from the same [rule](#rule). A common usage of transitions is with *split* transitions, where certain parts of the [target graph](#target-graph) is forked with distinct configurations for each fork. For example, one can build an Android APK with native binaries compiled for ARM and x86 using split transitions in a single build. **See also:** [User-defined transitions](/extending/config#user-defined-transitions) ### Tree artifact -An [artifact](#artifact) that represents a collection of files. Since these -files are not themselves artifacts, an [action](#action) operating on them must -instead register the tree artifact as its input or output. +An [artifact](#artifact) that represents a collection of files. Since these files are not themselves artifacts, an [action](#action) operating on them must instead register the tree artifact as its input or output. ### Visibility -One of two mechanisms for preventing unwanted dependencies in the build system: -*target visibility* for controlling whether a [target](#target) can be depended -upon by other targets; and *load visibility* for controlling whether a `BUILD` -or `.bzl` file may load a given `.bzl` file. Without context, usually -"visibility" refers to target visibility. +One of two mechanisms for preventing unwanted dependencies in the build system: *target visibility* for controlling whether a [target](#target) can be depended upon by other targets; and *load visibility* for controlling whether a `BUILD` or `.bzl` file may load a given `.bzl` file. Without context, usually "visibility" refers to target visibility. **See also:** [Visibility documentation](/concepts/visibility) ### Workspace -The environment shared by all Bazel commands run from the same [main -repository](#repository). +The environment shared by all Bazel commands run from the same [main repository](#repository). -Note that historically the concepts of "repository" and "workspace" have been -conflated; the term "workspace" has often been used to refer to the main -repository, and sometimes even used as a synonym of "repository". Such usage -should be avoided for clarity. +Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". Such usage should be avoided for clarity. diff --git a/reference/skyframe.mdx b/reference/skyframe.mdx index ba9149fa..7f99346a 100644 --- a/reference/skyframe.mdx +++ b/reference/skyframe.mdx @@ -2,197 +2,86 @@ title: 'Skyframe' --- - - The parallel evaluation and incrementality model of Bazel. ## Data model The data model consists of the following items: -* `SkyValue`. Also called nodes. `SkyValues` are immutable objects that - contain all the data built over the course of the build and the inputs of - the build. Examples are: input files, output files, targets and configured - targets. -* `SkyKey`. A short immutable name to reference a `SkyValue`, for example, - `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. -* `SkyFunction`. Builds nodes based on their keys and dependent nodes. -* Node graph. A data structure containing the dependency relationship between - nodes. -* `Skyframe`. Code name for the incremental evaluation framework Bazel is - based on. +- `SkyValue`. Also called nodes. `SkyValues` are immutable objects that contain all the data built over the course of the build and the inputs of the build. Examples are: input files, output files, targets and configured targets. +- `SkyKey`. A short immutable name to reference a `SkyValue`, for example, `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. +- `SkyFunction`. Builds nodes based on their keys and dependent nodes. +- Node graph. A data structure containing the dependency relationship between nodes. +- `Skyframe`. Code name for the incremental evaluation framework Bazel is based on. ## Evaluation A build is achieved by evaluating the node that represents the build request. -First, Bazel finds the `SkyFunction` corresponding to the key of the top-level -`SkyKey`. The function then requests the evaluation of the nodes it needs to -evaluate the top-level node, which in turn result in other `SkyFunction` calls, -until the leaf nodes are reached. Leaf nodes are usually ones that represent -input files in the file system. Finally, Bazel ends up with the value of the -top-level `SkyValue`, some side effects (such as output files in the file -system) and a directed acyclic graph of the dependencies between the nodes -involved in the build. - -A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in -advance all of the nodes it needs to do its job. A simple example is evaluating -an input file node that turns out to be a symlink: the function tries to read -the file, realizes that it is a symlink, and thus fetches the file system node -representing the target of the symlink. But that itself can be a symlink, in -which case the original function will need to fetch its target, too. - -The functions are represented in the code by the interface `SkyFunction` and the -services provided to it by an interface called `SkyFunction.Environment`. These -are the things functions can do: - -* Request the evaluation of another node by way of calling `env.getValue`. If - the node is available, its value is returned, otherwise, `null` is returned - and the function itself is expected to return `null`. In the latter case, - the dependent node is evaluated, and then the original node builder is - invoked again, but this time the same `env.getValue` call will return a - non-`null` value. -* Request the evaluation of multiple other nodes by calling `env.getValues()`. - This does essentially the same, except that the dependent nodes are - evaluated in parallel. -* Do computation during their invocation -* Have side effects, for example, writing files to the file system. Care needs - to be taken that two different functions avoid stepping on each other's - toes. In general, write side effects (where data flows outwards from Bazel) - are okay, read side effects (where data flows inwards into Bazel without a - registered dependency) are not, because they are an unregistered dependency - and as such, can cause incorrect incremental builds. - -Well-behaved `SkyFunction` implementations avoid accessing data in any other way -than requesting dependencies (such as by directly reading the file system), -because that results in Bazel not registering the data dependency on the file -that was read, thus resulting in incorrect incremental builds. - -Once a function has enough data to do its job, it should return a non-`null` -value indicating completion. +First, Bazel finds the `SkyFunction` corresponding to the key of the top-level `SkyKey`. The function then requests the evaluation of the nodes it needs to evaluate the top-level node, which in turn result in other `SkyFunction` calls, until the leaf nodes are reached. Leaf nodes are usually ones that represent input files in the file system. Finally, Bazel ends up with the value of the top-level `SkyValue`, some side effects (such as output files in the file system) and a directed acyclic graph of the dependencies between the nodes involved in the build. + +A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in advance all of the nodes it needs to do its job. A simple example is evaluating an input file node that turns out to be a symlink: the function tries to read the file, realizes that it is a symlink, and thus fetches the file system node representing the target of the symlink. But that itself can be a symlink, in which case the original function will need to fetch its target, too. + +The functions are represented in the code by the interface `SkyFunction` and the services provided to it by an interface called `SkyFunction.Environment`. These are the things functions can do: + +- Request the evaluation of another node by way of calling `env.getValue`. If the node is available, its value is returned, otherwise, `null` is returned and the function itself is expected to return `null`. In the latter case, the dependent node is evaluated, and then the original node builder is invoked again, but this time the same `env.getValue` call will return a non-`null` value. +- Request the evaluation of multiple other nodes by calling `env.getValues()`. This does essentially the same, except that the dependent nodes are evaluated in parallel. +- Do computation during their invocation +- Have side effects, for example, writing files to the file system. Care needs to be taken that two different functions avoid stepping on each other's toes. In general, write side effects (where data flows outwards from Bazel) are okay, read side effects (where data flows inwards into Bazel without a registered dependency) are not, because they are an unregistered dependency and as such, can cause incorrect incremental builds. + +Well-behaved `SkyFunction` implementations avoid accessing data in any other way than requesting dependencies (such as by directly reading the file system), because that results in Bazel not registering the data dependency on the file that was read, thus resulting in incorrect incremental builds. + +Once a function has enough data to do its job, it should return a non-`null` value indicating completion. This evaluation strategy has a number of benefits: -* Hermeticity. If functions only request input data by way of depending on - other nodes, Bazel can guarantee that if the input state is the same, the - same data is returned. If all sky functions are deterministic, this means - that the whole build will also be deterministic. -* Correct and perfect incrementality. If all the input data of all functions - is recorded, Bazel can invalidate only the exact set of nodes that need to - be invalidated when the input data changes. -* Parallelism. Since functions can only interact with each other by way of - requesting dependencies, functions that don't depend on each other can be - run in parallel and Bazel can guarantee that the result is the same as if - they were run sequentially. +- Hermeticity. If functions only request input data by way of depending on other nodes, Bazel can guarantee that if the input state is the same, the same data is returned. If all sky functions are deterministic, this means that the whole build will also be deterministic. +- Correct and perfect incrementality. If all the input data of all functions is recorded, Bazel can invalidate only the exact set of nodes that need to be invalidated when the input data changes. +- Parallelism. Since functions can only interact with each other by way of requesting dependencies, functions that don't depend on each other can be run in parallel and Bazel can guarantee that the result is the same as if they were run sequentially. ## Incrementality -Since functions can only access input data by depending on other nodes, Bazel -can build up a complete data flow graph from the input files to the output -files, and use this information to only rebuild those nodes that actually need -to be rebuilt: the reverse transitive closure of the set of changed input files. - -In particular, two possible incrementality strategies exist: the bottom-up one -and the top-down one. Which one is optimal depends on how the dependency graph -looks like. - -* During bottom-up invalidation, after a graph is built and the set of changed - inputs is known, all the nodes are invalidated that transitively depend on - changed files. This is optimal if the same top-level node will be built - again. Note that bottom-up invalidation requires running `stat()` on all - input files of the previous build to determine if they were changed. This - can be improved by using `inotify` or a similar mechanism to learn about - changed files. - -* During top-down invalidation, the transitive closure of the top-level node - is checked and only those nodes are kept whose transitive closure is clean. - This is better if the node graph is large, but the next build only needs a - small subset of it: bottom-up invalidation would invalidate the larger graph - of the first build, unlike top-down invalidation, which just walks the small - graph of second build. +Since functions can only access input data by depending on other nodes, Bazel can build up a complete data flow graph from the input files to the output files, and use this information to only rebuild those nodes that actually need to be rebuilt: the reverse transitive closure of the set of changed input files. + +In particular, two possible incrementality strategies exist: the bottom-up one and the top-down one. Which one is optimal depends on how the dependency graph looks like. + +- During bottom-up invalidation, after a graph is built and the set of changed inputs is known, all the nodes are invalidated that transitively depend on changed files. This is optimal if the same top-level node will be built again. Note that bottom-up invalidation requires running `stat()` on all input files of the previous build to determine if they were changed. This can be improved by using `inotify` or a similar mechanism to learn about changed files. + +- During top-down invalidation, the transitive closure of the top-level node is checked and only those nodes are kept whose transitive closure is clean. This is better if the node graph is large, but the next build only needs a small subset of it: bottom-up invalidation would invalidate the larger graph of the first build, unlike top-down invalidation, which just walks the small graph of second build. Bazel only does bottom-up invalidation. -To get further incrementality, Bazel uses _change pruning_: if a node is -invalidated, but upon rebuild, it is discovered that its new value is the same -as its old value, the nodes that were invalidated due to a change in this node -are "resurrected". +To get further incrementality, Bazel uses *change pruning*: if a node is invalidated, but upon rebuild, it is discovered that its new value is the same as its old value, the nodes that were invalidated due to a change in this node are "resurrected". -This is useful, for example, if one changes a comment in a C++ file: then the -`.o` file generated from it will be the same, thus, it is unnecessary to call -the linker again. +This is useful, for example, if one changes a comment in a C++ file: then the `.o` file generated from it will be the same, thus, it is unnecessary to call the linker again. ## Incremental Linking / Compilation -The main limitation of this model is that the invalidation of a node is an -all-or-nothing affair: when a dependency changes, the dependent node is always -rebuilt from scratch, even if a better algorithm would exist that would mutate -the old value of the node based on the changes. A few examples where this would -be useful: +The main limitation of this model is that the invalidation of a node is an all-or-nothing affair: when a dependency changes, the dependent node is always rebuilt from scratch, even if a better algorithm would exist that would mutate the old value of the node based on the changes. A few examples where this would be useful: -* Incremental linking -* When a single class file changes in a JAR file, it is possible - modify the JAR file in-place instead of building it from scratch again. +- Incremental linking +- When a single class file changes in a JAR file, it is possible modify the JAR file in-place instead of building it from scratch again. -The reason why Bazel does not support these things in a principled way -is twofold: +The reason why Bazel does not support these things in a principled way is twofold: -* There were limited performance gains. -* Difficulty to validate that the result of the mutation is the same as that - of a clean rebuild would be, and Google values builds that are bit-for-bit - repeatable. +- There were limited performance gains. +- Difficulty to validate that the result of the mutation is the same as that of a clean rebuild would be, and Google values builds that are bit-for-bit repeatable. -Until now, it was possible to achieve good enough performance by decomposing an -expensive build step and achieving partial re-evaluation that way. For example, -in an Android app, you can split all the classes into multiple groups and dex -them separately. This way, if classes in a group are unchanged, the dexing does -not have to be redone. +Until now, it was possible to achieve good enough performance by decomposing an expensive build step and achieving partial re-evaluation that way. For example, in an Android app, you can split all the classes into multiple groups and dex them separately. This way, if classes in a group are unchanged, the dexing does not have to be redone. ## Mapping to Bazel concepts -This is high level summary of the key `SkyFunction` and `SkyValue` -implementations Bazel uses to perform a build: - -* **FileStateValue**. The result of an `lstat()`. For existent files, the - function also computes additional information in order to detect changes to - the file. This is the lowest level node in the Skyframe graph and has no - dependencies. -* **FileValue**. Used by anything that cares about the actual contents or - resolved path of a file. Depends on the corresponding `FileStateValue` and - any symlinks that need to be resolved (such as the `FileValue` for `a/b` - needs the resolved path of `a` and the resolved path of `a/b`). The - distinction between `FileValue` and `FileStateValue` is important because - the latter can be used in cases where the contents of the file are not - actually needed. For example, the file contents are irrelevant when - evaluating file system globs (such as `srcs=glob(["*/*.java"])`). -* **DirectoryListingStateValue**. The result of `readdir()`. Like - `FileStateValue`, this is the lowest level node and has no dependencies. -* **DirectoryListingValue**. Used by anything that cares about the entries of - a directory. Depends on the corresponding `DirectoryListingStateValue`, as - well as the associated `FileValue` of the directory. -* **PackageValue**. Represents the parsed version of a BUILD file. Depends on - the `FileValue` of the associated `BUILD` file, and also transitively on any - `DirectoryListingValue` that is used to resolve the globs in the package - (the data structure representing the contents of a `BUILD` file internally). -* **ConfiguredTargetValue**. Represents a configured target, which is a tuple - of the set of actions generated during the analysis of a target and - information provided to dependent configured targets. Depends on the - `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` - of direct dependencies, and a special node representing the build - configuration. -* **ArtifactValue**. Represents a file in the build, be it a source or an - output artifact. Artifacts are almost equivalent to files, and are used to - refer to files during the actual execution of build steps. Source files - depends on the `FileValue` of the associated node, and output artifacts - depend on the `ActionExecutionValue` of whatever action generates the - artifact. -* **ActionExecutionValue**. Represents the execution of an action. Depends on - the `ArtifactValues` of its input files. The action it executes is contained - within its SkyKey, which is contrary to the concept that SkyKeys should be - small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if - the execution phase does not run. - -As a visual aid, this diagram shows the relationships between -SkyFunction implementations after a build of Bazel itself: +This is high level summary of the key `SkyFunction` and `SkyValue` implementations Bazel uses to perform a build: + +- **FileStateValue**. The result of an `lstat()`. For existent files, the function also computes additional information in order to detect changes to the file. This is the lowest level node in the Skyframe graph and has no dependencies. +- **FileValue**. Used by anything that cares about the actual contents or resolved path of a file. Depends on the corresponding `FileStateValue` and any symlinks that need to be resolved (such as the `FileValue` for `a/b` needs the resolved path of `a` and the resolved path of `a/b`). The distinction between `FileValue` and `FileStateValue` is important because the latter can be used in cases where the contents of the file are not actually needed. For example, the file contents are irrelevant when evaluating file system globs (such as `srcs=glob(["*/*.java"])`). +- **DirectoryListingStateValue**. The result of `readdir()`. Like `FileStateValue`, this is the lowest level node and has no dependencies. +- **DirectoryListingValue**. Used by anything that cares about the entries of a directory. Depends on the corresponding `DirectoryListingStateValue`, as well as the associated `FileValue` of the directory. +- **PackageValue**. Represents the parsed version of a BUILD file. Depends on the `FileValue` of the associated `BUILD` file, and also transitively on any `DirectoryListingValue` that is used to resolve the globs in the package (the data structure representing the contents of a `BUILD` file internally). +- **ConfiguredTargetValue**. Represents a configured target, which is a tuple of the set of actions generated during the analysis of a target and information provided to dependent configured targets. Depends on the `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` of direct dependencies, and a special node representing the build configuration. +- **ArtifactValue**. Represents a file in the build, be it a source or an output artifact. Artifacts are almost equivalent to files, and are used to refer to files during the actual execution of build steps. Source files depends on the `FileValue` of the associated node, and output artifacts depend on the `ActionExecutionValue` of whatever action generates the artifact. +- **ActionExecutionValue**. Represents the execution of an action. Depends on the `ArtifactValues` of its input files. The action it executes is contained within its SkyKey, which is contrary to the concept that SkyKeys should be small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if the execution phase does not run. + +As a visual aid, this diagram shows the relationships between SkyFunction implementations after a build of Bazel itself: ![A graph of SkyFunction implementation relationships](/reference/skyframe.png) diff --git a/reference/test-encyclopedia.mdx b/reference/test-encyclopedia.mdx new file mode 100644 index 00000000..b56909fe --- /dev/null +++ b/reference/test-encyclopedia.mdx @@ -0,0 +1,308 @@ +--- +title: 'Test encyclopedia' +--- + +An exhaustive specification of the test execution environment. + +## Background + +The Bazel BUILD language includes rules which can be used to define automated test programs in many languages. + +Tests are run using [`bazel test`](/docs/user-manual#test). + +Users may also execute test binaries directly. This is allowed but not endorsed, as such an invocation will not adhere to the mandates described below. + +Tests should be *hermetic*: that is, they ought to access only those resources on which they have a declared dependency. If tests are not properly hermetic then they do not give historically reproducible results. This could be a significant problem for culprit finding (determining which change broke a test), release engineering auditability, and resource isolation of tests (automated testing frameworks ought not DDOS a server because some tests happen to talk to it). + +## Objective + +The goal of this page is to formally establish the runtime environment for and expected behavior of Bazel tests. It will also impose requirements on the test runner and the build system. + +The test environment specification helps test authors avoid relying on unspecified behavior, and thus gives the testing infrastructure more freedom to make implementation changes. The specification tightens up some holes that currently allow many tests to pass despite not being properly hermetic, deterministic, and reentrant. + +This page is intended to be both normative and authoritative. If this specification and the implemented behavior of test runner disagree, the specification takes precedence. + +## Proposed Specification + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in IETF RFC 2119. + +## Purpose of tests + +The purpose of Bazel tests is to confirm some property of the source files checked into the repository. (On this page, "source files" includes test data, golden outputs, and anything else kept under version control.) One user writes a test to assert an invariant which they expect to be maintained. Other users execute the test later to check whether the invariant has been broken. If the test depends on any variables other than source files (non-hermetic), its value is diminished, because the later users cannot be sure their changes are at fault when the test stops passing. + +Therefore the outcome of a test must depend only on: + +- source files on which the test has a declared dependency +- products of the build system on which the test has a declared dependency +- resources whose behavior is guaranteed by the test runner to remain constant + +Currently, such behavior is not enforced. However, test runners reserve the right to add such enforcement in the future. + +## Role of the build system + +Test rules are analogous to binary rules in that each must yield an executable program. For some languages, this is a stub program which combines a language-specific harness with the test code. Test rules must produce other outputs as well. In addition to the primary test executable, the test runner will need a manifest of **runfiles**, input files which should be made available to the test at runtime, and it may need information about the type, size, and tags of a test. + +The build system may use the runfiles to deliver code as well as data. (This might be used as an optimization to make each test binary smaller by sharing files across tests, such as through the use of dynamic linking.) The build system should ensure that the generated executable loads these files via the runfiles image provided by the test runner, rather than hardcoded references to absolute locations in the source or output tree. + +## Role of the test runner + +From the point of view of the test runner, each test is a program which can be invoked with `execve()`. There may be other ways to execute tests; for example, an IDE might allow the execution of Java tests in-process. However, the result of running the test as a standalone process must be considered authoritative. If a test process runs to completion and terminates normally with an exit code of zero, the test has passed. Any other result is considered a test failure. In particular, writing any of the strings `PASS` or `FAIL` to stdout has no significance to the test runner. + +If a test takes too long to execute, exceeds some resource limit, or the test runner otherwise detects prohibited behavior, it may choose to kill the test and treat the run as a failure. The runner must not report the test as passing after sending a signal to the test process or any children thereof. + +The whole test target (not individual methods or tests) is given a limited amount of time to run to completion. The time limit for a test is based on its [`timeout`](/reference/be/common-definitions#test.timeout) attribute according to the following table: + +| timeout | Time Limit (sec.) | +| -------- | ----------------- | +| short | 60 | +| moderate | 300 | +| long | 900 | +| eternal | 3600 | + +Tests which do not explicitly specify a timeout have one implied based on the test's [`size`](/reference/be/common-definitions#test.size) as follows: + +| size | Implied timeout label | +| -------- | --------------------- | +| small | short | +| medium | moderate | +| large | long | +| enormous | eternal | + +A "large" test with no explicit timeout setting will be allotted 900 seconds to run. A "medium" test with a timeout of "short" will be allotted 60 seconds. + +Unlike `timeout`, the `size` additionally determines the assumed peak usage of other resources (like RAM) when running the test locally, as described in [Common definitions](/reference/be/common-definitions#common-attributes-tests). + +All combinations of `size` and `timeout` labels are legal, so an "enormous" test may be declared to have a timeout of "short". Presumably it would do some really horrible things very quickly. + +Tests may return arbitrarily fast regardless of timeout. A test is not penalized for an overgenerous timeout, although a warning may be issued: you should generally set your timeout as tight as you can without incurring any flakiness. + +The test timeout can be overridden with the `--test_timeout` bazel flag when manually running under conditions that are known to be slow. The `--test_timeout` values are in seconds. For example, `--test_timeout=120` sets the test timeout to two minutes. + +There is also a recommended lower bound for test timeouts as follows: + +| timeout | Time minimum (sec.) | +| -------- | ------------------- | +| short | 0 | +| moderate | 30 | +| long | 300 | +| eternal | 900 | + +For example, if a "moderate" test completes in 5.5s, consider setting `timeout = "short"` or `size = "small"`. Using the bazel `--test_verbose_timeout_warnings` command line option will show the tests whose specified size is too big. + +Test sizes and timeouts are specified in the BUILD file according to the specification [here](/reference/be/common-definitions#common-attributes-tests). If unspecified, a test's size will default to "medium". + +If the main process of a test exits, but some of its children are still running, the test runner should consider the run complete and count it as a success or failure based on the exit code observed from the main process. The test runner may kill any stray processes. Tests should not leak processes in this fashion. + +## Test sharding + +Tests can be parallelized via test sharding. See [`--test_sharding_strategy`](/reference/command-line-reference#flag--test_sharding_strategy) and [`shard_count`](/reference/be/common-definitions#common-attributes-tests) to enable test sharding. When sharding is enabled, the test runner is launched once per shard. The environment variable [`TEST_TOTAL_SHARDS`](#initial-conditions) is the number of shards, and [`TEST_SHARD_INDEX`](#initial-conditions) is the shard index, beginning at 0. Runners use this information to select which tests to run - for example, using a round-robin strategy. Not all test runners support sharding. If a runner supports sharding, it must create or update the last modified date of the file specified by [`TEST_SHARD_STATUS_FILE`](#initial-conditions). Otherwise, if [`--incompatible_check_sharding_support`](/reference/command-line-reference#flag--incompatible_check_sharding_support) is enabled, Bazel will fail the test if it is sharded. + +## Initial conditions + +When executing a test, the test runner must establish certain initial conditions. + +The test runner must invoke each test with the path to the test executable in `argv[0]`. This path must be relative and beneath the test's current directory (which is in the runfiles tree, see below). The test runner should not pass any other arguments to a test unless the user explicitly requests it. + +The initial environment block shall be composed as follows: + +| Variable | Value | Status | +| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | +| `HOME` | value of `$TEST_TMPDIR` | recommended | +| `LANG` | *unset* | required | +| `LANGUAGE` | *unset* | required | +| `LC_ALL` | *unset* | required | +| `LC_COLLATE` | *unset* | required | +| `LC_CTYPE` | *unset* | required | +| `LC_MESSAGES` | *unset* | required | +| `LC_MONETARY` | *unset* | required | +| `LC_NUMERIC` | *unset* | required | +| `LC_TIME` | *unset* | required | +| `LD_LIBRARY_PATH` | colon-separated list of directories containing shared libraries | optional | +| `JAVA_RUNFILES` | value of `$TEST_SRCDIR` | deprecated | +| `LOGNAME` | value of `$USER` | required | +| `PATH` | `/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.` | recommended | +| `PWD` | `$TEST_SRCDIR/workspace-name` | recommended | +| `SHLVL` | `2` | recommended | +| `TEST_INFRASTRUCTURE_FAILURE_FILE` | absolute path to a private file in a writable directory (This file should only be used to report failures originating from the testing infrastructure, not as a general mechanism for reporting flaky failures of tests. In this context, testing infrastructure is defined as systems or libraries that are not test-specific, but can cause test failures by malfunctioning. The first line is the name of the testing infrastructure component that caused the failure, the second one a human-readable description of the failure. Additional lines are ignored.) | optional | +| `TEST_LOGSPLITTER_OUTPUT_FILE` | absolute path to a private file in a writable directory (used to write Logsplitter protobuffer log) | optional | +| `TEST_PREMATURE_EXIT_FILE` | absolute path to a private file in a writable directory (used for catching calls to `exit()`) | optional | +| `TEST_RANDOM_SEED` | If the `--runs_per_test` option is used, `TEST_RANDOM_SEED` is set to the `run number` (starting with 1) for each individual test run. | optional | +| `TEST_RUN_NUMBER` | If the `--runs_per_test` option is used, `TEST_RUN_NUMBER` is set to the `run number` (starting with 1) for each individual test run. | optional | +| `TEST_TARGET` | The name of the target being tested | optional | +| `TEST_SIZE` | The test [`size`](#size) | optional | +| `TEST_TIMEOUT` | The test [`timeout`](#timeout) in seconds | optional | +| `TEST_SHARD_INDEX` | shard index, if [`sharding`](#test-sharding) is used | optional | +| `TEST_SHARD_STATUS_FILE` | path to file to touch to indicate support for [`sharding`](#test-sharding) | optional | +| `TEST_SRCDIR` | absolute path to the base of the runfiles tree | required | +| `TEST_TOTAL_SHARDS` | total [`shard count`](/reference/be/common-definitions#test.shard_count), if [`sharding`](#test-sharding) is used | optional | +| `TEST_TMPDIR` | absolute path to a private writable directory | required | +| `TEST_WORKSPACE` | the local repository's workspace name | optional | +| `TEST_UNDECLARED_OUTPUTS_DIR` | absolute path to a private writable directory (used to write undeclared test outputs). Any files written to the `TEST_UNDECLARED_OUTPUTS_DIR` directory will be zipped up and added to an `outputs.zip` file under `bazel-testlogs`. | optional | +| `TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR` | absolute path to a private writable directory (used to write undeclared test output annotation `.part` and `.pb` files). | optional | +| `TEST_WARNINGS_OUTPUT_FILE` | absolute path to a private file in a writable directory (used to write test target warnings) | optional | +| `TESTBRIDGE_TEST_ONLY` | value of [`--test_filter`](/docs/user-manual#flag--test_filter), if specified | optional | +| `TZ` | `UTC` | required | +| `USER` | value of `getpwuid(getuid())->pw_name` | required | +| `XML_OUTPUT_FILE` | Location to which test actions should write a test result XML output file. Otherwise, Bazel generates a default XML output file wrapping the test log as part of the test action. The XML schema is based on the [JUnit test result schema](https://windyroad.com.au/dl/Open%20Source/JUnit.xsd). | optional | +| `BAZEL_TEST` | Signifies test executable is being driven by `bazel test` | required | + +The environment may contain additional entries. Tests should not depend on the presence, absence, or value of any environment variable not listed above. + +The initial working directory shall be `$TEST_SRCDIR/$TEST_WORKSPACE`. + +The current process id, process group id, session id, and parent process id are unspecified. The process may or may not be a process group leader or a session leader. The process may or may not have a controlling terminal. The process may have zero or more running or unreaped child processes. The process should not have multiple threads when the test code gains control. + +File descriptor 0 (`stdin`) shall be open for reading, but what it is attached to is unspecified. Tests must not read from it. File descriptors 1 (`stdout`) and 2 (`stderr`) shall be open for writing, but what they are attached to is unspecified. It could be a terminal, a pipe, a regular file, or anything else to which characters can be written. They may share an entry in the open file table (meaning that they cannot seek independently). Tests should not inherit any other open file descriptors. + +The initial umask shall be `022` or `027`. + +No alarm or interval timer shall be pending. + +The initial mask of blocked signals shall be empty. All signals shall be set to their default action. + +The initial resource limits, both soft and hard, should be set as follows: + +| Resource | Limit | +| ------------------- | -------------------------------------- | +| `RLIMIT_AS` | unlimited | +| `RLIMIT_CORE` | unspecified | +| `RLIMIT_CPU` | unlimited | +| `RLIMIT_DATA` | unlimited | +| `RLIMIT_FSIZE` | unlimited | +| `RLIMIT_LOCKS` | unlimited | +| `RLIMIT_MEMLOCK` | unlimited | +| `RLIMIT_MSGQUEUE` | unspecified | +| `RLIMIT_NICE` | unspecified | +| `RLIMIT_NOFILE` | at least 1024 | +| `RLIMIT_NPROC` | unspecified | +| `RLIMIT_RSS` | unlimited | +| `RLIMIT_RTPRIO` | unspecified | +| `RLIMIT_SIGPENDING` | unspecified | +| `RLIMIT_STACK` | unlimited, or 2044KB <= rlim <= 8192KB | + +The initial process times (as returned by `times()`) and resource utilization (as returned by `getrusage()`) are unspecified. + +The initial scheduling policy and priority are unspecified. + +## Role of the host system + +In addition to the aspects of user context under direct control of the test runner, the operating system on which tests execute must satisfy certain properties for a test run to be valid. + +#### Filesystem + +The root directory observed by a test may or may not be the real root directory. + +`/proc` shall be mounted. + +All build tools shall be present at the absolute paths under `/usr` used by a local installation. + +Paths starting with `/home` may not be available. Tests should not access any such paths. + +`/tmp` shall be writable, but tests should avoid using these paths. + +Tests must not assume that any constant path is available for their exclusive use. + +Tests must not assume that atimes are enabled for any mounted filesystem. + +#### Users and groups + +The users root, nobody, and unittest must exist. The groups root, nobody, and eng must exist. + +Tests must be executed as a non-root user. The real and effective user ids must be equal; likewise for group ids. Beyond this, the current user id, group id, user name, and group name are unspecified. The set of supplementary group ids is unspecified. + +The current user id and group id must have corresponding names which can be retrieved with `getpwuid()` and `getgrgid()`. The same may not be true for supplementary group ids. + +The current user must have a home directory. It may not be writable. Tests must not attempt to write to it. + +#### Networking + +The hostname is unspecified. It may or may not contain a dot. Resolving the hostname must give an IP address of the current host. Resolving the hostname cut after the first dot must also work. The hostname localhost must resolve. + +#### Other resources + +Tests are granted at least one CPU core. Others may be available but this is not guaranteed. Other performance aspects of this core are not specified. You can increase the reservation to a higher number of CPU cores by adding the tag "cpu:n" (where n is a positive number) to a test rule. If a machine has less total CPU cores than requested, Bazel will still run the test. If a test uses [sharding](#test-sharding), each individual shard will reserve the number of CPU cores specified here. + +Tests may create subprocesses, but not process groups or sessions. + +There is a limit on the number of input files a test may consume. This limit is subject to change, but is currently in the range of tens of thousands of inputs. + +#### Time and date + +The current time and date are unspecified. The system timezone is unspecified. + +X Windows may or may not be available. Tests that need an X server should start Xvfb. + +## Test interaction with the filesystem + +All file paths specified in test environment variables point to somewhere on the local filesystem, unless otherwise specified. + +Tests should create files only within the directories specified by `$TEST_TMPDIR` and `$TEST_UNDECLARED_OUTPUTS_DIR` (if set). + +These directories will be initially empty. + +Tests must not attempt to remove, chmod, or otherwise alter these directories. + +These directories may be a symbolic links. + +The filesystem type of `$TEST_TMPDIR/.` remains unspecified. + +Tests may also write .part files to the `$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR` to annotate undeclared output files. + +In rare cases, a test may be forced to create files in `/tmp`. For example, [path length limits for Unix domain sockets](https://serverfault.com/questions/641347) typically require creating the socket under `/tmp`. Bazel will be unable to track such files; the test itself must take care to be hermetic, to use unique paths to avoid colliding with other, simultaneously running tests and non-test processes, and to clean up the files it creates in `/tmp`. + +Some popular testing frameworks, such as [JUnit4 `TemporaryFolder`](https://junit.org/junit4/javadoc/latest/org/junit/rules/TemporaryFolder.html) or [Go `TempDir`](https://golang.org/pkg/testing/#T.TempDir), have their own ways to create a temporary directory under `/tmp`. These testing frameworks include functionality that cleans up files in `/tmp`, so you may use them even though they create files outside of `TEST_TMPDIR`. + +Tests must access inputs through the **runfiles** mechanism, or other parts of the execution environment which are specifically intended to make input files available. + +Tests must not access other outputs of the build system at paths inferred from the location of their own executable. + +It is unspecified whether the runfiles tree contains regular files, symbolic links, or a mixture. The runfiles tree may contain symlinks to directories. Tests should avoid using paths containing `..` components within the runfiles tree. + +No directory, file, or symlink within the runfiles tree (including paths which traverse symlinks) should be writable. (It follows that the initial working directory should not be writable.) Tests must not assume that any part of the runfiles is writable, or owned by the current user (for example, `chmod` and `chgrp` may fail). + +The runfiles tree (including paths which traverse symlinks) must not change during test execution. Parent directories and filesystem mounts must not change in any way which affects the result of resolving a path within the runfiles tree. + +In order to catch early exit, a test may create a file at the path specified by `TEST_PREMATURE_EXIT_FILE` upon start and remove it upon exit. If Bazel sees the file when the test finishes, it will assume that the test exited prematurely and mark it as having failed. + +## Execution platform + +The [execution platform](/extending/platforms) for a test action is determined via [toolchain resolution](/extending/toolchains#toolchain-resolution), just like for any other action. Each test rule has an implicitly defined [`test` exec group](/extending/exec-groups#exec-groups-for-native-rules) that, unless overridden, has a mandatory toolchain requirement on `@bazel_tools//tools/test:default_test_toolchain_type`. Toolchains of this type do not carry any data in the form of providers, but can be used to influence the execution platform of the test action. By default, Bazel registers two such toolchains: + +- If `--@bazel_tools//tools/test:incompatible_use_default_test_toolchain` is disabled (the current default), the active test toolchain is `@bazel_tools//tools/test:legacy_test_toolchain`. This toolchain does not impose any constraints and thus test actions without manually specified exec constraints are configured for the first registered execution platform. This is often not the intended behavior in multi-platform builds as it can result in e.g. a test binary built for Linux on a Windows machine to be executed on Windows. +- If `--@bazel_tools//tools/test:incompatible_use_default_test_toolchain` is enabled, the active test toolchain is `@bazel_tools//tools/test:default_test_toolchain`. This toolchain requires an execution platform to match all the constraints of the test rule's target platform. In particular, the target platform is compatible with this toolchain if it is also registered as an execution platform. If no such platform is found, the test rule fails with a toolchain resolution error. + +Users can register additional toolchains for this type to influence this behavior and their toolchains will take precedence over the default ones. Test rule authors can define their own test toolchain type and also register a default toolchain for it. + +## Tag conventions + +Some tags in the test rules have a special meaning. See also the [Bazel Build Encyclopedia on the `tags` attribute](/reference/be/common-definitions#common.tags). + +| Tag | Meaning | +| ----------- | -------------------------------------------------------------------------------------------------------------- | +| `exclusive` | run no other test at the same time | +| `external` | test has an external dependency; disable test caching | +| `large` | `test_suite` convention; suite of large tests | +| `manual *` | don't include test target in wildcard target patterns like `:...`, `:*`, or `:all` | +| `medium` | `test_suite` convention; suite of medium tests | +| `small` | `test_suite` convention; suite of small tests | +| `smoke` | `test_suite` convention; means it should be run before committing code changes into the version control system | + +Note: bazel `query` does not respect the manual tag. + +## Runfiles + +In the following, assume there is a \*\_binary() rule labeled `//foo/bar:unittest`, with a run-time dependency on the rule labeled `//deps/server:server`. + +#### Location + +The runfiles directory for a target `//foo/bar:unittest` is the directory `$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles`. This path is referred to as the `runfiles_dir`. + +#### Dependencies + +The runfiles directory is declared as a compile-time dependency of the `*_binary()` rule. The runfiles directory itself depends on the set of BUILD files that affect the `*_binary()` rule or any of its compile-time or run-time dependencies. Modifying source files does not affect the structure of the runfiles directory, and thus does not trigger any rebuilding. + +#### Contents + +The runfiles directory contains the following: + +- **Symlinks to run-time dependencies**: each OutputFile and CommandRule that is a run-time dependency of the `*_binary()` rule is represented by one symlink in the runfiles directory. The name of the symlink is `$(WORKSPACE)/package_name/rule_name`. For example, the symlink for server would be named `$(WORKSPACE)/deps/server/server`, and the full path would be `$(WORKSPACE)/foo/bar/unittest.runfiles/$(WORKSPACE)/deps/server/server`. The destination of the symlink is the OutputFileName() of the OutputFile or CommandRule, expressed as an absolute path. Thus, the destination of the symlink might be `$(WORKSPACE)/linux-dbg/deps/server/42/server`. diff --git a/release/backward-compatibility.mdx b/release/backward-compatibility.mdx index af653ccd..42f40956 100644 --- a/release/backward-compatibility.mdx +++ b/release/backward-compatibility.mdx @@ -2,75 +2,51 @@ title: 'Backward Compatibility' --- +This page provides information about how to handle backward compatibility, including migrating from one release to another and how to communicate incompatible changes. - -This page provides information about how to handle backward compatibility, -including migrating from one release to another and how to communicate -incompatible changes. - -Bazel is evolving. Minor versions released as part of an [LTS major -version](/release#bazel-versioning) are fully backward-compatible. New major LTS -releases may contain incompatible changes that require some migration effort. -For more information about Bazel's release model, please check out the [Release -Model](/release) page. +Bazel is evolving. Minor versions released as part of an [LTS major version](/release#bazel-versioning) are fully backward-compatible. New major LTS releases may contain incompatible changes that require some migration effort. For more information about Bazel's release model, please check out the [Release Model](/release) page. ## Summary -1. It is recommended to use `--incompatible_*` flags for breaking changes. -1. For every `--incompatible_*` flag, a GitHub issue explains the change in - behavior and aims to provide a migration recipe. -1. Incompatible flags are recommended to be back-ported to the latest LTS - release without enabling the flag by default. -1. APIs and behavior guarded by an `--experimental_*` flag can change at any - time. -1. Never run production builds with `--experimental_*` or `--incompatible_*` - flags. +1. It is recommended to use `--incompatible_*` flags for breaking changes. +2. For every `--incompatible_*` flag, a GitHub issue explains the change in behavior and aims to provide a migration recipe. +3. Incompatible flags are recommended to be back-ported to the latest LTS release without enabling the flag by default. +4. APIs and behavior guarded by an `--experimental_*` flag can change at any time. +5. Never run production builds with `--experimental_*` or `--incompatible_*` flags. ## How to follow this policy -* [For Bazel users - how to update Bazel](/install/bazelisk) -* [For contributors - best practices for incompatible changes](/contribute/breaking-changes) -* [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) +- [For Bazel users - how to update Bazel](/install/bazelisk) +- [For contributors - best practices for incompatible changes](/contribute/breaking-changes) +- [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) ## What is stable functionality? -In general, APIs or behaviors without `--experimental_...` flags are considered -stable, supported features in Bazel. +In general, APIs or behaviors without `--experimental_...` flags are considered stable, supported features in Bazel. This includes: -* Starlark language and APIs -* Rules bundled with Bazel -* Bazel APIs such as Remote Execution APIs or Build Event Protocol -* Flags and their semantics +- Starlark language and APIs +- Rules bundled with Bazel +- Bazel APIs such as Remote Execution APIs or Build Event Protocol +- Flags and their semantics ## Incompatible changes and migration recipes -For every incompatible change in a new release, the Bazel team aims to provide a -_migration recipe_ that helps you update your code (`BUILD` and `.bzl` files, as -well as any Bazel usage in scripts, usage of Bazel API, and so on). +For every incompatible change in a new release, the Bazel team aims to provide a *migration recipe* that helps you update your code (`BUILD` and `.bzl` files, as well as any Bazel usage in scripts, usage of Bazel API, and so on). -Incompatible changes should have an associated `--incompatible_*` flag and a -corresponding GitHub issue. +Incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue. -The incompatible flag and relevant changes are recommended to be back-ported to -the latest LTS release without enabling the flag by default. This allows users -to migrate for the incompatible changes before the next LTS release is -available. +The incompatible flag and relevant changes are recommended to be back-ported to the latest LTS release without enabling the flag by default. This allows users to migrate for the incompatible changes before the next LTS release is available. ## Communicating incompatible changes -The primary source of information about incompatible changes are GitHub issues -marked with an ["incompatible-change" -label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). +The primary source of information about incompatible changes are GitHub issues marked with an ["incompatible-change" label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). For every incompatible change, the issue specifies the following: -* Name of the flag controlling the incompatible change -* Description of the changed functionality -* Migration recipe +- Name of the flag controlling the incompatible change +- Description of the changed functionality +- Migration recipe -When an incompatible change is ready for migration with Bazel at HEAD -(therefore, also with the next Bazel rolling release), it should be marked with -the `migration-ready` label. The incompatible change issue is closed when the -incompatible flag is flipped at HEAD. +When an incompatible change is ready for migration with Bazel at HEAD (therefore, also with the next Bazel rolling release), it should be marked with the `migration-ready` label. The incompatible change issue is closed when the incompatible flag is flipped at HEAD. diff --git a/release/index.mdx b/release/index.mdx index f58e8f5f..0d2629e7 100644 --- a/release/index.mdx +++ b/release/index.mdx @@ -2,66 +2,48 @@ title: 'Release Model' --- - - -As announced in [the original blog -post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel -4.0 and higher versions provides support for two release tracks: rolling -releases and long term support (LTS) releases. This page covers the latest -information about Bazel's release model. +As announced in [the original blog post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel 4.0 and higher versions provides support for two release tracks: rolling releases and long term support (LTS) releases. This page covers the latest information about Bazel's release model. ## Support matrix -| LTS release | Support stage | Latest version | End of support | -| ----------- | ------------- | -------------- | -------------- | -| Bazel 9 | Rolling| [Check rolling release page](/release/rolling) | N/A | -| Bazel 8 | Active| [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | -| Bazel 7 | Maintenance| [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | -| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | -| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | -| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | +| LTS release | Support stage | Latest version | End of support | +| ----------- | ------------- | --------------------------------------------------------------- | -------------- | +| Bazel 9 | Rolling | [Check rolling release page](/release/rolling) | N/A | +| Bazel 8 | Active | [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | +| Bazel 7 | Maintenance | [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | +| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | +| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | +| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | -All Bazel LTS releases can be found on the [release -page](https://github.com/bazelbuild/bazel/releases) on GitHub. +All Bazel LTS releases can be found on the [release page](https://github.com/bazelbuild/bazel/releases) on GitHub. -Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are -recommended to upgrade to the latest LTS release or use rolling releases if you -want to keep up with the latest changes at HEAD. +Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are recommended to upgrade to the latest LTS release or use rolling releases if you want to keep up with the latest changes at HEAD. ## Release versioning -Bazel uses a _major.minor.patch_ [Semantic -Versioning](https://semver.org/) scheme. +Bazel uses a *major.minor.patch* [Semantic Versioning](https://semver.org/) scheme. -* A _major release_ contains features that are not backward compatible with - the previous release. Each major Bazel version is an LTS release. -* A _minor release_ contains backward-compatible bug fixes and features - back-ported from the main branch. -* A _patch release_ contains critical bug fixes. +- A *major release* contains features that are not backward compatible with the previous release. Each major Bazel version is an LTS release. +- A *minor release* contains backward-compatible bug fixes and features back-ported from the main branch. +- A *patch release* contains critical bug fixes. -Additionally, pre-release versions are indicated by appending a hyphen and a -date suffix to the next major version number. +Additionally, pre-release versions are indicated by appending a hyphen and a date suffix to the next major version number. For example, a new release of each type would result in these version numbers: -* Major: 6.0.0 -* Minor: 6.1.0 -* Patch: 6.1.2 -* Pre-release: 7.0.0-pre.20230502.1 +- Major: 6.0.0 +- Minor: 6.1.0 +- Patch: 6.1.2 +- Pre-release: 7.0.0-pre.20230502.1 ## Support stages For each major Bazel version, there are four support stages: -* **Rolling**: This major version is still in pre-release, the Bazel team - publishes rolling releases from HEAD. -* **Active**: This major version is the current active LTS release. The Bazel - team backports important features and bug fixes into its minor releases. -* **Maintenance**: This major version is an old LTS release in maintenance - mode. The Bazel team only promises to backport critical bug fixes for - security issues and OS-compatibility issues into this LTS release. -* **Deprecated**: The Bazel team no longer provides support for this major - version, all users should migrate to newer Bazel LTS releases. +- **Rolling**: This major version is still in pre-release, the Bazel team publishes rolling releases from HEAD. +- **Active**: This major version is the current active LTS release. The Bazel team backports important features and bug fixes into its minor releases. +- **Maintenance**: This major version is an old LTS release in maintenance mode. The Bazel team only promises to backport critical bug fixes for security issues and OS-compatibility issues into this LTS release. +- **Deprecated**: The Bazel team no longer provides support for this major version, all users should migrate to newer Bazel LTS releases. ## Release cadence @@ -69,148 +51,90 @@ Bazel regularly publish releases for two release tracks. ### Rolling releases -* Rolling releases are coordinated with Google Blaze release and are released - from HEAD around every two weeks. It is a preview of the next Bazel LTS - release. -* Rolling releases can ship incompatible changes. Incompatible flags are - recommended for major breaking changes, rolling out incompatible changes - should follow our [backward compatibility - policy](/release/backward-compatibility). +- Rolling releases are coordinated with Google Blaze release and are released from HEAD around every two weeks. It is a preview of the next Bazel LTS release. +- Rolling releases can ship incompatible changes. Incompatible flags are recommended for major breaking changes, rolling out incompatible changes should follow our [backward compatibility policy](/release/backward-compatibility). ### LTS releases -* _Major release_: A new LTS release is expected to be cut from HEAD roughly - every - 12 months. Once a new LTS release is out, it immediately enters the Active - stage, and the previous LTS release enters the Maintenance stage. -* _Minor release_: New minor verions on the Active LTS track are expected to - be released once every 2 months. -* _Patch release_: New patch versions for LTS releases in Active and - Maintenance stages are expected to be released on demand for critical bug - fixes. -* A Bazel LTS release enters the Deprecated stage after being in ​​the - Maintenance stage for 2 years. - -For planned releases, please check our [release -issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) -on Github. +- *Major release*: A new LTS release is expected to be cut from HEAD roughly every 12 months. Once a new LTS release is out, it immediately enters the Active stage, and the previous LTS release enters the Maintenance stage. +- *Minor release*: New minor verions on the Active LTS track are expected to be released once every 2 months. +- *Patch release*: New patch versions for LTS releases in Active and Maintenance stages are expected to be released on demand for critical bug fixes. +- A Bazel LTS release enters the Deprecated stage after being in ​​the Maintenance stage for 2 years. + +For planned releases, please check our [release issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) on Github. ## Release procedure & policies -For rolling releases, the process is straightforward: about every two weeks, a -new release is created, aligning with the same baseline as the Google internal -Blaze release. Due to the rapid release schedule, we don't backport any changes -to rolling releases. +For rolling releases, the process is straightforward: about every two weeks, a new release is created, aligning with the same baseline as the Google internal Blaze release. Due to the rapid release schedule, we don't backport any changes to rolling releases. For LTS releases, the procedure and policies below are followed: -1. Determine a baseline commit for the release. - * For a new major LTS release, the baseline commit is the HEAD of the main - branch. - * For a minor or patch release, the baseline commit is the HEAD of the - current latest version of the same LTS release. -1. Create a release branch in the name of `release-` from the baseline - commit. -1. Backport changes via PRs to the release branch. - * The community can suggest certain commits to be back-ported by replying - "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential - release blockers, the Bazel team triages them and decide whether to - back-port the commits. - * Only backward-compatible commits on the main branch can be back-ported, - additional minor changes to resolve merge conflicts are acceptable. -1. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. - * Bazel maintainers can request to cherry-pick specific commit(s) - to a release branch. This process is initiated by creating a - cherry-pick request on GitHub. Here's how to do it. - 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=&labels=&projects=&template=cherry_pick_request.yml) - 2. Fill in the request details - * Title: Provide a concise and descriptive title for the request. - * Commit ID(s): Enter the ID(s) of the commit(s) you want to - cherry-pick. If there are multiple commits, then separate - them with commas. - * Category: Specify the category of the request. - * Reviewer(s): For multiple reviewers, separate their GitHub - ID's with commas. - 3. Set the milestone - * Find the "Milestone" section and click the setting. - * Select the appropriate X.Y.Z release blockers. This action - triggers the cherry-pick bot to process your request - for the "release-X.Y.Z" branch. - 4. Submit the Issue - * Once all details are filled in and the miestone is set, - submit the issue. - - * The cherry-pick bot will process the request and notify - if the commit(s) are eligible for cherry-picking. If - the commits are cherry-pickable, which means there's no - merge conflict while cherry-picking the commit, then - the bot will create a new pull request. When the pull - request is approved by a member of the Bazel team, the - commits are cherry-picked and merged to the release branch. - For a visual example of a completed cherry-pick request, - refer to this - [example](https://github.com/bazelbuild/bazel/issues/20230) - . - -1. Identify release blockers and fix issues found on the release branch. - * The release branch is tested with the same test suite in - [postsubmit](https://buildkite.com/bazel/bazel-bazel) and - [downstream test pipeline] - (https://buildkite.com/bazel/bazel-at-head-plus-downstream) - on Bazel CI. The Bazel team monitors testing results of the release - branch and fixes any regressions found. -1. Create a new release candidate from the release branch when all known - release blockers are resolved. - * The release candidate is announced on - [bazel-discuss](https://groups.google.com/g/bazel-discuss), - the Bazel team monitors community bug reports for the candidate. - * If new release blockers are identified, go back to the last step and - create a new release candidate after resolving all the issues. - * New features are not allowed to be added to the release branch after the - first release candidate is created; cherry-picks are limited to critical - fixes only. If a cherry-pick is needed, the requester must answer the - following questions: Why is this change critical, and what benefits does - it provide? What is the likelihood of this change introducing a - regression? -1. Push the release candidate as the official release if no further release - blockers are found - * For patch releases, push the release at least two business days after - the last release candidate is out. - * For major and minor releases, push the release two business days after - the last release candidate is out, but not earlier than one week after - the first release candidate is out. - * The release is only pushed on a day where the next day is a business - day. - * The release is announced on - [bazel-discuss](https://groups.google.com/g/bazel-discuss), - the Bazel team monitors and addresses community bug reports for the new - release. +1. Determine a baseline commit for the release. + + - For a new major LTS release, the baseline commit is the HEAD of the main branch. + - For a minor or patch release, the baseline commit is the HEAD of the current latest version of the same LTS release. + +2. Create a release branch in the name of `release-<version>` from the baseline commit. + +3. Backport changes via PRs to the release branch. + + - The community can suggest certain commits to be back-ported by replying "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential release blockers, the Bazel team triages them and decide whether to back-port the commits. + - Only backward-compatible commits on the main branch can be back-ported, additional minor changes to resolve merge conflicts are acceptable. + +4. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. + + - Bazel maintainers can request to cherry-pick specific commit(s) to a release branch. This process is initiated by creating a cherry-pick request on GitHub. Here's how to do it. + + 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=\&labels=\&projects=\&template=cherry_pick_request.yml) + + 2. Fill in the request details + + - Title: Provide a concise and descriptive title for the request. + - Commit ID(s): Enter the ID(s) of the commit(s) you want to cherry-pick. If there are multiple commits, then separate them with commas. + - Category: Specify the category of the request. + - Reviewer(s): For multiple reviewers, separate their GitHub ID's with commas. + + 3. Set the milestone + + - Find the "Milestone" section and click the setting. + - Select the appropriate X.Y.Z release blockers. This action triggers the cherry-pick bot to process your request for the "release-X.Y.Z" branch. + + 4. Submit the Issue + - Once all details are filled in and the miestone is set, submit the issue. + + - The cherry-pick bot will process the request and notify if the commit(s) are eligible for cherry-picking. If the commits are cherry-pickable, which means there's no merge conflict while cherry-picking the commit, then the bot will create a new pull request. When the pull request is approved by a member of the Bazel team, the commits are cherry-picked and merged to the release branch. For a visual example of a completed cherry-pick request, refer to this [example](https://github.com/bazelbuild/bazel/issues/20230) . + +5. Identify release blockers and fix issues found on the release branch. + + - The release branch is tested with the same test suite in [postsubmit](https://buildkite.com/bazel/bazel-bazel) and [downstream test pipeline](https://buildkite.com/bazel/bazel-at-head-plus-downstream) on Bazel CI. The Bazel team monitors testing results of the release branch and fixes any regressions found. + +6. Create a new release candidate from the release branch when all known release blockers are resolved. + + - The release candidate is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors community bug reports for the candidate. + - If new release blockers are identified, go back to the last step and create a new release candidate after resolving all the issues. + - New features are not allowed to be added to the release branch after the first release candidate is created; cherry-picks are limited to critical fixes only. If a cherry-pick is needed, the requester must answer the following questions: Why is this change critical, and what benefits does it provide? What is the likelihood of this change introducing a regression? + +7. Push the release candidate as the official release if no further release blockers are found + + - For patch releases, push the release at least two business days after the last release candidate is out. + - For major and minor releases, push the release two business days after the last release candidate is out, but not earlier than one week after the first release candidate is out. + - The release is only pushed on a day where the next day is a business day. + - The release is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors and addresses community bug reports for the new release. ## Report regressions -If a user finds a regression in a new Bazel release, release candidate or even -Bazel at HEAD, please file a bug on -[GitHub](https://github.com/bazelbuild/bazel/issues). You can use -Bazelisk to bisect the culprit commit and include this information in the bug -report. +If a user finds a regression in a new Bazel release, release candidate or even Bazel at HEAD, please file a bug on [GitHub](https://github.com/bazelbuild/bazel/issues). You can use Bazelisk to bisect the culprit commit and include this information in the bug report. -For example, if your build succeeds with Bazel 6.1.0 but fails with the second -release candidate of 6.2.0, you can do bisect via +For example, if your build succeeds with Bazel 6.1.0 but fails with the second release candidate of 6.2.0, you can do bisect via ```bash bazelisk --bisect=6.1.0..release-6.2.0rc2 build //foo:bar ``` -You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run -corresponding bazel commands to reset the build state if it's needed to -reproduce the issue. For more details, check out documentation about Bazelisk -[bisect feature] (https://github.com/bazelbuild/bazelisk#--bisect). +You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run corresponding bazel commands to reset the build state if it's needed to reproduce the issue. For more details, check out documentation about Bazelisk [bisect feature](https://github.com/bazelbuild/bazelisk#--bisect). -Remember to upgrade Bazelisk to the latest version to use the bisect -feature. +Remember to upgrade Bazelisk to the latest version to use the bisect feature. ## Rule compatibility -If you are a rule authors and want to maintain compatibility with different -Bazel versions, please check out the [Rule -Compatibility](/release/rule-compatibility) page. +If you are a rule authors and want to maintain compatibility with different Bazel versions, please check out the [Rule Compatibility](/release/rule-compatibility) page. diff --git a/release/rolling.mdx b/release/rolling.mdx index 8e79d6f5..4cb413ad 100644 --- a/release/rolling.mdx +++ b/release/rolling.mdx @@ -2,13 +2,6 @@ title: 'Rolling Releases' --- +This page contains an overview of all rolling releases, as per our [release policy](https://bazel.build/release#rolling-releases). [Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use these releases. - -This page contains an overview of all rolling releases, as per our -[release policy](https://bazel.build/release#rolling-releases). -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use -these releases. - -## Index - - +## Index diff --git a/release/rule-compatibility.mdx b/release/rule-compatibility.mdx index 05a8a95e..93027953 100644 --- a/release/rule-compatibility.mdx +++ b/release/rule-compatibility.mdx @@ -2,89 +2,55 @@ title: 'Rule Compatibility' --- +Bazel Starlark rules can break compatibility with Bazel LTS releases in the following two scenarios: +1. The rule breaks compatibility with future LTS releases because a feature it depends on is removed from Bazel at HEAD. +2. The rule breaks compatibility with the current or older LTS releases because a feature it depends on is only available in newer Bazel LTS releases. -Bazel Starlark rules can break compatibility with Bazel LTS releases in the -following two scenarios: - -1. The rule breaks compatibility with future LTS releases because a feature it - depends on is removed from Bazel at HEAD. -1. The rule breaks compatibility with the current or older LTS releases because - a feature it depends on is only available in newer Bazel LTS releases. - -Meanwhile, the rule itself can ship incompatible changes for their users as -well. When combined with breaking changes in Bazel, upgrading the rule version -and Bazel version can often be a source of frustration for Bazel users. This -page covers how rules authors should maintain rule compatibility with Bazel to -make it easier for users to upgrade Bazel and rules. +Meanwhile, the rule itself can ship incompatible changes for their users as well. When combined with breaking changes in Bazel, upgrading the rule version and Bazel version can often be a source of frustration for Bazel users. This page covers how rules authors should maintain rule compatibility with Bazel to make it easier for users to upgrade Bazel and rules. ## Manageable migration process -While it's obviously not feasible to guarantee compatibility between every -version of Bazel and every version of the rule, our aim is to ensure that the -migration process remains manageable for Bazel users. A manageable migration -process is defined as a process where **users are not forced to upgrade the -rule's major version and Bazel's major version simultaneously**, thereby -allowing users to handle incompatible changes from one source at a time. +While it's obviously not feasible to guarantee compatibility between every version of Bazel and every version of the rule, our aim is to ensure that the migration process remains manageable for Bazel users. A manageable migration process is defined as a process where **users are not forced to upgrade the rule's major version and Bazel's major version simultaneously**, thereby allowing users to handle incompatible changes from one source at a time. For example, with the following compatibility matrix: -* Migrating from rules_foo 1.x + Bazel 4.x to rules_foo 2.x + Bazel 5.x is not - considered manageable, as the users need to upgrade the major version of - rules_foo and Bazel at the same time. -* Migrating from rules_foo 2.x + Bazel 5.x to rules_foo 3.x + Bazel 6.x is - considered manageable, as the users can first upgrade rules_foo from 2.x to - 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to - 6.x. +- Migrating from rules\_foo 1.x + Bazel 4.x to rules\_foo 2.x + Bazel 5.x is not considered manageable, as the users need to upgrade the major version of rules\_foo and Bazel at the same time. +- Migrating from rules\_foo 2.x + Bazel 5.x to rules\_foo 3.x + Bazel 6.x is considered manageable, as the users can first upgrade rules\_foo from 2.x to 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to 6.x. -| | rules_foo 1.x | rules_foo 2.x | rules_foo 3.x | HEAD | -| --- | --- | --- | --- | --- | -| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | -| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | -| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | -| HEAD | ❌ | ❌ | ❌ | ✅ | +| | rules\_foo 1.x | rules\_foo 2.x | rules\_foo 3.x | HEAD | +| --------- | -------------- | -------------- | -------------- | ---- | +| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | +| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | +| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | +| HEAD | ❌ | ❌ | ❌ | ✅ | -❌: No version of the major rule version is compatible with the Bazel LTS -release. +❌: No version of the major rule version is compatible with the Bazel LTS release. -✅: At least one version of the rule is compatible with the latest version of the -Bazel LTS release. +✅: At least one version of the rule is compatible with the latest version of the Bazel LTS release. ## Best practices -As Bazel rules authors, you can ensure a manageable migration process for users -by following these best practices: - -1. The rule should follow [Semantic - Versioning](https://semver.org/): minor versions of the same - major version are backward compatible. -1. The rule at HEAD should be compatible with the latest Bazel LTS release. -1. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, - you can - * Set up your own CI testing with Bazel at HEAD - * Add your project to [Bazel downstream - testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); - the Bazel team files issues to your project if breaking changes in Bazel - affect your project, and you must follow our [downstream project - policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) - to address issues timely. -1. The latest major version of the rule must be compatible with the latest - Bazel LTS release. -1. A new major version of the rule should be compatible with the last Bazel LTS - release supported by the previous major version of the rule. - -Achieving 2. and 3. is the most important task since it allows achieving 4. and -5. naturally. - -To make it easier to keep compatibility with both Bazel at HEAD and the latest -Bazel LTS release, rules authors can: - -* Request backward-compatible features to be back-ported to the latest LTS - release, check out [release process](/release#release-procedure-policies) - for more details. -* Use [bazel_features](https://github.com/bazel-contrib/bazel_features) - to do Bazel feature detection. - -In general, with the recommended approaches, rules should be able to migrate for -Bazel incompatible changes and make use of new Bazel features at HEAD without -dropping compatibility with the latest Bazel LTS release. +As Bazel rules authors, you can ensure a manageable migration process for users by following these best practices: + +1. The rule should follow [Semantic Versioning](https://semver.org/): minor versions of the same major version are backward compatible. + +2. The rule at HEAD should be compatible with the latest Bazel LTS release. + +3. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, you can + + - Set up your own CI testing with Bazel at HEAD + - Add your project to [Bazel downstream testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); the Bazel team files issues to your project if breaking changes in Bazel affect your project, and you must follow our [downstream project policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) to address issues timely. + +4. The latest major version of the rule must be compatible with the latest Bazel LTS release. + +5. A new major version of the rule should be compatible with the last Bazel LTS release supported by the previous major version of the rule. + +Achieving 2. and 3. is the most important task since it allows achieving 4. and 5. naturally. + +To make it easier to keep compatibility with both Bazel at HEAD and the latest Bazel LTS release, rules authors can: + +- Request backward-compatible features to be back-ported to the latest LTS release, check out [release process](/release#release-procedure-policies) for more details. +- Use [bazel\_features](https://github.com/bazel-contrib/bazel_features) to do Bazel feature detection. + +In general, with the recommended approaches, rules should be able to migrate for Bazel incompatible changes and make use of new Bazel features at HEAD without dropping compatibility with the latest Bazel LTS release. diff --git a/remote/bep-examples.mdx b/remote/bep-examples.mdx index faf11bf9..68e4d42c 100644 --- a/remote/bep-examples.mdx +++ b/remote/bep-examples.mdx @@ -2,14 +2,9 @@ title: 'Build Event Protocol Examples' --- +The full specification of the Build Event Protocol can be found in its protocol buffer definition. However, it might be helpful to build up some intuition before looking at the specification. - -The full specification of the Build Event Protocol can be found in its protocol -buffer definition. However, it might be helpful to build up some intuition -before looking at the specification. - -Consider a simple Bazel workspace that consists of two empty shell scripts -`foo.sh` and `foo_test.sh` and the following `BUILD` file: +Consider a simple Bazel workspace that consists of two empty shell scripts `foo.sh` and `foo_test.sh` and the following `BUILD` file: ```bash sh_library( @@ -24,58 +19,32 @@ sh_test( ) ``` -When running `bazel test ...` on this project the build graph of the generated -build events will resemble the graph below. The arrows indicate the -aforementioned parent and child relationship. Note that some build events and -most fields have been omitted for brevity. +When running `bazel test ...` on this project the build graph of the generated build events will resemble the graph below. The arrows indicate the aforementioned parent and child relationship. Note that some build events and most fields have been omitted for brevity. ![bep-graph](/docs/images/bep-graph.png "BEP graph") **Figure 1.** BEP graph. -Initially, a `BuildStarted` event is published. The event informs us that the -build was invoked through the `bazel test` command and announces child events: +Initially, a `BuildStarted` event is published. The event informs us that the build was invoked through the `bazel test` command and announces child events: -* `OptionsParsed` -* `WorkspaceStatus` -* `CommandLine` -* `UnstructuredCommandLine` -* `BuildMetadata` -* `BuildFinished` -* `PatternExpanded` -* `Progress` +- `OptionsParsed` +- `WorkspaceStatus` +- `CommandLine` +- `UnstructuredCommandLine` +- `BuildMetadata` +- `BuildFinished` +- `PatternExpanded` +- `Progress` The first three events provide information about how Bazel was invoked. -The `PatternExpanded` build event provides insight -into which specific targets the `...` pattern expanded to: -`//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two -`TargetConfigured` events as children. Note that the `TargetConfigured` event -declares the `Configuration` event as a child event, even though `Configuration` -has been posted before the `TargetConfigured` event. - -Besides the parent and child relationship, events may also refer to each other -using their build event identifiers. For example, in the above graph the -`TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` -field. - -Build events that refer to files don’t usually embed the file -names and paths in the event. Instead, they contain the build event identifier -of a `NamedSetOfFiles` event, which will then contain the actual file names and -paths. The `NamedSetOfFiles` event allows a set of files to be reported once and -referred to by many targets. This structure is necessary because otherwise in -some cases the Build Event Protocol output size would grow quadratically with -the number of files. A `NamedSetOfFiles` event may also not have all its files -embedded, but instead refer to other `NamedSetOfFiles` events through their -build event identifiers. - -Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` -target from the above graph, printed in protocol buffer’s JSON representation. -The build event identifier contains the target as an opaque string and refers to -the `Configuration` event using its build event identifier. The event does not -announce any child events. The payload contains information about whether the -target was built successfully, the set of output files, and the kind of target -built. +The `PatternExpanded` build event provides insight into which specific targets the `...` pattern expanded to: `//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two `TargetConfigured` events as children. Note that the `TargetConfigured` event declares the `Configuration` event as a child event, even though `Configuration` has been posted before the `TargetConfigured` event. + +Besides the parent and child relationship, events may also refer to each other using their build event identifiers. For example, in the above graph the `TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` field. + +Build events that refer to files don’t usually embed the file names and paths in the event. Instead, they contain the build event identifier of a `NamedSetOfFiles` event, which will then contain the actual file names and paths. The `NamedSetOfFiles` event allows a set of files to be reported once and referred to by many targets. This structure is necessary because otherwise in some cases the Build Event Protocol output size would grow quadratically with the number of files. A `NamedSetOfFiles` event may also not have all its files embedded, but instead refer to other `NamedSetOfFiles` events through their build event identifiers. + +Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` target from the above graph, printed in protocol buffer’s JSON representation. The build event identifier contains the target as an opaque string and refers to the `Configuration` event using its build event identifier. The event does not announce any child events. The payload contains information about whether the target was built successfully, the set of output files, and the kind of target built. ```json { @@ -102,18 +71,9 @@ built. ## Aspect Results in BEP -Ordinary builds evaluate actions associated with `(target, configuration)` -pairs. When building with [aspects](/extending/aspects) enabled, Bazel -additionally evaluates targets associated with `(target, configuration, -aspect)` triples, for each target affected by a given enabled aspect. +Ordinary builds evaluate actions associated with `(target, configuration)` pairs. When building with [aspects](/extending/aspects) enabled, Bazel additionally evaluates targets associated with `(target, configuration, aspect)` triples, for each target affected by a given enabled aspect. -Evaluation results for aspects are available in BEP despite the absence of -aspect-specific event types. For each `(target, configuration)` pair with an -applicable aspect, Bazel publishes an additional `TargetConfigured` and -`TargetComplete` event bearing the result from applying the aspect to the -target. For example, if `//:foo_lib` is built with -`--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in -the BEP: +Evaluation results for aspects are available in BEP despite the absence of aspect-specific event types. For each `(target, configuration)` pair with an applicable aspect, Bazel publishes an additional `TargetConfigured` and `TargetComplete` event bearing the result from applying the aspect to the target. For example, if `//:foo_lib` is built with `--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in the BEP: ```json { @@ -138,37 +98,21 @@ the BEP: } ``` -Note: The only difference between the IDs is the presence of the `aspect` -field. A tool that does not check the `aspect` ID field and accumulates output -files by target may conflate target outputs with aspect outputs. +Note: The only difference between the IDs is the presence of the `aspect` field. A tool that does not check the `aspect` ID field and accumulates output files by target may conflate target outputs with aspect outputs. ## Consuming `NamedSetOfFiles` -Determining the artifacts produced by a given target (or aspect) is a common -BEP use-case that can be done efficiently with some preparation. This section -discusses the recursive, shared structure offered by the `NamedSetOfFiles` -event, which matches the structure of a Starlark [Depset](/extending/depsets). +Determining the artifacts produced by a given target (or aspect) is a common BEP use-case that can be done efficiently with some preparation. This section discusses the recursive, shared structure offered by the `NamedSetOfFiles` event, which matches the structure of a Starlark [Depset](/extending/depsets). -Consumers must take care to avoid quadratic algorithms when processing -`NamedSetOfFiles` events because large builds can contain tens of thousands of -such events, requiring hundreds of millions operations in a traversal with -quadratic complexity. +Consumers must take care to avoid quadratic algorithms when processing `NamedSetOfFiles` events because large builds can contain tens of thousands of such events, requiring hundreds of millions operations in a traversal with quadratic complexity. ![namedsetoffiles-bep-graph](/docs/images/namedsetoffiles-bep-graph.png "NamedSetOfFiles BEP graph") **Figure 2.** `NamedSetOfFiles` BEP graph. -A `NamedSetOfFiles` event always appears in the BEP stream *before* a -`TargetComplete` or `NamedSetOfFiles` event that references it. This is the -inverse of the "parent-child" event relationship, where all but the first event -appears after at least one event announcing it. A `NamedSetOfFiles` event is -announced by a `Progress` event with no semantics. - -Given these ordering and sharing constraints, a typical consumer must buffer all -`NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON -event stream and Python code demonstrate how to populate a map from -target/aspect to built artifacts in the "default" output group, and how to -process the outputs for a subset of built targets/aspects: +A `NamedSetOfFiles` event always appears in the BEP stream *before* a `TargetComplete` or `NamedSetOfFiles` event that references it. This is the inverse of the "parent-child" event relationship, where all but the first event appears after at least one event announcing it. A `NamedSetOfFiles` event is announced by a `Progress` event with no semantics. + +Given these ordering and sharing constraints, a typical consumer must buffer all `NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON event stream and Python code demonstrate how to populate a map from target/aspect to built artifacts in the "default" output group, and how to process the outputs for a subset of built targets/aspects: ```python named_sets = {} # type: dict[str, NamedSetOfFiles] diff --git a/remote/bep-glossary.mdx b/remote/bep-glossary.mdx index 3bd11eea..66f77ea5 100644 --- a/remote/bep-glossary.mdx +++ b/remote/bep-glossary.mdx @@ -2,22 +2,13 @@ title: 'Build Event Protocol Glossary' --- - - -Each BEP event type has its own semantics, minimally documented in -[build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). -The following glossary describes each event type. +Each BEP event type has its own semantics, minimally documented in [build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). The following glossary describes each event type. ## Aborted -Unlike other events, `Aborted` does not have a corresponding ID type, because -the `Aborted` event *replaces* events of other types. This event indicates that -the build terminated early and the event ID it appears under was not produced -normally. `Aborted` contains an enum and human-friendly description to explain -why the build did not complete. +Unlike other events, `Aborted` does not have a corresponding ID type, because the `Aborted` event *replaces* events of other types. This event indicates that the build terminated early and the event ID it appears under was not produced normally. `Aborted` contains an enum and human-friendly description to explain why the build did not complete. -For example, if a build is evaluating a target when the user interrupts Bazel, -BEP contains an event like the following: +For example, if a build is evaluating a target when the user interrupts Bazel, BEP contains an event like the following: ```json { @@ -37,35 +28,21 @@ BEP contains an event like the following: ## ActionExecuted -Provides details about the execution of a specific -[Action](/rules/lib/actions) in a build. By default, this event is -included in the BEP only for failed actions, to support identifying the root cause -of build failures. Users may set the `--build_event_publish_all_actions` flag -to include all `ActionExecuted` events. +Provides details about the execution of a specific [Action](/rules/lib/actions) in a build. By default, this event is included in the BEP only for failed actions, to support identifying the root cause of build failures. Users may set the `--build_event_publish_all_actions` flag to include all `ActionExecuted` events. ## BuildFinished -A single `BuildFinished` event is sent after the command is complete and -includes the exit code for the command. This event provides authoritative -success/failure information. +A single `BuildFinished` event is sent after the command is complete and includes the exit code for the command. This event provides authoritative success/failure information. ## BuildMetadata -Contains the parsed contents of the `--build_metadata` flag. This event exists -to support Bazel integration with other tooling by plumbing external data (such as -identifiers). +Contains the parsed contents of the `--build_metadata` flag. This event exists to support Bazel integration with other tooling by plumbing external data (such as identifiers). ## BuildMetrics -A single `BuildMetrics` event is sent at the end of every command and includes -counters/gauges useful for quantifying the build tool's behavior during the -command. These metrics indicate work actually done and does not count cached -work that is reused. +A single `BuildMetrics` event is sent at the end of every command and includes counters/gauges useful for quantifying the build tool's behavior during the command. These metrics indicate work actually done and does not count cached work that is reused. -Note that `memory_metrics` may not be populated if there was no Java garbage -collection during the command's execution. Users may set the -`--memory_profile=/dev/null` option which forces the garbage -collector to run at the end of the command to populate `memory_metrics`. +Note that `memory_metrics` may not be populated if there was no Java garbage collection during the command's execution. Users may set the `--memory_profile=/dev/null` option which forces the garbage collector to run at the end of the command to populate `memory_metrics`. ```json { @@ -94,14 +71,11 @@ collector to run at the end of the command to populate `memory_metrics`. ## BuildStarted -The first event in a BEP stream, `BuildStarted` includes metadata describing the -command before any meaningful work begins. +The first event in a BEP stream, `BuildStarted` includes metadata describing the command before any meaningful work begins. ## BuildToolLogs -A single `BuildToolLogs` event is sent at the end of a command, including URIs -of files generated by the build tool that may aid in understanding or debugging -build tool behavior. Some information may be included inline. +A single `BuildToolLogs` event is sent at the end of a command, including URIs of files generated by the build tool that may aid in understanding or debugging build tool behavior. Some information may be included inline. ```json { @@ -130,28 +104,15 @@ build tool behavior. Some information may be included inline. ## CommandLine -The BEP contains multiple `CommandLine` events containing representations of all -command-line arguments (including options and uninterpreted arguments). -Each `CommandLine` event has a label in its `StructuredCommandLineId` that -indicates which representation it conveys; three such events appear in the BEP: - -* `"original"`: Reconstructed commandline as Bazel received it from the Bazel - client, without startup options sourced from .rc files. -* `"canonical"`: The effective commandline with .rc files expanded and - invocation policy applied. -* `"tool"`: Populated from the `--experimental_tool_command_line` option. This - is useful to convey the command-line of a tool wrapping Bazel through the BEP. - This could be a base64-encoded `CommandLine` binary protocol buffer message - which is used directly, or a string which is parsed but not interpreted (as - the tool's options may differ from Bazel's). +The BEP contains multiple `CommandLine` events containing representations of all command-line arguments (including options and uninterpreted arguments). Each `CommandLine` event has a label in its `StructuredCommandLineId` that indicates which representation it conveys; three such events appear in the BEP: + +- `"original"`: Reconstructed commandline as Bazel received it from the Bazel client, without startup options sourced from .rc files. +- `"canonical"`: The effective commandline with .rc files expanded and invocation policy applied. +- `"tool"`: Populated from the `--experimental_tool_command_line` option. This is useful to convey the command-line of a tool wrapping Bazel through the BEP. This could be a base64-encoded `CommandLine` binary protocol buffer message which is used directly, or a string which is parsed but not interpreted (as the tool's options may differ from Bazel's). ## Configuration -A `Configuration` event is sent for every [`configuration`](/extending/config) -used in the top-level targets in a build. At least one configuration event is -always be present. The `id` is reused by the `TargetConfigured` and -`TargetComplete` event IDs and is necessary to disambiguate those events in -multi-configuration builds. +A `Configuration` event is sent for every [`configuration`](/extending/config) used in the top-level targets in a build. At least one configuration event is always be present. The `id` is reused by the `TargetConfigured` and `TargetComplete` event IDs and is necessary to disambiguate those events in multi-configuration builds. ```json { @@ -176,11 +137,7 @@ multi-configuration builds. ## ConvenienceSymlinksIdentified -**Experimental.** If the `--experimental_convenience_symlinks_bep_event` -option is set, a single `ConvenienceSymlinksIdentified` event is produced by -`build` commands to indicate how symlinks in the workspace should be managed. -This enables building tools that invoke Bazel remotely then arrange the local -workspace as if Bazel had been run locally. +**Experimental.** If the `--experimental_convenience_symlinks_bep_event` option is set, a single `ConvenienceSymlinksIdentified` event is produced by `build` commands to indicate how symlinks in the workspace should be managed. This enables building tools that invoke Bazel remotely then arrange the local workspace as if Bazel had been run locally. ```json { @@ -211,24 +168,17 @@ workspace as if Bazel had been run locally. ## Fetch -Indicates that a Fetch operation occurred as a part of the command execution. -Unlike other events, if a cached fetch result is re-used, this event does not -appear in the BEP stream. +Indicates that a Fetch operation occurred as a part of the command execution. Unlike other events, if a cached fetch result is re-used, this event does not appear in the BEP stream. ## NamedSetOfFiles -`NamedSetOfFiles` events report a structure matching a -[`depset`](/extending/depsets) of files produced during command evaluation. -Transitively included depsets are identified by `NamedSetOfFilesId`. +`NamedSetOfFiles` events report a structure matching a [`depset`](/extending/depsets) of files produced during command evaluation. Transitively included depsets are identified by `NamedSetOfFilesId`. -For more information on interpreting a stream's `NamedSetOfFiles` events, see the -[BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). +For more information on interpreting a stream's `NamedSetOfFiles` events, see the [BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). ## OptionsParsed -A single `OptionsParsed` event lists all options applied to the command, -separating startup options from command options. It also includes the -[InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. +A single `OptionsParsed` event lists all options applied to the command, separating startup options from command options. It also includes the [InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. ```json { @@ -264,13 +214,7 @@ separating startup options from command options. It also includes the ## PatternExpanded -`PatternExpanded` events indicate the set of all targets that match the patterns -supplied on the commandline. For successful commands, a single event is present -with all patterns in the `PatternExpandedId` and all targets in the -`PatternExpanded` event's *children*. If the pattern expands to any -`test_suite`s the set of test targets included by the `test_suite`. For each -pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) -event with a `PatternExpandedId` identifying the pattern. +`PatternExpanded` events indicate the set of all targets that match the patterns supplied on the commandline. For successful commands, a single event is present with all patterns in the `PatternExpandedId` and all targets in the `PatternExpanded` event's *children*. If the pattern expands to any `test_suite`s the set of test targets included by the `test_suite`. For each pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) event with a `PatternExpandedId` identifying the pattern. ```json { @@ -294,16 +238,11 @@ event with a `PatternExpandedId` identifying the pattern. ## Progress -Progress events contain the standard output and standard error produced by Bazel -during command execution. These events are also auto-generated as needed to -announce events that have not been announced by a logical "parent" event (in -particular, [NamedSetOfFiles](#namedsetoffiles).) +Progress events contain the standard output and standard error produced by Bazel during command execution. These events are also auto-generated as needed to announce events that have not been announced by a logical "parent" event (in particular, [NamedSetOfFiles](#namedsetoffiles).) ## TargetComplete -For each `(target, configuration, aspect)` combination that completes the -execution phase, a `TargetComplete` event is included in BEP. The event contains -the target's success/failure and the target's requested output groups. +For each `(target, configuration, aspect)` combination that completes the execution phase, a `TargetComplete` event is included in BEP. The event contains the target's success/failure and the target's requested output groups. ```json { @@ -333,14 +272,9 @@ the target's success/failure and the target's requested output groups. ## TargetConfigured -For each Target that completes the analysis phase, a `TargetConfigured` event is -included in BEP. This is the authoritative source for a target's "rule kind" -attribute. The configuration(s) applied to the target appear in the announced -*children* of the event. +For each Target that completes the analysis phase, a `TargetConfigured` event is included in BEP. This is the authoritative source for a target's "rule kind" attribute. The configuration(s) applied to the target appear in the announced *children* of the event. -For example, building with the `--experimental_multi_cpu` options may produce -the following `TargetConfigured` event for a single target with two -configurations: +For example, building with the `--experimental_multi_cpu` options may produce the following `TargetConfigured` event for a single target with two configurations: ```json { @@ -375,42 +309,26 @@ configurations: ## TargetSummary -For each `(target, configuration)` pair that is executed, a `TargetSummary` -event is included with an aggregate success result encompassing the configured -target's execution and all aspects applied to that configured target. +For each `(target, configuration)` pair that is executed, a `TargetSummary` event is included with an aggregate success result encompassing the configured target's execution and all aspects applied to that configured target. ## TestResult -If testing is requested, a `TestResult` event is sent for each test attempt, -shard, and run per test. This allows BEP consumers to identify precisely which -test actions failed their tests and identify the test outputs (such as logs, -test.xml files) for each test action. +If testing is requested, a `TestResult` event is sent for each test attempt, shard, and run per test. This allows BEP consumers to identify precisely which test actions failed their tests and identify the test outputs (such as logs, test.xml files) for each test action. ## TestSummary -If testing is requested, a `TestSummary` event is sent for each test `(target, -configuration)`, containing information necessary to interpret the test's -results. The number of attempts, shards and runs per test are included to enable -BEP consumers to differentiate artifacts across these dimensions. The attempts -and runs per test are considered while producing the aggregate `TestStatus` to -differentiate `FLAKY` tests from `FAILED` tests. +If testing is requested, a `TestSummary` event is sent for each test `(target, configuration)`, containing information necessary to interpret the test's results. The number of attempts, shards and runs per test are included to enable BEP consumers to differentiate artifacts across these dimensions. The attempts and runs per test are considered while producing the aggregate `TestStatus` to differentiate `FLAKY` tests from `FAILED` tests. ## UnstructuredCommandLine -Unlike [CommandLine](#commandline), this event carries the unparsed commandline -flags in string form as encountered by the build tool after expanding all -[`.bazelrc`](/run/bazelrc) files and -considering the `--config` flag. +Unlike [CommandLine](#commandline), this event carries the unparsed commandline flags in string form as encountered by the build tool after expanding all [`.bazelrc`](/run/bazelrc) files and considering the `--config` flag. -The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a -given command execution. +The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a given command execution. ## WorkspaceConfig -A single `WorkspaceConfig` event contains configuration information regarding the -workspace, such as the execution root. +A single `WorkspaceConfig` event contains configuration information regarding the workspace, such as the execution root. ## WorkspaceStatus -A single `WorkspaceStatus` event contains the result of the [workspace status -command](/docs/user-manual#workspace-status). +A single `WorkspaceStatus` event contains the result of the [workspace status command](/docs/user-manual#workspace-status). diff --git a/remote/bep.mdx b/remote/bep.mdx index bafdaa9e..4589f8b0 100644 --- a/remote/bep.mdx +++ b/remote/bep.mdx @@ -2,67 +2,28 @@ title: 'Build Event Protocol' --- +The [Build Event Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) (BEP) allows third-party programs to gain insight into a Bazel invocation. For example, you could use the BEP to gather information for an IDE plugin or a dashboard that displays build results. +The protocol is a set of [protocol buffer](https://developers.google.com/protocol-buffers/) messages with some semantics defined on top of it. It includes information about build and test results, build progress, the build configuration and much more. The BEP is intended to be consumed programmatically and makes parsing Bazel’s command line output a thing of the past. -The [Build Event -Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) -(BEP) allows third-party programs to gain insight into a Bazel invocation. For -example, you could use the BEP to gather information for an IDE -plugin or a dashboard that displays build results. - -The protocol is a set of [protocol -buffer](https://developers.google.com/protocol-buffers/) messages with some -semantics defined on top of it. It includes information about build and test -results, build progress, the build configuration and much more. The BEP is -intended to be consumed programmatically and makes parsing Bazel’s -command line output a thing of the past. - -The Build Event Protocol represents information about a build as events. A -build event is a protocol buffer message consisting of a build event identifier, -a set of child event identifiers, and a payload. - -* __Build Event Identifier:__ Depending on the kind of build event, it might be -an [opaque -string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) -or [structured -information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) -revealing more about the build event. A build event identifier is unique within -a build. - -* __Children:__ A build event may announce other build events, by including -their build event identifiers in its [children -field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). -For example, the `PatternExpanded` build event announces the targets it expands -to as children. The protocol guarantees that all events, except for the first -event, are announced by a previous event. - -* __Payload:__ The payload contains structured information about a build event, -encoded as a protocol buffer message specific to that event. Note that the -payload might not be the expected type, but could be an `Aborted` message -if the build aborted prematurely. +The Build Event Protocol represents information about a build as events. A build event is a protocol buffer message consisting of a build event identifier, a set of child event identifiers, and a payload. + +- **Build Event Identifier:** Depending on the kind of build event, it might be an [opaque string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) or [structured information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) revealing more about the build event. A build event identifier is unique within a build. + +- **Children:** A build event may announce other build events, by including their build event identifiers in its [children field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). For example, the `PatternExpanded` build event announces the targets it expands to as children. The protocol guarantees that all events, except for the first event, are announced by a previous event. + +- **Payload:** The payload contains structured information about a build event, encoded as a protocol buffer message specific to that event. Note that the payload might not be the expected type, but could be an `Aborted` message if the build aborted prematurely. ### Build event graph -All build events form a directed acyclic graph through their parent and child -relationship. Every build event except for the initial build event has one or -more parent events. Please note that not all parent events of a child event must -necessarily be posted before it. When a build is complete (succeeded or failed) -all announced events will have been posted. In case of a Bazel crash or a failed -network transport, some announced build events may never be posted. - -The event graph's structure reflects the lifecycle of a command. Every BEP -graph has the following characteristic shape: - -1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) - event. All other events are its descendants. -1. Immediate children of the BuildStarted event contain metadata about the - command. -1. Events containing data produced by the command, such as files built and test - results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) - event. -1. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed - by events containing summary information about the build (for example, metric - or profiling data). +All build events form a directed acyclic graph through their parent and child relationship. Every build event except for the initial build event has one or more parent events. Please note that not all parent events of a child event must necessarily be posted before it. When a build is complete (succeeded or failed) all announced events will have been posted. In case of a Bazel crash or a failed network transport, some announced build events may never be posted. + +The event graph's structure reflects the lifecycle of a command. Every BEP graph has the following characteristic shape: + +1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) event. All other events are its descendants. +2. Immediate children of the BuildStarted event contain metadata about the command. +3. Events containing data produced by the command, such as files built and test results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) event. +4. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed by events containing summary information about the build (for example, metric or profiling data). ## Consuming Build Event Protocol @@ -70,21 +31,13 @@ graph has the following characteristic shape: To consume the BEP in a binary format: -1. Have Bazel serialize the protocol buffer messages to a file by specifying the - option `--build_event_binary_file=/path/to/file`. The file will contain - serialized protocol buffer messages with each message being length delimited. - Each message is prefixed with its length encoded as a variable length integer. - This format can be read using the protocol buffer library’s - [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) - method. +1. Have Bazel serialize the protocol buffer messages to a file by specifying the option `--build_event_binary_file=/path/to/file`. The file will contain serialized protocol buffer messages with each message being length delimited. Each message is prefixed with its length encoded as a variable length integer. This format can be read using the protocol buffer library’s [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) method. -2. Then, write a program that extracts the relevant information from the - serialized protocol buffer message. +2. Then, write a program that extracts the relevant information from the serialized protocol buffer message. ### Consume in text or JSON formats -The following Bazel command line flags will output the BEP in -human-readable formats, such as text and JSON: +The following Bazel command line flags will output the BEP in human-readable formats, such as text and JSON: ``` --build_event_text_file @@ -93,56 +46,34 @@ human-readable formats, such as text and JSON: ## Build Event Service -The [Build Event -Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) -Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event -Service protocol is independent of the BEP and treats BEP events as opaque bytes. -Bazel ships with a gRPC client implementation of the Build Event Service protocol that -publishes Build Event Protocol events. One can specify the endpoint to send the -events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, -you must prefix the address with the appropriate scheme: `grpc://` for plaintext -gRPC and `grpcs://` for gRPC with TLS enabled. +The [Build Event Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event Service protocol is independent of the BEP and treats BEP events as opaque bytes. Bazel ships with a gRPC client implementation of the Build Event Service protocol that publishes Build Event Protocol events. One can specify the endpoint to send the events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, you must prefix the address with the appropriate scheme: `grpc://` for plaintext gRPC and `grpcs://` for gRPC with TLS enabled. ### Build Event Service flags Bazel has several flags related to the Build Event Service protocol, including: -* `--bes_backend` -* `--[no]bes_lifecycle_events` -* `--bes_results_url` -* `--bes_timeout` -* `--bes_instance_name` +- `--bes_backend` +- `--[no]bes_lifecycle_events` +- `--bes_results_url` +- `--bes_timeout` +- `--bes_instance_name` -For a description of each of these flags, see the -[Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). ### Authentication and security -Bazel’s Build Event Service implementation also supports authentication and TLS. -These settings can be controlled using the below flags. Please note that these -flags are also used for Bazel’s Remote Execution. This implies that the Build -Event Service and Remote Execution Endpoints need to share the same -authentication and TLS infrastructure. +Bazel’s Build Event Service implementation also supports authentication and TLS. These settings can be controlled using the below flags. Please note that these flags are also used for Bazel’s Remote Execution. This implies that the Build Event Service and Remote Execution Endpoints need to share the same authentication and TLS infrastructure. -* `--[no]google_default_credentials` -* `--google_credentials` -* `--google_auth_scopes` -* `--tls_certificate` -* `--[no]tls_enabled` +- `--[no]google_default_credentials` +- `--google_credentials` +- `--google_auth_scopes` +- `--tls_certificate` +- `--[no]tls_enabled` -For a description of each of these flags, see the -[Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). ### Build Event Service and remote caching -The BEP typically contains many references to log files (test.log, test.xml, -etc. ) stored on the machine where Bazel is running. A remote BES server -typically can't access these files as they are on different machines. A way to -work around this issue is to use Bazel with [remote -caching](/remote/caching). -Bazel will upload all output files to the remote cache (including files -referenced in the BEP) and the BES server can then fetch the referenced files -from the cache. - -See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for -more details. +The BEP typically contains many references to log files (test.log, test.xml, etc. ) stored on the machine where Bazel is running. A remote BES server typically can't access these files as they are on different machines. A way to work around this issue is to use Bazel with [remote caching](/remote/caching). Bazel will upload all output files to the remote cache (including files referenced in the BEP) and the BES server can then fetch the referenced files from the cache. + +See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for more details. diff --git a/remote/cache-local.mdx b/remote/cache-local.mdx index a3415a64..49b8d33b 100644 --- a/remote/cache-local.mdx +++ b/remote/cache-local.mdx @@ -2,39 +2,23 @@ title: 'Debugging Remote Cache Hits for Local Execution' --- +This page describes how to investigate cache misses in the context of local execution. +This page assumes that you have a build and/or test that successfully builds locally and is set up to utilize remote caching, and that you want to ensure that the remote cache is being effectively utilized. -This page describes how to investigate cache misses in the context of local -execution. - -This page assumes that you have a build and/or test that successfully builds -locally and is set up to utilize remote caching, and that you want to ensure -that the remote cache is being effectively utilized. - -For tips on how to check your cache hit rate and how to compare the execution -logs between two Bazel invocations, see -[Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). -Everything presented in that guide also applies to remote caching with local -execution. However, local execution presents some additional challenges. +For tips on how to check your cache hit rate and how to compare the execution logs between two Bazel invocations, see [Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). Everything presented in that guide also applies to remote caching with local execution. However, local execution presents some additional challenges. ## Checking your cache hit rate -Successful remote cache hits will show up in the status line, similar to -[Cache Hits rate with Remote -Execution](/remote/cache-remote#check-cache-hits). +Successful remote cache hits will show up in the status line, similar to [Cache Hits rate with Remote Execution](/remote/cache-remote#check-cache-hits). -In the standard output of your Bazel run, you will see something like the -following: +In the standard output of your Bazel run, you will see something like the following: -```none {:.devsite-disable-click-to-copy} +```none INFO: 7 processes: 3 remote cache hit, 4 linux-sandbox. ``` -This means that out of 7 attempted actions, 3 got a remote cache hit and 4 -actions did not have cache hits and were executed locally using `linux-sandbox` -strategy. Local cache hits are not included in this summary. If you are getting -0 processes (or a number lower than expected), run `bazel clean` followed by -your build/test command. +This means that out of 7 attempted actions, 3 got a remote cache hit and 4 actions did not have cache hits and were executed locally using `linux-sandbox` strategy. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. ## Troubleshooting cache hits @@ -42,50 +26,30 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure successful communication with the remote endpoint -To ensure your build is successfully communicating with the remote cache, follow -the steps in this section. +To ensure your build is successfully communicating with the remote cache, follow the steps in this section. 1. Check your output for warnings - With remote execution, a failure to talk to the remote endpoint would fail - your build. On the other hand, a cacheable local build would not fail if it - cannot cache. Check the output of your Bazel invocation for warnings, such - as: + With remote execution, a failure to talk to the remote endpoint would fail your build. On the other hand, a cacheable local build would not fail if it cannot cache. Check the output of your Bazel invocation for warnings, such as: - ```none + ```none WARNING: Error reading from the remote cache: ``` - or - ```none + ```none WARNING: Error writing to the remote cache: ``` + Such warnings will be followed by the error message detailing the connection problem that should help you debug: for example, mistyped endpoint name or incorrectly set credentials. Find and address any such errors. If the error message you see does not give you enough information, try adding `--verbose_failures`. - Such warnings will be followed by the error message detailing the connection - problem that should help you debug: for example, mistyped endpoint name or - incorrectly set credentials. Find and address any such errors. If the error - message you see does not give you enough information, try adding - `--verbose_failures`. - -2. Follow the steps from [Troubleshooting cache hits for remote - execution](/remote/cache-remote#troubleshooting_cache_hits) to - ensure that your cache-writing Bazel invocations are able to get cache hits - on the same machine and across machines. +2. Follow the steps from [Troubleshooting cache hits for remote execution](/remote/cache-remote#troubleshooting_cache_hits) to ensure that your cache-writing Bazel invocations are able to get cache hits on the same machine and across machines. 3. Ensure your cache-reading Bazel invocations can get cache hits. - a. Since cache-reading Bazel invocations will have a different command-line set - up, take additional care to ensure that they are properly set up to - communicate with the remote cache. Ensure the `--remote_cache` flag is set - and there are no warnings in the output. + a. Since cache-reading Bazel invocations will have a different command-line set up, take additional care to ensure that they are properly set up to communicate with the remote cache. Ensure the `--remote_cache` flag is set and there are no warnings in the output. - b. Ensure your cache-reading Bazel invocations build the same targets as the - cache-writing Bazel invocations. + b. Ensure your cache-reading Bazel invocations build the same targets as the cache-writing Bazel invocations. - c. Follow the same steps as to [ensure caching across - machines](/remote/cache-remote#caching-across-machines), - to ensure caching from your cache-writing Bazel invocation to your - cache-reading Bazel invocation. + c. Follow the same steps as to [ensure caching across machines](/remote/cache-remote#caching-across-machines), to ensure caching from your cache-writing Bazel invocation to your cache-reading Bazel invocation. diff --git a/remote/cache-remote.mdx b/remote/cache-remote.mdx index 896f1e58..3b96276a 100644 --- a/remote/cache-remote.mdx +++ b/remote/cache-remote.mdx @@ -2,36 +2,21 @@ title: 'Debugging Remote Cache Hits for Remote Execution' --- +This page describes how to check your cache hit rate and how to investigate cache misses in the context of remote execution. - -This page describes how to check your cache hit rate and how to investigate -cache misses in the context of remote execution. - -This page assumes that you have a build and/or test that successfully -utilizes remote execution, and you want to ensure that you are effectively -utilizing remote cache. +This page assumes that you have a build and/or test that successfully utilizes remote execution, and you want to ensure that you are effectively utilizing remote cache. ## Checking your cache hit rate -In the standard output of your Bazel run, look at the `INFO` line that lists -processes, which roughly correspond to Bazel actions. That line details -where the action was run. Look for the `remote` label, which indicates an action -executed remotely, `linux-sandbox` for actions executed in a local sandbox, -and other values for other execution strategies. An action whose result came -from a remote cache is displayed as `remote cache hit`. +In the standard output of your Bazel run, look at the `INFO` line that lists processes, which roughly correspond to Bazel actions. That line details where the action was run. Look for the `remote` label, which indicates an action executed remotely, `linux-sandbox` for actions executed in a local sandbox, and other values for other execution strategies. An action whose result came from a remote cache is displayed as `remote cache hit`. For example: -```none {:.devsite-disable-click-to-copy} +```none INFO: 11 processes: 6 remote cache hit, 3 internal, 2 remote. ``` -In this example there were 6 remote cache hits, and 2 actions did not have -cache hits and were executed remotely. The 3 internal part can be ignored. -It is typically tiny internal actions, such as creating symbolic links. Local -cache hits are not included in this summary. If you are getting 0 processes -(or a number lower than expected), run `bazel clean` followed by your build/test -command. +In this example there were 6 remote cache hits, and 2 actions did not have cache hits and were executed remotely. The 3 internal part can be ignored. It is typically tiny internal actions, such as creating symbolic links. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. ## Troubleshooting cache hits @@ -39,80 +24,47 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure re-running the same build/test command produces cache hits -1. Run the build(s) and/or test(s) that you expect to populate the cache. The - first time a new build is run on a particular stack, you can expect no remote - cache hits. As part of remote execution, action results are stored in the - cache and a subsequent run should pick them up. +1. Run the build(s) and/or test(s) that you expect to populate the cache. The first time a new build is run on a particular stack, you can expect no remote cache hits. As part of remote execution, action results are stored in the cache and a subsequent run should pick them up. -2. Run `bazel clean`. This command cleans your local cache, which allows - you to investigate remote cache hits without the results being masked by - local cache hits. +2. Run `bazel clean`. This command cleans your local cache, which allows you to investigate remote cache hits without the results being masked by local cache hits. -3. Run the build(s) and test(s) that you are investigating again (on the same - machine). +3. Run the build(s) and test(s) that you are investigating again (on the same machine). -4. Check the `INFO` line for cache hit rate. If you see no processes except - `remote cache hit` and `internal`, then your cache is being correctly populated and - accessed. In that case, skip to the next section. +4. Check the `INFO` line for cache hit rate. If you see no processes except `remote cache hit` and `internal`, then your cache is being correctly populated and accessed. In that case, skip to the next section. -5. A likely source of discrepancy is something non-hermetic in the build causing - the actions to receive different action keys across the two runs. To find - those actions, do the following: +5. A likely source of discrepancy is something non-hermetic in the build causing the actions to receive different action keys across the two runs. To find those actions, do the following: a. Re-run the build(s) or test(s) in question to obtain execution logs: - ```posix-terminal - bazel clean + ```posix-terminal + bazel clean - bazel --optional-flags build //your:target --execution_log_compact_file=/tmp/exec1.log - ``` + bazel <var>--optional-flags</var> build //<var>your:target</var> --execution_log_compact_file=/tmp/exec1.log + ``` - b. [Compare the execution logs](#compare-logs) between the - two runs. Ensure that the actions are identical across the two log files. - Discrepancies provide a clue about the changes that occurred between the - runs. Update your build to eliminate those discrepancies. + b. [Compare the execution logs](#compare-logs) between the two runs. Ensure that the actions are identical across the two log files. Discrepancies provide a clue about the changes that occurred between the runs. Update your build to eliminate those discrepancies. - If you are able to resolve the caching problems and now the repeated run - produces all cache hits, skip to the next section. + If you are able to resolve the caching problems and now the repeated run produces all cache hits, skip to the next section. - If your action IDs are identical but there are no cache hits, then something - in your configuration is preventing caching. Continue with this section to - check for common problems. + If your action IDs are identical but there are no cache hits, then something in your configuration is preventing caching. Continue with this section to check for common problems. -5. Check that all actions in the execution log have `cacheable` set to true. If - `cacheable` does not appear in the execution log for a give action, that - means that the corresponding rule may have a `no-cache` tag in its - definition in the `BUILD` file. Look at the `mnemonic` and `target_label` - fields in the execution log to help determine where the action is coming - from. +6. Check that all actions in the execution log have `cacheable` set to true. If `cacheable` does not appear in the execution log for a give action, that means that the corresponding rule may have a `no-cache` tag in its definition in the `BUILD` file. Look at the `mnemonic` and `target_label` fields in the execution log to help determine where the action is coming from. -6. If the actions are identical and `cacheable` but there are no cache hits, it - is possible that your command line includes `--noremote_accept_cached` which - would disable cache lookups for a build. +7. If the actions are identical and `cacheable` but there are no cache hits, it is possible that your command line includes `--noremote_accept_cached` which would disable cache lookups for a build. - If figuring out the actual command line is difficult, use the canonical - command line from the - [Build Event Protocol](/remote/bep) - as follows: + If figuring out the actual command line is difficult, use the canonical command line from the [Build Event Protocol](/remote/bep) as follows: - a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get - the text version of the log. + a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get the text version of the log. - b. Open the text version of the log and search for the - `structured_command_line` message with `command_line_label: "canonical"`. - It will list all the options after expansion. + b. Open the text version of the log and search for the `structured_command_line` message with `command_line_label: "canonical"`. It will list all the options after expansion. c. Search for `remote_accept_cached` and check whether it's set to `false`. - d. If `remote_accept_cached` is `false`, determine where it is being - set to `false`: either at the command line or in a - [bazelrc](/run/bazelrc#bazelrc-file-locations) file. + d. If `remote_accept_cached` is `false`, determine where it is being set to `false`: either at the command line or in a [bazelrc](/run/bazelrc#bazelrc-file-locations) file. ### Ensure caching across machines -After cache hits are happening as expected on the same machine, run the -same build(s)/test(s) on a different machine. If you suspect that caching is -not happening across machines, do the following: +After cache hits are happening as expected on the same machine, run the same build(s)/test(s) on a different machine. If you suspect that caching is not happening across machines, do the following: 1. Make a small modification to your build to avoid hitting existing caches. @@ -124,8 +76,7 @@ not happening across machines, do the following: bazel ... build ... --execution_log_compact_file=/tmp/exec1.log ``` -3. Run the build on the second machine, ensuring the modification from step 1 - is included: +3. Run the build on the second machine, ensuring the modification from step 1 is included: ```posix-terminal bazel clean @@ -133,47 +84,34 @@ not happening across machines, do the following: bazel ... build ... --execution_log_compact_file=/tmp/exec2.log ``` -4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two - runs. If the logs are not identical, investigate your build configurations - for discrepancies as well as properties from the host environment leaking - into either of the builds. +4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two runs. If the logs are not identical, investigate your build configurations for discrepancies as well as properties from the host environment leaking into either of the builds. ## Comparing the execution logs -The execution log contains records of actions executed during the build. -Each record describes both the inputs (not only files, but also command line -arguments, environment variables, etc) and the outputs of the action. Thus, -examination of the log can reveal why an action was reexecuted. +The execution log contains records of actions executed during the build. Each record describes both the inputs (not only files, but also command line arguments, environment variables, etc) and the outputs of the action. Thus, examination of the log can reveal why an action was reexecuted. -The execution log can be produced in one of three formats: -compact (`--execution_log_compact_file`), -binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). -The compact format is recommended, as it produces much smaller files with very -little runtime overhead. The following instructions work for any format. You -can also convert between them using the `//src/tools/execlog:converter` tool. +The execution log can be produced in one of three formats: compact (`--execution_log_compact_file`), binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). The compact format is recommended, as it produces much smaller files with very little runtime overhead. The following instructions work for any format. You can also convert between them using the `//src/tools/execlog:converter` tool. -To compare logs for two builds that are not sharing cache hits as expected, -do the following: +To compare logs for two builds that are not sharing cache hits as expected, do the following: -1. Get the execution logs from each build and store them as `/tmp/exec1.log` and - `/tmp/exec2.log`. +1. Get the execution logs from each build and store them as `/tmp/exec1.log` and `/tmp/exec2.log`. -2. Download the Bazel source code and build the `//src/tools/execlog:parser` - tool: +2. Download the Bazel source code and build the `//src/tools/execlog:parser` tool: - git clone https://github.com/bazelbuild/bazel.git - cd bazel - bazel build //src/tools/execlog:parser + ``` + git clone https://github.com/bazelbuild/bazel.git + cd bazel + bazel build //src/tools/execlog:parser + ``` -3. Use the `//src/tools/execlog:parser` tool to convert the logs into a - human-readable text format. In this format, the actions in the second log are - sorted to match the order in the first log, making a comparison easier. +3. Use the `//src/tools/execlog:parser` tool to convert the logs into a human-readable text format. In this format, the actions in the second log are sorted to match the order in the first log, making a comparison easier. - bazel-bin/src/tools/execlog/parser \ - --log_path=/tmp/exec1.log \ - --log_path=/tmp/exec2.log \ - --output_path=/tmp/exec1.log.txt \ - --output_path=/tmp/exec2.log.txt + ``` + bazel-bin/src/tools/execlog/parser \ + --log_path=/tmp/exec1.log \ + --log_path=/tmp/exec2.log \ + --output_path=/tmp/exec1.log.txt \ + --output_path=/tmp/exec2.log.txt + ``` -4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and - `/tmp/exec2.log.txt`. +4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and `/tmp/exec2.log.txt`. diff --git a/remote/caching.mdx b/remote/caching.mdx index f85f2c8e..1729ee02 100644 --- a/remote/caching.mdx +++ b/remote/caching.mdx @@ -2,94 +2,65 @@ title: 'Remote Caching' --- +This page covers remote caching, setting up a server to host the cache, and running builds using the remote cache. - -This page covers remote caching, setting up a server to host the cache, and -running builds using the remote cache. - -A remote cache is used by a team of developers and/or a continuous integration -(CI) system to share build outputs. If your build is reproducible, the -outputs from one machine can be safely reused on another machine, which can -make builds significantly faster. +A remote cache is used by a team of developers and/or a continuous integration (CI) system to share build outputs. If your build is reproducible, the outputs from one machine can be safely reused on another machine, which can make builds significantly faster. ## Overview -Bazel breaks a build into discrete steps, which are called actions. Each action -has inputs, output names, a command line, and environment variables. Required -inputs and expected outputs are declared explicitly for each action. +Bazel breaks a build into discrete steps, which are called actions. Each action has inputs, output names, a command line, and environment variables. Required inputs and expected outputs are declared explicitly for each action. -You can set up a server to be a remote cache for build outputs, which are these -action outputs. These outputs consist of a list of output file names and the -hashes of their contents. With a remote cache, you can reuse build outputs -from another user's build rather than building each new output locally. +You can set up a server to be a remote cache for build outputs, which are these action outputs. These outputs consist of a list of output file names and the hashes of their contents. With a remote cache, you can reuse build outputs from another user's build rather than building each new output locally. To use remote caching: -* Set up a server as the cache's backend -* Configure the Bazel build to use the remote cache -* Use Bazel version 0.10.0 or later +- Set up a server as the cache's backend +- Configure the Bazel build to use the remote cache +- Use Bazel version 0.10.0 or later The remote cache stores two types of data: -* The action cache, which is a map of action hashes to action result metadata. -* A content-addressable store (CAS) of output files. +- The action cache, which is a map of action hashes to action result metadata. +- A content-addressable store (CAS) of output files. -Note that the remote cache additionally stores the stdout and stderr for every -action. Inspecting the stdout/stderr of Bazel thus is not a good signal for -[estimating cache hits](/remote/cache-local). +Note that the remote cache additionally stores the stdout and stderr for every action. Inspecting the stdout/stderr of Bazel thus is not a good signal for [estimating cache hits](/remote/cache-local). ### How a build uses remote caching -Once a server is set up as the remote cache, you use the cache in multiple -ways: - -* Read and write to the remote cache -* Read and/or write to the remote cache except for specific targets -* Only read from the remote cache -* Not use the remote cache at all - -When you run a Bazel build that can read and write to the remote cache, -the build follows these steps: - -1. Bazel creates the graph of targets that need to be built, and then creates -a list of required actions. Each of these actions has declared inputs -and output filenames. -2. Bazel checks your local machine for existing build outputs and reuses any -that it finds. -3. Bazel checks the cache for existing build outputs. If the output is found, -Bazel retrieves the output. This is a cache hit. -4. For required actions where the outputs were not found, Bazel executes the -actions locally and creates the required build outputs. +Once a server is set up as the remote cache, you use the cache in multiple ways: + +- Read and write to the remote cache +- Read and/or write to the remote cache except for specific targets +- Only read from the remote cache +- Not use the remote cache at all + +When you run a Bazel build that can read and write to the remote cache, the build follows these steps: + +1. Bazel creates the graph of targets that need to be built, and then creates a list of required actions. Each of these actions has declared inputs and output filenames. +2. Bazel checks your local machine for existing build outputs and reuses any that it finds. +3. Bazel checks the cache for existing build outputs. If the output is found, Bazel retrieves the output. This is a cache hit. +4. For required actions where the outputs were not found, Bazel executes the actions locally and creates the required build outputs. 5. New build outputs are uploaded to the remote cache. ## Setting up a server as the cache's backend -You need to set up a server to act as the cache's backend. A HTTP/1.1 -server can treat Bazel's data as opaque bytes and so many existing servers -can be used as a remote caching backend. Bazel's -[HTTP Caching Protocol](#http-caching) is what supports remote -caching. +You need to set up a server to act as the cache's backend. A HTTP/1.1 server can treat Bazel's data as opaque bytes and so many existing servers can be used as a remote caching backend. Bazel's [HTTP Caching Protocol](#http-caching) is what supports remote caching. -You are responsible for choosing, setting up, and maintaining the backend -server that will store the cached outputs. When choosing a server, consider: +You are responsible for choosing, setting up, and maintaining the backend server that will store the cached outputs. When choosing a server, consider: -* Networking speed. For example, if your team is in the same office, you may -want to run your own local server. -* Security. The remote cache will have your binaries and so needs to be secure. -* Ease of management. For example, Google Cloud Storage is a fully managed service. +- Networking speed. For example, if your team is in the same office, you may want to run your own local server. +- Security. The remote cache will have your binaries and so needs to be secure. +- Ease of management. For example, Google Cloud Storage is a fully managed service. -There are many backends that can be used for a remote cache. Some options -include: +There are many backends that can be used for a remote cache. Some options include: -* [nginx](#nginx) -* [bazel-remote](#bazel-remote) -* [Google Cloud Storage](#cloud-storage) +- [nginx](#nginx) +- [bazel-remote](#bazel-remote) +- [Google Cloud Storage](#cloud-storage) ### nginx -nginx is an open source web server. With its [WebDAV module], it can be -used as a remote cache for Bazel. On Debian and Ubuntu you can install the -`nginx-extras` package. On macOS nginx is available via Homebrew: +nginx is an open source web server. With its \[WebDAV module], it can be used as a remote cache for Bazel. On Debian and Ubuntu you can install the `nginx-extras` package. On macOS nginx is available via Homebrew: ```posix-terminal brew tap denji/nginx @@ -97,12 +68,7 @@ brew tap denji/nginx brew install nginx-full --with-webdav ``` -Below is an example configuration for nginx. Note that you will need to -change `/path/to/cache/dir` to a valid directory where nginx has permission -to write and read. You may need to change `client_max_body_size` option to a -larger value if you have larger output files. The server will require other -configuration such as authentication. - +Below is an example configuration for nginx. Note that you will need to change `/path/to/cache/dir` to a valid directory where nginx has permission to write and read. You may need to change `client_max_body_size` option to a larger value if you have larger output files. The server will require other configuration such as authentication. Example configuration for `server` section in `nginx.conf`: @@ -122,72 +88,44 @@ location /cache/ { ### bazel-remote -bazel-remote is an open source remote build cache that you can use on -your infrastructure. It has been successfully used in production at -several companies since early 2018. Note that the Bazel project does -not provide technical support for bazel-remote. +bazel-remote is an open source remote build cache that you can use on your infrastructure. It has been successfully used in production at several companies since early 2018. Note that the Bazel project does not provide technical support for bazel-remote. -This cache stores contents on disk and also provides garbage collection -to enforce an upper storage limit and clean unused artifacts. The cache is -available as a [docker image] and its code is available on -[GitHub](https://github.com/buchgr/bazel-remote/). -Both the REST and gRPC remote cache APIs are supported. +This cache stores contents on disk and also provides garbage collection to enforce an upper storage limit and clean unused artifacts. The cache is available as a \[docker image] and its code is available on [GitHub](https://github.com/buchgr/bazel-remote/). Both the REST and gRPC remote cache APIs are supported. -Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) -page for instructions on how to use it. +Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) page for instructions on how to use it. ### Google Cloud Storage -[Google Cloud Storage] is a fully managed object store which provides an -HTTP API that is compatible with Bazel's remote caching protocol. It requires -that you have a Google Cloud account with billing enabled. +\[Google Cloud Storage] is a fully managed object store which provides an HTTP API that is compatible with Bazel's remote caching protocol. It requires that you have a Google Cloud account with billing enabled. To use Cloud Storage as the cache: -1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). -Ensure that you select a bucket location that's closest to you, as network bandwidth -is important for the remote cache. +1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). Ensure that you select a bucket location that's closest to you, as network bandwidth is important for the remote cache. -2. Create a service account for Bazel to authenticate to Cloud Storage. See -[Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). +2. Create a service account for Bazel to authenticate to Cloud Storage. See [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). -3. Generate a secret JSON key and then pass it to Bazel for authentication. Store -the key securely, as anyone with the key can read and write arbitrary data -to/from your GCS bucket. +3. Generate a secret JSON key and then pass it to Bazel for authentication. Store the key securely, as anyone with the key can read and write arbitrary data to/from your GCS bucket. 4. Connect to Cloud Storage by adding the following flags to your Bazel command: - * Pass the following URL to Bazel by using the flag: - `--remote_cache=https://storage.googleapis.com/bucket-name` where `bucket-name` is the name of your storage bucket. - * Pass the authentication key using the flag: `--google_credentials=/path/to/your/secret-key.json`, or - `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). -5. You can configure Cloud Storage to automatically delete old files. To do so, see -[Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). + - Pass the following URL to Bazel by using the flag: `--remote_cache=https://storage.googleapis.com<var>/bucket-name</var>` where `bucket-name` is the name of your storage bucket. + - Pass the authentication key using the flag: `--google_credentials=<var>/path/to/your/secret-key</var>.json`, or `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). + +5. You can configure Cloud Storage to automatically delete old files. To do so, see [Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). ### Other servers -You can set up any HTTP/1.1 server that supports PUT and GET as the cache's -backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), -[Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). +You can set up any HTTP/1.1 server that supports PUT and GET as the cache's backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), [Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). ## Authentication -As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. -You can pass a username and password to Bazel via the remote cache URL. The -syntax is `https://username:password@hostname.com:port/path`. Note that -HTTP Basic Authentication transmits username and password in plaintext over the -network and it's thus critical to always use it with HTTPS. +As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. You can pass a username and password to Bazel via the remote cache URL. The syntax is `https://username:password@hostname.com:port/path`. Note that HTTP Basic Authentication transmits username and password in plaintext over the network and it's thus critical to always use it with HTTPS. ## HTTP caching protocol -Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: -Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. -Action result metadata is stored under the path `/ac/` and output files are stored -under the path `/cas/`. +Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. Action result metadata is stored under the path `/ac/` and output files are stored under the path `/cas/`. -For example, consider a remote cache running under `http://localhost:8080/cache`. -A Bazel request to download action result metadata for an action with the SHA256 -hash `01ba4719...` will look as follows: +For example, consider a remote cache running under `http://localhost:8080/cache`. A Bazel request to download action result metadata for an action with the SHA256 hash `01ba4719...` will look as follows: ```http GET /cache/ac/01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b HTTP/1.1 @@ -196,8 +134,7 @@ Accept: */* Connection: Keep-Alive ``` -A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to -the CAS will look as follows: +A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to the CAS will look as follows: ```http PUT /cache/cas/15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225 HTTP/1.1 @@ -211,36 +148,29 @@ Connection: Keep-Alive ## Run Bazel using the remote cache -Once a server is set up as the remote cache, to use the remote cache you -need to add flags to your Bazel command. See list of configurations and -their flags below. +Once a server is set up as the remote cache, to use the remote cache you need to add flags to your Bazel command. See list of configurations and their flags below. -You may also need configure authentication, which is specific to your -chosen server. +You may also need configure authentication, which is specific to your chosen server. -You may want to add these flags in a `.bazelrc` file so that you don't -need to specify them every time you run Bazel. Depending on your project and -team dynamics, you can add flags to a `.bazelrc` file that is: +You may want to add these flags in a `.bazelrc` file so that you don't need to specify them every time you run Bazel. Depending on your project and team dynamics, you can add flags to a `.bazelrc` file that is: -* On your local machine -* In your project's workspace, shared with the team -* On the CI system +- On your local machine +- In your project's workspace, shared with the team +- On the CI system ### Read from and write to the remote cache -Take care in who has the ability to write to the remote cache. You may want -only your CI system to be able to write to the remote cache. +Take care in who has the ability to write to the remote cache. You may want only your CI system to be able to write to the remote cache. Use the following flag to read from and write to the remote cache: ```posix-terminal -build --remote_cache=http://{{ '' }}your.host:port{{ '' }} +build --remote_cache=http://<var>your.host:port</var> ``` Besides `HTTP`, the following protocols are also supported: `HTTPS`, `grpc`, `grpcs`. -Use the following flag in addition to the one above to only read from the -remote cache: +Use the following flag in addition to the one above to only read from the remote cache: ```posix-terminal build --remote_upload_local_results=false @@ -248,8 +178,7 @@ build --remote_upload_local_results=false ### Exclude specific targets from using the remote cache -To exclude specific targets from using the remote cache, tag the target with -`no-remote-cache`. For example: +To exclude specific targets from using the remote cache, tag the target with `no-remote-cache`. For example: ```starlark java_library( @@ -260,121 +189,100 @@ java_library( ### Delete content from the remote cache -Deleting content from the remote cache is part of managing your server. -How you delete content from the remote cache depends on the server you have -set up as the cache. When deleting outputs, either delete the entire cache, -or delete old outputs. +Deleting content from the remote cache is part of managing your server. How you delete content from the remote cache depends on the server you have set up as the cache. When deleting outputs, either delete the entire cache, or delete old outputs. -The cached outputs are stored as a set of names and hashes. When deleting -content, there's no way to distinguish which output belongs to a specific -build. +The cached outputs are stored as a set of names and hashes. When deleting content, there's no way to distinguish which output belongs to a specific build. You may want to delete content from the cache to: -* Create a clean cache after a cache was poisoned -* Reduce the amount of storage used by deleting old outputs +- Create a clean cache after a cache was poisoned +- Reduce the amount of storage used by deleting old outputs ### Unix sockets -The remote HTTP cache supports connecting over unix domain sockets. The behavior -is similar to curl's `--unix-socket` flag. Use the following to configure unix -domain socket: +The remote HTTP cache supports connecting over unix domain sockets. The behavior is similar to curl's `--unix-socket` flag. Use the following to configure unix domain socket: ```posix-terminal - build --remote_cache=http://{{ '' }}your.host:port{{ '' }} - build --remote_proxy=unix:/{{ '' }}path/to/socket{{ '' }} + build --remote_cache=http://<var>your.host:port</var> + build --remote_proxy=unix:/<var>path/to/socket</var> ``` This feature is unsupported on Windows. ## Disk cache -Bazel can use a directory on the file system as a remote cache. This is -useful for sharing build artifacts when switching branches and/or working -on multiple workspaces of the same project, such as multiple checkouts. -Enable the disk cache as follows: +Bazel can use a directory on the file system as a remote cache. This is useful for sharing build artifacts when switching branches and/or working on multiple workspaces of the same project, such as multiple checkouts. Enable the disk cache as follows: ```posix-terminal -build --disk_cache={{ '' }}path/to/build/cache{{ '' }} +build --disk_cache=<var>path/to/build/cache</var> ``` -You can pass a user-specific path to the `--disk_cache` flag using the `~` alias -(Bazel will substitute the current user's home directory). This comes in handy -when enabling the disk cache for all developers of a project via the project's -checked in `.bazelrc` file. +You can pass a user-specific path to the `--disk_cache` flag using the `~` alias (Bazel will substitute the current user's home directory). This comes in handy when enabling the disk cache for all developers of a project via the project's checked in `.bazelrc` file. ### Garbage collection -Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and -`--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache -or for the age of individual cache entries. Bazel will automatically garbage -collect the disk cache while idling between builds; the idle timer can be set -with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). +Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and `--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache or for the age of individual cache entries. Bazel will automatically garbage collect the disk cache while idling between builds; the idle timer can be set with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). -As an alternative to automatic garbage collection, we also provide a [tool]( -https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a -garbage collection on demand. +As an alternative to automatic garbage collection, we also provide a [tool](https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a garbage collection on demand. ## Known issues **Input file modification during a build** -When an input file is modified during a build, Bazel might upload invalid -results to the remote cache. You can enable a change detection with -the `--experimental_guard_against_concurrent_changes` flag. There -are no known issues and it will be enabled by default in a future release. -See [issue #3360] for updates. Generally, avoid modifying source files during a -build. +When an input file is modified during a build, Bazel might upload invalid results to the remote cache. You can enable a change detection with the `--experimental_guard_against_concurrent_changes` flag. There are no known issues and it will be enabled by default in a future release. See \[issue #3360] for updates. Generally, avoid modifying source files during a build. **Environment variables leaking into an action** -An action definition contains environment variables. This can be a problem for -sharing remote cache hits across machines. For example, environments with -different `$PATH` variables won't share cache hits. Only environment variables -explicitly whitelisted via `--action_env` are included in an action -definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` -with a whitelist of environment variables including `$PATH`. If you are getting -fewer cache hits than expected, check that your environment doesn't have an old -`/etc/bazel.bazelrc` file. +An action definition contains environment variables. This can be a problem for sharing remote cache hits across machines. For example, environments with different `$PATH` variables won't share cache hits. Only environment variables explicitly whitelisted via `--action_env` are included in an action definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` with a whitelist of environment variables including `$PATH`. If you are getting fewer cache hits than expected, check that your environment doesn't have an old `/etc/bazel.bazelrc` file. **Bazel does not track tools outside a workspace** -Bazel currently does not track tools outside a workspace. This can be a -problem if, for example, an action uses a compiler from `/usr/bin/`. Then, -two users with different compilers installed will wrongly share cache hits -because the outputs are different but they have the same action hash. See -[issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. +Bazel currently does not track tools outside a workspace. This can be a problem if, for example, an action uses a compiler from `/usr/bin/`. Then, two users with different compilers installed will wrongly share cache hits because the outputs are different but they have the same action hash. See [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. -**Incremental in-memory state is lost when running builds inside docker containers** -Bazel uses server/client architecture even when running in single docker container. -On the server side, Bazel maintains an in-memory state which speeds up builds. -When running builds inside docker containers such as in CI, the in-memory state is lost -and Bazel must rebuild it before using the remote cache. +**Incremental in-memory state is lost when running builds inside docker containers** Bazel uses server/client architecture even when running in single docker container. On the server side, Bazel maintains an in-memory state which speeds up builds. When running builds inside docker containers such as in CI, the in-memory state is lost and Bazel must rebuild it before using the remote cache. ## External links -* **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. - -* **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) -in which he benchmarks remote caching in Bazel. - -* [Adapting Rules for Remote Execution](/remote/rules) -* [Troubleshooting Remote Execution](/remote/sandbox) -* [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) -* [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) -* [bazel-remote](https://github.com/buchgr/bazel-remote/) -* [Google Cloud Storage](https://cloud.google.com/storage) -* [Google Cloud Console](https://cloud.google.com/console) -* [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) -* [Hazelcast](https://hazelcast.com) -* [Apache httpd](http://httpd.apache.org) -* [AWS S3](https://aws.amazon.com/s3) -* [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) -* [gRPC](https://grpc.io/) -* [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) -* [Buildbarn](https://github.com/buildbarn) -* [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) -* [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) -* [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) -* [Application Authentication](https://cloud.google.com/docs/authentication/production) -* [NativeLink](https://github.com/TraceMachina/nativelink) +- **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. + +- **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) in which he benchmarks remote caching in Bazel. + +- [Adapting Rules for Remote Execution](/remote/rules) + +- [Troubleshooting Remote Execution](/remote/sandbox) + +- [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) + +- [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) + +- [bazel-remote](https://github.com/buchgr/bazel-remote/) + +- [Google Cloud Storage](https://cloud.google.com/storage) + +- [Google Cloud Console](https://cloud.google.com/console) + +- [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) + +- [Hazelcast](https://hazelcast.com) + +- [Apache httpd](http://httpd.apache.org) + +- [AWS S3](https://aws.amazon.com/s3) + +- [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) + +- [gRPC](https://grpc.io/) + +- [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) + +- [Buildbarn](https://github.com/buildbarn) + +- [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + +- [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + +- [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) + +- [Application Authentication](https://cloud.google.com/docs/authentication/production) + +- [NativeLink](https://github.com/TraceMachina/nativelink) diff --git a/remote/ci.mdx b/remote/ci.mdx index 0a4e4488..d2b1a97f 100644 --- a/remote/ci.mdx +++ b/remote/ci.mdx @@ -2,37 +2,24 @@ title: 'Configuring Bazel CI to Test Rules for Remote Execution' --- - - -This page is for owners and maintainers of Bazel rule repositories. It -describes how to configure the Bazel Continuous Integration (CI) system for -your repository to test your rules for compatibility against a remote execution -scenario. The instructions on this page apply to projects stored in -GitHub repositories. +This page is for owners and maintainers of Bazel rule repositories. It describes how to configure the Bazel Continuous Integration (CI) system for your repository to test your rules for compatibility against a remote execution scenario. The instructions on this page apply to projects stored in GitHub repositories. ## Prerequisites Before completing the steps on this page, ensure the following: -* Your GitHub repository is part of the - [Bazel GitHub organization](https://github.com/bazelbuild). -* You have configured Buildkite for your repository as described in - [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). +- Your GitHub repository is part of the [Bazel GitHub organization](https://github.com/bazelbuild). +- You have configured Buildkite for your repository as described in [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). ## Setting up the Bazel CI for testing -1. In your `.bazelci/presubmit.yml` file, do the following: +1. In your `.bazelci/presubmit.yml` file, do the following: - a. Add a config named `rbe_ubuntu1604`. + a. Add a config named `rbe_ubuntu1604`. - b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. + b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. -2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) - GitHub repository to your `WORKSPACE` file, pinned to the - [latest release](https://releases.bazel.build/bazel-toolchains.html). Also - add an `rbe_autoconfig` target with name `buildkite_config`. This example - creates toolchain configuration for remote execution with BuildKite CI - for `rbe_ubuntu1604`. +2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) GitHub repository to your `WORKSPACE` file, pinned to the [latest release](https://releases.bazel.build/bazel-toolchains.html). Also add an `rbe_autoconfig` target with name `buildkite_config`. This example creates toolchain configuration for remote execution with BuildKite CI for `rbe_ubuntu1604`. ```posix-terminal load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") @@ -40,43 +27,25 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") rbe_autoconfig(name = "buildkite_config") ``` -3. Send a pull request with your changes to the `presubmit.yml` file. (See - [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) +3. Send a pull request with your changes to the `presubmit.yml` file. (See [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) -4. To view build results, click **Details** for the RBE (Ubuntu - 16.04) pull request check in GitHub, as shown in the figure below. This link - becomes available after the pull request has been merged and the CI tests - have run. (See - [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) +4. To view build results, click **Details** for the RBE (Ubuntu 16.04) pull request check in GitHub, as shown in the figure below. This link becomes available after the pull request has been merged and the CI tests have run. (See [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) - ![Example results](/docs/images/rbe-ci-1.png "Example results") + ![Example results](/docs/images/rbe-ci-1.png "Example results") -5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test - required to pass before merging in your branch protection rule. The setting - is located in GitHub in **Settings > Branches > Branch protection rules**, - as shown in the following figure. +5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test required to pass before merging in your branch protection rule. The setting is located in GitHub in **Settings > Branches > Branch protection rules**, as shown in the following figure. - ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") + ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") ## Troubleshooting failed builds and tests If your build or tests fail, it's likely due to the following: -* **Required build or test tools are not installed in the default container.** - Builds using the `rbe_ubuntu1604` config run by default inside an - [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) - container, which includes tools common to many Bazel builds. However, if - your rules require tools not present in the default container, you must - create a custom container based on the - [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) - container and include those tools as described later. +- **Required build or test tools are not installed in the default container.** Builds using the `rbe_ubuntu1604` config run by default inside an [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container, which includes tools common to many Bazel builds. However, if your rules require tools not present in the default container, you must create a custom container based on the [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container and include those tools as described later. -* **Build or test targets are using rules that are incompatible with remote - execution.** See - [Adapting Bazel Rules for Remote Execution](/remote/rules) for - details about compatibility with remote execution. +- **Build or test targets are using rules that are incompatible with remote execution.** See [Adapting Bazel Rules for Remote Execution](/remote/rules) for details about compatibility with remote execution. -## Using a custom container in the rbe_ubuntu1604 CI config +## Using a custom container in the rbe\_ubuntu1604 CI config The `rbe-ubuntu16-04` container is publicly available at the following URL: @@ -84,121 +53,99 @@ The `rbe-ubuntu16-04` container is publicly available at the following URL: http://gcr.io/cloud-marketplace/google/rbe-ubuntu16-04 ``` -You can pull it directly from Container Registry or build it from source. The -next sections describe both options. +You can pull it directly from Container Registry or build it from source. The next sections describe both options. -Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. -If you are building the container from source, you must also install the latest -version of Bazel. +Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. If you are building the container from source, you must also install the latest version of Bazel. ### Pulling the rbe-ubuntu16-04 from Container Registry -To pull the `rbe-ubuntu16-04` container from Container Registry, run the -following command: +To pull the `rbe-ubuntu16-04` container from Container Registry, run the following command: ```posix-terminal -gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} +gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> ``` -Replace sha256-checksum with the SHA256 checksum value for -[the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). +Replace \{\{ '`' }}sha256-checksum{{ '`' \}\} with the SHA256 checksum value for [the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). ### Building the rbe-ubuntu16-04 container from source To build the `rbe-ubuntu16-04` container from source, do the following: -1. Clone the `bazel-toolchains` repository: +1. Clone the `bazel-toolchains` repository: - ```posix-terminal - git clone https://github.com/bazelbuild/bazel-toolchains - ``` + ```posix-terminal + git clone https://github.com/bazelbuild/bazel-toolchains + ``` -2. Set up toolchain container targets and build the container as explained in - [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). +2. Set up toolchain container targets and build the container as explained in [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). -3. Pull the freshly built container: +3. Pull the freshly built container: - ```posix-terminal -gcloud docker -- pull gcr.io/project-id/custom-container-namesha256-checksum - ``` + ```posix-terminal + ``` + +gcloud docker -- pull gcr.io/\{\{ '`' }}project-id{{ '`' \}\}/\{\{ '`' }}custom-container-name{{ '`' \}\}\{\{ '`' }}sha256-checksum{{ '`' \}\} \`\`\` ### Running the custom container To run the custom container, do one of the following: -* If you pulled the container from Container Registry, run the following - command: +- If you pulled the container from Container Registry, run the following command: - ```posix-terminal - docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:sha256-checksum/bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var>/bin/bash + ``` - Replace `sha256-checksum` with the SHA256 checksum value for the - [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). + Replace `sha256-checksum` with the SHA256 checksum value for the [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). -* If you built the container from source, run the following command: +- If you built the container from source, run the following command: - ```posix-terminal - docker run -it gcr.io/project-id/custom-container-name@sha256:sha256sum /bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/<var>project-id</var>/<var>custom-container-name</var>@sha256:<var>sha256sum</var> /bin/bash + ``` ### Adding resources to the custom container -Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or -[`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or -alternate versions of the original resources to the `rbe-ubuntu16-04` container. -If you are new to Docker, read the following: +Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or [`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or alternate versions of the original resources to the `rbe-ubuntu16-04` container. If you are new to Docker, read the following: -* [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) -* [Docker Samples](https://docs.docker.com/samples/) +- [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) +- [Docker Samples](https://docs.docker.com/samples/) -For example, the following `Dockerfile` snippet installs `my_tool_package`: +For example, the following `Dockerfile` snippet installs `<var>my_tool_package</var>`: ``` -FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} -RUN apt-get update && yes | apt-get install -y {{ '' }}my_tool_package{{ '' }} +FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> +RUN apt-get update && yes | apt-get install -y <var>my_tool_package</var> ``` ### Pushing the custom container to Container Registry -Once you have customized the container, build the container image and push it to -Container Registry as follows: +Once you have customized the container, build the container image and push it to Container Registry as follows: 1. Build the container image: - ```posix-terminal - docker build -t custom-container-name. - - docker tag custom-container-name gcr.io/project-id/custom-container-name - ``` + ```posix-terminal + docker build -t <var>custom-container-name</var>. -2. Push the container image to Container Registry: + docker tag <var>custom-container-name</var> gcr.io/<var>project-id</var>/<var>custom-container-name</var> + ``` - ```posix-terminal - gcloud docker -- push gcr.io/project-id/custom-container-name - ``` +2. Push the container image to Container Registry: -3. Navigate to the following URL to verify the container has been pushed: + ```posix-terminal + gcloud docker -- push gcr.io/<var>project-id</var>/<var>custom-container-name</var> + ``` - https://console.cloud.google.com/gcr/images/project-id/GLOBAL/custom-container-name +3. Navigate to the following URL to verify the container has been pushed: -4. Take note of the SHA256 checksum of your custom container. You will need to - provide it in your build platform definition later. + [https://console.cloud.google.com/gcr/images/\{\{](https://console.cloud.google.com/gcr/images/%7B%7B) '`' }}project-id{{ '`' \}\}/GLOBAL/\{\{ '`' }}custom-container-name{{ '`' \}\} -5. Configure the container for public access as described in publicly - accessible as explained in - [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). +4. Take note of the SHA256 checksum of your custom container. You will need to provide it in your build platform definition later. - For more information, see - [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). +5. Configure the container for public access as described in publicly accessible as explained in [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). + For more information, see [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). ### Specifying the build platform definition -You must include a [Bazel platform](/extending/platforms) configuration in your -custom toolchain configuration, which allows Bazel to select a toolchain -appropriate to the desired hardware/software platform. To generate -automatically a valid platform, you can add to your `WORKSPACE` an -`rbe_autoconfig` target with name `buildkite_config` which includes additional -attrs to select your custom container. For details on this setup, read -the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). +You must include a [Bazel platform](/extending/platforms) configuration in your custom toolchain configuration, which allows Bazel to select a toolchain appropriate to the desired hardware/software platform. To generate automatically a valid platform, you can add to your `WORKSPACE` an `rbe_autoconfig` target with name `buildkite_config` which includes additional attrs to select your custom container. For details on this setup, read the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). diff --git a/remote/creating.mdx b/remote/creating.mdx index 0e46a07a..4b2cf226 100644 --- a/remote/creating.mdx +++ b/remote/creating.mdx @@ -2,49 +2,30 @@ title: 'Creating Persistent Workers' --- +[Persistent workers](/remote/persistent) can make your build faster. If you have repeated actions in your build that have a high startup cost or would benefit from cross-action caching, you may want to implement your own persistent worker to perform these actions. - -[Persistent workers](/remote/persistent) can make your build faster. If -you have repeated actions in your build that have a high startup cost or would -benefit from cross-action caching, you may want to implement your own persistent -worker to perform these actions. - -The Bazel server communicates with the worker using `stdin`/`stdout`. It -supports the use of protocol buffers or JSON strings. +The Bazel server communicates with the worker using `stdin`/`stdout`. It supports the use of protocol buffers or JSON strings. The worker implementation has two parts: -* The [worker](#making-worker). -* The [rule that uses the worker](#rule-uses-worker). +- The [worker](#making-worker). +- The [rule that uses the worker](#rule-uses-worker). ## Making the worker A persistent worker upholds a few requirements: -* It reads - [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) - from its `stdin`. -* It writes - [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) - (and only `WorkResponse`s) to its `stdout`. -* It accepts the `--persistent_worker` flag. The wrapper must recognize the - `--persistent_worker` command-line flag and only make itself persistent if - that flag is passed, otherwise it must do a one-shot compilation and exit. +- It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) from its `stdin`. +- It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) (and only `WorkResponse`s) to its `stdout`. +- It accepts the `--persistent_worker` flag. The wrapper must recognize the `--persistent_worker` command-line flag and only make itself persistent if that flag is passed, otherwise it must do a one-shot compilation and exit. -If your program upholds these requirements, it can be used as a persistent -worker! +If your program upholds these requirements, it can be used as a persistent worker! ### Work requests -A `WorkRequest` contains a list of arguments to the worker, a list of -path-digest pairs representing the inputs the worker can access (this isn’t -enforced, but you can use this info for caching), and a request id, which is 0 -for singleplex workers. +A `WorkRequest` contains a list of arguments to the worker, a list of path-digest pairs representing the inputs the worker can access (this isn’t enforced, but you can use this info for caching), and a request id, which is 0 for singleplex workers. -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), -the JSON protocol uses "camel case" (`requestId`). This document uses camel case -in the JSON examples, but snake case when talking about the field regardless of -protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). This document uses camel case in the JSON examples, but snake case when talking about the field regardless of protocol. ```json { @@ -57,24 +38,13 @@ protocol. } ``` -The optional `verbosity` field can be used to request extra debugging output -from the worker. It is entirely up to the worker what and how to output. Higher -values indicate more verbose output. Passing the `--worker_verbose` flag to -Bazel sets the `verbosity` field to 10, but smaller or larger values can be used -manually for different amounts of output. +The optional `verbosity` field can be used to request extra debugging output from the worker. It is entirely up to the worker what and how to output. Higher values indicate more verbose output. Passing the `--worker_verbose` flag to Bazel sets the `verbosity` field to 10, but smaller or larger values can be used manually for different amounts of output. -The optional `sandbox_dir` field is used only by workers that support -[multiplex sandboxing](/remote/multiplex). +The optional `sandbox_dir` field is used only by workers that support [multiplex sandboxing](/remote/multiplex). ### Work responses -A `WorkResponse` contains a request id, a zero or nonzero exit code, and an -output message describing any errors encountered in processing or executing -the request. A worker should capture the `stdout` and `stderr` of any tool it -calls and report them through the `WorkResponse`. Writing it to the `stdout` of -the worker process is unsafe, as it will interfere with the worker protocol. -Writing it to the `stderr` of the worker process is safe, but the result is -collected in a per-worker log file instead of ascribed to individual actions. +A `WorkResponse` contains a request id, a zero or nonzero exit code, and an output message describing any errors encountered in processing or executing the request. A worker should capture the `stdout` and `stderr` of any tool it calls and report them through the `WorkResponse`. Writing it to the `stdout` of the worker process is unsafe, as it will interfere with the worker protocol. Writing it to the `stderr` of the worker process is safe, but the result is collected in a per-worker log file instead of ascribed to individual actions. ```json { @@ -85,10 +55,7 @@ collected in a per-worker log file instead of ascribed to individual actions. } ``` -As per the norm for protobufs, all fields are optional. However, Bazel requires -the `WorkRequest` and the corresponding `WorkResponse`, to have the same request -id, so the request id must be specified if it is nonzero. This is a valid -`WorkResponse`. +As per the norm for protobufs, all fields are optional. However, Bazel requires the `WorkRequest` and the corresponding `WorkResponse`, to have the same request id, so the request id must be specified if it is nonzero. This is a valid `WorkResponse`. ```json { @@ -96,69 +63,35 @@ id, so the request id must be specified if it is nonzero. This is a valid } ``` -A `request_id` of 0 indicates a "singleplex" request, used when this request -cannot be processed in parallel with other requests. The server guarantees that -a given worker receives requests with either only `request_id` 0 or only -`request_id` greater than zero. Singleplex requests are sent in serial, for -example if the server doesn't send another request until it has received a -response (except for cancel requests, see below). +A `request_id` of 0 indicates a "singleplex" request, used when this request cannot be processed in parallel with other requests. The server guarantees that a given worker receives requests with either only `request_id` 0 or only `request_id` greater than zero. Singleplex requests are sent in serial, for example if the server doesn't send another request until it has received a response (except for cancel requests, see below). **Notes** -* Each protocol buffer is preceded by its length in `varint` format (see - [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). -* JSON requests and responses are not preceded by a size indicator. -* JSON requests uphold the same structure as the protobuf, but use standard - JSON and use camel case for all field names. -* In order to maintain the same backward and forward compatibility properties - as protobuf, JSON workers must tolerate unknown fields in these messages, - and use the protobuf defaults for missing values. -* Bazel stores requests as protobufs and converts them to JSON using - [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) +- Each protocol buffer is preceded by its length in `varint` format (see [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). +- JSON requests and responses are not preceded by a size indicator. +- JSON requests uphold the same structure as the protobuf, but use standard JSON and use camel case for all field names. +- In order to maintain the same backward and forward compatibility properties as protobuf, JSON workers must tolerate unknown fields in these messages, and use the protobuf defaults for missing values. +- Bazel stores requests as protobufs and converts them to JSON using [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) ### Cancellation -Workers can optionally allow work requests to be cancelled before they finish. -This is particularly useful in connection with dynamic execution, where local -execution can regularly be interrupted by a faster remote execution. To allow -cancellation, add `supports-worker-cancellation: 1` to the -`execution-requirements` field (see below) and set the -`--experimental_worker_cancellation` flag. - -A **cancel request** is a `WorkRequest` with the `cancel` field set (and -similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` -field set). The only other field that must be in a cancel request or cancel -response is `request_id`, indicating which request to cancel. The `request_id` -field will be 0 for singleplex workers or the non-0 `request_id` of a previously -sent `WorkRequest` for multiplex workers. The server may send cancel requests -for requests that the worker has already responded to, in which case the cancel -request must be ignored. - -Each non-cancel `WorkRequest` message must be answered exactly once, whether or -not it was cancelled. Once the server has sent a cancel request, the worker may -respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` -field set to true. Sending a regular `WorkResponse` is also accepted, but the -`output` and `exit_code` fields will be ignored. - -Once a response has been sent for a `WorkRequest`, the worker must not touch the -files in its working directory. The server is free to clean up the files, -including temporary files. +Workers can optionally allow work requests to be cancelled before they finish. This is particularly useful in connection with dynamic execution, where local execution can regularly be interrupted by a faster remote execution. To allow cancellation, add `supports-worker-cancellation: 1` to the `execution-requirements` field (see below) and set the `--experimental_worker_cancellation` flag. + +A **cancel request** is a `WorkRequest` with the `cancel` field set (and similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` field set). The only other field that must be in a cancel request or cancel response is `request_id`, indicating which request to cancel. The `request_id` field will be 0 for singleplex workers or the non-0 `request_id` of a previously sent `WorkRequest` for multiplex workers. The server may send cancel requests for requests that the worker has already responded to, in which case the cancel request must be ignored. + +Each non-cancel `WorkRequest` message must be answered exactly once, whether or not it was cancelled. Once the server has sent a cancel request, the worker may respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` field set to true. Sending a regular `WorkResponse` is also accepted, but the `output` and `exit_code` fields will be ignored. + +Once a response has been sent for a `WorkRequest`, the worker must not touch the files in its working directory. The server is free to clean up the files, including temporary files. ## Making the rule that uses the worker -You'll also need to create a rule that generates actions to be performed by the -worker. Making a Starlark rule that uses a worker is just like -[creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). +You'll also need to create a rule that generates actions to be performed by the worker. Making a Starlark rule that uses a worker is just like [creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). -In addition, the rule needs to contain a reference to the worker itself, and -there are some requirements for the actions it produces. +In addition, the rule needs to contain a reference to the worker itself, and there are some requirements for the actions it produces. ### Referring to the worker -The rule that uses the worker needs to contain a field that refers to the worker -itself, so you'll need to create an instance of a `\*\_binary` rule to define -your worker. If your worker is called `MyWorker.Java`, this might be the -associated rule: +The rule that uses the worker needs to contain a field that refers to the worker itself, so you'll need to create an instance of a `\*\_binary` rule to define your worker. If your worker is called `MyWorker.Java`, this might be the associated rule: ```python java_binary( @@ -167,12 +100,9 @@ java_binary( ) ``` -This creates the "worker" label, which refers to the worker binary. You'll then -define a rule that *uses* the worker. This rule should define an attribute that -refers to the worker binary. +This creates the "worker" label, which refers to the worker binary. You'll then define a rule that *uses* the worker. This rule should define an attribute that refers to the worker binary. -If the worker binary you built is in a package named "work", which is at the top -level of the build, this might be the attribute definition: +If the worker binary you built is in a package named "work", which is at the top level of the build, this might be the attribute definition: ```python "worker": attr.label( @@ -182,46 +112,25 @@ level of the build, this might be the attribute definition: ) ``` -`cfg = "exec"` indicates that the worker should be built to run on your -execution platform rather than on the target platform (i.e., the worker is used -as tool during the build). +`cfg = "exec"` indicates that the worker should be built to run on your execution platform rather than on the target platform (i.e., the worker is used as tool during the build). ### Work action requirements -The rule that uses the worker creates actions for the worker to perform. These -actions have a couple of requirements. +The rule that uses the worker creates actions for the worker to perform. These actions have a couple of requirements. -* The *"arguments"* field. This takes a list of strings, all but the last of - which are arguments passed to the worker upon startup. The last element in - the "arguments" list is a `flag-file` (@-preceded) argument. Workers read - the arguments from the specified flagfile on a per-WorkRequest basis. Your - rule can write non-startup arguments for the worker to this flagfile. +- The *"arguments"* field. This takes a list of strings, all but the last of which are arguments passed to the worker upon startup. The last element in the "arguments" list is a `flag-file` (@-preceded) argument. Workers read the arguments from the specified flagfile on a per-WorkRequest basis. Your rule can write non-startup arguments for the worker to this flagfile. -* The *"execution-requirements"* field, which takes a dictionary containing - `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. +- The *"execution-requirements"* field, which takes a dictionary containing `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. - The "arguments" and "execution-requirements" fields are required for all - actions sent to workers. Additionally, actions that should be executed by - JSON workers need to include `"requires-worker-protocol" : "json"` in the - execution requirements field. `"requires-worker-protocol" : "proto"` is also - a valid execution requirement, though it’s not required for proto workers, - since they are the default. + The "arguments" and "execution-requirements" fields are required for all actions sent to workers. Additionally, actions that should be executed by JSON workers need to include `"requires-worker-protocol" : "json"` in the execution requirements field. `"requires-worker-protocol" : "proto"` is also a valid execution requirement, though it’s not required for proto workers, since they are the default. - You can also set a `worker-key-mnemonic` in the execution requirements. This - may be useful if you're reusing the executable for multiple action types and - want to distinguish actions by this worker. + You can also set a `worker-key-mnemonic` in the execution requirements. This may be useful if you're reusing the executable for multiple action types and want to distinguish actions by this worker. -* Temporary files generated in the course of the action should be saved to the - worker's directory. This enables sandboxing. +- Temporary files generated in the course of the action should be saved to the worker's directory. This enables sandboxing. -Note: To pass an argument starting with a literal `@`, start the argument with -`@@` instead. If an argument is also an external repository label, it will not -be considered a flagfile argument. +Note: To pass an argument starting with a literal `@`, start the argument with `@@` instead. If an argument is also an external repository label, it will not be considered a flagfile argument. -Assuming a rule definition with "worker" attribute described above, in addition -to a "srcs" attribute representing the inputs, an "output" attribute -representing the outputs, and an "args" attribute representing the worker -startup args, the call to `ctx.actions.run` might be: +Assuming a rule definition with "worker" attribute described above, in addition to a "srcs" attribute representing the inputs, an "output" attribute representing the outputs, and an "args" attribute representing the worker startup args, the call to `ctx.actions.run` might be: ```python ctx.actions.run( @@ -236,26 +145,14 @@ ctx.actions.run( ) ``` -For another example, see -[Implementing persistent workers](/remote/persistent#implementation). +For another example, see [Implementing persistent workers](/remote/persistent#implementation). ## Examples -The Bazel code base uses -[Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), -in addition to an -[example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) -that is used in our integration tests. +The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), in addition to an [example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) that is used in our integration tests. -You can use their -[scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) -to make any Java-based tool into a worker by passing in the correct callback. +You can use their [scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) to make any Java-based tool into a worker by passing in the correct callback. -For an example of a rule that uses a worker, take a look at Bazel's -[worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). +For an example of a rule that uses a worker, take a look at Bazel's [worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). -External contributors have implemented workers in a variety of languages; take a -look at -[Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). -You can -[find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)! +External contributors have implemented workers in a variety of languages; take a look at [Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). You can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest\&type=Code)! diff --git a/remote/dynamic.mdx b/remote/dynamic.mdx new file mode 100644 index 00000000..002d7373 --- /dev/null +++ b/remote/dynamic.mdx @@ -0,0 +1,63 @@ +--- +title: 'Dynamic Execution' +--- + +**Dynamic execution** is a feature in Bazel where local and remote execution of the same action are started in parallel, using the output from the first branch that finishes, cancelling the other branch. It combines the execution power and/or large shared cache of a remote build system with the low latency of local execution, providing the best of both worlds for clean and incremental builds alike. + +This page describes how to enable, tune, and debug dynamic execution. If you have both local and remote execution set up and are trying to adjust Bazel settings for better performance, this page is for you. If you don't already have remote execution set up, go to the Bazel [Remote Execution Overview](/remote/rbe) first. + +## Enabling dynamic execution? + +The dynamic execution module is part of Bazel, but to make use of dynamic execution, you must already be able to compile both locally and remotely from the same Bazel setup. + +To enable the dynamic execution module, pass the `--internal_spawn_scheduler` flag to Bazel. This adds a new execution strategy called `dynamic`. You can now use this as your strategy for the mnemonics you want to run dynamically, such as `--strategy=Javac=dynamic`. See the next section for how to pick which mnemonics to enable dynamic execution for. + +For any mnemonic using the dynamic strategy, the remote execution strategies are taken from the `--dynamic_remote_strategy` flag, and local strategies from the `--dynamic_local_strategy` flag. Passing `--dynamic_local_strategy=worker,sandboxed` sets the default for the local branch of dynamic execution to try with workers or sandboxed execution in that order. Passing `--dynamic_local_strategy=Javac=worker` overrides the default for the Javac mnemonic only. The remote version works the same way. Both flags can be specified multiple times. If an action cannot be executed locally, it is executed remotely as normal, and vice-versa. + +If your remote system has a cache, the `--dynamic_local_execution_delay` flag adds a delay in milliseconds to the local execution after the remote system has indicated a cache hit. This avoids running local execution when more cache hits are likely. The default value is 1000ms, but should be tuned to being just a bit longer than cache hits usually take. The actual time depends both on the remote system and on how long a round-trip takes. Usually, the value will be the same for all users of a given remote system, unless some of them are far enough away to add roundtrip latency. You can use the [Bazel profiling features](/rules/performance#performance-profiling) to look at how long typical cache hits take. + +Dynamic execution can be used with local sandboxed strategy as well as with [persistent workers](/remote/persistent). Persistent workers will automatically run with sandboxing when used with dynamic execution, and cannot use [multiplex workers](/remote/multiplex). On Darwin and Windows systems, the sandboxed strategy can be slow; you can pass `--reuse_sandbox_directories` to reduce overhead of creating sandboxes on these systems. + +Dynamic execution can also run with the `standalone` strategy, though since the `standalone` strategy must take the output lock when it starts executing, it effectively blocks the remote strategy from finishing first. The `--experimental_local_lockfree_output` flag enables a way around this problem by allowing the local execution to write directly to the output, but be aborted by the remote execution, should that finish first. + +If one of the branches of dynamic execution finishes first but is a failure, the entire action fails. This is an intentional choice to prevent differences between local and remote execution from going unnoticed. + +For more background on how dynamic execution and its locking works, see Julio Merino's excellent [blog posts](https://jmmv.dev/series/bazel-dynamic-execution/) + +## When should I use dynamic execution? + +Dynamic execution requires some form of [remote execution system](/remote/rbe). It is not currently possible to use a cache-only remote system, as a cache miss would be considered a failed action. + +Not all types of actions are well suited for remote execution. The best candidates are those that are inherently faster locally, for instance through the use of [persistent workers](/remote/persistent), or those that run fast enough that the overhead of remote execution dominates execution time. Since each locally executed action locks some amount of CPU and memory resources, running actions that don't fall into those categories merely delays execution for those that do. + +As of release [5.0.0-pre.20210708.4](https://github.com/bazelbuild/bazel/releases/tag/5.0.0-pre.20210708.4), [performance profiling](/rules/performance#performance-profiling) contains data about worker execution, including time spent finishing a work request after losing a dynamic execution race. If you see dynamic execution worker threads spending significant time acquiring resources, or a lot of time in the `async-worker-finish`, you may have some slow local actions delaying the worker threads. + +![Profiling data with poor dynamic execution performance](/docs/images/dyn-trace-alldynamic.png) + +In the profile above, which uses 8 Javac workers, we see many Javac workers having lost the races and finishing their work on the `async-worker-finish` threads. This was caused by a non-worker mnemonic taking enough resources to delay the workers. + +![Profiling data with better dynamic execution performance](/docs/images/dyn-trace-javaconly.png) + +When only Javac is run with dynamic execution, only about half of the started workers end up losing the race after starting their work. + +The previously recommended `--experimental_spawn_scheduler` flag is deprecated. It turns on dynamic execution and sets `dynamic` as the default strategy for all mnemonics, which would often lead to these kinds of problems. + +## Performance + +The dynamic execution approach assumes there are enough resources available locally and remotely that it's worth spending some extra resources to improve overall performance. But excessive resource usage may slow down Bazel itself or the machine it runs on, or put unexpected pressure on a remote system. There are several options for changing the behaviour of dynamic execution: + +`--dynamic_local_execution_delay` delays the start of a local branch by a number of milliseconds after the remote branch has started, but only if there has been a remote cache hit during the current build. This makes builds that benefit from remote caching not waste local resources when it is likely that most outputs can be found in the cache. Depending on the quality of the cache, reducing this might improve build speeds, at the cost of using more local resources. + +`--experimental_dynamic_local_load_factor` is an experimental advanced resource management option. It takes a value from 0 to 1, 0 turning off this feature. When set to a value above 0, Bazel adjusts the number of locally scheduled actions when many actions waiting to be scheduled. Setting it to 1 allows as many actions to be scheduled as there are CPUs available (as per `--local_resources`). Lower values set the number of actions scheduled to correspondingly fewer as higher numbers of actions are available to run. This may sound counter-intuitive, but with a good remote system, local execution does not help much when many actions are being run, and the local CPU is better spent managing remote actions. + +`--experimental_dynamic_slow_remote_time` prioritizes starting local branches when the remote branch has been running for at least this long. Normally the most recently scheduled action gets priority, as it has the greatest chance of winning the race, but if the remote system sometimes hangs or takes extra long, this can get a build to move along. This is not enabled by default, because it could hide issues with the remote system that should rather be fixed. Make sure to monitor your remote system performance if you enable this option. + +`--experimental_dynamic_ignore_local_signals` can be used to let the remote branch take over when a local spawn exits due to a given signal. This is is mainly useful together with worker resource limits (see [`--experimental_worker_memory_limit_mb`](https://bazel.build/reference/command-line-reference#flag--experimental_worker_memory_limit_mb), [`--experimental_worker_sandbox_hardening`](https://bazel.build/reference/command-line-reference#flag--experimental_worker_sandbox_hardening), and [`--experimental_sandbox_memory_limit_mb`)](https://bazel.build/reference/command-line-reference#flag--experimental_sandbox_memory_limit_mb)), where worker processes may be killed when they use too many resources. + +The [JSON trace profile](/advanced/performance/json-trace-profile) contains a number of performance-related graphs that can help identify ways to improve the trade-off of performance and resource usage. + +## Troubleshooting + +Problems with dynamic execution can be subtle and hard to debug, as they can manifest only under some specific combinations of local and remote execution. The `--debug_spawn_scheduler` adds extra output from the dynamic execution system that can help debug these problems. You can also adjust the `--dynamic_local_execution_delay` flag and number of remote vs. local jobs to make it easier to reproduce the problems. + +If you are experiencing problems with dynamic execution using the `standalone` strategy, try running without `--experimental_local_lockfree_output`, or run your local actions sandboxed. This may slow down your build a bit (see above if you're on Mac or Windows), but removes some possible causes for failures. diff --git a/remote/multiplex.mdx b/remote/multiplex.mdx index b4b0a0d4..d9075a8a 100644 --- a/remote/multiplex.mdx +++ b/remote/multiplex.mdx @@ -2,112 +2,38 @@ title: 'Multiplex Workers (Experimental Feature)' --- - - -This page describes multiplex workers, how to write multiplex-compatible -rules, and workarounds for certain limitations. +This page describes multiplex workers, how to write multiplex-compatible rules, and workarounds for certain limitations. Caution: Experimental features are subject to change at any time. -_Multiplex workers_ allow Bazel to handle multiple requests with a single worker -process. For multi-threaded workers, Bazel can use fewer resources to -achieve the same, or better performance. For example, instead of having one -worker process per worker, Bazel can have four multiplexed workers talking to -the same worker process, which can then handle requests in parallel. For -languages like Java and Scala, this saves JVM warm-up time and JIT compilation -time, and in general it allows using one shared cache between all workers of -the same type. +*Multiplex workers* allow Bazel to handle multiple requests with a single worker process. For multi-threaded workers, Bazel can use fewer resources to achieve the same, or better performance. For example, instead of having one worker process per worker, Bazel can have four multiplexed workers talking to the same worker process, which can then handle requests in parallel. For languages like Java and Scala, this saves JVM warm-up time and JIT compilation time, and in general it allows using one shared cache between all workers of the same type. ## Overview -There are two layers between the Bazel server and the worker process. For certain -mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from -the worker pool. The `WorkerProxy` forwards requests to the worker process -sequentially along with a `request_id`, the worker process processes the request -and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` -receives a response, it parses the `request_id` and then forwards the responses -back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all -communication is done over standard in/out, but the tool cannot just use -`stderr` for user-visible output ([see below](#output)). - -Each worker has a key. Bazel uses the key's hash code (composed of environment -variables, the execution root, and the mnemonic) to determine which -`WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same -`WorkerMultiplexer` if they have the same hash code. Therefore, assuming -environment variables and the execution root are the same in a single Bazel -invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one -worker process. The total number of workers, including regular workers and -`WorkerProxy`s, is still limited by `--worker_max_instances`. +There are two layers between the Bazel server and the worker process. For certain mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from the worker pool. The `WorkerProxy` forwards requests to the worker process sequentially along with a `request_id`, the worker process processes the request and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` receives a response, it parses the `request_id` and then forwards the responses back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all communication is done over standard in/out, but the tool cannot just use `stderr` for user-visible output ([see below](#output)). + +Each worker has a key. Bazel uses the key's hash code (composed of environment variables, the execution root, and the mnemonic) to determine which `WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same `WorkerMultiplexer` if they have the same hash code. Therefore, assuming environment variables and the execution root are the same in a single Bazel invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one worker process. The total number of workers, including regular workers and `WorkerProxy`s, is still limited by `--worker_max_instances`. ## Writing multiplex-compatible rules -The rule's worker process should be multi-threaded to take advantage of -multiplex workers. Protobuf allows a ruleset to parse a single request even -though there might be multiple requests piling up in the stream. Whenever the -worker process parses a request from the stream, it should handle the request in -a new thread. Because different thread could complete and write to the stream at -the same time, the worker process needs to make sure the responses are written -atomically (messages don't overlap). Responses must contain the -`request_id` of the request they're handling. +The rule's worker process should be multi-threaded to take advantage of multiplex workers. Protobuf allows a ruleset to parse a single request even though there might be multiple requests piling up in the stream. Whenever the worker process parses a request from the stream, it should handle the request in a new thread. Because different thread could complete and write to the stream at the same time, the worker process needs to make sure the responses are written atomically (messages don't overlap). Responses must contain the `request_id` of the request they're handling. ### Handling multiplex output -Multiplex workers need to be more careful about handling their output than -singleplex workers. Anything sent to `stderr` will go into a single log file -shared among all `WorkerProxy`s of the same type, -randomly interleaved between concurrent requests. While redirecting `stdout` -into `stderr` is a good idea, do not collect that output into the `output` -field of `WorkResponse`, as that could show the user mangled pieces of output. -If your tool only sends user-oriented output to `stdout` or `stderr`, you will -need to change that behaviour before you can enable multiplex workers. +Multiplex workers need to be more careful about handling their output than singleplex workers. Anything sent to `stderr` will go into a single log file shared among all `WorkerProxy`s of the same type, randomly interleaved between concurrent requests. While redirecting `stdout` into `stderr` is a good idea, do not collect that output into the `output` field of `WorkResponse`, as that could show the user mangled pieces of output. If your tool only sends user-oriented output to `stdout` or `stderr`, you will need to change that behaviour before you can enable multiplex workers. ## Enabling multiplex workers -Multiplex workers are not enabled by default. A ruleset can turn on multiplex -workers by using the `supports-multiplex-workers` tag in the -`execution_requirements` of an action (just like the `supports-workers` tag -enables regular workers). As is the case when using regular workers, a worker -strategy needs to be specified, either at the ruleset level (for example, -`--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for -example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are -necessary, and `supports-multiplex-workers` takes precedence over -`supports-workers`, if both are set. You can turn off multiplex workers -globally by passing `--noworker_multiplex`. - -A ruleset is encouraged to use multiplex workers if possible, to reduce memory -pressure and improve performance. However, multiplex workers are not currently -compatible with [dynamic execution](/remote/dynamic) unless they -implement multiplex sandboxing. Attempting to run non-sandboxed multiplex -workers with dynamic execution will silently use sandboxed -singleplex workers instead. +Multiplex workers are not enabled by default. A ruleset can turn on multiplex workers by using the `supports-multiplex-workers` tag in the `execution_requirements` of an action (just like the `supports-workers` tag enables regular workers). As is the case when using regular workers, a worker strategy needs to be specified, either at the ruleset level (for example, `--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are necessary, and `supports-multiplex-workers` takes precedence over `supports-workers`, if both are set. You can turn off multiplex workers globally by passing `--noworker_multiplex`. + +A ruleset is encouraged to use multiplex workers if possible, to reduce memory pressure and improve performance. However, multiplex workers are not currently compatible with [dynamic execution](/remote/dynamic) unless they implement multiplex sandboxing. Attempting to run non-sandboxed multiplex workers with dynamic execution will silently use sandboxed singleplex workers instead. ## Multiplex sandboxing -Multiplex workers can be sandboxed by adding explicit support for it in the -worker implementations. While singleplex worker sandboxing can be done by -running each worker process in its own sandbox, multiplex workers share the -process working directory between multiple parallel requests. To allow -sandboxing of multiplex workers, the worker must support reading from and -writing to a subdirectory specified in each request, instead of directly in -its working directory. - -To support multiplex sandboxing, the worker must use the `sandbox_dir` field -from the `WorkRequest` and use that as a prefix for all file reads and writes. -While the `arguments` and `inputs` fields remain unchanged from an unsandboxed -request, the actual inputs are relative to the `sandbox_dir`. The worker must -translate file paths found in `arguments` and `inputs` to read from this -modified path, and must also write all outputs relative to the `sandbox_dir`. -This includes paths such as '.', as well as paths found in files specified -in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). - -Once a worker supports multiplex sandboxing, the ruleset can declare this -support by adding `supports-multiplex-sandboxing` to the -`execution_requirements` of an action. Bazel will then use multiplex sandboxing -if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if -the worker is used with dynamic execution. - -The worker files of a sandboxed multiplex worker are still relative to the -working directory of the worker process. Thus, if a file is -used both for running the worker and as an input, it must be specified both as -an input in the flagfile argument as well as in `tools`, `executable`, or -`runfiles`. +Multiplex workers can be sandboxed by adding explicit support for it in the worker implementations. While singleplex worker sandboxing can be done by running each worker process in its own sandbox, multiplex workers share the process working directory between multiple parallel requests. To allow sandboxing of multiplex workers, the worker must support reading from and writing to a subdirectory specified in each request, instead of directly in its working directory. + +To support multiplex sandboxing, the worker must use the `sandbox_dir` field from the `WorkRequest` and use that as a prefix for all file reads and writes. While the `arguments` and `inputs` fields remain unchanged from an unsandboxed request, the actual inputs are relative to the `sandbox_dir`. The worker must translate file paths found in `arguments` and `inputs` to read from this modified path, and must also write all outputs relative to the `sandbox_dir`. This includes paths such as '.', as well as paths found in files specified in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). + +Once a worker supports multiplex sandboxing, the ruleset can declare this support by adding `supports-multiplex-sandboxing` to the `execution_requirements` of an action. Bazel will then use multiplex sandboxing if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if the worker is used with dynamic execution. + +The worker files of a sandboxed multiplex worker are still relative to the working directory of the worker process. Thus, if a file is used both for running the worker and as an input, it must be specified both as an input in the flagfile argument as well as in `tools`, `executable`, or `runfiles`. diff --git a/remote/output-directories.mdx b/remote/output-directories.mdx index 9c1ba329..26a36c2a 100644 --- a/remote/output-directories.mdx +++ b/remote/output-directories.mdx @@ -2,141 +2,108 @@ title: 'Output Directory Layout' --- - - This page covers requirements and layout for output directories. ## Requirements Requirements for an output directory layout: -* Doesn't collide if multiple users are building on the same box. -* Supports building in multiple workspaces at the same time. -* Supports building for multiple target configurations in the same workspace. -* Doesn't collide with any other tools. -* Is easy to access. -* Is easy to clean, even selectively. -* Is unambiguous, even if the user relies on symbolic links when changing into - their client directory. -* All the build state per user should be underneath one directory ("I'd like to - clean all the .o files from all my clients.") +- Doesn't collide if multiple users are building on the same box. +- Supports building in multiple workspaces at the same time. +- Supports building for multiple target configurations in the same workspace. +- Doesn't collide with any other tools. +- Is easy to access. +- Is easy to clean, even selectively. +- Is unambiguous, even if the user relies on symbolic links when changing into their client directory. +- All the build state per user should be underneath one directory ("I'd like to clean all the .o files from all my clients.") ## Current layout The solution that's currently implemented: -* Bazel must be invoked from a directory containing a repo boundary file, or a - subdirectory thereof. In other words, Bazel must be invoked from inside a - [repository](../external/overview#repository). Otherwise, an error is - reported. -* The _outputRoot_ directory defaults to `~/.cache/bazel` on Linux, - `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if - set, else `%USERPROFILE%` if set, else the result of calling - `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the - environment variable `$XDG_CACHE_HOME` is set on either Linux or - macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. - If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel - itself, then that value overrides any defaults. -* The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. - This is called the _outputUserRoot_ directory. -* Beneath the `outputUserRoot` directory there is an `install` directory, and in - it is an `installBase` directory whose name is the MD5 hash of the Bazel - installation manifest. -* Beneath the `outputUserRoot` directory, an `outputBase` directory - is also created whose name is the MD5 hash of the path name of the workspace - root. So, for example, if Bazel is running in the workspace root - `/home/user/src/my-project` (or in a directory symlinked to that one), then - an output base directory is created called: - `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You - can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. -* You can use Bazel's `--output_base` startup option to override the default - output base directory. For example, - `bazel --output_base=/tmp/bazel/output build x/y:z`. -* You can also use Bazel's `--output_user_root` startup option to override the - default install base and output base directories. For example: - `bazel --output_user_root=/tmp/bazel build x/y:z`. - -The symlinks for "bazel-<workspace-name>", "bazel-out", "bazel-testlogs", -and "bazel-bin" are put in the workspace directory; these symlinks point to some -directories inside a target-specific directory inside the output directory. -These symlinks are only for the user's convenience, as Bazel itself does not -use them. Also, this is done only if the workspace root is writable. +- Bazel must be invoked from a directory containing a repo boundary file, or a subdirectory thereof. In other words, Bazel must be invoked from inside a [repository](../external/overview#repository). Otherwise, an error is reported. +- The *outputRoot* directory defaults to `~/.cache/bazel` on Linux, `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if set, else `%USERPROFILE%` if set, else the result of calling `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the environment variable `$XDG_CACHE_HOME` is set on either Linux or macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel itself, then that value overrides any defaults. +- The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. This is called the *outputUserRoot* directory. +- Beneath the `outputUserRoot` directory there is an `install` directory, and in it is an `installBase` directory whose name is the MD5 hash of the Bazel installation manifest. +- Beneath the `outputUserRoot` directory, an `outputBase` directory is also created whose name is the MD5 hash of the path name of the workspace root. So, for example, if Bazel is running in the workspace root `/home/user/src/my-project` (or in a directory symlinked to that one), then an output base directory is created called: `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. +- You can use Bazel's `--output_base` startup option to override the default output base directory. For example, `bazel --output_base=/tmp/bazel/output build x/y:z`. +- You can also use Bazel's `--output_user_root` startup option to override the default install base and output base directories. For example: `bazel --output_user_root=/tmp/bazel build x/y:z`. + +The symlinks for "bazel-\", "bazel-out", "bazel-testlogs", and "bazel-bin" are put in the workspace directory; these symlinks point to some directories inside a target-specific directory inside the output directory. These symlinks are only for the user's convenience, as Bazel itself does not use them. Also, this is done only if the workspace root is writable. ## Layout diagram The directories are laid out as follows: ``` -<workspace-name>/ <== The workspace root - bazel-my-project => <..._main> <== Symlink to execRoot - bazel-out => <...bazel-out> <== Convenience symlink to outputPath - bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) - bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory - -/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot - _bazel_$USER/ <== Top level directory for a given user depends on the user name: +<workspace-name>/ <== The workspace root + bazel-my-project => <..._main> <== Symlink to execRoot + bazel-out => <...bazel-out> <== Convenience symlink to outputPath + bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) + bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory + +/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot + _bazel_$USER/ <== Top level directory for a given user depends on the user name: outputUserRoot install/ - fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase - _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of + fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase + _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of the bazel executable on first run (such as helper scripts and the main Java file BazelServer_deploy.jar) - 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as + 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as /home/user/src/my-project): outputBase - action_cache/ <== Action cache directory hierarchy + action_cache/ <== Action cache directory hierarchy This contains the persistent record of the file metadata (timestamps, and perhaps eventually also MD5 sums) used by the FilesystemValueChecker. - command.log <== A copy of the stdout/stderr output from the most + command.log <== A copy of the stdout/stderr output from the most recent bazel command. - external/ <== The directory that remote repositories are + external/ <== The directory that remote repositories are downloaded/symlinked into. - server/ <== The Bazel server puts all server-related files (such + server/ <== The Bazel server puts all server-related files (such as socket file, logs, etc) here. - jvm.out <== The debugging output for the server. - execroot/ <== The working directory for all actions. For special + jvm.out <== The debugging output for the server. + execroot/ <== The working directory for all actions. For special cases such as sandboxing and remote execution, the actions run in a directory that mimics execroot. Implementation details, such as where the directories are created, are intentionally hidden from the action. Every action can access its inputs and outputs relative to the execroot directory. - _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot - _bin/ <== Helper tools are linked from or copied to here. + _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot + _bin/ <== Helper tools are linked from or copied to here. - bazel-out/ <== All actual output of the build is under here: outputPath - _tmp/actions/ <== Action output directory. This contains a file with the + bazel-out/ <== All actual output of the build is under here: outputPath + _tmp/actions/ <== Action output directory. This contains a file with the stdout/stderr for every action from the most recent bazel run that produced output. - local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; + local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; this is currently encoded - bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) - foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz - foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc - other_package/other.o <== Object files from source //other_package:other.cc - foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named + bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) + foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz + foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc + other_package/other.o <== Object files from source //other_package:other.cc + foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named //foo/bar:baz - foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. + foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. MANIFEST _main/ ... - genfiles/ <== Bazel puts generated source for the target configuration here: + genfiles/ <== Bazel puts generated source for the target configuration here: $(GENDIR) foo/bar.h such as foo/bar.h might be a headerfile generated by //foo:bargen - testlogs/ <== Bazel internal test runner puts test log files here + testlogs/ <== Bazel internal test runner puts test log files here foo/bartest.log such as foo/bar.log might be an output of the //foo:bartest test with foo/bartest.status foo/bartest.status containing exit status of the test (such as PASSED or FAILED (Exit 1), etc) - host/ <== BuildConfiguration for build host (user's workstation), for + host/ <== BuildConfiguration for build host (user's workstation), for building prerequisite tools, that will be used in later stages of the build (ex: Protocol Compiler) - <packages>/ <== Packages referenced in the build appear as if under a regular workspace + <packages>/ <== Packages referenced in the build appear as if under a regular workspace ``` The layout of the \*.runfiles directories is documented in more detail in the places pointed to by RunfilesSupport. ## `bazel clean` -`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` -directory. It also removes the workspace symlinks. The `--expunge` option -will clean the entire outputBase. +`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` directory. It also removes the workspace symlinks. The `--expunge` option will clean the entire outputBase. diff --git a/remote/persistent.mdx b/remote/persistent.mdx index 1a56946a..bd9029f5 100644 --- a/remote/persistent.mdx +++ b/remote/persistent.mdx @@ -2,158 +2,65 @@ title: 'Persistent Workers' --- +This page covers how to use persistent workers, the benefits, requirements, and how workers affect sandboxing. +A persistent worker is a long-running process started by the Bazel server, which functions as a *wrapper* around the actual *tool* (typically a compiler), or is the *tool* itself. In order to benefit from persistent workers, the tool must support doing a sequence of compilations, and the wrapper needs to translate between the tool's API and the request/response format described below. The same worker might be called with and without the `--persistent_worker` flag in the same build, and is responsible for appropriately starting and talking to the tool, as well as shutting down workers on exit. Each worker instance is assigned (but not chrooted to) a separate working directory under `<outputBase>/bazel-workers`. -This page covers how to use persistent workers, the benefits, requirements, and -how workers affect sandboxing. +Using persistent workers is an [execution strategy](/docs/user-manual#execution-strategy) that decreases start-up overhead, allows more JIT compilation, and enables caching of for example the abstract syntax trees in the action execution. This strategy achieves these improvements by sending multiple requests to a long-running process. -A persistent worker is a long-running process started by the Bazel server, which -functions as a *wrapper* around the actual *tool* (typically a compiler), or is -the *tool* itself. In order to benefit from persistent workers, the tool must -support doing a sequence of compilations, and the wrapper needs to translate -between the tool's API and the request/response format described below. The same -worker might be called with and without the `--persistent_worker` flag in the -same build, and is responsible for appropriately starting and talking to the -tool, as well as shutting down workers on exit. Each worker instance is assigned -(but not chrooted to) a separate working directory under -`/bazel-workers`. +Persistent workers are implemented for multiple languages, including Java, [Scala](https://github.com/bazelbuild/rules_scala), [Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. -Using persistent workers is an -[execution strategy](/docs/user-manual#execution-strategy) that decreases -start-up overhead, allows more JIT compilation, and enables caching of for -example the abstract syntax trees in the action execution. This strategy -achieves these improvements by sending multiple requests to a long-running -process. - -Persistent workers are implemented for multiple languages, including Java, -[Scala](https://github.com/bazelbuild/rules_scala), -[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. - -Programs using a NodeJS runtime can use the -[@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to -implement the worker protocol. +Programs using a NodeJS runtime can use the [@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to implement the worker protocol. ## Using persistent workers -[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) -uses persistent workers by default when executing builds, though remote -execution takes precedence. For actions that do not support persistent workers, -Bazel falls back to starting a tool instance for each action. You can explicitly -set your build to use persistent workers by setting the `worker` -[strategy](/docs/user-manual#execution-strategy) for the applicable tool -mnemonics. As a best practice, this example includes specifying `local` as a -fallback to the `worker` strategy: +[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) uses persistent workers by default when executing builds, though remote execution takes precedence. For actions that do not support persistent workers, Bazel falls back to starting a tool instance for each action. You can explicitly set your build to use persistent workers by setting the `worker` [strategy](/docs/user-manual#execution-strategy) for the applicable tool mnemonics. As a best practice, this example includes specifying `local` as a fallback to the `worker` strategy: ```posix-terminal -bazel build //{{ '' }}my:target{{ '' }} --strategy=Javac=worker,local +bazel build //<var>my:target</var> --strategy=Javac=worker,local ``` -Using the workers strategy instead of the local strategy can boost compilation -speed significantly, depending on implementation. For Java, builds can be 2–4 -times faster, sometimes more for incremental compilation. Compiling Bazel is -about 2.5 times as fast with workers. For more details, see the -"[Choosing number of workers](#number-of-workers)" section. - -If you also have a remote build environment that matches your local build -environment, you can use the experimental -[*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), -which races a remote execution and a worker execution. To enable the dynamic -strategy, pass the -[--experimental_spawn_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) -flag. This strategy automatically enables workers, so there is no need to -specify the `worker` strategy, but you can still use `local` or `sandboxed` as -fallbacks. +Using the workers strategy instead of the local strategy can boost compilation speed significantly, depending on implementation. For Java, builds can be 2–4 times faster, sometimes more for incremental compilation. Compiling Bazel is about 2.5 times as fast with workers. For more details, see the "[Choosing number of workers](#number-of-workers)" section. + +If you also have a remote build environment that matches your local build environment, you can use the experimental [*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), which races a remote execution and a worker execution. To enable the dynamic strategy, pass the [--experimental\_spawn\_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) flag. This strategy automatically enables workers, so there is no need to specify the `worker` strategy, but you can still use `local` or `sandboxed` as fallbacks. ## Choosing number of workers -The default number of worker instances per mnemonic is 4, but can be adjusted -with the -[`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) -flag. There is a trade-off between making good use of the available CPUs and the -amount of JIT compilation and cache hits you get. With more workers, more -targets will pay start-up costs of running non-JITted code and hitting cold -caches. If you have a small number of targets to build, a single worker may give -the best trade-off between compilation speed and resource usage (for example, -see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). -The `worker_max_instances` flag sets the maximum number of worker instances per -mnemonic and flag set (see below), so in a mixed system you could end up using -quite a lot of memory if you keep the default value. For incremental builds the -benefit of multiple worker instances is even smaller. - -This graph shows the from-scratch compilation times for Bazel (target -`//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation -with 64 GB of RAM. For each worker configuration, five clean builds are run and -the average of the last four are taken. +The default number of worker instances per mnemonic is 4, but can be adjusted with the [`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) flag. There is a trade-off between making good use of the available CPUs and the amount of JIT compilation and cache hits you get. With more workers, more targets will pay start-up costs of running non-JITted code and hitting cold caches. If you have a small number of targets to build, a single worker may give the best trade-off between compilation speed and resource usage (for example, see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). The `worker_max_instances` flag sets the maximum number of worker instances per mnemonic and flag set (see below), so in a mixed system you could end up using quite a lot of memory if you keep the default value. For incremental builds the benefit of multiple worker instances is even smaller. + +This graph shows the from-scratch compilation times for Bazel (target `//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation with 64 GB of RAM. For each worker configuration, five clean builds are run and the average of the last four are taken. ![Graph of performance improvements of clean builds](/docs/images/workers-clean-chart.png "Performance improvements of clean builds") **Figure 1.** Graph of performance improvements of clean builds. -For this configuration, two workers give the fastest compile, though at only 14% -improvement compared to one worker. One worker is a good option if you want to -use less memory. +For this configuration, two workers give the fastest compile, though at only 14% improvement compared to one worker. One worker is a good option if you want to use less memory. -Incremental compilation typically benefits even more. Clean builds are -relatively rare, but changing a single file between compiles is common, in -particular in test-driven development. The above example also has some non-Java -packaging actions to it that can overshadow the incremental compile time. +Incremental compilation typically benefits even more. Clean builds are relatively rare, but changing a single file between compiles is common, in particular in test-driven development. The above example also has some non-Java packaging actions to it that can overshadow the incremental compile time. -Recompiling the Java sources only -(`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) -after changing an internal string constant in -[AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) -gives a 3x speed-up (average of 20 incremental builds with one warmup build -discarded): +Recompiling the Java sources only (`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) after changing an internal string constant in [AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) gives a 3x speed-up (average of 20 incremental builds with one warmup build discarded): ![Graph of performance improvements of incremental builds](/docs/images/workers-incremental-chart.png "Performance improvements of incremental builds") **Figure 2.** Graph of performance improvements of incremental builds. -The speed-up depends on the change being made. A speed-up of a factor 6 is -measured in the above situation when a commonly used constant is changed. +The speed-up depends on the change being made. A speed-up of a factor 6 is measured in the above situation when a commonly used constant is changed. ## Modifying persistent workers -You can pass the -[`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) -flag to specify start-up flags to workers, keyed by mnemonic. For instance, -passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. -Only one worker flag can be set per use of this flag, and only for one mnemonic. -Workers are not just created separately for each mnemonic, but also for -variations in their start-up flags. Each combination of mnemonic and start-up -flags is combined into a `WorkerKey`, and for each `WorkerKey` up to -`worker_max_instances` workers may be created. See the next section for how the -action configuration can also specify set-up flags. - -Passing the -[`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) -flag makes each worker request use a separate sandbox directory for all its -inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, -especially on macOS, but gives a better correctness guarantee. - -The -[`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) -flag is mainly useful for debugging and profiling. This flag forces all workers -to quit once a build is done. You can also pass -[`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to -get more output about what the workers are doing. This flag is reflected in the -`verbosity` field in `WorkRequest`, allowing worker implementations to also be -more verbose. - -Workers store their logs in the `/bazel-workers` directory, for -example -`/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. -The file name includes the worker id and the mnemonic. Since there can be more -than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` -log files for a given mnemonic. - -For Android builds, see details at the -[Android Build Performance page](/docs/android-build-performance). +You can pass the [`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) flag to specify start-up flags to workers, keyed by mnemonic. For instance, passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. Only one worker flag can be set per use of this flag, and only for one mnemonic. Workers are not just created separately for each mnemonic, but also for variations in their start-up flags. Each combination of mnemonic and start-up flags is combined into a `WorkerKey`, and for each `WorkerKey` up to `worker_max_instances` workers may be created. See the next section for how the action configuration can also specify set-up flags. + +Passing the [`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) flag makes each worker request use a separate sandbox directory for all its inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, especially on macOS, but gives a better correctness guarantee. + +The [`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) flag is mainly useful for debugging and profiling. This flag forces all workers to quit once a build is done. You can also pass [`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to get more output about what the workers are doing. This flag is reflected in the `verbosity` field in `WorkRequest`, allowing worker implementations to also be more verbose. + +Workers store their logs in the `<outputBase>/bazel-workers` directory, for example `/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. The file name includes the worker id and the mnemonic. Since there can be more than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` log files for a given mnemonic. + +For Android builds, see details at the [Android Build Performance page](/docs/android-build-performance). ## Implementing persistent workers -See the [creating persistent workers](/remote/creating) page for more -information on how to make a worker. +See the [creating persistent workers](/remote/creating) page for more information on how to make a worker. This example shows a Starlark configuration for a worker that uses JSON: @@ -174,14 +81,9 @@ ctx.actions.run( ) ``` -With this definition, the first use of this action would start with executing -the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request -to compile `Foo.java` would then look like: +With this definition, the first use of this action would start with executing the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request to compile `Foo.java` would then look like: -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), -the JSON protocol uses "camel case" (`requestId`). In this document, we will use -camel case in the JSON examples, but snake case when talking about the field -regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). In this document, we will use camel case in the JSON examples, but snake case when talking about the field regardless of protocol. ```json { @@ -193,12 +95,7 @@ regardless of protocol. } ``` -The worker receives this on `stdin` in newline-delimited JSON format (because -`requires-worker-protocol` is set to JSON). The worker then performs the action, -and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then -parses this response and manually converts it to a `WorkResponse` proto. To -communicate with the associated worker using binary-encoded protobuf instead of -JSON, `requires-worker-protocol` would be set to `proto`, like this: +The worker receives this on `stdin` in newline-delimited JSON format (because `requires-worker-protocol` is set to JSON). The worker then performs the action, and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then parses this response and manually converts it to a `WorkResponse` proto. To communicate with the associated worker using binary-encoded protobuf instead of JSON, `requires-worker-protocol` would be set to `proto`, like this: ``` execution_requirements = { @@ -207,59 +104,31 @@ JSON, `requires-worker-protocol` would be set to `proto`, like this: } ``` -If you do not include `requires-worker-protocol` in the execution requirements, -Bazel will default the worker communication to use protobuf. +If you do not include `requires-worker-protocol` in the execution requirements, Bazel will default the worker communication to use protobuf. -Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this -configuration allowed changing the `max_mem` parameter, a separate worker would -be spawned for each value used. This can lead to excessive memory consumption if -too many variations are used. +Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this configuration allowed changing the `max_mem` parameter, a separate worker would be spawned for each value used. This can lead to excessive memory consumption if too many variations are used. -Each worker can currently only process one request at a time. The experimental -[multiplex workers](/remote/multiplex) feature allows using multiple -threads, if the underlying tool is multithreaded and the wrapper is set up to -understand this. +Each worker can currently only process one request at a time. The experimental [multiplex workers](/remote/multiplex) feature allows using multiple threads, if the underlying tool is multithreaded and the wrapper is set up to understand this. -In -[this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), -you can see example worker wrappers written in Java as well as in Python. If you -are working in JavaScript or TypeScript, the -[@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) -and -[nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) -might be helpful. +In [this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), you can see example worker wrappers written in Java as well as in Python. If you are working in JavaScript or TypeScript, the [@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) and [nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) might be helpful. ## How do workers affect sandboxing? -Using the `worker` strategy by default does not run the action in a -[sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the -`--worker_sandboxing` flag to run all workers inside sandboxes, making sure each -execution of the tool only sees the input files it's supposed to have. The tool -may still leak information between requests internally, for instance through a -cache. Using `dynamic` strategy -[requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). +Using the `worker` strategy by default does not run the action in a [sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the `--worker_sandboxing` flag to run all workers inside sandboxes, making sure each execution of the tool only sees the input files it's supposed to have. The tool may still leak information between requests internally, for instance through a cache. Using `dynamic` strategy [requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). -To allow correct use of compiler caches with workers, a digest is passed along -with each input file. Thus the compiler or the wrapper can check if the input is -still valid without having to read the file. +To allow correct use of compiler caches with workers, a digest is passed along with each input file. Thus the compiler or the wrapper can check if the input is still valid without having to read the file. -Even when using the input digests to guard against unwanted caching, sandboxed -workers offer less strict sandboxing than a pure sandbox, because the tool may -keep other internal state that has been affected by previous requests. +Even when using the input digests to guard against unwanted caching, sandboxed workers offer less strict sandboxing than a pure sandbox, because the tool may keep other internal state that has been affected by previous requests. -Multiplex workers can only be sandboxed if the worker implementation support it, -and this sandboxing must be separately enabled with the -`--experimental_worker_multiplex_sandboxing` flag. See more details in -[the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). +Multiplex workers can only be sandboxed if the worker implementation support it, and this sandboxing must be separately enabled with the `--experimental_worker_multiplex_sandboxing` flag. See more details in [the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). ## Further reading For more information on persistent workers, see: -* [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) -* [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) -* [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) -* [Front End Development with Bazel: Angular/TypeScript and Persistent Workers - w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) -* [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) -* [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) +- [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) +- [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) +- [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) +- [Front End Development with Bazel: Angular/TypeScript and Persistent Workers w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) +- [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) +- [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) diff --git a/remote/rbe.mdx b/remote/rbe.mdx index 75d4a155..0d8bf255 100644 --- a/remote/rbe.mdx +++ b/remote/rbe.mdx @@ -2,32 +2,20 @@ title: 'Remote Execution Overview' --- +This page covers the benefits, requirements, and options for running Bazel with remote execution. - -This page covers the benefits, requirements, and options for running Bazel -with remote execution. - -By default, Bazel executes builds and tests on your local machine. Remote -execution of a Bazel build allows you to distribute build and test actions -across multiple machines, such as a datacenter. +By default, Bazel executes builds and tests on your local machine. Remote execution of a Bazel build allows you to distribute build and test actions across multiple machines, such as a datacenter. Remote execution provides the following benefits: -* Faster build and test execution through scaling of nodes available - for parallel actions -* A consistent execution environment for a development team -* Reuse of build outputs across a development team +- Faster build and test execution through scaling of nodes available for parallel actions +- A consistent execution environment for a development team +- Reuse of build outputs across a development team -Bazel uses an open-source -[gRPC protocol](https://github.com/bazelbuild/remote-apis) -to allow for remote execution and remote caching. +Bazel uses an open-source [gRPC protocol](https://github.com/bazelbuild/remote-apis) to allow for remote execution and remote caching. -For a list of commercially supported remote execution services as well as -self-service tools, see -[Remote Execution Services](https://www.bazel.build/remote-execution-services.html) +For a list of commercially supported remote execution services as well as self-service tools, see [Remote Execution Services](https://www.bazel.build/remote-execution-services.html) ## Requirements -Remote execution of Bazel builds imposes a set of mandatory configuration -constraints on the build. For more information, see -[Adapting Bazel Rules for Remote Execution](/remote/rules). +Remote execution of Bazel builds imposes a set of mandatory configuration constraints on the build. For more information, see [Adapting Bazel Rules for Remote Execution](/remote/rules). diff --git a/remote/rules.mdx b/remote/rules.mdx index 340ab02c..4ea4c9c3 100644 --- a/remote/rules.mdx +++ b/remote/rules.mdx @@ -2,179 +2,79 @@ title: 'Adapting Bazel Rules for Remote Execution' --- +This page is intended for Bazel users writing custom build and test rules who want to understand the requirements for Bazel rules in the context of remote execution. +Remote execution allows Bazel to execute actions on a separate platform, such as a datacenter. Bazel uses a [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) for its remote execution. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source project that aims to provide a distributed remote execution platform. -This page is intended for Bazel users writing custom build and test rules -who want to understand the requirements for Bazel rules in the context of -remote execution. +This page uses the following terminology when referring to different environment types or *platforms*: -Remote execution allows Bazel to execute actions on a separate platform, such as -a datacenter. Bazel uses a -[gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) -for its remote execution. You can try remote execution with -[bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), -an open-source project that aims to provide a distributed remote execution -platform. - -This page uses the following terminology when referring to different -environment types or *platforms*: - -* **Host platform** - where Bazel runs. -* **Execution platform** - where Bazel actions run. -* **Target platform** - where the build outputs (and some actions) run. +- **Host platform** - where Bazel runs. +- **Execution platform** - where Bazel actions run. +- **Target platform** - where the build outputs (and some actions) run. ## Overview -When configuring a Bazel build for remote execution, you must follow the -guidelines described in this page to ensure the build executes remotely -error-free. This is due to the nature of remote execution, namely: +When configuring a Bazel build for remote execution, you must follow the guidelines described in this page to ensure the build executes remotely error-free. This is due to the nature of remote execution, namely: -* **Isolated build actions.** Build tools do not retain state and dependencies - cannot leak between them. +- **Isolated build actions.** Build tools do not retain state and dependencies cannot leak between them. -* **Diverse execution environments.** Local build configuration is not always - suitable for remote execution environments. +- **Diverse execution environments.** Local build configuration is not always suitable for remote execution environments. -This page describes the issues that can arise when implementing custom Bazel -build and test rules for remote execution and how to avoid them. It covers the -following topics: +This page describes the issues that can arise when implementing custom Bazel build and test rules for remote execution and how to avoid them. It covers the following topics: -* [Invoking build tools through toolchain rules](#toolchain-rules) -* [Managing implicit dependencies](#manage-dependencies) -* [Managing platform-dependent binaries](#manage-binaries) -* [Managing configure-style WORKSPACE rules](#manage-workspace-rules) +- [Invoking build tools through toolchain rules](#toolchain-rules) +- [Managing implicit dependencies](#manage-dependencies) +- [Managing platform-dependent binaries](#manage-binaries) +- [Managing configure-style WORKSPACE rules](#manage-workspace-rules) ## Invoking build tools through toolchain rules -A Bazel toolchain rule is a configuration provider that tells a build rule what -build tools, such as compilers and linkers, to use and how to configure them -using parameters defined by the rule's creator. A toolchain rule allows build -and test rules to invoke build tools in a predictable, preconfigured manner -that's compatible with remote execution. For example, use a toolchain rule -instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local -variables that may not be set to equivalent values (or at all) in the remote -execution environment. - -Toolchain rules currently exist for Bazel build and test rules for -[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch -ain.bzl), -[Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), -and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), -and new toolchain rules are under way for other languages and tools such as -[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). -If a toolchain rule does not exist for the tool your rule uses, consider -[creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). +A Bazel toolchain rule is a configuration provider that tells a build rule what build tools, such as compilers and linkers, to use and how to configure them using parameters defined by the rule's creator. A toolchain rule allows build and test rules to invoke build tools in a predictable, preconfigured manner that's compatible with remote execution. For example, use a toolchain rule instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local variables that may not be set to equivalent values (or at all) in the remote execution environment. + +Toolchain rules currently exist for Bazel build and test rules for \[Scala]\([https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch) ain.bzl), [Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), and new toolchain rules are under way for other languages and tools such as [bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). If a toolchain rule does not exist for the tool your rule uses, consider [creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). ## Managing implicit dependencies -If a build tool can access dependencies across build actions, those actions will -fail when remotely executed because each remote build action is executed -separately from others. Some build tools retain state across build actions and -access dependencies that have not been explicitly included in the tool -invocation, which will cause remotely executed build actions to fail. - -For example, when Bazel instructs a stateful compiler to locally build _foo_, -the compiler retains references to foo's build outputs. When Bazel then -instructs the compiler to build _bar_, which depends on _foo_, without -explicitly stating that dependency in the BUILD file for inclusion in the -compiler invocation, the action executes successfully as long as the same -compiler instance executes for both actions (as is typical for local execution). -However, since in a remote execution scenario each build action executes a -separate compiler instance, compiler state and _bar_'s implicit dependency on -_foo_ will be lost and the build will fail. - -To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the -local Docker sandbox, which has the same restrictions for dependencies as remote -execution. Use the sandbox to prepare your build for remote execution by -identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) -for more information. +If a build tool can access dependencies across build actions, those actions will fail when remotely executed because each remote build action is executed separately from others. Some build tools retain state across build actions and access dependencies that have not been explicitly included in the tool invocation, which will cause remotely executed build actions to fail. + +For example, when Bazel instructs a stateful compiler to locally build *foo*, the compiler retains references to foo's build outputs. When Bazel then instructs the compiler to build *bar*, which depends on *foo*, without explicitly stating that dependency in the BUILD file for inclusion in the compiler invocation, the action executes successfully as long as the same compiler instance executes for both actions (as is typical for local execution). However, since in a remote execution scenario each build action executes a separate compiler instance, compiler state and *bar*'s implicit dependency on *foo* will be lost and the build will fail. + +To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the local Docker sandbox, which has the same restrictions for dependencies as remote execution. Use the sandbox to prepare your build for remote execution by identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) for more information. ## Managing platform-dependent binaries -Typically, a binary built on the host platform cannot safely execute on an -arbitrary remote execution platform due to potentially mismatched dependencies. -For example, the SingleJar binary supplied with Bazel targets the host platform. -However, for remote execution, SingleJar must be compiled as part of the process -of building your code so that it targets the remote execution platform. (See the -[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) +Typically, a binary built on the host platform cannot safely execute on an arbitrary remote execution platform due to potentially mismatched dependencies. For example, the SingleJar binary supplied with Bazel targets the host platform. However, for remote execution, SingleJar must be compiled as part of the process of building your code so that it targets the remote execution platform. (See the [target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) -Do not ship binaries of build tools required by your build with your source code -unless you are sure they will safely run in your execution platform. Instead, do -one of the following: +Do not ship binaries of build tools required by your build with your source code unless you are sure they will safely run in your execution platform. Instead, do one of the following: -* Ship or externally reference the source code for the tool so that it can be - built for the remote execution platform. +- Ship or externally reference the source code for the tool so that it can be built for the remote execution platform. -* Pre-install the tool into the remote execution environment (for example, a - toolchain container) if it's stable enough and use toolchain rules to run it - in your build. +- Pre-install the tool into the remote execution environment (for example, a toolchain container) if it's stable enough and use toolchain rules to run it in your build. ## Managing configure-style WORKSPACE rules -Bazel's `WORKSPACE` rules can be used for probing the host platform for tools -and libraries required by the build, which, for local builds, is also Bazel's -execution platform. If the build explicitly depends on local build tools and -artifacts, it will fail during remote execution if the remote execution platform -is not identical to the host platform. - -The following actions performed by `WORKSPACE` rules are not compatible with -remote execution: - -* **Building binaries.** Executing compilation actions in `WORKSPACE` rules - results in binaries that are incompatible with the remote execution platform - if different from the host platform. - -* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` - rules require that their dependencies be pre-installed on the host platform. - Such packages, built specifically for the host platform, will be - incompatible with the remote execution platform if different from the host - platform. - -* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries - installed on the host platform created via `WORKSPACE` rules will cause the - build to fail on the remote execution platform as Bazel will not be able to - locate them. Instead, create symlinks using standard build actions so that - the symlinked tools and libraries are accessible from Bazel's `runfiles` - tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) - to symlink target files outside of the external repo directory. - -* **Mutating the host platform.** Avoid creating files outside of the Bazel - `runfiles` tree, creating environment variables, and similar actions, as - they may behave unexpectedly on the remote execution platform. +Bazel's `WORKSPACE` rules can be used for probing the host platform for tools and libraries required by the build, which, for local builds, is also Bazel's execution platform. If the build explicitly depends on local build tools and artifacts, it will fail during remote execution if the remote execution platform is not identical to the host platform. + +The following actions performed by `WORKSPACE` rules are not compatible with remote execution: + +- **Building binaries.** Executing compilation actions in `WORKSPACE` rules results in binaries that are incompatible with the remote execution platform if different from the host platform. + +- **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` rules require that their dependencies be pre-installed on the host platform. Such packages, built specifically for the host platform, will be incompatible with the remote execution platform if different from the host platform. + +- **Symlinking to local tools or artifacts.** Symlinks to tools or libraries installed on the host platform created via `WORKSPACE` rules will cause the build to fail on the remote execution platform as Bazel will not be able to locate them. Instead, create symlinks using standard build actions so that the symlinked tools and libraries are accessible from Bazel's `runfiles` tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) to symlink target files outside of the external repo directory. + +- **Mutating the host platform.** Avoid creating files outside of the Bazel `runfiles` tree, creating environment variables, and similar actions, as they may behave unexpectedly on the remote execution platform. To help find potential non-hermetic behavior you can use [Workspace rules log](/remote/workspace). -If an external dependency executes specific operations dependent on the host -platform, you should split those operations between `WORKSPACE` and build -rules as follows: - -* **Platform inspection and dependency enumeration.** These operations are - safe to execute locally via `WORKSPACE` rules, which can check which - libraries are installed, download packages that must be built, and prepare - required artifacts for compilation. For remote execution, these rules must - also support using pre-checked artifacts to provide the information that - would normally be obtained during host platform inspection. Pre-checked - artifacts allow Bazel to describe dependencies as if they were local. Use - conditional statements or the `--override_repository` flag for this. - -* **Generating or compiling target-specific artifacts and platform mutation**. - These operations must be executed via regular build rules. Actions that - produce target-specific artifacts for external dependencies must execute - during the build. - -To more easily generate pre-checked artifacts for remote execution, you can use -`WORKSPACE` rules to emit generated files. You can run those rules on each new -execution environment, such as inside each toolchain container, and check the -outputs of your remote execution build in to your source repo to reference. - -For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) -and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), -the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). -For local execution, files produced by checking the host environment are used. -For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) -on an environment variable allows the rule to use files that are checked into -the repo. - -The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) -that can run both locally and remotely, and perform the necessary processing -that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). +If an external dependency executes specific operations dependent on the host platform, you should split those operations between `WORKSPACE` and build rules as follows: + +- **Platform inspection and dependency enumeration.** These operations are safe to execute locally via `WORKSPACE` rules, which can check which libraries are installed, download packages that must be built, and prepare required artifacts for compilation. For remote execution, these rules must also support using pre-checked artifacts to provide the information that would normally be obtained during host platform inspection. Pre-checked artifacts allow Bazel to describe dependencies as if they were local. Use conditional statements or the `--override_repository` flag for this. + +- **Generating or compiling target-specific artifacts and platform mutation**. These operations must be executed via regular build rules. Actions that produce target-specific artifacts for external dependencies must execute during the build. + +To more easily generate pre-checked artifacts for remote execution, you can use `WORKSPACE` rules to emit generated files. You can run those rules on each new execution environment, such as inside each toolchain container, and check the outputs of your remote execution build in to your source repo to reference. + +For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). For local execution, files produced by checking the host environment are used. For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) on an environment variable allows the rule to use files that are checked into the repo. + +The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) that can run both locally and remotely, and perform the necessary processing that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/remote/sandbox.mdx b/remote/sandbox.mdx index 8c3f9565..98485efc 100644 --- a/remote/sandbox.mdx +++ b/remote/sandbox.mdx @@ -2,223 +2,143 @@ title: 'Troubleshooting Bazel Remote Execution with Docker Sandbox' --- +Bazel builds that succeed locally may fail when executed remotely due to restrictions and requirements that do not affect local builds. The most common causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). +This page describes how to identify and resolve the most common issues that arise with remote execution using the Docker sandbox feature, which imposes restrictions upon the build equal to those of remote execution. This allows you to troubleshoot your build without the need for a remote execution service. -Bazel builds that succeed locally may fail when executed remotely due to -restrictions and requirements that do not affect local builds. The most common -causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). +The Docker sandbox feature mimics the restrictions of remote execution as follows: -This page describes how to identify and resolve the most common issues that -arise with remote execution using the Docker sandbox feature, which imposes -restrictions upon the build equal to those of remote execution. This allows you -to troubleshoot your build without the need for a remote execution service. +- **Build actions execute in toolchain containers.** You can use the same toolchain containers to run your build locally and remotely via a service supporting containerized remote execution. -The Docker sandbox feature mimics the restrictions of remote execution as -follows: +- **No extraneous data crosses the container boundary.** Only explicitly declared inputs and outputs enter and leave the container, and only after the associated build action successfully completes. -* **Build actions execute in toolchain containers.** You can use the same - toolchain containers to run your build locally and remotely via a service - supporting containerized remote execution. +- **Each action executes in a fresh container.** A new, unique container is created for each spawned build action. -* **No extraneous data crosses the container boundary.** Only explicitly - declared inputs and outputs enter and leave the container, and only after - the associated build action successfully completes. - -* **Each action executes in a fresh container.** A new, unique container is - created for each spawned build action. - -Note: Builds take noticeably more time to complete when the Docker sandbox -feature is enabled. This is normal. +Note: Builds take noticeably more time to complete when the Docker sandbox feature is enabled. This is normal. You can troubleshoot these issues using one of the following methods: -* **[Troubleshooting natively.](#troubleshooting-natively)** With this method, - Bazel and its build actions run natively on your local machine. The Docker - sandbox feature imposes restrictions upon the build equal to those of remote - execution. However, this method will not detect local tools, states, and - data leaking into your build, which will cause problems with remote execution. +- **[Troubleshooting natively.](#troubleshooting-natively)** With this method, Bazel and its build actions run natively on your local machine. The Docker sandbox feature imposes restrictions upon the build equal to those of remote execution. However, this method will not detect local tools, states, and data leaking into your build, which will cause problems with remote execution. -* **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** - With this method, Bazel and its build actions run inside a Docker container, - which allows you to detect tools, states, and data leaking from the local - machine into the build in addition to imposing restrictions - equal to those of remote execution. This method provides insight into your - build even if portions of the build are failing. This method is experimental - and not officially supported. +- **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** With this method, Bazel and its build actions run inside a Docker container, which allows you to detect tools, states, and data leaking from the local machine into the build in addition to imposing restrictions equal to those of remote execution. This method provides insight into your build even if portions of the build are failing. This method is experimental and not officially supported. ## Prerequisites Before you begin troubleshooting, do the following if you have not already done so: -* Install Docker and configure the permissions required to run it. -* Install Bazel 0.14.1 or later. Earlier versions do not support the Docker - sandbox feature. -* Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) - repo, pinned to the latest release version, to your build's `WORKSPACE` file - as described [here](https://releases.bazel.build/bazel-toolchains.html). -* Add flags to your `.bazelrc` file to enable the feature. Create the file in - the root directory of your Bazel project if it does not exist. Flags below - are a reference sample. Please see the latest - [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) - file in the bazel-toolchains repo and copy the values of the flags defined - there for config `docker-sandbox`. +- Install Docker and configure the permissions required to run it. +- Install Bazel 0.14.1 or later. Earlier versions do not support the Docker sandbox feature. +- Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) repo, pinned to the latest release version, to your build's `WORKSPACE` file as described [here](https://releases.bazel.build/bazel-toolchains.html). +- Add flags to your `.bazelrc` file to enable the feature. Create the file in the root directory of your Bazel project if it does not exist. Flags below are a reference sample. Please see the latest [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) file in the bazel-toolchains repo and copy the values of the flags defined there for config `docker-sandbox`. ``` # Docker Sandbox Mode -build:docker-sandbox --host_javabase=<...> -build:docker-sandbox --javabase=<...> -build:docker-sandbox --crosstool_top=<...> -build:docker-sandbox --experimental_docker_image=<...> +build:docker-sandbox --host_javabase=<...> +build:docker-sandbox --javabase=<...> +build:docker-sandbox --crosstool_top=<...> +build:docker-sandbox --experimental_docker_image=<...> build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker build:docker-sandbox --experimental_docker_verbose build:docker-sandbox --experimental_enable_docker_sandbox ``` -Note: The flags referenced in the `.bazelrc` file shown above are configured -to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) -container. +Note: The flags referenced in the `.bazelrc` file shown above are configured to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) container. If your rules require additional tools, do the following: -1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) - and [building](https://docs.docker.com/engine/reference/commandline/build/) - the image locally. - -2. Replace the value of the `--experimental_docker_image` flag above with the - name of your custom container image. +1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) and [building](https://docs.docker.com/engine/reference/commandline/build/) the image locally. +2. Replace the value of the `--experimental_docker_image` flag above with the name of your custom container image. ## Troubleshooting natively -This method executes Bazel and all of its build actions directly on the local -machine and is a reliable way to confirm whether your build will succeed when -executed remotely. +This method executes Bazel and all of its build actions directly on the local machine and is a reliable way to confirm whether your build will succeed when executed remotely. -However, with this method, locally installed tools, binaries, and data may leak -into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). -Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) -in addition to troubleshooting natively. +However, with this method, locally installed tools, binaries, and data may leak into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) in addition to troubleshooting natively. ### Step 1: Run the build -1. Add the `--config=docker-sandbox` flag to the Bazel command that executes - your build. For example: +1. Add the `--config=docker-sandbox` flag to the Bazel command that executes your build. For example: - ```posix-terminal - bazel --bazelrc=.bazelrc build --config=docker-sandbox target - ``` + ```posix-terminal + bazel --bazelrc=.bazelrc build --config=docker-sandbox <var>target</var> + ``` -2. Run the build and wait for it to complete. The build will run up to four - times slower than normal due to the Docker sandbox feature. +2. Run the build and wait for it to complete. The build will run up to four times slower than normal due to the Docker sandbox feature. You may encounter the following error: -```none {:.devsite-disable-click-to-copy} +```none ERROR: 'docker' is an invalid value for docker spawn strategy. ``` -If you do, run the build again with the `--experimental_docker_verbose` flag. -This flag enables verbose error messages. This error is typically caused by a -faulty Docker installation or lack of permissions to execute it under the -current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) -for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). +If you do, run the build again with the `--experimental_docker_verbose` flag. This flag enables verbose error messages. This error is typically caused by a faulty Docker installation or lack of permissions to execute it under the current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). ### Step 2: Resolve detected issues The following are the most commonly encountered issues and their workarounds. -* **A file, tool, binary, or resource referenced by the Bazel runfiles tree is - missing.**. Confirm that all dependencies of the affected targets have been - [explicitly declared](/concepts/dependencies). See - [Managing implicit dependencies](/remote/rules#manage-dependencies) - for more information. +- **A file, tool, binary, or resource referenced by the Bazel runfiles tree is missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/concepts/dependencies). See [Managing implicit dependencies](/remote/rules#manage-dependencies) for more information. -* **A file, tool, binary, or resource referenced by an absolute path or the `PATH` - variable is missing.** Confirm that all required tools are installed within - the toolchain container and use [toolchain rules](/extending/toolchains) to properly - declare dependencies pointing to the missing resource. See - [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) - for more information. +- **A file, tool, binary, or resource referenced by an absolute path or the `PATH` variable is missing.** Confirm that all required tools are installed within the toolchain container and use [toolchain rules](/extending/toolchains) to properly declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) for more information. -* **A binary execution fails.** One of the build rules is referencing a binary - incompatible with the execution environment (the Docker container). See - [Managing platform-dependent binaries](/remote/rules#manage-binaries) - for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) - for help. +- **A binary execution fails.** One of the build rules is referencing a binary incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote/rules#manage-binaries) for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. -* **A file from `@local-jdk` is missing or causing errors.** The Java binaries - on your local machine are leaking into the build while being incompatible with - it. Use [`java_toolchain`](/reference/be/java#java_toolchain) - in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. +- **A file from `@local-jdk` is missing or causing errors.** The Java binaries on your local machine are leaking into the build while being incompatible with it. Use [`java_toolchain`](/reference/be/java#java_toolchain) in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. -* **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +- **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. ## Troubleshooting in a Docker container -With this method, Bazel runs inside a host Docker container, and Bazel's build -actions execute inside individual toolchain containers spawned by the Docker -sandbox feature. The sandbox spawns a brand new toolchain container for each -build action and only one action executes in each toolchain container. +With this method, Bazel runs inside a host Docker container, and Bazel's build actions execute inside individual toolchain containers spawned by the Docker sandbox feature. The sandbox spawns a brand new toolchain container for each build action and only one action executes in each toolchain container. -This method provides more granular control of tools installed in the host -environment. By separating the execution of the build from the execution of its -build actions and keeping the installed tooling to a minimum, you can verify -whether your build has any dependencies on the local execution environment. +This method provides more granular control of tools installed in the host environment. By separating the execution of the build from the execution of its build actions and keeping the installed tooling to a minimum, you can verify whether your build has any dependencies on the local execution environment. ### Step 1: Build the container -Note: The commands below are tailored specifically for a `debian:stretch` base. -For other bases, modify them as necessary. +Note: The commands below are tailored specifically for a `debian:stretch` base. For other bases, modify them as necessary. -1. Create a `Dockerfile` that creates the Docker container and installs Bazel - with a minimal set of build tools: +1. Create a `Dockerfile` that creates the Docker container and installs Bazel with a minimal set of build tools: - ``` - FROM debian:stretch + ``` + FROM debian:stretch - RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim + RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim - RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - + RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - - RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" - RUN apt-get update && apt-get install -y docker-ce + RUN apt-get update && apt-get install -y docker-ce - RUN wget https://releases.bazel.build//release/bazel--installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh + RUN wget https://releases.bazel.build/<latest Bazel version>/release/bazel-<latest Bazel version>-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh - RUN ./bazel-installer.sh - ``` + RUN ./bazel-installer.sh + ``` -2. Build the container as `bazel_container`: +2. Build the container as `bazel_container`: - ```posix-terminal - docker build -t bazel_container - < Dockerfile - ``` + ```posix-terminal + docker build -t bazel_container - < Dockerfile + ``` ### Step 2: Start the container -Start the Docker container using the command shown below. In the command, -substitute the path to the source code on your host that you want to build. +Start the Docker container using the command shown below. In the command, substitute the path to the source code on your host that you want to build. ```posix-terminal docker run -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp:/tmp \ - -v {{ '' }}your source code directory{{ '' }}:/src \ + -v <var>your source code directory</var>:/src \ -w /src \ bazel_container \ /bin/bash ``` -This command runs the container as root, mapping the docker socket, and mounting -the `/tmp` directory. This allows Bazel to spawn other Docker containers and to -use directories under `/tmp` to share files with those containers. Your source -code is available at `/src` inside the container. +This command runs the container as root, mapping the docker socket, and mounting the `/tmp` directory. This allows Bazel to spawn other Docker containers and to use directories under `/tmp` to share files with those containers. Your source code is available at `/src` inside the container. -The command intentionally starts from a `debian:stretch` base container that -includes binaries incompatible with the `rbe-ubuntu16-04` container used as a -toolchain container. If binaries from the local environment are leaking into the -toolchain container, they will cause build errors. +The command intentionally starts from a `debian:stretch` base container that includes binaries incompatible with the `rbe-ubuntu16-04` container used as a toolchain container. If binaries from the local environment are leaking into the toolchain container, they will cause build errors. ### Step 3: Test the container @@ -232,28 +152,18 @@ bazel version ### Step 4: Run the build -Run the build as shown below. The output user is root so that it corresponds to -a directory that is accessible with the same absolute path from inside the host -container in which Bazel runs, from the toolchain containers spawned by the Docker -sandbox feature in which Bazel's build actions are running, and from the local -machine on which the host and action containers run. +Run the build as shown below. The output user is root so that it corresponds to a directory that is accessible with the same absolute path from inside the host container in which Bazel runs, from the toolchain containers spawned by the Docker sandbox feature in which Bazel's build actions are running, and from the local machine on which the host and action containers run. ```posix-terminal -bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox {{ '' }}target{{ '' }} +bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <var>target</var> ``` ### Step 5: Resolve detected issues You can resolve build failures as follows: -* If the build fails with an "out of disk space" error, you can increase this - limit by starting the host container with the flag `--memory=XX` where `XX` - is the allocated disk space in gigabytes. This is experimental and may - result in unpredictable behavior. +- If the build fails with an "out of disk space" error, you can increase this limit by starting the host container with the flag `--memory=XX` where `XX` is the allocated disk space in gigabytes. This is experimental and may result in unpredictable behavior. -* If the build fails during the analysis or loading phases, one or more of - your build rules declared in the WORKSPACE file are not compatible with - remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) - for possible causes and workarounds. +- If the build fails during the analysis or loading phases, one or more of your build rules declared in the WORKSPACE file are not compatible with remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) for possible causes and workarounds. -* If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). +- If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). diff --git a/remote/workspace.mdx b/remote/workspace.mdx index ae0aea50..726f8bd3 100644 --- a/remote/workspace.mdx +++ b/remote/workspace.mdx @@ -2,126 +2,78 @@ title: 'Finding Non-Hermetic Behavior in WORKSPACE Rules' --- - - In the following, a host machine is the machine where Bazel runs. -When using remote execution, the actual build and/or test steps are not -happening on the host machine, but are instead sent off to the remote execution -system. However, the steps involved in resolving workspace rules are happening -on the host machine. If your workspace rules access information about the -host machine for use during execution, your build is likely to break due to -incompatibilities between the environments. - -As part of [adapting Bazel rules for remote -execution](/remote/rules), you need to find such workspace rules -and fix them. This page describes how to find potentially problematic workspace -rules using the workspace log. +When using remote execution, the actual build and/or test steps are not happening on the host machine, but are instead sent off to the remote execution system. However, the steps involved in resolving workspace rules are happening on the host machine. If your workspace rules access information about the host machine for use during execution, your build is likely to break due to incompatibilities between the environments. +As part of [adapting Bazel rules for remote execution](/remote/rules), you need to find such workspace rules and fix them. This page describes how to find potentially problematic workspace rules using the workspace log. ## Finding non-hermetic rules -[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to -external workspaces, but they are rich enough to allow arbitrary processing to -happen in the process. All related commands are happening locally and can be a -potential source of non-hermeticity. Usually non-hermetic behavior is -introduced through -[`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting -with the host machine. +[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. All related commands are happening locally and can be a potential source of non-hermeticity. Usually non-hermetic behavior is introduced through [`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting with the host machine. -Starting with Bazel 0.18, you can get a log of some potentially non-hermetic -actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to -your Bazel command. Here `[PATH]` is a filename under which the log will be -created. +Starting with Bazel 0.18, you can get a log of some potentially non-hermetic actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to your Bazel command. Here `[PATH]` is a filename under which the log will be created. Things to note: -* the log captures the events as they are executed. If some steps are - cached, they will not show up in the log, so to get a full result, don't - forget to run `bazel clean --expunge` beforehand. +- the log captures the events as they are executed. If some steps are cached, they will not show up in the log, so to get a full result, don't forget to run `bazel clean --expunge` beforehand. -* Sometimes functions might be re-executed, in which case the related - events will show up in the log multiple times. +- Sometimes functions might be re-executed, in which case the related events will show up in the log multiple times. -* Workspace rules currently only log Starlark events. +- Workspace rules currently only log Starlark events. - Note: These particular rules do not cause hermiticity concerns as long - as a hash is specified. + Note: These particular rules do not cause hermiticity concerns as long as a hash is specified. To find what was executed during workspace initialization: -1. Run `bazel clean --expunge`. This command will clean your local cache and - any cached repositories, ensuring that all initialization will be re-run. +1. Run `bazel clean --expunge`. This command will clean your local cache and any cached repositories, ensuring that all initialization will be re-run. -2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your - Bazel command and run the build. +2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your Bazel command and run the build. - This produces a binary proto file listing messages of type - [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) + This produces a binary proto file listing messages of type [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -3. Download the Bazel source code and navigate to the Bazel folder by using - the command below. You need the source code to be able to parse the - workspace log with the - [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). +3. Download the Bazel source code and navigate to the Bazel folder by using the command below. You need the source code to be able to parse the workspace log with the [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). - ```posix-terminal - git clone https://github.com/bazelbuild/bazel.git + ```posix-terminal + git clone https://github.com/bazelbuild/bazel.git - cd bazel - ``` + cd bazel + ``` -4. In the Bazel source code repo, convert the whole workspace log to text. +4. In the Bazel source code repo, convert the whole workspace log to text. - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt + ``` -5. The output may be quite verbose and include output from built in Bazel - rules. +5. The output may be quite verbose and include output from built in Bazel rules. - To exclude specific rules from the output, use `--exclude_rule` option. - For example: + To exclude specific rules from the output, use `--exclude_rule` option. For example: - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ - --exclude_rule "//external:local_config_cc" \ - --exclude_rule "//external:dep" > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ + --exclude_rule "//external:local_config_cc" \ + --exclude_rule "//external:dep" > /tmp/workspacelog.txt + ``` -5. Open `/tmp/workspacelog.txt` and check for unsafe operations. +6. Open `/tmp/workspacelog.txt` and check for unsafe operations. -The log consists of -[WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -messages outlining certain potentially non-hermetic actions performed on a -[`repository_ctx`](/rules/lib/builtins/repository_ctx). +The log consists of [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) messages outlining certain potentially non-hermetic actions performed on a [`repository_ctx`](/rules/lib/builtins/repository_ctx). The actions that have been highlighted as potentially non-hermetic are as follows: -* `execute`: executes an arbitrary command on the host environment. Check if - these may introduce any dependencies on the host environment. +- `execute`: executes an arbitrary command on the host environment. Check if these may introduce any dependencies on the host environment. -* `download`, `download_and_extract`: to ensure hermetic builds, make sure - that sha256 is specified +- `download`, `download_and_extract`: to ensure hermetic builds, make sure that sha256 is specified -* `file`, `template`: this is not non-hermetic in itself, but may be a mechanism - for introducing dependencies on the host environment into the repository. - Ensure that you understand where the input comes from, and that it does not - depend on the host environment. +- `file`, `template`: this is not non-hermetic in itself, but may be a mechanism for introducing dependencies on the host environment into the repository. Ensure that you understand where the input comes from, and that it does not depend on the host environment. -* `os`: this is not non-hermetic in itself, but an easy way to get dependencies - on the host environment. A hermetic build would generally not call this. - In evaluating whether your usage is hermetic, keep in mind that this is - running on the host and not on the workers. Getting environment specifics - from the host is generally not a good idea for remote builds. +- `os`: this is not non-hermetic in itself, but an easy way to get dependencies on the host environment. A hermetic build would generally not call this. In evaluating whether your usage is hermetic, keep in mind that this is running on the host and not on the workers. Getting environment specifics from the host is generally not a good idea for remote builds. -* `symlink`: this is normally safe, but look for red flags. Any symlinks to - outside the repository or to an absolute path would cause problems on the - remote worker. If the symlink is created based on host machine properties - it would probably be problematic as well. +- `symlink`: this is normally safe, but look for red flags. Any symlinks to outside the repository or to an absolute path would cause problems on the remote worker. If the symlink is created based on host machine properties it would probably be problematic as well. -* `which`: checking for programs installed on the host is usually problematic - since the workers may have different configurations. +- `which`: checking for programs installed on the host is usually problematic since the workers may have different configurations. diff --git a/rules/bzl-style.mdx b/rules/bzl-style.mdx index 65f589f4..8d85a7b5 100644 --- a/rules/bzl-style.mdx +++ b/rules/bzl-style.mdx @@ -2,92 +2,49 @@ title: '.bzl style guide' --- +This page covers basic style guidelines for Starlark and also includes information on macros and rules. +[Starlark](/rules/language) is a language that defines how software is built, and as such it is both a programming and a configuration language. -This page covers basic style guidelines for Starlark and also includes -information on macros and rules. +You will use Starlark to write `BUILD` files, macros, and build rules. Macros and rules are essentially meta-languages - they define how `BUILD` files are written. `BUILD` files are intended to be simple and repetitive. -[Starlark](/rules/language) is a -language that defines how software is built, and as such it is both a -programming and a configuration language. +All software is read more often than it is written. This is especially true for Starlark, as engineers read `BUILD` files to understand dependencies of their targets and details of their builds. This reading will often happen in passing, in a hurry, or in parallel to accomplishing some other task. Consequently, simplicity and readability are very important so that users can parse and comprehend `BUILD` files quickly. -You will use Starlark to write `BUILD` files, macros, and build rules. Macros and -rules are essentially meta-languages - they define how `BUILD` files are written. -`BUILD` files are intended to be simple and repetitive. +When a user opens a `BUILD` file, they quickly want to know the list of targets in the file; or review the list of sources of that C++ library; or remove a dependency from that Java binary. Each time you add a layer of abstraction, you make it harder for a user to do these tasks. -All software is read more often than it is written. This is especially true for -Starlark, as engineers read `BUILD` files to understand dependencies of their -targets and details of their builds. This reading will often happen in passing, -in a hurry, or in parallel to accomplishing some other task. Consequently, -simplicity and readability are very important so that users can parse and -comprehend `BUILD` files quickly. +`BUILD` files are also analyzed and updated by many different tools. Tools may not be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` files simple will allow you to get better tooling. As a code base grows, it becomes more and more frequent to do changes across many `BUILD` files in order to update a library or do a cleanup. -When a user opens a `BUILD` file, they quickly want to know the list of targets in -the file; or review the list of sources of that C++ library; or remove a -dependency from that Java binary. Each time you add a layer of abstraction, you -make it harder for a user to do these tasks. - -`BUILD` files are also analyzed and updated by many different tools. Tools may not -be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` -files simple will allow you to get better tooling. As a code base grows, it -becomes more and more frequent to do changes across many `BUILD` files in order to -update a library or do a cleanup. - -Important: Do not create a variable or macro just to avoid some amount of -repetition in `BUILD` files. Your `BUILD` file should be easily readable both by -developers and tools. The -[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't -really apply here. +Important: Do not create a variable or macro just to avoid some amount of repetition in `BUILD` files. Your `BUILD` file should be easily readable both by developers and tools. The [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't really apply here. ## General advice -* Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) - as a formatter and linter. -* Follow [testing guidelines](/rules/testing). +- Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) as a formatter and linter. +- Follow [testing guidelines](/rules/testing). ## Style ### Python style -When in doubt, follow the -[PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. -In particular, use four rather than two spaces for indentation to follow the -Python convention. - -Since -[Starlark is not Python](/rules/language#differences-with-python), -some aspects of Python style do not apply. For example, PEP 8 advises that -comparisons to singletons be done with `is`, which is not an operator in -Starlark. +When in doubt, follow the [PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. In particular, use four rather than two spaces for indentation to follow the Python convention. +Since [Starlark is not Python](/rules/language#differences-with-python), some aspects of Python style do not apply. For example, PEP 8 advises that comparisons to singletons be done with `is`, which is not an operator in Starlark. ### Docstring -Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). -Use a docstring at the top of each `.bzl` file, and a docstring for each public -function. +Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Use a docstring at the top of each `.bzl` file, and a docstring for each public function. ### Document rules and aspects -Rules and aspects, along with their attributes, as well as providers and their -fields, should be documented using the `doc` argument. +Rules and aspects, along with their attributes, as well as providers and their fields, should be documented using the `doc` argument. ### Naming convention -* Variables and function names use lowercase with words separated by - underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. -* Top-level private values start with one underscore. Bazel enforces that - private values cannot be used from other files. Local variables should not - use the underscore prefix. +- Variables and function names use lowercase with words separated by underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. +- Top-level private values start with one underscore. Bazel enforces that private values cannot be used from other files. Local variables should not use the underscore prefix. ### Line length -As in `BUILD` files, there is no strict line length limit as labels can be long. -When possible, try to use at most 79 characters per line (following Python's -style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline -should not be enforced strictly: editors should display more than 80 columns, -automated changes will frequently introduce longer lines, and humans shouldn't -spend time splitting lines that are already readable. +As in `BUILD` files, there is no strict line length limit as labels can be long. When possible, try to use at most 79 characters per line (following Python's style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline should not be enforced strictly: editors should display more than 80 columns, automated changes will frequently introduce longer lines, and humans shouldn't spend time splitting lines that are already readable. ### Keyword arguments @@ -105,67 +62,41 @@ def fct(name, srcs): ### Boolean values -Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values -(such as when using a boolean attribute in a rule). +Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values (such as when using a boolean attribute in a rule). ### Use print only for debugging -Do not use the `print()` function in production code; it is only intended for -debugging, and will spam all direct and indirect users of your `.bzl` file. The -only exception is that you may submit code that uses `print()` if it is disabled -by default and can only be enabled by editing the source -- for example, if all -uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to -`False`. Be mindful of whether these statements are useful enough to justify -their impact on readability. +Do not use the `print()` function in production code; it is only intended for debugging, and will spam all direct and indirect users of your `.bzl` file. The only exception is that you may submit code that uses `print()` if it is disabled by default and can only be enabled by editing the source -- for example, if all uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to `False`. Be mindful of whether these statements are useful enough to justify their impact on readability. ## Macros -A macro is a function which instantiates one or more rules during the loading -phase. In general, use rules whenever possible instead of macros. The build -graph seen by the user is not the same as the one used by Bazel during the -build - macros are expanded *before Bazel does any build graph analysis.* - -Because of this, when something goes wrong, the user will need to understand -your macro's implementation to troubleshoot build problems. Additionally, `bazel -query` results can be hard to interpret because targets shown in the results -come from macro expansion. Finally, aspects are not aware of macros, so tooling -depending on aspects (IDEs and others) might fail. - -A safe use for macros is for defining additional targets intended to be -referenced directly at the Bazel CLI or in BUILD files: In that case, only the -*end users* of those targets need to know about them, and any build problems -introduced by macros are never far from their usage. - -For macros that define generated targets (implementation details of the macro -which are not supposed to be referred to at the CLI or depended on by targets -not instantiated by that macro), follow these best practices: - -* A macro should take a `name` argument and define a target with that name. - That target becomes that macro's *main target*. -* Generated targets, that is all other targets defined by a macro, should: - * Have their names prefixed by ``. For example, using - `name = '%s_bar' % (name)`. - * Have restricted visibility (`//visibility:private`), and - * Have a `manual` tag to avoid expansion in wildcard targets (`:all`, - `...`, `:*`, etc). -* The `name` should only be used to derive names of targets defined by the - macro, and not for anything else. For example, don't use the name to derive - a dependency or input file that is not generated by the macro itself. -* All the targets created in the macro should be coupled in some way to the - main target. -* Conventionally, `name` should be the first argument when defining a macro. -* Keep the parameter names in the macro consistent. If a parameter is passed - as an attribute value to the main target, keep its name the same. If a macro - parameter serves the same purpose as a common rule attribute, such as - `deps`, name as you would the attribute (see below). -* When calling a macro, use only keyword arguments. This is consistent with - rules, and greatly improves readability. - -Engineers often write macros when the Starlark API of relevant rules is -insufficient for their specific use case, regardless of whether the rule is -defined within Bazel in native code, or in Starlark. If you're facing this -problem, ask the rule author if they can extend the API to accomplish your -goals. +A macro is a function which instantiates one or more rules during the loading phase. In general, use rules whenever possible instead of macros. The build graph seen by the user is not the same as the one used by Bazel during the build - macros are expanded *before Bazel does any build graph analysis.* + +Because of this, when something goes wrong, the user will need to understand your macro's implementation to troubleshoot build problems. Additionally, `bazel query` results can be hard to interpret because targets shown in the results come from macro expansion. Finally, aspects are not aware of macros, so tooling depending on aspects (IDEs and others) might fail. + +A safe use for macros is for defining additional targets intended to be referenced directly at the Bazel CLI or in BUILD files: In that case, only the *end users* of those targets need to know about them, and any build problems introduced by macros are never far from their usage. + +For macros that define generated targets (implementation details of the macro which are not supposed to be referred to at the CLI or depended on by targets not instantiated by that macro), follow these best practices: + +- A macro should take a `name` argument and define a target with that name. That target becomes that macro's *main target*. + +- Generated targets, that is all other targets defined by a macro, should: + + - Have their names prefixed by `<name>`. For example, using `name = '%s_bar' % (name)`. + - Have restricted visibility (`//visibility:private`), and + - Have a `manual` tag to avoid expansion in wildcard targets (`:all`, `...`, `:*`, etc). + +- The `name` should only be used to derive names of targets defined by the macro, and not for anything else. For example, don't use the name to derive a dependency or input file that is not generated by the macro itself. + +- All the targets created in the macro should be coupled in some way to the main target. + +- Conventionally, `name` should be the first argument when defining a macro. + +- Keep the parameter names in the macro consistent. If a parameter is passed as an attribute value to the main target, keep its name the same. If a macro parameter serves the same purpose as a common rule attribute, such as `deps`, name as you would the attribute (see below). + +- When calling a macro, use only keyword arguments. This is consistent with rules, and greatly improves readability. + +Engineers often write macros when the Starlark API of relevant rules is insufficient for their specific use case, regardless of whether the rule is defined within Bazel in native code, or in Starlark. If you're facing this problem, ask the rule author if they can extend the API to accomplish your goals. As a rule of thumb, the more macros resemble the rules, the better. @@ -173,40 +104,28 @@ See also [macros](/extending/macros#conventions). ## Rules -* Rules, aspects, and their attributes should use lower_case names ("snake - case"). -* Rule names are nouns that describe the main kind of artifact produced by the - rule, from the point of view of its dependencies (or for leaf rules, the - user). This is not necessarily a file suffix. For instance, a rule that - produces C++ artifacts meant to be used as Python extensions might be called - `py_extension`. For most languages, typical rules include: - * `*_library` - a compilation unit or "module". - * `*_binary` - a target producing an executable or a deployment unit. - * `*_test` - a test target. This can include multiple tests. Expect all - tests in a `*_test` target to be variations on the same theme, for - example, testing a single library. - * `*_import`: a target encapsulating a pre-compiled artifact, such as a - `.jar`, or a `.dll` that is used during compilation. -* Use consistent names and types for attributes. Some generally applicable - attributes include: - * `srcs`: `label_list`, allowing files: source files, typically - human-authored. - * `deps`: `label_list`, typically *not* allowing files: compilation - dependencies. - * `data`: `label_list`, allowing files: data files, such as test data etc. - * `runtime_deps`: `label_list`: runtime dependencies that are not needed - for compilation. -* For any attributes with non-obvious behavior (for example, string templates - with special substitutions, or tools that are invoked with specific - requirements), provide documentation using the `doc` keyword argument to the - attribute's declaration (`attr.label_list()` or similar). -* Rule implementation functions should almost always be private functions - (named with a leading underscore). A common style is to give the - implementation function for `myrule` the name `_myrule_impl`. -* Pass information between your rules using a well-defined - [provider](/extending/rules#providers) interface. Declare and document provider - fields. -* Design your rule with extensibility in mind. Consider that other rules might - want to interact with your rule, access your providers, and reuse the - actions you create. -* Follow [performance guidelines](/rules/performance) in your rules. +- Rules, aspects, and their attributes should use lower\_case names ("snake case"). + +- Rule names are nouns that describe the main kind of artifact produced by the rule, from the point of view of its dependencies (or for leaf rules, the user). This is not necessarily a file suffix. For instance, a rule that produces C++ artifacts meant to be used as Python extensions might be called `py_extension`. For most languages, typical rules include: + + - `*_library` - a compilation unit or "module". + - `*_binary` - a target producing an executable or a deployment unit. + - `*_test` - a test target. This can include multiple tests. Expect all tests in a `*_test` target to be variations on the same theme, for example, testing a single library. + - `*_import`: a target encapsulating a pre-compiled artifact, such as a `.jar`, or a `.dll` that is used during compilation. + +- Use consistent names and types for attributes. Some generally applicable attributes include: + + - `srcs`: `label_list`, allowing files: source files, typically human-authored. + - `deps`: `label_list`, typically *not* allowing files: compilation dependencies. + - `data`: `label_list`, allowing files: data files, such as test data etc. + - `runtime_deps`: `label_list`: runtime dependencies that are not needed for compilation. + +- For any attributes with non-obvious behavior (for example, string templates with special substitutions, or tools that are invoked with specific requirements), provide documentation using the `doc` keyword argument to the attribute's declaration (`attr.label_list()` or similar). + +- Rule implementation functions should almost always be private functions (named with a leading underscore). A common style is to give the implementation function for `myrule` the name `_myrule_impl`. + +- Pass information between your rules using a well-defined [provider](/extending/rules#providers) interface. Declare and document provider fields. + +- Design your rule with extensibility in mind. Consider that other rules might want to interact with your rule, access your providers, and reuse the actions you create. + +- Follow [performance guidelines](/rules/performance) in your rules. diff --git a/rules/challenges.mdx b/rules/challenges.mdx index 10ff7371..e31c5d45 100644 --- a/rules/challenges.mdx +++ b/rules/challenges.mdx @@ -2,222 +2,88 @@ title: 'Challenges of Writing Rules' --- - - -This page gives a high-level overview of the specific issues and challenges -of writing efficient Bazel rules. +This page gives a high-level overview of the specific issues and challenges of writing efficient Bazel rules. ## Summary Requirements -* Assumption: Aim for Correctness, Throughput, Ease of Use & Latency -* Assumption: Large Scale Repositories -* Assumption: BUILD-like Description Language -* Historic: Hard Separation between Loading, Analysis, and Execution is - Outdated, but still affects the API -* Intrinsic: Remote Execution and Caching are Hard -* Intrinsic: Using Change Information for Correct and Fast Incremental Builds - requires Unusual Coding Patterns -* Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard +- Assumption: Aim for Correctness, Throughput, Ease of Use & Latency +- Assumption: Large Scale Repositories +- Assumption: BUILD-like Description Language +- Historic: Hard Separation between Loading, Analysis, and Execution is Outdated, but still affects the API +- Intrinsic: Remote Execution and Caching are Hard +- Intrinsic: Using Change Information for Correct and Fast Incremental Builds requires Unusual Coding Patterns +- Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard ## Assumptions -Here are some assumptions made about the build system, such as need for -correctness, ease of use, throughput, and large scale repositories. The -following sections address these assumptions and offer guidelines to ensure -rules are written in an effective manner. +Here are some assumptions made about the build system, such as need for correctness, ease of use, throughput, and large scale repositories. The following sections address these assumptions and offer guidelines to ensure rules are written in an effective manner. ### Aim for correctness, throughput, ease of use & latency -We assume that the build system needs to be first and foremost correct with -respect to incremental builds. For a given source tree, the output of the -same build should always be the same, regardless of what the output tree looks -like. In the first approximation, this means Bazel needs to know every single -input that goes into a given build step, such that it can rerun that step if any -of the inputs change. There are limits to how correct Bazel can get, as it leaks -some information such as date / time of the build, and ignores certain types of -changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) -helps ensure correctness by preventing reads to undeclared input files. Besides -the intrinsic limits of the system, there are a few known correctness issues, -most of which are related to Fileset or the C++ rules, which are both hard -problems. We have long-term efforts to fix these. - -The second goal of the build system is to have high throughput; we are -permanently pushing the boundaries of what can be done within the current -machine allocation for a remote execution service. If the remote execution -service gets overloaded, nobody can get work done. - -Ease of use comes next. Of multiple correct approaches with the same (or -similar) footprint of the remote execution service, we choose the one that is -easier to use. - -Latency denotes the time it takes from starting a build to getting the intended -result, whether that is a test log from a passing or failing test, or an error -message that a `BUILD` file has a typo. - -Note that these goals often overlap; latency is as much a function of throughput -of the remote execution service as is correctness relevant for ease of use. +We assume that the build system needs to be first and foremost correct with respect to incremental builds. For a given source tree, the output of the same build should always be the same, regardless of what the output tree looks like. In the first approximation, this means Bazel needs to know every single input that goes into a given build step, such that it can rerun that step if any of the inputs change. There are limits to how correct Bazel can get, as it leaks some information such as date / time of the build, and ignores certain types of changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) helps ensure correctness by preventing reads to undeclared input files. Besides the intrinsic limits of the system, there are a few known correctness issues, most of which are related to Fileset or the C++ rules, which are both hard problems. We have long-term efforts to fix these. + +The second goal of the build system is to have high throughput; we are permanently pushing the boundaries of what can be done within the current machine allocation for a remote execution service. If the remote execution service gets overloaded, nobody can get work done. + +Ease of use comes next. Of multiple correct approaches with the same (or similar) footprint of the remote execution service, we choose the one that is easier to use. + +Latency denotes the time it takes from starting a build to getting the intended result, whether that is a test log from a passing or failing test, or an error message that a `BUILD` file has a typo. + +Note that these goals often overlap; latency is as much a function of throughput of the remote execution service as is correctness relevant for ease of use. ### Large scale repositories -The build system needs to operate at the scale of large repositories where large -scale means that it does not fit on a single hard drive, so it is impossible to -do a full checkout on virtually all developer machines. A medium-sized build -will need to read and parse tens of thousands of `BUILD` files, and evaluate -hundreds of thousands of globs. While it is theoretically possible to read all -`BUILD` files on a single machine, we have not yet been able to do so within a -reasonable amount of time and memory. As such, it is critical that `BUILD` files -can be loaded and parsed independently. +The build system needs to operate at the scale of large repositories where large scale means that it does not fit on a single hard drive, so it is impossible to do a full checkout on virtually all developer machines. A medium-sized build will need to read and parse tens of thousands of `BUILD` files, and evaluate hundreds of thousands of globs. While it is theoretically possible to read all `BUILD` files on a single machine, we have not yet been able to do so within a reasonable amount of time and memory. As such, it is critical that `BUILD` files can be loaded and parsed independently. ### BUILD-like description language -In this context, we assume a configuration language that is -roughly similar to `BUILD` files in declaration of library and binary rules -and their interdependencies. `BUILD` files can be read and parsed independently, -and we avoid even looking at source files whenever we can (except for -existence). +In this context, we assume a configuration language that is roughly similar to `BUILD` files in declaration of library and binary rules and their interdependencies. `BUILD` files can be read and parsed independently, and we avoid even looking at source files whenever we can (except for existence). ## Historic -There are differences between Bazel versions that cause challenges and some -of these are outlined in the following sections. +There are differences between Bazel versions that cause challenges and some of these are outlined in the following sections. ### Hard separation between loading, analysis, and execution is outdated but still affects the API -Technically, it is sufficient for a rule to know the input and output files of -an action just before the action is sent to remote execution. However, the -original Bazel code base had a strict separation of loading packages, then -analyzing rules using a configuration (command-line flags, essentially), and -only then running any actions. This distinction is still part of the rules API -today, even though the core of Bazel no longer requires it (more details below). - -That means that the rules API requires a declarative description of the rule -interface (what attributes it has, types of attributes). There are some -exceptions where the API allows custom code to run during the loading phase to -compute implicit names of output files and implicit values of attributes. For -example, a java_library rule named 'foo' implicitly generates an output named -'libfoo.jar', which can be referenced from other rules in the build graph. - -Furthermore, the analysis of a rule cannot read any source files or inspect the -output of an action; instead, it needs to generate a partial directed bipartite -graph of build steps and output file names that is only determined from the rule -itself and its dependencies. +Technically, it is sufficient for a rule to know the input and output files of an action just before the action is sent to remote execution. However, the original Bazel code base had a strict separation of loading packages, then analyzing rules using a configuration (command-line flags, essentially), and only then running any actions. This distinction is still part of the rules API today, even though the core of Bazel no longer requires it (more details below). + +That means that the rules API requires a declarative description of the rule interface (what attributes it has, types of attributes). There are some exceptions where the API allows custom code to run during the loading phase to compute implicit names of output files and implicit values of attributes. For example, a java\_library rule named 'foo' implicitly generates an output named 'libfoo.jar', which can be referenced from other rules in the build graph. + +Furthermore, the analysis of a rule cannot read any source files or inspect the output of an action; instead, it needs to generate a partial directed bipartite graph of build steps and output file names that is only determined from the rule itself and its dependencies. ## Intrinsic -There are some intrinsic properties that make writing rules challenging and -some of the most common ones are described in the following sections. +There are some intrinsic properties that make writing rules challenging and some of the most common ones are described in the following sections. ### Remote execution and caching are hard -Remote execution and caching improve build times in large repositories by -roughly two orders of magnitude compared to running the build on a single -machine. However, the scale at which it needs to perform is staggering: Google's -remote execution service is designed to handle a huge number of requests per -second, and the protocol carefully avoids unnecessary roundtrips as well as -unnecessary work on the service side. +Remote execution and caching improve build times in large repositories by roughly two orders of magnitude compared to running the build on a single machine. However, the scale at which it needs to perform is staggering: Google's remote execution service is designed to handle a huge number of requests per second, and the protocol carefully avoids unnecessary roundtrips as well as unnecessary work on the service side. -At this time, the protocol requires that the build system knows all inputs to a -given action ahead of time; the build system then computes a unique action -fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, -the scheduler replies with the digests of the output files; the files itself are -addressed by digest later on. However, this imposes restrictions on the Bazel -rules, which need to declare all input files ahead of time. +At this time, the protocol requires that the build system knows all inputs to a given action ahead of time; the build system then computes a unique action fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, the scheduler replies with the digests of the output files; the files itself are addressed by digest later on. However, this imposes restrictions on the Bazel rules, which need to declare all input files ahead of time. ### Using change information for correct and fast incremental builds requires unusual coding patterns -Above, we argued that in order to be correct, Bazel needs to know all the input -files that go into a build step in order to detect whether that build step is -still up-to-date. The same is true for package loading and rule analysis, and we -have designed [Skyframe](/reference/skyframe) to handle this -in general. Skyframe is a graph library and evaluation framework that takes a -goal node (such as 'build //foo with these options'), and breaks it down into -its constituent parts, which are then evaluated and combined to yield this -result. As part of this process, Skyframe reads packages, analyzes rules, and -executes actions. - -At each node, Skyframe tracks exactly which nodes any given node used to compute -its own output, all the way from the goal node down to the input files (which -are also Skyframe nodes). Having this graph explicitly represented in memory -allows the build system to identify exactly which nodes are affected by a given -change to an input file (including creation or deletion of an input file), doing -the minimal amount of work to restore the output tree to its intended state. - -As part of this, each node performs a dependency discovery process. Each -node can declare dependencies, and then use the contents of those dependencies -to declare even further dependencies. In principle, this maps well to a -thread-per-node model. However, medium-sized builds contain hundreds of -thousands of Skyframe nodes, which isn't easily possible with current Java -technology (and for historical reasons, we're currently tied to using Java, so -no lightweight threads and no continuations). - -Instead, Bazel uses a fixed-size thread pool. However, that means that if a node -declares a dependency that isn't available yet, we may have to abort that -evaluation and restart it (possibly in another thread), when the dependency is -available. This, in turn, means that nodes should not do this excessively; a -node that declares N dependencies serially can potentially be restarted N times, -costing O(N^2) time. Instead, we aim for up-front bulk declaration of -dependencies, which sometimes requires reorganizing the code, or even splitting -a node into multiple nodes to limit the number of restarts. - -Note that this technology isn't currently available in the rules API; instead, -the rules API is still defined using the legacy concepts of loading, analysis, -and execution phases. However, a fundamental restriction is that all accesses to -other nodes have to go through the framework so that it can track the -corresponding dependencies. Regardless of the language in which the build system -is implemented or in which the rules are written (they don't have to be the -same), rule authors must not use standard libraries or patterns that bypass -Skyframe. For Java, that means avoiding java.io.File as well as any form of -reflection, and any library that does either. Libraries that support dependency -injection of these low-level interfaces still need to be setup correctly for -Skyframe. - -This strongly suggests to avoid exposing rule authors to a full language runtime -in the first place. The danger of accidental use of such APIs is just too big - -several Bazel bugs in the past were caused by rules using unsafe APIs, even -though the rules were written by the Bazel team or other domain experts. +Above, we argued that in order to be correct, Bazel needs to know all the input files that go into a build step in order to detect whether that build step is still up-to-date. The same is true for package loading and rule analysis, and we have designed [Skyframe](/reference/skyframe) to handle this in general. Skyframe is a graph library and evaluation framework that takes a goal node (such as 'build //foo with these options'), and breaks it down into its constituent parts, which are then evaluated and combined to yield this result. As part of this process, Skyframe reads packages, analyzes rules, and executes actions. + +At each node, Skyframe tracks exactly which nodes any given node used to compute its own output, all the way from the goal node down to the input files (which are also Skyframe nodes). Having this graph explicitly represented in memory allows the build system to identify exactly which nodes are affected by a given change to an input file (including creation or deletion of an input file), doing the minimal amount of work to restore the output tree to its intended state. + +As part of this, each node performs a dependency discovery process. Each node can declare dependencies, and then use the contents of those dependencies to declare even further dependencies. In principle, this maps well to a thread-per-node model. However, medium-sized builds contain hundreds of thousands of Skyframe nodes, which isn't easily possible with current Java technology (and for historical reasons, we're currently tied to using Java, so no lightweight threads and no continuations). + +Instead, Bazel uses a fixed-size thread pool. However, that means that if a node declares a dependency that isn't available yet, we may have to abort that evaluation and restart it (possibly in another thread), when the dependency is available. This, in turn, means that nodes should not do this excessively; a node that declares N dependencies serially can potentially be restarted N times, costing O(N^2) time. Instead, we aim for up-front bulk declaration of dependencies, which sometimes requires reorganizing the code, or even splitting a node into multiple nodes to limit the number of restarts. + +Note that this technology isn't currently available in the rules API; instead, the rules API is still defined using the legacy concepts of loading, analysis, and execution phases. However, a fundamental restriction is that all accesses to other nodes have to go through the framework so that it can track the corresponding dependencies. Regardless of the language in which the build system is implemented or in which the rules are written (they don't have to be the same), rule authors must not use standard libraries or patterns that bypass Skyframe. For Java, that means avoiding java.io.File as well as any form of reflection, and any library that does either. Libraries that support dependency injection of these low-level interfaces still need to be setup correctly for Skyframe. + +This strongly suggests to avoid exposing rule authors to a full language runtime in the first place. The danger of accidental use of such APIs is just too big - several Bazel bugs in the past were caused by rules using unsafe APIs, even though the rules were written by the Bazel team or other domain experts. ### Avoiding quadratic time and memory consumption is hard -To make matters worse, apart from the requirements imposed by Skyframe, the -historical constraints of using Java, and the outdatedness of the rules API, -accidentally introducing quadratic time or memory consumption is a fundamental -problem in any build system based on library and binary rules. There are two -very common patterns that introduce quadratic memory consumption (and therefore -quadratic time consumption). - -1. Chains of Library Rules - -Consider the case of a chain of library rules A depends on B, depends on C, and -so on. Then, we want to compute some property over the transitive closure of -these rules, such as the Java runtime classpath, or the C++ linker command for -each library. Naively, we might take a standard list implementation; however, -this already introduces quadratic memory consumption: the first library -contains one entry on the classpath, the second two, the third three, and so -on, for a total of 1+2+3+...+N = O(N^2) entries. - -2. Binary Rules Depending on the Same Library Rules - -Consider the case where a set of binaries that depend on the same library -rules — such as if you have a number of test rules that test the same -library code. Let's say out of N rules, half the rules are binary rules, and -the other half library rules. Now consider that each binary makes a copy of -some property computed over the transitive closure of library rules, such as -the Java runtime classpath, or the C++ linker command line. For example, it -could expand the command line string representation of the C++ link action. N/2 -copies of N/2 elements is O(N^2) memory. +To make matters worse, apart from the requirements imposed by Skyframe, the historical constraints of using Java, and the outdatedness of the rules API, accidentally introducing quadratic time or memory consumption is a fundamental problem in any build system based on library and binary rules. There are two very common patterns that introduce quadratic memory consumption (and therefore quadratic time consumption). + +1. Chains of Library Rules - Consider the case of a chain of library rules A depends on B, depends on C, and so on. Then, we want to compute some property over the transitive closure of these rules, such as the Java runtime classpath, or the C++ linker command for each library. Naively, we might take a standard list implementation; however, this already introduces quadratic memory consumption: the first library contains one entry on the classpath, the second two, the third three, and so on, for a total of 1+2+3+...+N = O(N^2) entries. + +2. Binary Rules Depending on the Same Library Rules - Consider the case where a set of binaries that depend on the same library rules — such as if you have a number of test rules that test the same library code. Let's say out of N rules, half the rules are binary rules, and the other half library rules. Now consider that each binary makes a copy of some property computed over the transitive closure of library rules, such as the Java runtime classpath, or the C++ linker command line. For example, it could expand the command line string representation of the C++ link action. N/2 copies of N/2 elements is O(N^2) memory. #### Custom collections classes to avoid quadratic complexity -Bazel is heavily affected by both of these scenarios, so we introduced a set of -custom collection classes that effectively compress the information in memory by -avoiding the copy at each step. Almost all of these data structures have set -semantics, so we called it -[depset](/rules/lib/depset) -(also known as `NestedSet` in the internal implementation). The majority of -changes to reduce Bazel's memory consumption over the past several years were -changes to use depsets instead of whatever was previously used. - -Unfortunately, usage of depsets does not automatically solve all the issues; -in particular, even just iterating over a depset in each rule re-introduces -quadratic time consumption. Internally, NestedSets also has some helper methods -to facilitate interoperability with normal collections classes; unfortunately, -accidentally passing a NestedSet to one of these methods leads to copying -behavior, and reintroduces quadratic memory consumption. +Bazel is heavily affected by both of these scenarios, so we introduced a set of custom collection classes that effectively compress the information in memory by avoiding the copy at each step. Almost all of these data structures have set semantics, so we called it [depset](/rules/lib/depset) (also known as `NestedSet` in the internal implementation). The majority of changes to reduce Bazel's memory consumption over the past several years were changes to use depsets instead of whatever was previously used. + +Unfortunately, usage of depsets does not automatically solve all the issues; in particular, even just iterating over a depset in each rule re-introduces quadratic time consumption. Internally, NestedSets also has some helper methods to facilitate interoperability with normal collections classes; unfortunately, accidentally passing a NestedSet to one of these methods leads to copying behavior, and reintroduces quadratic memory consumption. diff --git a/rules/deploying.mdx b/rules/deploying.mdx index 3fe2c869..4395dd28 100644 --- a/rules/deploying.mdx +++ b/rules/deploying.mdx @@ -2,48 +2,30 @@ title: 'Deploying Rules' --- +This page is for rule writers who are planning to make their rules available to others. - -This page is for rule writers who are planning to make their rules available -to others. - -We recommend you start a new ruleset from the template repository: -https://github.com/bazel-contrib/rules-template -That template follows the recommendations below, and includes API documentation generation -and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. +We recommend you start a new ruleset from the template repository: [https://github.com/bazel-contrib/rules-template](https://github.com/bazel-contrib/rules-template) That template follows the recommendations below, and includes API documentation generation and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. ## Hosting and naming rules -New rules should go into their own GitHub repository under your organization. -Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) -if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) -organization. +New rules should go into their own GitHub repository under your organization. Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) organization. -Repository names for Bazel rules are standardized on the following format: -`$ORGANIZATION/rules_$NAME`. -See [examples on GitHub](https://github.com/search?q=rules+bazel&type=Repositories). -For consistency, you should follow this same format when publishing your Bazel rules. +Repository names for Bazel rules are standardized on the following format: `$ORGANIZATION/rules_$NAME`. See [examples on GitHub](https://github.com/search?q=rules+bazel\&type=Repositories). For consistency, you should follow this same format when publishing your Bazel rules. -Make sure to use a descriptive GitHub repository description and `README.md` -title, example: +Make sure to use a descriptive GitHub repository description and `README.md` title, example: -* Repository name: `bazelbuild/rules_go` -* Repository description: *Go rules for Bazel* -* Repository tags: `golang`, `bazel` -* `README.md` header: *Go rules for [Bazel](https://bazel.build)* -(note the link to https://bazel.build which will guide users who are unfamiliar -with Bazel to the right place) +- Repository name: `bazelbuild/rules_go` +- Repository description: *Go rules for Bazel* +- Repository tags: `golang`, `bazel` +- `README.md` header: *Go rules for [Bazel](https://bazel.build)* (note the link to [https://bazel.build](https://bazel.build) which will guide users who are unfamiliar with Bazel to the right place) -Rules can be grouped either by language (such as Scala), runtime platform -(such as Android), or framework (such as Spring). +Rules can be grouped either by language (such as Scala), runtime platform (such as Android), or framework (such as Spring). ## Repository content -Every rule repository should have a certain layout so that users can quickly -understand new rules. +Every rule repository should have a certain layout so that users can quickly understand new rules. -For example, when writing new rules for the (make-believe) -`mockascript` language, the rule repository would have the following structure: +For example, when writing new rules for the (make-believe) `mockascript` language, the rule repository would have the following structure: ``` / @@ -71,17 +53,9 @@ For example, when writing new rules for the (make-believe) ### MODULE.bazel -In the project's `MODULE.bazel`, you should define the name that users will use -to reference your rules. If your rules belong to the -[bazelbuild](https://github.com/bazelbuild) organization, you must use -`rules_` (such as `rules_mockascript`). Otherwise, you should name your -repository `_rules_` (such as `build_stack_rules_proto`). Please -start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) -if you feel like your rules should follow the convention for rules in the -[bazelbuild](https://github.com/bazelbuild) organization. +In the project's `MODULE.bazel`, you should define the name that users will use to reference your rules. If your rules belong to the [bazelbuild](https://github.com/bazelbuild) organization, you must use `rules_<lang>` (such as `rules_mockascript`). Otherwise, you should name your repository `<org>_rules_<lang>` (such as `build_stack_rules_proto`). Please start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules should follow the convention for rules in the [bazelbuild](https://github.com/bazelbuild) organization. -In the following sections, assume the repository belongs to the -[bazelbuild](https://github.com/bazelbuild) organization. +In the following sections, assume the repository belongs to the [bazelbuild](https://github.com/bazelbuild) organization. ``` module(name = "rules_mockascript") @@ -89,16 +63,11 @@ module(name = "rules_mockascript") ### README -At the top level, there should be a `README` that contains a brief description -of your ruleset, and the API users should expect. +At the top level, there should be a `README` that contains a brief description of your ruleset, and the API users should expect. ### Rules -Often times there will be multiple rules provided by your repository. Create a -directory named by the language and provide an entry point - `defs.bzl` file -exporting all rules (also include a `BUILD` file so the directory is a package). -For `rules_mockascript` that means there will be a directory named -`mockascript`, and a `BUILD` file and a `defs.bzl` file inside: +Often times there will be multiple rules provided by your repository. Create a directory named by the language and provide an entry point - `defs.bzl` file exporting all rules (also include a `BUILD` file so the directory is a package). For `rules_mockascript` that means there will be a directory named `mockascript`, and a `BUILD` file and a `defs.bzl` file inside: ``` / @@ -109,11 +78,7 @@ For `rules_mockascript` that means there will be a directory named ### Constraints -If your rule defines -[toolchain](/extending/toolchains) rules, -it's possible that you'll need to define custom `constraint_setting`s and/or -`constraint_value`s. Put these into a `///constraints` package. Your -directory structure will look like this: +If your rule defines [toolchain](/extending/toolchains) rules, it's possible that you'll need to define custom `constraint_setting`s and/or `constraint_value`s. Put these into a `//<LANG>/constraints` package. Your directory structure will look like this: ``` / @@ -124,100 +89,58 @@ directory structure will look like this: defs.bzl ``` -Please read -[github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) -for best practices, and to see what constraints are already present, and -consider contributing your constraints there if they are language independent. -Be mindful of introducing custom constraints, all users of your rules will -use them to perform platform specific logic in their `BUILD` files (for example, -using [selects](/reference/be/functions#select)). -With custom constraints, you define a language that the whole Bazel ecosystem -will speak. +Please read [github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) for best practices, and to see what constraints are already present, and consider contributing your constraints there if they are language independent. Be mindful of introducing custom constraints, all users of your rules will use them to perform platform specific logic in their `BUILD` files (for example, using [selects](/reference/be/functions#select)). With custom constraints, you define a language that the whole Bazel ecosystem will speak. ### Runfiles library -If your rule provides a standard library for accessing runfiles, it should be -in the form of a library target located at `///runfiles` (an abbreviation -of `///runfiles:runfiles`). User targets that need to access their data -dependencies will typically add this target to their `deps` attribute. +If your rule provides a standard library for accessing runfiles, it should be in the form of a library target located at `//<LANG>/runfiles` (an abbreviation of `//<LANG>/runfiles:runfiles`). User targets that need to access their data dependencies will typically add this target to their `deps` attribute. ### Repository rules #### Dependencies -Your rules might have external dependencies, which you'll need to specify in -your MODULE.bazel file. +Your rules might have external dependencies, which you'll need to specify in your MODULE.bazel file. #### Registering toolchains -Your rules might also register toolchains, which you can also specify in the -MODULE.bazel file. - -Note that in order to resolve toolchains in the analysis phase Bazel needs to -analyze all `toolchain` targets that are registered. Bazel will not need to -analyze all targets referenced by `toolchain.toolchain` attribute. If in order -to register toolchains you need to perform complex computation in the -repository, consider splitting the repository with `toolchain` targets from the -repository with `_toolchain` targets. Former will be always fetched, and -the latter will only be fetched when user actually needs to build `` code. +Your rules might also register toolchains, which you can also specify in the MODULE.bazel file. +Note that in order to resolve toolchains in the analysis phase Bazel needs to analyze all `toolchain` targets that are registered. Bazel will not need to analyze all targets referenced by `toolchain.toolchain` attribute. If in order to register toolchains you need to perform complex computation in the repository, consider splitting the repository with `toolchain` targets from the repository with `<LANG>_toolchain` targets. Former will be always fetched, and the latter will only be fetched when user actually needs to build `<LANG>` code. #### Release snippet -In your release announcement provide a snippet that your users can copy-paste -into their `MODULE.bazel` file. This snippet in general will look as follows: +In your release announcement provide a snippet that your users can copy-paste into their `MODULE.bazel` file. This snippet in general will look as follows: ``` -bazel_dep(name = "rules_", version = "") +bazel_dep(name = "rules_<LANG>", version = "<VERSION>") ``` - ### Tests -There should be tests that verify that the rules are working as expected. This -can either be in the standard location for the language the rules are for or a -`tests/` directory at the top level. +There should be tests that verify that the rules are working as expected. This can either be in the standard location for the language the rules are for or a `tests/` directory at the top level. ### Examples (optional) -It is useful to users to have an `examples/` directory that shows users a couple -of basic ways that the rules can be used. +It is useful to users to have an `examples/` directory that shows users a couple of basic ways that the rules can be used. ## CI/CD -Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib -org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. -See comments in the rules-template repo for more information. +Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. See comments in the rules-template repo for more information. -If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), -you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md&title=Request+to+add+new+project+%5BPROJECT_NAME%5D&labels=new-project) -it to [ci.bazel.build](http://ci.bazel.build). +If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md\&title=Request+to+add+new+project+%5BPROJECT_NAME%5D\&labels=new-project) it to [ci.bazel.build](http://ci.bazel.build). ## Documentation -See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for -instructions on how to comment your rules so that documentation can be generated -automatically. +See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for instructions on how to comment your rules so that documentation can be generated automatically. -The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) -shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date -as Starlark files are updated. +The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date as Starlark files are updated. ## FAQs ### Why can't we add our rule to the main Bazel GitHub repository? -We want to decouple rules from Bazel releases as much as possible. It's clearer -who owns individual rules, reducing the load on Bazel developers. For our users, -decoupling makes it easier to modify, upgrade, downgrade, and replace rules. -Contributing to rules can be lighter weight than contributing to Bazel - -depending on the rules -, including full submit access to the corresponding -GitHub repository. Getting submit access to Bazel itself is a much more involved -process. +We want to decouple rules from Bazel releases as much as possible. It's clearer who owns individual rules, reducing the load on Bazel developers. For our users, decoupling makes it easier to modify, upgrade, downgrade, and replace rules. Contributing to rules can be lighter weight than contributing to Bazel - depending on the rules -, including full submit access to the corresponding GitHub repository. Getting submit access to Bazel itself is a much more involved process. -The downside is a more complicated one-time installation process for our users: -they have to add a dependency on your ruleset in their `MODULE.bazel` file. +The downside is a more complicated one-time installation process for our users: they have to add a dependency on your ruleset in their `MODULE.bazel` file. -We used to have all of the rules in the Bazel repository (under -`//tools/build_rules` or `//tools/build_defs`). We still have a couple rules -there, but we are working on moving the remaining rules out. +We used to have all of the rules in the Bazel repository (under `//tools/build_rules` or `//tools/build_defs`). We still have a couple rules there, but we are working on moving the remaining rules out. diff --git a/rules/errors/read-only-variable.mdx b/rules/errors/read-only-variable.mdx index 2bfde654..ab4e4619 100644 --- a/rules/errors/read-only-variable.mdx +++ b/rules/errors/read-only-variable.mdx @@ -2,11 +2,7 @@ title: 'Error: Variable x is read only' --- - - -A global variable cannot be reassigned. It will always point to the same object. -However, its content might change, if the value is mutable (for example, the -content of a list). Local variables don't have this restriction. +A global variable cannot be reassigned. It will always point to the same object. However, its content might change, if the value is mutable (for example, the content of a list). Local variables don't have this restriction. ```python a = [1, 2] @@ -20,8 +16,7 @@ b = 4 # forbidden `ERROR: /path/ext.bzl:7:1: Variable b is read only` -You will get a similar error if you try to redefine a function (function -overloading is not supported), for example: +You will get a similar error if you try to redefine a function (function overloading is not supported), for example: ```python def foo(x): return x + 1 diff --git a/rules/faq.mdx b/rules/faq.mdx index 5321f0b7..89481fa7 100644 --- a/rules/faq.mdx +++ b/rules/faq.mdx @@ -2,79 +2,48 @@ title: 'Frequently Asked Questions' --- - - These are some common issues and questions with writing extensions. ## Why is my file not produced / my action never executed? Bazel only executes the actions needed to produce the *requested* output files. -* If the file you want has a label, you can request it directly: - `bazel build //pkg:myfile.txt` +- If the file you want has a label, you can request it directly: `bazel build //pkg:myfile.txt` -* If the file is in an output group of the target, you may need to specify that - output group on the command line: - `bazel build //pkg:mytarget --output_groups=foo` +- If the file is in an output group of the target, you may need to specify that output group on the command line: `bazel build //pkg:mytarget --output_groups=foo` -* If you want the file to be built automatically whenever your target is - mentioned on the command line, add it to your rule's default outputs by - returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. +- If you want the file to be built automatically whenever your target is mentioned on the command line, add it to your rule's default outputs by returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. See the [Rules page](/extending/rules#requesting-output-files) for more information. ## Why is my implementation function not executed? -Bazel analyzes only the targets that are requested for the build. You should -either name the target on the command line, or something that depends on the -target. +Bazel analyzes only the targets that are requested for the build. You should either name the target on the command line, or something that depends on the target. ## A file is missing when my action or binary is executed -Make sure that 1) the file has been registered as an input to the action or -binary, and 2) the script or tool being executed is accessing the file using the -correct path. +Make sure that 1) the file has been registered as an input to the action or binary, and 2) the script or tool being executed is accessing the file using the correct path. -For actions, you declare inputs by passing them to the `ctx.actions.*` function -that creates the action. The proper path for the file can be obtained using -[`File.path`](lib/File#path). +For actions, you declare inputs by passing them to the `ctx.actions.*` function that creates the action. The proper path for the file can be obtained using [`File.path`](lib/File#path). -For binaries (the executable outputs run by a `bazel run` or `bazel test` -command), you declare inputs by including them in the -[runfiles](/extending/rules#runfiles). Instead of using the `path` field, use -[`File.short_path`](lib/File#short_path), which is file's path relative to -the runfiles directory in which the binary executes. +For binaries (the executable outputs run by a `bazel run` or `bazel test` command), you declare inputs by including them in the [runfiles](/extending/rules#runfiles). Instead of using the `path` field, use [`File.short_path`](lib/File#short_path), which is file's path relative to the runfiles directory in which the binary executes. ## How can I control which files are built by `bazel build //pkg:mytarget`? -Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to -[set the default outputs](/extending/rules#requesting-output-files). +Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to [set the default outputs](/extending/rules#requesting-output-files). ## How can I run a program or do file I/O as part of my build? -A tool can be declared as a target, just like any other part of your build, and -run during the execution phase to help build other targets. To create an action -that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the -tool as the `executable` parameter. +A tool can be declared as a target, just like any other part of your build, and run during the execution phase to help build other targets. To create an action that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the tool as the `executable` parameter. -During the loading and analysis phases, a tool *cannot* run, nor can you perform -file I/O. This means that tools and file contents (except the contents of BUILD -and .bzl files) cannot affect how the target and action graphs get created. +During the loading and analysis phases, a tool *cannot* run, nor can you perform file I/O. This means that tools and file contents (except the contents of BUILD and .bzl files) cannot affect how the target and action graphs get created. ## What if I need to access the same structured data both before and during the execution phase? -You can format the structured data as a .bzl file. You can `load()` the file to -access it during the loading and analysis phases. You can pass it as an input or -runfile to actions and executables that need it during the execution phase. +You can format the structured data as a .bzl file. You can `load()` the file to access it during the loading and analysis phases. You can pass it as an input or runfile to actions and executables that need it during the execution phase. ## How should I document Starlark code? -For rules and rule attributes, you can pass a docstring literal (possibly -triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper -functions and macros, use a triple-quoted docstring literal following the format -given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). -Rule implementation functions generally do not need their own docstring. +For rules and rule attributes, you can pass a docstring literal (possibly triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper functions and macros, use a triple-quoted docstring literal following the format given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Rule implementation functions generally do not need their own docstring. -Using string literals in the expected places makes it easier for automated -tooling to extract documentation. Feel free to use standard non-string comments -wherever it may help the reader of your code. +Using string literals in the expected places makes it easier for automated tooling to extract documentation. Feel free to use standard non-string comments wherever it may help the reader of your code. diff --git a/rules/index.mdx b/rules/index.mdx index 2a6c3eb7..44590f11 100644 --- a/rules/index.mdx +++ b/rules/index.mdx @@ -2,11 +2,7 @@ title: 'Rules' --- - - -The Bazel ecosystem has a growing and evolving set of rules to support popular -languages and packages. Much of Bazel's strength comes from the ability to -[define new rules](/extending/concepts) that can be used by others. +The Bazel ecosystem has a growing and evolving set of rules to support popular languages and packages. Much of Bazel's strength comes from the ability to [define new rules](/extending/concepts) that can be used by others. This page describes the recommended, native, and non-native Bazel rules. @@ -14,58 +10,57 @@ This page describes the recommended, native, and non-native Bazel rules. Here is a selection of recommended rules: -* [Android](/docs/bazel-and-android) -* [C / C++](/docs/bazel-and-cpp) -* [Docker/OCI](https://github.com/bazel-contrib/rules_oci) -* [Go](https://github.com/bazelbuild/rules_go) -* [Haskell](https://github.com/tweag/rules_haskell) -* [Java](/docs/bazel-and-java) -* [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) -* [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) -* [Objective-C](/docs/bazel-and-apple) -* [Package building](https://github.com/bazelbuild/rules_pkg) -* [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) -* [Python](https://github.com/bazelbuild/rules_python) -* [Rust](https://github.com/bazelbuild/rules_rust) -* [Scala](https://github.com/bazelbuild/rules_scala) -* [Shell](/reference/be/shell) -* [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) - -The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains -additional functions that can be useful when writing new rules and new -macros. - -The rules above were reviewed and follow our -[requirements for recommended rules](/community/recommended-rules). -Contact the respective rule set's maintainers regarding issues and feature -requests. - -To find more Bazel rules, use a search engine, take a look on -[awesomebazel.com](https://awesomebazel.com/), or search on -[GitHub](https://github.com/search?o=desc&q=bazel+rules&s=stars&type=Repositories). +- [Android](/docs/bazel-and-android) +- [C / C++](/docs/bazel-and-cpp) +- [Docker/OCI](https://github.com/bazel-contrib/rules_oci) +- [Go](https://github.com/bazelbuild/rules_go) +- [Haskell](https://github.com/tweag/rules_haskell) +- [Java](/docs/bazel-and-java) +- [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) +- [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) +- [Objective-C](/docs/bazel-and-apple) +- [Package building](https://github.com/bazelbuild/rules_pkg) +- [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) +- [Python](https://github.com/bazelbuild/rules_python) +- [Rust](https://github.com/bazelbuild/rules_rust) +- [Scala](https://github.com/bazelbuild/rules_scala) +- [Shell](/reference/be/shell) +- [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) + +The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains additional functions that can be useful when writing new rules and new macros. + +The rules above were reviewed and follow our [requirements for recommended rules](/community/recommended-rules). Contact the respective rule set's maintainers regarding issues and feature requests. + +To find more Bazel rules, use a search engine, take a look on [awesomebazel.com](https://awesomebazel.com/), or search on [GitHub](https://github.com/search?o=desc\&q=bazel+rules\&s=stars\&type=Repositories). ## Native rules that do not apply to a specific programming language -Native rules are shipped with the Bazel binary, they are always available in -BUILD files without a `load` statement. +Native rules are shipped with the Bazel binary, they are always available in BUILD files without a `load` statement. + +- Extra actions -* Extra actions - [`extra_action`](/reference/be/extra-actions#extra_action) - [`action_listener`](/reference/be/extra-actions#action_listener) -* General + +- General + - [`filegroup`](/reference/be/general#filegroup) - [`genquery`](/reference/be/general#genquery) - [`test_suite`](/reference/be/general#test_suite) - [`alias`](/reference/be/general#alias) - [`config_setting`](/reference/be/general#config_setting) - [`genrule`](/reference/be/general#genrule) -* Platform + +- Platform + - [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) - [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) - [`platform`](/reference/be/platforms-and-toolchains#platform) - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - [`toolchain_type`](/reference/be/platforms-and-toolchains#toolchain_type) -* Workspace + +- Workspace + - [`bind`](/reference/be/workspace#bind) - [`local_repository`](/reference/be/workspace#local_repository) - [`new_local_repository`](/reference/be/workspace#new_local_repository) @@ -74,10 +69,10 @@ BUILD files without a `load` statement. ## Embedded non-native rules -Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from -the `@bazel_tools` built-in external repository. +Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from the `@bazel_tools` built-in external repository. + +- Repository rules -* Repository rules - [`git_repository`](/rules/lib/repo/git#git_repository) - [`http_archive`](/rules/lib/repo/http#http_archive) - [`http_file`](/rules/lib/repo/http#http_archive) diff --git a/rules/language.mdx b/rules/language.mdx index 13e33a4b..9fad7248 100644 --- a/rules/language.mdx +++ b/rules/language.mdx @@ -2,18 +2,11 @@ title: 'Starlark Language' --- - - -{/* [TOC] */} - -This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), -formerly known as Skylark, the language used in Bazel. For a complete list of -functions and types, see the [Bazel API reference](/rules/lib/overview). +This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), formerly known as Skylark, the language used in Bazel. For a complete list of functions and types, see the [Bazel API reference](/rules/lib/overview). For more information about the language, see [Starlark's GitHub repo](https://github.com/bazelbuild/starlark/). -For the authoritative specification of the Starlark syntax and -behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). +For the authoritative specification of the Starlark syntax and behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). ## Syntax @@ -33,44 +26,32 @@ def fizz_buzz(n): fizz_buzz(20) ``` -Starlark's semantics can differ from Python, but behavioral differences are -rare, except for cases where Starlark raises an error. The following Python -types are supported: +Starlark's semantics can differ from Python, but behavioral differences are rare, except for cases where Starlark raises an error. The following Python types are supported: -* [None](lib/globals#None) -* [bool](lib/bool) -* [dict](lib/dict) -* [tuple](lib/tuple) -* [function](lib/function) -* [int](lib/int) -* [list](lib/list) -* [string](lib/string) +- [None](lib/globals#None) +- [bool](lib/bool) +- [dict](lib/dict) +- [tuple](lib/tuple) +- [function](lib/function) +- [int](lib/int) +- [list](lib/list) +- [string](lib/string) ## Type annotations -**Experimental**. Type annotations are an experimental feature and may change -at any time. Don't depend on it. It may be enabled in Bazel at HEAD -by using the `--experimental_starlark_types` flag. +**Experimental**. Type annotations are an experimental feature and may change at any time. Don't depend on it. It may be enabled in Bazel at HEAD by using the `--experimental_starlark_types` flag. -Starlark in Bazel at HEAD is incrementally adding support for type annotations -with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). +Starlark in Bazel at HEAD is incrementally adding support for type annotations with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). -- Starlark type annotations are under active development. The progress is - tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). +- Starlark type annotations are under active development. The progress is tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). - The specification is incrementally extended: [starlark-with-types/spec.md](https://github.com/bazelbuild/starlark/blob/starlark-with-types/spec.md) - Initial proposal: [SEP-001 Bootstrapping Starlark types](https://docs.google.com/document/d/1Sid7EAbBd_w_T7D94Li_f_bK3zMTztFbzIMvcpzo1wY/edit?tab=t.0#heading=h.5mcn15i0e1ch) ## Mutability -Starlark favors immutability. Two mutable data structures are available: -[lists](lib/list) and [dicts](lib/dict). Changes to mutable -data-structures, such as appending a value to a list or deleting an entry in a -dictionary are valid only for objects created in the current context. After a -context finishes, its values become immutable. +Starlark favors immutability. Two mutable data structures are available: [lists](lib/list) and [dicts](lib/dict). Changes to mutable data-structures, such as appending a value to a list or deleting an entry in a dictionary are valid only for objects created in the current context. After a context finishes, its values become immutable. -This is because Bazel builds use parallel execution. During a build, each `.bzl` -file and each `BUILD` file get their own execution context. Each rule is also -analyzed in its own context. +This is because Bazel builds use parallel execution. During a build, each `.bzl` file and each `BUILD` file get their own execution context. Each rule is also analyzed in its own context. Let's go through an example with the file `foo.bzl`: @@ -84,13 +65,9 @@ def fct(): # declare a function fct() # execute the fct function ``` -Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s -context. When `fct()` runs, it does so within the context of `foo.bzl`. After -evaluation for `foo.bzl` completes, the environment contains an immutable entry, -`var`, with the value `[5]`. +Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s context. When `fct()` runs, it does so within the context of `foo.bzl`. After evaluation for `foo.bzl` completes, the environment contains an immutable entry, `var`, with the value `[5]`. -When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain -immutable. For this reason, the following code in `bar.bzl` is illegal: +When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain immutable. For this reason, the following code in `bar.bzl` is illegal: ```python # `bar.bzl` @@ -101,66 +78,52 @@ var.append(6) # runtime error, the list stored in var is frozen fct() # runtime error, fct() attempts to modify a frozen list ``` -Global variables defined in `bzl` files cannot be changed outside of the -`bzl` file that defined them. Just like the above example using `bzl` files, -values returned by rules are immutable. +Global variables defined in `bzl` files cannot be changed outside of the `bzl` file that defined them. Just like the above example using `bzl` files, values returned by rules are immutable. ## Differences between BUILD and .bzl files -`BUILD` files register targets via making calls to rules. `.bzl` files provide -definitions for constants, rules, macros, and functions. +`BUILD` files register targets via making calls to rules. `.bzl` files provide definitions for constants, rules, macros, and functions. -[Native functions](/reference/be/functions) and [native rules]( -/reference/be/overview#language-specific-native-rules) are global symbols in -`BUILD` files. `bzl` files need to load them using the [`native` module]( -/rules/lib/toplevel/native). +[Native functions](/reference/be/functions) and [native rules](/reference/be/overview#language-specific-native-rules) are global symbols in `BUILD` files. `bzl` files need to load them using the [`native` module](/rules/lib/toplevel/native). -There are two syntactic restrictions in `BUILD` files: 1) declaring functions is -illegal, and 2) `*args` and `**kwargs` arguments are not allowed. +There are two syntactic restrictions in `BUILD` files: 1) declaring functions is illegal, and 2) `*args` and `**kwargs` arguments are not allowed. ## Differences with Python -* Global variables are immutable. +- Global variables are immutable. -* `for` statements are not allowed at the top-level. Use them within functions - instead. In `BUILD` files, you may use list comprehensions. +- `for` statements are not allowed at the top-level. Use them within functions instead. In `BUILD` files, you may use list comprehensions. -* `if` statements are not allowed at the top-level. However, `if` expressions - can be used: `first = data[0] if len(data) > 0 else None`. +- `if` statements are not allowed at the top-level. However, `if` expressions can be used: `first = data[0] if len(data) > 0 else None`. -* Deterministic order for iterating through Dictionaries. +- Deterministic order for iterating through Dictionaries. -* Recursion is not allowed. +- Recursion is not allowed. -* Int type is limited to 32-bit signed integers. Overflows will throw an error. +- Int type is limited to 32-bit signed integers. Overflows will throw an error. -* Modifying a collection during iteration is an error. +- Modifying a collection during iteration is an error. -* Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are -not defined across value types. In short: `5 < 'foo'` will throw an error and -`5 == "5"` will return false. +- Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are not defined across value types. In short: `5 < 'foo'` will throw an error and `5 == "5"` will return false. -* In tuples, a trailing comma is valid only when the tuple is between - parentheses — when you write `(1,)` instead of `1,`. +- In tuples, a trailing comma is valid only when the tuple is between parentheses — when you write `(1,)` instead of `1,`. -* Dictionary literals cannot have duplicated keys. For example, this is an - error: `{"a": 4, "b": 7, "a": 1}`. +- Dictionary literals cannot have duplicated keys. For example, this is an error: `{"a": 4, "b": 7, "a": 1}`. -* Strings are represented with double-quotes (such as when you call - [repr](lib/globals#repr)). +- Strings are represented with double-quotes (such as when you call [repr](lib/globals#repr)). -* Strings aren't iterable. +- Strings aren't iterable. The following Python features are not supported: -* implicit string concatenation (use explicit `+` operator). -* Chained comparisons (such as `1 < x < 5`). -* `class` (see [`struct`](lib/struct#struct) function). -* `import` (see [`load`](/extending/concepts#loading-an-extension) statement). -* `while`, `yield`. -* float and set types. -* generators and generator expressions. -* `is` (use `==` instead). -* `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). -* `global`, `nonlocal`. -* most builtin functions, most methods. +- implicit string concatenation (use explicit `+` operator). +- Chained comparisons (such as `1 < x < 5`). +- `class` (see [`struct`](lib/struct#struct) function). +- `import` (see [`load`](/extending/concepts#loading-an-extension) statement). +- `while`, `yield`. +- float and set types. +- generators and generator expressions. +- `is` (use `==` instead). +- `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). +- `global`, `nonlocal`. +- most builtin functions, most methods. diff --git a/rules/legacy-macro-tutorial.mdx b/rules/legacy-macro-tutorial.mdx index 28b0fca8..8322ec42 100644 --- a/rules/legacy-macro-tutorial.mdx +++ b/rules/legacy-macro-tutorial.mdx @@ -2,20 +2,11 @@ title: 'Creating a Legacy Macro' --- +IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If you only need to support Bazel 8 or newer, we recommend using [symbolic macros](/extending/macros) instead; take a look at [Creating a Symbolic Macro](macro-tutorial). +Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a legacy macro that resizes an image. -IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If -you only need to support Bazel 8 or newer, we recommend using [symbolic -macros](/extending/macros) instead; take a look at [Creating a Symbolic -Macro](macro-tutorial). - -Imagine that you need to run a tool as part of your build. For example, you -may want to generate or preprocess a source file, or compress a binary. In this -tutorial, you are going to create a legacy macro that resizes an image. - -Macros are suitable for simple tasks. If you want to do anything more -complicated, for example add support for a new programming language, consider -creating a [rule](/extending/rules). Rules give you more control and flexibility. +Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -24,7 +15,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -34,8 +25,7 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, -define a function in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, define a function in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark def miniature(name, src, size = "100x100", **kwargs): @@ -48,25 +38,20 @@ def miniature(name, src, size = "100x100", **kwargs): srcs = [src], # Note that the line below will fail if `src` is not a filename string outs = ["small_" + src], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs ) ``` A few remarks: - * By convention, legacy macros have a `name` argument, just like rules. +- By convention, legacy macros have a `name` argument, just like rules. - * To document the behavior of a legacy macro, use - [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. +- To document the behavior of a legacy macro, use [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. - * To call a `genrule`, or any other native rule, prefix with `native.`. +- To call a `genrule`, or any other native rule, prefix with `native.`. - * Use `**kwargs` to forward the extra arguments to the underlying `genrule` - (it works just like in - [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). - This is useful, so that a user can use standard attributes like - `visibility`, or `tags`. +- Use `**kwargs` to forward the extra arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful, so that a user can use standard attributes like `visibility`, or `tags`. Now, use the macro from the `BUILD` file: @@ -85,14 +70,6 @@ cc_binary( ) ``` -And finally, a **warning note**: the macro assumes that `src` is a filename -string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` -works; but what happens if the `BUILD` file instead used `src = -"//other/package:image.png"`, or even `src = select(...)`? - -You should make sure to declare such assumptions in your macro's documentation. -Unfortunately, legacy macros, especially large ones, tend to be fragile because -it can be hard to notice and document all such assumptions in your code – and, -of course, some users of the macro won't read the documentation. We recommend, -if possible, instead using [symbolic macros](/extending/macros), which have -built\-in checks on attribute types. +And finally, a **warning note**: the macro assumes that `src` is a filename string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` works; but what happens if the `BUILD` file instead used `src = "//other/package:image.png"`, or even `src = select(...)`? + +You should make sure to declare such assumptions in your macro's documentation. Unfortunately, legacy macros, especially large ones, tend to be fragile because it can be hard to notice and document all such assumptions in your code – and, of course, some users of the macro won't read the documentation. We recommend, if possible, instead using [symbolic macros](/extending/macros), which have built-in checks on attribute types. diff --git a/rules/macro-tutorial.mdx b/rules/macro-tutorial.mdx index 2b3d8e4a..4cf91617 100644 --- a/rules/macro-tutorial.mdx +++ b/rules/macro-tutorial.mdx @@ -2,20 +2,11 @@ title: 'Creating a Symbolic Macro' --- +IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new macro system introduced in Bazel 8. If you need to support older Bazel versions, you will want to write a [legacy macro](/extending/legacy-macros) instead; take a look at [Creating a Legacy Macro](legacy-macro-tutorial). +Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a symbolic macro that resizes an image. -IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new -macro system introduced in Bazel 8. If you need to support older Bazel versions, -you will want to write a [legacy macro](/extending/legacy-macros) instead; take -a look at [Creating a Legacy Macro](legacy-macro-tutorial). - -Imagine that you need to run a tool as part of your build. For example, you -may want to generate or preprocess a source file, or compress a binary. In this -tutorial, you are going to create a symbolic macro that resizes an image. - -Macros are suitable for simple tasks. If you want to do anything more -complicated, for example add support for a new programming language, consider -creating a [rule](/extending/rules). Rules give you more control and flexibility. +Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -24,7 +15,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -34,9 +25,7 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, -define an *implementation function* and a *macro declaration* in a separate -`.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, define an *implementation function* and a *macro declaration* in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark # Implementation function @@ -46,7 +35,7 @@ def _miniature_impl(name, visibility, src, size, **kwargs): visibility = visibility, srcs = [src], outs = [name + "_small_" + src.name], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs, ) @@ -84,19 +73,13 @@ miniature = macro( A few remarks: - * Symbolic macro implementation functions must have `name` and `visibility` - parameters. They should used for the macro's main target. +- Symbolic macro implementation functions must have `name` and `visibility` parameters. They should used for the macro's main target. - * To document the behavior of a symbolic macro, use `doc` parameters for - `macro()` and its attributes. +- To document the behavior of a symbolic macro, use `doc` parameters for `macro()` and its attributes. - * To call a `genrule`, or any other native rule, use `native.`. +- To call a `genrule`, or any other native rule, use `native.`. - * Use `**kwargs` to forward the extra inherited arguments to the underlying - `genrule` (it works just like in - [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). - This is useful so that a user can set standard attributes like `tags` or - `testonly`. +- Use `**kwargs` to forward the extra inherited arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful so that a user can set standard attributes like `tags` or `testonly`. Now, use the macro from the `BUILD` file: diff --git a/rules/performance.mdx b/rules/performance.mdx index c415cf14..edc0a2fd 100644 --- a/rules/performance.mdx +++ b/rules/performance.mdx @@ -2,30 +2,20 @@ title: 'Optimizing Performance' --- +When writing rules, the most common performance pitfall is to traverse or copy data that is accumulated from dependencies. When aggregated over the whole build, these operations can easily take O(N^2) time or space. To avoid this, it is crucial to understand how to use depsets effectively. - -When writing rules, the most common performance pitfall is to traverse or copy -data that is accumulated from dependencies. When aggregated over the whole -build, these operations can easily take O(N^2) time or space. To avoid this, it -is crucial to understand how to use depsets effectively. - -This can be hard to get right, so Bazel also provides a memory profiler that -assists you in finding spots where you might have made a mistake. Be warned: -The cost of writing an inefficient rule may not be evident until it is in -widespread use. +This can be hard to get right, so Bazel also provides a memory profiler that assists you in finding spots where you might have made a mistake. Be warned: The cost of writing an inefficient rule may not be evident until it is in widespread use. ## Use depsets -Whenever you are rolling up information from rule dependencies you should use -[depsets](lib/depset). Only use plain lists or dicts to publish information -local to the current rule. +Whenever you are rolling up information from rule dependencies you should use [depsets](lib/depset). Only use plain lists or dicts to publish information local to the current rule. A depset represents information as a nested graph which enables sharing. Consider the following graph: ``` -C -> B -> A +C -> B -> A D ---^ ``` @@ -47,12 +37,9 @@ c = ['c', 'b', 'a'] d = ['d', 'b', 'a'] ``` -Note that in this case `'a'` is mentioned four times! With larger graphs this -problem will only get worse. +Note that in this case `'a'` is mentioned four times! With larger graphs this problem will only get worse. -Here is an example of a rule implementation that uses depsets correctly to -publish transitive information. Note that it is OK to publish rule-local -information using lists if you want since this is not O(N^2). +Here is an example of a rule implementation that uses depsets correctly to publish transitive information. Note that it is OK to publish rule-local information using lists if you want since this is not O(N^2). ``` MyProvider = provider() @@ -74,21 +61,13 @@ See the [depset overview](/extending/depsets) page for more information. ### Avoid calling `depset.to_list()` -You can coerce a depset to a flat list using -[`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) -cost. If at all possible, avoid any flattening of depsets except for debugging -purposes. +You can coerce a depset to a flat list using [`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) cost. If at all possible, avoid any flattening of depsets except for debugging purposes. -A common misconception is that you can freely flatten depsets if you only do it -at top-level targets, such as an `_binary` rule, since then the cost is not -accumulated over each level of the build graph. But this is *still* O(N^2) when -you build a set of targets with overlapping dependencies. This happens when -building your tests `//foo/tests/...`, or when importing an IDE project. +A common misconception is that you can freely flatten depsets if you only do it at top-level targets, such as an `<xx>_binary` rule, since then the cost is not accumulated over each level of the build graph. But this is *still* O(N^2) when you build a set of targets with overlapping dependencies. This happens when building your tests `//foo/tests/...`, or when importing an IDE project. ### Reduce the number of calls to `depset` -Calling `depset` inside a loop is often a mistake. It can lead to depsets with -very deep nesting, which perform poorly. For example: +Calling `depset` inside a loop is often a mistake. It can lead to depsets with very deep nesting, which perform poorly. For example: ```python x = depset() @@ -97,8 +76,7 @@ for i in inputs: x = depset(transitive = [x, i.deps]) ``` -This code can be replaced easily. First, collect the transitive depsets and -merge them all at once: +This code can be replaced easily. First, collect the transitive depsets and merge them all at once: ```python transitive = [] @@ -117,33 +95,19 @@ x = depset(transitive = [i.deps for i in inputs]) ## Use ctx.actions.args() for command lines -When building command lines you should use [ctx.actions.args()](lib/Args). -This defers expansion of any depsets to the execution phase. +When building command lines you should use [ctx.actions.args()](lib/Args). This defers expansion of any depsets to the execution phase. -Apart from being strictly faster, this will reduce the memory consumption of -your rules -- sometimes by 90% or more. +Apart from being strictly faster, this will reduce the memory consumption of your rules -- sometimes by 90% or more. Here are some tricks: -* Pass depsets and lists directly as arguments, instead of flattening them -yourself. They will get expanded by `ctx.actions.args()` for you. -If you need any transformations on the depset contents, look at -[ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. +- Pass depsets and lists directly as arguments, instead of flattening them yourself. They will get expanded by `ctx.actions.args()` for you. If you need any transformations on the depset contents, look at [ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. -* Are you passing `File#path` as arguments? No need. Any -[File](lib/File) is automatically turned into its -[path](lib/File#path), deferred to expansion time. +- Are you passing `File#path` as arguments? No need. Any [File](lib/File) is automatically turned into its [path](lib/File#path), deferred to expansion time. -* Avoid constructing strings by concatenating them together. -The best string argument is a constant as its memory will be shared between -all instances of your rule. +- Avoid constructing strings by concatenating them together. The best string argument is a constant as its memory will be shared between all instances of your rule. -* If the args are too long for the command line an `ctx.actions.args()` object -can be conditionally or unconditionally written to a param file using -[`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is -done behind the scenes when the action is executed. If you need to explicitly -control the params file you can write it manually using -[`ctx.actions.write`](lib/actions#write). +- If the args are too long for the command line an `ctx.actions.args()` object can be conditionally or unconditionally written to a param file using [`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is done behind the scenes when the action is executed. If you need to explicitly control the params file you can write it manually using [`ctx.actions.write`](lib/actions#write). Example: @@ -154,15 +118,15 @@ def _impl(ctx): file = ctx.declare_file(...) files = depset(...) - # Bad, constructs a full string "--foo=" for each rule instance + # Bad, constructs a full string "--foo=<file path>" for each rule instance args.add("--foo=" + file.path) # Good, shares "--foo" among all rule instances, and defers file.path to later - # It will however pass ["--foo", ] to the action command line, - # instead of ["--foo="] + # It will however pass ["--foo", <file path>] to the action command line, + # instead of ["--foo=<file_path>"] args.add("--foo", file) - # Use format if you prefer ["--foo="] to ["--foo", ] + # Use format if you prefer ["--foo=<file path>"] to ["--foo", <file path>] args.add(file, format="--foo=%s") # Bad, makes a giant string of a whole depset @@ -178,9 +142,7 @@ def _to_short_path(f): ## Transitive action inputs should be depsets -When building an action using [ctx.actions.run](lib/actions?#run), do not -forget that the `inputs` field accepts a depset. Use this whenever inputs are -collected from dependencies transitively. +When building an action using [ctx.actions.run](lib/actions?#run), do not forget that the `inputs` field accepts a depset. Use this whenever inputs are collected from dependencies transitively. ``` inputs = depset(...) @@ -192,54 +154,39 @@ ctx.actions.run( ## Hanging -If Bazel appears to be hung, you can hit Ctrl-\ or send -Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread -dump in the file `$(bazel info output_base)/server/jvm.out`. +If Bazel appears to be hung, you can hit `Ctrl-\` or send Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread dump in the file `$(bazel info output_base)/server/jvm.out`. -Since you may not be able to run `bazel info` if bazel is hung, the -`output_base` directory is usually the parent of the `bazel-` -symlink in your workspace directory. +Since you may not be able to run `bazel info` if bazel is hung, the `output_base` directory is usually the parent of the `bazel-<workspace>` symlink in your workspace directory. ## Performance profiling -The [JSON trace profile](/advanced/performance/json-trace-profile) can be very -useful to quickly understand what Bazel spent time on during the invocation. +The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. -The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) -flag may be used to capture Java Flight Recorder profiles of various kinds -(cpu time, wall time, memory allocations and lock contention). +The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) flag may be used to capture Java Flight Recorder profiles of various kinds (cpu time, wall time, memory allocations and lock contention). -The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) -flag may be used to write a pprof profile of CPU usage by all Starlark threads. +The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) flag may be used to write a pprof profile of CPU usage by all Starlark threads. ## Memory profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s -memory use. If there is a problem you can dump the heap to find the -exact line of code that is causing the problem. +Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. If there is a problem you can dump the heap to find the exact line of code that is causing the problem. ### Enabling memory tracking You must pass these two startup flags to *every* Bazel invocation: - ``` - STARTUP_FLAGS=\ - --host_jvm_args=-javaagent: \

- --host_jvm_args=-DRULE_MEMORY_TRACKER=1 - ``` -Note: You can download the allocation instrumenter jar file from [Maven Central -Repository][allocation-instrumenter-link]. +``` +STARTUP_FLAGS=\ +--host_jvm_args=-javaagent:<path to java-allocation-instrumenter-3.3.4.jar> \ +--host_jvm_args=-DRULE_MEMORY_TRACKER=1 +``` -[allocation-instrumenter-link]: https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4 +Note: You can download the allocation instrumenter jar file from [Maven Central Repository](https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4). -These start the server in memory tracking mode. If you forget these for even -one Bazel invocation the server will restart and you will have to start over. +These start the server in memory tracking mode. If you forget these for even one Bazel invocation the server will restart and you will have to start over. ### Using the Memory Tracker -As an example, look at the target `foo` and see what it does. To only -run the analysis and not run the build execution phase, add the -`--nobuild` flag. +As an example, look at the target `foo` and see what it does. To only run the analysis and not run the build execution phase, add the `--nobuild` flag. ``` $ bazel $(STARTUP_FLAGS) build --nobuild //foo:foo @@ -249,14 +196,14 @@ Next, see how much memory the whole Bazel instance consumes: ``` $ bazel $(STARTUP_FLAGS) info used-heap-size-after-gc -> 2594MB +> 2594MB ``` Break it down by rule class by using `bazel dump --rules`: ``` $ bazel $(STARTUP_FLAGS) dump --rules -> +> RULE COUNT ACTIONS BYTES EACH genrule 33,762 33,801 291,538,824 8,635 @@ -271,16 +218,14 @@ _check_proto_library_deps 719 668 1,835,288 2,5 ... (more output) ``` -Look at where the memory is going by producing a `pprof` file -using `bazel dump --skylark_memory`: +Look at where the memory is going by producing a `pprof` file using `bazel dump --skylark_memory`: ``` $ bazel $(STARTUP_FLAGS) dump --skylark_memory=$HOME/prof.gz -> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz +> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz ``` -Use the `pprof` tool to investigate the heap. A good starting point is -getting a flame graph by using `pprof -flame $HOME/prof.gz`. +Use the `pprof` tool to investigate the heap. A good starting point is getting a flame graph by using `pprof -flame $HOME/prof.gz`. Get `pprof` from [https://github.com/google/pprof](https://github.com/google/pprof). @@ -288,13 +233,13 @@ Get a text dump of the hottest call sites annotated with lines: ``` $ pprof -text -lines $HOME/prof.gz -> +> flat flat% sum% cum cum% - 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library :-1 - 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule :-1 - 74.11MB 9.96% 44.80% 74.11MB 9.96% glob :-1 - 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup :-1 - 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test :-1 + 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library <native>:-1 + 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule <native>:-1 + 74.11MB 9.96% 44.80% 74.11MB 9.96% glob <native>:-1 + 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup <native>:-1 + 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test <native>:-1 26.55MB 3.57% 63.07% 26.55MB 3.57% _generate_foo_files /foo/tc/tc.bzl:491 26.01MB 3.50% 66.57% 26.01MB 3.50% _build_foo_impl /foo/build_test.bzl:78 22.01MB 2.96% 69.53% 22.01MB 2.96% _build_foo_impl /foo/build_test.bzl:73 diff --git a/rules/rules-tutorial.mdx b/rules/rules-tutorial.mdx index 7700f20c..30ad839d 100644 --- a/rules/rules-tutorial.mdx +++ b/rules/rules-tutorial.mdx @@ -2,24 +2,9 @@ title: 'Rules Tutorial' --- +[Starlark](https://github.com/bazelbuild/starlark) is a Python-like configuration language originally developed for use in Bazel and since adopted by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of Starlark properly known as the "Build Language", though it is often simply referred to as "Starlark", especially when emphasizing that a feature is expressed in the Build Language as opposed to being a built-in or "native" part of Bazel. Bazel augments the core language with numerous build-related functions such as `glob`, `genrule`, `java_binary`, and so on. - -{/* [TOC] */} - -[Starlark](https://github.com/bazelbuild/starlark) is a Python-like -configuration language originally developed for use in Bazel and since adopted -by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of -Starlark properly known as the "Build Language", though it is often simply -referred to as "Starlark", especially when emphasizing that a feature is -expressed in the Build Language as opposed to being a built-in or "native" part -of Bazel. Bazel augments the core language with numerous build-related functions -such as `glob`, `genrule`, `java_binary`, and so on. - -See the -[Bazel](/start/) and [Starlark](/extending/concepts) documentation for -more details, and the -[Rules SIG template](https://github.com/bazel-contrib/rules-template) as a -starting point for new rulesets. +See the [Bazel](/start/) and [Starlark](/extending/concepts) documentation for more details, and the [Rules SIG template](https://github.com/bazel-contrib/rules-template) as a starting point for new rulesets. ## The empty rule @@ -34,10 +19,7 @@ foo_binary = rule( ) ``` -When you call the [`rule`](lib/globals#rule) function, you -must define a callback function. The logic will go there, but you -can leave the function empty for now. The [`ctx`](lib/ctx) argument -provides information about the target. +When you call the [`rule`](lib/globals#rule) function, you must define a callback function. The logic will go there, but you can leave the function empty for now. The [`ctx`](lib/ctx) argument provides information about the target. You can load the rule and use it from a `BUILD` file. @@ -58,9 +40,7 @@ INFO: Found 1 target... Target //:bin up-to-date (nothing to build) ``` -Even though the rule does nothing, it already behaves like other rules: it has a -mandatory name, it supports common attributes like `visibility`, `testonly`, and -`tags`. +Even though the rule does nothing, it already behaves like other rules: it has a mandatory name, it supports common attributes like `visibility`, `testonly`, and `tags`. ## Evaluation model @@ -89,10 +69,7 @@ foo_binary(name = "bin1") foo_binary(name = "bin2") ``` -[`ctx.label`](lib/ctx#label) -corresponds to the label of the target being analyzed. The `ctx` object has -many useful fields and methods; you can find an exhaustive list in the -[API reference](lib/ctx). +[`ctx.label`](lib/ctx#label) corresponds to the label of the target being analyzed. The `ctx` object has many useful fields and methods; you can find an exhaustive list in the [API reference](lib/ctx). Query the code: @@ -106,15 +83,10 @@ DEBUG: /usr/home/bazel-codelab/BUILD:2:1: BUILD file Make a few observations: -* "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, - Bazel evaluates all the files it loads. If multiple `BUILD` files are loading - foo.bzl, you would see only one occurrence of "bzl file evaluation" because - Bazel caches the result of the evaluation. -* The callback function `_foo_binary_impl` is not called. Bazel query loads - `BUILD` files, but doesn't analyze targets. +- "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, Bazel evaluates all the files it loads. If multiple `BUILD` files are loading foo.bzl, you would see only one occurrence of "bzl file evaluation" because Bazel caches the result of the evaluation. +- The callback function `_foo_binary_impl` is not called. Bazel query loads `BUILD` files, but doesn't analyze targets. -To analyze the targets, use the [`cquery`](/query/cquery) ("configured -query") or the `build` command: +To analyze the targets, use the [`cquery`](/query/cquery) ("configured query") or the `build` command: ``` $ bazel build :all @@ -126,15 +98,11 @@ INFO: Found 2 targets... As you can see, `_foo_binary_impl` is now called twice - once for each target. -Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, -because the evaluation of `foo.bzl` is cached after the call to `bazel query`. -Bazel only emits `print` statements when they are actually executed. +Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, because the evaluation of `foo.bzl` is cached after the call to `bazel query`. Bazel only emits `print` statements when they are actually executed. ## Creating a file -To make your rule more useful, update it to generate a file. First, declare the -file and give it a name. In this example, create a file with the same name as -the target: +To make your rule more useful, update it to generate a file. First, declare the file and give it a name. In this example, create a file with the same name as the target: ```python ctx.actions.declare_file(ctx.label.name) @@ -147,9 +115,7 @@ The following files have no generating action: bin2 ``` -Whenever you declare a file, you have to tell Bazel how to generate it by -creating an action. Use [`ctx.actions.write`](lib/actions#write), -to create a file with the given content. +Whenever you declare a file, you have to tell Bazel how to generate it by creating an action. Use [`ctx.actions.write`](lib/actions#write), to create a file with the given content. ```python def _foo_binary_impl(ctx): @@ -167,11 +133,7 @@ $ bazel build bin1 Target //:bin1 up-to-date (nothing to build) ``` -The `ctx.actions.write` function registered an action, which taught Bazel -how to generate the file. But Bazel won't create the file until it is -actually requested. So the last thing to do is tell Bazel that the file -is an output of the rule, and not a temporary file used within the rule -implementation. +The `ctx.actions.write` function registered an action, which taught Bazel how to generate the file. But Bazel won't create the file until it is actually requested. So the last thing to do is tell Bazel that the file is an output of the rule, and not a temporary file used within the rule implementation. ```python def _foo_binary_impl(ctx): @@ -183,8 +145,7 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Look at the `DefaultInfo` and `depset` functions later. For now, -assume that the last line is the way to choose the outputs of a rule. +Look at the `DefaultInfo` and `depset` functions later. For now, assume that the last line is the way to choose the outputs of a rule. Now, run Bazel: @@ -202,8 +163,7 @@ You have successfully generated a file! ## Attributes -To make the rule more useful, add new attributes using -[the `attr` module](lib/attr) and update the rule definition. +To make the rule more useful, add new attributes using [the `attr` module](lib/attr) and update the rule definition. Add a string attribute called `username`: @@ -225,8 +185,7 @@ foo_binary( ) ``` -To access the value in the callback function, use `ctx.attr.username`. For -example: +To access the value in the callback function, use `ctx.attr.username`. For example: ```python def _foo_binary_impl(ctx): @@ -238,38 +197,23 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Note that you can make the attribute mandatory or set a default value. Look at -the documentation of [`attr.string`](lib/attr#string). -You may also use other types of attributes, such as [boolean](lib/attr#bool) -or [list of integers](lib/attr#int_list). +Note that you can make the attribute mandatory or set a default value. Look at the documentation of [`attr.string`](lib/attr#string). You may also use other types of attributes, such as [boolean](lib/attr#bool) or [list of integers](lib/attr#int_list). ## Dependencies -Dependency attributes, such as [`attr.label`](lib/attr#label) -and [`attr.label_list`](lib/attr#label_list), -declare a dependency from the target that owns the attribute to the target whose -label appears in the attribute's value. This kind of attribute forms the basis -of the target graph. +Dependency attributes, such as [`attr.label`](lib/attr#label) and [`attr.label_list`](lib/attr#label_list), declare a dependency from the target that owns the attribute to the target whose label appears in the attribute's value. This kind of attribute forms the basis of the target graph. -In the `BUILD` file, the target label appears as a string object, such as -`//pkg:name`. In the implementation function, the target will be accessible as a -[`Target`](lib/Target) object. For example, view the files returned -by the target using [`Target.files`](lib/Target#modules.Target.files). +In the `BUILD` file, the target label appears as a string object, such as `//pkg:name`. In the implementation function, the target will be accessible as a [`Target`](lib/Target) object. For example, view the files returned by the target using [`Target.files`](lib/Target#modules.Target.files). ### Multiple files -By default, only targets created by rules may appear as dependencies (such as a -`foo_library()` target). If you want the attribute to accept targets that are -input files (such as source files in the repository), you can do it with -`allow_files` and specify the list of accepted file extensions (or `True` to -allow any file extension): +By default, only targets created by rules may appear as dependencies (such as a `foo_library()` target). If you want the attribute to accept targets that are input files (such as source files in the repository), you can do it with `allow_files` and specify the list of accepted file extensions (or `True` to allow any file extension): ```python "srcs": attr.label_list(allow_files = [".java"]), ``` -The list of files can be accessed with `ctx.files.`. For -example, the list of files in the `srcs` attribute can be accessed through +The list of files can be accessed with `ctx.files.<attribute name>`. For example, the list of files in the `srcs` attribute can be accessed through ```python ctx.files.srcs @@ -283,7 +227,7 @@ If you need only one file, use `allow_single_file`: "src": attr.label(allow_single_file = [".java"]) ``` -This file is then accessible under `ctx.file.`: +This file is then accessible under `ctx.file.<attribute name>`: ```python ctx.file.src @@ -291,17 +235,9 @@ ctx.file.src ## Create a file with a template -You can create a rule that generates a .cc file based on a template. Also, you -can use `ctx.actions.write` to output a string constructed in the rule -implementation function, but this has two problems. First, as the template gets -bigger, it becomes more memory efficient to put it in a separate file and avoid -constructing large strings during the analysis phase. Second, using a separate -file is more convenient for the user. Instead, use -[`ctx.actions.expand_template`](lib/actions#expand_template), -which performs substitutions on a template file. +You can create a rule that generates a .cc file based on a template. Also, you can use `ctx.actions.write` to output a string constructed in the rule implementation function, but this has two problems. First, as the template gets bigger, it becomes more memory efficient to put it in a separate file and avoid constructing large strings during the analysis phase. Second, using a separate file is more convenient for the user. Instead, use [`ctx.actions.expand_template`](lib/actions#expand_template), which performs substitutions on a template file. -Create a `template` attribute to declare a dependency on the template -file: +Create a `template` attribute to declare a dependency on the template file: ```python def _hello_world_impl(ctx): @@ -340,8 +276,7 @@ cc_binary( ) ``` -If you don't want to expose the template to the end-user and always use the -same one, you can set a default value and make the attribute private: +If you don't want to expose the template to the end-user and always use the same one, you can set a default value and make the attribute private: ```python "_template": attr.label( @@ -350,11 +285,7 @@ same one, you can set a default value and make the attribute private: ), ``` -Attributes that start with an underscore are private and cannot be set in a -`BUILD` file. The template is now an _implicit dependency_: Every `hello_world` -target has a dependency on this file. Don't forget to make this file visible -to other packages by updating the `BUILD` file and using -[`exports_files`](/reference/be/functions#exports_files): +Attributes that start with an underscore are private and cannot be set in a `BUILD` file. The template is now an *implicit dependency*: Every `hello_world` target has a dependency on this file. Don't forget to make this file visible to other packages by updating the `BUILD` file and using [`exports_files`](/reference/be/functions#exports_files): ```python exports_files(["file.cc.tpl"]) @@ -362,7 +293,6 @@ exports_files(["file.cc.tpl"]) ## Going further -* Take a look at the [reference documentation for rules](/extending/rules#contents). -* Get familiar with [depsets](/extending/depsets). -* Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) - which includes additional examples of rules. +- Take a look at the [reference documentation for rules](/extending/rules#contents). +- Get familiar with [depsets](/extending/depsets). +- Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) which includes additional examples of rules. diff --git a/rules/testing.mdx b/rules/testing.mdx index 2996e08c..4a191c15 100644 --- a/rules/testing.mdx +++ b/rules/testing.mdx @@ -2,47 +2,23 @@ title: 'Testing' --- - - -There are several different approaches to testing Starlark code in Bazel. This -page gathers the current best practices and frameworks by use case. +There are several different approaches to testing Starlark code in Bazel. This page gathers the current best practices and frameworks by use case. ## Testing rules -[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called -[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) -for checking the analysis-time behavior of rules, such as their actions and -providers. Such tests are called "analysis tests" and are currently the best -option for testing the inner workings of rules. +[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) for checking the analysis-time behavior of rules, such as their actions and providers. Such tests are called "analysis tests" and are currently the best option for testing the inner workings of rules. Some caveats: -* Test assertions occur within the build, not a separate test runner process. - Targets that are created by the test must be named such that they do not - collide with targets from other tests or from the build. An error that - occurs during the test is seen by Bazel as a build breakage rather than a - test failure. - -* It requires a fair amount of boilerplate to set up the rules under test and - the rules containing test assertions. This boilerplate may seem daunting at - first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros - are evaluated and targets generated during the loading phase, while rule - implementation functions don't run until later, during the analysis phase. - -* Analysis tests are intended to be fairly small and lightweight. Certain - features of the analysis testing framework are restricted to verifying - targets with a maximum number of transitive dependencies (currently 500). - This is due to performance implications of using these features with larger - tests. - -The basic principle is to define a testing rule that depends on the -rule-under-test. This gives the testing rule access to the rule-under-test's -providers. - -The testing rule's implementation function carries out assertions. If there are -any failures, these are not raised immediately by calling `fail()` (which would -trigger an analysis-time build error), but rather by storing the errors in a -generated script that fails at test execution time. +- Test assertions occur within the build, not a separate test runner process. Targets that are created by the test must be named such that they do not collide with targets from other tests or from the build. An error that occurs during the test is seen by Bazel as a build breakage rather than a test failure. + +- It requires a fair amount of boilerplate to set up the rules under test and the rules containing test assertions. This boilerplate may seem daunting at first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros are evaluated and targets generated during the loading phase, while rule implementation functions don't run until later, during the analysis phase. + +- Analysis tests are intended to be fairly small and lightweight. Certain features of the analysis testing framework are restricted to verifying targets with a maximum number of transitive dependencies (currently 500). This is due to performance implications of using these features with larger tests. + +The basic principle is to define a testing rule that depends on the rule-under-test. This gives the testing rule access to the rule-under-test's providers. + +The testing rule's implementation function carries out assertions. If there are any failures, these are not raised immediately by calling `fail()` (which would trigger an analysis-time build error), but rather by storing the errors in a generated script that fails at test execution time. See below for a minimal toy example, followed by an example that checks actions. @@ -69,7 +45,6 @@ myrule = rule( `//mypkg/myrules_test.bzl`: - ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest") load(":myrules.bzl", "myrule", "MyInfo") @@ -139,47 +114,31 @@ myrules_test_suite(name = "myrules_test") The test can be run with `bazel test //mypkg:myrules_test`. -Aside from the initial `load()` statements, there are two main parts to the -file: +Aside from the initial `load()` statements, there are two main parts to the file: -* The tests themselves, each of which consists of 1) an analysis-time - implementation function for the testing rule, 2) a declaration of the - testing rule via `analysistest.make()`, and 3) a loading-time function - (macro) for declaring the rule-under-test (and its dependencies) and testing - rule. If the assertions do not change between test cases, 1) and 2) may be - shared by multiple test cases. +- The tests themselves, each of which consists of 1) an analysis-time implementation function for the testing rule, 2) a declaration of the testing rule via `analysistest.make()`, and 3) a loading-time function (macro) for declaring the rule-under-test (and its dependencies) and testing rule. If the assertions do not change between test cases, 1) and 2) may be shared by multiple test cases. -* The test suite function, which calls the loading-time functions for each - test, and declares a `test_suite` target bundling all tests together. +- The test suite function, which calls the loading-time functions for each test, and declares a `test_suite` target bundling all tests together. -For consistency, follow the recommended naming convention: Let `foo` stand for -the part of the test name that describes what the test is checking -(`provider_contents` in the above example). For example, a JUnit test method -would be named `testFoo`. +For consistency, follow the recommended naming convention: Let `foo` stand for the part of the test name that describes what the test is checking (`provider_contents` in the above example). For example, a JUnit test method would be named `testFoo`. Then: -* the macro which generates the test and target under test should should be - named `_test_foo` (`_test_provider_contents`) +- the macro which generates the test and target under test should should be named `_test_foo` (`_test_provider_contents`) -* its test rule type should be named `foo_test` (`provider_contents_test`) +- its test rule type should be named `foo_test` (`provider_contents_test`) -* the label of the target of this rule type should be `foo_test` - (`provider_contents_test`) +- the label of the target of this rule type should be `foo_test` (`provider_contents_test`) -* the implementation function for the testing rule should be named - `_foo_test_impl` (`_provider_contents_test_impl`) +- the implementation function for the testing rule should be named `_foo_test_impl` (`_provider_contents_test_impl`) -* the labels of the targets of the rules under test and their dependencies - should be prefixed with `foo_` (`provider_contents_`) +- the labels of the targets of the rules under test and their dependencies should be prefixed with `foo_` (`provider_contents_`) -Note that the labels of all targets can conflict with other labels in the same -BUILD package, so it's helpful to use a unique name for the test. +Note that the labels of all targets can conflict with other labels in the same BUILD package, so it's helpful to use a unique name for the test. ### Failure testing -It may be useful to verify that a rule fails given certain inputs or in certain -state. This can be done using the analysis test framework: +It may be useful to verify that a rule fails given certain inputs or in certain state. This can be done using the analysis test framework: The test rule created with `analysistest.make` should specify `expect_failure`: @@ -190,8 +149,7 @@ failure_testing_test = analysistest.make( ) ``` -The test rule implementation should make assertions on the nature of the failure -that took place (specifically, the failure message): +The test rule implementation should make assertions on the nature of the failure that took place (specifically, the failure message): ```python def _failure_testing_test_impl(ctx): @@ -200,11 +158,7 @@ def _failure_testing_test_impl(ctx): return analysistest.end(env) ``` -Also make sure that your target under test is specifically tagged 'manual'. -Without this, building all targets in your package using `:all` will result in a -build of the intentionally-failing target and will exhibit a build failure. With -'manual', your target under test will build only if explicitly specified, or as -a dependency of a non-manual target (such as your test rule): +Also make sure that your target under test is specifically tagged 'manual'. Without this, building all targets in your package using `:all` will result in a build of the intentionally-failing target and will exhibit a build failure. With 'manual', your target under test will build only if explicitly specified, or as a dependency of a non-manual target (such as your test rule): ```python def _test_failure(): @@ -219,9 +173,7 @@ def _test_failure(): ### Verifying registered actions -You may want to write tests which make assertions about the actions that your -rule registers, for example, using `ctx.actions.run()`. This can be done in your -analysis test rule implementation function. An example: +You may want to write tests which make assertions about the actions that your rule registers, for example, using `ctx.actions.run()`. This can be done in your analysis test rule implementation function. An example: ```python def _inspect_actions_test_impl(ctx): @@ -236,14 +188,11 @@ def _inspect_actions_test_impl(ctx): return analysistest.end(env) ``` -Note that `analysistest.target_actions(env)` returns a list of -[`Action`](lib/Action) objects which represent actions registered by the -target under test. +Note that `analysistest.target_actions(env)` returns a list of [`Action`](lib/Action) objects which represent actions registered by the target under test. ### Verifying rule behavior under different flags -You may want to verify your real rule behaves a certain way given certain build -flags. For example, your rule may behave differently if a user specifies: +You may want to verify your real rule behaves a certain way given certain build flags. For example, your rule may behave differently if a user specifies: ```shell bazel build //mypkg:real_target -c opt @@ -255,20 +204,15 @@ versus bazel build //mypkg:real_target -c dbg ``` -At first glance, this could be done by testing the target under test using the -desired build flags: +At first glance, this could be done by testing the target under test using the desired build flags: ```shell bazel test //mypkg:myrules_test -c opt ``` -But then it becomes impossible for your test suite to simultaneously contain a -test which verifies the rule behavior under `-c opt` and another test which -verifies the rule behavior under `-c dbg`. Both tests would not be able to run -in the same build! +But then it becomes impossible for your test suite to simultaneously contain a test which verifies the rule behavior under `-c opt` and another test which verifies the rule behavior under `-c dbg`. Both tests would not be able to run in the same build! -This can be solved by specifying the desired build flags when defining the test -rule: +This can be solved by specifying the desired build flags when defining the test rule: ```python myrule_c_opt_test = analysistest.make( @@ -279,33 +223,21 @@ myrule_c_opt_test = analysistest.make( ) ``` -Normally, a target under test is analyzed given the current build flags. -Specifying `config_settings` overrides the values of the specified command line -options. (Any unspecified options will retain their values from the actual -command line). - -In the specified `config_settings` dictionary, command line flags must be -prefixed with a special placeholder value `//command_line_option:`, as is shown -above. +Normally, a target under test is analyzed given the current build flags. Specifying `config_settings` overrides the values of the specified command line options. (Any unspecified options will retain their values from the actual command line). +In the specified `config_settings` dictionary, command line flags must be prefixed with a special placeholder value `//command_line_option:`, as is shown above. ## Validating artifacts The main ways to check that your generated files are correct are: -* You can write a test script in shell, Python, or another language, and - create a target of the appropriate `*_test` rule type. +- You can write a test script in shell, Python, or another language, and create a target of the appropriate `*_test` rule type. -* You can use a specialized rule for the kind of test you want to perform. +- You can use a specialized rule for the kind of test you want to perform. ### Using a test target -The most straightforward way to validate an artifact is to write a script and -add a `*_test` target to your BUILD file. The specific artifacts you want to -check should be data dependencies of this target. If your validation logic is -reusable for multiple tests, it should be a script that takes command line -arguments that are controlled by the test target's `args` attribute. Here's an -example that validates that the output of `myrule` from above is `"abc"`. +The most straightforward way to validate an artifact is to write a script and add a `*_test` target to your BUILD file. The specific artifacts you want to check should be data dependencies of this target. If your validation logic is reusable for multiple tests, it should be a script that takes command line arguments that are controlled by the test target's `args` attribute. Here's an example that validates that the output of `myrule` from above is `"abc"`. `//mypkg/myrule_validator.sh`: @@ -341,12 +273,7 @@ sh_test( ### Using a custom rule -A more complicated alternative is to write the shell script as a template that -gets instantiated by a new rule. This involves more indirection and Starlark -logic, but leads to cleaner BUILD files. As a side-benefit, any argument -preprocessing can be done in Starlark instead of the script, and the script is -slightly more self-documenting since it uses symbolic placeholders (for -substitutions) instead of numeric ones (for arguments). +A more complicated alternative is to write the shell script as a template that gets instantiated by a new rule. This involves more indirection and Starlark logic, but leads to cleaner BUILD files. As a side-benefit, any argument preprocessing can be done in Starlark instead of the script, and the script is slightly more self-documenting since it uses symbolic placeholders (for substitutions) instead of numeric ones (for arguments). `//mypkg/myrule_validator.sh.template`: @@ -417,18 +344,11 @@ myrule_validation_test( ) ``` -Alternatively, instead of using a template expansion action, you could have -inlined the template into the .bzl file as a string and expanded it during the -analysis phase using the `str.format` method or `%`-formatting. +Alternatively, instead of using a template expansion action, you could have inlined the template into the .bzl file as a string and expanded it during the analysis phase using the `str.format` method or `%`-formatting. ## Testing Starlark utilities -[Skylib](https://github.com/bazelbuild/bazel-skylib)'s -[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) -framework can be used to test utility functions (that is, functions that are -neither macros nor rule implementations). Instead of using `unittest.bzl`'s -`analysistest` library, `unittest` may be used. For such test suites, the -convenience function `unittest.suite()` can be used to reduce boilerplate. +[Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) framework can be used to test utility functions (that is, functions that are neither macros nor rule implementations). Instead of using `unittest.bzl`'s `analysistest` library, `unittest` may be used. For such test suites, the convenience function `unittest.suite()` can be used to reduce boilerplate. `//mypkg/myhelpers.bzl`: @@ -439,7 +359,6 @@ def myhelper(): `//mypkg/myhelpers_test.bzl`: - ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":myhelpers.bzl", "myhelper") diff --git a/rules/verbs-tutorial.mdx b/rules/verbs-tutorial.mdx index db7757e1..4cd6c990 100644 --- a/rules/verbs-tutorial.mdx +++ b/rules/verbs-tutorial.mdx @@ -2,29 +2,17 @@ title: 'Using Macros to Create Custom Verbs' --- - - -Day-to-day interaction with Bazel happens primarily through a few commands: -`build`, `test`, and `run`. At times, though, these can feel limited: you may -want to push packages to a repository, publish documentation for end-users, or -deploy an application with Kubernetes. But Bazel doesn't have a `publish` or -`deploy` command – where do these actions fit in? +Day-to-day interaction with Bazel happens primarily through a few commands: `build`, `test`, and `run`. At times, though, these can feel limited: you may want to push packages to a repository, publish documentation for end-users, or deploy an application with Kubernetes. But Bazel doesn't have a `publish` or `deploy` command – where do these actions fit in? ## The bazel run command -Bazel's focus on hermeticity, reproducibility, and incrementality means the -`build` and `test` commands aren't helpful for the above tasks. These actions -may run in a sandbox, with limited network access, and aren't guaranteed to be -re-run with every `bazel build`. +Bazel's focus on hermeticity, reproducibility, and incrementality means the `build` and `test` commands aren't helpful for the above tasks. These actions may run in a sandbox, with limited network access, and aren't guaranteed to be re-run with every `bazel build`. + +Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have side effects. Bazel users are accustomed to rules that create executables, and rule authors can follow a common set of patterns to extend this to "custom verbs". -Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have -side effects. Bazel users are accustomed to rules that create executables, and -rule authors can follow a common set of patterns to extend this to -"custom verbs". +### In the wild: rules\_k8s -### In the wild: rules_k8s -For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), -the Kubernetes rules for Bazel. Suppose you have the following target: +For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), the Kubernetes rules for Bazel. Suppose you have the following target: ```python # BUILD file in //application/k8s @@ -36,51 +24,21 @@ k8s_object( ) ``` -The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a -standard Kubernetes YAML file when `bazel build` is used on the `staging` -target. However, the additional targets are also created by the `k8s_object` -macro with names like `staging.apply` and `:staging.delete`. These build -scripts to perform those actions, and when executed with `bazel run -staging.apply`, these behave like our own `bazel k8s-apply` or `bazel -k8s-delete` commands. - -### Another example: ts_api_guardian_test - -This pattern can also be seen in the Angular project. The -[`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) -produces two targets. The first is a standard `nodejs_test` target which compares -some generated output against a "golden" file (that is, a file containing the -expected output). This can be built and run with a normal `bazel -test` invocation. In `angular-cli`, you can run [one such -target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) -with `bazel test //etc/api:angular_devkit_core_api`. - -Over time, this golden file may need to be updated for legitimate reasons. -Updating this manually is tedious and error-prone, so this macro also provides -a `nodejs_binary` target that updates the golden file, instead of comparing -against it. Effectively, the same test script can be written to run in "verify" -or "accept" mode, based on how it's invoked. This follows the same pattern -you've learned already: there is no native `bazel test-accept` command, but the -same effect can be achieved with -`bazel run //etc/api:angular_devkit_core_api.accept`. - -This pattern can be quite powerful, and turns out to be quite common once you -learn to recognize it. +The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a standard Kubernetes YAML file when `bazel build` is used on the `staging` target. However, the additional targets are also created by the `k8s_object` macro with names like `staging.apply` and `:staging.delete`. These build scripts to perform those actions, and when executed with `bazel run staging.apply`, these behave like our own `bazel k8s-apply` or `bazel k8s-delete` commands. + +### Another example: ts\_api\_guardian\_test + +This pattern can also be seen in the Angular project. The [`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) produces two targets. The first is a standard `nodejs_test` target which compares some generated output against a "golden" file (that is, a file containing the expected output). This can be built and run with a normal `bazel test` invocation. In `angular-cli`, you can run [one such target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) with `bazel test //etc/api:angular_devkit_core_api`. + +Over time, this golden file may need to be updated for legitimate reasons. Updating this manually is tedious and error-prone, so this macro also provides a `nodejs_binary` target that updates the golden file, instead of comparing against it. Effectively, the same test script can be written to run in "verify" or "accept" mode, based on how it's invoked. This follows the same pattern you've learned already: there is no native `bazel test-accept` command, but the same effect can be achieved with `bazel run //etc/api:angular_devkit_core_api.accept`. + +This pattern can be quite powerful, and turns out to be quite common once you learn to recognize it. ## Adapting your own rules -[Macros](/extending/macros) are the heart of this pattern. Macros are used like -rules, but they can create several targets. Typically, they will create a -target with the specified name which performs the primary build action: perhaps -it builds a normal binary, a Docker image, or an archive of source code. In -this pattern, additional targets are created to produce scripts performing side -effects based on the output of the primary target, like publishing the -resulting binary or updating the expected test output. +[Macros](/extending/macros) are the heart of this pattern. Macros are used like rules, but they can create several targets. Typically, they will create a target with the specified name which performs the primary build action: perhaps it builds a normal binary, a Docker image, or an archive of source code. In this pattern, additional targets are created to produce scripts performing side effects based on the output of the primary target, like publishing the resulting binary or updating the expected test output. -To illustrate this, wrap an imaginary rule that generates a website with -[Sphinx](https://www.sphinx-doc.org) with a macro to create an additional -target that allows the user to publish it when ready. Consider the following -existing rule for generating a website with Sphinx: +To illustrate this, wrap an imaginary rule that generates a website with [Sphinx](https://www.sphinx-doc.org) with a macro to create an additional target that allows the user to publish it when ready. Consider the following existing rule for generating a website with Sphinx: ```python _sphinx_site = rule( @@ -89,8 +47,7 @@ _sphinx_site = rule( ) ``` -Next, consider a rule like the following, which builds a script that, when run, -publishes the generated pages: +Next, consider a rule like the following, which builds a script that, when run, publishes the generated pages: ```python _sphinx_publisher = rule( @@ -106,8 +63,7 @@ _sphinx_publisher = rule( ) ``` -Finally, define the following symbolic macro (available in Bazel 8 or newer) to -create targets for both of the above rules together: +Finally, define the following symbolic macro (available in Bazel 8 or newer) to create targets for both of the above rules together: ```starlark def _sphinx_site_impl(name, visibility, srcs, **kwargs): @@ -128,8 +84,7 @@ sphinx_site = macro( ) ``` -Or, if you need to support Bazel releases older than Bazel 8, you would instead -define a legacy macro: +Or, if you need to support Bazel releases older than Bazel 8, you would instead define a legacy macro: ```starlark def sphinx_site(name, srcs = [], **kwargs): @@ -140,8 +95,7 @@ def sphinx_site(name, srcs = [], **kwargs): _sphinx_publisher(name = "%s.publish" % name, site = name, **kwargs) ``` -In the `BUILD` files, use the macro as though it just creates the primary -target: +In the `BUILD` files, use the macro as though it just creates the primary target: ```python sphinx_site( @@ -150,28 +104,8 @@ sphinx_site( ) ``` -In this example, a "docs" target is created, just as though the macro were a -standard, single Bazel rule. When built, the rule generates some configuration -and runs Sphinx to produce an HTML site, ready for manual inspection. However, -an additional "docs.publish" target is also created, which builds a script for -publishing the site. Once you check the output of the primary target, you can -use `bazel run :docs.publish` to publish it for public consumption, just like -an imaginary `bazel publish` command. - -It's not immediately obvious what the implementation of the `_sphinx_publisher` -rule might look like. Often, actions like this write a _launcher_ shell script. -This method typically involves using -[`ctx.actions.expand_template`](lib/actions#expand_template) -to write a very simple shell script, in this case invoking the publisher binary -with a path to the output of the primary target. This way, the publisher -implementation can remain generic, the `_sphinx_site` rule can just produce -HTML, and this small script is all that's necessary to combine the two -together. - -In `rules_k8s`, this is indeed what `.apply` does: -[`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) -writes a very simple Bash script, based on -[`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), -which runs `kubectl` with the output of the primary target. This script can -then be build and run with `bazel run :staging.apply`, effectively providing a -`k8s-apply` command for `k8s_object` targets. +In this example, a "docs" target is created, just as though the macro were a standard, single Bazel rule. When built, the rule generates some configuration and runs Sphinx to produce an HTML site, ready for manual inspection. However, an additional "docs.publish" target is also created, which builds a script for publishing the site. Once you check the output of the primary target, you can use `bazel run :docs.publish` to publish it for public consumption, just like an imaginary `bazel publish` command. + +It's not immediately obvious what the implementation of the `_sphinx_publisher` rule might look like. Often, actions like this write a *launcher* shell script. This method typically involves using [`ctx.actions.expand_template`](lib/actions#expand_template) to write a very simple shell script, in this case invoking the publisher binary with a path to the output of the primary target. This way, the publisher implementation can remain generic, the `_sphinx_site` rule can just produce HTML, and this small script is all that's necessary to combine the two together. + +In `rules_k8s`, this is indeed what `.apply` does: [`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) writes a very simple Bash script, based on [`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), which runs `kubectl` with the output of the primary target. This script can then be build and run with `bazel run :staging.apply`, effectively providing a `k8s-apply` command for `k8s_object` targets. diff --git a/rules/windows.mdx b/rules/windows.mdx new file mode 100644 index 00000000..f6422687 --- /dev/null +++ b/rules/windows.mdx @@ -0,0 +1,189 @@ +--- +title: 'Writing Rules on Windows' +--- + +This page focuses on writing Windows-compatible rules, common problems of writing portable rules, and some solutions. + +## Paths + +Problems: + +- **Length limit**: maximum path length is 259 characters. + + Though Windows also supports longer paths (up to 32767 characters), many programs are built with the lower limit. + + Be aware of this about programs you run in the actions. + +- **Working directory**: is also limited to 259 characters. + + Processes cannot `cd` into a directory longer than 259 characters. + +- **Case-sensitivity**: Windows paths are case-insensitive, Unix paths are case-sensitive. + + Be aware of this when creating command lines for actions. + +- **Path separators**: are backslash (`\`), not forward slash (`/`). + + Bazel stores paths Unix-style with `/` separators. Though some Windows programs support Unix-style paths, others don't. Some built-in commands in cmd.exe support them, some don't. + + It's best to always use `\` separators on Windows: replace `/` with `\` when you create command lines and environment variables for actions. + +- **Absolute paths**: don't start with slash (`/`). + + Absolute paths on Windows start with a drive letter, such as `C:\foo\bar.txt`. There's no single filesystem root. + + Be aware of this if your rule checks if a path is absolute. Absolute paths should be avoided since they are often non-portable. + +Solutions: + +- **Keep paths short.** + + Avoid long directory names, deeply nested directory structures, long file names, long workspace names, long target names. + + All of these may become path components of actions' input files, and may exhaust the path length limit. + +- **Use a short output root.** + + Use the `--output_user_root=<path>` flag to specify a short path for Bazel outputs. A good idea is to have a drive (or virtual drive) just for Bazel outputs (such as `D:\`), and adding this line to your `.bazelrc` file: + + ``` + build --output_user_root=D:/ + ``` + + or + + ``` + build --output_user_root=C:/_bzl + ``` + +- **Use junctions.** + + Junctions are, loosely speaking\[1], directory symlinks. Junctions are easy to create and can point to directories (on the same computer) with long paths. If a build action creates a junction whose path is short but whose target is long, then tools with short path limit can access the files in the junction'ed directory. + + In `.bat` files or in cmd.exe you can create junctions like so: + + ``` + mklink /J c:\path\to\junction c:\path\to\very\long\target\path + ``` + + \[1]: Strictly speaking [Junctions are not Symbolic Links](https://superuser.com/a/343079), but for the sake of build actions you may regard Junctions as Directory Symlinks. + +- **Replace `/` with `\` in paths in actions / envvars.** + + When you create the command line or environment variables for an action, make the paths Windows-style. Example: + + ```python + def as_path(p, is_windows): + if is_windows: + return p.replace("/", "\\") + else: + return p + ``` + +## Environment variables + +Problems: + +- **Case-sensitivity**: Windows environment variable names are case-insensitive. + + For example, in Java `System.getenv("SystemRoot")` and `System.getenv("SYSTEMROOT")` yields the same result. (This applies to other languages too.) + +- **Hermeticity**: actions should use as few custom environment variables as possible. + + Environment variables are part of the action's cache key. If an action uses environment variables that change often, or are custom to users, that makes the rule less cache-able. + +Solutions: + +- **Only use upper-case environment variable names.** + + This works on Windows, macOS, and Linux. + +- **Minimize action environments.** + + When using `ctx.actions.run`, set the environment to `ctx.configuration.default_shell_env`. If the action needs more environment variables, put them all in a dictionary and pass that to the action. Example: + + ```python + load("@bazel_skylib//lib:dicts.bzl", "dicts") + + def _make_env(ctx, output_file, is_windows): + out_path = output_file.path + if is_windows: + out_path = out_path.replace("/", "\\") + return dicts.add(ctx.configuration.default_shell_env, {"MY_OUTPUT": out_path}) + ``` + +## Actions + +Problems: + +- **Executable outputs**: Every executable file must have an executable extension. + + The most common extensions are `.exe` (binary files) and `.bat` (Batch scripts). + + Be aware that shell scripts (`.sh`) are NOT executable on Windows; you cannot specify them as `ctx.actions.run`'s `executable`. There's also no `+x` permission that files can have, so you can't execute arbitrary files like on Linux. + +- **Bash commands**: For sake of portability, avoid running Bash commands directly in actions. + + Bash is widespread on Unix-like systems, but it's often unavailable on Windows. Bazel itself is relying less and less on Bash (MSYS2), so in the future users would be less likely to have MSYS2 installed along with Bazel. To make rules easier to use on Windows, avoid running Bash commands in actions. + +- **Line endings**: Windows uses CRLF (`\r\n`), Unix-like systems uses LF (`\n`). + + Be aware of this when comparing text files. Be mindful of your Git settings, especially of line endings when checking out or committing. (See Git's `core.autocrlf` setting.) + +Solutions: + +- **Use a Bash-less purpose-made rule.** + + `native.genrule()` is a wrapper for Bash commands, and it's often used to solve simple problems like copying a file or writing a text file. You can avoid relying on Bash (and reinventing the wheel): see if bazel-skylib has a purpose-made rule for your needs. None of them depends on Bash when built/tested on Windows. + + Build rule examples: + + - `copy_file()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/copy_file_doc.md)): copies a file somewhere else, optionally making it executable + + - `write_file()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/write_file_doc.md)): writes a text file, with the desired line endings (`auto`, `unix`, or `windows`), optionally making it executable (if it's a script) + + - `run_binary()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/run_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/run_binary_doc.md)): runs a binary (or `*_binary` rule) with given inputs and expected outputs as a build action (this is a build rule wrapper for `ctx.actions.run`) + + - `native_binary()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/native_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/native_binary_doc.md#native_binary)): wraps a native binary in a `*_binary` rule, which you can `bazel run` or use in `run_binary()`'s `tool` attribute or `native.genrule()`'s `tools` attribute + + Test rule examples: + + - `diff_test()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/diff_test.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/diff_test_doc.md)): test that compares contents of two files + + - `native_test()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/native_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/native_binary_doc.md#native_test)): wraps a native binary in a `*_test` rule, which you can `bazel test` + +- **On Windows, consider using `.bat` scripts for trivial things.** + + Instead of `.sh` scripts, you can solve trivial tasks with `.bat` scripts. + + For example, if you need a script that does nothing, or prints a message, or exits with a fixed error code, then a simple `.bat` file will suffice. If your rule returns a `DefaultInfo()` provider, the `executable` field may refer to that `.bat` file on Windows. + + And since file extensions don't matter on macOS and Linux, you can always use `.bat` as the extension, even for shell scripts. + + Be aware that empty `.bat` files cannot be executed. If you need an empty script, write one space in it. + +- **Use Bash in a principled way.** + + In Starlark build and test rules, use `ctx.actions.run_shell` to run Bash scripts and Bash commands as actions. + + In Starlark macros, wrap Bash scripts and commands in a `native.sh_binary()` or `native.genrule()`. Bazel will check if Bash is available and run the script or command through Bash. + + In Starlark repository rules, try avoiding Bash altogether. Bazel currently offers no way to run Bash commands in a principled way in repository rules. + +## Deleting files + +Problems: + +- **Files cannot be deleted while open.** + + Open files cannot be deleted (by default), attempts result in "Access Denied" errors. If you cannot delete a file, maybe a running process still holds it open. + +- **Working directory of a running process cannot be deleted.** + + Processes have an open handle to their working directory, and the directory cannot be deleted until the process terminates. + +Solutions: + +- **In your code, try to close files eagerly.** + + In Java, use `try-with-resources`. In Python, use `with open(...) as f:`. In principle, try closing handles as soon as possible. diff --git a/run/bazelrc.mdx b/run/bazelrc.mdx index 90d4464a..2ca427d7 100644 --- a/run/bazelrc.mdx +++ b/run/bazelrc.mdx @@ -2,129 +2,85 @@ title: 'Write bazelrc configuration files' --- - - -Bazel accepts many options. Some options are varied frequently (for example, -`--subcommands`) while others stay the same across several builds (such as -`--package_path`). To avoid specifying these unchanged options for every build -(and other commands), you can specify options in a configuration file, called -`.bazelrc`. +Bazel accepts many options. Some options are varied frequently (for example, `--subcommands`) while others stay the same across several builds (such as `--package_path`). To avoid specifying these unchanged options for every build (and other commands), you can specify options in a configuration file, called `.bazelrc`. ### Where are the `.bazelrc` files? -Bazel looks for optional configuration files in the following locations, -in the order shown below. The options are interpreted in this order, so -options in later files can override a value from an earlier file if a -conflict arises. All options that control which of these files are loaded are -startup options, which means they must occur after `bazel` and -before the command (`build`, `test`, etc). - -1. **The system RC file**, unless `--nosystem_rc` is present. +Bazel looks for optional configuration files in the following locations, in the order shown below. The options are interpreted in this order, so options in later files can override a value from an earlier file if a conflict arises. All options that control which of these files are loaded are startup options, which means they must occur after `bazel` and before the command (`build`, `test`, etc). - Path: +1. **The system RC file**, unless `--nosystem_rc` is present. - - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` - - On Windows: `%ProgramData%\bazel.bazelrc` + Path: - It is not an error if this file does not exist. + - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` + - On Windows: `%ProgramData%\bazel.bazelrc` - If another system-specified location is required, you must build a custom - Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in - [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). - The system-specified location may contain environment variable references, - such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. + It is not an error if this file does not exist. -2. **The workspace RC file**, unless `--noworkspace_rc` is present. + If another system-specified location is required, you must build a custom Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). The system-specified location may contain environment variable references, such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. - Path: `.bazelrc` in your workspace directory (next to the main - `MODULE.bazel` file). +2. **The workspace RC file**, unless `--noworkspace_rc` is present. - It is not an error if this file does not exist. + Path: `.bazelrc` in your workspace directory (next to the main `MODULE.bazel` file). -3. **The home RC file**, unless `--nohome_rc` is present. + It is not an error if this file does not exist. - Path: +3. **The home RC file**, unless `--nohome_rc` is present. - - On Linux/macOS/Unixes: `$HOME/.bazelrc` - - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` + Path: - It is not an error if this file does not exist. + - On Linux/macOS/Unixes: `$HOME/.bazelrc` + - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` -4. **The environment variable RC file**, if its path is set with the `BAZELRC` - environment variable. + It is not an error if this file does not exist. - The environment variable can include multiple comma-separated paths. +4. **The environment variable RC file**, if its path is set with the `BAZELRC` environment variable. -5. **The user-specified RC file**, if specified with - --bazelrc=file + The environment variable can include multiple comma-separated paths. - This flag is optional but can also be specified multiple times. +5. **The user-specified RC file**, if specified with `--bazelrc=file` - `/dev/null` indicates that all further `--bazelrc`s will be ignored, which - is useful to disable the search for a user rc file, such as in release - builds. + This flag is optional but can also be specified multiple times. - For example: + `/dev/null` indicates that all further `--bazelrc`s will be ignored, which is useful to disable the search for a user rc file, such as in release builds. - ``` - --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc - ``` + For example: - - `x.rc` and `y.rc` are read. - - `z.rc` is ignored due to the prior `/dev/null`. + ``` + --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc + ``` -In addition to this optional configuration file, Bazel looks for a global rc -file. For more details, see the [global bazelrc section](#global-bazelrc). + - `x.rc` and `y.rc` are read. + - `z.rc` is ignored due to the prior `/dev/null`. +In addition to this optional configuration file, Bazel looks for a global rc file. For more details, see the [global bazelrc section](#global-bazelrc). ### `.bazelrc` syntax and semantics -Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based -grammar. Empty lines and lines starting with `#` (comments) are ignored. Each -line contains a sequence of words, which are tokenized according to the same -rules as the Bourne shell. +Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based grammar. Empty lines and lines starting with `#` (comments) are ignored. Each line contains a sequence of words, which are tokenized according to the same rules as the Bourne shell. #### Imports -Lines that start with `import` or `try-import` are special: use these to load -other "rc" files. To specify a path that is relative to the workspace root, -write `import %workspace%/path/to/bazelrc`. +Lines that start with `import` or `try-import` are special: use these to load other "rc" files. To specify a path that is relative to the workspace root, write `import %workspace%/path/to/bazelrc`. -The difference between `import` and `try-import` is that Bazel fails if the -`import`'ed file is missing (or can't be read), but not so for a `try-import`'ed -file. +The difference between `import` and `try-import` is that Bazel fails if the `import`'ed file is missing (or can't be read), but not so for a `try-import`'ed file. Import precedence: -- Options in the imported file take precedence over options specified before - the import statement. -- Options specified after the import statement take precedence over the - options in the imported file. -- Options in files imported later take precedence over files imported earlier. +- Options in the imported file take precedence over options specified before the import statement. +- Options specified after the import statement take precedence over the options in the imported file. +- Options in files imported later take precedence over files imported earlier. #### Option defaults -Most lines of a bazelrc define default option values. The first word on each -line specifies when these defaults are applied: - -- `startup`: startup options, which go before the command, and are described - in `bazel help startup_options`. -- `common`: options that should be applied to all Bazel commands that support - them. If a command does not support an option specified in this way, the - option is ignored so long as it is valid for *some* other Bazel command. - Note that this only applies to option names: If the current command accepts - an option with the specified name, but doesn't support the specified value, - it will fail. -- `always`: options that apply to all Bazel commands. If a command does not - support an option specified in this way, it will fail. -- _`command`_: Bazel command, such as `build` or `query` to which the options - apply. These options also apply to all commands that inherit from the - specified command. (For example, `test` inherits from `build`.) - -Each of these lines may be used more than once and the arguments that follow the -first word are combined as if they had appeared on a single line. (Users of CVS, -another tool with a "Swiss army knife" command-line interface, will find the -syntax similar to that of `.cvsrc`.) For example, the lines: +Most lines of a bazelrc define default option values. The first word on each line specifies when these defaults are applied: + +- `startup`: startup options, which go before the command, and are described in `bazel help startup_options`. +- `common`: options that should be applied to all Bazel commands that support them. If a command does not support an option specified in this way, the option is ignored so long as it is valid for *some* other Bazel command. Note that this only applies to option names: If the current command accepts an option with the specified name, but doesn't support the specified value, it will fail. +- `always`: options that apply to all Bazel commands. If a command does not support an option specified in this way, it will fail. +- *`command`*: Bazel command, such as `build` or `query` to which the options apply. These options also apply to all commands that inherit from the specified command. (For example, `test` inherits from `build`.) + +Each of these lines may be used more than once and the arguments that follow the first word are combined as if they had appeared on a single line. (Users of CVS, another tool with a "Swiss army knife" command-line interface, will find the syntax similar to that of `.cvsrc`.) For example, the lines: ```posix-terminal build --test_tmpdir=/tmp/foo --verbose_failures @@ -142,89 +98,49 @@ so the effective flags are `--verbose_failures` and `--test_tmpdir=/tmp/bar`. Option precedence: -- Options on the command line always take precedence over those in rc files. - For example, if a rc file says `build -c opt` but the command line flag is - `-c dbg`, the command line flag takes precedence. -- Within the rc file, precedence is governed by specificity: lines for a more - specific command take precedence over lines for a less specific command. +- Options on the command line always take precedence over those in rc files. For example, if a rc file says `build -c opt` but the command line flag is `-c dbg`, the command line flag takes precedence. - Specificity is defined by inheritance. Some commands inherit options from - other commands, making the inheriting command more specific than the base - command. For example `test` inherits from the `build` command, so all `bazel - build` flags are valid for `bazel test`, and all `build` lines apply also to - `bazel test` unless there's a `test` line for the same option. If the rc - file says: +- Within the rc file, precedence is governed by specificity: lines for a more specific command take precedence over lines for a less specific command. - ```posix-terminal - test -c dbg --test_env=PATH + Specificity is defined by inheritance. Some commands inherit options from other commands, making the inheriting command more specific than the base command. For example `test` inherits from the `build` command, so all `bazel build` flags are valid for `bazel test`, and all `build` lines apply also to `bazel test` unless there's a `test` line for the same option. If the rc file says: - build -c opt --verbose_failures - ``` + ```posix-terminal + test -c dbg --test_env=PATH - then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel - test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. + build -c opt --verbose_failures + ``` - The inheritance (specificity) graph is: + then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. - * Every command inherits from `common` - * The following commands inherit from (and are more specific than) - `build`: `test`, `run`, `clean`, `mobile-install`, `info`, - `print_action`, `config`, `cquery`, and `aquery` - * `coverage`, `fetch`, and `vendor` inherit from `test` + The inheritance (specificity) graph is: -- Two lines specifying options for the same command at equal specificity are - parsed in the order in which they appear within the file. + - Every command inherits from `common` + - The following commands inherit from (and are more specific than) `build`: `test`, `run`, `clean`, `mobile-install`, `info`, `print_action`, `config`, `cquery`, and `aquery` + - `coverage`, `fetch`, and `vendor` inherit from `test` -- Because this precedence rule does not match the file order, it helps - readability if you follow the precedence order within rc files: start with - `common` options at the top, and end with the most-specific commands at the - bottom of the file. This way, the order in which the options are read is the - same as the order in which they are applied, which is more intuitive. +- Two lines specifying options for the same command at equal specificity are parsed in the order in which they appear within the file. -The arguments specified on a line of an rc file may include arguments that are -not options, such as the names of build targets, and so on. These, like the -options specified in the same files, have lower precedence than their siblings -on the command line, and are always prepended to the explicit list of non- -option arguments. +- Because this precedence rule does not match the file order, it helps readability if you follow the precedence order within rc files: start with `common` options at the top, and end with the most-specific commands at the bottom of the file. This way, the order in which the options are read is the same as the order in which they are applied, which is more intuitive. + +The arguments specified on a line of an rc file may include arguments that are not options, such as the names of build targets, and so on. These, like the options specified in the same files, have lower precedence than their siblings on the command line, and are always prepended to the explicit list of non- option arguments. #### `--config` -In addition to setting option defaults, the rc file can be used to group options -and provide a shorthand for common groupings. This is done by adding a `:name` -suffix to the command. These options are ignored by default, but will be -included when the option --config=name is present, -either on the command line or in a `.bazelrc` file, recursively, even inside of -another config definition. The options specified by `command:name` will only be -expanded for applicable commands, in the precedence order described above. - -Note: Configs can be defined in any `.bazelrc` file, and that all lines of -the form `command:name` (for applicable commands) will be expanded, across the -different rc files. In order to avoid name conflicts, we suggest that configs -defined in personal rc files start with an underscore (`_`) to avoid -unintentional name sharing. - -`--config=foo` expands to the options defined in -[the rc files](#bazelrc-file-locations) "in-place" so that the options -specified for the config have the same precedence that the `--config=foo` option -had. - -This syntax does not extend to the use of `startup` to set -[startup options](#option-defaults). Setting -`startup:config-name --some_startup_option` in the .bazelrc will be ignored. +In addition to setting option defaults, the rc file can be used to group options and provide a shorthand for common groupings. This is done by adding a `:name` suffix to the command. These options are ignored by default, but will be included when the option `--config=name` is present, either on the command line or in a `.bazelrc` file, recursively, even inside of another config definition. The options specified by `command:name` will only be expanded for applicable commands, in the precedence order described above. + +Note: Configs can be defined in any `.bazelrc` file, and that all lines of the form `command:name` (for applicable commands) will be expanded, across the different rc files. In order to avoid name conflicts, we suggest that configs defined in personal rc files start with an underscore (`_`) to avoid unintentional name sharing. + +`--config=foo` expands to the options defined in [the rc files](#bazelrc-file-locations) "in-place" so that the options specified for the config have the same precedence that the `--config=foo` option had. + +This syntax does not extend to the use of `startup` to set [startup options](#option-defaults). Setting `startup:config-name --some_startup_option` in the .bazelrc will be ignored. #### `--enable_platform_specific_config` -In the `.bazelrc` you can use platform specific configs that will be -automatically enabled based on the host OS. For example, if the host OS -is Linux and the `build` command is run, the `build:linux` configuration -will be automatically enabled. Supported OS identifiers are `linux`, -`macos`, `windows`, `freebsd`, and `openbsd`. +In the `.bazelrc` you can use platform specific configs that will be automatically enabled based on the host OS. For example, if the host OS is Linux and the `build` command is run, the `build:linux` configuration will be automatically enabled. Supported OS identifiers are `linux`, `macos`, `windows`, `freebsd`, and `openbsd`. -This is equivalent to using `--config=linux` on Linux, -`--config=windows` on Windows, and so on. This can be disabled with -`--enable_platform_specific_config=false`. +This is equivalent to using `--config=linux` on Linux, `--config=windows` on Windows, and so on. This can be disabled with `--enable_platform_specific_config=false`. -See [--enable_platform_specific_config](/reference/command-line-reference#flag--enable_platform_specific_config). +See [--enable\_platform\_specific\_config](/reference/command-line-reference#flag--enable_platform_specific_config). #### Example @@ -247,27 +163,16 @@ build:memcheck --strip=never --test_timeout=3600 #### `.bazelignore` -You can specify directories within the workspace -that you want Bazel to ignore, such as related projects -that use other build systems. Place a file called -`.bazelignore` at the root of the workspace -and add the directories you want Bazel to ignore, one per -line. Entries are relative to the workspace root. +You can specify directories within the workspace that you want Bazel to ignore, such as related projects that use other build systems. Place a file called `.bazelignore` at the root of the workspace and add the directories you want Bazel to ignore, one per line. Entries are relative to the workspace root. -The `.bazelignore` file does not permit glob semantics. -Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. -It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. -See [#24203](https://github.com/bazelbuild/bazel/pull/24203). +The `.bazelignore` file does not permit glob semantics. Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. See [#24203](https://github.com/bazelbuild/bazel/pull/24203). ### The global bazelrc file Bazel reads optional bazelrc files in this order: -1. System rc-file located at `/etc/bazel.bazelrc`. -2. Workspace rc-file located at `$workspace/tools/bazel.rc`. -3. Home rc-file located at `$HOME/.bazelrc` +1. System rc-file located at `/etc/bazel.bazelrc`. +2. Workspace rc-file located at `$workspace/tools/bazel.rc`. +3. Home rc-file located at `$HOME/.bazelrc` -Each bazelrc file listed here has a corresponding flag which can be used to -disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can -also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` -startup option. +Each bazelrc file listed here has a corresponding flag which can be used to disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` startup option. diff --git a/run/build.mdx b/run/build.mdx new file mode 100644 index 00000000..30dca42c --- /dev/null +++ b/run/build.mdx @@ -0,0 +1,369 @@ +--- +title: 'Build programs with Bazel' +--- + +This page covers how to build a program with Bazel, build command syntax, and target pattern syntax. + +## Quickstart + +To run Bazel, go to your base [workspace](/concepts/build-ref#workspace) directory or any of its subdirectories and type `bazel`. See [build](#bazel-build) if you need to make a new workspace. + +```posix-terminal +bazel help + [Bazel release bazel <var>version</var>] +Usage: bazel <var>command</var> <var>options</var> ... +``` + +### Available commands + +- [`analyze-profile`](/docs/user-manual#analyze-profile): Analyzes build profile data. +- [`aquery`](/docs/user-manual#aquery): Executes a query on the [post-analysis](#analysis) action graph. +- [`build`](#bazel-build): Builds the specified targets. +- [`canonicalize-flags`](/docs/user-manual#canonicalize-flags): Canonicalize Bazel flags. +- [`clean`](/docs/user-manual#clean): Removes output files and optionally stops the server. +- [`cquery`](/query/cquery): Executes a [post-analysis](#analysis) dependency graph query. +- [`dump`](/docs/user-manual#dump): Dumps the internal state of the Bazel server process. +- [`help`](/docs/user-manual#help): Prints help for commands, or the index. +- [`info`](/docs/user-manual#info): Displays runtime info about the bazel server. +- [`fetch`](#fetching-external-dependencies): Fetches all external dependencies of a target. +- [`mobile-install`](/docs/user-manual#mobile-install): Installs apps on mobile devices. +- [`query`](/query/guide): Executes a dependency graph query. +- [`run`](/docs/user-manual#running-executables): Runs the specified target. +- [`shutdown`](/docs/user-manual#shutdown): Stops the Bazel server. +- [`test`](/docs/user-manual#running-tests): Builds and runs the specified test targets. +- [`version`](/docs/user-manual#version): Prints version information for Bazel. + +### Getting help + +- `bazel help <var>command</var>`: Prints help and options for `<var>command</var>`. +- `bazel help `[`startup_options`](/docs/user-manual#startup-options): Options for the JVM hosting Bazel. +- `bazel help `[`target-syntax`](#specifying-build-targets): Explains the syntax for specifying targets. +- `bazel help info-keys`: Displays a list of keys used by the info command. + +The `bazel` tool performs many functions, called commands. The most commonly used ones are `bazel build` and `bazel test`. You can browse the online help messages using `bazel help`. + +### Building one target + +Before you can start a build, you need a *workspace*. A workspace is a directory tree that contains all the source files needed to build your application. Bazel allows you to perform a build from a completely read-only volume. + +To build a program with Bazel, type `bazel build` followed by the [target](#specifying-build-targets) you want to build. + +```posix-terminal +bazel build //foo +``` + +After issuing the command to build `//foo`, you'll see output similar to this: + +``` +INFO: Analyzed target //foo:foo (14 packages loaded, 48 targets configured). +INFO: Found 1 target... +Target //foo:foo up-to-date: + bazel-bin/foo/foo +INFO: Elapsed time: 9.905s, Critical Path: 3.25s +INFO: Build completed successfully, 6 total actions +``` + +First, Bazel **loads** all packages in your target's dependency graph. This includes *declared dependencies*, files listed directly in the target's `BUILD` file, and *transitive dependencies*, files listed in the `BUILD` files of your target's dependencies. After identifying all dependencies, Bazel **analyzes** them for correctness and creates the *build actions*. Last, Bazel **executes** the compilers and other tools of the build. + +During the build's execution phase, Bazel prints progress messages. The progress messages include the current build step (such as, compiler or linker) as it starts, and the number completed over the total number of build actions. As the build starts, the number of total actions often increases as Bazel discovers the entire action graph, but the number stabilizes within a few seconds. + +At the end of the build, Bazel prints which targets were requested, whether or not they were successfully built, and if so, where the output files can be found. Scripts that run builds can reliably parse this output; see [`--show_result`](/docs/user-manual#show-result) for more details. + +If you type the same command again, the build finishes much faster. + +```posix-terminal +bazel build //foo +INFO: Analyzed target //foo:foo (0 packages loaded, 0 targets configured). +INFO: Found 1 target... +Target //foo:foo up-to-date: + bazel-bin/foo/foo +INFO: Elapsed time: 0.144s, Critical Path: 0.00s +INFO: Build completed successfully, 1 total action +``` + +This is a *null build*. Because nothing changed, there are no packages to reload and no build steps to execute. If something changed in 'foo' or its dependencies, Bazel would re-execute some build actions, or complete an *incremental build*. + +### Building multiple targets + +Bazel allows a number of ways to specify the targets to be built. Collectively, these are known as *target patterns*. This syntax is used in commands like `build`, `test`, or `query`. + +Whereas [labels](/concepts/labels) are used to specify individual targets, such as for declaring dependencies in `BUILD` files, Bazel's target patterns specify multiple targets. Target patterns are a generalization of the label syntax for *sets* of targets, using wildcards. In the simplest case, any valid label is also a valid target pattern, identifying a set of exactly one target. + +All target patterns starting with `//` are resolved relative to the current workspace. + +| | | +| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| `//foo/bar:wiz` | Just the single target `//foo/bar:wiz`. | +| `//foo/bar` | Equivalent to `//foo/bar:bar`. | +| `//foo/bar:all` | All rule targets in the package `foo/bar`. | +| `//foo/...` | All rule targets in all packages beneath the directory `foo`. | +| `//foo/...:all` | All rule targets in all packages beneath the directory `foo`. | +| `//foo/...:*` | All targets (rules and files) in all packages beneath the directory `foo`. | +| `//foo/...:all-targets` | All targets (rules and files) in all packages beneath the directory `foo`. | +| `//...` | All rule targets in packages in the main repository. Does not include targets from [external repositories](/docs/external). | +| `//:all` | All rule targets in the top-level package, if there is a \`BUILD\` file at the root of the workspace. | + +Target patterns that do not begin with `//` are resolved relative to the current *working directory*. These examples assume a working directory of `foo`: + +| | | +| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `:foo` | Equivalent to `//foo:foo`. | +| `bar:wiz` | Equivalent to `//foo/bar:wiz`. | +| `bar/wiz` | Equivalent to:* `//foo/bar/wiz:wiz` if `foo/bar/wiz` is a package +* `//foo/bar:wiz` if `foo/bar` is a package +* `//foo:bar/wiz` otherwise | +| `bar:all` | Equivalent to `//foo/bar:all`. | +| `:all` | Equivalent to `//foo:all`. | +| `...:all` | Equivalent to `//foo/...:all`. | +| `...` | Equivalent to `//foo/...:all`. | +| `bar/...:all` | Equivalent to `//foo/bar/...:all`. | + +By default, directory symlinks are followed for recursive target patterns, except those that point to under the output base, such as the convenience symlinks that are created in the root directory of the workspace. + +In addition, Bazel does not follow symlinks when evaluating recursive target patterns in any directory that contains a file named as follows: `DONT_FOLLOW_SYMLINKS_WHEN_TRAVERSING_THIS_DIRECTORY_VIA_A_RECURSIVE_TARGET_PATTERN` + +`foo/...` is a wildcard over *packages*, indicating all packages recursively beneath directory `foo` (for all roots of the package path). `:all` is a wildcard over *targets*, matching all rules within a package. These two may be combined, as in `foo/...:all`, and when both wildcards are used, this may be abbreviated to `foo/...`. + +In addition, `:*` (or `:all-targets`) is a wildcard that matches *every target* in the matched packages, including files that aren't normally built by any rule, such as `_deploy.jar` files associated with `java_binary` rules. + +This implies that `:*` denotes a *superset* of `:all`; while potentially confusing, this syntax does allow the familiar `:all` wildcard to be used for typical builds, where building targets like the `_deploy.jar` is not desired. + +In addition, Bazel allows a slash to be used instead of the colon required by the label syntax; this is often convenient when using Bash filename expansion. For example, `foo/bar/wiz` is equivalent to `//foo/bar:wiz` (if there is a package `foo/bar`) or to `//foo:bar/wiz` (if there is a package `foo`). + +Many Bazel commands accept a list of target patterns as arguments, and they all honor the prefix negation operator `-`. This can be used to subtract a set of targets from the set specified by the preceding arguments. Note that this means order matters. For example, + +```posix-terminal +bazel build foo/... bar/... +``` + +means "build all targets beneath `foo` *and* all targets beneath `bar`", whereas + +```posix-terminal +bazel build -- foo/... -foo/bar/... +``` + +means "build all targets beneath `foo` *except* those beneath `foo/bar`". (The `--` argument is required to prevent the subsequent arguments starting with `-` from being interpreted as additional options.) + +It's important to point out though that subtracting targets this way will not guarantee that they are not built, since they may be dependencies of targets that weren't subtracted. For example, if there were a target `//foo:all-apis` that among others depended on `//foo/bar:api`, then the latter would be built as part of building the former. + +Targets with `tags = ["manual"]` are not included in wildcard target patterns (`...`, `:*`, `:all`, etc.) when specified in commands like `bazel build` and `bazel test` (but they are included in negative wildcard target patterns, that is they will be subtracted). You should specify such test targets with explicit target patterns on the command line if you want Bazel to build/test them. In contrast, `bazel query` doesn't perform any such filtering automatically (that would defeat the purpose of `bazel query`). + +### Fetching external dependencies + +By default, Bazel will download and symlink external dependencies during the build. However, this can be undesirable, either because you'd like to know when new external dependencies are added or because you'd like to "prefetch" dependencies (say, before a flight where you'll be offline). If you would like to prevent new dependencies from being added during builds, you can specify the `--fetch=false` flag. Note that this flag only applies to repository rules that do not point to a directory in the local file system. Changes, for example, to `local_repository`, `new_local_repository` and Android SDK and NDK repository rules will always take effect regardless of the value `--fetch` . + +If you disallow fetching during builds and Bazel finds new external dependencies, your build will fail. + +You can manually fetch dependencies by running `bazel fetch`. If you disallow during-build fetching, you'll need to run `bazel fetch`: + +- Before you build for the first time. +- After you add a new external dependency. + +Once it has been run, you should not need to run it again until the MODULE.bazel file changes. + +`fetch` takes a list of targets to fetch dependencies for. For example, this would fetch dependencies needed to build `//foo:bar` and `//bar:baz`: + +```posix-terminal +bazel fetch //foo:bar //bar:baz +``` + +To fetch all external dependencies for a workspace, run: + +```posix-terminal +bazel fetch //... +``` + +With Bazel 7 or later, if you have Bzlmod enabled, you can also fetch all external dependencies by running + +```posix-terminal +bazel fetch +``` + +You do not need to run bazel fetch at all if you have all of the tools you are using (from library jars to the JDK itself) under your workspace root. However, if you're using anything outside of the workspace directory then Bazel will automatically run `bazel fetch` before running `bazel build`. + +#### The repository cache + +Bazel tries to avoid fetching the same file several times, even if the same file is needed in different workspaces, or if the definition of an external repository changed but it still needs the same file to download. To do so, bazel caches all files downloaded in the repository cache which, by default, is located at `~/.cache/bazel/_bazel_$USER/cache/repos/v1/`. The location can be changed by the `--repository_cache` option. The cache is shared between all workspaces and installed versions of bazel. An entry is taken from the cache if Bazel knows for sure that it has a copy of the correct file, that is, if the download request has a SHA256 sum of the file specified and a file with that hash is in the cache. So specifying a hash for each external file is not only a good idea from a security perspective; it also helps avoiding unnecessary downloads. + +Upon each cache hit, the modification time of the file in the cache is updated. In this way, the last use of a file in the cache directory can easily be determined, for example to manually clean up the cache. The cache is never cleaned up automatically, as it might contain a copy of a file that is no longer available upstream. + +#### \[Deprecated] Distribution files directories + +**Deprecated**: *Using repository cache is preferred to achieve offline build.* + +The distribution directory is another Bazel mechanism to avoid unnecessary downloads. Bazel searches distribution directories before the repository cache. The primary difference is that the distribution directory requires manual preparation. + +Using the [`--distdir=/path/to-directory`](/reference/command-line-reference#flag--distdir) option, you can specify additional read-only directories to look for files instead of fetching them. A file is taken from such a directory if the file name is equal to the base name of the URL and additionally the hash of the file is equal to the one specified in the download request. This only works if the file hash is specified in the repo rule declaration. + +While the condition on the file name is not necessary for correctness, it reduces the number of candidate files to one per specified directory. In this way, specifying distribution files directories remains efficient, even if the number of files in such a directory grows large. + +#### Running Bazel in an airgapped environment + +To keep Bazel's binary size small, Bazel's implicit dependencies are fetched over the network while running for the first time. These implicit dependencies contain toolchains and rules that may not be necessary for everyone. For example, Android tools are unbundled and fetched only when building Android projects. + +However, these implicit dependencies may cause problems when running Bazel in an airgapped environment, even if you have vendored all of your external dependencies. To solve that, you can prepare a repository cache (with Bazel 7 or later) or distribution directory (with Bazel prior to 7) containing these dependencies on a machine with network access, and then transfer them to the airgapped environment with an offline approach. + +##### Repository cache (with Bazel 7 or later) + +To prepare the [repository cache](#repository-cache), use the [`--repository_cache`](/reference/command-line-reference#flag--repository_cache) flag. You will need to do this once for every new Bazel binary version, since the implicit dependencies can be different for every release. + +To fetch those dependencies outside of your airgapped environment, first create an empty workspace: + +```posix-terminal +mkdir empty_workspace && cd empty_workspace + +touch MODULE.bazel +``` + +To fetch built-in Bzlmod dependencies, run + +```posix-terminal +bazel fetch --repository_cache="path/to/repository/cache" +``` + +If you still rely on the legacy WORKSPACE file, to fetch built-in WORKSPACE dependencies, run + +```posix-terminal +bazel sync --repository_cache="path/to/repository/cache" +``` + +Finally, when you use Bazel in your airgapped environment, pass the same `--repository_cache` flag. For convenience, you can add it as an `.bazelrc` entry: + +```python +common --repository_cache="path/to/repository/cache" +``` + +In addition, you may also need to clone the [BCR](https://github.com/bazelbuild/bazel-central-registry) locally and use `--registry` flag to point your local copy to prevent Bazel from accessing the BCR through internet. Add the following line to your `.bazelrc`: + +```python +common --registry="path/to/local/bcr/registry" +``` + +##### Distribution directory (with Bazel prior to 7) + +To prepare the [distribution directory](#distribution-directory), use the [`--distdir`](/reference/command-line-reference#flag--distdir) flag. You will need to do this once for every new Bazel binary version, since the implicit dependencies can be different for every release. + +To build these dependencies outside of your airgapped environment, first checkout the Bazel source tree at the right version: + +```posix-terminal +git clone https://github.com/bazelbuild/bazel "$BAZEL_DIR" + +cd "$BAZEL_DIR" + +git checkout "$BAZEL_VERSION" +``` + +Then, build the tarball containing the implicit runtime dependencies for that specific Bazel version: + +```posix-terminal +bazel build @additional_distfiles//:archives.tar +``` + +Export this tarball to a directory that can be copied into your airgapped environment. Note the `--strip-components` flag, because `--distdir` can be quite finicky with the directory nesting level: + +```posix-terminal +tar xvf bazel-bin/external/additional_distfiles/archives.tar \ + -C "$NEW_DIRECTORY" --strip-components=3 +``` + +Finally, when you use Bazel in your airgapped environment, pass the `--distdir` flag pointing to the directory. For convenience, you can add it as an `.bazelrc` entry: + +```posix-terminal +build --distdir=<var>path</var>/to/<var>directory</var> +``` + +### Build configurations and cross-compilation + +All the inputs that specify the behavior and result of a given build can be divided into two distinct categories. The first kind is the intrinsic information stored in the `BUILD` files of your project: the build rule, the values of its attributes, and the complete set of its transitive dependencies. The second kind is the external or environmental data, supplied by the user or by the build tool: the choice of target architecture, compilation and linking options, and other toolchain configuration options. We refer to a complete set of environmental data as a **configuration**. + +In any given build, there may be more than one configuration. Consider a cross-compile, in which you build a `//foo:bin` executable for a 64-bit architecture, but your workstation is a 32-bit machine. Clearly, the build will require building `//foo:bin` using a toolchain capable of creating 64-bit executables, but the build system must also build various tools used during the build itself—for example tools that are built from source, then subsequently used in, say, a genrule—and these must be built to run on your workstation. Thus we can identify two configurations: the **exec configuration**, which is used for building tools that run during the build, and the **target configuration** (or *request configuration*, but we say "target configuration" more often even though that word already has many meanings), which is used for building the binary you ultimately requested. + +Typically, there are many libraries that are prerequisites of both the requested build target (`//foo:bin`) and one or more of the exec tools, for example some base libraries. Such libraries must be built twice, once for the exec configuration, and once for the target configuration. Bazel takes care of ensuring that both variants are built, and that the derived files are kept separate to avoid interference; usually such targets can be built concurrently, since they are independent of each other. If you see progress messages indicating that a given target is being built twice, this is most likely the explanation. + +The exec configuration is derived from the target configuration as follows: + +- Use the same version of Crosstool (`--crosstool_top`) as specified in the request configuration, unless `--host_crosstool_top` is specified. +- Use the value of `--host_cpu` for `--cpu` (default: `k8`). +- Use the same values of these options as specified in the request configuration: `--compiler`, `--use_ijars`, and if `--host_crosstool_top` is used, then the value of `--host_cpu` is used to look up a `default_toolchain` in the Crosstool (ignoring `--compiler`) for the exec configuration. +- Use the value of `--host_javabase` for `--javabase` +- Use the value of `--host_java_toolchain` for `--java_toolchain` +- Use optimized builds for C++ code (`-c opt`). +- Generate no debugging information (`--copt=-g0`). +- Strip debug information from executables and shared libraries (`--strip=always`). +- Place all derived files in a special location, distinct from that used by any possible request configuration. +- Suppress stamping of binaries with build data (see `--embed_*` options). +- All other values remain at their defaults. + +There are many reasons why it might be preferable to select a distinct exec configuration from the request configuration. Most importantly: + +Firstly, by using stripped, optimized binaries, you reduce the time spent linking and executing the tools, the disk space occupied by the tools, and the network I/O time in distributed builds. + +Secondly, by decoupling the exec and request configurations in all builds, you avoid very expensive rebuilds that would result from minor changes to the request configuration (such as changing a linker options does), as described earlier. + +### Correct incremental rebuilds + +One of the primary goals of the Bazel project is to ensure correct incremental rebuilds. Previous build tools, especially those based on Make, make several unsound assumptions in their implementation of incremental builds. + +Firstly, that timestamps of files increase monotonically. While this is the typical case, it is very easy to fall afoul of this assumption; syncing to an earlier revision of a file causes that file's modification time to decrease; Make-based systems will not rebuild. + +More generally, while Make detects changes to files, it does not detect changes to commands. If you alter the options passed to the compiler in a given build step, Make will not re-run the compiler, and it is necessary to manually discard the invalid outputs of the previous build using `make clean`. + +Also, Make is not robust against the unsuccessful termination of one of its subprocesses after that subprocess has started writing to its output file. While the current execution of Make will fail, the subsequent invocation of Make will blindly assume that the truncated output file is valid (because it is newer than its inputs), and it will not be rebuilt. Similarly, if the Make process is killed, a similar situation can occur. + +Bazel avoids these assumptions, and others. Bazel maintains a database of all work previously done, and will only omit a build step if it finds that the set of input files (and their timestamps) to that build step, and the compilation command for that build step, exactly match one in the database, and, that the set of output files (and their timestamps) for the database entry exactly match the timestamps of the files on disk. Any change to the input files or output files, or to the command itself, will cause re-execution of the build step. + +The benefit to users of correct incremental builds is: less time wasted due to confusion. (Also, less time spent waiting for rebuilds caused by use of `make clean`, whether necessary or pre-emptive.) + +#### Build consistency and incremental builds + +Formally, we define the state of a build as *consistent* when all the expected output files exist, and their contents are correct, as specified by the steps or rules required to create them. When you edit a source file, the state of the build is said to be *inconsistent*, and remains inconsistent until you next run the build tool to successful completion. We describe this situation as *unstable inconsistency*, because it is only temporary, and consistency is restored by running the build tool. + +There is another kind of inconsistency that is pernicious: *stable inconsistency*. If the build reaches a stable inconsistent state, then repeated successful invocation of the build tool does not restore consistency: the build has gotten "stuck", and the outputs remain incorrect. Stable inconsistent states are the main reason why users of Make (and other build tools) type `make clean`. Discovering that the build tool has failed in this manner (and then recovering from it) can be time consuming and very frustrating. + +Conceptually, the simplest way to achieve a consistent build is to throw away all the previous build outputs and start again: make every build a clean build. This approach is obviously too time-consuming to be practical (except perhaps for release engineers), and therefore to be useful, the build tool must be able to perform incremental builds without compromising consistency. + +Correct incremental dependency analysis is hard, and as described above, many other build tools do a poor job of avoiding stable inconsistent states during incremental builds. In contrast, Bazel offers the following guarantee: after a successful invocation of the build tool during which you made no edits, the build will be in a consistent state. (If you edit your source files during a build, Bazel makes no guarantee about the consistency of the result of the current build. But it does guarantee that the results of the *next* build will restore consistency.) + +As with all guarantees, there comes some fine print: there are some known ways of getting into a stable inconsistent state with Bazel. We won't guarantee to investigate such problems arising from deliberate attempts to find bugs in the incremental dependency analysis, but we will investigate and do our best to fix all stable inconsistent states arising from normal or "reasonable" use of the build tool. + +If you ever detect a stable inconsistent state with Bazel, please report a bug. + +#### Sandboxed execution + +Note: Sandboxing is enabled by default for local execution. + +Bazel uses sandboxes to guarantee that actions run hermetically and correctly. Bazel runs *spawns* (loosely speaking: actions) in sandboxes that only contain the minimal set of files the tool requires to do its job. Currently sandboxing works on Linux 3.12 or newer with the `CONFIG_USER_NS` option enabled, and also on macOS 10.11 or newer. + +Bazel will print a warning if your system does not support sandboxing to alert you to the fact that builds are not guaranteed to be hermetic and might affect the host system in unknown ways. To disable this warning you can pass the `--ignore_unsupported_sandboxing` flag to Bazel. + +Note: Hermeticity means that the action only uses its declared input files and no other files in the filesystem, and it only produces its declared output files. See [Hermeticity](/basics/hermeticity) for more details. + +On some platforms such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. This can be checked by looking at the file `/proc/sys/kernel/unprivileged_userns_clone`: if it exists and contains a 0, then user namespaces can be activated with `sudo sysctl kernel.unprivileged_userns_clone=1`. + +In some cases, the Bazel sandbox fails to execute rules because of the system setup. The symptom is generally a failure that output a message similar to `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`. In that case, try to deactivate the sandbox for genrules with `--strategy=Genrule=standalone` and for other rules with `--spawn_strategy=standalone`. Also please report a bug on our issue tracker and mention which Linux distribution you're using so that we can investigate and provide a fix in a subsequent release. + +### Phases of a build + +In Bazel, a build occurs in three distinct phases; as a user, understanding the difference between them provides insight into the options which control a build (see below). + +#### Loading phase + +The first is **loading** during which all the necessary BUILD files for the initial targets, and their transitive closure of dependencies, are loaded, parsed, evaluated and cached. + +For the first build after a Bazel server is started, the loading phase typically takes many seconds as many BUILD files are loaded from the file system. In subsequent builds, especially if no BUILD files have changed, loading occurs very quickly. + +Errors reported during this phase include: package not found, target not found, lexical and grammatical errors in a BUILD file, and evaluation errors. + +#### Analysis phase + +The second phase, **analysis**, involves the semantic analysis and validation of each build rule, the construction of a build dependency graph, and the determination of exactly what work is to be done in each step of the build. + +Like loading, analysis also takes several seconds when computed in its entirety. However, Bazel caches the dependency graph from one build to the next and only reanalyzes what it has to, which can make incremental builds extremely fast in the case where the packages haven't changed since the previous build. + +Errors reported at this stage include: inappropriate dependencies, invalid inputs to a rule, and all rule-specific error messages. + +The loading and analysis phases are fast because Bazel avoids unnecessary file I/O at this stage, reading only BUILD files in order to determine the work to be done. This is by design, and makes Bazel a good foundation for analysis tools, such as Bazel's [query](/query/guide) command, which is implemented atop the loading phase. + +#### Execution phase + +The third and final phase of the build is **execution**. This phase ensures that the outputs of each step in the build are consistent with its inputs, re-running compilation/linking/etc. tools as necessary. This step is where the build spends the majority of its time, ranging from a few seconds to over an hour for a large build. Errors reported during this phase include: missing source files, errors in a tool executed by some build action, or failure of a tool to produce the expected set of outputs. diff --git a/run/client-server.mdx b/run/client-server.mdx index 1868635f..b335b0ca 100644 --- a/run/client-server.mdx +++ b/run/client-server.mdx @@ -2,57 +2,21 @@ title: 'Client/server implementation' --- +The Bazel system is implemented as a long-lived server process. This allows it to perform many optimizations not possible with a batch-oriented implementation, such as caching of BUILD files, dependency graphs, and other metadata from one build to the next. This improves the speed of incremental builds, and allows different commands, such as `build` and `query` to share the same cache of loaded packages, making queries very fast. Each server can handle at most one invocation at a time; further concurrent invocations will either block or fail-fast (see `--block_for_lock`). +When you run `bazel`, you're running the client. The client finds the server based on the [output base](/run/scripts#output-base-option), which by default is determined by the path of the base workspace directory and your userid, so if you build in multiple workspaces, you'll have multiple output bases and thus multiple Bazel server processes. Multiple users on the same workstation can build concurrently in the same workspace because their output bases will differ (different userids). -The Bazel system is implemented as a long-lived server process. This allows it -to perform many optimizations not possible with a batch-oriented implementation, -such as caching of BUILD files, dependency graphs, and other metadata from one -build to the next. This improves the speed of incremental builds, and allows -different commands, such as `build` and `query` to share the same cache of -loaded packages, making queries very fast. Each server can handle at most one -invocation at a time; further concurrent invocations will either block or -fail-fast (see `--block_for_lock`). - -When you run `bazel`, you're running the client. The client finds the server -based on the [output base](/run/scripts#output-base-option), which by default is -determined by the path of the base workspace directory and your userid, so if -you build in multiple workspaces, you'll have multiple output bases and thus -multiple Bazel server processes. Multiple users on the same workstation can -build concurrently in the same workspace because their output bases will differ -(different userids). - -If the client cannot find a running server instance, it starts a new one. It -does this by checking if the output base already exists, implying the blaze -archive has already been unpacked. Otherwise if the output base doesn't exist, -the client unzips the archive's files and sets their `mtime`s to a date 9 years -in the future. Once installed, the client confirms that the `mtime`s of the -unzipped files are equal to the far off date to ensure no installation tampering -has occurred. - -The server process will stop after a period of inactivity (3 hours, by default, -which can be modified using the startup option `--max_idle_secs`). For the most -part, the fact that there is a server running is invisible to the user, but -sometimes it helps to bear this in mind. For example, if you're running scripts -that perform a lot of automated builds in different directories, it's important -to ensure that you don't accumulate a lot of idle servers; you can do this by -explicitly shutting them down when you're finished with them, or by specifying -a short timeout period. - -The name of a Bazel server process appears in the output of `ps x` or `ps -e f` -as bazel(dirname), where _dirname_ is the basename of the -directory enclosing the root of your workspace directory. For example: +If the client cannot find a running server instance, it starts a new one. It does this by checking if the output base already exists, implying the blaze archive has already been unpacked. Otherwise if the output base doesn't exist, the client unzips the archive's files and sets their `mtime`s to a date 9 years in the future. Once installed, the client confirms that the `mtime`s of the unzipped files are equal to the far off date to ensure no installation tampering has occurred. + +The server process will stop after a period of inactivity (3 hours, by default, which can be modified using the startup option `--max_idle_secs`). For the most part, the fact that there is a server running is invisible to the user, but sometimes it helps to bear this in mind. For example, if you're running scripts that perform a lot of automated builds in different directories, it's important to ensure that you don't accumulate a lot of idle servers; you can do this by explicitly shutting them down when you're finished with them, or by specifying a short timeout period. + +The name of a Bazel server process appears in the output of `ps x` or `ps -e f` as `bazel(dirname)`, where *dirname* is the basename of the directory enclosing the root of your workspace directory. For example: ```posix-terminal ps -e f 16143 ? Sl 3:00 bazel(src-johndoe2) -server -Djava.library.path=... ``` -This makes it easier to find out which server process belongs to a given -workspace. (Beware that with certain other options to `ps`, Bazel server -processes may be named just `java`.) Bazel servers can be stopped using the -[shutdown](/docs/user-manual#shutdown) command. +This makes it easier to find out which server process belongs to a given workspace. (Beware that with certain other options to `ps`, Bazel server processes may be named just `java`.) Bazel servers can be stopped using the [shutdown](/docs/user-manual#shutdown) command. -When running `bazel`, the client first checks that the server is the appropriate -version; if not, the server is stopped and a new one started. This ensures that -the use of a long-running server process doesn't interfere with proper -versioning. +When running `bazel`, the client first checks that the server is the appropriate version; if not, the server is stopped and a new one started. This ensures that the use of a long-running server process doesn't interfere with proper versioning. diff --git a/run/scripts.mdx b/run/scripts.mdx index f267c903..25cb1f65 100644 --- a/run/scripts.mdx +++ b/run/scripts.mdx @@ -2,129 +2,93 @@ title: 'Calling Bazel from scripts' --- +You can call Bazel from scripts to perform a build, run tests, or query the dependency graph. Bazel has been designed to enable effective scripting, but this section lists some details to bear in mind to make your scripts more robust. +### Choosing the output base -You can call Bazel from scripts to perform a build, run tests, or query -the dependency graph. Bazel has been designed to enable effective scripting, but -this section lists some details to bear in mind to make your scripts more -robust. +The `--output_base` option controls where the Bazel process should write the outputs of a build to, as well as various working files used internally by Bazel, one of which is a lock that guards against concurrent mutation of the output base by multiple Bazel processes. -### Choosing the output base +Choosing the correct output base directory for your script depends on several factors. If you need to put the build outputs in a specific location, this will dictate the output base you need to use. If you are making a "read only" call to Bazel (such as `bazel query`), the locking factors will be more important. In particular, if you need to run multiple instances of your script concurrently, you should be mindful that each Blaze server process can handle at most one invocation [at a time](/run/client-server#clientserver-implementation). Depending on your situation it may make sense for each instance of your script to wait its turn, or it may make sense to use `--output_base` to run multiple Blaze servers and use those. -The `--output_base` option controls where the Bazel process should write the -outputs of a build to, as well as various working files used internally by -Bazel, one of which is a lock that guards against concurrent mutation of the -output base by multiple Bazel processes. - -Choosing the correct output base directory for your script depends on several -factors. If you need to put the build outputs in a specific location, this will -dictate the output base you need to use. If you are making a "read only" call to -Bazel (such as `bazel query`), the locking factors will be more important. In -particular, if you need to run multiple instances of your script concurrently, -you should be mindful that each Blaze server process can handle at most one -invocation [at a time](/run/client-server#clientserver-implementation). -Depending on your situation it may make sense for each instance of your script -to wait its turn, or it may make sense to use `--output_base` to run multiple -Blaze servers and use those. - -If you use the default output base value, you will be contending for the same -lock used by the user's interactive Bazel commands. If the user issues -long-running commands such as builds, your script will have to wait for those -commands to complete before it can continue. +If you use the default output base value, you will be contending for the same lock used by the user's interactive Bazel commands. If the user issues long-running commands such as builds, your script will have to wait for those commands to complete before it can continue. ### Notes about server mode -By default, Bazel uses a long-running [server process](/run/client-server) as an -optimization. When running Bazel in a script, don't forget to call `shutdown` -when you're finished with the server, or, specify `--max_idle_secs=5` so that -idle servers shut themselves down promptly. +By default, Bazel uses a long-running [server process](/run/client-server) as an optimization. When running Bazel in a script, don't forget to call `shutdown` when you're finished with the server, or, specify `--max_idle_secs=5` so that idle servers shut themselves down promptly. ### What exit code will I get? -Bazel attempts to differentiate failures due to the source code under -consideration from external errors that prevent Bazel from executing properly. -Bazel execution can result in following exit codes: +Bazel attempts to differentiate failures due to the source code under consideration from external errors that prevent Bazel from executing properly. Bazel execution can result in following exit codes: **Exit Codes common to all commands:** -- `0` - Success -- `2` - Command Line Problem, Bad or Illegal flags or command combination, or - Bad Environment Variables. Your command line must be modified. -- `8` - Build Interrupted but we terminated with an orderly shutdown. -- `9` - The server lock is held and `--noblock_for_lock` was passed. -- `32` - External Environment Failure not on this machine. - -- `33` - Bazel ran out of memory and crashed. You need to modify your command line. -- `34` - Reserved for Google-internal use. -- `35` - Reserved for Google-internal use. -- `36` - Local Environmental Issue, suspected permanent. -- `37` - Unhandled Exception / Internal Bazel Error. -- `38` - Transient error publishing results to the Build Event Service. -- `39` - Blobs required by Bazel are evicted from Remote Cache. -- `41-44` - Reserved for Google-internal use. -- `45` - Persistent error publishing results to the Build Event Service. -- `47` - Reserved for Google-internal use. -- `49` - Reserved for Google-internal use. +- `0` - Success -**Return codes for commands `bazel build`, `bazel test`:** +- `2` - Command Line Problem, Bad or Illegal flags or command combination, or Bad Environment Variables. Your command line must be modified. + +- `8` - Build Interrupted but we terminated with an orderly shutdown. + +- `9` - The server lock is held and `--noblock_for_lock` was passed. + +- `32` - External Environment Failure not on this machine. + +- `33` - Bazel ran out of memory and crashed. You need to modify your command line. + +- `34` - Reserved for Google-internal use. + +- `35` - Reserved for Google-internal use. + +- `36` - Local Environmental Issue, suspected permanent. -- `1` - Build failed. -- `3` - Build OK, but some tests failed or timed out. -- `4` - Build successful but no tests were found even though testing was - requested. +- `37` - Unhandled Exception / Internal Bazel Error. +- `38` - Transient error publishing results to the Build Event Service. + +- `39` - Blobs required by Bazel are evicted from Remote Cache. + +- `41-44` - Reserved for Google-internal use. + +- `45` - Persistent error publishing results to the Build Event Service. + +- `47` - Reserved for Google-internal use. + +- `49` - Reserved for Google-internal use. + +**Return codes for commands `bazel build`, `bazel test`:** + +- `1` - Build failed. +- `3` - Build OK, but some tests failed or timed out. +- `4` - Build successful but no tests were found even though testing was requested. **For `bazel run`:** -- `1` - Build failed. -- If the build succeeds but the executed subprocess returns a non-zero exit - code it will be the exit code of the command as well. +- `1` - Build failed. +- If the build succeeds but the executed subprocess returns a non-zero exit code it will be the exit code of the command as well. **For `bazel query`:** -- `3` - Partial success, but the query encountered 1 or more errors in the - input BUILD file set and therefore the results of the operation are not 100% - reliable. This is likely due to a `--keep_going` option on the command line. -- `7` - Command failure. - -Future Bazel versions may add additional exit codes, replacing generic failure -exit code `1` with a different non-zero value with a particular meaning. -However, all non-zero exit values will always constitute an error. +- `3` - Partial success, but the query encountered 1 or more errors in the input BUILD file set and therefore the results of the operation are not 100% reliable. This is likely due to a `--keep_going` option on the command line. +- `7` - Command failure. +Future Bazel versions may add additional exit codes, replacing generic failure exit code `1` with a different non-zero value with a particular meaning. However, all non-zero exit values will always constitute an error. ### Reading the .bazelrc file -By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base -workspace directory or the user's home directory. Whether or not this is -desirable is a choice for your script; if your script needs to be perfectly -hermetic (such as when doing release builds), you should disable reading the -.bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform -a build using the user's preferred settings, the default behavior is better. +By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base workspace directory or the user's home directory. Whether or not this is desirable is a choice for your script; if your script needs to be perfectly hermetic (such as when doing release builds), you should disable reading the .bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform a build using the user's preferred settings, the default behavior is better. ### Command log -The Bazel output is also available in a command log file which you can find with -the following command: +The Bazel output is also available in a command log file which you can find with the following command: ```posix-terminal bazel info command_log ``` -The command log file contains the interleaved stdout and stderr streams of the -most recent Bazel command. Note that running `bazel info` will overwrite the -contents of this file, since it then becomes the most recent Bazel command. -However, the location of the command log file will not change unless you change -the setting of the `--output_base` or `--output_user_root` options. +The command log file contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. ### Parsing output -The Bazel output is quite easy to parse for many purposes. Two options that may -be helpful for your script are `--noshow_progress` which suppresses progress -messages, and --show_result n, which controls whether or -not "build up-to-date" messages are printed; these messages may be parsed to -discover which targets were successfully built, and the location of the output -files they created. Be sure to specify a very large value of _n_ if you rely on -these messages. +The Bazel output is quite easy to parse for many purposes. Two options that may be helpful for your script are `--noshow_progress` which suppresses progress messages, and `--show_result n`, which controls whether or not "build up-to-date" messages are printed; these messages may be parsed to discover which targets were successfully built, and the location of the output files they created. Be sure to specify a very large value of *n* if you rely on these messages. ## Troubleshooting performance by profiling diff --git a/start/android-app.mdx b/start/android-app.mdx index 35da5828..3ae739fd 100644 --- a/start/android-app.mdx +++ b/start/android-app.mdx @@ -2,36 +2,22 @@ title: 'Bazel Tutorial: Build an Android App' --- - -**Note:** There are known limitations on using Bazel for building Android apps. -Visit the [rules_android issues page](https://github.com/bazelbuild/rules_android/issues) -to see the list of known issues. While the Bazel team and Open Source Software -(OSS) contributors work actively to address known issues, users should be aware -that Android Studio does not officially support Bazel projects. - This tutorial covers how to build a simple Android app using Bazel. -Bazel supports building Android apps using the -[Android rules](/reference/be/android). +Bazel supports building Android apps using the [Android rules](/reference/be/android). -This tutorial is intended for Windows, macOS and Linux users and does not -require experience with Bazel or Android app development. You do not need to -write any Android code in this tutorial. +This tutorial is intended for Windows, macOS and Linux users and does not require experience with Bazel or Android app development. You do not need to write any Android code in this tutorial. ## What you'll learn In this tutorial you learn how to: -* Set up your environment by installing Bazel and Android Studio, and - downloading the sample project. -* Set up a Bazel workspace that contains the source code - for the app and a `MODULE.bazel` file that identifies the top level of the - workspace directory. -* Update the `MODULE.bazel` file to contain references to the required - external dependencies, like the Android SDK. -* Create a `BUILD` file. -* Build the app with Bazel. -* Deploy and run the app on an Android emulator or physical device. +- Set up your environment by installing Bazel and Android Studio, and downloading the sample project. +- Set up a Bazel workspace that contains the source code for the app and a `MODULE.bazel` file that identifies the top level of the workspace directory. +- Update the `MODULE.bazel` file to contain references to the required external dependencies, like the Android SDK. +- Create a `BUILD` file. +- Build the app with Bazel. +- Deploy and run the app on an Android emulator or physical device. ## Before you begin @@ -39,16 +25,13 @@ In this tutorial you learn how to: Before you begin the tutorial, install the following software: -* **Bazel.** To install, follow the [installation instructions](/install). -* **Android Studio.** To install, follow the steps to [download Android - Studio](https://developer.android.com/sdk/index.html). - Execute the setup wizard to download the SDK and configure your environment. -* (Optional) **Git.** Use `git` to download the Android app project. +- **Bazel.** To install, follow the [installation instructions](/install). +- **Android Studio.** To install, follow the steps to [download Android Studio](https://developer.android.com/sdk/index.html). Execute the setup wizard to download the SDK and configure your environment. +- (Optional) **Git.** Use `git` to download the Android app project. ### Get the sample project -For the sample project, use the tutorial Android app project in -[Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). +For the sample project, use the tutorial Android app project in [Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). This app has a single button that prints a greeting when clicked: @@ -56,15 +39,13 @@ This app has a single button that prints a greeting when clicked: **Figure 1.** Android app button greeting. -Clone the repository with `git` (or [download the ZIP file -directly](https://github.com/bazelbuild/examples/archive/master.zip)): +Clone the repository with `git` (or [download the ZIP file directly](https://github.com/bazelbuild/examples/archive/master.zip)): ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in `examples/android/tutorial`. For -the rest of the tutorial, you will be executing commands in this directory. +The sample project for this tutorial is in `examples/android/tutorial`. For the rest of the tutorial, you will be executing commands in this directory. ### Review the source files @@ -99,24 +80,20 @@ The key files and directories are: | Android source files | `src/main/java/com/example/bazel/MainActivity.java` and `Greeter.java` | | Resource file directory | `src/main/java/com/example/bazel/res/` | - ## Build with Bazel ### Set up the workspace -A [workspace](/concepts/build-ref#workspace) is a directory that contains the -source files for one or more software projects, and has a `MODULE.bazel` file at -its root. +A [workspace](/concepts/build-ref#workspace) is a directory that contains the source files for one or more software projects, and has a `MODULE.bazel` file at its root. -The `MODULE.bazel` file may be empty or may contain references to [external -dependencies](/external/overview) required to build your project. +The `MODULE.bazel` file may be empty or may contain references to [external dependencies](/external/overview) required to build your project. First, run the following command to create an empty `MODULE.bazel` file: -| OS | Command | -| ------------------------ | ----------------------------------- | +| OS | Command | +| ------------------------ | -------------------------------------- | | Linux, macOS | `touch MODULE.bazel` | -| Windows (Command Prompt) | `type nul > MODULE.bazel` | +| Windows (Command Prompt) | `type nul > MODULE.bazel` | | Windows (PowerShell) | `New-Item MODULE.bazel -ItemType file` | ### Running Bazel @@ -127,8 +104,7 @@ You can now check if Bazel is running correctly with the command: bazel info workspace ``` -If Bazel prints the path of the current directory, you're good to go! If the -`MODULE.bazel` file does not exist, you may see an error message like: +If Bazel prints the path of the current directory, you're good to go! If the `MODULE.bazel` file does not exist, you may see an error message like: ``` ERROR: The 'info' command is only supported from within a workspace. @@ -136,10 +112,7 @@ ERROR: The 'info' command is only supported from within a workspace. ### Integrate with the Android SDK -Bazel needs to run the Android SDK -[build tools](https://developer.android.com/tools/revisions/build-tools.html) -to build the app. This means that you need to add some information to your -`MODULE.bazel` file so that Bazel knows where to find them. +Bazel needs to run the Android SDK [build tools](https://developer.android.com/tools/revisions/build-tools.html) to build the app. This means that you need to add some information to your `MODULE.bazel` file so that Bazel knows where to find them. Add the following line to your `MODULE.bazel` file: @@ -155,94 +128,53 @@ android_sdk_repository_extension = use_extension("@rules_android//rules/android_ use_repo(android_sdk_repository_extension, "androidsdk") ``` -This will use the Android SDK at the path referenced by the `ANDROID_HOME` -environment variable, and automatically detect the highest API level and the -latest version of build tools installed within that location. +This will use the Android SDK at the path referenced by the `ANDROID_HOME` environment variable, and automatically detect the highest API level and the latest version of build tools installed within that location. -You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find -the path to the installed SDK using Android Studio's [SDK -Manager](https://developer.android.com/studio/intro/update#sdk-manager). -Assuming the SDK is installed to default locations, you can use the following -commands to set the `ANDROID_HOME` variable: +You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find the path to the installed SDK using Android Studio's [SDK Manager](https://developer.android.com/studio/intro/update#sdk-manager). Assuming the SDK is installed to default locations, you can use the following commands to set the `ANDROID_HOME` variable: -| OS | Command | +| OS | Command | | ------------------------ | --------------------------------------------------- | | Linux | `export ANDROID_HOME=$HOME/Android/Sdk/` | | macOS | `export ANDROID_HOME=$HOME/Library/Android/sdk` | | Windows (Command Prompt) | `set ANDROID_HOME=%LOCALAPPDATA%\Android\Sdk` | | Windows (PowerShell) | `$env:ANDROID_HOME="$env:LOCALAPPDATA\Android\Sdk"` | -The above commands set the variable only for the current shell session. To make -them permanent, run the following commands: +The above commands set the variable only for the current shell session. To make them permanent, run the following commands: -| OS | Command | -| ------------------------ | --------------------------------------------------- | -| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | -| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | +| OS | Command | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | +| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | +| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | | Windows (Command Prompt) | `setx ANDROID_HOME "%LOCALAPPDATA%\Android\Sdk"` | | Windows (PowerShell) | `[System.Environment]::SetEnvironmentVariable('ANDROID_HOME', "$env:LOCALAPPDATA\Android\Sdk", [System.EnvironmentVariableTarget]::User)` | - -**Optional:** If you want to compile native code into your Android app, you -also need to download the [Android -NDK](https://developer.android.com/ndk/downloads/index.html) -and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: +**Optional:** If you want to compile native code into your Android app, you also need to download the [Android NDK](https://developer.android.com/ndk/downloads/index.html) and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: ```python bazel_dep(name = "rules_android_ndk", version = "0.1.3") ``` +For more information, read [Using the Android Native Development Kit with Bazel](/docs/android-ndk). -For more information, read [Using the Android Native Development Kit with -Bazel](/docs/android-ndk). - -It's not necessary to set the API levels to the same value for the SDK and NDK. -[This page](https://developer.android.com/ndk/guides/stable_apis.html) -contains a map from Android releases to NDK-supported API levels. +It's not necessary to set the API levels to the same value for the SDK and NDK. [This page](https://developer.android.com/ndk/guides/stable_apis.html) contains a map from Android releases to NDK-supported API levels. ### Create a BUILD file -A [`BUILD` file](/concepts/build-files) describes the relationship -between a set of build outputs, like compiled Android resources from `aapt` or -class files from `javac`, and their dependencies. These dependencies may be -source files (Java, C++) in your workspace or other build outputs. `BUILD` files -are written in a language called **Starlark**. - -`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. -The package hierarchy is a logical structure that overlays the directory -structure in your workspace. Each [package](/concepts/build-ref#packages) is a -directory (and its subdirectories) that contains a related set of source files -and a `BUILD` file. The package also includes any subdirectories, excluding -those that contain their own `BUILD` file. The *package name* is the path to the -`BUILD` file relative to the `MODULE.bazel` file. - -Note that Bazel's package hierarchy is conceptually different from the Java -package hierarchy of your Android App directory where the `BUILD` file is -located, although the directories may be organized identically. - -For the simple Android app in this tutorial, the source files in `src/main/` -comprise a single Bazel package. A more complex project may have many nested -packages. - -#### Add an android_library rule - -A `BUILD` file contains several different types of declarations for Bazel. The -most important type is the -[build rule](/concepts/build-files#types-of-build-rules), which tells -Bazel how to build an intermediate or final software output from a set of source -files or other dependencies. Bazel provides two build rules, -[`android_library`](/reference/be/android#android_library) and -[`android_binary`](/reference/be/android#android_binary), that you can use to -build an Android app. - -For this tutorial, you'll first use the -`android_library` rule to tell Bazel to build an [Android library -module](http://developer.android.com/tools/projects/index.html#LibraryProjects) -from the app source code and resource files. You'll then use the -`android_binary` rule to tell Bazel how to build the Android application package. - -Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, -and declare a new `android_library` target: +A [`BUILD` file](/concepts/build-files) describes the relationship between a set of build outputs, like compiled Android resources from `aapt` or class files from `javac`, and their dependencies. These dependencies may be source files (Java, C++) in your workspace or other build outputs. `BUILD` files are written in a language called **Starlark**. + +`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. The package hierarchy is a logical structure that overlays the directory structure in your workspace. Each [package](/concepts/build-ref#packages) is a directory (and its subdirectories) that contains a related set of source files and a `BUILD` file. The package also includes any subdirectories, excluding those that contain their own `BUILD` file. The *package name* is the path to the `BUILD` file relative to the `MODULE.bazel` file. + +Note that Bazel's package hierarchy is conceptually different from the Java package hierarchy of your Android App directory where the `BUILD` file is located, although the directories may be organized identically. + +For the simple Android app in this tutorial, the source files in `src/main/` comprise a single Bazel package. A more complex project may have many nested packages. + +#### Add an android\_library rule + +A `BUILD` file contains several different types of declarations for Bazel. The most important type is the [build rule](/concepts/build-files#types-of-build-rules), which tells Bazel how to build an intermediate or final software output from a set of source files or other dependencies. Bazel provides two build rules, [`android_library`](/reference/be/android#android_library) and [`android_binary`](/reference/be/android#android_binary), that you can use to build an Android app. + +For this tutorial, you'll first use the `android_library` rule to tell Bazel to build an [Android library module](http://developer.android.com/tools/projects/index.html#LibraryProjects) from the app source code and resource files. You'll then use the `android_binary` rule to tell Bazel how to build the Android application package. + +Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, and declare a new `android_library` target: `src/main/java/com/example/bazel/BUILD`: @@ -264,18 +196,13 @@ android_library( ) ``` -The `android_library` build rule contains a set of attributes that specify the -information that Bazel needs to build a library module from the source files. -Note also that the name of the rule is `greeter_activity`. You'll reference the -rule using this name as a dependency in the `android_binary` rule. +The `android_library` build rule contains a set of attributes that specify the information that Bazel needs to build a library module from the source files. Note also that the name of the rule is `greeter_activity`. You'll reference the rule using this name as a dependency in the `android_binary` rule. -#### Add an android_binary rule +#### Add an android\_binary rule -The [`android_binary`](/reference/be/android#android_binary) rule builds -the Android application package (`.apk` file) for your app. +The [`android_binary`](/reference/be/android#android_binary) rule builds the Android application package (`.apk` file) for your app. -Create a new `BUILD` file in the `src/main/` directory, -and declare a new `android_binary` target: +Create a new `BUILD` file in the `src/main/` directory, and declare a new `android_binary` target: `src/main/BUILD`: @@ -289,35 +216,23 @@ android_binary( ) ``` -Here, the `deps` attribute references the output of the `greeter_activity` rule -you added to the `BUILD` file above. This means that when Bazel builds the -output of this rule it checks first to see if the output of the -`greeter_activity` library rule has been built and is up-to-date. If not, Bazel -builds it and then uses that output to build the application package file. +Here, the `deps` attribute references the output of the `greeter_activity` rule you added to the `BUILD` file above. This means that when Bazel builds the output of this rule it checks first to see if the output of the `greeter_activity` library rule has been built and is up-to-date. If not, Bazel builds it and then uses that output to build the application package file. Now, save and close the file. ### Build the app -Try building the app! Run the following command to build the -`android_binary` target: +Try building the app! Run the following command to build the `android_binary` target: ```posix-terminal bazel build //src/main:app ``` -The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the -target that follows. The target is specified as the name of a build rule inside -a `BUILD` file, with along with the package path relative to your workspace -directory. For this example, the target is `app` and the package path is -`//src/main/`. +The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the target that follows. The target is specified as the name of a build rule inside a `BUILD` file, with along with the package path relative to your workspace directory. For this example, the target is `app` and the package path is `//src/main/`. -Note that you can sometimes omit the package path or target name, depending on -your current working directory at the command line and the name of the target. -For more details about target labels and paths, see [Labels](/concepts/labels). +Note that you can sometimes omit the package path or target name, depending on your current working directory at the command line and the name of the target. For more details about target labels and paths, see [Labels](/concepts/labels). -Bazel will start to build the sample app. During the build process, its output -will appear similar to the following: +Bazel will start to build the sample app. During the build process, its output will appear similar to the following: ```bash INFO: Analysed target //src/main:app (0 packages loaded, 0 targets configured). @@ -330,40 +245,25 @@ Target //src/main:app up-to-date: #### Locate the build outputs -Bazel puts the outputs of both intermediate and final build operations in a set -of per-user, per-workspace output directories. These directories are symlinked -from the following locations at the top-level of the project directory, where -the `MODULE.bazel` file is: +Bazel puts the outputs of both intermediate and final build operations in a set of per-user, per-workspace output directories. These directories are symlinked from the following locations at the top-level of the project directory, where the `MODULE.bazel` file is: -* `bazel-bin` stores binary executables and other runnable build outputs -* `bazel-genfiles` stores intermediary source files that are generated by - Bazel rules -* `bazel-out` stores other types of build outputs +- `bazel-bin` stores binary executables and other runnable build outputs +- `bazel-genfiles` stores intermediary source files that are generated by Bazel rules +- `bazel-out` stores other types of build outputs -Bazel stores the Android `.apk` file generated using the `android_binary` rule -in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is -derived from the name of the Bazel package. +Bazel stores the Android `.apk` file generated using the `android_binary` rule in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is derived from the name of the Bazel package. -At a command prompt, list the contents of this directory and find the `app.apk` -file: +At a command prompt, list the contents of this directory and find the `app.apk` file: -| OS | Command | +| OS | Command | | ------------------------ | ------------------------ | | Linux, macOS | `ls bazel-bin/src/main` | | Windows (Command Prompt) | `dir bazel-bin\src\main` | | Windows (PowerShell) | `ls bazel-bin\src\main` | - ### Run the app -You can now deploy the app to a connected Android device or emulator from the -command line using `bazel mobile-install`. -This command uses the Android Debug Bridge (`adb`) to communicate with the -device. You must set up your device to use `adb` following the instructions in -[Android Debug Bridge](http://developer.android.com/tools/help/adb.html) -before deployment. You can also choose to install the app on the Android emulator -included in Android Studio. Make sure the emulator is running before executing -the command below. +You can now deploy the app to a connected Android device or emulator from the command line using `bazel mobile-install`. This command uses the Android Debug Bridge (`adb`) to communicate with the device. You must set up your device to use `adb` following the instructions in [Android Debug Bridge](http://developer.android.com/tools/help/adb.html) before deployment. You can also choose to install the app on the Android emulator included in Android Studio. Make sure the emulator is running before executing the command below. Enter the following: @@ -378,12 +278,7 @@ bazel mobile-install //src/main:app \ --tool_java_language_version=17 ``` -Note that the extra flags required for mobile-install can be added to your -project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags -(`--mode`, `--mobile_install*`) will no longer be required starting from -Bazel 8.4.0 and onwards. The various Java flags for language and runtime version -may be required depending on your workspace's Java configuration. -_Mobile-install sub-tools require a language and runtime level of 17 or higher._ +Note that the extra flags required for mobile-install can be added to your project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags (`--mode`, `--mobile_install*`) will no longer be required starting from Bazel 8.4.0 and onwards. The various Java flags for language and runtime version may be required depending on your workspace's Java configuration. *Mobile-install sub-tools require a language and runtime level of 17 or higher.* Now the "Bazel Tutorial App" should install and launch automatically: @@ -397,16 +292,20 @@ Now the "Bazel Tutorial App" should install and launch automatically: For more details, see these pages: -* Open issues on [rules_android GitHub](https://github.com/bazelbuild/rules_android/issues) -* More information on [mobile-install](/docs/mobile-install) -* Integrate external dependencies like AppCompat, Guava and JUnit from Maven - repositories using [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) -* Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) - integration. -* Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) -* See more Bazel example projects of: - * [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) - * [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) - * [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) +- Open issues on [rules\_android GitHub](https://github.com/bazelbuild/rules_android/issues) + +- More information on [mobile-install](/docs/mobile-install) + +- Integrate external dependencies like AppCompat, Guava and JUnit from Maven repositories using [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) + +- Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) integration. + +- Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) + +- See more Bazel example projects of: + + - [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) + - [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) + - [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) Happy building! diff --git a/start/cpp.mdx b/start/cpp.mdx index adb7c71a..e583146e 100644 --- a/start/cpp.mdx +++ b/start/cpp.mdx @@ -2,37 +2,25 @@ title: 'Bazel Tutorial: Build a C++ Project' --- - - ## Introduction -New to Bazel? You're in the right place. Follow this First Build tutorial for a -simplified introduction to using Bazel. This tutorial defines key terms as they -are used in Bazel's context and walks you through the basics of the Bazel -workflow. Starting with the tools you need, you will build and run three -projects with increasing complexity and learn how and why they get more complex. +New to Bazel? You're in the right place. Follow this First Build tutorial for a simplified introduction to using Bazel. This tutorial defines key terms as they are used in Bazel's context and walks you through the basics of the Bazel workflow. Starting with the tools you need, you will build and run three projects with increasing complexity and learn how and why they get more complex. -While Bazel is a [build system](https://bazel.build/basics/build-systems) that -supports multi-language builds, this tutorial uses a C++ project as an example -and provides the general guidelines and flow that apply to most languages. +While Bazel is a [build system](https://bazel.build/basics/build-systems) that supports multi-language builds, this tutorial uses a C++ project as an example and provides the general guidelines and flow that apply to most languages. Estimated completion time: 30 minutes. ### Prerequisites -Start by [installing Bazel](https://bazel.build/install), if you haven't -already. This tutorial uses Git for source control, so for best results [install -Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. +Start by [installing Bazel](https://bazel.build/install), if you haven't already. This tutorial uses Git for source control, so for best results [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. -Next, retrieve the sample project from Bazel's GitHub repository by running the -following in your command-line tool of choice: +Next, retrieve the sample project from Bazel's GitHub repository by running the following in your command-line tool of choice: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/cpp-tutorial` -directory. +The sample project for this tutorial is in the `examples/cpp-tutorial` directory. Take a look at how it's structured: @@ -64,52 +52,24 @@ examples └── MODULE.bazel ``` -There are three sets of files, each set representing a stage in this tutorial. -In the first stage, you will build a single [target] -(https://bazel.build/reference/glossary#target) residing in a single [package] -(https://bazel.build/reference/glossary#package). In the second stage, you will -build both a binary and a library from a single package. In the third and final -stage, you will build a project with multiple packages and build it with -multiple targets. +There are three sets of files, each set representing a stage in this tutorial. In the first stage, you will build a single \[target] ([https://bazel.build/reference/glossary#target](https://bazel.build/reference/glossary#target)) residing in a single \[package] ([https://bazel.build/reference/glossary#package](https://bazel.build/reference/glossary#package)). In the second stage, you will build both a binary and a library from a single package. In the third and final stage, you will build a project with multiple packages and build it with multiple targets. ### Summary: Introduction -By installing Bazel (and Git) and cloning the repository for this tutorial, you -have laid the foundation for your first build with Bazel. Continue to the next -section to define some terms and set up your -[workspace](https://bazel.build/reference/glossary#workspace). +By installing Bazel (and Git) and cloning the repository for this tutorial, you have laid the foundation for your first build with Bazel. Continue to the next section to define some terms and set up your [workspace](https://bazel.build/reference/glossary#workspace). ## Getting started -Before you can build a project, you need to set up its workspace. A workspace -is a directory that holds your project's source files and Bazel's build outputs. -It also contains these significant files: +Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains these significant files: -* The `MODULE.bazel` file, which identifies the directory and its contents as - a Bazel workspace and lives at the root of the project's directory - structure. It's also where you specify your external dependencies. -* One or more [`BUILD` - files](https://bazel.build/reference/glossary#build-file), which tell Bazel - how to build different parts of the project. A directory within the - workspace that contains a `BUILD` file is a - [package](https://bazel.build/reference/glossary#package). (More on packages - later in this tutorial.) +- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure. It's also where you specify your external dependencies. +- One or more [`BUILD` files](https://bazel.build/reference/glossary#build-file), which tell Bazel how to build different parts of the project. A directory within the workspace that contains a `BUILD` file is a [package](https://bazel.build/reference/glossary#package). (More on packages later in this tutorial.) -In future projects, to designate a directory as a Bazel workspace, create an -empty file named `MODULE.bazel` in that directory. For the purposes of this -tutorial, a `MODULE.bazel` file is already present in each stage. +In future projects, to designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. For the purposes of this tutorial, a `MODULE.bazel` file is already present in each stage. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. Each -`BUILD` file requires at least one -[rule](https://bazel.build/reference/glossary#rule) as a set of instructions, -which tells Bazel how to build the outputs you want, such as executable binaries -or libraries. Each instance of a build rule in the `BUILD` file is called a -[target](https://bazel.build/reference/glossary#target) and points to a specific -set of source files and -[dependencies](https://bazel.build/reference/glossary#dependency). A target can -also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. Each `BUILD` file requires at least one [rule](https://bazel.build/reference/glossary#rule) as a set of instructions, which tells Bazel how to build the outputs you want, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a [target](https://bazel.build/reference/glossary#target) and points to a specific set of source files and [dependencies](https://bazel.build/reference/glossary#dependency). A target can also point to other targets. Take a look at the `BUILD` file in the `cpp-tutorial/stage1/main` directory: @@ -120,21 +80,15 @@ cc_binary( ) ``` -In our example, the `hello-world` target instantiates Bazel's built-in -[`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule -tells Bazel to build a self-contained executable binary from the -`hello-world.cc`> source file with no dependencies. +In our example, the `hello-world` target instantiates Bazel's built-in [`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule tells Bazel to build a self-contained executable binary from the `hello-world.cc`> source file with no dependencies. ### Summary: getting started -Now you are familiar with some key terms, and what they mean in the context of -this project and Bazel in general. In the next section, you will build and test -Stage 1 of the project. +Now you are familiar with some key terms, and what they mean in the context of this project and Bazel in general. In the next section, you will build and test Stage 1 of the project. ## Stage 1: single target, single package -It's time to build the first part of the project. For a visual reference, the -structure of the Stage 1 section of the project is: +It's time to build the first part of the project. For a visual reference, the structure of the Stage 1 section of the project is: ```none examples @@ -158,9 +112,7 @@ Next, run: bazel build //main:hello-world ``` -In the target label, the `//main:` part is the location of the `BUILD` file -relative to the root of the workspace, and `hello-world` is the target name in -the `BUILD` file. +In the target label, the `//main:` part is the location of the `BUILD` file relative to the root of the workspace, and `hello-world` is the target name in the `BUILD` file. Bazel produces something that looks like this: @@ -171,8 +123,7 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.267s, Critical Path: 0.25s ``` -You just built your first Bazel target. Bazel places build outputs in the -`bazel-bin` directory at the root of the workspace. +You just built your first Bazel target. Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Now test your freshly built binary, which is: @@ -184,23 +135,15 @@ This results in a printed "`Hello world`" message. Here's the dependency graph of Stage 1: -![Dependency graph for hello-world displays a single target with a single source -file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world -displays a single target with a single source file.") +![Dependency graph for hello-world displays a single target with a single source file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world displays a single target with a single source file.") ### Summary: stage 1 -Now that you have completed your first build, you have a basic idea of how a -build is structured. In the next stage, you will add complexity by adding -another target. +Now that you have completed your first build, you have a basic idea of how a build is structured. In the next stage, you will add complexity by adding another target. ## Stage 2: multiple build targets -While a single target is sufficient for small projects, you may want to split -larger projects into multiple targets and packages. This allows for fast -incremental builds – that is, Bazel only rebuilds what's changed – and speeds up -your builds by building multiple parts of a project at once. This stage of the -tutorial adds a target, and the next adds a package. +While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages. This allows for fast incremental builds – that is, Bazel only rebuilds what's changed – and speeds up your builds by building multiple parts of a project at once. This stage of the tutorial adds a target, and the next adds a package. This is the directory you are working with for Stage 2: @@ -232,15 +175,9 @@ cc_binary( ) ``` -With this `BUILD` file, Bazel first builds the `hello-greet` library (using -Bazel's built-in [`cc_library` -rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the -`hello-world` binary. The `deps` attribute in the `hello-world` target tells -Bazel that the `hello-greet` library is required to build the `hello-world` -binary. +With this `BUILD` file, Bazel first builds the `hello-greet` library (using Bazel's built-in [`cc_library` rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the `hello-world` binary. The `deps` attribute in the `hello-world` target tells Bazel that the `hello-greet` library is required to build the `hello-world` binary. -Before you can build this new version of the project, you need to change -directories, switching to the `cpp-tutorial/stage2` directory by running: +Before you can build this new version of the project, you need to change directories, switching to the `cpp-tutorial/stage2` directory by running: ```posix-terminal cd ../stage2 @@ -261,36 +198,25 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.399s, Critical Path: 0.30s ``` -Now you can test your freshly built binary, which returns another "`Hello -world`": +Now you can test your freshly built binary, which returns another "`Hello world`": ```posix-terminal bazel-bin/main/hello-world ``` -If you now modify `hello-greet.cc` and rebuild the project, Bazel only -recompiles that file. +If you now modify `hello-greet.cc` and rebuild the project, Bazel only recompiles that file. -Looking at the dependency graph, you can see that `hello-world` depends on an -extra input named `hello-greet`: +Looking at the dependency graph, you can see that `hello-world` depends on an extra input named `hello-greet`: -![Dependency graph for `hello-world` displays dependency changes after -modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency -graph for `hello-world` displays dependency changes after modification to the -file.") +![Dependency graph for hello-world displays dependency changes after modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency graph for `hello-world` displays dependency changes after modification to the file.") ### Summary: stage 2 -You've now built the project with two targets. The `hello-world` target builds -one source file and depends on one other target (`//main:hello-greet`), which -builds two additional source files. In the next section, take it a step further -and add another package. +You've now built the project with two targets. The `hello-world` target builds one source file and depends on one other target (`//main:hello-greet`), which builds two additional source files. In the next section, take it a step further and add another package. ## Stage 3: multiple packages -This next stage adds another layer of complication and builds a project with -multiple packages. Take a look at the structure and contents of the -`cpp-tutorial/stage3` directory: +This next stage adds another layer of complication and builds a project with multiple packages. Take a look at the structure and contents of the `cpp-tutorial/stage3` directory: ```none └──stage3 @@ -306,9 +232,7 @@ multiple packages. Take a look at the structure and contents of the └── MODULE.bazel ``` -You can see that now there are two sub-directories, and each contains a `BUILD` -file. Therefore, to Bazel, the workspace now contains two packages: `lib` and -`main`. +You can see that now there are two sub-directories, and each contains a `BUILD` file. Therefore, to Bazel, the workspace now contains two packages: `lib` and `main`. Take a look at the `lib/BUILD` file: @@ -340,25 +264,13 @@ cc_binary( ) ``` -The `hello-world` target in the main package depends on the` hello-time` target -in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows -this through the `deps` attribute. You can see this reflected in the dependency -graph: +The `hello-world` target in the main package depends on the` hello-time` target in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows this through the `deps` attribute. You can see this reflected in the dependency graph: -![Dependency graph for `hello-world` displays how the target in the main package -depends on the target in the `lib` -package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for -`hello-world` displays how the target in the main package depends on the target -in the `lib` package.") +![Dependency graph for hello-world displays how the target in the main package depends on the target in the lib package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for `hello-world` displays how the target in the main package depends on the target in the `lib` package.") -For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` -explicitly visible to targets in `main/BUILD` using the visibility attribute. -This is because by default targets are only visible to other targets in the same -`BUILD` file. Bazel uses target visibility to prevent issues such as libraries -containing implementation details leaking into public APIs. +For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` explicitly visible to targets in `main/BUILD` using the visibility attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs. -Now build this final version of the project. Switch to the `cpp-tutorial/stage3` -directory by running: +Now build this final version of the project. Switch to the `cpp-tutorial/stage3` directory by running: ```posix-terminal cd ../stage3 @@ -387,25 +299,15 @@ bazel-bin/main/hello-world ### Summary: stage 3 -You've now built the project as two packages with three targets and understand -the dependencies between them, which equips you to go forth and build future -projects with Bazel. In the next section, take a look at how to continue your -Bazel journey. +You've now built the project as two packages with three targets and understand the dependencies between them, which equips you to go forth and build future projects with Bazel. In the next section, take a look at how to continue your Bazel journey. ## Next steps -You've now completed your first basic build with Bazel, but this is just the -start. Here are some more resources to continue learning with Bazel: - -* To keep focusing on C++, read about common [C++ build use - cases](https://bazel.build/tutorials/cpp-use-cases). -* To get started with building other applications with Bazel, see the - tutorials for [Java](https://bazel.build/start/java), [Android - application](https://bazel.build/start/android-app), or [iOS - application](https://bazel.build/start/ios-app). -* To learn more about working with local and remote repositories, read about - [external dependencies](https://bazel.build/docs/external). -* To learn more about Bazel's other rules, see this [reference - guide](https://bazel.build/rules). +You've now completed your first basic build with Bazel, but this is just the start. Here are some more resources to continue learning with Bazel: + +- To keep focusing on C++, read about common [C++ build use cases](https://bazel.build/tutorials/cpp-use-cases). +- To get started with building other applications with Bazel, see the tutorials for [Java](https://bazel.build/start/java), [Android application](https://bazel.build/start/android-app), or [iOS application](https://bazel.build/start/ios-app). +- To learn more about working with local and remote repositories, read about [external dependencies](https://bazel.build/docs/external). +- To learn more about Bazel's other rules, see this [reference guide](https://bazel.build/rules). Happy building! diff --git a/start/go.mdx b/start/go.mdx index 3a095681..d5742314 100644 --- a/start/go.mdx +++ b/start/go.mdx @@ -2,12 +2,7 @@ title: 'Bazel Tutorial: Build a Go Project' --- - - -This tutorial introduces you to the basics of Bazel by showing you how to build -a Go (Golang) project. You'll learn how to set up your workspace, build a small -program, import a library, and run its test. Along the way, you'll learn key -Bazel concepts, such as targets and `BUILD` files. +This tutorial introduces you to the basics of Bazel by showing you how to build a Go (Golang) project. You'll learn how to set up your workspace, build a small program, import a library, and run its test. Along the way, you'll learn key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes @@ -15,35 +10,27 @@ Estimated completion time: 30 minutes ### Install Bazel -Before you get started, first [install bazel](/install) if you haven't done so -already. +Before you get started, first [install bazel](/install) if you haven't done so already. You can check if Bazel is installed by running `bazel version` in any directory. ### Install Go (optional) -You don't need to [install Go](https://go.dev/doc/install) to build Go projects -with Bazel. The Bazel Go rule set automatically downloads and uses a Go -toolchain instead of using the toolchain installed on your machine. This ensures -all developers on a project build with same version of Go. +You don't need to [install Go](https://go.dev/doc/install) to build Go projects with Bazel. The Bazel Go rule set automatically downloads and uses a Go toolchain instead of using the toolchain installed on your machine. This ensures all developers on a project build with same version of Go. -However, you may still want to install a Go toolchain to run commands like `go -get` and `go mod tidy`. +However, you may still want to install a Go toolchain to run commands like `go get` and `go mod tidy`. You can check if Go is installed by running `go version` in any directory. ### Get the sample project -The Bazel examples are stored in a Git repository, so you'll need to [install -Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you -haven't already. To download the examples repository, run this command: +The Bazel examples are stored in a Git repository, so you'll need to [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you haven't already. To download the examples repository, run this command: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/go-tutorial` directory. -See what it contains: +The sample project for this tutorial is in the `examples/go-tutorial` directory. See what it contains: ```none go-tutorial/ @@ -52,13 +39,11 @@ go-tutorial/ └── stage3 ``` -There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a -different section of this tutorial. Each stage builds on the previous one. +There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a different section of this tutorial. Each stage builds on the previous one. ## Build with Bazel -Start in the `stage1` directory, where we'll find a program. We can -build it with `bazel build`, then run it: +Start in the `stage1` directory, where we'll find a program. We can build it with `bazel build`, then run it: ```posix-shell $ cd go-tutorial/stage1/ @@ -107,9 +92,7 @@ func main() { } ``` -`BUILD` contains some instructions for Bazel, telling it what we want to build. -You'll typically write a file like this in each directory. For this project, we -have a single `go_binary` target that builds our program from `hello.go`. +`BUILD` contains some instructions for Bazel, telling it what we want to build. You'll typically write a file like this in each directory. For this project, we have a single `go_binary` target that builds our program from `hello.go`. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -120,17 +103,9 @@ go_binary( ) ``` -`MODULE.bazel` tracks your project's dependencies. It also marks your project's -root directory, so you'll only write one `MODULE.bazel` file per project. It -serves a similar purpose to Go's `go.mod` file. You don't actually need a -`go.mod` file in a Bazel project, but it may still be useful to have one so that -you can continue using `go get` and `go mod tidy` for dependency management. The -Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in -another tutorial. +`MODULE.bazel` tracks your project's dependencies. It also marks your project's root directory, so you'll only write one `MODULE.bazel` file per project. It serves a similar purpose to Go's `go.mod` file. You don't actually need a `go.mod` file in a Bazel project, but it may still be useful to have one so that you can continue using `go get` and `go mod tidy` for dependency management. The Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in another tutorial. -Our `MODULE.bazel` file contains a single dependency on -[rules_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need -this dependency because Bazel doesn't have built-in support for Go. +Our `MODULE.bazel` file contains a single dependency on [rules\_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need this dependency because Bazel doesn't have built-in support for Go. ```bazel bazel_dep( @@ -139,50 +114,25 @@ bazel_dep( ) ``` -Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes -and other metadata about our dependencies. It includes implicit dependencies -added by Bazel itself, so it's quite long, and we won't show it here. Just like -`go.sum`, you should commit your `MODULE.bazel.lock` file to source control to -ensure everyone on your project gets the same version of each dependency. You -shouldn't need to edit `MODULE.bazel.lock` manually. +Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes and other metadata about our dependencies. It includes implicit dependencies added by Bazel itself, so it's quite long, and we won't show it here. Just like `go.sum`, you should commit your `MODULE.bazel.lock` file to source control to ensure everyone on your project gets the same version of each dependency. You shouldn't need to edit `MODULE.bazel.lock` manually. ### Understand the BUILD file -Most of your interaction with Bazel will be through `BUILD` files (or -equivalently, `BUILD.bazel` files), so it's important to understand what they -do. +Most of your interaction with Bazel will be through `BUILD` files (or equivalently, `BUILD.bazel` files), so it's important to understand what they do. -`BUILD` files are written in a scripting language called -[Starlark](https://bazel.build/rules/language), a limited subset of Python. +`BUILD` files are written in a scripting language called [Starlark](https://bazel.build/rules/language), a limited subset of Python. -A `BUILD` file contains a list of -[targets](https://bazel.build/reference/glossary#target). A target is something -Bazel can build, like a binary, library, or test. +A `BUILD` file contains a list of [targets](https://bazel.build/reference/glossary#target). A target is something Bazel can build, like a binary, library, or test. -A target calls a rule function with a list of -[attributes](https://bazel.build/reference/glossary#attribute) to describe what -should be built. Our example has two attributes: `name` identifies the target on -the command line, and `srcs` is a list of source file paths (slash-separated, -relative to the directory containing the `BUILD` file). +A target calls a rule function with a list of [attributes](https://bazel.build/reference/glossary#attribute) to describe what should be built. Our example has two attributes: `name` identifies the target on the command line, and `srcs` is a list of source file paths (slash-separated, relative to the directory containing the `BUILD` file). -A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a -target. In our example, we used the -[`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) -rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) -(commands) that generate a set of output files. For example, `go_binary` defines -Go compile and link actions that produce an executable output file. +A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a target. In our example, we used the [`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) (commands) that generate a set of output files. For example, `go_binary` defines Go compile and link actions that produce an executable output file. -Bazel has built-in rules for a few languages like Java and C++. You can find -their [documentation in the Build -Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find -rule sets for many other languages and tools on the [Bazel Central Registry -(BCR)](https://registry.bazel.build/). +Bazel has built-in rules for a few languages like Java and C++. You can find their [documentation in the Build Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find rule sets for many other languages and tools on the [Bazel Central Registry (BCR)](https://registry.bazel.build/). ## Add a library -Move onto the `stage2` directory, where we'll build a new program that -prints your fortune. This program uses a separate Go package as a library that -selects a fortune from a predefined list of messages. +Move onto the `stage2` directory, where we'll build a new program that prints your fortune. This program uses a separate Go package as a library that selects a fortune from a predefined list of messages. ```none go-tutorial/stage2 @@ -195,11 +145,7 @@ go-tutorial/stage2 └── print_fortune.go ``` -`fortune.go` is the source file for the library. The `fortune` library is a -separate Go package, so its source files are in a separate directory. Bazel -doesn't require you to keep Go packages in separate directories, but it's a -strong convention in the Go ecosystem, and following it will help you stay -compatible with other Go tools. +`fortune.go` is the source file for the library. The `fortune` library is a separate Go package, so its source files are in a separate directory. Bazel doesn't require you to keep Go packages in separate directories, but it's a strong convention in the Go ecosystem, and following it will help you stay compatible with other Go tools. ```go package fortune @@ -217,19 +163,11 @@ func Get() string { } ``` -The `fortune` directory has its own `BUILD` file that tells Bazel how to build -this package. We use `go_library` here instead of `go_binary`. +The `fortune` directory has its own `BUILD` file that tells Bazel how to build this package. We use `go_library` here instead of `go_binary`. -We also need to set the `importpath` attribute to a string with which the -library can be imported into other Go source files. This name should be the -repository path (or module path) concatenated with the directory within the -repository. +We also need to set the `importpath` attribute to a string with which the library can be imported into other Go source files. This name should be the repository path (or module path) concatenated with the directory within the repository. -Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. -[`visibility`](https://bazel.build/concepts/visibility) may be set on any -target. It determines which Bazel packages may depend on this target. In our -case, we want any target to be able to depend on this library, so we use the -special value `//visibility:public`. +Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. [`visibility`](https://bazel.build/concepts/visibility) may be set on any target. It determines which Bazel packages may depend on this target. In our case, we want any target to be able to depend on this library, so we use the special value `//visibility:public`. ```bazel load("@rules_go//go:def.bzl", "go_library") @@ -264,11 +202,9 @@ func main() { } ``` -`print_fortune.go` imports the package using the same string declared in the -`importpath` attribute of the `fortune` library. +`print_fortune.go` imports the package using the same string declared in the `importpath` attribute of the `fortune` library. -We also need to declare this dependency to Bazel. Here's the `BUILD` file in the -`stage2` directory. +We also need to declare this dependency to Bazel. Here's the `BUILD` file in the `stage2` directory. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -286,57 +222,25 @@ You can run this with the command below. bazel run //:print_fortune ``` -The `print_fortune` target has a `deps` attribute, a list of other targets that -it depends on. It contains `"//fortune"`, a label string referring to the target -in the `fortune` directory named `fortune`. +The `print_fortune` target has a `deps` attribute, a list of other targets that it depends on. It contains `"//fortune"`, a label string referring to the target in the `fortune` directory named `fortune`. -Bazel requires that all targets declare their dependencies explicitly with -attributes like `deps`. This may seem cumbersome since dependencies are *also* -specified in source files, but Bazel's explictness gives it an advantage. Bazel -builds an [action graph](https://bazel.build/reference/glossary#action-graph) -containing all commands, inputs, and outputs before running any commands, -without reading any source files. Bazel can then cache action results or send -actions for [remote execution](https://bazel.build/remote/rbe) without built-in -language-specific logic. +Bazel requires that all targets declare their dependencies explicitly with attributes like `deps`. This may seem cumbersome since dependencies are *also* specified in source files, but Bazel's explictness gives it an advantage. Bazel builds an [action graph](https://bazel.build/reference/glossary#action-graph) containing all commands, inputs, and outputs before running any commands, without reading any source files. Bazel can then cache action results or send actions for [remote execution](https://bazel.build/remote/rbe) without built-in language-specific logic. ### Understanding labels -A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses -to identify a target or a file. Labels are used in command line arguments and in -`BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, -`//:print-fortune`, and `@rules_go//go:def.bzl`. - -A label has three parts: a repository name, a package name, and a target (or -file) name. - -The repository name is written between `@` and `//` and is used to refer to a -target from a different Bazel module (for historical reasons, *module* and -*repository* are sometimes used synonymously). In the label, -`@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name -can be omitted when referring to targets in the same repository. - -The package name is written between `//` and `:` and is used to refer to a -target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, -the package name is `go`. A Bazel -[package](https://bazel.build/reference/glossary#package) is a set of files and -targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. -Its package name is a slash-separated path from the module root directory -(containing `MODULE.bazel`) to the directory containing the `BUILD` file. A -package may include subdirectories, but only if they don't also contain `BUILD` -files defining their own packages. - -Most Go projects have one `BUILD` file per directory and one Go package per -`BUILD` file. The package name in a label may be omitted when referring to -targets in the same directory. - -The target name is written after `:` and refers to a target within a package. -The target name may be omitted if it's the same as the last component of the -package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is -the same as `//fortune`). - -On the command-line, you can use `...` as a wildcard to refer to all the targets -within a package. This is useful for building or testing all the targets in a -repository. +A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses to identify a target or a file. Labels are used in command line arguments and in `BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, `//:print-fortune`, and `@rules_go//go:def.bzl`. + +A label has three parts: a repository name, a package name, and a target (or file) name. + +The repository name is written between `@` and `//` and is used to refer to a target from a different Bazel module (for historical reasons, *module* and *repository* are sometimes used synonymously). In the label, `@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name can be omitted when referring to targets in the same repository. + +The package name is written between `//` and `:` and is used to refer to a target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, the package name is `go`. A Bazel [package](https://bazel.build/reference/glossary#package) is a set of files and targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. Its package name is a slash-separated path from the module root directory (containing `MODULE.bazel`) to the directory containing the `BUILD` file. A package may include subdirectories, but only if they don't also contain `BUILD` files defining their own packages. + +Most Go projects have one `BUILD` file per directory and one Go package per `BUILD` file. The package name in a label may be omitted when referring to targets in the same directory. + +The target name is written after `:` and refers to a target within a package. The target name may be omitted if it's the same as the last component of the package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is the same as `//fortune`). + +On the command-line, you can use `...` as a wildcard to refer to all the targets within a package. This is useful for building or testing all the targets in a repository. ```posix-shell # Build everything @@ -372,15 +276,13 @@ import ( // TestGet checks that Get returns one of the strings from fortunes. func TestGet(t *testing.T) { msg := Get() - if i := slices.Index(fortunes, msg); i < 0 { + if i := slices.Index(fortunes, msg); i < 0 { t.Errorf("Get returned %q, not one the expected messages", msg) } } ``` -This file uses the unexported `fortunes` variable, so it needs to be compiled -into the same Go package as `fortune.go`. Look at the `BUILD` file to see -how that works: +This file uses the unexported `fortunes` variable, so it needs to be compiled into the same Go package as `fortune.go`. Look at the `BUILD` file to see how that works: ```bazel load("@rules_go//go:def.bzl", "go_library", "go_test") @@ -399,19 +301,9 @@ go_test( ) ``` -We have a new `fortune_test` target that uses the `go_test` rule to compile and -link a test executable. `go_test` needs to compile `fortune.go` and -`fortune_test.go` together with the same command, so we use the `embed` -attribute here to incorporate the attributes of the `fortune` target into -`fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, -but it also works with `go_library`, which is sometimes useful for generated -code. +We have a new `fortune_test` target that uses the `go_test` rule to compile and link a test executable. `go_test` needs to compile `fortune.go` and `fortune_test.go` together with the same command, so we use the `embed` attribute here to incorporate the attributes of the `fortune` target into `fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, but it also works with `go_library`, which is sometimes useful for generated code. -You may be wondering if the `embed` attribute is related to Go's -[`embed`](https://pkg.go.dev/embed) package, which is used to access data files -copied into an executable. This is an unfortunate name collision: rules_go's -`embed` attribute was introduced before Go's `embed` package. Instead, rules_go -uses the `embedsrcs` to list files that can be loaded with the `embed` package. +You may be wondering if the `embed` attribute is related to Go's [`embed`](https://pkg.go.dev/embed) package, which is used to access data files copied into an executable. This is an unfortunate name collision: rules\_go's `embed` attribute was introduced before Go's `embed` package. Instead, rules\_go uses the `embedsrcs` to list files that can be loaded with the `embed` package. Try running our test with `bazel test`: @@ -430,9 +322,7 @@ Executed 0 out of 1 test: 1 test passes. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. ``` -You can use the `...` wildcard to run all tests. Bazel will also build targets -that aren't tests, so this can catch compile errors even in packages that don't -have tests. +You can use the `...` wildcard to run all tests. Bazel will also build targets that aren't tests, so this can catch compile errors even in packages that don't have tests. ```posix-shell $ bazel test //... @@ -440,21 +330,9 @@ $ bazel test //... ## Conclusion and further reading -In this tutorial, we built and tested a small Go project with Bazel, and we -learned some core Bazel concepts along the way. - -- To get started building other applications with Bazel, see the tutorials for - [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and - [iOS](/start/ios-app). -- You can also check the list of [recommended rules](/rules) for other - languages. -- For more information on Go, see the - [rules_go](https://github.com/bazel-contrib/rules_go) module, especially the - [Core Go - rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) - documentation. -- To learn more about working with Bazel modules outside your project, see - [external dependencies](/docs/external). In particular, for information on - how to depend on Go modules and toolchains through Bazel's module system, - see [Go with - bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). +In this tutorial, we built and tested a small Go project with Bazel, and we learned some core Bazel concepts along the way. + +- To get started building other applications with Bazel, see the tutorials for [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and [iOS](/start/ios-app). +- You can also check the list of [recommended rules](/rules) for other languages. +- For more information on Go, see the [rules\_go](https://github.com/bazel-contrib/rules_go) module, especially the [Core Go rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) documentation. +- To learn more about working with Bazel modules outside your project, see [external dependencies](/docs/external). In particular, for information on how to depend on Go modules and toolchains through Bazel's module system, see [Go with bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). diff --git a/start/ios-app.mdx b/start/ios-app.mdx index 0b860ab6..5004bbd2 100644 --- a/start/ios-app.mdx +++ b/start/ios-app.mdx @@ -2,5 +2,4 @@ title: 'Bazel Tutorial: Build an iOS App' --- - -This tutorial has been moved into the [bazelbuild/rules_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. +This tutorial has been moved into the [bazelbuild/rules\_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. diff --git a/start/java.mdx b/start/java.mdx index b892917d..c9c5916c 100644 --- a/start/java.mdx +++ b/start/java.mdx @@ -2,11 +2,7 @@ title: 'Bazel Tutorial: Build a Java Project' --- - - -This tutorial covers the basics of building Java applications with -Bazel. You will set up your workspace and build a simple Java project that -illustrates key Bazel concepts, such as targets and `BUILD` files. +This tutorial covers the basics of building Java applications with Bazel. You will set up your workspace and build a simple Java project that illustrates key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes. @@ -14,36 +10,40 @@ Estimated completion time: 30 minutes. In this tutorial you learn how to: -* Build a target -* Visualize the project's dependencies -* Split the project into multiple targets and packages -* Control target visibility across packages -* Reference targets through labels -* Deploy a target +- Build a target +- Visualize the project's dependencies +- Split the project into multiple targets and packages +- Control target visibility across packages +- Reference targets through labels +- Deploy a target ## Before you begin ### Install Bazel -To prepare for the tutorial, first [Install Bazel](/install) if -you don't have it installed already. +To prepare for the tutorial, first [Install Bazel](/install) if you don't have it installed already. ### Install the JDK -1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). +1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). + +2. Set the JAVA\_HOME environment variable to point to the JDK. + + - On Linux/macOS: + + ``` + export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" + ``` -2. Set the JAVA\_HOME environment variable to point to the JDK. - * On Linux/macOS: + - On Windows: - export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" - * On Windows: - 1. Open Control Panel. - 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . - 3. Under the "User variables" list (the one on the top), click "New...". - 4. In the "Variable name" field, enter `JAVA_HOME`. - 5. Click "Browse Directory...". - 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). - 7. Click "OK" on all dialog windows. + 1. Open Control Panel. + 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . + 3. Under the "User variables" list (the one on the top), click "New\...". + 4. In the "Variable name" field, enter `JAVA_HOME`. + 5. Click "Browse Directory...". + 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). + 7. Click "OK" on all dialog windows. ### Get the sample project @@ -53,8 +53,7 @@ Retrieve the sample project from Bazel's GitHub repository: git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/java-tutorial` -directory and is structured as follows: +The sample project for this tutorial is in the `examples/java-tutorial` directory and is structured as follows: ``` java-tutorial @@ -76,32 +75,19 @@ java-tutorial ### Set up the workspace -Before you can build a project, you need to set up its workspace. A workspace is -a directory that holds your project's source files and Bazel's build outputs. It -also contains files that Bazel recognizes as special: +Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains files that Bazel recognizes as special: -* The `MODULE.bazel` file, which identifies the directory and its contents as a - Bazel workspace and lives at the root of the project's directory structure, +- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure, -* One or more `BUILD` files, which tell Bazel how to build different parts of - the project. (A directory within the workspace that contains a `BUILD` file - is a *package*. You will learn about packages later in this tutorial.) +- One or more `BUILD` files, which tell Bazel how to build different parts of the project. (A directory within the workspace that contains a `BUILD` file is a *package*. You will learn about packages later in this tutorial.) -To designate a directory as a Bazel workspace, create an empty file named -`MODULE.bazel` in that directory. +To designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. -When Bazel builds the project, all inputs and dependencies must be in the same -workspace. Files residing in different workspaces are independent of one -another unless linked, which is beyond the scope of this tutorial. +When Bazel builds the project, all inputs and dependencies must be in the same workspace. Files residing in different workspaces are independent of one another unless linked, which is beyond the scope of this tutorial. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. -The most important type is the *build rule*, which tells Bazel how to build the -desired outputs, such as executable binaries or libraries. Each instance -of a build rule in the `BUILD` file is called a *target* and points to a -specific set of source files and dependencies. A target can also point to other -targets. +A `BUILD` file contains several different types of instructions for Bazel. The most important type is the *build rule*, which tells Bazel how to build the desired outputs, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a *target* and points to a specific set of source files and dependencies. A target can also point to other targets. Take a look at the `java-tutorial/BUILD` file: @@ -112,30 +98,19 @@ java_binary( ) ``` -In our example, the `ProjectRunner` target instantiates Bazel's built-in -[`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to -build a `.jar` file and a wrapper shell script (both named after the target). +In our example, the `ProjectRunner` target instantiates Bazel's built-in [`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to build a `.jar` file and a wrapper shell script (both named after the target). -The attributes in the target explicitly state its dependencies and options. -While the `name` attribute is mandatory, many are optional. For example, in the -`ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies -the source files that Bazel uses to build the target, and `main_class` specifies -the class that contains the main method. (You may have noticed that our example -uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel -instead of listing them one by one.) +The attributes in the target explicitly state its dependencies and options. While the `name` attribute is mandatory, many are optional. For example, in the `ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies the source files that Bazel uses to build the target, and `main_class` specifies the class that contains the main method. (You may have noticed that our example uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel instead of listing them one by one.) ### Build the project -To build your sample project, navigate to the `java-tutorial` directory -and run: +To build your sample project, navigate to the `java-tutorial` directory and run: ```posix-terminal bazel build //:ProjectRunner ``` -In the target label, the `//` part is the location of the `BUILD` file -relative to the root of the workspace (in this case, the root itself), -and `ProjectRunner` is the target name in the `BUILD` file. (You will -learn about target labels in more detail at the end of this tutorial.) + +In the target label, the `//` part is the location of the `BUILD` file relative to the root of the workspace (in this case, the root itself), and `ProjectRunner` is the target name in the `BUILD` file. (You will learn about target labels in more detail at the end of this tutorial.) Bazel produces output similar to the following: @@ -147,9 +122,7 @@ Bazel produces output similar to the following: INFO: Elapsed time: 1.021s, Critical Path: 0.83s ``` -Congratulations, you just built your first Bazel target! Bazel places build -outputs in the `bazel-bin` directory at the root of the workspace. Browse -through its contents to get an idea for Bazel's output structure. +Congratulations, you just built your first Bazel target! Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Browse through its contents to get an idea for Bazel's output structure. Now test your freshly built binary: @@ -159,43 +132,31 @@ bazel-bin/ProjectRunner ### Review the dependency graph -Bazel requires build dependencies to be explicitly declared in BUILD files. -Bazel uses those statements to create the project's dependency graph, which -enables accurate incremental builds. +Bazel requires build dependencies to be explicitly declared in BUILD files. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text -representation of the dependency graph by running this command at the -workspace root: +To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: ```posix-terminal bazel query --notool_deps --noimplicit_deps "deps(//:ProjectRunner)" --output graph ``` -The above command tells Bazel to look for all dependencies for the target -`//:ProjectRunner` (excluding host and implicit dependencies) and format the -output as a graph. +The above command tells Bazel to look for all dependencies for the target `//:ProjectRunner` (excluding host and implicit dependencies) and format the output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -As you can see, the project has a single target that build two source files with -no additional dependencies: +As you can see, the project has a single target that build two source files with no additional dependencies: ![Dependency graph of the target 'ProjectRunner'](/docs/images/tutorial_java_01.svg) -After you set up your workspace, build your project, and examine its -dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. ## Refine your Bazel build -While a single target is sufficient for small projects, you may want to split -larger projects into multiple targets and packages to allow for fast incremental -builds (that is, only rebuild what's changed) and to speed up your builds by -building multiple parts of a project at once. +While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages to allow for fast incremental builds (that is, only rebuild what's changed) and to speed up your builds by building multiple parts of a project at once. ### Specify multiple build targets -You can split the sample project build into two targets. Replace the contents of -the `java-tutorial/BUILD` file with the following: +You can split the sample project build into two targets. Replace the contents of the `java-tutorial/BUILD` file with the following: ```python java_binary( @@ -211,9 +172,7 @@ java_library( ) ``` -With this configuration, Bazel first builds the `greeter` library, then the -`ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that -the `greeter` library is required to build the `ProjectRunner` binary. +With this configuration, Bazel first builds the `greeter` library, then the `ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that the `greeter` library is required to build the `ProjectRunner` binary. To build this new version of the project, run the following command: @@ -237,26 +196,17 @@ Now test your freshly built binary: bazel-bin/ProjectRunner ``` -If you now modify `ProjectRunner.java` and rebuild the project, Bazel only -recompiles that file. +If you now modify `ProjectRunner.java` and rebuild the project, Bazel only recompiles that file. -Looking at the dependency graph, you can see that `ProjectRunner` depends on the -same inputs as it did before, but the structure of the build is different: +Looking at the dependency graph, you can see that `ProjectRunner` depends on the same inputs as it did before, but the structure of the build is different: -![Dependency graph of the target 'ProjectRunner' after adding a dependency]( -/docs/images/tutorial_java_02.svg) +![Dependency graph of the target 'ProjectRunner' after adding a dependency](/docs/images/tutorial_java_02.svg) -You've now built the project with two targets. The `ProjectRunner` target builds -one source files and depends on one other target (`:greeter`), which builds -one additional source file. +You've now built the project with two targets. The `ProjectRunner` target builds one source files and depends on one other target (`:greeter`), which builds one additional source file. ### Use multiple packages -Let’s now split the project into multiple packages. If you take a look at the -`src/main/java/com/example/cmdline` directory, you can see that it also contains -a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now -contains two packages, `//src/main/java/com/example/cmdline` and `//` (since -there is a `BUILD` file at the root of the workspace). +Let’s now split the project into multiple packages. If you take a look at the `src/main/java/com/example/cmdline` directory, you can see that it also contains a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now contains two packages, `//src/main/java/com/example/cmdline` and `//` (since there is a `BUILD` file at the root of the workspace). Take a look at the `src/main/java/com/example/cmdline/BUILD` file: @@ -269,21 +219,13 @@ java_binary( ) ``` -The `runner` target depends on the `greeter` target in the `//` package (hence -the target label `//:greeter`) - Bazel knows this through the `deps` attribute. -Take a look at the dependency graph: +The `runner` target depends on the `greeter` target in the `//` package (hence the target label `//:greeter`) - Bazel knows this through the `deps` attribute. Take a look at the dependency graph: ![Dependency graph of the target 'runner'](/docs/images/tutorial_java_03.svg) -However, for the build to succeed, you must explicitly give the `runner` target -in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in -`//BUILD` using the `visibility` attribute. This is because by default targets -are only visible to other targets in the same `BUILD` file. (Bazel uses target -visibility to prevent issues such as libraries containing implementation details -leaking into public APIs.) +However, for the build to succeed, you must explicitly give the `runner` target in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in `//BUILD` using the `visibility` attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. (Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs.) -To do this, add the `visibility` attribute to the `greeter` target in -`java-tutorial/BUILD` as shown below: +To do this, add the `visibility` attribute to the `greeter` target in `java-tutorial/BUILD` as shown below: ```python java_library( @@ -293,8 +235,7 @@ java_library( ) ``` -Now you can build the new package by running the following command at the root -of the workspace: +Now you can build the new package by running the following command at the root of the workspace: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner @@ -316,48 +257,29 @@ Now test your freshly built binary: ./bazel-bin/src/main/java/com/example/cmdline/runner ``` -You've now modified the project to build as two packages, each containing one -target, and understand the dependencies between them. - +You've now modified the project to build as two packages, each containing one target, and understand the dependencies between them. ## Use labels to reference targets -In `BUILD` files and at the command line, Bazel uses target labels to reference -targets - for example, `//:ProjectRunner` or -`//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: +In `BUILD` files and at the command line, Bazel uses target labels to reference targets - for example, `//:ProjectRunner` or `//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path to the -directory containing the `BUILD` file, and `target-name` is what you named the -target in the `BUILD` file (the `name` attribute). If the target is a file -target, then `path/to/package` is the path to the root of the package, and -`target-name` is the name of the target file, including its full path. +If the target is a rule target, then `path/to/package` is the path to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path. -When referencing targets at the repository root, the package path is empty, -just use `//:target-name`. When referencing targets within the same `BUILD` -file, you can even skip the `//` workspace root identifier and just use -`:target-name`. +When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. -For example, for targets in the `java-tutorial/BUILD` file, you did not have to -specify a package path, since the workspace root is itself a package (`//`), and -your two target labels were simply `//:ProjectRunner` and `//:greeter`. +For example, for targets in the `java-tutorial/BUILD` file, you did not have to specify a package path, since the workspace root is itself a package (`//`), and your two target labels were simply `//:ProjectRunner` and `//:greeter`. -However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you -had to specify the full package path of `//src/main/java/com/example/cmdline` -and your target label was `//src/main/java/com/example/cmdline:runner`. +However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you had to specify the full package path of `//src/main/java/com/example/cmdline` and your target label was `//src/main/java/com/example/cmdline:runner`. ## Package a Java target for deployment -Let’s now package a Java target for deployment by building the binary with all -of its runtime dependencies. This lets you run the binary outside of your -development environment. +Let’s now package a Java target for deployment by building the binary with all of its runtime dependencies. This lets you run the binary outside of your development environment. -As you remember, the [java_binary](/reference/be/java#java_binary) build rule -produces a `.jar` and a wrapper shell script. Take a look at the contents of -`runner.jar` using this command: +As you remember, the [java\_binary](/reference/be/java#java_binary) build rule produces a `.jar` and a wrapper shell script. Take a look at the contents of `runner.jar` using this command: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner.jar @@ -373,12 +295,8 @@ com/example/ com/example/cmdline/ com/example/cmdline/Runner.class ``` -As you can see, `runner.jar` contains `Runner.class`, but not its dependency, -`Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` -to the classpath, so if you leave it like this, it will run locally, but it -won't run standalone on another machine. Fortunately, the `java_binary` rule -allows you to build a self-contained, deployable binary. To build it, append -`_deploy.jar` to the target name: + +As you can see, `runner.jar` contains `Runner.class`, but not its dependency, `Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` to the classpath, so if you leave it like this, it will run locally, but it won't run standalone on another machine. Fortunately, the `java_binary` rule allows you to build a self-contained, deployable binary. To build it, append `_deploy.jar` to the target name: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner_deploy.jar @@ -392,10 +310,8 @@ Target //src/main/java/com/example/cmdline:runner_deploy.jar up-to-date: bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar INFO: Elapsed time: 1.700s, Critical Path: 0.23s ``` -You have just built `runner_deploy.jar`, which you can run standalone away from -your development environment since it contains the required runtime -dependencies. Take a look at the contents of this standalone JAR using the -same command as before: + +You have just built `runner_deploy.jar`, which you can run standalone away from your development environment since it contains the required runtime dependencies. Take a look at the contents of this standalone JAR using the same command as before: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar @@ -418,19 +334,14 @@ com/example/Greeting.class For more details, see: -* [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) for - rules to manage transitive Maven dependencies. +- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) for rules to manage transitive Maven dependencies. -* [External Dependencies](/docs/external) to learn more about working with - local and remote repositories. +- [External Dependencies](/docs/external) to learn more about working with local and remote repositories. -* The [other rules](/rules) to learn more about Bazel. +- The [other rules](/rules) to learn more about Bazel. -* The [C++ build tutorial](/start/cpp) to get started with building - C++ projects with Bazel. +- The [C++ build tutorial](/start/cpp) to get started with building C++ projects with Bazel. -* The [Android application tutorial](/start/android-app ) and - [iOS application tutorial](/start/ios-app)) to get started with - building mobile applications for Android and iOS with Bazel. +- The [Android application tutorial](/start/android-app) and [iOS application tutorial](/start/ios-app)) to get started with building mobile applications for Android and iOS with Bazel. Happy building! diff --git a/tools/mdx-transform/transform.mjs b/tools/mdx-transform/transform.mjs index d4074d14..6dcf3147 100644 --- a/tools/mdx-transform/transform.mjs +++ b/tools/mdx-transform/transform.mjs @@ -351,6 +351,29 @@ export function transformContent(content) { .replace(/https\\:\/\//g, 'https://') .replace(/http\\:\/\//g, 'http://'); output = output.replace(/\\([{}])/g, '\$1'); + + // Escape comparison operators in table cells to prevent MDX parsing issues + // Process each line and escape < and > in table cell content + output = output.split('\n').map(line => { + // Only process table rows (lines with | characters) + if (line.includes('|') && !line.trim().startsWith('```')) { + // Split by | but preserve the delimiters + const cells = line.split('|'); + return cells.map(cell => { + // Don't escape in code blocks (content within backticks) + if (cell.includes('`')) { + // More complex: need to preserve code vs non-code + return cell.replace(/([^`]+)/g, (nonCode) => { + return nonCode.replace(/<=/g, '<=').replace(/>=/g, '>=').replace(/(?])([<>])(?!=)/g, '<'); + }); + } + // Escape comparison operators + return cell.replace(/<=/g, '<=').replace(/>=/g, '>='); + }).join('|'); + } + return line; + }).join('\n'); + return output; } diff --git a/tutorials/ccp-toolchain-config.mdx b/tutorials/ccp-toolchain-config.mdx index 0b14cb55..3595614f 100644 --- a/tutorials/ccp-toolchain-config.mdx +++ b/tutorials/ccp-toolchain-config.mdx @@ -2,473 +2,407 @@ title: 'Bazel Tutorial: Configure C++ Toolchains' --- +This tutorial uses an example scenario to describe how to configure C++ toolchains for a project. - -This tutorial uses an example scenario to describe how to configure C++ -toolchains for a project. - -## What you'll learn +## What you'll learn In this tutorial you learn how to: -* Set up the build environment -* Use `--toolchain_resolution_debug` to debug toolchain resolution -* Configure the C++ toolchain -* Create a Starlark rule that provides additional configuration for the - `cc_toolchain` so that Bazel can build the application with `clang` -* Build the C++ binary by running `bazel build //main:hello-world` on a - Linux machine -* Cross-compile the binary for android by running `bazel build - //main:hello-world --platforms=//:android_x86_64` +- Set up the build environment +- Use `--toolchain_resolution_debug` to debug toolchain resolution +- Configure the C++ toolchain +- Create a Starlark rule that provides additional configuration for the `cc_toolchain` so that Bazel can build the application with `clang` +- Build the C++ binary by running `bazel build //main:hello-world` on a Linux machine +- Cross-compile the binary for android by running `bazel build //main:hello-world --platforms=//:android_x86_64` -## Before you begin +## Before you begin -This tutorial assumes you are on Linux and have successfully built C++ -applications and installed the appropriate tooling and libraries. The tutorial -uses `clang version 19`, which you can install on your system. +This tutorial assumes you are on Linux and have successfully built C++ applications and installed the appropriate tooling and libraries. The tutorial uses `clang version 19`, which you can install on your system. -### Set up the build environment +### Set up the build environment Set up your build environment as follows: -1. If you have not already done so, [download and install Bazel - 7.0.2](https://bazel.build/install) or later. - -2. Add an empty `MODULE.bazel` file at the root folder. +1. If you have not already done so, [download and install Bazel 7.0.2](https://bazel.build/install) or later. + +2. Add an empty `MODULE.bazel` file at the root folder. + +3. Add the following `cc_binary` target to the `main/BUILD` file: + + ```python + cc_binary( + name = "hello-world", + srcs = ["hello-world.cc"], + ) + ``` + + Because Bazel uses many internal tools written in C++ during the build, such as `process-wrapper`, the pre-existing default C++ toolchain is specified for the host platform. This enables these internal tools to build using that toolchain of the one created in this tutorial. Hence, the `cc_binary` target is also built with the default toolchain. + +4. Run the build with the following command: + + ```bash + bazel build //main:hello-world + ``` -3. Add the following `cc_binary` target to the `main/BUILD` file: + The build succeeds without any toolchain registered in `MODULE.bazel`. - ```python - cc_binary( - name = "hello-world", - srcs = ["hello-world.cc"], - ) - ``` + To further see what's under the hood, run: - Because Bazel uses many internal tools written in C++ during the build, such - as `process-wrapper`, the pre-existing default C++ toolchain is specified - for the host platform. This enables these internal tools to build using that - toolchain of the one created in this tutorial. Hence, the `cc_binary` target - is also built with the default toolchain. + ```bash + bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' -4. Run the build with the following command: + INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 + ``` - ```bash - bazel build //main:hello-world - ``` + Without specifying `--platforms`, Bazel builds the target for `@platforms//host` using `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. - The build succeeds without any toolchain registered in `MODULE.bazel`. +## Configure the C++ toolchain - To further see what's under the hood, run: +To configure the C++ toolchain, repeatedly build the application and eliminate each error one by one as described as following. - ```bash - bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' - - INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 - ``` - - Without specifying `--platforms`, Bazel builds the target for - `@platforms//host` using - `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. - -## Configure the C++ toolchain - -To configure the C++ toolchain, repeatedly build the application and eliminate -each error one by one as described as following. - -Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using -an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` -flag to enable C++ toolchain resolution. - -It also assumes `clang version 9.0.1`, although the details should only change -slightly between different versions of clang. - -1. Add `toolchain/BUILD` with - - ```python - filegroup(name = "empty") - - cc_toolchain( - name = "linux_x86_64_toolchain", - toolchain_identifier = "linux_x86_64-toolchain", - toolchain_config = ":linux_x86_64_toolchain_config", - all_files = ":empty", - compiler_files = ":empty", - dwp_files = ":empty", - linker_files = ":empty", - objcopy_files = ":empty", - strip_files = ":empty", - supports_param_files = 0, - ) - - toolchain( - name = "cc_toolchain_for_linux_x86_64", - toolchain = ":linux_x86_64_toolchain", - toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", - exec_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - target_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - ) - ``` - - Then add appropriate dependencies and register the toolchain with - `MODULE.bazel` with - - ```python - bazel_dep(name = "platforms", version = "0.0.10") - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64" - ) - ``` - - This step defines a `cc_toolchain` and binds it to a `toolchain` target for - the host configuration. - -2. Run the build again. Because the `toolchain` package doesn't yet define the - `linux_x86_64_toolchain_config` target, Bazel throws the following error: - - ```bash - ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. - ``` - -3. In the `toolchain/BUILD` file, define an empty filegroup as follows: - - ```python - package(default_visibility = ["//visibility:public"]) - - filegroup(name = "linux_x86_64_toolchain_config") - ``` - -4. Run the build again. Bazel throws the following error: - - ```bash - '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. - ``` - - `CcToolchainConfigInfo` is a provider that you use to configure your C++ - toolchains. To fix this error, create a Starlark rule that provides - `CcToolchainConfigInfo` to Bazel by making a - `toolchain/cc_toolchain_config.bzl` file with the following content: - - ```python - def _impl(ctx): - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "k8-toolchain", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - `cc_common.create_cc_toolchain_config_info()` creates the needed provider - `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load - statement to `toolchain/BUILD` right below the package statement: - - ```python - load(":cc_toolchain_config.bzl", "cc_toolchain_config") - ``` - - And replace the "linux_x86_64_toolchain_config" filegroup with a declaration - of a `cc_toolchain_config` rule: - - ```python - cc_toolchain_config(name = "linux_x86_64_toolchain_config") - ``` - -5. Run the build again. Bazel throws the following error: - - ```bash - .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) - src/main/tools/linux-sandbox-pid1.cc:421: - "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory - Target //:hello-world failed to build` - ``` - - At this point, Bazel has enough information to attempt building the code but - it still does not know what tools to use to complete the required build - actions. You will modify the Starlark rule implementation to tell Bazel what - tools to use. For that, you need the `tool_path()` constructor from - [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): - - ```python - # toolchain/cc_toolchain_config.bzl: - # NEW - load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") - - def _impl(ctx): - tool_paths = [ # NEW - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/usr/bin/ar", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, # NEW - ) - ``` - - Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for - your system. Note that the compiler is referenced by the name "gcc" for - historic reasons. - -6. Run the build again. Bazel throws the following error: - - ```bash - ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': - the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): - '/usr/include/c++/13/ctime' - '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' - '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' - ... - ``` - - Bazel needs to know where to search for included headers. There are multiple - ways to solve this, such as using the `includes` attribute of `cc_binary`, - but here this is solved at the toolchain level with the - [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) - parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you - are using a different version of `clang`, the include path will be - different. These paths may also be different depending on the distribution. - - Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like - this: - - ```python - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - cxx_builtin_include_directories = [ # NEW - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - ``` - -7. Run the build command again, you will see an error like: - - ```bash - /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': - hello-world.cc:(.text+0x68): undefined reference to `std::cout' - ``` - - The reason for this is because the linker is missing the C++ standard - library and it can't find its symbols. There are many ways to solve this, - such as using the `linkopts` attribute of `cc_binary`. Here it is solved by - making sure that any target using the toolchain doesn't have to specify this - flag. - - Copy the following code to `toolchain/cc_toolchain_config.bzl`: - - ```python - # NEW - load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") - # NEW - load( - "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", - "feature", # NEW - "flag_group", # NEW - "flag_set", # NEW - "tool_path", - ) - - all_link_actions = [ # NEW - ACTION_NAMES.cpp_link_executable, - ACTION_NAMES.cpp_link_dynamic_library, - ACTION_NAMES.cpp_link_nodeps_dynamic_library, - ] - - def _impl(ctx): - tool_paths = [ - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/bin/false", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - features = [ # NEW - feature( - name = "default_linker_flags", - enabled = True, - flag_sets = [ - flag_set( - actions = all_link_actions, - flag_groups = ([ - flag_group( - flags = [ - "-lstdc++", - ], - ), - ]), - ), - ], - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - features = features, # NEW - cxx_builtin_include_directories = [ - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - Note that this code uses the GNU C++ library libstdc++. If you want to use - the LLVM C++ library, use "-lc++" instead of "-lstdc++". - -8. Running `bazel build //main:hello-world`, it should finally build the binary - successfully for host. - -9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and - `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in - target names. - - In `MODULE.bazel`, register the toolchain for android - - ```python - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64", - "//toolchain:cc_toolchain_for_android_x86_64" - ) - ``` - -10. Run `bazel build //main:hello-world - --android_platforms=//toolchain:android_x86_64` to build the binary for - Android. - -In practice, Linux and Android should have different C++ toolchain configs. You -can either modify the existing `cc_toolchain_config` for the differences or -create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate -platforms. - -## Review your work - -In this tutorial you learned how to configure a basic C++ toolchain, but -toolchains are more powerful than this example. +Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` flag to enable C++ toolchain resolution. + +It also assumes `clang version 9.0.1`, although the details should only change slightly between different versions of clang. + +1. Add `toolchain/BUILD` with + + ```python + filegroup(name = "empty") + + cc_toolchain( + name = "linux_x86_64_toolchain", + toolchain_identifier = "linux_x86_64-toolchain", + toolchain_config = ":linux_x86_64_toolchain_config", + all_files = ":empty", + compiler_files = ":empty", + dwp_files = ":empty", + linker_files = ":empty", + objcopy_files = ":empty", + strip_files = ":empty", + supports_param_files = 0, + ) + + toolchain( + name = "cc_toolchain_for_linux_x86_64", + toolchain = ":linux_x86_64_toolchain", + toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", + exec_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + ) + ``` + + Then add appropriate dependencies and register the toolchain with `MODULE.bazel` with + + ```python + bazel_dep(name = "platforms", version = "0.0.10") + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64" + ) + ``` + + This step defines a `cc_toolchain` and binds it to a `toolchain` target for the host configuration. + +2. Run the build again. Because the `toolchain` package doesn't yet define the `linux_x86_64_toolchain_config` target, Bazel throws the following error: + + ```bash + ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. + ``` + +3. In the `toolchain/BUILD` file, define an empty filegroup as follows: + + ```python + package(default_visibility = ["//visibility:public"]) + + filegroup(name = "linux_x86_64_toolchain_config") + ``` + +4. Run the build again. Bazel throws the following error: + + ```bash + '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. + ``` + + `CcToolchainConfigInfo` is a provider that you use to configure your C++ toolchains. To fix this error, create a Starlark rule that provides `CcToolchainConfigInfo` to Bazel by making a `toolchain/cc_toolchain_config.bzl` file with the following content: + + ```python + def _impl(ctx): + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "k8-toolchain", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + `cc_common.create_cc_toolchain_config_info()` creates the needed provider `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load statement to `toolchain/BUILD` right below the package statement: + + ```python + load(":cc_toolchain_config.bzl", "cc_toolchain_config") + ``` + + And replace the "linux\_x86\_64\_toolchain\_config" filegroup with a declaration of a `cc_toolchain_config` rule: + + ```python + cc_toolchain_config(name = "linux_x86_64_toolchain_config") + ``` + +5. Run the build again. Bazel throws the following error: + + ```bash + .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) + src/main/tools/linux-sandbox-pid1.cc:421: + "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory + Target //:hello-world failed to build` + ``` + + At this point, Bazel has enough information to attempt building the code but it still does not know what tools to use to complete the required build actions. You will modify the Starlark rule implementation to tell Bazel what tools to use. For that, you need the `tool_path()` constructor from [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): + + ```python + # toolchain/cc_toolchain_config.bzl: + # NEW + load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") + + def _impl(ctx): + tool_paths = [ # NEW + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/usr/bin/ar", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, # NEW + ) + ``` + + Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for your system. Note that the compiler is referenced by the name "gcc" for historic reasons. + +6. Run the build again. Bazel throws the following error: + + ```bash + ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': + the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): + '/usr/include/c++/13/ctime' + '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' + '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' + ... + ``` + + Bazel needs to know where to search for included headers. There are multiple ways to solve this, such as using the `includes` attribute of `cc_binary`, but here this is solved at the toolchain level with the [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you are using a different version of `clang`, the include path will be different. These paths may also be different depending on the distribution. + + Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like this: + + ```python + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + cxx_builtin_include_directories = [ # NEW + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + ``` + +7. Run the build command again, you will see an error like: + + ```bash + /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': + hello-world.cc:(.text+0x68): undefined reference to `std::cout' + ``` + + The reason for this is because the linker is missing the C++ standard library and it can't find its symbols. There are many ways to solve this, such as using the `linkopts` attribute of `cc_binary`. Here it is solved by making sure that any target using the toolchain doesn't have to specify this flag. + + Copy the following code to `toolchain/cc_toolchain_config.bzl`: + + ```python + # NEW + load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") + # NEW + load( + "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", + "feature", # NEW + "flag_group", # NEW + "flag_set", # NEW + "tool_path", + ) + + all_link_actions = [ # NEW + ACTION_NAMES.cpp_link_executable, + ACTION_NAMES.cpp_link_dynamic_library, + ACTION_NAMES.cpp_link_nodeps_dynamic_library, + ] + + def _impl(ctx): + tool_paths = [ + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/bin/false", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + features = [ # NEW + feature( + name = "default_linker_flags", + enabled = True, + flag_sets = [ + flag_set( + actions = all_link_actions, + flag_groups = ([ + flag_group( + flags = [ + "-lstdc++", + ], + ), + ]), + ), + ], + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + features = features, # NEW + cxx_builtin_include_directories = [ + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + Note that this code uses the GNU C++ library libstdc++. If you want to use the LLVM C++ library, use "-lc++" instead of "-lstdc++". + +8. Running `bazel build //main:hello-world`, it should finally build the binary successfully for host. + +9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in target names. + + In `MODULE.bazel`, register the toolchain for android + + ```python + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64", + "//toolchain:cc_toolchain_for_android_x86_64" + ) + ``` + +10. Run `bazel build //main:hello-world --android_platforms=//toolchain:android_x86_64` to build the binary for Android. + +In practice, Linux and Android should have different C++ toolchain configs. You can either modify the existing `cc_toolchain_config` for the differences or create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate platforms. + +## Review your work + +In this tutorial you learned how to configure a basic C++ toolchain, but toolchains are more powerful than this example. The key takeaways are: -- You need to specify a matching `platforms` flag in the command line for - Bazel to resolve to the toolchain for the same constraint values on the - platform. The documentation holds more [information about language specific - configuration flags](/concepts/platforms). -- You have to let the toolchain know where the tools live. In this tutorial - there is a simplified version where you access the tools from the system. If - you are interested in a more self-contained approach, you can read about - [external dependencies](/external/overview). Your tools could come from a - different module and you would have to make their files available to the - `cc_toolchain` with target dependencies on attributes, such as - `compiler_files`. The `tool_paths` would need to be changed as well. -- You can create features to customize which flags should be passed to - different actions, be it linking or any other type of action. - -## Further reading - -For more details, see [C++ toolchain -configuration](/docs/cc-toolchain-config-reference) +- You need to specify a matching `platforms` flag in the command line for Bazel to resolve to the toolchain for the same constraint values on the platform. The documentation holds more [information about language specific configuration flags](/concepts/platforms). +- You have to let the toolchain know where the tools live. In this tutorial there is a simplified version where you access the tools from the system. If you are interested in a more self-contained approach, you can read about [external dependencies](/external/overview). Your tools could come from a different module and you would have to make their files available to the `cc_toolchain` with target dependencies on attributes, such as `compiler_files`. The `tool_paths` would need to be changed as well. +- You can create features to customize which flags should be passed to different actions, be it linking or any other type of action. + +## Further reading + +For more details, see [C++ toolchain configuration](/docs/cc-toolchain-config-reference) diff --git a/tutorials/cpp-dependency.mdx b/tutorials/cpp-dependency.mdx index 194cc73c..24872bb8 100644 --- a/tutorials/cpp-dependency.mdx +++ b/tutorials/cpp-dependency.mdx @@ -2,49 +2,36 @@ title: 'Review the dependency graph' --- +A successful build has all of its dependencies explicitly stated in the `BUILD` file. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. - -A successful build has all of its dependencies explicitly stated in the `BUILD` -file. Bazel uses those statements to create the project's dependency graph, -which enables accurate incremental builds. - -To visualize the sample project's dependencies, you can generate a text -representation of the dependency graph by running this command at the -workspace root: +To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: ``` bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph ``` -The above command tells Bazel to look for all dependencies for the target -`//main:hello-world` (excluding host and implicit dependencies) and format the -output as a graph. +The above command tells Bazel to look for all dependencies for the target `//main:hello-world` (excluding host and implicit dependencies) and format the output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -On Ubuntu, you can view the graph locally by installing GraphViz and the xdot -Dot Viewer: +On Ubuntu, you can view the graph locally by installing GraphViz and the xdot Dot Viewer: ``` sudo apt update && sudo apt install graphviz xdot ``` -Then you can generate and view the graph by piping the text output above -straight to xdot: +Then you can generate and view the graph by piping the text output above straight to xdot: ``` -xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ +xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph) ``` -As you can see, the first stage of the sample project has a single target -that builds a single source file with no additional dependencies: +As you can see, the first stage of the sample project has a single target that builds a single source file with no additional dependencies: ![Dependency graph for 'hello-world'](/docs/images/cpp-tutorial-stage1.png "Dependency graph") -**Figure 1.** Dependency graph for `hello-world` displays a single target with a single -source file. +**Figure 1.** Dependency graph for `hello-world` displays a single target with a single source file. -After you set up your workspace, build your project, and examine its -dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. diff --git a/tutorials/cpp-labels.mdx b/tutorials/cpp-labels.mdx index 78d0dbce..17e062aa 100644 --- a/tutorials/cpp-labels.mdx +++ b/tutorials/cpp-labels.mdx @@ -2,26 +2,12 @@ title: 'Use labels to reference targets' --- - - -In `BUILD` files and at the command line, Bazel uses *labels* to reference -targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax -is: +In `BUILD` files and at the command line, Bazel uses *labels* to reference targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax is: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path from the -workspace root (the directory containing the `MODULE.bazel` file) to the directory -containing the `BUILD` file, and `target-name` is what you named the target -in the `BUILD` file (the `name` attribute). If the target is a file target, -then `path/to/package` is the path to the root of the package, and -`target-name` is the name of the target file, including its full -path relative to the root of the package (the directory containing the -package's `BUILD` file). +If the target is a rule target, then `path/to/package` is the path from the workspace root (the directory containing the `MODULE.bazel` file) to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path relative to the root of the package (the directory containing the package's `BUILD` file). -When referencing targets at the repository root, the package path is empty, -just use `//:target-name`. When referencing targets within the same `BUILD` -file, you can even skip the `//` workspace root identifier and just use -`:target-name`. +When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. diff --git a/tutorials/cpp-use-cases.mdx b/tutorials/cpp-use-cases.mdx index f25f80b3..f13e6be0 100644 --- a/tutorials/cpp-use-cases.mdx +++ b/tutorials/cpp-use-cases.mdx @@ -2,21 +2,13 @@ title: 'Common C++ Build Use Cases' --- +Here you will find some of the most common use cases for building C++ projects with Bazel. If you have not done so already, get started with building C++ projects with Bazel by completing the tutorial [Introduction to Bazel: Build a C++ Project](/start/cpp). - -Here you will find some of the most common use cases for building C++ projects -with Bazel. If you have not done so already, get started with building C++ -projects with Bazel by completing the tutorial -[Introduction to Bazel: Build a C++ Project](/start/cpp). - -For information on cc_library and hdrs header files, see -cc_library. +For information on cc\_library and hdrs header files, see [cc\_library](/reference/be/c-cpp#cc_library). ## Including multiple files in a target -You can include multiple files in a single target with -glob. -For example: +You can include multiple files in a single target with [glob](/reference/be/functions#glob). For example: ```python cc_library( @@ -26,19 +18,11 @@ cc_library( ) ``` -With this target, Bazel will build all the `.cc` and `.h` files it finds in the -same directory as the `BUILD` file that contains this target (excluding -subdirectories). +With this target, Bazel will build all the `.cc` and `.h` files it finds in the same directory as the `BUILD` file that contains this target (excluding subdirectories). ## Using transitive includes -If a file includes a header, then any rule with that file as a source (that is, -having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should -depend on the included header's library rule. Conversely, only direct -dependencies need to be specified as dependencies. For example, suppose -`sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` -doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` -file would look like this: +If a file includes a header, then any rule with that file as a source (that is, having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should depend on the included header's library rule. Conversely, only direct dependencies need to be specified as dependencies. For example, suppose `sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` file would look like this: ```python cc_library( @@ -62,15 +46,11 @@ cc_library( ) ``` -Here, the `sandwich` library depends on the `bread` library, which depends -on the `flour` library. +Here, the `sandwich` library depends on the `bread` library, which depends on the `flour` library. ## Adding include paths -Sometimes you cannot (or do not want to) root include paths at the workspace -root. Existing libraries might already have an include directory that doesn't -match its path in your workspace. For example, suppose you have the following -directory structure: +Sometimes you cannot (or do not want to) root include paths at the workspace root. Existing libraries might already have an include directory that doesn't match its path in your workspace. For example, suppose you have the following directory structure: ``` └── my-project @@ -83,11 +63,7 @@ directory structure: └── MODULE.bazel ``` -Bazel will expect `some_lib.h` to be included as -`legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes -`"some_lib.h"`. To make that include path valid, -`legacy/some_lib/BUILD` will need to specify that the `some_lib/include` -directory is an include directory: +Bazel will expect `some_lib.h` to be included as `legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes `"some_lib.h"`. To make that include path valid, `legacy/some_lib/BUILD` will need to specify that the `some_lib/include` directory is an include directory: ```python cc_library( @@ -98,15 +74,11 @@ cc_library( ) ``` -This is especially useful for external dependencies, as their header files -must otherwise be included with a `/` prefix. +This is especially useful for external dependencies, as their header files must otherwise be included with a `/` prefix. ## Include external libraries -Suppose you are using [Google Test](https://github.com/google/googletest) -. -You can add a dependency on it in the `MODULE.bazel` file to -download Google Test and make it available in your repository: +Suppose you are using [Google Test](https://github.com/google/googletest). You can add a dependency on it in the `MODULE.bazel` file to download Google Test and make it available in your repository: ```python bazel_dep(name = "googletest", version = "1.15.2") @@ -142,8 +114,7 @@ cc_test( ) ``` -To make `hello-greet` visible to `hello-test`, you must add -`"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. +To make `hello-greet` visible to `hello-test`, you must add `"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. Now you can use `bazel test` to run the test. @@ -163,11 +134,9 @@ INFO: Elapsed time: 4.497s, Critical Path: 2.53s Executed 1 out of 1 tests: 1 test passes. ``` - ## Adding dependencies on precompiled libraries -If you want to use a library of which you only have a compiled version (for -example, headers and a `.so` file) wrap it in a `cc_library` rule: +If you want to use a library of which you only have a compiled version (for example, headers and a `.so` file) wrap it in a `cc_library` rule: ```python cc_library( diff --git a/versions/index.mdx b/versions/index.mdx index 4290e57f..64608bf9 100644 --- a/versions/index.mdx +++ b/versions/index.mdx @@ -2,14 +2,8 @@ title: 'Documentation Versions' --- +The default documentation on this website represents the latest version at HEAD. Each major and minor supported release will have a snapshot of the narrative and reference documentation that follows the lifecycle of Bazel's version support. +To see documentation for stable Bazel versions, use the "Versioned docs" drop-down. -The default documentation on this website represents the latest version at HEAD. -Each major and minor supported release will have a snapshot of the narrative and -reference documentation that follows the lifecycle of Bazel's version support. - -To see documentation for stable Bazel versions, use the "Versioned docs" -drop-down. - -To see documentation for older Bazel versions prior to Feb 2022, go to -[docs.bazel.build](https://docs.bazel.build/). +To see documentation for older Bazel versions prior to Feb 2022, go to [docs.bazel.build](https://docs.bazel.build/). From b5fe045f65dd5f7e2b4d0be1936fd7fe91dc89d5 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Thu, 6 Nov 2025 13:23:53 -0800 Subject: [PATCH 08/10] clenaup --- about/faq.mdx | 74 +- about/intro.mdx | 95 +- about/roadmap.mdx | 88 +- about/vision.mdx | 110 +- about/why.mdx | 66 +- .../build-performance-breakdown.mdx | 248 ++- .../performance/build-performance-metrics.mdx | 76 +- advanced/performance/iteration-speed.mdx | 84 +- advanced/performance/json-trace-profile.mdx | 102 +- advanced/performance/memory.mdx | 60 +- basics/artifact-based-builds.mdx | 277 ++- basics/build-systems.mdx | 136 +- basics/dependencies.mdx | 306 ++- basics/distributed-builds.mdx | 128 +- basics/hermeticity.mdx | 117 +- basics/index.mdx | 54 +- basics/task-based-builds.mdx | 238 ++- brand/index.mdx | 84 +- build/share-variables.mdx | 31 +- build/style-guide.mdx | 279 ++- community/recommended-rules.mdx | 50 +- community/remote-execution-services.mdx | 31 +- community/sig.mdx | 146 +- community/users.mdx | 533 +++-- concepts/build-files.mdx | 73 - concepts/build-ref.mdx | 100 +- concepts/dependencies.mdx | 192 -- concepts/labels.mdx | 148 -- concepts/platforms.mdx | 283 ++- concepts/visibility.mdx | 331 ++- configure/attributes.mdx | 338 ++- configure/best-practices.mdx | 67 +- configure/coverage.mdx | 106 +- configure/integrate-cpp.mdx | 37 - configure/windows.mdx | 240 ++- contribute/breaking-changes.mdx | 112 +- contribute/codebase.mdx | 1881 ++++++++++++----- contribute/design-documents.mdx | 238 ++- contribute/docs-style-guide.mdx | 176 -- contribute/docs.mdx | 32 +- contribute/index.mdx | 76 +- contribute/maintainers-guide.mdx | 254 ++- contribute/naming.mdx | 62 +- contribute/patch-acceptance.mdx | 65 +- contribute/policy.mdx | 103 +- contribute/release-notes.mdx | 67 +- contribute/search.mdx | 167 -- contribute/statemachine-guide.mdx | 568 +++-- contribute/windows-chocolatey-maintenance.mdx | 50 +- contribute/windows-scoop-maintenance.mdx | 28 +- docs/android-build-performance.mdx | 70 +- docs/android-instrumentation-test.mdx | 192 +- docs/android-ndk.mdx | 147 +- docs/bazel-and-android.mdx | 43 +- docs/bazel-and-apple.mdx | 87 +- docs/bazel-and-cpp.mdx | 125 +- docs/bazel-and-java.mdx | 219 +- docs/bazel-and-javascript.mdx | 30 +- docs/cc-toolchain-config-reference.mdx | 488 ----- docs/configurable-attributes.mdx | 337 ++- docs/mobile-install.mdx | 183 +- docs/sandboxing.mdx | 144 +- docs/user-manual.mdx | 1443 ------------- extending/aspects.mdx | 202 +- extending/auto-exec-groups.mdx | 97 +- extending/concepts.mdx | 109 +- extending/config.mdx | 631 ------ extending/depsets.mdx | 130 +- extending/exec-groups.mdx | 106 +- extending/legacy-macros.mdx | 137 +- extending/macros.mdx | 270 ++- extending/platforms.mdx | 170 +- extending/rules.mdx | 879 ++++++-- extending/toolchains.mdx | 314 ++- external/extension.mdx | 202 +- external/faq.mdx | 327 ++- external/lockfile.mdx | 201 +- external/migration.mdx | 979 +++++---- external/migration_tool.mdx | 642 ------ external/mod-command.mdx | 418 ---- external/module.mdx | 222 +- external/overview.mdx | 261 ++- external/registry.mdx | 83 - external/repo.mdx | 141 +- external/vendor.mdx | 136 +- help.mdx | 52 +- install/bazelisk.mdx | 53 +- install/compile-source.mdx | 262 ++- install/completion.mdx | 177 +- install/docker-container.mdx | 28 +- install/ide.mdx | 81 +- install/index.mdx | 32 +- install/os-x.mdx | 82 +- install/suse.mdx | 16 +- install/ubuntu.mdx | 82 +- install/windows.mdx | 185 +- migrate/index.mdx | 6 +- migrate/maven.mdx | 250 ++- migrate/xcode.mdx | 229 +- query/aquery.mdx | 177 +- query/cquery.mdx | 350 ++- query/guide.mdx | 206 +- query/language.mdx | 872 -------- query/quickstart.mdx | 500 ----- reference/flag-cheatsheet.mdx | 91 - reference/glossary.mdx | 516 ++++- reference/skyframe.mdx | 211 +- reference/test-encyclopedia.mdx | 308 --- release/backward-compatibility.mdx | 70 +- release/index.mdx | 254 ++- release/rolling.mdx | 11 +- release/rule-compatibility.mdx | 110 +- remote/bep-examples.mdx | 110 +- remote/bep-glossary.mdx | 148 +- remote/bep.mdx | 143 +- remote/cache-local.mdx | 68 +- remote/cache-remote.mdx | 152 +- remote/caching.mdx | 330 +-- remote/ci.mdx | 173 +- remote/creating.mdx | 195 +- remote/dynamic.mdx | 63 - remote/multiplex.mdx | 108 +- remote/output-directories.mdx | 133 +- remote/persistent.mdx | 219 +- remote/rbe.mdx | 28 +- remote/rules.mdx | 196 +- remote/sandbox.mdx | 210 +- remote/workspace.mdx | 122 +- rules/bzl-style.mdx | 221 +- rules/challenges.mdx | 230 +- rules/deploying.mdx | 143 +- rules/errors/read-only-variable.mdx | 9 +- rules/faq.mdx | 57 +- rules/index.mdx | 81 +- rules/language.mdx | 127 +- rules/legacy-macro-tutorial.mdx | 49 +- rules/macro-tutorial.mdx | 37 +- rules/performance.mdx | 147 +- rules/rules-tutorial.mdx | 126 +- rules/testing.mdx | 157 +- rules/verbs-tutorial.mdx | 118 +- rules/windows.mdx | 189 -- run/bazelrc.mdx | 243 ++- run/build.mdx | 369 ---- run/client-server.mdx | 54 +- run/scripts.mdx | 136 +- start/android-app.mdx | 261 ++- start/cpp.mdx | 178 +- start/go.mdx | 224 +- start/ios-app.mdx | 3 +- start/java.mdx | 233 +- tutorials/ccp-toolchain-config.mdx | 832 ++++---- tutorials/cpp-dependency.mdx | 31 +- tutorials/cpp-labels.mdx | 20 +- tutorials/cpp-use-cases.mdx | 55 +- versions/index.mdx | 12 +- 156 files changed, 17465 insertions(+), 13858 deletions(-) delete mode 100644 concepts/build-files.mdx delete mode 100644 concepts/dependencies.mdx delete mode 100644 concepts/labels.mdx delete mode 100644 configure/integrate-cpp.mdx delete mode 100644 contribute/docs-style-guide.mdx delete mode 100644 contribute/search.mdx delete mode 100644 docs/cc-toolchain-config-reference.mdx delete mode 100644 docs/user-manual.mdx delete mode 100644 extending/config.mdx delete mode 100644 external/migration_tool.mdx delete mode 100644 external/mod-command.mdx delete mode 100644 external/registry.mdx delete mode 100644 query/language.mdx delete mode 100644 query/quickstart.mdx delete mode 100644 reference/flag-cheatsheet.mdx delete mode 100644 reference/test-encyclopedia.mdx delete mode 100644 remote/dynamic.mdx delete mode 100644 rules/windows.mdx delete mode 100644 run/build.mdx diff --git a/about/faq.mdx b/about/faq.mdx index fc0c31d1..dd5be8a9 100644 --- a/about/faq.mdx +++ b/about/faq.mdx @@ -2,6 +2,8 @@ title: 'FAQ' --- + + If you have questions or need support, see [Getting Help](/help). ## What is Bazel? @@ -12,19 +14,19 @@ Bazel is a tool that automates software builds and tests. Supported build tasks Bazel was designed to fit the way software is developed at Google. It has the following features: -- Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. -- High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. -- Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. -- Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. -- Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about \~200ms. +* Multi-language support: Bazel supports [many languages](/reference/be/overview), and can be extended to support arbitrary programming languages. +* High-level build language: Projects are described in the `BUILD` language, a concise text format that describes a project as sets of small interconnected libraries, binaries and tests. In contrast, with tools like Make, you have to describe individual files and compiler invocations. +* Multi-platform support: The same tool and the same `BUILD` files can be used to build software for different architectures, and even different platforms. At Google, we use Bazel to build everything from server applications running on systems in our data centers to client apps running on mobile phones. +* Reproducibility: In `BUILD` files, each library, test and binary must specify its direct dependencies completely. Bazel uses this dependency information to know what must be rebuilt when you make changes to a source file, and which tasks can run in parallel. This means that all builds are incremental and will always produce the same result. +* Scalable: Bazel can handle large builds; at Google, it is common for a server binary to have 100k source files, and builds where no files were changed take about ~200ms. ## Why doesn’t Google use...? -- Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. - - Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. -- Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. -- Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. -- Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. +* Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it’s up to the user to write rules that are correct. + * Users interact with Bazel on a higher level. For example, Bazel has built-in rules for “Java test”, “C++ binary”, and notions such as “target platform” and “host platform”. These rules have been battle tested to be foolproof. +* Ant and Maven: Ant and Maven are primarily geared toward Java, while Bazel handles multiple languages. Bazel encourages subdividing codebases in smaller reusable units, and can rebuild only ones that need rebuilding. This speeds up development when working with larger codebases. +* Gradle: Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility. +* Pants, Buck: Both tools were created and developed by ex-Googlers at Twitter and Foursquare, and Facebook respectively. They have been modeled after Bazel, but their feature sets are different, so they aren’t viable alternatives for us. ## Where did Bazel come from? @@ -46,10 +48,10 @@ Bazel runs build operations locally by default. However, Bazel can also connect For our server code base, we use the following development workflow: -- All our server code is in a single, gigantic version control system. -- Everybody builds their software with Bazel. -- Different teams own different parts of the source tree, and make their components available as `BUILD` targets. -- Branching is primarily used for managing releases, so everybody develops their software at the head revision. +* All our server code is in a single, gigantic version control system. +* Everybody builds their software with Bazel. +* Different teams own different parts of the source tree, and make their components available as `BUILD` targets. +* Branching is primarily used for managing releases, so everybody develops their software at the head revision. Bazel is a cornerstone of this philosophy: since Bazel requires all dependencies to be fully specified, we can predict which programs and tests are affected by a change, and vet them before submission. @@ -61,22 +63,24 @@ Building software should be fun and easy. Slow and unpredictable builds take the ## Why would I want to use Bazel? -- Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. -- Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. -- Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. +* Bazel may give you faster build times because it can recompile only the files that need to be recompiled. Similarly, it can skip re-running tests that it knows haven’t changed. +* Bazel produces deterministic results. This eliminates skew between incremental and clean builds, laptop and CI system, etc. +* Bazel can build different client and server apps with the same tool from the same workspace. For example, you can change a client/server protocol in a single commit, and test that the updated mobile app works with the updated server, building both with the same tool, reaping all the aforementioned benefits of Bazel. ## Can I see examples? -Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. +Yes; see a [simple example](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD) +or read the [Bazel source code](https://github.com/bazelbuild/bazel/blob/master/src/BUILD) for a more complex example. + ## What is Bazel best at? Bazel shines at building and testing projects with the following properties: -- Projects with a large codebase -- Projects written in (multiple) compiled languages -- Projects that deploy on multiple platforms -- Projects that have extensive tests +* Projects with a large codebase +* Projects written in (multiple) compiled languages +* Projects that deploy on multiple platforms +* Projects that have extensive tests ## Where can I run Bazel? @@ -86,13 +90,11 @@ Porting to other UNIX platforms should be relatively easy, as long as a JDK is a ## What should I not use Bazel for? -- Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: - - - A compilation step that fetches data from the internet. - - A test step that connects to the QA instance of your site. - - A deployment step that changes your site’s cloud configuration. - -- If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. +* Bazel tries to be smart about caching. This means that it is not good for running build operations whose outputs should not be cached. For example, the following steps should not be run from Bazel: + * A compilation step that fetches data from the internet. + * A test step that connects to the QA instance of your site. + * A deployment step that changes your site’s cloud configuration. +* If your build consists of a few long, sequential steps, Bazel may not be able to help much. You’ll get more speed by breaking long steps into smaller, discrete targets that Bazel can run in parallel. ## How stable is Bazel’s feature set? @@ -130,10 +132,10 @@ Yes, you can use our [Docker rules](https://github.com/bazelbuild/rules_docker) For Java and C++ binaries, yes, assuming you do not change the toolchain. If you have build steps that involve custom recipes (for example, executing binaries through a shell script inside a rule), you will need to take some extra care: -- Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. -- Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. -- Avoid connecting to the network. Sandboxed execution can help here too. -- Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. +* Do not use dependencies that were not declared. Sandboxed execution (–spawn\_strategy=sandboxed, only on Linux) can help find undeclared dependencies. +* Avoid storing timestamps and user-IDs in generated files. ZIP files and other archives are especially prone to this. +* Avoid connecting to the network. Sandboxed execution can help here too. +* Avoid processes that use random numbers, in particular, dictionary traversal is randomized in many programming languages. ## Do you have binary releases? @@ -177,8 +179,8 @@ We still have to refactor the interfaces between the public code in Bazel and ou Open sourcing Bazel is a work-in-progress. In particular, we’re still working on open sourcing: -- Many of our unit and integration tests (which should make contributing patches easier). -- Full IDE integration. +* Many of our unit and integration tests (which should make contributing patches easier). +* Full IDE integration. Beyond code, we’d like to eventually have all code reviews, bug tracking, and design decisions happen publicly, with the Bazel community involved. We are not there yet, so some changes will simply appear in the Bazel repository without clear explanation. Despite this lack of transparency, we want to support external developers and collaborate. Thus, we are opening up the code, even though some of the development is still happening internal to Google. Please let us know if anything seems unclear or unjustified as we transition to an open model. @@ -188,7 +190,7 @@ Yes, some of the code base either integrates with Google-specific technology or ## How do I contact the team? -We are reachable at [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com). +We are reachable at bazel-discuss@googlegroups.com. ## Where do I report bugs? diff --git a/about/intro.mdx b/about/intro.mdx index a70f715b..a531ac2a 100644 --- a/about/intro.mdx +++ b/about/intro.mdx @@ -2,63 +2,110 @@ title: 'Intro to Bazel' --- -Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users. + + +Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. +It uses a human-readable, high-level build language. Bazel supports projects in +multiple languages and builds outputs for multiple platforms. Bazel supports +large codebases across multiple repositories, and large numbers of users. ## Benefits Bazel offers the following advantages: -- **High-level build language.** Bazel uses an abstract, human-readable language to describe the build properties of your project at a high semantical level. Unlike other tools, Bazel operates on the *concepts* of libraries, binaries, scripts, and data sets, shielding you from the complexity of writing individual calls to tools such as compilers and linkers. +* **High-level build language.** Bazel uses an abstract, human-readable + language to describe the build properties of your project at a high + semantical level. Unlike other tools, Bazel operates on the *concepts* + of libraries, binaries, scripts, and data sets, shielding you from the + complexity of writing individual calls to tools such as compilers and + linkers. -- **Bazel is fast and reliable.** Bazel caches all previously done work and tracks changes to both file content and build commands. This way, Bazel knows when something needs to be rebuilt, and rebuilds only that. To further speed up your builds, you can set up your project to build in a highly parallel and incremental fashion. +* **Bazel is fast and reliable.** Bazel caches all previously done work and + tracks changes to both file content and build commands. This way, Bazel + knows when something needs to be rebuilt, and rebuilds only that. To further + speed up your builds, you can set up your project to build in a highly + parallel and incremental fashion. -- **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel can build binaries and deployable packages for multiple platforms, including desktop, server, and mobile, from the same project. +* **Bazel is multi-platform.** Bazel runs on Linux, macOS, and Windows. Bazel + can build binaries and deployable packages for multiple platforms, including + desktop, server, and mobile, from the same project. -- **Bazel scales.** Bazel maintains agility while handling builds with 100k+ source files. It works with multiple repositories and user bases in the tens of thousands. +* **Bazel scales.** Bazel maintains agility while handling builds with 100k+ + source files. It works with multiple repositories and user bases in the tens + of thousands. -- **Bazel is extensible.** Many [languages](/rules) are supported, and you can extend Bazel to support any other language or framework. +* **Bazel is extensible.** Many [languages](/rules) are + supported, and you can extend Bazel to support any other language or + framework. ## Using Bazel To build or test a project with Bazel, you typically do the following: -1. **Set up Bazel.** Download and [install Bazel](/install). +1. **Set up Bazel.** Download and [install Bazel](/install). -2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a directory where Bazel looks for build inputs and `BUILD` files, and where it stores build outputs. +2. **Set up a project [workspace](/concepts/build-ref#workspaces)**, which is a + directory where Bazel looks for build inputs and `BUILD` files, and where it + stores build outputs. -3. **Write a `BUILD` file**, which tells Bazel what to build and how to build it. +3. **Write a `BUILD` file**, which tells Bazel what to build and how to + build it. - You write your `BUILD` file by declaring build targets using [Starlark](/rules/language), a domain-specific language. (See example [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) + You write your `BUILD` file by declaring build targets using + [Starlark](/rules/language), a domain-specific language. (See example + [here](https://github.com/bazelbuild/bazel/blob/master/examples/cpp/BUILD).) - A build target specifies a set of input artifacts that Bazel will build plus their dependencies, the build rule Bazel will use to build it, and options that configure the build rule. + A build target specifies a set of input artifacts that Bazel will build plus + their dependencies, the build rule Bazel will use to build it, and options + that configure the build rule. - A build rule specifies the build tools Bazel will use, such as compilers and linkers, and their configurations. Bazel ships with a number of build rules covering the most common artifact types in the supported languages on supported platforms. + A build rule specifies the build tools Bazel will use, such as compilers and + linkers, and their configurations. Bazel ships with a number of build rules + covering the most common artifact types in the supported languages on + supported platforms. -4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel places your outputs within the workspace. +4. **Run Bazel** from the [command line](/reference/command-line-reference). Bazel + places your outputs within the workspace. -In addition to building, you can also use Bazel to run [tests](/reference/test-encyclopedia) and [query](/query/guide) the build to trace dependencies in your code. +In addition to building, you can also use Bazel to run +[tests](/reference/test-encyclopedia) and [query](/query/guide) the build +to trace dependencies in your code. ## Bazel build process When running a build or a test, Bazel does the following: -1. **Loads** the `BUILD` files relevant to the target. +1. **Loads** the `BUILD` files relevant to the target. -2. **Analyzes** the inputs and their [dependencies](/concepts/dependencies), applies the specified build rules, and produces an [action](/extending/concepts#evaluation-model) graph. +2. **Analyzes** the inputs and their + [dependencies](/concepts/dependencies), applies the specified build + rules, and produces an [action](/extending/concepts#evaluation-model) + graph. -3. **Executes** the build actions on the inputs until the final build outputs are produced. +3. **Executes** the build actions on the inputs until the final build outputs + are produced. -Since all previous build work is cached, Bazel can identify and reuse cached artifacts and only rebuild or retest what's changed. To further enforce correctness, you can set up Bazel to run builds and tests [hermetically](/basics/hermeticity) through sandboxing, minimizing skew and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). +Since all previous build work is cached, Bazel can identify and reuse cached +artifacts and only rebuild or retest what's changed. To further enforce +correctness, you can set up Bazel to run builds and tests +[hermetically](/basics/hermeticity) through sandboxing, minimizing skew +and maximizing [reproducibility](/run/build#correct-incremental-rebuilds). ### Action graph -The action graph represents the build artifacts, the relationships between them, and the build actions that Bazel will perform. Thanks to this graph, Bazel can [track](/run/build#build-consistency) changes to file content as well as changes to actions, such as build or test commands, and know what build work has previously been done. The graph also enables you to easily [trace dependencies](/query/guide) in your code. +The action graph represents the build artifacts, the relationships between them, +and the build actions that Bazel will perform. Thanks to this graph, Bazel can +[track](/run/build#build-consistency) changes to +file content as well as changes to actions, such as build or test commands, and +know what build work has previously been done. The graph also enables you to +easily [trace dependencies](/query/guide) in your code. ## Getting started tutorials -To get started with Bazel, see [Getting Started](/start/) or jump directly to the Bazel tutorials: +To get started with Bazel, see [Getting Started](/start/) or jump +directly to the Bazel tutorials: -- [Tutorial: Build a C++ Project](/start/cpp) -- [Tutorial: Build a Java Project](/start/java) -- [Tutorial: Build an Android Application](/start/android-app) -- [Tutorial: Build an iOS Application](/start/ios-app) +* [Tutorial: Build a C++ Project](/start/cpp) +* [Tutorial: Build a Java Project](/start/java) +* [Tutorial: Build an Android Application](/start/android-app) +* [Tutorial: Build an iOS Application](/start/ios-app) diff --git a/about/roadmap.mdx b/about/roadmap.mdx index c17485d2..42e63e9b 100644 --- a/about/roadmap.mdx +++ b/about/roadmap.mdx @@ -2,50 +2,98 @@ title: 'Bazel roadmap' --- -As Bazel continues to evolve in response to your needs, we want to share our 2025 roadmap update. -We plan to bring Bazel 9.0 [long term support (LTS)](https://bazel.build/release/versioning) to you in late 2025. + +As Bazel continues to evolve in response to your needs, we want to share our +2025 roadmap update. + +We plan to bring Bazel 9.0 +[long term support (LTS)](https://bazel.build/release/versioning) to you in late +2025. ## Full transition to Bzlmod -[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) hosts more than 650 modules. +[Bzlmod](https://bazel.build/docs/bzlmod) has been the standard external +dependency system in Bazel since Bazel 7, replacing the legacy WORKSPACE system. +As of March 2025, the [Bazel Central Registry](https://registry.bazel.build/) +hosts more than 650 modules. -With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will be the only way to introduce external dependencies in Bazel. To minimize the migration cost for the community, we'll focus on further improving our migration [guide](https://bazel.build/external/migration) and [tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). +With Bazel 9, we will completely remove WORKSPACE functionality, and Bzlmod will +be the only way to introduce external dependencies in Bazel. To minimize the +migration cost for the community, we'll focus on further improving our migration +[guide](https://bazel.build/external/migration) and +[tool](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools#migrate_to_bzlmodpy). -Additionally, we aim to implement an improved shared repository cache (see [#12227](https://github.com/bazelbuild/bazel/issues/12227)) with garbage collection, and may backport it to Bazel 8. The Bazel Central Registry will also support verifying SLSA attestations. +Additionally, we aim to implement an improved shared repository cache (see +[#12227](https://github.com/bazelbuild/bazel/issues/12227)) +with garbage collection, and may backport it to Bazel 8. The Bazel Central +Registry will also support verifying SLSA attestations. ## Migration of Android, C++, Java, Python, and Proto rules -With Bazel 8, we have migrated support for Android, Java, Python, and Proto rules out of the Bazel codebase into Starlark rules in their corresponding repositories. To ease the migration, we implemented the autoload features in Bazel, which can be controlled with [--incompatible\_autoload\_externally](https://github.com/bazelbuild/bazel/issues/23043) and [--incompatible\_disable\_autoloads\_in\_main\_repo](https://github.com/bazelbuild/bazel/issues/25755) flags. +With Bazel 8, we have migrated support for Android, Java, Python, and Proto +rules out of the Bazel codebase into Starlark rules in their corresponding +repositories. To ease the migration, we implemented the autoload features in +Bazel, which can be controlled with +[--incompatible_autoload_externally](https://github.com/bazelbuild/bazel/issues/23043) +and [--incompatible_disable_autoloads_in_main_repo](https://github.com/bazelbuild/bazel/issues/25755) +flags. -With Bazel 9, we aim to disable autoloads by default and require every project to explicitly load required rules in BUILD files. +With Bazel 9, we aim to disable autoloads by default and require every project +to explicitly load required rules in BUILD files. -We will rewrite most of C++ language support to Starlark, detach it from Bazel binary and move it into the [/rules\_cc](https://github.com/bazelbuild/rules_cc) repository. This is the last remaining major language support that is still part of Bazel. +We will rewrite most of C++ language support to Starlark, detach it from Bazel +binary and move it into the [/rules_cc](https://github.com/bazelbuild/rules_cc) +repository. This is the last remaining major language support that is still part +of Bazel. -We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving them to repositories next to the implementation to increase velocity of rule authors. +We're also porting unit tests for C++, Java, and Proto rules to Starlark, moving +them to repositories next to the implementation to increase velocity of rule +authors. ## Starlark improvements -Bazel will have the ability to evaluate symbolic macros lazily. This means that a symbolic macro won't run if the targets it declares are not requested, improving performance for very large packages. +Bazel will have the ability to evaluate symbolic macros lazily. This means that +a symbolic macro won't run if the targets it declares are not requested, +improving performance for very large packages. -Starlark will have an experimental type system, similar to Python's type annotations. We expect the type system to stabilize *after* Bazel 9 is launched. +Starlark will have an experimental type system, similar to Python's type +annotations. We expect the type system to stabilize _after_ Bazel 9 is launched. ## Configurability Our main focus is reducing the cost and confusion of build flags. -We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a new project configuration model that doesn't make users have to know which build and test flags to set where. So `$ bazel test //foo` automatically sets the right flags based on `foo`'s project's policy. This will likely remain experimental in 9.0 but guiding feedback is welcome. - -[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip out Starlark flags when they leave project boundaries, so they don't break caching on transitive dependencies that don't need them. This makes builds that use [transitions](https://bazel.build/extending/config#user-defined-transitions) cheaper and faster. [Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) an example. We're extending the idea to control which flags propagate to [exec configurations](https://bazel.build/extending/rules#configurations) and are considering even more flexible support like custom Starlark to determine which dependency edges should propagate flags. - -We're up-prioritizing effort to move built-in language flags out of Bazel and into Starlark, where they can live with related rule definitions. +We're [experimenting](https://github.com/bazelbuild/bazel/issues/24839) with a +new project configuration model that doesn't make users have to know which build +and test flags to set where. So `$ bazel test //foo` automatically sets the +right flags based on `foo`'s project's policy. This will likely remain +experimental in 9.0 but guiding feedback is welcome. + +[Flag scoping](https://github.com/bazelbuild/bazel/issues/24042) lets you strip +out Starlark flags when they leave project boundaries, so they don't break +caching on transitive dependencies that don't need them. This makes builds that +use [transitions](https://bazel.build/extending/config#user-defined-transitions) +cheaper and faster. +[Here's](https://github.com/gregestren/snippets/tree/master/project_scoped_flags) +an example. We're extending the idea to control which flags propagate to +[exec configurations](https://bazel.build/extending/rules#configurations) and +are considering even more flexible support like custom Starlark to determine +which dependency edges should propagate flags. + +We're up-prioritizing effort to move built-in language flags out of Bazel and +into Starlark, where they can live with related rule definitions. ## Remote execution improvements -We plan to add support for asynchronous execution, speeding up remote execution by increasing parallelism. +We plan to add support for asynchronous execution, speeding up remote execution +by increasing parallelism. -*** +--- -To follow updates to the roadmap and discuss planned features, join the community Slack server at [slack.bazel.build](https://slack.bazel.build/). +To follow updates to the roadmap and discuss planned features, join the +community Slack server at [slack.bazel.build](https://slack.bazel.build/). -*This roadmap is intended to help inform the community about the team's intentions for Bazel 9.0. Priorities are subject to change in response to developer and customer feedback, or to new market opportunities.* +*This roadmap is intended to help inform the community about the team's +intentions for Bazel 9.0. Priorities are subject to change in response to +developer and customer feedback, or to new market opportunities.* diff --git a/about/vision.mdx b/about/vision.mdx index b80ca03a..da0ed02d 100644 --- a/about/vision.mdx +++ b/about/vision.mdx @@ -2,48 +2,96 @@ title: 'Bazel Vision' --- -Any software developer can efficiently build, test, and package any project, of any size or complexity, with tooling that's easy to adopt and extend. -- **Engineers can take build fundamentals for granted.** Software developers focus on the creative process of authoring code because the mechanical process of build and test is solved. When customizing the build system to support new languages or unique organizational needs, users focus on the aspects of extensibility that are unique to their use case, without having to reinvent the basic plumbing. -- **Engineers can easily contribute to any project.** A developer who wants to start working on a new project can simply clone the project and run the build. There's no need for local configuration - it just works. With cross-platform remote execution, they can work on any machine anywhere and fully test their changes against all platforms the project targets. Engineers can quickly configure the build for a new project or incrementally migrate an existing build. +Any software developer can efficiently build, test, and package +any project, of any size or complexity, with tooling that's easy to adopt and +extend. -- **Projects can scale to any size codebase, any size team.** Fast, incremental testing allows teams to fully validate every change before it is committed. This remains true even as repos grow, projects span multiple repos, and multiple languages are introduced. Infrastructure does not force developers to trade test coverage for build speed. +* **Engineers can take build fundamentals for granted.** Software developers + focus on the creative process of authoring code because the mechanical + process of build and test is solved. When customizing the build system to + support new languages or unique organizational needs, users focus on the + aspects of extensibility that are unique to their use case, without having + to reinvent the basic plumbing. + +* **Engineers can easily contribute to any project.** A developer who wants to + start working on a new project can simply clone the project and run the + build. There's no need for local configuration - it just works. With + cross-platform remote execution, they can work on any machine anywhere and + fully test their changes against all platforms the project targets. + Engineers can quickly configure the build for a new project or incrementally + migrate an existing build. + +* **Projects can scale to any size codebase, any size team.** Fast, + incremental testing allows teams to fully validate every change before it is + committed. This remains true even as repos grow, projects span multiple + repos, and multiple languages are introduced. Infrastructure does not force + developers to trade test coverage for build speed. **We believe Bazel has the potential to fulfill this vision.** -Bazel was built from the ground up to enable builds that are reproducible (a given set of inputs will always produce the same outputs) and portable (a build can be run on any machine without affecting the output). +Bazel was built from the ground up to enable builds that are reproducible (a +given set of inputs will always produce the same outputs) and portable (a build +can be run on any machine without affecting the output). -These characteristics support safe incrementality (rebuilding only changed inputs doesn't introduce the risk of corruption) and distributability (build actions are isolated and can be offloaded). By minimizing the work needed to do a correct build and parallelizing that work across multiple cores and remote systems, Bazel can make any build fast. +These characteristics support safe incrementality (rebuilding only changed +inputs doesn't introduce the risk of corruption) and distributability (build +actions are isolated and can be offloaded). By minimizing the work needed to do +a correct build and parallelizing that work across multiple cores and remote +systems, Bazel can make any build fast. -Bazel's abstraction layer — instructions specific to languages, platforms, and toolchains implemented in a simple extensibility language — allows it to be easily applied to any context. +Bazel's abstraction layer — instructions specific to languages, platforms, and +toolchains implemented in a simple extensibility language — allows it to be +easily applied to any context. ## Bazel core competencies -1. Bazel supports **multi-language, multi-platform** builds and tests. You can run a single command to build and test your entire source tree, no matter which combination of languages and platforms you target. -2. Bazel builds are **fast and correct**. Every build and test run is incremental, on your developers' machines and on CI. -3. Bazel provides a **uniform, extensible language** to define builds for any language or platform. -4. Bazel allows your builds **to scale** by connecting to remote execution and caching services. -5. Bazel works across **all major development platforms** (Linux, MacOS, and Windows). -6. We accept that adopting Bazel requires effort, but **gradual adoption** is possible. Bazel interfaces with de-facto standard tools for a given language/platform. +1. Bazel supports **multi-language, multi-platform** builds and tests. You can + run a single command to build and test your entire source tree, no matter + which combination of languages and platforms you target. +1. Bazel builds are **fast and correct**. Every build and test run is + incremental, on your developers' machines and on CI. +1. Bazel provides a **uniform, extensible language** to define builds for any + language or platform. +1. Bazel allows your builds **to scale** by connecting to remote execution and + caching services. +1. Bazel works across **all major development platforms** (Linux, MacOS, and + Windows). +1. We accept that adopting Bazel requires effort, but **gradual adoption** is + possible. Bazel interfaces with de-facto standard tools for a given + language/platform. ## Serving language communities -Software engineering evolves in the context of language communities — typically, self-organizing groups of people who use common tools and practices. - -To be of use to members of a language community, high-quality Bazel rules must be available that integrate with the workflows and conventions of that community. - -Bazel is committed to be extensible and open, and to support good rulesets for any language. - -### Requirements of a good ruleset - -1. The rules need to support efficient **building and testing** for the language, including code coverage. -2. The rules need to **interface with a widely-used "package manager"** for the language (such as Maven for Java), and support incremental migration paths from other widely-used build systems. -3. The rules need to be **extensible and interoperable**, following ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) principles. -4. The rules need to be **remote-execution ready**. In practice, this means **configurable using the [toolchains](/extending/toolchains) mechanism**. -5. The rules (and Bazel) need to interface with a **widely-used IDE** for the language, if there is one. -6. The rules need to have **thorough, usable documentation,** with introductory material for new users, comprehensive docs for expert users. - -Each of these items is essential and only together do they deliver on Bazel's competencies for their particular ecosystem. - -They are also, by and large, sufficient - once all are fulfilled, Bazel fully delivers its value to members of that language community. +Software engineering evolves in the context of language communities — typically, +self-organizing groups of people who use common tools and practices. + +To be of use to members of a language community, high-quality Bazel rules must be +available that integrate with the workflows and conventions of that community. + +Bazel is committed to be extensible and open, and to support good rulesets for +any language. + +### Requirements of a good ruleset + +1. The rules need to support efficient **building and testing** for the + language, including code coverage. +1. The rules need to **interface with a widely-used "package manager"** for the + language (such as Maven for Java), and support incremental migration paths + from other widely-used build systems. +1. The rules need to be **extensible and interoperable**, following + ["Bazel sandwich"](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-08-04-extensibility-for-native-rules.md) + principles. +1. The rules need to be **remote-execution ready**. In practice, this means + **configurable using the [toolchains](/extending/toolchains) mechanism**. +1. The rules (and Bazel) need to interface with a **widely-used IDE** for the + language, if there is one. +1. The rules need to have **thorough, usable documentation,** with introductory + material for new users, comprehensive docs for expert users. + +Each of these items is essential and only together do they deliver on Bazel's +competencies for their particular ecosystem. + +They are also, by and large, sufficient - once all are fulfilled, Bazel fully +delivers its value to members of that language community. diff --git a/about/why.mdx b/about/why.mdx index dcdec18e..6224abf6 100644 --- a/about/why.mdx +++ b/about/why.mdx @@ -2,44 +2,84 @@ title: 'Why Bazel?' --- -Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) build tool with [integrated testing](#integrated-testing) that supports multiple [languages](#multi-language), [repositories](#multi-repository), and [platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). + + +Bazel is a [fast](#fast), [correct](#correct), and [extensible](#extensible) +build tool with [integrated testing](#integrated-testing) that supports multiple +[languages](#multi-language), [repositories](#multi-repository), and +[platforms](#multi-platform) in an industry-leading [ecosystem](#ecosystem). ## Bazel is fast -Bazel knows exactly what input files each build command needs, avoiding unnecessary work by re-running only when the set of input files have changed between each build. It runs build commands with as much parallelism as possible, either within the same computer or on [remote build nodes](/remote/rbe). If the structure of build allows for it, it can run thousands of build or test commands at the same time. +Bazel knows exactly what input files each build command needs, avoiding +unnecessary work by re-running only when the set of input files have +changed between each build. +It runs build commands with as much parallelism as possible, either within the +same computer or on [remote build nodes](/remote/rbe). If the structure of build +allows for it, it can run thousands of build or test commands at the same time. -This is supported by multiple caching layers, in memory, on disk and on the remote build farm, if available. At Google, we routinely achieve cache hit rates north of 99%. +This is supported by multiple caching layers, in memory, on disk and on the +remote build farm, if available. At Google, we routinely achieve cache hit rates +north of 99%. ## Bazel is correct -Bazel ensures that your binaries are built *only* from your own source code. Bazel actions run in individual sandboxes and Bazel tracks every input file of the build, only and always re-running build commands when it needs to. This keeps your binaries up-to-date so that the [same source code always results in the same binary](/basics/hermeticity), bit by bit. +Bazel ensures that your binaries are built *only* from your own +source code. Bazel actions run in individual sandboxes and Bazel tracks +every input file of the build, only and always re-running build +commands when it needs to. This keeps your binaries up-to-date so that the +[same source code always results in the same binary](/basics/hermeticity), bit +by bit. -Say goodbye to endless `make clean` invocations and to chasing phantom bugs that were in fact resolved in source code that never got built. +Say goodbye to endless `make clean` invocations and to chasing phantom bugs +that were in fact resolved in source code that never got built. ## Bazel is extensible -Harness the full power of Bazel by writing your own rules and macros to customize Bazel for your specific needs across a wide range of projects. +Harness the full power of Bazel by writing your own rules and macros to +customize Bazel for your specific needs across a wide range of projects. -Bazel rules are written in [Starlark](/rules/language), our in-house programming language that's a subset of Python. Starlark makes rule-writing accessible to most developers, while also creating rules that can be used across the ecosystem. +Bazel rules are written in [Starlark](/rules/language), our +in-house programming language that's a subset of Python. Starlark makes +rule-writing accessible to most developers, while also creating rules that can +be used across the ecosystem. ## Integrated testing -Bazel's [integrated test runner](/docs/user-manual#running-tests) knows and runs only those tests needing to be re-run, using remote execution (if available) to run them in parallel. Detect flakes early by using remote execution to quickly run a test thousands of times. +Bazel's [integrated test runner](/docs/user-manual#running-tests) +knows and runs only those tests needing to be re-run, using remote execution +(if available) to run them in parallel. Detect flakes early by using remote +execution to quickly run a test thousands of times. -Bazel [provides facilities](/remote/bep) to upload test results to a central location, thereby facilitating efficient communication of test outcomes, be it on CI or by individual developers. +Bazel [provides facilities](/remote/bep) to upload test results to a central +location, thereby facilitating efficient communication of test outcomes, be it +on CI or by individual developers. ## Multi-language support -Bazel supports many common programming languages including C++, Java, Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, backend, web UI and mobile app) in the same Bazel invocation without being constrained to one language's idiomatic build tool. +Bazel supports many common programming languages including C++, Java, +Kotlin, Python, Go, and Rust. You can build multiple binaries (for example, +backend, web UI and mobile app) in the same Bazel invocation without being +constrained to one language's idiomatic build tool. ## Multi-repository support -Bazel can [gather source code from multiple locations](/external/overview): you don't need to vendor your dependencies (but you can!), you can instead point Bazel to the location of your source code or prebuilt artifacts (e.g. a git repository or Maven Central), and it takes care of the rest. +Bazel can [gather source code from multiple locations](/external/overview): you +don't need to vendor your dependencies (but you can!), you can instead point +Bazel to the location of your source code or prebuilt artifacts (e.g. a git +repository or Maven Central), and it takes care of the rest. ## Multi-platform support -Bazel can simultaneously build projects for multiple platforms including Linux, macOS, Windows, and Android. It also provides powerful [cross-compilation capabilities](/extending/platforms) to build code for one platform while running the build on another. +Bazel can simultaneously build projects for multiple platforms including Linux, +macOS, Windows, and Android. It also provides powerful +[cross-compilation capabilities](/extending/platforms) to build code for one +platform while running the build on another. ## Wide ecosystem -[Industry leaders](/community/users) love Bazel, building a large community of developers who use and contribute to Bazel. Find a tools, services and documentation, including [consulting and SaaS offerings](/community/experts) Bazel can use. Explore extensions like support for programming languages in our [open source software repositories](/rules). +[Industry leaders](/community/users) love Bazel, building a large +community of developers who use and contribute to Bazel. Find a tools, services +and documentation, including [consulting and SaaS offerings](/community/experts) +Bazel can use. Explore extensions like support for programming languages in +our [open source software repositories](/rules). diff --git a/advanced/performance/build-performance-breakdown.mdx b/advanced/performance/build-performance-breakdown.mdx index 8bebff8c..477e7578 100644 --- a/advanced/performance/build-performance-breakdown.mdx +++ b/advanced/performance/build-performance-breakdown.mdx @@ -2,120 +2,234 @@ title: 'Breaking down build performance' --- -Bazel is complex and does a lot of different things over the course of a build, some of which can have an impact on build performance. This page attempts to map some of these Bazel concepts to their implications on build performance. While not extensive, we have included some examples of how to detect build performance issues through [extracting metrics](/configure/build-performance-metrics) and what you can do to fix them. With this, we hope you can apply these concepts when investigating build performance regressions. -### Clean vs Incremental builds -A clean build is one that builds everything from scratch, while an incremental build reuses some already completed work. +Bazel is complex and does a lot of different things over the course of a build, +some of which can have an impact on build performance. This page attempts to map +some of these Bazel concepts to their implications on build performance. While +not extensive, we have included some examples of how to detect build performance +issues through [extracting metrics](/configure/build-performance-metrics) +and what you can do to fix them. With this, we hope you can apply these concepts +when investigating build performance regressions. -We suggest looking at clean and incremental builds separately, especially when you are collecting / aggregating metrics that are dependent on the state of Bazel’s caches (for example [build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) ). They also represent two different user experiences. As compared to starting a clean build from scratch (which takes longer due to a cold cache), incremental builds happen far more frequently as developers iterate on code (typically faster since the cache is usually already warm). +### Clean vs Incremental builds -You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly categorize it to likely be an incremental build - the user could have switched to different flags or different targets causing an effectively clean build. Any more rigorous definition of incrementality will likely have to come in the form of a heuristic, for example looking at the number of packages loaded (`PackageMetrics.packages_loaded`). +A clean build is one that builds everything from scratch, while an incremental +build reuses some already completed work. + +We suggest looking at clean and incremental builds separately, especially when +you are collecting / aggregating metrics that are dependent on the state of +Bazel’s caches (for example +[build request size metrics](#deterministic-build-metrics-as-a-proxy-for-build-performance) +). They also represent two different user experiences. As compared to starting +a clean build from scratch (which takes longer due to a cold cache), incremental +builds happen far more frequently as developers iterate on code (typically +faster since the cache is usually already warm). + +You can use the `CumulativeMetrics.num_analyses` field in the BEP to classify +builds. If `num_analyses <= 1`, it is a clean build; otherwise, we can broadly +categorize it to likely be an incremental build - the user could have switched +to different flags or different targets causing an effectively clean build. Any +more rigorous definition of incrementality will likely have to come in the form +of a heuristic, for example looking at the number of packages loaded +(`PackageMetrics.packages_loaded`). ### Deterministic build metrics as a proxy for build performance -Measuring build performance can be difficult due to the non-deterministic nature of certain metrics (for example Bazel’s CPU time or queue times on a remote cluster). As such, it can be useful to use deterministic metrics as a proxy for the amount of work done by Bazel, which in turn affects its performance. - -The size of a build request can have a significant implication on build performance. A larger build could represent more work in analyzing and constructing the build graphs. Organic growth of builds comes naturally with development, as more dependencies are added/created, and thus grow in complexity and become more expensive to build. - -We can slice this problem into the various build phases, and use the following metrics as proxy metrics for work done at each phase: - -1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. A regression here represents more work that needs to be done to read and parse each additional BUILD file in the loading phase. - - - This is often due to the addition of dependencies and having to load their transitive closure. - - Use [query](/query/quickstart) / [cquery](/query/cquery) to find where new dependencies might have been added. - -2. `TargetMetrics.targets_configured`: representing the number of targets and aspects configured in the build. A regression represents more work in constructing and traversing the configured target graph. - - - This is often due to the addition of dependencies and having to construct the graph of their transitive closure. - - Use [cquery](/query/cquery) to find where new dependencies might have been added. - -3. `ActionSummary.actions_created`: represents the actions created in the build, and a regression represents more work in constructing the action graph. Note that this also includes unused actions that might not have been executed. - - - Use [aquery](/query/aquery) for debugging regressions; we suggest starting with [`--output=summary`](/reference/command-line-reference#flag--output) before further drilling down with [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). - -4. `ActionSummary.actions_executed`: the number of actions executed, a regression directly represents more work in executing these actions. - - - The [BEP](/remote/bep) writes out the action statistics `ActionData` that shows the most executed action types. By default, it collects the top 20 action types, but you can pass in the [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) to collect this data for all action types that were executed. - - This should help you to figure out what kind of actions were executed (additionally). - -5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by the executed actions. - - - If the number of actions executed did not increase, then it is likely that a rule implementation was changed. - -These metrics are all affected by the state of the local cache, hence you will want to ensure that the builds you extract these metrics from are **clean builds**. - -We have noted that a regression in any of these metrics can be accompanied by regressions in wall time, cpu time and memory usage. +Measuring build performance can be difficult due to the non-deterministic nature +of certain metrics (for example Bazel’s CPU time or queue times on a remote +cluster). As such, it can be useful to use deterministic metrics as a proxy for +the amount of work done by Bazel, which in turn affects its performance. + +The size of a build request can have a significant implication on build +performance. A larger build could represent more work in analyzing and +constructing the build graphs. Organic growth of builds comes naturally with +development, as more dependencies are added/created, and thus grow in complexity +and become more expensive to build. + +We can slice this problem into the various build phases, and use the following +metrics as proxy metrics for work done at each phase: + +1. `PackageMetrics.packages_loaded`: the number of packages successfully loaded. + A regression here represents more work that needs to be done to read and parse + each additional BUILD file in the loading phase. + - This is often due to the addition of dependencies and having to load their + transitive closure. + - Use [query](/query/quickstart) / [cquery](/query/cquery) to find + where new dependencies might have been added. + +2. `TargetMetrics.targets_configured`: representing the number of targets and + aspects configured in the build. A regression represents more work in + constructing and traversing the configured target graph. + - This is often due to the addition of dependencies and having to construct + the graph of their transitive closure. + - Use [cquery](/query/cquery) to find where new + dependencies might have been added. + +3. `ActionSummary.actions_created`: represents the actions created in the build, + and a regression represents more work in constructing the action graph. Note + that this also includes unused actions that might not have been executed. + - Use [aquery](/query/aquery) for debugging regressions; + we suggest starting with + [`--output=summary`](/reference/command-line-reference#flag--output) + before further drilling down with + [`--skyframe_state`](/reference/command-line-reference#flag--skyframe_state). + +4. `ActionSummary.actions_executed`: the number of actions executed, a + regression directly represents more work in executing these actions. + - The [BEP](/remote/bep) writes out the action statistics + `ActionData` that shows the most executed action types. By default, it + collects the top 20 action types, but you can pass in the + [`--experimental_record_metrics_for_all_mnemonics`](/reference/command-line-reference#flag--experimental_record_metrics_for_all_mnemonics) + to collect this data for all action types that were executed. + - This should help you to figure out what kind of actions were executed + (additionally). + +5. `BuildGraphSummary.outputArtifactCount`: the number of artifacts created by + the executed actions. + - If the number of actions executed did not increase, then it is likely that + a rule implementation was changed. + + +These metrics are all affected by the state of the local cache, hence you will +want to ensure that the builds you extract these metrics from are +**clean builds**. + +We have noted that a regression in any of these metrics can be accompanied by +regressions in wall time, cpu time and memory usage. ### Usage of local resources -Bazel consumes a variety of resources on your local machine (both for analyzing the build graph and driving the execution, and for running local actions), this can affect the performance / availability of your machine in performing the build, and also other tasks. +Bazel consumes a variety of resources on your local machine (both for analyzing +the build graph and driving the execution, and for running local actions), this +can affect the performance / availability of your machine in performing the +build, and also other tasks. #### Time spent -Perhaps the metrics most susceptible to noise (and can vary greatly from build to build) is time; in particular - wall time, cpu time and system time. You can use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get a benchmark for these metrics, and with a sufficient number of `--runs`, you can increase the statistical significance of your measurement. +Perhaps the metrics most susceptible to noise (and can vary greatly from build +to build) is time; in particular - wall time, cpu time and system time. You can +use [bazel-bench](https://github.com/bazelbuild/bazel-bench) to get +a benchmark for these metrics, and with a sufficient number of `--runs`, you can +increase the statistical significance of your measurement. - **Wall time** is the real world time elapsed. - - - If *only* wall time regresses, we suggest collecting a [JSON trace profile](/advanced/performance/json-trace-profile) and looking for differences. Otherwise, it would likely be more efficient to investigate other regressed metrics as they could have affected the wall time. + - If _only_ wall time regresses, we suggest collecting a + [JSON trace profile](/advanced/performance/json-trace-profile) and looking + for differences. Otherwise, it would likely be more efficient to + investigate other regressed metrics as they could have affected the wall + time. - **CPU time** is the time spent by the CPU executing user code. - - - If the CPU time regresses across two project commits, we suggest collecting a Starlark CPU profile. You should probably also use `--nobuild` to restrict the build to the analysis phase since that is where most of the CPU heavy work is done. + - If the CPU time regresses across two project commits, we suggest collecting + a Starlark CPU profile. You should probably also use `--nobuild` to + restrict the build to the analysis phase since that is where most of the + CPU heavy work is done. - System time is the time spent by the CPU in the kernel. - - - If system time regresses, it is mostly correlated with I/O when Bazel reads files from your file system. + - If system time regresses, it is mostly correlated with I/O when Bazel reads + files from your file system. #### System-wide load profiling -Using the [`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) flag introduced in Bazel 6.0, the [JSON trace profiler](/advanced/performance/json-trace-profile) collects the system load average during the invocation. +Using the +[`--experimental_collect_load_average_in_profiler`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L306-L312) +flag introduced in Bazel 6.0, the +[JSON trace profiler](/advanced/performance/json-trace-profile) collects the +system load average during the invocation. ![Profile that includes system load average](/docs/images/json-trace-profile-system-load-average.png "Profile that includes system load average") **Figure 1.** Profile that includes system load average. -A high load during a Bazel invocation can be an indication that Bazel schedules too many local actions in parallel for your machine. You might want to look into adjusting [`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), especially in container environments (at least until [#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). +A high load during a Bazel invocation can be an indication that Bazel schedules +too many local actions in parallel for your machine. You might want to look into +adjusting +[`--local_cpu_resources`](/reference/command-line-reference#flag--local_cpu_resources) +and [`--local_ram_resources`](/reference/command-line-reference#flag--local_ram_resources), +especially in container environments (at least until +[#16512](https://github.com/bazelbuild/bazel/pull/16512) is merged). -#### Monitoring Bazel memory usage - -There are two main sources to get Bazel’s memory usage, Bazel `info` and the [BEP](/remote/bep). - -- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after a call to `System.gc()`. - - [Bazel bench](https://github.com/bazelbuild/bazel-bench) provides benchmarks for this metric as well. - - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` and `committed-heap-size` (see [documentation](/docs/user-manual#configuration-independent-data)), but are less relevant. - -- [BEP](/remote/bep)’s `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in bytes post GC (requires setting [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) that attempts to force a full GC). - -A regression in memory usage is usually a result of a regression in [build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), which are often due to addition of dependencies or a change in the rule implementation. +#### Monitoring Bazel memory usage -To analyze Bazel’s memory footprint on a more granular level, we recommend using the [built-in memory profiler](/rules/performance#memory-profiling) for rules. +There are two main sources to get Bazel’s memory usage, Bazel `info` and the +[BEP](/remote/bep). + +- `bazel info used-heap-size-after-gc`: The amount of used memory in bytes after + a call to `System.gc()`. + - [Bazel bench](https://github.com/bazelbuild/bazel-bench) + provides benchmarks for this metric as well. + - Additionally, there are `peak-heap-size`, `max-heap-size`, `used-heap-size` + and `committed-heap-size` (see + [documentation](/docs/user-manual#configuration-independent-data)), but are + less relevant. + +- [BEP](/remote/bep)’s + `MemoryMetrics.peak_post_gc_heap_size`: Size of the peak JVM heap size in + bytes post GC (requires setting + [`--memory_profile`](/reference/command-line-reference#flag--memory_profile) + that attempts to force a full GC). + +A regression in memory usage is usually a result of a regression in +[build request size metrics](#deterministic_build_metrics_as_a_proxy_for_build_performance), +which are often due to addition of dependencies or a change in the rule +implementation. + +To analyze Bazel’s memory footprint on a more granular level, we recommend using +the [built-in memory profiler](/rules/performance#memory-profiling) +for rules. #### Memory profiling of persistent workers -While [persistent workers](/remote/persistent) can help to speed up builds significantly (especially for interpreted languages) their memory footprint can be problematic. Bazel collects metrics on its workers, in particular, the `WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory workers use (by mnemonic). +While [persistent workers](/remote/persistent) can help to speed up builds +significantly (especially for interpreted languages) their memory footprint can +be problematic. Bazel collects metrics on its workers, in particular, the +`WorkerMetrics.WorkerStats.worker_memory_in_kb` field tells how much memory +workers use (by mnemonic). -The [JSON trace profiler](/advanced/performance/json-trace-profile) also collects persistent worker memory usage during the invocation by passing in the [`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) flag (new in Bazel 6.0). +The [JSON trace profiler](/advanced/performance/json-trace-profile) also +collects persistent worker memory usage during the invocation by passing in the +[`--experimental_collect_system_network_usage`](https://github.com/bazelbuild/bazel/blob/6.0.0/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java#L314-L320) +flag (new in Bazel 6.0). ![Profile that includes workers memory usage](/docs/images/json-trace-profile-workers-memory-usage.png "Profile that includes workers memory usage") **Figure 2.** Profile that includes workers memory usage. -Lowering the value of [`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) (default 4) might help to reduce the amount of memory used by persistent workers. We are actively working on making Bazel’s resource manager and scheduler smarter so that such fine tuning will be required less often in the future. +Lowering the value of +[`--worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) +(default 4) might help to reduce +the amount of memory used by persistent workers. We are actively working on +making Bazel’s resource manager and scheduler smarter so that such fine tuning +will be required less often in the future. ### Monitoring network traffic for remote builds -In remote execution, Bazel downloads artifacts that were built as a result of executing actions. As such, your network bandwidth can affect the performance of your build. +In remote execution, Bazel downloads artifacts that were built as a result of +executing actions. As such, your network bandwidth can affect the performance +of your build. -If you are using remote execution for your builds, you might want to consider monitoring the network traffic during the invocation using the `NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) (requires passing `--experimental_collect_system_network_usage`). +If you are using remote execution for your builds, you might want to consider +monitoring the network traffic during the invocation using the +`NetworkMetrics.SystemNetworkStats` proto from the [BEP](/remote/bep) +(requires passing `--experimental_collect_system_network_usage`). -Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) allow you to view system-wide network usage throughout the course of the build by passing the `--experimental_collect_system_network_usage` flag (new in Bazel 6.0). +Furthermore, [JSON trace profiles](/advanced/performance/json-trace-profile) +allow you to view system-wide network usage throughout the course of the build +by passing the `--experimental_collect_system_network_usage` flag (new in Bazel +6.0). ![Profile that includes system-wide network usage](/docs/images/json-trace-profile-network-usage.png "Profile that includes system-wide network usage") **Figure 3.** Profile that includes system-wide network usage. -A high but rather flat network usage when using remote execution might indicate that network is the bottleneck in your build; if you are not using it already, consider turning on Build without the Bytes by passing [`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. +A high but rather flat network usage when using remote execution might indicate +that network is the bottleneck in your build; if you are not using it already, +consider turning on Build without the Bytes by passing +[`--remote_download_minimal`](/reference/command-line-reference#flag--remote_download_minimal). +This will speed up your builds by avoiding the download of unnecessary intermediate artifacts. -Another option is to configure a local [disk cache](/reference/command-line-reference#flag--disk_cache) to save on download bandwidth. +Another option is to configure a local +[disk cache](/reference/command-line-reference#flag--disk_cache) to save on +download bandwidth. diff --git a/advanced/performance/build-performance-metrics.mdx b/advanced/performance/build-performance-metrics.mdx index 8f92e75a..8391ea87 100644 --- a/advanced/performance/build-performance-metrics.mdx +++ b/advanced/performance/build-performance-metrics.mdx @@ -2,50 +2,96 @@ title: 'Extracting build performance metrics' --- -Probably every Bazel user has experienced builds that were slow or slower than anticipated. Improving the performance of individual builds has particular value for targets with significant impact, such as: + + +Probably every Bazel user has experienced builds that were slow or slower than +anticipated. Improving the performance of individual builds has particular value +for targets with significant impact, such as: 1. Core developer targets that are frequently iterated on and (re)built. 2. Common libraries widely depended upon by other targets. -3. A representative target from a class of targets (e.g. custom rules), diagnosing and fixing issues in one build might help to resolve issues at the larger scale. +3. A representative target from a class of targets (e.g. custom rules), + diagnosing and fixing issues in one build might help to resolve issues at the + larger scale. -An important step to improving the performance of builds is to understand where resources are spent. This page lists different metrics you can collect. [Breaking down build performance](/configure/build-performance-breakdown) showcases how you can use these metrics to detect and fix build performance issues. +An important step to improving the performance of builds is to understand where +resources are spent. This page lists different metrics you can collect. +[Breaking down build performance](/configure/build-performance-breakdown) showcases +how you can use these metrics to detect and fix build performance issues. There are a few main ways to extract metrics from your Bazel builds, namely: ## Build Event Protocol (BEP) -Bazel outputs a variety of protocol buffers [`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) through the [Build Event Protocol (BEP)](/remote/bep), which can be aggregated by a backend specified by you. Depending on your use cases, you might decide to aggregate the metrics in various ways, but here we will go over some concepts and proto fields that would be useful in general to consider. +Bazel outputs a variety of protocol buffers +[`build_event_stream.proto`](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) +through the [Build Event Protocol (BEP)](/remote/bep), which +can be aggregated by a backend specified by you. Depending on your use cases, +you might decide to aggregate the metrics in various ways, but here we will go +over some concepts and proto fields that would be useful in general to consider. ## Bazel’s query / cquery / aquery commands -Bazel provides 3 different query modes ([query](/query/quickstart), [cquery](/query/cquery) and [aquery](/query/aquery)) that allow users to query the target graph, configured target graph and action graph respectively. The query language provides a [suite of functions](/query/language#functions) usable across the different query modes, that allows you to customize your queries according to your needs. +Bazel provides 3 different query modes ([query](/query/quickstart), +[cquery](/query/cquery) and [aquery](/query/aquery)) that allow users +to query the target graph, configured target graph and action graph +respectively. The query language provides a +[suite of functions](/query/language#functions) usable across the different +query modes, that allows you to customize your queries according to your needs. ## JSON Trace Profiles -For every build-like Bazel invocation, Bazel writes a trace profile in JSON format. The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. +For every build-like Bazel invocation, Bazel writes a trace profile in JSON +format. The [JSON trace profile](/advanced/performance/json-trace-profile) can +be very useful to quickly understand what Bazel spent time on during the +invocation. ## Execution Log -The [execution log](/remote/cache-remote) can help you to troubleshoot and fix missing remote cache hits due to machine and environment differences or non-deterministic actions. If you pass the flag [`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) (available from Bazel 5.2) it will also contain detailed spawn metrics, both for locally and remotely executed actions. You can use these metrics for example to make comparisons between local and remote machine performance or to find out which part of the spawn execution is consistently slower than expected (for example due to queuing). +The [execution log](/remote/cache-remote) can help you to troubleshoot and fix +missing remote cache hits due to machine and environment differences or +non-deterministic actions. If you pass the flag +[`--experimental_execution_log_spawn_metrics`](/reference/command-line-reference#flag--experimental_execution_log_spawn_metrics) +(available from Bazel 5.2) it will also contain detailed spawn metrics, both for +locally and remotely executed actions. You can use these metrics for example to +make comparisons between local and remote machine performance or to find out +which part of the spawn execution is consistently slower than expected (for +example due to queuing). ## Execution Graph Log -While the JSON trace profile contains the critical path information, sometimes you need additional information on the dependency graph of the executed actions. Starting with Bazel 6.0, you can pass the flags `--experimental_execution_graph_log` and `--experimental_execution_graph_log_dep_type=all` to write out a log about the executed actions and their inter-dependencies. +While the JSON trace profile contains the critical path information, sometimes +you need additional information on the dependency graph of the executed actions. +Starting with Bazel 6.0, you can pass the flags +`--experimental_execution_graph_log` and +`--experimental_execution_graph_log_dep_type=all` to write out a log about the +executed actions and their inter-dependencies. -This information can be used to understand the drag that is added by a node on the critical path. The drag is the amount of time that can potentially be saved by removing a particular node from the execution graph. +This information can be used to understand the drag that is added by a node on +the critical path. The drag is the amount of time that can potentially be saved +by removing a particular node from the execution graph. -The data helps you predict the impact of changes to the build and action graph before you actually do them. +The data helps you predict the impact of changes to the build and action graph +before you actually do them. ## Benchmarking with bazel-bench -[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a benchmarking tool for Git projects to benchmark build performance in the following cases: +[Bazel bench](https://github.com/bazelbuild/bazel-bench) is a +benchmarking tool for Git projects to benchmark build performance in the +following cases: -- **Project benchmark:** Benchmarking two git commits against each other at a single Bazel version. Used to detect regressions in your build (often through the addition of dependencies). +* **Project benchmark:** Benchmarking two git commits against each other at a + single Bazel version. Used to detect regressions in your build (often through + the addition of dependencies). -- **Bazel benchmark:** Benchmarking two versions of Bazel against each other at a single git commit. Used to detect regressions within Bazel itself (if you happen to maintain / fork Bazel). +* **Bazel benchmark:** Benchmarking two versions of Bazel against each other at + a single git commit. Used to detect regressions within Bazel itself (if you + happen to maintain / fork Bazel). -Benchmarks monitor wall time, CPU time and system time and Bazel’s retained heap size. +Benchmarks monitor wall time, CPU time and system time and Bazel’s retained +heap size. -It is also recommended to run Bazel bench on dedicated, physical machines that are not running other processes so as to reduce sources of variability. +It is also recommended to run Bazel bench on dedicated, physical machines that +are not running other processes so as to reduce sources of variability. diff --git a/advanced/performance/iteration-speed.mdx b/advanced/performance/iteration-speed.mdx index 79240cd0..2bbf8398 100644 --- a/advanced/performance/iteration-speed.mdx +++ b/advanced/performance/iteration-speed.mdx @@ -2,44 +2,92 @@ title: 'Optimize Iteration Speed' --- -This page describes how to optimize Bazel's build performance when running Bazel repeatedly. + + +This page describes how to optimize Bazel's build performance when running Bazel +repeatedly. ## Bazel's Runtime State A Bazel invocation involves several interacting parts. -- The `bazel` command line interface (CLI) is the user-facing front-end tool and receives commands from the user. +* The `bazel` command line interface (CLI) is the user-facing front-end tool + and receives commands from the user. -- The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) for each distinct [output base](https://bazel.build/remote/output-directories). The Bazel server is generally persistent, but will shut down after some idle time so as to not waste resources. +* The CLI tool starts a [*Bazel server*](https://bazel.build/run/client-server) + for each distinct [output base](https://bazel.build/remote/output-directories). + The Bazel server is generally persistent, but will shut down after some idle + time so as to not waste resources. -- The Bazel server performs the loading and analysis steps for a given command (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts of the build graph in memory. The resulting data structures are retained in the Bazel server as part of the *analysis cache*. +* The Bazel server performs the loading and analysis steps for a given command + (`build`, `run`, `cquery`, etc.), in which it constructs the necessary parts + of the build graph in memory. The resulting data structures are retained in + the Bazel server as part of the *analysis cache*. -- The Bazel server can also perform the action execution, or it can send actions off for remote execution if it is set up to do so. The results of action executions are also cached, namely in the *action cache* (or *execution cache*, which may be either local or remote, and it may be shared among Bazel servers). +* The Bazel server can also perform the action execution, or it can send + actions off for remote execution if it is set up to do so. The results of + action executions are also cached, namely in the *action cache* (or + *execution cache*, which may be either local or remote, and it may be shared + among Bazel servers). -- The result of the Bazel invocation is made available in the output tree. +* The result of the Bazel invocation is made available in the output tree. ## Running Bazel Iteratively -In a typical developer workflow, it is common to build (or run) a piece of code repeatedly, often at a very high frequency (e.g. to resolve some compilation error or investigate a failing test). In this situation, it is important that repeated invocations of `bazel` have as little overhead as possible relative to the underlying, repeated action (e.g. invoking a compiler, or executing a test). +In a typical developer workflow, it is common to build (or run) a piece of code +repeatedly, often at a very high frequency (e.g. to resolve some compilation +error or investigate a failing test). In this situation, it is important that +repeated invocations of `bazel` have as little overhead as possible relative to +the underlying, repeated action (e.g. invoking a compiler, or executing a test). With this in mind, we take another look at Bazel's runtime state: -The analysis cache is a critical piece of data. A significant amount of time can be spent just on the loading and analysis phases of a cold run (i.e. a run just after the Bazel server was started or when the analysis cache was discarded). For a single, successful cold build (e.g. for a production release) this cost is bearable, but for repeatedly building the same target it is important that this cost be amortized and not repeated on each invocation. - -The analysis cache is rather volatile. First off, it is part of the in-process state of the Bazel server, so losing the server loses the cache. But the cache is also *invalidated* very easily: for example, many `bazel` command line flags cause the cache to be discarded. This is because many flags affect the build graph (e.g. because of [configurable attributes](https://bazel.build/configure/attributes)). Some flag changes can also cause the Bazel server to be restarted (e.g. changing [startup options](https://bazel.build/docs/user-manual#startup-options)). - -A good execution cache is also valuable for build performance. An execution cache can be kept locally [on disk](https://bazel.build/remote/caching#disk-cache), or [remotely](https://bazel.build/remote/caching). The cache can be shared among Bazel servers, and indeed among developers. +The analysis cache is a critical piece of data. A significant amount of time can +be spent just on the loading and analysis phases of a cold run (i.e. a run just +after the Bazel server was started or when the analysis cache was discarded). +For a single, successful cold build (e.g. for a production release) this cost is +bearable, but for repeatedly building the same target it is important that this +cost be amortized and not repeated on each invocation. + +The analysis cache is rather volatile. First off, it is part of the in-process +state of the Bazel server, so losing the server loses the cache. But the cache +is also *invalidated* very easily: for example, many `bazel` command line flags +cause the cache to be discarded. This is because many flags affect the build +graph (e.g. because of +[configurable attributes](https://bazel.build/configure/attributes)). Some flag +changes can also cause the Bazel server to be restarted (e.g. changing +[startup options](https://bazel.build/docs/user-manual#startup-options)). + +A good execution cache is also valuable for build performance. An execution +cache can be kept locally +[on disk](https://bazel.build/remote/caching#disk-cache), or +[remotely](https://bazel.build/remote/caching). The cache can be shared among +Bazel servers, and indeed among developers. ## Avoid discarding the analysis cache -Bazel will print a warning if either the analysis cache was discarded or the server was restarted. Either of these should be avoided during iterative use: +Bazel will print a warning if either the analysis cache was discarded or the +server was restarted. Either of these should be avoided during iterative use: -- Be mindful of changing `bazel` flags in the middle of an iterative workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` causes each command to discard the analysis cache of the other. In general, try to use a fixed set of flags for the duration of a particular workflow. +* Be mindful of changing `bazel` flags in the middle of an iterative + workflow. For example, mixing a `bazel build -c opt` with a `bazel cquery` + causes each command to discard the analysis cache of the other. In general, + try to use a fixed set of flags for the duration of a particular workflow. -- Losing the Bazel server loses the analysis cache. The Bazel server has a [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle time, after which it shuts down. You can configure this time via your bazelrc file to suit your needs. The server also restarted when startup flags change, so, again, avoid changing those flags if possible. +* Losing the Bazel server loses the analysis cache. The Bazel server has a + [configurable](https://bazel.build/docs/user-manual#max-idle-secs) idle + time, after which it shuts down. You can configure this time via your + bazelrc file to suit your needs. The server also restarted when startup + flags change, so, again, avoid changing those flags if possible. -- [Beware]() that the Bazel server is killed if you press Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time by interrupting a running build that is no longer needed, but only press Ctrl-C once to request a graceful end of the current invocation. +* Beware that the Bazel server is killed if you press + Ctrl-C repeatedly while Bazel is running. It is tempting to try to save time + by interrupting a running build that is no longer needed, but only press + Ctrl-C once to request a graceful end of the current invocation. -- If you want to use multiple sets of flags from the same workspace, you can use multiple, distinct output bases, switched with the `--output_base` flag. Each output base gets its own Bazel server. +* If you want to use multiple sets of flags from the same workspace, you can + use multiple, distinct output bases, switched with the `--output_base` + flag. Each output base gets its own Bazel server. -To make this condition an error rather than a warning, you can use the `--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) +To make this condition an error rather than a warning, you can use the +`--noallow_analysis_cache_discard` flag (introduced in Bazel 6.4.0) diff --git a/advanced/performance/json-trace-profile.mdx b/advanced/performance/json-trace-profile.mdx index c452c4b0..80c698c0 100644 --- a/advanced/performance/json-trace-profile.mdx +++ b/advanced/performance/json-trace-profile.mdx @@ -2,17 +2,34 @@ title: 'JSON Trace Profile' --- -The JSON trace profile can be very useful to quickly understand what Bazel spent time on during the invocation. -By default, for all build-like commands and query, Bazel writes a profile into the output base named `command-$INVOCATION_ID.profile.gz`, where `$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates a symlink called `command.profile.gz` in the output base that points the profile of the latest command. You can configure whether a profile is written with the [`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) flag, and the location it is written to with the [`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are compressed with GZIP. Bazel keeps the last 5 profiles, configurable by [`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), in the output base by default for post-build analysis. Explicitly passing a profile path with `--profile` disables automatic garbage collection. + +The JSON trace profile can be very useful to quickly understand what Bazel spent +time on during the invocation. + +By default, for all build-like commands and query, Bazel writes a profile into +the output base named `command-$INVOCATION_ID.profile.gz`, where +`$INVOCATION_ID` is the invocation identifier of the command. Bazel also creates +a symlink called `command.profile.gz` in the output base that points the profile +of the latest command. You can configure whether a profile is written with the +[`--generate_json_trace_profile`](/reference/command-line-reference#flag--generate_json_trace_profile) +flag, and the location it is written to with the +[`--profile`](/docs/user-manual#profile) flag. Locations ending with `.gz` are +compressed with GZIP. Bazel keeps the last 5 profiles, configurable by +[`--profiles_to_retain`](/reference/command-line-reference#flag--generate_json_trace_profile), +in the output base by default for post-build analysis. Explicitly passing a +profile path with `--profile` disables automatic garbage collection. ## Tools -You can load this profile into `chrome://tracing` or analyze and post-process it with other tools. +You can load this profile into `chrome://tracing` or analyze and +post-process it with other tools. ### `chrome://tracing` -To visualize the profile, open `chrome://tracing` in a Chrome browser tab, click "Load" and pick the (potentially compressed) profile file. For more detailed results, click the boxes in the lower left corner. +To visualize the profile, open `chrome://tracing` in a Chrome browser tab, +click "Load" and pick the (potentially compressed) profile file. For more +detailed results, click the boxes in the lower left corner. Example profile: @@ -22,19 +39,29 @@ Example profile: You can use these keyboard controls to navigate: -- Press `1` for "select" mode. In this mode, you can select particular boxes to inspect the event details (see lower left corner). Select multiple events to get a summary and aggregated statistics. -- Press `2` for "pan" mode. Then drag the mouse to move the view. You can also use `a`/`d` to move left/right. -- Press `3` for "zoom" mode. Then drag the mouse to zoom. You can also use `w`/`s` to zoom in/out. -- Press `4` for "timing" mode where you can measure the distance between two events. -- Press `?` to learn about all controls. +* Press `1` for "select" mode. In this mode, you can select + particular boxes to inspect the event details (see lower left corner). + Select multiple events to get a summary and aggregated statistics. +* Press `2` for "pan" mode. Then drag the mouse to move the view. You + can also use `a`/`d` to move left/right. +* Press `3` for "zoom" mode. Then drag the mouse to zoom. You can + also use `w`/`s` to zoom in/out. +* Press `4` for "timing" mode where you can measure the distance + between two events. +* Press `?` to learn about all controls. ### Bazel Invocation Analyzer -The open-source [Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) consumes a profile format and prints suggestions on how to improve the build’s performance. This analysis can be performed using its CLI or on [https://analyzer.engflow.com](https://analyzer.engflow.com). +The open-source +[Bazel Invocation Analyzer](https://github.com/EngFlow/bazel_invocation_analyzer) +consumes a profile format and prints suggestions on how to improve +the build’s performance. This analysis can be performed using its CLI or on +[https://analyzer.engflow.com](https://analyzer.engflow.com). ### `jq` -`jq` is like `sed` for JSON data. An example usage of `jq` to extract all durations of the sandbox creation step in local action execution: +`jq` is like `sed` for JSON data. An example usage of `jq` to extract all +durations of the sandbox creation step in local action execution: ``` $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | jq '.traceEvents | .[] | select(.name == "sandbox.createFileSystem") | .dur' @@ -51,29 +78,50 @@ $ zcat $(../bazel-6.0.0rc1-linux-x86_64 info output_base)/command.profile.gz | j ## Profile information -The profile contains multiple rows. Usually the bulk of rows represent Bazel threads and their corresponding events, but some special rows are also included. +The profile contains multiple rows. Usually the bulk of rows represent Bazel +threads and their corresponding events, but some special rows are also included. -The special rows included depend on the version of Bazel invoked when the profile was created, and may be customized by different flags. +The special rows included depend on the version of Bazel invoked when the +profile was created, and may be customized by different flags. Figure 1 shows a profile created with Bazel v5.3.1 and includes these rows: -- `action count`: Displays how many concurrent actions were in flight. Click on it to see the actual value. Should go up to the value of [`--jobs`](/reference/command-line-reference#flag--jobs) in clean builds. -- `CPU usage (Bazel)`: For each second of the build, displays the amount of CPU that was used by Bazel (a value of 1 equals one core being 100% busy). -- `Critical Path`: Displays one block for each action on the critical path. -- `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", and "runAnalysisPhase". -- `Garbage Collector`: Displays minor and major Garbage Collection (GC) pauses. +* `action count`: Displays how many concurrent actions were in flight. Click + on it to see the actual value. Should go up to the value of + [`--jobs`](/reference/command-line-reference#flag--jobs) in clean + builds. +* `CPU usage (Bazel)`: For each second of the build, displays the amount of + CPU that was used by Bazel (a value of 1 equals one core being 100% busy). +* `Critical Path`: Displays one block for each action on the critical path. +* `Main Thread`: Bazel’s main thread. Useful to get a high-level picture of + what Bazel is doing, for example "Launch Blaze", "evaluateTargetPatterns", + and "runAnalysisPhase". +* `Garbage Collector`: Displays minor and major Garbage Collection (GC) + pauses. ## Common performance issues When analyzing performance profiles, look for: -- Slower than expected analysis phase (`runAnalysisPhase`), especially on incremental builds. This can be a sign of a poor rule implementation, for example one that flattens depsets. Package loading can be slow by an excessive amount of targets, complex macros or recursive globs. -- Individual slow actions, especially those on the critical path. It might be possible to split large actions into multiple smaller actions or reduce the set of (transitive) dependencies to speed them up. Also check for an unusual high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). -- Bottlenecks, that is a small number of threads is busy while all others are idling / waiting for the result (see around 22s and 29s in Figure 1). Optimizing this will most likely require touching the rule implementations or Bazel itself to introduce more parallelism. This can also happen when there is an unusual amount of GC. +* Slower than expected analysis phase (`runAnalysisPhase`), especially on + incremental builds. This can be a sign of a poor rule implementation, for + example one that flattens depsets. Package loading can be slow by an + excessive amount of targets, complex macros or recursive globs. +* Individual slow actions, especially those on the critical path. It might be + possible to split large actions into multiple smaller actions or reduce the + set of (transitive) dependencies to speed them up. Also check for an unusual + high non-`PROCESS_TIME` (such as `REMOTE_SETUP` or `FETCH`). +* Bottlenecks, that is a small number of threads is busy while all others are + idling / waiting for the result (see around 22s and 29s in Figure 1). + Optimizing this will most likely require touching the rule implementations + or Bazel itself to introduce more parallelism. This can also happen when + there is an unusual amount of GC. ## Profile file format -The top-level object contains metadata (`otherData`) and the actual tracing data (`traceEvents`). The metadata contains extra info, for example the invocation ID and date of the Bazel invocation. +The top-level object contains metadata (`otherData`) and the actual tracing data +(`traceEvents`). The metadata contains extra info, for example the invocation ID +and date of the Bazel invocation. Example: @@ -100,6 +148,12 @@ Example: } ``` -Timestamps (`ts`) and durations (`dur`) in the trace events are given in microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. Note that some events are merged together if they are very short and close to each other; pass [`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) if you would like to prevent event merging. +Timestamps (`ts`) and durations (`dur`) in the trace events are given in +microseconds. The category (`cat`) is one of enum values of `ProfilerTask`. +Note that some events are merged together if they are very short and close to +each other; pass +[`--noslim_profile`](/reference/command-line-reference#flag--slim_profile) +if you would like to prevent event merging. -See also the [Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). +See also the +[Chrome Trace Event Format Specification](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview). diff --git a/advanced/performance/memory.mdx b/advanced/performance/memory.mdx index 8f6c8b5f..844e691b 100644 --- a/advanced/performance/memory.mdx +++ b/advanced/performance/memory.mdx @@ -2,27 +2,50 @@ title: 'Optimize Memory' --- + + This page describes how to limit and reduce the memory Bazel uses. ## Running Bazel with Limited RAM -In certain situations, you may want Bazel to use minimal memory. You can set the maximum heap via the startup flag [`--host_jvm_args`](/docs/user-manual#host-jvm-args), like `--host_jvm_args=-Xmx2g`. +In certain situations, you may want Bazel to use minimal memory. You can set the +maximum heap via the startup flag +[`--host_jvm_args`](/docs/user-manual#host-jvm-args), +like `--host_jvm_args=-Xmx2g`. ### Trade incremental build speeds for memory -If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when it doesn't have enough memory. You can make Bazel use less memory, at the cost of slower incremental builds, by passing the following command flags: [`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), [`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), and [`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). +If your builds are too big, Bazel may throw an `OutOfMemoryError` (OOM) when +it doesn't have enough memory. You can make Bazel use less memory, at the cost +of slower incremental builds, by passing the following command flags: +[`--discard_analysis_cache`](/docs/user-manual#discard-analysis-cache), +[`--nokeep_state_after_build`](/reference/command-line-reference#flag--keep_state_after_build), +and +[`--notrack_incremental_state`](/reference/command-line-reference#flag--track_incremental_state). -These flags will minimize the memory that Bazel uses in a build, at the cost of making future builds slower than a standard incremental build would be. +These flags will minimize the memory that Bazel uses in a build, at the cost of +making future builds slower than a standard incremental build would be. You can also pass any one of these flags individually: -- `--discard_analysis_cache` will reduce the memory used during execution (not analysis). Incremental builds will not have to redo package loading, but will have to redo analysis and execution (although the on-disk action cache can prevent most re-execution). -- `--notrack_incremental_state` will not store any edges in Bazel's internal dependency graph, so that it is unusable for incremental builds. The next build will discard that data, but it is preserved until then, for internal debugging, unless `--nokeep_state_after_build` is specified. -- `--nokeep_state_after_build` will discard all data after the build, so that incremental builds have to build from scratch (except for the on-disk action cache). Alone, it does not affect the high-water mark of the current build. + * `--discard_analysis_cache` will reduce the memory used during execution (not +analysis). Incremental builds will not have to redo package loading, but will +have to redo analysis and execution (although the on-disk action cache can +prevent most re-execution). + * `--notrack_incremental_state` will not store any edges in Bazel's internal + dependency graph, so that it is unusable for incremental builds. The next build + will discard that data, but it is preserved until then, for internal debugging, + unless `--nokeep_state_after_build` is specified. + * `--nokeep_state_after_build` will discard all data after the build, so that + incremental builds have to build from scratch (except for the on-disk action + cache). Alone, it does not affect the high-water mark of the current build. ### Trade build flexibility for memory with Skyfocus (Experimental) -If you want to make Bazel use less memory *and* retain incremental build speeds, you can tell Bazel the working set of files that you will be modifying, and Bazel will only keep state needed to correctly incrementally rebuild changes to those files. This feature is called **Skyfocus**. +If you want to make Bazel use less memory *and* retain incremental build speeds, +you can tell Bazel the working set of files that you will be modifying, and +Bazel will only keep state needed to correctly incrementally rebuild changes to +those files. This feature is called **Skyfocus**. To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: @@ -30,16 +53,21 @@ To use Skyfocus, pass the `--experimental_enable_skyfocus` flag: bazel build //pkg:target --experimental_enable_skyfocus ``` -By default, the working set will be the set of files next to the target being built. In the example, all files in `//pkg` will be kept in the working set, and changes to files outside of the working set will be disallowed, until you issue `bazel clean` or restart the Bazel server. +By default, the working set will be the set of files next to the target being +built. In the example, all files in `//pkg` will be kept in the working set, and +changes to files outside of the working set will be disallowed, until you issue +`bazel clean` or restart the Bazel server. -If you want to specify an exact set of files or directories, use the `--experimental_working_set` flag, like so: +If you want to specify an exact set of files or directories, use the +`--experimental_working_set` flag, like so: ```sh bazel build //pkg:target --experimental_enable_skyfocus --experimental_working_set=path/to/another/dir,path/to/tests/dir ``` -You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the memory reduction amount: +You can also pass `--experimental_skyfocus_dump_post_gc_stats` to show the +memory reduction amount: Putting it altogether, you should see something like this: @@ -51,13 +79,19 @@ INFO: Analyzed 149 targets (4533 packages loaded, 169438 targets configured). INFO: Found 25 targets and 124 test targets... INFO: Updated working set successfully. INFO: Focusing on 334 roots, 3 leafs... (use --experimental_skyfocus_dump_keys to show them) -INFO: Heap: 1237MB -> 676MB (-45.31%) +INFO: Heap: 1237MB -> 676MB (-45.31%) INFO: Elapsed time: 192.670s ... INFO: Build completed successfully, 62303 total actions ``` -For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, and incremental builds to handle changes to files under `dir1`, `dir2`, and `dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot rebuild changed files outside of these directories. +For this example, using Skyfocus allowed Bazel to drop 561MB (45%) of memory, +and incremental builds to handle changes to files under `dir1`, `dir2`, and +`dir3/subdir` will retain their fast speeds, with the tradeoff that Bazel cannot +rebuild changed files outside of these directories. ## Memory Profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. Read more about this process on the [Memory Profiling section](/rules/performance#memory-profiling) of our documentation on how to improve the performance of custom rules. +Bazel comes with a built-in memory profiler that can help you check your rule’s +memory use. Read more about this process on the +[Memory Profiling section](/rules/performance#memory-profiling) of our +documentation on how to improve the performance of custom rules. diff --git a/basics/artifact-based-builds.mdx b/basics/artifact-based-builds.mdx index 5dfc5896..4176a288 100644 --- a/basics/artifact-based-builds.mdx +++ b/basics/artifact-based-builds.mdx @@ -2,19 +2,57 @@ title: 'Artifact-Based Build Systems' --- -This page covers artifact-based build systems and the philosophy behind their creation. Bazel is an artifact-based build system. While task-based build systems are good step above build scripts, they give too much power to individual engineers by letting them define their own tasks. -Artifact-based build systems have a small number of tasks defined by the system that engineers can configure in a limited way. Engineers still tell the system **what** to build, but the build system determines **how** to build it. As with task-based build systems, artifact-based build systems, such as Bazel, still have buildfiles, but the contents of those buildfiles are very different. Rather than being an imperative set of commands in a Turing-complete scripting language describing how to produce an output, buildfiles in Bazel are a declarative manifest describing a set of artifacts to build, their dependencies, and a limited set of options that affect how they’re built. When engineers run `bazel` on the command line, they specify a set of targets to build (the **what**), and Bazel is responsible for configuring, running, and scheduling the compilation steps (the **how**). Because the build system now has full control over what tools to run when, it can make much stronger guarantees that allow it to be far more efficient while still guaranteeing correctness. -## A functional perspective +This page covers artifact-based build systems and the philosophy behind their +creation. Bazel is an artifact-based build system. While task-based build +systems are good step above build scripts, they give too much power to +individual engineers by letting them define their own tasks. + +Artifact-based build systems have a small number of tasks defined by the system +that engineers can configure in a limited way. Engineers still tell the system +**what** to build, but the build system determines **how** to build it. As with +task-based build systems, artifact-based build systems, such as Bazel, still +have buildfiles, but the contents of those buildfiles are very different. Rather +than being an imperative set of commands in a Turing-complete scripting language +describing how to produce an output, buildfiles in Bazel are a declarative +manifest describing a set of artifacts to build, their dependencies, and a +limited set of options that affect how they’re built. When engineers run `bazel` +on the command line, they specify a set of targets to build (the **what**), and +Bazel is responsible for configuring, running, and scheduling the compilation +steps (the **how**). Because the build system now has full control over what +tools to run when, it can make much stronger guarantees that allow it to be far +more efficient while still guaranteeing correctness. -It’s easy to make an analogy between artifact-based build systems and functional programming. Traditional imperative programming languages (such as, Java, C, and Python) specify lists of statements to be executed one after another, in the same way that task-based build systems let programmers define a series of steps to execute. Functional programming languages (such as, Haskell and ML), in contrast, are structured more like a series of mathematical equations. In functional languages, the programmer describes a computation to perform, but leaves the details of when and exactly how that computation is executed to the compiler. +## A functional perspective -This maps to the idea of declaring a manifest in an artifact-based build system and letting the system figure out how to execute the build. Many problems can't be easily expressed using functional programming, but the ones that do benefit greatly from it: the language is often able to trivially parallelize such programs and make strong guarantees about their correctness that would be impossible in an imperative language. The easiest problems to express using functional programming are the ones that simply involve transforming one piece of data into another using a series of rules or functions. And that’s exactly what a build system is: the whole system is effectively a mathematical function that takes source files (and tools like the compiler) as inputs and produces binaries as outputs. So, it’s not surprising that it works well to base a build system around the tenets of functional programming. +It’s easy to make an analogy between artifact-based build systems and functional +programming. Traditional imperative programming languages (such as, Java, C, and +Python) specify lists of statements to be executed one after another, in the +same way that task-based build systems let programmers define a series of steps +to execute. Functional programming languages (such as, Haskell and ML), in +contrast, are structured more like a series of mathematical equations. In +functional languages, the programmer describes a computation to perform, but +leaves the details of when and exactly how that computation is executed to the +compiler. + +This maps to the idea of declaring a manifest in an artifact-based build system +and letting the system figure out how to execute the build. Many problems can't +be easily expressed using functional programming, but the ones that do benefit +greatly from it: the language is often able to trivially parallelize such +programs and make strong guarantees about their correctness that would be +impossible in an imperative language. The easiest problems to express using +functional programming are the ones that simply involve transforming one piece +of data into another using a series of rules or functions. And that’s exactly +what a build system is: the whole system is effectively a mathematical function +that takes source files (and tools like the compiler) as inputs and produces +binaries as outputs. So, it’s not surprising that it works well to base a build +system around the tenets of functional programming. ## Understanding artifact-based build systems -Google's build system, Blaze, was the first artifact-based build system. Bazel is the open-sourced version of Blaze. +Google's build system, Blaze, was the first artifact-based build system. Bazel +is the open-sourced version of Blaze. Here’s what a buildfile (normally named `BUILD`) looks like in Bazel: @@ -37,64 +75,205 @@ java_library( ) ``` -In Bazel, `BUILD` files define targets—the two types of targets here are `java_binary` and `java_library`. Every target corresponds to an artifact that can be created by the system: binary targets produce binaries that can be executed directly, and library targets produce libraries that can be used by binaries or other libraries. Every target has: - -- `name`: how the target is referenced on the command line and by other targets -- `srcs`: the source files to be compiled to create the artifact for the target -- `deps`: other targets that must be built before this target and linked into it - -Dependencies can either be within the same package (such as `MyBinary`’s dependency on `:mylib`) or on a different package in the same source hierarchy (such as `mylib`’s dependency on `//java/com/example/common`). - -As with task-based build systems, you perform builds using Bazel’s command-line tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After entering that command for the first time in a clean repository, Bazel: - -1. Parses every `BUILD` file in the workspace to create a graph of dependencies among artifacts. -2. Uses the graph to determine the transitive dependencies of `MyBinary`; that is, every target that `MyBinary` depends on and every target that those targets depend on, recursively. -3. Builds each of those dependencies, in order. Bazel starts by building each target that has no other dependencies and keeps track of which dependencies still need to be built for each target. As soon as all of a target’s dependencies are built, Bazel starts building that target. This process continues until every one of `MyBinary`’s transitive dependencies have been built. -4. Builds `MyBinary` to produce a final executable binary that links in all of the dependencies that were built in step 3. - -Fundamentally, it might not seem like what’s happening here is that much different than what happened when using a task-based build system. Indeed, the end result is the same binary, and the process for producing it involved analyzing a bunch of steps to find dependencies among them, and then running those steps in order. But there are critical differences. The first one appears in step 3: because Bazel knows that each target only produces a Java library, it knows that all it has to do is run the Java compiler rather than an arbitrary user-defined script, so it knows that it’s safe to run these steps in parallel. This can produce an order of magnitude performance improvement over building targets one at a time on a multicore machine, and is only possible because the artifact-based approach leaves the build system in charge of its own execution strategy so that it can make stronger guarantees about parallelism. - -The benefits extend beyond parallelism, though. The next thing that this approach gives us becomes apparent when the developer types `bazel build :MyBinary` a second time without making any changes: Bazel exits in less than a second with a message saying that the target is up to date. This is possible due to the functional programming paradigm we talked about earlier—Bazel knows that each target is the result only of running a Java compiler, and it knows that the output from the Java compiler depends only on its inputs, so as long as the inputs haven’t changed, the output can be reused. And this analysis works at every level; if `MyBinary.java` changes, Bazel knows to rebuild `MyBinary` but reuse `mylib`. If a source file for `//java/com/example/common` changes, Bazel knows to rebuild that library, `mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. Because Bazel knows about the properties of the tools it runs at every step, it’s able to rebuild only the minimum set of artifacts each time while guaranteeing that it won’t produce stale builds. - -Reframing the build process in terms of artifacts rather than tasks is subtle but powerful. By reducing the flexibility exposed to the programmer, the build system can know more about what is being done at every step of the build. It can use this knowledge to make the build far more efficient by parallelizing build processes and reusing their outputs. But this is really just the first step, and these building blocks of parallelism and reuse form the basis for a distributed and highly scalable build system. +In Bazel, `BUILD` files define targets—the two types of targets here are +`java_binary` and `java_library`. Every target corresponds to an artifact that +can be created by the system: binary targets produce binaries that can be +executed directly, and library targets produce libraries that can be used by +binaries or other libraries. Every target has: + +* `name`: how the target is referenced on the command line and by other + targets +* `srcs`: the source files to be compiled to create the artifact for the target +* `deps`: other targets that must be built before this target and linked into + it + +Dependencies can either be within the same package (such as `MyBinary`’s +dependency on `:mylib`) or on a different package in the same source hierarchy +(such as `mylib`’s dependency on `//java/com/example/common`). + +As with task-based build systems, you perform builds using Bazel’s command-line +tool. To build the `MyBinary` target, you run `bazel build :MyBinary`. After +entering that command for the first time in a clean repository, Bazel: + +1. Parses every `BUILD` file in the workspace to create a graph of dependencies + among artifacts. +1. Uses the graph to determine the transitive dependencies of `MyBinary`; that + is, every target that `MyBinary` depends on and every target that those + targets depend on, recursively. +1. Builds each of those dependencies, in order. Bazel starts by building each + target that has no other dependencies and keeps track of which dependencies + still need to be built for each target. As soon as all of a target’s + dependencies are built, Bazel starts building that target. This process + continues until every one of `MyBinary`’s transitive dependencies have been + built. +1. Builds `MyBinary` to produce a final executable binary that links in all of + the dependencies that were built in step 3. + +Fundamentally, it might not seem like what’s happening here is that much +different than what happened when using a task-based build system. Indeed, the +end result is the same binary, and the process for producing it involved +analyzing a bunch of steps to find dependencies among them, and then running +those steps in order. But there are critical differences. The first one appears +in step 3: because Bazel knows that each target only produces a Java library, it +knows that all it has to do is run the Java compiler rather than an arbitrary +user-defined script, so it knows that it’s safe to run these steps in parallel. +This can produce an order of magnitude performance improvement over building +targets one at a time on a multicore machine, and is only possible because the +artifact-based approach leaves the build system in charge of its own execution +strategy so that it can make stronger guarantees about parallelism. + +The benefits extend beyond parallelism, though. The next thing that this +approach gives us becomes apparent when the developer types `bazel +build :MyBinary` a second time without making any changes: Bazel exits in less +than a second with a message saying that the target is up to date. This is +possible due to the functional programming paradigm we talked about +earlier—Bazel knows that each target is the result only of running a Java +compiler, and it knows that the output from the Java compiler depends only on +its inputs, so as long as the inputs haven’t changed, the output can be reused. +And this analysis works at every level; if `MyBinary.java` changes, Bazel knows +to rebuild `MyBinary` but reuse `mylib`. If a source file for +`//java/com/example/common` changes, Bazel knows to rebuild that library, +`mylib`, and `MyBinary`, but reuse `//java/com/example/myproduct/otherlib`. +Because Bazel knows about the properties of the tools it runs at every step, +it’s able to rebuild only the minimum set of artifacts each time while +guaranteeing that it won’t produce stale builds. + +Reframing the build process in terms of artifacts rather than tasks is subtle +but powerful. By reducing the flexibility exposed to the programmer, the build +system can know more about what is being done at every step of the build. It can +use this knowledge to make the build far more efficient by parallelizing build +processes and reusing their outputs. But this is really just the first step, and +these building blocks of parallelism and reuse form the basis for a distributed +and highly scalable build system. ## Other nifty Bazel tricks -Artifact-based build systems fundamentally solve the problems with parallelism and reuse that are inherent in task-based build systems. But there are still a few problems that came up earlier that we haven’t addressed. Bazel has clever ways of solving each of these, and we should discuss them before moving on. +Artifact-based build systems fundamentally solve the problems with parallelism +and reuse that are inherent in task-based build systems. But there are still a +few problems that came up earlier that we haven’t addressed. Bazel has clever +ways of solving each of these, and we should discuss them before moving on. ### Tools as dependencies -One problem we ran into earlier was that builds depended on the tools installed on our machine, and reproducing builds across systems could be difficult due to different tool versions or locations. The problem becomes even more difficult when your project uses languages that require different tools based on which platform they’re being built on or compiled for (such as, Windows versus Linux), and each of those platforms requires a slightly different set of tools to do the same job. +One problem we ran into earlier was that builds depended on the tools installed +on our machine, and reproducing builds across systems could be difficult due to +different tool versions or locations. The problem becomes even more difficult +when your project uses languages that require different tools based on which +platform they’re being built on or compiled for (such as, Windows versus Linux), +and each of those platforms requires a slightly different set of tools to do the +same job. -Bazel solves the first part of this problem by treating tools as dependencies to each target. Every `java_library` in the workspace implicitly depends on a Java compiler, which defaults to a well-known compiler. Whenever Bazel builds a `java_library`, it checks to make sure that the specified compiler is available at a known location. Just like any other dependency, if the Java compiler changes, every artifact that depends on it is rebuilt. +Bazel solves the first part of this problem by treating tools as dependencies to +each target. Every `java_library` in the workspace implicitly depends on a Java +compiler, which defaults to a well-known compiler. Whenever Bazel builds a +`java_library`, it checks to make sure that the specified compiler is available +at a known location. Just like any other dependency, if the Java compiler +changes, every artifact that depends on it is rebuilt. -Bazel solves the second part of the problem, platform independence, by setting [build configurations](/run/build#build-config-cross-compilation). Rather than targets depending directly on their tools, they depend on types of configurations: +Bazel solves the second part of the problem, platform independence, by setting +[build configurations](/run/build#build-config-cross-compilation). Rather than +targets depending directly on their tools, they depend on types of configurations: -- **Host configuration**: building tools that run during the build -- **Target configuration**: building the binary you ultimately requested +* **Host configuration**: building tools that run during the build +* **Target configuration**: building the binary you ultimately requested ### Extending the build system -Bazel comes with targets for several popular programming languages out of the box, but engineers will always want to do more—part of the benefit of task-based systems is their flexibility in supporting any kind of build process, and it would be better not to give that up in an artifact-based build system. Fortunately, Bazel allows its supported target types to be extended by [adding custom rules](/extending/rules). - -To define a rule in Bazel, the rule author declares the inputs that the rule requires (in the form of attributes passed in the `BUILD` file) and the fixed set of outputs that the rule produces. The author also defines the actions that will be generated by that rule. Each action declares its inputs and outputs, runs a particular executable or writes a particular string to a file, and can be connected to other actions via its inputs and outputs. This means that actions are the lowest-level composable unit in the build system—an action can do whatever it wants so long as it uses only its declared inputs and outputs, and Bazel takes care of scheduling actions and caching their results as appropriate. - -The system isn’t foolproof given that there’s no way to stop an action developer from doing something like introducing a nondeterministic process as part of their action. But this doesn’t happen very often in practice, and pushing the possibilities for abuse all the way down to the action level greatly decreases opportunities for errors. Rules supporting many common languages and tools are widely available online, and most projects will never need to define their own rules. Even for those that do, rule definitions only need to be defined in one central place in the repository, meaning most engineers will be able to use those rules without ever having to worry about their implementation. +Bazel comes with targets for several popular programming languages out of the +box, but engineers will always want to do more—part of the benefit of task-based +systems is their flexibility in supporting any kind of build process, and it +would be better not to give that up in an artifact-based build system. +Fortunately, Bazel allows its supported target types to be extended by +[adding custom rules](/extending/rules). + +To define a rule in Bazel, the rule author declares the inputs that the rule +requires (in the form of attributes passed in the `BUILD` file) and the fixed +set of outputs that the rule produces. The author also defines the actions that +will be generated by that rule. Each action declares its inputs and outputs, +runs a particular executable or writes a particular string to a file, and can be +connected to other actions via its inputs and outputs. This means that actions +are the lowest-level composable unit in the build system—an action can do +whatever it wants so long as it uses only its declared inputs and outputs, and +Bazel takes care of scheduling actions and caching their results as appropriate. + +The system isn’t foolproof given that there’s no way to stop an action developer +from doing something like introducing a nondeterministic process as part of +their action. But this doesn’t happen very often in practice, and pushing the +possibilities for abuse all the way down to the action level greatly decreases +opportunities for errors. Rules supporting many common languages and tools are +widely available online, and most projects will never need to define their own +rules. Even for those that do, rule definitions only need to be defined in one +central place in the repository, meaning most engineers will be able to use +those rules without ever having to worry about their implementation. ### Isolating the environment -Actions sound like they might run into the same problems as tasks in other systems—isn’t it still possible to write actions that both write to the same file and end up conflicting with one another? Actually, Bazel makes these conflicts impossible by using *[sandboxing](/docs/sandboxing)*. On supported systems, every action is isolated from every other action via a filesystem sandbox. Effectively, each action can see only a restricted view of the filesystem that includes the inputs it has declared and any outputs it has produced. This is enforced by systems such as LXC on Linux, the same technology behind Docker. This means that it’s impossible for actions to conflict with one another because they are unable to read any files they don’t declare, and any files that they write but don’t declare will be thrown away when the action finishes. Bazel also uses sandboxes to restrict actions from communicating via the network. +Actions sound like they might run into the same problems as tasks in other +systems—isn’t it still possible to write actions that both write to the same +file and end up conflicting with one another? Actually, Bazel makes these +conflicts impossible by using _[sandboxing](/docs/sandboxing)_. On supported +systems, every action is isolated from every other action via a filesystem +sandbox. Effectively, each action can see only a restricted view of the +filesystem that includes the inputs it has declared and any outputs it has +produced. This is enforced by systems such as LXC on Linux, the same technology +behind Docker. This means that it’s impossible for actions to conflict with one +another because they are unable to read any files they don’t declare, and any +files that they write but don’t declare will be thrown away when the action +finishes. Bazel also uses sandboxes to restrict actions from communicating via +the network. ### Making external dependencies deterministic -There’s still one problem remaining: build systems often need to download dependencies (whether tools or libraries) from external sources rather than directly building them. This can be seen in the example via the `@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file from Maven. - -Depending on files outside of the current workspace is risky. Those files could change at any time, potentially requiring the build system to constantly check whether they’re fresh. If a remote file changes without a corresponding change in the workspace source code, it can also lead to unreproducible builds—a build might work one day and fail the next for no obvious reason due to an unnoticed dependency change. Finally, an external dependency can introduce a huge security risk when it is owned by a third party: if an attacker is able to infiltrate that third-party server, they can replace the dependency file with something of their own design, potentially giving them full control over your build environment and its output. - -The fundamental problem is that we want the build system to be aware of these files without having to check them into source control. Updating a dependency should be a conscious choice, but that choice should be made once in a central place rather than managed by individual engineers or automatically by the system. This is because even with a “Live at Head” model, we still want builds to be deterministic, which implies that if you check out a commit from last week, you should see your dependencies as they were then rather than as they are now. - -Bazel and some other build systems address this problem by requiring a workspacewide manifest file that lists a *cryptographic hash* for every external dependency in the workspace. The hash is a concise way to uniquely represent the file without checking the entire file into source control. Whenever a new external dependency is referenced from a workspace, that dependency’s hash is added to the manifest, either manually or automatically. When Bazel runs a build, it checks the actual hash of its cached dependency against the expected hash defined in the manifest and redownloads the file only if the hash differs. - -If the artifact we download has a different hash than the one declared in the manifest, the build will fail unless the hash in the manifest is updated. This can be done automatically, but that change must be approved and checked into source control before the build will accept the new dependency. This means that there’s always a record of when a dependency was updated, and an external dependency can’t change without a corresponding change in the workspace source. It also means that, when checking out an older version of the source code, the build is guaranteed to use the same dependencies that it was using at the point when that version was checked in (or else it will fail if those dependencies are no longer available). - -Of course, it can still be a problem if a remote server becomes unavailable or starts serving corrupt data—this can cause all of your builds to begin failing if you don’t have another copy of that dependency available. To avoid this problem, we recommend that, for any nontrivial project, you mirror all of its dependencies onto servers or services that you trust and control. Otherwise you will always be at the mercy of a third party for your build system’s availability, even if the checked-in hashes guarantee its security. +There’s still one problem remaining: build systems often need to download +dependencies (whether tools or libraries) from external sources rather than +directly building them. This can be seen in the example via the +`@com_google_common_guava_guava//jar` dependency, which downloads a `JAR` file +from Maven. + +Depending on files outside of the current workspace is risky. Those files could +change at any time, potentially requiring the build system to constantly check +whether they’re fresh. If a remote file changes without a corresponding change +in the workspace source code, it can also lead to unreproducible builds—a build +might work one day and fail the next for no obvious reason due to an unnoticed +dependency change. Finally, an external dependency can introduce a huge security +risk when it is owned by a third party: if an attacker is able to infiltrate +that third-party server, they can replace the dependency file with something of +their own design, potentially giving them full control over your build +environment and its output. + +The fundamental problem is that we want the build system to be aware of these +files without having to check them into source control. Updating a dependency +should be a conscious choice, but that choice should be made once in a central +place rather than managed by individual engineers or automatically by the +system. This is because even with a “Live at Head” model, we still want builds +to be deterministic, which implies that if you check out a commit from last +week, you should see your dependencies as they were then rather than as they are +now. + +Bazel and some other build systems address this problem by requiring a +workspacewide manifest file that lists a _cryptographic hash_ for every external +dependency in the workspace. The hash is a concise way to uniquely represent the +file without checking the entire file into source control. Whenever a new +external dependency is referenced from a workspace, that dependency’s hash is +added to the manifest, either manually or automatically. When Bazel runs a +build, it checks the actual hash of its cached dependency against the expected +hash defined in the manifest and redownloads the file only if the hash differs. + +If the artifact we download has a different hash than the one declared in the +manifest, the build will fail unless the hash in the manifest is updated. This +can be done automatically, but that change must be approved and checked into +source control before the build will accept the new dependency. This means that +there’s always a record of when a dependency was updated, and an external +dependency can’t change without a corresponding change in the workspace source. +It also means that, when checking out an older version of the source code, the +build is guaranteed to use the same dependencies that it was using at the point +when that version was checked in (or else it will fail if those dependencies are +no longer available). + +Of course, it can still be a problem if a remote server becomes unavailable or +starts serving corrupt data—this can cause all of your builds to begin failing +if you don’t have another copy of that dependency available. To avoid this +problem, we recommend that, for any nontrivial project, you mirror all of its +dependencies onto servers or services that you trust and control. Otherwise you +will always be at the mercy of a third party for your build system’s +availability, even if the checked-in hashes guarantee its security. diff --git a/basics/build-systems.mdx b/basics/build-systems.mdx index e2f80cb6..b3c63389 100644 --- a/basics/build-systems.mdx +++ b/basics/build-systems.mdx @@ -2,44 +2,126 @@ title: 'Why a Build System?' --- -This page discusses what build systems are, what they do, why you should use a build system, and why compilers and build scripts aren't the best choice as your organization starts to scale. It's intended for developers who don't have much experience with a build system. + + +This page discusses what build systems are, what they do, why you should use a +build system, and why compilers and build scripts aren't the best choice as your +organization starts to scale. It's intended for developers who don't have much +experience with a build system. ## What is a build system? -Fundamentally, all build systems have a straightforward purpose: they transform the source code written by engineers into executable binaries that can be read by machines. Build systems aren't just for human-authored code; they also allow machines to create builds automatically, whether for testing or for releases to production. In an organization with thousands of engineers, it's common that most builds are triggered automatically rather than directly by engineers. +Fundamentally, all build systems have a straightforward purpose: they transform +the source code written by engineers into executable binaries that can be read +by machines. Build systems aren't just for human-authored code; they also allow +machines to create builds automatically, whether for testing or for releases to +production. In an organization with thousands of engineers, it's common that +most builds are triggered automatically rather than directly by engineers. ### Can't I just use a compiler? -The need for a build system might not be immediately obvious. Most engineers don't use a build system while learning to code: most start by invoking tools like `gcc` or `javac` directly from the command line, or the equivalent in an integrated development environment (IDE). As long as all the source code is in the same directory, a command like this works fine: +The need for a build system might not be immediately obvious. Most engineers +don't use a build system while learning to code: most start by invoking tools +like `gcc` or `javac` directly from the command line, or the equivalent in an +integrated development environment (IDE). As long as all the source code is in +the same directory, a command like this works fine: ```posix-terminal javac *.java ``` -This instructs the Java compiler to take every Java source file in the current directory and turn it into a binary class file. In the simplest case, this is all you need. - -However, as soon as code expands, the complications begin. `javac` is smart enough to look in subdirectories of the current directory to find code to import. But it has no way of finding code stored in *other parts* of the filesystem (perhaps a library shared by several projects). It also only knows how to build Java code. Large systems often involve different pieces written in a variety of programming languages with webs of dependencies among those pieces, meaning no compiler for a single language can possibly build the entire system. - -Once you're dealing with code from multiple languages or multiple compilation units, building code is no longer a one-step process. Now you must evaluate what your code depends on and build those pieces in the proper order, possibly using a different set of tools for each piece. If any dependencies change, you must repeat this process to avoid depending on stale binaries. For a codebase of even moderate size, this process quickly becomes tedious and error-prone. - -The compiler also doesn’t know anything about how to handle external dependencies, such as third-party `JAR` files in Java. Without a build system, you could manage this by downloading the dependency from the internet, sticking it in a `lib` folder on the hard drive, and configuring the compiler to read libraries from that directory. Over time, it's difficult to maintain the updates, versions, and source of these external dependencies. +This instructs the Java compiler to take every Java source file in the current +directory and turn it into a binary class file. In the simplest case, this is +all you need. + +However, as soon as code expands, the complications begin. `javac` is smart +enough to look in subdirectories of the current directory to find code to +import. But it has no way of finding code stored in _other parts_ of the +filesystem (perhaps a library shared by several projects). It also only knows +how to build Java code. Large systems often involve different pieces written in +a variety of programming languages with webs of dependencies among those pieces, +meaning no compiler for a single language can possibly build the entire system. + +Once you're dealing with code from multiple languages or multiple compilation +units, building code is no longer a one-step process. Now you must evaluate what +your code depends on and build those pieces in the proper order, possibly using +a different set of tools for each piece. If any dependencies change, you must +repeat this process to avoid depending on stale binaries. For a codebase of even +moderate size, this process quickly becomes tedious and error-prone. + +The compiler also doesn’t know anything about how to handle external +dependencies, such as third-party `JAR` files in Java. Without a build system, +you could manage this by downloading the dependency from the internet, sticking +it in a `lib` folder on the hard drive, and configuring the compiler to read +libraries from that directory. Over time, it's difficult to maintain the +updates, versions, and source of these external dependencies. ### What about shell scripts? -Suppose that your hobby project starts out simple enough that you can build it using just a compiler, but you begin running into some of the problems described previously. Maybe you still don’t think you need a build system and can automate away the tedious parts using some simple shell scripts that take care of building things in the correct order. This helps out for a while, but pretty soon you start running into even more problems: - -- It becomes tedious. As your system grows more complex, you begin spending almost as much time working on your build scripts as on real code. Debugging shell scripts is painful, with more and more hacks being layered on top of one another. - -- It’s slow. To make sure you weren’t accidentally relying on stale libraries, you have your build script build every dependency in order every time you run it. You think about adding some logic to detect which parts need to be rebuilt, but that sounds awfully complex and error prone for a script. Or you think about specifying which parts need to be rebuilt each time, but then you’re back to square one. - -- Good news: it’s time for a release! Better go figure out all the arguments you need to pass to the jar command to make your final build. And remember how to upload it and push it out to the central repository. And build and push the documentation updates, and send out a notification to users. Hmm, maybe this calls for another script... - -- Disaster! Your hard drive crashes, and now you need to recreate your entire system. You were smart enough to keep all of your source files in version control, but what about those libraries you downloaded? Can you find them all again and make sure they were the same version as when you first downloaded them? Your scripts probably depended on particular tools being installed in particular places—can you restore that same environment so that the scripts work again? What about all those environment variables you set a long time ago to get the compiler working just right and then forgot about? - -- Despite the problems, your project is successful enough that you’re able to begin hiring more engineers. Now you realize that it doesn’t take a disaster for the previous problems to arise—you need to go through the same painful bootstrapping process every time a new developer joins your team. And despite your best efforts, there are still small differences in each person’s system. Frequently, what works on one person’s machine doesn’t work on another’s, and each time it takes a few hours of debugging tool paths or library versions to figure out where the difference is. - -- You decide that you need to automate your build system. In theory, this is as simple as getting a new computer and setting it up to run your build script every night using cron. You still need to go through the painful setup process, but now you don’t have the benefit of a human brain being able to detect and resolve minor problems. Now, every morning when you get in, you see that last night’s build failed because yesterday a developer made a change that worked on their system but didn’t work on the automated build system. Each time it’s a simple fix, but it happens so often that you end up spending a lot of time each day discovering and applying these simple fixes. - -- Builds become slower and slower as the project grows. One day, while waiting for a build to complete, you gaze mournfully at the idle desktop of your coworker, who is on vacation, and wish there were a way to take advantage of all that wasted computational power. - -You’ve run into a classic problem of scale. For a single developer working on at most a couple hundred lines of code for at most a week or two (which might have been the entire experience thus far of a junior developer who just graduated university), a compiler is all you need. Scripts can maybe take you a little bit farther. But as soon as you need to coordinate across multiple developers and their machines, even a perfect build script isn’t enough because it becomes very difficult to account for the minor differences in those machines. At this point, this simple approach breaks down and it’s time to invest in a real build system. +Suppose that your hobby project starts out simple enough that you can build it +using just a compiler, but you begin running into some of the problems described +previously. Maybe you still don’t think you need a build system and can automate +away the tedious parts using some simple shell scripts that take care of +building things in the correct order. This helps out for a while, but pretty +soon you start running into even more problems: + +* It becomes tedious. As your system grows more complex, you begin spending + almost as much time working on your build scripts as on real code. Debugging + shell scripts is painful, with more and more hacks being layered on top of + one another. + +* It’s slow. To make sure you weren’t accidentally relying on stale libraries, + you have your build script build every dependency in order every time you + run it. You think about adding some logic to detect which parts need to be + rebuilt, but that sounds awfully complex and error prone for a script. Or + you think about specifying which parts need to be rebuilt each time, but + then you’re back to square one. + +* Good news: it’s time for a release! Better go figure out all the arguments + you need to pass to the jar command to make your final build. And remember + how to upload it and push it out to the central repository. And build and + push the documentation updates, and send out a notification to users. Hmm, + maybe this calls for another script... + +* Disaster! Your hard drive crashes, and now you need to recreate your entire + system. You were smart enough to keep all of your source files in version + control, but what about those libraries you downloaded? Can you find them + all again and make sure they were the same version as when you first + downloaded them? Your scripts probably depended on particular tools being + installed in particular places—can you restore that same environment so that + the scripts work again? What about all those environment variables you set a + long time ago to get the compiler working just right and then forgot about? + +* Despite the problems, your project is successful enough that you’re able to + begin hiring more engineers. Now you realize that it doesn’t take a disaster + for the previous problems to arise—you need to go through the same painful + bootstrapping process every time a new developer joins your team. And + despite your best efforts, there are still small differences in each + person’s system. Frequently, what works on one person’s machine doesn’t work + on another’s, and each time it takes a few hours of debugging tool paths or + library versions to figure out where the difference is. + +* You decide that you need to automate your build system. In theory, this is + as simple as getting a new computer and setting it up to run your build + script every night using cron. You still need to go through the painful + setup process, but now you don’t have the benefit of a human brain being + able to detect and resolve minor problems. Now, every morning when you get + in, you see that last night’s build failed because yesterday a developer + made a change that worked on their system but didn’t work on the automated + build system. Each time it’s a simple fix, but it happens so often that you + end up spending a lot of time each day discovering and applying these simple + fixes. + +* Builds become slower and slower as the project grows. One day, while waiting + for a build to complete, you gaze mournfully at the idle desktop of your + coworker, who is on vacation, and wish there were a way to take advantage of + all that wasted computational power. + +You’ve run into a classic problem of scale. For a single developer working on at +most a couple hundred lines of code for at most a week or two (which might have +been the entire experience thus far of a junior developer who just graduated +university), a compiler is all you need. Scripts can maybe take you a little bit +farther. But as soon as you need to coordinate across multiple developers and +their machines, even a perfect build script isn’t enough because it becomes very +difficult to account for the minor differences in those machines. At this point, +this simple approach breaks down and it’s time to invest in a real build system. diff --git a/basics/dependencies.mdx b/basics/dependencies.mdx index 308b627e..1d3bf8f1 100644 --- a/basics/dependencies.mdx +++ b/basics/dependencies.mdx @@ -2,84 +2,300 @@ title: 'Dependency Management' --- -In looking through the previous pages, one theme repeats over and over: managing your own code is fairly straightforward, but managing its dependencies is much more difficult. There are all sorts of dependencies: sometimes there’s a dependency on a task (such as “push the documentation before I mark a release as complete”), and sometimes there’s a dependency on an artifact (such as “I need to have the latest version of the computer vision library to build my code”). Sometimes, you have internal dependencies on another part of your codebase, and sometimes you have external dependencies on code or data owned by another team (either in your organization or a third party). But in any case, the idea of “I need that before I can have this” is something that recurs repeatedly in the design of build systems, and managing dependencies is perhaps the most fundamental job of a build system. -## Dealing with Modules and Dependencies - -Projects that use artifact-based build systems like Bazel are broken into a set of modules, with modules expressing dependencies on one another via `BUILD` files. Proper organization of these modules and dependencies can have a huge effect on both the performance of the build system and how much work it takes to maintain. -## Using Fine-Grained Modules and the 1:1:1 Rule +In looking through the previous pages, one theme repeats over and over: managing +your own code is fairly straightforward, but managing its dependencies is much +more difficult. There are all sorts of dependencies: sometimes there’s a +dependency on a task (such as “push the documentation before I mark a release as +complete”), and sometimes there’s a dependency on an artifact (such as “I need +to have the latest version of the computer vision library to build my code”). +Sometimes, you have internal dependencies on another part of your codebase, and +sometimes you have external dependencies on code or data owned by another team +(either in your organization or a third party). But in any case, the idea of “I +need that before I can have this” is something that recurs repeatedly in the +design of build systems, and managing dependencies is perhaps the most +fundamental job of a build system. -The first question that comes up when structuring an artifact-based build is deciding how much functionality an individual module should encompass. In Bazel, a *module* is represented by a target specifying a buildable unit like a `java_library` or a `go_binary`. At one extreme, the entire project could be contained in a single module by putting one `BUILD` file at the root and recursively globbing together all of that project’s source files. At the other extreme, nearly every source file could be made into its own module, effectively requiring each file to list in a `BUILD` file every other file it depends on. - -Most projects fall somewhere between these extremes, and the choice involves a trade-off between performance and maintainability. Using a single module for the entire project might mean that you never need to touch the `BUILD` file except when adding an external dependency, but it means that the build system must always build the entire project all at once. This means that it won’t be able to parallelize or distribute parts of the build, nor will it be able to cache parts that it’s already built. One-module-per-file is the opposite: the build system has the maximum flexibility in caching and scheduling steps of the build, but engineers need to expend more effort maintaining lists of dependencies whenever they change which files reference which. +## Dealing with Modules and Dependencies -Though the exact granularity varies by language (and often even within language), Google tends to favor significantly smaller modules than one might typically write in a task-based build system. A typical production binary at Google often depends on tens of thousands of targets, and even a moderate-sized team can own several hundred targets within its codebase. For languages like Java that have a strong built-in notion of packaging, each directory usually contains a single package, target, and `BUILD` file (Pants, another build system based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging conventions frequently define multiple targets per `BUILD` file. +Projects that use artifact-based build systems like Bazel are broken into a set +of modules, with modules expressing dependencies on one another via `BUILD` +files. Proper organization of these modules and dependencies can have a huge +effect on both the performance of the build system and how much work it takes to +maintain. -The benefits of smaller build targets really begin to show at scale because they lead to faster distributed builds and a less frequent need to rebuild targets. The advantages become even more compelling after testing enters the picture, as finer-grained targets mean that the build system can be much smarter about running only a limited subset of tests that could be affected by any given change. Because Google believes in the systemic benefits of using smaller targets, we’ve made some strides in mitigating the downside by investing in tooling to automatically manage `BUILD` files to avoid burdening developers. +## Using Fine-Grained Modules and the 1:1:1 Rule -Some of these tools, such as `buildifier` and `buildozer`, are available with Bazel in the [`buildtools` directory](https://github.com/bazelbuild/buildtools). +The first question that comes up when structuring an artifact-based build is +deciding how much functionality an individual module should encompass. In Bazel, +a _module_ is represented by a target specifying a buildable unit like a +`java_library` or a `go_binary`. At one extreme, the entire project could be +contained in a single module by putting one `BUILD` file at the root and +recursively globbing together all of that project’s source files. At the other +extreme, nearly every source file could be made into its own module, effectively +requiring each file to list in a `BUILD` file every other file it depends on. + +Most projects fall somewhere between these extremes, and the choice involves a +trade-off between performance and maintainability. Using a single module for the +entire project might mean that you never need to touch the `BUILD` file except +when adding an external dependency, but it means that the build system must +always build the entire project all at once. This means that it won’t be able to +parallelize or distribute parts of the build, nor will it be able to cache parts +that it’s already built. One-module-per-file is the opposite: the build system +has the maximum flexibility in caching and scheduling steps of the build, but +engineers need to expend more effort maintaining lists of dependencies whenever +they change which files reference which. + +Though the exact granularity varies by language (and often even within +language), Google tends to favor significantly smaller modules than one might +typically write in a task-based build system. A typical production binary at +Google often depends on tens of thousands of targets, and even a moderate-sized +team can own several hundred targets within its codebase. For languages like +Java that have a strong built-in notion of packaging, each directory usually +contains a single package, target, and `BUILD` file (Pants, another build system +based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging +conventions frequently define multiple targets per `BUILD` file. + +The benefits of smaller build targets really begin to show at scale because they +lead to faster distributed builds and a less frequent need to rebuild targets. +The advantages become even more compelling after testing enters the picture, as +finer-grained targets mean that the build system can be much smarter about +running only a limited subset of tests that could be affected by any given +change. Because Google believes in the systemic benefits of using smaller +targets, we’ve made some strides in mitigating the downside by investing in +tooling to automatically manage `BUILD` files to avoid burdening developers. + +Some of these tools, such as `buildifier` and `buildozer`, are available with +Bazel in the [`buildtools` +directory](https://github.com/bazelbuild/buildtools). ## Minimizing Module Visibility -Bazel and other build systems allow each target to specify a visibility — a property that determines which other targets may depend on it. A private target can only be referenced within its own `BUILD` file. A target may grant broader visibility to the targets of an explicitly defined list of `BUILD` files, or, in the case of public visibility, to every target in the workspace. - -As with most programming languages, it is usually best to minimize visibility as much as possible. Generally, teams at Google will make targets public only if those targets represent widely used libraries available to any team at Google. Teams that require others to coordinate with them before using their code will maintain an allowlist of customer targets as their target’s visibility. Each team’s internal implementation targets will be restricted to only directories owned by the team, and most `BUILD` files will have only one target that isn’t private. +Bazel and other build systems allow each target to specify a visibility — a +property that determines which other targets may depend on it. A private target +can only be referenced within its own `BUILD` file. A target may grant broader +visibility to the targets of an explicitly defined list of `BUILD` files, or, in +the case of public visibility, to every target in the workspace. + +As with most programming languages, it is usually best to minimize visibility as +much as possible. Generally, teams at Google will make targets public only if +those targets represent widely used libraries available to any team at Google. +Teams that require others to coordinate with them before using their code will +maintain an allowlist of customer targets as their target’s visibility. Each +team’s internal implementation targets will be restricted to only directories +owned by the team, and most `BUILD` files will have only one target that isn’t +private. ## Managing Dependencies -Modules need to be able to refer to one another. The downside of breaking a codebase into fine-grained modules is that you need to manage the dependencies among those modules (though tools can help automate this). Expressing these dependencies usually ends up being the bulk of the content in a `BUILD` file. +Modules need to be able to refer to one another. The downside of breaking a +codebase into fine-grained modules is that you need to manage the dependencies +among those modules (though tools can help automate this). Expressing these +dependencies usually ends up being the bulk of the content in a `BUILD` file. ### Internal dependencies -In a large project broken into fine-grained modules, most dependencies are likely to be internal; that is, on another target defined and built in the same source repository. Internal dependencies differ from external dependencies in that they are built from source rather than downloaded as a prebuilt artifact while running the build. This also means that there’s no notion of “version” for internal dependencies—a target and all of its internal dependencies are always built at the same commit/revision in the repository. One issue that should be handled carefully with regard to internal dependencies is how to treat transitive dependencies (Figure 1). Suppose target A depends on target B, which depends on a common library target C. Should target A be able to use classes defined in target C? - -[![Transitive dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) +In a large project broken into fine-grained modules, most dependencies are +likely to be internal; that is, on another target defined and built in the same +source repository. Internal dependencies differ from external dependencies in +that they are built from source rather than downloaded as a prebuilt artifact +while running the build. This also means that there’s no notion of “version” for +internal dependencies—a target and all of its internal dependencies are always +built at the same commit/revision in the repository. One issue that should be +handled carefully with regard to internal dependencies is how to treat +transitive dependencies (Figure 1). Suppose target A depends on target B, which +depends on a common library target C. Should target A be able to use classes +defined in target C? + +[![Transitive +dependencies](/images/transitive-dependencies.png)](/images/transitive-dependencies.png) **Figure 1**. Transitive dependencies -As far as the underlying tools are concerned, there’s no problem with this; both B and C will be linked into target A when it is built, so any symbols defined in C are known to A. Bazel allowed this for many years, but as Google grew, we began to see problems. Suppose that B was refactored such that it no longer needed to depend on C. If B’s dependency on C was then removed, A and any other target that used C via a dependency on B would break. Effectively, a target’s dependencies became part of its public contract and could never be safely changed. This meant that dependencies accumulated over time and builds at Google started to slow down. - -Google eventually solved this issue by introducing a “strict transitive dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to reference a symbol without depending on it directly and, if so, fails with an error and a shell command that can be used to automatically insert the dependency. Rolling this change out across Google’s entire codebase and refactoring every one of our millions of build targets to explicitly list their dependencies was a multiyear effort, but it was well worth it. Our builds are now much faster given that targets have fewer unnecessary dependencies, and engineers are empowered to remove dependencies they don’t need without worrying about breaking targets that depend on them. - -As usual, enforcing strict transitive dependencies involved a trade-off. It made build files more verbose, as frequently used libraries now need to be listed explicitly in many places rather than pulled in incidentally, and engineers needed to spend more effort adding dependencies to `BUILD` files. We’ve since developed tools that reduce this toil by automatically detecting many missing dependencies and adding them to a `BUILD` files without any developer intervention. But even without such tools, we’ve found the trade-off to be well worth it as the codebase scales: explicitly adding a dependency to `BUILD` file is a one-time cost, but dealing with implicit transitive dependencies can cause ongoing problems as long as the build target exists. Bazel [enforces strict transitive dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) on Java code by default. +As far as the underlying tools are concerned, there’s no problem with this; both +B and C will be linked into target A when it is built, so any symbols defined in +C are known to A. Bazel allowed this for many years, but as Google grew, we +began to see problems. Suppose that B was refactored such that it no longer +needed to depend on C. If B’s dependency on C was then removed, A and any other +target that used C via a dependency on B would break. Effectively, a target’s +dependencies became part of its public contract and could never be safely +changed. This meant that dependencies accumulated over time and builds at Google +started to slow down. + +Google eventually solved this issue by introducing a “strict transitive +dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to +reference a symbol without depending on it directly and, if so, fails with an +error and a shell command that can be used to automatically insert the +dependency. Rolling this change out across Google’s entire codebase and +refactoring every one of our millions of build targets to explicitly list their +dependencies was a multiyear effort, but it was well worth it. Our builds are +now much faster given that targets have fewer unnecessary dependencies, and +engineers are empowered to remove dependencies they don’t need without worrying +about breaking targets that depend on them. + +As usual, enforcing strict transitive dependencies involved a trade-off. It made +build files more verbose, as frequently used libraries now need to be listed +explicitly in many places rather than pulled in incidentally, and engineers +needed to spend more effort adding dependencies to `BUILD` files. We’ve since +developed tools that reduce this toil by automatically detecting many missing +dependencies and adding them to a `BUILD` files without any developer +intervention. But even without such tools, we’ve found the trade-off to be well +worth it as the codebase scales: explicitly adding a dependency to `BUILD` file +is a one-time cost, but dealing with implicit transitive dependencies can cause +ongoing problems as long as the build target exists. Bazel [enforces strict +transitive +dependencies](https://blog.bazel.build/2017/06/28/sjd-unused_deps.html) +on Java code by default. ### External dependencies -If a dependency isn’t internal, it must be external. External dependencies are those on artifacts that are built and stored outside of the build system. The dependency is imported directly from an artifact repository (typically accessed over the internet) and used as-is rather than being built from source. One of the biggest differences between external and internal dependencies is that external dependencies have versions, and those versions exist independently of the project’s source code. +If a dependency isn’t internal, it must be external. External dependencies are +those on artifacts that are built and stored outside of the build system. The +dependency is imported directly from an artifact repository (typically accessed +over the internet) and used as-is rather than being built from source. One of +the biggest differences between external and internal dependencies is that +external dependencies have versions, and those versions exist independently of +the project’s source code. ### Automatic versus manual dependency management -Build systems can allow the versions of external dependencies to be managed either manually or automatically. When managed manually, the buildfile explicitly lists the version it wants to download from the artifact repository, often using a [semantic version string](https://semver.org/) such as `1.1.4`. When managed automatically, the source file specifies a range of acceptable versions, and the build system always downloads the latest one. For example, Gradle allows a dependency version to be declared as “1.+” to specify that any minor or patch version of a dependency is acceptable so long as the major version is 1. - -Automatically managed dependencies can be convenient for small projects, but they’re usually a recipe for disaster on projects of nontrivial size or that are being worked on by more than one engineer. The problem with automatically managed dependencies is that you have no control over when the version is updated. There’s no way to guarantee that external parties won’t make breaking updates (even when they claim to use semantic versioning), so a build that worked one day might be broken the next with no easy way to detect what changed or to roll it back to a working state. Even if the build doesn’t break, there can be subtle behavior or performance changes that are impossible to track down. - -In contrast, because manually managed dependencies require a change in source control, they can be easily discovered and rolled back, and it’s possible to check out an older version of the repository to build with older dependencies. Bazel requires that versions of all dependencies be specified manually. At even moderate scales, the overhead of manual version management is well worth it for the stability it provides. +Build systems can allow the versions of external dependencies to be managed +either manually or automatically. When managed manually, the buildfile +explicitly lists the version it wants to download from the artifact repository, +often using a [semantic version string](https://semver.org/) such +as `1.1.4`. When managed automatically, the source file specifies a range of +acceptable versions, and the build system always downloads the latest one. For +example, Gradle allows a dependency version to be declared as “1.+” to specify +that any minor or patch version of a dependency is acceptable so long as the +major version is 1. + +Automatically managed dependencies can be convenient for small projects, but +they’re usually a recipe for disaster on projects of nontrivial size or that are +being worked on by more than one engineer. The problem with automatically +managed dependencies is that you have no control over when the version is +updated. There’s no way to guarantee that external parties won’t make breaking +updates (even when they claim to use semantic versioning), so a build that +worked one day might be broken the next with no easy way to detect what changed +or to roll it back to a working state. Even if the build doesn’t break, there +can be subtle behavior or performance changes that are impossible to track down. + +In contrast, because manually managed dependencies require a change in source +control, they can be easily discovered and rolled back, and it’s possible to +check out an older version of the repository to build with older dependencies. +Bazel requires that versions of all dependencies be specified manually. At even +moderate scales, the overhead of manual version management is well worth it for +the stability it provides. ### The One-Version Rule -Different versions of a library are usually represented by different artifacts, so in theory there’s no reason that different versions of the same external dependency couldn’t both be declared in the build system under different names. That way, each target could choose which version of the dependency it wanted to use. This causes a lot of problems in practice, so Google enforces a strict [One-Version Rule](https://opensource.google/docs/thirdparty/oneversion/) for all third-party dependencies in our codebase. - -The biggest problem with allowing multiple versions is the diamond dependency issue. Suppose that target A depends on target B and on v1 of an external library. If target B is later refactored to add a dependency on v2 of the same external library, target A will break because it now depends implicitly on two different versions of the same library. Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, because any of that target’s users could already be depending on a different version. Following the One-Version Rule makes this conflict impossible—if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist. +Different versions of a library are usually represented by different artifacts, +so in theory there’s no reason that different versions of the same external +dependency couldn’t both be declared in the build system under different names. +That way, each target could choose which version of the dependency it wanted to +use. This causes a lot of problems in practice, so Google enforces a strict +[One-Version +Rule](https://opensource.google/docs/thirdparty/oneversion/) for +all third-party dependencies in our codebase. + +The biggest problem with allowing multiple versions is the diamond dependency +issue. Suppose that target A depends on target B and on v1 of an external +library. If target B is later refactored to add a dependency on v2 of the same +external library, target A will break because it now depends implicitly on two +different versions of the same library. Effectively, it’s never safe to add a +new dependency from a target to any third-party library with multiple versions, +because any of that target’s users could already be depending on a different +version. Following the One-Version Rule makes this conflict impossible—if a +target adds a dependency on a third-party library, any existing dependencies +will already be on that same version, so they can happily coexist. ### Transitive external dependencies -Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central, allow artifacts to specify dependencies on particular versions of other artifacts in the repository. Build tools like Maven or Gradle often recursively download each transitive dependency by default, meaning that adding a single dependency in your project could potentially cause dozens of artifacts to be downloaded in total. - -This is very convenient: when adding a dependency on a new library, it would be a big pain to have to track down each of that library’s transitive dependencies and add them all manually. But there’s also a huge downside: because different libraries can depend on different versions of the same third-party library, this strategy necessarily violates the One-Version Rule and leads to the diamond dependency problem. If your target depends on two external libraries that use different versions of the same dependency, there’s no telling which one you’ll get. This also means that updating an external dependency could cause seemingly unrelated failures throughout the codebase if the new version begins pulling in conflicting versions of some of its dependencies. - -Bazel did not use to automatically download transitive dependencies. It used to employ a `WORKSPACE` file that required all transitive dependencies to be listed, which led to a lot of pain when managing external dependencies. Bazel has since added support for automatic transitive external dependency management in the form of the `MODULE.bazel` file. See [external dependency overview](/external/overview) for more details. - -Yet again, the choice here is one between convenience and scalability. Small projects might prefer not having to worry about managing transitive dependencies themselves and might be able to get away with using automatic transitive dependencies. This strategy becomes less and less appealing as the organization and codebase grows, and conflicts and unexpected results become more and more frequent. At larger scales, the cost of manually managing dependencies is much less than the cost of dealing with issues caused by automatic dependency management. +Dealing with the transitive dependencies of an external dependency can be +particularly difficult. Many artifact repositories such as Maven Central, allow +artifacts to specify dependencies on particular versions of other artifacts in +the repository. Build tools like Maven or Gradle often recursively download each +transitive dependency by default, meaning that adding a single dependency in +your project could potentially cause dozens of artifacts to be downloaded in +total. + +This is very convenient: when adding a dependency on a new library, it would be +a big pain to have to track down each of that library’s transitive dependencies +and add them all manually. But there’s also a huge downside: because different +libraries can depend on different versions of the same third-party library, this +strategy necessarily violates the One-Version Rule and leads to the diamond +dependency problem. If your target depends on two external libraries that use +different versions of the same dependency, there’s no telling which one you’ll +get. This also means that updating an external dependency could cause seemingly +unrelated failures throughout the codebase if the new version begins pulling in +conflicting versions of some of its dependencies. + +Bazel did not use to automatically download transitive dependencies. It used to +employ a `WORKSPACE` file that required all transitive dependencies to be +listed, which led to a lot of pain when managing external dependencies. Bazel +has since added support for automatic transitive external dependency management +in the form of the `MODULE.bazel` file. See [external dependency +overview](/external/overview) for more details. + +Yet again, the choice here is one between convenience and scalability. Small +projects might prefer not having to worry about managing transitive dependencies +themselves and might be able to get away with using automatic transitive +dependencies. This strategy becomes less and less appealing as the organization +and codebase grows, and conflicts and unexpected results become more and more +frequent. At larger scales, the cost of manually managing dependencies is much +less than the cost of dealing with issues caused by automatic dependency +management. ### Caching build results using external dependencies -External dependencies are most often provided by third parties that release stable versions of libraries, perhaps without providing source code. Some organizations might also choose to make some of their own code available as artifacts, allowing other pieces of code to depend on them as third-party rather than internal dependencies. This can theoretically speed up builds if artifacts are slow to build but quick to download. - -However, this also introduces a lot of overhead and complexity: someone needs to be responsible for building each of those artifacts and uploading them to the artifact repository, and clients need to ensure that they stay up to date with the latest version. Debugging also becomes much more difficult because different parts of the system will have been built from different points in the repository, and there is no longer a consistent view of the source tree. - -A better way to solve the problem of artifacts taking a long time to build is to use a build system that supports remote caching, as described earlier. Such a build system saves the resulting artifacts from every build to a location that is shared across engineers, so if a developer depends on an artifact that was recently built by someone else, the build system automatically downloads it instead of building it. This provides all of the performance benefits of depending directly on artifacts while still ensuring that builds are as consistent as if they were always built from the same source. This is the strategy used internally by Google, and Bazel can be configured to use a remote cache. +External dependencies are most often provided by third parties that release +stable versions of libraries, perhaps without providing source code. Some +organizations might also choose to make some of their own code available as +artifacts, allowing other pieces of code to depend on them as third-party rather +than internal dependencies. This can theoretically speed up builds if artifacts +are slow to build but quick to download. + +However, this also introduces a lot of overhead and complexity: someone needs to +be responsible for building each of those artifacts and uploading them to the +artifact repository, and clients need to ensure that they stay up to date with +the latest version. Debugging also becomes much more difficult because different +parts of the system will have been built from different points in the +repository, and there is no longer a consistent view of the source tree. + +A better way to solve the problem of artifacts taking a long time to build is to +use a build system that supports remote caching, as described earlier. Such a +build system saves the resulting artifacts from every build to a location that +is shared across engineers, so if a developer depends on an artifact that was +recently built by someone else, the build system automatically downloads it +instead of building it. This provides all of the performance benefits of +depending directly on artifacts while still ensuring that builds are as +consistent as if they were always built from the same source. This is the +strategy used internally by Google, and Bazel can be configured to use a remote +cache. ### Security and reliability of external dependencies -Depending on artifacts from third-party sources is inherently risky. There’s an availability risk if the third-party source (such as an artifact repository) goes down, because your entire build might grind to a halt if it’s unable to download an external dependency. There’s also a security risk: if the third-party system is compromised by an attacker, the attacker could replace the referenced artifact with one of their own design, allowing them to inject arbitrary code into your build. Both problems can be mitigated by mirroring any artifacts you depend on onto servers you control and blocking your build system from accessing third-party artifact repositories like Maven Central. The trade-off is that these mirrors take effort and resources to maintain, so the choice of whether to use them often depends on the scale of the project. The security issue can also be completely prevented with little overhead by requiring the hash of each third-party artifact to be specified in the source repository, causing the build to fail if the artifact is tampered with. Another alternative that completely sidesteps the issue is to vendor your project’s dependencies. When a project vendors its dependencies, it checks them into source control alongside the project’s source code, either as source or as binaries. This effectively means that all of the project’s external dependencies are converted to internal dependencies. Google uses this approach internally, checking every third-party library referenced throughout Google into a `third_party` directory at the root of Google’s source tree. However, this works at Google only because Google’s source control system is custom built to handle an extremely large monorepo, so vendoring might not be an option for all organizations. +Depending on artifacts from third-party sources is inherently risky. There’s an +availability risk if the third-party source (such as an artifact repository) +goes down, because your entire build might grind to a halt if it’s unable to +download an external dependency. There’s also a security risk: if the +third-party system is compromised by an attacker, the attacker could replace the +referenced artifact with one of their own design, allowing them to inject +arbitrary code into your build. Both problems can be mitigated by mirroring any +artifacts you depend on onto servers you control and blocking your build system +from accessing third-party artifact repositories like Maven Central. The +trade-off is that these mirrors take effort and resources to maintain, so the +choice of whether to use them often depends on the scale of the project. The +security issue can also be completely prevented with little overhead by +requiring the hash of each third-party artifact to be specified in the source +repository, causing the build to fail if the artifact is tampered with. Another +alternative that completely sidesteps the issue is to vendor your project’s +dependencies. When a project vendors its dependencies, it checks them into +source control alongside the project’s source code, either as source or as +binaries. This effectively means that all of the project’s external dependencies +are converted to internal dependencies. Google uses this approach internally, +checking every third-party library referenced throughout Google into a +`third_party` directory at the root of Google’s source tree. However, this works +at Google only because Google’s source control system is custom built to handle +an extremely large monorepo, so vendoring might not be an option for all +organizations. diff --git a/basics/distributed-builds.mdx b/basics/distributed-builds.mdx index b11e7c9b..c32f44ff 100644 --- a/basics/distributed-builds.mdx +++ b/basics/distributed-builds.mdx @@ -2,46 +2,136 @@ title: 'Distributed Builds' --- -When you have a large codebase, chains of dependencies can become very deep. Even simple binaries can often depend on tens of thousands of build targets. At this scale, it’s simply impossible to complete a build in a reasonable amount of time on a single machine: no build system can get around the fundamental laws of physics imposed on a machine’s hardware. The only way to make this work is with a build system that supports distributed builds wherein the units of work being done by the system are spread across an arbitrary and scalable number of machines. Assuming we’ve broken the system’s work into small enough units (more on this later), this would allow us to complete any build of any size as quickly as we’re willing to pay for. This scalability is the holy grail we’ve been working toward by defining an artifact-based build system. + + +When you have a large codebase, chains of dependencies can become very deep. +Even simple binaries can often depend on tens of thousands of build targets. At +this scale, it’s simply impossible to complete a build in a reasonable amount +of time on a single machine: no build system can get around the fundamental +laws of physics imposed on a machine’s hardware. The only way to make this work +is with a build system that supports distributed builds wherein the units of +work being done by the system are spread across an arbitrary and scalable +number of machines. Assuming we’ve broken the system’s work into small enough +units (more on this later), this would allow us to complete any build of any +size as quickly as we’re willing to pay for. This scalability is the holy grail +we’ve been working toward by defining an artifact-based build system. ## Remote caching -The simplest type of distributed build is one that only leverages *remote caching*, which is shown in Figure 1. +The simplest type of distributed build is one that only leverages _remote +caching_, which is shown in Figure 1. [![Distributed build with remote caching](/images/distributed-build-remote-cache.png)](/images/distributed-build-remote-cache.png) **Figure 1**. A distributed build showing remote caching -Every system that performs builds, including both developer workstations and continuous integration systems, shares a reference to a common remote cache service. This service might be a fast and local short-term storage system like Redis or a cloud service like Google Cloud Storage. Whenever a user needs to build an artifact, whether directly or as a dependency, the system first checks with the remote cache to see if that artifact already exists there. If so, it can download the artifact instead of building it. If not, the system builds the artifact itself and uploads the result back to the cache. This means that low-level dependencies that don’t change very often can be built once and shared across users rather than having to be rebuilt by each user. At Google, many artifacts are served from a cache rather than built from scratch, vastly reducing the cost of running our build system. - -For a remote caching system to work, the build system must guarantee that builds are completely reproducible. That is, for any build target, it must be possible to determine the set of inputs to that target such that the same set of inputs will produce exactly the same output on any machine. This is the only way to ensure that the results of downloading an artifact are the same as the results of building it oneself. Note that this requires that each artifact in the cache be keyed on both its target and a hash of its inputs—that way, different engineers could make different modifications to the same target at the same time, and the remote cache would store all of the resulting artifacts and serve them appropriately without conflict. - -Of course, for there to be any benefit from a remote cache, downloading an artifact needs to be faster than building it. This is not always the case, especially if the cache server is far from the machine doing the build. Google’s network and build system is carefully tuned to be able to quickly share build results. +Every system that performs builds, including both developer workstations and +continuous integration systems, shares a reference to a common remote cache +service. This service might be a fast and local short-term storage system like +Redis or a cloud service like Google Cloud Storage. Whenever a user needs to +build an artifact, whether directly or as a dependency, the system first checks +with the remote cache to see if that artifact already exists there. If so, it +can download the artifact instead of building it. If not, the system builds the +artifact itself and uploads the result back to the cache. This means that +low-level dependencies that don’t change very often can be built once and shared +across users rather than having to be rebuilt by each user. At Google, many +artifacts are served from a cache rather than built from scratch, vastly +reducing the cost of running our build system. + +For a remote caching system to work, the build system must guarantee that builds +are completely reproducible. That is, for any build target, it must be possible +to determine the set of inputs to that target such that the same set of inputs +will produce exactly the same output on any machine. This is the only way to +ensure that the results of downloading an artifact are the same as the results +of building it oneself. Note that this requires that each artifact in the cache +be keyed on both its target and a hash of its inputs—that way, different +engineers could make different modifications to the same target at the same +time, and the remote cache would store all of the resulting artifacts and serve +them appropriately without conflict. + +Of course, for there to be any benefit from a remote cache, downloading an +artifact needs to be faster than building it. This is not always the case, +especially if the cache server is far from the machine doing the build. Google’s +network and build system is carefully tuned to be able to quickly share build +results. ## Remote execution -Remote caching isn’t a true distributed build. If the cache is lost or if you make a low-level change that requires everything to be rebuilt, you still need to perform the entire build locally on your machine. The true goal is to support remote execution, in which the actual work of doing the build can be spread across any number of workers. Figure 2 depicts a remote execution system. +Remote caching isn’t a true distributed build. If the cache is lost or if you +make a low-level change that requires everything to be rebuilt, you still need +to perform the entire build locally on your machine. The true goal is to support +remote execution, in which the actual work of doing the build can be spread +across any number of workers. Figure 2 depicts a remote execution system. [![Remote execution system](/images/remote-execution-system.png)](/images/remote-execution-system.png) **Figure 2**. A remote execution system -The build tool running on each user’s machine (where users are either human engineers or automated build systems) sends requests to a central build master. The build master breaks the requests into their component actions and schedules the execution of those actions over a scalable pool of workers. Each worker performs the actions asked of it with the inputs specified by the user and writes out the resulting artifacts. These artifacts are shared across the other machines executing actions that require them until the final output can be produced and sent to the user. - -The trickiest part of implementing such a system is managing the communication between the workers, the master, and the user’s local machine. Workers might depend on intermediate artifacts produced by other workers, and the final output needs to be sent back to the user’s local machine. To do this, we can build on top of the distributed cache described previously by having each worker write its results to and read its dependencies from the cache. The master blocks workers from proceeding until everything they depend on has finished, in which case they’ll be able to read their inputs from the cache. The final product is also cached, allowing the local machine to download it. Note that we also need a separate means of exporting the local changes in the user’s source tree so that workers can apply those changes before building. - -For this to work, all of the parts of the artifact-based build systems described earlier need to come together. Build environments must be completely self-describing so that we can spin up workers without human intervention. Build processes themselves must be completely self-contained because each step might be executed on a different machine. Outputs must be completely deterministic so that each worker can trust the results it receives from other workers. Such guarantees are extremely difficult for a task-based system to provide, which makes it nigh-impossible to build a reliable remote execution system on top of one. +The build tool running on each user’s machine (where users are either human +engineers or automated build systems) sends requests to a central build master. +The build master breaks the requests into their component actions and schedules +the execution of those actions over a scalable pool of workers. Each worker +performs the actions asked of it with the inputs specified by the user and +writes out the resulting artifacts. These artifacts are shared across the other +machines executing actions that require them until the final output can be +produced and sent to the user. + +The trickiest part of implementing such a system is managing the communication +between the workers, the master, and the user’s local machine. Workers might +depend on intermediate artifacts produced by other workers, and the final output +needs to be sent back to the user’s local machine. To do this, we can build on +top of the distributed cache described previously by having each worker write +its results to and read its dependencies from the cache. The master blocks +workers from proceeding until everything they depend on has finished, in which +case they’ll be able to read their inputs from the cache. The final product is +also cached, allowing the local machine to download it. Note that we also need a +separate means of exporting the local changes in the user’s source tree so that +workers can apply those changes before building. + +For this to work, all of the parts of the artifact-based build systems described +earlier need to come together. Build environments must be completely +self-describing so that we can spin up workers without human intervention. Build +processes themselves must be completely self-contained because each step might +be executed on a different machine. Outputs must be completely deterministic so +that each worker can trust the results it receives from other workers. Such +guarantees are extremely difficult for a task-based system to provide, which +makes it nigh-impossible to build a reliable remote execution system on top of +one. ## Distributed builds at Google -Since 2008, Google has been using a distributed build system that employs both remote caching and remote execution, which is illustrated in Figure 3. +Since 2008, Google has been using a distributed build system that employs both +remote caching and remote execution, which is illustrated in Figure 3. [![High-level build system](/images/high-level-build-system.png)](/images/high-level-build-system.png) **Figure 3**. Google’s distributed build system -Google’s remote cache is called ObjFS. It consists of a backend that stores build outputs in Bigtables distributed throughout our fleet of production machines and a frontend FUSE daemon named objfsd that runs on each developer’s machine. The FUSE daemon allows engineers to browse build outputs as if they were normal files stored on the workstation, but with the file content downloaded on-demand only for the few files that are directly requested by the user. Serving file contents on-demand greatly reduces both network and disk usage, and the system is able to build twice as fast compared to when we stored all build output on the developer’s local disk. - -Google’s remote execution system is called Forge. A Forge client in Blaze (Bazel's internal equivalent) called the Distributor sends requests for each action to a job running in our datacenters called the Scheduler. The Scheduler maintains a cache of action results, allowing it to return a response immediately if the action has already been created by any other user of the system. If not, it places the action into a queue. A large pool of Executor jobs continually read actions from this queue, execute them, and store the results directly in the ObjFS Bigtables. These results are available to the executors for future actions, or to be downloaded by the end user via objfsd. - -The end result is a system that scales to efficiently support all builds performed at Google. And the scale of Google’s builds is truly massive: Google runs millions of builds executing millions of test cases and producing petabytes of build outputs from billions of lines of source code every day. Not only does such a system let our engineers build complex codebases quickly, it also allows us to implement a huge number of automated tools and systems that rely on our build. +Google’s remote cache is called ObjFS. It consists of a backend that stores +build outputs in Bigtables distributed throughout our fleet of production +machines and a frontend FUSE daemon named objfsd that runs on each developer’s +machine. The FUSE daemon allows engineers to browse build outputs as if they +were normal files stored on the workstation, but with the file content +downloaded on-demand only for the few files that are directly requested by the +user. Serving file contents on-demand greatly reduces both network and disk +usage, and the system is able to build twice as fast compared to when we stored +all build output on the developer’s local disk. + +Google’s remote execution system is called Forge. A Forge client in Blaze +(Bazel's internal equivalent) called +the Distributor sends requests for each action to a job running in our +datacenters called the Scheduler. The Scheduler maintains a cache of action +results, allowing it to return a response immediately if the action has already +been created by any other user of the system. If not, it places the action into +a queue. A large pool of Executor jobs continually read actions from this queue, +execute them, and store the results directly in the ObjFS Bigtables. These +results are available to the executors for future actions, or to be downloaded +by the end user via objfsd. + +The end result is a system that scales to efficiently support all builds +performed at Google. And the scale of Google’s builds is truly massive: Google +runs millions of builds executing millions of test cases and producing petabytes +of build outputs from billions of lines of source code every day. Not only does +such a system let our engineers build complex codebases quickly, it also allows +us to implement a huge number of automated tools and systems that rely on our +build. diff --git a/basics/hermeticity.mdx b/basics/hermeticity.mdx index a9765af8..dcee3a9e 100644 --- a/basics/hermeticity.mdx +++ b/basics/hermeticity.mdx @@ -2,59 +2,108 @@ title: 'Hermeticity' --- -This page covers hermeticity, the benefits of using hermetic builds, and strategies for identifying non-hermetic behavior in your builds. + + +This page covers hermeticity, the benefits of using hermetic builds, and +strategies for identifying non-hermetic behavior in your builds. ## Overview -When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system. +When given the same input source code and product configuration, a hermetic +build system always returns the same output by isolating the build from changes +to the host system. -In order to isolate the build, hermetic builds are insensitive to libraries and other software installed on the local or remote host machine. They depend on specific versions of build tools, such as compilers, and dependencies, such as libraries. This makes the build process self-contained as it doesn't rely on services external to the build environment. +In order to isolate the build, hermetic builds are insensitive to libraries and +other software installed on the local or remote host machine. They depend on +specific versions of build tools, such as compilers, and dependencies, such as +libraries. This makes the build process self-contained as it doesn't rely on +services external to the build environment. The two important aspects of hermeticity are: -- **Isolation**: Hermetic build systems treat tools as source code. They download copies of tools and manage their storage and use inside managed file trees. This creates isolation between the host machine and local user, including installed versions of languages. -- **Source identity**: Hermetic build systems try to ensure the sameness of inputs. Code repositories, such as Git, identify sets of code mutations with a unique hash code. Hermetic build systems use this hash to identify changes to the build's input. +* **Isolation**: Hermetic build systems treat tools as source code. They + download copies of tools and manage their storage and use inside managed file + trees. This creates isolation between the host machine and local user, + including installed versions of languages. +* **Source identity**: Hermetic build systems try to ensure the sameness of + inputs. Code repositories, such as Git, identify sets of code mutations with a + unique hash code. Hermetic build systems use this hash to identify changes to + the build's input. ## Benefits The major benefits of hermetic builds are: -- **Speed**: The output of an action can be cached, and the action need not be run again unless inputs change. -- **Parallel execution**: For given input and output, the build system can construct a graph of all actions to calculate efficient and parallel execution. The build system loads the rules and calculates an action graph and hash inputs to look up in the cache. -- **Multiple builds**: You can build multiple hermetic builds on the same machine, each build using different tools and versions. -- **Reproducibility**: Hermetic builds are good for troubleshooting because you know the exact conditions that produced the build. +* **Speed**: The output of an action can be cached, and the action need not be + run again unless inputs change. +* **Parallel execution**: For given input and output, the build system can + construct a graph of all actions to calculate efficient and parallel + execution. The build system loads the rules and calculates an action graph + and hash inputs to look up in the cache. +* **Multiple builds**: You can build multiple hermetic builds on the same + machine, each build using different tools and versions. +* **Reproducibility**: Hermetic builds are good for troubleshooting because you + know the exact conditions that produced the build. ## Identifying non-hermeticity -If you are preparing to switch to Bazel, migration is easier if you improve your existing builds' hermeticity in advance. Some common sources of non-hermeticity in builds are: +If you are preparing to switch to Bazel, migration is easier if you improve +your existing builds' hermeticity in advance. Some common sources of +non-hermeticity in builds are: -- Arbitrary processing in `.mk` files -- Actions or tooling that create files non-deterministically, usually involving build IDs or timestamps -- System binaries that differ across hosts (such as `/usr/bin` binaries, absolute paths, system C++ compilers for native C++ rules autoconfiguration) -- Writing to the source tree during the build. This prevents the same source tree from being used for another target. The first build writes to the source tree, fixing the source tree for target A. Then trying to build target B may fail. +* Arbitrary processing in `.mk` files +* Actions or tooling that create files non-deterministically, usually involving + build IDs or timestamps +* System binaries that differ across hosts (such as `/usr/bin` binaries, absolute + paths, system C++ compilers for native C++ rules autoconfiguration) +* Writing to the source tree during the build. This prevents the same source + tree from being used for another target. The first build writes to the source + tree, fixing the source tree for target A. Then trying to build target B may + fail. ## Troubleshooting non-hermetic builds -Starting with local execution, issues that affect local cache hits reveal non-hermetic actions. - -- Ensure null sequential builds: If you run `make` and get a successful build, running the build again should not rebuild any targets. If you run each build step twice or on different systems, compare a hash of the file contents and get results that differ, the build is not reproducible. -- Run steps to [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) from a variety of potential client machines to ensure that you catch any cases of client environment leaking into the actions. -- Execute a build within a docker container that contains nothing but the checked-out source tree and explicit list of host tools. Build breakages and error messages will catch implicit system dependencies. -- Discover and fix hermeticity problems using [remote execution rules](/remote/rules#overview). -- Enable strict [sandboxing](/docs/sandboxing) at the per-action level, since actions in a build can be stateful and affect the build or the output. -- [Workspace rules](/remote/workspace) allow developers to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. You can get a log of some potentially non-hermetic actions in Bazel workspace rules by adding the flag `--experimental_workspace_rules_log_file=<var>PATH</var>` to your Bazel command. - -Note: Make your build fully hermetic when mixing remote and local execution, using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote Docker container will enable the build to execute the same in both environments. +Starting with local execution, issues that affect local cache hits reveal +non-hermetic actions. + +* Ensure null sequential builds: If you run `make` and get a successful build, + running the build again should not rebuild any targets. If you run each build + step twice or on different systems, compare a hash of the file contents and + get results that differ, the build is not reproducible. +* Run steps to + [debug local cache hits](/remote/cache-remote#troubleshooting-cache-hits) + from a variety of potential client machines to ensure that you catch any + cases of client environment leaking into the actions. +* Execute a build within a docker container that contains nothing but the + checked-out source tree and explicit list of host tools. Build breakages and + error messages will catch implicit system dependencies. +* Discover and fix hermeticity problems using + [remote execution rules](/remote/rules#overview). +* Enable strict [sandboxing](/docs/sandboxing) + at the per-action level, since actions in a build can be stateful and affect + the build or the output. +* [Workspace rules](/remote/workspace) + allow developers to add dependencies to external workspaces, but they are + rich enough to allow arbitrary processing to happen in the process. You can + get a log of some potentially non-hermetic actions in Bazel workspace rules by + adding the flag + `--experimental_workspace_rules_log_file=PATH` to + your Bazel command. + +Note: Make your build fully hermetic when mixing remote and local execution, +using Bazel’s “dynamic strategy” functionality. Running Bazel inside the remote +Docker container will enable the build to execute the same in both environments. ## Hermeticity with Bazel -For more information about how other projects have had success using hermetic builds with Bazel, see these BazelCon talks: - -- [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) -- [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) -- [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) -- [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) -- [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) -- [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) -- [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=4\&t=0s) (BMW) -- [Building Self Driving Cars with Bazel + Q\&A](https://www.youtube.com/watch?v=fjfFe98LTm8\&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj\&index=29) (GM Cruise) +For more information about how other projects have had success using hermetic +builds with Bazel, see these BazelCon talks: + +* [Building Real-time Systems with Bazel](https://www.youtube.com/watch?v=t_3bckhV_YI) (SpaceX) +* [Bazel Remote Execution and Remote Caching](https://www.youtube.com/watch?v=_bPyEbAyC0s) (Uber and TwoSigma) +* [Faster Builds With Remote Execution and Caching](https://www.youtube.com/watch?v=MyuJRUwT5LI) +* [Fusing Bazel: Faster Incremental Builds](https://www.youtube.com/watch?v=rQd9Zd1ONOw) +* [Remote Execution vs Local Execution](https://www.youtube.com/watch?v=C8wHmIln--g) +* [Improving the Usability of Remote Caching](https://www.youtube.com/watch?v=u5m7V3ZRHLA) (IBM) +* [Building Self Driving Cars with Bazel](https://www.youtube.com/watch?v=Gh4SJuYUoQI&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=4&t=0s) (BMW) +* [Building Self Driving Cars with Bazel + Q&A](https://www.youtube.com/watch?v=fjfFe98LTm8&list=PLxNYxgaZ8Rsf-7g43Z8LyXct9ax6egdSj&index=29) (GM Cruise) diff --git a/basics/index.mdx b/basics/index.mdx index ecd09227..f3c833fa 100644 --- a/basics/index.mdx +++ b/basics/index.mdx @@ -2,28 +2,56 @@ title: 'Build Basics' --- -A build system is one of the most important parts of an engineering organization because each developer interacts with it potentially dozens or hundreds of times per day. A fully featured build system is necessary to enable developer productivity as an organization scales. For individual developers, it's straightforward to just compile your code and so a build system might seem excessive. But at a larger scale, having a build system helps with managing shared dependencies, such as relying on another part of the code base, or an external resource, such as a library. Build systems help to make sure that you have everything you need to build your code before it starts building. Build systems also increase velocity when they're set up to help engineers share resources and results. -This section covers some history and basics of building and build systems, including design decisions that went into making Bazel. If you're familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you can skip this section, but it's a helpful overview to understand why artifact-based build systems are excellent at enabling scale. -Note: Much of this section's content comes from the *Build Systems and Build Philosophy* chapter of the [*Software Engineering at Google* book](https://abseil.io/resources/swe-book/html/ch18.html). Thank you to the original author, Erik Kuefler, for allowing its reuse and modification here! +A build system is one of the most important parts of an engineering organization +because each developer interacts with it potentially dozens or hundreds of times +per day. A fully featured build system is necessary to enable developer +productivity as an organization scales. For individual developers, it's +straightforward to just compile your code and so a build system might seem +excessive. But at a larger scale, having a build system helps with managing +shared dependencies, such as relying on another part of the code base, or an +external resource, such as a library. Build systems help to make sure that you +have everything you need to build your code before it starts building. Build +systems also increase velocity when they're set up to help engineers share +resources and results. -- **[Why a Build System?](/basics/build-systems)** +This section covers some history and basics of building and build systems, +including design decisions that went into making Bazel. If you're +familiar with artifact-based build systems, such as Bazel, Buck, and Pants, you +can skip this section, but it's a helpful overview to understand why +artifact-based build systems are excellent at enabling scale. - If you haven't used a build system before, start here. This page covers why you should use a build system, and why compilers and build scripts aren't the best choice once your organization starts to scale beyond a few developers. +Note: Much of this section's content comes from the _Build Systems and +Build Philosophy_ chapter of the +[_Software Engineering at Google_ book](https://abseil.io/resources/swe-book/html/ch18.html). +Thank you to the original author, Erik Kuefler, for allowing its reuse and +modification here! -- **[Task-Based Build Systems](/basics/task-based-builds)** +* **[Why a Build System?](/basics/build-systems)** - This page discusses task-based build systems (such as Make, Maven, and Gradle) and some of their challenges. + If you haven't used a build system before, start here. This page covers why + you should use a build system, and why compilers and build scripts aren't + the best choice once your organization starts to scale beyond a few + developers. -- **[Artifact-Based Build Systems](/basics/artifact-based-builds)** +* **[Task-Based Build Systems](/basics/task-based-builds)** - This page discusses artifact-based build systems in response to the pain points of task-based build systems. + This page discusses task-based build systems (such as Make, Maven, and + Gradle) and some of their challenges. -- **[Distributed Builds](/basics/distributed-builds)** +* **[Artifact-Based Build Systems](/basics/artifact-based-builds)** - This page covers distributed builds, or builds that are executed outside of your local machine. This requires more robust infrastructure to share resources and build results (and is where the true wizardry happens!) + This page discusses artifact-based build systems in response to the pain + points of task-based build systems. -- **[Dependency Management](/basics/dependencies)** +* **[Distributed Builds](/basics/distributed-builds)** - This page covers some complications of dependencies at a large scale and strategies to counteract those complications. + This page covers distributed builds, or builds that are executed outside of + your local machine. This requires more robust infrastructure to share + resources and build results (and is where the true wizardry happens!) + +* **[Dependency Management](/basics/dependencies)** + + This page covers some complications of dependencies at a large scale and + strategies to counteract those complications. diff --git a/basics/task-based-builds.mdx b/basics/task-based-builds.mdx index daef39b3..9dd3f8c0 100644 --- a/basics/task-based-builds.mdx +++ b/basics/task-based-builds.mdx @@ -2,68 +2,94 @@ title: 'Task-Based Build Systems' --- -This page covers task-based build systems, how they work and some of the complications that can occur with task-based systems. After shell scripts, task-based build systems are the next logical evolution of building. + + +This page covers task-based build systems, how they work and some of the +complications that can occur with task-based systems. After shell scripts, +task-based build systems are the next logical evolution of building. + ## Understanding task-based build systems -In a task-based build system, the fundamental unit of work is the task. Each task is a script that can execute any sort of logic, and tasks specify other tasks as dependencies that must run before them. Most major build systems in use today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of shell scripts, most modern build systems require engineers to create build files that describe how to perform the build. +In a task-based build system, the fundamental unit of work is the task. Each +task is a script that can execute any sort of logic, and tasks specify other +tasks as dependencies that must run before them. Most major build systems in use +today, such as Ant, Maven, Gradle, Grunt, and Rake, are task based. Instead of +shell scripts, most modern build systems require engineers to create build files +that describe how to perform the build. -Take this example from the [Ant manual](https://ant.apache.org/manual/using.html): +Take this example from the +[Ant manual](https://ant.apache.org/manual/using.html): ```xml -<project name="MyProject" default="dist" basedir="."> - <description> + + simple example build file - </description> - - <property name="src" location="src"/> - <property name="build" location="build"/> - <property name="dist" location="dist"/> - - <target name="init"> - - <tstamp/> - - <mkdir dir="${build}"/> - </target> - <target name="compile" depends="init" - description="compile the source"> - - <javac srcdir="${src}" destdir="${build}"/> - </target> - <target name="dist" depends="compile" - description="generate the distribution"> - - <mkdir dir="${dist}/lib"/> - - <jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar" basedir="${build}"/> - </target> - <target name="clean" - description="clean up"> - - <delete dir="${build}"/> - <delete dir="${dist}"/> - </target> -</project> + + + + + + + + + + + + + + + + + + + + + + + + + + + + ``` -The buildfile is written in XML and defines some simple metadata about the build along with a list of tasks (the `<target>` tags in the XML). (Ant uses the word *target* to represent a *task*, and it uses the word *task* to refer to *commands*.) Each task executes a list of possible commands defined by Ant, which here include creating and deleting directories, running `javac`, and creating a JAR file. This set of commands can be extended by user-provided plug-ins to cover any sort of logic. Each task can also define the tasks it depends on via the depends attribute. These dependencies form an acyclic graph, as seen in Figure 1. +The buildfile is written in XML and defines some simple metadata about the build +along with a list of tasks (the `` tags in the XML). (Ant uses the word +_target_ to represent a _task_, and it uses the word _task_ to refer to +_commands_.) Each task executes a list of possible commands defined by Ant, +which here include creating and deleting directories, running `javac`, and +creating a JAR file. This set of commands can be extended by user-provided +plug-ins to cover any sort of logic. Each task can also define the tasks it +depends on via the depends attribute. These dependencies form an acyclic graph, +as seen in Figure 1. [![Acrylic graph showing dependencies](/images/task-dependencies.png)](/images/task-dependencies.png) Figure 1. An acyclic graph showing dependencies -Users perform builds by providing tasks to Ant’s command-line tool. For example, when a user types `ant dist`, Ant takes the following steps: - -1. Loads a file named `build.xml` in the current directory and parses it to create the graph structure shown in Figure 1. -2. Looks for the task named `dist` that was provided on the command line and discovers that it has a dependency on the task named `compile`. -3. Looks for the task named `compile` and discovers that it has a dependency on the task named `init`. -4. Looks for the task named `init` and discovers that it has no dependencies. -5. Executes the commands defined in the `init` task. -6. Executes the commands defined in the `compile` task given that all of that task’s dependencies have been run. -7. Executes the commands defined in the `dist` task given that all of that task’s dependencies have been run. - -In the end, the code executed by Ant when running the `dist` task is equivalent to the following shell script: +Users perform builds by providing tasks to Ant’s command-line tool. For example, +when a user types `ant dist`, Ant takes the following steps: + +1. Loads a file named `build.xml` in the current directory and parses it to + create the graph structure shown in Figure 1. +1. Looks for the task named `dist` that was provided on the command line and + discovers that it has a dependency on the task named `compile`. +1. Looks for the task named `compile` and discovers that it has a dependency on + the task named `init`. +1. Looks for the task named `init` and discovers that it has no dependencies. +1. Executes the commands defined in the `init` task. +1. Executes the commands defined in the `compile` task given that all of that + task’s dependencies have been run. +1. Executes the commands defined in the `dist` task given that all of that + task’s dependencies have been run. + +In the end, the code executed by Ant when running the `dist` task is equivalent +to the following shell script: ```posix-terminal ./createTimestamp.sh @@ -77,32 +103,114 @@ mkdir -p dist/lib/ jar cf dist/lib/MyProject-$(date --iso-8601).jar build/* ``` -When the syntax is stripped away, the buildfile and the build script actually aren’t too different. But we’ve already gained a lot by doing this. We can create new buildfiles in other directories and link them together. We can easily add new tasks that depend on existing tasks in arbitrary and complex ways. We need only pass the name of a single task to the `ant` command-line tool, and it determines everything that needs to be run. - -Ant is an old piece of software, originally released in 2000. Other tools like Maven and Gradle have improved on Ant in the intervening years and essentially replaced it by adding features like automatic management of external dependencies and a cleaner syntax without any XML. But the nature of these newer systems remains the same: they allow engineers to write build scripts in a principled and modular way as tasks and provide tools for executing those tasks and managing dependencies among them. +When the syntax is stripped away, the buildfile and the build script actually +aren’t too different. But we’ve already gained a lot by doing this. We can +create new buildfiles in other directories and link them together. We can easily +add new tasks that depend on existing tasks in arbitrary and complex ways. We +need only pass the name of a single task to the `ant` command-line tool, and it +determines everything that needs to be run. + +Ant is an old piece of software, originally released in 2000. Other tools like +Maven and Gradle have improved on Ant in the intervening years and essentially +replaced it by adding features like automatic management of external +dependencies and a cleaner syntax without any XML. But the nature of these newer +systems remains the same: they allow engineers to write build scripts in a +principled and modular way as tasks and provide tools for executing those tasks +and managing dependencies among them. ## The dark side of task-based build systems -Because these tools essentially let engineers define any script as a task, they are extremely powerful, allowing you to do pretty much anything you can imagine with them. But that power comes with drawbacks, and task-based build systems can become difficult to work with as their build scripts grow more complex. The problem with such systems is that they actually end up giving *too much power to engineers and not enough power to the system*. Because the system has no idea what the scripts are doing, performance suffers, as it must be very conservative in how it schedules and executes build steps. And there’s no way for the system to confirm that each script is doing what it should, so scripts tend to grow in complexity and end up being another thing that needs debugging. +Because these tools essentially let engineers define any script as a task, they +are extremely powerful, allowing you to do pretty much anything you can imagine +with them. But that power comes with drawbacks, and task-based build systems can +become difficult to work with as their build scripts grow more complex. The +problem with such systems is that they actually end up giving _too much power to +engineers and not enough power to the system_. Because the system has no idea +what the scripts are doing, performance suffers, as it must be very conservative +in how it schedules and executes build steps. And there’s no way for the system +to confirm that each script is doing what it should, so scripts tend to grow in +complexity and end up being another thing that needs debugging. ### Difficulty of parallelizing build steps -Modern development workstations are quite powerful, with multiple cores that are capable of executing several build steps in parallel. But task-based systems are often unable to parallelize task execution even when it seems like they should be able to. Suppose that task A depends on tasks B and C. Because tasks B and C have no dependency on each other, is it safe to run them at the same time so that the system can more quickly get to task A? Maybe, if they don’t touch any of the same resources. But maybe not—perhaps both use the same file to track their statuses and running them at the same time causes a conflict. There’s no way in general for the system to know, so either it has to risk these conflicts (leading to rare but very difficult-to-debug build problems), or it has to restrict the entire build to running on a single thread in a single process. This can be a huge waste of a powerful developer machine, and it completely rules out the possibility of distributing the build across multiple machines. +Modern development workstations are quite powerful, with multiple cores that are +capable of executing several build steps in parallel. But task-based systems are +often unable to parallelize task execution even when it seems like they should +be able to. Suppose that task A depends on tasks B and C. Because tasks B and C +have no dependency on each other, is it safe to run them at the same time so +that the system can more quickly get to task A? Maybe, if they don’t touch any +of the same resources. But maybe not—perhaps both use the same file to track +their statuses and running them at the same time causes a conflict. There’s no +way in general for the system to know, so either it has to risk these conflicts +(leading to rare but very difficult-to-debug build problems), or it has to +restrict the entire build to running on a single thread in a single process. +This can be a huge waste of a powerful developer machine, and it completely +rules out the possibility of distributing the build across multiple machines. ### Difficulty performing incremental builds -A good build system allows engineers to perform reliable incremental builds such that a small change doesn’t require the entire codebase to be rebuilt from scratch. This is especially important if the build system is slow and unable to parallelize build steps for the aforementioned reasons. But unfortunately, task-based build systems struggle here, too. Because tasks can do anything, there’s no way in general to check whether they’ve already been done. Many tasks simply take a set of source files and run a compiler to create a set of binaries; thus, they don’t need to be rerun if the underlying source files haven’t changed. But without additional information, the system can’t say this for sure—maybe the task downloads a file that could have changed, or maybe it writes a timestamp that could be different on each run. To guarantee correctness, the system typically must rerun every task during each build. Some build systems try to enable incremental builds by letting engineers specify the conditions under which a task needs to be rerun. Sometimes this is feasible, but often it’s a much trickier problem than it appears. For example, in languages like C++ that allow files to be included directly by other files, it’s impossible to determine the entire set of files that must be watched for changes without parsing the input sources. Engineers often end up taking shortcuts, and these shortcuts can lead to rare and frustrating problems where a task result is reused even when it shouldn’t be. When this happens frequently, engineers get into the habit of running clean before every build to get a fresh state, completely defeating the purpose of having an incremental build in the first place. Figuring out when a task needs to be rerun is surprisingly subtle, and is a job better handled by machines than humans. +A good build system allows engineers to perform reliable incremental builds such +that a small change doesn’t require the entire codebase to be rebuilt from +scratch. This is especially important if the build system is slow and unable to +parallelize build steps for the aforementioned reasons. But unfortunately, +task-based build systems struggle here, too. Because tasks can do anything, +there’s no way in general to check whether they’ve already been done. Many tasks +simply take a set of source files and run a compiler to create a set of +binaries; thus, they don’t need to be rerun if the underlying source files +haven’t changed. But without additional information, the system can’t say this +for sure—maybe the task downloads a file that could have changed, or maybe it +writes a timestamp that could be different on each run. To guarantee +correctness, the system typically must rerun every task during each build. Some +build systems try to enable incremental builds by letting engineers specify the +conditions under which a task needs to be rerun. Sometimes this is feasible, but +often it’s a much trickier problem than it appears. For example, in languages +like C++ that allow files to be included directly by other files, it’s +impossible to determine the entire set of files that must be watched for changes +without parsing the input sources. Engineers often end up taking shortcuts, and +these shortcuts can lead to rare and frustrating problems where a task result is +reused even when it shouldn’t be. When this happens frequently, engineers get +into the habit of running clean before every build to get a fresh state, +completely defeating the purpose of having an incremental build in the first +place. Figuring out when a task needs to be rerun is surprisingly subtle, and is +a job better handled by machines than humans. ### Difficulty maintaining and debugging scripts -Finally, the build scripts imposed by task-based build systems are often just difficult to work with. Though they often receive less scrutiny, build scripts are code just like the system being built, and are easy places for bugs to hide. Here are some examples of bugs that are very common when working with a task-based build system: - -- Task A depends on task B to produce a particular file as output. The owner of task B doesn’t realize that other tasks rely on it, so they change it to produce output in a different location. This can’t be detected until someone tries to run task A and finds that it fails. -- Task A depends on task B, which depends on task C, which is producing a particular file as output that’s needed by task A. The owner of task B decides that it doesn’t need to depend on task C any more, which causes task A to fail even though task B doesn’t care about task C at all! -- The developer of a new task accidentally makes an assumption about the machine running the task, such as the location of a tool or the value of particular environment variables. The task works on their machine, but fails whenever another developer tries it. -- A task contains a nondeterministic component, such as downloading a file from the internet or adding a timestamp to a build. Now, people get potentially different results each time they run the build, meaning that engineers won’t always be able to reproduce and fix one another’s failures or failures that occur on an automated build system. -- Tasks with multiple dependencies can create race conditions. If task A depends on both task B and task C, and task B and C both modify the same file, task A gets a different result depending on which one of tasks B and C finishes first. - -There’s no general-purpose way to solve these performance, correctness, or maintainability problems within the task-based framework laid out here. So long as engineers can write arbitrary code that runs during the build, the system can’t have enough information to always be able to run builds quickly and correctly. To solve the problem, we need to take some power out of the hands of engineers and put it back in the hands of the system and reconceptualize the role of the system not as running tasks, but as producing artifacts. - -This approach led to the creation of artifact-based build systems, like Blaze and Bazel. +Finally, the build scripts imposed by task-based build systems are often just +difficult to work with. Though they often receive less scrutiny, build scripts +are code just like the system being built, and are easy places for bugs to hide. +Here are some examples of bugs that are very common when working with a +task-based build system: + +* Task A depends on task B to produce a particular file as output. The owner + of task B doesn’t realize that other tasks rely on it, so they change it to + produce output in a different location. This can’t be detected until someone + tries to run task A and finds that it fails. +* Task A depends on task B, which depends on task C, which is producing a + particular file as output that’s needed by task A. The owner of task B + decides that it doesn’t need to depend on task C any more, which causes task + A to fail even though task B doesn’t care about task C at all! +* The developer of a new task accidentally makes an assumption about the + machine running the task, such as the location of a tool or the value of + particular environment variables. The task works on their machine, but fails + whenever another developer tries it. +* A task contains a nondeterministic component, such as downloading a file + from the internet or adding a timestamp to a build. Now, people get + potentially different results each time they run the build, meaning that + engineers won’t always be able to reproduce and fix one another’s failures + or failures that occur on an automated build system. +* Tasks with multiple dependencies can create race conditions. If task A + depends on both task B and task C, and task B and C both modify the same + file, task A gets a different result depending on which one of tasks B and C + finishes first. + +There’s no general-purpose way to solve these performance, correctness, or +maintainability problems within the task-based framework laid out here. So long +as engineers can write arbitrary code that runs during the build, the system +can’t have enough information to always be able to run builds quickly and +correctly. To solve the problem, we need to take some power out of the hands of +engineers and put it back in the hands of the system and reconceptualize the +role of the system not as running tasks, but as producing artifacts. + +This approach led to the creation of artifact-based build systems, like Blaze +and Bazel. diff --git a/brand/index.mdx b/brand/index.mdx index f6b1bd31..2a21cd43 100644 --- a/brand/index.mdx +++ b/brand/index.mdx @@ -2,57 +2,87 @@ title: 'Bazel Brand Guidelines' --- -The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and are treated separately from the copyright or patent license grants contained in the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel Trademarks other than those permitted in these guidelines must be approved in advance. + + +The Bazel trademark and logo ("Bazel Trademarks") are trademarks of Google, and +are treated separately from the copyright or patent license grants contained in +the Apache-licensed Bazel repositories on GitHub. Any use of the Bazel +Trademarks other than those permitted in these guidelines must be approved in +advance. ## Purpose of the Brand Guidelines -These guidelines exist to ensure that the Bazel project can share its technology under open source licenses while making sure that the "Bazel" brand is protected as a meaningful source identifier in a way that's consistent with trademark law. By adhering to these guidelines, you help to promote the freedom to use and develop high-quality Bazel technology. +These guidelines exist to ensure that the Bazel project can share its technology +under open source licenses while making sure that the "Bazel" brand is protected +as a meaningful source identifier in a way that's consistent with trademark law. +By adhering to these guidelines, you help to promote the freedom to use and +develop high-quality Bazel technology. ## Acceptable Uses -Given the open nature of Bazel, you may use the Bazel trademark to refer to the project without prior written permission. Examples of these approved references include the following: +Given the open nature of Bazel, you may use the Bazel trademark to refer to the +project without prior written permission. Examples of these approved references +include the following: -- To refer to the Bazel Project itself; -- To link to bazel.build; -- To refer to unmodified source code or other files shared by the Bazel repositories on GitHub; -- In blog posts, news articles, or educational materials about Bazel; -- To accurately identify that your design or implementation is based on, is for use with, or is compatible with Bazel technology. +* To refer to the Bazel Project itself; +* To link to bazel.build; +* To refer to unmodified source code or other files shared by the Bazel + repositories on GitHub; +* In blog posts, news articles, or educational materials about Bazel; +* To accurately identify that your design or implementation is based on, is + for use with, or is compatible with Bazel technology. Examples: -- \[Your Product] for Bazel -- \[Your Product] is compatible with Bazel -- \[XYZ] Conference for Bazel Users +* \[Your Product\] for Bazel +* \[Your Product\] is compatible with Bazel +* \[XYZ\] Conference for Bazel Users ## General Guidelines -- The Bazel name may never be used or registered in a manner that would cause confusion as to Google's sponsorship, affiliation, or endorsement. -- Don't use the Bazel name as part of your company name, product name, domain name, or social media profile. -- Other than as permitted by these guidelines, the Bazel name should not be combined with other trademarks, terms, or source identifiers. -- Don't remove, distort or alter any element of the Bazel Trademarks. That includes modifying the Bazel Trademark, for example, through hyphenation, combination or abbreviation. Do not shorten, abbreviate, or create acronyms out of the Bazel Trademarks. -- Don't display the word Bazel using any different stylization, color, or font from the surrounding text. -- Don't use the term Bazel as a verb or use it in possessive form. -- Don't use the Bazel logo on any website, product UI, or promotional materials without prior written permission from [product@bazel.build](mailto:product@bazel.build). +* The Bazel name may never be used or registered in a manner that would cause + confusion as to Google's sponsorship, affiliation, or endorsement. +* Don't use the Bazel name as part of your company name, product name, domain + name, or social media profile. +* Other than as permitted by these guidelines, the Bazel name should not be + combined with other trademarks, terms, or source identifiers. +* Don't remove, distort or alter any element of the Bazel Trademarks. That + includes modifying the Bazel Trademark, for example, through hyphenation, + combination or abbreviation. Do not shorten, abbreviate, or create acronyms + out of the Bazel Trademarks. +* Don't display the word Bazel using any different stylization, color, or font + from the surrounding text. +* Don't use the term Bazel as a verb or use it in possessive form. +* Don't use the Bazel logo on any website, product UI, or promotional + materials without prior written permission from + [product@bazel.build](mailto:product@bazel.build). ## Usage for Events and Community Groups -The Bazel word mark may be used referentially in events, community groups, or other gatherings related to the Bazel build system, but it may not be used in a manner that implies official status or endorsement. +The Bazel word mark may be used referentially in events, community groups, or +other gatherings related to the Bazel build system, but it may not be used in a +manner that implies official status or endorsement. Examples of appropriate naming conventions are: -- \[XYZ] Bazel User Group -- Bazel Community Day at \[XYZ] -- \[XYZ] Conference for Bazel Users +* \[XYZ\] Bazel User Group +* Bazel Community Day at \[XYZ\] +* \[XYZ\] Conference for Bazel Users -where \[XYZ] represents the location and optionally other wordings. +where \[XYZ\] represents the location and optionally other wordings. -Any naming convention that may imply official status or endorsement requires review for approval from [product@bazel.build](mailto:product@bazel.build). +Any naming convention that may imply official status or endorsement requires +review for approval from [product@bazel.build](mailto:product@bazel.build). Examples of naming conventions that require prior written permission: -- BazelCon -- Bazel Conference +* BazelCon +* Bazel Conference ## Contact Us -Please do not hesitate to contact us at [product@bazel.build](mailto:product@bazel.build) if you are unsure whether your intended use of the Bazel Trademarks is in compliance with these guidelines, or to ask for permission to use the Bazel Trademarks, clearly describing the intended usage and duration. +Please do not hesitate to contact us at +[product@bazel.build](mailto:product@bazel.build) if you are unsure whether your +intended use of the Bazel Trademarks is in compliance with these guidelines, or +to ask for permission to use the Bazel Trademarks, clearly describing the +intended usage and duration. diff --git a/build/share-variables.mdx b/build/share-variables.mdx index 3bf6683b..b248034e 100644 --- a/build/share-variables.mdx +++ b/build/share-variables.mdx @@ -2,9 +2,13 @@ title: 'Sharing Variables' --- -`BUILD` files are intended to be simple and declarative. They will typically consist of a series of target declarations. As your code base and your `BUILD` files get larger, you will probably notice some duplication, such as: -```python + +`BUILD` files are intended to be simple and declarative. They will typically +consist of a series of target declarations. As your code base and your `BUILD` +files get larger, you will probably notice some duplication, such as: + +``` python cc_library( name = "foo", copts = ["-DVERSION=5"], @@ -19,11 +23,17 @@ cc_library( ) ``` -Code duplication in `BUILD` files is usually fine. This can make the file more readable: each declaration can be read and understood without any context. This is important, not only for humans, but also for external tools. For example, a tool might be able to read and update `BUILD` files to add missing dependencies. Code refactoring and code reuse might prevent this kind of automated modification. +Code duplication in `BUILD` files is usually fine. This can make the file more +readable: each declaration can be read and understood without any context. This +is important, not only for humans, but also for external tools. For example, a +tool might be able to read and update `BUILD` files to add missing dependencies. +Code refactoring and code reuse might prevent this kind of automated +modification. -If it is useful to share values (for example, if values must be kept in sync), you can introduce a variable: +If it is useful to share values (for example, if values must be kept in sync), +you can introduce a variable: -```python +``` python COPTS = ["-DVERSION=5"] cc_library( @@ -40,21 +50,24 @@ cc_library( ) ``` -Multiple declarations now use the value `COPTS`. By convention, use uppercase letters to name global constants. +Multiple declarations now use the value `COPTS`. By convention, use uppercase +letters to name global constants. ## Sharing variables across multiple BUILD files -If you need to share a value across multiple `BUILD` files, you have to put it in a `.bzl` file. `.bzl` files contain definitions (variables and functions) that can be used in `BUILD` files. +If you need to share a value across multiple `BUILD` files, you have to put it +in a `.bzl` file. `.bzl` files contain definitions (variables and functions) +that can be used in `BUILD` files. In `path/to/variables.bzl`, write: -```python +``` python COPTS = ["-DVERSION=5"] ``` Then, you can update your `BUILD` files to access the variable: -```python +``` python load("//path/to:variables.bzl", "COPTS") cc_library( diff --git a/build/style-guide.mdx b/build/style-guide.mdx index d0a01fe0..cfc3cf86 100644 --- a/build/style-guide.mdx +++ b/build/style-guide.mdx @@ -2,17 +2,31 @@ title: 'BUILD Style Guide' --- + + ## Prefer DAMP BUILD files over DRY -The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by introducing abstractions such as variables and functions to avoid redundancy in code. +The DRY principle — "Don't Repeat Yourself" — encourages uniqueness by +introducing abstractions such as variables and functions to avoid redundancy in +code. -In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — encourages readability over uniqueness to make files easier to understand and maintain. +In contrast, the DAMP principle — "Descriptive and Meaningful Phrases" — +encourages readability over uniqueness to make files easier to understand and +maintain. -`BUILD` files aren't code, they are configurations. They aren't tested like code, but do need to be maintained by people and tools. That makes DAMP better for them than DRY. +`BUILD` files aren't code, they are configurations. They aren't tested like +code, but do need to be maintained by people and tools. That makes DAMP better +for them than DRY. ## BUILD.bazel file formatting -`BUILD` file formatting follows the same approach as Go, where a standardized tool takes care of most formatting issues. [Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and emits the source code in a standard style. Every `BUILD` file is therefore formatted in the same automated way, which makes formatting a non-issue during code reviews. It also makes it easier for tools to understand, edit, and generate `BUILD` files. +`BUILD` file formatting follows the same approach as Go, where a standardized +tool takes care of most formatting issues. +[Buildifier](https://github.com/bazelbuild/buildifier) is a tool that parses and +emits the source code in a standard style. Every `BUILD` file is therefore +formatted in the same automated way, which makes formatting a non-issue during +code reviews. It also makes it easier for tools to understand, edit, and +generate `BUILD` files. `BUILD` file formatting must match the output of `buildifier`. @@ -45,15 +59,18 @@ py_test( **Recommendation**: Use the following order (every element is optional): -- Package description (a comment) +* Package description (a comment) -- All `load()` statements +* All `load()` statements -- The `package()` function. +* The `package()` function. -- Calls to rules and macros +* Calls to rules and macros -Buildifier makes a distinction between a standalone comment and a comment attached to an element. If a comment is not attached to a specific element, use an empty line after it. The distinction is important when doing automated changes (for example, to keep or remove a comment when deleting a rule). +Buildifier makes a distinction between a standalone comment and a comment +attached to an element. If a comment is not attached to a specific element, use +an empty line after it. The distinction is important when doing automated +changes (for example, to keep or remove a comment when deleting a rule). ```python # Standalone comment (such as to make a section in a file) @@ -64,7 +81,11 @@ cc_library(name = "cc") ## References to targets in the current package -Files should be referred to by their paths relative to the package directory (without ever using up-references, such as `..`). Generated files should be prefixed with "`:`" to indicate that they are not sources. Source files should not be prefixed with `:`. Rules should be prefixed with `:`. For example, assuming `x.cc` is a source file: +Files should be referred to by their paths relative to the package directory +(without ever using up-references, such as `..`). Generated files should be +prefixed with "`:`" to indicate that they are not sources. Source files +should not be prefixed with `:`. Rules should be prefixed with `:`. For +example, assuming `x.cc` is a source file: ```python cc_library( @@ -77,70 +98,100 @@ genrule( name = "gen_header", srcs = [], outs = ["x.h"], - cmd = "echo 'int x();' > $@", + cmd = "echo 'int x();' > $@", ) ``` ## Target naming -Target names should be descriptive. If a target contains one source file, the target should generally have a name derived from that source (for example, a `cc_library` for `chat.cc` could be named `chat`, or a `java_library` for `DirectMessage.java` could be named `direct_message`). - -The eponymous target for a package (the target with the same name as the containing directory) should provide the functionality described by the directory name. If there is no such target, do not create an eponymous target. - -Prefer using the short name when referring to an eponymous target (`//x` instead of `//x:x`). If you are in the same package, prefer the local reference (`:x` instead of `//x`). - -Avoid using "reserved" target names which have special meaning. This includes `all`, `__pkg__`, and `__subpackages__`, these names have special semantics and can cause confusion and unexpected behaviors when they are used. - -In the absence of a prevailing team convention these are some non-binding recommendations that are broadly used at Google: - -- In general, use ["snake\_case"](https://en.wikipedia.org/wiki/Snake_case) - - - For a `java_library` with one `src` this means using a name that is not the same as the filename without the extension - - For Java `*_binary` and `*_test` rules, use ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). This allows for the target name to match one of the `src`s. For `java_test`, this makes it possible for the `test_class` attribute to be inferred from the name of the target. - -- If there are multiple variants of a particular target then add a suffix to disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) - -- Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` - -- Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to avoid conflicts between a `_library` target and its corresponding `_binary`) - -- For proto related targets: - - - `proto_library` targets should have names ending in `_proto` - - - Languages specific `*_proto_library` rules should match the underlying proto but replace `_proto` with a language specific suffix such as: - - - **`cc_proto_library`**: `_cc_proto` - - **`java_proto_library`**: `_java_proto` - - **`java_lite_proto_library`**: `_java_proto_lite` +Target names should be descriptive. If a target contains one source file, +the target should generally have a name derived from that source (for example, a +`cc_library` for `chat.cc` could be named `chat`, or a `java_library` for +`DirectMessage.java` could be named `direct_message`). + +The eponymous target for a package (the target with the same name as the +containing directory) should provide the functionality described by the +directory name. If there is no such target, do not create an eponymous +target. + +Prefer using the short name when referring to an eponymous target (`//x` +instead of `//x:x`). If you are in the same package, prefer the local +reference (`:x` instead of `//x`). + +Avoid using "reserved" target names which have special meaning. This includes +`all`, `__pkg__`, and `__subpackages__`, these names have special +semantics and can cause confusion and unexpected behaviors when they are used. + +In the absence of a prevailing team convention these are some non-binding +recommendations that are broadly used at Google: + +* In general, use ["snake_case"](https://en.wikipedia.org/wiki/Snake_case) + * For a `java_library` with one `src` this means using a name that is not + the same as the filename without the extension + * For Java `*_binary` and `*_test` rules, use + ["Upper CamelCase"](https://en.wikipedia.org/wiki/Camel_case). + This allows for the target name to match one of the `src`s. For + `java_test`, this makes it possible for the `test_class` attribute to be + inferred from the name of the target. +* If there are multiple variants of a particular target then add a suffix to + disambiguate (such as. `:foo_dev`, `:foo_prod` or `:bar_x86`, `:bar_x64`) +* Suffix `_test` targets with `_test`, `_unittest`, `Test`, or `Tests` +* Avoid meaningless suffixes like `_lib` or `_library` (unless necessary to + avoid conflicts between a `_library` target and its corresponding `_binary`) +* For proto related targets: + * `proto_library` targets should have names ending in `_proto` + * Languages specific `*_proto_library` rules should match the underlying + proto but replace `_proto` with a language specific suffix such as: + * **`cc_proto_library`**: `_cc_proto` + * **`java_proto_library`**: `_java_proto` + * **`java_lite_proto_library`**: `_java_proto_lite` ## Visibility -Visibility should be scoped as tightly as possible, while still allowing access by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as appropriate. +Visibility should be scoped as tightly as possible, while still allowing access +by tests and reverse dependencies. Use `__pkg__` and `__subpackages__` as +appropriate. -Avoid setting package `default_visibility` to `//visibility:public`. `//visibility:public` should be individually set only for targets in the project's public API. These could be libraries that are designed to be depended on by external projects or binaries that could be used by an external project's build process. +Avoid setting package `default_visibility` to `//visibility:public`. +`//visibility:public` should be individually set only for targets in the +project's public API. These could be libraries that are designed to be depended +on by external projects or binaries that could be used by an external project's +build process. ## Dependencies -Dependencies should be restricted to direct dependencies (dependencies needed by the sources listed in the rule). Do not list transitive dependencies. +Dependencies should be restricted to direct dependencies (dependencies +needed by the sources listed in the rule). Do not list transitive dependencies. -Package-local dependencies should be listed first and referred to in a way compatible with the [References to targets in the current package](#targets-current-package) section above (not by their absolute package name). +Package-local dependencies should be listed first and referred to in a way +compatible with the +[References to targets in the current package](#targets-current-package) +section above (not by their absolute package name). -Prefer to list dependencies directly, as a single list. Putting the "common" dependencies of several targets into a variable reduces maintainability, makes it impossible for tools to change the dependencies of a target, and can lead to unused dependencies. +Prefer to list dependencies directly, as a single list. Putting the "common" +dependencies of several targets into a variable reduces maintainability, makes +it impossible for tools to change the dependencies of a target, and can lead to +unused dependencies. ## Globs -Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it is more error-prone and less obvious than an empty list. +Indicate "no targets" with `[]`. Do not use a glob that matches nothing: it +is more error-prone and less obvious than an empty list. ### Recursive -Do not use recursive globs to match source files (for example, `glob(["**/*.java"])`). +Do not use recursive globs to match source files (for example, +`glob(["**/*.java"])`). -Recursive globs make `BUILD` files difficult to reason about because they skip subdirectories containing `BUILD` files. +Recursive globs make `BUILD` files difficult to reason about because they skip +subdirectories containing `BUILD` files. -Recursive globs are generally less efficient than having a `BUILD` file per directory with a dependency graph defined between them as this enables better remote caching and parallelism. +Recursive globs are generally less efficient than having a `BUILD` file per +directory with a dependency graph defined between them as this enables better +remote caching and parallelism. -It is good practice to author a `BUILD` file in each directory and define a dependency graph between them. +It is good practice to author a `BUILD` file in each directory and define a +dependency graph between them. ### Non-recursive @@ -148,16 +199,21 @@ Non-recursive globs are generally acceptable. ## Avoid list comprehensions -Avoid using list comprehensions at the top level of a `BUILD.bazel` file. Automate repetitive calls by creating each named target with a separate top-level rule or macro call. Give each a short `name` parameter for clarity. +Avoid using list comprehensions at the top level of a `BUILD.bazel` file. +Automate repetitive calls by creating each named target with a separate +top-level rule or macro call. Give each a short `name` parameter for clarity. List comprehension reduces the following: -- Maintainability. It's difficult or impossible for human maintainers and large scale automated changes to update list comprehensions correctly. -- Discoverability. Since the pattern doesn't have `name` parameters, it's hard to find the rule by name. +* Maintainability. It's difficult or impossible for human maintainers and + large scale automated changes to update list comprehensions correctly. +* Discoverability. Since the pattern doesn't have `name` parameters, + it's hard to find the rule by name. -A common application of the list comprehension pattern is to generate tests. For example: +A common application of the list comprehension pattern is to generate tests. For +example: -```build +```build {.bad} [[java_test( name = "test_%s_%s" % (backend, count), srcs = [ ... ], @@ -172,7 +228,8 @@ A common application of the list comprehension pattern is to generate tests. For ]] ``` -We recommend using simpler alternatives. For example, define a macro that generates one test and invoke it for each top-level `name`: +We recommend using simpler alternatives. For example, define a macro that +generates one test and invoke it for each top-level `name`: ```build my_java_test(name = "test_fake_1", @@ -186,7 +243,7 @@ my_java_test(name = "test_fake_10", Don't use list variables to encapsulate common dependencies: -```build +```build {.bad} COMMON_DEPS = [ "//d:e", "//x/y:z", @@ -203,11 +260,12 @@ cc_library(name = "b", ) ``` -Similarly, don't use a library target with [`exports`](/reference/be/java#java_library.exports) to group dependencies. +Similarly, don't use a library target with +[`exports`](/reference/be/java#java_library.exports) to group dependencies. Instead, list the dependencies separately for each target: -```build +```build {.good} cc_library(name = "a", srcs = ["a.cc"], deps = [ @@ -227,25 +285,38 @@ cc_library(name = "b", ) ``` -Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools maintain them. There will be repetition, but you won't have to think about how to manage the dependencies. +Let [Gazelle](https://github.com/bazel-contrib/bazel-gazelle) and other tools +maintain them. There will be repetition, but you won't have to think about how +to manage the dependencies. ## Prefer literal strings -Although Starlark provides string operators for concatenation (`+`) and formatting (`%`), use them with caution. It is tempting to factor out common string parts to make expressions more concise or break long lines. However, +Although Starlark provides string operators for concatenation (`+`) and +formatting (`%`), use them with caution. It is tempting to factor out common +string parts to make expressions more concise or break long lines. However, -- It is harder to read broken-up string values at a glance. +* It is harder to read broken-up string values at a glance. -- Automated tools such as [buildozer](https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md) and Code Search have trouble finding values and updating them correctly when the values broken up. +* Automated tools such as + [buildozer][buildozer] and Code Search have trouble finding values and + updating them correctly when the values broken up. -- In `BUILD` files, readability is more important than avoiding repetition (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). +* In `BUILD` files, readability is more important than avoiding repetition + (see [DAMP versus DRY](#prefer-damp-build-files-over-dry)). -- This Style Guide [warns against splitting label-valued strings](#other-conventions) and [explicitly permits long lines](#differences-python-style-guide). +* This Style Guide + [warns against splitting label-valued strings](#other-conventions) + and + [explicitly permits long lines](#differences-python-style-guide). -- Buildifier automatically fuses concatenated strings when it detects that they are labels. +* Buildifier automatically fuses concatenated strings when it detects that + they are labels. -Therefore, prefer explicit, literal strings over concatenated or formatted strings, especially in label-type attributes such as `name` and `deps`. For example, this `BUILD` fragment: +Therefore, prefer explicit, literal strings over concatenated or formatted +strings, especially in label-type attributes such as `name` and `deps`. For +example, this `BUILD` fragment: -```build +```build {.bad} NAME = "foo" PACKAGE = "//a/b" @@ -259,7 +330,7 @@ proto_library( would be better rewritten as -```build +```build {.good} proto_library( name = "foo_proto", deps = ["//a/b:other_proto"], @@ -267,32 +338,66 @@ proto_library( ) ``` +[buildozer]: https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md + ## Limit the symbols exported by each `.bzl` file -Minimize the number of symbols (rules, macros, constants, functions) exported by each public `.bzl` (Starlark) file. We recommend that a file should export multiple symbols only if they are certain to be used together. Otherwise, split it into multiple `.bzl` files, each with its own [bzl\_library](https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library). +Minimize the number of symbols (rules, macros, constants, functions) exported by +each public `.bzl` (Starlark) file. We recommend that a file should export +multiple symbols only if they are certain to be used together. Otherwise, split +it into multiple `.bzl` files, each with its own [bzl_library][bzl_library]. -Excessive symbols can cause `.bzl` files to grow into broad "libraries" of symbols, causing changes to single files to force Bazel to rebuild many targets. +Excessive symbols can cause `.bzl` files to grow into broad "libraries" of +symbols, causing changes to single files to force Bazel to rebuild many targets. + +[bzl_library]: https://github.com/bazelbuild/bazel-skylib/blob/main/README.md#bzl_library ## Other conventions -- Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), use lowercase and underscores to declare variables (such as `my_variable`). + * Use uppercase and underscores to declare constants (such as `GLOBAL_CONSTANT`), + use lowercase and underscores to declare variables (such as `my_variable`). -- Labels should never be split, even if they are longer than 79 characters. Labels should be string literals whenever possible. *Rationale*: It makes find and replace easy. It also improves readability. + * Labels should never be split, even if they are longer than 79 characters. + Labels should be string literals whenever possible. *Rationale*: It makes + find and replace easy. It also improves readability. -- The value of the name attribute should be a literal constant string (except in macros). *Rationale*: External tools use the name attribute to refer a rule. They need to find rules without having to interpret code. + * The value of the name attribute should be a literal constant string (except + in macros). *Rationale*: External tools use the name attribute to refer a + rule. They need to find rules without having to interpret code. -- When setting boolean-type attributes, use boolean values, not integer values. For legacy reasons, rules still convert integers to booleans as needed, but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying "deflake this target by rerunning it once". `flaky = True` unambiguously says "this test is flaky". + * When setting boolean-type attributes, use boolean values, not integer values. + For legacy reasons, rules still convert integers to booleans as needed, + but this is discouraged. *Rationale*: `flaky = 1` could be misread as saying + "deflake this target by rerunning it once". `flaky = True` unambiguously says + "this test is flaky". ## Differences with Python style guide -Although compatibility with [Python style guide](https://www.python.org/dev/peps/pep-0008/) is a goal, there are a few differences: - -- No strict line length limit. Long comments and long strings are often split to 79 columns, but it is not required. It should not be enforced in code reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this limit. It is common for `BUILD` files to be generated or edited by tools, which does not go well with a line length limit. - -- Implicit string concatenation is not supported. Use the `+` operator. *Rationale*: `BUILD` files contain many string lists. It is easy to forget a comma, which leads to a complete different result. This has created many bugs in the past. [See also this discussion.](https://lwn.net/Articles/551438/) - -- Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: Named arguments are much more frequent than in Python and are always on a separate line. Spaces improve readability. This convention has been around for a long time, and it is not worth modifying all existing `BUILD` files. - -- By default, use double quotation marks for strings. *Rationale*: This is not specified in the Python style guide, but it recommends consistency. So we decided to use only double-quoted strings. Many languages use double-quotes for string literals. - -- Use a single blank line between two top-level definitions. *Rationale*: The structure of a `BUILD` file is not like a typical Python file. It has only top-level statements. Using a single-blank line makes `BUILD` files shorter. +Although compatibility with +[Python style guide](https://www.python.org/dev/peps/pep-0008/) +is a goal, there are a few differences: + + * No strict line length limit. Long comments and long strings are often split + to 79 columns, but it is not required. It should not be enforced in code + reviews or presubmit scripts. *Rationale*: Labels can be long and exceed this + limit. It is common for `BUILD` files to be generated or edited by tools, + which does not go well with a line length limit. + + * Implicit string concatenation is not supported. Use the `+` operator. + *Rationale*: `BUILD` files contain many string lists. It is easy to forget a + comma, which leads to a complete different result. This has created many bugs + in the past. [See also this discussion.](https://lwn.net/Articles/551438/) + + * Use spaces around the `=` sign for keywords arguments in rules. *Rationale*: + Named arguments are much more frequent than in Python and are always on a + separate line. Spaces improve readability. This convention has been around + for a long time, and it is not worth modifying all existing `BUILD` files. + + * By default, use double quotation marks for strings. *Rationale*: This is not + specified in the Python style guide, but it recommends consistency. So we + decided to use only double-quoted strings. Many languages use double-quotes + for string literals. + + * Use a single blank line between two top-level definitions. *Rationale*: The + structure of a `BUILD` file is not like a typical Python file. It has only + top-level statements. Using a single-blank line makes `BUILD` files shorter. diff --git a/community/recommended-rules.mdx b/community/recommended-rules.mdx index 426d8784..86daa056 100644 --- a/community/recommended-rules.mdx +++ b/community/recommended-rules.mdx @@ -2,33 +2,53 @@ title: 'Recommended Rules' --- -In the documentation, we provide a list of [recommended rules](/rules). -This is a set of high quality rules, which will provide a good experience to our users. We make a distinction between the supported rules, and the hundreds of rules you can find on the Internet. + +In the documentation, we provide a list of +[recommended rules](/rules). + +This is a set of high quality rules, which will provide a good experience to our +users. We make a distinction between the supported rules, and the hundreds of +rules you can find on the Internet. ## Nomination -If a ruleset meets the requirements below, a rule maintainer can nominate it to be part of the *recommended rules* by filing a [GitHub issue](https://github.com/bazelbuild/bazel/). +If a ruleset meets the requirements below, a rule maintainer can nominate it +to be part of the _recommended rules_ by filing a +[GitHub issue](https://github.com/bazelbuild/bazel/). -After a review by the [Bazel core team](/contribute/policy), it will be recommended on the Bazel website. +After a review by the [Bazel core team](/contribute/policy), it +will be recommended on the Bazel website. ## Requirements for the rule maintainers -- The ruleset provides an important feature, useful to a large number of Bazel users (for example, support for a widely popular language). -- The ruleset is well maintained. There must be at least two active maintainers. -- The ruleset is well documented, with examples, and easy to use. -- The ruleset follows the best practices and is performant (see [the performance guide](/rules/performance)). -- The ruleset has sufficient test coverage. -- The ruleset is tested on [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) with the latest version of Bazel. Tests should always pass (when used as a presubmit check). -- The ruleset is also tested with the upcoming incompatible changes. Breakages should be fixed within two weeks. Migration issues should be reported to the Bazel team quickly. +* The ruleset provides an important feature, useful to a large number of Bazel + users (for example, support for a widely popular language). +* The ruleset is well maintained. There must be at least two active maintainers. +* The ruleset is well documented, with examples, and easy to use. +* The ruleset follows the best practices and is performant (see + [the performance guide](/rules/performance)). +* The ruleset has sufficient test coverage. +* The ruleset is tested on + [BuildKite](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) + with the latest version of Bazel. Tests should always pass (when used as a + presubmit check). +* The ruleset is also tested with the upcoming incompatible changes. Breakages + should be fixed within two weeks. Migration issues should be reported to the + Bazel team quickly. ## Requirements for Bazel developers -- Recommended rules are frequently tested with Bazel at head (at least once a day). -- No change in Bazel may break a recommended rule (with the default set of flags). If it happens, the change should be fixed or rolled back. +* Recommended rules are frequently tested with Bazel at head (at least once a + day). +* No change in Bazel may break a recommended rule (with the default set of + flags). If it happens, the change should be fixed or rolled back. ## Demotion -If there is a concern that a particular ruleset is no longer meeting the requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be filed. +If there is a concern that a particular ruleset is no longer meeting the +requirements, a [GitHub issue](https://github.com/bazelbuild/bazel/) should be +filed. -Rule maintainers will be contacted and need to respond in 2 weeks. Based on the outcome, Bazel core team might make a decision to demote the rule set. +Rule maintainers will be contacted and need to respond in 2 weeks. Based on the +outcome, Bazel core team might make a decision to demote the rule set. diff --git a/community/remote-execution-services.mdx b/community/remote-execution-services.mdx index 7a971881..6dee80fa 100644 --- a/community/remote-execution-services.mdx +++ b/community/remote-execution-services.mdx @@ -2,23 +2,28 @@ title: 'Remote Execution Services' --- + + Use the following services to run Bazel with remote execution: -- Manual +* Manual - - Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) directly to create your own remote execution service. + * Use the [gRPC protocol](https://github.com/bazelbuild/remote-apis) + directly to create your own remote execution service. -- Self-service +* Self-service - - [Buildbarn](https://github.com/buildbarn) - - [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - - [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - - [NativeLink](https://github.com/TraceMachina/nativelink) + * [Buildbarn](https://github.com/buildbarn) + * [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) + * [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) + * [NativeLink](https://github.com/TraceMachina/nativelink) -- Commercial +* Commercial - - [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. - - [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. - - [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, caching, and results UI. - - [EngFlow Remote Execution](https://www.engflow.com) - Remote execution and remote caching service with Build and Test UI. Can be self-hosted or hosted. - - [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. + * [Aspect Build](https://www.aspect.build/) – Self-hosted remote cache and remote execution services. + * [Bitrise](https://bitrise.io/why/features/mobile-build-caching-for-better-build-test-performance) - Providing the world's leading mobile-first CI/CD and remote build caching platform. + * [BuildBuddy](https://www.buildbuddy.io) - Remote build execution, + caching, and results UI. + * [EngFlow Remote Execution](https://www.engflow.com) - Remote execution + and remote caching service with Build and Test UI. Can be self-hosted or hosted. + * [NativeLink](https://github.com/TraceMachina/nativelink) - Remote build execution, caching, analytics, and simulation. diff --git a/community/sig.mdx b/community/sig.mdx index 33ef6b15..ae5f9189 100644 --- a/community/sig.mdx +++ b/community/sig.mdx @@ -2,23 +2,42 @@ title: 'Bazel Special Interest Groups' --- -Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular areas and to support communication and coordination between [Bazel owners, maintainers, and contributors](/contribute/policy). This policy applies to [`bazelbuild`](http://github.com/bazelbuild). -SIGs do their work in public. The ideal scope for a SIG covers a well-defined domain, where the majority of participation is from the community. SIGs may focus on community maintained repositories in `bazelbuild` (such as language rules) or focus on areas of code in the Bazel repository (such as Remote Execution). -While not all SIGs will have the same level of energy, breadth of scope, or governance models, there should be sufficient evidence that there are community members willing to engage and contribute should the interest group be established. Before joining, review the group's work, and then get in touch with the SIG leader. Membership policies vary on a per-SIG basis. +Bazel hosts Special Interest Groups (SIGs) to focus collaboration on particular +areas and to support communication and coordination between [Bazel owners, +maintainers, and contributors](/contribute/policy). This policy +applies to [`bazelbuild`](http://github.com/bazelbuild). -See the complete list of [Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). +SIGs do their work in public. The ideal scope for a SIG covers a well-defined +domain, where the majority of participation is from the community. SIGs may +focus on community maintained repositories in `bazelbuild` (such as language +rules) or focus on areas of code in the Bazel repository (such as Remote +Execution). + +While not all SIGs will have the same level of energy, breadth of scope, or +governance models, there should be sufficient evidence that there are community +members willing to engage and contribute should the interest group be +established. Before joining, review the group's work, and then get in touch +with the SIG leader. Membership policies vary on a per-SIG basis. + +See the complete list of +[Bazel SIGs](https://github.com/bazelbuild/community/tree/main/sigs). ### Non-goals: What a SIG is not -SIGs are intended to facilitate collaboration on shared work. A SIG is therefore: +SIGs are intended to facilitate collaboration on shared work. A SIG is +therefore: -- *Not a support forum:* a mailing list and a SIG is not the same thing -- *Not immediately required:* early on in a project's life, you may not know if you have shared work or collaborators -- *Not free labor:* energy is required to grow and coordinate the work collaboratively +- *Not a support forum:* a mailing list and a SIG is not the same thing +- *Not immediately required:* early on in a project's life, you may not know + if you have shared work or collaborators +- *Not free labor:* energy is required to grow and coordinate the work + collaboratively -Bazel Owners take a conservative approach to SIG creation—thanks to the ease of starting projects on GitHub, there are many avenues where collaboration can happen without the need for a SIG. +Bazel Owners take a conservative approach to SIG creation—thanks to the ease of +starting projects on GitHub, there are many avenues where collaboration can +happen without the need for a SIG. ## SIG lifecycle @@ -26,65 +45,114 @@ This section covers how to create a SIG. ### Research and consultation -To propose a new SIG group, first gather evidence for approval, as specified below. Some possible avenues to consider are: +To propose a new SIG group, first gather evidence for approval, as specified +below. Some possible avenues to consider are: -- A well-defined problem or set of problems the group would solve -- Consultation with community members who would benefit, assessing both the benefit and their willingness to commit -- For existing projects, evidence from issues and PRs that contributors care about the topic -- Potential goals for the group to achieve -- Resource requirements of running the group +- A well-defined problem or set of problems the group would solve +- Consultation with community members who would benefit, assessing both the + benefit and their willingness to commit +- For existing projects, evidence from issues and PRs that contributors care + about the topic +- Potential goals for the group to achieve +- Resource requirements of running the group -Even if the need for a SIG seems self-evident, the research and consultation is still important to the success of the group. +Even if the need for a SIG seems self-evident, the research and consultation is +still important to the success of the group. ### Create the new group -The new group should follow the below process for chartering. In particular, it must demonstrate: - -- A clear purpose and benefit to Bazel (either around a sub-project or application area) -- Two or more contributors willing to act as group leads, existence of other contributors, and evidence of demand for the group -- Each group needs to use at least one publicly accessible mailing list. A SIG may reuse one of the public lists, such as [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list for @bazel.build, or create their own list -- Resources the SIG initially requires (usually, mailing list and regular video call.) -- SIGs can serve documents and files from their directory in [`bazelbuild/community`](https://github.com/bazelbuild/community) or from their own repository in the [`bazelbuild`](https://github.com/bazelbuild) GitHub organization. SIGs may link to external resources if they choose to organize their work outside of the `bazelbuild` GitHub organization -- Bazel Owners approve or reject SIG applications and consult other stakeholders as necessary - -Before entering the formal parts of the process, you should consult with the Bazel product team, at [product@bazel.build](mailto:product@bazel.build). Most SIGs require conversation and iteration before approval. - -The formal request for the new group is done by submitting a charter as a PR to [`bazelbuild/community`](https://github.com/bazelbuild/community), and including the request in the comments on the PR following the template below. On approval, the PR for the group is merged and the required resources created. +The new group should follow the below process for chartering. In particular, it +must demonstrate: + +- A clear purpose and benefit to Bazel (either around a sub-project or + application area) +- Two or more contributors willing to act as group leads, existence of other + contributors, and evidence of demand for the group +- Each group needs to use at least one publicly accessible mailing list. A SIG + may reuse one of the public lists, such as + [bazel-discuss](https://groups.google.com/g/bazel-discuss), ask for a list + for @bazel.build, or create their own list +- Resources the SIG initially requires (usually, mailing list and regular + video call.) +- SIGs can serve documents and files from their directory in + [`bazelbuild/community`](https://github.com/bazelbuild/community) + or from their own repository in the + [`bazelbuild`](https://github.com/bazelbuild) GitHub + organization. SIGs may link to external resources if they choose to organize + their work outside of the `bazelbuild` GitHub organization +- Bazel Owners approve or reject SIG applications and consult other + stakeholders as necessary + +Before entering the formal parts of the process, you should consult with +the Bazel product team, at product@bazel.build. Most SIGs require conversation +and iteration before approval. + +The formal request for the new group is done by submitting a charter as a PR to +[`bazelbuild/community`](https://github.com/bazelbuild/community), +and including the request in the comments on the PR following the template +below. On approval, the PR for the group is merged and the required resources +created. ### Template Request for New SIG -To request a new SIG, use the template in the community repo: [SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). +To request a new SIG, use the template in the community repo: +[SIG-request-template.md](https://github.com/bazelbuild/community/blob/main/governance/SIG-request-template.md). ### Chartering -To establish a group, you need a charter and must follow the Bazel [code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). Archives of the group will be public. Membership may either be open to all without approval, or available on request, pending approval of the group administrator. +To establish a group, you need a charter and must follow the Bazel +[code of conduct](https://github.com/bazelbuild/bazel/blob/HEAD/CODE_OF_CONDUCT.md). +Archives of the group will be public. Membership may either be open to all +without approval, or available on request, pending approval of the group +administrator. -The charter must nominate an administrator. As well as an administrator, the group must include at least one person as lead (these may be the same person), who serves as point of contact for coordination as required with the Bazel product team. +The charter must nominate an administrator. As well as an administrator, the +group must include at least one person as lead (these may be the same person), +who serves as point of contact for coordination as required with the Bazel +product team. -Group creators must post their charter to the group mailing list. The community repository in the Bazel GitHub organization archives such documents and policies. As groups evolve their practices and conventions, they should update their charters within the relevant part of the community repository. +Group creators must post their charter to the group mailing list. The community +repository in the Bazel GitHub organization archives such documents and +policies. As groups evolve their practices and conventions, they should update +their charters within the relevant part of the community repository. ### Collaboration and inclusion -While not mandated, the group should choose to make use of collaboration via scheduled conference calls or chat channels to conduct meetings. Any such meetings should be advertised on the mailing list, and notes posted to the mailing list afterwards. Regular meetings help drive accountability and progress in a SIG. +While not mandated, the group should choose to make use of collaboration +via scheduled conference calls or chat channels to conduct meetings. Any such +meetings should be advertised on the mailing list, and notes posted to the +mailing list afterwards. Regular meetings help drive accountability and progress +in a SIG. -Bazel product team members may proactively monitor and encourage the group to discussion and action as appropriate. +Bazel product team members may proactively monitor and encourage the group to +discussion and action as appropriate. ### Launch a SIG Required activities: -- Notify Bazel general discussion groups ([bazel-discuss](https://groups.google.com/g/bazel-discuss), [bazel-dev](https://groups.google.com/g/bazel-dev)). +- Notify Bazel general discussion groups + ([bazel-discuss](https://groups.google.com/g/bazel-discuss), + [bazel-dev](https://groups.google.com/g/bazel-dev)). Optional activities: -- Create a blog post for the Bazel blog +- Create a blog post for the Bazel blog ### Health and termination of SIGs -The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners occasionally request the SIG lead to report on the SIG's work, to inform the broader Bazel community of the group's activity. +The Bazel owners make a best effort to ensure the health of SIGs. Bazel owners +occasionally request the SIG lead to report on the SIG's work, to inform the +broader Bazel community of the group's activity. -If a SIG no longer has a useful purpose or interested community, it may be archived and cease operation. The Bazel product team reserves the right to archive such inactive SIGs to maintain the overall health of the project, though it is a less preferable outcome. A SIG may also opt to disband if it recognizes it has reached the end of its useful life. +If a SIG no longer has a useful purpose or interested community, it may be +archived and cease operation. The Bazel product team reserves the right to +archive such inactive SIGs to maintain the overall health of the project, +though it is a less preferable outcome. A SIG may also opt to disband if +it recognizes it has reached the end of its useful life. ## Note -*This content has been adopted from Tensorflow’s [SIG playbook](https://www.tensorflow.org/community/sig_playbook) with modifications.* +*This content has been adopted from Tensorflow’s +[SIG playbook](https://www.tensorflow.org/community/sig_playbook) +with modifications.* diff --git a/community/users.mdx b/community/users.mdx index b7cf01fc..91e26c47 100644 --- a/community/users.mdx +++ b/community/users.mdx @@ -2,189 +2,276 @@ title: 'Who''s Using Bazel' --- -Note: Using Bazel? You can add your company on [StackShare](https://stackshare.io/bazel). To add yourself to this page, contact [product@bazel.build](mailto:product@bazel.build). -This page lists companies and OSS projects that are known to use Bazel. This does not constitute an endorsement. + +Note: Using Bazel? You can add your company on +[StackShare](https://stackshare.io/bazel). To add yourself to this page, +contact [product@bazel.build](mailto:product@bazel.build). + +This page lists companies and OSS projects that are known to use Bazel. +This does not constitute an endorsement. ## Companies using Bazel ### [acqio](https://acqio.com.br) -![](/community/images/acqio_logo.svg) + -Acqio is a Fintech that provides payment products and services for small and medium merchants. Acqio has a handful of monorepos and uses Bazel along with Kubernetes to deliver fast and reliable microservices. +Acqio is a Fintech that provides payment products and services for small and +medium merchants. Acqio has a handful of monorepos and uses Bazel along with +Kubernetes to deliver fast and reliable microservices. ### [Adobe](https://www.adobe.com/) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Adobe_logo_and_wordmark_%282017%29.svg/440px-Adobe_logo_and_wordmark_%282017%29.svg.png) + -Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for continuous, GitOps driven Kubernetes deployments. +Adobe has released Bazel [rules](https://github.com/adobe/rules_gitops) for +continuous, GitOps driven Kubernetes deployments. ### [Asana](https://asana.com) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Asana_logo.svg/256px-Asana_logo.svg.png) + -Asana is a web and mobile application designed to help teams track their work. In their own words: +Asana is a web and mobile application designed to help teams track their work. +In their own words: -> Bazel has increased reliability, stability, and speed for all of builds/tests at Asana. We no longer need to clean because of incorrect caches. +> Bazel has increased reliability, stability, and speed for all of builds/tests +at Asana. We no longer need to clean because of incorrect caches. ### [Ascend.io](https://ascend.io) -Ascend is a Palo Alto startup that offers solutions for large data sets analysis. Their motto is *Big data is hard. We make it easy*. +Ascend is a Palo Alto startup that offers solutions for large data sets +analysis. Their motto is _Big data is hard. We make it easy_. ### [ASML](https://asml.com) -![](https://upload.wikimedia.org/wikipedia/en/6/6c/ASML_Holding_N.V._logo.svg) + -ASML is an innovation leader in the semiconductor industry. We provide chipmakers with everything they need – hardware, software and services – to mass produce patterns on silicon through lithography. +ASML is an innovation leader in the semiconductor industry. We provide chipmakers +with everything they need – hardware, software and services – to mass produce +patterns on silicon through lithography. ### [Beeswax](https://www.beeswax.com/) -> Beeswax is a New York based startup that provides real time bidding as service. Bazel powers their Jenkins based continuous integration and deployment framework. Beeswax loves Bazel because it is blazingly fast, correct and well supported across many languages and platforms. +> Beeswax is a New York based startup that provides real time bidding as +service. Bazel powers their Jenkins based continuous integration and deployment +framework. Beeswax loves Bazel because it is blazingly fast, correct and well +supported across many languages and platforms. ### [Braintree](https://www.braintreepayments.com) -![](https://upload.wikimedia.org/wikipedia/commons/0/00/Braintree-logo1.png) + -Braintree, a PayPal subsidiary, develops payment solutions for websites and applications. They use Bazel for parts of their internal build and Paul Gross even posted a [nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). +Braintree, a PayPal subsidiary, develops payment solutions for websites and +applications. They use Bazel for parts of their internal build and Paul Gross +even posted a +[nice piece about how their switch to Bazel went](https://www.pgrs.net/2015/09/01/migrating-from-gradle-to-bazel/). ### [Canva](https://www.canva.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/b/bb/Canva_Logo.svg) - -Canva leverages Bazel to manage its large polyglot codebase, which includes Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered significant developer and compute infrastructure efficiencies, for example 5-6x decreases in average CI build times, and it continues to become the foundation of fast, reproducible, and standardised software builds at the company. +Canva leverages Bazel to manage its large polyglot codebase, which includes +Java, TypeScript, Scala, Python, and more. Migration to Bazel has delivered +significant developer and compute infrastructure efficiencies, for example 5-6x +decreases in average CI build times, and it continues to become the foundation +of fast, reproducible, and standardised software builds at the company. ### [CarGurus](https://www.cargurus.com) + -![](https://www.cargurus.com/gfx/reskin/logos/logo_CarGurus.svg) - -CarGurus is on a mission to build the world's most trusted and transparent automotive marketplace and uses Bazel to build their polyglot monorepo. +CarGurus is on a mission to build the world's most trusted and transparent +automotive marketplace and uses Bazel to build their polyglot monorepo. ### [Compass](https://www.compass.com) -Compass is a tech-driven real estate platform. With an elite team of real estate, technology and business professionals, we aim to be the best and most trusted source for home seekers. +Compass is a tech-driven real estate platform. With an elite team of real +estate, technology and business professionals, we aim to be the best and most +trusted source for home seekers. ### [Databricks](https://databricks.com) -![](https://databricks.com/wp-content/uploads/2021/10/db-nav-logo.svg) Databricks provides cloud-based integrated workspaces based on Apache Spark™. + +Databricks provides cloud-based integrated workspaces based on Apache Spark™. -> The Databricks codebase is a Monorepo, containing the Scala code that powers most of our services, Javascript for front-end UI, Python for scripting, Jsonnet to configure our infrastructure, and much more \[...] Even though our monorepo contains a million lines of Scala, working with code within is fast and snappy. ([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) +> The Databricks codebase is a Monorepo, containing the Scala code that powers +most of our services, Javascript for front-end UI, Python for scripting, +Jsonnet to configure our infrastructure, and much more [...] Even though our +monorepo contains a million lines of Scala, working with code within is fast +and snappy. +([Speedy Scala Builds with Bazel at Databricks](https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html)) ### [Dataform](https://dataform.co) -Dataform provides scalable analytics for data teams. They maintain a handful of NPM packages and a documentation site in one single monorepo and they do it all with Bazel. +Dataform provides scalable analytics for data teams. They maintain a handful of +NPM packages and a documentation site in one single monorepo and they do it all +with Bazel. -After the migration to Bazel, they [reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), including: +After the migration to Bazel, they +[reported many benefits](https://github.com/bazelbuild/rules_nodejs#user-testimonials), +including: -> - Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). -> - Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes -> - Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop +> * Faster CI: we enabled the remote build caching which has reduced our average build time from 30 minutes to 5 (for the entire repository). +> * Improvements to local development: no more random bash scripts that you forget to run, incremental builds reduced to seconds from minutes +> * Developer setup time: New engineers can build all our code with just 3 dependencies - bazel, docker and the JVM. The last engineer to join our team managed to build all our code in < 30 minutes on a brand new, empty laptop ### [Deep Silver FISHLABS](https://www.dsfishlabs.com) - -Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with C++/Python/Go/C as a base for their internal build tooling and especially for baking and deploying all their 3D Assets. +Deep Silver FISHLABS is a developer of high-end 3D games. They use Bazel with +C++/Python/Go/C as a base for their internal build tooling and especially for +baking and deploying all their 3D Assets. ### [Dropbox](https://www.dropbox.com/) - -![](/community/images/dropbox.png) At Dropbox, Bazel is a key component to our distributed build and test environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable production releases. + +At Dropbox, Bazel is a key component to our distributed build and test +environment. We use Bazel to combine TypeScript/Python/Go/C/Rust into reliable +production releases. ### [Engel & Völkers](https://www.engelvoelkers.com) -Engel & Völkers AG is a privately owned German company that, via a series of franchised offices, provides services related to real estate transactions. +Engel & Völkers AG is a privately owned German company that, via a series of +franchised offices, provides services related to real estate transactions. -> One of our internal project has seen a decrease of compilation time from 11 minutes to roughly 1 minute, this was an impressive achievement and we are currently working on bringing Bazel to more projects. ([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) +> One of our internal project has seen a decrease of compilation time from 11 +minutes to roughly 1 minute, this was an impressive achievement and we are +currently working on bringing Bazel to more projects. +([Experimenting with Google Cloud Build and Bazel](https://www.engelvoelkers.com/en/tech/engineering/software-engineering/experimenting-with-google-cloud-build-and-bazel/)) ### [Etsy](https://www.etsy.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/a/aa/Etsy_logo_lg_rgb.png) +Etsy is an e-commerce website focused on handmade or vintage items and supplies, +as well as unique factory-manufactured items. -Etsy is an e-commerce website focused on handmade or vintage items and supplies, as well as unique factory-manufactured items. - -They use Bazel to build and test its Java-based search platform. Bazel produces both packages for bare metal servers and repeatable Docker images. +They use Bazel to build and test its Java-based search platform. Bazel produces +both packages for bare metal servers and repeatable Docker images. ### [Evertz.io](https://www.evertz.io/) -Evertz.io is a multi-tenant, serverless SaaS platform for offering cost effective, multi-regional services worldwide to the Broadcast Media Industry, created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). +Evertz.io is a multi-tenant, serverless SaaS platform for offering cost +effective, multi-regional services worldwide to the Broadcast Media Industry, +created by [Evertz Microsystems](https://en.wikipedia.org/wiki/Evertz_Microsystems). -The website is fully built and deployed with an Angular and Bazel workflow ([source](https://twitter.com/MattMackay/status/1113947685508341762)). +The website is fully built and deployed with an Angular and Bazel workflow +([source](https://twitter.com/MattMackay/status/1113947685508341762)). ### [FINDMINE](http://www.findmine.com) + -![](https://www.findmine.com/static/assets/landpage/findmine-color-logo.png) - -FINDMINE is a automation technology for the retail industry that uses machine learning to scale the currently manual and tedious process of product curation. We use Bazel to mechanize our entire python package building, testing, and deployment process. +FINDMINE is a automation technology for the retail industry that uses machine +learning to scale the currently manual and tedious process of product curation. +We use Bazel to mechanize our entire python package building, testing, and +deployment process. ### [Flexport](https://www.flexport.com/) -Flexport is a tech-enabled global freight forwarder; our mission is to make global trade easier for everyone. At Flexport, we use Bazel to build/test our Java/JavaScript services and client libraries and to generate Java and Ruby code from protobuf definitions. [Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) +Flexport is a tech-enabled global freight forwarder; our mission is to make +global trade easier for everyone. At Flexport, we use Bazel to build/test our +Java/JavaScript services and client libraries and to generate Java and Ruby +code from protobuf definitions. +[Read about how we run individual JUnit 5 tests in isolation with Bazel.](https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068) ### [Foursquare](https://foursquare.com) + -![](https://upload.wikimedia.org/wikipedia/commons/9/99/FSQ_logo.png) - -Foursquare's mission is to create technology that constructs meaningful bridges between digital spaces and physical places. We manage millions of lines of primarily Scala and Python code powering data-intensive applications, including complex codegen and container build processes, with Bazel. +Foursquare's mission is to create technology that constructs meaningful +bridges between digital spaces and physical places. We manage millions of +lines of primarily Scala and Python code powering data-intensive +applications, including complex codegen and container build processes, with +Bazel. ### [GermanTechJobs](https://germantechjobs.de) + -![](https://upload.wikimedia.org/wikipedia/commons/9/98/GermanTechJobs_Logo.png) - -Bazel has simplified our workflows 10-fold and enabled shipping features at scale. +Bazel has simplified our workflows 10-fold and enabled shipping features at +scale. ### [Google](https://google.com) + -![](https://upload.wikimedia.org/wikipedia/commons/2/2f/Google_2015_logo.svg) - -Bazel was designed to be able to scale to Google's needs and meet Google's requirements of reproducibility and platform/language support. All software at Google is built using Bazel. Google uses Bazel and its rules for millions of builds every day. +Bazel was designed to be able to scale to Google's needs and meet Google's +requirements of reproducibility and platform/language support. All software at +Google is built using Bazel. Google uses Bazel and its rules for millions of +builds every day. ### [Huawei](http://www.huawei.com/) -> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go projects, except for Go projects, others originally were built by Maven. We write a simple tool to translate a Maven-built project into Bazel-built one. More and more projects will use Bazel in recent future. +> Huawei Technologies is using Bazel in about 30 projects, they are Java/Scala/Go +projects, except for Go projects, others originally were built by Maven. We +write a simple tool to translate a Maven-built project into Bazel-built one. +More and more projects will use Bazel in recent future. ### [IMC Trading](https://imc.com) + -![](https://upload.wikimedia.org/wikipedia/commons/1/17/IMC_Logo.svg) - -> IMC is a global proprietary trading firm and market maker headquarted in Amsterdam. We are using Bazel to continuously build and test our Java/C++/Python/SystemVerilog projects. +> IMC is a global proprietary trading firm and market maker headquarted in +Amsterdam. We are using Bazel to continuously build and test our +Java/C++/Python/SystemVerilog projects. ### [Improbable.io](https://improbable.io/) -Improbable.io develops SpatialOS, a distributed operating system that enables creating huge simulations inhabited by millions of complex entities. +Improbable.io develops SpatialOS, a distributed operating system that enables +creating huge simulations inhabited by millions of complex entities. ### [Interaxon](https://www.choosemuse.com/) -InteraXon is a thought-controlled computing firm that creates hardware and software platforms to convert brainwaves into digital signals. +InteraXon is a thought-controlled computing firm that creates hardware and +software platforms to convert brainwaves into digital signals. ### [Jupiter](https://jupiter.co/) -Jupiter is a company that provides delivery of groceries and household essentials every week. +Jupiter is a company that provides delivery of groceries and household +essentials every week. -They use Bazel in their backend code, specifically to compile protos and Kotlin to JVM binaries, using remote caching. ([source](https://starship.jupiter.co/jupiter-stack/)) +They use Bazel in their backend code, specifically to compile protos and Kotlin +to JVM binaries, using remote caching. +([source](https://starship.jupiter.co/jupiter-stack/)) ### [Just](https://gojust.com/) -Just is an enterprise financial technology company, headquartered in Norway, creating software solutions to transform how global corporate treasurers manage risk and liquidity. Their entire application stack is built with Bazel. +Just is an enterprise financial technology company, headquartered in Norway, +creating software solutions to transform how global corporate treasurers manage +risk and liquidity. Their entire application stack is built with Bazel. ### [Line](https://line.me/) -Line provides an app for instant communications, which is the most popular messaging application in Japan. They use Bazel on their codebase consisting of about 60% Swift and 40% C/C++/Objective-C/Objective-C++ ([source](https://twitter.com/thi_dt/status/1253334262020886532)). +Line provides an app for instant communications, which is the most popular +messaging application in Japan. +They use Bazel on their codebase consisting of about 60% Swift and 40% +C/C++/Objective-C/Objective-C++ +([source](https://twitter.com/thi_dt/status/1253334262020886532)). -> After switching to Bazel, we were able to achieve a huge improvement in the build times. This brought a significant improvement in the turn-around time during a QA period. Distributing a new build to our testers no longer means another hour waiting for building and testing. ([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) +> After switching to Bazel, we were able to achieve a huge improvement in the +build times. This brought a significant improvement in the turn-around time +during a QA period. Distributing a new build to our testers no longer means +another hour waiting for building and testing. +([Improving Build Performance of LINE for iOS with Bazel](https://engineering.linecorp.com/en/blog/improving-build-performance-line-ios-bazel/)) ### [LingoChamp](https://www.liulishuo.com/en) -![](/community/images/liulishuo.png) LingoChamp provides professional solutions to English learners. We use Bazel for our go, java and python projects. + +LingoChamp provides professional solutions to English learners. We use Bazel +for our go, java and python projects. ### [LinkedIn](https://linkedin.com/) -![](/community/images/Linkedin-Logo.png) LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social network. LinkedIn uses Bazel for building its iOS Apps. + +LinkedIn, a subsidiary of Microsoft, is the world’s largest professional social +network. LinkedIn uses Bazel for building its iOS Apps. ### [Lucid Software](https://lucid.co/) -![](/community/images/Lucid_Software-logo.svg) + -Lucid Software is a leader in visual collaboration, helping teams see and build the future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), [Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams can align around a shared vision, clarify complexity, and collaborate visually, no matter where they’re located. +Lucid Software is a leader in visual collaboration, helping teams see and build the +future from idea to reality. With its products—[Lucidchart](https://www.lucidchart.com/), +[Lucidspark](https://lucidspark.com/), and [Lucidscale](https://lucidscale.com/)—teams +can align around a shared vision, clarify complexity, and collaborate visually, no +matter where they’re located. -Lucid uses Bazel to build millions of lines of Scala and TypeScript. Migrating to Bazel has tremendously sped up its builds, reduced external dependencies on the build environment, and simplified developers' experience with the build system. Bazel has improved developer productivity at Lucid and unlocked further growth. +Lucid uses Bazel to build millions of lines of Scala and TypeScript. +Migrating to Bazel has tremendously sped up its builds, reduced external +dependencies on the build environment, and simplified developers' experience +with the build system. Bazel has improved developer productivity at Lucid and +unlocked further growth. ### [Lyft](https://www.lyft.com/) @@ -192,81 +279,131 @@ Lyft is using Bazel for their iOS ([source](https://twitter.com/SmileyKeith/stat ### [Meetup](http://www.meetup.com/) -Meetup is an online social networking portal that facilitates offline group meetings. The Meetup engineering team contributes to [rules\_scala](https://github.com/bazelbuild/rules_scala) and is the maintainer of [rules\_avro](https://github.com/meetup/rules_avro) and [rules\_openapi](https://github.com/meetup/rules_openapi). +Meetup is an online social networking portal that facilitates offline group +meetings. +The Meetup engineering team contributes to +[rules_scala](https://github.com/bazelbuild/rules_scala) and is the +maintainer of [rules_avro](https://github.com/meetup/rules_avro) +and [rules_openapi](https://github.com/meetup/rules_openapi). + ### [Nvidia](https://www.nvidia.com/) -> At Nvidia we have been using dazel(docker bazel) for python to work around some of bazel's python short comings. Everything else runs in normal bazel (Mostly Go / Scala/ C++/ Cuda) ([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) +> At Nvidia we have been using dazel(docker bazel) for python to work around +some of bazel's python short comings. Everything else runs in normal bazel +(Mostly Go / Scala/ C++/ Cuda) +([source](https://twitter.com/rwhitcomb/status/1080887723433447424)) + ### [Peloton Technology](http://www.peloton-tech.com) -Peloton Technology is an automated vehicle technology company that tackles truck accidents and fuel use. They use Bazel to *enable reliable builds for automotive safety systems*. +Peloton Technology is an automated vehicle technology company that tackles truck +accidents and fuel use. They use Bazel to _enable reliable builds for automotive +safety systems_. ### [Pigweed](https://pigweed.dev) -![](https://pigweed.dev/_static/pw_logo.svg) + -Pigweed is an open-source solution for sustained, robust, and rapid embedded product development for large teams. Pigweed has shipped in millions of devices, including Google's suite of Pixel devices, Nest thermostats, [satellites](https://www.spinlaunch.com/), and [autonomous aerial drones](https://www.flyzipline.com/). +Pigweed is an open-source solution for sustained, robust, and rapid embedded +product development for large teams. Pigweed has shipped in millions of +devices, including Google's suite of Pixel devices, Nest thermostats, +[satellites](https://www.spinlaunch.com/), and [autonomous aerial +drones](https://www.flyzipline.com/). -Pigweed [uses Bazel as its primary build system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for Embedded](https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded) blog post discusses why we think it's a great build system for embedded projects! +Pigweed [uses Bazel as its primary build +system](https://pigweed.dev/seed/0111-build-systems.html). The [Bazel for +Embedded][pw-bazel-great] blog post discusses why we think it's a great build +system for embedded projects! + +[pw-bazel-great]: https://blog.bazel.build/2024/08/08/bazel-for-embedded.html#why-bazel-for-embedded ### [Pinterest](https://www.pinterest.com/) -![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Pinterest_Logo.svg/200px-Pinterest_Logo.svg.png) + -Pinterest is the world’s catalog of ideas. They use Bazel to build various backend services (Java/C++) and the iOS application (Objective-C/C++). +Pinterest is the world’s catalog of ideas. They use Bazel to build various +backend services (Java/C++) and the iOS application (Objective-C/C++). -> We identified Bazel was the best fit for our goals to build a foundation for an order of magnitude improvement in performance, eliminate variability in build environments and adopt incrementally. As a result, we’re now shipping all our iOS releases using Bazel. [Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) +> We identified Bazel was the best fit for our goals to build a foundation for +an order of magnitude improvement in performance, eliminate variability in +build environments and adopt incrementally. As a result, we’re now shipping all +our iOS releases using Bazel. +[Developing fast & reliable iOS builds at Pinterest](https://medium.com/@Pinterest_Engineering/developing-fast-reliable-ios-builds-at-pinterest-part-one-cb1810407b92) ### [PubRef](https://github.com/pubref) -PubRef is an emerging scientific publishing platform. They use Bazel with [rules\_closure](https://github.com/bazelbuild/rules_closure) to build the frontend, native java rules to build the main backend, [rules\_go](https://github.com/bazelbuild/rules_go), [rules\_node](https://github.com/pubref/rules_node), and [rules\_kotlin](https://github.com/pubref/rules_kotlin) to build assorted backend services. [rules\_protobuf](https://github.com/pubref/rules_protobuf) is used to assist with gRPC-based communication between backend services. PubRef.org is based in Boulder, CO. +PubRef is an emerging scientific publishing platform. They use Bazel with +[rules_closure](https://github.com/bazelbuild/rules_closure) to build the +frontend, native java rules to build the main backend, +[rules_go](https://github.com/bazelbuild/rules_go), +[rules_node](https://github.com/pubref/rules_node), and +[rules_kotlin](https://github.com/pubref/rules_kotlin) to build assorted +backend services. [rules_protobuf](https://github.com/pubref/rules_protobuf) is +used to assist with gRPC-based communication between backend services. +PubRef.org is based in Boulder, CO. ### [Redfin](https://redfin.com/) - -Redfin is a next-generation real estate brokerage with full-service local agents. They use Bazel to build and deploy the website and various backend services. - -> With the conversion mostly behind us, things are greatly improved! Our CI builds are faster (*way* faster: they used to take 40–90 minutes, and now dev builds average 5–6 minutes). Reliability is far higher, too. This is harder to quantify, but the shift from unexplained build failures being something that “just happens” to being viewed as real problems to be solved has put us on a virtuous cycle of ever-increasing reliability. ([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) +Redfin is a next-generation real estate brokerage with full-service local +agents. They use Bazel to build and deploy the website and various backend +services. + +> With the conversion mostly behind us, things are greatly improved! Our CI +builds are faster (*way* faster: they used to take 40–90 minutes, and now dev +builds average 5–6 minutes). Reliability is far higher, too. This is harder to +quantify, but the shift from unexplained build failures being something that +“just happens” to being viewed as real problems to be solved has put us on a +virtuous cycle of ever-increasing reliability. +([We Switched from Maven to Bazel and Builds Got 10x Faster](https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854)) ### [Ritual](https://ritual.co) + -![](https://lh3.googleusercontent.com/7Ir6j25ROnsXhtQXveOzup33cizxLf-TiifSC1cI6op0bQVB-WePmPjJOfXUBQ0L3KpkheObAiS28e-TS8hZtDzxOIc) - -Ritual is a mobile pick up app, connecting restaurants with customers to offer a simple, time-saving tool to get the food and beverages they want, without the wait. Ritual uses Bazel for their backend services. +Ritual is a mobile pick up app, connecting restaurants with customers to offer +a simple, time-saving tool to get the food and beverages they want, without the +wait. Ritual uses Bazel for their backend services. ### [Snap](https://www.snap.com/en-US/) -Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more details about their process, see their [engineering blog](https://eng.snap.com/blog/). +Snap, the developer of Snapchat messaging app, has migrated from Buck to Bazel +in 2020 ([source](https://twitter.com/wew/status/1326957862816509953)). For more +details about their process, see their [engineering blog](https://eng.snap.com/blog/). ### [Stripe](https://stripe.com) - -![](https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Stripe_Logo%2C_revised_2016.svg/320px-Stripe_Logo%2C_revised_2016.svg.png) + Stripe provides mobile payment solutions. They use Bazel in their build and test pipelines, as detailed in their [engineering blog](https://stripe.com/blog/fast-secure-builds-choose-two). ### [Tinder](https://tinder.com) + -![](https://policies.tinder.com/static/b0327365f4c0a31c4337157c10e9fadf/c1b63/tinder_full_color_watermark.png) - -Tinder migrated its iOS app from CocoaPods to Bazel in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). +Tinder migrated its iOS app from CocoaPods to Bazel +in 2021 ([source](https://medium.com/tinder/bazel-hermetic-toolchain-and-tooling-migration-c244dc0d3ae)). ### [Tink](https://tink.com/) + -![](https://cdn.tink.se/tink-logos/LOW/Tink_Black.png) +Tink is a european fintech, building the best way to connect to banks across +Europe. -Tink is a european fintech, building the best way to connect to banks across Europe. - -They are using Bazel to build their backend services from a polyglot monorepo. Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) meetup group. +They are using Bazel to build their backend services from a polyglot monorepo. +Engineers at Tink are organizing the [bazel build //stockholm/...](https://www.meetup.com/BazelSTHLM/) +meetup group. ### [Tokopedia](https://www.tokopedia.com/) -Tokopedia is an Indonesian technology company specializing in e-commerce, with over 90 million monthly active users and over 7 million merchants on the platform. +Tokopedia is an Indonesian technology company specializing in e-commerce, with +over 90 million monthly active users and over 7 million merchants on the +platform. -They wrote the article [How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), where they explain how Bazel sped up their builds. The build duration went from 55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote caching. +They wrote the article +[How Tokopedia Achieved 1000% Faster iOS Build Time](https://medium.com/tokopedia-engineering/how-tokopedia-achieved-1000-faster-ios-build-time-7664b2d8ae5), +where they explain how Bazel sped up their builds. The build duration went from +55 minutes to 10 minutes by using Bazel, and down to 5 minutes with remote +caching. ### [Trunk.io](https://trunk.io/merge/trunk-merge-and-bazel) - -![](/community/images/trunk-logo-dark.svg) + Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initialized Capital. Trunk offers a powerful pull request merge service with first-class support for the Bazel build system. By leveraging Bazel's understanding of dependencies within a codebase, Trunk's merge service intelligently creates parallel merge lanes, allowing independent changes to be tested and merged simultaneously. @@ -274,49 +411,71 @@ Trunk is a San Francisco-based company backed by Andreessen Horowitz and Initial ### [Twitter](https://twitter.com/) -Twitter has made the decision to migrate from Pants to Bazel as their primary build tool ([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). +Twitter has made the decision to migrate from Pants to Bazel as their primary +build tool +([source](https://groups.google.com/forum/#!msg/pants-devel/PHVIbVDLhx8/LpSKIP5cAwAJ)). ### [Two Sigma](https://www.twosigma.com/) + -![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c6/Two_Sigma_logo.svg/2880px-Two_Sigma_logo.svg.png) - -Two Sigma is a New York-headquartered technology company dedicated to finding value in the world’s data. +Two Sigma is a New York-headquartered technology company dedicated to finding +value in the world’s data. ### [TypeDB](https://typedb.com) +TypeDB Logo -![TypeDB Logo](/community/images/typedb.png) - -TypeDB is a database technology that can be used to intuitively model interconnected data. Through its type-theoretic and polymorphic query language, TypeQL, the data can be accessed with simple, human-readable queries that run at lightspeed. +TypeDB is a database technology that can be used to intuitively model +interconnected data. Through its type-theoretic and polymorphic query language, +TypeQL, the data can be accessed with simple, human-readable queries that run at +lightspeed. -Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution pipeline that manages many repositories in a wide variety of languages, and deploys to numerous platforms seamlessly. The TypeDB team has also released Bazel rules for assembling and deploying software distributions. +Bazel enables the TypeDB team to build a highly-orchestrated CI and distribution +pipeline that manages many repositories in a wide variety of languages, and +deploys to numerous platforms seamlessly. The TypeDB team has also released +Bazel rules for assembling and deploying software distributions. ### [Uber](https://www.uber.com) -Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo is likely one of the largest Go repositories using Bazel. See the article [Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) to learn more about their experience. +Uber is a ride-hailing company. With 900 active developers, Uber’s Go monorepo +is likely one of the largest Go repositories using Bazel. See the article +[Building Uber’s Go Monorepo with Bazel](https://eng.uber.com/go-monorepo-bazel/) +to learn more about their experience. ### [Uber Advanced Technologies Group](https://www.uber.com/info/atg/) + -![](https://upload.wikimedia.org/wikipedia/commons/thumb/6/62/Uber_logo.svg/220px-Uber_logo.svg.png) - -Uber Advanced Technologies Group is focused on autonomous vehicle efforts at Uber, including trucking/freight and autonomous ride sharing. The organization uses Bazel as its primary build system. +Uber Advanced Technologies Group is focused on autonomous vehicle efforts at +Uber, including trucking/freight and autonomous ride sharing. The organization +uses Bazel as its primary build system. ### [Vistar Media](http://vistarmedia.com) - -Vistar Media is an advertising platform that enables brands to reach consumers based on their behavior in the physical world. Their engineering team is primarily based out of Philadelphia and is using Bazel for builds, deploys, to speed up testing, and to consolidate repositories written with a variety of different technologies. +Vistar Media is an advertising platform that enables brands to reach consumers +based on their behavior in the physical world. Their engineering team is +primarily based out of Philadelphia and is using Bazel for builds, deploys, to +speed up testing, and to consolidate repositories written with a variety of +different technologies. ### [VMware](https://www.vmware.com/) - -VMware uses Bazel to produce deterministic, reliable builds while developing innovative products for their customers. +VMware uses Bazel to produce deterministic, reliable builds while developing +innovative products for their customers. ### [Wix](https://www.wix.com/) -Wix is a cloud-based web development platform. Their backend uses Java and Scala code. They use remote execution with Google Cloud Build. +Wix is a cloud-based web development platform. Their backend uses Java and Scala +code. They use remote execution with Google Cloud Build. -> We have seen about 5 times faster clean builds when running with bazel remote execution which utilizes bazel’s great build/test parallelism capabilities when it dispatches build/test actions to a worker farm. Average build times are more than 10 times faster due to the utilization of bazel’s aggressive caching mechanism. ([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) +> We have seen about 5 times faster clean builds when running with bazel remote +execution which utilizes bazel’s great build/test parallelism capabilities when +it dispatches build/test actions to a worker farm. Average build times are more +than 10 times faster due to the utilization of bazel’s aggressive caching +mechanism. +([Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself](https://medium.com/wix-engineering/migrating-to-bazel-from-maven-or-gradle-5-crucial-questions-you-should-ask-yourself-f23ac6bca070)) ### [Zenly](https://zen.ly/) -Zenly is a live map of your friends and family. It’s the most fun way to meet up — or just see what’s up! — so you can feel together, even when you're apart. +Zenly is a live map of your friends and family. It’s the most fun way to meet up +— or just see what’s up! — so you can feel together, even when you're apart. + *** @@ -324,33 +483,44 @@ Zenly is a live map of your friends and family. It’s the most fun way to meet ### [Abseil](https://abseil.io/) -Abseil is an open-source collection of C++ code (compliant to C++11) designed to augment the C++ standard library. +Abseil is an open-source collection of C++ code (compliant to C++11) designed +to augment the C++ standard library. ### [Angular](https://angular.io) -![](https://upload.wikimedia.org/wikipedia/commons/c/cf/Angular_full_color_logo.svg) + -Angular is a popular web framework. Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). +Angular is a popular web framework. +Angular is [built with Bazel](https://github.com/angular/angular/blob/master/docs/BAZEL.md). ### [Apollo](https://github.com/ApolloAuto/apollo) -Apollo is a high performance, flexible architecture which accelerates the development, testing, and deployment of Autonomous Vehicles. +Apollo is a high performance, flexible architecture which accelerates the +development, testing, and deployment of Autonomous Vehicles. ### [brpc](https://github.com/brpc/brpc) -An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ instances(not counting clients) and thousands kinds of services, called "baidu-rpc" inside Baidu. +An industrial-grade RPC framework used throughout Baidu, with 1,000,000+ +instances(not counting clients) and thousands kinds of services, called +"baidu-rpc" inside Baidu. ### [cert-manager](https://github.com/jetstack/cert-manager) -cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry. +cert-manager is a Kubernetes add-on to automate the management and issuance of +TLS certificates from various issuing sources. It will ensure certificates are +valid and up to date periodically, and attempt to renew certificates at an +appropriate time before expiry. ### [CallBuilder](https://github.com/google/CallBuilder) -A Java code generator that allows you to create a builder by writing one function. +A Java code generator that allows you to create a builder by writing one +function. ### [CPPItertools](https://github.com/ryanhaining/cppitertools) -C++ library providing range-based for loop add-ons inspired by the Python builtins and itertools library. Like itertools and the Python3 builtins, this library uses lazy evaluation wherever possible. +C++ library providing range-based for loop add-ons inspired by the Python +builtins and itertools library. Like itertools and the Python3 builtins, this +library uses lazy evaluation wherever possible. ### [Copybara](https://github.com/google/copybara) @@ -358,11 +528,13 @@ Copybara is a tool for transforming and moving code between repositories. ### [Dagger](https://google.github.io/dagger/) -Dagger is a fully static, compile-time dependency injection framework for both Java and Android. +Dagger is a fully static, compile-time dependency injection framework for both +Java and Android. ### [DAML](https://github.com/digital-asset/daml) -DAML is a smart contract language for building future-proof distributed applications on a safe, privacy-aware runtime. +DAML is a smart contract language for building future-proof distributed +applications on a safe, privacy-aware runtime. ### [DeepMind Lab](https://github.com/deepmind/lab) @@ -370,7 +542,10 @@ A customisable 3D platform for agent-based AI research. ### [Drake](https://github.com/RobotLocomotion/drake) -Drake is a C++ toolbox started at MIT and now led by the Toyota Research Institute. It is a collection of tools for analyzing the dynamics of our robots and building control systems for them, with a heavy emphasis on optimization-based design/analysis. +Drake is a C++ toolbox started at MIT and now led by the Toyota Research +Institute. It is a collection of tools for analyzing the dynamics of our robots +and building control systems for them, with a heavy emphasis on +optimization-based design/analysis. ### [Envoy](https://github.com/lyft/envoy) @@ -378,15 +553,19 @@ C++ L7 proxy and communication bus ### [Error Prone](https://github.com/google/error-prone) -Catches common Java mistakes as compile-time errors. (Migration to Bazel is in progress.) +Catches common Java mistakes as compile-time errors. (Migration to Bazel is in +progress.) ### [Extensible Service Proxy](https://github.com/cloudendpoints/esp) -Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management capabilities for JSON/REST or gRPC API services. The current implementation is based on an NGINX HTTP reverse proxy server. +Extensible Service Proxy, a.k.a. ESP is a proxy which enables API management +capabilities for JSON/REST or gRPC API services. The current implementation is +based on an NGINX HTTP reverse proxy server. ### [FFruit](https://gitlab.com/perezd/ffruit/) -FFruit is a free & open source Android application to the popular service [Falling Fruit](https://fallingfruit.org). +FFruit is a free & open source Android application to the popular service +[Falling Fruit](https://fallingfruit.org). ### [Gerrit Code Review](https://gerritcodereview.com) @@ -398,51 +577,61 @@ Gitiles is a simple repository browser for Git repositories, built on JGit. ### [Grakn](https://github.com/graknlabs/grakn) -Grakn ([https://grakn.ai/](https://grakn.ai/)) is the knowledge graph engine to organise complex networks of data and make it queryable. +Grakn (https://grakn.ai/) is the knowledge graph engine to organise complex +networks of data and make it queryable. ### [GRPC](http://www.grpc.io) - -A language-and-platform-neutral remote procedure call system. (Bazel is a supported, although not primary, build system.) +A language-and-platform-neutral remote procedure call system. +(Bazel is a supported, although not primary, build system.) ### [gVisor](https://github.com/google/gvisor) - gVisor is a container runtime sandbox. ### [Guetzli](https://github.com/google/guetzli/) -Guetzli is a JPEG encoder that aims for excellent compression density at high visual quality. +Guetzli is a JPEG encoder that aims for excellent compression density at high +visual quality. ### [Gulava](http://www.github.com/google/gulava/) -A Java code generator that lets you write Prolog-style predicates and use them seamlessly from normal Java code. +A Java code generator that lets you write Prolog-style predicates and use them +seamlessly from normal Java code. ### [Heron](https://github.com/apache/incubator-heron) -Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter. +Heron is a realtime, distributed, fault-tolerant stream processing engine from +Twitter. ### [Internet Computer Protocol](https://internetcomputer.org/) -![](https://internetcomputer.org/img/IC_logo_horizontal_white.svg) + -The Internet Computer Protocol is a publicly available blockchain network that enables replicated execution of general-purpose computation, serving hundreds of thousands of applications and their users. +The Internet Computer Protocol is a publicly available blockchain network that +enables replicated execution of general-purpose computation, serving hundreds +of thousands of applications and their users. ### [Jazzer](https://github.com/CodeIntelligenceTesting/jazzer) -![](https://www.code-intelligence.com/hubfs/Logos/CI%20Logos/Jazzer_einfach.png) + Jazzer is a fuzzer for Java and other JVM-based languages that integrates with JUnit 5. ### [JGit](https://eclipse.org/jgit/) -JGit is a lightweight, pure Java library implementing the Git version control system. +JGit is a lightweight, pure Java library implementing the Git version control +system. ### [Jsonnet](https://jsonnet.org/) -An elegant, formally-specified config generation language for JSON. (Bazel is a supported build system.) +An elegant, formally-specified config generation language for JSON. +(Bazel is a supported build system.) ### [Kubernetes](https://github.com/kubernetes/kubernetes) -![](https://raw.githubusercontent.com/kubernetes/kubernetes/master/logo/logo.png) Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. + +Kubernetes is an open source system for managing containerized applications +across multiple hosts, providing basic mechanisms for deployment, maintenance, +and scaling of applications. ### [Kythe](https://github.com/google/kythe) @@ -450,9 +639,10 @@ An ecosystem for building tools that work with code. ### [ls-lint](https://github.com/loeffel-io/ls-lint) -![](https://raw.githubusercontent.com/loeffel-io/ls-lint/master/assets/logo/ls-lint.png) + -An extremely fast directory and filename linter - Bring some structure to your project file system. +An extremely fast directory and filename linter - Bring some structure to your +project file system. ### [Nomulus](https://github.com/google/nomulus) @@ -460,11 +650,19 @@ Top-level domain name registry service on Google App Engine. ### [ONOS : Open Network Operating System](https://github.com/opennetworkinglab/onos) -![](https://upload.wikimedia.org/wikipedia/en/thumb/d/d3/Logo_for_the_ONOS_open_source_project.png/175px-Logo_for_the_ONOS_open_source_project.png) ONOS is the only SDN controller platform that supports the transition from legacy “brown field” networks to SDN “green field” networks. This enables exciting new capabilities, and disruptive deployment and operational cost points for network operators. + +ONOS is the only SDN controller platform that supports the transition from +legacy “brown field” networks to SDN “green field” networks. This enables +exciting new capabilities, and disruptive deployment and operational cost points +for network operators. ### [PetitParser for Java](https://github.com/petitparser/java-petitparser) -Grammars for programming languages are traditionally specified statically. They are hard to compose and reuse due to ambiguities that inevitably arise. PetitParser combines ideas from scannnerless parsing, parser combinators, parsing expression grammars and packrat parsers to model grammars and parsers as objects that can be reconfigured dynamically. +Grammars for programming languages are traditionally specified statically. +They are hard to compose and reuse due to ambiguities that inevitably arise. +PetitParser combines ideas from scannnerless parsing, parser combinators, +parsing expression grammars and packrat parsers to model grammars and parsers +as objects that can be reconfigured dynamically. ### [PlaidML](https://github.com/plaidml/plaidml) @@ -472,11 +670,14 @@ PlaidML is a framework for making deep learning work everywhere. ### [Project V](https://www.v2ray.com/) -![](https://www.v2ray.com/resources/v2ray_1024.png) Project V is a set of tools to help you build your own privacy network over internet. + +Project V is a set of tools to help you build your own privacy network over +internet. ### [Prysmatic Labs Ethereum 2.0 Implementation](https://github.com/prysmaticlabs/prysm) -Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed computing platform. +Prysm is a sharding client for Ethereum 2.0, a blockchain-based distributed +computing platform. ### [Ray](https://github.com/ray-project/ray) @@ -484,7 +685,8 @@ Ray is a flexible, high-performance distributed execution framework. ### [Resty](https://github.com/go-resty/resty) -Resty is a Simple HTTP and REST client library for Go (inspired by Ruby rest-client). +Resty is a Simple HTTP and REST client library for Go (inspired by Ruby +rest-client). ### [Roughtime](https://roughtime.googlesource.com/roughtime) @@ -496,7 +698,9 @@ Selenium is a portable framework for testing web applications. ### [Semantic](https://github.com/github/semantic) -Semantic is a Haskell library and command line tool for parsing, analyzing, and comparing source code. It is developed by GitHub (and used for example for the code navigation). +Semantic is a Haskell library and command line tool for parsing, analyzing, and +comparing source code. It is developed by GitHub (and used for example for the +code navigation). ### [Served](https://github.com/meltwater/served) @@ -504,11 +708,13 @@ Served is a C++ library for building high performance RESTful web servers. ### [Sonnet](https://github.com/deepmind/sonnet) -Sonnet is a library built on top of TensorFlow for building complex neural networks. +Sonnet is a library built on top of TensorFlow for building complex neural +networks. ### [Sorbet](https://github.com/sorbet/sorbet) -Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to codebases with millions of lines of code and can be adopted incrementally. +Sorbet is a fast, powerful type checker for a subset of Ruby. It scales to +codebases with millions of lines of code and can be adopted incrementally. ### [Spotify](https://spotify.com) @@ -516,11 +722,12 @@ Spotify is using Bazel to build their iOS and Android Apps ([source](https://twi ### [Tink](https://github.com/google/tink) -Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. +Tink is a multi-language, cross-platform, open source library that provides +cryptographic APIs that are secure, easy to use correctly, and hard(er) to +misuse. ### [TensorFlow](http://tensorflow.org) - -![](https://upload.wikimedia.org/wikipedia/commons/a/a4/TensorFlowLogo.png) + An open source software library for machine intelligence. @@ -534,8 +741,10 @@ Project Wycheproof tests crypto libraries against known attacks. ### [XIOSim](https://github.com/s-kanev/XIOSim) -XIOSim is a detailed user-mode microarchitectural simulator for the x86 architecture. +XIOSim is a detailed user-mode microarchitectural simulator for the x86 +architecture. ### [ZhihuDailyPurify](https://github.com/izzyleung/ZhihuDailyPurify) -ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese question-and-answer webs. +ZhihuDailyPurify is a light weight version of Zhihu Daily, a Chinese +question-and-answer webs. diff --git a/concepts/build-files.mdx b/concepts/build-files.mdx deleted file mode 100644 index 87f9d167..00000000 --- a/concepts/build-files.mdx +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: 'BUILD files' ---- - -The previous sections described packages, targets and labels, and the build dependency graph abstractly. This section describes the concrete syntax used to define a package. - -By definition, every package contains a `BUILD` file, which is a short program. - -Note: The `BUILD` file can be named either `BUILD` or `BUILD.bazel`. If both files exist, `BUILD.bazel` takes precedence over `BUILD`. For simplicity's sake, the documentation refers to these files simply as `BUILD` files. - -`BUILD` files are evaluated using an imperative language, [Starlark](https://github.com/bazelbuild/starlark/). - -They are interpreted as a sequential list of statements. - -In general, order does matter: variables must be defined before they are used, for example. However, most `BUILD` files consist only of declarations of build rules, and the relative order of these statements is immaterial; all that matters is *which* rules were declared, and with what values, by the time package evaluation completes. - -When a build rule function, such as `cc_library`, is executed, it creates a new target in the graph. This target can later be referred using a label. - -In simple `BUILD` files, rule declarations can be re-ordered freely without changing the behavior. - -To encourage a clean separation between code and data, `BUILD` files cannot contain function definitions, `for` statements or `if` statements (but list comprehensions and `if` expressions are allowed). Functions can be declared in `.bzl` files instead. Additionally, `*args` and `**kwargs` arguments are not allowed in `BUILD` files; instead list all the arguments explicitly. - -Crucially, programs in Starlark can't perform arbitrary I/O. This invariant makes the interpretation of `BUILD` files hermetic — dependent only on a known set of inputs, which is essential for ensuring that builds are reproducible. For more details, see [Hermeticity](/basics/hermeticity). - -Because `BUILD` files need to be updated whenever the dependencies of the underlying code change, they are typically maintained by multiple people on a team. `BUILD` file authors should comment liberally to document the role of each build target, whether or not it is intended for public use, and to document the role of the package itself. - -## Loading an extension - -Bazel extensions are files ending in `.bzl`. Use the `load` statement to import a symbol from an extension. - -``` -load("//foo/bar:file.bzl", "some_library") -``` - -This code loads the file `foo/bar/file.bzl` and adds the `some_library` symbol to the environment. This can be used to load new rules, functions, or constants (for example, a string or a list). Multiple symbols can be imported by using additional arguments to the call to `load`. Arguments must be string literals (no variable) and `load` statements must appear at top-level — they cannot be in a function body. - -The first argument of `load` is a [label](/concepts/labels) identifying a `.bzl` file. If it's a relative label, it is resolved with respect to the package (not directory) containing the current `bzl` file. Relative labels in `load` statements should use a leading `:`. - -`load` also supports aliases, therefore, you can assign different names to the imported symbols. - -``` -load("//foo/bar:file.bzl", library_alias = "some_library") -``` - -You can define multiple aliases within one `load` statement. Moreover, the argument list can contain both aliases and regular symbol names. The following example is perfectly legal (please note when to use quotation marks). - -``` -load(":my_rules.bzl", "some_rule", nice_alias = "some_other_rule") -``` - -In a `.bzl` file, symbols starting with `_` are not exported and cannot be loaded from another file. - -You can use [load visibility](/concepts/visibility#load-visibility) to restrict who may load a `.bzl` file. - -## Types of build rules - -The majority of build rules come in families, grouped together by language. For example, `cc_binary`, `cc_library` and `cc_test` are the build rules for C++ binaries, libraries, and tests, respectively. Other languages use the same naming scheme, with a different prefix, such as `java_*` for Java. Some of these functions are documented in the [Build Encyclopedia](/reference/be/overview), but it is possible for anyone to create new rules. - -- `*_binary` rules build executable programs in a given language. After a build, the executable will reside in the build tool's binary output tree at the corresponding name for the rule's label, so `//my:program` would appear at (for example) `$(BINDIR)/my/program`. - - In some languages, such rules also create a runfiles directory containing all the files mentioned in a `data` attribute belonging to the rule, or any rule in its transitive closure of dependencies; this set of files is gathered together in one place for ease of deployment to production. - -- `*_test` rules are a specialization of a `*_binary` rule, used for automated testing. Tests are simply programs that return zero on success. - - Like binaries, tests also have runfiles trees, and the files beneath it are the only files that a test may legitimately open at runtime. For example, a program `cc_test(name='x', data=['//foo:bar'])` may open and read `$TEST_SRCDIR/workspace/foo/bar` during execution. (Each programming language has its own utility function for accessing the value of `$TEST_SRCDIR`, but they are all equivalent to using the environment variable directly.) Failure to observe the rule will cause the test to fail when it is executed on a remote testing host. - -- `*_library` rules specify separately-compiled modules in the given programming language. Libraries can depend on other libraries, and binaries and tests can depend on libraries, with the expected separate-compilation behavior. - -[← Labels](/concepts/labels) · [Dependencies →](/concepts/dependencies) - -## File encoding - -`BUILD` and `.bzl` files should be encoded in UTF-8, of which ASCII is a valid subset. Arbitrary byte sequences are currently allowed, but may stop being supported in the future. diff --git a/concepts/build-ref.mdx b/concepts/build-ref.mdx index 33c1e76c..e8839d40 100644 --- a/concepts/build-ref.mdx +++ b/concepts/build-ref.mdx @@ -2,27 +2,51 @@ title: 'Repositories, workspaces, packages, and targets' --- -Bazel builds software from source code organized in directory trees called repositories. A defined set of repositories comprises the workspace. Source files in repositories are organized in a nested hierarchy of packages, where each package is a directory that contains a set of related source files and one `BUILD` file. The `BUILD` file specifies what software outputs can be built from the source. + + +Bazel builds software from source code organized in directory trees called +repositories. A defined set of repositories comprises the workspace. Source +files in repositories are organized in a nested hierarchy of packages, where +each package is a directory that contains a set of related source files and one +`BUILD` file. The `BUILD` file specifies what software outputs can be built from +the source. ### Repositories -Source files used in a Bazel build are organized in *repositories* (often shortened to *repos*). A repo is a directory tree with a boundary marker file at its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. +Source files used in a Bazel build are organized in _repositories_ (often +shortened to _repos_). A repo is a directory tree with a boundary marker file at +its root; such a boundary marker file could be `MODULE.bazel`, `REPO.bazel`, or +in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. -The repo in which the current Bazel command is being run is called the *main repo*. Other, (external) repos are defined by *repo rules*; see [external dependencies overview](/external/overview) for more information. +The repo in which the current Bazel command is being run is called the _main +repo_. Other, (external) repos are defined by _repo rules_; see [external +dependencies overview](/external/overview) for more information. ## Workspace -A *workspace* is the environment shared by all Bazel commands run from the same main repo. It encompasses the main repo and the set of all defined external repos. +A _workspace_ is the environment shared by all Bazel commands run from the same +main repo. It encompasses the main repo and the set of all defined external +repos. -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". ## Packages -The primary unit of code organization in a repository is the *package*. A package is a collection of related files and a specification of how they can be used to produce output artifacts. +The primary unit of code organization in a repository is the _package_. A +package is a collection of related files and a specification of how they can be +used to produce output artifacts. -A package is defined as a directory containing a [`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A package includes all files in its directory, plus all subdirectories beneath it, except those which themselves contain a `BUILD` file. From this definition, no file or directory may be a part of two different packages. +A package is defined as a directory containing a +[`BUILD` file](/concepts/build-files) named either `BUILD` or `BUILD.bazel`. A +package includes all files in its directory, plus all subdirectories beneath it, +except those which themselves contain a `BUILD` file. From this definition, no +file or directory may be a part of two different packages. -For example, in the following directory tree there are two packages, `my/app`, and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but a directory belonging to package `my/app`. +For example, in the following directory tree there are two packages, `my/app`, +and the subpackage `my/app/tests`. Note that `my/app/data` is not a package, but +a directory belonging to package `my/app`. ``` src/my/app/BUILD @@ -34,18 +58,48 @@ src/my/app/tests/test.cc ## Targets -A package is a container of *targets*, which are defined in the package's `BUILD` file. Most targets are one of two principal kinds, *files* and *rules*. - -Files are further divided into two kinds. *Source files* are usually written by the efforts of people, and checked in to the repository. *Generated files*, sometimes called derived files or output files, are not checked in, but are generated from source files. - -The second kind of target is declared with a *rule*. Each rule instance specifies the relationship between a set of input and a set of output files. The inputs to a rule may be source files, but they also may be the outputs of other rules. - -Whether the input to a rule is a source file or a generated file is in most cases immaterial; what matters is only the contents of that file. This fact makes it easy to replace a complex source file with a generated file produced by a rule, such as happens when the burden of manually maintaining a highly structured file becomes too tiresome, and someone writes a program to derive it. No change is required to the consumers of that file. Conversely, a generated file may easily be replaced by a source file with only local changes. - -The inputs to a rule may also include *other rules*. The precise meaning of such relationships is often quite complex and language- or rule-dependent, but intuitively it is simple: a C++ library rule A might have another C++ library rule B for an input. The effect of this dependency is that B's header files are available to A during compilation, B's symbols are available to A during linking, and B's runtime data is available to A during execution. - -An invariant of all rules is that the files generated by a rule always belong to the same package as the rule itself; it is not possible to generate files into another package. It is not uncommon for a rule's inputs to come from another package, though. - -Package groups are sets of packages whose purpose is to limit accessibility of certain rules. Package groups are defined by the `package_group` function. They have three properties: the list of packages they contain, their name, and other package groups they include. The only allowed ways to refer to them are from the `visibility` attribute of rules or from the `default_visibility` attribute of the `package` function; they do not generate or consume files. For more information, refer to the [`package_group` documentation](/reference/be/functions#package_group). - -[Labels→](/concepts/labels) +A package is a container of _targets_, which are defined in the package's +`BUILD` file. Most targets are one of two principal kinds, _files_ and _rules_. + +Files are further divided into two kinds. _Source files_ are usually written by +the efforts of people, and checked in to the repository. _Generated files_, +sometimes called derived files or output files, are not checked in, but are +generated from source files. + +The second kind of target is declared with a _rule_. Each rule instance +specifies the relationship between a set of input and a set of output files. The +inputs to a rule may be source files, but they also may be the outputs of other +rules. + +Whether the input to a rule is a source file or a generated file is in most +cases immaterial; what matters is only the contents of that file. This fact +makes it easy to replace a complex source file with a generated file produced by +a rule, such as happens when the burden of manually maintaining a highly +structured file becomes too tiresome, and someone writes a program to derive it. +No change is required to the consumers of that file. Conversely, a generated +file may easily be replaced by a source file with only local changes. + +The inputs to a rule may also include _other rules_. The precise meaning of such +relationships is often quite complex and language- or rule-dependent, but +intuitively it is simple: a C++ library rule A might have another C++ library +rule B for an input. The effect of this dependency is that B's header files are +available to A during compilation, B's symbols are available to A during +linking, and B's runtime data is available to A during execution. + +An invariant of all rules is that the files generated by a rule always belong to +the same package as the rule itself; it is not possible to generate files into +another package. It is not uncommon for a rule's inputs to come from another +package, though. + +Package groups are sets of packages whose purpose is to limit accessibility of +certain rules. Package groups are defined by the `package_group` function. They +have three properties: the list of packages they contain, their name, and other +package groups they include. The only allowed ways to refer to them are from the +`visibility` attribute of rules or from the `default_visibility` attribute of +the `package` function; they do not generate or consume files. For more +information, refer to the [`package_group` +documentation](/reference/be/functions#package_group). + + + Labels + diff --git a/concepts/dependencies.mdx b/concepts/dependencies.mdx deleted file mode 100644 index 6d8c88cf..00000000 --- a/concepts/dependencies.mdx +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: 'Dependencies' ---- - -A target `A` *depends upon* a target `B` if `B` is needed by `A` at build or execution time. The *depends upon* relation induces a [Directed Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) over targets, and it is called a *dependency graph*. - -A target's *direct* dependencies are those other targets reachable by a path of length 1 in the dependency graph. A target's *transitive* dependencies are those targets upon which it depends via a path of any length through the graph. - -In fact, in the context of builds, there are two dependency graphs, the graph of *actual dependencies* and the graph of *declared dependencies*. Most of the time, the two graphs are so similar that this distinction need not be made, but it is useful for the discussion below. - -## Actual and declared dependencies - -A target `X` is *actually dependent* on target `Y` if `Y` must be present, built, and up-to-date in order for `X` to be built correctly. *Built* could mean generated, processed, compiled, linked, archived, compressed, executed, or any of the other kinds of tasks that routinely occur during a build. - -A target `X` has a *declared dependency* on target `Y` if there is a dependency edge from `X` to `Y` in the package of `X`. - -For correct builds, the graph of actual dependencies *A* must be a subgraph of the graph of declared dependencies *D*. That is, every pair of directly-connected nodes `x --> y` in *A* must also be directly connected in *D*. It can be said that *D* is an *overapproximation* of *A*. - -Important: *D* should not be too much of an overapproximation of *A* because redundant declared dependencies can make builds slower and binaries larger. - -`BUILD` file writers must explicitly declare all of the actual direct dependencies for every rule to the build system, and no more. - -Failure to observe this principle causes undefined behavior: the build may fail, but worse, the build may depend on some prior operations, or upon transitive declared dependencies the target happens to have. Bazel checks for missing dependencies and report errors, but it's not possible for this checking to be complete in all cases. - -You need not (and should not) attempt to list everything indirectly imported, even if it is *needed* by `A` at execution time. - -During a build of target `X`, the build tool inspects the entire transitive closure of dependencies of `X` to ensure that any changes in those targets are reflected in the final result, rebuilding intermediates as needed. - -The transitive nature of dependencies leads to a common mistake. Sometimes, code in one file may use code provided by an *indirect* dependency — a transitive but not direct edge in the declared dependency graph. Indirect dependencies don't appear in the `BUILD` file. Because the rule doesn't directly depend on the provider, there is no way to track changes, as shown in the following example timeline: - -### 1. Declared dependencies match actual dependencies - -At first, everything works. The code in package `a` uses code in package `b`. The code in package `b` uses code in package `c`, and thus `a` transitively depends on `c`. - -| `a/BUILD` | `b/BUILD` | -| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ``` -rule( - name = "a", - srcs = "a.in", - deps = "//b:b", -) - -``` | ``` -rule( - name = "b", - srcs = "b.in", - deps = "//c:c", -) - -``` | -| `a / a.in` | `b / b.in` | -| ``` -import b; -b.foo(); - -``` | ``` -import c; -function foo() { - c.bar(); -} - -``` | -| ![Declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Declared** dependency graph | ![Actual dependency graph that matches the declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Actual** dependency graph | - -The declared dependencies overapproximate the actual dependencies. All is well. - -### 2. Adding an undeclared dependency - -A latent hazard is introduced when someone adds code to `a` that creates a direct *actual* dependency on `c`, but forgets to declare it in the build file `a/BUILD`. - -| `a / a.in` |   | -| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| ``` - import b; - import c; - b.foo(); - c.garply(); - -``` |   | -| ![Declared dependency graph with arrows connecting a, b, and c](/docs/images/a_b_c.svg)**Declared** dependency graph | ![Actual dependency graph with arrows connecting a, b, and c. An arrow now connects A to C as well. This does not match the declared dependency graph](/docs/images/a_b_c_ac.svg)**Actual** dependency graph | - -The declared dependencies no longer overapproximate the actual dependencies. This may build ok, because the transitive closures of the two graphs are equal, but masks a problem: `a` has an actual but undeclared dependency on `c`. - -### 3. Divergence between declared and actual dependency graphs - -The hazard is revealed when someone refactors `b` so that it no longer depends on `c`, inadvertently breaking `a` through no fault of their own. - -|   | `b/BUILD` | -| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -|   | ``` -rule( - name = "b", - srcs = "b.in", - deps = "//d:d", -) - -``` | -|   | `b / b.in` | -|   | ``` - import d; - function foo() { - d.baz(); - } - -``` | -| ![Declared dependency graph with arrows connecting a and b. b no longer connects to c, which breaks a's connection to c](/docs/images/ab_c.svg)**Declared** dependency graph | ![Actual dependency graph that shows a connecting to b and c, but b no longer connects to c](/docs/images/a_b_a_c.svg)**Actual** dependency graph | - -The declared dependency graph is now an underapproximation of the actual dependencies, even when transitively closed; the build is likely to fail. - -The problem could have been averted by ensuring that the actual dependency from `a` to `c` introduced in Step 2 was properly declared in the `BUILD` file. - -## Types of dependencies - -Most build rules have three attributes for specifying different kinds of generic dependencies: `srcs`, `deps` and `data`. These are explained below. For more details, see [Attributes common to all rules](/reference/be/common-definitions). - -Many rules also have additional attributes for rule-specific kinds of dependencies, for example, `compiler` or `resources`. These are detailed in the [Build Encyclopedia](/reference/be/). - -### `srcs` dependencies - -Files consumed directly by the rule or rules that output source files. - -### `deps` dependencies - -Rule pointing to separately-compiled modules providing header files, symbols, libraries, data, etc. - -### `data` dependencies - -A build target might need some data files to run correctly. These data files aren't source code: they don't affect how the target is built. For example, a unit test might compare a function's output to the contents of a file. When you build the unit test you don't need the file, but you do need it when you run the test. The same applies to tools that are launched during execution. - -The build system runs tests in an isolated directory where only files listed as `data` are available. Thus, if a binary/library/test needs some files to run, specify them (or a build rule containing them) in `data`. For example: - -``` -# I need a config file from a directory named env: -java_binary( - name = "setenv", - ... - data = [":env/default_env.txt"], -) - -# I need test data from another directory -sh_test( - name = "regtest", - srcs = ["regtest.sh"], - data = [ - "//data:file1.txt", - "//data:file2.txt", - ... - ], -) -``` - -These files are available using the relative path `path/to/data/file`. In tests, you can refer to these files by joining the paths of the test's source directory and the workspace-relative path, for example, `${TEST_SRCDIR}/workspace/path/to/data/file`. - -## Using labels to reference directories - -As you look over our `BUILD` files, you might notice that some `data` labels refer to directories. These labels end with `/.` or `/` like these examples, which you should not use: - - -\`data = \["//data/regression:unittest/."]\` - -\`data = \["testdata/."]\` - -\`data = \["testdata/"]\` - - -This seems convenient, particularly for tests because it allows a test to use all the data files in the directory. - -But try not to do this. In order to ensure correct incremental rebuilds (and re-execution of tests) after a change, the build system must be aware of the complete set of files that are inputs to the build (or test). When you specify a directory, the build system performs a rebuild only when the directory itself changes (due to addition or deletion of files), but won't be able to detect edits to individual files as those changes don't affect the enclosing directory. Rather than specifying directories as inputs to the build system, you should enumerate the set of files contained within them, either explicitly or using the [`glob()`](/reference/be/functions#glob) function. (Use `**` to force the `glob()` to be recursive.) - - -\`data = glob(\["testdata/\*\*"])\` - - -Unfortunately, there are some scenarios where directory labels must be used. For example, if the `testdata` directory contains files whose names don't conform to the [label syntax](/concepts/labels#labels-lexical-specification), then explicit enumeration of files, or use of the [`glob()`](/reference/be/functions#glob) function produces an invalid labels error. You must use directory labels in this case, but beware of the associated risk of incorrect rebuilds described above. - -If you must use directory labels, keep in mind that you can't refer to the parent package with a relative `../` path; instead, use an absolute path like `//data/regression:unittest/.`. - -Note: Directory labels are only valid for data dependencies. If you try to use a directory as a label in an argument other than `data`, it will fail and you will get a (probably cryptic) error message. - -Any external rule, such as a test, that needs to use multiple files must explicitly declare its dependence on all of them. You can use `filegroup()` to group files together in the `BUILD` file: - -``` -filegroup( - name = 'my_data', - srcs = glob(['my_unittest_data/*']) -) -``` - -You can then reference the label `my_data` as the data dependency in your test. - -[← BUILD files](/concepts/build-files) · [Visibility →](/concepts/visibility) diff --git a/concepts/labels.mdx b/concepts/labels.mdx deleted file mode 100644 index bbef34cb..00000000 --- a/concepts/labels.mdx +++ /dev/null @@ -1,148 +0,0 @@ ---- -title: 'Labels' ---- - -A **label** is an identifier for a target. A typical label in its full canonical form looks like: - -```none -@@myrepo//my/app/main:app_binary -``` - -The first part of the label is the repository name, `@@myrepo`. The double-`@` syntax signifies that this is a [*canonical* repo name](/external/overview#canonical-repo-name), which is unique within the workspace. Labels with canonical repo names unambiguously identify a target no matter which context they appear in. - -Often the canonical repo name is an arcane string that looks like `@@rules_java++toolchains+local_jdk`. What is much more commonly seen is labels with an [*apparent* repo name](/external/overview#apparent-repo-name), which looks like: - -``` -@myrepo//my/app/main:app_binary -``` - -The only difference is the repo name being prefixed with one `@` instead of two. This refers to a repo with the apparent name `myrepo`, which could be different based on the context this label appears in. - -In the typical case that a label refers to the same repository from which it is used, the repo name part may be omitted. So, inside `@@myrepo` the first label is usually written as - -``` -//my/app/main:app_binary -``` - -The second part of the label is the un-qualified package name `my/app/main`, the path to the package relative to the repository root. Together, the repository name and the un-qualified package name form the fully-qualified package name `@@myrepo//my/app/main`. When the label refers to the same package it is used in, the package name (and optionally, the colon) may be omitted. So, inside `@@myrepo//my/app/main`, this label may be written either of the following ways: - -``` -app_binary -:app_binary -``` - -It is a matter of convention that the colon is omitted for files, but retained for rules, but it is not otherwise significant. - -The part of the label after the colon, `app_binary` is the un-qualified target name. When it matches the last component of the package path, it, and the colon, may be omitted. So, these two labels are equivalent: - -``` -//my/app/lib -//my/app/lib:lib -``` - -The name of a file target in a subdirectory of the package is the file's path relative to the package root (the directory containing the `BUILD` file). So, this file is in the `my/app/main/testdata` subdirectory of the repository: - -``` -//my/app/main:testdata/input.txt -``` - -Strings like `//my/app` and `@@some_repo//my/app` have two meanings depending on the context in which they are used: when Bazel expects a label, they mean `//my/app:app` and `@@some_repo//my/app:app`, respectively. But, when Bazel expects a package (e.g. in `package_group` specifications), they reference the package that contains that label. - -A common mistake in `BUILD` files is using `//my/app` to refer to a package, or to *all* targets in a package--it does not. Remember, it is equivalent to `//my/app:app`, so it names the `app` target in the `my/app` package of the current repository. - -However, the use of `//my/app` to refer to a package is encouraged in the specification of a `package_group` or in `.bzl` files, because it clearly communicates that the package name is absolute and rooted in the top-level directory of the workspace. - -Relative labels cannot be used to refer to targets in other packages; the repository identifier and package name must always be specified in this case. For example, if the source tree contains both the package `my/app` and the package `my/app/testdata` (each of these two directories has its own `BUILD` file), the latter package contains a file named `testdepot.zip`. Here are two ways (one wrong, one correct) to refer to this file within `//my/app:BUILD`: - - -\`testdata\` is a different package, so you can't use a relative path - - -``` -testdata/testdepot.zip -``` - - -refer to \`testdata\` with its full path - - -``` -//my/app/testdata:testdepot.zip -``` - -Labels starting with `@@//` are references to the main repository, which will still work even from external repositories. Therefore `@@//a/b/c` is different from `//a/b/c` when referenced from an external repository. The former refers back to the main repository, while the latter looks for `//a/b/c` in the external repository itself. This is especially relevant when writing rules in the main repository that refer to targets in the main repository, and will be used from external repositories. - -For information about the different ways you can refer to targets, see [target patterns](/run/build#specifying-build-targets). - -### Lexical specification of a label - -Label syntax discourages use of metacharacters that have special meaning to the shell. This helps to avoid inadvertent quoting problems, and makes it easier to construct tools and scripts that manipulate labels, such as the [Bazel Query Language](/query/language). - -The precise details of allowed target names are below. - -### Target names — `<var>package-name</var>:target-name` - -`target-name` is the name of the target within the package. The name of a rule is the value of the `name` attribute in the rule's declaration in a `BUILD` file; the name of a file is its pathname relative to the directory containing the `BUILD` file. - -Target names must be composed entirely of characters drawn from the set `a`–`z`, `A`–`Z`, `0`–`9`, and the punctuation symbols `!%-@^_"#$&'()*-+,;<=>?[]{|}~/.`. - -Filenames must be relative pathnames in normal form, which means they must neither start nor end with a slash (for example, `/foo` and `foo/` are forbidden) nor contain multiple consecutive slashes as path separators (for example, `foo//bar`). Similarly, up-level references (`..`) and current-directory references (`./`) are forbidden. - - -Do not use \`..\` to refer to files in other packages - -Use \`//package-name:filename\` - - -While it is common to use `/` in the name of a file target, avoid the use of `/` in the names of rules. Especially when the shorthand form of a label is used, it may confuse the reader. The label `//foo/bar/wiz` is always a shorthand for `//foo/bar/wiz:wiz`, even if there is no such package `foo/bar/wiz`; it never refers to `//foo:bar/wiz`, even if that target exists. - -However, there are some situations where use of a slash is convenient, or sometimes even necessary. For example, the name of certain rules must match their principal source file, which may reside in a subdirectory of the package. - -### Package names — `//package-name:<var>target-name</var>` - -The name of a package is the name of the directory containing its `BUILD` file, relative to the top-level directory of the containing repository. For example: `my/app`. - -On a technical level, Bazel enforces the following: - -- Allowed characters in package names are the lowercase letters `a` through `z`, the uppercase letters `A` through `Z`, the digits `0` through `9`, the characters ``! \"#$%&'()*+,-.;<=>?@[]^_`{|}`` (yes, there's a space character in there!), and of course forward slash `/` (since it's the directory separator). -- Package names may not start or end with a forward slash character `/`. -- Package names may not contain the substring `//`. This wouldn't make sense---what would the corresponding directory path be? -- Package names may not contain the substring `/./` or `/../` or `/.../` etc. This enforcement is done to avoid confusion when translating between a logical package name and a physical directory name, given the semantic meaning of the dot character in path strings. - -On a practical level: - -- For a language with a directory structure that is significant to its module system (for example, Java), it's important to choose directory names that are valid identifiers in the language. For example, don't start with a leading digit and avoid special characters, especially underscores and hyphens. -- Although Bazel supports targets in the workspace's root package (for example, `//:foo`), it's best to leave that package empty so all meaningful packages have descriptive names. - -## Rules - -A rule specifies the relationship between inputs and outputs, and the steps to build the outputs. Rules can be of one of many different kinds (sometimes called the *rule class*), which produce compiled executables and libraries, test executables and other supported outputs as described in the [Build Encyclopedia](/reference/be/overview). - -`BUILD` files declare *targets* by invoking *rules*. - -In the example below, we see the declaration of the target `my_app` using the `cc_binary` rule. - -```python -cc_binary( - name = "my_app", - srcs = ["my_app.cc"], - deps = [ - "//absl/base", - "//absl/strings", - ], -) -``` - -Every rule invocation has a `name` attribute (which must be a valid [target name](#target-names)), that declares a target within the package of the `BUILD` file. - -Every rule has a set of *attributes*; the applicable attributes for a given rule, and the significance and semantics of each attribute are a function of the rule's kind; see the [Build Encyclopedia](/reference/be/overview) for a list of rules and their corresponding attributes. Each attribute has a name and a type. Some of the common types an attribute can have are integer, label, list of labels, string, list of strings, output label, list of output labels. Not all attributes need to be specified in every rule. Attributes thus form a dictionary from keys (names) to optional, typed values. - -The `srcs` attribute present in many rules has type "list of labels"; its value, if present, is a list of labels, each being the name of a target that is an input to this rule. - -In some cases, the name of the rule kind is somewhat arbitrary, and more interesting are the names of the files generated by the rule, and this is true of genrules. For more information, see [General Rules: genrule](/reference/be/general#genrule). - -In other cases, the name is significant: for `*_binary` and `*_test` rules, for example, the rule name determines the name of the executable produced by the build. - -This directed acyclic graph over targets is called the *target graph* or *build dependency graph*, and is the domain over which the [Bazel Query tool](/query/guide) operates. - -[← Targets](/concepts/build-ref) · [BUILD files →](/concepts/build-files) diff --git a/concepts/platforms.mdx b/concepts/platforms.mdx index e2ecbde8..e560ea4d 100644 --- a/concepts/platforms.mdx +++ b/concepts/platforms.mdx @@ -2,23 +2,30 @@ title: 'Migrating to Platforms' --- -Bazel has sophisticated [support](#background) for modeling [platforms](/extending/platforms) and [toolchains](/extending/toolchains) for multi-architecture and cross-compiled builds. + + +Bazel has sophisticated [support](#background) for modeling +[platforms][Platforms] and [toolchains][Toolchains] for multi-architecture and +cross-compiled builds. This page summarizes the state of this support. -Key Point: Bazel's platform and toolchain APIs are available today. Not all languages support them. Use these APIs with your project if you can. Bazel is migrating all major languages so eventually all builds will be platform-based. +Key Point: Bazel's platform and toolchain APIs are available today. Not all +languages support them. Use these APIs with your project if you can. Bazel is +migrating all major languages so eventually all builds will be platform-based. See also: -- [Platforms](/extending/platforms) -- [Toolchains](/extending/toolchains) -- [Background](#background) +* [Platforms][Platforms] +* [Toolchains][Toolchains] +* [Background][Background] ## Status ### C++ -C++ rules use platforms to select toolchains when `--incompatible_enable_cc_toolchain_resolution` is set. +C++ rules use platforms to select toolchains when +`--incompatible_enable_cc_toolchain_resolution` is set. This means you can configure a C++ project with: @@ -34,19 +41,23 @@ bazel build //:my_cpp_project` --cpu=... --crosstool_top=... --compiler=... This will be enabled by default in Bazel 7.0 ([#7260](https://github.com/bazelbuild/bazel/issues/7260)). -To test your C++ project with platforms, see [Migrating Your Project](#migrating-your-project) and [Configuring C++ toolchains](/tutorials/ccp-toolchain-config). +To test your C++ project with platforms, see +[Migrating Your Project](#migrating-your-project) and +[Configuring C++ toolchains]. ### Java Java rules use platforms to select toolchains. -This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, `--javabase`, and `--host_javabase`. +This replaces legacy flags `--java_toolchain`, `--host_java_toolchain`, +`--javabase`, and `--host_javabase`. See [Java and Bazel](/docs/bazel-and-java) for details. ### Android -Android rules use platforms to select toolchains when `--incompatible_enable_android_toolchain_resolution` is set. +Android rules use platforms to select toolchains when +`--incompatible_enable_android_toolchain_resolution` is set. This means you can configure an Android project with: @@ -54,42 +65,63 @@ This means you can configure an Android project with: bazel build //:my_android_project --android_platforms=//:my_android_platform ``` -instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, and `--fat_apk_cpu`. +instead of with legacy flags like `--android_crosstool_top`, `--android_cpu`, +and `--fat_apk_cpu`. This will be enabled by default in Bazel 7.0 ([#16285](https://github.com/bazelbuild/bazel/issues/16285)). -To test your Android project with platforms, see [Migrating Your Project](#migrating-your-project). +To test your Android project with platforms, see +[Migrating Your Project](#migrating-your-project). ### Apple -[Apple rules](https://github.com/bazelbuild/rules_apple) do not support platforms and are not yet scheduled for support. +[Apple rules] do not support platforms and are not yet scheduled +for support. -You can still use platform APIs with Apple builds (for example, when building with a mixture of Apple rules and pure C++) with [platform mappings](#platform-mappings). +You can still use platform APIs with Apple builds (for example, when building +with a mixture of Apple rules and pure C++) with [platform +mappings](#platform-mappings). ### Other languages -- [Go rules](https://github.com/bazelbuild/rules_go) fully support platforms -- [Rust rules](https://github.com/bazelbuild/rules_rust) fully support platforms. +* [Go rules] fully support platforms +* [Rust rules] fully support platforms. -If you own a language rule set, see [Migrating your rule set](#migrating-your-rule-set) for adding support. +If you own a language rule set, see [Migrating your rule set] for adding +support. ## Background -*Platforms* and *toolchains* were introduced to standardize how software projects target different architectures and cross-compile. +*Platforms* and *toolchains* were introduced to standardize how software +projects target different architectures and cross-compile. -This was [inspired](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) by the observation that language maintainers were already doing this in ad hoc, incompatible ways. For example, C++ rules used `--cpu` and `--crosstool_top` to declare a target CPU and toolchain. Neither of these correctly models a "platform". This produced awkward and incorrect builds. +This was +[inspired][Inspiration] +by the observation that language maintainers were already doing this in ad +hoc, incompatible ways. For example, C++ rules used `--cpu` and + `--crosstool_top` to declare a target CPU and toolchain. Neither of these +correctly models a "platform". This produced awkward and incorrect builds. -Java, Android, and other languages evolved their own flags for similar purposes, none of which interoperated with each other. This made cross-language builds confusing and complicated. +Java, Android, and other languages evolved their own flags for similar purposes, +none of which interoperated with each other. This made cross-language builds +confusing and complicated. -Bazel is intended for large, multi-language, multi-platform projects. This demands more principled support for these concepts, including a clear standard API. +Bazel is intended for large, multi-language, multi-platform projects. This +demands more principled support for these concepts, including a clear +standard API. ### Need for migration -Upgrading to the new API requires two efforts: releasing the API and upgrading rule logic to use it. +Upgrading to the new API requires two efforts: releasing the API and upgrading +rule logic to use it. -The first is done but the second is ongoing. This consists of ensuring language-specific platforms and toolchains are defined, language logic reads toolchains through the new API instead of old flags like `--crosstool_top`, and `config_setting`s select on the new API instead of old flags. +The first is done but the second is ongoing. This consists of ensuring +language-specific platforms and toolchains are defined, language logic reads +toolchains through the new API instead of old flags like `--crosstool_top`, and +`config_setting`s select on the new API instead of old flags. -This work is straightforward but requires a distinct effort for each language, plus fair warning for project owners to test against upcoming changes. +This work is straightforward but requires a distinct effort for each language, +plus fair warning for project owners to test against upcoming changes. This is why this is an ongoing migration. @@ -104,18 +136,25 @@ bazel build //:myproject --platforms=//:myplatform This implies: 1. Your project's rules choose the right toolchains for `//:myplatform`. -2. Your project's dependencies choose the right toolchains for `//:myplatform`. -3. `//:myplatform` references [common declarations](https://github.com/bazelbuild/platforms) of `CPU`, `OS`, and other generic, language-independent properties -4. All relevant [`select()`s](/docs/configurable-attributes) properly match `//:myplatform`. -5. `//:myplatform` is defined in a clear, accessible place: in your project's repo if the platform is unique to your project, or some common place all consuming projects can find it - -Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be deprecated and removed as soon as it's safe to do so. +1. Your project's dependencies choose the right toolchains for `//:myplatform`. +1. `//:myplatform` references +[common declarations][Common Platform Declarations] +of `CPU`, `OS`, and other generic, language-independent properties +1. All relevant [`select()`s][select()] properly match `//:myplatform`. +1. `//:myplatform` is defined in a clear, accessible place: in your project's +repo if the platform is unique to your project, or some common place all +consuming projects can find it + +Old flags like `--cpu`, `--crosstool_top`, and `--fat_apk_cpu` will be +deprecated and removed as soon as it's safe to do so. Ultimately, this will be the *sole* way to configure architectures. + ## Migrating your project -If you build with languages that support platforms, your build should already work with an invocation like: +If you build with languages that support platforms, your build should already +work with an invocation like: ```posix-terminal bazel build //:myproject --platforms=//:myplatform @@ -123,29 +162,49 @@ bazel build //:myproject --platforms=//:myplatform See [Status](#status) and your language's documentation for precise details. -If a language requires a flag to enable platform support, you also need to set that flag. See [Status](#status) for details. +If a language requires a flag to enable platform support, you also need to set +that flag. See [Status](#status) for details. For your project to build, you need to check the following: -1. `//:myplatform` must exist. It's generally the project owner's responsibility to define platforms because different projects target different machines. See [Default platforms](#default-platforms). +1. `//:myplatform` must exist. It's generally the project owner's responsibility + to define platforms because different projects target different machines. + See [Default platforms](#default-platforms). -2. The toolchains you want to use must exist. If using stock toolchains, the language owners should include instructions for how to register them. If writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). +1. The toolchains you want to use must exist. If using stock toolchains, the + language owners should include instructions for how to register them. If + writing your own custom toolchains, you need to [register](https://bazel.build/extending/toolchains#registering-building-toolchains) them in your + `MODULE.bazel` file or with [`--extra_toolchains`](https://bazel.build/reference/command-line-reference#flag--extra_toolchains). -3. `select()`s and [configuration transitions](/extending/config#user-defined-transitions) must resolve properly. See [select()](#select) and [Transitions](#transitions). +1. `select()`s and [configuration transitions][Starlark transitions] must + resolve properly. See [select()](#select) and [Transitions](#transitions). -4. If your build mixes languages that do and don't support platforms, you may need platform mappings to help the legacy languages work with the new API. See [Platform mappings](#platform-mappings) for details. +1. If your build mixes languages that do and don't support platforms, you may + need platform mappings to help the legacy languages work with the new API. + See [Platform mappings](#platform-mappings) for details. If you still have problems, [reach out](#questions) for support. ### Default platforms -Project owners should define explicit [platforms](/extending/platforms#constraints-platforms) to describe the architectures they want to build for. These are then triggered with `--platforms`. +Project owners should define explicit +[platforms][Defining Constraints and Platforms] to describe the architectures +they want to build for. These are then triggered with `--platforms`. -When `--platforms` isn't set, Bazel defaults to a `platform` representing the local build machine. This is auto-generated at `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`) so there's no need to explicitly define it. It maps the local machine's `OS` and `CPU` with `constraint_value`s declared in [`@platforms`](https://github.com/bazelbuild/platforms). +When `--platforms` isn't set, Bazel defaults to a `platform` representing the +local build machine. This is auto-generated at `@platforms//host` (aliased as +`@bazel_tools//tools:host_platform`) +so there's no need to explicitly define it. It maps the local machine's `OS` +and `CPU` with `constraint_value`s declared in +[`@platforms`](https://github.com/bazelbuild/platforms). ### `select()` -Projects can [`select()`](/docs/configurable-attributes) on [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value) but not complete platforms. This is intentional so `select()` supports as wide a variety of machines as possible. A library with `ARM`-specific sources should support *all* `ARM`-powered machines unless there's reason to be more specific. +Projects can [`select()`][select()] on +[`constraint_value` targets][constraint_value Rule] but not complete +platforms. This is intentional so `select()` supports as wide a variety of +machines as possible. A library with `ARM`-specific sources should support *all* +`ARM`-powered machines unless there's reason to be more specific. To select on one or more `constraint_value`s, use: @@ -169,47 +228,75 @@ config_setting( ) ``` -More details [here](/docs/configurable-attributes#platforms). +More details [here][select() Platforms]. -`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. When migrating your project to platforms, you must either convert them to `constraint_values` or use [platform mappings](#platform-mappings) to support both styles during migration. +`select`s on `--cpu`, `--crosstool_top`, etc. don't understand `--platforms`. +When migrating your project to platforms, you must either convert them to +`constraint_values` or use [platform mappings](#platform-mappings) to support +both styles during migration. ### Transitions -[Starlark transitions](/extending/config#user-defined-transitions) change flags down parts of your build graph. If your project uses a transition that sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read `--platforms` won't see these changes. +[Starlark transitions][Starlark transitions] change +flags down parts of your build graph. If your project uses a transition that +sets `--cpu`, `--crossstool_top`, or other legacy flags, rules that read +`--platforms` won't see these changes. -When migrating your project to platforms, you must either convert changes like `return { "//command_line_option:cpu": "arm" }` to `return { "//command_line_option:platforms": "//:my_arm_platform" }` or use [platform mappings](#platform-mappings) to support both styles during migration. window. +When migrating your project to platforms, you must either convert changes like +`return { "//command_line_option:cpu": "arm" }` to `return { +"//command_line_option:platforms": "//:my_arm_platform" }` or use [platform +mappings](#platform-mappings) to support both styles during migration. +window. ## Migrating your rule set If you own a rule set and want to support platforms, you need to: -1. Have rule logic resolve toolchains with the toolchain API. See [toolchain API](/extending/toolchains) (`ctx.toolchains`). +1. Have rule logic resolve toolchains with the toolchain API. See + [toolchain API][Toolchains] (`ctx.toolchains`). -2. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so rule logic alternately resolves toolchains through the new API or old flags like `--crosstool_top` during migration testing. +1. Optional: define an `--incompatible_enable_platforms_for_my_language` flag so + rule logic alternately resolves toolchains through the new API or old flags + like `--crosstool_top` during migration testing. -3. Define the relevant properties that make up platform components. See [Common platform properties](#common-platform-properties) +1. Define the relevant properties that make up platform components. See + [Common platform properties](#common-platform-properties) -4. Define standard toolchains and make them accessible to users through your rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) +1. Define standard toolchains and make them accessible to users through your + rule's registration instructions ([details](https://bazel.build/extending/toolchains#registering-building-toolchains)) -5. Ensure [`select()`s](#select) and [configuration transitions](#transitions) support platforms. This is the biggest challenge. It's particularly challenging for multi-language projects (which may fail if *all* languages can't read `--platforms`). +1. Ensure [`select()`s](#select) and + [configuration transitions](#transitions) support platforms. This is the + biggest challenge. It's particularly challenging for multi-language projects + (which may fail if *all* languages can't read `--platforms`). -If you need to mix with rules that don't support platforms, you may need [platform mappings](#platform-mappings) to bridge the gap. +If you need to mix with rules that don't support platforms, you may need +[platform mappings](#platform-mappings) to bridge the gap. ### Common platform properties -Common, cross-language platform properties like `OS` and `CPU` should be declared in [`@platforms`](https://github.com/bazelbuild/platforms). This encourages sharing, standardization, and cross-language compatibility. +Common, cross-language platform properties like `OS` and `CPU` should be +declared in [`@platforms`](https://github.com/bazelbuild/platforms). +This encourages sharing, standardization, and cross-language compatibility. -Properties unique to your rules should be declared in your rule's repo. This lets you maintain clear ownership over the specific concepts your rules are responsible for. +Properties unique to your rules should be declared in your rule's repo. This +lets you maintain clear ownership over the specific concepts your rules are +responsible for. -If your rules use custom-purpose OSes or CPUs, these should be declared in your rule's repo vs. [`@platforms`](https://github.com/bazelbuild/platforms). +If your rules use custom-purpose OSes or CPUs, these should be declared in your +rule's repo vs. +[`@platforms`](https://github.com/bazelbuild/platforms). ## Platform mappings -*Platform mappings* is a temporary API that lets platform-aware logic mix with legacy logic in the same build. This is a blunt tool that's only intended to smooth incompatibilities with different migration timeframes. +*Platform mappings* is a temporary API that lets platform-aware logic mix with +legacy logic in the same build. This is a blunt tool that's only intended to +smooth incompatibilities with different migration timeframes. -Caution: Only use this if necessary, and expect to eventually eliminate it. +Caution: Only use this if necessary, and expect to eventually eliminate it. -A platform mapping is a map of either a `platform()` to a corresponding set of legacy flags or the reverse. For example: +A platform mapping is a map of either a `platform()` to a +corresponding set of legacy flags or the reverse. For example: ```python platforms: @@ -230,15 +317,20 @@ flags: //platforms:macos ``` -Bazel uses this to guarantee all settings, both platform-based and legacy, are consistently applied throughout the build, including through [transitions](#transitions). +Bazel uses this to guarantee all settings, both platform-based and +legacy, are consistently applied throughout the build, including through +[transitions](#transitions). -By default Bazel reads mappings from the `platform_mappings` file in your workspace root. You can also set `--platform_mappings=//:my_custom_mapping`. +By default Bazel reads mappings from the `platform_mappings` file in your +workspace root. You can also set +`--platform_mappings=//:my_custom_mapping`. -See the [platform mappings design](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit) for details. +See the [platform mappings design] for details. ## API review -A [`platform`](/reference/be/platforms-and-toolchains#platform) is a collection of [`constraint_value` targets](/reference/be/platforms-and-toolchains#constraint_value): +A [`platform`][platform Rule] is a collection of +[`constraint_value` targets][constraint_value Rule]: ```python platform( @@ -250,7 +342,9 @@ platform( ) ``` -A [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) is a machine property. Values of the same "kind" are grouped under a common [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting): +A [`constraint_value`][constraint_value Rule] is a machine +property. Values of the same "kind" are grouped under a common +[`constraint_setting`][constraint_setting Rule]: ```python constraint_setting(name = "os") @@ -264,27 +358,72 @@ constraint_value( ) ``` -A [`toolchain`](/extending/toolchains) is a [Starlark rule](/extending/rules). Its attributes declare a language's tools (like `compiler = "//mytoolchain:custom_gcc"`). Its [providers](/extending/rules#providers) pass this information to rules that need to build with these tools. +A [`toolchain`][Toolchains] is a [Starlark rule][Starlark rule]. Its +attributes declare a language's tools (like `compiler = +"//mytoolchain:custom_gcc"`). Its [providers][Starlark Provider] pass +this information to rules that need to build with these tools. -Toolchains declare the `constraint_value`s of machines they can [target](/reference/be/platforms-and-toolchains#toolchain.target_compatible_with) (`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can [run on](/reference/be/platforms-and-toolchains#toolchain.exec_compatible_with) (`exec_compatible_with = ["@platforms//os:mac"]`). +Toolchains declare the `constraint_value`s of machines they can +[target][target_compatible_with Attribute] +(`target_compatible_with = ["@platforms//os:linux"]`) and machines their tools can +[run on][exec_compatible_with Attribute] +(`exec_compatible_with = ["@platforms//os:mac"]`). -When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel automatically selects a toolchain that can run on the build machine and build binaries for `//:myplatform`. This is known as *toolchain resolution*. +When building `$ bazel build //:myproject --platforms=//:myplatform`, Bazel +automatically selects a toolchain that can run on the build machine and +build binaries for `//:myplatform`. This is known as *toolchain resolution*. -The set of available toolchains can be registered in the `MODULE.bazel` file with [`register_toolchains`](/rules/lib/globals/module#register_toolchains) or at the command line with [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). +The set of available toolchains can be registered in the `MODULE.bazel` file +with [`register_toolchains`][register_toolchains Function] or at the +command line with [`--extra_toolchains`][extra_toolchains Flag]. -For more information see [here](/extending/toolchains). +For more information see [here][Toolchains]. ## Questions -For general support and questions about the migration timeline, contact [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) or the owners of the appropriate rules. +For general support and questions about the migration timeline, contact +[bazel-discuss] or the owners of the appropriate rules. -For discussions on the design and evolution of the platform/toolchain APIs, contact [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev). +For discussions on the design and evolution of the platform/toolchain APIs, +contact [bazel-dev]. ## See also -- [Configurable Builds - Part 1](https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html) -- [Platforms](/extending/platforms) -- [Toolchains](/extending/toolchains) -- [Bazel Platforms Cookbook](https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/) -- [Platforms examples](https://github.com/hlopko/bazel_platforms_examples) -- [Example C++ toolchain](https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms) +* [Configurable Builds - Part 1] +* [Platforms] +* [Toolchains] +* [Bazel Platforms Cookbook] +* [Platforms examples] +* [Example C++ toolchain] + +[Android Rules]: /docs/bazel-and-android +[Apple Rules]: https://github.com/bazelbuild/rules_apple +[Background]: #background +[Bazel platforms Cookbook]: https://docs.google.com/document/d/1UZaVcL08wePB41ATZHcxQV4Pu1YfA1RvvWm8FbZHuW8/ +[bazel-dev]: https://groups.google.com/forum/#!forum/bazel-dev +[bazel-discuss]: https://groups.google.com/forum/#!forum/bazel-discuss +[Common Platform Declarations]: https://github.com/bazelbuild/platforms +[constraint_setting Rule]: /reference/be/platforms-and-toolchains#constraint_setting +[constraint_value Rule]: /reference/be/platforms-and-toolchains#constraint_value +[Configurable Builds - Part 1]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html +[Configuring C++ toolchains]: /tutorials/ccp-toolchain-config +[Defining Constraints and Platforms]: /extending/platforms#constraints-platforms +[Example C++ toolchain]: https://github.com/gregestren/snippets/tree/master/custom_cc_toolchain_with_platforms +[exec_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.exec_compatible_with +[extra_toolchains Flag]: /reference/command-line-reference#flag--extra_toolchains +[Go Rules]: https://github.com/bazelbuild/rules_go +[Inspiration]: https://blog.bazel.build/2019/02/11/configurable-builds-part-1.html +[Migrating your rule set]: #migrating-your-rule-set +[Platforms]: /extending/platforms +[Platforms examples]: https://github.com/hlopko/bazel_platforms_examples +[platform mappings design]: https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls/edit +[platform Rule]: /reference/be/platforms-and-toolchains#platform +[register_toolchains Function]: /rules/lib/globals/module#register_toolchains +[Rust rules]: https://github.com/bazelbuild/rules_rust +[select()]: /docs/configurable-attributes +[select() Platforms]: /docs/configurable-attributes#platforms +[Starlark provider]: /extending/rules#providers +[Starlark rule]: /extending/rules +[Starlark transitions]: /extending/config#user-defined-transitions +[target_compatible_with Attribute]: /reference/be/platforms-and-toolchains#toolchain.target_compatible_with +[Toolchains]: /extending/toolchains diff --git a/concepts/visibility.mdx b/concepts/visibility.mdx index 10fd9eec..982f0a0e 100644 --- a/concepts/visibility.mdx +++ b/concepts/visibility.mdx @@ -2,49 +2,97 @@ title: 'Visibility' --- -This page covers Bazel's two visibility systems: [target visibility](#target-visibility) and [load visibility](#load-visibility). -Both types of visibility help other developers distinguish between your library's public API and its implementation details, and help enforce structure as your workspace grows. You can also use visibility when deprecating a public API to allow current users while denying new ones. + +This page covers Bazel's two visibility systems: +[target visibility](#target-visibility) and [load visibility](#load-visibility). + +Both types of visibility help other developers distinguish between your +library's public API and its implementation details, and help enforce structure +as your workspace grows. You can also use visibility when deprecating a public +API to allow current users while denying new ones. ## Target visibility -**Target visibility** controls who may depend on your target — that is, who may use your target's label inside an attribute such as `deps`. A target will fail to build during the [analysis](/reference/glossary#analysis-phase) phase if it violates the visibility of one of its dependencies. +**Target visibility** controls who may depend on your target — that is, who may +use your target's label inside an attribute such as `deps`. A target will fail +to build during the [analysis](/reference/glossary#analysis-phase) phase if it +violates the visibility of one of its dependencies. -Generally, a target `A` is visible to a target `B` if they are in the same location, or if `A` grants visibility to `B`'s location. In the absence of [symbolic macros](/extending/macros), the term "location" can be simplified to just "package"; see [below](#symbolic-macros) for more on symbolic macros. +Generally, a target `A` is visible to a target `B` if they are in the same +location, or if `A` grants visibility to `B`'s location. In the absence of +[symbolic macros](/extending/macros), the term "location" can be simplified +to just "package"; see [below](#symbolic-macros) for more on symbolic macros. -Visibility is specified by listing allowed packages. Allowing a package does not necessarily mean that its subpackages are also allowed. For more details on packages and subpackages, see [Concepts and terminology](/concepts/build-ref). +Visibility is specified by listing allowed packages. Allowing a package does not +necessarily mean that its subpackages are also allowed. For more details on +packages and subpackages, see [Concepts and terminology](/concepts/build-ref). -For prototyping, you can disable target visibility enforcement by setting the flag `--check_visibility=false`. This shouldn't be done for production usage in submitted code. +For prototyping, you can disable target visibility enforcement by setting the +flag `--check_visibility=false`. This shouldn't be done for production usage in +submitted code. -The primary way to control visibility is with a rule's [`visibility`](/reference/be/common-definitions#common.visibility) attribute. The following subsections describe the attribute's format, how to apply it to various kinds of targets, and the interaction between the visibility system and symbolic macros. +The primary way to control visibility is with a rule's +[`visibility`](/reference/be/common-definitions#common.visibility) attribute. +The following subsections describe the attribute's format, how to apply it to +various kinds of targets, and the interaction between the visibility system and +symbolic macros. ### Visibility specifications -All rule targets have a `visibility` attribute that takes a list of labels. Each label has one of the following forms. With the exception of the last form, these are just syntactic placeholders that don't correspond to any actual target. +All rule targets have a `visibility` attribute that takes a list of labels. Each +label has one of the following forms. With the exception of the last form, these +are just syntactic placeholders that don't correspond to any actual target. -- `"//visibility:public"`: Grants access to all packages. +* `"//visibility:public"`: Grants access to all packages. -- `"//visibility:private"`: Does not grant any additional access; only targets in this location's package can use this target. +* `"//visibility:private"`: Does not grant any additional access; only targets + in this location's package can use this target. -- `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its subpackages). +* `"//foo/bar:__pkg__"`: Grants access to `//foo/bar` (but not its + subpackages). -- `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its direct and indirect subpackages. +* `"//foo/bar:__subpackages__"`: Grants access to `//foo/bar` and all of its + direct and indirect subpackages. -- `"//some_pkg:my_package_group"`: Grants access to all of the packages that are part of the given [`package_group`](/reference/be/functions#package_group). +* `"//some_pkg:my_package_group"`: Grants access to all of the packages that + are part of the given [`package_group`](/reference/be/functions#package_group). - - Package groups use a [different syntax](/reference/be/functions#package_group.packages) for specifying packages. Within a package group, the forms `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, `"//visibility:public"` and `"//visibility:private"` are just `"public"` and `"private"`. + * Package groups use a + [different syntax](/reference/be/functions#package_group.packages) for + specifying packages. Within a package group, the forms + `"//foo/bar:__pkg__"` and `"//foo/bar:__subpackages__"` are respectively + replaced by `"//foo/bar"` and `"//foo/bar/..."`. Likewise, + `"//visibility:public"` and `"//visibility:private"` are just `"public"` + and `"private"`. -For example, if `//some/package:mytarget` has its `visibility` set to `[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target that is part of the `//some/package/...` source tree, as well as targets declared in `//tests/BUILD`, but not by targets defined in `//tests/integration/BUILD`. +For example, if `//some/package:mytarget` has its `visibility` set to +`[":__subpackages__", "//tests:__pkg__"]`, then it could be used by any target +that is part of the `//some/package/...` source tree, as well as targets +declared in `//tests/BUILD`, but not by targets defined in +`//tests/integration/BUILD`. -**Best practice:** To make several targets visible to the same set of packages, use a `package_group` instead of repeating the list in each target's `visibility` attribute. This increases readability and prevents the lists from getting out of sync. +**Best practice:** To make several targets visible to the same set +of packages, use a `package_group` instead of repeating the list in each +target's `visibility` attribute. This increases readability and prevents the +lists from getting out of sync. -**Best practice:** When granting visibility to another team's project, prefer `__subpackages__` over `__pkg__` to avoid needless visibility churn as that project evolves and adds new subpackages. +**Best practice:** When granting visibility to another team's project, prefer +`__subpackages__` over `__pkg__` to avoid needless visibility churn as that +project evolves and adds new subpackages. -Note: The `visibility` attribute may not specify non-`package_group` targets. Doing so triggers a "Label does not refer to a package group" or "Cycle in dependency graph" error. +Note: The `visibility` attribute may not specify non-`package_group` targets. +Doing so triggers a "Label does not refer to a package group" or "Cycle in +dependency graph" error. ### Rule target visibility -A rule target's visibility is determined by taking its `visibility` attribute -- or a suitable default if not given -- and appending the location where the target was declared. For targets not declared in a symbolic macro, if the package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), this default is used; for all other packages and for targets declared in a symbolic macro, the default is just `["//visibility:private"]`. +A rule target's visibility is determined by taking its `visibility` attribute +-- or a suitable default if not given -- and appending the location where the +target was declared. For targets not declared in a symbolic macro, if the +package specifies a [`default_visibility`](/reference/be/functions#package.default_visibility), +this default is used; for all other packages and for targets declared in a +symbolic macro, the default is just `["//visibility:private"]`. ```starlark # //mypkg/BUILD @@ -82,11 +130,15 @@ package_group( ) ``` -**Best practice:** Avoid setting `default_visibility` to public. It may be convenient for prototyping or in small codebases, but the risk of inadvertently creating public targets increases as the codebase grows. It's better to be explicit about which targets are part of a package's public interface. +**Best practice:** Avoid setting `default_visibility` to public. It may be +convenient for prototyping or in small codebases, but the risk of inadvertently +creating public targets increases as the codebase grows. It's better to be +explicit about which targets are part of a package's public interface. ### Generated file target visibility -A generated file target has the same visibility as the rule target that generates it. +A generated file target has the same visibility as the rule target that +generates it. ```starlark # //mypkg/BUILD @@ -116,21 +168,36 @@ some_rule( ### Source file target visibility -Source file targets can either be explicitly declared using [`exports_files`](/reference/be/functions#exports_files), or implicitly created by referring to their filename in a label attribute of a rule (outside of a symbolic macro). As with rule targets, the location of the call to `exports_files`, or the BUILD file that referred to the input file, is always automatically appended to the file's visibility. +Source file targets can either be explicitly declared using +[`exports_files`](/reference/be/functions#exports_files), or implicitly created +by referring to their filename in a label attribute of a rule (outside of a +symbolic macro). As with rule targets, the location of the call to +`exports_files`, or the BUILD file that referred to the input file, is always +automatically appended to the file's visibility. -Files declared by `exports_files` can have their visibility set by the `visibility` parameter to that function. If this parameter is not given, the visibility is public. +Files declared by `exports_files` can have their visibility set by the +`visibility` parameter to that function. If this parameter is not given, the visibility is public. -Note: `exports_files` may not be used to override the visibility of a generated file. +Note: `exports_files` may not be used to override the visibility of a generated +file. -For files that do not appear in a call to `exports_files`, the visibility depends on the value of the flag [`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): +For files that do not appear in a call to `exports_files`, the visibility +depends on the value of the flag +[`--incompatible_no_implicit_file_export`](https://github.com/bazelbuild/bazel/issues/10225): -- If the flag is true, the visibility is private. +* If the flag is true, the visibility is private. -- Else, the legacy behavior applies: The visibility is the same as the `BUILD` file's `default_visibility`, or private if a default visibility is not specified. +* Else, the legacy behavior applies: The visibility is the same as the + `BUILD` file's `default_visibility`, or private if a default visibility is + not specified. -Avoid relying on the legacy behavior. Always write an `exports_files` declaration whenever a source file target needs non-private visibility. +Avoid relying on the legacy behavior. Always write an `exports_files` +declaration whenever a source file target needs non-private visibility. -**Best practice:** When possible, prefer to expose a rule target rather than a source file. For example, instead of calling `exports_files` on a `.java` file, wrap the file in a non-private `java_library` target. Generally, rule targets should only directly reference source files that live in the same package. +**Best practice:** When possible, prefer to expose a rule target rather than a +source file. For example, instead of calling `exports_files` on a `.java` file, +wrap the file in a non-private `java_library` target. Generally, rule targets +should only directly reference source files that live in the same package. #### Example @@ -151,45 +218,98 @@ cc_binary( ### Config setting visibility -Historically, Bazel has not enforced visibility for [`config_setting`](/reference/be/general#config_setting) targets that are referenced in the keys of a [`select()`](/reference/be/functions#select). There are two flags to remove this legacy behavior: +Historically, Bazel has not enforced visibility for +[`config_setting`](/reference/be/general#config_setting) targets that are +referenced in the keys of a [`select()`](/reference/be/functions#select). There +are two flags to remove this legacy behavior: -- [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) enables visibility checking for these targets. To assist with migration, it also causes any `config_setting` that does not specify a `visibility` to be considered public (regardless of package-level `default_visibility`). +* [`--incompatible_enforce_config_setting_visibility`](https://github.com/bazelbuild/bazel/issues/12932) + enables visibility checking for these targets. To assist with migration, it + also causes any `config_setting` that does not specify a `visibility` to be + considered public (regardless of package-level `default_visibility`). -- [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) causes `config_setting`s that do not specify a `visibility` to respect the package's `default_visibility` and to fallback on private visibility, just like any other rule target. It is a no-op if `--incompatible_enforce_config_setting_visibility` is not set. +* [`--incompatible_config_setting_private_default_visibility`](https://github.com/bazelbuild/bazel/issues/12933) + causes `config_setting`s that do not specify a `visibility` to respect the + package's `default_visibility` and to fallback on private visibility, just + like any other rule target. It is a no-op if + `--incompatible_enforce_config_setting_visibility` is not set. -Avoid relying on the legacy behavior. Any `config_setting` that is intended to be used outside the current package should have an explicit `visibility`, if the package does not already specify a suitable `default_visibility`. +Avoid relying on the legacy behavior. Any `config_setting` that is intended to +be used outside the current package should have an explicit `visibility`, if the +package does not already specify a suitable `default_visibility`. ### Package group target visibility -`package_group` targets do not have a `visibility` attribute. They are always publicly visible. +`package_group` targets do not have a `visibility` attribute. They are always +publicly visible. ### Visibility of implicit dependencies -Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — dependencies that are not spelled out in a `BUILD` file but are inherent to every instance of that rule. For example, a `cc_library` rule might create an implicit dependency from each of its rule targets to an executable target representing a C++ compiler. +Some rules have [implicit dependencies](/extending/rules#private_attributes_and_implicit_dependencies) — +dependencies that are not spelled out in a `BUILD` file but are inherent to +every instance of that rule. For example, a `cc_library` rule might create an +implicit dependency from each of its rule targets to an executable target +representing a C++ compiler. -The visibility of such an implicit dependency is checked with respect to the package containing the `.bzl` file in which the rule (or aspect) is defined. In our example, the C++ compiler could be private so long as it lives in the same package as the definition of the `cc_library` rule. As a fallback, if the implicit dependency is not visible from the definition, it is checked with respect to the `cc_library` target. +The visibility of such an implicit dependency is checked with respect to the +package containing the `.bzl` file in which the rule (or aspect) is defined. In +our example, the C++ compiler could be private so long as it lives in the same +package as the definition of the `cc_library` rule. As a fallback, if the +implicit dependency is not visible from the definition, it is checked with +respect to the `cc_library` target. -If you want to restrict the usage of a rule to certain packages, use [load visibility](#load-visibility) instead. +If you want to restrict the usage of a rule to certain packages, use +[load visibility](#load-visibility) instead. ### Visibility and symbolic macros -This section describes how the visibility system interacts with [symbolic macros](/extending/macros). +This section describes how the visibility system interacts with +[symbolic macros](/extending/macros). #### Locations within symbolic macros -A key detail of the visibility system is how we determine the location of a declaration. For targets that are not declared in a symbolic macro, the location is just the package where the target lives -- the package of the `BUILD` file. But for targets created in a symbolic macro, the location is the package containing the `.bzl` file where the macro's definition (the `my_macro = macro(...)` statement) appears. When a target is created inside multiple nested targets, it is always the innermost symbolic macro's definition that is used. - -The same system is used to determine what location to check against a given dependency's visibility. If the consuming target was created inside a macro, we look at the innermost macro's definition rather than the package the consuming target lives in. - -This means that all macros whose code is defined in the same package are automatically "friends" with one another. Any target directly created by a macro defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, regardless of what packages the macros are actually instantiated in. Likewise, they can see, and can be seen by, targets declared directly in `//lib/BUILD` and its legacy macros. Conversely, targets that live in the same package cannot necessarily see one another if at least one of them is created by a symbolic macro. - -Within a symbolic macro's implementation function, the `visibility` parameter has the effective value of the macro's `visibility` attribute after appending the location where the macro was called. The standard way for a macro to export one of its targets to its caller is to forward this value along to the target's declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit this attribute won't be visible to the caller of the macro unless the caller happens to be in the same package as the macro definition. This behavior composes, in the sense that a chain of nested calls to submacros may each pass `visibility = visibility`, re-exporting the inner macro's exported targets to the caller at each level, without exposing any of the macros' implementation details. +A key detail of the visibility system is how we determine the location of a +declaration. For targets that are not declared in a symbolic macro, the location +is just the package where the target lives -- the package of the `BUILD` file. +But for targets created in a symbolic macro, the location is the package +containing the `.bzl` file where the macro's definition (the +`my_macro = macro(...)` statement) appears. When a target is created inside +multiple nested targets, it is always the innermost symbolic macro's definition +that is used. + +The same system is used to determine what location to check against a given +dependency's visibility. If the consuming target was created inside a macro, we +look at the innermost macro's definition rather than the package the consuming +target lives in. + +This means that all macros whose code is defined in the same package are +automatically "friends" with one another. Any target directly created by a macro +defined in `//lib:defs.bzl` can be seen from any other macro defined in `//lib`, +regardless of what packages the macros are actually instantiated in. Likewise, +they can see, and can be seen by, targets declared directly in `//lib/BUILD` and +its legacy macros. Conversely, targets that live in the same package cannot +necessarily see one another if at least one of them is created by a symbolic +macro. + +Within a symbolic macro's implementation function, the `visibility` parameter +has the effective value of the macro's `visibility` attribute after appending +the location where the macro was called. The standard way for a macro to export +one of its targets to its caller is to forward this value along to the target's +declaration, as in `some_rule(..., visibility = visibility)`. Targets that omit +this attribute won't be visible to the caller of the macro unless the caller +happens to be in the same package as the macro definition. This behavior +composes, in the sense that a chain of nested calls to submacros may each pass +`visibility = visibility`, re-exporting the inner macro's exported targets to +the caller at each level, without exposing any of the macros' implementation +details. #### Delegating privileges to a submacro -The visibility model has a special feature to allow a macro to delegate its permissions to a submacro. This is important for factoring and composing macros. +The visibility model has a special feature to allow a macro to delegate its +permissions to a submacro. This is important for factoring and composing macros. -Suppose you have a macro `my_macro` that creates a dependency edge using a rule `some_library` from another package: +Suppose you have a macro `my_macro` that creates a dependency edge using a rule +`some_library` from another package: ```starlark # //macro/defs.bzl @@ -218,7 +338,10 @@ load("//macro:defs.bzl", "my_macro") my_macro(name = "foo", ...) ``` -The `//pkg:foo_dependency` target has no `visibility` specified, so it is only visible within `//macro`, which works fine for the consuming target. Now, what happens if the author of `//lib` refactors `some_library` to instead be implemented using a macro? +The `//pkg:foo_dependency` target has no `visibility` specified, so it is only +visible within `//macro`, which works fine for the consuming target. Now, what +happens if the author of `//lib` refactors `some_library` to instead be +implemented using a macro? ```starlark # //lib:defs.bzl @@ -234,43 +357,88 @@ def _impl(name, visibility, deps, ...): some_library = macro(implementation = _impl, ...) ``` -With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than `//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's visibility. The author of `my_macro` can't be expected to pass `visibility = ["//lib"]` to the declaration of the dependency just to work around this implementation detail. +With this change, `//pkg:foo_consumer`'s location is now `//lib` rather than +`//macro`, so its usage of `//pkg:foo_dependency` violates the dependency's +visibility. The author of `my_macro` can't be expected to pass +`visibility = ["//lib"]` to the declaration of the dependency just to work +around this implementation detail. -For this reason, when a dependency of a target is also an attribute value of the macro that declared the target, we check the dependency's visibility against the location of the macro instead of the location of the consuming target. +For this reason, when a dependency of a target is also an attribute value of the +macro that declared the target, we check the dependency's visibility against the +location of the macro instead of the location of the consuming target. -In this example, to validate whether `//pkg:foo_consumer` can see `//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an input to the call to `some_library` inside of `my_macro`, and instead check the dependency's visibility against the location of this call, `//macro`. +In this example, to validate whether `//pkg:foo_consumer` can see +`//pkg:foo_dependency`, we see that `//pkg:foo_dependency` was also passed as an +input to the call to `some_library` inside of `my_macro`, and instead check the +dependency's visibility against the location of this call, `//macro`. -This process can repeat recursively, as long as a target or macro declaration is inside of another symbolic macro taking the dependency's label in one of its label-typed attributes. +This process can repeat recursively, as long as a target or macro declaration is +inside of another symbolic macro taking the dependency's label in one of its +label-typed attributes. -Note: Visibility delegation does not work for labels that were not passed into the macro, such as labels derived by string manipulation. +Note: Visibility delegation does not work for labels that were not passed into +the macro, such as labels derived by string manipulation. #### Finalizers -Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. +Targets declared in a rule finalizer (a symbolic macro with `finalizer = True`), +in addition to seeing targets following the usual symbolic macro visibility +rules, can *also* see all targets which are visible to the finalizer target's +package. -In other words, if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. +In other words, if you migrate a `native.existing_rules()`-based legacy macro to +a finalizer, the targets declared by the finalizer will still be able to see +their old dependencies. -It is possible to define targets that a finalizer can introspect using `native.existing_rules()`, but which it cannot use as dependencies under the visibility system. For example, if a macro-defined target is not visible to its own package or to the finalizer macro's definition, and is not delegated to the finalizer, the finalizer cannot see such a target. Note, however, that a `native.existing_rules()`-based legacy macro will also be unable to see such a target. +It is possible to define targets that a finalizer can introspect using +`native.existing_rules()`, but which it cannot use as dependencies under the +visibility system. For example, if a macro-defined target is not visible to its +own package or to the finalizer macro's definition, and is not delegated to the +finalizer, the finalizer cannot see such a target. Note, however, that a +`native.existing_rules()`-based legacy macro will also be unable to see such a +target. ## Load visibility -**Load visibility** controls whether a `.bzl` file may be loaded from other `BUILD` or `.bzl` files outside the current package. +**Load visibility** controls whether a `.bzl` file may be loaded from other +`BUILD` or `.bzl` files outside the current package. -In the same way that target visibility protects source code that is encapsulated by targets, load visibility protects build logic that is encapsulated by `.bzl` files. For instance, a `BUILD` file author might wish to factor some repetitive target declarations into a macro in a `.bzl` file. Without the protection of load visibility, they might find their macro reused by other collaborators in the same workspace, so that modifying the macro breaks other teams' builds. +In the same way that target visibility protects source code that is encapsulated +by targets, load visibility protects build logic that is encapsulated by `.bzl` +files. For instance, a `BUILD` file author might wish to factor some repetitive +target declarations into a macro in a `.bzl` file. Without the protection of +load visibility, they might find their macro reused by other collaborators in +the same workspace, so that modifying the macro breaks other teams' builds. -Note that a `.bzl` file may or may not have a corresponding source file target. If it does, there is no guarantee that the load visibility and the target visibility coincide. That is, the same `BUILD` file might be able to load the `.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), or vice versa. This can sometimes cause problems for rules that wish to consume `.bzl` files as source code, such as for documentation generation or testing. +Note that a `.bzl` file may or may not have a corresponding source file target. +If it does, there is no guarantee that the load visibility and the target +visibility coincide. That is, the same `BUILD` file might be able to load the +`.bzl` file but not list it in the `srcs` of a [`filegroup`](/reference/be/general#filegroup), +or vice versa. This can sometimes cause problems for rules that wish to consume +`.bzl` files as source code, such as for documentation generation or testing. -For prototyping, you may disable load visibility enforcement by setting `--check_bzl_visibility=false`. As with `--check_visibility=false`, this should not be done for submitted code. +For prototyping, you may disable load visibility enforcement by setting +`--check_bzl_visibility=false`. As with `--check_visibility=false`, this should +not be done for submitted code. Load visibility is available as of Bazel 6.0. ### Declaring load visibility -To set the load visibility of a `.bzl` file, call the [`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. The argument to `visibility()` is a list of package specifications, just like the [`packages`](/reference/be/functions#package_group.packages) attribute of `package_group`. However, `visibility()` does not accept negative package specifications. +To set the load visibility of a `.bzl` file, call the +[`visibility()`](/rules/lib/globals/bzl#visibility) function from within the file. +The argument to `visibility()` is a list of package specifications, just like +the [`packages`](/reference/be/functions#package_group.packages) attribute of +`package_group`. However, `visibility()` does not accept negative package +specifications. -The call to `visibility()` must only occur once per file, at the top level (not inside a function), and ideally immediately following the `load()` statements. +The call to `visibility()` must only occur once per file, at the top level (not +inside a function), and ideally immediately following the `load()` statements. -Unlike target visibility, the default load visibility is always public. Files that do not call `visibility()` are always loadable from anywhere in the workspace. It is a good idea to add `visibility("private")` to the top of any new `.bzl` file that is not specifically intended for use outside the package. +Unlike target visibility, the default load visibility is always public. Files +that do not call `visibility()` are always loadable from anywhere in the +workspace. It is a good idea to add `visibility("private")` to the top of any +new `.bzl` file that is not specifically intended for use outside the package. ### Example @@ -312,7 +480,8 @@ This section describes tips for managing load visibility declarations. #### Factoring visibilities -When multiple `.bzl` files should have the same visibility, it can be helpful to factor their package specifications into a common list. For example: +When multiple `.bzl` files should have the same visibility, it can be helpful to +factor their package specifications into a common list. For example: ```starlark # //mylib/internal_defs.bzl @@ -344,13 +513,18 @@ visibility(clients) ... ``` -This helps prevent accidental skew between the various `.bzl` files' visibilities. It also is more readable when the `clients` list is large. +This helps prevent accidental skew between the various `.bzl` files' +visibilities. It also is more readable when the `clients` list is large. #### Composing visibilities -Sometimes a `.bzl` file might need to be visible to an allowlist that is composed of multiple smaller allowlists. This is analogous to how a `package_group` can incorporate other `package_group`s via its [`includes`](/reference/be/functions#package_group.includes) attribute. +Sometimes a `.bzl` file might need to be visible to an allowlist that is +composed of multiple smaller allowlists. This is analogous to how a +`package_group` can incorporate other `package_group`s via its +[`includes`](/reference/be/functions#package_group.includes) attribute. -Suppose you are deprecating a widely used macro. You want it to be visible only to existing users and to the packages owned by your own team. You might write: +Suppose you are deprecating a widely used macro. You want it to be visible only +to existing users and to the packages owned by your own team. You might write: ```starlark # //mylib/macros.bzl @@ -364,7 +538,12 @@ visibility(our_packages + their_remaining_uses) #### Deduplicating with package groups -Unlike target visibility, you cannot define a load visibility in terms of a `package_group`. If you want to reuse the same allowlist for both target visibility and load visibility, it's best to move the list of package specifications into a .bzl file, where both kinds of declarations may refer to it. Building off the example in [Factoring visibilities](#factoring-visibilities) above, you might write: +Unlike target visibility, you cannot define a load visibility in terms of a +`package_group`. If you want to reuse the same allowlist for both target +visibility and load visibility, it's best to move the list of package +specifications into a .bzl file, where both kinds of declarations may refer to +it. Building off the example in [Factoring visibilities](#factoring-visibilities) +above, you might write: ```starlark # //mylib/BUILD @@ -377,11 +556,17 @@ package_group( ) ``` -This only works if the list does not contain any negative package specifications. +This only works if the list does not contain any negative package +specifications. #### Protecting individual symbols -Any Starlark symbol whose name begins with an underscore cannot be loaded from another file. This makes it easy to create private symbols, but does not allow you to share these symbols with a limited set of trusted files. On the other hand, load visibility gives you control over what other packages may see your `.bzl file`, but does not allow you to prevent any non-underscored symbol from being loaded. +Any Starlark symbol whose name begins with an underscore cannot be loaded from +another file. This makes it easy to create private symbols, but does not allow +you to share these symbols with a limited set of trusted files. On the other +hand, load visibility gives you control over what other packages may see your +`.bzl file`, but does not allow you to prevent any non-underscored symbol from +being loaded. Luckily, you can combine these two features to get fine-grained control. @@ -418,4 +603,8 @@ public_util = _public_util #### bzl-visibility Buildifier lint -There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) that provides a warning if users load a file from a directory named `internal` or `private`, when the user's file is not itself underneath the parent of that directory. This lint predates the load visibility feature and is unnecessary in workspaces where `.bzl` files declare visibilities. +There is a [Buildifier lint](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#bzl-visibility) +that provides a warning if users load a file from a directory named `internal` +or `private`, when the user's file is not itself underneath the parent of that +directory. This lint predates the load visibility feature and is unnecessary in +workspaces where `.bzl` files declare visibilities. diff --git a/configure/attributes.mdx b/configure/attributes.mdx index df76c4ba..7bc3f41e 100644 --- a/configure/attributes.mdx +++ b/configure/attributes.mdx @@ -2,9 +2,15 @@ title: 'Configurable Build Attributes' --- -***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. -This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. + +**_Configurable attributes_**, commonly known as [`select()`]( +/reference/be/functions#select), is a Bazel feature that lets users toggle the values +of build rule attributes at the command line. + +This can be used, for example, for a multiplatform library that automatically +chooses the appropriate implementation for the architecture, or for a +feature-configurable binary that can be customized at build time. ## Example @@ -35,30 +41,59 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: - -| | | -| ----------------------------------------------- | ------------------ | -| Command | deps = | -| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | -| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | - -`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the +command line. Specifically, `deps` becomes: + + + + + + + + + + + + + + + + + + + + + + +
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
+ +`select()` serves as a placeholder for a value that will be chosen based on +*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) +targets. By using `select()` in a configurable attribute, the attribute +effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either +* They all resolve to the same value. For example, when running on linux x86, this is unambiguous + `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` + is an unambiguous specialization of `values = {"cpu": "x86"}`. -- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. - -The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when +nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, +`resources`, `cmd`, and most other attributes. Only a small number of attributes +are *non-configurable*, and these are clearly annotated. For example, +`config_setting`'s own +[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies +under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of +the machine running Bazel (which, thanks to cross-compilation, may be different +than the CPU the target is built for). This is known as a +[configuration transition](/reference/glossary#transition). Given @@ -102,19 +137,30 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and +`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s +build parameters, which include `--cpu=arm`. The `tools` attribute changes +`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on +`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a +[`config_setting`](/reference/be/general#config_setting) or +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of +expected command line flag settings. By encapsulating these in a target, it's +easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands +them for all builds in all projects. These are specified with +[`config_setting`](/reference/be/general#config_setting)'s +[`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -127,21 +173,31 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` +is the expected value for that flag. `:meaningful_condition_name` matches if +*every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, +[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because +it only affects how Bazel reports progress to the user. Targets can't use that +flag to construct their results. The exact set of supported flags isn't +documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with +[Starlark build settings][BuildSettings]. Unlike built-in flags, these are +defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s +[`flag_values`](/reference/be/general#config_setting.flag_values) +attribute: ```python config_setting( @@ -154,17 +210,29 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) +for a working example. -[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) +is an alternative legacy syntax for custom flags (for example +`--define foo=bar`). This can be expressed either in the +[values](/reference/be/general#config_setting.values) attribute +(`values = {"define": "foo=bar"}`) or the +[define_values](/reference/be/general#config_setting.define_values) attribute +(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards +compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The +`config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition matches. +The built-in condition `//conditions:default` matches when no other condition +matches. -Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match +and no default condition emits a `"no matching conditions"` error. This can +protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -190,11 +258,16 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s +[`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides +flexibility, it can also be burdensome to individually set each one every time +you want to build a target. + [Platforms](/extending/platforms) +let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -252,9 +325,12 @@ platform( ) ``` -The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the +`config_setting`s that contain a subset of the platform's `constraint_values`, +allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, +you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -281,9 +357,11 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to check against single values. +This saves the need for boilerplate `config_setting`s when you only need to +check against single values. -Platforms are still under development. See the [documentation](/concepts/platforms) for details. +Platforms are still under development. See the +[documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -305,12 +383,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: + - Duplicate labels can appear in different paths of the same `select`. + - Duplicate labels can *not* appear within the same path of a `select`. + - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -- Duplicate labels can appear in different paths of the same `select`. -- Duplicate labels can *not* appear within the same path of a `select`. -- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) - -`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: +`select` cannot appear inside another `select`. If you need to nest `selects` +and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -331,7 +409,8 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND +chaining](#and-chaining). ## OR chaining @@ -350,7 +429,9 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. +Most conditions evaluate to the same dep. But this syntax is hard to read and +maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple +times. One option is to predefine the value as a BUILD variable: @@ -369,13 +450,18 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary duplication. +This makes it easier to manage the dependency. But it still causes unnecessary +duplication. For more direct support, use one of the following: ### `selects.with_or` -The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: +The +[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -394,12 +480,18 @@ sh_binary( ### `selects.config_setting_group` -The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: + +The +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` + ```python config_setting( name = "config1", @@ -423,13 +515,17 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across +different attributes. -It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous +"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the +[Skylib](https://github.com/bazelbuild/bazel-skylib) macro +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -454,11 +550,13 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed +inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to fails with the error: +By default, when no condition matches, the target the `select()` is attached to +fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -468,7 +566,8 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) +attribute: ```python cc_library( @@ -491,7 +590,8 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable attributes. For example, given: +Rule implementations receive the *resolved values* of configurable +attributes. For example, given: ```python # myapp/BUILD @@ -511,7 +611,9 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert +Macros can accept `select()` clauses and pass them through to native +rules. But *they cannot directly manipulate them*. For example, there's no way +for a macro to convert ```python select({"foo": "val"}, ...) @@ -525,22 +627,32 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* +because macros are evaluated in Bazel's [loading phase](/run/build#loading), +which occurs before flag values are known. +This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. +Second, macros that just need to iterate over *all* `select` paths, while +technically feasible, lack a coherent UI. Further design is necessary to change +this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's +[loading phase](/reference/glossary#loading-phase). +This means it doesn't know what command line flags a target uses since those +flags aren't evaluated until later in the build (in the +[analysis phase](/reference/glossary#analysis-phase)). +So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has +all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` - ```python # myapp/BUILD @@ -589,9 +701,14 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for +details. -The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in +*macros*. These are different than *rules*. See the +documentation on [rules](/extending/rules) and [macros](/extending/macros) +to understand the difference. +Here's an end-to-end example: Define a rule and macro: @@ -671,7 +788,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before +Bazel reads the build's command line flags. That means there isn't enough +information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -694,7 +813,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: +Because *macros* (but not rules) by definition +[can't evaluate `select()`s](#faq-select-macro), any attempt to do so +usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -707,7 +828,8 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly vigilant with them: +Booleans are a special case that fail silently, so you should be particularly +vigilant with them: ```sh $ cat myapp/defs.bzl @@ -729,13 +851,21 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. +This happens because macros don't understand the contents of `select()`. +So what they're really evaluting is the `select()` object itself. According to +[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design +standards, all objects aside from a very small number of exceptions +automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before +Bazel knows what the build's command line parameters are. Can they at least read +the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but it isn't yet a Bazel feature. What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: +Conceptually this is possible, but it isn't yet a Bazel feature. +What you *can* do today is prepare a straight dictionary, then feed it into a +`select()`: ```sh $ cat myapp/defs.bzl @@ -747,7 +877,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -779,7 +909,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -787,11 +917,17 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo +rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in +the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't +actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in +the `actual` attribute, to perform this type of run-time determination. This +works correctly, since `alias()` is a BUILD rule, and is evaluated with a +specific configuration. ```sh $ cat WORKSPACE @@ -817,31 +953,37 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target +that depends on either `//:ssl` or `//external:ssl` will see the alternative +located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, +use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo <desired build flags> +$ bazel cquery //myapp:foo //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on +//myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> +$ bazel cquery 'somepath(//bar, //myapp:foo)' //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the +configuration that resolves `//myapp:foo`'s `select()`. You can inspect its +values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -860,13 +1002,18 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. +`//myapp:foo` may exist in different configurations in the same build. See the +[cquery docs](/query/cquery) for guidance on using `somepath` to get the right +one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the +same command line flags as the `bazel cquery`. The `config` command relies on +the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform +is the target platform because the semantics are unclear. For example: @@ -889,11 +1036,15 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the +`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the +`:x86_linux_platform` defined here? The author of the `BUILD` file and the user +who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with these constraints: +Instead, define a `config_setting` that matches **any** platform with +these constraints: ```py config_setting( @@ -914,11 +1065,13 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what +platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you +can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -938,4 +1091,7 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. +The Bazel team doesn't endorse doing this; it overly constrains your build and +confuses users when the expected condition does not match. + +[BuildSettings]: /extending/config#user-defined-build-settings diff --git a/configure/best-practices.mdx b/configure/best-practices.mdx index 90eec9fc..deecf9dd 100644 --- a/configure/best-practices.mdx +++ b/configure/best-practices.mdx @@ -2,7 +2,10 @@ title: 'Best Practices' --- -This page assumes you are familiar with Bazel and provides guidelines and advice on structuring your projects to take full advantage of Bazel's features. + + +This page assumes you are familiar with Bazel and provides guidelines and +advice on structuring your projects to take full advantage of Bazel's features. The overall goals are: @@ -11,45 +14,81 @@ The overall goals are: - To make code well-structured and testable. - To create a build configuration that is easy to understand and maintain. -These guidelines are not requirements: few projects will be able to adhere to all of them. As the man page for lint says, "A special reward will be presented to the first person to produce a real program that produces no errors with strict checking." However, incorporating as many of these principles as possible should make a project more readable, less error-prone, and faster to build. +These guidelines are not requirements: few projects will be able to adhere to +all of them. As the man page for lint says, "A special reward will be presented +to the first person to produce a real program that produces no errors with +strict checking." However, incorporating as many of these principles as possible +should make a project more readable, less error-prone, and faster to build. -This page uses the requirement levels described in [this RFC](https://www.ietf.org/rfc/rfc2119.txt). +This page uses the requirement levels described in +[this RFC](https://www.ietf.org/rfc/rfc2119.txt). ## Running builds and tests -A project should always be able to run `bazel build //...` and `bazel test //...` successfully on its stable branch. Targets that are necessary but do not build under certain circumstances (such as,require specific build flags, don't build on a certain platform, require license agreements) should be tagged as specifically as possible (for example, "`requires-osx`"). This tagging allows targets to be filtered at a more fine-grained level than the "manual" tag and allows someone inspecting the `BUILD` file to understand what a target's restrictions are. +A project should always be able to run `bazel build //...` and +`bazel test //...` successfully on its stable branch. Targets that are necessary +but do not build under certain circumstances (such as,require specific build +flags, don't build on a certain platform, require license agreements) should be +tagged as specifically as possible (for example, "`requires-osx`"). This +tagging allows targets to be filtered at a more fine-grained level than the +"manual" tag and allows someone inspecting the `BUILD` file to understand what +a target's restrictions are. ## Third-party dependencies You may declare third-party dependencies: -- Either declare them as remote repositories in the `MODULE.bazel` file. -- Or put them in a directory called `third_party/` under your workspace directory. +* Either declare them as remote repositories in the `MODULE.bazel` file. +* Or put them in a directory called `third_party/` under your workspace directory. ## Depending on binaries -Everything should be built from source whenever possible. Generally this means that, instead of depending on a library `some-library.so`, you'd create a `BUILD` file and build `some-library.so` from its sources, then depend on that target. +Everything should be built from source whenever possible. Generally this means +that, instead of depending on a library `some-library.so`, you'd create a +`BUILD` file and build `some-library.so` from its sources, then depend on that +target. -Always building from source ensures that a build is not using a library that was built with incompatible flags or a different architecture. There are also some features like coverage, static analysis, or dynamic analysis that only work on the source. +Always building from source ensures that a build is not using a library that +was built with incompatible flags or a different architecture. There are also +some features like coverage, static analysis, or dynamic analysis that only +work on the source. ## Versioning -Prefer building all code from head whenever possible. When versions must be used, avoid including the version in the target name (for example, `//guava`, not `//guava-20.0`). This naming makes the library easier to update (only one target needs to be updated). It's also more resilient to diamond dependency issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, you could end up with a library that tries to depend on two different versions. If you created a misleading alias to point both targets to one `guava` library, then the `BUILD` files are misleading. +Prefer building all code from head whenever possible. When versions must be +used, avoid including the version in the target name (for example, `//guava`, +not `//guava-20.0`). This naming makes the library easier to update (only one +target needs to be updated). It's also more resilient to diamond dependency +issues: if one library depends on `guava-19.0` and one depends on `guava-20.0`, +you could end up with a library that tries to depend on two different versions. +If you created a misleading alias to point both targets to one `guava` library, +then the `BUILD` files are misleading. ## Using the `.bazelrc` file -For project-specific options, use the configuration file your `<var>workspace</var>/.bazelrc` (see [bazelrc format](/run/bazelrc)). +For project-specific options, use the configuration file your +`workspace/.bazelrc` (see [bazelrc format](/run/bazelrc)). -If you want to support per-user options for your project that you **do not** want to check into source control, include the line: +If you want to support per-user options for your project that you **do not** +want to check into source control, include the line: ``` try-import %workspace%/user.bazelrc ``` +(or any other file name) in your `workspace/.bazelrc` +and add `user.bazelrc` to your `.gitignore`. -(or any other file name) in your `<var>workspace</var>/.bazelrc` and add `user.bazelrc` to your `.gitignore`. +The open-source +[bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) -The open-source [bazelrc-preset.bzl](https://github.com/bazel-contrib/bazelrc-preset.bzl) generates a custom bazelrc file that matches your Bazel version and provides a preset of recommended flags. +generates a custom bazelrc file that matches your Bazel version and provides a +preset of recommended flags. ## Packages -Every directory that contains buildable files should be a package. If a `BUILD` file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's a sign that a `BUILD` file should be added to that subdirectory. The longer this structure exists, the more likely circular dependencies will be inadvertently created, a target's scope will creep, and an increasing number of reverse dependencies will have to be updated. +Every directory that contains buildable files should be a package. If a `BUILD` +file refers to files in subdirectories (such as, `srcs = ["a/b/C.java"]`) it's +a sign that a `BUILD` file should be added to that subdirectory. The longer +this structure exists, the more likely circular dependencies will be +inadvertently created, a target's scope will creep, and an increasing number +of reverse dependencies will have to be updated. diff --git a/configure/coverage.mdx b/configure/coverage.mdx index 72cde9c5..03a8ab9b 100644 --- a/configure/coverage.mdx +++ b/configure/coverage.mdx @@ -2,67 +2,129 @@ title: 'Code coverage with Bazel' --- -Bazel features a `coverage` sub-command to produce code coverage reports on repositories that can be tested with `bazel coverage`. Due to the idiosyncrasies of the various language ecosystems, it is not always trivial to make this work for a given project. -This page documents the general process for creating and viewing coverage reports, and also features some language-specific notes for languages whose configuration is well-known. It is best read by first reading [the general section](#creating-a-coverage-report), and then reading about the requirements for a specific language. Note also the [remote execution section](#remote-execution), which requires some additional considerations. -While a lot of customization is possible, this document focuses on producing and consuming [`lcov`](https://github.com/linux-test-project/lcov) reports, which is currently the most well-supported route. +Bazel features a `coverage` sub-command to produce code coverage +reports on repositories that can be tested with `bazel coverage`. Due +to the idiosyncrasies of the various language ecosystems, it is not +always trivial to make this work for a given project. + +This page documents the general process for creating and viewing +coverage reports, and also features some language-specific notes for +languages whose configuration is well-known. It is best read by first +reading [the general section](#creating-a-coverage-report), and then +reading about the requirements for a specific language. Note also the +[remote execution section](#remote-execution), which requires some +additional considerations. + +While a lot of customization is possible, this document focuses on +producing and consuming [`lcov`][lcov] reports, which is currently the +most well-supported route. ## Creating a coverage report ### Preparation -The basic workflow for creating coverage reports requires the following: +The basic workflow for creating coverage reports requires the +following: - A basic repository with test targets - A toolchain with the language-specific code coverage tools installed - A correct "instrumentation" configuration -The former two are language-specific and mostly straightforward, however the latter can be more difficult for complex projects. +The former two are language-specific and mostly straightforward, +however the latter can be more difficult for complex projects. -"Instrumentation" in this case refers to the coverage tools that are used for a specific target. Bazel allows turning this on for a specific subset of files using the [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) flag, which specifies a filter for targets that are tested with the instrumentation enabled. To enable instrumentation for tests, the [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) flag is required. +"Instrumentation" in this case refers to the coverage tools that are +used for a specific target. Bazel allows turning this on for a +specific subset of files using the +[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter) +flag, which specifies a filter for targets that are tested with the +instrumentation enabled. To enable instrumentation for tests, the +[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) +flag is required. -By default, bazel tries to match the target package(s), and prints the relevant filter as an `INFO` message. +By default, bazel tries to match the target package(s), and prints the +relevant filter as an `INFO` message. ### Running coverage -To produce a coverage report, use [`bazel coverage --combined_report=lcov [target]`](/reference/command-line-reference#coverage). This runs the tests for the target, generating coverage reports in the lcov format for each file. +To produce a coverage report, use [`bazel coverage +--combined_report=lcov +[target]`](/reference/command-line-reference#coverage). This runs the +tests for the target, generating coverage reports in the lcov format +for each file. -Once finished, bazel runs an action that collects all the produced coverage files, and merges them into one, which is then finally created under `$(bazel info output_path)/_coverage/_coverage_report.dat`. +Once finished, bazel runs an action that collects all the produced +coverage files, and merges them into one, which is then finally +created under `$(bazel info +output_path)/_coverage/_coverage_report.dat`. -Coverage reports are also produced if tests fail, though note that this does not extend to the failed tests - only passing tests are reported. +Coverage reports are also produced if tests fail, though note that +this does not extend to the failed tests - only passing tests are +reported. ### Viewing coverage -The coverage report is only output in the non-human-readable `lcov` format. From this, we can use the `genhtml` utility (part of [the lcov project](https://github.com/linux-test-project/lcov)) to produce a report that can be viewed in a web browser: +The coverage report is only output in the non-human-readable `lcov` +format. From this, we can use the `genhtml` utility (part of [the lcov +project][lcov]) to produce a report that can be viewed in a web +browser: ```console genhtml --branch-coverage --output genhtml "$(bazel info output_path)/_coverage/_coverage_report.dat" ``` -Note that `genhtml` reads the source code as well, to annotate missing coverage in these files. For this to work, it is expected that `genhtml` is executed in the root of the bazel project. +Note that `genhtml` reads the source code as well, to annotate missing +coverage in these files. For this to work, it is expected that +`genhtml` is executed in the root of the bazel project. -To view the result, simply open the `index.html` file produced in the `genhtml` directory in any web browser. +To view the result, simply open the `index.html` file produced in the +`genhtml` directory in any web browser. -For further help and information around the `genhtml` tool, or the `lcov` coverage format, see [the lcov project](https://github.com/linux-test-project/lcov). +For further help and information around the `genhtml` tool, or the +`lcov` coverage format, see [the lcov project][lcov]. ## Remote execution Running with remote test execution currently has a few caveats: -- The report combination action cannot yet run remotely. This is because Bazel does not consider the coverage output files as part of its graph (see [this issue](https://github.com/bazelbuild/bazel/issues/4685)), and can therefore not correctly treat them as inputs to the combination action. To work around this, use `--strategy=CoverageReport=local`. - - Note: It may be necessary to specify something like `--strategy=CoverageReport=local,remote` instead, if Bazel is set up to try `local,remote`, due to how Bazel resolves strategies. -- `--remote_download_minimal` and similar flags can also not be used as a consequence of the former. -- Bazel will currently fail to create coverage information if tests have been cached previously. To work around this, `--nocache_test_results` can be set specifically for coverage runs, although this of course incurs a heavy cost in terms of test times. -- `--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs` - - Usually coverage is run as part of the test action, and so by default, we don't get all coverage back as outputs of the remote execution by default. These flags override the default and obtain the coverage data. See [this issue](https://github.com/bazelbuild/bazel/issues/4685) for more details. +- The report combination action cannot yet run remotely. This is + because Bazel does not consider the coverage output files as part of + its graph (see [this issue][remote_report_issue]), and can therefore + not correctly treat them as inputs to the combination action. To + work around this, use `--strategy=CoverageReport=local`. + - Note: It may be necessary to specify something like + `--strategy=CoverageReport=local,remote` instead, if Bazel is set + up to try `local,remote`, due to how Bazel resolves strategies. +- `--remote_download_minimal` and similar flags can also not be used + as a consequence of the former. +- Bazel will currently fail to create coverage information if tests + have been cached previously. To work around this, + `--nocache_test_results` can be set specifically for coverage runs, + although this of course incurs a heavy cost in terms of test times. +- `--experimental_split_coverage_postprocessing` and + `--experimental_fetch_all_coverage_outputs` + - Usually coverage is run as part of the test action, and so by + default, we don't get all coverage back as outputs of the remote + execution by default. These flags override the default and obtain + the coverage data. See [this issue][split_coverage_issue] for more + details. ## Language-specific configuration ### Java -Java should work out-of-the-box with the default configuration. The [bazel toolchains](https://github.com/bazelbuild/bazel-toolchains) contain everything necessary for remote execution, as well, including JUnit. +Java should work out-of-the-box with the default configuration. The +[bazel toolchains][bazel_toolchains] contain everything necessary for +remote execution, as well, including JUnit. ### Python -See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) for additional steps needed to enable coverage support in Python. +See the [`rules_python` coverage docs](https://rules-python.readthedocs.io/en/latest/coverage.html) +for additional steps needed to enable coverage support in Python. + +[lcov]: https://github.com/linux-test-project/lcov +[bazel_toolchains]: https://github.com/bazelbuild/bazel-toolchains +[remote_report_issue]: https://github.com/bazelbuild/bazel/issues/4685 +[split_coverage_issue]: https://github.com/bazelbuild/bazel/issues/4685 diff --git a/configure/integrate-cpp.mdx b/configure/integrate-cpp.mdx deleted file mode 100644 index 1b9aac1f..00000000 --- a/configure/integrate-cpp.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: 'Integrating with C++ Rules' ---- - -This page describes how to integrate with C++ rules on various levels. - -## Accessing the C++ toolchain - -You should use the helper functions available at [`@rules_cc//cc:find_cc_toolchain.bzl`](https://github.com/bazelbuild/rules_cc/blob/main/cc/find_cc_toolchain.bzl) to depend on a CC toolchain from a Starlark rule. - -To depend on a C++ toolchain in your rule, set the `toolchains` parameter to `use_cc_toolchain()`. Then, in the rule implementation, use `find_cpp_toolchain(ctx)` to get the [`CcToolchainInfo`](/rules/lib/providers/CcToolchainInfo). A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/write_cc_toolchain_cpu/write_cc_toolchain_cpu.bzl). - -## Generating command lines and environment variables using the C++ toolchain - -Typically, you would integrate with the C++ toolchain to have the same command line flags as C++ rules do, but without using C++ actions directly. This is because when writing our own actions, they must behave consistently with the C++ toolchain - for example, passing C++ command line flags to a tool that invokes the C++ compiler behind the scenes. - -C++ rules use a special way of constructing command lines based on [feature configuration](/docs/cc-toolchain-config-reference). To construct a command line, you need the following: - -- `features` and `action_configs` - these come from the `CcToolchainConfigInfo` and encapsulated in `CcToolchainInfo` -- `FeatureConfiguration` - returned by [cc\_common.configure\_features](/rules/lib/toplevel/cc_common#configure_features) -- cc toolchain config variables - returned by [cc\_common.create\_compile\_variables](/rules/lib/toplevel/cc_common#create_compile_variables) or [cc\_common.create\_link\_variables](/rules/lib/toplevel/cc_common#create_link_variables). - -There still are tool-specific getters, such as [compiler\_executable](/rules/lib/providers/CcToolchainInfo#compiler_executable). Prefer `get_tool_for_action` over these, as tool-specific getters will eventually be removed. - -A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/my_c_compile/my_c_compile.bzl). - -## Implementing Starlark rules that depend on C++ rules and/or that C++ rules can depend on - -Most C++ rules provide [`CcInfo`](/rules/lib/providers/CcInfo), a provider containing [`CompilationContext`](/rules/lib/builtins/CompilationContext) and [`LinkingContext`](/rules/lib/builtins/LinkingContext). Through these it is possible to access information such as all transitive headers or libraries to link. From `CcInfo` and from the `CcToolchainInfo` custom Starlark rules should be able to get all the information they need. - -If a custom Starlark rule provides `CcInfo`, it's a signal to the C++ rules that they can also depend on it. Be careful, however - if you only need to propagate `CcInfo` through the graph to the binary rule that then makes use of it, wrap `CcInfo` in a different provider. For example, if `java_library` rule wanted to propagate native dependencies up to the `java_binary`, it shouldn't provide `CcInfo` directly (`cc_binary` depending on `java_library` doesn't make sense), it should wrap it in, for example, `JavaCcInfo`. - -A complete working example can be found [in the rules\_cc examples](https://github.com/bazelbuild/rules_cc/blob/main/examples/my_c_archive/my_c_archive.bzl). - -## Reusing logic and actions of C++ rules - -*Not stable yet; This section will be updated once the API stabilizes. Follow [#4570](https://github.com/bazelbuild/bazel/issues/4570) for up-to-date information.* diff --git a/configure/windows.mdx b/configure/windows.mdx index 566c425d..2dbb90ee 100644 --- a/configure/windows.mdx +++ b/configure/windows.mdx @@ -2,17 +2,24 @@ title: 'Using Bazel on Windows' --- -This page covers Best Practices for using Bazel on Windows. For installation instructions, see [Install Bazel on Windows](/install/windows). + + +This page covers Best Practices for using Bazel on Windows. For installation +instructions, see [Install Bazel on Windows](/install/windows). ## Known issues -Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. [GitHub-Windows](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows). +Windows-related Bazel issues are marked with the "area-Windows" label on GitHub. +[GitHub-Windows]. + +[GitHub-Windows]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Windows ## Best practices ### Avoid long path issues -Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. To avoid hitting this issue, you can specify a short output directory for Bazel by the [--output\_user\_root](/reference/command-line-reference#flag--output_user_root) flag. +Some tools have the [Maximum Path Length Limitation](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation) on Windows, including the MSVC compiler. +To avoid hitting this issue, you can specify a short output directory for Bazel by the [\-\-output_user_root](/reference/command-line-reference#flag--output_user_root) flag. For example, add the following line to your bazelrc file: @@ -22,10 +29,14 @@ startup --output_user_root=C:/tmp ### Enable symlink support -Some features require Bazel to be able to create file symlinks on Windows, either by enabling [Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) (on Windows 10 version 1703 or newer), or by running Bazel as an administrator. This enables the following features: +Some features require Bazel to be able to create file symlinks on Windows, +either by enabling +[Developer Mode](https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development) +(on Windows 10 version 1703 or newer), or by running Bazel as an administrator. +This enables the following features: -- [--windows\_enable\_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) -- [--enable\_runfiles](/reference/command-line-reference#flag--enable_runfiles) +* [\-\-windows_enable_symlinks](/reference/command-line-reference#flag--windows_enable_symlinks) +* [\-\-enable_runfiles](/reference/command-line-reference#flag--enable_runfiles) To make it easier, add the following lines to your bazelrc file: @@ -37,11 +48,23 @@ build --enable_runfiles **Note**: Creating symlinks on Windows is an expensive operation. The `--enable_runfiles` flag can potentially create a large amount of file symlinks. Only enable this feature when you need it. +{/* TODO(pcloudy): https://github.com/bazelbuild/bazel/issues/6402 + Write a doc about runfiles library and add a link to it here */} + ### Running Bazel: MSYS2 shell vs. command prompt vs. PowerShell -**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from PowerShell. +**Recommendation:** Run Bazel from the command prompt (`cmd.exe`) or from +PowerShell. -As of 2020-01-15, **do not** run Bazel from `bash` -- either from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel may work for most use cases, some things are broken, like [interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). Also, if you choose to run under MSYS2, you need to disable MSYS2's automatic path conversion, otherwise MSYS will convert command line arguments that *look like* Unix paths (such as `//foo:bar`) into Windows paths. See [this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) for details. +As of 2020-01-15, **do not** run Bazel from `bash` -- either +from MSYS2 shell, or Git Bash, or Cygwin, or any other Bash variant. While Bazel +may work for most use cases, some things are broken, like +[interrupting the build with Ctrl+C from MSYS2](https://github.com/bazelbuild/bazel/issues/10573)). +Also, if you choose to run under MSYS2, you need to disable MSYS2's +automatic path conversion, otherwise MSYS will convert command line arguments +that _look like_ Unix paths (such as `//foo:bar`) into Windows paths. See +[this StackOverflow answer](https://stackoverflow.com/a/49004265/7778502) +for details. ### Using Bazel without Bash (MSYS2) @@ -55,7 +78,13 @@ Starting with Bazel 1.0, you can build any rule without Bash unless it is a: - `sh_binary` or `sh_test` rule, because these inherently need Bash - Starlark rule that uses `ctx.actions.run_shell()` or `ctx.resolve_command()` -However, `genrule` is often used for simple tasks like [copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). Instead of using `genrule` (and depending on Bash) you may find a suitable rule in the [bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). When built on Windows, **these rules do not require Bash**. +However, `genrule` is often used for simple tasks like +[copying a file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl) +or [writing a text file](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl). +Instead of using `genrule` (and depending on Bash) you may find a suitable rule +in the +[bazel-skylib repository](https://github.com/bazelbuild/bazel-skylib/tree/main/rules). +When built on Windows, **these rules do not require Bash**. #### Using bazel test without Bash @@ -75,15 +104,24 @@ Starting with Bazel 1.0, you can run any rule without Bash, except when: - you use `--run_under` or `--script_path` - the test rule itself requires Bash (because its executable is a shell script) -#### Using sh\_binary and sh\_\* rules, and ctx.actions.run\_shell() without Bash +#### Using sh\_binary and sh\_* rules, and ctx.actions.run_shell() without Bash -You need Bash to build and test `sh_*` rules, and to build and test Starlark rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This applies not only to rules in your project, but to rules in any of the external repositories your project depends on (even transitively). +You need Bash to build and test `sh_*` rules, and to build and test Starlark +rules that use `ctx.actions.run_shell()` and `ctx.resolve_command()`. This +applies not only to rules in your project, but to rules in any of the external +repositories your project depends on (even transitively). -In the future, there may be an option to use Windows Subsystem for Linux (WSL) to build these rules, but currently it is not a priority for the Bazel-on-Windows subteam. +In the future, there may be an option to use Windows Subsystem for +Linux (WSL) to build these rules, but currently it is not a priority for +the Bazel-on-Windows subteam. ### Setting environment variables -Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only set in that command prompt session. If you start a new `cmd.exe`, you need to set the variables again. To always set the variables when `cmd.exe` starts, you can add them to the User variables or System variables in the `Control Panel > System Properties > Advanced > Environment Variables...` dialog box. +Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only +set in that command prompt session. If you start a new `cmd.exe`, you need to +set the variables again. To always set the variables when `cmd.exe` starts, you +can add them to the User variables or System variables in the `Control Panel > +System Properties > Advanced > Environment Variables...` dialog box. ## Build on Windows @@ -91,54 +129,71 @@ Environment variables you set in the Windows Command Prompt (`cmd.exe`) are only To build C++ targets with MSVC, you need: -- [The Visual C++ compiler](/install/windows#install-vc). +* [The Visual C++ compiler](/install/windows#install-vc). -- (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. +* (Optional) The `BAZEL_VC` and `BAZEL_VC_FULL_VERSION` environment variable. - Bazel automatically detects the Visual C++ compiler on your system. To tell Bazel to use a specific VC installation, you can set the following environment variables: + Bazel automatically detects the Visual C++ compiler on your system. + To tell Bazel to use a specific VC installation, you can set the + following environment variables: - For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. + For Visual Studio 2017 and 2019, set one of `BAZEL_VC`. Additionally you may also set `BAZEL_VC_FULL_VERSION`. - - `BAZEL_VC` the Visual C++ Build Tools installation directory + * `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC + ``` - - `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel will choose the latest version. + * `BAZEL_VC_FULL_VERSION` (Optional) Only for Visual Studio 2017 and 2019, the full version + number of your Visual C++ Build Tools. You can choose the exact Visual C++ Build Tools + version via `BAZEL_VC_FULL_VERSION` if more than one version are installed, otherwise Bazel + will choose the latest version. - ``` - set BAZEL_VC_FULL_VERSION=14.16.27023 - ``` + ``` + set BAZEL_VC_FULL_VERSION=14.16.27023 + ``` - For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) + For Visual Studio 2015 or older, set `BAZEL_VC`. (`BAZEL_VC_FULL_VERSION` is not supported.) - - `BAZEL_VC` the Visual C++ Build Tools installation directory + * `BAZEL_VC` the Visual C++ Build Tools installation directory - ``` - set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC - ``` + ``` + set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC + ``` -- The [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). +* The [Windows + SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk). - The Windows SDK contains header files and libraries you need when building Windows applications, including Bazel itself. By default, the latest Windows SDK installed will be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified Windows SDK installed. + The Windows SDK contains header files and libraries you need when building + Windows applications, including Bazel itself. By default, the latest Windows SDK installed will + be used. You also can specify Windows SDK version by setting `BAZEL_WINSDK_FULL_VERSION`. You + can use a full Windows 10 SDK number such as 10.0.10240.0, or specify 8.1 to use the Windows 8.1 + SDK (only one version of Windows 8.1 SDK is available). Please make sure you have the specified + Windows SDK installed. - **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise `BAZEL_WINSDK_FULL_VERSION` will be ignored. + **Requirement**: This is supported with VC 2017 and 2019. The standalone VC 2015 Build Tools doesn't + support selecting Windows SDK, you'll need the full Visual Studio 2015 installation, otherwise + `BAZEL_WINSDK_FULL_VERSION` will be ignored. - ``` - set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 - ``` + ``` + set BAZEL_WINSDK_FULL_VERSION=10.0.10240.0 + ``` If everything is set up, you can build a C++ target now! -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` -bazel build //examples/cpp:hello-world -bazel-bin\examples\cpp\hello-world.exe +bazel build //examples/cpp:hello-world +bazel-bin\examples\cpp\hello-world.exe ``` -By default, the built binaries target x64 architecture. To build for ARM64 architecture, use +By default, the built binaries target x64 architecture. To build for ARM64 +architecture, use ```none --platforms=//:windows_arm64 --extra_toolchains=@local_config_cc//:cc-toolchain-arm64_windows @@ -152,88 +207,111 @@ cc_configure = use_extension("@rules_cc//cc:extensions.bzl", "cc_configure_exten use_repo(cc_configure, "local_config_cc") ``` -To build and use Dynamically Linked Libraries (DLL files), see [this example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). +To build and use Dynamically Linked Libraries (DLL files), see [this +example](https://github.com/bazelbuild/bazel/tree/master/examples/windows/dll). -**Command Line Length Limit**: To prevent the [Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), enable the compiler parameter file feature via `--features=compiler_param_file`. +**Command Line Length Limit**: To prevent the +[Windows command line length limit issue](https://github.com/bazelbuild/bazel/issues/5163), +enable the compiler parameter file feature via `--features=compiler_param_file`. ### Build C++ with Clang From 0.29.0, Bazel supports building with LLVM's MSVC-compatible compiler driver (`clang-cl.exe`). -**Requirement**: To build with Clang, you have to install **both** [LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, because although you use `clang-cl.exe` as compiler, you still need to link to Visual C++ libraries. +**Requirement**: To build with Clang, you have to install **both** +[LLVM](http://releases.llvm.org/download.html) and Visual C++ Build tools, +because although you use `clang-cl.exe` as compiler, you still need to link to +Visual C++ libraries. -Bazel can automatically detect LLVM installation on your system, or you can explicitly tell Bazel where LLVM is installed by `BAZEL_LLVM`. +Bazel can automatically detect LLVM installation on your system, or you can explicitly tell +Bazel where LLVM is installed by `BAZEL_LLVM`. -- `BAZEL_LLVM` the LLVM installation directory +* `BAZEL_LLVM` the LLVM installation directory - ```posix-terminal - set BAZEL_LLVM=C:\Program Files\LLVM - ``` + ```posix-terminal + set BAZEL_LLVM=C:\Program Files\LLVM + ``` To enable the Clang toolchain for building C++, there are several situations. -- In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the top level `BUILD` file): +* In Bazel 7.0.0 and newer: Add a platform target to your `BUILD file` (eg. the + top level `BUILD` file): - ``` - platform( - name = "x64_windows-clang-cl", - constraint_values = [ - "@platforms//cpu:x86_64", - "@platforms//os:windows", - "@bazel_tools//tools/cpp:clang-cl", - ], - ) - ``` + ``` + platform( + name = "x64_windows-clang-cl", + constraint_values = [ + "@platforms//cpu:x86_64", + "@platforms//os:windows", + "@bazel_tools//tools/cpp:clang-cl", + ], + ) + ``` - Then enable the Clang toolchain by specifying the following build flags: + Then enable the Clang toolchain by specifying the following build flags: - ``` - --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl - ``` + ``` + --extra_toolchains=@local_config_cc//:cc-toolchain-x64_windows-clang-cl --extra_execution_platforms=//:x64_windows-clang-cl + ``` -- In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by a build flag `--compiler=clang-cl`. +* In Bazel older than 7.0.0 but newer than 0.28: Enable the Clang toolchain by + a build flag `--compiler=clang-cl`. - If your build sets the flag [--incompatible\_enable\_cc\_toolchain\_resolution](https://github.com/bazelbuild/bazel/issues/7260) to `true`, then use the approach for Bazel 7.0.0. + If your build sets the flag + [\-\-incompatible_enable_cc_toolchain_resolution] + (https://github.com/bazelbuild/bazel/issues/7260) + to `true`, then use the approach for Bazel 7.0.0. -- In Bazel 0.28 and older: Clang is not supported. +* In Bazel 0.28 and older: Clang is not supported. ### Build Java To build Java targets, you need: -- [The Java SE Development Kit](/install/windows#install-jdk) +* [The Java SE Development Kit](/install/windows#install-jdk) On Windows, Bazel builds two output files for `java_binary` rules: -- a `.jar` file -- a `.exe` file that can set up the environment for the JVM and run the binary +* a `.jar` file +* a `.exe` file that can set up the environment for the JVM and run the binary -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world - bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe + bazel build //examples/java-native/src/main/java/com/example/myproject:hello-world + bazel-bin\examples\java-native\src\main\java\com\example\myproject\hello-world.exe ``` ### Build Python To build Python targets, you need: -- The [Python interpreter](/install/windows#install-python) +* The [Python interpreter](/install/windows#install-python) On Windows, Bazel builds two output files for `py_binary` rules: -- a self-extracting zip file -- an executable file that can launch the Python interpreter with the self-extracting zip file as the argument +* a self-extracting zip file +* an executable file that can launch the Python interpreter with the + self-extracting zip file as the argument -You can either run the executable file (it has a `.exe` extension) or you can run Python with the self-extracting zip file as the argument. +You can either run the executable file (it has a `.exe` extension) or you can run +Python with the self-extracting zip file as the argument. -Try building a target from one of our [sample projects](https://github.com/bazelbuild/bazel/tree/master/examples): +Try building a target from one of our [sample +projects](https://github.com/bazelbuild/bazel/tree/master/examples): ``` - bazel build //examples/py_native:bin - bazel-bin\examples\py_native\bin.exe - python bazel-bin\examples\py_native\bin.zip + bazel build //examples/py_native:bin + bazel-bin\examples\py_native\bin.exe + python bazel-bin\examples\py_native\bin.zip ``` -If you are interested in details about how Bazel builds Python targets on Windows, check out this [design doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). +If you are interested in details about how Bazel builds Python targets on +Windows, check out this [design +doc](https://github.com/bazelbuild/bazel-website/blob/master/designs/_posts/2016-09-05-build-python-on-windows.md). diff --git a/contribute/breaking-changes.mdx b/contribute/breaking-changes.mdx index c30b163a..5dda1b9d 100644 --- a/contribute/breaking-changes.mdx +++ b/contribute/breaking-changes.mdx @@ -2,41 +2,65 @@ title: 'Guide for rolling out breaking changes' --- -It is inevitable that we will make breaking changes to Bazel. We will have to change our designs and fix the things that do not quite work. However, we need to make sure that community and Bazel ecosystem can follow along. To that end, Bazel project has adopted a [backward compatibility policy](/release/backward-compatibility). This document describes the process for Bazel contributors to make a breaking change in Bazel to adhere to this policy. + + +It is inevitable that we will make breaking changes to Bazel. We will have to +change our designs and fix the things that do not quite work. However, we need +to make sure that community and Bazel ecosystem can follow along. To that end, +Bazel project has adopted a +[backward compatibility policy](/release/backward-compatibility). +This document describes the process for Bazel contributors to make a breaking +change in Bazel to adhere to this policy. 1. Follow the [design document policy](/contribute/design-documents). -2. [File a GitHub issue.](#github-issue) +1. [File a GitHub issue.](#github-issue) -3. [Implement the change.](#implementation) +1. [Implement the change.](#implementation) -4. [Update labels.](#labels) +1. [Update labels.](#labels) -5. [Update repositories.](#update-repos) +1. [Update repositories.](#update-repos) -6. [Flip the incompatible flag.](#flip-flag) +1. [Flip the incompatible flag.](#flip-flag) ## GitHub issue -[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) in the Bazel repository. [See example.](https://github.com/bazelbuild/bazel/issues/6611) +[File a GitHub issue](https://github.com/bazelbuild/bazel/issues) +in the Bazel repository. +[See example.](https://github.com/bazelbuild/bazel/issues/6611) We recommend that: -- The title starts with the name of the flag (the flag name will start with `incompatible_`). +* The title starts with the name of the flag (the flag name will start with + `incompatible_`). -- You add the label [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). +* You add the label + [`incompatible-change`](https://github.com/bazelbuild/bazel/labels/incompatible-change). -- The description contains a description of the change and a link to relevant design documents. +* The description contains a description of the change and a link to relevant + design documents. -- The description contains a migration recipe, to explain users how they should update their code. Ideally, when the change is mechanical, include a link to a migration tool. +* The description contains a migration recipe, to explain users how they should + update their code. Ideally, when the change is mechanical, include a link to a + migration tool. -- The description includes an example of the error message users will get if they don't migrate. This will make the GitHub issue more discoverable from search engines. Make sure that the error message is helpful and actionable. When possible, the error message should include the name of the incompatible flag. +* The description includes an example of the error message users will get if + they don't migrate. This will make the GitHub issue more discoverable from + search engines. Make sure that the error message is helpful and actionable. + When possible, the error message should include the name of the incompatible + flag. -For the migration tool, consider contributing to [Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. It may also report warnings. +For the migration tool, consider contributing to +[Buildifier](https://github.com/bazelbuild/buildtools/blob/master/buildifier/README.md). +It is able to apply automated fixes to `BUILD`, `WORKSPACE`, and `.bzl` files. +It may also report warnings. ## Implementation -Create a new flag in Bazel. The default value must be false. The help text should contain the URL of the GitHub issue. As the flag name starts with `incompatible_`, it needs metadata tags: +Create a new flag in Bazel. The default value must be false. The help text +should contain the URL of the GitHub issue. As the flag name starts with +`incompatible_`, it needs metadata tags: ```java metadataTags = { @@ -44,60 +68,80 @@ Create a new flag in Bazel. The default value must be false. The help text shoul }, ``` -In the commit description, add a brief summary of the flag. Also add [`RELNOTES:`](release-notes.md) in the following form: `RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` +In the commit description, add a brief summary of the flag. +Also add [`RELNOTES:`](release-notes.md) in the following form: +`RELNOTES: --incompatible_name_of_flag has been added. See #xyz for details` -The commit should also update the relevant documentation, so that there is no window of commits in which the code is inconsistent with the docs. Since our documentation is versioned, changes to the docs will not be inadvertently released prematurely. +The commit should also update the relevant documentation, so that there is no +window of commits in which the code is inconsistent with the docs. Since our +documentation is versioned, changes to the docs will not be inadvertently +released prematurely. ## Labels -Once the commit is merged and the incompatible change is ready to be adopted, add the label [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) to the GitHub issue. +Once the commit is merged and the incompatible change is ready to be adopted, add the label +[`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) +to the GitHub issue. -If a problem is found with the flag and users are not expected to migrate yet: remove the flags `migration-ready`. +If a problem is found with the flag and users are not expected to migrate yet: +remove the flags `migration-ready`. -If you plan to flip the flag in the next major release, add label \`breaking-change-X.0" to the issue. +If you plan to flip the flag in the next major release, add label `breaking-change-X.0" to the issue. ## Updating repositories -Bazel CI tests a list of important projects at [Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). +Bazel CI tests a list of important projects at +[Bazel@HEAD + Downstream](https://buildkite.com/bazel/bazel-at-head-plus-downstream). Most of them are often +dependencies of other Bazel projects, therefore it's important to migrate them to unblock the migration for the broader community. To monitor the migration status of those projects, you can use the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags). +Check how this pipeline works [here](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#checking-incompatible-changes-status-for-downstream-projects). Our dev support team monitors the [`migration-ready`](https://github.com/bazelbuild/bazel/labels/migration-ready) label. Once you add this label to the GitHub issue, they will handle the following: 1. Create a comment in the GitHub issue to track the list of failures and downstream projects that need to be migrated ([see example](https://github.com/bazelbuild/bazel/issues/17032#issuecomment-1353077469)) -2. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) +1. File Github issues to notify the owners of every downstream project broken by your incompatible change ([see example](https://github.com/bazelbuild/intellij/issues/4208)) -3. Follow up to make sure all issues are addressed before the target release date +1. Follow up to make sure all issues are addressed before the target release date Migrating projects in the downstream pipeline is NOT entirely the responsibility of the incompatible change author, but you can do the following to accelerate the migration and make life easier for both Bazel users and the Bazel Green Team. 1. Send PRs to fix downstream projects. -2. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). +1. Reach out to the Bazel community for help on migration (e.g. [Bazel Rules Authors SIG](https://bazel-contrib.github.io/SIG-rules-authors/)). ## Flipping the flag Before flipping the default value of the flag to true, please make sure that: -- Core repositories in the ecosystem are migrated. +* Core repositories in the ecosystem are migrated. - On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. + On the [`bazelisk-plus-incompatible-flags` pipeline](https://buildkite.com/bazel/bazelisk-plus-incompatible-flags), + the flag should appear under `The following flags didn't break any passing Bazel team owned/co-owned projects`. -- All issues in the checklist are marked as fixed/closed. +* All issues in the checklist are marked as fixed/closed. -- User concerns and questions have been resolved. +* User concerns and questions have been resolved. When the flag is ready to flip in Bazel, but blocked on internal migration at Google, please consider setting the flag value to false in the internal `blazerc` file to unblock the flag flip. By doing this, we can ensure Bazel users depend on the new behaviour by default as early as possible. When changing the flag default to true, please: -- Use `RELNOTES[INC]` in the commit description, with the following format: `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for details` You can include additional information in the rest of the commit description. -- Use `Fixes #xyz` in the description, so that the GitHub issue gets closed when the commit is merged. -- Review and update documentation if needed. -- File a new issue `#abc` to track the removal of the flag. +* Use `RELNOTES[INC]` in the commit description, with the + following format: + `RELNOTES[INC]: --incompatible_name_of_flag is flipped to true. See #xyz for + details` + You can include additional information in the rest of the commit description. +* Use `Fixes #xyz` in the description, so that the GitHub issue gets closed + when the commit is merged. +* Review and update documentation if needed. +* File a new issue `#abc` to track the removal of the flag. ## Removing the flag -After the flag is flipped at HEAD, it should be removed from Bazel eventually. When you plan to remove the incompatible flag: +After the flag is flipped at HEAD, it should be removed from Bazel eventually. +When you plan to remove the incompatible flag: -- Consider leaving more time for users to migrate if it's a major incompatible change. Ideally, the flag should be available in at least one major release. -- For the commit that removes the flag, use `Fixes #abc` in the description so that the GitHub issue gets closed when the commit is merged. +* Consider leaving more time for users to migrate if it's a major incompatible change. + Ideally, the flag should be available in at least one major release. +* For the commit that removes the flag, use `Fixes #abc` in the description + so that the GitHub issue gets closed when the commit is merged. diff --git a/contribute/codebase.mdx b/contribute/codebase.mdx index 99109430..44e0150d 100644 --- a/contribute/codebase.mdx +++ b/contribute/codebase.mdx @@ -2,773 +2,1652 @@ title: 'The Bazel codebase' --- -This document is a description of the codebase and how Bazel is structured. It is intended for people willing to contribute to Bazel, not for end-users. + + +This document is a description of the codebase and how Bazel is structured. It +is intended for people willing to contribute to Bazel, not for end-users. ## Introduction -The codebase of Bazel is large (\~350KLOC production code and \~260 KLOC test code) and no one is familiar with the whole landscape: everyone knows their particular valley very well, but few know what lies over the hills in every direction. +The codebase of Bazel is large (~350KLOC production code and ~260 KLOC test +code) and no one is familiar with the whole landscape: everyone knows their +particular valley very well, but few know what lies over the hills in every +direction. -In order for people midway upon the journey not to find themselves within a forest dark with the straightforward pathway being lost, this document tries to give an overview of the codebase so that it's easier to get started with working on it. +In order for people midway upon the journey not to find themselves within a +forest dark with the straightforward pathway being lost, this document tries to +give an overview of the codebase so that it's easier to get started with +working on it. -The public version of the source code of Bazel lives on GitHub at [github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not the "source of truth"; it's derived from a Google-internal source tree that contains additional functionality that is not useful outside Google. The long-term goal is to make GitHub the source of truth. +The public version of the source code of Bazel lives on GitHub at +[github.com/bazelbuild/bazel](http://github.com/bazelbuild/bazel). This is not +the "source of truth"; it's derived from a Google-internal source tree that +contains additional functionality that is not useful outside Google. The +long-term goal is to make GitHub the source of truth. -Contributions are accepted through the regular GitHub pull request mechanism, and manually imported by a Googler into the internal source tree, then re-exported back out to GitHub. +Contributions are accepted through the regular GitHub pull request mechanism, +and manually imported by a Googler into the internal source tree, then +re-exported back out to GitHub. ## Client/server architecture -The bulk of Bazel resides in a server process that stays in RAM between builds. This allows Bazel to maintain state between builds. +The bulk of Bazel resides in a server process that stays in RAM between builds. +This allows Bazel to maintain state between builds. -This is why the Bazel command line has two kinds of options: startup and command. In a command line like this: +This is why the Bazel command line has two kinds of options: startup and +command. In a command line like this: ``` bazel --host_jvm_args=-Xmx8G build -c opt //foo:bar ``` -Some options (`--host_jvm_args=`) are before the name of the command to be run and some are after (`-c opt`); the former kind is called a "startup option" and affects the server process as a whole, whereas the latter kind, the "command option", only affects a single command. - -Each server instance has a single associated workspace (collection of source trees known as "repositories") and each workspace usually has a single active server instance. This can be circumvented by specifying a custom output base (see the "Directory layout" section for more information). - -Bazel is distributed as a single ELF executable that is also a valid .zip file. When you type `bazel`, the above ELF executable implemented in C++ (the "client") gets control. It sets up an appropriate server process using the following steps: - -1. Checks whether it has already extracted itself. If not, it does that. This is where the implementation of the server comes from. -2. Checks whether there is an active server instance that works: it is running, it has the right startup options and uses the right workspace directory. It finds the running server by looking at the directory `$OUTPUT_BASE/server` where there is a lock file with the port the server is listening on. -3. If needed, kills the old server process -4. If needed, starts up a new server process - -After a suitable server process is ready, the command that needs to be run is communicated to it over a gRPC interface, then the output of Bazel is piped back to the terminal. Only one command can be running at the same time. This is implemented using an elaborate locking mechanism with parts in C++ and parts in Java. There is some infrastructure for running multiple commands in parallel, since the inability to run `bazel version` in parallel with another command is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s and some state in `BlazeRuntime`. - -At the end of a command, the Bazel server transmits the exit code the client should return. An interesting wrinkle is the implementation of `bazel run`: the job of this command is to run something Bazel just built, but it can't do that from the server process because it doesn't have a terminal. So instead it tells the client what binary it should `exec()` and with what arguments. - -When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC connection, which tries to terminate the command as soon as possible. After the third Ctrl-C, the client sends a SIGKILL to the server instead. - -The source code of the client is under `src/main/cpp` and the protocol used to communicate with the server is in `src/main/protobuf/command_server.proto` . - -The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls from the client are handled by `GrpcServerImpl.run()`. +Some options (`--host_jvm_args=`) are before the name of the command to be run +and some are after (`-c opt`); the former kind is called a "startup option" and +affects the server process as a whole, whereas the latter kind, the "command +option", only affects a single command. + +Each server instance has a single associated workspace (collection of source +trees known as "repositories") and each workspace usually has a single active +server instance. This can be circumvented by specifying a custom output base +(see the "Directory layout" section for more information). + +Bazel is distributed as a single ELF executable that is also a valid .zip file. +When you type `bazel`, the above ELF executable implemented in C++ (the +"client") gets control. It sets up an appropriate server process using the +following steps: + +1. Checks whether it has already extracted itself. If not, it does that. This + is where the implementation of the server comes from. +2. Checks whether there is an active server instance that works: it is running, + it has the right startup options and uses the right workspace directory. It + finds the running server by looking at the directory `$OUTPUT_BASE/server` + where there is a lock file with the port the server is listening on. +3. If needed, kills the old server process +4. If needed, starts up a new server process + +After a suitable server process is ready, the command that needs to be run is +communicated to it over a gRPC interface, then the output of Bazel is piped back +to the terminal. Only one command can be running at the same time. This is +implemented using an elaborate locking mechanism with parts in C++ and parts in +Java. There is some infrastructure for running multiple commands in parallel, +since the inability to run `bazel version` in parallel with another command +is somewhat embarrassing. The main blocker is the life cycle of `BlazeModule`s +and some state in `BlazeRuntime`. + +At the end of a command, the Bazel server transmits the exit code the client +should return. An interesting wrinkle is the implementation of `bazel run`: the +job of this command is to run something Bazel just built, but it can't do that +from the server process because it doesn't have a terminal. So instead it tells +the client what binary it should `exec()` and with what arguments. + +When one presses Ctrl-C, the client translates it to a Cancel call on the gRPC +connection, which tries to terminate the command as soon as possible. After the +third Ctrl-C, the client sends a SIGKILL to the server instead. + +The source code of the client is under `src/main/cpp` and the protocol used to +communicate with the server is in `src/main/protobuf/command_server.proto` . + +The main entry point of the server is `BlazeRuntime.main()` and the gRPC calls +from the client are handled by `GrpcServerImpl.run()`. ## Directory layout -Bazel creates a somewhat complicated set of directories during a build. A full description is available in [Output directory layout](/remote/output-directories). +Bazel creates a somewhat complicated set of directories during a build. A full +description is available in [Output directory layout](/remote/output-directories). -The "main repo" is the source tree Bazel is run in. It usually corresponds to something you checked out from source control. The root of this directory is known as the "workspace root". +The "main repo" is the source tree Bazel is run in. It usually corresponds to +something you checked out from source control. The root of this directory is +known as the "workspace root". -Bazel puts all of its data under the "output user root". This is usually `$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the `--output_user_root` startup option. +Bazel puts all of its data under the "output user root". This is usually +`$HOME/.cache/bazel/_bazel_${USER}`, but can be overridden using the +`--output_user_root` startup option. -The "install base" is where Bazel is extracted to. This is done automatically and each Bazel version gets a subdirectory based on its checksum under the install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed using the `--install_base` command line option. +The "install base" is where Bazel is extracted to. This is done automatically +and each Bazel version gets a subdirectory based on its checksum under the +install base. It's at `$OUTPUT_USER_ROOT/install` by default and can be changed +using the `--install_base` command line option. -The "output base" is the place where the Bazel instance attached to a specific workspace writes to. Each output base has at most one Bazel server instance running at any time. It's usually at `$OUTPUT_USER_ROOT/<checksum of the path to the workspace>`. It can be changed using the `--output_base` startup option, which is, among other things, useful for getting around the limitation that only one Bazel instance can be running in any workspace at any given time. +The "output base" is the place where the Bazel instance attached to a specific +workspace writes to. Each output base has at most one Bazel server instance +running at any time. It's usually at `$OUTPUT_USER_ROOT/`. It can be changed using the `--output_base` startup option, +which is, among other things, useful for getting around the limitation that only +one Bazel instance can be running in any workspace at any given time. The output directory contains, among other things: -- The fetched external repositories at `$OUTPUT_BASE/external`. -- The exec root, a directory that contains symlinks to all the source code for the current build. It's located at `$OUTPUT_BASE/execroot`. During the build, the working directory is `$EXECROOT/<name of main repository>`. We are planning to change this to `$EXECROOT`, although it's a long term plan because it's a very incompatible change. -- Files built during the build. +* The fetched external repositories at `$OUTPUT_BASE/external`. +* The exec root, a directory that contains symlinks to all the source + code for the current build. It's located at `$OUTPUT_BASE/execroot`. During + the build, the working directory is `$EXECROOT/`. We are planning to change this to `$EXECROOT`, although it's a + long term plan because it's a very incompatible change. +* Files built during the build. ## The process of executing a command -Once the Bazel server gets control and is informed about a command it needs to execute, the following sequence of events happens: +Once the Bazel server gets control and is informed about a command it needs to +execute, the following sequence of events happens: -1. `BlazeCommandDispatcher` is informed about the new request. It decides whether the command needs a workspace to run in (almost every command except for ones that don't have anything to do with source code, such as version or help) and whether another command is running. +1. `BlazeCommandDispatcher` is informed about the new request. It decides + whether the command needs a workspace to run in (almost every command except + for ones that don't have anything to do with source code, such as version or + help) and whether another command is running. -2. The right command is found. Each command must implement the interface `BlazeCommand` and must have the `@Command` annotation (this is a bit of an antipattern, it would be nice if all the metadata a command needs was described by methods on `BlazeCommand`) +2. The right command is found. Each command must implement the interface + `BlazeCommand` and must have the `@Command` annotation (this is a bit of an + antipattern, it would be nice if all the metadata a command needs was + described by methods on `BlazeCommand`) -3. The command line options are parsed. Each command has different command line options, which are described in the `@Command` annotation. +3. The command line options are parsed. Each command has different command line + options, which are described in the `@Command` annotation. -4. An event bus is created. The event bus is a stream for events that happen during the build. Some of these are exported to outside of Bazel under the aegis of the Build Event Protocol in order to tell the world how the build goes. +4. An event bus is created. The event bus is a stream for events that happen + during the build. Some of these are exported to outside of Bazel under the + aegis of the Build Event Protocol in order to tell the world how the build + goes. -5. The command gets control. The most interesting commands are those that run a build: build, test, run, coverage and so on: this functionality is implemented by `BuildTool`. +5. The command gets control. The most interesting commands are those that run a + build: build, test, run, coverage and so on: this functionality is + implemented by `BuildTool`. -6. The set of target patterns on the command line is parsed and wildcards like `//pkg:all` and `//pkg/...` are resolved. This is implemented in `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as `TargetPatternPhaseValue`. +6. The set of target patterns on the command line is parsed and wildcards like + `//pkg:all` and `//pkg/...` are resolved. This is implemented in + `AnalysisPhaseRunner.evaluateTargetPatterns()` and reified in Skyframe as + `TargetPatternPhaseValue`. -7. The loading/analysis phase is run to produce the action graph (a directed acyclic graph of commands that need to be executed for the build). +7. The loading/analysis phase is run to produce the action graph (a directed + acyclic graph of commands that need to be executed for the build). -8. The execution phase is run. This means running every action required to build the top-level targets that are requested are run. +8. The execution phase is run. This means running every action required to + build the top-level targets that are requested are run. ## Command line options -The command line options for a Bazel invocation are described in an `OptionsParsingResult` object, which in turn contains a map from "option classes" to the values of the options. An "option class" is a subclass of `OptionsBase` and groups command line options together that are related to each other. For example: - -1. Options related to a programming language (`CppOptions` or `JavaOptions`). These should be a subclass of `FragmentOptions` and are eventually wrapped into a `BuildOptions` object. -2. Options related to the way Bazel executes actions (`ExecutionOptions`) - -These options are designed to be consumed in the analysis phase and (either through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). Some of them (for example, whether to do C++ include scanning or not) are read in the execution phase, but that always requires explicit plumbing since `BuildConfiguration` is not available then. For more information, see the section "Configurations". - -**WARNING:** We like to pretend that `OptionsBase` instances are immutable and use them that way (such as a part of `SkyKeys`). This is not the case and modifying them is a really good way to break Bazel in subtle ways that are hard to debug. Unfortunately, making them actually immutable is a large endeavor. (Modifying a `FragmentOptions` immediately after construction before anyone else gets a chance to keep a reference to it and before `equals()` or `hashCode()` is called on it is okay.) +The command line options for a Bazel invocation are described in an +`OptionsParsingResult` object, which in turn contains a map from "option +classes" to the values of the options. An "option class" is a subclass of +`OptionsBase` and groups command line options together that are related to each +other. For example: + +1. Options related to a programming language (`CppOptions` or `JavaOptions`). + These should be a subclass of `FragmentOptions` and are eventually wrapped + into a `BuildOptions` object. +2. Options related to the way Bazel executes actions (`ExecutionOptions`) + +These options are designed to be consumed in the analysis phase and (either +through `RuleContext.getFragment()` in Java or `ctx.fragments` in Starlark). +Some of them (for example, whether to do C++ include scanning or not) are read +in the execution phase, but that always requires explicit plumbing since +`BuildConfiguration` is not available then. For more information, see the +section "Configurations". + +**WARNING:** We like to pretend that `OptionsBase` instances are immutable and +use them that way (such as a part of `SkyKeys`). This is not the case and +modifying them is a really good way to break Bazel in subtle ways that are hard +to debug. Unfortunately, making them actually immutable is a large endeavor. +(Modifying a `FragmentOptions` immediately after construction before anyone else +gets a chance to keep a reference to it and before `equals()` or `hashCode()` is +called on it is okay.) Bazel learns about option classes in the following ways: -1. Some are hard-wired into Bazel (`CommonCommandOptions`) -2. From the `@Command` annotation on each Bazel command -3. From `ConfiguredRuleClassProvider` (these are command line options related to individual programming languages) -4. Starlark rules can also define their own options (see [here](/extending/config)) +1. Some are hard-wired into Bazel (`CommonCommandOptions`) +2. From the `@Command` annotation on each Bazel command +3. From `ConfiguredRuleClassProvider` (these are command line options related + to individual programming languages) +4. Starlark rules can also define their own options (see + [here](/extending/config)) -Each option (excluding Starlark-defined options) is a member variable of a `FragmentOptions` subclass that has the `@Option` annotation, which specifies the name and the type of the command line option along with some help text. +Each option (excluding Starlark-defined options) is a member variable of a +`FragmentOptions` subclass that has the `@Option` annotation, which specifies +the name and the type of the command line option along with some help text. -The Java type of the value of a command line option is usually something simple (a string, an integer, a Boolean, a label, etc.). However, we also support options of more complicated types; in this case, the job of converting from the command line string to the data type falls to an implementation of `com.google.devtools.common.options.Converter`. +The Java type of the value of a command line option is usually something simple +(a string, an integer, a Boolean, a label, etc.). However, we also support +options of more complicated types; in this case, the job of converting from the +command line string to the data type falls to an implementation of +`com.google.devtools.common.options.Converter`. ## The source tree, as seen by Bazel -Bazel is in the business of building software, which happens by reading and interpreting the source code. The totality of the source code Bazel operates on is called "the workspace" and it is structured into repositories, packages and rules. +Bazel is in the business of building software, which happens by reading and +interpreting the source code. The totality of the source code Bazel operates on +is called "the workspace" and it is structured into repositories, packages and +rules. ### Repositories -A "repository" is a source tree on which a developer works; it usually represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, that is, a single source tree that contains all source code used to run the build. Bazel, in contrast, supports projects whose source code spans multiple repositories. The repository from which Bazel is invoked is called the "main repository", the others are called "external repositories". +A "repository" is a source tree on which a developer works; it usually +represents a single project. Bazel's ancestor, Blaze, operated on a monorepo, +that is, a single source tree that contains all source code used to run the build. +Bazel, in contrast, supports projects whose source code spans multiple +repositories. The repository from which Bazel is invoked is called the "main +repository", the others are called "external repositories". -A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The main repo is the source tree where you're invoking Bazel from. External repos are defined in various ways; see [external dependencies overview](/external/overview) for more information. +A repository is marked by a repo boundary file (`MODULE.bazel`, `REPO.bazel`, or +in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`) in its root directory. The +main repo is the source tree where you're invoking Bazel from. External repos +are defined in various ways; see [external dependencies +overview](/external/overview) for more information. -Code of external repositories is symlinked or downloaded under `$OUTPUT_BASE/external`. +Code of external repositories is symlinked or downloaded under +`$OUTPUT_BASE/external`. -When running the build, the whole source tree needs to be pieced together; this is done by `SymlinkForest`, which symlinks every package in the main repository to `$EXECROOT` and every external repository to either `$EXECROOT/external` or `$EXECROOT/..`. +When running the build, the whole source tree needs to be pieced together; this +is done by `SymlinkForest`, which symlinks every package in the main repository +to `$EXECROOT` and every external repository to either `$EXECROOT/external` or +`$EXECROOT/..`. ### Packages -Every repository is composed of packages, a collection of related files and a specification of the dependencies. These are specified by a file called `BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this file name. However, it turned out to be a commonly used path segment, especially on Windows, where file names are case-insensitive. - -Packages are independent of each other: changes to the `BUILD` file of a package cannot cause other packages to change. The addition or removal of `BUILD` files \_can \_change other packages, since recursive globs stop at package boundaries and thus the presence of a `BUILD` file stops the recursion. - -The evaluation of a `BUILD` file is called "package loading". It's implemented in the class `PackageFactory`, works by calling the Starlark interpreter and requires knowledge of the set of available rule classes. The result of package loading is a `Package` object. It's mostly a map from a string (the name of a target) to the target itself. - -A large chunk of complexity during package loading is globbing: Bazel does not require every source file to be explicitly listed and instead can run globs (such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that descend into subdirectories (but not into subpackages). This requires access to the file system and since that can be slow, we implement all sorts of tricks to make it run in parallel and as efficiently as possible. +Every repository is composed of packages, a collection of related files and +a specification of the dependencies. These are specified by a file called +`BUILD` or `BUILD.bazel`. If both exist, Bazel prefers `BUILD.bazel`; the reason +why `BUILD` files are still accepted is that Bazel's ancestor, Blaze, used this +file name. However, it turned out to be a commonly used path segment, especially +on Windows, where file names are case-insensitive. + +Packages are independent of each other: changes to the `BUILD` file of a package +cannot cause other packages to change. The addition or removal of `BUILD` files +_can _change other packages, since recursive globs stop at package boundaries +and thus the presence of a `BUILD` file stops the recursion. + +The evaluation of a `BUILD` file is called "package loading". It's implemented +in the class `PackageFactory`, works by calling the Starlark interpreter and +requires knowledge of the set of available rule classes. The result of package +loading is a `Package` object. It's mostly a map from a string (the name of a +target) to the target itself. + +A large chunk of complexity during package loading is globbing: Bazel does not +require every source file to be explicitly listed and instead can run globs +(such as `glob(["**/*.java"])`). Unlike the shell, it supports recursive globs that +descend into subdirectories (but not into subpackages). This requires access to +the file system and since that can be slow, we implement all sorts of tricks to +make it run in parallel and as efficiently as possible. Globbing is implemented in the following classes: -- `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber -- `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to the legacy globber in order to avoid "Skyframe restarts" (described below) +* `LegacyGlobber`, a fast and blissfully Skyframe-unaware globber +* `SkyframeHybridGlobber`, a version that uses Skyframe and reverts back to + the legacy globber in order to avoid "Skyframe restarts" (described below) -The `Package` class itself contains some members that are exclusively used to parse the "external" package (related to external dependencies) and which do not make sense for real packages. This is a design flaw because objects describing regular packages should not contain fields that describe something else. These include: +The `Package` class itself contains some members that are exclusively used to +parse the "external" package (related to external dependencies) and which do not +make sense for real packages. This is +a design flaw because objects describing regular packages should not contain +fields that describe something else. These include: -- The repository mappings -- The registered toolchains -- The registered execution platforms +* The repository mappings +* The registered toolchains +* The registered execution platforms -Ideally, there would be more separation between parsing the "external" package from parsing regular packages so that `Package` does not need to cater for the needs of both. This is unfortunately difficult to do because the two are intertwined quite deeply. +Ideally, there would be more separation between parsing the "external" package +from parsing regular packages so that `Package` does not need to cater for the +needs of both. This is unfortunately difficult to do because the two are +intertwined quite deeply. ### Labels, Targets, and Rules Packages are composed of targets, which have the following types: -1. **Files:** things that are either the input or the output of the build. In Bazel parlance, we call them *artifacts* (discussed elsewhere). Not all files created during the build are targets; it's common for an output of Bazel not to have an associated label. -2. **Rules:** these describe steps to derive its outputs from its inputs. They are generally associated with a programming language (such as `cc_library`, `java_library` or `py_library`), but there are some language-agnostic ones (such as `genrule` or `filegroup`) -3. **Package groups:** discussed in the [Visibility](#visibility) section. - -The name of a target is called a *Label*. The syntax of labels is `@repo//pac/kage:name`, where `repo` is the name of the repository the Label is in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of the file (if the label refers to a source file) relative to the directory of the package. When referring to a target on the command line, some parts of the label can be omitted: - -1. If the repository is omitted, the label is taken to be in the main repository. -2. If the package part is omitted (such as `name` or `:name`), the label is taken to be in the package of the current working directory (relative paths containing uplevel references (..) are not allowed) - -A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may be implemented either in Starlark (the `rule()` function) or in Java (so called "native rules", type `RuleClass`). In the long term, every language-specific rule will be implemented in Starlark, but some legacy rule families (such as Java or C++) are still in Java for the time being. - -Starlark rule classes need to be imported at the beginning of `BUILD` files using the `load()` statement, whereas Java rule classes are "innately" known by Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. +1. **Files:** things that are either the input or the output of the build. In + Bazel parlance, we call them _artifacts_ (discussed elsewhere). Not all + files created during the build are targets; it's common for an output of + Bazel not to have an associated label. +2. **Rules:** these describe steps to derive its outputs from its inputs. They + are generally associated with a programming language (such as `cc_library`, + `java_library` or `py_library`), but there are some language-agnostic ones + (such as `genrule` or `filegroup`) +3. **Package groups:** discussed in the [Visibility](#visibility) section. + +The name of a target is called a _Label_. The syntax of labels is +`@repo//pac/kage:name`, where `repo` is the name of the repository the Label is +in, `pac/kage` is the directory its `BUILD` file is in and `name` is the path of +the file (if the label refers to a source file) relative to the directory of the +package. When referring to a target on the command line, some parts of the label +can be omitted: + +1. If the repository is omitted, the label is taken to be in the main + repository. +2. If the package part is omitted (such as `name` or `:name`), the label is taken + to be in the package of the current working directory (relative paths + containing uplevel references (..) are not allowed) + +A kind of a rule (such as "C++ library") is called a "rule class". Rule classes may +be implemented either in Starlark (the `rule()` function) or in Java (so called +"native rules", type `RuleClass`). In the long term, every language-specific +rule will be implemented in Starlark, but some legacy rule families (such as Java +or C++) are still in Java for the time being. + +Starlark rule classes need to be imported at the beginning of `BUILD` files +using the `load()` statement, whereas Java rule classes are "innately" known by +Bazel, by virtue of being registered with the `ConfiguredRuleClassProvider`. Rule classes contain information such as: -1. Its attributes (such as `srcs`, `deps`): their types, default values, constraints, etc. -2. The configuration transitions and aspects attached to each attribute, if any -3. The implementation of the rule -4. The transitive info providers the rule "usually" creates +1. Its attributes (such as `srcs`, `deps`): their types, default values, + constraints, etc. +2. The configuration transitions and aspects attached to each attribute, if any +3. The implementation of the rule +4. The transitive info providers the rule "usually" creates -**Terminology note:** In the codebase, we often use "Rule" to mean the target created by a rule class. But in Starlark and in user-facing documentation, "Rule" should be used exclusively to refer to the rule class itself; the target is just a "target". Also note that despite `RuleClass` having "class" in its name, there is no Java inheritance relationship between a rule class and targets of that type. +**Terminology note:** In the codebase, we often use "Rule" to mean the target +created by a rule class. But in Starlark and in user-facing documentation, +"Rule" should be used exclusively to refer to the rule class itself; the target +is just a "target". Also note that despite `RuleClass` having "class" in its +name, there is no Java inheritance relationship between a rule class and targets +of that type. ## Skyframe -The evaluation framework underlying Bazel is called Skyframe. Its model is that everything that needs to be built during a build is organized into a directed acyclic graph with edges pointing from any pieces of data to its dependencies, that is, other pieces of data that need to be known to construct it. - -The nodes in the graph are called `SkyValue`s and their names are called `SkyKey`s. Both are deeply immutable; only immutable objects should be reachable from them. This invariant almost always holds, and in case it doesn't (such as for the individual options classes `BuildOptions`, which is a member of `BuildConfigurationValue` and its `SkyKey`) we try really hard not to change them or to change them in only ways that are not observable from the outside. From this it follows that everything that is computed within Skyframe (such as configured targets) must also be immutable. - -The most convenient way to observe the Skyframe graph is to run `bazel dump --skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best to do it for tiny builds, since it can get pretty large. - -Skyframe lives in the `com.google.devtools.build.skyframe` package. The similarly-named package `com.google.devtools.build.lib.skyframe` contains the implementation of Bazel on top of Skyframe. More information about Skyframe is available [here](/reference/skyframe). - -To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the `SkyFunction` corresponding to the type of the key. During the function's evaluation, it may request other dependencies from Skyframe by calling the various overloads of `SkyFunction.Environment.getValue()`. This has the side-effect of registering those dependencies into Skyframe's internal graph, so that Skyframe will know to re-evaluate the function when any of its dependencies change. In other words, Skyframe's caching and incremental computation work at the granularity of `SkyFunction`s and `SkyValue`s. - -Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` will return null. The function should then yield control back to Skyframe by itself returning null. At some later point, Skyframe will evaluate the unavailable dependency, then restart the function from the beginning — only this time the `getValue()` call will succeed with a non-null result. - -A consequence of this is that any computation performed inside the `SkyFunction` prior to the restart must be repeated. But this does not include work done to evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work around this issue by: - -1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to limit the number of restarts. -2. Breaking up a `SkyValue` into separate pieces computed by different `SkyFunction`s, so that they can be computed and cached independently. This should be done strategically, since it has the potential to increases memory usage. -3. Storing state between restarts, either using `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache "behind the back of Skyframe". With complex SkyFunctions, state management between restarts can get tricky, so [`StateMachine`s](/contribute/statemachine-guide) were introduced for a structured approach to logical concurrency, including hooks to suspend and resume hierarchical computations within a `SkyFunction`. Example: [`DependencyResolver#computeDependencies`](https://developers.google.com/devsite/reference/markdown/links#reference_links) uses a `StateMachine` with `getState()` to compute the potentially huge set of direct dependencies of a configured target, which otherwise can result in expensive restarts. - -Fundamentally, Bazel need these types of workarounds because hundreds of thousands of in-flight Skyframe nodes is common, and Java's support of lightweight threads [does not outperform](/contribute/statemachine-guide#epilogue_eventually_removing_callbacks) the `StateMachine` implementation as of 2023. +The evaluation framework underlying Bazel is called Skyframe. Its model is that +everything that needs to be built during a build is organized into a directed +acyclic graph with edges pointing from any pieces of data to its dependencies, +that is, other pieces of data that need to be known to construct it. + +The nodes in the graph are called `SkyValue`s and their names are called +`SkyKey`s. Both are deeply immutable; only immutable objects should be +reachable from them. This invariant almost always holds, and in case it doesn't +(such as for the individual options classes `BuildOptions`, which is a member of +`BuildConfigurationValue` and its `SkyKey`) we try really hard not to change +them or to change them in only ways that are not observable from the outside. +From this it follows that everything that is computed within Skyframe (such as +configured targets) must also be immutable. + +The most convenient way to observe the Skyframe graph is to run `bazel dump +--skyframe=deps`, which dumps the graph, one `SkyValue` per line. It's best +to do it for tiny builds, since it can get pretty large. + +Skyframe lives in the `com.google.devtools.build.skyframe` package. The +similarly-named package `com.google.devtools.build.lib.skyframe` contains the +implementation of Bazel on top of Skyframe. More information about Skyframe is +available [here](/reference/skyframe). + +To evaluate a given `SkyKey` into a `SkyValue`, Skyframe will invoke the +`SkyFunction` corresponding to the type of the key. During the function's +evaluation, it may request other dependencies from Skyframe by calling the +various overloads of `SkyFunction.Environment.getValue()`. This has the +side-effect of registering those dependencies into Skyframe's internal graph, so +that Skyframe will know to re-evaluate the function when any of its dependencies +change. In other words, Skyframe's caching and incremental computation work at +the granularity of `SkyFunction`s and `SkyValue`s. + +Whenever a `SkyFunction` requests a dependency that is unavailable, `getValue()` +will return null. The function should then yield control back to Skyframe by +itself returning null. At some later point, Skyframe will evaluate the +unavailable dependency, then restart the function from the beginning — only this +time the `getValue()` call will succeed with a non-null result. + +A consequence of this is that any computation performed inside the `SkyFunction` +prior to the restart must be repeated. But this does not include work done to +evaluate dependency `SkyValues`, which are cached. Therefore, we commonly work +around this issue by: + +1. Declaring dependencies in batches (by using `getValuesAndExceptions()`) to + limit the number of restarts. +2. Breaking up a `SkyValue` into separate pieces computed by different + `SkyFunction`s, so that they can be computed and cached independently. This + should be done strategically, since it has the potential to increases memory + usage. +3. Storing state between restarts, either using + `SkyFunction.Environment.getState()`, or keeping an ad hoc static cache + "behind the back of Skyframe". With complex SkyFunctions, state management + between restarts can get tricky, so + [`StateMachine`s](/contribute/statemachine-guide) were introduced for a + structured approach to logical concurrency, including hooks to suspend and + resume hierarchical computations within a `SkyFunction`. Example: + [`DependencyResolver#computeDependencies`][statemachine_example] + uses a `StateMachine` with `getState()` to compute the potentially huge set + of direct dependencies of a configured target, which otherwise can result in + expensive restarts. + +[statemachine_example]: https://developers.google.com/devsite/reference/markdown/links#reference_links + +Fundamentally, Bazel need these types of workarounds because hundreds of +thousands of in-flight Skyframe nodes is common, and Java's support of +lightweight threads [does not outperform][virtual_threads] the +`StateMachine` implementation as of 2023. + +[virtual_threads]: /contribute/statemachine-guide#epilogue_eventually_removing_callbacks ## Starlark -Starlark is the domain-specific language people use to configure and extend Bazel. It's conceived as a restricted subset of Python that has far fewer types, more restrictions on control flow, and most importantly, strong immutability guarantees to enable concurrent reads. It is not Turing-complete, which discourages some (but not all) users from trying to accomplish general programming tasks within the language. +Starlark is the domain-specific language people use to configure and extend +Bazel. It's conceived as a restricted subset of Python that has far fewer types, +more restrictions on control flow, and most importantly, strong immutability +guarantees to enable concurrent reads. It is not Turing-complete, which +discourages some (but not all) users from trying to accomplish general +programming tasks within the language. -Starlark is implemented in the `net.starlark.java` package. It also has an independent Go implementation [here](https://github.com/google/starlark-go). The Java implementation used in Bazel is currently an interpreter. +Starlark is implemented in the `net.starlark.java` package. +It also has an independent Go implementation +[here](https://github.com/google/starlark-go). The Java +implementation used in Bazel is currently an interpreter. Starlark is used in several contexts, including: -1. **`BUILD` files.** This is where new build targets are defined. Starlark code running in this context only has access to the contents of the `BUILD` file itself and `.bzl` files loaded by it. -2. **The `MODULE.bazel` file.** This is where external dependencies are defined. Starlark code running in this context only has very limited access to a few predefined directives. -3. **`.bzl` files.** This is where new build rules, repo rules, module extensions are defined. Starlark code here can define new functions and load from other `.bzl` files. +1. **`BUILD` files.** This is where new build targets are defined. Starlark + code running in this context only has access to the contents of the `BUILD` + file itself and `.bzl` files loaded by it. +2. **The `MODULE.bazel` file.** This is where external dependencies are + defined. Starlark code running in this context only has very limited access + to a few predefined directives. +3. **`.bzl` files.** This is where new build rules, repo rules, module + extensions are defined. Starlark code here can define new functions and load + from other `.bzl` files. -The dialects available for `BUILD` and `.bzl` files are slightly different because they express different things. A list of differences is available [here](/rules/language#differences-between-build-and-bzl-files). +The dialects available for `BUILD` and `.bzl` files are slightly different +because they express different things. A list of differences is available +[here](/rules/language#differences-between-build-and-bzl-files). More information about Starlark is available [here](/rules/language). ## The loading/analysis phase -The loading/analysis phase is where Bazel determines what actions are needed to build a particular rule. Its basic unit is a "configured target", which is, quite sensibly, a (target, configuration) pair. - -It's called the "loading/analysis phase" because it can be split into two distinct parts, which used to be serialized, but they can now overlap in time: - -1. Loading packages, that is, turning `BUILD` files into the `Package` objects that represent them -2. Analyzing configured targets, that is, running the implementation of the rules to produce the action graph - -Each configured target in the transitive closure of the configured targets requested on the command line must be analyzed bottom-up; that is, leaf nodes first, then up to the ones on the command line. The inputs to the analysis of a single configured target are: - -1. **The configuration.** ("how" to build that rule; for example, the target platform but also things like command line options the user wants to be passed to the C++ compiler) -2. **The direct dependencies.** Their transitive info providers are available to the rule being analyzed. They are called like that because they provide a "roll-up" of the information in the transitive closure of the configured target, such as all the .jar files on the classpath or all the .o files that need to be linked into a C++ binary) -3. **The target itself**. This is the result of loading the package the target is in. For rules, this includes its attributes, which is usually what matters. -4. **The implementation of the configured target.** For rules, this can either be in Starlark or in Java. All non-rule configured targets are implemented in Java. +The loading/analysis phase is where Bazel determines what actions are needed to +build a particular rule. Its basic unit is a "configured target", which is, +quite sensibly, a (target, configuration) pair. + +It's called the "loading/analysis phase" because it can be split into two +distinct parts, which used to be serialized, but they can now overlap in time: + +1. Loading packages, that is, turning `BUILD` files into the `Package` objects + that represent them +2. Analyzing configured targets, that is, running the implementation of the + rules to produce the action graph + +Each configured target in the transitive closure of the configured targets +requested on the command line must be analyzed bottom-up; that is, leaf nodes +first, then up to the ones on the command line. The inputs to the analysis of +a single configured target are: + +1. **The configuration.** ("how" to build that rule; for example, the target + platform but also things like command line options the user wants to be + passed to the C++ compiler) +2. **The direct dependencies.** Their transitive info providers are available + to the rule being analyzed. They are called like that because they provide a + "roll-up" of the information in the transitive closure of the configured + target, such as all the .jar files on the classpath or all the .o files that + need to be linked into a C++ binary) +3. **The target itself**. This is the result of loading the package the target + is in. For rules, this includes its attributes, which is usually what + matters. +4. **The implementation of the configured target.** For rules, this can either + be in Starlark or in Java. All non-rule configured targets are implemented + in Java. The output of analyzing a configured target is: -1. The transitive info providers that configured targets that depend on it can access -2. The artifacts it can create and the actions that produce them. +1. The transitive info providers that configured targets that depend on it can + access +2. The artifacts it can create and the actions that produce them. -The API offered to Java rules is `RuleContext`, which is the equivalent of the `ctx` argument of Starlark rules. Its API is more powerful, but at the same time, it's easier to do Bad Things™, for example to write code whose time or space complexity is quadratic (or worse), to make the Bazel server crash with a Java exception or to violate invariants (such as by inadvertently modifying an `Options` instance or by making a configured target mutable) +The API offered to Java rules is `RuleContext`, which is the equivalent of the +`ctx` argument of Starlark rules. Its API is more powerful, but at the same +time, it's easier to do Bad Things™, for example to write code whose time or +space complexity is quadratic (or worse), to make the Bazel server crash with a +Java exception or to violate invariants (such as by inadvertently modifying an +`Options` instance or by making a configured target mutable) -The algorithm that determines the direct dependencies of a configured target lives in `DependencyResolver.dependentNodeMap()`. +The algorithm that determines the direct dependencies of a configured target +lives in `DependencyResolver.dependentNodeMap()`. ### Configurations -Configurations are the "how" of building a target: for what platform, with what command line options, etc. - -The same target can be built for multiple configurations in the same build. This is useful, for example, when the same code is used for a tool that's run during the build and for the target code and we are cross-compiling or when we are building a fat Android app (one that contains native code for multiple CPU architectures) - -Conceptually, the configuration is a `BuildOptions` instance. However, in practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides additional sundry pieces of functionality. It propagates from the top of the dependency graph to the bottom. If it changes, the build needs to be re-analyzed. - -This results in anomalies like having to re-analyze the whole build if, for example, the number of requested test runs changes, even though that only affects test targets (we have plans to "trim" configurations so that this is not the case, but it's not ready yet). - -When a rule implementation needs part of the configuration, it needs to declare it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` . This is both to avoid mistakes (such as Python rules using the Java fragment) and to facilitate configuration trimming so that such as if Python options change, C++ targets don't need to be re-analyzed. - -The configuration of a rule is not necessarily the same as that of its "parent" rule. The process of changing the configuration in a dependency edge is called a "configuration transition". It can happen in two places: - -1. On a dependency edge. These transitions are specified in `Attribute.Builder.cfg()` and are functions from a `Rule` (where the transition happens) and a `BuildOptions` (the original configuration) to one or more `BuildOptions` (the output configuration). -2. On any incoming edge to a configured target. These are specified in `RuleClass.Builder.cfg()`. +Configurations are the "how" of building a target: for what platform, with what +command line options, etc. + +The same target can be built for multiple configurations in the same build. This +is useful, for example, when the same code is used for a tool that's run during +the build and for the target code and we are cross-compiling or when we are +building a fat Android app (one that contains native code for multiple CPU +architectures) + +Conceptually, the configuration is a `BuildOptions` instance. However, in +practice, `BuildOptions` is wrapped by `BuildConfiguration` that provides +additional sundry pieces of functionality. It propagates from the top of the +dependency graph to the bottom. If it changes, the build needs to be +re-analyzed. + +This results in anomalies like having to re-analyze the whole build if, for +example, the number of requested test runs changes, even though that only +affects test targets (we have plans to "trim" configurations so that this is +not the case, but it's not ready yet). + +When a rule implementation needs part of the configuration, it needs to declare +it in its definition using `RuleClass.Builder.requiresConfigurationFragments()` +. This is both to avoid mistakes (such as Python rules using the Java fragment) and +to facilitate configuration trimming so that such as if Python options change, C++ +targets don't need to be re-analyzed. + +The configuration of a rule is not necessarily the same as that of its "parent" +rule. The process of changing the configuration in a dependency edge is called a +"configuration transition". It can happen in two places: + +1. On a dependency edge. These transitions are specified in + `Attribute.Builder.cfg()` and are functions from a `Rule` (where the + transition happens) and a `BuildOptions` (the original configuration) to one + or more `BuildOptions` (the output configuration). +2. On any incoming edge to a configured target. These are specified in + `RuleClass.Builder.cfg()`. The relevant classes are `TransitionFactory` and `ConfigurationTransition`. Configuration transitions are used, for example: -1. To declare that a particular dependency is used during the build and it should thus be built in the execution architecture -2. To declare that a particular dependency must be built for multiple architectures (such as for native code in fat Android APKs) +1. To declare that a particular dependency is used during the build and it + should thus be built in the execution architecture +2. To declare that a particular dependency must be built for multiple + architectures (such as for native code in fat Android APKs) -If a configuration transition results in multiple configurations, it's called a *split transition.* +If a configuration transition results in multiple configurations, it's called a +_split transition._ -Configuration transitions can also be implemented in Starlark (documentation [here](/extending/config)) +Configuration transitions can also be implemented in Starlark (documentation +[here](/extending/config)) ### Transitive info providers -Transitive info providers are a way (and the \_only \_way) for configured targets to learn things about other configured targets that they depend on, and the only way to tell things about themselves to other configured targets that depend on them. The reason why "transitive" is in their name is that this is usually some sort of roll-up of the transitive closure of a configured target. - -There is generally a 1:1 correspondence between Java transitive info providers and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of `FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was deemed to be more Starlark-ish than a direct transliteration of the Java one). Their key is one of the following things: - -1. A Java Class object. This is only available for providers that are not accessible from Starlark. These providers are a subclass of `TransitiveInfoProvider`. -2. A string. This is legacy and heavily discouraged since it's susceptible to name clashes. Such transitive info providers are direct subclasses of `build.lib.packages.Info` . -3. A provider symbol. This can be created from Starlark using the `provider()` function and is the recommended way to create new providers. The symbol is represented by a `Provider.Key` instance in Java. - -New providers implemented in Java should be implemented using `BuiltinProvider`. `NativeProvider` is deprecated (we haven't had time to remove it yet) and `TransitiveInfoProvider` subclasses cannot be accessed from Starlark. +Transitive info providers are a way (and the _only _way) for configured targets +to learn things about other configured targets that they depend on, and the only +way to tell things about themselves to other configured targets that depend on +them. The reason why "transitive" is in their name is that this is usually some +sort of roll-up of the transitive closure of a configured target. + +There is generally a 1:1 correspondence between Java transitive info providers +and Starlark ones (the exception is `DefaultInfo` which is an amalgamation of +`FileProvider`, `FilesToRunProvider` and `RunfilesProvider` because that API was +deemed to be more Starlark-ish than a direct transliteration of the Java one). +Their key is one of the following things: + +1. A Java Class object. This is only available for providers that are not + accessible from Starlark. These providers are a subclass of + `TransitiveInfoProvider`. +2. A string. This is legacy and heavily discouraged since it's susceptible to + name clashes. Such transitive info providers are direct subclasses of + `build.lib.packages.Info` . +3. A provider symbol. This can be created from Starlark using the `provider()` + function and is the recommended way to create new providers. The symbol is + represented by a `Provider.Key` instance in Java. + +New providers implemented in Java should be implemented using `BuiltinProvider`. +`NativeProvider` is deprecated (we haven't had time to remove it yet) and +`TransitiveInfoProvider` subclasses cannot be accessed from Starlark. ### Configured targets -Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a subclass for each rule class implemented in Java. Starlark configured targets are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . +Configured targets are implemented as `RuleConfiguredTargetFactory`. There is a +subclass for each rule class implemented in Java. Starlark configured targets +are created through `StarlarkRuleConfiguredTargetUtil.buildRule()` . -Configured target factories should use `RuleConfiguredTargetBuilder` to construct their return value. It consists of the following things: +Configured target factories should use `RuleConfiguredTargetBuilder` to +construct their return value. It consists of the following things: -1. Their `filesToBuild`, the hazy concept of "the set of files this rule represents." These are the files that get built when the configured target is on the command line or in the srcs of a genrule. -2. Their runfiles, regular and data. -3. Their output groups. These are various "other sets of files" the rule can build. They can be accessed using the output\_group attribute of the filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. +1. Their `filesToBuild`, the hazy concept of "the set of files this rule + represents." These are the files that get built when the configured target + is on the command line or in the srcs of a genrule. +2. Their runfiles, regular and data. +3. Their output groups. These are various "other sets of files" the rule can + build. They can be accessed using the output\_group attribute of the + filegroup rule in BUILD and using the `OutputGroupInfo` provider in Java. ### Runfiles -Some binaries need data files to run. A prominent example is tests that need input files. This is represented in Bazel by the concept of "runfiles". A "runfiles tree" is a directory tree of the data files for a particular binary. It is created in the file system as a symlink tree with individual symlinks pointing to the files in the source or output trees. - -A set of runfiles is represented as a `Runfiles` instance. It is conceptually a map from the path of a file in the runfiles tree to the `Artifact` instance that represents it. It's a little more complicated than a single `Map` for two reasons: - -- Most of the time, the runfiles path of a file is the same as its execpath. We use this to save some RAM. -- There are various legacy kinds of entries in runfiles trees, which also need to be represented. - -Runfiles are collected using `RunfilesProvider`: an instance of this class represents the runfiles a configured target (such as a library) and its transitive closure needs and they are gathered like a nested set (in fact, they are implemented using nested sets under the cover): each target unions the runfiles of its dependencies, adds some of its own, then sends the resulting set upwards in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` instances, one for when the rule is depended on through the "data" attribute and one for every other kind of incoming dependency. This is because a target sometimes presents different runfiles when depended on through a data attribute than otherwise. This is undesired legacy behavior that we haven't gotten around removing yet. - -Runfiles of binaries are represented as an instance of `RunfilesSupport`. This is different from `Runfiles` because `RunfilesSupport` has the capability of actually being built (unlike `Runfiles`, which is just a mapping). This necessitates the following additional components: - -- **The input runfiles manifest.** This is a serialized description of the runfiles tree. It is used as a proxy for the contents of the runfiles tree and Bazel assumes that the runfiles tree changes if and only if the contents of the manifest change. -- **The output runfiles manifest.** This is used by runtime libraries that handle runfiles trees, notably on Windows, which sometimes doesn't support symbolic links. -- **Command line arguments** for running the binary whose runfiles the `RunfilesSupport` object represents. +Some binaries need data files to run. A prominent example is tests that need +input files. This is represented in Bazel by the concept of "runfiles". A +"runfiles tree" is a directory tree of the data files for a particular binary. +It is created in the file system as a symlink tree with individual symlinks +pointing to the files in the source or output trees. + +A set of runfiles is represented as a `Runfiles` instance. It is conceptually a +map from the path of a file in the runfiles tree to the `Artifact` instance that +represents it. It's a little more complicated than a single `Map` for two +reasons: + +* Most of the time, the runfiles path of a file is the same as its execpath. + We use this to save some RAM. +* There are various legacy kinds of entries in runfiles trees, which also need + to be represented. + +Runfiles are collected using `RunfilesProvider`: an instance of this class +represents the runfiles a configured target (such as a library) and its transitive +closure needs and they are gathered like a nested set (in fact, they are +implemented using nested sets under the cover): each target unions the runfiles +of its dependencies, adds some of its own, then sends the resulting set upwards +in the dependency graph. A `RunfilesProvider` instance contains two `Runfiles` +instances, one for when the rule is depended on through the "data" attribute and +one for every other kind of incoming dependency. This is because a target +sometimes presents different runfiles when depended on through a data attribute +than otherwise. This is undesired legacy behavior that we haven't gotten around +removing yet. + +Runfiles of binaries are represented as an instance of `RunfilesSupport`. This +is different from `Runfiles` because `RunfilesSupport` has the capability of +actually being built (unlike `Runfiles`, which is just a mapping). This +necessitates the following additional components: + +* **The input runfiles manifest.** This is a serialized description of the + runfiles tree. It is used as a proxy for the contents of the runfiles tree + and Bazel assumes that the runfiles tree changes if and only if the contents + of the manifest change. +* **The output runfiles manifest.** This is used by runtime libraries that + handle runfiles trees, notably on Windows, which sometimes doesn't support + symbolic links. +* **Command line arguments** for running the binary whose runfiles the + `RunfilesSupport` object represents. ### Aspects -Aspects are a way to "propagate computation down the dependency graph". They are described for users of Bazel [here](/extending/aspects). A good motivating example is protocol buffers: a `proto_library` rule should not know about any particular language, but building the implementation of a protocol buffer message (the "basic unit" of protocol buffers) in any programming language should be coupled to the `proto_library` rule so that if two targets in the same language depend on the same protocol buffer, it gets built only once. - -Just like configured targets, they are represented in Skyframe as a `SkyValue` and the way they are constructed is very similar to how configured targets are built: they have a factory class called `ConfiguredAspectFactory` that has access to a `RuleContext`, but unlike configured target factories, it also knows about the configured target it is attached to and its providers. - -The set of aspects propagated down the dependency graph is specified for each attribute using the `Attribute.Builder.aspects()` function. There are a few confusingly-named classes that participate in the process: - -1. `AspectClass` is the implementation of the aspect. It can be either in Java (in which case it's a subclass) or in Starlark (in which case it's an instance of `StarlarkAspectClass`). It's analogous to `RuleConfiguredTargetFactory`. -2. `AspectDefinition` is the definition of the aspect; it includes the providers it requires, the providers it provides and contains a reference to its implementation, such as the appropriate `AspectClass` instance. It's analogous to `RuleClass`. -3. `AspectParameters` is a way to parametrize an aspect that is propagated down the dependency graph. It's currently a string to string map. A good example of why it's useful is protocol buffers: if a language has multiple APIs, the information as to which API the protocol buffers should be built for should be propagated down the dependency graph. -4. `Aspect` represents all the data that's needed to compute an aspect that propagates down the dependency graph. It consists of the aspect class, its definition and its parameters. -5. `RuleAspect` is the function that determines which aspects a particular rule should propagate. It's a `Rule` -> `Aspect` function. - -A somewhat unexpected complication is that aspects can attach to other aspects; for example, an aspect collecting the classpath for a Java IDE will probably want to know about all the .jar files on the classpath, but some of them are protocol buffers. In that case, the IDE aspect will want to attach to the (`proto_library` rule + Java proto aspect) pair. - -The complexity of aspects on aspects is captured in the class `AspectCollection`. +Aspects are a way to "propagate computation down the dependency graph". They are +described for users of Bazel +[here](/extending/aspects). A good +motivating example is protocol buffers: a `proto_library` rule should not know +about any particular language, but building the implementation of a protocol +buffer message (the "basic unit" of protocol buffers) in any programming +language should be coupled to the `proto_library` rule so that if two targets in +the same language depend on the same protocol buffer, it gets built only once. + +Just like configured targets, they are represented in Skyframe as a `SkyValue` +and the way they are constructed is very similar to how configured targets are +built: they have a factory class called `ConfiguredAspectFactory` that has +access to a `RuleContext`, but unlike configured target factories, it also knows +about the configured target it is attached to and its providers. + +The set of aspects propagated down the dependency graph is specified for each +attribute using the `Attribute.Builder.aspects()` function. There are a few +confusingly-named classes that participate in the process: + +1. `AspectClass` is the implementation of the aspect. It can be either in Java + (in which case it's a subclass) or in Starlark (in which case it's an + instance of `StarlarkAspectClass`). It's analogous to + `RuleConfiguredTargetFactory`. +2. `AspectDefinition` is the definition of the aspect; it includes the + providers it requires, the providers it provides and contains a reference to + its implementation, such as the appropriate `AspectClass` instance. It's + analogous to `RuleClass`. +3. `AspectParameters` is a way to parametrize an aspect that is propagated down + the dependency graph. It's currently a string to string map. A good example + of why it's useful is protocol buffers: if a language has multiple APIs, the + information as to which API the protocol buffers should be built for should + be propagated down the dependency graph. +4. `Aspect` represents all the data that's needed to compute an aspect that + propagates down the dependency graph. It consists of the aspect class, its + definition and its parameters. +5. `RuleAspect` is the function that determines which aspects a particular rule + should propagate. It's a `Rule` -> `Aspect` function. + +A somewhat unexpected complication is that aspects can attach to other aspects; +for example, an aspect collecting the classpath for a Java IDE will probably +want to know about all the .jar files on the classpath, but some of them are +protocol buffers. In that case, the IDE aspect will want to attach to the +(`proto_library` rule + Java proto aspect) pair. + +The complexity of aspects on aspects is captured in the class +`AspectCollection`. ### Platforms and toolchains -Bazel supports multi-platform builds, that is, builds where there may be multiple architectures where build actions run and multiple architectures for which code is built. These architectures are referred to as *platforms* in Bazel parlance (full documentation [here](/extending/platforms)) - -A platform is described by a key-value mapping from *constraint settings* (such as the concept of "CPU architecture") to *constraint values* (such as a particular CPU like x86\_64). We have a "dictionary" of the most commonly used constraint settings and values in the `@platforms` repository. - -The concept of *toolchain* comes from the fact that depending on what platforms the build is running on and what platforms are targeted, one may need to use different compilers; for example, a particular C++ toolchain may run on a specific OS and be able to target some other OSes. Bazel must determine the C++ compiler that is used based on the set execution and target platform (documentation for toolchains [here](/extending/toolchains)). - -In order to do this, toolchains are annotated with the set of execution and target platform constraints they support. In order to do this, the definition of a toolchain are split into two parts: - -1. A `toolchain()` rule that describes the set of execution and target constraints a toolchain supports and tells what kind (such as C++ or Java) of toolchain it is (the latter is represented by the `toolchain_type()` rule) -2. A language-specific rule that describes the actual toolchain (such as `cc_toolchain()`) - -This is done in this way because we need to know the constraints for every toolchain in order to do toolchain resolution and language-specific `*_toolchain()` rules contain much more information than that, so they take more time to load. +Bazel supports multi-platform builds, that is, builds where there may be +multiple architectures where build actions run and multiple architectures for +which code is built. These architectures are referred to as _platforms_ in Bazel +parlance (full documentation +[here](/extending/platforms)) + +A platform is described by a key-value mapping from _constraint settings_ (such as +the concept of "CPU architecture") to _constraint values_ (such as a particular CPU +like x86\_64). We have a "dictionary" of the most commonly used constraint +settings and values in the `@platforms` repository. + +The concept of _toolchain_ comes from the fact that depending on what platforms +the build is running on and what platforms are targeted, one may need to use +different compilers; for example, a particular C++ toolchain may run on a +specific OS and be able to target some other OSes. Bazel must determine the C++ +compiler that is used based on the set execution and target platform +(documentation for toolchains +[here](/extending/toolchains)). + +In order to do this, toolchains are annotated with the set of execution and +target platform constraints they support. In order to do this, the definition of +a toolchain are split into two parts: + +1. A `toolchain()` rule that describes the set of execution and target + constraints a toolchain supports and tells what kind (such as C++ or Java) of + toolchain it is (the latter is represented by the `toolchain_type()` rule) +2. A language-specific rule that describes the actual toolchain (such as + `cc_toolchain()`) + +This is done in this way because we need to know the constraints for every +toolchain in order to do toolchain resolution and language-specific +`*_toolchain()` rules contain much more information than that, so they take more +time to load. Execution platforms are specified in one of the following ways: -1. In the MODULE.bazel file using the `register_execution_platforms()` function -2. On the command line using the --extra\_execution\_platforms command line option - -The set of available execution platforms is computed in `RegisteredExecutionPlatformsFunction` . - -The target platform for a configured target is determined by `PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we eventually want to support multiple target platforms, but it's not implemented yet. - -The set of toolchains to be used for a configured target is determined by `ToolchainResolutionFunction`. It is a function of: - -- The set of registered toolchains (in the MODULE.bazel file and the configuration) -- The desired execution and target platforms (in the configuration) -- The set of toolchain types that are required by the configured target (in `UnloadedToolchainContextKey)` -- The set of execution platform constraints of the configured target (the `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` - -Its result is an `UnloadedToolchainContext`, which is essentially a map from toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of the selected toolchain. It's called "unloaded" because it does not contain the toolchains themselves, only their labels. - -Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` and used by the implementation of the configured target that requested them. - -We also have a legacy system that relies on there being one single "host" configuration and target configurations being represented by various configuration flags, such as `--cpu` . We are gradually transitioning to the above system. In order to handle cases where people rely on the legacy configuration values, we have implemented [platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) to translate between the legacy flags and the new-style platform constraints. Their code is in `PlatformMappingFunction` and uses a non-Starlark "little language". +1. In the MODULE.bazel file using the `register_execution_platforms()` function +2. On the command line using the --extra\_execution\_platforms command line + option + +The set of available execution platforms is computed in +`RegisteredExecutionPlatformsFunction` . + +The target platform for a configured target is determined by +`PlatformOptions.computeTargetPlatform()` . It's a list of platforms because we +eventually want to support multiple target platforms, but it's not implemented +yet. + +The set of toolchains to be used for a configured target is determined by +`ToolchainResolutionFunction`. It is a function of: + +* The set of registered toolchains (in the MODULE.bazel file and the + configuration) +* The desired execution and target platforms (in the configuration) +* The set of toolchain types that are required by the configured target (in + `UnloadedToolchainContextKey)` +* The set of execution platform constraints of the configured target (the + `exec_compatible_with` attribute), in `UnloadedToolchainContextKey` + +Its result is an `UnloadedToolchainContext`, which is essentially a map from +toolchain type (represented as a `ToolchainTypeInfo` instance) to the label of +the selected toolchain. It's called "unloaded" because it does not contain the +toolchains themselves, only their labels. + +Then the toolchains are actually loaded using `ResolvedToolchainContext.load()` +and used by the implementation of the configured target that requested them. + +We also have a legacy system that relies on there being one single "host" +configuration and target configurations being represented by various +configuration flags, such as `--cpu` . We are gradually transitioning to the above +system. In order to handle cases where people rely on the legacy configuration +values, we have implemented +[platform mappings](https://docs.google.com/document/d/1Vg_tPgiZbSrvXcJ403vZVAGlsWhH9BUDrAxMOYnO0Ls) +to translate between the legacy flags and the new-style platform constraints. +Their code is in `PlatformMappingFunction` and uses a non-Starlark "little +language". ### Constraints -Sometimes one wants to designate a target as being compatible with only a few platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: +Sometimes one wants to designate a target as being compatible with only a few +platforms. Bazel has (unfortunately) multiple mechanisms to achieve this end: -- Rule-specific constraints -- `environment_group()` / `environment()` -- Platform constraints +* Rule-specific constraints +* `environment_group()` / `environment()` +* Platform constraints -Rule-specific constraints are mostly used within Google for Java rules; they are on their way out and they are not available in Bazel, but the source code may contain references to it. The attribute that governs this is called `constraints=` . +Rule-specific constraints are mostly used within Google for Java rules; they are +on their way out and they are not available in Bazel, but the source code may +contain references to it. The attribute that governs this is called +`constraints=` . -#### environment\_group() and environment() +#### environment_group() and environment() These rules are a legacy mechanism and are not widely used. -All build rules can declare which "environments" they can be built for, where an "environment" is an instance of the `environment()` rule. +All build rules can declare which "environments" they can be built for, where an +"environment" is an instance of the `environment()` rule. There are various ways supported environments can be specified for a rule: -1. Through the `restricted_to=` attribute. This is the most direct form of specification; it declares the exact set of environments the rule supports. -2. Through the `compatible_with=` attribute. This declares environments a rule supports in addition to "standard" environments that are supported by default. -3. Through the package-level attributes `default_restricted_to=` and `default_compatible_with=`. -4. Through default specifications in `environment_group()` rules. Every environment belongs to a group of thematically related peers (such as "CPU architectures", "JDK versions" or "mobile operating systems"). The definition of an environment group includes which of these environments should be supported by "default" if not otherwise specified by the `restricted_to=` / `environment()` attributes. A rule with no such attributes inherits all defaults. -5. Through a rule class default. This overrides global defaults for all instances of the given rule class. This can be used, for example, to make all `*_test` rules testable without each instance having to explicitly declare this capability. - -`environment()` is implemented as a regular rule whereas `environment_group()` is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a function that is available by default from Starlark (`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous target. This is to avoid a cyclic dependency that would arise because each environment needs to declare the environment group it belongs to and each environment group needs to declare its default environments. - -A build can be restricted to a certain environment with the `--target_environment` command line option. - -The implementation of the constraint check is in `RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. +1. Through the `restricted_to=` attribute. This is the most direct form of + specification; it declares the exact set of environments the rule supports. +2. Through the `compatible_with=` attribute. This declares environments a rule + supports in addition to "standard" environments that are supported by + default. +3. Through the package-level attributes `default_restricted_to=` and + `default_compatible_with=`. +4. Through default specifications in `environment_group()` rules. Every + environment belongs to a group of thematically related peers (such as "CPU + architectures", "JDK versions" or "mobile operating systems"). The + definition of an environment group includes which of these environments + should be supported by "default" if not otherwise specified by the + `restricted_to=` / `environment()` attributes. A rule with no such + attributes inherits all defaults. +5. Through a rule class default. This overrides global defaults for all + instances of the given rule class. This can be used, for example, to make + all `*_test` rules testable without each instance having to explicitly + declare this capability. + +`environment()` is implemented as a regular rule whereas `environment_group()` +is both a subclass of `Target` but not `Rule` (`EnvironmentGroup`) and a +function that is available by default from Starlark +(`StarlarkLibrary.environmentGroup()`) which eventually creates an eponymous +target. This is to avoid a cyclic dependency that would arise because each +environment needs to declare the environment group it belongs to and each +environment group needs to declare its default environments. + +A build can be restricted to a certain environment with the +`--target_environment` command line option. + +The implementation of the constraint check is in +`RuleContextConstraintSemantics` and `TopLevelConstraintSemantics`. #### Platform constraints -The current "official" way to describe what platforms a target is compatible with is by using the same constraints used to describe toolchains and platforms. It was implemented in pull request [#10945](https://github.com/bazelbuild/bazel/pull/10945). +The current "official" way to describe what platforms a target is compatible +with is by using the same constraints used to describe toolchains and platforms. +It was implemented in pull request +[#10945](https://github.com/bazelbuild/bazel/pull/10945). ### Visibility -If you work on a large codebase with a lot of developers (like at Google), you want to take care to prevent everyone else from arbitrarily depending on your code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), people *will* come to rely on behaviors that you considered to be implementation details. +If you work on a large codebase with a lot of developers (like at Google), you +want to take care to prevent everyone else from arbitrarily depending on your +code. Otherwise, as per [Hyrum's law](https://www.hyrumslaw.com/), +people _will_ come to rely on behaviors that you considered to be implementation +details. -Bazel supports this by the mechanism called *visibility*: you can limit which targets can depend on a particular target using the [visibility](/reference/be/common-definitions#common-attributes) attribute. This attribute is a little special because, although it holds a list of labels, these labels may encode a pattern over package names rather than a pointer to any particular target. (Yes, this is a design flaw.) +Bazel supports this by the mechanism called _visibility_: you can limit which +targets can depend on a particular target using the +[visibility](/reference/be/common-definitions#common-attributes) attribute. This +attribute is a little special because, although it holds a list of labels, these +labels may encode a pattern over package names rather than a pointer to any +particular target. (Yes, this is a design flaw.) This is implemented in the following places: -- The `RuleVisibility` interface represents a visibility declaration. It can be either a constant (fully public or fully private) or a list of labels. -- Labels can refer to either package groups (predefined list of packages), to packages directly (`//pkg:__pkg__`) or subtrees of packages (`//pkg:__subpackages__`). This is different from the command line syntax, which uses `//pkg:*` or `//pkg/...`. -- Package groups are implemented as their own target (`PackageGroup`) and configured target (`PackageGroupConfiguredTarget`). We could probably replace these with simple rules if we wanted to. Their logic is implemented with the help of: `PackageSpecification`, which corresponds to a single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds to a single `package_group`'s `packages` attribute; and `PackageSpecificationProvider`, which aggregates over a `package_group` and its transitive `includes`. -- The conversion from visibility label lists to dependencies is done in `DependencyResolver.visitTargetVisibility` and a few other miscellaneous places. -- The actual check is done in `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` +* The `RuleVisibility` interface represents a visibility declaration. It can + be either a constant (fully public or fully private) or a list of labels. +* Labels can refer to either package groups (predefined list of packages), to + packages directly (`//pkg:__pkg__`) or subtrees of packages + (`//pkg:__subpackages__`). This is different from the command line syntax, + which uses `//pkg:*` or `//pkg/...`. +* Package groups are implemented as their own target (`PackageGroup`) and + configured target (`PackageGroupConfiguredTarget`). We could probably + replace these with simple rules if we wanted to. Their logic is implemented + with the help of: `PackageSpecification`, which corresponds to a + single pattern like `//pkg/...`; `PackageGroupContents`, which corresponds + to a single `package_group`'s `packages` attribute; and + `PackageSpecificationProvider`, which aggregates over a `package_group` and + its transitive `includes`. +* The conversion from visibility label lists to dependencies is done in + `DependencyResolver.visitTargetVisibility` and a few other miscellaneous + places. +* The actual check is done in + `CommonPrerequisiteValidator.validateDirectPrerequisiteVisibility()` ### Nested sets -Oftentimes, a configured target aggregates a set of files from its dependencies, adds its own, and wraps the aggregate set into a transitive info provider so that configured targets that depend on it can do the same. Examples: +Oftentimes, a configured target aggregates a set of files from its dependencies, +adds its own, and wraps the aggregate set into a transitive info provider so +that configured targets that depend on it can do the same. Examples: -- The C++ header files used for a build -- The object files that represent the transitive closure of a `cc_library` -- The set of .jar files that need to be on the classpath for a Java rule to compile or run -- The set of Python files in the transitive closure of a Python rule +* The C++ header files used for a build +* The object files that represent the transitive closure of a `cc_library` +* The set of .jar files that need to be on the classpath for a Java rule to + compile or run +* The set of Python files in the transitive closure of a Python rule -If we did this the naive way by using, for example, `List` or `Set`, we'd end up with quadratic memory usage: if there is a chain of N rules and each rule adds a file, we'd have 1+2+...+N collection members. +If we did this the naive way by using, for example, `List` or `Set`, we'd end up with +quadratic memory usage: if there is a chain of N rules and each rule adds a +file, we'd have 1+2+...+N collection members. -In order to get around this problem, we came up with the concept of a `NestedSet`. It's a data structure that is composed of other `NestedSet` instances and some members of its own, thereby forming a directed acyclic graph of sets. They are immutable and their members can be iterated over. We define multiple iteration order (`NestedSet.Order`): preorder, postorder, topological (a node always comes after its ancestors) and "don't care, but it should be the same each time". +In order to get around this problem, we came up with the concept of a +`NestedSet`. It's a data structure that is composed of other `NestedSet` +instances and some members of its own, thereby forming a directed acyclic graph +of sets. They are immutable and their members can be iterated over. We define +multiple iteration order (`NestedSet.Order`): preorder, postorder, topological +(a node always comes after its ancestors) and "don't care, but it should be the +same each time". The same data structure is called `depset` in Starlark. ### Artifacts and Actions -The actual build consists of a set of commands that need to be run to produce the output the user wants. The commands are represented as instances of the class `Action` and the files are represented as instances of the class `Artifact`. They are arranged in a bipartite, directed, acyclic graph called the "action graph". - -Artifacts come in two kinds: source artifacts (ones that are available before Bazel starts executing) and derived artifacts (ones that need to be built). Derived artifacts can themselves be multiple kinds: - -1. **Regular artifacts.** These are checked for up-to-dateness by computing their checksum, with mtime as a shortcut; we don't checksum the file if its ctime hasn't changed. -2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by calling readlink(). Unlike regular artifacts, these can be dangling symlinks. Usually used in cases where one then packs up some files into an archive of some sort. -3. **Tree artifacts.** These are not single files, but directory trees. They are checked for up-to-dateness by checking the set of files in it and their contents. They are represented as a `TreeArtifact`. -4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a rebuild. This is used exclusively for build stamp information: we don't want to do a rebuild just because the current time changed. - -There is no fundamental reason why source artifacts cannot be tree artifacts or unresolved symlink artifacts; it's just that we haven't implemented it yet. At the moment, source symlinks are always resolved, and while source directories are supported, their contents are entirely opaque to build rules and thus don't support the same kind of lazy command line expansion as tree artifacts do. - -Actions are best understood as a command that needs to be run, the environment it needs and the set of outputs it produces. The following things are the main components of the description of an action: - -- The command line that needs to be run -- The input artifacts it needs -- The environment variables that need to be set -- Annotations that describe the environment (such as platform) it needs to run in \\ - -There are also a few other special cases, like writing a file whose content is known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be separate classes), although Java and C++ have their own action types (`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). - -We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is pretty close, but C++ is a bit of a special-case due to .d file parsing and include scanning. - -The action graph is mostly "embedded" into the Skyframe graph: conceptually, the execution of an action is represented as an invocation of `ActionExecutionFunction`. The mapping from an action graph dependency edge to a Skyframe dependency edge is described in `ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few optimizations in order to keep the number of Skyframe edges low: - -- Derived artifacts do not have their own `SkyValue`s. Instead, `Artifact.getGeneratingActionKey()` is used to find out the key for the action that generates it -- Nested sets have their own Skyframe key. +The actual build consists of a set of commands that need to be run to produce +the output the user wants. The commands are represented as instances of the +class `Action` and the files are represented as instances of the class +`Artifact`. They are arranged in a bipartite, directed, acyclic graph called the +"action graph". + +Artifacts come in two kinds: source artifacts (ones that are available +before Bazel starts executing) and derived artifacts (ones that need to be +built). Derived artifacts can themselves be multiple kinds: + +1. **Regular artifacts.** These are checked for up-to-dateness by computing + their checksum, with mtime as a shortcut; we don't checksum the file if its + ctime hasn't changed. +2. **Unresolved symlink artifacts.** These are checked for up-to-dateness by + calling readlink(). Unlike regular artifacts, these can be dangling + symlinks. Usually used in cases where one then packs up some files into an + archive of some sort. +3. **Tree artifacts.** These are not single files, but directory trees. They + are checked for up-to-dateness by checking the set of files in it and their + contents. They are represented as a `TreeArtifact`. +4. **Constant metadata artifacts.** Changes to these artifacts don't trigger a + rebuild. This is used exclusively for build stamp information: we don't want + to do a rebuild just because the current time changed. + +There is no fundamental reason why source artifacts cannot be tree artifacts or +unresolved symlink artifacts; it's just that we haven't implemented it yet. At +the moment, source symlinks are always resolved, and while source directories +are supported, their contents are entirely opaque to build rules and thus don't +support the same kind of lazy command line expansion as tree artifacts do. + +Actions are best understood as a command that needs to be run, the environment +it needs and the set of outputs it produces. The following things are the main +components of the description of an action: + +* The command line that needs to be run +* The input artifacts it needs +* The environment variables that need to be set +* Annotations that describe the environment (such as platform) it needs to run in + \ + +There are also a few other special cases, like writing a file whose content is +known to Bazel. They are a subclass of `AbstractAction`. Most of the actions are +a `SpawnAction` or a `StarlarkAction` (the same, they should arguably not be +separate classes), although Java and C++ have their own action types +(`JavaCompileAction`, `CppCompileAction` and `CppLinkAction`). + +We eventually want to move everything to `SpawnAction`; `JavaCompileAction` is +pretty close, but C++ is a bit of a special-case due to .d file parsing and +include scanning. + +The action graph is mostly "embedded" into the Skyframe graph: conceptually, the +execution of an action is represented as an invocation of +`ActionExecutionFunction`. The mapping from an action graph dependency edge to a +Skyframe dependency edge is described in +`ActionExecutionFunction.getInputDeps()` and `Artifact.key()` and has a few +optimizations in order to keep the number of Skyframe edges low: + +* Derived artifacts do not have their own `SkyValue`s. Instead, + `Artifact.getGeneratingActionKey()` is used to find out the key for the + action that generates it +* Nested sets have their own Skyframe key. ### Shared actions -Some actions are generated by multiple configured targets; Starlark rules are more limited since they are only allowed to put their derived actions into a directory determined by their configuration and their package (but even so, rules in the same package can conflict), but rules implemented in Java can put derived artifacts anywhere. - -This is considered to be a misfeature, but getting rid of it is really hard because it produces significant savings in execution time when, for example, a source file needs to be processed somehow and that file is referenced by multiple rules (handwave-handwave). This comes at the cost of some RAM: each instance of a shared action needs to be stored in memory separately. - -If two actions generate the same output file, they must be exactly the same: have the same inputs, the same outputs and run the same command line. This equivalence relation is implemented in `Actions.canBeShared()` and it is verified between the analysis and execution phases by looking at every Action. This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` and is one of the few places in Bazel that requires a "global" view of the build. +Some actions are generated by multiple configured targets; Starlark rules are +more limited since they are only allowed to put their derived actions into a +directory determined by their configuration and their package (but even so, +rules in the same package can conflict), but rules implemented in Java can put +derived artifacts anywhere. + +This is considered to be a misfeature, but getting rid of it is really hard +because it produces significant savings in execution time when, for example, a +source file needs to be processed somehow and that file is referenced by +multiple rules (handwave-handwave). This comes at the cost of some RAM: each +instance of a shared action needs to be stored in memory separately. + +If two actions generate the same output file, they must be exactly the same: +have the same inputs, the same outputs and run the same command line. This +equivalence relation is implemented in `Actions.canBeShared()` and it is +verified between the analysis and execution phases by looking at every Action. +This is implemented in `SkyframeActionExecutor.findAndStoreArtifactConflicts()` +and is one of the few places in Bazel that requires a "global" view of the +build. ## The execution phase -This is when Bazel actually starts running build actions, such as commands that produce outputs. - -The first thing Bazel does after the analysis phase is to determine what Artifacts need to be built. The logic for this is encoded in `TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the configured targets on the command line and the contents of a special output group for the explicit purpose of expressing "if this target is on the command line, build these artifacts". - -The next step is creating the execution root. Since Bazel has the option to read source packages from different locations in the file system (`--package_path`), it needs to provide locally executed actions with a full source tree. This is handled by the class `SymlinkForest` and works by taking note of every target used in the analysis phase and building up a single directory tree that symlinks every package with a used target from its actual location. An alternative would be to pass the correct paths to commands (taking `--package_path` into account). This is undesirable because: - -- It changes action command lines when a package is moved from a package path entry to another (used to be a common occurrence) -- It results in different command lines if an action is run remotely than if it's run locally -- It requires a command line transformation specific to the tool in use (consider the difference between such as Java classpaths and C++ include paths) -- Changing the command line of an action invalidates its action cache entry -- `--package_path` is slowly and steadily being deprecated - -Then, Bazel starts traversing the action graph (the bipartite, directed graph composed of actions and their input and output artifacts) and running actions. The execution of each action is represented by an instance of the `SkyValue` class `ActionExecutionValue`. - -Since running an action is expensive, we have a few layers of caching that can be hit behind Skyframe: - -- `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts of `ActionExecutionFunction` cheap -- The local action cache contains data about the state of the file system -- Remote execution systems usually also contain their own cache +This is when Bazel actually starts running build actions, such as commands that +produce outputs. + +The first thing Bazel does after the analysis phase is to determine what +Artifacts need to be built. The logic for this is encoded in +`TopLevelArtifactHelper`; roughly speaking, it's the `filesToBuild` of the +configured targets on the command line and the contents of a special output +group for the explicit purpose of expressing "if this target is on the command +line, build these artifacts". + +The next step is creating the execution root. Since Bazel has the option to read +source packages from different locations in the file system (`--package_path`), +it needs to provide locally executed actions with a full source tree. This is +handled by the class `SymlinkForest` and works by taking note of every target +used in the analysis phase and building up a single directory tree that symlinks +every package with a used target from its actual location. An alternative would +be to pass the correct paths to commands (taking `--package_path` into account). +This is undesirable because: + +* It changes action command lines when a package is moved from a package path + entry to another (used to be a common occurrence) +* It results in different command lines if an action is run remotely than if + it's run locally +* It requires a command line transformation specific to the tool in use + (consider the difference between such as Java classpaths and C++ include paths) +* Changing the command line of an action invalidates its action cache entry +* `--package_path` is slowly and steadily being deprecated + +Then, Bazel starts traversing the action graph (the bipartite, directed graph +composed of actions and their input and output artifacts) and running actions. +The execution of each action is represented by an instance of the `SkyValue` +class `ActionExecutionValue`. + +Since running an action is expensive, we have a few layers of caching that can +be hit behind Skyframe: + +* `ActionExecutionFunction.stateMap` contains data to make Skyframe restarts + of `ActionExecutionFunction` cheap +* The local action cache contains data about the state of the file system +* Remote execution systems usually also contain their own cache ### The local action cache -This cache is another layer that sits behind Skyframe; even if an action is re-executed in Skyframe, it can still be a hit in the local action cache. It represents the state of the local file system and it's serialized to disk which means that when one starts up a new Bazel server, one can get local action cache hits even though the Skyframe graph is empty. +This cache is another layer that sits behind Skyframe; even if an action is +re-executed in Skyframe, it can still be a hit in the local action cache. It +represents the state of the local file system and it's serialized to disk which +means that when one starts up a new Bazel server, one can get local action cache +hits even though the Skyframe graph is empty. -This cache is checked for hits using the method `ActionCacheChecker.getTokenIfNeedToExecute()` . +This cache is checked for hits using the method +`ActionCacheChecker.getTokenIfNeedToExecute()` . -Contrary to its name, it's a map from the path of a derived artifact to the action that emitted it. The action is described as: +Contrary to its name, it's a map from the path of a derived artifact to the +action that emitted it. The action is described as: -1. The set of its input and output files and their checksum -2. Its "action key", which is usually the command line that was executed, but in general, represents everything that's not captured by the checksum of the input files (such as for `FileWriteAction`, it's the checksum of the data that's written) +1. The set of its input and output files and their checksum +2. Its "action key", which is usually the command line that was executed, but + in general, represents everything that's not captured by the checksum of the + input files (such as for `FileWriteAction`, it's the checksum of the data + that's written) -There is also a highly experimental "top-down action cache" that is still under development, which uses transitive hashes to avoid going to the cache as many times. +There is also a highly experimental "top-down action cache" that is still under +development, which uses transitive hashes to avoid going to the cache as many +times. ### Input discovery and input pruning -Some actions are more complicated than just having a set of inputs. Changes to the set of inputs of an action come in two forms: - -- An action may discover new inputs before its execution or decide that some of its inputs are not actually necessary. The canonical example is C++, where it's better to make an educated guess about what header files a C++ file uses from its transitive closure so that we don't heed to send every file to remote executors; therefore, we have an option not to register every header file as an "input", but scan the source file for transitively included headers and only mark those header files as inputs that are mentioned in `#include` statements (we overestimate so that we don't need to implement a full C preprocessor) This option is currently hard-wired to "false" in Bazel and is only used at Google. -- An action may realize that some files were not used during its execution. In C++, this is called ".d files": the compiler tells which header files were used after the fact, and in order to avoid the embarrassment of having worse incrementality than Make, Bazel makes use of this fact. This offers a better estimate than the include scanner because it relies on the compiler. +Some actions are more complicated than just having a set of inputs. Changes to +the set of inputs of an action come in two forms: + +* An action may discover new inputs before its execution or decide that some + of its inputs are not actually necessary. The canonical example is C++, + where it's better to make an educated guess about what header files a C++ + file uses from its transitive closure so that we don't heed to send every + file to remote executors; therefore, we have an option not to register every + header file as an "input", but scan the source file for transitively + included headers and only mark those header files as inputs that are + mentioned in `#include` statements (we overestimate so that we don't need to + implement a full C preprocessor) This option is currently hard-wired to + "false" in Bazel and is only used at Google. +* An action may realize that some files were not used during its execution. In + C++, this is called ".d files": the compiler tells which header files were + used after the fact, and in order to avoid the embarrassment of having worse + incrementality than Make, Bazel makes use of this fact. This offers a better + estimate than the include scanner because it relies on the compiler. These are implemented using methods on Action: -1. `Action.discoverInputs()` is called. It should return a nested set of Artifacts that are determined to be required. These must be source artifacts so that there are no dependency edges in the action graph that don't have an equivalent in the configured target graph. -2. The action is executed by calling `Action.execute()`. -3. At the end of `Action.execute()`, the action can call `Action.updateInputs()` to tell Bazel that not all of its inputs were needed. This can result in incorrect incremental builds if a used input is reported as unused. +1. `Action.discoverInputs()` is called. It should return a nested set of + Artifacts that are determined to be required. These must be source artifacts + so that there are no dependency edges in the action graph that don't have an + equivalent in the configured target graph. +2. The action is executed by calling `Action.execute()`. +3. At the end of `Action.execute()`, the action can call + `Action.updateInputs()` to tell Bazel that not all of its inputs were + needed. This can result in incorrect incremental builds if a used input is + reported as unused. -When an action cache returns a hit on a fresh Action instance (such as created after a server restart), Bazel calls `updateInputs()` itself so that the set of inputs reflects the result of input discovery and pruning done before. +When an action cache returns a hit on a fresh Action instance (such as created +after a server restart), Bazel calls `updateInputs()` itself so that the set of +inputs reflects the result of input discovery and pruning done before. -Starlark actions can make use of the facility to declare some inputs as unused using the `unused_inputs_list=` argument of `ctx.actions.run()`. +Starlark actions can make use of the facility to declare some inputs as unused +using the `unused_inputs_list=` argument of +`ctx.actions.run()`. ### Various ways to run actions: Strategies/ActionContexts -Some actions can be run in different ways. For example, a command line can be executed locally, locally but in various kinds of sandboxes, or remotely. The concept that embodies this is called an `ActionContext` (or `Strategy`, since we successfully went only halfway with a rename...) +Some actions can be run in different ways. For example, a command line can be +executed locally, locally but in various kinds of sandboxes, or remotely. The +concept that embodies this is called an `ActionContext` (or `Strategy`, since we +successfully went only halfway with a rename...) The life cycle of an action context is as follows: -1. When the execution phase is started, `BlazeModule` instances are asked what action contexts they have. This happens in the constructor of `ExecutionTool`. Action context types are identified by a Java `Class` instance that refers to a sub-interface of `ActionContext` and which interface the action context must implement. -2. The appropriate action context is selected from the available ones and is forwarded to `ActionExecutionContext` and `BlazeExecutor` . -3. Actions request contexts using `ActionExecutionContext.getContext()` and `BlazeExecutor.getStrategy()` (there should really be only one way to do it…) - -Strategies are free to call other strategies to do their jobs; this is used, for example, in the dynamic strategy that starts actions both locally and remotely, then uses whichever finishes first. - -One notable strategy is the one that implements persistent worker processes (`WorkerSpawnStrategy`). The idea is that some tools have a long startup time and should therefore be reused between actions instead of starting one anew for every action (This does represent a potential correctness issue, since Bazel relies on the promise of the worker process that it doesn't carry observable state between individual requests) - -If the tool changes, the worker process needs to be restarted. Whether a worker can be reused is determined by computing a checksum for the tool used using `WorkerFilesHash`. It relies on knowing which inputs of the action represent part of the tool and which represent inputs; this is determined by the creator of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are counted as parts of the tool. +1. When the execution phase is started, `BlazeModule` instances are asked what + action contexts they have. This happens in the constructor of + `ExecutionTool`. Action context types are identified by a Java `Class` + instance that refers to a sub-interface of `ActionContext` and which + interface the action context must implement. +2. The appropriate action context is selected from the available ones and is + forwarded to `ActionExecutionContext` and `BlazeExecutor` . +3. Actions request contexts using `ActionExecutionContext.getContext()` and + `BlazeExecutor.getStrategy()` (there should really be only one way to do + it…) + +Strategies are free to call other strategies to do their jobs; this is used, for +example, in the dynamic strategy that starts actions both locally and remotely, +then uses whichever finishes first. + +One notable strategy is the one that implements persistent worker processes +(`WorkerSpawnStrategy`). The idea is that some tools have a long startup time +and should therefore be reused between actions instead of starting one anew for +every action (This does represent a potential correctness issue, since Bazel +relies on the promise of the worker process that it doesn't carry observable +state between individual requests) + +If the tool changes, the worker process needs to be restarted. Whether a worker +can be reused is determined by computing a checksum for the tool used using +`WorkerFilesHash`. It relies on knowing which inputs of the action represent +part of the tool and which represent inputs; this is determined by the creator +of the Action: `Spawn.getToolFiles()` and the runfiles of the `Spawn` are +counted as parts of the tool. More information about strategies (or action contexts!): -- Information about various strategies for running actions is available [here](https://jmmv.dev/2019/12/bazel-strategies.html). -- Information about the dynamic strategy, one where we run an action both locally and remotely to see whichever finishes first is available [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). -- Information about the intricacies of executing actions locally is available [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). +* Information about various strategies for running actions is available + [here](https://jmmv.dev/2019/12/bazel-strategies.html). +* Information about the dynamic strategy, one where we run an action both + locally and remotely to see whichever finishes first is available + [here](https://jmmv.dev/series.html#Bazel%20dynamic%20execution). +* Information about the intricacies of executing actions locally is available + [here](https://jmmv.dev/2019/11/bazel-process-wrapper.html). ### The local resource manager -Bazel *can* run many actions in parallel. The number of local actions that *should* be run in parallel differs from action to action: the more resources an action requires, the less instances should be running at the same time to avoid overloading the local machine. +Bazel _can_ run many actions in parallel. The number of local actions that +_should_ be run in parallel differs from action to action: the more resources an +action requires, the less instances should be running at the same time to avoid +overloading the local machine. -This is implemented in the class `ResourceManager`: each action has to be annotated with an estimate of the local resources it requires in the form of a `ResourceSet` instance (CPU and RAM). Then when action contexts do something that requires local resources, they call `ResourceManager.acquireResources()` and are blocked until the required resources are available. +This is implemented in the class `ResourceManager`: each action has to be +annotated with an estimate of the local resources it requires in the form of a +`ResourceSet` instance (CPU and RAM). Then when action contexts do something +that requires local resources, they call `ResourceManager.acquireResources()` +and are blocked until the required resources are available. -A more detailed description of local resource management is available [here](https://jmmv.dev/2019/12/bazel-local-resources.html). +A more detailed description of local resource management is available +[here](https://jmmv.dev/2019/12/bazel-local-resources.html). ### The structure of the output directory -Each action requires a separate place in the output directory where it places its outputs. The location of derived artifacts is usually as follows: +Each action requires a separate place in the output directory where it places +its outputs. The location of derived artifacts is usually as follows: ``` -$EXECROOT/bazel-out/<configuration>/bin/<package>/<artifact name> +$EXECROOT/bazel-out//bin// ``` -How is the name of the directory that is associated with a particular configuration determined? There are two conflicting desirable properties: - -1. If two configurations can occur in the same build, they should have different directories so that both can have their own version of the same action; otherwise, if the two configurations disagree about such as the command line of an action producing the same output file, Bazel doesn't know which action to choose (an "action conflict") -2. If two configurations represent "roughly" the same thing, they should have the same name so that actions executed in one can be reused for the other if the command lines match: for example, changes to the command line options to the Java compiler should not result in C++ compile actions being re-run. - -So far, we have not come up with a principled way of solving this problem, which has similarities to the problem of configuration trimming. A longer discussion of options is available [here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). The main problematic areas are Starlark rules (whose authors usually aren't intimately familiar with Bazel) and aspects, which add another dimension to the space of things that can produce the "same" output file. - -The current approach is that the path segment for the configuration is `<CPU>-<compilation mode>` with various suffixes added so that configuration transitions implemented in Java don't result in action conflicts. In addition, a checksum of the set of Starlark configuration transitions is added so that users can't cause action conflicts. It is far from perfect. This is implemented in `OutputDirectories.buildMnemonic()` and relies on each configuration fragment adding its own part to the name of the output directory. +How is the name of the directory that is associated with a particular +configuration determined? There are two conflicting desirable properties: + +1. If two configurations can occur in the same build, they should have + different directories so that both can have their own version of the same + action; otherwise, if the two configurations disagree about such as the command + line of an action producing the same output file, Bazel doesn't know which + action to choose (an "action conflict") +2. If two configurations represent "roughly" the same thing, they should have + the same name so that actions executed in one can be reused for the other if + the command lines match: for example, changes to the command line options to + the Java compiler should not result in C++ compile actions being re-run. + +So far, we have not come up with a principled way of solving this problem, which +has similarities to the problem of configuration trimming. A longer discussion +of options is available +[here](https://docs.google.com/document/d/1fZI7wHoaS-vJvZy9SBxaHPitIzXE_nL9v4sS4mErrG4/edit). +The main problematic areas are Starlark rules (whose authors usually aren't +intimately familiar with Bazel) and aspects, which add another dimension to the +space of things that can produce the "same" output file. + +The current approach is that the path segment for the configuration is +`-` with various suffixes added so that configuration +transitions implemented in Java don't result in action conflicts. In addition, a +checksum of the set of Starlark configuration transitions is added so that users +can't cause action conflicts. It is far from perfect. This is implemented in +`OutputDirectories.buildMnemonic()` and relies on each configuration fragment +adding its own part to the name of the output directory. ## Tests Bazel has rich support for running tests. It supports: -- Running tests remotely (if a remote execution backend is available) -- Running tests multiple times in parallel (for deflaking or gathering timing data) -- Sharding tests (splitting test cases in same test over multiple processes for speed) -- Re-running flaky tests -- Grouping tests into test suites +* Running tests remotely (if a remote execution backend is available) +* Running tests multiple times in parallel (for deflaking or gathering timing + data) +* Sharding tests (splitting test cases in same test over multiple processes + for speed) +* Re-running flaky tests +* Grouping tests into test suites -Tests are regular configured targets that have a TestProvider, which describes how the test should be run: +Tests are regular configured targets that have a TestProvider, which describes +how the test should be run: -- The artifacts whose building result in the test being run. This is a "cache status" file that contains a serialized `TestResultData` message -- The number of times the test should be run -- The number of shards the test should be split into -- Some parameters about how the test should be run (such as the test timeout) +* The artifacts whose building result in the test being run. This is a "cache + status" file that contains a serialized `TestResultData` message +* The number of times the test should be run +* The number of shards the test should be split into +* Some parameters about how the test should be run (such as the test timeout) ### Determining which tests to run Determining which tests are run is an elaborate process. -First, during target pattern parsing, test suites are recursively expanded. The expansion is implemented in `TestsForTargetPatternFunction`. A somewhat surprising wrinkle is that if a test suite declares no tests, it refers to *every* test in its package. This is implemented in `Package.beforeBuild()` by adding an implicit attribute called `$implicit_tests` to test suite rules. - -Then, tests are filtered for size, tags, timeout and language according to the command line options. This is implemented in `TestFilter` and is called from `TargetPatternPhaseFunction.determineTests()` during target parsing and the result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason why rule attributes which can be filtered for are not configurable is that this happens before the analysis phase, therefore, the configuration is not available. - -This is then processed further in `BuildView.createResult()`: targets whose analysis failed are filtered out and tests are split into exclusive and non-exclusive tests. It's then put into `AnalysisResult`, which is how `ExecutionTool` knows which tests to run. - -In order to lend some transparency to this elaborate process, the `tests()` query operator (implemented in `TestsFunction`) is available to tell which tests are run when a particular target is specified on the command line. It's unfortunately a reimplementation, so it probably deviates from the above in multiple subtle ways. +First, during target pattern parsing, test suites are recursively expanded. The +expansion is implemented in `TestsForTargetPatternFunction`. A somewhat +surprising wrinkle is that if a test suite declares no tests, it refers to +_every_ test in its package. This is implemented in `Package.beforeBuild()` by +adding an implicit attribute called `$implicit_tests` to test suite rules. + +Then, tests are filtered for size, tags, timeout and language according to the +command line options. This is implemented in `TestFilter` and is called from +`TargetPatternPhaseFunction.determineTests()` during target parsing and the +result is put into `TargetPatternPhaseValue.getTestsToRunLabels()`. The reason +why rule attributes which can be filtered for are not configurable is that this +happens before the analysis phase, therefore, the configuration is not +available. + +This is then processed further in `BuildView.createResult()`: targets whose +analysis failed are filtered out and tests are split into exclusive and +non-exclusive tests. It's then put into `AnalysisResult`, which is how +`ExecutionTool` knows which tests to run. + +In order to lend some transparency to this elaborate process, the `tests()` +query operator (implemented in `TestsFunction`) is available to tell which tests +are run when a particular target is specified on the command line. It's +unfortunately a reimplementation, so it probably deviates from the above in +multiple subtle ways. ### Running tests -The way the tests are run is by requesting cache status artifacts. This then results in the execution of a `TestRunnerAction`, which eventually calls the `TestActionContext` chosen by the `--test_strategy` command line option that runs the test in the requested way. - -Tests are run according to an elaborate protocol that uses environment variables to tell tests what's expected from them. A detailed description of what Bazel expects from tests and what tests can expect from Bazel is available [here](/reference/test-encyclopedia). At the simplest, an exit code of 0 means success, anything else means failure. - -In addition to the cache status file, each test process emits a number of other files. They are put in the "test log directory" which is the subdirectory called `testlogs` of the output directory of the target configuration: - -- `test.xml`, a JUnit-style XML file detailing the individual test cases in the test shard -- `test.log`, the console output of the test. stdout and stderr are not separated. -- `test.outputs`, the "undeclared outputs directory"; this is used by tests that want to output files in addition to what they print to the terminal. - -There are two things that can happen during test execution that cannot during building regular targets: exclusive test execution and output streaming. - -Some tests need to be executed in exclusive mode, for example not in parallel with other tests. This can be elicited either by adding `tags=["exclusive"]` to the test rule or running the test with `--test_strategy=exclusive` . Each exclusive test is run by a separate Skyframe invocation requesting the execution of the test after the "main" build. This is implemented in `SkyframeExecutor.runExclusiveTest()`. - -Unlike regular actions, whose terminal output is dumped when the action finishes, the user can request the output of tests to be streamed so that they get informed about the progress of a long-running test. This is specified by the `--test_output=streamed` command line option and implies exclusive test execution so that outputs of different tests are not interspersed. - -This is implemented in the aptly-named `StreamedTestOutput` class and works by polling changes to the `test.log` file of the test in question and dumping new bytes to the terminal where Bazel rules. - -Results of the executed tests are available on the event bus by observing various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). They are dumped to the Build Event Protocol and they are emitted to the console by `AggregatingTestListener`. +The way the tests are run is by requesting cache status artifacts. This then +results in the execution of a `TestRunnerAction`, which eventually calls the +`TestActionContext` chosen by the `--test_strategy` command line option that +runs the test in the requested way. + +Tests are run according to an elaborate protocol that uses environment variables +to tell tests what's expected from them. A detailed description of what Bazel +expects from tests and what tests can expect from Bazel is available +[here](/reference/test-encyclopedia). At the +simplest, an exit code of 0 means success, anything else means failure. + +In addition to the cache status file, each test process emits a number of other +files. They are put in the "test log directory" which is the subdirectory called +`testlogs` of the output directory of the target configuration: + +* `test.xml`, a JUnit-style XML file detailing the individual test cases in + the test shard +* `test.log`, the console output of the test. stdout and stderr are not + separated. +* `test.outputs`, the "undeclared outputs directory"; this is used by tests + that want to output files in addition to what they print to the terminal. + +There are two things that can happen during test execution that cannot during +building regular targets: exclusive test execution and output streaming. + +Some tests need to be executed in exclusive mode, for example not in parallel with +other tests. This can be elicited either by adding `tags=["exclusive"]` to the +test rule or running the test with `--test_strategy=exclusive` . Each exclusive +test is run by a separate Skyframe invocation requesting the execution of the +test after the "main" build. This is implemented in +`SkyframeExecutor.runExclusiveTest()`. + +Unlike regular actions, whose terminal output is dumped when the action +finishes, the user can request the output of tests to be streamed so that they +get informed about the progress of a long-running test. This is specified by the +`--test_output=streamed` command line option and implies exclusive test +execution so that outputs of different tests are not interspersed. + +This is implemented in the aptly-named `StreamedTestOutput` class and works by +polling changes to the `test.log` file of the test in question and dumping new +bytes to the terminal where Bazel rules. + +Results of the executed tests are available on the event bus by observing +various events (such as `TestAttempt`, `TestResult` or `TestingCompleteEvent`). +They are dumped to the Build Event Protocol and they are emitted to the console +by `AggregatingTestListener`. ### Coverage collection -Coverage is reported by the tests in LCOV format in the files `bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . - -To collect coverage, each test execution is wrapped in a script called `collect_coverage.sh` . - -This script sets up the environment of the test to enable coverage collection and determine where the coverage files are written by the coverage runtime(s). It then runs the test. A test may itself run multiple subprocesses and consist of parts written in multiple different programming languages (with separate coverage collection runtimes). The wrapper script is responsible for converting the resulting files to LCOV format if necessary, and merges them into a single file. - -The interposition of `collect_coverage.sh` is done by the test strategies and requires `collect_coverage.sh` to be on the inputs of the test. This is accomplished by the implicit attribute `:coverage_support` which is resolved to the value of the configuration flag `--coverage_support` (see `TestConfiguration.TestOptions.coverageSupport`) - -Some languages do offline instrumentation, meaning that the coverage instrumentation is added at compile time (such as C++) and others do online instrumentation, meaning that coverage instrumentation is added at execution time. - -Another core concept is *baseline coverage*. This is the coverage of a library, binary, or test if no code in it was run. The problem it solves is that if you want to compute the test coverage for a binary, it is not enough to merge the coverage of all of the tests because there may be code in the binary that is not linked into any test. Therefore, what we do is to emit a coverage file for every binary which contains only the files we collect coverage for with no covered lines. The default baseline coverage file for a target is at `bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are encouraged to generate their own baseline coverage files with more meaningful content than just the names of the source files. - -We track two groups of files for coverage collection for each rule: the set of instrumented files and the set of instrumentation metadata files. - -The set of instrumented files is just that, a set of files to instrument. For online coverage runtimes, this can be used at runtime to decide which files to instrument. It is also used to implement baseline coverage. - -The set of instrumentation metadata files is the set of extra files a test needs to generate the LCOV files Bazel requires from it. In practice, this consists of runtime-specific files; for example, gcc emits .gcno files during compilation. These are added to the set of inputs of test actions if coverage mode is enabled. - -Whether or not coverage is being collected is stored in the `BuildConfiguration`. This is handy because it is an easy way to change the test action and the action graph depending on this bit, but it also means that if this bit is flipped, all targets need to be re-analyzed (some languages, such as C++ require different compiler options to emit code that can collect coverage, which mitigates this issue somewhat, since then a re-analysis is needed anyway). - -The coverage support files are depended on through labels in an implicit dependency so that they can be overridden by the invocation policy, which allows them to differ between the different versions of Bazel. Ideally, these differences would be removed, and we standardized on one of them. - -We also generate a "coverage report" which merges the coverage collected for every test in a Bazel invocation. This is handled by `CoverageReportActionFactory` and is called from `BuildView.createResult()` . It gets access to the tools it needs by looking at the `:coverage_report_generator` attribute of the first test that is executed. +Coverage is reported by the tests in LCOV format in the files +`bazel-testlogs/$PACKAGE/$TARGET/coverage.dat` . + +To collect coverage, each test execution is wrapped in a script called +`collect_coverage.sh` . + +This script sets up the environment of the test to enable coverage collection +and determine where the coverage files are written by the coverage runtime(s). +It then runs the test. A test may itself run multiple subprocesses and consist +of parts written in multiple different programming languages (with separate +coverage collection runtimes). The wrapper script is responsible for converting +the resulting files to LCOV format if necessary, and merges them into a single +file. + +The interposition of `collect_coverage.sh` is done by the test strategies and +requires `collect_coverage.sh` to be on the inputs of the test. This is +accomplished by the implicit attribute `:coverage_support` which is resolved to +the value of the configuration flag `--coverage_support` (see +`TestConfiguration.TestOptions.coverageSupport`) + +Some languages do offline instrumentation, meaning that the coverage +instrumentation is added at compile time (such as C++) and others do online +instrumentation, meaning that coverage instrumentation is added at execution +time. + +Another core concept is _baseline coverage_. This is the coverage of a library, +binary, or test if no code in it was run. The problem it solves is that if you +want to compute the test coverage for a binary, it is not enough to merge the +coverage of all of the tests because there may be code in the binary that is not +linked into any test. Therefore, what we do is to emit a coverage file for every +binary which contains only the files we collect coverage for with no covered +lines. The default baseline coverage file for a target is at +`bazel-testlogs/$PACKAGE/$TARGET/baseline_coverage.dat`, but rules are +encouraged to generate their own baseline coverage files with more meaningful +content than just the names of the source files. + +We track two groups of files for coverage collection for each rule: the set of +instrumented files and the set of instrumentation metadata files. + +The set of instrumented files is just that, a set of files to instrument. For +online coverage runtimes, this can be used at runtime to decide which files to +instrument. It is also used to implement baseline coverage. + +The set of instrumentation metadata files is the set of extra files a test needs +to generate the LCOV files Bazel requires from it. In practice, this consists of +runtime-specific files; for example, gcc emits .gcno files during compilation. +These are added to the set of inputs of test actions if coverage mode is +enabled. + +Whether or not coverage is being collected is stored in the +`BuildConfiguration`. This is handy because it is an easy way to change the test +action and the action graph depending on this bit, but it also means that if +this bit is flipped, all targets need to be re-analyzed (some languages, such as +C++ require different compiler options to emit code that can collect coverage, +which mitigates this issue somewhat, since then a re-analysis is needed anyway). + +The coverage support files are depended on through labels in an implicit +dependency so that they can be overridden by the invocation policy, which allows +them to differ between the different versions of Bazel. Ideally, these +differences would be removed, and we standardized on one of them. + +We also generate a "coverage report" which merges the coverage collected for +every test in a Bazel invocation. This is handled by +`CoverageReportActionFactory` and is called from `BuildView.createResult()` . It +gets access to the tools it needs by looking at the `:coverage_report_generator` +attribute of the first test that is executed. ## The query engine -Bazel has a [little language](/query/guide) used to ask it various things about various graphs. The following query kinds are provided: - -- `bazel query` is used to investigate the target graph -- `bazel cquery` is used to investigate the configured target graph -- `bazel aquery` is used to investigate the action graph - -Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. Additional additional query functions can be done by subclassing `QueryFunction` . In order to allow streaming query results, instead of collecting them to some data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which calls it for results it wants to return. - -The result of a query can be emitted in various ways: labels, labels and rule classes, XML, protobuf and so on. These are implemented as subclasses of `OutputFormatter`. - -A subtle requirement of some query output formats (proto, definitely) is that Bazel needs to emit \_all \_the information that package loading provides so that one can diff the output and determine whether a particular target has changed. As a consequence, attribute values need to be serializable, which is why there are only so few attribute types without any attributes having complex Starlark values. The usual workaround is to use a label, and attach the complex information to the rule with that label. It's not a very satisfying workaround and it would be very nice to lift this requirement. +Bazel has a +[little language](/query/guide) +used to ask it various things about various graphs. The following query kinds +are provided: + +* `bazel query` is used to investigate the target graph +* `bazel cquery` is used to investigate the configured target graph +* `bazel aquery` is used to investigate the action graph + +Each of these is implemented by subclassing `AbstractBlazeQueryEnvironment`. +Additional additional query functions can be done by subclassing `QueryFunction` +. In order to allow streaming query results, instead of collecting them to some +data structure, a `query2.engine.Callback` is passed to `QueryFunction`, which +calls it for results it wants to return. + +The result of a query can be emitted in various ways: labels, labels and rule +classes, XML, protobuf and so on. These are implemented as subclasses of +`OutputFormatter`. + +A subtle requirement of some query output formats (proto, definitely) is that +Bazel needs to emit _all _the information that package loading provides so that +one can diff the output and determine whether a particular target has changed. +As a consequence, attribute values need to be serializable, which is why there +are only so few attribute types without any attributes having complex Starlark +values. The usual workaround is to use a label, and attach the complex +information to the rule with that label. It's not a very satisfying workaround +and it would be very nice to lift this requirement. ## The module system -Bazel can be extended by adding modules to it. Each module must subclass `BlazeModule` (the name is a relic of the history of Bazel when it used to be called Blaze) and gets information about various events during the execution of a command. +Bazel can be extended by adding modules to it. Each module must subclass +`BlazeModule` (the name is a relic of the history of Bazel when it used to be +called Blaze) and gets information about various events during the execution of +a command. -They are mostly used to implement various pieces of "non-core" functionality that only some versions of Bazel (such as the one we use at Google) need: +They are mostly used to implement various pieces of "non-core" functionality +that only some versions of Bazel (such as the one we use at Google) need: -- Interfaces to remote execution systems -- New commands +* Interfaces to remote execution systems +* New commands -The set of extension points `BlazeModule` offers is somewhat haphazard. Don't use it as an example of good design principles. +The set of extension points `BlazeModule` offers is somewhat haphazard. Don't +use it as an example of good design principles. ## The event bus -The main way BlazeModules communicate with the rest of Bazel is by an event bus (`EventBus`): a new instance is created for every build, various parts of Bazel can post events to it and modules can register listeners for the events they are interested in. For example, the following things are represented as events: +The main way BlazeModules communicate with the rest of Bazel is by an event bus +(`EventBus`): a new instance is created for every build, various parts of Bazel +can post events to it and modules can register listeners for the events they are +interested in. For example, the following things are represented as events: -- The list of build targets to be built has been determined (`TargetParsingCompleteEvent`) -- The top-level configurations have been determined (`BuildConfigurationEvent`) -- A target was built, successfully or not (`TargetCompleteEvent`) -- A test was run (`TestAttempt`, `TestSummary`) +* The list of build targets to be built has been determined + (`TargetParsingCompleteEvent`) +* The top-level configurations have been determined + (`BuildConfigurationEvent`) +* A target was built, successfully or not (`TargetCompleteEvent`) +* A test was run (`TestAttempt`, `TestSummary`) -Some of these events are represented outside of Bazel in the [Build Event Protocol](/remote/bep) (they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things outside the Bazel process to observe the build. They are accessible either as a file that contains protocol messages or Bazel can connect to a server (called the Build Event Service) to stream events. +Some of these events are represented outside of Bazel in the +[Build Event Protocol](/remote/bep) +(they are `BuildEvent`s). This allows not only `BlazeModule`s, but also things +outside the Bazel process to observe the build. They are accessible either as a +file that contains protocol messages or Bazel can connect to a server (called +the Build Event Service) to stream events. -This is implemented in the `build.lib.buildeventservice` and `build.lib.buildeventstream` Java packages. +This is implemented in the `build.lib.buildeventservice` and +`build.lib.buildeventstream` Java packages. ## External repositories -Note: The information in this section is out of date, as code in this area has undergone extensive change in the past couple of years. Please refer to [external dependencies overview](/external/overview) for more up-to-date information. +Note: The information in this section is out of date, as code in this area has +undergone extensive change in the past couple of years. Please refer to +[external dependencies overview](/external/overview) for more up-to-date +information. -Whereas Bazel was originally designed to be used in a monorepo (a single source tree containing everything one needs to build), Bazel lives in a world where this is not necessarily true. "External repositories" are an abstraction used to bridge these two worlds: they represent code that is necessary for the build but is not in the main source tree. +Whereas Bazel was originally designed to be used in a monorepo (a single source +tree containing everything one needs to build), Bazel lives in a world where +this is not necessarily true. "External repositories" are an abstraction used to +bridge these two worlds: they represent code that is necessary for the build but +is not in the main source tree. ### The WORKSPACE file -The set of external repositories is determined by parsing the WORKSPACE file. For example, a declaration like this: +The set of external repositories is determined by parsing the WORKSPACE file. +For example, a declaration like this: ``` local_repository(name="foo", path="/foo/bar") ``` -Results in the repository called `@foo` being available. Where this gets complicated is that one can define new repository rules in Starlark files, which can then be used to load new Starlark code, which can be used to define new repository rules and so on… +Results in the repository called `@foo` being available. Where this gets +complicated is that one can define new repository rules in Starlark files, which +can then be used to load new Starlark code, which can be used to define new +repository rules and so on… -To handle this case, the parsing of the WORKSPACE file (in `WorkspaceFileFunction`) is split up into chunks delineated by `load()` statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and computing `WorkspaceFileFunction` until index X means evaluating it until the Xth `load()` statement. +To handle this case, the parsing of the WORKSPACE file (in +`WorkspaceFileFunction`) is split up into chunks delineated by `load()` +statements. The chunk index is indicated by `WorkspaceFileKey.getIndex()` and +computing `WorkspaceFileFunction` until index X means evaluating it until the +Xth `load()` statement. ### Fetching repositories -Before the code of the repository is available to Bazel, it needs to be *fetched*. This results in Bazel creating a directory under `$OUTPUT_BASE/external/<repository name>`. +Before the code of the repository is available to Bazel, it needs to be +_fetched_. This results in Bazel creating a directory under +`$OUTPUT_BASE/external/`. Fetching the repository happens in the following steps: -1. `PackageLookupFunction` realizes that it needs a repository and creates a `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` -2. `RepositoryLoaderFunction` forwards the request to `RepositoryDelegatorFunction` for unclear reasons (the code says it's to avoid re-downloading things in case of Skyframe restarts, but it's not a very solid reasoning) -3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to fetch by iterating over the chunks of the WORKSPACE file until the requested repository is found -4. The appropriate `RepositoryFunction` is found that implements the repository fetching; it's either the Starlark implementation of the repository or a hard-coded map for repositories that are implemented in Java. - -There are various layers of caching since fetching a repository can be very expensive: - -1. There is a cache for downloaded files that is keyed by their checksum (`RepositoryCache`). This requires the checksum to be available in the WORKSPACE file, but that's good for hermeticity anyway. This is shared by every Bazel server instance on the same workstation, regardless of which workspace or output base they are running in. -2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` that contains a checksum of the rule that was used to fetch it. If the Bazel server restarts but the checksum does not change, it's not re-fetched. This is implemented in `RepositoryDelegatorFunction.DigestWriter` . -3. The `--distdir` command line option designates another cache that is used to look up artifacts to be downloaded. This is useful in enterprise settings where Bazel should not fetch random things from the Internet. This is implemented by `DownloadManager` . - -Once a repository is downloaded, the artifacts in it are treated as source artifacts. This poses a problem because Bazel usually checks for up-to-dateness of source artifacts by calling stat() on them, and these artifacts are also invalidated when the definition of the repository they are in changes. Thus, `FileStateValue`s for an artifact in an external repository need to depend on their external repository. This is handled by `ExternalFilesHelper`. +1. `PackageLookupFunction` realizes that it needs a repository and creates a + `RepositoryName` as a `SkyKey`, which invokes `RepositoryLoaderFunction` +2. `RepositoryLoaderFunction` forwards the request to + `RepositoryDelegatorFunction` for unclear reasons (the code says it's to + avoid re-downloading things in case of Skyframe restarts, but it's not a + very solid reasoning) +3. `RepositoryDelegatorFunction` finds out the repository rule it's asked to + fetch by iterating over the chunks of the WORKSPACE file until the requested + repository is found +4. The appropriate `RepositoryFunction` is found that implements the repository + fetching; it's either the Starlark implementation of the repository or a + hard-coded map for repositories that are implemented in Java. + +There are various layers of caching since fetching a repository can be very +expensive: + +1. There is a cache for downloaded files that is keyed by their checksum + (`RepositoryCache`). This requires the checksum to be available in the + WORKSPACE file, but that's good for hermeticity anyway. This is shared by + every Bazel server instance on the same workstation, regardless of which + workspace or output base they are running in. +2. A "marker file" is written for each repository under `$OUTPUT_BASE/external` + that contains a checksum of the rule that was used to fetch it. If the Bazel + server restarts but the checksum does not change, it's not re-fetched. This + is implemented in `RepositoryDelegatorFunction.DigestWriter` . +3. The `--distdir` command line option designates another cache that is used to + look up artifacts to be downloaded. This is useful in enterprise settings + where Bazel should not fetch random things from the Internet. This is + implemented by `DownloadManager` . + +Once a repository is downloaded, the artifacts in it are treated as source +artifacts. This poses a problem because Bazel usually checks for up-to-dateness +of source artifacts by calling stat() on them, and these artifacts are also +invalidated when the definition of the repository they are in changes. Thus, +`FileStateValue`s for an artifact in an external repository need to depend on +their external repository. This is handled by `ExternalFilesHelper`. ### Repository mappings -It can happen that multiple repositories want to depend on the same repository, but in different versions (this is an instance of the "diamond dependency problem"). For example, if two binaries in separate repositories in the build want to depend on Guava, they will presumably both refer to Guava with labels starting `@guava//` and expect that to mean different versions of it. - -Therefore, Bazel allows one to re-map external repository labels so that the string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the repository of one binary and another Guava repository (such as `@guava2//`) the repository of the other. - -Alternatively, this can also be used to **join** diamonds. If a repository depends on `@guava1//`, and another depends on `@guava2//`, repository mapping allows one to re-map both repositories to use a canonical `@guava//` repository. - -The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute of individual repository definitions. It then appears in Skyframe as a member of `WorkspaceFileValue`, where it is plumbed to: - -- `Package.Builder.repositoryMapping` which is used to transform label-valued attributes of rules in the package by `RuleClass.populateRuleAttributeValues()` -- `Package.repositoryMapping` which is used in the analysis phase (for resolving things like `$(location)` which are not parsed in the loading phase) -- `BzlLoadFunction` for resolving labels in load() statements +It can happen that multiple repositories want to depend on the same repository, +but in different versions (this is an instance of the "diamond dependency +problem"). For example, if two binaries in separate repositories in the build +want to depend on Guava, they will presumably both refer to Guava with labels +starting `@guava//` and expect that to mean different versions of it. + +Therefore, Bazel allows one to re-map external repository labels so that the +string `@guava//` can refer to one Guava repository (such as `@guava1//`) in the +repository of one binary and another Guava repository (such as `@guava2//`) the +repository of the other. + +Alternatively, this can also be used to **join** diamonds. If a repository +depends on `@guava1//`, and another depends on `@guava2//`, repository mapping +allows one to re-map both repositories to use a canonical `@guava//` repository. + +The mapping is specified in the WORKSPACE file as the `repo_mapping` attribute +of individual repository definitions. It then appears in Skyframe as a member of +`WorkspaceFileValue`, where it is plumbed to: + +* `Package.Builder.repositoryMapping` which is used to transform label-valued + attributes of rules in the package by + `RuleClass.populateRuleAttributeValues()` +* `Package.repositoryMapping` which is used in the analysis phase (for + resolving things like `$(location)` which are not parsed in the loading + phase) +* `BzlLoadFunction` for resolving labels in load() statements ## JNI bits -The server of Bazel is *mostly* written in Java. The exception is the parts that Java cannot do by itself or couldn't do by itself when we implemented it. This is mostly limited to interaction with the file system, process control and various other low-level things. +The server of Bazel is _mostly_ written in Java. The exception is the parts that +Java cannot do by itself or couldn't do by itself when we implemented it. This +is mostly limited to interaction with the file system, process control and +various other low-level things. -The C++ code lives under src/main/native and the Java classes with native methods are: +The C++ code lives under src/main/native and the Java classes with native +methods are: -- `NativePosixFiles` and `NativePosixFileSystem` -- `ProcessUtils` -- `WindowsFileOperations` and `WindowsFileProcesses` -- `com.google.devtools.build.lib.platform` +* `NativePosixFiles` and `NativePosixFileSystem` +* `ProcessUtils` +* `WindowsFileOperations` and `WindowsFileProcesses` +* `com.google.devtools.build.lib.platform` ## Console output -Emitting console output seems like a simple thing, but the confluence of running multiple processes (sometimes remotely), fine-grained caching, the desire to have a nice and colorful terminal output and having a long-running server makes it non-trivial. - -Right after the RPC call comes in from the client, two `RpcOutputStream` instances are created (for stdout and stderr) that forward the data printed into them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) pair). Anything that needs to be printed on the console goes through these streams. Then these streams are handed over to `BlazeCommandDispatcher.execExclusively()`. - -Output is by default printed with ANSI escape sequences. When these are not desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In addition, `System.out` and `System.err` are redirected to these output streams. This is so that debugging information can be printed using `System.err.println()` and still end up in the terminal output of the client (which is different from that of the server). Care is taken that if a process produces binary output (such as `bazel query --output=proto`), no munging of stdout takes place. - -Short messages (errors, warnings and the like) are expressed through the `EventHandler` interface. Notably, these are different from what one posts to the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, warning, info, and a few others) and they may have a `Location` (the place in the source code that caused the event to happen). - -Some `EventHandler` implementations store the events they received. This is used to replay information to the UI caused by various kinds of cached processing, for example, the warnings emitted by a cached configured target. - -Some `EventHandler`s also allow posting events that eventually find their way to the event bus (regular `Event`s do \_not \_appear there). These are implementations of `ExtendedEventHandler` and their main use is to replay cached `EventBus` events. These `EventBus` events all implement `Postable`, but not everything that is posted to `EventBus` necessarily implements this interface; only those that are cached by an `ExtendedEventHandler` (it would be nice and most of the things do; it's not enforced, though) - -Terminal output is *mostly* emitted through `UiEventHandler`, which is responsible for all the fancy output formatting and progress reporting Bazel does. It has two inputs: - -- The event bus -- The event stream piped into it through Reporter - -The only direct connection the command execution machinery (for example the rest of Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, which allows direct access to these streams. It's only used when a command needs to dump large amounts of possible binary data (such as `bazel query`). +Emitting console output seems like a simple thing, but the confluence of running +multiple processes (sometimes remotely), fine-grained caching, the desire to +have a nice and colorful terminal output and having a long-running server makes +it non-trivial. + +Right after the RPC call comes in from the client, two `RpcOutputStream` +instances are created (for stdout and stderr) that forward the data printed into +them to the client. These are then wrapped in an `OutErr` (an (stdout, stderr) +pair). Anything that needs to be printed on the console goes through these +streams. Then these streams are handed over to +`BlazeCommandDispatcher.execExclusively()`. + +Output is by default printed with ANSI escape sequences. When these are not +desired (`--color=no`), they are stripped by an `AnsiStrippingOutputStream`. In +addition, `System.out` and `System.err` are redirected to these output streams. +This is so that debugging information can be printed using +`System.err.println()` and still end up in the terminal output of the client +(which is different from that of the server). Care is taken that if a process +produces binary output (such as `bazel query --output=proto`), no munging of stdout +takes place. + +Short messages (errors, warnings and the like) are expressed through the +`EventHandler` interface. Notably, these are different from what one posts to +the `EventBus` (this is confusing). Each `Event` has an `EventKind` (error, +warning, info, and a few others) and they may have a `Location` (the place in +the source code that caused the event to happen). + +Some `EventHandler` implementations store the events they received. This is used +to replay information to the UI caused by various kinds of cached processing, +for example, the warnings emitted by a cached configured target. + +Some `EventHandler`s also allow posting events that eventually find their way to +the event bus (regular `Event`s do _not _appear there). These are +implementations of `ExtendedEventHandler` and their main use is to replay cached +`EventBus` events. These `EventBus` events all implement `Postable`, but not +everything that is posted to `EventBus` necessarily implements this interface; +only those that are cached by an `ExtendedEventHandler` (it would be nice and +most of the things do; it's not enforced, though) + +Terminal output is _mostly_ emitted through `UiEventHandler`, which is +responsible for all the fancy output formatting and progress reporting Bazel +does. It has two inputs: + +* The event bus +* The event stream piped into it through Reporter + +The only direct connection the command execution machinery (for example the rest of +Bazel) has to the RPC stream to the client is through `Reporter.getOutErr()`, +which allows direct access to these streams. It's only used when a command needs +to dump large amounts of possible binary data (such as `bazel query`). ## Profiling Bazel -Bazel is fast. Bazel is also slow, because builds tend to grow until just the edge of what's bearable. For this reason, Bazel includes a profiler which can be used to profile builds and Bazel itself. It's implemented in a class that's aptly named `Profiler`. It's turned on by default, although it records only abridged data so that its overhead is tolerable; The command line `--record_full_profiler_data` makes it record everything it can. - -It emits a profile in the Chrome profiler format; it's best viewed in Chrome. It's data model is that of task stacks: one can start tasks and end tasks and they are supposed to be neatly nested within each other. Each Java thread gets its own task stack. **TODO:** How does this work with actions and continuation-passing style? - -The profiler is started and stopped in `BlazeRuntime.initProfiler()` and `BlazeRuntime.afterCommand()` respectively and attempts to be live for as long as possible so that we can profile everything. To add something to the profile, call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure represents the end of the task. It's best used with try-with-resources statements. - -We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on and it mostly records maximum heap sizes and GC behavior. +Bazel is fast. Bazel is also slow, because builds tend to grow until just the +edge of what's bearable. For this reason, Bazel includes a profiler which can be +used to profile builds and Bazel itself. It's implemented in a class that's +aptly named `Profiler`. It's turned on by default, although it records only +abridged data so that its overhead is tolerable; The command line +`--record_full_profiler_data` makes it record everything it can. + +It emits a profile in the Chrome profiler format; it's best viewed in Chrome. +It's data model is that of task stacks: one can start tasks and end tasks and +they are supposed to be neatly nested within each other. Each Java thread gets +its own task stack. **TODO:** How does this work with actions and +continuation-passing style? + +The profiler is started and stopped in `BlazeRuntime.initProfiler()` and +`BlazeRuntime.afterCommand()` respectively and attempts to be live for as long +as possible so that we can profile everything. To add something to the profile, +call `Profiler.instance().profile()`. It returns a `Closeable`, whose closure +represents the end of the task. It's best used with try-with-resources +statements. + +We also do rudimentary memory profiling in `MemoryProfiler`. It's also always on +and it mostly records maximum heap sizes and GC behavior. ## Testing Bazel -Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and ones that only run the analysis phase. We call the former "integration tests" and the latter "unit tests", although they are more like integration tests that are, well, less integrated. We also have some actual unit tests, where they are necessary. +Bazel has two main kinds of tests: ones that observe Bazel as a "black box" and +ones that only run the analysis phase. We call the former "integration tests" +and the latter "unit tests", although they are more like integration tests that +are, well, less integrated. We also have some actual unit tests, where they are +necessary. Of integration tests, we have two kinds: -1. Ones implemented using a very elaborate bash test framework under `src/test/shell` -2. Ones implemented in Java. These are implemented as subclasses of `BuildIntegrationTestCase` - -`BuildIntegrationTestCase` is the preferred integration testing framework as it is well-equipped for most testing scenarios. As it is a Java framework, it provides debuggability and seamless integration with many common development tools. There are many examples of `BuildIntegrationTestCase` classes in the Bazel repository. - -Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a scratch file system you can use to write `BUILD` files, then various helper methods can request configured targets, change the configuration and assert various things about the result of the analysis. +1. Ones implemented using a very elaborate bash test framework under + `src/test/shell` +2. Ones implemented in Java. These are implemented as subclasses of + `BuildIntegrationTestCase` + +`BuildIntegrationTestCase` is the preferred integration testing framework as it +is well-equipped for most testing scenarios. As it is a Java framework, it +provides debuggability and seamless integration with many common development +tools. There are many examples of `BuildIntegrationTestCase` classes in the +Bazel repository. + +Analysis tests are implemented as subclasses of `BuildViewTestCase`. There is a +scratch file system you can use to write `BUILD` files, then various helper +methods can request configured targets, change the configuration and assert +various things about the result of the analysis. diff --git a/contribute/design-documents.mdx b/contribute/design-documents.mdx index 199af600..1fe70b9d 100644 --- a/contribute/design-documents.mdx +++ b/contribute/design-documents.mdx @@ -2,94 +2,145 @@ title: 'Design Documents' --- -If you're planning to add, change, or remove a user-facing feature, or make a *significant architectural change* to Bazel, you **must** write a design document and have it reviewed before you can submit the change. + + +If you're planning to add, change, or remove a user-facing feature, or make a +*significant architectural change* to Bazel, you **must** write a design +document and have it reviewed before you can submit the change. Here are some examples of significant changes: -- Addition or deletion of native build rules -- Breaking-changes to native rules -- Changes to a native build rule semantics that affect the behavior of more than a single rule -- Changes to Bazel's rule definition API -- Changes to the APIs that Bazel uses to connect to other systems -- Changes to the Starlark language, semantics, or APIs -- Changes that could have a pervasive effect on Bazel performance or memory usage (for better or for worse) -- Changes to widely used internal APIs -- Changes to flags and command-line interface. +* Addition or deletion of native build rules +* Breaking-changes to native rules +* Changes to a native build rule semantics that affect the behavior of more + than a single rule +* Changes to Bazel's rule definition API +* Changes to the APIs that Bazel uses to connect to other systems +* Changes to the Starlark language, semantics, or APIs +* Changes that could have a pervasive effect on Bazel performance or memory + usage (for better or for worse) +* Changes to widely used internal APIs +* Changes to flags and command-line interface. ## Reasons for design reviews -When you write a design document, you can coordinate with other Bazel developers and seek guidance from Bazel's core team. For example, when a proposal adds, removes, or modifies any function or object available in BUILD, MODULE.bazel, or bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. Design documents are reviewed before submission because: - -- Bazel is a very complex system; seemingly innocuous local changes can have significant global consequences. -- The team gets many feature requests from users; such requests need to be evaluated not only for technical feasibility but importance with regards to other feature requests. -- Bazel features are frequently implemented by people outside the core team; such contributors have widely varying levels of Bazel expertise. -- The Bazel team itself has varying levels of expertise; no single team member has a complete understanding of every corner of Bazel. -- Changes to Bazel must account for backward compatibility and avoid breaking changes. +When you write a design document, you can coordinate with other Bazel developers +and seek guidance from Bazel's core team. For example, when a proposal adds, +removes, or modifies any function or object available in BUILD, MODULE.bazel, or +bzl files, add the [Starlark team](maintainers-guide.md) as reviewers. +Design documents are reviewed before submission because: + +* Bazel is a very complex system; seemingly innocuous local changes can have + significant global consequences. +* The team gets many feature requests from users; such requests need to be + evaluated not only for technical feasibility but importance with regards to + other feature requests. +* Bazel features are frequently implemented by people outside the core team; + such contributors have widely varying levels of Bazel expertise. +* The Bazel team itself has varying levels of expertise; no single team member + has a complete understanding of every corner of Bazel. +* Changes to Bazel must account for backward compatibility and avoid breaking + changes. Bazel's design review policy helps to maximize the likelihood that: -- all feature requests get a baseline level of scrutiny. -- the right people will weigh in on designs before we've invested in an implementation that may not work. +* all feature requests get a baseline level of scrutiny. +* the right people will weigh in on designs before we've invested in an + implementation that may not work. -To help you get started, take a look at the design documents in the [Bazel Proposals Repository](https://github.com/bazelbuild/proposals). Designs are works in progress, so implementation details can change over time and with feedback. The published design documents capture the initial design, and *not* the ongoing changes as designs are implemented. Always go to the documentation for descriptions of current Bazel functionality. +To help you get started, take a look at the design documents in the +[Bazel Proposals Repository](https://github.com/bazelbuild/proposals). +Designs are works in progress, so implementation details can change over time +and with feedback. The published design documents capture the initial design, +and *not* the ongoing changes as designs are implemented. Always go to the +documentation for descriptions of current Bazel functionality. ## Contributor Workflow -As a contributor, you can write a design document, send pull requests and request reviewers for your proposal. +As a contributor, you can write a design document, send pull requests and +request reviewers for your proposal. ### Write the design document All design documents must have a header that includes: -- author -- date of last major change -- list of reviewers, including one (and only one) [lead reviewer](#lead-reviewer) -- current status (*draft*, *in review*, *approved*, *rejected*, *being implemented*, *implemented*) -- link to discussion thread (*to be added after the announcement*) +* author +* date of last major change +* list of reviewers, including one (and only one) + [lead reviewer](#lead-reviewer) +* current status (_draft_, _in review_, _approved_, _rejected_, + _being implemented_, _implemented_) +* link to discussion thread (_to be added after the announcement_) -The document can be written either [as a world-readable Google Doc](#gdocs) or [using Markdown](#markdown). Read below about for a [Markdown / Google Docs comparison](#markdown-versus-gdocs). +The document can be written either [as a world-readable Google Doc](#gdocs) +or [using Markdown](#markdown). Read below about for a +[Markdown / Google Docs comparison](#markdown-versus-gdocs). -Proposals that have a user-visible impact must have a section documenting the impact on backward compatibility (and a rollout plan if needed). +Proposals that have a user-visible impact must have a section documenting the +impact on backward compatibility (and a rollout plan if needed). ### Create a Pull Request -Share your design doc by creating a pull request (PR) to add the document to [the design index](https://github.com/bazelbuild/proposals). Add your markdown file or a document link to your PR. +Share your design doc by creating a pull request (PR) to add the document to +[the design index](https://github.com/bazelbuild/proposals). Add +your markdown file or a document link to your PR. -When possible, [choose a lead reviewer](#lead-reviewer). and cc other reviewers. If you don't choose a lead reviewer, a Bazel maintainer will assign one to your PR. +When possible, [choose a lead reviewer](#lead-reviewer). +and cc other reviewers. If you don't choose a lead reviewer, a Bazel +maintainer will assign one to your PR. -After you create your PR, reviewers can make preliminary comments during the code review. For example, the lead reviewer can suggest extra reviewers, or point out missing information. The lead reviewer approves the PR when they believe the review process can start. This doesn't mean the proposal is perfect or will be approved; it means that the proposal contains enough information to start the discussion. +After you create your PR, reviewers can make preliminary comments during the +code review. For example, the lead reviewer can suggest extra reviewers, or +point out missing information. The lead reviewer approves the PR when they +believe the review process can start. This doesn't mean the proposal is perfect +or will be approved; it means that the proposal contains enough information to +start the discussion. ### Announce the new proposal -Send an announcement to [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when the PR is submitted. +Send an announcement to +[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) when +the PR is submitted. -You may copy other groups (for example, [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), to get feedback from Bazel end-users). +You may copy other groups (for example, +[bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss), +to get feedback from Bazel end-users). ### Iterate with reviewers -Anyone interested can comment on your proposal. Try to answer questions, clarify the proposal, and address concerns. +Anyone interested can comment on your proposal. Try to answer questions, +clarify the proposal, and address concerns. -Discussion should happen on the announcement thread. If the proposal is in a Google Doc, comments may be used instead (Note that anonymous comments are allowed). +Discussion should happen on the announcement thread. If the proposal is in a +Google Doc, comments may be used instead (Note that anonymous comments are +allowed). ### Update the status -Create a new PR to update the status of the proposal, when iteration is complete. Send the PR to the same lead reviewer and cc the other reviewers. +Create a new PR to update the status of the proposal, when iteration is +complete. Send the PR to the same lead reviewer and cc the other reviewers. -To officially accept the proposal, the lead reviewer approves the PR after ensuring that the other reviewers agree with the decision. +To officially accept the proposal, the lead reviewer approves the PR after +ensuring that the other reviewers agree with the decision. -There must be at least 1 week between the first announcement and the approval of a proposal. This ensures that users had enough time to read the document and share their concerns. +There must be at least 1 week between the first announcement and the approval of +a proposal. This ensures that users had enough time to read the document and +share their concerns. -Implementation can begin before the proposal is accepted, for example as a proof-of-concept or an experimentation. However, you cannot submit the change before the review is complete. +Implementation can begin before the proposal is accepted, for example as a +proof-of-concept or an experimentation. However, you cannot submit the change +before the review is complete. ### Choosing a lead reviewer A lead reviewer should be a domain expert who is: -- Knowledgeable of the relevant subsystems -- Objective and capable of providing constructive feedback -- Available for the entire review period to lead the process +* Knowledgeable of the relevant subsystems +* Objective and capable of providing constructive feedback +* Available for the entire review period to lead the process -Consider checking the contacts for various [team labels](/contribute/maintainers-guide#team-labels). +Consider checking the contacts for various [team +labels](/contribute/maintainers-guide#team-labels). ## Markdown vs Google Docs @@ -97,34 +148,49 @@ Decide what works best for you, since both are accepted. Benefits of using Google Docs: -- Effective for brainstorming, since it is easy to get started with. -- Collaborative editing. -- Quick iteration. -- Easy way to suggest edits. +* Effective for brainstorming, since it is easy to get started with. +* Collaborative editing. +* Quick iteration. +* Easy way to suggest edits. Benefits of using Markdown files: -- Clean URLs for linking. -- Explicit record of revisions. -- No forgetting to set up access rights before publicizing a link. -- Easily searchable with search engines. -- Future-proof: Plain text is not at the mercy of any specific tool and doesn't require an Internet connection. -- It is possible to update them even if the author is not around anymore. -- They can be processed automatically (update/detect dead links, fetch list of authors, etc.). +* Clean URLs for linking. +* Explicit record of revisions. +* No forgetting to set up access rights before publicizing a link. +* Easily searchable with search engines. +* Future-proof: Plain text is not at the mercy of any specific tool + and doesn't require an Internet connection. +* It is possible to update them even if the author is not around anymore. +* They can be processed automatically (update/detect dead links, fetch + list of authors, etc.). -You can choose to first iterate on a Google Doc, and then convert it to Markdown for posterity. +You can choose to first iterate on a Google Doc, and then convert it to +Markdown for posterity. ### Using Google Docs -For consistency, use the [Bazel design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). It includes the necessary header and creates visual consistency with other Bazel related documents. To do that, click on **File** > **Make a copy** or click this link to [make a copy of the design doc template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). +For consistency, use the [Bazel design doc template]( +https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/edit). +It includes the necessary header and creates visual +consistency with other Bazel related documents. To do that, click on **File** > +**Make a copy** or click this link to [make a copy of the design doc +template](https://docs.google.com/document/d/1cE5zrjrR40RXNg64XtRFewSv6FrLV6slGkkqxBumS1w/copy). -To make your document readable to the world, click on **Share** > **Advanced** > **Change…**, and choose "On - Anyone with the link". If you allow comments on the document, anyone can comment anonymously, even without a Google account. +To make your document readable to the world, click on +**Share** > **Advanced** > **Change…**, and +choose "On - Anyone with the link". If you allow comments on the document, +anyone can comment anonymously, even without a Google account. ### Using Markdown -Documents are stored on GitHub and use the [GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) ([Specification](https://github.github.com/gfm/)). +Documents are stored on GitHub and use the +[GitHub flavor of Markdown](https://guides.github.com/features/mastering-markdown/) +([Specification](https://github.github.com/gfm/)). -Create a PR to update an existing document. Significant changes should be reviewed by the document reviewers. Trivial changes (such as typos, formatting) can be approved by anyone. +Create a PR to update an existing document. Significant changes should be +reviewed by the document reviewers. Trivial changes (such as typos, formatting) +can be approved by anyone. ## Reviewer workflow @@ -132,41 +198,57 @@ A reviewer comments, reviews and approves design documents. ### General reviewer responsibilities -You're responsible for reviewing design documents, asking for additional information if needed, and approving a design that passes the review process. +You're responsible for reviewing design documents, asking for additional +information if needed, and approving a design that passes the review process. #### When you receive a new proposal -1. Take a quick look at the document. -2. Comment if critical information is missing, or if the design doesn't fit with the goals of the project. -3. Suggest additional reviewers. -4. Approve the PR when it is ready for review. +1. Take a quick look at the document. +1. Comment if critical information is missing, or if the design doesn't fit + with the goals of the project. +1. Suggest additional reviewers. +1. Approve the PR when it is ready for review. #### During the review process -1. Engage in a dialogue with the design author about issues that are problematic or require clarification. -2. If appropriate, invite comments from non-reviewers who should be aware of the design. -3. Decide which comments must be addressed by the author as a prerequisite to approval. -4. Write "LGTM" (*Looks Good To Me*) in the discussion thread when you are happy with the current state of the proposal. +1. Engage in a dialogue with the design author about issues that are problematic + or require clarification. +1. If appropriate, invite comments from non-reviewers who should be aware of + the design. +1. Decide which comments must be addressed by the author as a prerequisite to + approval. +1. Write "LGTM" (_Looks Good To Me_) in the discussion thread when you are + happy with the current state of the proposal. -Follow this process for all design review requests. Do not approve designs affecting Bazel if they are not in the [design index](https://github.com/bazelbuild/proposals). +Follow this process for all design review requests. Do not approve designs +affecting Bazel if they are not in the +[design index](https://github.com/bazelbuild/proposals). ### Lead reviewer responsibilities -You're responsible for making the go / no-go decision on implementation of a pending design. If you're not able to do this, you should identify a suitable delegate (reassign the PR to the delegate), or reassign the bug to a Bazel manager for further disposition. +You're responsible for making the go / no-go decision on implementation +of a pending design. If you're not able to do this, you should identify a +suitable delegate (reassign the PR to the delegate), or reassign the bug to a +Bazel manager for further disposition. #### During the review process -1. Ensure that the comment and design iteration process moves forward constructively. -2. Prior to approval, ensure that concerns from other reviewers have been resolved. +1. Ensure that the comment and design iteration process moves forward + constructively. +1. Prior to approval, ensure that concerns from other reviewers have been + resolved. #### After approval by all reviewers -1. Make sure there has been at least 1 week since the announcement on the mailing list. -2. Make sure the PR updates the status. -3. Approve the PR sent by the proposal author. +1. Make sure there has been at least 1 week since the announcement on the + mailing list. +1. Make sure the PR updates the status. +1. Approve the PR sent by the proposal author. #### Rejecting designs -1. Make sure the PR author sends a PR; or send them a PR. -2. The PR updates the status of the document. -3. Add a comment to the document explaining why the design can't be approved in its current state, and outlining next steps, if any (such as "revisit invalid assumptions and resubmit"). +1. Make sure the PR author sends a PR; or send them a PR. +1. The PR updates the status of the document. +1. Add a comment to the document explaining why the design can't be approved in + its current state, and outlining next steps, if any (such as "revisit invalid + assumptions and resubmit"). diff --git a/contribute/docs-style-guide.mdx b/contribute/docs-style-guide.mdx deleted file mode 100644 index a2e60e67..00000000 --- a/contribute/docs-style-guide.mdx +++ /dev/null @@ -1,176 +0,0 @@ ---- -title: 'Bazel docs style guide' ---- - -Thank you for contributing to Bazel's documentation. This serves as a quick documentation style guide to get you started. For any style questions not answered by this guide, follow the [Google developer documentation style guide](https://developers.google.com/style). - -## Defining principles - -Bazel docs should uphold these principles: - -- **Concise.** Use as few words as possible. -- **Clear.** Use plain language. Write without jargon for a fifth-grade reading level. -- **Consistent.** Use the same words or phrases for repeated concepts throughout the docs. -- **Correct.** Write in a way where the content stays correct for as long as possible by avoiding time-based information and promises for the future. - -## Writing - -This section contains basic writing tips. - -### Headings - -- Page-level headings start at H2. (H1 headings are used as page titles.) - -- Make headers as short as is sensible. This way, they fit in the TOC without wrapping. - - - ✅ Yes: : Permissions - - ⚠️ No: : A brief note on permissions - -- Use sentence case for headings - - - ✅ Yes: : Set up your workspace - - ⚠️ No: : Set Up Your Workspace - -- Try to make headings task-based or actionable. If headings are conceptual, it may be based around understanding, but write to what the user does. - - - ✅ Yes: : Preserving graph order - - ⚠️ No: : On the preservation of graph order - -### Names - -- Capitalize proper nouns, such as Bazel and Starlark. - - - ✅ Yes: : At the end of the build, Bazel prints the requested targets. - - ⚠️ No: : At the end of the build, bazel prints the requested targets. - -- Keep it consistent. Don't introduce new names for existing concepts. Where applicable, use the term defined in the [Glossary](/reference/glossary). - - - For example, if you're writing about issuing commands on a terminal, don't use both terminal and command line on the page. - -### Page scope - -- Each page should have one purpose and that should be defined at the beginning. This helps readers find what they need quicker. - - - ✅ Yes: : This page covers how to install Bazel on Windows. - - ⚠️ No: : (No introductory sentence.) - -- At the end of the page, tell the reader what to do next. For pages where there is no clear action, you can include links to similar concepts, examples, or other avenues for exploration. - -### Subject - -In Bazel documentation, the audience should primarily be users—the people using Bazel to build their software. - -- Address your reader as "you". (If for some reason you can't use "you", use gender-neutral language, such as they.) - - - ✅ Yes: : To build Java code using Bazel, you must install a JDK. - - **MAYBE:** For users to build Java code with Bazel, they must install a JDK. - - ⚠️ No: : For a user to build Java code with Bazel, he or she must install a JDK. - -- If your audience is NOT general Bazel users, define the audience at the beginning of the page or in the section. Other audiences can include maintainers, contributors, migrators, or other roles. - -- Avoid "we". In user docs, there is no author; just tell people what's possible. - - - ✅ Yes: : As Bazel evolves, you should update your code base to maintain compatibility. - - ⚠️ No: : Bazel is evolving, and we will make changes to Bazel that at times will be incompatible and require some changes from Bazel users. - -### Temporal - -Where possible, avoid terms that orient things in time, such as referencing specific dates (Q2 2022) or saying "now", "currently", or "soon." These go stale quickly and could be incorrect if it's a future projection. Instead, specify a version level instead, such as "Bazel X.x and higher supports \ or a GitHub issue link. - -- ✅ Yes: : Bazel 0.10.0 or later supports remote caching. -- ⚠️ No: : Bazel will soon support remote caching, likely in October 2017. - -### Tense - -- Use present tense. Avoid past or future tense unless absolutely necessary for clarity. - - - ✅ Yes: : Bazel issues an error when it finds dependencies that don't conform to this rule. - - ⚠️ No: : If Bazel finds a dependency that does not conform to this rule, Bazel will issue an error. - -- Where possible, use active voice (where a subject acts upon an object) not passive voice (where an object is acted upon by a subject). Generally, active voice makes sentences clearer because it shows who is responsible. If using active voice detracts from clarity, use passive voice. - - - ✅ Yes: : Bazel initiates X and uses the output to build Y. - - ⚠️ No: : X is initiated by Bazel and then afterward Y will be built with the output. - -### Tone - -Write with a business friendly tone. - -- Avoid colloquial language. It's harder to translate phrases that are specific to English. - - - ✅ Yes: : Good rulesets - - ⚠️ No: : So what is a good ruleset? - -- Avoid overly formal language. Write as though you're explaining the concept to someone who is curious about tech, but doesn't know the details. - -## Formatting - -### File type - -For readability, wrap lines at 80 characters. Long links or code snippets may be longer, but should start on a new line. For example: - -Note: Where possible, use Markdown instead of HTML in your files. Follow the [GitHub Markdown Syntax Guide](https://guides.github.com/features/mastering-markdown/#syntax) for recommended Markdown style. - -### Links - -- Use descriptive link text instead of "here" or "below". This practice makes it easier to scan a doc and is better for screen readers. - - - ✅ Yes: : For more details, see \[Installing Bazel]. - - ⚠️ No: : For more details, see \[here]. - -- End the sentence with the link, if possible. - - - ✅ Yes: : For more details, see \[link]. - - ⚠️ No: : See \[link] for more information. - -### Lists - -- Use an ordered list to describe how to accomplish a task with steps - -- Use an unordered list to list things that aren't task based. (There should still be an order of sorts, such as alphabetical, importance, etc.) - -- Write with parallel structure. For example: - - 1. Make all the list items sentences. - 2. Start with verbs that are the same tense. - 3. Use an ordered list if there are steps to follow. - -### Placeholders - -- Use angle brackets to denote a variable that users should change. In Markdown, escape the angle brackets with a back slash: `\<example\>`. - - - ✅ Yes: : `bazel help <command>`: Prints help and options for `<command>` - - ⚠️ No: : bazel help *command*: Prints help and options for "command" - -- Especially for complicated code samples, use placeholders that make sense in context. - -### Table of contents - -Use the auto-generated TOC supported by the site. Don't add a manual TOC. - -## Code - -Code samples are developers' best friends. You probably know how to write these already, but here are a few tips. - -If you're referencing a small snippet of code, you can embed it in a sentence. If you want the reader to use the code, such as copying a command, use a code block. - -### Code blocks - -- Keep it short. Eliminate all redundant or unnecessary text from a code sample. -- In Markdown, specify the type of code block by adding the sample's language. - -```` -```shell -... -```` - -- Separate commands and output into different code blocks. - -### Inline code formatting - -- Use code style for filenames, directories, paths, and small bits of code. - -- Use inline code styling instead of *italics*, "quotes," or **bolding**. - - - ✅ Yes: : `bazel help <command>`: Prints help and options for `<command>` - - ⚠️ No: : bazel help *command*: Prints help and options for "command" diff --git a/contribute/docs.mdx b/contribute/docs.mdx index 940b20e2..cc240cc4 100644 --- a/contribute/docs.mdx +++ b/contribute/docs.mdx @@ -2,26 +2,42 @@ title: 'Contribute to Bazel documentation' --- -Thank you for contributing to Bazel's documentation! There are a few ways to help create better docs for our community. + + +Thank you for contributing to Bazel's documentation! There are a few ways to +help create better docs for our community. ## Documentation types This site includes a few types of content. -- *Narrative documentation*, which is written by technical writers and engineers. Most of this site is narrative documentation that covers conceptual and task-based guides. -- *Reference documentation*, which is generated documentation from code comments. You can't make changes to the reference doc pages directly, but instead need to change their source. + - *Narrative documentation*, which is written by technical writers and + engineers. Most of this site is narrative documentation that covers + conceptual and task-based guides. + - *Reference documentation*, which is generated documentation from code comments. + You can't make changes to the reference doc pages directly, but instead need + to change their source. ## Documentation infrastructure -Bazel documentation is served from Google and the source files are mirrored in Bazel's GitHub repository. You can make changes to the source files in GitHub. If approved, you can merge the changes and a Bazel maintainer will update the website source to publish your updates. +Bazel documentation is served from Google and the source files are mirrored in +Bazel's GitHub repository. You can make changes to the source files in GitHub. +If approved, you can merge the changes and a Bazel maintainer will update the +website source to publish your updates. + ## Small changes -You can approach small changes, such as fixing errors or typos, in a couple of ways. +You can approach small changes, such as fixing errors or typos, in a couple of +ways. -- **Pull request**. You can create a pull request in GitHub with the [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. -- **Bug**. You can file a bug with details and suggested changes and the Bazel documentation owners will make the update. + - **Pull request**. You can create a pull request in GitHub with the + [web-based editor](https://docs.github.com/repositories/working-with-files/managing-files/editing-files) or on a branch. + - **Bug**. You can file a bug with details and suggested changes and the Bazel + documentation owners will make the update. ## Large changes -If you want to make substantial changes to existing documentation or propose new documentation, you can either create a pull request or start with a Google doc and contact the Bazel Owners to collaborate. +If you want to make substantial changes to existing documentation or propose +new documentation, you can either create a pull request or start with a Google +doc and contact the Bazel Owners to collaborate. diff --git a/contribute/index.mdx b/contribute/index.mdx index 18b12854..ee667729 100644 --- a/contribute/index.mdx +++ b/contribute/index.mdx @@ -2,41 +2,57 @@ title: 'Contributing to Bazel' --- + + There are many ways to help the Bazel project and ecosystem. ## Provide feedback -As you use Bazel, you may find things that can be improved. You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) when: +As you use Bazel, you may find things that can be improved. +You can help by [reporting issues](http://github.com/bazelbuild/bazel/issues) +when: -- Bazel crashes or you encounter a bug that can [only be resolved using `bazel clean`](/run/build#correct-incremental-rebuilds). -- The documentation is incomplete or unclear. You can also report issues from the page you are viewing by using the "Create issue" link at the top right corner of the page. -- An error message could be improved. + - Bazel crashes or you encounter a bug that can [only be resolved using `bazel + clean`](/run/build#correct-incremental-rebuilds). + - The documentation is incomplete or unclear. You can also report issues + from the page you are viewing by using the "Create issue" + link at the top right corner of the page. + - An error message could be improved. ## Participate in the community You can engage with the Bazel community by: -- Answering questions [on Stack Overflow](https://stackoverflow.com/questions/tagged/bazel). -- Helping other users [on Slack](https://slack.bazel.build). -- Improving documentation or [contributing examples](https://github.com/bazelbuild/examples). -- Sharing your experience or your tips, for example, on a blog or social media. + - Answering questions [on Stack Overflow]( + https://stackoverflow.com/questions/tagged/bazel). + - Helping other users [on Slack](https://slack.bazel.build). + - Improving documentation or [contributing examples]( + https://github.com/bazelbuild/examples). + - Sharing your experience or your tips, for example, on a blog or social media. ## Contribute code -Bazel is a large project and making a change to the Bazel source code can be difficult. +Bazel is a large project and making a change to the Bazel source code +can be difficult. You can contribute to the Bazel ecosystem by: -- Helping rules maintainers by contributing pull requests. -- Creating new rules and open-sourcing them. -- Contributing to Bazel-related tools, for example, migration tools. -- Improving Bazel integration with other IDEs and tools. + - Helping rules maintainers by contributing pull requests. + - Creating new rules and open-sourcing them. + - Contributing to Bazel-related tools, for example, migration tools. + - Improving Bazel integration with other IDEs and tools. -Before making a change, [create a GitHub issue](http://github.com/bazelbuild/bazel/issues) or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). +Before making a change, [create a GitHub +issue](http://github.com/bazelbuild/bazel/issues) +or email [bazel-discuss@](mailto:bazel-discuss@googlegroups.com). -The most helpful contributions fix bugs or add features (as opposed to stylistic, refactoring, or "cleanup" changes). Your change should include tests and documentation, keeping in mind backward-compatibility, portability, and the impact on memory usage and performance. +The most helpful contributions fix bugs or add features (as opposed +to stylistic, refactoring, or "cleanup" changes). Your change should +include tests and documentation, keeping in mind backward-compatibility, +portability, and the impact on memory usage and performance. -To learn about how to submit a change, see the [patch acceptance process](/contribute/patch-acceptance). +To learn about how to submit a change, see the +[patch acceptance process](/contribute/patch-acceptance). ## Bazel's code description @@ -44,19 +60,23 @@ Bazel has a large codebase with code in multiple locations. See the [codebase gu Bazel is organized as follows: -- Client code is in `src/main/cpp` and provides the command-line interface. - -- Protocol buffers are in `src/main/protobuf`. - -- Server code is in `src/main/java` and `src/test/java`. - - - Core code which is mostly composed of [SkyFrame](/reference/skyframe) and some utilities. - - Built-in rules are in `com.google.devtools.build.lib.rules` and in `com.google.devtools.build.lib.bazel.rules`. You might want to read about the [Challenges of Writing Rules](/rules/challenges) first. - -- Java native interfaces are in `src/main/native`. +* Client code is in `src/main/cpp` and provides the command-line interface. +* Protocol buffers are in `src/main/protobuf`. +* Server code is in `src/main/java` and `src/test/java`. + * Core code which is mostly composed of [SkyFrame](/reference/skyframe) + and some utilities. + * Built-in rules are in `com.google.devtools.build.lib.rules` and in + `com.google.devtools.build.lib.bazel.rules`. You might want to read about + the [Challenges of Writing Rules](/rules/challenges) first. +* Java native interfaces are in `src/main/native`. +* Various tooling for language support are described in the list in the + [compiling Bazel](/install/compile-source) section. -- Various tooling for language support are described in the list in the [compiling Bazel](/install/compile-source) section. ### Searching Bazel's source code -To quickly search through Bazel's source code, use [Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's repositories, branches, and files. You can also view history, diffs, and blame information. To learn more, see the [Bazel Code Search User Guide](/contribute/search). +To quickly search through Bazel's source code, use +[Bazel Code Search](https://source.bazel.build/). You can navigate Bazel's +repositories, branches, and files. You can also view history, diffs, and blame +information. To learn more, see the +[Bazel Code Search User Guide](/contribute/search). diff --git a/contribute/maintainers-guide.mdx b/contribute/maintainers-guide.mdx index b98a408e..9d745afb 100644 --- a/contribute/maintainers-guide.mdx +++ b/contribute/maintainers-guide.mdx @@ -2,141 +2,205 @@ title: 'Guide for Bazel Maintainers' --- + + This is a guide for the maintainers of the Bazel open source project. -If you are looking to contribute to Bazel, please read [Contributing to Bazel](/contribute) instead. +If you are looking to contribute to Bazel, please read [Contributing to +Bazel](/contribute) instead. The objectives of this page are to: -1. Serve as the maintainers' source of truth for the project’s contribution process. -2. Set expectations between the community contributors and the project maintainers. +1. Serve as the maintainers' source of truth for the project’s contribution + process. +1. Set expectations between the community contributors and the project + maintainers. -Bazel's [core group of contributors](/contribute/policy) has dedicated subteams to manage aspects of the open source project. These are: +Bazel's [core group of contributors](/contribute/policy) has dedicated +subteams to manage aspects of the open source project. These are: -- **Release Process**: Manage Bazel's release process. -- **Green Team**: Grow a healthy ecosystem of rules and tools. -- **Developer Experience Gardeners**: Encourage external contributions, review issues and pull requests, and make our development workflow more open. +* **Release Process**: Manage Bazel's release process. +* **Green Team**: Grow a healthy ecosystem of rules and tools. +* **Developer Experience Gardeners**: Encourage external contributions, review + issues and pull requests, and make our development workflow more open. ## Releases -- [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) -- [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) +* [Release Playbook](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md) +* [Testing local changes with downstream projects](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md) ## Continuous Integration -Read the Green team's guide to Bazel's CI infrastructure on the [bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) repository. +Read the Green team's guide to Bazel's CI infrastructure on the +[bazelbuild/continuous-integration](https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md) +repository. ## Lifecycle of an Issue -1. A user creates an issue by choosing one of the [issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) and it enters the pool of [unreviewed open issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93\&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). - -2. A member on the Developer Experience (DevEx) subteam rotation reviews the issue. - - 1. If the issue is **not a bug** or a **feature request**, the DevEx member will usually close the issue and redirect the user to [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for higher visibility on the question. - 2. If the issue belongs in one of the rules repositories owned by the community, like [rules\_apple](https://github.com.bazelbuild/rules_apple), the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) to the correct repository. - 3. If the issue is vague or has missing information, the DevEx member will assign the issue back to the user to request for more information before continuing. This usually occurs when the user does not choose the right [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) or provides incomplete information. - -3. After reviewing the issue, the DevEx member decides if the issue requires immediate attention. If it does, they will assign the **P0** [priority](#priority) label and an owner from the list of team leads. - -4. The DevEx member assigns the `untriaged` label and exactly one [team label](#team-labels) for routing. - -5. The DevEx member also assigns exactly one `type:` label, such as `type: bug` or `type: feature request`, according to the type of the issue. - -6. For platform-specific issues, the DevEx member assigns one `platform:` label, such as `platform:apple` for Mac-specific issues. - -7. If the issue is low priority and can be worked on by a new community contributor, the DevEx member assigns the `good first issue` label. At this stage, the issue enters the pool of [untriaged open issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). - -Each Bazel subteam will triage all issues under labels they own, preferably on a weekly basis. The subteam will review and evaluate the issue and provide a resolution, if possible. If you are an owner of a team label, see [this section ](#label-own)for more information. +1. A user creates an issue by choosing one of the +[issue templates](https://github.com/bazelbuild/bazel/issues/new/choose) + and it enters the pool of [unreviewed open + issues](https://github.com/bazelbuild/bazel/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+-label%3Auntriaged+-label%3Ap2+-label%3Ap1+-label%3Ap3+-label%3Ap4+-label%3Ateam-Starlark+-label%3Ateam-Rules-CPP+-label%3Ateam-Rules-Java+-label%3Ateam-XProduct+-label%3Ateam-Android+-label%3Ateam-Apple+-label%3Ateam-Configurability++-label%3Ateam-Performance+-label%3Ateam-Rules-Server+-label%3Ateam-Core+-label%3Ateam-Rules-Python+-label%3Ateam-Remote-Exec+-label%3Ateam-Local-Exec+-label%3Ateam-Bazel). +1. A member on the Developer Experience (DevEx) subteam rotation reviews the + issue. + 1. If the issue is **not a bug** or a **feature request**, the DevEx member + will usually close the issue and redirect the user to + [StackOverflow](https://stackoverflow.com/questions/tagged/bazel) and + [bazel-discuss](https://groups.google.com/forum/#!forum/bazel-discuss) for + higher visibility on the question. + 1. If the issue belongs in one of the rules repositories owned by the + community, like [rules_apple](https://github.com.bazelbuild/rules_apple), + the DevEx member will [transfer this issue](https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/transferring-an-issue-to-another-repository) + to the correct repository. + 1. If the issue is vague or has missing information, the DevEx member will + assign the issue back to the user to request for more information before + continuing. This usually occurs when the user does not choose the right + [issue template](https://github.com/bazelbuild/bazel/issues/new/choose) + or provides incomplete information. +1. After reviewing the issue, the DevEx member decides if the issue requires + immediate attention. If it does, they will assign the **P0** + [priority](#priority) label and an owner from the list of team leads. +1. The DevEx member assigns the `untriaged` label and exactly one [team + label](#team-labels) for routing. +1. The DevEx member also assigns exactly one `type:` label, such as `type: bug` + or `type: feature request`, according to the type of the issue. +1. For platform-specific issues, the DevEx member assigns one `platform:` label, + such as `platform:apple` for Mac-specific issues. +1. If the issue is low priority and can be worked on by a new community + contributor, the DevEx member assigns the `good first issue` label. +At this stage, the issue enters the pool of [untriaged open +issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aissue+is%3Aopen+label%3Auntriaged). + +Each Bazel subteam will triage all issues under labels they own, preferably on a +weekly basis. The subteam will review and evaluate the issue and provide a +resolution, if possible. If you are an owner of a team label, see [this section +](#label-own) for more information. When an issue is resolved, it can be closed. ## Lifecycle of a Pull Request 1. A user creates a pull request. -2. If you a member of a Bazel team and sending a PR against your own area, you are responsible for assigning your team label and finding the best reviewer. -3. Otherwise, during daily triage, a DevEx member assigns one [team label](#team-labels) and the team's technical lead (TL) for routing. +1. If you a member of a Bazel team and sending a PR against your own area, + you are responsible for assigning your team label and finding the best + reviewer. +1. Otherwise, during daily triage, a DevEx member assigns one + [team label](#team-labels) and the team's technical lead (TL) for routing. 1. The TL may optionally assign someone else to review the PR. -4. The assigned reviewer reviews the PR and works with the author until it is approved or dropped. -5. If approved, the reviewer **imports** the PR's commit(s) into Google's internal version control system for further tests. As Bazel is the same build system used internally at Google, we need to test all PR commits against the internal test suite. This is the reason why we do not merge PRs directly. -6. If the imported commit passes all internal tests, the commit will be squashed and exported back out to GitHub. -7. When the commit merges into master, GitHub automatically closes the PR. +1. The assigned reviewer reviews the PR and works with the author until it is + approved or dropped. +1. If approved, the reviewer **imports** the PR's commit(s) into Google's + internal version control system for further tests. As Bazel is the same build + system used internally at Google, we need to test all PR commits against the + internal test suite. This is the reason why we do not merge PRs directly. +1. If the imported commit passes all internal tests, the commit will be squashed + and exported back out to GitHub. +1. When the commit merges into master, GitHub automatically closes the PR. + ## My team owns a label. What should I do? -Subteams need to triage all issues in the [labels they own](#team-labels), preferably on a weekly basis. +Subteams need to triage all issues in the [labels they own](#team-labels), +preferably on a weekly basis. ### Issues 1. Filter the list of issues by your team label **and** the `untriaged` label. -2. Review the issue. -3. Identify a [priority level](#priority) and assign the label. -4. The issue may have already been prioritized by the DevEx subteam if it's a P0. Re-prioritize if needed. -5. Each issue needs to have exactly one [priority label](#priority). If an issue is either P0 or P1 we assume that is actively worked on. -6. Remove the `untriaged` label. +1. Review the issue. +1. Identify a [priority level](#priority) and assign the label. + 1. The issue may have already been prioritized by the DevEx subteam if it's a + P0. Re-prioritize if needed. + 1. Each issue needs to have exactly one [priority label](#priority). If an + issue is either P0 or P1 we assume that is actively worked on. +1. Remove the `untriaged` label. -Note that you need to be in the [bazelbuild organization](https://github.com/bazelbuild) to be able to add or remove labels. +Note that you need to be in the [bazelbuild +organization](https://github.com/bazelbuild) to be able to add or remove labels. ### Pull Requests 1. Filter the list of pull requests by your team label. -2. Review open pull requests. -3. **Optional**: If you are assigned for the review but is not the right fit for it, re-assign the appropriate reviewer to perform a code review. -4. Work with the pull request creator to complete a code review. -5. Approve the PR. -6. Ensure that all tests pass. -7. Import the patch to the internal version control system and run the internal presubmits. -8. Submit the internal patch. If the patch submits and exports successfully, the PR will be closed automatically by GitHub. +1. Review open pull requests. + 1. **Optional**: If you are assigned for the review but is not the right fit + for it, re-assign the appropriate reviewer to perform a code review. +1. Work with the pull request creator to complete a code review. +1. Approve the PR. +1. Ensure that all tests pass. +1. Import the patch to the internal version control system and run the internal + presubmits. +1. Submit the internal patch. If the patch submits and exports successfully, the + PR will be closed automatically by GitHub. ## Priority -The following definitions for priority will be used by the maintainers to triage issues. - -- [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken functionality that causes a Bazel release (minus release candidates) to be unusable, or a downed service that severely impacts development of the Bazel project. This includes regressions introduced in a new release that blocks a significant number of users, or an incompatible breaking change that was not compliant to the [Breaking Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) policy. No practical workaround exists. -- [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or feature which should be addressed in the next release, or a serious issue that impacts many users (including the development of the Bazel project), but a practical workaround exists. Typically does not require immediate action. In high demand and planned in the current quarter's roadmap. -- [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature that should be addressed but we don't currently work on. Moderate live issue in a released Bazel version that is inconvenient for a user that needs to be addressed in an future release and/or an easy workaround exists. -- [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug fix or enhancement with small impact. Not prioritized into Bazel roadmaps or any imminent release, however community contributions are encouraged. -- [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect or feature request that is unlikely to get closed. Can also be kept open for a potential re-prioritization if more users are impacted. +The following definitions for priority will be used by the maintainers to triage +issues. + +* [**P0**](https://github.com/bazelbuild/bazel/labels/P0) - Major broken + functionality that causes a Bazel release (minus release candidates) to be + unusable, or a downed service that severely impacts development of the Bazel + project. This includes regressions introduced in a new release that blocks a + significant number of users, or an incompatible breaking change that was not + compliant to the [Breaking + Change](https://docs.google.com/document/d/1q5GGRxKrF_mnwtaPKI487P8OdDRh2nN7jX6U-FXnHL0/edit?pli=1#heading=h.ceof6vpkb3ik) + policy. No practical workaround exists. +* [**P1**](https://github.com/bazelbuild/bazel/labels/P1) - Critical defect or + feature which should be addressed in the next release, or a serious issue that + impacts many users (including the development of the Bazel project), but a + practical workaround exists. Typically does not require immediate action. In + high demand and planned in the current quarter's roadmap. +* [**P2**](https://github.com/bazelbuild/bazel/labels/P2) - Defect or feature + that should be addressed but we don't currently work on. Moderate live issue + in a released Bazel version that is inconvenient for a user that needs to be + addressed in an future release and/or an easy workaround exists. +* [**P3**](https://github.com/bazelbuild/bazel/labels/P3) - Desirable minor bug + fix or enhancement with small impact. Not prioritized into Bazel roadmaps or + any imminent release, however community contributions are encouraged. +* [**P4**](https://github.com/bazelbuild/bazel/labels/P4) - Low priority defect + or feature request that is unlikely to get closed. Can also be kept open for a + potential re-prioritization if more users are impacted. ## Team labels -- [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team - - Contact: [ahumesky](https://github.com/ahumesky) -- [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues - - Contact: [meisterT](https://github.com/meisterT) -- [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI - - Contact: [meisterT](https://github.com/meisterT) -- [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags - - Contact: [gregestren](https://github.com/gregestren) -- [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc - - Contact: [haxorz](https://github.com/haxorz) -- [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team -- [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file - - Contact: [meteorcloudy](https://github.com/meteorcloudy) -- [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob - - Contact: [brandjon](https://github.com/brandjon) -- [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team - - Contact: [meisterT](https://github.com/meisterT) -- [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure - - Contact: [meteorcloudy](https://github.com/meteorcloudy) -- [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team - - Contact: [meisterT](https://github.com/meisterT) -- [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team - - Contact: [coeuvre](https://github.com/coeuvre) -- [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts - - Contact: [comius](https://github.com/comius) -- [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic - - Contact: [pzembrod](https://github.com/pzembrod) -- [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules - - Contact: [hvadehra](https://github.com/hvadehra) -- [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules - - Contact: [rickeylev](https://github.com/rickeylev) -- [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel - - Contact: [comius](https://github.com/comius) -- [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. - - Contact: [brandjon](https://github.com/brandjon) -- [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. - - Contact: [brandjon](https://github.com/brandjon) - -For new issues, we deprecated the `category: *` labels in favor of the team labels. +* [`team-Android`](https://github.com/bazelbuild/bazel/labels/team-Android): Issues for Android team + * Contact: [ahumesky](https://github.com/ahumesky) +* [`team-Bazel`](https://github.com/bazelbuild/bazel/labels/team-Bazel): General Bazel product/strategy issues + * Contact: [meisterT](https://github.com/meisterT) +* [`team-CLI`](https://github.com/bazelbuild/bazel/labels/team-CLI): Console UI + * Contact: [meisterT](https://github.com/meisterT) +* [`team-Configurability`](https://github.com/bazelbuild/bazel/labels/team-Configurability): Issues for Configurability team. Includes: Core build configuration and transition system. Does *not* include: Changes to new or existing flags + * Contact: [gregestren](https://github.com/gregestren) +* [`team-Core`](https://github.com/bazelbuild/bazel/labels/team-Core): Skyframe, bazel query, BEP, options parsing, bazelrc + * Contact: [haxorz](https://github.com/haxorz) +* [`team-Documentation`](https://github.com/bazelbuild/bazel/labels/team-Documentation): Issues for Documentation team +* [`team-ExternalDeps`](https://github.com/bazelbuild/bazel/labels/team-ExternalDeps): External dependency handling, Bzlmod, remote repositories, WORKSPACE file + * Contact: [meteorcloudy](https://github.com/meteorcloudy) +* [`team-Loading-API`](https://github.com/bazelbuild/bazel/labels/team-Loading-API): BUILD file and macro processing: labels, package(), visibility, glob + * Contact: [brandjon](https://github.com/brandjon) +* [`team-Local-Exec`](https://github.com/bazelbuild/bazel/labels/team-Local-Exec): Issues for Execution (Local) team + * Contact: [meisterT](https://github.com/meisterT) +* [`team-OSS`](https://github.com/bazelbuild/bazel/labels/team-OSS): Issues for Bazel OSS team: installation, release process, Bazel packaging, website, docs infrastructure + * Contact: [meteorcloudy](https://github.com/meteorcloudy) +* [`team-Performance`](https://github.com/bazelbuild/bazel/labels/team-Performance): Issues for Bazel Performance team + * Contact: [meisterT](https://github.com/meisterT) +* [`team-Remote-Exec`](https://github.com/bazelbuild/bazel/labels/team-Remote-Exec): Issues for Execution (Remote) team + * Contact: [coeuvre](https://github.com/coeuvre) +* [`team-Rules-API`](https://github.com/bazelbuild/bazel/labels/team-Rules-API): API for writing rules/aspects: providers, runfiles, actions, artifacts + * Contact: [comius](https://github.com/comius) +* [`team-Rules-CPP`](https://github.com/bazelbuild/bazel/labels/team-Rules-CPP) / [`team-Rules-ObjC`](https://github.com/bazelbuild/bazel/labels/team-Rules-ObjC): Issues for C++/Objective-C rules, including native Apple rule logic + * Contact: [pzembrod](https://github.com/pzembrod) +* [`team-Rules-Java`](https://github.com/bazelbuild/bazel/labels/team-Rules-Java): Issues for Java rules + * Contact: [hvadehra](https://github.com/hvadehra) +* [`team-Rules-Python`](https://github.com/bazelbuild/bazel/labels/team-Rules-Python): Issues for the native Python rules + * Contact: [rickeylev](https://github.com/rickeylev) +* [`team-Rules-Server`](https://github.com/bazelbuild/bazel/labels/team-Rules-Server): Issues for server-side rules included with Bazel + * Contact: [comius](https://github.com/comius) +* [`team-Starlark-Integration`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Integration): Non-API Bazel + Starlark integration. Includes: how Bazel triggers the Starlark interpreter, Stardoc, builtins injection, character encoding. Does *not* include: BUILD or .bzl language issues. + * Contact: [brandjon](https://github.com/brandjon) +* [`team-Starlark-Interpreter`](https://github.com/bazelbuild/bazel/labels/team-Starlark-Interpreter): Issues for the Starlark interpreter (anything in [java.net.starlark](https://github.com/bazelbuild/bazel/tree/master/src/main/java/net/starlark/java)). BUILD and .bzl API issues (which represent Bazel's *integration* with Starlark) go in `team-Build-Language`. + * Contact: [brandjon](https://github.com/brandjon) + +For new issues, we deprecated the `category: *` labels in favor of the team +labels. See the full list of labels [here](https://github.com/bazelbuild/bazel/labels). diff --git a/contribute/naming.mdx b/contribute/naming.mdx index 806f7cd2..144b08af 100644 --- a/contribute/naming.mdx +++ b/contribute/naming.mdx @@ -2,42 +2,72 @@ title: 'Naming a Bazel related project' --- -First, thank you for contributing to the Bazel ecosystem! Please reach out to the Bazel community on the [bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss) to share your project and its suggested name. -If you are building a Bazel related tool or sharing your Skylark rules, we recommend following these guidelines for the name of your project: + +First, thank you for contributing to the Bazel ecosystem! Please reach out to +the Bazel community on the +[bazel-discuss mailing list](https://groups.google.com/forum/#!forum/bazel-discuss +) to share your project and its suggested name. + +If you are building a Bazel related tool or sharing your Skylark rules, +we recommend following these guidelines for the name of your project: ## Naming Starlark rules -See [Deploying new Starlark rules](/rules/deploying) in the docs. +See [Deploying new Starlark rules](/rules/deploying) +in the docs. ## Naming other Bazel related tools -This section applies if you are building a tool to enrich the Bazel ecosystem. For example, a new IDE plugin or a new build system migrator. +This section applies if you are building a tool to enrich the Bazel ecosystem. +For example, a new IDE plugin or a new build system migrator. -Picking a good name for your tool can be hard. If we’re not careful and use too many codenames, the Bazel ecosystem could become very difficult to understand for newcomers. +Picking a good name for your tool can be hard. If we’re not careful and use too +many codenames, the Bazel ecosystem could become very difficult to understand +for newcomers. Follow these guidelines for naming Bazel tools: -1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand for our users, we should avoid confusing them with too many new names. +1. Prefer **not introducing a new brand name**: "*Bazel*" is already a new brand +for our users, we should avoid confusing them with too many new names. + +2. Prefer **using a name that includes "Bazel"**: This helps to express that it +is a Bazel related tool, it also helps people find it with a search engine. -2. Prefer **using a name that includes "Bazel"**: This helps to express that it is a Bazel related tool, it also helps people find it with a search engine. +3. Prefer **using names that are descriptive about what the tool is doing**: +Ideally, the name should not need a subtitle for users to have a first good +guess at what the tool does. Using english words separated by spaces is a good +way to achieve this. -3. Prefer **using names that are descriptive about what the tool is doing**: Ideally, the name should not need a subtitle for users to have a first good guess at what the tool does. Using english words separated by spaces is a good way to achieve this. +4. **It is not a requirement to use a floral or food theme**: Bazel evokes +[basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to +look for a name that is a plant, food or that relates to "basil." -4. **It is not a requirement to use a floral or food theme**: Bazel evokes [basil](https://en.wikipedia.org/wiki/Basil), the plant. You do not need to look for a name that is a plant, food or that relates to "basil." +5. **If your tool relates to another third party brand, use it only as a +descriptor**: For example, use "Bazel migrator for Cmake" instead of +"Cmake Bazel migrator". -5. **If your tool relates to another third party brand, use it only as a descriptor**: For example, use "Bazel migrator for Cmake" instead of "Cmake Bazel migrator". +These guidelines also apply to the GitHub repository URL. Reading the repository +URL should help people understand what the tool does. Of course, the repository +name can be shorter and must use dashes instead of spaces and lower case letters. -These guidelines also apply to the GitHub repository URL. Reading the repository URL should help people understand what the tool does. Of course, the repository name can be shorter and must use dashes instead of spaces and lower case letters. Examples of good names: -- *Bazel for Eclipse*: Users will understand that if they want to use Bazel with Eclipse, this is where they should be looking. It uses a third party brand as a descriptor. -- *Bazel buildfarm*: A "buildfarm" is a [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users will understand that this project relates to building on servers. +* *Bazel for Eclipse*: Users will understand that if they want to use Bazel + with Eclipse, this is where they should be looking. It uses a third party brand + as a descriptor. +* *Bazel buildfarm*: A "buildfarm" is a + [compile farm](https://en.wikipedia.org/wiki/Compile_farm). Users + will understand that this project relates to building on servers. Examples of names to avoid: -- *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) does not relate enough to the Bazel project. -- *Bazelizer*: The tool behind this name could do a lot of things, this name is not descriptive enough. +* *Ocimum*: The [scientific name of basil](https://en.wikipedia.org/wiki/Ocimum) + does not relate enough to the Bazel project. +* *Bazelizer*: The tool behind this name could do a lot of things, this name is + not descriptive enough. -Note that these recommendations are aligned with the [guidelines](https://opensource.google.com/docs/releasing/preparing/#name) Google uses when open sourcing a project. +Note that these recommendations are aligned with the +[guidelines](https://opensource.google.com/docs/releasing/preparing/#name) +Google uses when open sourcing a project. diff --git a/contribute/patch-acceptance.mdx b/contribute/patch-acceptance.mdx index c877c554..87376afd 100644 --- a/contribute/patch-acceptance.mdx +++ b/contribute/patch-acceptance.mdx @@ -2,26 +2,51 @@ title: 'Patch Acceptance Process' --- -This page outlines how contributors can propose and make changes to the Bazel code base. -1. Read the [Bazel Contribution policy](/contribute/policy). - -2. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to discuss your plan and design. Pull requests that change or add behavior need a corresponding issue for tracking. - -3. If you're proposing significant changes, write a [design document](/contribute/design-documents). - -4. Ensure you've signed a [Contributor License Agreement](https://cla.developers.google.com). - -5. Prepare a git commit that implements the feature. Don't forget to add tests and update the documentation. If your change has user-visible effects, please [add release notes](/contribute/release-notes). If it is an incompatible change, read the [guide for rolling out breaking changes](/contribute/breaking-changes). -6. Create a pull request on [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, read [about pull requests](https://help.github.com/articles/about-pull-requests/). Note that we restrict permissions to create branches on the main Bazel repository, so you will need to push your commit to [your own fork of the repository](https://help.github.com/articles/working-with-forks/). +This page outlines how contributors can propose and make changes to the Bazel +code base. -7. A Bazel maintainer should assign you a reviewer within two business days (excluding holidays in the USA and Germany). If you aren't assigned a reviewer in that time, you can request one by emailing \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). - -8. Work with the reviewer to complete a code review. For each change, create a new commit and push it to make changes to your pull request. If the review takes too long (for instance, if the reviewer is unresponsive), send an email to \[[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)] (mailto:[bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com)). - -9. After your review is complete, a Bazel maintainer applies your patch to Google's internal version control system. - - This triggers internal presubmit checks that may suggest more changes. If you haven't expressed a preference, the maintainer submitting your change adds "trivial" changes (such as [linting](https://en.wikipedia.org/wiki/Lint_\(software\))) that don't affect design. If deeper changes are required or you'd prefer to apply changes directly, you and the reviewer should communicate preferences clearly in review comments. - - After internal submission, the patch is exported as a Git commit, at which point the GitHub pull request is closed. All final changes are attributed to you. +1. Read the [Bazel Contribution policy](/contribute/policy). +1. Create a [GitHub issue](https://github.com/bazelbuild/bazel/) to + discuss your plan and design. Pull requests that change or add behavior + need a corresponding issue for tracking. +1. If you're proposing significant changes, write a + [design document](/contribute/design-documents). +1. Ensure you've signed a [Contributor License + Agreement](https://cla.developers.google.com). +1. Prepare a git commit that implements the feature. Don't forget to add tests + and update the documentation. If your change has user-visible effects, please + [add release notes](/contribute/release-notes). If it is an incompatible change, + read the [guide for rolling out breaking changes](/contribute/breaking-changes). +1. Create a pull request on + [GitHub](https://github.com/bazelbuild/bazel/pulls). If you're new to GitHub, + read [about pull + requests](https://help.github.com/articles/about-pull-requests/). Note that + we restrict permissions to create branches on the main Bazel repository, so + you will need to push your commit to [your own fork of the + repository](https://help.github.com/articles/working-with-forks/). +1. A Bazel maintainer should assign you a reviewer within two business days + (excluding holidays in the USA and Germany). If you aren't assigned a + reviewer in that time, you can request one by emailing + [bazel-discuss@googlegroups.com] + (mailto:bazel-discuss@googlegroups.com). +1. Work with the reviewer to complete a code review. For each change, create a + new commit and push it to make changes to your pull request. If the review + takes too long (for instance, if the reviewer is unresponsive), send an email to + [bazel-discuss@googlegroups.com] + (mailto:bazel-discuss@googlegroups.com). +1. After your review is complete, a Bazel maintainer applies your patch to + Google's internal version control system. + + This triggers internal presubmit checks + that may suggest more changes. If you haven't expressed a preference, the + maintainer submitting your change adds "trivial" changes (such as + [linting](https://en.wikipedia.org/wiki/Lint_(software))) that don't affect + design. If deeper changes are required or you'd prefer to apply + changes directly, you and the reviewer should communicate preferences + clearly in review comments. + + After internal submission, the patch is exported as a Git commit, + at which point the GitHub pull request is closed. All final changes + are attributed to you. diff --git a/contribute/policy.mdx b/contribute/policy.mdx index 407f4c68..1bf00290 100644 --- a/contribute/policy.mdx +++ b/contribute/policy.mdx @@ -1,59 +1,78 @@ +translation: human +page_type: lcat --- title: 'Contribution policy' --- + + This page covers Bazel's governance model and contribution policy. ## Governance model -The [Bazel project](https://github.com/bazelbuild) is led and managed by Google and has a large community of contributors outside of Google. Some Bazel components (such as specific rules repositories under the [bazelbuild](https://github.com/bazelbuild) organization) are led, maintained, and managed by members of the community. The Google Bazel team reviews suggestions to add community-owned repositories (such as rules) to the [bazelbuild](https://github.com/bazelbuild) GitHub organization. +The [Bazel project](https://github.com/bazelbuild) is led and managed by Google +and has a large community of contributors outside of Google. Some Bazel +components (such as specific rules repositories under the +[bazelbuild](https://github.com/bazelbuild) organization) are led, +maintained, and managed by members of the community. The Google Bazel team +reviews suggestions to add community-owned repositories (such as rules) to the +[bazelbuild](https://github.com/bazelbuild) GitHub organization. ### Contributor roles -Here are outlines of the roles in the Bazel project, including their responsibilities: - -- **Owners**: The Google Bazel team. Owners are responsible for: - - - Strategy, maintenance, and leadership of the Bazel project. - - Building and maintaining Bazel's core functionality. - - Appointing Maintainers and approving new repositories. - -- **Maintainers**: The Google Bazel team and designated GitHub users. Maintainers are responsible for: - - - Building and maintaining the primary functionality of their repository. - - Reviewing and approving contributions to areas of the Bazel code base. - - Supporting users and contributors with timely and transparent issue management, PR review, and documentation. - - Releasing, testing and collaborating with Bazel Owners. - -- **Contributors**: All users who contribute code or documentation to the Bazel project. - - - Creating well-written PRs to contribute to Bazel's codebase and documentation. - - Using standard channels, such as GitHub Issues, to propose changes and report issues. +Here are outlines of the roles in the Bazel project, including their +responsibilities: + +* **Owners**: The Google Bazel team. Owners are responsible for: + * Strategy, maintenance, and leadership of the Bazel project. + * Building and maintaining Bazel's core functionality. + * Appointing Maintainers and approving new repositories. +* **Maintainers**: The Google Bazel team and designated GitHub users. + Maintainers are responsible for: + * Building and maintaining the primary functionality of their repository. + * Reviewing and approving contributions to areas of the Bazel code base. + * Supporting users and contributors with timely and transparent issue + management, PR review, and documentation. + * Releasing, testing and collaborating with Bazel Owners. +* **Contributors**: All users who contribute code or documentation to the + Bazel project. + * Creating well-written PRs to contribute to Bazel's codebase and + documentation. + * Using standard channels, such as GitHub Issues, to propose changes and + report issues. ### Becoming a Maintainer -Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as rule sets. Contributors with a record of consistent, responsible past contributions who are planning major contributions in the future could be considered to become qualified Maintainers. +Bazel Owners may appoint Maintainers to lead well-defined areas of code, such as +rule sets. Contributors with a record of consistent, responsible past +contributions who are planning major contributions in the future could be +considered to become qualified Maintainers. ## Contribution policy -The Bazel project accepts contributions from external contributors. Here are the contribution policies for Google-managed and Community-managed areas of code. - -- **Licensing**. All Maintainers and Contributors must sign the [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). - -- **Contributions**. Owners and Maintainers should make every effort to accept worthwhile contributions. All contributions must be: - - - Well written and well tested - - Discussed and approved by the Maintainers of the relevant area of code. Discussions and approvals happen on GitHub Issues and in GitHub PRs. Larger contributions require a [design review](/contribute/design-documents). - - Added to Bazel's Continuous Integration system if not already present. - - Supportable and aligned with Bazel product direction - -- **Code review**. All changes in all `bazelbuild` repositories require review: - - - All PRs must be approved by an Owner or Maintainer. - - Only Owners and Maintainers can merge PRs. - -- **Compatibility**. Owners may need to reject or request modifications to PRs in the unlikely event that the change requires substantial modifications to internal Google systems. - -- **Documentation**. Where relevant, feature contributions should include documentation updates. - -For more details on contributing to Bazel, see our [contribution guidelines](/contribute/). +The Bazel project accepts contributions from external contributors. Here are the +contribution policies for Google-managed and Community-managed areas of code. + +* **Licensing**. All Maintainers and Contributors must sign the + [Google’s Contributor License Agreement](https://cla.developers.google.com/clas). +* **Contributions**. Owners and Maintainers should make every effort to accept + worthwhile contributions. All contributions must be: + * Well written and well tested + * Discussed and approved by the Maintainers of the relevant area of code. + Discussions and approvals happen on GitHub Issues and in GitHub PRs. + Larger contributions require a + [design review](/contribute/design-documents). + * Added to Bazel's Continuous Integration system if not already present. + * Supportable and aligned with Bazel product direction +* **Code review**. All changes in all `bazelbuild` repositories require + review: + * All PRs must be approved by an Owner or Maintainer. + * Only Owners and Maintainers can merge PRs. +* **Compatibility**. Owners may need to reject or request modifications to PRs + in the unlikely event that the change requires substantial modifications to + internal Google systems. +* **Documentation**. Where relevant, feature contributions should include + documentation updates. + +For more details on contributing to Bazel, see our +[contribution guidelines](/contribute/). diff --git a/contribute/release-notes.mdx b/contribute/release-notes.mdx index bd0d9467..83e1d75b 100644 --- a/contribute/release-notes.mdx +++ b/contribute/release-notes.mdx @@ -2,44 +2,77 @@ title: 'Writing release notes' --- + + This document is targeted at Bazel contributors. -Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release note. This is used by the Bazel team to track changes in each release and write the release announcement. +Commit descriptions in Bazel include a `RELNOTES:` tag followed by a release +note. This is used by the Bazel team to track changes in each release and write +the release announcement. ## Overview -- Is your change a bugfix? In that case, you don't need a release note. Please include a reference to the GitHub issue. +* Is your change a bugfix? In that case, you don't need a release note. Please + include a reference to the GitHub issue. -- If the change adds / removes / changes Bazel in a user-visible way, then it may be advantageous to mention it. +* If the change adds / removes / changes Bazel in a user-visible way, then it + may be advantageous to mention it. -If the change is significant, follow the [design document policy](/contribute/design-documents) first. +If the change is significant, follow the [design document +policy](/contribute/design-documents) first. ## Guidelines -The release notes will be read by our users, so it should be short (ideally one sentence), avoid jargon (Bazel-internal terminology), should focus on what the change is about. +The release notes will be read by our users, so it should be short (ideally one +sentence), avoid jargon (Bazel-internal terminology), should focus on what the +change is about. -- Include a link to the relevant documentation. Almost any release note should contain a link. If the description mentions a flag, a feature, a command name, users will probably want to know more about it. +* Include a link to the relevant documentation. Almost any release note should + contain a link. If the description mentions a flag, a feature, a command name, + users will probably want to know more about it. -- Use backquotes around code, symbols, flags, or any word containing an underscore. +* Use backquotes around code, symbols, flags, or any word containing an + underscore. -- Do not just copy and paste bug descriptions. They are often cryptic and only make sense to us and leave the user scratching their head. Release notes are meant to explain what has changed and why in user-understandable language. +* Do not just copy and paste bug descriptions. They are often cryptic and only + make sense to us and leave the user scratching their head. Release notes are + meant to explain what has changed and why in user-understandable language. -- Always use present tense and the format "Bazel now supports Y" or "X now does Z." We don't want our release notes to sound like bug entries. All release note entries should be informative and use a consistent style and language. +* Always use present tense and the format "Bazel now supports Y" or "X now does + Z." We don't want our release notes to sound like bug entries. All release + note entries should be informative and use a consistent style and language. -- If something has been deprecated or removed, use "X has been deprecated" or "X has been removed." Not "is removed" or "was removed." +* If something has been deprecated or removed, use "X has been deprecated" or "X + has been removed." Not "is removed" or "was removed." -- If Bazel now does something differently, use "X now $newBehavior instead of $oldBehavior" in present tense. This lets the user know in detail what to expect when they use the new release. +* If Bazel now does something differently, use "X now $newBehavior instead of + $oldBehavior" in present tense. This lets the user know in detail what to + expect when they use the new release. -- If Bazel now supports or no longer supports something, use "Bazel now supports / no longer supports X". +* If Bazel now supports or no longer supports something, use "Bazel now supports + / no longer supports X". -- Explain why something has been removed / deprecated / changed. One sentence is enough but we want the user to be able to evaluate impact on their builds. +* Explain why something has been removed / deprecated / changed. One sentence is + enough but we want the user to be able to evaluate impact on their builds. -- Do NOT make any promises about future functionality. Avoid "this flag will be removed" or "this will be changed." It introduces uncertainty. The first thing the user will wonder is "when?" and we don't want them to start worrying about their current builds breaking at some unknown time. +* Do NOT make any promises about future functionality. Avoid "this flag will be + removed" or "this will be changed." It introduces uncertainty. The first thing + the user will wonder is "when?" and we don't want them to start worrying about + their current builds breaking at some unknown time. ## Process -As part of the [release process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), we collect the `RELNOTES` tags of every commit. We copy everything in a [Google Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) where we review, edit, and organize the notes. +As part of the [release +process](https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md), +we collect the `RELNOTES` tags of every commit. We copy everything in a [Google +Doc](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) +where we review, edit, and organize the notes. -The release manager sends an email to the [bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. Bazel contributors are invited to contribute to the document and make sure their changes are correctly reflected in the announcement. +The release manager sends an email to the +[bazel-dev](https://groups.google.com/forum/#!forum/bazel-dev) mailing-list. +Bazel contributors are invited to contribute to the document and make sure +their changes are correctly reflected in the announcement. -Later, the announcement will be submitted to the [Bazel blog](https://blog.bazel.build/), using the [bazel-blog repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). +Later, the announcement will be submitted to the [Bazel +blog](https://blog.bazel.build/), using the [bazel-blog +repository](https://github.com/bazelbuild/bazel-blog/tree/master/_posts). diff --git a/contribute/search.mdx b/contribute/search.mdx deleted file mode 100644 index 63f2526c..00000000 --- a/contribute/search.mdx +++ /dev/null @@ -1,167 +0,0 @@ ---- -title: 'Searching the codebase' ---- - -## Product overview - -Bazel's [code search and source browsing interface](https://source.bazel.build) is a web-based tool for browsing Bazel source code repositories. You can use these features to navigate among different repositories, branches, and files. You can also view history, diffs, and blame information. - -## Getting started - -Note: For the best experience, use the latest version of Chrome, Safari, or Firefox. - -To access the code search and source browsing interface, open [https://source.bazel.build](https://source.bazel.build) in your web browser. - -The main screen appears. This screen contains the following components: - -1. The Breadcrumb toolbar. This toolbar displays your current location in the repository and allows you to move quickly to another location such as another repository, or another location within a repository, such as a file, branch, or commit. - -2. A list of repositories that you can browse. - -At the top of the screen is a search box. You can use this box to search for specific files and code. - -## Working with repositories - -### Opening a repository - -To open a repository, click its name from the main screen. - -Alternatively, you can use the Breadcrumb toolbar to browse for a specificrepository. This toolbar displays your current location in the repository and allows you to move quickly to another location such as another repository, or another location within a repository, such as a file, branch, or commit. - -### Switch repositories - -To switch to a different repository, select the repository from the Breadcrumb toolbar. - -### View a repository at a specific commit - -To view a repository at a specific commit: - -1. From the view of the repository, select the file. -2. From the Breadcrumb toolbar, open the **Branch** menu. -3. In the submenu that appears, click **Commit**. -4. Select the commit you want to view. - -The interface now shows the repository as it existed at that commit. - -### Open a branch, commit, or tag - -By default, the code search and source browsing interface opens a repository to the default branch. To open a different branch, from the Breadcrumb toolbar, click the **Branch/Commit/Tag** menu. A submenu opens, allowing you to select a branch using a branch name, a tag name, or through a search box. - -- To select a branch using a branch name, select **Branch** and then click the name of the branch. -- To select a branch using a tag name, select **Tag** and then click the tag name. -- To select a branch using a commit id, select **Commit** and then click the commit id. -- To search for a branch, commit, or tag, select the corresponding item and type a search term in the search box. - -## Working with files - -When you select a repository from the main screen, the screen changes to display a view of that repository. If a README file exists, its contents appear in the file pane, located on the right side of the screen. Otherwise, a list of repository's files and folders appear. On the left side of the screen is a tree view of the repository's files and folders. You can use this tree to browse and open specific files. - -Notice that, when you are viewing a repository, the Breadcrumb toolbar now has three components: - -- A **Repository** menu, from which you can select different repositories -- A **Branch/Commit/Tag** menu, from which you can select specific branches, tags, or commits -- A **File path** box, which displays the name of the current file or folder and its corresponding path - -### Open a file - -You can open a file by browsing to its directory and selecting it. The view of the repository updates to show the contents of the file in the file pane, and its location in the repository in the tree pane. - -### View file changes - -To view file changes: - -1. From the view of the repository, select the file. -2. Click **BLAME**, located in the upper-right corner. - -The file pane updates to display who made changes to the file and when. - -### View change history - -To view the change history of a file: - -1. From the view of the repository, select the file. -2. Click **HISTORY**, located in the upper-right corner. The **Change history** pane appears, showing the commits for this file. - -### View code reviews - -For Gerrit code reviews, you can open the tool directly from the Change History pane. - -To view the code review for a file: - -1. From the view of the repository, select the file. -2. Click **HISTORY**, located in the upper-right corner. The Change History pane appears, showing the commits for this file. -3. Hover over a commit. A **More** button (three vertical dots) appears. -4. Click the **More** button. -5. Select **View code review**. - -The Gerrit Code Review tool opens in a new browser window. - -### Open a file at a specific commit - -To open a file at a specific commit: - -1. From the view of the repository, select the file. -2. Click **HISTORY**, located in the upper-right corner. The Change History pane appears, showing the commits for this file. -3. Hover over a commit. A **VIEW** button appears. -4. Click the **VIEW** button. - -### Compare a file to a different commit - -To compare a file at a different commit: - -1. From the view of the repository, select the file. To compare from two different commits, first open the file at that commit. -2. Hover over a commit. A **DIFF** button appears. -3. Click the **DIFF** button. - -The file pane updates to display a side-by-side comparison between the two files. The oldest of the two commits is always on the left. - -In the Change History pane, both commits are highlighted, and a label indicates if the commit is displayed on the left or the right. - -To change either file, hover over the commit in the Change History pane. Then, click either the **Left** or **Right** button to have the open the commit on the left or right side of the diff. - -### Browsing cross references - -Another way to browse source repositories is through the use of cross references. These references appear automatically as hyperlinks within a given source file. - -To make cross references easier to identify, click **Cross References**, located in the upper-right corner. This option displays an underline below all cross references in a file. - -**Note:** If **Cross References** is grayed out, it indicates that cross references are not available for that file. - -Click a cross reference to open the Cross Reference pane. This pane contains two sections: - -- A **Definition** section, which lists the file or files that define the reference -- A **References** section, which lists the files in which the reference also appears - -Both sections display the name of the file, as well as the line or lines that contains the reference. To open a file from the Cross Reference pane, click the line number entry. The file appears in a new section of the pane, allowing you to continue to browse the file while keeping the original file in view. - -You can continue to browse cross references using the Cross Reference pane, just as you can in the File pane. When you do, the pane displays a breadcrumb trail, which you can use to navigate between different cross references. - -## Searching for code - -You can search for specific files or code snippets using the search box located at the top of the screen. Searches are always against the default branch. - -All searches use [RE2 regular expressions](https://github.com/google/re2/wiki/Syntax) by default. If you do not want to use regular expressions, enclose your search in double quotes ( " ). - -**Note:** To quickly search for a specific file, either add a backslash in front of the period, or enclose the entire file name in quotes. - -``` -foo\.java -"foo.java" -``` - -You can refine your search using the following filters. - -| **Filter** | **Other options** | **Description** | **Example** | -| ---------- | ------------------ | ---------------------------------------------------------------------------- | -------------------- | -| lang: | language: | Perform an exact match by file language. | lang:java test | -| file: | filepath: path: f: | | | -| case:yes | | Make the search case sensitive. By default, searches are not case-sensitive. | case:yes Hello World | -| class: | | Search for a class name. | class:MainClass | -| function: | func: | Search for a function name. | function:print | -| - | | Negates the term from the search. | hello -world | -| \\ | | Escapes special characters, such as ., \\, or (. | run\\(\\) | -| "\[term]" | | Perform a literal search. | "class:main" | - -## Additional Support - -To report an issue, click the **Feedback** button that appears in the top right-hand corner of the screen and enter your feedback in the provided form. diff --git a/contribute/statemachine-guide.mdx b/contribute/statemachine-guide.mdx index 46993892..e98a96e8 100644 --- a/contribute/statemachine-guide.mdx +++ b/contribute/statemachine-guide.mdx @@ -1,28 +1,64 @@ --- -title: 'A Guide to Skyframe StateMachines' +title: 'A Guide to Skyframe `StateMachine`s' --- + + ## Overview -A Skyframe `StateMachine` is a *deconstructed* function-object that resides on the heap. It supports flexible and evaluation without redundancy[1](#user-content-fn-1) when required values are not immediately available but computed asynchronously. The `StateMachine` cannot tie up a thread resource while waiting, but instead has to be suspended and resumed. The deconstruction thus exposes explicit re-entry points so that prior computations can be skipped. +A Skyframe `StateMachine` is a *deconstructed* function-object that resides on +the heap. It supports flexible and evaluation without redundancy[^1] when +required values are not immediately available but computed asynchronously. The +`StateMachine` cannot tie up a thread resource while waiting, but instead has to +be suspended and resumed. The deconstruction thus exposes explicit re-entry +points so that prior computations can be skipped. -`StateMachine`s can be used to express sequences, branching, structured logical concurrency and are tailored specifically for Skyframe interaction. `StateMachine`s can be composed into larger `StateMachine`s and share sub-`StateMachine`s. Concurrency is always hierarchical by construction and purely logical. Every concurrent subtask runs in the single shared parent SkyFunction thread. +`StateMachine`s can be used to express sequences, branching, structured logical +concurrency and are tailored specifically for Skyframe interaction. +`StateMachine`s can be composed into larger `StateMachine`s and share +sub-`StateMachine`s. Concurrency is always hierarchical by construction and +purely logical. Every concurrent subtask runs in the single shared parent +SkyFunction thread. ## Introduction -This section briefly motivates and introduces `StateMachine`s, found in the [`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) package. +This section briefly motivates and introduces `StateMachine`s, found in the +[`java.com.google.devtools.build.skyframe.state`](https://github.com/bazelbuild/bazel/tree/master/src/main/java/com/google/devtools/build/skyframe/state) +package. ### A brief introduction to Skyframe restarts -Skyframe is a framework that performs parallel evaluation of dependency graphs. Each node in the graph corresponds with the evaluation of a SkyFunction with a SkyKey specifying its parameters and SkyValue specifying its result. The computational model is such that a SkyFunction may lookup SkyValues by SkyKey, triggering recursive, parallel evaluation of additional SkyFunctions. Instead of blocking, which would tie up a thread, when a requested SkyValue is not yet ready because some subgraph of computation is incomplete, the requesting SkyFunction observes a `null` `getValue` response and should return `null` instead of a SkyValue, signaling that it is incomplete due to missing inputs. Skyframe *restarts* the SkyFunctions when all previously requested SkyValues become available. - -Before the introduction of `SkyKeyComputeState`, the traditional way of handling a restart was to fully rerun the computation. Although this has quadratic complexity, functions written this way eventually complete because each rerun, fewer lookups return `null`. With `SkyKeyComputeState` it is possible to associate hand-specified check-point data with a SkyFunction, saving significant recomputation. - -`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate virtually all recomputation when a SkyFunction restarts (assuming that `SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume execution hooks. +Skyframe is a framework that performs parallel evaluation of dependency graphs. +Each node in the graph corresponds with the evaluation of a SkyFunction with a +SkyKey specifying its parameters and SkyValue specifying its result. The +computational model is such that a SkyFunction may lookup SkyValues by SkyKey, +triggering recursive, parallel evaluation of additional SkyFunctions. Instead of +blocking, which would tie up a thread, when a requested SkyValue is not yet +ready because some subgraph of computation is incomplete, the requesting +SkyFunction observes a `null` `getValue` response and should return `null` +instead of a SkyValue, signaling that it is incomplete due to missing inputs. +Skyframe *restarts* the SkyFunctions when all previously requested SkyValues +become available. + +Before the introduction of `SkyKeyComputeState`, the traditional way of handling +a restart was to fully rerun the computation. Although this has quadratic +complexity, functions written this way eventually complete because each rerun, +fewer lookups return `null`. With `SkyKeyComputeState` it is possible to +associate hand-specified check-point data with a SkyFunction, saving significant +recomputation. + +`StateMachine`s are objects that live inside `SkyKeyComputeState` and eliminate +virtually all recomputation when a SkyFunction restarts (assuming that +`SkyKeyComputeState` does not fall out of cache) by exposing suspend and resume +execution hooks. ### Stateful computations inside `SkyKeyComputeState` -From an object-oriented design standpoint, it makes sense to consider storing computational objects inside `SkyKeyComputeState` instead of pure data values. In *Java*, the bare minimum description of a behavior carrying object is a *functional interface* and it turns out to be sufficient. A `StateMachine` has the following, curiously recursive, definition[2](#user-content-fn-2). +From an object-oriented design standpoint, it makes sense to consider storing +computational objects inside `SkyKeyComputeState` instead of pure data values. +In *Java*, the bare minimum description of a behavior carrying object is a +*functional interface* and it turns out to be sufficient. A `StateMachine` has +the following, curiously recursive, definition[^2]. ``` @FunctionalInterface @@ -31,9 +67,12 @@ public interface StateMachine { } ``` -The `Tasks` interface is analogous to `SkyFunction.Environment` but it is designed for asynchrony and adds support for logically concurrent subtasks[3](#user-content-fn-3). +The `Tasks` interface is analogous to `SkyFunction.Environment` but it is +designed for asynchrony and adds support for logically concurrent subtasks[^3]. -The return value of `step` is another `StateMachine`, allowing the specification of a sequence of steps, inductively. `step` returns `DONE` when the `StateMachine` is done. For example: +The return value of `step` is another `StateMachine`, allowing the specification +of a sequence of steps, inductively. `step` returns `DONE` when the +`StateMachine` is done. For example: ``` class HelloWorld implements StateMachine { @@ -59,66 +98,98 @@ hello world ``` -Note that the method reference `this::step2` is also a `StateMachine` due to `step2` satisfying `StateMachine`'s functional interface definition. Method references are the most common way to specify the next state in a `StateMachine`. +Note that the method reference `this::step2` is also a `StateMachine` due to +`step2` satisfying `StateMachine`'s functional interface definition. Method +references are the most common way to specify the next state in a +`StateMachine`. ![Suspending and resuming](/contribute/images/suspend-resume.svg) -Intuitively, breaking a computation down into `StateMachine` steps, instead of a monolithic function, provides the hooks needed to *suspend* and *resume* a computation. When `StateMachine.step` returns, there is an explicit *suspension* point. The continuation specified by the returned `StateMachine` value is an explicit *resume* point. Recomputation can thus be avoided because the computation can be picked up exactly where it left off. +Intuitively, breaking a computation down into `StateMachine` steps, instead of a +monolithic function, provides the hooks needed to *suspend* and *resume* a +computation. When `StateMachine.step` returns, there is an explicit *suspension* +point. The continuation specified by the returned `StateMachine` value is an +explicit *resume* point. Recomputation can thus be avoided because the +computation can be picked up exactly where it left off. ### Callbacks, continuations and asynchronous computation -In technical terms, a `StateMachine` serves as a *continuation*, determining the subsequent computation to be executed. Instead of blocking, a `StateMachine` can voluntarily *suspend* by returning from the `step` function, which transfers control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can then switch to a ready `StateMachine` or relinquish control back to Skyframe. +In technical terms, a `StateMachine` serves as a *continuation*, determining the +subsequent computation to be executed. Instead of blocking, a `StateMachine` can +voluntarily *suspend* by returning from the `step` function, which transfers +control back to a [`Driver`](#drivers-and-bridging) instance. The `Driver` can +then switch to a ready `StateMachine` or relinquish control back to Skyframe. -Traditionally, *callbacks* and *continuations* are conflated into one concept. However, `StateMachine`s maintain a distinction between the two. +Traditionally, *callbacks* and *continuations* are conflated into one concept. +However, `StateMachine`s maintain a distinction between the two. -- *Callback* - describes where to store the result of an asynchronous computation. -- *Continuation* - specifies the next execution state. +* *Callback* - describes where to store the result of an asynchronous + computation. +* *Continuation* - specifies the next execution state. -Callbacks are required when invoking an asynchronous operation, which means that the actual operation doesn't occur immediately upon calling the method, as in the case of a SkyValue lookup. Callbacks should be kept as simple as possible. +Callbacks are required when invoking an asynchronous operation, which means that +the actual operation doesn't occur immediately upon calling the method, as in +the case of a SkyValue lookup. Callbacks should be kept as simple as possible. -Caution: A common pitfall of callbacks is that the asynchronous computation must ensure the callback is called by the end of every reachable path. It's possible to overlook some branches and the compiler doesn't give warnings about this. +Caution: A common pitfall of callbacks is that the asynchronous computation must +ensure the callback is called by the end of every reachable path. It's possible +to overlook some branches and the compiler doesn't give warnings about this. -*Continuations* are the `StateMachine` return values of `StateMachine`s and encapsulate the complex execution that follows once all asynchronous computations resolve. This structured approach helps to keep the complexity of callbacks manageable. +*Continuations* are the `StateMachine` return values of `StateMachine`s and +encapsulate the complex execution that follows once all asynchronous +computations resolve. This structured approach helps to keep the complexity of +callbacks manageable. ## Tasks -The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues by SkyKey and to schedule concurrent subtasks. +The `Tasks` interface provides `StateMachine`s with an API to lookup SkyValues +by SkyKey and to schedule concurrent subtasks. ``` interface Tasks { void enqueue(StateMachine subtask); - void lookUp(SkyKey key, Consumer<SkyValue> sink); + void lookUp(SkyKey key, Consumer sink); - <E extends Exception> - void lookUp(SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); + + void lookUp(SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); // lookUp overloads for 2 and 3 exception types exist, but are elided here. } ``` -Tip: When any state uses the `Tasks` interface to perform lookups or create subtasks, those lookups and subtasks will complete before the next state begins. +Tip: When any state uses the `Tasks` interface to perform lookups or create +subtasks, those lookups and subtasks will complete before the next state begins. -Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create subtasks, they all *transitively* complete before the next state begins. +Tip: (Corollary) If subtasks are complex `StateMachine`s or recursively create +subtasks, they all *transitively* complete before the next state begins. ### SkyValue lookups -`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are analogous to `SkyFunction.Environment.getValue` and `SkyFunction.Environment.getValueOrThrow` and have similar exception handling semantics. The implementation does not immediately perform the lookup, but instead, batches[4](#user-content-fn-4) as many lookups as possible before doing so. The value might not be immediately available, for example, requiring a Skyframe restart, so the caller specifies what to do with the resulting value using a callback. +`StateMachine`s use `Tasks.lookUp` overloads to look up SkyValues. They are +analogous to `SkyFunction.Environment.getValue` and +`SkyFunction.Environment.getValueOrThrow` and have similar exception handling +semantics. The implementation does not immediately perform the lookup, but +instead, batches[^4] as many lookups as possible before doing so. The value +might not be immediately available, for example, requiring a Skyframe restart, +so the caller specifies what to do with the resulting value using a callback. -The `StateMachine` processor ([`Driver`s and bridging to SkyFrame](#drivers-and-bridging)) guarantees that the value is available before the next state begins. An example follows. +The `StateMachine` processor ([`Driver`s and bridging to +SkyFrame](#drivers-and-bridging)) guarantees that the value is available before +the next state begins. An example follows. ``` -class DoesLookup implements StateMachine, Consumer<SkyValue> { +class DoesLookup implements StateMachine, Consumer { private Value value; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new Key(), (Consumer<SkyValue>) this); + tasks.lookUp(new Key(), (Consumer) this); return this::processValue; } // The `lookUp` call in `step` causes this to be called before `processValue`. - @Override // Implementation of Consumer<SkyValue>. + @Override // Implementation of Consumer. public void accept(SkyValue value) { this.value = (Value)value; } @@ -130,15 +201,25 @@ class DoesLookup implements StateMachine, Consumer<SkyValue> { } ``` -In the above example, the first step does a lookup for `new Key()`, passing `this` as the consumer. That is possible because `DoesLookup` implements `Consumer<SkyValue>`. +In the above example, the first step does a lookup for `new Key()`, passing +`this` as the consumer. That is possible because `DoesLookup` implements +`Consumer`. -Tip: When passing `this` as a value sink, it's helpful to readers to upcast it to the receiver type to narrow down the purpose of passing `this`. The example passes `(Consumer<SkyValue>) this`. +Tip: When passing `this` as a value sink, it's helpful to readers to upcast it +to the receiver type to narrow down the purpose of passing `this`. The example +passes `(Consumer) this`. -By contract, before the next state `DoesLookup.processValue` begins, all the lookups of `DoesLookup.step` are complete. Therefore `value` is available when it is accessed in `processValue`. +By contract, before the next state `DoesLookup.processValue` begins, all the +lookups of `DoesLookup.step` are complete. Therefore `value` is available when +it is accessed in `processValue`. ### Subtasks -`Tasks.enqueue` requests the execution of logically concurrent subtasks. Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s can do, including recursively creating more subtasks or looking up SkyValues. Much like `lookUp`, the state machine driver ensures that all subtasks are complete before proceeding to the next step. An example follows. +`Tasks.enqueue` requests the execution of logically concurrent subtasks. +Subtasks are also `StateMachine`s and can do anything regular `StateMachine`s +can do, including recursively creating more subtasks or looking up SkyValues. +Much like `lookUp`, the state machine driver ensures that all subtasks are +complete before proceeding to the next step. An example follows. ``` class Subtasks implements StateMachine { @@ -176,15 +257,22 @@ class Subtasks implements StateMachine { } ``` -Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a single thread so the "concurrent" update of `i` does not need any synchronization. +Though `Subtask1` and `Subtask2` are logically concurrent, everything runs in a +single thread so the "concurrent" update of `i` does not need any +synchronization. ### Structured concurrency -Since every `lookUp` and `enqueue` must resolve before advancing to the next state, it means that concurrency is naturally limited to tree-structures. It's possible to create hierarchical[5](#user-content-fn-5) concurrency as shown in the following example. +Since every `lookUp` and `enqueue` must resolve before advancing to the next +state, it means that concurrency is naturally limited to tree-structures. It's +possible to create hierarchical[^5] concurrency as shown in the following +example. ![Structured Concurrency](/contribute/images/structured-concurrency.svg) -It's hard to tell from the *UML* that the concurrency structure forms a tree. There's an [alternate view](#concurrency-tree-diagram) that better shows the tree structure. +It's hard to tell from the *UML* that the concurrency structure forms a tree. +There's an [alternate view](#concurrency-tree-diagram) that better shows the +tree structure. ![Unstructured Concurrency](/contribute/images/unstructured-concurrency.svg) @@ -192,15 +280,19 @@ Structured concurrency is much easier to reason about. ## Composition and control flow patterns -This section presents examples for how multiple `StateMachine`s can be composed and solutions to certain control flow problems. +This section presents examples for how multiple `StateMachine`s can be composed +and solutions to certain control flow problems. ### Sequential states -This is the most common and straightforward control flow pattern. An example of this is shown in [Stateful computations inside `SkyKeyComputeState`](#stateful-computations). +This is the most common and straightforward control flow pattern. An example of +this is shown in [Stateful computations inside +`SkyKeyComputeState`](#stateful-computations). ### Branching -Branching states in `StateMachine`s can be achieved by returning different values using regular *Java* control flow, as shown in the following example. +Branching states in `StateMachine`s can be achieved by returning different +values using regular *Java* control flow, as shown in the following example. ``` class Branch implements StateMachine { @@ -220,11 +312,18 @@ It’s very common for certain branches to return `DONE`, for early completion. ### Advanced sequential composition -Since the `StateMachine` control structure is memoryless, sharing `StateMachine` definitions as subtasks can sometimes be awkward. Let *M1* and *M2* be `StateMachine` instances that share a `StateMachine`, *S*, with *M1* and *M2* being the sequences *\* and *\* respectively. The problem is that *S* doesn’t know whether to continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a call stack. This section reviews some techniques for achieving this. +Since the `StateMachine` control structure is memoryless, sharing `StateMachine` +definitions as subtasks can sometimes be awkward. Let *M1* and +*M2* be `StateMachine` instances that share a `StateMachine`, *S*, +with *M1* and *M2* being the sequences *<A, S, B>* and +*<X, S, Y>* respectively. The problem is that *S* doesn’t know whether to +continue to *B* or *Y* after it completes and `StateMachine`s don't quite keep a +call stack. This section reviews some techniques for achieving this. #### `StateMachine` as terminal sequence element -This doesn’t solve the initial problem posed. It only demonstrates sequential composition when the shared `StateMachine` is terminal in the sequence. +This doesn’t solve the initial problem posed. It only demonstrates sequential +composition when the shared `StateMachine` is terminal in the sequence. ``` // S is the shared state machine. @@ -251,7 +350,8 @@ This works even if *S* is itself a complex state machine. #### Subtask for sequential composition -Since enqueued subtasks are guaranteed to complete before the next state, it’s sometimes possible to slightly abuse[6](#user-content-fn-6) the subtask mechanism. +Since enqueued subtasks are guaranteed to complete before the next state, it’s +sometimes possible to slightly abuse[^6] the subtask mechanism. ``` class M1 implements StateMachine { @@ -259,7 +359,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // S starts after `step` returns and by contract must complete before `doB` - // begins. It is effectively sequential, inducing the sequence < A, S, B >. + // begins. It is effectively sequential, inducing the sequence < A, S, B >. tasks.enqueue(new S()); return this::doB; } @@ -274,7 +374,7 @@ class M2 implements StateMachine { @Override public StateMachine step(Tasks tasks) { performX(); - // Similarly, this induces the sequence < X, S, Y>. + // Similarly, this induces the sequence < X, S, Y>. tasks.enqueue(new S()); return this::doY; } @@ -288,7 +388,10 @@ class M2 implements StateMachine { #### `runAfter` injection -Sometimes, abusing `Tasks.enqueue` is impossible because there are other parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* executes. In this case, injecting a `runAfter` parameter into *S* can be used to inform *S* of what to do next. +Sometimes, abusing `Tasks.enqueue` is impossible because there are other +parallel subtasks or `Tasks.lookUp` calls that must be completed before *S* +executes. In this case, injecting a `runAfter` parameter into *S* can be used to +inform *S* of what to do next. ``` class S implements StateMachine { @@ -315,7 +418,7 @@ class M1 implements StateMachine { public StateMachine step(Tasks tasks) { performA(); // Passes `this::doB` as the `runAfter` parameter of S, resulting in the - // sequence < A, S, B >. + // sequence < A, S, B >. return new S(/* runAfter= */ this::doB); } @@ -330,7 +433,7 @@ class M2 implements StateMachine { public StateMachine step(Tasks tasks) { performX(); // Passes `this::doY` as the `runAfter` parameter of S, resulting in the - // sequence < X, S, Y >. + // sequence < X, S, Y >. return new S(/* runAfter= */ this::doY); } @@ -341,7 +444,10 @@ class M2 implements StateMachine { } ``` -This approach is cleaner than abusing subtasks. However, applying this too liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is the road to [Callback Hell](#callback-hell). It’s better to break up sequential `runAfter`s with ordinary sequential states instead. +This approach is cleaner than abusing subtasks. However, applying this too +liberally, for example, by nesting multiple `StateMachine`s with `runAfter`, is +the road to [Callback Hell](#callback-hell). It’s better to break up sequential +`runAfter`s with ordinary sequential states instead. ``` return new S(/* runAfter= */ new T(/* runAfter= */ this::nextStep)) @@ -360,21 +466,37 @@ can be replaced with the following. } ``` -Note: It's possible to pass `DONE` as the `runAfter` parameter when there's nothing to run afterwards. +Note: It's possible to pass `DONE` as the `runAfter` parameter when there's +nothing to run afterwards. -Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` to let the reader know the meaning at the callsite. +Tip: When using `runAfter`, always annotate the parameter with `/* runAfter= */` +to let the reader know the meaning at the callsite. #### *Forbidden* alternative: `runAfterUnlessError` -In an earlier draft, we had considered a `runAfterUnlessError` that would abort early on errors. This was motivated by the fact that errors often end up getting checked twice, once by the `StateMachine` that has a `runAfter` reference and once by the `runAfter` machine itself. +In an earlier draft, we had considered a `runAfterUnlessError` that would abort +early on errors. This was motivated by the fact that errors often end up getting +checked twice, once by the `StateMachine` that has a `runAfter` reference and +once by the `runAfter` machine itself. -After some deliberation, we decided that uniformity of the code is more important than deduplicating the error checking. It would be confusing if the `runAfter` mechanism did not work in a consistent manner with the `tasks.enqueue` mechanism, which always requires error checking. +After some deliberation, we decided that uniformity of the code is more +important than deduplicating the error checking. It would be confusing if the +`runAfter` mechanism did not work in a consistent manner with the +`tasks.enqueue` mechanism, which always requires error checking. -Warning: When using `runAfter`, the machine that has the injected `runAfter` should invoke it unconditionally at completion, even on error, for consistency. +Warning: When using `runAfter`, the machine that has the injected `runAfter` +should invoke it unconditionally at completion, even on error, for consistency. ### Direct delegation -Each time there is a formal state transition, the main `Driver` loop advances. As per contract, advancing states means that all previously enqueued SkyValue lookups and subtasks resolve before the next state executes. Sometimes the logic of a delegate `StateMachine` makes a phase advance unnecessary or counterproductive. For example, if the first `step` of the delegate performs SkyKey lookups that could be parallelized with lookups of the delegating state then a phase advance would make them sequential. It could make more sense to perform direct delegation, as shown in the example below. +Each time there is a formal state transition, the main `Driver` loop advances. +As per contract, advancing states means that all previously enqueued SkyValue +lookups and subtasks resolve before the next state executes. Sometimes the logic +of a delegate `StateMachine` makes a phase advance unnecessary or +counterproductive. For example, if the first `step` of the delegate performs +SkyKey lookups that could be parallelized with lookups of the delegating state +then a phase advance would make them sequential. It could make more sense to +perform direct delegation, as shown in the example below. ``` class Parent implements StateMachine { @@ -420,37 +542,49 @@ class Delegate implements StateMachine { ## Data flow -The focus of the previous discussion has been on managing control flow. This section describes the propagation of data values. +The focus of the previous discussion has been on managing control flow. This +section describes the propagation of data values. ### Implementing `Tasks.lookUp` callbacks -There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue lookups](#skyvalue-lookups). This section provides rationale and suggests approaches for handling multiple SkyValues. +There’s an example of implementing a `Tasks.lookUp` callback in [SkyValue +lookups](#skyvalue-lookups). This section provides rationale and suggests +approaches for handling multiple SkyValues. #### `Tasks.lookUp` callbacks The `Tasks.lookUp` method takes a callback, `sink`, as a parameter. ``` - void lookUp(SkyKey key, Consumer<SkyValue> sink); + void lookUp(SkyKey key, Consumer sink); ``` The idiomatic approach would be to use a *Java* lambda to implement this: ``` - tasks.lookUp(key, value -> myValue = (MyValueClass)value); + tasks.lookUp(key, value -> myValue = (MyValueClass)value); ``` -with `myValue` being a member variable of the `StateMachine` instance doing the lookup. However, the lambda requires an extra memory allocation compared to implementing the `Consumer<SkyValue>` interface in the `StateMachine` implementation. The lambda is still useful when there are multiple lookups that would be ambiguous. +with `myValue` being a member variable of the `StateMachine` instance doing the +lookup. However, the lambda requires an extra memory allocation compared to +implementing the `Consumer` interface in the `StateMachine` +implementation. The lambda is still useful when there are multiple lookups that +would be ambiguous. -Note: Bikeshed warning. There is a noticeable difference of approximately 1% end-to-end CPU usage when implementing callbacks systematically in `StateMachine` implementations compared to using lambdas, which makes this recommendation debatable. To avoid unnecessary debates, it is advised to leave the decision up to the individual implementing the solution. +Note: Bikeshed warning. There is a noticeable difference of approximately 1% +end-to-end CPU usage when implementing callbacks systematically in +`StateMachine` implementations compared to using lambdas, which makes this +recommendation debatable. To avoid unnecessary debates, it is advised to leave +the decision up to the individual implementing the solution. -There are also error handling overloads of `Tasks.lookUp`, that are analogous to `SkyFunction.Environment.getValueOrThrow`. +There are also error handling overloads of `Tasks.lookUp`, that are analogous to +`SkyFunction.Environment.getValueOrThrow`. ``` - <E extends Exception> void lookUp( - SkyKey key, Class<E> exceptionClass, ValueOrExceptionSink<E> sink); + void lookUp( + SkyKey key, Class exceptionClass, ValueOrExceptionSink sink); - interface ValueOrExceptionSink<E extends Exception> { + interface ValueOrExceptionSink { void acceptValueOrException(@Nullable SkyValue value, @Nullable E exception); } ``` @@ -458,13 +592,13 @@ There are also error handling overloads of `Tasks.lookUp`, that are analogous to An example implementation is shown below. ``` -class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyException> { +class PerformLookupWithError extends StateMachine, ValueOrExceptionSink { private MyValue value; private MyException error; @Override public StateMachine step(Tasks tasks) { - tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink<MyException>) this); + tasks.lookUp(new MyKey(), MyException.class, ValueOrExceptionSink) this); return this::processResult; } @@ -493,29 +627,33 @@ class PerformLookupWithError extends StateMachine, ValueOrExceptionSink<MyExc } ``` -As with lookups without error handling, having the `StateMachine` class directly implement the callback saves a memory allocation for the lamba. +As with lookups without error handling, having the `StateMachine` class directly +implement the callback saves a memory allocation for the lamba. -[Error handling](#error-handling) provides a bit more detail, but essentially, there's not much difference between the propagation of errors and normal values. +[Error handling](#error-handling) provides a bit more detail, but essentially, +there's not much difference between the propagation of errors and normal values. #### Consuming multiple SkyValues -Multiple SkyValue lookups are often required. An approach that works much of the time is to switch on the type of SkyValue. The following is an example that has been simplified from prototype production code. +Multiple SkyValue lookups are often required. An approach that works much of the +time is to switch on the type of SkyValue. The following is an example that has +been simplified from prototype production code. ``` @Nullable private StateMachine fetchConfigurationAndPackage(Tasks tasks) { var configurationKey = configuredTarget.getConfigurationKey(); if (configurationKey != null) { - tasks.lookUp(configurationKey, (Consumer<SkyValue>) this); + tasks.lookUp(configurationKey, (Consumer) this); } var packageId = configuredTarget.getLabel().getPackageIdentifier(); - tasks.lookUp(PackageValue.key(packageId), (Consumer<SkyValue>) this); + tasks.lookUp(PackageValue.key(packageId), (Consumer) this); return this::constructResult; } - @Override // Implementation of `Consumer<SkyValue>`. + @Override // Implementation of `Consumer`. public void accept(SkyValue value) { if (value instanceof BuildConfigurationValue) { this.configurationValue = (BuildConfigurationValue) value; @@ -529,11 +667,18 @@ Multiple SkyValue lookups are often required. An approach that works much of the } ``` -The `Consumer<SkyValue>` callback implementation can be shared unambiguously because the value types are different. When that’s not the case, falling back to lambda-based implementations or full inner-class instances that implement the appropriate callbacks is viable. +The `Consumer` callback implementation can be shared unambiguously +because the value types are different. When that’s not the case, falling back to +lambda-based implementations or full inner-class instances that implement the +appropriate callbacks is viable. ### Propagating values between `StateMachine`s -So far, this document has only explained how to arrange work in a subtask, but subtasks also need to report a values back to the caller. Since subtasks are logically asynchronous, their results are communicated back to the caller using a *callback*. To make this work, the subtask defines a sink interface that is injected via its constructor. +So far, this document has only explained how to arrange work in a subtask, but +subtasks also need to report a values back to the caller. Since subtasks are +logically asynchronous, their results are communicated back to the caller using +a *callback*. To make this work, the subtask defines a sink interface that is +injected via its constructor. ``` class BarProducer implements StateMachine { @@ -564,9 +709,16 @@ class BarProducer implements StateMachine { } ``` -Tip: It would be tempting to use the more concise signature void `accept(Bar value)` rather than the stuttery `void acceptBarValue(Bar value)` above. However, `Consumer<SkyValue>` is a common overload of `void accept(Bar value)`, so doing this often leads to violations of the [Overloads: never split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) style-guide rule. +Tip: It would be tempting to use the more concise signature void `accept(Bar +value)` rather than the stuttery `void acceptBarValue(Bar value)` above. +However, `Consumer` is a common overload of `void accept(Bar value)`, +so doing this often leads to violations of the [Overloads: never +split](https://google.github.io/styleguide/javaguide.html#s3.4.2-ordering-class-contents) +style-guide rule. -Tip: Using a custom `ResultSink` type instead of a generic one from `java.util.function` makes it easy to find implementations in the code base, improving readability. +Tip: Using a custom `ResultSink` type instead of a generic one from +`java.util.function` makes it easy to find implementations in the code base, +improving readability. A caller `StateMachine` would then look like the following. @@ -615,37 +767,70 @@ class Caller implements StateMachine, BarProducer.ResultSink { } ``` -The preceding example demonstrates a few things. `Caller` has to propagate its results back and defines its own `Caller.ResultSink`. `Caller` implements the `BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if `value` is null to determine if an error occurred. This is a common behavior pattern after accepting output from either a subtask or SkyValue lookup. +The preceding example demonstrates a few things. `Caller` has to propagate its +results back and defines its own `Caller.ResultSink`. `Caller` implements the +`BarProducer.ResultSink` callbacks. Upon resumption, `processResult` checks if +`value` is null to determine if an error occurred. This is a common behavior +pattern after accepting output from either a subtask or SkyValue lookup. -Note that the implementation of `acceptBarError` eagerly forwards the result to the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). +Note that the implementation of `acceptBarError` eagerly forwards the result to +the `Caller.ResultSink`, as required by [Error bubbling](#error-bubbling). -Alternatives for top-level `StateMachine`s are described in [`Driver`s and bridging to SkyFunctions](#drivers-and-bridging). +Alternatives for top-level `StateMachine`s are described in [`Driver`s and +bridging to SkyFunctions](#drivers-and-bridging). ### Error handling -There's a couple of examples of error handling already in [`Tasks.lookUp` callbacks](#tasks-lookup-callbacks) and [Propagating values between `StateMachines`](#propagating-values). Exceptions, other than `InterruptedException` are not thrown, but instead passed around through callbacks as values. Such callbacks often have exclusive-or semantics, with exactly one of a value or error being passed. +There's a couple of examples of error handling already in [`Tasks.lookUp` +callbacks](#tasks-lookup-callbacks) and [Propagating values between +`StateMachines`](#propagating-values). Exceptions, other than +`InterruptedException` are not thrown, but instead passed around through +callbacks as values. Such callbacks often have exclusive-or semantics, with +exactly one of a value or error being passed. -The next section describes a a subtle, but important interaction with Skyframe error handling. +The next section describes a a subtle, but important interaction with Skyframe +error handling. #### Error bubbling (--nokeep\_going) -Warning: Errors need to be eagerly propagated all the way back to the SkyFunction for error bubbling to function correctly. +Warning: Errors need to be eagerly propagated all the way back to the +SkyFunction for error bubbling to function correctly. -During error bubbling, a SkyFunction may be restarted even if not all requested SkyValues are available. In such cases, the subsequent state will never be reached due to the `Tasks` API contract. However, the `StateMachine` should still propagate the exception. +During error bubbling, a SkyFunction may be restarted even if not all requested +SkyValues are available. In such cases, the subsequent state will never be +reached due to the `Tasks` API contract. However, the `StateMachine` should +still propagate the exception. -Since propagation must occur regardless of whether the next state is reached, the error handling callback must perform this task. For an inner `StateMachine`, this is achieved by invoking the parent callback. +Since propagation must occur regardless of whether the next state is reached, +the error handling callback must perform this task. For an inner `StateMachine`, +this is achieved by invoking the parent callback. -At the top-level `StateMachine`, which interfaces with the SkyFunction, this can be done by calling the `setException` method of `ValueOrExceptionProducer`. `ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even if there are missing SkyValues. +At the top-level `StateMachine`, which interfaces with the SkyFunction, this can +be done by calling the `setException` method of `ValueOrExceptionProducer`. +`ValueOrExceptionProducer.tryProduceValue` will then throw the exception, even +if there are missing SkyValues. -If a `Driver` is being utilized directly, it is essential to check for propagated errors from the SkyFunction, even if the machine has not finished processing. +If a `Driver` is being utilized directly, it is essential to check for +propagated errors from the SkyFunction, even if the machine has not finished +processing. ### Event Handling -For SkyFunctions that need to emit events, a `StoredEventHandler` is injected into SkyKeyComputeState and further injected into `StateMachine`s that require them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping certain events unless they are replayed but this was subsequently fixed. `StoredEventHandler` injection is preserved because it simplifies the implementation of events emitted from error handling callbacks. +For SkyFunctions that need to emit events, a `StoredEventHandler` is injected +into SkyKeyComputeState and further injected into `StateMachine`s that require +them. Historically, the `StoredEventHandler` was needed due to Skyframe dropping +certain events unless they are replayed but this was subsequently fixed. +`StoredEventHandler` injection is preserved because it simplifies the +implementation of events emitted from error handling callbacks. ## `Driver`s and bridging to SkyFunctions -A `Driver` is responsible for managing the execution of `StateMachine`s, beginning with a specified root `StateMachine`. As `StateMachine`s can recursively enqueue subtask `StateMachine`s, a single `Driver` can manage numerous subtasks. These subtasks create a tree structure, a result of [Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue lookups across subtasks for improved efficiency. +A `Driver` is responsible for managing the execution of `StateMachine`s, +beginning with a specified root `StateMachine`. As `StateMachine`s can +recursively enqueue subtask `StateMachine`s, a single `Driver` can manage +numerous subtasks. These subtasks create a tree structure, a result of +[Structured concurrency](#structured-concurrency). The `Driver` batches SkyValue +lookups across subtasks for improved efficiency. There are a number of classes built around the `Driver`, with the following API. @@ -656,15 +841,24 @@ public final class Driver { } ``` -`Driver` takes a single root `StateMachine` as a parameter. Calling `Driver.drive` executes the `StateMachine` as far as it can go without a Skyframe restart. It returns true when the `StateMachine` completes and false otherwise, indicating that not all values were available. +`Driver` takes a single root `StateMachine` as a parameter. Calling +`Driver.drive` executes the `StateMachine` as far as it can go without a +Skyframe restart. It returns true when the `StateMachine` completes and false +otherwise, indicating that not all values were available. -`Driver` maintains the concurrent state of the `StateMachine` and it is well suited for embedding in `SkyKeyComputeState`. +`Driver` maintains the concurrent state of the `StateMachine` and it is well +suited for embedding in `SkyKeyComputeState`. ### Directly instantiating `Driver` -`StateMachine` implementations conventionally communicate their results via callbacks. It's possible to directly instantiate a `Driver` as shown in the following example. +`StateMachine` implementations conventionally communicate their results via +callbacks. It's possible to directly instantiate a `Driver` as shown in the +following example. -The `Driver` is embedded in the `SkyKeyComputeState` implementation along with an implementation of the corresponding `ResultSink` to be defined a bit further down. At the top level, the `State` object is an appropriate receiver for the result of the computation as it is guaranteed to outlive `Driver`. +The `Driver` is embedded in the `SkyKeyComputeState` implementation along with +an implementation of the corresponding `ResultSink` to be defined a bit further +down. At the top level, the `State` object is an appropriate receiver for the +result of the computation as it is guaranteed to outlive `Driver`. ``` class State implements SkyKeyComputeState, ResultProducer.ResultSink { @@ -746,7 +940,8 @@ private Result computeResult(State state, Skyfunction.Environment env) ### Embedding `Driver` -If the `StateMachine` produces a value and raises no exceptions, embedding `Driver` is another possible implementation, as shown in the following example. +If the `StateMachine` produces a value and raises no exceptions, embedding +`Driver` is another possible implementation, as shown in the following example. ``` class ResultProducer implements StateMachine { @@ -775,7 +970,8 @@ class ResultProducer implements StateMachine { } ``` -The SkyFunction may have code that looks like the following (where `State` is the function specific type of `SkyKeyComputeState`). +The SkyFunction may have code that looks like the following (where `State` is +the function specific type of `SkyKeyComputeState`). ``` @Nullable // Null when a Skyframe restart is needed. @@ -796,16 +992,19 @@ Result computeResult(SkyFunction.Environment env, State state) } ``` -Embedding `Driver` in the `StateMachine` implementation is a better fit for Skyframe's synchronous coding style. +Embedding `Driver` in the `StateMachine` implementation is a better fit for +Skyframe's synchronous coding style. ### StateMachines that may produce exceptions -Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` and `ValueOrException2Producer` classes that have synchronous APIs to match synchronous SkyFunction code. +Otherwise, there are `SkyKeyComputeState`-embeddable `ValueOrExceptionProducer` +and `ValueOrException2Producer` classes that have synchronous APIs to match +synchronous SkyFunction code. The `ValueOrExceptionProducer` abstract class includes the following methods. ``` -public abstract class ValueOrExceptionProducer<V, E extends Exception> +public abstract class ValueOrExceptionProducer implements StateMachine { @Nullable public final V tryProduceValue(Environment env) @@ -818,34 +1017,65 @@ public abstract class ValueOrExceptionProducer<V, E extends Exception> } ``` -It includes an embedded `Driver` instance and closely resembles the `ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, implementations call `setValue` or `setException` when either of those occur. When both occur, the exception takes priority. The `tryProduceValue` method bridges the asynchronous callback code to synchronous code and throws an exception when one is set. +It includes an embedded `Driver` instance and closely resembles the +`ResultProducer` class in [Embedding driver](#embedding-driver) and interfaces +with the SkyFunction in a similar manner. Instead of defining a `ResultSink`, +implementations call `setValue` or `setException` when either of those occur. +When both occur, the exception takes priority. The `tryProduceValue` method +bridges the asynchronous callback code to synchronous code and throws an +exception when one is set. -As previously noted, during error bubbling, it's possible for an error to occur even if the machine is not yet done because not all inputs are available. To accommodate this, `tryProduceValue` throws any set exceptions, even before the machine is done. +As previously noted, during error bubbling, it's possible for an error to occur +even if the machine is not yet done because not all inputs are available. To +accommodate this, `tryProduceValue` throws any set exceptions, even before the +machine is done. ## Epilogue: Eventually removing callbacks -`StateMachine`s are a highly efficient, but boilerplate intensive way to perform asynchronous computation. Continuations (particularly in the form of `Runnable`s passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and there are no efficient asynchronous APIs for disk I/O. Eventually, it would be good to optimize away callbacks as they have a learning curve and impede readability. - -One of the most promising alternatives is *Java* virtual threads. Instead of having to write callbacks, everything is replaced with synchronous, blocking calls. This is possible because tying up a virtual thread resource, unlike a platform thread, is supposed to be cheap. However, even with virtual threads, replacing simple synchronous operations with thread creation and synchronization primitives is too expensive. We performed a migration from `StateMachine`s to *Java* virtual threads and they were orders of magnitude slower, leading to almost a 3x increase in end-to-end analysis latency. Since virtual threads are still a preview feature, it's possible that this migration can be performed at a later date when performance improves. - -Another approach to consider is waiting for *Loom* coroutines, if they ever become available. The advantage here is that it might be possible to reduce synchronization overhead by using cooperative multitasking. - -If all else fails, low-level bytecode rewriting could also be a viable alternative. With enough optimization, it might be possible to achieve performance that approaches hand-written callback code. +`StateMachine`s are a highly efficient, but boilerplate intensive way to perform +asynchronous computation. Continuations (particularly in the form of `Runnable`s +passed to `ListenableFuture`) are widespread in certain parts of *Bazel* code, +but aren't prevalent in analysis SkyFunctions. Analysis is mostly CPU bound and +there are no efficient asynchronous APIs for disk I/O. Eventually, it would be +good to optimize away callbacks as they have a learning curve and impede +readability. + +One of the most promising alternatives is *Java* virtual threads. Instead of +having to write callbacks, everything is replaced with synchronous, blocking +calls. This is possible because tying up a virtual thread resource, unlike a +platform thread, is supposed to be cheap. However, even with virtual threads, +replacing simple synchronous operations with thread creation and synchronization +primitives is too expensive. We performed a migration from `StateMachine`s to +*Java* virtual threads and they were orders of magnitude slower, leading to +almost a 3x increase in end-to-end analysis latency. Since virtual threads are +still a preview feature, it's possible that this migration can be performed at a +later date when performance improves. + +Another approach to consider is waiting for *Loom* coroutines, if they ever +become available. The advantage here is that it might be possible to reduce +synchronization overhead by using cooperative multitasking. + +If all else fails, low-level bytecode rewriting could also be a viable +alternative. With enough optimization, it might be possible to achieve +performance that approaches hand-written callback code. ## Appendix ### Callback Hell -Callback hell is an infamous problem in asynchronous code that uses callbacks. It stems from the fact that the continuation for a subsequent step is nested within the previous step. If there are many steps, this nesting can be extremely deep. If coupled with control flow the code becomes unmanageable. +Callback hell is an infamous problem in asynchronous code that uses callbacks. +It stems from the fact that the continuation for a subsequent step is nested +within the previous step. If there are many steps, this nesting can be extremely +deep. If coupled with control flow the code becomes unmanageable. ``` class CallbackHell implements StateMachine { @Override public StateMachine step(Tasks task) { doA(); - return (t, l) -> { + return (t, l) -> { doB(); - return (t1, l2) -> { + return (t1, l2) -> { doC(); return DONE; }; @@ -854,7 +1084,11 @@ class CallbackHell implements StateMachine { } ``` -One of the advantages of nested implementations is that the stack frame of the outer step can be preserved. In *Java*, captured lambda variables must be effectively final so using such variables can be cumbersome. Deep nesting is avoided by returning method references as continuations instead of lambdas as shown as follows. +One of the advantages of nested implementations is that the stack frame of the +outer step can be preserved. In *Java*, captured lambda variables must be +effectively final so using such variables can be cumbersome. Deep nesting is +avoided by returning method references as continuations instead of lambdas as +shown as follows. ``` class CallbackHellAvoided implements StateMachine { @@ -876,18 +1110,23 @@ class CallbackHellAvoided implements StateMachine { } ``` -Callback hell may also occur if the [`runAfter` injection](#runafter-injection) pattern is used too densely, but this can be avoided by interspersing injections with sequential steps. +Callback hell may also occur if the [`runAfter` injection](#runafter-injection) +pattern is used too densely, but this can be avoided by interspersing injections +with sequential steps. #### Example: Chained SkyValue lookups -It is often the case that the application logic requires dependent chains of SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. Thinking about this naively, this would result in a complex, deeply nested callback structure. +It is often the case that the application logic requires dependent chains of +SkyValue lookups, for example, if a second SkyKey depends on the first SkyValue. +Thinking about this naively, this would result in a complex, deeply nested +callback structure. ``` private ValueType1 value1; private ValueType2 value2; private StateMachine step1(...) { - tasks.lookUp(key1, (Consumer<SkyValue>) this); // key1 has type KeyType1. + tasks.lookUp(key1, (Consumer) this); // key1 has type KeyType1. return this::step2; } @@ -907,48 +1146,91 @@ private void acceptValueType2(SkyValue value) { } ``` -However, since continuations are specified as method references, the code looks procedural across state transitions: `step2` follows `step1`. Note that here, a lambda is used to assign `value2`. This makes the ordering of the code match the ordering of the computation from top-to-bottom. +However, since continuations are specified as method references, the code looks +procedural across state transitions: `step2` follows `step1`. Note that here, a +lambda is used to assign `value2`. This makes the ordering of the code match the +ordering of the computation from top-to-bottom. ### Miscellaneous Tips #### Readability: Execution Ordering -To improve readability, strive to keep the `StateMachine.step` implementations in execution order and callback implementations immediately following where they are passed in the code. This isn't always possible where the control flow branches. Additional comments might be helpful in such cases. +To improve readability, strive to keep the `StateMachine.step` implementations +in execution order and callback implementations immediately following where they +are passed in the code. This isn't always possible where the control flow +branches. Additional comments might be helpful in such cases. -In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an intermediate method reference is created to achieve this. This trades a small amount of performance for readability, which is likely worthwhile here. +In [Example: Chained SkyValue lookups](#chained-skyvalue-lookups), an +intermediate method reference is created to achieve this. This trades a small +amount of performance for readability, which is likely worthwhile here. #### Generational Hypothesis -Medium-lived *Java* objects break the generational hypothesis of the *Java* garbage collector, which is designed to handle objects that live for a very short time or objects that live forever. By definition, objects in `SkyKeyComputeState` violate this hypothesis. Such objects, containing the constructed tree of all still-running `StateMachine`s, rooted at `Driver` have an intermediate lifespan as they suspend, waiting for asynchronous computations to complete. - -It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes possible to observe an increase in GC time, even with dramatic decreases in actual garbage generated. Since `StateMachine`s have an intermediate lifespan they could be promoted to old gen, causing it to fill up more quickly, thus necessitating more expensive major or full GCs to clean up. - -The initial precaution is to minimize the use of `StateMachine` variables, but it is not always feasible, for example, if a value is needed across multiple states. Where it is possible, local stack `step` variables are young generation variables and efficiently GC'd. - -For `StateMachine` variables, breaking things down into subtasks and following the recommended pattern for [Propagating values between `StateMachine`s](#propagating-values) is also helpful. Observe that when following the pattern, only child `StateMachine`s have references to parent `StateMachine`s and not vice versa. This means that as children complete and update the parents using result callbacks, the children naturally fall out of scope and become eligible for GC. - -Finally, in some cases, a `StateMachine` variable is needed in earlier states but not in later states. It can be beneficial to null out references of large objects once it is known that they are no longer needed. +Medium-lived *Java* objects break the generational hypothesis of the *Java* +garbage collector, which is designed to handle objects that live for a very +short time or objects that live forever. By definition, objects in +`SkyKeyComputeState` violate this hypothesis. Such objects, containing the +constructed tree of all still-running `StateMachine`s, rooted at `Driver` have +an intermediate lifespan as they suspend, waiting for asynchronous computations +to complete. + +It seems less bad in JDK19, but when using `StateMachine`s, it's sometimes +possible to observe an increase in GC time, even with dramatic decreases in +actual garbage generated. Since `StateMachine`s have an intermediate lifespan +they could be promoted to old gen, causing it to fill up more quickly, thus +necessitating more expensive major or full GCs to clean up. + +The initial precaution is to minimize the use of `StateMachine` variables, but +it is not always feasible, for example, if a value is needed across multiple +states. Where it is possible, local stack `step` variables are young generation +variables and efficiently GC'd. + +For `StateMachine` variables, breaking things down into subtasks and following +the recommended pattern for [Propagating values between +`StateMachine`s](#propagating-values) is also helpful. Observe that when +following the pattern, only child `StateMachine`s have references to parent +`StateMachine`s and not vice versa. This means that as children complete and +update the parents using result callbacks, the children naturally fall out of +scope and become eligible for GC. + +Finally, in some cases, a `StateMachine` variable is needed in earlier states +but not in later states. It can be beneficial to null out references of large +objects once it is known that they are no longer needed. #### Naming states -When naming a method, it's usually possible to name a method for the behavior that happens within that method. It's less clear how to do this in `StateMachine`s because there is no stack. For example, suppose method `foo` calls a sub-method `bar`. In a `StateMachine`, this could be translated into the state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior `bar`. As a result, method names for states tend to be narrower in scope, potentially reflecting local behavior. +When naming a method, it's usually possible to name a method for the behavior +that happens within that method. It's less clear how to do this in +`StateMachine`s because there is no stack. For example, suppose method `foo` +calls a sub-method `bar`. In a `StateMachine`, this could be translated into the +state sequence `foo`, followed by `bar`. `foo` no longer includes the behavior +`bar`. As a result, method names for states tend to be narrower in scope, +potentially reflecting local behavior. ### Concurrency tree diagram -The following is an alternative view of the diagram in [Structured concurrency](#structured-concurrency) that better depicts the tree structure. The blocks form a small tree. +The following is an alternative view of the diagram in [Structured +concurrency](#structured-concurrency) that better depicts the tree structure. +The blocks form a small tree. ![Structured Concurrency 3D](/contribute/images/structured-concurrency-3d.svg) -## Footnotes - -1. In contrast to Skyframe's convention of restarting from the beginning when values are not available. [↩](#user-content-fnref-1) - -2. Note that `step` is permitted to throw `InterruptedException`, but the examples omit this. There are a few low methods in *Bazel* code that throw this exception and it propagates up to the `Driver`, to be described later, that runs the `StateMachine`. It's fine to not declare it to be thrown when unneeded. [↩](#user-content-fnref-2) - -3. Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which performs *independent* work for each dependency. Instead of manipulating complex data structures that process all the dependencies at once, introducing inefficiencies, each dependency has its own independent `StateMachine`. [↩](#user-content-fnref-3) - -4. Multiple `tasks.lookUp` calls within a single step are batched together. Additional batching can be created by lookups occurring within concurrent subtasks. [↩](#user-content-fnref-4) - -5. This is conceptually similar to Java’s structured concurrency [jeps/428](https://openjdk.org/jeps/428). [↩](#user-content-fnref-5) - -6. Doing this is similar to spawning a thread and joining it to achieve sequential composition. [↩](#user-content-fnref-6) +[^1]: In contrast to Skyframe's convention of restarting from the beginning when + values are not available. +[^2]: Note that `step` is permitted to throw `InterruptedException`, but the + examples omit this. There are a few low methods in *Bazel* code that throw + this exception and it propagates up to the `Driver`, to be described later, + that runs the `StateMachine`. It's fine to not declare it to be thrown when + unneeded. +[^3]: Concurrent subtasks were motivated by the `ConfiguredTargetFunction` which + performs *independent* work for each dependency. Instead of manipulating + complex data structures that process all the dependencies at once, + introducing inefficiencies, each dependency has its own independent + `StateMachine`. +[^4]: Multiple `tasks.lookUp` calls within a single step are batched together. + Additional batching can be created by lookups occurring within concurrent + subtasks. +[^5]: This is conceptually similar to Java’s structured concurrency + [jeps/428](https://openjdk.org/jeps/428). +[^6]: Doing this is similar to spawning a thread and joining it to achieve + sequential composition. diff --git a/contribute/windows-chocolatey-maintenance.mdx b/contribute/windows-chocolatey-maintenance.mdx index a570547d..c6aee8fb 100644 --- a/contribute/windows-chocolatey-maintenance.mdx +++ b/contribute/windows-chocolatey-maintenance.mdx @@ -2,16 +2,22 @@ title: 'Maintaining Bazel Chocolatey package on Windows' --- -Note: The Chocolatey package is experimental; please provide feedback (`@petemounce` in issue tracker). + + +Note: The Chocolatey package is experimental; please provide feedback +(`@petemounce` in issue tracker). ## Prerequisites You need: -- [chocolatey package manager](https://chocolatey.org) installed -- (to publish) a chocolatey API key granting you permission to publish the `bazel` package - - [@petemounce](https://github.com/petemounce) currently maintains this unofficial package. -- (to publish) to have set up that API key for the chocolatey source locally via `choco apikey -k <your key here> -s https://chocolatey.org/` +* [chocolatey package manager](https://chocolatey.org) installed +* (to publish) a chocolatey API key granting you permission to publish the + `bazel` package + * [@petemounce](https://github.com/petemounce) currently + maintains this unofficial package. +* (to publish) to have set up that API key for the chocolatey source locally + via `choco apikey -k -s https://chocolatey.org/` ## Build @@ -23,38 +29,44 @@ pushd scripts/packages/chocolatey popd ``` -Should result in `scripts/packages/chocolatey/bazel.<version>.nupkg` being created. +Should result in `scripts/packages/chocolatey/bazel..nupkg` being +created. The `build.ps1` script supports `mode` values `local`, `rc` and `release`. ## Test -1. Build the package (with `-mode local`) +0. Build the package (with `-mode local`) - - run a webserver (`python -m SimpleHTTPServer` in `scripts/packages/chocolatey` is convenient and starts one on `http://localhost:8000`) + * run a webserver (`python -m SimpleHTTPServer` in + `scripts/packages/chocolatey` is convenient and starts one on + `http://localhost:8000`) -2. Test the install +0. Test the install - The `test.ps1` should install the package cleanly (and error if it did not install cleanly), then tell you what to do next. + The `test.ps1` should install the package cleanly (and error if it did not + install cleanly), then tell you what to do next. -3. Test the uninstall +0. Test the uninstall - ```sh - choco uninstall bazel - # should remove bazel from the system - ``` + ```sh + choco uninstall bazel + # should remove bazel from the system + ``` Chocolatey's moderation process automates checks here as well. ## Release -Modify `tools/parameters.json` for the new release's URI and checksum once the release has been published to github releases. +Modify `tools/parameters.json` for the new release's URI and checksum once the +release has been published to github releases. ```powershell -./build.ps1 -version <version> -isRelease -./test.ps1 -version <version> +./build.ps1 -version -isRelease +./test.ps1 -version # if the test.ps1 passes choco push bazel.x.y.z.nupkg --source https://chocolatey.org/ ``` -Chocolatey.org will then run automated checks and respond to the push via email to the maintainers. +Chocolatey.org will then run automated checks and respond to the push via email +to the maintainers. diff --git a/contribute/windows-scoop-maintenance.mdx b/contribute/windows-scoop-maintenance.mdx index dc4aadd9..58e2a6c4 100644 --- a/contribute/windows-scoop-maintenance.mdx +++ b/contribute/windows-scoop-maintenance.mdx @@ -2,26 +2,38 @@ title: 'Maintaining Bazel Scoop package on Windows' --- -Note: The Scoop package is experimental. To provide feedback, go to `@excitoon` in issue tracker. + + +Note: The Scoop package is experimental. To provide feedback, go to +`@excitoon` in issue tracker. ## Prerequisites You need: -- [Scoop package manager](https://scoop.sh/) installed -- GitHub account in order to publish and create pull requests to [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) - - [@excitoon](https://github.com/excitoon) currently maintains this unofficial package. Feel free to ask questions by [e-mail](mailto:vladimir.chebotarev@gmail.com) or [Telegram](http://telegram.me/excitoon). +* [Scoop package manager](https://scoop.sh/) installed +* GitHub account in order to publish and create pull requests to + [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) + * [@excitoon](https://github.com/excitoon) currently maintains this + unofficial package. Feel free to ask questions by + [e-mail](mailto:vladimir.chebotarev@gmail.com) or + [Telegram](http://telegram.me/excitoon). ## Release process -Scoop packages are very easy to maintain. Once you have the URL of released Bazel, you need to make appropriate changes in [this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): +Scoop packages are very easy to maintain. Once you have the URL of released +Bazel, you need to make appropriate changes in +[this file](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json): - update version - update dependencies if needed - update URL - update hash (`sha256` by default) -In your filesystem, `bazel.json` is located in the directory `%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to your clone of a Git repository [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). +In your filesystem, `bazel.json` is located in the directory +`%UserProfile%/scoop/buckets/main/bucket` by default. This directory belongs to +your clone of a Git repository +[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main). Test the result: @@ -32,7 +44,9 @@ bazel version bazel something_else ``` -The first time, make a fork of [scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and specify it as your own remote for `%UserProfile%/scoop/buckets/main`: +The first time, make a fork of +[scoopinstaller/scoop-main](https://github.com/scoopinstaller/scoop-main) and +specify it as your own remote for `%UserProfile%/scoop/buckets/main`: ``` git remote add mine FORK_URL diff --git a/docs/android-build-performance.mdx b/docs/android-build-performance.mdx index a96a199d..0d5edc77 100644 --- a/docs/android-build-performance.mdx +++ b/docs/android-build-performance.mdx @@ -2,39 +2,53 @@ title: 'Android Build Performance' --- -This page contains information on optimizing build performance for Android apps specifically. For general build performance optimization with Bazel, see [Optimizing Performance](/rules/performance). + + +This page contains information on optimizing build performance for Android +apps specifically. For general build performance optimization with Bazel, see +[Optimizing Performance](/rules/performance). ## Recommended flags -The flags are in the [`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so they can be pasted directly into a `bazelrc` file and invoked with `--config=<configuration_name>` on the command line. +The flags are in the +[`bazelrc` configuration syntax](/run/bazelrc#bazelrc-syntax-semantics), so +they can be pasted directly into a `bazelrc` file and invoked with +`--config=` on the command line. **Profiling performance** -Bazel writes a JSON trace profile by default to a file called `command.profile.gz` in Bazel's output base. See the [JSON Profile documentation](/rules/performance#performance-profiling) for how to read and interact with the profile. +Bazel writes a JSON trace profile by default to a file called +`command.profile.gz` in Bazel's output base. +See the [JSON Profile documentation](/rules/performance#performance-profiling) for +how to read and interact with the profile. **Persistent workers for Android build actions**. -A subset of Android build actions has support for [persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). +A subset of Android build actions has support for +[persistent workers](https://blog.bazel.build/2015/12/10/java-workers.html). These actions' mnemonics are: -- DexBuilder -- Javac -- Desugar -- AaptPackage -- AndroidResourceParser -- AndroidResourceValidator -- AndroidResourceCompiler -- RClassGenerator -- AndroidResourceLink -- AndroidAapt2 -- AndroidAssetMerger -- AndroidResourceMerger -- AndroidCompiledResourceMerger - -Enabling workers can result in better build performance by saving on JVM startup costs from invoking each of these tools, but at the cost of increased memory usage on the system by persisting them. - -To enable workers for these actions, apply these flags with `--config=android_workers` on the command line: +* DexBuilder +* Javac +* Desugar +* AaptPackage +* AndroidResourceParser +* AndroidResourceValidator +* AndroidResourceCompiler +* RClassGenerator +* AndroidResourceLink +* AndroidAapt2 +* AndroidAssetMerger +* AndroidResourceMerger +* AndroidCompiledResourceMerger + +Enabling workers can result in better build performance by saving on JVM +startup costs from invoking each of these tools, but at the cost of increased +memory usage on the system by persisting them. + +To enable workers for these actions, apply these flags with +`--config=android_workers` on the command line: ``` build:android_workers --strategy=DexBuilder=worker @@ -54,7 +68,11 @@ build:android_workers --strategy=Desugar=worker build:android_workers --persistent_android_resource_processor ``` -The default number of persistent workers created per action is `4`. We have [measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) by capping the number of instances for each action to `1` or `2`, although this may vary depending on the system Bazel is running on, and the project being built. +The default number of persistent workers created per action is `4`. We have +[measured improved build performance](https://github.com/bazelbuild/bazel/issues/8586#issuecomment-500070549) +by capping the number of instances for each action to `1` or `2`, although this +may vary depending on the system Bazel is running on, and the project being +built. To cap the number of instances for an action, apply these flags: @@ -68,8 +86,12 @@ build:android_workers --worker_max_instances=AaptPackage=2 **Using AAPT2** -[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the `--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on `android_binary` and `android_local_test`. +[`aapt2`](https://developer.android.com/studio/command-line/aapt2) has improved +performance over `aapt` and also creates smaller APKs. To use `aapt2`, use the +`--android_aapt=aapt2` flag or set `aapt2` on the `aapt_version` on +`android_binary` and `android_local_test`. **SSD optimizations** -The `--experimental_multi_threaded_digest` flag is useful for optimizing digest computation on SSDs. +The `--experimental_multi_threaded_digest` flag is useful for optimizing digest +computation on SSDs. diff --git a/docs/android-instrumentation-test.mdx b/docs/android-instrumentation-test.mdx index e5b23f88..fca7b577 100644 --- a/docs/android-instrumentation-test.mdx +++ b/docs/android-instrumentation-test.mdx @@ -2,23 +2,34 @@ title: 'Android Instrumentation Tests' --- -*If you're new to Bazel, start with the [Building Android with Bazel](/start/android-app) tutorial.* + + +_If you're new to Bazel, start with the [Building Android with +Bazel](/start/android-app ) tutorial._ ![Running Android instrumentation tests in parallel](/docs/images/android_test.gif "Android instrumentation test") **Figure 1.** Running parallel Android instrumentation tests. -[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) allows developers to test their apps on Android emulators and devices. It utilizes real Android framework APIs and the Android Test Library. +[`android_instrumentation_test`](/reference/be/android#android_instrumentation_test) +allows developers to test their apps on Android emulators and devices. +It utilizes real Android framework APIs and the Android Test Library. -For hermeticity and reproducibility, Bazel creates and launches Android emulators in a sandbox, ensuring that tests always run from a clean state. Each test gets an isolated emulator instance, allowing tests to run in parallel without passing states between them. +For hermeticity and reproducibility, Bazel creates and launches Android +emulators in a sandbox, ensuring that tests always run from a clean state. Each +test gets an isolated emulator instance, allowing tests to run in parallel +without passing states between them. -For more information on Android instrumentation tests, check out the [Android developer documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). +For more information on Android instrumentation tests, check out the [Android +developer +documentation](https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests.html). Please file issues in the [GitHub issue tracker](https://github.com/bazelbuild/bazel/issues). ## How it works -When you run `bazel test` on an `android_instrumentation_test` target for the first time, Bazel performs the following steps: +When you run `bazel test` on an `android_instrumentation_test` target for the +first time, Bazel performs the following steps: 1. Builds the test APK, APK under test, and their transitive dependencies 2. Creates, boots, and caches clean emulator states @@ -28,7 +39,9 @@ When you run `bazel test` on an `android_instrumentation_test` target for the fi 6. Shuts down the emulator 7. Reports the results -In subsequent test runs, Bazel boots the emulator from the clean, cached state created in step 2, so there are no leftover states from previous runs. Caching emulator state also speeds up test runs. +In subsequent test runs, Bazel boots the emulator from the clean, cached state +created in step 2, so there are no leftover states from previous runs. Caching +emulator state also speeds up test runs. ## Prerequisites @@ -41,14 +54,17 @@ Ensure your environment satisfies the following prerequisites: ```posix-terminal bazel info release ``` - This results in output similar to the following: -```none +```none {:.devsite-disable-click-to-copy} release 4.1.0 ``` -- **KVM**. Bazel requires emulators to have [hardware acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) with KVM on Linux. You can follow these [installation instructions](https://help.ubuntu.com/community/KVM/Installation) for Ubuntu. +- **KVM**. Bazel requires emulators to have [hardware + acceleration](https://developer.android.com/studio/run/emulator-acceleration.html#accel-check) + with KVM on Linux. You can follow these + [installation instructions](https://help.ubuntu.com/community/KVM/Installation) + for Ubuntu. To verify that KVM has the correct configuration, run: @@ -58,32 +74,34 @@ apt-get install cpu-checker && kvm-ok If it prints the following message, you have the correct configuration: -```none +```none {:.devsite-disable-click-to-copy} INFO: /dev/kvm exists KVM acceleration can be used ``` -- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). +- **Xvfb**. To run headless tests (for example, on CI servers), Bazel requires + the [X virtual framebuffer](https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml). To install it, run: ```posix-terminal apt-get install xvfb ``` - -Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` by running: +Verify that `Xvfb` is installed correctly and is installed at `/usr/bin/Xvfb` +by running: ```posix-terminal which Xvfb ``` - The output is the following: ```{:.devsite-disable-click-to-copy} /usr/bin/Xvfb ``` -- **32-bit Libraries**. Some of the binaries used by the test infrastructure are 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For Ubuntu, install these 32-bit libraries: +- **32-bit Libraries**. Some of the binaries used by the test infrastructure are + 32-bit, so on 64-bit machines, ensure that 32-bit binaries can be run. For + Ubuntu, install these 32-bit libraries: ```posix-terminal sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 lib32z1 libbz2-1.0:i386 @@ -97,6 +115,7 @@ Here is a typical target dependency graph of an `android_instrumentation_test`: **Figure 2.** Target dependency graph of an `android_instrumentation_test`. + ### BUILD file The graph translates into a `BUILD` file like this: @@ -150,45 +169,59 @@ android_library( The main attributes of the rule `android_instrumentation_test` are: -- `test_app`: An `android_binary` target. This target contains test code and dependencies like Espresso and UIAutomator. The selected `android_binary` target is required to specify an `instruments` attribute pointing to another `android_binary`, which is the app under test. +- `test_app`: An `android_binary` target. This target contains test code and + dependencies like Espresso and UIAutomator. The selected `android_binary` + target is required to specify an `instruments` attribute pointing to another + `android_binary`, which is the app under test. -- `target_device`: An `android_device` target. This target describes the specifications of the Android emulator which Bazel uses to create, launch and run the tests. See the [section on choosing an Android device](#android-device-target) for more information. +- `target_device`: An `android_device` target. This target describes the + specifications of the Android emulator which Bazel uses to create, launch and + run the tests. See the [section on choosing an Android + device](#android-device-target) for more information. -The test app's `AndroidManifest.xml` must include [an `<instrumentation>` tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). This tag must specify the attributes for the **package of the target app** and the **fully qualified class name of the instrumentation test runner**, `androidx.test.runner.AndroidJUnitRunner`. +The test app's `AndroidManifest.xml` must include [an `` +tag](https://developer.android.com/studio/test/#configure_instrumentation_manifest_settings). +This tag must specify the attributes for the **package of the target app** and +the **fully qualified class name of the instrumentation test runner**, +`androidx.test.runner.AndroidJUnitRunner`. Here is an example `AndroidTestManifest.xml` for the test app: ```xml -<?xml version="1.0" encoding="UTF-8"?> -<manifest xmlns:android="http://schemas.android.com/apk/res/android" + + - <instrumentation + - <uses-sdk + - <application > - - </application> -</manifest> + + + + ``` ### WORKSPACE dependencies -In order to use this rule, your project needs to depend on these external repositories: +In order to use this rule, your project needs to depend on these external +repositories: - `@androidsdk`: The Android SDK. Download this through Android Studio. -- `@android_test_support`: Hosts the test runner, emulator launcher, and `android_device` targets. You can find the [latest release here](https://github.com/android/android-test/releases). +- `@android_test_support`: Hosts the test runner, emulator launcher, and + `android_device` targets. You can find the [latest release + here](https://github.com/android/android-test/releases). -Enable these dependencies by adding the following lines to your `WORKSPACE` file: +Enable these dependencies by adding the following lines to your `WORKSPACE` +file: ```python # Android SDK @@ -210,20 +243,25 @@ android_test_repositories() ## Maven dependencies -For managing dependencies on Maven artifacts from repositories, such as [Google Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), you should use a Maven resolver, such as [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). +For managing dependencies on Maven artifacts from repositories, such as [Google +Maven](https://maven.google.com) or [Maven Central](https://central.maven.org), +you should use a Maven resolver, such as +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external). -The rest of this page shows how to use `rules_jvm_external` to resolve and fetch dependencies from Maven repositories. +The rest of this page shows how to use `rules_jvm_external` to +resolve and fetch dependencies from Maven repositories. -## Choosing an android\_device target +## Choosing an android_device target -`android_instrumentation_test.target_device` specifies which Android device to run the tests on. These `android_device` targets are defined in [`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). +`android_instrumentation_test.target_device` specifies which Android device to +run the tests on. These `android_device` targets are defined in +[`@android_test_support`](https://github.com/google/android-testing-support-library/tree/master/tools/android/emulated_devices). For example, you can query for the sources for a particular target by running: ```posix-terminal bazel query --output=build @android_test_support//tools/android/emulated_devices/generic_phone:android_23_x86 ``` - Which results in output that looks similar to: ```python @@ -249,22 +287,29 @@ android_device( The device target names use this template: ``` -@android_test_support//tools/android/emulated_devices/<var>device_type</var>:<var>system</var>_<var>api_level</var>_x86_qemu2 +@android_test_support//tools/android/emulated_devices/{{ "" }}device_type{{ "" }}:{{ "" }}system{{ "" }}_{{ "" }}api_level{{ "" }}_x86_qemu2 ``` -In order to launch an `android_device`, the `system_image` for the selected API level is required. To download the system image, use Android SDK's `tools/bin/sdkmanager`. For example, to download the system image for `generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager "system-images;android-23;default;x86"`. +In order to launch an `android_device`, the `system_image` for the selected API +level is required. To download the system image, use Android SDK's +`tools/bin/sdkmanager`. For example, to download the system image for +`generic_phone:android_23_x86`, run `$sdk/tools/bin/sdkmanager +"system-images;android-23;default;x86"`. -To see the full list of supported `android_device` targets in `@android_test_support`, run the following command: +To see the full list of supported `android_device` targets in +`@android_test_support`, run the following command: ```posix-terminal bazel query 'filter("x86_qemu2$", kind(android_device, @android_test_support//tools/android/emulated_devices/...:*))' ``` -Bazel currently supports x86-based emulators only. For better performance, use `QEMU2` `android_device` targets instead of `QEMU` ones. +Bazel currently supports x86-based emulators only. For better performance, use +`QEMU2` `android_device` targets instead of `QEMU` ones. ## Running tests -To run tests, add these lines to your project's `<var>project root</var>:<var>/.bazelrc` file. +To run tests, add these lines to your project's +`project root:/.bazelrc` file. ``` # Configurations for testing with Bazel @@ -294,11 +339,13 @@ Then, use one of the configurations to run tests: - `bazel test //my/test:target --config=headless` - `bazel test //my/test:target --config=local_device` -Use **only one configuration** or tests will fail. +Use __only one configuration__ or tests will fail. ### Headless testing -With `Xvfb`, it is possible to test with emulators without the graphical interface, also known as headless testing. To disable the graphical interface when running tests, pass the test argument `--enable_display=false` to Bazel: +With `Xvfb`, it is possible to test with emulators without the graphical +interface, also known as headless testing. To disable the graphical interface +when running tests, pass the test argument `--enable_display=false` to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=false @@ -306,7 +353,9 @@ bazel test //my/test:target --test_arg=--enable_display=false ### GUI testing -If the `$DISPLAY` environment variable is set, it's possible to enable the graphical interface of the emulator while the test is running. To do this, pass these test arguments to Bazel: +If the `$DISPLAY` environment variable is set, it's possible to enable the +graphical interface of the emulator while the test is running. To do this, pass +these test arguments to Bazel: ```posix-terminal bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY @@ -314,15 +363,26 @@ bazel test //my/test:target --test_arg=--enable_display=true --test_env=DISPLAY ### Testing with a local emulator or device -Bazel also supports testing directly on a locally launched emulator or connected device. Pass the flags `--test_strategy=exclusive` and `--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. If there is more than one connected device, pass the flag `--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of the device/emulator listed in `adb devices`. +Bazel also supports testing directly on a locally launched emulator or connected +device. Pass the flags +`--test_strategy=exclusive` and +`--test_arg=--device_broker_type=LOCAL_ADB_SERVER` to enable local testing mode. +If there is more than one connected device, pass the flag +`--test_arg=--device_serial_number=$device_id` where `$device_id` is the id of +the device/emulator listed in `adb devices`. ## Sample projects -If you are looking for canonical project samples, see the [Android testing samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) for projects using Espresso and UIAutomator. +If you are looking for canonical project samples, see the [Android testing +samples](https://github.com/googlesamples/android-testing#experimental-bazel-support) +for projects using Espresso and UIAutomator. ## Espresso setup -If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) (`androidx.test.espresso`), you can use the following snippets to set up your Bazel workspace with the list of commonly used Espresso artifacts and their dependencies: +If you write UI tests with [Espresso](https://developer.android.com/training/testing/espresso/) +(`androidx.test.espresso`), you can use the following snippets to set up your +Bazel workspace with the list of commonly used Espresso artifacts and their +dependencies: ``` androidx.test.espresso:espresso-core @@ -333,7 +393,8 @@ org.hamcrest:java-hamcrest junit:junit ``` -One way to organize these dependencies is to create a `//:test_deps` shared library in your `<var>project root</var>/BUILD.bazel` file: +One way to organize these dependencies is to create a `//:test_deps` shared +library in your `project root/BUILD.bazel` file: ```python java_library( @@ -350,7 +411,7 @@ java_library( ) ``` -Then, add the required dependencies in `<var>project root</var>/WORKSPACE`: +Then, add the required dependencies in `project root/WORKSPACE`: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -383,7 +444,8 @@ maven_install( ) ``` -Finally, in your test `android_binary` target, add the `//:test_deps` dependency: +Finally, in your test `android_binary` target, add the `//:test_deps` +dependency: ```python android_binary( @@ -401,14 +463,17 @@ android_binary( ### Reading test logs -Use `--test_output=errors` to print logs for failing tests, or `--test_output=all` to print all test output. If you're looking for an individual test log, go to `$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. +Use `--test_output=errors` to print logs for failing tests, or +`--test_output=all` to print all test output. If you're looking for an +individual test log, go to +`$PROJECT_ROOT/bazel-testlogs/path/to/InstrumentationTestTargetName`. -For example, the test logs for `BasicSample` canonical project are in `bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: +For example, the test logs for `BasicSample` canonical project are in +`bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest`, run: ```posix-terminal tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ``` - This results in the following output: ```none @@ -466,7 +531,9 @@ $ tree bazel-testlogs/ui/espresso/BasicSample/BasicSampleInstrumentationTest ### Reading emulator logs -The emulator logs for `android_device` targets are stored in the `/tmp/` directory with the name `emulator_xxxxx.log`, where `xxxxx` is a randomly-generated sequence of characters. +The emulator logs for `android_device` targets are stored in the `/tmp/` +directory with the name `emulator_xxxxx.log`, where `xxxxx` is a +randomly-generated sequence of characters. Use this command to find the latest emulator log: @@ -476,7 +543,8 @@ ls -1t /tmp/emulator_*.log | head -n 1 ### Testing against multiple API levels -If you would like to test against multiple API levels, you can use a list comprehension to create test targets for each API level. For example: +If you would like to test against multiple API levels, you can use a list +comprehension to create test targets for each API level. For example: ```python API_LEVELS = [ @@ -495,10 +563,14 @@ API_LEVELS = [ ## Known issues -- [Forked adb server processes are not terminated after tests](https://github.com/bazelbuild/bazel/issues/4853) -- While APK building works on all platforms (Linux, macOS, Windows), testing only works on Linux. -- Even with `--config=local_adb`, users still need to specify `android_instrumentation_test.target_device`. -- If using a local device or emulator, Bazel does not uninstall the APKs after the test. Clean the packages by running this command: +- [Forked adb server processes are not terminated after + tests](https://github.com/bazelbuild/bazel/issues/4853) +- While APK building works on all platforms (Linux, macOS, Windows), testing + only works on Linux. +- Even with `--config=local_adb`, users still need to specify + `android_instrumentation_test.target_device`. +- If using a local device or emulator, Bazel does not uninstall the APKs after + the test. Clean the packages by running this command: ```posix-terminal adb shell pm list diff --git a/docs/android-ndk.mdx b/docs/android-ndk.mdx index 292231dc..b434797d 100644 --- a/docs/android-ndk.mdx +++ b/docs/android-ndk.mdx @@ -2,13 +2,25 @@ title: 'Using the Android Native Development Kit with Bazel' --- -*If you're new to Bazel, please start with the [Building Android with Bazel](/start/android-app) tutorial.* + + +_If you're new to Bazel, please start with the [Building Android with +Bazel](/start/android-app ) tutorial._ ## Overview -Bazel can run in many different build configurations, including several that use the Android Native Development Kit (NDK) toolchain. This means that normal `cc_library` and `cc_binary` rules can be compiled for Android directly within Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository rule and its related bzlmod extension. +Bazel can run in many different build configurations, including several that use +the Android Native Development Kit (NDK) toolchain. This means that normal +`cc_library` and `cc_binary` rules can be compiled for Android directly within +Bazel. Bazel accomplishes this by using the `android_ndk_repository` repository +rule and its related bzlmod extension. -For general Android compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). This tutorial demonstrates how to integrate C++ library dependencies into Android apps and uses [`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK toolchain discovery and registration. +For general Android +compilation, use [`rules_android`](https://github.com/bazelbuild/rules_android). +This tutorial demonstrates how to integrate C++ library dependencies into +Android apps and uses +[`rules_android_ndk`](https://github.com/bazelbuild/rules_android_ndk) for NDK +toolchain discovery and registration. ## Prerequisites @@ -16,7 +28,12 @@ Please ensure that you have installed the Android SDK and NDK. ### NDK and SDK setup -External repository setup varies depending on whether you are using WORKSPACE or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of each other. If you are using one dependency management solution, you don't need to add the boilerplate for the other. +External repository setup varies depending on whether you are using WORKSPACE +or bzlmod (MODULE.bazel). *Bzlmod is the preferred solution for Bazel 7+.* +Note that the MODULE.bazel and WORKSPACE setup stanzas are independent of +each other. +If you are using one dependency management solution, you don't need to add +the boilerplate for the other. #### Bzlmod MODULE.bazel setup @@ -60,18 +77,23 @@ android_ndk_repository( Compatibility notes for WORKSPACE: -- Both `rules_android` and `rules_android_ndk` rules require extra boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) file of `rules_android_ndk`'s basic example app. +* Both `rules_android` and `rules_android_ndk` rules require extra + boilerplate not depicted in the WORKSPACE snippet above. For an up-to-date + and fully-formed instantiation stanza, see the [WORKSPACE](https://github.com/bazelbuild/rules_android_ndk/blob/main/examples/basic/WORKSPACE) + file of `rules_android_ndk`'s basic example app. -For more information about the `android_ndk_repository` rule, see its [docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). +For more information about the `android_ndk_repository` rule, see its +[docstring](https://github.com/bazelbuild/rules_android_ndk/blob/7b4300f6d731139ca097f3332a5aebae5b0d91d0/rules.bzl#L18-L25). ## Quick start -To build C++ for Android, simply add `cc_library` dependencies to your `android_binary` or `android_library` rules. +To build C++ for Android, simply add `cc_library` dependencies to your +`android_binary` or `android_library` rules. For example, given the following `BUILD` file for an Android app: ```python -# In <project>/app/src/main/BUILD.bazel +# In /app/src/main/BUILD.bazel load("@rules_cc//cc:cc_library.bzl", "cc_library") load("@rules_android//rules:rules.bzl", "android_binary", "android_library") @@ -100,17 +122,19 @@ This `BUILD` file results in the following target graph: ![Example results](/docs/images/android_ndk.png "Build graph results") -**Figure 1.** Build graph of Android project with cc\_library dependencies. +**Figure 1.** Build graph of Android project with cc_library dependencies. To build the app, simply run: ```posix-terminal -bazel build //app/src/main:app --android_platforms=<your platform> +bazel build //app/src/main:app --android_platforms= ``` -Note that if you don't specify `--android_platforms`, your build will fail with errors about missing JNI headers. +Note that if you don't specify `--android_platforms`, your build will fail with +errors about missing JNI headers. -The `bazel build` command compiles the Java files, Android resource files, and `cc_library` rules, and packages everything into an APK: +The `bazel build` command compiles the Java files, Android resource files, and +`cc_library` rules, and packages everything into an APK: ```posix-terminal $ zipinfo -1 bazel-bin/app/src/main/app.apk @@ -126,21 +150,27 @@ META-INF/CERT.RSA META-INF/MANIFEST.MF ``` -Bazel compiles all of the cc\_libraries into a single shared object (`.so`) file, targeted the architectures specified by `--android_platforms`. See the section on [configuring the target ABI](#configuring-target-abi) for more details. +Bazel compiles all of the cc_libraries into a single shared object (`.so`) file, +targeted the architectures specified by `--android_platforms`. +See the section on [configuring the target ABI](#configuring-target-abi) for +more details. ## Example setup -This example is available in the [Bazel examples repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). +This example is available in the [Bazel examples +repository](https://github.com/bazelbuild/examples/tree/master/android/ndk). -In the `BUILD.bazel` file, three targets are defined with the `android_binary`, `android_library`, and `cc_library` rules. +In the `BUILD.bazel` file, three targets are defined with the `android_binary`, +`android_library`, and `cc_library` rules. The `android_binary` top-level target builds the APK. -The `cc_library` target contains a single C++ source file with a JNI function implementation: +The `cc_library` target contains a single C++ source file with a JNI function +implementation: ```c++ -#include <jni.h> -#include <string> +#include +#include extern "C" JNIEXPORT jstring @@ -150,11 +180,14 @@ Java_com_example_android_bazel_MainActivity_stringFromJNI( JNIEnv *env, jobject /* this */) { std::string hello = "Hello from C++"; - return env->NewStringUTF(hello.c_str()); + return env->NewStringUTF(hello.c_str()); } ``` -The `android_library` target specifies the Java sources, resource files, and the dependency on a `cc_library` target. For this example, `MainActivity.java` loads the shared object file `libapp.so`, and defines the method signature for the JNI function: +The `android_library` target specifies the Java sources, resource files, and the +dependency on a `cc_library` target. For this example, `MainActivity.java` loads +the shared object file `libapp.so`, and defines the method signature for the JNI +function: ```java public class MainActivity extends AppCompatActivity { @@ -173,19 +206,23 @@ public class MainActivity extends AppCompatActivity { } ``` -Note: The name of the native library is derived from the name of the top level `android_binary` target. In this example, it is `app`. +Note: The name of the native library is derived from the name of the top +level `android_binary` target. In this example, it is `app`. ## Configuring the target ABI To configure the target ABI, use the `--android_platforms` flag as follows: ```posix-terminal -bazel build //:app --android_platforms=<var>comma-separated list of platforms</var> +bazel build //:app --android_platforms={{ "" }}comma-separated list of platforms{{ "" }} ``` -Just like the `--platforms` flag, the values passed to `--android_platforms` are the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) targets, using standard constraint values to describe your device. +Just like the `--platforms` flag, the values passed to `--android_platforms` are +the labels of [`platform`](https://bazel.build/reference/be/platforms-and-toolchains#platform) +targets, using standard constraint values to describe your device. -For example, for an Android device with a 64-bit ARM processor, you'd define your platform like this: +For example, for an Android device with a 64-bit ARM processor, you'd define +your platform like this: ```py platform( @@ -197,36 +234,43 @@ platform( ) ``` -Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) OS constraint. To migrate the CPU constraint, check this chart: +Every Android `platform` should use the [`@platforms//os:android`](https://github.com/bazelbuild/platforms/blob/33a3b209f94856193266871b1545054afb90bb28/os/BUILD#L36) +OS constraint. To migrate the CPU constraint, check this chart: -| CPU Value | Platform | -| ------------- | ------------------------ | -| `armeabi-v7a` | `@platforms//cpu:armv7` | -| `arm64-v8a` | `@platforms//cpu:arm64` | -| `x86` | `@platforms//cpu:x86_32` | -| `x86_64` | `@platforms//cpu:x86_64` | +CPU Value | Platform +------------- | ------------------------------------------ +`armeabi-v7a` | `@platforms//cpu:armv7` +`arm64-v8a` | `@platforms//cpu:arm64` +`x86` | `@platforms//cpu:x86_32` +`x86_64` | `@platforms//cpu:x86_64` -And, of course, for a multi-architecture APK, you pass multiple labels, for example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in your top-level `BUILD.bazel` file). +And, of course, for a multi-architecture APK, you pass multiple labels, for +example: `--android_platforms=//:arm64,//:x86_64` (assuming you defined those in +your top-level `BUILD.bazel` file). -Bazel is unable to select a default Android platform, so one must be defined and specified with `--android_platforms`. +Bazel is unable to select a default Android platform, so one must be defined and +specified with `--android_platforms`. -Depending on the NDK revision and Android API level, the following ABIs are available: +Depending on the NDK revision and Android API level, the following ABIs are +available: | NDK revision | ABIs | -| ------------ | ----------------------------------------------------------- | +|--------------|-------------------------------------------------------------| | 16 and lower | armeabi, armeabi-v7a, arm64-v8a, mips, mips64, x86, x86\_64 | | 17 and above | armeabi-v7a, arm64-v8a, x86, x86\_64 | -See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) for more information on these ABIs. +See [the NDK docs](https://developer.android.com/ndk/guides/abis.html) +for more information on these ABIs. -Multi-ABI Fat APKs are not recommended for release builds since they increase the size of the APK, but can be useful for development and QA builds. +Multi-ABI Fat APKs are not recommended for release builds since they increase +the size of the APK, but can be useful for development and QA builds. ## Selecting a C++ standard Use the following flags to build according to a C++ standard: | C++ Standard | Flag | -| ------------ | ----------------------- | +|--------------|-------------------------| | C++98 | Default, no flag needed | | C++11 | `--cxxopt=-std=c++11` | | C++14 | `--cxxopt=-std=c++14` | @@ -238,9 +282,11 @@ For example: bazel build //:app --cxxopt=-std=c++11 ``` -Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and `--linkopt` in the [User Manual](/docs/user-manual#cxxopt). +Read more about passing compiler and linker flags with `--cxxopt`, `--copt`, and +`--linkopt` in the [User Manual](/docs/user-manual#cxxopt). -Compiler and linker flags can also be specified as attributes in `cc_library` using `copts` and `linkopts`. For example: +Compiler and linker flags can also be specified as attributes in `cc_library` +using `copts` and `linkopts`. For example: ```python cc_library( @@ -253,9 +299,11 @@ cc_library( ## Building a `cc_library` for Android without using `android_binary` -To build a standalone `cc_binary` or `cc_library` for Android without using an `android_binary`, use the `--platforms` flag. +To build a standalone `cc_binary` or `cc_library` for Android without using an +`android_binary`, use the `--platforms` flag. -For example, assuming you have defined Android platforms in `my/platforms/BUILD`: +For example, assuming you have defined Android platforms in +`my/platforms/BUILD`: ```posix-terminal bazel build //my/cc/jni:target \ @@ -264,9 +312,13 @@ bazel build //my/cc/jni:target \ With this approach, the entire build tree is affected. -Note: All of the targets on the command line must be compatible with building for Android when specifying these flags, which may make it difficult to use [Bazel wild-cards](/run/build#specifying-build-targets) like `/...` and `:all`. +Note: All of the targets on the command line must be compatible with +building for Android when specifying these flags, which may make it difficult to +use [Bazel wild-cards](/run/build#specifying-build-targets) like +`/...` and `:all`. -These flags can be put into a `bazelrc` config (one for each ABI), in `<var>project</var>/.bazelrc`: +These flags can be put into a `bazelrc` config (one for each ABI), in +`project/.bazelrc`: ``` common:android_x86 --platforms=//my/platforms:x86 @@ -274,7 +326,7 @@ common:android_x86 --platforms=//my/platforms:x86 common:android_armeabi-v7a --platforms=//my/platforms:armeabi-v7a # In general -common:android_<abi> --platforms=//my/platforms:<abi> +common:android_ --platforms=//my/platforms: ``` Then, to build a `cc_library` for `x86` for example, run: @@ -283,4 +335,7 @@ Then, to build a `cc_library` for `x86` for example, run: bazel build //my/cc/jni:target --config=android_x86 ``` -In general, use this method for low-level targets (like `cc_library`) or when you know exactly what you're building; rely on the automatic configuration transitions from `android_binary` for high-level targets where you're expecting to build a lot of targets you don't control. +In general, use this method for low-level targets (like `cc_library`) or when +you know exactly what you're building; rely on the automatic configuration +transitions from `android_binary` for high-level targets where you're expecting +to build a lot of targets you don't control. diff --git a/docs/bazel-and-android.mdx b/docs/bazel-and-android.mdx index b043a8bc..bf3625c9 100644 --- a/docs/bazel-and-android.mdx +++ b/docs/bazel-and-android.mdx @@ -2,27 +2,44 @@ title: 'Android and Bazel' --- -This page contains resources that help you use Bazel with Android projects. It links to a tutorial, build rules, and other information specific to building Android projects with Bazel. + + +This page contains resources that help you use Bazel with Android projects. It +links to a tutorial, build rules, and other information specific to building +Android projects with Bazel. ## Getting started The following resources will help you work with Bazel on Android projects: -- [Tutorial: Building an Android app](/start/android-app). This tutorial is a good place to start learning about Bazel commands and concepts, and how to build Android apps with Bazel. -- [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). This codelab explains how to build Android apps with Bazel. +* [Tutorial: Building an Android app](/start/android-app ). This + tutorial is a good place to start learning about Bazel commands and concepts, + and how to build Android apps with Bazel. +* [Codelab: Building Android Apps with Bazel](https://developer.android.com/codelabs/bazel-android-intro#0). + This codelab explains how to build Android apps with Bazel. ## Features -Bazel has Android rules for building and testing Android apps, integrating with the SDK/NDK, and creating emulator images. There are also Bazel plugins for Android Studio and IntelliJ. - -- [Android rules](/reference/be/android). The Build Encyclopedia describes the rules for building and testing Android apps with Bazel. -- [Integration with Android Studio](/install/ide). Bazel is compatible with Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) plugin. -- [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` feature provides automated build-and-deploy functionality for building and testing Android apps directly on Android devices and emulators. -- [Android instrumentation testing](/docs/android-instrumentation-test) on emulators and devices. -- [Android NDK integration](/docs/android-ndk). Bazel supports compiling to native code through direct NDK integration and the C++ rules. -- [Android build performance](/docs/android-build-performance). This page provides information on optimizing build performance for Android apps. +Bazel has Android rules for building and testing Android apps, integrating with +the SDK/NDK, and creating emulator images. There are also Bazel plugins for +Android Studio and IntelliJ. + +* [Android rules](/reference/be/android). The Build Encyclopedia describes the rules + for building and testing Android apps with Bazel. +* [Integration with Android Studio](/install/ide). Bazel is compatible with + Android Studio using the [Android Studio with Bazel](https://ij.bazel.build/) + plugin. +* [`mobile-install` for Android](/docs/mobile-install). Bazel's `mobile-install` + feature provides automated build-and-deploy functionality for building and + testing Android apps directly on Android devices and emulators. +* [Android instrumentation testing](/docs/android-instrumentation-test) on + emulators and devices. +* [Android NDK integration](/docs/android-ndk). Bazel supports compiling to + native code through direct NDK integration and the C++ rules. +* [Android build performance](/docs/android-build-performance). This page + provides information on optimizing build performance for Android apps. ## Further reading -- Integrating with dependencies from Google Maven and Maven Central with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external). -- Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). +* Integrating with dependencies from Google Maven and Maven Central with [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external). +* Learn [How Android Builds Work in Bazel](https://blog.bazel.build/2018/02/14/how-android-builds-work-in-bazel.html). diff --git a/docs/bazel-and-apple.mdx b/docs/bazel-and-apple.mdx index 430c0f1f..6e4a06fe 100644 --- a/docs/bazel-and-apple.mdx +++ b/docs/bazel-and-apple.mdx @@ -2,52 +2,85 @@ title: 'Apple Apps and Bazel' --- -This page contains resources that help you use Bazel to build macOS and iOS projects. It links to a tutorial, build rules, and other information specific to using Bazel to build and test for those platforms. + + +This page contains resources that help you use Bazel to build macOS and iOS +projects. It links to a tutorial, build rules, and other information specific to +using Bazel to build and test for those platforms. ## Working with Bazel The following resources will help you work with Bazel on macOS and iOS projects: -- [Tutorial: Building an iOS app](/start/ios-app) -- [Objective-C build rules](/reference/be/objective-c) -- [General Apple rules](https://github.com/bazelbuild/rules_apple) -- [Integration with Xcode](/install/ide) +* [Tutorial: Building an iOS app](/start/ios-app) +* [Objective-C build rules](/reference/be/objective-c) +* [General Apple rules](https://github.com/bazelbuild/rules_apple) +* [Integration with Xcode](/install/ide) ## Migrating to Bazel -If you currently build your macOS and iOS projects with Xcode, follow the steps in the migration guide to start building them with Bazel: +If you currently build your macOS and iOS projects with Xcode, follow the steps +in the migration guide to start building them with Bazel: -- [Migrating from Xcode to Bazel](/migrate/xcode) +* [Migrating from Xcode to Bazel](/migrate/xcode) ## Apple apps and new rules -**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. +**Note**: Creating new rules is for advanced build and test scenarios. +You do not need it when getting started with Bazel. -The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your macOS and iOS projects: +The following modules, configuration fragments, and providers will help you +[extend Bazel's capabilities](/extending/concepts) +when building your macOS and iOS projects: -- Modules: +* Modules: - - [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) - - [`apple_common`](/rules/lib/toplevel/apple_common) - - [`apple_platform`](/rules/lib/builtins/apple_platform) - - [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) - - [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) + * [`apple_bitcode_mode`](/rules/lib/builtins/apple_bitcode_mode) + * [`apple_common`](/rules/lib/toplevel/apple_common) + * [`apple_platform`](/rules/lib/builtins/apple_platform) + * [`apple_platform_type`](/rules/lib/builtins/apple_platform_type) + * [`apple_toolchain`](/rules/lib/builtins/apple_toolchain) -- Configuration fragments: +* Configuration fragments: - - [`apple`](/rules/lib/fragments/apple) + * [`apple`](/rules/lib/fragments/apple) -- Providers: +* Providers: - - [`ObjcProvider`](/rules/lib/providers/ObjcProvider) - - [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) + * [`ObjcProvider`](/rules/lib/providers/ObjcProvider) + * [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) ## Xcode selection -If your build requires Xcode, Bazel will select an appropriate version based on the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes the set of available Xcode versions and sets a default version if `--xcode_version` is not passed. This default is overridden by the `--xcode_version` flag, as long as it is set to an Xcode version that is represented in the `--xcode_config` target. - -If you do not pass `--xcode_config`, Bazel will use the autogenerated [`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the Xcode versions available on your host machine. The default version is the newest available Xcode version. This is appropriate for local execution. - -If you are performing remote builds, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `versions` attribute is a list of remotely available [`xcode_version`](/reference/be/objective-c#xcode_version) targets, and whose `default` attribute is one of these [`xcode_versions`](/reference/be/objective-c#xcode_version). - -If you are using dynamic execution, you should set `--xcode_config` to an [`xcode_config`](/reference/be/objective-c#xcode_config) target whose `remote_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the remotely available Xcode versions, and whose `local_versions` attribute is an [`available_xcodes`](/reference/be/workspace#available_xcodes) target containing the locally available Xcode versions. For `local_versions`, you probably want to use the autogenerated `@local_config_xcode//:host_available_xcodes`. The default Xcode version is the newest mutually available version, if there is one, otherwise the default of the `local_versions` target. If you prefer to use the `local_versions` default as the default, you can pass `--experimental_prefer_mutual_default=false`. +If your build requires Xcode, Bazel will select an appropriate version based on +the `--xcode_config` and `--xcode_version` flags. The `--xcode_config` consumes +the set of available Xcode versions and sets a default version if +`--xcode_version` is not passed. This default is overridden by the +`--xcode_version` flag, as long as it is set to an Xcode version that is +represented in the `--xcode_config` target. + +If you do not pass `--xcode_config`, Bazel will use the autogenerated +[`XcodeVersionConfig`](/rules/lib/providers/XcodeVersionConfig) that represents the +Xcode versions available on your host machine. The default version is +the newest available Xcode version. This is appropriate for local execution. + +If you are performing remote builds, you should set `--xcode_config` to an +[`xcode_config`](/reference/be/objective-c#xcode_config) +target whose `versions` attribute is a list of remotely available +[`xcode_version`](/reference/be/objective-c#xcode_version) +targets, and whose `default` attribute is one of these +[`xcode_versions`](/reference/be/objective-c#xcode_version). + +If you are using dynamic execution, you should set `--xcode_config` to an +[`xcode_config`](/reference/be/objective-c#xcode_config) +target whose `remote_versions` attribute is an +[`available_xcodes`](/reference/be/workspace#available_xcodes) +target containing the remotely available Xcode versions, and whose +`local_versions` attribute is an +[`available_xcodes`](/reference/be/workspace#available_xcodes) +target containing the locally available Xcode versions. For `local_versions`, +you probably want to use the autogenerated +`@local_config_xcode//:host_available_xcodes`. The default Xcode version is the +newest mutually available version, if there is one, otherwise the default of the +`local_versions` target. If you prefer to use the `local_versions` default +as the default, you can pass `--experimental_prefer_mutual_default=false`. diff --git a/docs/bazel-and-cpp.mdx b/docs/bazel-and-cpp.mdx index 17ecb7c8..73e502cb 100644 --- a/docs/bazel-and-cpp.mdx +++ b/docs/bazel-and-cpp.mdx @@ -2,78 +2,101 @@ title: 'C++ and Bazel' --- -This page contains resources that help you use Bazel with C++ projects. It links to a tutorial, build rules, and other information specific to building C++ projects with Bazel. -## Working with Bazel - -The following resources will help you work with Bazel on C++ projects: - -- [Tutorial: Building a C++ project](/start/cpp) -- [C++ common use cases](/tutorials/cpp-use-cases) +This page contains resources that help you use Bazel with C++ projects. It links +to a tutorial, build rules, and other information specific to building C++ +projects with Bazel. -- [C/C++ rules](/reference/be/c-cpp) - -- Essential Libraries - - - [Abseil](https://abseil.io/docs/cpp/quickstart) - - [Boost](https://github.com/nelhage/rules_boost) - - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) - -- [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +## Working with Bazel -- [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) +The following resources will help you work with Bazel on C++ projects: -- [Integrating with C++ rules](/configure/integrate-cpp) +* [Tutorial: Building a C++ project](/start/cpp) +* [C++ common use cases](/tutorials/cpp-use-cases) +* [C/C++ rules](/reference/be/c-cpp) +* Essential Libraries + - [Abseil](https://abseil.io/docs/cpp/quickstart) + - [Boost](https://github.com/nelhage/rules_boost) + - [HTTPS Requests: CPR and libcurl](https://github.com/hedronvision/bazel-make-cc-https-easy) +* [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +* [Tutorial: Configuring C++ toolchains](/tutorials/ccp-toolchain-config) +* [Integrating with C++ rules](/configure/integrate-cpp) ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to C++ projects. +In addition to [general Bazel best practices](/configure/best-practices), below are +best practices specific to C++ projects. ### BUILD files Follow the guidelines below when creating your BUILD files: -- Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) rule target per compilation unit in the directory. - -- You should granularize your C++ libraries as much as possible to maximize incrementality and parallelize the build. - -- If there is a single source file in `srcs`, name the library the same as that C++ file's name. This library should contain C++ file(s), any matching header file(s), and the library's direct dependencies. For example: - - ```python - cc_library( - name = "mylib", - srcs = ["mylib.cc"], - hdrs = ["mylib.h"], - deps = [":lower-level-lib"] - ) - ``` - -- Use one `cc_test` rule target per `cc_library` target in the file. Name the target `[library-name]_test` and the source file `[library-name]_test.cc`. For example, a test target for the `mylib` library target shown above would look like this: - - ```python - cc_test( - name = "mylib_test", - srcs = ["mylib_test.cc"], - deps = [":mylib"] - ) - ``` +* Each `BUILD` file should contain one [`cc_library`](/reference/be/c-cpp#cc_library) + rule target per compilation unit in the directory. + +* You should granularize your C++ libraries as much as + possible to maximize incrementality and parallelize the build. + +* If there is a single source file in `srcs`, name the library the same as + that C++ file's name. This library should contain C++ file(s), any matching + header file(s), and the library's direct dependencies. For example: + + ```python + cc_library( + name = "mylib", + srcs = ["mylib.cc"], + hdrs = ["mylib.h"], + deps = [":lower-level-lib"] + ) + ``` + +* Use one `cc_test` rule target per `cc_library` target in the file. Name the + target `[library-name]_test` and the source file `[library-name]_test.cc`. + For example, a test target for the `mylib` library target shown above would + look like this: + + ```python + cc_test( + name = "mylib_test", + srcs = ["mylib_test.cc"], + deps = [":mylib"] + ) + ``` ### Include paths Follow these guidelines for include paths: -- Make all include paths relative to the workspace directory. +* Make all include paths relative to the workspace directory. -- Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not angle-brackets (`#include <foo/bar/baz.h>`). +* Use quoted includes (`#include "foo/bar/baz.h"`) for non-system headers, not + angle-brackets (`#include <foo/bar/baz.h>`). -- Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` (parent directory). +* Avoid using UNIX directory shortcuts, such as `.` (current directory) or `..` + (parent directory). -- For legacy or `third_party` code that requires includes pointing outside the project repository, such as external repository includes requiring a prefix, use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) arguments on the `cc_library` rule target. +* For legacy or `third_party` code that requires includes pointing outside the + project repository, such as external repository includes requiring a prefix, + use the [`include_prefix`](/reference/be/c-cpp#cc_library.include_prefix) and + [`strip_include_prefix`](/reference/be/c-cpp#cc_library.strip_include_prefix) + arguments on the `cc_library` rule target. ### Toolchain features -The following optional [features](/docs/cc-toolchain-config-reference#features) can improve the hygiene of a C++ project. They can be enabled using the `--features` command-line flag or the `features` attribute of [`repo`](/external/overview#repo.bazel), [`package`](/reference/be/functions#package) or `cc_*` rules: - -- The `parse_headers` feature makes it so that the C++ compiler is used to parse (but not compile) all header files in the built targets and their dependencies when using the [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) flag. This can help catch issues in header-only libraries and ensure that headers are self-contained and independent of the order in which they are included. -- The `layering_check` feature enforces that targets only include headers provided by their direct dependencies. The default toolchain supports this feature on Linux with `clang` as the compiler. +The following optional [features](/docs/cc-toolchain-config-reference#features) +can improve the hygiene of a C++ project. They can be enabled using the +`--features` command-line flag or the `features` attribute of +[`repo`](/external/overview#repo.bazel), +[`package`](/reference/be/functions#package) or `cc_*` rules: + +* The `parse_headers` feature makes it so that the C++ compiler is used to parse + (but not compile) all header files in the built targets and their dependencies + when using the + [`--process_headers_in_dependencies`](/reference/command-line-reference#flag--process_headers_in_dependencies) + flag. This can help catch issues in header-only libraries and ensure that + headers are self-contained and independent of the order in which they are + included. +* The `layering_check` feature enforces that targets only include headers + provided by their direct dependencies. The default toolchain supports this + feature on Linux with `clang` as the compiler. diff --git a/docs/bazel-and-java.mdx b/docs/bazel-and-java.mdx index 463fa6dd..bd5af010 100644 --- a/docs/bazel-and-java.mdx +++ b/docs/bazel-and-java.mdx @@ -2,120 +2,164 @@ title: 'Java and Bazel' --- -This page contains resources that help you use Bazel with Java projects. It links to a tutorial, build rules, and other information specific to building Java projects with Bazel. + + +This page contains resources that help you use Bazel with Java projects. It +links to a tutorial, build rules, and other information specific to building +Java projects with Bazel. ## Working with Bazel The following resources will help you work with Bazel on Java projects: -- [Tutorial: Building a Java Project](/start/java) -- [Java rules](/reference/be/java) +* [Tutorial: Building a Java Project](/start/java) +* [Java rules](/reference/be/java) ## Migrating to Bazel -If you currently build your Java projects with Maven, follow the steps in the migration guide to start building your Maven projects with Bazel: +If you currently build your Java projects with Maven, follow the steps in the +migration guide to start building your Maven projects with Bazel: -- [Migrating from Maven to Bazel](/migrate/maven) +* [Migrating from Maven to Bazel](/migrate/maven) ## Java versions There are two relevant versions of Java that are set with configuration flags: -- the version of the source files in the repository -- the version of the Java runtime that is used to execute the code and to test it +* the version of the source files in the repository +* the version of the Java runtime that is used to execute the code and to test + it ### Configuring the version of the source code in your repository -Without an additional configuration, Bazel assumes all Java source files in the repository are written in a single Java version. To specify the version of the sources in the repository add `build --java_language_version={ver}` to `.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners should set this flag so that Bazel and its users can reference the source code's Java version number. For more details, see [Java language version flag](/docs/user-manual#java-language-version). +Without an additional configuration, Bazel assumes all Java source files in the +repository are written in a single Java version. To specify the version of the +sources in the repository add `build --java_language_version={ver}` to +`.bazelrc` file, where `{ver}` is for example `11`. Bazel repository owners +should set this flag so that Bazel and its users can reference the source code's +Java version number. For more details, see +[Java language version flag](/docs/user-manual#java-language-version). ### Configuring the JVM used to execute and test the code Bazel uses one JDK for compilation and another JVM to execute and test the code. -By default Bazel compiles the code using a JDK it downloads and it executes and tests the code with the JVM installed on the local machine. Bazel searches for the JVM using `JAVA_HOME` or path. +By default Bazel compiles the code using a JDK it downloads and it executes and +tests the code with the JVM installed on the local machine. Bazel searches for +the JVM using `JAVA_HOME` or path. -The resulting binaries are compatible with locally installed JVM in system libraries, which means the resulting binaries depend on what is installed on the machine. +The resulting binaries are compatible with locally installed JVM in system +libraries, which means the resulting binaries depend on what is installed on the +machine. -To configure the JVM used for execution and testing use `--java_runtime_version` flag. The default value is `local_jdk`. +To configure the JVM used for execution and testing use `--java_runtime_version` +flag. The default value is `local_jdk`. ### Hermetic testing and compilation -To create a hermetic compile, you can use command line flag `--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and tested on the JVM downloaded from a remote repository. For more details, see [Java runtime version flag](/docs/user-manual#java_runtime_version). +To create a hermetic compile, you can use command line flag +`--java_runtime_version=remotejdk_11`. The code is compiled for, executed, and +tested on the JVM downloaded from a remote repository. For more details, see +[Java runtime version flag](/docs/user-manual#java_runtime_version). ### Configuring compilation and execution of build tools in Java -There is a second pair of JDK and JVM used to build and execute tools, which are used in the build process, but are not in the build results. That JDK and JVM are controlled using `--tool_java_language_version` and `--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, respectively. +There is a second pair of JDK and JVM used to build and execute tools, which are +used in the build process, but are not in the build results. That JDK and JVM +are controlled using `--tool_java_language_version` and +`--tool_java_runtime_version`. Default values are `11` and `remotejdk_11`, +respectively. #### Compiling using locally installed JDK -Bazel by default compiles using remote JDK, because it is overriding JDK's internals. The compilation toolchains using locally installed JDK are configured, however not used. +Bazel by default compiles using remote JDK, because it is overriding JDK's +internals. The compilation toolchains using locally installed JDK are configured, +however not used. -To compile using locally installed JDK, that is use the compilation toolchains for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, however, mind that this may not work on JDK of arbitrary vendors. +To compile using locally installed JDK, that is use the compilation toolchains +for local JDK, use additional flag `--extra_toolchains=@local_jdk//:all`, +however, mind that this may not work on JDK of arbitrary vendors. -For more details, see [configuring Java toolchains](#config-java-toolchains). +For more details, see +[configuring Java toolchains](#config-java-toolchains). ## Best practices -In addition to [general Bazel best practices](/configure/best-practices), below are best practices specific to Java projects. +In addition to [general Bazel best practices](/configure/best-practices), below are +best practices specific to Java projects. ### Directory structure -Prefer Maven's standard directory layout (sources under `src/main/java`, tests under `src/test/java`). +Prefer Maven's standard directory layout (sources under `src/main/java`, tests +under `src/test/java`). ### BUILD files Follow these guidelines when creating your `BUILD` files: -- Use one `BUILD` file per directory containing Java sources, because this improves build performance. +* Use one `BUILD` file per directory containing Java sources, because this + improves build performance. -- Every `BUILD` file should contain one `java_library` rule that looks like this: +* Every `BUILD` file should contain one `java_library` rule that looks like + this: - ```python - java_library( - name = "directory-name", - srcs = glob(["*.java"]), - deps = [...], - ) - ``` + ```python + java_library( + name = "directory-name", + srcs = glob(["*.java"]), + deps = [...], + ) + ``` -- The name of the library should be the name of the directory containing the `BUILD` file. This makes the label of the library shorter, that is use `"//package"` instead of `"//package:package"`. +* The name of the library should be the name of the directory containing the + `BUILD` file. This makes the label of the library shorter, that is use + `"//package"` instead of `"//package:package"`. -- The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of all Java files in the directory. +* The sources should be a non-recursive [`glob`](/reference/be/functions#glob) of + all Java files in the directory. -- Tests should be in a matching directory under `src/test` and depend on this library. +* Tests should be in a matching directory under `src/test` and depend on this + library. ## Creating new rules for advanced Java builds -**Note**: Creating new rules is for advanced build and test scenarios. You do not need it when getting started with Bazel. - -The following modules, configuration fragments, and providers will help you [extend Bazel's capabilities](/extending/concepts) when building your Java projects: - -- Main Java module: [`java_common`](/rules/lib/toplevel/java_common) - -- Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) +**Note**: Creating new rules is for advanced build and test scenarios. You do +not need it when getting started with Bazel. -- Configuration fragment: [`java`](/rules/lib/fragments/java) +The following modules, configuration fragments, and providers will help you +[extend Bazel's capabilities](/extending/concepts) when building your Java +projects: -- Other modules: +* Main Java module: [`java_common`](/rules/lib/toplevel/java_common) +* Main Java provider: [`JavaInfo`](/rules/lib/providers/JavaInfo) +* Configuration fragment: [`java`](/rules/lib/fragments/java) +* Other modules: - - [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) - - [`java_compilation_info`](/rules/lib/providers/java_compilation_info) - - [`java_output_jars`](/rules/lib/providers/java_output_jars) - - [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) - - [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) + * [`java_annotation_processing`](/rules/lib/builtins/java_annotation_processing) + * [`java_compilation_info`](/rules/lib/providers/java_compilation_info) + * [`java_output_jars`](/rules/lib/providers/java_output_jars) + * [`JavaRuntimeInfo`](/rules/lib/providers/JavaRuntimeInfo) + * [`JavaToolchainInfo`](/rules/lib/providers/JavaToolchainInfo) ## Configuring the Java toolchains Bazel uses two types of Java toolchains: -- execution, used to execute and test Java binaries, controlled with the `--java_runtime_version` flag -- compilation, used to compile Java sources, controlled with the `--java_language_version` flag +* execution, used to execute and test Java binaries, controlled with the + `--java_runtime_version` flag +* compilation, used to compile Java sources, controlled with the + `--java_language_version` flag ### Configuring additional execution toolchains -Execution toolchain is the JVM, either local or from a repository, with some additional information about its version, operating system, and CPU architecture. +Execution toolchain is the JVM, either local or from a repository, with some +additional information about its version, operating system, and CPU +architecture. -Java execution toolchains may added using the `local_java_repository` or `remote_java_repository` repo rules in a module extension. Adding the rule makes the JVM available using a flag. When multiple definitions for the same operating system and CPU architecture are given, the first one is used. +Java execution toolchains may added using the `local_java_repository` or +`remote_java_repository` repo rules in a module extension. Adding the rule makes +the JVM available using a flag. When multiple definitions for the same operating +system and CPU architecture are given, the first one is used. Example configuration of local JVM in MODULE.bazel: @@ -155,29 +199,51 @@ register_toolchains("@openjdk_canary_linux_arm_toolchain_config_repo//:all") ### Configuring additional compilation toolchains -Compilation toolchain is composed of JDK and multiple tools that Bazel uses during the compilation and that provides additional features, such as: Error Prone, strict Java dependencies, header compilation, Android desugaring, coverage instrumentation, and genclass handling for IDEs. +Compilation toolchain is composed of JDK and multiple tools that Bazel uses +during the compilation and that provides additional features, such as: Error +Prone, strict Java dependencies, header compilation, Android desugaring, +coverage instrumentation, and genclass handling for IDEs. -JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the aforementioned features. Actual compilation is executed using the internal compiler by the JDK. The JDK used for compilation is specified by `java_runtime` attribute of the toolchain. +JavaBuilder is a Bazel-bundled tool that executes compilation, and provides the +aforementioned features. Actual compilation is executed using the internal +compiler by the JDK. The JDK used for compilation is specified by `java_runtime` +attribute of the toolchain. -Bazel overrides some JDK internals. In case of JDK version > 9, `java.compiler` and `jdk.compiler` modules are patched using JDK's flag `--patch_module`. In case of JDK version 8, the Java compiler is patched using `-Xbootclasspath` flag. +Bazel overrides some JDK internals. In case of JDK version > 9, +`java.compiler` and `jdk.compiler` modules are patched using JDK's flag +`--patch_module`. In case of JDK version 8, the Java compiler is patched using +`-Xbootclasspath` flag. -VanillaJavaBuilder is a second implementation of JavaBuilder, which does not modify JDK's internal compiler and does not have any of the additional features. VanillaJavaBuilder is not used by any of the built-in toolchains. +VanillaJavaBuilder is a second implementation of JavaBuilder, +which does not modify JDK's internal compiler and does not have any of the +additional features. VanillaJavaBuilder is not used by any of the built-in +toolchains. In addition to JavaBuilder, Bazel uses several other tools during compilation. -The `ijar` tool processes `jar` files to remove everything except call signatures. Resulting jars are called header jars. They are used to improve the compilation incrementality by only recompiling downstream dependents when the body of a function changes. +The `ijar` tool processes `jar` files to remove everything except call +signatures. Resulting jars are called header jars. They are used to improve the +compilation incrementality by only recompiling downstream dependents when the +body of a function changes. The `singlejar` tool packs together multiple `jar` files into a single one. -The `genclass` tool post-processes the output of a Java compilation, and produces a `jar` containing only the class files for sources that were generated by annotation processors. +The `genclass` tool post-processes the output of a Java compilation, and produces +a `jar` containing only the class files for sources that were generated by +annotation processors. -The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in LCOV format. +The `JacocoRunner` tool runs Jacoco over instrumented files and outputs results in +LCOV format. The `TestRunner` tool executes JUnit 4 tests in a controlled environment. -You can reconfigure the compilation by adding `default_java_toolchain` macro to a `BUILD` file and registering it either by adding `register_toolchains` rule to the `MODULE.bazel` file or by using [`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. +You can reconfigure the compilation by adding `default_java_toolchain` macro to +a `BUILD` file and registering it either by adding `register_toolchains` rule to +the `MODULE.bazel` file or by using +[`--extra_toolchains`](/docs/user-manual#extra-toolchains) flag. -The toolchain is only used when the `source_version` attribute matches the value specified by `--java_language_version` flag. +The toolchain is only used when the `source_version` attribute matches the +value specified by `--java_language_version` flag. Example toolchain configuration: @@ -198,26 +264,37 @@ default_java_toolchain( ) ``` -which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` or by adding `register_toolchains("//:repository_default_toolchain_definition")` to the workpace. +which can be used using `--extra_toolchains=//:repository_default_toolchain_definition` +or by adding `register_toolchains("//:repository_default_toolchain_definition")` +to the workpace. Predefined configurations: -- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 -- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of arbitrary vendors. -- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt tools (`ijar`, `singlejar`) -- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are built from sources (this may be useful on operating system with different libc) +- `DEFAULT_TOOLCHAIN_CONFIGURATION`: all features, supports JDK versions >= 9 +- `VANILLA_TOOLCHAIN_CONFIGURATION`: no additional features, supports JDKs of + arbitrary vendors. +- `PREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but only use prebuilt + tools (`ijar`, `singlejar`) +- `NONPREBUILT_TOOLCHAIN_CONFIGURATION`: same as default, but all tools are + built from sources (this may be useful on operating system with different + libc) #### Configuring JVM and Java compiler flags -You may configure JVM and javac flags either with flags or with `default_java_toolchain` attributes. +You may configure JVM and javac flags either with flags or with + `default_java_toolchain` attributes. -The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and `--host_javacopt`. +The relevant flags are `--jvmopt`, `--host_jvmopt`, `--javacopt`, and +`--host_javacopt`. -The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, `javabuilder_jvm_opts`, and `turbine_jvm_opts`. +The relevant `default_java_toolchain` attributes are `javacopts`, `jvm_opts`, +`javabuilder_jvm_opts`, and `turbine_jvm_opts`. #### Package specific Java compiler flags configuration -You can configure different Java compiler flags for specific source files using `package_configuration` attribute of `default_java_toolchain`. Please refer to the example below. +You can configure different Java compiler flags for specific source +files using `package_configuration` attribute of `default_java_toolchain`. +Please refer to the example below. ```python load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolchain") @@ -252,11 +329,14 @@ package_group( #### Multiple versions of Java source code in a single repository -Bazel only supports compiling a single version of Java sources in a build. build. This means that when building a Java test or an application, all dependencies are built against the same Java version. +Bazel only supports compiling a single version of Java sources in a build. +build. This means that when building a Java test or an application, all + dependencies are built against the same Java version. However, separate builds may be executed using different flags. -To make the task of using different flags easier, sets of flags for a specific version may be grouped with `.bazelrc` configs": +To make the task of using different flags easier, sets of flags for a specific +version may be grouped with `.bazelrc` configs": ```python build:java8 --java_language_version=8 @@ -265,4 +345,5 @@ build:java11 --java_language_version=11 build:java11 --java_runtime_version=remotejdk_11 ``` -These configs can be used with the `--config` flag, for example `bazel test --config=java11 //:java11_test`. +These configs can be used with the `--config` flag, for example +`bazel test --config=java11 //:java11_test`. diff --git a/docs/bazel-and-javascript.mdx b/docs/bazel-and-javascript.mdx index 4a2c176a..63d80189 100644 --- a/docs/bazel-and-javascript.mdx +++ b/docs/bazel-and-javascript.mdx @@ -2,19 +2,23 @@ title: 'JavaScript and Bazel' --- -This page contains resources that help you use Bazel with JavaScript projects. It links to build rules and other information specific to building JavaScript with Bazel. + + +This page contains resources that help you use Bazel with JavaScript projects. +It links to build rules and other information specific to building JavaScript +with Bazel. The following resources will help you work with Bazel on JavaScript projects: -- [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) -- [rules\_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs -- [rules\_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler -- [rules\_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier -- [rules\_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) -- [rules\_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) -- [rules\_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) -- [rules\_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler -- [rules\_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) -- [rules\_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) -- [rules\_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) -- [rules\_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) +* [NodeJS toolchain](https://github.com/bazelbuild/rules_nodejs) +* [rules_js](https://github.com/aspect-build/rules_js) - Bazel rules for building JavaScript programs +* [rules_esbuild](https://github.com/aspect-build/rules_esbuild) - Bazel rules for [esbuild](https://esbuild.github.io) JS bundler +* [rules_terser](https://github.com/aspect-build/rules_terser) - Bazel rules for [Terser](https://terser.org) - a JavaScript minifier +* [rules_swc](https://github.com/aspect-build/rules_swc) - Bazel rules for [swc](https://swc.rs) +* [rules_ts](https://github.com/aspect-build/rules_ts) - Bazel rules for [TypeScript](http://typescriptlang.org) +* [rules_webpack](https://github.com/aspect-build/rules_webpack) - Bazel rules for [Webpack](https://webpack.js.org) +* [rules_rollup](https://github.com/aspect-build/rules_rollup) - Bazel rules for [Rollup](https://rollupjs.org) - a JavaScript bundler +* [rules_jest](https://github.com/aspect-build/rules_jest) - Bazel rules to run tests using [Jest](https://jestjs.io) +* [rules_jasmine](https://github.com/aspect-build/rules_jasmine) - Bazel rules to run tests using [Jasmine](https://jasmine.github.io/) +* [rules_cypress](https://github.com/aspect-build/rules_cypress) - Bazel rules to run tests using [Cypress](https://cypress.io) +* [rules_deno](https://github.com/aspect-build/rules_deno) - Bazel rules for [Deno](http://deno.land) diff --git a/docs/cc-toolchain-config-reference.mdx b/docs/cc-toolchain-config-reference.mdx deleted file mode 100644 index d000204c..00000000 --- a/docs/cc-toolchain-config-reference.mdx +++ /dev/null @@ -1,488 +0,0 @@ ---- -title: 'C++ Toolchain Configuration' ---- - -## Overview - -To invoke the compiler with the right options, Bazel needs some knowledge about the compiler internals, such as include directories and important flags. In other words, Bazel needs a simplified model of the compiler to understand its workings. - -Bazel needs to know the following: - -- Whether the compiler supports thinLTO, modules, dynamic linking, or PIC (position independent code). -- Paths to the required tools such as gcc, ld, ar, objcopy, and so on. -- The built-in system include directories. Bazel needs these to validate that all headers that were included in the source file were properly declared in the `BUILD` file. -- The default sysroot. -- Which flags to use for compilation, linking, archiving. -- Which flags to use for the supported compilation modes (opt, dbg, fastbuild). -- Make variables specifically required by the compiler. - -If the compiler has support for multiple architectures, Bazel needs to configure them separately. - -[`CcToolchainConfigInfo`](/rules/lib/providers/CcToolchainConfigInfo) is a provider that provides the necessary level of granularity for configuring the behavior of Bazel's C++ rules. By default, Bazel automatically configures `CcToolchainConfigInfo` for your build, but you have the option to configure it manually. For that, you need a Starlark rule that provides the `CcToolchainConfigInfo` and you need to point the [`toolchain_config`](/reference/be/c-cpp#cc_toolchain.toolchain_config) attribute of the [`cc_toolchain`](/reference/be/c-cpp#cc_toolchain) to your rule. You can create the `CcToolchainConfigInfo` by calling [`cc_common.create_cc_toolchain_config_info()`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info). You can find Starlark constructors for all structs you'll need in the process in [`@rules_cc//cc:cc_toolchain_config_lib.bzl`](https://github.com/bazelbuild/rules_cc/blob/master/cc/cc_toolchain_config_lib.bzl). - -When a C++ target enters the analysis phase, Bazel selects the appropriate `cc_toolchain` target based on the `BUILD` file, and obtains the `CcToolchainConfigInfo` provider from the target specified in the `cc_toolchain.toolchain_config` attribute. The `cc_toolchain` target passes this information to the C++ target through a `CcToolchainProvider`. - -For example, a compile or link action, instantiated by a rule such as `cc_binary` or `cc_library`, needs the following information: - -- The compiler or linker to use -- Command-line flags for the compiler/linker -- Configuration flags passed through the `--copt/--linkopt` options -- Environment variables -- Artifacts needed in the sandbox in which the action executes - -All of the above information except the artifacts required in the sandbox is specified in the Starlark target that the `cc_toolchain` points to. - -The artifacts to be shipped to the sandbox are declared in the `cc_toolchain` target. For example, with the `cc_toolchain.linker_files` attribute you can specify the linker binary and toolchain libraries to ship into the sandbox. - -## Toolchain selection - -The toolchain selection logic operates as follows: - -1. User specifies a `cc_toolchain_suite` target in the `BUILD` file and points Bazel to the target using the [`--crosstool_top` option](/docs/user-manual#flag--crosstool_top). - -2. The `cc_toolchain_suite` target references multiple toolchains. The values of the `--cpu` and `--compiler` flags determine which of those toolchains is selected, either based only on the `--cpu` flag value, or based on a joint `--cpu | --compiler` value. The selection process is as follows: - -- If the `--compiler` option is specified, Bazel selects the corresponding entry from the `cc_toolchain_suite.toolchains` attribute with `--cpu | --compiler`. If Bazel does not find a corresponding entry, it throws an error. - -- If the `--compiler` option is not specified, Bazel selects the corresponding entry from the `cc_toolchain_suite.toolchains` attribute with just `--cpu`. - -- If no flags are specified, Bazel inspects the host system and selects a `--cpu` value based on its findings. See the [inspection mechanism code](https://source.bazel.build/bazel/+/1b73bc37e184e71651eb631223dcce321ba16211:src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java). - -Once a toolchain has been selected, corresponding `feature` and `action_config` objects in the Starlark rule govern the configuration of the build (that is, items described later). These messages allow the implementation of fully fledged C++ features in Bazel without modifying the Bazel binary. C++ rules support multiple unique actions documented in detail [in the Bazel source code](https://source.bazel.build/bazel/+/4f547a7ea86df80e4c76145ffdbb0c8b75ba3afa:tools/build_defs/cc/action_names.bzl). - -## Features - -A feature is an entity that requires command-line flags, actions, constraints on the execution environment, or dependency alterations. A feature can be something as simple as allowing `BUILD` files to select configurations of flags, such as `treat_warnings_as_errors`, or interact with the C++ rules and include new compile actions and inputs to the compilation, such as `header_modules` or `thin_lto`. - -Ideally, `CcToolchainConfigInfo` contains a list of features, where each feature consists of one or more flag groups, each defining a list of flags that apply to specific Bazel actions. - -A feature is specified by name, which allows full decoupling of the Starlark rule configuration from Bazel releases. In other words, a Bazel release does not affect the behavior of `CcToolchainConfigInfo` configurations as long as those configurations do not require the use of new features. - -A feature is enabled in one of the following ways: - -- The feature's `enabled` field is set to `true`. -- Bazel or the rule owner explicitly enable it. -- The user enables it through the `--feature` Bazel option or `features` rule attribute. - -Features can have interdependencies, depend on command line flags, `BUILD` file settings, and other variables. - -### Feature relationships - -Dependencies are typically managed directly with Bazel, which simply enforces the requirements and manages conflicts intrinsic to the nature of the features defined in the build. The toolchain specification allows for more granular constraints for use directly within the Starlark rule that govern feature support and expansion. These are: - -| | | -| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Constraint** | **Description** | -| ``` -requires = [ - feature_set (features = [ - 'feature-name-1', - 'feature-name-2' - ]), -] -``` | Feature-level. The feature is supported only if the specified required features are enabled. For example, when a feature is only supported in certain build modes (`opt`, `dbg`, or `fastbuild`). If \`requires\` contains multiple \`feature\_set\`s the feature is supported if any of the \`feature\_set\`s is satisfied (when all specified features are enabled). | -| ``` -implies = ['feature'] -``` | Feature-level. This feature implies the specified feature(s). Enabling a feature also implicitly enables all features implied by it (that is, it functions recursively).Also provides the ability to factor common subsets of functionality out of a set of features, such as the common parts of sanitizers. Implied features cannot be disabled. | -| ``` -provides = ['feature'] -``` | Feature-level. Indicates that this feature is one of several mutually exclusive alternate features. For example, all of the sanitizers could specify `provides = ["sanitizer"]`.This improves error handling by listing the alternatives if the user asks for two or more mutually exclusive features at once. | -| ``` -with_features = [ - with_feature_set( - features = ['feature-1'], - not_features = ['feature-2'], - ), -] -``` | Flag set-level. A feature can specify multiple flag sets with multiple. When `with_features` is specified, the flag set will only expand to the build command if there is at least one `with_feature_set` for which all of the features in the specified `features` set are enabled, and all the features specified in `not_features` set are disabled. If `with_features` is not specified, the flag set will be applied unconditionally for every action specified. | - -## Actions - -Actions provide the flexibility to modify the circumstances under which an action executes without assuming how the action will be run. An `action_config` specifies the tool binary that an action invokes, while a `feature` specifies the configuration (flags) that determine how that tool behaves when the action is invoked. - -[Features](#features) reference actions to signal which Bazel actions they affect since actions can modify the Bazel action graph. The `CcToolchainConfigInfo` provider contains actions that have flags and tools associated with them, such as `c++-compile`. Flags are assigned to each action by associating them with a feature. - -Each action name represents a single type of action performed by Bazel, such as compiling or linking. There is, however, a many-to-one relationship between actions and Bazel action types, where a Bazel action type refers to a Java class that implements an action (such as `CppCompileAction`). In particular, the "assembler actions" and "compiler actions" in the table below are `CppCompileAction`, while the link actions are `CppLinkAction`. - -### Assembler actions - -| | | -| --------------------- | --------------------------------------------------------- | -| **Action** | **Description** | -| `preprocess-assemble` | Assemble with preprocessing. Typically for `.S` files. | -| `assemble` | Assemble without preprocessing. Typically for `.s` files. | - -### Compiler actions - -| | | -| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Action** | **Description** | -| `cc-flags-make-variable` | Propagates `CC_FLAGS` to genrules. | -| `c-compile` | Compile as C. | -| `c++-compile` | Compile as C++. | -| `c++-header-parsing` | Run the compiler's parser on a header file to ensure that the header is self-contained, as it will otherwise produce compilation errors. Applies only to toolchains that support modules. | - -### Link actions - -| | | -| --------------------------------- | ----------------------------------------------------------- | -| **Action** | **Description** | -| `c++-link-dynamic-library` | Link a shared library containing all of its dependencies. | -| `c++-link-nodeps-dynamic-library` | Link a shared library only containing `cc_library` sources. | -| `c++-link-executable` | Link a final ready-to-run library. | - -### AR actions - -AR actions assemble object files into archive libraries (`.a` files) via `ar` and encode some semantics into the name. - -| | | -| ------------------------- | ---------------------------------- | -| **Action** | **Description** | -| `c++-link-static-library` | Create a static library (archive). | - -### LTO actions - -| | | -| ------------- | ------------------------------------------------------ | -| **Action** | **Description** | -| `lto-backend` | ThinLTO action compiling bitcodes into native objects. | -| `lto-index` | ThinLTO action generating global index. | - -## Using action\_config - -The `action_config` is a Starlark struct that describes a Bazel action by specifying the tool (binary) to invoke during the action and sets of flags, defined by features. These flags apply constraints to the action's execution. - -The `action_config()` constructor has the following parameters: - -| | | -| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Attribute** | **Description** | -| `action_name` | The Bazel action to which this action corresponds. Bazel uses this attribute to discover per-action tool and execution requirements. | -| `tools` | The executable to invoke. The tool applied to the action will be the first tool in the list with a feature set that matches the feature configuration. Default value must be provided. | -| `flag_sets` | A list of flags that applies to a group of actions. Same as for a feature. | -| `env_sets` | A list of environment constraints that applies to a group of actions. Same as for a feature. | - -An `action_config` can require and imply other features and `action_config`s as dictated by the [feature relationships](#feature-relationships) described earlier. This behavior is similar to that of a feature. - -The last two attributes are redundant against the corresponding attributes on features and are included because some Bazel actions require certain flags or environment variables and the goal is to avoid unnecessary `action_config`+`feature` pairs. Typically, sharing a single feature across multiple `action_config`s is preferred. - -You can not define more than one `action_config` with the same `action_name` within the same toolchain. This prevents ambiguity in tool paths and enforces the intention behind `action_config` - that an action's properties are clearly described in a single place in the toolchain. - -### Using tool constructor - -An`action_config` can specify a set of tools via its `tools` parameter. The `tool()` constructor takes in the following parameters: - -| | | -| --------------- | ------------------------------------------------------------------------------------------ | -| **Field** | **Description** | -| `path` | Path to the tool in question (relative to the current location). | -| `with_features` | A list of feature sets out of which at least one must be satisfied for this tool to apply. | - -For a given `action_config`, only a single `tool` applies its tool path and execution requirements to the Bazel action. A tool is selected by iterating through the `tools` attribute on an `action_config` until a tool with a `with_feature` set matching the feature configuration is found (see [Feature relationships](#feature-relationships) earlier on this page for more information). You should end your tool lists with a default tool that corresponds to an empty feature configuration. - -### Example usage - -Features and actions can be used together to implement Bazel actions with diverse cross-platform semantics. For example, debug symbol generation on macOS requires generating symbols in the compile action, then invoking a specialized tool during the link action to create compressed dsym archive, and then decompressing that archive to produce the application bundle and `.plist` files consumable by Xcode. - -With Bazel, this process can instead be implemented as follows, with `unbundle-debuginfo` being a Bazel action: - -``` -load("@rules_cc//cc:defs.bzl", "ACTION_NAMES") - -action_configs = [ - action_config ( - action_name = ACTION_NAMES.cpp_link_executable, - tools = [ - tool( - with_features = [ - with_feature(features=["generate-debug-symbols"]), - ], - path = "toolchain/mac/ld-with-dsym-packaging", - ), - tool (path = "toolchain/mac/ld"), - ], - ), -] - -features = [ - feature( - name = "generate-debug-symbols", - flag_sets = [ - flag_set ( - actions = [ - ACTION_NAMES.c_compile, - ACTION_NAMES.cpp_compile - ], - flag_groups = [ - flag_group( - flags = ["-g"], - ), - ], - ) - ], - implies = ["unbundle-debuginfo"], - ), -] -``` - -This same feature can be implemented entirely differently for Linux, which uses `fission`, or for Windows, which produces `.pdb` files. For example, the implementation for `fission`-based debug symbol generation might look as follows: - -``` -load("@rules_cc//cc:defs.bzl", "ACTION_NAMES") - -action_configs = [ - action_config ( - name = ACTION_NAMES.cpp_compile, - tools = [ - tool( - path = "toolchain/bin/gcc", - ), - ], - ), -] - -features = [ - feature ( - name = "generate-debug-symbols", - requires = [with_feature_set(features = ["dbg"])], - flag_sets = [ - flag_set( - actions = [ACTION_NAMES.cpp_compile], - flag_groups = [ - flag_group( - flags = ["-gsplit-dwarf"], - ), - ], - ), - flag_set( - actions = [ACTION_NAMES.cpp_link_executable], - flag_groups = [ - flag_group( - flags = ["-Wl", "--gdb-index"], - ), - ], - ), - ], - ), -] -``` - -### Flag groups - -`CcToolchainConfigInfo` allows you to bundle flags into groups that serve a specific purpose. You can specify a flag within using pre-defined variables within the flag value, which the compiler expands when adding the flag to the build command. For example: - -``` -flag_group ( - flags = ["%{output_execpath}"], -) -``` - -In this case, the contents of the flag will be replaced by the output file path of the action. - -Flag groups are expanded to the build command in the order in which they appear in the list, top-to-bottom, left-to-right. - -For flags that need to repeat with different values when added to the build command, the flag group can iterate variables of type `list`. For example, the variable `include_path` of type `list`: - -``` -flag_group ( - iterate_over = "include_paths", - flags = ["-I%{include_paths}"], -) -``` - -expands to `-I<path>` for each path element in the `include_paths` list. All flags (or `flag_group`s) in the body of a flag group declaration are expanded as a unit. For example: - -``` -flag_group ( - iterate_over = "include_paths", - flags = ["-I", "%{include_paths}"], -) -``` - -expands to `-I <path>` for each path element in the `include_paths` list. - -A variable can repeat multiple times. For example: - -``` -flag_group ( - iterate_over = "include_paths", - flags = ["-iprefix=%{include_paths}", "-isystem=%{include_paths}"], -) -``` - -expands to: - -``` --iprefix=<inc0> -isystem=<inc0> -iprefix=<inc1> -isystem=<inc1> -``` - -Variables can correspond to structures accessible using dot-notation. For example: - -``` -flag_group ( - flags = ["-l%{libraries_to_link.name}"], -) -``` - -Structures can be nested and may also contain sequences. To prevent name clashes and to be explicit, you must specify the full path through the fields. For example: - -``` -flag_group ( - iterate_over = "libraries_to_link", - flag_groups = [ - flag_group ( - iterate_over = "libraries_to_link.shared_libraries", - flags = ["-l%{libraries_to_link.shared_libraries.name}"], - ), - ], -) -``` - -### Conditional expansion - -Flag groups support conditional expansion based on the presence of a particular variable or its field using the `expand_if_available`, `expand_if_not_available`, `expand_if_true`, `expand_if_false`, or `expand_if_equal` attributes. For example: - -``` -flag_group ( - iterate_over = "libraries_to_link", - flag_groups = [ - flag_group ( - iterate_over = "libraries_to_link.shared_libraries", - flag_groups = [ - flag_group ( - expand_if_available = "libraries_to_link.shared_libraries.is_whole_archive", - flags = ["--whole_archive"], - ), - flag_group ( - flags = ["-l%{libraries_to_link.shared_libraries.name}"], - ), - flag_group ( - expand_if_available = "libraries_to_link.shared_libraries.is_whole_archive", - flags = ["--no_whole_archive"], - ), - ], - ), - ], -) -``` - -Note: The `--whole_archive` and `--no_whole_archive` options are added to the build command only when a currently iterated library has an `is_whole_archive` field. - -## CcToolchainConfigInfo reference - -This section provides a reference of build variables, features, and other information required to successfully configure C++ rules. - -### CcToolchainConfigInfo build variables - -The following is a reference of `CcToolchainConfigInfo` build variables. - -Note: The **Action** column indicates the relevant action type, if applicable. - -| | | | -| ---------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Variable** | **Action** | **Description** | -| **`source_file`** | compile | Source file to compile. | -| **`input_file`** | strip | Artifact to strip. | -| **`output_file`** | compile, strip | Compilation output. | -| **`output_assembly_file`** | compile | Emitted assembly file. Applies only when the `compile` action emits assembly text, typically when using the `--save_temps` flag. The contents are the same as for `output_file`. | -| **`output_preprocess_file`** | compile | Preprocessed output. Applies only to compile actions that only preprocess the source files, typically when using the `--save_temps` flag. The contents are the same as for `output_file`. | -| **`includes`** | compile | Sequence of files the compiler must unconditionally include in the compiled source. | -| **`include_paths`** | compile | Sequence directories in which the compiler searches for headers included using `#include<foo.h>` and `#include "foo.h"`. | -| **`quote_include_paths`** | compile | Sequence of `-iquote` includes - directories in which the compiler searches for headers included using `#include "foo.h"`. | -| **`system_include_paths`** | compile | Sequence of `-isystem` includes - directories in which the compiler searches for headers included using `#include <foo.h>`. | -| **`dependency_file`** | compile | The `.d` dependency file generated by the compiler. | -| **`preprocessor_defines`** | compile | Sequence of `defines`, such as `--DDEBUG`. | -| **`pic`** | compile | Compiles the output as position-independent code. | -| **`gcov_gcno_file`** | compile | The `gcov` coverage file. | -| **`per_object_debug_info_file`** | compile | The per-object debug info (`.dwp`) file. | -| **`stripopts`** | strip | Sequence of `stripopts`. | -| **`legacy_compile_flags`** | compile | Sequence of flags from legacy `CROSSTOOL` fields such as `compiler_flag`, `optional_compiler_flag`, `cxx_flag`, and `optional_cxx_flag`. | -| **`user_compile_flags`** | compile | Sequence of flags from either the `copt` rule attribute or the `--copt`, `--cxxopt`, and `--conlyopt` flags. | -| **`unfiltered_compile_flags`** | compile | Sequence of flags from the `unfiltered_cxx_flag` legacy `CROSSTOOL` field or the `unfiltered_compile_flags` feature. These are not filtered by the `nocopts` rule attribute. | -| **`sysroot`** | | The `sysroot`. | -| **`runtime_library_search_directories`** | link | Entries in the linker runtime search path (usually set with the `-rpath` flag). | -| **`library_search_directories`** | link | Entries in the linker search path (usually set with the `-L` flag). | -| **`libraries_to_link`** | link | Flags providing files to link as inputs in the linker invocation. | -| **`def_file_path`** | link | Location of def file used on Windows with MSVC. | -| **`linker_param_file`** | link | Location of linker param file created by bazel to overcome command line length limit. | -| **`output_execpath`** | link | Execpath of the output of the linker. | -| **`generate_interface_library`** | link | `"yes"` or `"no"` depending on whether interface library should be generated. | -| **`interface_library_builder_path`** | link | Path to the interface library builder tool. | -| **`interface_library_input_path`** | link | Input for the interface library `ifso` builder tool. | -| **`interface_library_output_path`** | link | Path where to generate interface library using the `ifso` builder tool. | -| **`legacy_link_flags`** | link | Linker flags coming from the legacy `CROSSTOOL` fields. | -| **`user_link_flags`** | link | Linker flags coming from the `--linkopt` or `linkopts` attribute. | -| **`linkstamp_paths`** | link | A build variable giving linkstamp paths. | -| **`force_pic`** | link | Presence of this variable indicates that PIC/PIE code should be generated (Bazel option \`--force\_pic\` was passed). | -| **`strip_debug_symbols`** | link | Presence of this variable indicates that the debug symbols should be stripped. | -| **`is_cc_test`** | link | Truthy when current action is a `cc_test` linking action, false otherwise. | -| **`is_using_fission`** | compile, link | Presence of this variable indicates that fission (per-object debug info) is activated. Debug info will be in `.dwo` files instead of `.o` files and the compiler and linker need to know this. | -| **`fdo_instrument_path`** | compile, link | Path to the directory that stores FDO instrumentation profile. | -| **`fdo_profile_path`** | compile | Path to FDO profile. | -| **`fdo_prefetch_hints_path`** | compile | Path to the cache prefetch profile. | -| **`cs_fdo_instrument_path`** | compile, link | Path to the directory that stores context sensitive FDO instrumentation profile. | - -### Well-known features - -The following is a reference of features and their activation conditions. - -| | | -| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Feature** | **Documentation** | -| **`opt \| dbg \| fastbuild`** | Enabled by default based on compilation mode. | -| **`static_linking_mode \| dynamic_linking_mode`** | Enabled by default based on linking mode. | -| **`per_object_debug_info`** | Enabled if the `supports_fission` feature is specified and enabled and the current compilation mode is specified in the `--fission` flag. | -| **`supports_start_end_lib`** | If enabled (and the option `--start_end_lib` is set), Bazel will not link against static libraries but instead use the `--start-lib/--end-lib` linker options to link against objects directly. This speeds up the build since Bazel doesn't have to build static libraries. | -| **`supports_interface_shared_libraries`** | If enabled (and the option `--interface_shared_objects` is set), Bazel will link targets that have `linkstatic` set to False (`cc_test`s by default) against interface shared libraries. This makes incremental relinking faster. | -| **`supports_dynamic_linker`** | If enabled, C++ rules will know the toolchain can produce shared libraries. | -| **`static_link_cpp_runtimes`** | If enabled, Bazel will link the C++ runtime statically in static linking mode and dynamically in dynamic linking mode. Artifacts specified in the `cc_toolchain.static_runtime_lib` or `cc_toolchain.dynamic_runtime_lib` attribute (depending on the linking mode) will be added to the linking actions. | -| **`supports_pic`** | If enabled, toolchain will know to use PIC objects for dynamic libraries. The \`pic\` variable is present whenever PIC compilation is needed. If not enabled by default, and \`--force\_pic\` is passed, Bazel will request \`supports\_pic\` and validate that the feature is enabled. If the feature is missing, or couldn't be enabled, \`--force\_pic\` cannot be used. | -| **`static_linking_mode \| dynamic_linking_mode`** | Enabled by default based on linking mode. | -| **`no_legacy_features`** | Prevents Bazel from adding legacy features to the C++ configuration when present. See the complete list of features below. | -| **`shorten_virtual_includes`** | If enabled, virtual include header files are linked under `bin/_virtual_includes/<hash of target path>` instead of `bin/<target package path>/_virtual_includes/<target name>`. Useful on Windows to avoid long path issue with MSVC. | - -#### Legacy features patching logic - -Bazel applies the following changes to the toolchain's features for backwards compatibility: - -- Moves `legacy_compile_flags` feature to the top of the toolchain -- Moves `default_compile_flags` feature to the top of the toolchain -- Adds `dependency_file` (if not present) feature to the top of the toolchain -- Adds `pic` (if not present) feature to the top of the toolchain -- Adds `per_object_debug_info` (if not present) feature to the top of the toolchain -- Adds `preprocessor_defines` (if not present) feature to the top of the toolchain -- Adds `includes` (if not present) feature to the top of the toolchain -- Adds `include_paths` (if not present) feature to the top of the toolchain -- Adds `fdo_instrument` (if not present) feature to the top of the toolchain -- Adds `fdo_optimize` (if not present) feature to the top of the toolchain -- Adds `cs_fdo_instrument` (if not present) feature to the top of the toolchain -- Adds `cs_fdo_optimize` (if not present) feature to the top of the toolchain -- Adds `fdo_prefetch_hints` (if not present) feature to the top of the toolchain -- Adds `autofdo` (if not present) feature to the top of the toolchain -- Adds `build_interface_libraries` (if not present) feature to the top of the toolchain -- Adds `dynamic_library_linker_tool` (if not present) feature to the top of the toolchain -- Adds `shared_flag` (if not present) feature to the top of the toolchain -- Adds `linkstamps` (if not present) feature to the top of the toolchain -- Adds `output_execpath_flags` (if not present) feature to the top of the toolchain -- Adds `runtime_library_search_directories` (if not present) feature to the top of the toolchain -- Adds `library_search_directories` (if not present) feature to the top of the toolchain -- Adds `archiver_flags` (if not present) feature to the top of the toolchain -- Adds `libraries_to_link` (if not present) feature to the top of the toolchain -- Adds `force_pic_flags` (if not present) feature to the top of the toolchain -- Adds `user_link_flags` (if not present) feature to the top of the toolchain -- Adds `legacy_link_flags` (if not present) feature to the top of the toolchain -- Adds `static_libgcc` (if not present) feature to the top of the toolchain -- Adds `fission_support` (if not present) feature to the top of the toolchain -- Adds `strip_debug_symbols` (if not present) feature to the top of the toolchain -- Adds `coverage` (if not present) feature to the top of the toolchain -- Adds `llvm_coverage_map_format` (if not present) feature to the top of the toolchain -- Adds `gcc_coverage_map_format` (if not present) feature to the top of the toolchain -- Adds `fully_static_link` (if not present) feature to the bottom of the toolchain -- Adds `user_compile_flags` (if not present) feature to the bottom of the toolchain -- Adds `sysroot` (if not present) feature to the bottom of the toolchain -- Adds `unfiltered_compile_flags` (if not present) feature to the bottom of the toolchain -- Adds `linker_param_file` (if not present) feature to the bottom of the toolchain -- Adds `compiler_input_flags` (if not present) feature to the bottom of the toolchain -- Adds `compiler_output_flags` (if not present) feature to the bottom of the toolchain - -This is a long list of features. The plan is to get rid of them once [Crosstool in Starlark](https://github.com/bazelbuild/bazel/issues/5380) is done. For the curious reader see the implementation in [CppActionConfigs](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java?q=cppactionconfigs\&ss=bazel), and for production toolchains consider adding `no_legacy_features` to make the toolchain more standalone. diff --git a/docs/configurable-attributes.mdx b/docs/configurable-attributes.mdx index 6ec62150..958341f3 100644 --- a/docs/configurable-attributes.mdx +++ b/docs/configurable-attributes.mdx @@ -2,9 +2,15 @@ title: 'Configurable Build Attributes' --- -***Configurable attributes***, commonly known as [`select()`](/reference/be/functions#select), is a Bazel feature that lets users toggle the values of build rule attributes at the command line. -This can be used, for example, for a multiplatform library that automatically chooses the appropriate implementation for the architecture, or for a feature-configurable binary that can be customized at build time. + +**_Configurable attributes_**, commonly known as [`select()`]( +/reference/be/functions#select), is a Bazel feature that lets users toggle the values +of build rule attributes at the command line. + +This can be used, for example, for a multiplatform library that automatically +chooses the appropriate implementation for the architecture, or for a +feature-configurable binary that can be customized at build time. ## Example @@ -35,30 +41,60 @@ config_setting( ) ``` -This declares a `cc_binary` that "chooses" its deps based on the flags at the command line. Specifically, `deps` becomes: - -| | | -| ----------------------------------------------- | ------------------ | -| Command | deps = | -| `bazel build //myapp:mybinary --cpu=arm` | `[":arm_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=x86` | `[":x86_dev_lib"]` | -| `bazel build //myapp:mybinary --cpu=ppc` | `[":generic_lib"]` | -| `bazel build //myapp:mybinary -c dbg --cpu=ppc` | `[":generic_lib"]` | - -`select()` serves as a placeholder for a value that will be chosen based on *configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) targets. By using `select()` in a configurable attribute, the attribute effectively adopts different values when different conditions hold. +This declares a `cc_binary` that "chooses" its deps based on the flags at the +command line. Specifically, `deps` becomes: + + + + + + + + + + + + + + + + + + + + + + +
Commanddeps =
bazel build //myapp:mybinary --cpu=arm[":arm_lib"]
bazel build //myapp:mybinary -c dbg --cpu=x86[":x86_dev_lib"]
bazel build //myapp:mybinary --cpu=ppc[":generic_lib"]
bazel build //myapp:mybinary -c dbg --cpu=ppc[":generic_lib"]
+ +`select()` serves as a placeholder for a value that will be chosen based on +*configuration conditions*, which are labels referencing [`config_setting`](/reference/be/general#config_setting) +targets. By using `select()` in a configurable attribute, the attribute +effectively adopts different values when different conditions hold. Matches must be unambiguous: if multiple conditions match then either: -- They all resolve to the same value. For example, when running on linux x86, this is unambiguous `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". -- One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` is an unambiguous specialization of `values = {"cpu": "x86"}`. +* They all resolve to the same value. For example, when running on linux x86, this is unambiguous + `{"@platforms//os:linux": "Hello", "@platforms//cpu:x86_64": "Hello"}` because both branches resolve to "hello". +* One's `values` is a strict superset of all others'. For example, `values = {"cpu": "x86", "compilation_mode": "dbg"}` + is an unambiguous specialization of `values = {"cpu": "x86"}`. -The built-in condition [`//conditions:default`](#default-condition) automatically matches when nothing else does. +The built-in condition [`//conditions:default`](#default-condition) automatically matches when +nothing else does. -While this example uses `deps`, `select()` works just as well on `srcs`, `resources`, `cmd`, and most other attributes. Only a small number of attributes are *non-configurable*, and these are clearly annotated. For example, `config_setting`'s own [`values`](/reference/be/general#config_setting.values) attribute is non-configurable. +While this example uses `deps`, `select()` works just as well on `srcs`, +`resources`, `cmd`, and most other attributes. Only a small number of attributes +are *non-configurable*, and these are clearly annotated. For example, +`config_setting`'s own +[`values`](/reference/be/general#config_setting.values) attribute is non-configurable. ## `select()` and dependencies -Certain attributes change the build parameters for all transitive dependencies under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of the machine running Bazel (which, thanks to cross-compilation, may be different than the CPU the target is built for). This is known as a [configuration transition](/reference/glossary#transition). +Certain attributes change the build parameters for all transitive dependencies +under a target. For example, `genrule`'s `tools` changes `--cpu` to the CPU of +the machine running Bazel (which, thanks to cross-compilation, may be different +than the CPU the target is built for). This is known as a +[configuration transition](/reference/glossary#transition). Given @@ -102,19 +138,30 @@ running $ bazel build //myapp:my_genrule --cpu=arm ``` -on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and `x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s build parameters, which include `--cpu=arm`. The `tools` attribute changes `--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on `tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. +on an `x86` developer machine binds the build to `g_arm.src`, `tool1`, and +`x86tool.cc`. Both of the `select`s attached to `my_genrule` use `my_genrule`'s +build parameters, which include `--cpu=arm`. The `tools` attribute changes +`--cpu` to `x86` for `tool1` and its transitive dependencies. The `select` on +`tool1` uses `tool1`'s build parameters, which include `--cpu=x86`. ## Configuration conditions -Each key in a configurable attribute is a label reference to a [`config_setting`](/reference/be/general#config_setting) or [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). +Each key in a configurable attribute is a label reference to a +[`config_setting`](/reference/be/general#config_setting) or +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value). -`config_setting` is just a collection of expected command line flag settings. By encapsulating these in a target, it's easy to maintain "standard" conditions users can reference from multiple places. +`config_setting` is just a collection of +expected command line flag settings. By encapsulating these in a target, it's +easy to maintain "standard" conditions users can reference from multiple places. `constraint_value` provides support for [multi-platform behavior](#platforms). ### Built-in flags -Flags like `--cpu` are built into Bazel: the build tool natively understands them for all builds in all projects. These are specified with [`config_setting`](/reference/be/general#config_setting)'s [`values`](/reference/be/general#config_setting.values) attribute: +Flags like `--cpu` are built into Bazel: the build tool natively understands +them for all builds in all projects. These are specified with +[`config_setting`](/reference/be/general#config_setting)'s +[`values`](/reference/be/general#config_setting.values) attribute: ```python config_setting( @@ -127,21 +174,31 @@ config_setting( ) ``` -`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` is the expected value for that flag. `:meaningful_condition_name` matches if *every* entry in `values` matches. Order is irrelevant. +`flagN` is a flag name (without `--`, so `"cpu"` instead of `"--cpu"`). `valueN` +is the expected value for that flag. `:meaningful_condition_name` matches if +*every* entry in `values` matches. Order is irrelevant. `valueN` is parsed as if it was set on the command line. This means: -- `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` -- `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` -- `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` +* `values = { "compilation_mode": "opt" }` matches `bazel build -c opt` +* `values = { "force_pic": "true" }` matches `bazel build --force_pic=1` +* `values = { "force_pic": "0" }` matches `bazel build --noforce_pic` -`config_setting` only supports flags that affect target behavior. For example, [`--show_progress`](/docs/user-manual#show-progress) isn't allowed because it only affects how Bazel reports progress to the user. Targets can't use that flag to construct their results. The exact set of supported flags isn't documented. In practice, most flags that "make sense" work. +`config_setting` only supports flags that affect target behavior. For example, +[`--show_progress`](/docs/user-manual#show-progress) isn't allowed because +it only affects how Bazel reports progress to the user. Targets can't use that +flag to construct their results. The exact set of supported flags isn't +documented. In practice, most flags that "make sense" work. ### Custom flags -You can model your own project-specific flags with [Starlark build settings](/extending/config#user-defined-build-settings). Unlike built-in flags, these are defined as build targets, so Bazel references them with target labels. +You can model your own project-specific flags with +[Starlark build settings][BuildSettings]. Unlike built-in flags, these are +defined as build targets, so Bazel references them with target labels. -These are triggered with [`config_setting`](/reference/be/general#config_setting)'s [`flag_values`](/reference/be/general#config_setting.flag_values) attribute: +These are triggered with [`config_setting`](/reference/be/general#config_setting)'s +[`flag_values`](/reference/be/general#config_setting.flag_values) +attribute: ```python config_setting( @@ -154,17 +211,29 @@ config_setting( ) ``` -Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) for a working example. +Behavior is the same as for [built-in flags](#built-in-flags). See [here](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) +for a working example. -[`--define`](/reference/command-line-reference#flag--define) is an alternative legacy syntax for custom flags (for example `--define foo=bar`). This can be expressed either in the [values](/reference/be/general#config_setting.values) attribute (`values = {"define": "foo=bar"}`) or the [define\_values](/reference/be/general#config_setting.define_values) attribute (`define_values = {"foo": "bar"}`). `--define` is only supported for backwards compatibility. Prefer Starlark build settings whenever possible. +[`--define`](/reference/command-line-reference#flag--define) +is an alternative legacy syntax for custom flags (for example +`--define foo=bar`). This can be expressed either in the +[values](/reference/be/general#config_setting.values) attribute +(`values = {"define": "foo=bar"}`) or the +[define_values](/reference/be/general#config_setting.define_values) attribute +(`define_values = {"foo": "bar"}`). `--define` is only supported for backwards +compatibility. Prefer Starlark build settings whenever possible. -`values`, `flag_values`, and `define_values` evaluate independently. The `config_setting` matches if all values across all of them match. +`values`, `flag_values`, and `define_values` evaluate independently. The +`config_setting` matches if all values across all of them match. ## The default condition -The built-in condition `//conditions:default` matches when no other condition matches. +The built-in condition `//conditions:default` matches when no other condition +matches. -Because of the "exactly one match" rule, a configurable attribute with no match and no default condition emits a `"no matching conditions"` error. This can protect against silent failures from unexpected settings: +Because of the "exactly one match" rule, a configurable attribute with no match +and no default condition emits a `"no matching conditions"` error. This can +protect against silent failures from unexpected settings: ```python # myapp/BUILD @@ -190,11 +259,16 @@ Conditions checked: //myapp:x86_cpu ``` -For even clearer errors, you can set custom messages with `select()`'s [`no_match_error`](#custom-error-messages) attribute. +For even clearer errors, you can set custom messages with `select()`'s +[`no_match_error`](#custom-error-messages) attribute. ## Platforms -While the ability to specify multiple flags on the command line provides flexibility, it can also be burdensome to individually set each one every time you want to build a target. [Platforms](/extending/platforms) let you consolidate these into simple bundles. +While the ability to specify multiple flags on the command line provides +flexibility, it can also be burdensome to individually set each one every time +you want to build a target. + [Platforms](/extending/platforms) +let you consolidate these into simple bundles. ```python # myapp/BUILD @@ -252,9 +326,12 @@ platform( ) ``` -The platform can be specified on the command line. It activates the `config_setting`s that contain a subset of the platform's `constraint_values`, allowing those `config_setting`s to match in `select()` expressions. +The platform can be specified on the command line. It activates the +`config_setting`s that contain a subset of the platform's `constraint_values`, +allowing those `config_setting`s to match in `select()` expressions. -For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, you can simply run +For example, in order to set the `srcs` attribute of `my_rocks` to `calcite.sh`, +you can simply run ```sh bazel build //my_app:my_rocks --platforms=//myapp:marble_platform @@ -281,9 +358,11 @@ sh_binary( ) ``` -This saves the need for boilerplate `config_setting`s when you only need to check against single values. +This saves the need for boilerplate `config_setting`s when you only need to +check against single values. -Platforms are still under development. See the [documentation](/concepts/platforms) for details. +Platforms are still under development. See the +[documentation](/concepts/platforms) for details. ## Combining `select()`s @@ -305,12 +384,12 @@ sh_binary( ``` Note: Some restrictions apply on what can be combined in the `select`s values: + - Duplicate labels can appear in different paths of the same `select`. + - Duplicate labels can *not* appear within the same path of a `select`. + - Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) -- Duplicate labels can appear in different paths of the same `select`. -- Duplicate labels can *not* appear within the same path of a `select`. -- Duplicate labels can *not* appear across multiple combined `select`s (no matter what path) - -`select` cannot appear inside another `select`. If you need to nest `selects` and your attribute takes other targets as values, use an intermediate target: +`select` cannot appear inside another `select`. If you need to nest `selects` +and your attribute takes other targets as values, use an intermediate target: ```python sh_binary( @@ -331,7 +410,8 @@ sh_library( ) ``` -If you need a `select` to match when multiple conditions match, consider [AND chaining](#and-chaining). +If you need a `select` to match when multiple conditions match, consider [AND +chaining](#and-chaining). ## OR chaining @@ -350,7 +430,9 @@ sh_binary( ) ``` -Most conditions evaluate to the same dep. But this syntax is hard to read and maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple times. +Most conditions evaluate to the same dep. But this syntax is hard to read and +maintain. It would be nice to not have to repeat `[":standard_lib"]` multiple +times. One option is to predefine the value as a BUILD variable: @@ -369,13 +451,18 @@ sh_binary( ) ``` -This makes it easier to manage the dependency. But it still causes unnecessary duplication. +This makes it easier to manage the dependency. But it still causes unnecessary +duplication. For more direct support, use one of the following: ### `selects.with_or` -The [with\_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing conditions directly inside a `select`: +The +[with_or](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing conditions directly inside a `select`: ```python load("@bazel_skylib//lib:selects.bzl", "selects") @@ -394,12 +481,18 @@ sh_binary( ### `selects.config_setting_group` -The [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) module supports `OR`ing multiple `config_setting`s: + +The +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group) +macro in [Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md) +module supports `OR`ing multiple `config_setting`s: ```python load("@bazel_skylib//lib:selects.bzl", "selects") ``` + ```python config_setting( name = "config1", @@ -423,13 +516,17 @@ sh_binary( ) ``` -Unlike `selects.with_or`, different targets can share `:config1_or_2` across different attributes. +Unlike `selects.with_or`, different targets can share `:config1_or_2` across +different attributes. -It's an error for multiple conditions to match unless one is an unambiguous "specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. +It's an error for multiple conditions to match unless one is an unambiguous +"specialization" of the others or they all resolve to the same value. See [here](#configurable-build-example) for details. ## AND chaining -If you need a `select` branch to match when multiple conditions match, use the [Skylib](https://github.com/bazelbuild/bazel-skylib) macro [config\_setting\_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): +If you need a `select` branch to match when multiple conditions match, use the +[Skylib](https://github.com/bazelbuild/bazel-skylib) macro +[config_setting_group](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectsconfig_setting_group): ```python config_setting( @@ -454,11 +551,13 @@ sh_binary( ) ``` -Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed inside a `select`. You have to explicitly wrap them in a `config_setting_group`. +Unlike OR chaining, existing `config_setting`s can't be directly `AND`ed +inside a `select`. You have to explicitly wrap them in a `config_setting_group`. ## Custom error messages -By default, when no condition matches, the target the `select()` is attached to fails with the error: +By default, when no condition matches, the target the `select()` is attached to +fails with the error: ```sh ERROR: Configurable attribute "deps" doesn't match this configuration (would @@ -468,7 +567,8 @@ Conditions checked: //tools/cc_target_os:android ``` -This can be customized with the [`no_match_error`](/reference/be/functions#select) attribute: +This can be customized with the [`no_match_error`](/reference/be/functions#select) +attribute: ```python cc_library( @@ -491,7 +591,8 @@ build with an Android or Windows toolchain ## Rules compatibility -Rule implementations receive the *resolved values* of configurable attributes. For example, given: +Rule implementations receive the *resolved values* of configurable +attributes. For example, given: ```python # myapp/BUILD @@ -511,7 +612,9 @@ $ bazel build //myapp/my_target --define mode=foo Rule implementation code sees `ctx.attr.some_attr` as `[":foo"]`. -Macros can accept `select()` clauses and pass them through to native rules. But *they cannot directly manipulate them*. For example, there's no way for a macro to convert +Macros can accept `select()` clauses and pass them through to native +rules. But *they cannot directly manipulate them*. For example, there's no way +for a macro to convert ```python select({"foo": "val"}, ...) @@ -525,22 +628,32 @@ select({"foo": "val_with_suffix"}, ...) This is for two reasons. -First, macros that need to know which path a `select` will choose *cannot work* because macros are evaluated in Bazel's [loading phase](/run/build#loading), which occurs before flag values are known. This is a core Bazel design restriction that's unlikely to change any time soon. +First, macros that need to know which path a `select` will choose *cannot work* +because macros are evaluated in Bazel's [loading phase](/run/build#loading), +which occurs before flag values are known. +This is a core Bazel design restriction that's unlikely to change any time soon. -Second, macros that just need to iterate over *all* `select` paths, while technically feasible, lack a coherent UI. Further design is necessary to change this. +Second, macros that just need to iterate over *all* `select` paths, while +technically feasible, lack a coherent UI. Further design is necessary to change +this. ## Bazel query and cquery -Bazel [`query`](/query/guide) operates over Bazel's [loading phase](/reference/glossary#loading-phase). This means it doesn't know what command line flags a target uses since those flags aren't evaluated until later in the build (in the [analysis phase](/reference/glossary#analysis-phase)). So it can't determine which `select()` branches are chosen. +Bazel [`query`](/query/guide) operates over Bazel's +[loading phase](/reference/glossary#loading-phase). +This means it doesn't know what command line flags a target uses since those +flags aren't evaluated until later in the build (in the +[analysis phase](/reference/glossary#analysis-phase)). +So it can't determine which `select()` branches are chosen. -Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has all this information and can accurately resolve `select()`s. +Bazel [`cquery`](/query/cquery) operates after Bazel's analysis phase, so it has +all this information and can accurately resolve `select()`s. Consider: ```python load("@bazel_skylib//rules:common_settings.bzl", "string_flag") ``` - ```python # myapp/BUILD @@ -589,9 +702,14 @@ $ bazel cquery 'deps(//myapp:my_lib)' --//myapp:dog_type=pug ### Why doesn't select() work in macros? -select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for details. +select() *does* work in rules! See [Rules compatibility](#rules-compatibility) for +details. -The key issue this question usually means is that select() doesn't work in *macros*. These are different than *rules*. See the documentation on [rules](/extending/rules) and [macros](/extending/macros) to understand the difference. Here's an end-to-end example: +The key issue this question usually means is that select() doesn't work in +*macros*. These are different than *rules*. See the +documentation on [rules](/extending/rules) and [macros](/extending/macros) +to understand the difference. +Here's an end-to-end example: Define a rule and macro: @@ -671,7 +789,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:5:3: My name is happy_macro with custom messa DEBUG: /myworkspace/myapp/hi.bzl:15:3: My name is happy_rule with custom message: FIRST STRING. ``` -This is impossible to change because *by definition* macros are evaluated before Bazel reads the build's command line flags. That means there isn't enough information to evaluate select()s. +This is impossible to change because *by definition* macros are evaluated before +Bazel reads the build's command line flags. That means there isn't enough +information to evaluate select()s. Macros can, however, pass `select()`s as opaque blobs to rules: @@ -694,7 +814,9 @@ DEBUG: /myworkspace/myapp/defs.bzl:15:3: My name is sad_macro_less_sad with cust ### Why does select() always return true? -Because *macros* (but not rules) by definition [can't evaluate `select()`s](#faq-select-macro), any attempt to do so usually produces an error: +Because *macros* (but not rules) by definition +[can't evaluate `select()`s](#faq-select-macro), any attempt to do so +usually produces an error: ```sh ERROR: /myworkspace/myapp/BUILD:17:1: Traceback @@ -707,7 +829,8 @@ my_config_string.upper() type 'select' has no method upper(). ``` -Booleans are a special case that fail silently, so you should be particularly vigilant with them: +Booleans are a special case that fail silently, so you should be particularly +vigilant with them: ```sh $ cat myapp/defs.bzl @@ -729,13 +852,21 @@ $ bazel build //mypro:all --cpu=ppc DEBUG: /myworkspace/myapp/defs.bzl:4:3: TRUE. ``` -This happens because macros don't understand the contents of `select()`. So what they're really evaluting is the `select()` object itself. According to [Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design standards, all objects aside from a very small number of exceptions automatically return true. +This happens because macros don't understand the contents of `select()`. +So what they're really evaluting is the `select()` object itself. According to +[Pythonic](https://docs.python.org/release/2.5.2/lib/truth.html) design +standards, all objects aside from a very small number of exceptions +automatically return true. ### Can I read select() like a dict? -Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before Bazel knows what the build's command line parameters are. Can they at least read the `select()`'s dictionary to, for example, add a suffix to each value? +Macros [can't](#faq-select-macro) evaluate select(s) because macros evaluate before +Bazel knows what the build's command line parameters are. Can they at least read +the `select()`'s dictionary to, for example, add a suffix to each value? -Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). What you *can* do today is prepare a straight dictionary, then feed it into a `select()`: +Conceptually this is possible, but [it isn't yet a Bazel feature](https://github.com/bazelbuild/bazel/issues/8419). +What you *can* do today is prepare a straight dictionary, then feed it into a +`select()`: ```sh $ cat myapp/defs.bzl @@ -747,7 +878,7 @@ def selecty_genrule(name, select_cmd): outs = [name + ".out"], srcs = [], cmd = "echo " + select(select_cmd + {"//conditions:default": "default"}) - + " > $@" + + " > $@" ) $ cat myapp/BUILD @@ -779,7 +910,7 @@ def selecty_genrule(name, select_cmd): name = name, outs = [name + ".out"], srcs = [], - cmd = "echo " + cmd_suffix + "> $@", + cmd = "echo " + cmd_suffix + "> $@", ) ``` @@ -787,11 +918,17 @@ def selecty_genrule(name, select_cmd): First of all, do not use `bind()`. It is deprecated in favor of `alias()`. -The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo rule, not a BUILD rule. +The technical answer is that [`bind()`](/reference/be/workspace#bind) is a repo +rule, not a BUILD rule. -Repo rules do not have a specific configuration, and aren't evaluated in the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't actually evaluate to any specific branch. +Repo rules do not have a specific configuration, and aren't evaluated in +the same way as BUILD rules. Therefore, a `select()` in a `bind()` can't +actually evaluate to any specific branch. -Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in the `actual` attribute, to perform this type of run-time determination. This works correctly, since `alias()` is a BUILD rule, and is evaluated with a specific configuration. +Instead, you should use [`alias()`](/reference/be/general#alias), with a `select()` in +the `actual` attribute, to perform this type of run-time determination. This +works correctly, since `alias()` is a BUILD rule, and is evaluated with a +specific configuration. You can even have a `bind()` target point to an `alias()`, if needed. @@ -819,31 +956,37 @@ alias( ) ``` -With this setup, you can pass `--define ssl_library=alternative`, and any target that depends on either `//:ssl` or `//external:ssl` will see the alternative located at `@alternative//:ssl`. +With this setup, you can pass `--define ssl_library=alternative`, and any target +that depends on either `//:ssl` or `//external:ssl` will see the alternative +located at `@alternative//:ssl`. But really, stop using `bind()`. ### Why doesn't my select() choose what I expect? -If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, use [cquery](/query/cquery) and `bazel config` to debug: +If `//myapp:foo` has a `select()` that doesn't choose the condition you expect, +use [cquery](/query/cquery) and `bazel config` to debug: If `//myapp:foo` is the top-level target you're building, run: ```sh -$ bazel cquery //myapp:foo <desired build flags> +$ bazel cquery //myapp:foo //myapp:foo (12e23b9a2b534a) ``` -If you're building some other target `//bar` that depends on //myapp:foo somewhere in its subgraph, run: +If you're building some other target `//bar` that depends on +//myapp:foo somewhere in its subgraph, run: ```sh -$ bazel cquery 'somepath(//bar, //myapp:foo)' <desired build flags> +$ bazel cquery 'somepath(//bar, //myapp:foo)' //bar:bar (3ag3193fee94a2) //bar:intermediate_dep (12e23b9a2b534a) //myapp:foo (12e23b9a2b534a) ``` -The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the configuration that resolves `//myapp:foo`'s `select()`. You can inspect its values with `bazel config`: +The `(12e23b9a2b534a)` next to `//myapp:foo` is a *hash* of the +configuration that resolves `//myapp:foo`'s `select()`. You can inspect its +values with `bazel config`: ```sh $ bazel config 12e23b9a2b534a @@ -862,13 +1005,18 @@ Fragment com.google.devtools.build.lib.rules.cpp.CppOptions { Then compare this output against the settings expected by each `config_setting`. -`//myapp:foo` may exist in different configurations in the same build. See the [cquery docs](/query/cquery) for guidance on using `somepath` to get the right one. +`//myapp:foo` may exist in different configurations in the same build. See the +[cquery docs](/query/cquery) for guidance on using `somepath` to get the right +one. -Caution: To prevent restarting the Bazel server, invoke `bazel config` with the same command line flags as the `bazel cquery`. The `config` command relies on the configuration nodes from the still-running server of the previous command. +Caution: To prevent restarting the Bazel server, invoke `bazel config` with the +same command line flags as the `bazel cquery`. The `config` command relies on +the configuration nodes from the still-running server of the previous command. ### Why doesn't `select()` work with platforms? -Bazel doesn't support configurable attributes checking whether a given platform is the target platform because the semantics are unclear. +Bazel doesn't support configurable attributes checking whether a given platform +is the target platform because the semantics are unclear. For example: @@ -891,11 +1039,15 @@ cc_library( ) ``` -In this `BUILD` file, which `select()` should be used if the target platform has both the `@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the `:x86_linux_platform` defined here? The author of the `BUILD` file and the user who defined the separate platform may have different ideas. +In this `BUILD` file, which `select()` should be used if the target platform has both the +`@platforms//cpu:x86` and `@platforms//os:linux` constraints, but is **not** the +`:x86_linux_platform` defined here? The author of the `BUILD` file and the user +who defined the separate platform may have different ideas. #### What should I do instead? -Instead, define a `config_setting` that matches **any** platform with these constraints: +Instead, define a `config_setting` that matches **any** platform with +these constraints: ```py config_setting( @@ -916,11 +1068,13 @@ cc_library( ) ``` -This process defines specific semantics, making it clearer to users what platforms meet the desired conditions. +This process defines specific semantics, making it clearer to users what +platforms meet the desired conditions. #### What if I really, really want to `select` on the platform? -If your build requirements specifically require checking the platform, you can flip the value of the `--platforms` flag in a `config_setting`: +If your build requirements specifically require checking the platform, you +can flip the value of the `--platforms` flag in a `config_setting`: ```py config_setting( @@ -940,4 +1094,7 @@ cc_library( ) ``` -The Bazel team doesn't endorse doing this; it overly constrains your build and confuses users when the expected condition does not match. +The Bazel team doesn't endorse doing this; it overly constrains your build and +confuses users when the expected condition does not match. + +[BuildSettings]: /extending/config#user-defined-build-settings diff --git a/docs/mobile-install.mdx b/docs/mobile-install.mdx index 1d08dcd0..8bc4a679 100644 --- a/docs/mobile-install.mdx +++ b/docs/mobile-install.mdx @@ -2,100 +2,203 @@ title: 'bazel mobile-install' --- -Fast iterative development for Android -This page describes how `bazel mobile-install` makes iterative development for Android much faster. It describes the benefits of this approach versus the drawbacks of separate build and install steps. + + +

Fast iterative development for Android

+ +This page describes how `bazel mobile-install` makes iterative development +for Android much faster. It describes the benefits of this approach versus the +drawbacks of separate build and install steps. ## Summary To install small changes to an Android app very quickly, do the following: -1. Find the `android_binary` rule of the app you want to install. -2. Connect your device to `adb`. -3. Run `bazel mobile-install :your_target`. App startup will be a little slower than usual. -4. Edit the code or Android resources. -5. Run `bazel mobile-install :your_target`. -6. Enjoy a fast and minimal incremental installation! + 1. Find the `android_binary` rule of the app you want to install. + 2. Connect your device to `adb`. + 3. Run `bazel mobile-install :your_target`. App startup will be a little + slower than usual. + 4. Edit the code or Android resources. + 5. Run `bazel mobile-install :your_target`. + 6. Enjoy a fast and minimal incremental installation! Some command line options to Bazel that may be useful: -- `--adb` tells Bazel which adb binary to use -- `--adb_arg` can be used to add extra arguments to the command line of `adb`. One useful application of this is to select which device you want to install to if you have multiple devices connected to your workstation: `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=<SERIAL>` + - `--adb` tells Bazel which adb binary to use + - `--adb_arg` can be used to add extra arguments to the command line of `adb`. + One useful application of this is to select which device you want to install + to if you have multiple devices connected to your workstation: + `bazel mobile-install :your_target -- --adb_arg=-s --adb_arg=` -When in doubt, look at the [example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) +When in doubt, look at the +[example](https://github.com/bazelbuild/rules_android/tree/main/examples/basicapp), +contact us on [Google Groups](https://groups.google.com/forum/#!forum/bazel-discuss), +or [file a GitHub issue](https://github.com/bazelbuild/rules_android/issues) ## Introduction -One of the most important attributes of a developer's toolchain is speed: there is a world of difference between changing the code and seeing it run within a second and having to wait minutes, sometimes hours, before you get any feedback on whether your changes do what you expect them to. +One of the most important attributes of a developer's toolchain is speed: there +is a world of difference between changing the code and seeing it run within a +second and having to wait minutes, sometimes hours, before you get any feedback +on whether your changes do what you expect them to. -Unfortunately, the traditional Android toolchain for building an .apk entails many monolithic, sequential steps and all of these have to be done in order to build an Android app. At Google, waiting five minutes to build a single-line change was not unusual on larger projects like Google Maps. +Unfortunately, the traditional Android toolchain for building an .apk entails +many monolithic, sequential steps and all of these have to be done in order to +build an Android app. At Google, waiting five minutes to build a single-line +change was not unusual on larger projects like Google Maps. -`bazel mobile-install` makes iterative development for Android much faster by using a combination of change pruning, work sharding, and clever manipulation of Android internals, all without changing any of your app's code. +`bazel mobile-install` makes iterative development for Android much faster by +using a combination of change pruning, work sharding, and clever manipulation of +Android internals, all without changing any of your app's code. ## Problems with traditional app installation Building an Android app has some issues, including: -- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) is invoked exactly once in the build and it does not know how to reuse work from previous builds: it dexes every method again, even though only one method was changed. +- Dexing. By default, the Dexer tool (historically `dx`, now `d8` or `r8`) +is invoked exactly once in the build and it does not know how to reuse work from +previous builds: it dexes every method again, even though only one method was +changed. -- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 connection, and larger apps can take a lot of time to upload. The entire app is uploaded, even if only small parts have changed, for example, a resource or a single method, so this can be a major bottleneck. +- Uploading data to the device. adb does not use the full bandwidth of a USB 2.0 +connection, and larger apps can take a lot of time to upload. The entire app is +uploaded, even if only small parts have changed, for example, a resource or a +single method, so this can be a major bottleneck. -- Compilation to native code. Android L introduced ART, a new Android runtime, which compiles apps ahead-of-time rather than compiling them just-in-time like Dalvik. This makes apps much faster at the cost of longer installation time. This is a good tradeoff for users because they typically install an app once and use it many times, but results in slower development where an app is installed many times and each version is run at most a handful of times. +- Compilation to native code. Android L introduced ART, a new Android runtime, +which compiles apps ahead-of-time rather than compiling them just-in-time like +Dalvik. This makes apps much faster at the cost of longer installation +time. This is a good tradeoff for users because they typically install an app +once and use it many times, but results in slower development where an app is +installed many times and each version is run at most a handful of times. ## The approach of `bazel mobile-install` `bazel mobile-install `makes the following improvements: -- Sharded desugaring and dexing. After building the app's Java code, Bazel shards the class files into approximately equal-sized parts and invokes `d8` separately on them. `d8` is not invoked on shards that did not change since the last build. These shards are then compiled into separate sharded APKs. + - Sharded desugaring and dexing. After building the app's Java code, Bazel + shards the class files into approximately equal-sized parts and invokes `d8` + separately on them. `d8` is not invoked on shards that did not change since + the last build. These shards are then compiled into separate sharded APKs. -- Incremental file transfer. Android resources, .dex files, and native libraries are removed from the main .apk and are stored in under a separate mobile-install directory. This makes it possible to update code and Android resources independently without reinstalling the whole app. Thus, transferring the files takes less time and only the .dex files that have changed are recompiled on-device. + - Incremental file transfer. Android resources, .dex files, and native + libraries are removed from the main .apk and are stored in under a separate + mobile-install directory. This makes it possible to update code and Android + resources independently without reinstalling the whole app. Thus, + transferring the files takes less time and only the .dex files that have + changed are recompiled on-device. -- Sharded installation. Mobile-install uses Android Studio's [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) tool to combine sharded APKs on the connected device and provide a cohesive experience. + - Sharded installation. Mobile-install uses Android Studio's + [`apkdeployer`](https://maven.google.com/web/index.html?q=deployer#com.android.tools.apkdeployer:apkdeployer) + tool to combine sharded APKs on the connected device and provide a cohesive + experience. ### Sharded Dexing -Sharded dexing is reasonably straightforward: once the .jar files are built, a [tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) shards them into separate .jar files of approximately equal size, then invokes `d8` on those that were changed since the previous build. The logic that determines which shards to dex is not specific to Android: it just uses the general change pruning algorithm of Bazel. - -The first version of the sharding algorithm simply ordered the .class files alphabetically, then cut the list up into equal-sized parts, but this proved to be suboptimal: if a class was added or removed (even a nested or an anonymous one), it would cause all the classes alphabetically after it to shift by one, resulting in dexing those shards again. Thus, it was decided to shard Java packages rather than individual classes. Of course, this still results in dexing many shards if a new package is added or removed, but that is much less frequent than adding or removing a single class. - -The number of shards is controlled by command-line configuration, using the `--define=num_dex_shards=N` flag. In an ideal world, Bazel would automatically determine how many shards are best, but Bazel currently must know the set of actions (for example, commands to be executed during the build) before executing any of them, so it cannot determine the optimal number of shards because it doesn't know how many Java classes there will eventually be in the app. Generally speaking, the more shards, the faster the build and the installation will be, but the slower app startup becomes, because the dynamic linker has to do more work. The sweet spot is usually between 10 and 50 shards. +Sharded dexing is reasonably straightforward: once the .jar files are built, a +[tool](https://github.com/bazelbuild/rules_android/blob/main/src/tools/java/com/google/devtools/build/android/ziputils/DexMapper.java) +shards them into separate .jar files of approximately equal size, then invokes +`d8` on those that were changed since the previous build. The logic that +determines which shards to dex is not specific to Android: it just uses the +general change pruning algorithm of Bazel. + +The first version of the sharding algorithm simply ordered the .class files +alphabetically, then cut the list up into equal-sized parts, but this proved to +be suboptimal: if a class was added or removed (even a nested or an anonymous +one), it would cause all the classes alphabetically after it to shift by one, +resulting in dexing those shards again. Thus, it was decided to shard Java +packages rather than individual classes. Of course, this still results in +dexing many shards if a new package is added or removed, but that is much less +frequent than adding or removing a single class. + +The number of shards is controlled by command-line configuration, using the +`--define=num_dex_shards=N` flag. In an ideal world, Bazel would +automatically determine how many shards are best, but Bazel currently must know +the set of actions (for example, commands to be executed during the build) before +executing any of them, so it cannot determine the optimal number of shards +because it doesn't know how many Java classes there will eventually be in the +app. Generally speaking, the more shards, the faster the build and the +installation will be, but the slower app startup becomes, because the dynamic +linker has to do more work. The sweet spot is usually between 10 and 50 shards. ### Incremental deployment -Incremental APK shard transfer and installation is now handled by the `apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). Whereas earlier (native) versions of mobile-install required manually tracking first-time installations and selectively apply the `--incremental` flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) has been greatly simplified. The same mobile-install invocation can be used regardless of how many times the app has been installed or reinstalled. - -At a high level, the `apkdeployer` tool is a wrapper around various `adb` sub-commands. The main entrypoint logic can be found in the [`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) class, with other utility classes colocated in the same package. The `Deployer` class ingests, among other things, a list of paths to split APKs and a protobuf with information about the installation, and leverages deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) in order to create an install session and incrementally deploy app splits. See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) classes for implementation details. +Incremental APK shard transfer and installation is now handled by the +`apkdeployer` utility described in ["The approach of mobile-install"](#approach-mobile-install). +Whereas earlier (native) versions of mobile-install required manually tracking +first-time installations and selectively apply the `--incremental` +flag on subsequent installation, the most recent version in [`rules_android`](https://github.com/bazelbuild/rules_android/tree/main/mobile_install) +has been greatly simplified. The same mobile-install +invocation can be used regardless of how many times the app has been installed +or reinstalled. + +At a high level, the `apkdeployer` tool is a wrapper around various `adb` +sub-commands. The main entrypoint logic can be found in the +[`com.android.tools.deployer.Deployer`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/Deployer.java) +class, with other utility classes colocated in the same package. +The `Deployer` class ingests, among other things, a list of paths to split +APKs and a protobuf with information about the installation, and leverages +deployment features for [Android app bundles](https://developer.android.com/guide/app-bundle) +in order to create an install session and incrementally deploy app splits. +See the [`ApkPreInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkPreInstaller.java) +and [`ApkInstaller`](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:deploy/deployer/src/main/java/com/android/tools/deployer/ApkInstaller.java) +classes for implementation details. ## Results ### Performance -In general, `bazel mobile-install` results in a 4x to 10x speedup of building and installing large apps after a small change. +In general, `bazel mobile-install` results in a 4x to 10x speedup of building +and installing large apps after a small change. The following numbers were computed for a few Google products: -![](/docs/images/mobile-install-performance.svg) + -This, of course, depends on the nature of the change: recompilation after changing a base library takes more time. +This, of course, depends on the nature of the change: recompilation after +changing a base library takes more time. ### Limitations -The tricks the stub application plays don't work in every case. The following cases highlight where it does not work as expected: +The tricks the stub application plays don't work in every case. +The following cases highlight where it does not work as expected: -- Mobile-install is only supported via the Starlark rules of `rules_android`. See the ["brief history of mobile-install"](#mobile-install-history) for more detail. + - Mobile-install is only supported via the Starlark rules of `rules_android`. + See the ["brief history of mobile-install"](#mobile-install-history) for + more detail. -- Only devices running ART are supported. Mobile-install uses API and runtime features that only exist on devices running ART, not Dalvik. Any Android runtime more recent than Android L (API 21+) should be compatible. + - Only devices running ART are supported. Mobile-install uses API and runtime features + that only exist on devices running ART, not Dalvik. Any Android runtime more + recent than Android L (API 21+) should be compatible. -- Bazel itself must be run with a tool Java runtime *and* language version of 17 or higher. + - Bazel itself must be run with a tool Java runtime _and_ language version + of 17 or higher. -- Bazel versions prior to 8.4.0 must specify some additional flags for mobile-install. See [the Bazel Android tutorial](/start/android-app). These flags inform Bazel where the Starlark mobile-install aspect is and which rules are supported. + - Bazel versions prior to 8.4.0 must specify some additional flags for + mobile-install. See [the Bazel Android tutorial](/start/android-app). These + flags inform Bazel where the Starlark mobile-install aspect is and which + rules are supported. ### A brief history of mobile-install +Earlier Bazel versions _natively_ included built-in build and test rules for +popular languages and ecosystems such as C++, Java, and Android. These rules +were therefore referred to as _native_ rules. Bazel 8 (released in 2024) removed +support for these rules because many of them had been migrated to the +[Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) +for more details. + +The legacy native Android rules also supported a legacy _native_ version of +mobile-install functionality. This is referred to as "mobile-install v1" or +"native mobile-install" now. This functionality was deleted in Bazel 8, along +with the built-in Android rules. -Earlier Bazel versions *natively* included built-in build and test rules for popular languages and ecosystems such as C++, Java, and Android. These rules were therefore referred to as *native* rules. Bazel 8 (released in 2024) removed support for these rules because many of them had been migrated to the [Starlark](/rules/language) language. See the ["Bazel 8.0 LTS blog post"](https://blog.bazel.build/2024/12/09/bazel-8-release.html) for more details. +Now, all mobile-install functionality, as well as all Android build and test +rules, are implemented in Starlark and reside in the `rules_android` GitHub +repository. The latest version is known as "mobile-install v3" or "MIv3". -The legacy native Android rules also supported a legacy *native* version of mobile-install functionality. This is referred to as "mobile-install v1" or "native mobile-install" now. This functionality was deleted in Bazel 8, along with the built-in Android rules. +_Naming note_: There was a "mobile-install **v2**" available only internally +at Google at one point, but this was never published externally, and only v3 +continues to be used for both Google-internal and OSS rules_android deployment. -Now, all mobile-install functionality, as well as all Android build and test rules, are implemented in Starlark and reside in the `rules_android` GitHub repository. The latest version is known as "mobile-install v3" or "MIv3". -*Naming note*: There was a "mobile-install **v2**" available only internally at Google at one point, but this was never published externally, and only v3 continues to be used for both Google-internal and OSS rules\_android deployment. diff --git a/docs/sandboxing.mdx b/docs/sandboxing.mdx index b12027ec..68697953 100644 --- a/docs/sandboxing.mdx +++ b/docs/sandboxing.mdx @@ -2,49 +2,119 @@ title: 'Sandboxing' --- -This article covers sandboxing in Bazel and debugging your sandboxing environment. -*Sandboxing* is a permission restricting strategy that isolates processes from each other or from resources in a system. For Bazel, this means restricting file system access. -Bazel's file system sandbox runs processes in a working directory that only contains known inputs, such that compilers and other tools don't see source files they should not access, unless they know the absolute paths to them. +This article covers sandboxing in Bazel and debugging your sandboxing +environment. -Sandboxing doesn't hide the host environment in any way. Processes can freely access all files on the file system. However, on platforms that support user namespaces, processes can't modify any files outside their working directory. This ensures that the build graph doesn't have hidden dependencies that could affect the reproducibility of the build. +*Sandboxing* is a permission restricting strategy that isolates processes from +each other or from resources in a system. For Bazel, this means restricting file +system access. -More specifically, Bazel constructs an `execroot/` directory for each action, which acts as the action's work directory at execution time. `execroot/` contains all input files to the action and serves as the container for any generated outputs. Bazel then uses an operating-system-provided technique, containers on Linux and `sandbox-exec` on macOS, to constrain the action within `execroot/`. +Bazel's file system sandbox runs processes in a working directory that only +contains known inputs, such that compilers and other tools don't see source +files they should not access, unless they know the absolute paths to them. -## Reasons for sandboxing - -- Without action sandboxing, Bazel doesn't know if a tool uses undeclared input files (files that are not explicitly listed in the dependencies of an action). When one of the undeclared input files changes, Bazel still believes that the build is up-to-date and won’t rebuild the action. This can result in an incorrect incremental build. - -- Incorrect reuse of cache entries creates problems during remote caching. A bad cache entry in a shared cache affects every developer on the project, and wiping the entire remote cache is not a feasible solution. +Sandboxing doesn't hide the host environment in any way. Processes can freely +access all files on the file system. However, on platforms that support user +namespaces, processes can't modify any files outside their working directory. +This ensures that the build graph doesn't have hidden dependencies that could +affect the reproducibility of the build. -- Sandboxing mimics the behavior of remote execution — if a build works well with sandboxing, it will likely also work with remote execution. By making remote execution upload all necessary files (including local tools), you can significantly reduce maintenance costs for compile clusters compared to having to install the tools on every machine in the cluster every time you want to try out a new compiler or make a change to an existing tool. - -## What sandbox strategy to use +More specifically, Bazel constructs an `execroot/` directory for each action, +which acts as the action's work directory at execution time. `execroot/` +contains all input files to the action and serves as the container for any +generated outputs. Bazel then uses an operating-system-provided technique, +containers on Linux and `sandbox-exec` on macOS, to constrain the action within +`execroot/`. -You can choose which kind of sandboxing to use, if any, with the [strategy flags](user-manual.html#strategy-options). Using the `sandboxed` strategy makes Bazel pick one of the sandbox implementations listed below, preferring an OS-specific sandbox to the less hermetic generic one. [Persistent workers](/remote/persistent) run in a generic sandbox if you pass the `--worker_sandboxing` flag. - -The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. It simply executes the action's command line with the working directory set to the execroot of your workspace. - -`processwrapper-sandbox` is a sandboxing strategy that does not require any "advanced" features - it should work on any POSIX system out of the box. It builds a sandbox directory consisting of symlinks that point to the original source files, executes the action's command line with the working directory set to this directory instead of the execroot, then moves the known output artifacts out of the sandbox into the execroot and deletes the sandbox. This prevents the action from accidentally using any input files that are not declared and from littering the execroot with unknown output files. +## Reasons for sandboxing -`linux-sandbox` goes one step further and builds on top of the `processwrapper-sandbox`. Similar to what Docker does under the hood, it uses Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the action from the host. That is, it makes the entire filesystem read-only except for the sandbox directory, so the action cannot accidentally modify anything on the host filesystem. This prevents situations like a buggy test accidentally rm -rf'ing your $HOME directory. Optionally, you can also prevent the action from accessing the network. `linux-sandbox` uses PID namespaces to prevent the action from seeing any other processes and to reliably kill all processes (even daemons spawned by the action) at the end. +- Without action sandboxing, Bazel doesn't know if a tool uses undeclared + input files (files that are not explicitly listed in the dependencies of an + action). When one of the undeclared input files changes, Bazel still + believes that the build is up-to-date and won’t rebuild the action. This can + result in an incorrect incremental build. -`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool to achieve roughly the same as the Linux sandbox. +- Incorrect reuse of cache entries creates problems during remote caching. A + bad cache entry in a shared cache affects every developer on the project, + and wiping the entire remote cache is not a feasible solution. -Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" scenario due to restrictions in the mechanisms provided by the operating systems. Because Docker also uses Linux namespaces for its container magic, you cannot easily run `linux-sandbox` inside a Docker container, unless you use `docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a process that's already being sandboxed. Thus, in these cases, Bazel automatically falls back to using `processwrapper-sandbox`. +- Sandboxing mimics the behavior of remote execution — if a build works well + with sandboxing, it will likely also work with remote execution. By making + remote execution upload all necessary files (including local tools), you can + significantly reduce maintenance costs for compile clusters compared to + having to install the tools on every machine in the cluster every time you + want to try out a new compiler or make a change to an existing tool. -If you would rather get a build error — such as to not accidentally build with a less strict execution strategy — explicitly modify the list of execution strategies that Bazel tries to use (for example, `bazel build --spawn_strategy=worker,linux-sandbox`). +## What sandbox strategy to use -Dynamic execution usually requires sandboxing for local execution. To opt out, pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently sandboxes [persistent workers](/remote/persistent). +You can choose which kind of sandboxing to use, if any, with the +[strategy flags](user-manual.html#strategy-options). Using the `sandboxed` +strategy makes Bazel pick one of the sandbox implementations listed below, +preferring an OS-specific sandbox to the less hermetic generic one. +[Persistent workers](/remote/persistent) run in a generic sandbox if you pass +the `--worker_sandboxing` flag. + +The `local` (a.k.a. `standalone`) strategy does not do any kind of sandboxing. +It simply executes the action's command line with the working directory set to +the execroot of your workspace. + +`processwrapper-sandbox` is a sandboxing strategy that does not require any +"advanced" features - it should work on any POSIX system out of the box. It +builds a sandbox directory consisting of symlinks that point to the original +source files, executes the action's command line with the working directory set +to this directory instead of the execroot, then moves the known output artifacts +out of the sandbox into the execroot and deletes the sandbox. This prevents the +action from accidentally using any input files that are not declared and from +littering the execroot with unknown output files. + +`linux-sandbox` goes one step further and builds on top of the +`processwrapper-sandbox`. Similar to what Docker does under the hood, it uses +Linux Namespaces (User, Mount, PID, Network and IPC namespaces) to isolate the +action from the host. That is, it makes the entire filesystem read-only except +for the sandbox directory, so the action cannot accidentally modify anything on +the host filesystem. This prevents situations like a buggy test accidentally rm +-rf'ing your $HOME directory. Optionally, you can also prevent the action from +accessing the network. `linux-sandbox` uses PID namespaces to prevent the action +from seeing any other processes and to reliably kill all processes (even daemons +spawned by the action) at the end. + +`darwin-sandbox` is similar, but for macOS. It uses Apple's `sandbox-exec` tool +to achieve roughly the same as the Linux sandbox. + +Both the `linux-sandbox` and the `darwin-sandbox` do not work in a "nested" +scenario due to restrictions in the mechanisms provided by the operating +systems. Because Docker also uses Linux namespaces for its container magic, you +cannot easily run `linux-sandbox` inside a Docker container, unless you use +`docker run --privileged`. On macOS, you cannot run `sandbox-exec` inside a +process that's already being sandboxed. Thus, in these cases, Bazel +automatically falls back to using `processwrapper-sandbox`. + +If you would rather get a build error — such as to not accidentally build with a +less strict execution strategy — explicitly modify the list of execution +strategies that Bazel tries to use (for example, `bazel build +--spawn_strategy=worker,linux-sandbox`). + +Dynamic execution usually requires sandboxing for local execution. To opt out, +pass the `--experimental_local_lockfree_output` flag. Dynamic execution silently +sandboxes [persistent workers](/remote/persistent). ## Downsides to sandboxing -- Sandboxing incurs extra setup and teardown cost. How big this cost is depends on many factors, including the shape of the build and the performance of the host OS. For Linux, sandboxed builds are rarely more than a few percent slower. Setting `--reuse_sandbox_directories` can mitigate the setup and teardown cost. +- Sandboxing incurs extra setup and teardown cost. How big this cost is + depends on many factors, including the shape of the build and the + performance of the host OS. For Linux, sandboxed builds are rarely more than + a few percent slower. Setting `--reuse_sandbox_directories` can + mitigate the setup and teardown cost. -- Sandboxing effectively disables any cache the tool may have. You can mitigate this by using [persistent workers](/remote/persistent), at the cost of weaker sandbox guarantees. +- Sandboxing effectively disables any cache the tool may have. You can + mitigate this by using [persistent workers](/remote/persistent), at + the cost of weaker sandbox guarantees. -- [Multiplex workers](/remote/multiplex) require explicit worker support to be sandboxed. Workers that do not support multiplex sandboxing run as singleplex workers under dynamic execution, which can cost extra memory. +- [Multiplex workers](/remote/multiplex) require explicit worker support + to be sandboxed. Workers that do not support multiplex sandboxing run as + singleplex workers under dynamic execution, which can cost extra memory. ## Debugging @@ -52,7 +122,11 @@ Follow the strategies below to debug issues with sandboxing. ### Deactivated namespaces -On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file exists and contains a 0, you can activate user namespaces by running: +On some platforms, such as +[Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) +cluster nodes or Debian, user namespaces are deactivated by default due to +security concerns. If the `/proc/sys/kernel/unprivileged_userns_clone` file +exists and contains a 0, you can activate user namespaces by running: ```posix-terminal sudo sysctl kernel.unprivileged_userns_clone=1 @@ -60,11 +134,16 @@ On some platforms, such as [Google Kubernetes Engine](https://cloud.google.com/k ### Rule execution failures -The sandbox may fail to execute rules because of the system setup. If you see a message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for genrules, and `--spawn_strategy=local` for other rules. +The sandbox may fail to execute rules because of the system setup. If you see a +message like `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or +directory`, try to deactivate the sandbox with `--strategy=Genrule=local` for +genrules, and `--spawn_strategy=local` for other rules. ### Detailed debugging for build failures -If your build failed, use `--verbose_failures` and `--sandbox_debug` to make Bazel show the exact command it ran when your build failed, including the part that sets up the sandbox. +If your build failed, use `--verbose_failures` and `--sandbox_debug` to make +Bazel show the exact command it ran when your build failed, including the part +that sets up the sandbox. Example error message: @@ -87,6 +166,9 @@ namespace-sandbox failed: error executing command /some/path/to/your/some-compiler --some-params some-target) ``` -You can now inspect the generated sandbox directory and see which files Bazel created and run the command again to see how it behaves. +You can now inspect the generated sandbox directory and see which files Bazel +created and run the command again to see how it behaves. -Note that Bazel does not delete the sandbox directory when you use `--sandbox_debug`. Unless you are actively debugging, you should disable `--sandbox_debug` because it fills up your disk over time. +Note that Bazel does not delete the sandbox directory when you use +`--sandbox_debug`. Unless you are actively debugging, you should disable +`--sandbox_debug` because it fills up your disk over time. diff --git a/docs/user-manual.mdx b/docs/user-manual.mdx deleted file mode 100644 index cbe48b01..00000000 --- a/docs/user-manual.mdx +++ /dev/null @@ -1,1443 +0,0 @@ ---- -title: 'Commands and Options' ---- - -This page covers the options that are available with various Bazel commands, such as `bazel build`, `bazel run`, and `bazel test`. This page is a companion to the list of Bazel's commands in [Build with Bazel](/run/build). - -## Target syntax - -Some commands, like `build` or `test`, can operate on a list of targets. They use a syntax more flexible than labels, which is documented in [Specifying targets to build](/run/build#specifying-build-targets). - -## Options - -The following sections describe the options available during a build. When `--long` is used on a help command, the on-line help messages provide summary information about the meaning, type and default value for each option. - -Most options can only be specified once. When specified multiple times, the last instance wins. Options that can be specified multiple times are identified in the on-line help with the text 'may be used multiple times'. - -### Package location - -#### `--package_path` - -**WARNING:** The `--package_path` option is deprecated. Bazel prefers packages in the main repository to be under the workspace root. - -This option specifies the set of directories that are searched to find the BUILD file for a given package. - -Bazel finds its packages by searching the package path. This is a colon separated ordered list of bazel directories, each being the root of a partial source tree. - -*To specify a custom package path* using the `--package_path` option: - -``` - % bazel build --package_path %workspace%:/some/other/root -``` - -Package path elements may be specified in three formats: - -1. If the first character is `/`, the path is absolute. -2. If the path starts with `%workspace%`, the path is taken relative to the nearest enclosing bazel directory. For instance, if your working directory is `/home/bob/clients/bob_client/bazel/foo`, then the string `%workspace%` in the package-path is expanded to `/home/bob/clients/bob_client/bazel`. -3. Anything else is taken relative to the working directory. This is usually not what you mean to do, and may behave unexpectedly if you use Bazel from directories below the bazel workspace. For instance, if you use the package-path element `.`, and then cd into the directory `/home/bob/clients/bob_client/bazel/foo`, packages will be resolved from the `/home/bob/clients/bob_client/bazel/foo` directory. - -If you use a non-default package path, specify it in your [Bazel configuration file](/run/bazelrc) for convenience. - -*Bazel doesn't require any packages to be in the current directory*, so you can do a build from an empty bazel workspace if all the necessary packages can be found somewhere else on the package path. - -Example: Building from an empty client - -``` - % mkdir -p foo/bazel - % cd foo/bazel - % touch MODULE.bazel - % bazel build --package_path /some/other/path //foo -``` - -#### `--deleted_packages` - -This option specifies a comma-separated list of packages which Bazel should consider deleted, and not attempt to load from any directory on the package path. This can be used to simulate the deletion of packages without actually deleting them. This option can be passed multiple times, in which case the individual lists are concatenated. - -### Error checking - -These options control Bazel's error-checking and/or warnings. - -#### `--[no]check_visibility` - -If this option is set to false, visibility checks are demoted to warnings. The default value of this option is true, so that by default, visibility checking is done. - -#### `--output_filter=<var>regex</var>` - -The `--output_filter` option will only show build and compilation warnings for targets that match the regular expression. If a target does not match the given regular expression and its execution succeeds, its standard output and standard error are thrown away. - -Here are some typical values for this option: - -| | | -| ------------------------------------------------------------------------------ | --------------------------------------------- | -| \`--output\_filter='^//(first/project\|second/project):'\` | Show the output for the specified packages. | -| \`--output\_filter='^//((?!(first/bad\_project\|second/bad\_project):).)\*$'\` | Don't show output for the specified packages. | -| \`--output\_filter=\` | Show everything. | -| \`--output\_filter=DONT\_MATCH\_ANYTHING\` | Show nothing. | - -### Tool flags - -These options control which options Bazel will pass to other tools. - -#### `--copt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler. The argument will be passed to the compiler whenever it is invoked for preprocessing, compiling, and/or assembling C, C++, or assembler code. It will not be passed when linking. - -This option can be used multiple times. For example: - -``` - % bazel build --copt="-g0" --copt="-fpic" //foo -``` - -will compile the `foo` library without debug tables, generating position-independent code. - -Note: Changing `--copt` settings will force a recompilation of all affected object files. Also note that copts values listed in specific cc\_library or cc\_binary build rules will be placed on the compiler command line *after* these options. - -Warning: C++-specific options (such as `-fno-implicit-templates`) should be specified in `--cxxopt`, not in `--copt`. Likewise, C-specific options (such as -Wstrict-prototypes) should be specified in `--conlyopt`, not in `copt`. Similarly, compiler options that only have an effect at link time (such as `-l`) should be specified in `--linkopt`, not in `--copt`. - -#### `--host_copt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler for source files that are compiled in the exec configuration. This is analogous to the [`--copt`](#copt) option, but applies only to the exec configuration. - -#### `--host_conlyopt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler for C source files that are compiled in the exec configuration. This is analogous to the [`--conlyopt`](#cconlyopt) option, but applies only to the exec configuration. - -#### `--host_cxxopt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler for C++ source files that are compiled in the exec configuration. This is analogous to the [`--cxxopt`](#cxxopt) option, but applies only to the exec configuration. - -#### `--host_linkopt=<var>linker-option</var>` - -This option takes an argument which is to be passed to the linker for source files that are compiled in the exec configuration. This is analogous to the [`--linkopt`](#linkopt) option, but applies only to the exec configuration. - -#### `--conlyopt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler when compiling C source files. - -This is similar to `--copt`, but only applies to C compilation, not to C++ compilation or linking. So you can pass C-specific options (such as `-Wno-pointer-sign`) using `--conlyopt`. - -Note: copts parameters listed in specific cc\_library or cc\_binary build rules are placed on the compiler command line *after* these options. - -#### `--cxxopt=<var>cc-option</var>` - -This option takes an argument which is to be passed to the compiler when compiling C++ source files. - -This is similar to `--copt`, but only applies to C++ compilation, not to C compilation or linking. So you can pass C++-specific options (such as `-fpermissive` or `-fno-implicit-templates`) using `--cxxopt`. - -For example: - -``` - % bazel build --cxxopt="-fpermissive" --cxxopt="-Wno-error" //foo/cruddy_code -``` - -Note: copts parameters listed in specific cc\_library or cc\_binary build rules are placed on the compiler command line *after* these options. - -#### `--linkopt=<var>linker-option</var>` - -This option takes an argument which is to be passed to the compiler when linking. - -This is similar to `--copt`, but only applies to linking, not to compilation. So you can pass compiler options that only make sense at link time (such as `-lssp` or `-Wl,--wrap,abort`) using `--linkopt`. For example: - -``` - % bazel build --copt="-fmudflap" --linkopt="-lmudflap" //foo/buggy_code -``` - -Build rules can also specify link options in their attributes. This option's settings always take precedence. Also see [cc\_library.linkopts](/reference/be/c-cpp#cc_library.linkopts). - -#### `--strip (always|never|sometimes)` - -This option determines whether Bazel will strip debugging information from all binaries and shared libraries, by invoking the linker with the `-Wl,--strip-debug` option. `--strip=always` means always strip debugging information. `--strip=never` means never strip debugging information. The default value of `--strip=sometimes` means strip if the `--compilation_mode` is `fastbuild`. - -``` - % bazel build --strip=always //foo:bar -``` - -will compile the target while stripping debugging information from all generated binaries. - -Note: If you want debugging information, it's not enough to disable stripping; you also need to make sure that the debugging information was generated by the compiler, which you can do by using either `-c dbg` or `--copt -g`. - -Bazel's `--strip` option corresponds with ld's `--strip-debug` option: it only strips debugging information. If for some reason you want to strip *all* symbols, not just *debug* symbols, you would need to use ld's `--strip-all` option, which you can do by passing `--linkopt=-Wl,--strip-all` to Bazel. Also be aware that setting Bazel's `--strip` flag will override `--linkopt=-Wl,--strip-all`, so you should only set one or the other. - -If you are only building a single binary and want all symbols stripped, you could also pass `--stripopt=--strip-all` and explicitly build the `//foo:bar.stripped` version of the target. As described in the section on `--stripopt`, this applies a strip action after the final binary is linked rather than including stripping in all of the build's link actions. - -#### `--stripopt=<var>strip-option</var>` - -This is an additional option to pass to the `strip` command when generating a [`*.stripped` binary](/reference/be/c-cpp#cc_binary_implicit_outputs). The default is `-S -p`. This option can be used multiple times. - -Note: `--stripopt` does not apply to the stripping of the main binary with `[--strip](#flag--strip)=(always|sometimes)`. - -#### `--fdo_instrument=<var>profile-output-dir</var>` - -The `--fdo_instrument` option enables the generation of FDO (feedback directed optimization) profile output when the built C/C++ binary is executed. For GCC, the argument provided is used as a directory prefix for a per-object file directory tree of .gcda files containing profile information for each .o file. - -Once the profile data tree has been generated, the profile tree should be zipped up, and provided to the `--fdo_optimize=<var>profile-zip</var>` Bazel option to enable the FDO-optimized compilation. - -For the LLVM compiler the argument is also the directory under which the raw LLVM profile data file(s) is dumped. For example: `--fdo_instrument=<var>/path/to/rawprof/dir/</var>`. - -The options `--fdo_instrument` and `--fdo_optimize` cannot be used at the same time. - -#### `--fdo_optimize=<var>profile-zip</var>` - -The `--fdo_optimize` option enables the use of the per-object file profile information to perform FDO (feedback directed optimization) optimizations when compiling. For GCC, the argument provided is the zip file containing the previously-generated file tree of .gcda files containing profile information for each .o file. - -Alternatively, the argument provided can point to an auto profile identified by the extension .afdo. - -Note: This option also accepts labels that resolve to source files. You may need to add an `exports_files` directive to the corresponding package to make the file visible to Bazel. - -For the LLVM compiler the argument provided should point to the indexed LLVM profile output file prepared by the llvm-profdata tool, and should have a .profdata extension. - -The options `--fdo_instrument` and `--fdo_optimize` cannot be used at the same time. - -#### `--java_language_version=<var>version</var>` - -This option specifies the version of Java sources. For example: - -``` - % bazel build --java_language_version=8 java/com/example/common/foo:all -``` - -compiles and allows only constructs compatible with Java 8 specification. Default value is 11. --> Possible values are: 8, 9, 10, 11, 17, and 21 and may be extended by registering custom Java toolchains using `default_java_toolchain`. - -#### `--tool_java_language_version=<var>version</var>` - -The Java language version used to build tools that are executed during a build. Default value is 11. - -#### `--java_runtime_version=<var>version</var>` - -This option specifies the version of JVM to use to execute the code and run the tests. For example: - -``` - % bazel run --java_runtime_version=remotejdk_11 java/com/example/common/foo:java_application -``` - -downloads JDK 11 from a remote repository and run the Java application using it. - -Default value is `local_jdk`. Possible values are: `local_jdk`, `local_jdk_<var>version</var>`, `remotejdk_11`, `remotejdk_17`, and `remotejdk_21`. You can extend the values by registering custom JVM using either `local_java_repository` or `remote_java_repository` repository rules. - -#### `--tool_java_runtime_version=<var>version</var>` - -The version of JVM used to execute tools that are needed during a build. Default value is `remotejdk_11`. - -#### `--jvmopt=<var>jvm-option</var>` - -This option allows option arguments to be passed to the Java VM. It can be used with one big argument, or multiple times with individual arguments. For example: - -``` - % bazel build --jvmopt="-server -Xms256m" java/com/example/common/foo:all -``` - -will use the server VM for launching all Java binaries and set the startup heap size for the VM to 256 MB. - -#### `--javacopt=<var>javac-option</var>` - -This option allows option arguments to be passed to javac. It can be used with one big argument, or multiple times with individual arguments. For example: - -``` - % bazel build --javacopt="-g:source,lines" //myprojects:prog -``` - -will rebuild a java\_binary with the javac default debug info (instead of the bazel default). - -The option is passed to javac after the Bazel built-in default options for javac and before the per-rule options. The last specification of any option to javac wins. The default options for javac are: - -``` - -source 8 -target 8 -encoding UTF-8 -``` - -Note: Changing `--javacopt` settings will force a recompilation of all affected classes. Also note that javacopts parameters listed in specific java\_library or java\_binary build rules will be placed on the javac command line *after* these options. - -#### `--strict_java_deps (default|strict|off|warn|error)` - -This option controls whether javac checks for missing direct dependencies. Java targets must explicitly declare all directly used targets as dependencies. This flag instructs javac to determine the jars actually used for type checking each java file, and warn/error if they are not the output of a direct dependency of the current target. - -- `off` means checking is disabled. -- `warn` means javac will generate standard java warnings of type `[strict]` for each missing direct dependency. -- `default`, `strict` and `error` all mean javac will generate errors instead of warnings, causing the current target to fail to build if any missing direct dependencies are found. This is also the default behavior when the flag is unspecified. - -### Build semantics - -These options affect the build commands and/or the output file contents. - -#### `--compilation_mode (fastbuild|opt|dbg)` (-c) - -The `--compilation_mode` option (often shortened to `-c`, especially `-c opt`) takes an argument of `fastbuild`, `dbg` or `opt`, and affects various C/C++ code-generation options, such as the level of optimization and the completeness of debug tables. Bazel uses a different output directory for each different compilation mode, so you can switch between modes without needing to do a full rebuild *every* time. - -- `fastbuild` means build as fast as possible: generate minimal debugging information (`-gmlt -Wl,-S`), and don't optimize. This is the default. Note: `-DNDEBUG` will **not** be set. -- `dbg` means build with debugging enabled (`-g`), so that you can use gdb (or another debugger). -- `opt` means build with optimization enabled and with `assert()` calls disabled (`-O2 -DNDEBUG`). Debugging information will not be generated in `opt` mode unless you also pass `--copt -g`. - -#### `--cpu=<var>cpu</var>` - -This option specifies the target CPU architecture to be used for the compilation of binaries during the build. - -Note: A particular combination of crosstool version, compiler version, and target CPU is allowed only if it has been specified in the currently used CROSSTOOL file. - -#### `--action_env=<var>VAR=VALUE</var>` - -Specifies the set of environment variables available during the execution of all actions. Variables can be either specified by name, in which case the value will be taken from the invocation environment, or by the `name=value` pair which sets the value independent of the invocation environment. - -This `--action_env` flag can be specified multiple times. If a value is assigned to the same variable across multiple `--action_env` flags, the latest assignment wins. - -#### `--experimental_action_listener=<var>label</var>` - -Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. - -The `experimental_action_listener` option instructs Bazel to use details from the [`action_listener`](/reference/be/extra-actions#action_listener) rule specified by \{\{ "`" }}label{{ "`" \}\} to insert [`extra_actions`](/reference/be/extra-actions#extra_action) into the build graph. - -#### `--[no]experimental_extra_action_top_level_only` - -Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. - -If this option is set to true, extra actions specified by the [`--experimental_action_listener`](#experimental-action-listener) command line option will only be scheduled for top level targets. - -#### `--experimental_extra_action_filter=<var>regex</var>` - -Warning: Extra actions are deprecated. Use [aspects](/extending/aspects) instead. - -The `experimental_extra_action_filter` option instructs Bazel to filter the set of targets to schedule `extra_actions` for. - -This flag is only applicable in combination with the [`--experimental_action_listener`](#experimental-action-listener) flag. - -By default all `extra_actions` in the transitive closure of the requested targets-to-build get scheduled for execution. `--experimental_extra_action_filter` will restrict scheduling to `extra_actions` of which the owner's label matches the specified regular expression. - -The following example will limit scheduling of `extra_actions` to only apply to actions of which the owner's label contains '/bar/': - -``` -% bazel build --experimental_action_listener=//test:al //foo/... \ - --experimental_extra_action_filter=.*/bar/.* -``` - -#### `--host_cpu=<var>cpu</var>` - -This option specifies the name of the CPU architecture that should be used to build host tools. - -#### `--android_platforms=<var>platform[,platform]*</var>` - -The platforms to build the transitive `deps` of `android_binary` rules (specifically for native dependencies like C++). For example, if a `cc_library` appears in the transitive `deps` of an `android_binary` rule it is be built once for each platform specified with `--android_platforms` for the `android_binary` rule, and included in the final output. - -There is no default value for this flag: a custom Android platform must be defined and used. - -One `.so` file is created and packaged in the APK for each platform specified with `--android_platforms`. The `.so` file's name prefixes the name of the `android_binary` rule with "lib". For example, if the name of the `android_binary` is "foo", then the file is `libfoo.so`. - -#### `--per_file_copt=<var>[+-]regex[,[+-]regex]...@option[,option]...</var>` - -When present, any C++ file with a label or an execution path matching one of the inclusion regex expressions and not matching any of the exclusion expressions will be built with the given options. The label matching uses the canonical form of the label (i.e //`package`:`label_name`). - -The execution path is the relative path to your workspace directory including the base name (including extension) of the C++ file. It also includes any platform dependent prefixes. - -Note: If only one of the label or the execution path matches the options will be used. - -To match the generated files (such as genrule outputs) Bazel can only use the execution path. In this case the regexp shouldn't start with '//' since that doesn't match any execution paths. Package names can be used like this: `--per_file_copt=base/.*\.pb\.cc@-g0`. This will match every `.pb.cc` file under a directory called `base`. - -This option can be used multiple times. - -The option is applied regardless of the compilation mode used. For example, it is possible to compile with `--compilation_mode=opt` and selectively compile some files with stronger optimization turned on, or with optimization disabled. - -**Caveat**: If some files are selectively compiled with debug symbols the symbols might be stripped during linking. This can be prevented by setting `--strip=never`. - -**Syntax**: `[+-]regex[,[+-]regex]...@option[,option]...` Where `regex` stands for a regular expression that can be prefixed with a `+` to identify include patterns and with `-` to identify exclude patterns. `option` stands for an arbitrary option that is passed to the C++ compiler. If an option contains a `,` it has to be quoted like so `\,`. Options can also contain `@`, since only the first `@` is used to separate regular expressions from options. - -**Example**: `--per_file_copt=//foo:.*\.cc,-//foo:file\.cc@-O0,-fprofile-arcs` adds the `-O0` and the `-fprofile-arcs` options to the command line of the C++ compiler for all `.cc` files in `//foo/` except `file.cc`. - -#### `--dynamic_mode=<var>mode</var>` - -Determines whether C++ binaries will be linked dynamically, interacting with the [linkstatic attribute](/reference/be/c-cpp#cc_binary.linkstatic) on build rules. - -Modes: - -- `default`: Allows bazel to choose whether to link dynamically. See [linkstatic](/reference/be/c-cpp#cc_binary.linkstatic) for more information. -- `fully`: Links all targets dynamically. This will speed up linking time, and reduce the size of the resulting binaries. -- `off`: Links all targets in [mostly static](/reference/be/c-cpp#cc_binary.linkstatic) mode. If `-static` is set in linkopts, targets will change to fully static. - -#### `--fission (yes|no|[dbg][,opt][,fastbuild])` - -Enables [Fission](https://gcc.gnu.org/wiki/DebugFission), which writes C++ debug information to dedicated .dwo files instead of .o files, where it would otherwise go. This substantially reduces the input size to links and can reduce link times. - -When set to `[dbg][,opt][,fastbuild]` (example: `--fission=dbg,fastbuild`), Fission is enabled only for the specified set of compilation modes. This is useful for bazelrc settings. When set to `yes`, Fission is enabled universally. When set to `no`, Fission is disabled universally. Default is `no`. - -#### `--force_ignore_dash_static` - -If this flag is set, any `-static` options in linkopts of `cc_*` rules BUILD files are ignored. This is only intended as a workaround for C++ hardening builds. - -#### `--[no]force_pic` - -If enabled, all C++ compilations produce position-independent code ("-fPIC"), links prefer PIC pre-built libraries over non-PIC libraries, and links produce position-independent executables ("-pie"). Default is disabled. - -Note: Dynamically linked binaries (for example `--dynamic_mode fully`) generate PIC code regardless of this flag's setting. So this flag is for cases where users want PIC code explicitly generated for static links. - -#### `--android_resource_shrinking` - -Selects whether to perform resource shrinking for android\_binary rules. Sets the default for the [shrink\_resources attribute](/reference/be/android#android_binary.shrink_resources) on android\_binary rules; see the documentation for that rule for further details. Defaults to off. - -#### `--custom_malloc=<var>malloc-library-target</var>` - -When specified, always use the given malloc implementation, overriding all `malloc="target"` attributes, including in those targets that use the default (by not specifying any `malloc`). - -#### `--crosstool_top=<var>label</var>` - -This option specifies the location of the crosstool compiler suite to be used for all C++ compilation during a build. Bazel will look in that location for a CROSSTOOL file and uses that to automatically determine settings for `--compiler`. - -#### `--host_crosstool_top=<var>label</var>` - -If not specified, Bazel uses the value of `--crosstool_top` to compile code in the exec configuration, such as tools run during the build. The main purpose of this flag is to enable cross-compilation. - -#### `--apple_crosstool_top=<var>label</var>` - -The crosstool to use for compiling C/C++ rules in the transitive `deps` of objc\_*, ios\_\_*, and apple\_\* rules. For those targets, this flag overwrites `--crosstool_top`. - -#### `--compiler=<var>version</var>` - -This option specifies the C/C++ compiler version (such as `gcc-4.1.0`) to be used for the compilation of binaries during the build. If you want to build with a custom crosstool, you should use a CROSSTOOL file instead of specifying this flag. - -Note: Only certain combinations of crosstool version, compiler version, and target CPU are allowed. - -#### `--android_sdk=<var>label</var>` - -Deprecated. This shouldn't be directly specified. - -This option specifies the Android SDK/platform toolchain and Android runtime library that will be used to build any Android-related rule. - -The Android SDK will be automatically selected if an `android_sdk_repository` rule is defined in the WORKSPACE file. - -#### `--java_toolchain=<var>label</var>` - -No-op. Kept only for backwards compatibility. - -#### `--host_java_toolchain=<var>label</var>` - -No-op. Kept only for backwards compatibility. - -#### `--javabase=(<var>label</var>)` - -No-op. Kept only for backwards compatibility. - -#### `--host_javabase=<var>label</var>` - -No-op. Kept only for backwards compatibility. - -### Execution strategy - -These options affect how Bazel will execute the build. They should not have any significant effect on the output files generated by the build. Typically their main effect is on the speed of the build. - -#### `--spawn_strategy=<var>strategy</var>` - -This option controls where and how commands are executed. - -- `standalone` causes commands to be executed as local subprocesses. This value is deprecated. Please use `local` instead. -- `sandboxed` causes commands to be executed inside a sandbox on the local machine. This requires that all input files, data dependencies and tools are listed as direct dependencies in the `srcs`, `data` and `tools` attributes. Bazel enables local sandboxing by default, on systems that support sandboxed execution. -- `local` causes commands to be executed as local subprocesses. -- `worker` causes commands to be executed using a persistent worker, if available. -- `docker` causes commands to be executed inside a docker sandbox on the local machine. This requires that docker is installed. -- `remote` causes commands to be executed remotely; this is only available if a remote executor has been configured separately. - -#### `--strategy <var>mnemonic</var>=<var>strategy</var>` - -This option controls where and how commands are executed, overriding the [--spawn\_strategy](#spawn-strategy) (and [--genrule\_strategy](#genrule-strategy) with mnemonic Genrule) on a per-mnemonic basis. See [--spawn\_strategy](#spawn-strategy) for the supported strategies and their effects. - -#### `--strategy_regexp=<var><filter,filter,...>=<strategy></var>` - -This option specifies which strategy should be used to execute commands that have descriptions matching a certain `regex_filter`. See [--per\_file\_copt](#per-file-copt) for details on regex\_filter matching. See [--spawn\_strategy](#spawn-strategy) for the supported strategies and their effects. - -The last `regex_filter` that matches the description is used. This option overrides other flags for specifying strategy. - -- Example: `--strategy_regexp=//foo.*\\.cc,-//foo/bar=local` means to run actions using `local` strategy if their descriptions match //foo.\*.cc but not //foo/bar. -- Example: `--strategy_regexp='Compiling.*/bar=local' --strategy_regexp=Compiling=sandboxed` runs 'Compiling //foo/bar/baz' with the `sandboxed` strategy, but reversing the order runs it with `local`. -- Example: `--strategy_regexp='Compiling.*/bar=local,sandboxed'` runs 'Compiling //foo/bar/baz' with the `local` strategy and falls back to `sandboxed` if it fails. - -#### `--genrule_strategy=<var>strategy</var>` - -This is a deprecated short-hand for `--strategy=Genrule=<var>strategy</var>`. - -#### `--jobs=<var>n</var>` (-j) - -This option, which takes an integer argument, specifies a limit on the number of jobs that should be executed concurrently during the execution phase of the build. - -Note : The number of concurrent jobs that Bazel will run is determined not only by the `--jobs` setting, but also by Bazel's scheduler, which tries to avoid running concurrent jobs that will use up more resources (RAM or CPU) than are available, based on some (very crude) estimates of the resource consumption of each job. The behavior of the scheduler can be controlled by the `--local_resources` option. - -#### `--progress_report_interval=<var>n</var>` - -Bazel periodically prints a progress report on jobs that are not finished yet (such as long running tests). This option sets the reporting frequency, progress will be printed every `n` seconds. - -The default is 0, that means an incremental algorithm: the first report will be printed after 10 seconds, then 30 seconds and after that progress is reported once every minute. - -When bazel is using cursor control, as specified by [`--curses`](#curses), progress is reported every second. - -#### `--local_resources <var>resources or resource expression</var>` - -These options specify the amount of local resources (RAM in MB and number of CPU logical cores) that Bazel can take into consideration when scheduling build and test activities to run locally. They take an float, or a keyword (HOST\_RAM or HOST\_CPUS) optionally followed by `[-|*`float`]` (for example, `--local_resources=cpu=2`, `--local_resources=memory=HOST_RAM*.5`, `--local_resources=cpu=HOST_CPUS-1`). The flags are independent; one or both may be set. By default, Bazel estimates the amount of RAM and number of CPU cores directly from the local system's configuration. - -#### `--[no]build_runfile_links` - -This option, which is enabled by default, specifies whether the runfiles symlinks for tests and binaries should be built in the output directory. Using `--nobuild_runfile_links` can be useful to validate if all targets compile without incurring the overhead for building the runfiles trees. - -When tests (or applications) are executed, their run-time data dependencies are gathered together in one place. Within Bazel's output tree, this "runfiles" tree is typically rooted as a sibling of the corresponding binary or test. During test execution, runfiles may be accessed using paths of the form `$TEST_SRCDIR/<var>canonical_repo_name</var>/<var>packagename</var>/<var>filename</var>`. The runfiles tree ensures that tests have access to all the files upon which they have a declared dependence, and nothing more. By default, the runfiles tree is implemented by constructing a set of symbolic links to the required files. As the set of links grows, so does the cost of this operation, and for some large builds it can contribute significantly to overall build time, particularly because each individual test (or application) requires its own runfiles tree. - -#### `--[no]build_runfile_manifests` - -This option, which is enabled by default, specifies whether runfiles manifests should be written to the output tree. Disabling it implies `--nobuild_runfile_links`. - -It can be disabled when executing tests remotely, as runfiles trees will be created remotely from in-memory manifests. - -#### `--[no]discard_analysis_cache` - -When this option is enabled, Bazel will discard the analysis cache right before execution starts, thus freeing up additional memory (around 10%) for the [execution phase](/run/build#execution). The drawback is that further incremental builds will be slower. See also [memory-saving mode](/configure/memory). - -#### `--[no]keep_going` (-k) - -As in GNU Make, the execution phase of a build stops when the first error is encountered. Sometimes it is useful to try to build as much as possible even in the face of errors. This option enables that behavior, and when it is specified, the build will attempt to build every target whose prerequisites were successfully built, but will ignore errors. - -While this option is usually associated with the execution phase of a build, it also affects the analysis phase: if several targets are specified in a build command, but only some of them can be successfully analyzed, the build will stop with an error unless `--keep_going` is specified, in which case the build will proceed to the execution phase, but only for the targets that were successfully analyzed. - -#### `--[no]use_ijars` - -This option changes the way `java_library` targets are compiled by Bazel. Instead of using the output of a `java_library` for compiling dependent `java_library` targets, Bazel will create interface jars that contain only the signatures of non-private members (public, protected, and default (package) access methods and fields) and use the interface jars to compile the dependent targets. This makes it possible to avoid recompilation when changes are only made to method bodies or private members of a class. - -Note: Using `--use_ijars` might give you a different error message when you are accidentally referring to a non visible member of another class: Instead of getting an error that the member is not visible you will get an error that the member does not exist. Changing the `--use_ijars` setting will force a recompilation of all affected classes. - -#### `--[no]interface_shared_objects` - -This option enables *interface shared objects*, which makes binaries and other shared libraries depend on the *interface* of a shared object, rather than its implementation. When only the implementation changes, Bazel can avoid rebuilding targets that depend on the changed shared library unnecessarily. - -### Output selection - -These options determine what to build or test. - -#### `--[no]build` - -This option causes the execution phase of the build to occur; it is on by default. When it is switched off, the execution phase is skipped, and only the first two phases, loading and analysis, occur. - -This option can be useful for validating BUILD files and detecting errors in the inputs, without actually building anything. - -#### `--[no]build_tests_only` - -If specified, Bazel will build only what is necessary to run the `*_test` and `test_suite` rules that were not filtered due to their [size](#test-size-filters), [timeout](#test-timeout-filters), [tag](#test-tag-filters), or [language](#test-lang-filters). If specified, Bazel will ignore other targets specified on the command line. By default, this option is disabled and Bazel will build everything requested, including `*_test` and `test_suite` rules that are filtered out from testing. This is useful because running `bazel test --build_tests_only foo/...` may not detect all build breakages in the `foo` tree. - -#### `--[no]check_up_to_date` - -This option causes Bazel not to perform a build, but merely check whether all specified targets are up-to-date. If so, the build completes successfully, as usual. However, if any files are out of date, instead of being built, an error is reported and the build fails. This option may be useful to determine whether a build has been performed more recently than a source edit (for example, for pre-submit checks) without incurring the cost of a build. - -See also [`--check_tests_up_to_date`](#check-tests-up-to-date). - -#### `--[no]compile_one_dependency` - -Compile a single dependency of the argument files. This is useful for syntax checking source files in IDEs, for example, by rebuilding a single target that depends on the source file to detect errors as early as possible in the edit/build/test cycle. This argument affects the way all non-flag arguments are interpreted: each argument must be a file target label or a plain filename relative to the current working directory, and one rule that depends on each source filename is built. For C++ and Java sources, rules in the same language space are preferentially chosen. For multiple rules with the same preference, the one that appears first in the BUILD file is chosen. An explicitly named target pattern which does not reference a source file results in an error. - -#### `--save_temps` - -The `--save_temps` option causes temporary outputs from the compiler to be saved. These include .s files (assembler code), .i (preprocessed C) and .ii (preprocessed C++) files. These outputs are often useful for debugging. Temps will only be generated for the set of targets specified on the command line. - -Note: The implementation of `--save_temps` does not use the compiler's `-save-temps` flag. Instead, there are two passes, one with `-S` and one with `-E`. A consequence of this is that if your build fails, Bazel may not yet have produced the ".i" or ".ii" and ".s" files. If you're trying to use `--save_temps` to debug a failed compilation, you may need to also use `--keep_going` so that Bazel will still try to produce the preprocessed files after the compilation fails. - -The `--save_temps` flag currently works only for cc\_\* rules. - -To ensure that Bazel prints the location of the additional output files, check that your [`--show_result <var>n</var>`](#show-result) setting is high enough. - -#### `--build_tag_filters=<var>tag[,tag]*</var>` - -If specified, Bazel will build only targets that have at least one required tag (if any of them are specified) and does not have any excluded tags. Build tag filter is specified as comma delimited list of tag keywords, optionally preceded with '-' sign used to denote excluded tags. Required tags may also have a preceding '+' sign. - -When running tests, Bazel ignores `--build_tag_filters` for test targets, which are built and run even if they do not match this filter. To avoid building them, filter test targets using `--test_tag_filters` or by explicitly excluding them. - -#### `--test_size_filters=<var>size[,size]*</var>` - -If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets with the given size. Test size filter is specified as comma delimited list of allowed test size values (small, medium, large or enormous), optionally preceded with '-' sign used to denote excluded test sizes. For example, - -``` - % bazel test --test_size_filters=small,medium //foo:all -``` - -and - -``` - % bazel test --test_size_filters=-large,-enormous //foo:all -``` - -will test only small and medium tests inside //foo. - -By default, test size filtering is not applied. - -#### `--test_timeout_filters=<var>timeout[,timeout]*</var>` - -If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets with the given timeout. Test timeout filter is specified as comma delimited list of allowed test timeout values (short, moderate, long or eternal), optionally preceded with '-' sign used to denote excluded test timeouts. See [--test\_size\_filters](#test-size-filters) for example syntax. - -By default, test timeout filtering is not applied. - -#### `--test_tag_filters=<var>tag[,tag]*</var>` - -If specified, Bazel will test (or build if `--build_tests_only` is also specified) only test targets that have at least one required tag (if any of them are specified) and does not have any excluded tags. Test tag filter is specified as comma delimited list of tag keywords, optionally preceded with '-' sign used to denote excluded tags. Required tags may also have a preceding '+' sign. - -For example, - -``` - % bazel test --test_tag_filters=performance,stress,-flaky //myproject:all -``` - -will test targets that are tagged with either `performance` or `stress` tag but are **not** tagged with the `flaky` tag. - -By default, test tag filtering is not applied. Note that you can also filter on test's `size` and `local` tags in this manner. - -#### `--test_lang_filters=<var>string[,string]*</var>` - -Specifies a comma-separated list of strings referring to names of test rule classes. To refer to the rule class `foo_test`, use the string "foo". Bazel will test (or build if `--build_tests_only` is also specified) only targets of the referenced rule classes. To instead exclude those targets, use the string "-foo". For example, - -``` - % bazel test --test_lang_filters=foo,bar //baz/... -``` - -will test only targets that are instances of \`foo\_test\` or \`bar\_test\` in \`//baz/...\`, while - -``` - % bazel test --test_lang_filters=-foo,-bar //baz/... -``` - -will test all the targets in \`//baz/...\` except for the \`foo\_test\` and \`bar\_test\` instances. - -Tip: You can use `bazel query --output=label_kind "//p:t"` to learn the rule class name of the target `//p:t`. And you can look at the pair of instantiation stacks in the output of `bazel query --output=build "//p:t"` to learn why that target is an instance of that rule class. - -Warning: The option name "--test\_lang\_filter" is vestigal and is therefore unfortunately misleading; don't make assumptions about the semantics based on the name. - -#### `--test_filter=<var>filter-expression</var>` - -Specifies a filter that the test runner may use to pick a subset of tests for running. All targets specified in the invocation are built, but depending on the expression only some of them may be executed; in some cases, only certain test methods are run. - -The particular interpretation of \{\{ "`" }}filter-expression{{ "`" \}\} is up to the test framework responsible for running the test. It may be a glob, substring, or regexp. `--test_filter` is a convenience over passing different `--test_arg` filter arguments, but not all frameworks support it. - -### Verbosity - -These options control the verbosity of Bazel's output, either to the terminal, or to additional log files. - -#### `--explain=<var>logfile</var>` - -This option, which requires a filename argument, causes the dependency checker in `bazel build`'s execution phase to explain, for each build step, either why it is being executed, or that it is up-to-date. The explanation is written to *logfile*. - -If you are encountering unexpected rebuilds, this option can help to understand the reason. Add it to your `.bazelrc` so that logging occurs for all subsequent builds, and then inspect the log when you see an execution step executed unexpectedly. This option may carry a small performance penalty, so you might want to remove it when it is no longer needed. - -#### `--verbose_explanations` - -This option increases the verbosity of the explanations generated when the [--explain](#explain) option is enabled. - -In particular, if verbose explanations are enabled, and an output file is rebuilt because the command used to build it has changed, then the output in the explanation file will include the full details of the new command (at least for most commands). - -Using this option may significantly increase the length of the generated explanation file and the performance penalty of using `--explain`. - -If `--explain` is not enabled, then `--verbose_explanations` has no effect. - -#### `--profile=<var>file</var>` - -This option, which takes a filename argument, causes Bazel to write profiling data into a file. The data then can be analyzed or parsed using the `bazel analyze-profile` command. The Build profile can be useful in understanding where Bazel's `build` command is spending its time. - -#### `--[no]show_loading_progress` - -This option causes Bazel to output package-loading progress messages. If it is disabled, the messages won't be shown. - -#### `--[no]show_progress` - -This option causes progress messages to be displayed; it is on by default. When disabled, progress messages are suppressed. - -#### `--show_progress_rate_limit=<var>n</var>` - -This option causes bazel to display at most one progress message per `n` seconds, where \{\{ "`" }}n{{ "`" \}\} is a real number. The default value for this option is 0.02, meaning bazel will limit the progress messages to one per every 0.02 seconds. - -#### `--show_result=<var>n</var>` - -This option controls the printing of result information at the end of a `bazel build` command. By default, if a single build target was specified, Bazel prints a message stating whether or not the target was successfully brought up-to-date, and if so, the list of output files that the target created. If multiple targets were specified, result information is not displayed. - -While the result information may be useful for builds of a single target or a few targets, for large builds (such as an entire top-level project tree), this information can be overwhelming and distracting; this option allows it to be controlled. `--show_result` takes an integer argument, which is the maximum number of targets for which full result information should be printed. By default, the value is 1. Above this threshold, no result information is shown for individual targets. Thus zero causes the result information to be suppressed always, and a very large value causes the result to be printed always. - -Users may wish to choose a value in-between if they regularly alternate between building a small group of targets (for example, during the compile-edit-test cycle) and a large group of targets (for example, when establishing a new workspace or running regression tests). In the former case, the result information is very useful whereas in the latter case it is less so. As with all options, this can be specified implicitly via the [`.bazelrc`](/run/bazelrc) file. - -The files are printed so as to make it easy to copy and paste the filename to the shell, to run built executables. The "up-to-date" or "failed" messages for each target can be easily parsed by scripts which drive a build. - -#### `--sandbox_debug` - -This option causes Bazel to print extra debugging information when using sandboxing for action execution. This option also preserves sandbox directories, so that the files visible to actions during execution can be examined. - -#### `--subcommands` (`-s`) - -This option causes Bazel's execution phase to print the full command line for each command prior to executing it. - -``` - >>>>> # //examples/cpp:hello-world [action 'Linking examples/cpp/hello-world'] - (cd /home/johndoe/.cache/bazel/_bazel_johndoe/4c084335afceb392cfbe7c31afee3a9f/bazel && \ - exec env - \ - /usr/bin/gcc -o bazel-out/local-fastbuild/bin/examples/cpp/hello-world -B/usr/bin/ -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes -Wl,-S -Wl,@bazel-out/local_linux-fastbuild/bin/examples/cpp/hello-world-2.params) -``` - -Where possible, commands are printed in a Bourne shell compatible syntax, so that they can be easily copied and pasted to a shell command prompt. (The surrounding parentheses are provided to protect your shell from the `cd` and `exec` calls; be sure to copy them!) However some commands are implemented internally within Bazel, such as creating symlink trees. For these there's no command line to display. - -`--subcommands=pretty_print` may be passed to print the arguments of the command as a list rather than as a single line. This may help make long command lines more readable. - -See also [--verbose\_failures](#verbose-failures), below. - -For logging subcommands to a file in a tool-friendly format, see [--execution\_log\_json\_file](/reference/command-line-reference#flag--execution_log_json_file) and [--execution\_log\_binary\_file](/reference/command-line-reference#flag--execution_log_binary_file). - -#### `--verbose_failures` - -This option causes Bazel's execution phase to print the full command line for commands that failed. This can be invaluable for debugging a failing build. - -Failing commands are printed in a Bourne shell compatible syntax, suitable for copying and pasting to a shell prompt. - -### Workspace status - -Use these options to "stamp" Bazel-built binaries: to embed additional information into the binaries, such as the source control revision or other workspace-related information. You can use this mechanism with rules that support the `stamp` attribute, such as `genrule`, `cc_binary`, and more. - -#### `--workspace_status_command=<var>program</var>` - -This flag lets you specify a binary that Bazel runs before each build. The program can report information about the status of the workspace, such as the current source control revision. - -The flag's value must be a path to a native program. On Linux/macOS this may be any executable. On Windows this must be a native binary, typically an ".exe", ".bat", or a ".cmd" file. - -The program should print zero or more key/value pairs to standard output, one entry on each line, then exit with zero (otherwise the build fails). The key names can be anything but they may only use upper case letters and underscores. The first space after the key name separates it from the value. The value is the rest of the line (including additional whitespaces). Neither the key nor the value may span multiple lines. Keys must not be duplicated. - -Bazel partitions the keys into two buckets: "stable" and "volatile". (The names "stable" and "volatile" are a bit counter-intuitive, so don't think much about them.) - -Bazel then writes the key-value pairs into two files: - -- `bazel-out/stable-status.txt` contains all keys and values where the key's name starts with `STABLE_` -- `bazel-out/volatile-status.txt` contains the rest of the keys and their values - -The contract is: - -- "stable" keys' values should change rarely, if possible. If the contents of `bazel-out/stable-status.txt` change, Bazel invalidates the actions that depend on them. In other words, if a stable key's value changes, Bazel will rerun stamped actions. Therefore the stable status should not contain things like timestamps, because they change all the time, and would make Bazel rerun stamped actions with each build. - - Bazel always outputs the following stable keys: - - - `BUILD_EMBED_LABEL`: value of `--embed_label` - - `BUILD_HOST`: the name of the host machine that Bazel is running on - - `BUILD_USER`: the name of the user that Bazel is running as - -- "volatile" keys' values may change often. Bazel expects them to change all the time, like timestamps do, and duly updates the `bazel-out/volatile-status.txt` file. In order to avoid rerunning stamped actions all the time though, **Bazel pretends that the volatile file never changes**. In other words, if the volatile status file is the only file whose contents has changed, Bazel will not invalidate actions that depend on it. If other inputs of the actions have changed, then Bazel reruns that action, and the action will see the updated volatile status, but just the volatile status changing alone will not invalidate the action. - - Bazel always outputs the following volatile keys: - - - `BUILD_TIMESTAMP`: time of the build in seconds since the Unix Epoch (the value of `System.currentTimeMillis()` divided by a thousand) - - `FORMATTED_DATE`: time of the build Formatted as `yyyy MMM d HH mm ss EEE`(for example 2023 Jun 2 01 44 29 Fri) in UTC. - -On Linux/macOS you can pass `--workspace_status_command=/bin/true` to disable retrieving workspace status, because `true` does nothing, successfully (exits with zero) and prints no output. On Windows you can pass the path of MSYS's `true.exe` for the same effect. - -If the workspace status command fails (exits non-zero) for any reason, the build will fail. - -Example program on Linux using Git: - -``` -#!/bin/bash -echo "CURRENT_TIME $(date +%s)" -echo "RANDOM_HASH $(cat /proc/sys/kernel/random/uuid)" -echo "STABLE_GIT_COMMIT $(git rev-parse HEAD)" -echo "STABLE_USER_NAME $USER" -``` - -Pass this program's path with `--workspace_status_command`, and the stable status file will include the STABLE lines and the volatile status file will include the rest of the lines. - -#### `--[no]stamp` - -This option, in conjunction with the `stamp` rule attribute, controls whether to embed build information in binaries. - -Stamping can be enabled or disabled explicitly on a per-rule basis using the `stamp` attribute. Please refer to the Build Encyclopedia for details. When a rule sets `stamp = -1` (the default for `*_binary` rules), this option determines whether stamping is enabled. - -Bazel never stamps binaries that are built for the exec configuration, regardless of this option or the `stamp` attribute. For rules that set `stamp = 0` (the default for `*_test` rules), stamping is disabled regardless of `--[no]stamp`. Specifying `--stamp` does not force targets to be rebuilt if their dependencies have not changed. - -Setting `--nostamp` is generally desireable for build performance, as it reduces input volatility and maximizes build caching. - -### Platform - -Use these options to control the host and target platforms that configure how builds work, and to control what execution platforms and toolchains are available to Bazel rules. - -Please see background information on [Platforms](/extending/platforms) and [Toolchains](/extending/toolchains). - -#### `--platforms=<var>labels</var>` - -The labels of the platform rules describing the target platforms for the current command. - -#### `--host_platform=<var>label</var>` - -The label of a platform rule that describes the host system. - -#### `--extra_execution_platforms=<var>labels</var>` - -The platforms that are available as execution platforms to run actions. Platforms can be specified by exact target, or as a target pattern. These platforms will be considered before those declared in MODULE.bazel files by [register\_execution\_platforms()](/rules/lib/globals/module#register_execution_platforms). This option accepts a comma-separated list of platforms in order of priority. If the flag is passed multiple times, the most recent overrides. - -#### `--extra_toolchains=<var>labels</var>` - -The toolchain rules to be considered during toolchain resolution. Toolchains can be specified by exact target, or as a target pattern. These toolchains will be considered before those declared in MODULE.bazel files by [register\_toolchains()](/rules/lib/globals/module#register_toolchains). - -#### `--toolchain_resolution_debug=<var>regex</var>` - -Print debug information while finding toolchains if the toolchain type matches the regex. Multiple regexes can be separated by commas. The regex can be negated by using a `-` at the beginning. This might help developers of Bazel or Starlark rules with debugging failures due to missing toolchains. - -### Miscellaneous - -#### `--flag_alias=<var>alias_name=target_path</var>` - -A convenience flag used to bind longer Starlark build settings to a shorter name. For more details, see the [Starlark Configurations](/extending/config#using-build-setting-aliases). - -#### `--symlink_prefix=<var>string</var>` - -Changes the prefix of the generated convenience symlinks. The default value for the symlink prefix is `bazel-` which will create the symlinks `bazel-bin`, `bazel-testlogs`, and `bazel-genfiles`. - -If the symbolic links cannot be created for any reason, a warning is issued but the build is still considered a success. In particular, this allows you to build in a read-only directory or one that you have no permission to write into. Any paths printed in informational messages at the conclusion of a build will only use the symlink-relative short form if the symlinks point to the expected location; in other words, you can rely on the correctness of those paths, even if you cannot rely on the symlinks being created. - -Some common values of this option: - -- **Suppress symlink creation:** `--symlink_prefix=/` will cause Bazel to not create or update any symlinks, including the `bazel-out` and `bazel-<workspace>` symlinks. Use this option to suppress symlink creation entirely. - -- **Reduce clutter:** `--symlink_prefix=.bazel/` will cause Bazel to create symlinks called `bin` (etc) inside a hidden directory `.bazel`. - -#### `--platform_suffix=<var>string</var>` - -Adds a suffix to the configuration short name, which is used to determine the output directory. Setting this option to different values puts the files into different directories, for example to improve cache hit rates for builds that otherwise clobber each others output files, or to keep the output files around for comparisons. - -#### `--default_visibility=<var>(private|public)</var>` - -Temporary flag for testing bazel default visibility changes. Not intended for general use but documented for completeness' sake. - -#### `--starlark_cpu_profile=_file_` - -This flag, whose value is the name of a file, causes Bazel to gather statistics about CPU usage by all Starlark threads, and write the profile, in [pprof](https://github.com/google/pprof) format, to the named file. - -Use this option to help identify Starlark functions that make loading and analysis slow due to excessive computation. For example: - -``` -$ bazel build --nobuild --starlark_cpu_profile=/tmp/pprof.gz my/project/... -$ pprof /tmp/pprof.gz -(pprof) top -Type: CPU -Time: Feb 6, 2020 at 12:06pm (PST) -Duration: 5.26s, Total samples = 3.34s (63.55%) -Showing nodes accounting for 3.34s, 100% of 3.34s total - flat flat% sum% cum cum% - 1.86s 55.69% 55.69% 1.86s 55.69% sort_source_files - 1.02s 30.54% 86.23% 1.02s 30.54% expand_all_combinations - 0.44s 13.17% 99.40% 0.44s 13.17% range - 0.02s 0.6% 100% 3.34s 100% sorted - 0 0% 100% 1.38s 41.32% my/project/main/BUILD - 0 0% 100% 1.96s 58.68% my/project/library.bzl - 0 0% 100% 3.34s 100% main -``` - -For different views of the same data, try the `pprof` commands `svg`, `web`, and `list`. - -## Using Bazel for releases - -Bazel is used both by software engineers during the development cycle, and by release engineers when preparing binaries for deployment to production. This section provides a list of tips for release engineers using Bazel. - -### Significant options - -When using Bazel for release builds, the same issues arise as for other scripts that perform a build. For more details, see [Call Bazel from scripts](/run/scripts). In particular, the following options are strongly recommended: - -- [`--bazelrc=/dev/null`](/run/bazelrc) -- [`--nokeep_state_after_build`](/reference/command-line-reference#common_options-flag--keep_state_after_build) - -These options are also important: - -- [`--package_path`](#package-path) -- [`--symlink_prefix`](#symlink-prefix): for managing builds for multiple configurations, it may be convenient to distinguish each build with a distinct identifier, such as "64bit" vs. "32bit". This option differentiates the `bazel-bin` (etc.) symlinks. - -## Running tests - -To build and run tests with bazel, type `bazel test` followed by the name of the test targets. - -By default, this command performs simultaneous build and test activity, building all specified targets (including any non-test targets specified on the command line) and testing `*_test` and `test_suite` targets as soon as their prerequisites are built, meaning that test execution is interleaved with building. Doing so usually results in significant speed gains. - -### Options for `bazel test` - -#### `--cache_test_results=(yes|no|auto)` (`-t`) - -If this option is set to 'auto' (the default) then Bazel will only rerun a test if any of the following conditions applies: - -- Bazel detects changes in the test or its dependencies -- the test is marked as `external` -- multiple test runs were requested with `--runs_per_test` -- the test failed. - -If 'no', all tests will be executed unconditionally. - -If 'yes', the caching behavior will be the same as auto except that it may cache test failures and test runs with `--runs_per_test`. - -Note: Test results are *always* saved in Bazel's output tree, regardless of whether this option is enabled, so you needn't have used `--cache_test_results` on the prior run(s) of `bazel test` in order to get cache hits. The option only affects whether Bazel will *use* previously saved results, not whether it will save results of the current run. - -Users who have enabled this option by default in their `.bazelrc` file may find the abbreviations `-t` (on) or `-t-` (off) convenient for overriding the default on a particular run. - -#### `--check_tests_up_to_date` - -This option tells Bazel not to run the tests, but to merely check and report the cached test results. If there are any tests which have not been previously built and run, or whose tests results are out-of-date (for example, because the source code or the build options have changed), then Bazel will report an error message ("test result is not up-to-date"), will record the test's status as "NO STATUS" (in red, if color output is enabled), and will return a non-zero exit code. - -This option also implies [`--check_up_to_date`](#check-up-to-date) behavior. - -This option may be useful for pre-submit checks. - -#### `--test_verbose_timeout_warnings` - -This option tells Bazel to explicitly warn the user if a test's timeout is significantly longer than the test's actual execution time. While a test's timeout should be set such that it is not flaky, a test that has a highly over-generous timeout can hide real problems that crop up unexpectedly. - -For instance, a test that normally executes in a minute or two should not have a timeout of ETERNAL or LONG as these are much, much too generous. - -This option is useful to help users decide on a good timeout value or sanity check existing timeout values. - -Note: Each test shard is allotted the timeout of the entire `XX_test` target. Using this option does not affect a test's timeout value, merely warns if Bazel thinks the timeout could be restricted further. - -#### `--[no]test_keep_going` - -By default, all tests are run to completion. If this flag is disabled, however, the build is aborted on any non-passing test. Subsequent build steps and test invocations are not run, and in-flight invocations are canceled. Do not specify both `--notest_keep_going` and `--keep_going`. - -#### `--flaky_test_attempts=<var>attempts</var>` - -This option specifies the maximum number of times a test should be attempted if it fails for any reason. A test that initially fails but eventually succeeds is reported as `FLAKY` on the test summary. It is, however, considered to be passed when it comes to identifying Bazel exit code or total number of passed tests. Tests that fail all allowed attempts are considered to be failed. - -By default (when this option is not specified, or when it is set to default), only a single attempt is allowed for regular tests, and 3 for test rules with the `flaky` attribute set. You can specify an integer value to override the maximum limit of test attempts. Bazel allows a maximum of 10 test attempts in order to prevent abuse of the system. - -#### `--runs_per_test=<var>[regex@]number</var>` - -This option specifies the number of times each test should be executed. All test executions are treated as separate tests (fallback functionality will apply to each of them independently). - -The status of a target with failing runs depends on the value of the `--runs_per_test_detects_flakes` flag: - -- If absent, any failing run causes the entire test to fail. -- If present and two runs from the same shard return PASS and FAIL, the test will receive a status of flaky (unless other failing runs cause it to fail). - -If a single number is specified, all tests will run that many times. Alternatively, a regular expression may be specified using the syntax regex\@number. This constrains the effect of `--runs_per_test` to targets which match the regex (`--runs_per_test=^//pizza:.*@4` runs all tests under `//pizza/` 4 times). This form of `--runs_per_test` may be specified more than once. - -#### `--[no]runs_per_test_detects_flakes` - -If this option is specified (by default it is not), Bazel will detect flaky test shards through `--runs_per_test`. If one or more runs for a single shard fail and one or more runs for the same shard pass, the target will be considered flaky with the flag. If unspecified, the target will report a failing status. - -#### `--test_summary=<var>output_style</var>` - -Specifies how the test result summary should be displayed. - -- `short` prints the results of each test along with the name of the file containing the test output if the test failed. This is the default value. -- `terse` like `short`, but even shorter: only print information about tests which did not pass. -- `detailed` prints each individual test case that failed, not only each test. The names of test output files are omitted. -- `none` does not print test summary. - -#### `--test_output=<var>output_style</var>` - -Specifies how test output should be displayed: - -- `summary` shows a summary of whether each test passed or failed. Also shows the output log file name for failed tests. The summary will be printed at the end of the build (during the build, one would see just simple progress messages when tests start, pass or fail). This is the default behavior. -- `errors` sends combined stdout/stderr output from failed tests only into the stdout immediately after test is completed, ensuring that test output from simultaneous tests is not interleaved with each other. Prints a summary at the build as per summary output above. -- `all` is similar to `errors` but prints output for all tests, including those which passed. -- `streamed` streams stdout/stderr output from each test in real-time. - -#### `--java_debug` - -This option causes the Java virtual machine of a java test to wait for a connection from a JDWP-compliant debugger before starting the test. This option implies `--test_output=streamed`. - -#### `--[no]verbose_test_summary` - -By default this option is enabled, causing test times and other additional information (such as test attempts) to be printed to the test summary. If `--noverbose_test_summary` is specified, test summary will include only test name, test status and cached test indicator and will be formatted to stay within 80 characters when possible. - -#### `--test_tmpdir=<var>path</var>` - -Specifies temporary directory for tests executed locally. Each test will be executed in a separate subdirectory inside this directory. The directory will be cleaned at the beginning of the each `bazel test` command. By default, bazel will place this directory under Bazel output base directory. - -Note: This is a directory for running tests, not storing test results (those are always stored under the `bazel-out` directory). - -#### `--test_timeout=<var>seconds</var>` OR `--test_timeout=<var>seconds</var>,<var>seconds</var>,<var>seconds</var>,<var>seconds</var>` - -Overrides the timeout value for all tests by using specified number of seconds as a new timeout value. If only one value is provided, then it will be used for all test timeout categories. - -Alternatively, four comma-separated values may be provided, specifying individual timeouts for short, moderate, long and eternal tests (in that order). In either form, zero or a negative value for any of the test sizes will be substituted by the default timeout for the given timeout categories as defined by the page [Writing Tests](/reference/test-encyclopedia). By default, Bazel will use these timeouts for all tests by inferring the timeout limit from the test's size whether the size is implicitly or explicitly set. - -Tests which explicitly state their timeout category as distinct from their size will receive the same value as if that timeout had been implicitly set by the size tag. So a test of size 'small' which declares a 'long' timeout will have the same effective timeout that a 'large' tests has with no explicit timeout. - -#### `--test_arg=<var>arg</var>` - -Passes command-line options/flags/arguments to each test process. This option can be used multiple times to pass several arguments. For example, `--test_arg=--logtostderr --test_arg=--v=3`. - -Note that, unlike the `bazel run` command, you can't pass test arguments directly as in `bazel test -- target --logtostderr --v=3`. That's because extraneous arguments passed to `bazel test` are interpreted as additional test targets. That is, `--logtostderr` and `--v=3` would each be interpreted as a test target. This ambiguity doesn't exist for a `bazel run` command, which only accepts one target. - -`--test_arg` can be passed to a `bazel run` command, but it's ignored unless the target being run is a test target. (As with any other flag, if it's passed in a `bazel run` command after a `--` token, it's not processed by Bazel but forwarded verbatim to the executed target.) - -#### `--test_env=<var>variable</var>=_value_` OR `--test_env=<var>variable</var>` - -Specifies additional variables that must be injected into the test environment for each test. If \{\{ "`" }}value{{ "`" \}\} is not specified it will be inherited from the shell environment used to start the `bazel test` command. - -The environment can be accessed from within a test by using `System.getenv("var")` (Java), `getenv("var")` (C or C++), - -#### `--run_under=<var>command-prefix</var>` - -This specifies a prefix that the test runner will insert in front of the test command before running it. The \{\{ "`" }}command-prefix{{ "`" \}\} is split into words using Bourne shell tokenization rules, and then the list of words is prepended to the command that will be executed. - -If the first word is a fully-qualified label (starts with `//`) it is built. Then the label is substituted by the corresponding executable location that is prepended to the command that will be executed along with the other words. - -Some caveats apply: - -- The PATH used for running tests may be different than the PATH in your environment, so you may need to use an **absolute path** for the `--run_under` command (the first word in \{\{ "`" }}command-prefix{{ "`" \}\}). -- **`stdin` is not connected**, so `--run_under` can't be used for interactive commands. - -Examples: - -``` - --run_under=/usr/bin/strace - --run_under='/usr/bin/strace -c' - --run_under=/usr/bin/valgrind - --run_under='/usr/bin/valgrind --quiet --num-callers=20' -``` - -#### Test selection - -As documented under [Output selection options](#output-selection), you can filter tests by [size](#test-size-filters), [timeout](#test-timeout-filters), [tag](#test-tag-filters), or [language](#test-lang-filters). A convenience [general name filter](#test-filter) can forward particular filter args to the test runner. - -#### Other options for `bazel test` - -The syntax and the remaining options are exactly like [`bazel build`](/run/build). - -## Running executables - -The `bazel run` command is similar to `bazel build`, except it is used to build *and run* a single target. Here is a typical session (`//java/myapp:myapp` says hello and prints out its args): - -``` - % bazel run java/myapp:myapp -- --arg1 --arg2 - INFO: Analyzed target //java/myapp:myapp (13 packages loaded, 27 targets configured). - INFO: Found 1 target... - Target //java/myapp:myapp up-to-date: - bazel-bin/java/myapp/myapp - INFO: Elapsed time: 14.290s, Critical Path: 5.54s, ... - INFO: Build completed successfully, 4 total actions - INFO: Running command line: bazel-bin/java/myapp/myapp <args omitted> - Hello there - $EXEC_ROOT/java/myapp/myapp - --arg1 - --arg2 -``` - -Note: `--` is needed so that Bazel does not interpret `--arg1` and `--arg2` as Bazel options, but rather as part of the command line for running the binary. Additionally, Bazel will avoid logging these arguments to the console in case they contain sensitive information. - -`bazel run` is similar, but not identical, to directly invoking the binary built by Bazel and its behavior is different depending on whether the binary to be invoked is a test or not. - -When the binary is not a test, the current working directory will be the runfiles tree of the binary. - -When the binary is a test, the current working directory will be the exec root and a good-faith attempt is made to replicate the environment tests are usually run in. The emulation is not perfect, though, and tests that have multiple shards cannot be run this way (the `--test_sharding_strategy=disabled` command line option can be used to work around this) - -The following extra environment variables are also available to the binary: - -- `BUILD_WORKSPACE_DIRECTORY`: the root of the workspace where the build was run. -- `BUILD_WORKING_DIRECTORY`: the current working directory where Bazel was run from. -- `BUILD_ID`: the build ID of the `bazel run` invocation. This is usually unique, except if Bazel was run with `--script_path` and the resulting script is re-used. -- `BUILD_EXECROOT`: the execution root of the `bazel run` invocation. - -These can be used, for example, to interpret file names on the command line in a user-friendly way. - -### Options for `bazel run` - -#### `--run_under=<var>command-prefix</var>` - -This has the same effect as the `--run_under` option for `bazel test` ([see above](#test-run-under)), except that it applies to the command being run by `bazel run` rather than to the tests being run by `bazel test` and cannot run under label. - -#### Filtering logging outputs from Bazel - -When invoking a binary with `bazel run`, Bazel prints logging output from Bazel itself and the binary under invocation. To make the logs less noisy, you can suppress the outputs from Bazel itself with the `--ui_event_filters` and `--noshow_progress` flags. - -For example: `bazel run --ui_event_filters=-info,-stdout,-stderr --noshow_progress //java/myapp:myapp` - -### Executing tests - -`bazel run` can also execute test binaries, which has the effect of running the test in a close approximation of the environment described at [Writing Tests](/reference/test-encyclopedia). Note that none of the `--test_*` arguments have an effect when running a test in this manner except `--test_arg` . - -## Cleaning build outputs - -### The `clean` command - -Bazel has a `clean` command, analogous to that of Make. It deletes the output directories for all build configurations performed by this Bazel instance, or the entire working tree created by this Bazel instance, and resets internal caches. If executed without any command-line options, then the output directory for all configurations will be cleaned. - -Recall that each Bazel instance is associated with a single workspace, thus the `clean` command will delete all outputs from all builds you've done with that Bazel instance in that workspace. - -To completely remove the entire working tree created by a Bazel instance, you can specify the `--expunge` option. When executed with `--expunge`, the clean command simply removes the entire output base tree which, in addition to the build output, contains all temp files created by Bazel. It also stops the Bazel server after the clean, equivalent to the [`shutdown`](#shutdown) command. For example, to clean up all disk and memory traces of a Bazel instance, you could specify: - -``` - % bazel clean --expunge -``` - -Alternatively, you can expunge in the background by using `--expunge_async`. It is safe to invoke a Bazel command in the same client while the asynchronous expunge continues to run. - -Note: This may introduce IO contention. - -The `clean` command is provided primarily as a means of reclaiming disk space for workspaces that are no longer needed. Bazel's incremental rebuilds may not be perfect so `clean` can be used to recover a consistent state when problems arise. - -Bazel's design is such that these problems are fixable and these bugs are a high priority to be fixed. If you ever find an incorrect incremental build, file a bug report, and report bugs in the tools rather than using `clean`. - -## Querying the dependency graph - -Bazel includes a query language for asking questions about the dependency graph used during the build. The query language is used by two commands: query and cquery. The major difference between the two commands is that query runs after the [loading phase](/run/build#loading) and cquery runs after the [analysis phase](/run/build#analysis). These tools are an invaluable aid to many software engineering tasks. - -The query language is based on the idea of algebraic operations over graphs; it is documented in detail in - -[Bazel Query Reference](/query/language). Please refer to that document for reference, for examples, and for query-specific command-line options. - -The query tool accepts several command-line option. `--output` selects the output format. `--[no]keep_going` (disabled by default) causes the query tool to continue to make progress upon errors; this behavior may be disabled if an incomplete result is not acceptable in case of errors. - -The `--[no]tool_deps` option, enabled by default, causes dependencies in non-target configurations to be included in the dependency graph over which the query operates. - -The `--[no]implicit_deps` option, enabled by default, causes implicit dependencies to be included in the dependency graph over which the query operates. An implicit dependency is one that is not explicitly specified in the BUILD file but added by bazel. - -Example: "Show the locations of the definitions (in BUILD files) of all genrules required to build all the tests in the PEBL tree." - -``` - bazel query --output location 'kind(genrule, deps(kind(".*_test rule", foo/bar/pebl/...)))' -``` - -## Querying the action graph - -Caution: The aquery command is still experimental and its API will change. - -The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis configured target graph and exposes information about actions, artifacts and their relationships. - -The tool accepts several command-line options. `--output` selects the output format. The default output format (`text`) is human-readable, use `proto` or `textproto` for machine-readable format. Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. - -It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. - -For more details, see [Action Graph Query](/query/aquery). - -## Miscellaneous commands and options - -### `help` - -The `help` command provides on-line help. By default, it shows a summary of available commands and help topics, as shown in [Building with Bazel](/run/build#quickstart). Specifying an argument displays detailed help for a particular topic. Most topics are Bazel commands, such as `build` or `query`, but there are some additional help topics that do not correspond to commands. - -#### `--[no]long` (`-l`) - -By default, `bazel help [<var>topic</var>]` prints only a summary of the relevant options for a topic. If the `--long` option is specified, the type, default value and full description of each option is also printed. - -### `shutdown` - -Bazel server processes may be stopped by using the `shutdown` command. This command causes the Bazel server to exit as soon as it becomes idle (for example, after the completion of any builds or other commands that are currently in progress). For more details, see [Client/server implementation](/run/client-server). - -Bazel servers stop themselves after an idle timeout, so this command is rarely necessary; however, it can be useful in scripts when it is known that no further builds will occur in a given workspace. - -`shutdown` accepts one option, `--iff_heap_size_greater_than _n_`, which requires an integer argument (in MB). If specified, this makes the shutdown conditional on the amount of memory already consumed. This is useful for scripts that initiate a lot of builds, as any memory leaks in the Bazel server could cause it to crash spuriously on occasion; performing a conditional restart preempts this condition. - -### `info` - -The `info` command prints various values associated with the Bazel server instance, or with a specific build configuration. (These may be used by scripts that drive a build.) - -The `info` command also permits a single (optional) argument, which is the name of one of the keys in the list below. In this case, `bazel info <var>key</var>` will print only the value for that one key. (This is especially convenient when scripting Bazel, as it avoids the need to pipe the result through `sed -ne /key:/s/key://p`: - -#### Configuration-independent data - -- `release`: the release label for this Bazel instance, or "development version" if this is not a released binary. - -- `workspace` the absolute path to the base workspace directory. - -- `install_base`: the absolute path to the installation directory used by this Bazel instance for the current user. Bazel installs its internally required executables below this directory. - -- `output_base`: the absolute path to the base output directory used by this Bazel instance for the current user and workspace combination. Bazel puts all of its scratch and build output below this directory. - -- `execution_root`: the absolute path to the execution root directory under output\_base. This directory is the root for all files accessible to commands executed during the build, and is the working directory for those commands. If the workspace directory is writable, a symlink named `bazel-<workspace>` is placed there pointing to this directory. - -- `output_path`: the absolute path to the output directory beneath the execution root used for all files actually generated as a result of build commands. If the workspace directory is writable, a symlink named `bazel-out` is placed there pointing to this directory. - -- `server_pid`: the process ID of the Bazel server process. - -- `server_log`: the absolute path to the Bazel server's debug log file. This file contains debugging information for all commands over the lifetime of the Bazel server, and is intended for human consumption by Bazel developers and power users. - -- `command_log`: the absolute path to the command log file; this contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. - -- `used-heap-size`, `committed-heap-size`, `max-heap-size`: reports various JVM heap size parameters. Respectively: memory currently used, memory currently guaranteed to be available to the JVM from the system, maximum possible allocation. - -- `gc-count`, `gc-time`: The cumulative count of garbage collections since the start of this Bazel server and the time spent to perform them. Note that these values are not reset at the start of every build. - -- `package_path`: A colon-separated list of paths which would be searched for packages by bazel. Has the same format as the `--package_path` build command line argument. - -Example: the process ID of the Bazel server. - -``` -% bazel info server_pid -1285 -``` - -#### Configuration-specific data - -These data may be affected by the configuration options passed to `bazel info`, for example `--cpu`, `--compilation_mode`, etc. The `info` command accepts all the options that control dependency analysis, since some of these determine the location of the output directory of a build, the choice of compiler, etc. - -- `bazel-bin`, `bazel-testlogs`, `bazel-genfiles`: reports the absolute path to the `bazel-*` directories in which programs generated by the build are located. This is usually, though not always, the same as the `bazel-*` symlinks created in the base workspace directory after a successful build. However, if the workspace directory is read-only, no `bazel-*` symlinks can be created. Scripts that use the value reported by `bazel info`, instead of assuming the existence of the symlink, will be more robust. -- The complete ["Make" environment](/reference/be/make-variables). If the `--show_make_env` flag is specified, all variables in the current configuration's "Make" environment are also displayed (such as `CC`, `GLIBC_VERSION`, etc). These are the variables accessed using the `$(CC)` or `varref("CC")` syntax inside BUILD files. - -Example: the C++ compiler for the current configuration. This is the `$(CC)` variable in the "Make" environment, so the `--show_make_env` flag is needed. - -``` - % bazel info --show_make_env -c opt COMPILATION_MODE - opt -``` - -Example: the `bazel-bin` output directory for the current configuration. This is guaranteed to be correct even in cases where the `bazel-bin` symlink cannot be created for some reason (such as if you are building from a read-only directory). - -``` -% bazel info --cpu=piii bazel-bin -/var/tmp/_bazel_johndoe/fbd0e8a34f61ce5d491e3da69d959fe6/execroot/io_bazel/bazel-out/piii-opt/bin -% bazel info --cpu=k8 bazel-bin -/var/tmp/_bazel_johndoe/fbd0e8a34f61ce5d491e3da69d959fe6/execroot/io_bazel/bazel-out/k8-opt/bin -``` - -### `version` and `--version` - -The version command prints version details about the built Bazel binary, including the changelist at which it was built and the date. These are particularly useful in determining if you have the latest Bazel, or if you are reporting bugs. Some of the interesting values are: - -- `changelist`: the changelist at which this version of Bazel was released. -- `label`: the release label for this Bazel instance, or "development version" if this is not a released binary. Very useful when reporting bugs. - -`bazel --version`, with no other args, will emit the same output as `bazel version --gnu_format`, except without the side-effect of potentially starting a Bazel server or unpacking the server archive. `bazel --version` can be run from anywhere - it does not require a workspace directory. - -### `mobile-install` - -The `mobile-install` command installs apps to mobile devices. Currently only Android devices running ART are supported. - -See [bazel mobile-install](/docs/mobile-install) for more information. - -Note: This command does not install the same thing that `bazel build` produces: Bazel tweaks the app so that it can be built, installed and re-installed quickly. This should, however, be mostly transparent to the app. - -The following options are supported: - -#### `--incremental` - -If set, Bazel tries to install the app incrementally, that is, only those parts that have changed since the last build. This cannot update resources referenced from `AndroidManifest.xml`, native code or Java resources (such as those referenced by `Class.getResource()`). If these things change, this option must be omitted. Contrary to the spirit of Bazel and due to limitations of the Android platform, it is the **responsibility of the user** to know when this command is good enough and when a full install is needed. - -If you are using a device with Marshmallow or later, consider the [`--split_apks`](#split-apks) flag. - -#### `--split_apks` - -Whether to use split apks to install and update the application on the device. Works only with devices with Marshmallow or later. Note that the [`--incremental`](#incremental) flag is not necessary when using `--split_apks`. - -#### `--start_app` - -Starts the app in a clean state after installing. Equivalent to `--start=COLD`. - -#### `--debug_app` - -Waits for debugger to be attached before starting the app in a clean state after installing. Equivalent to `--start=DEBUG`. - -#### `--start=_start_type_` - -How the app should be started after installing it. Supported \_start\_type\_s are: - -- `NO` Does not start the app. This is the default. -- `COLD` Starts the app from a clean state after install. -- `WARM` Preserves and restores the application state on incremental installs. -- `DEBUG` Waits for the debugger before starting the app in a clean state after install. - -Note: If more than one of `--start=_start_type_`, `--start_app` or `--debug_app` is set, the last value is used. - -#### `--adb=<var>path</var>` - -Indicates the `adb` binary to be used. - -The default is to use the adb in the Android SDK specified by [`--android_sdk`](#android-sdk). - -#### `--adb_arg=<var>serial</var>` - -Extra arguments to `adb`. These come before the subcommand in the command line and are typically used to specify which device to install to. For example, to select the Android device or emulator to use: - -``` -% bazel mobile-install --adb_arg=-s --adb_arg=deadbeef -``` - -invokes `adb` as - -``` -adb -s deadbeef install ... -``` - -#### `--incremental_install_verbosity=<var>number</var>` - -The verbosity for incremental install. Set to 1 for debug logging to be printed to the console. - -### `dump` - -The `dump` command prints to stdout a dump of the internal state of the Bazel server. This command is intended primarily for use by Bazel developers, so the output of this command is not specified, and is subject to change. - -By default, command will just print help message outlining possible options to dump specific areas of the Bazel state. In order to dump internal state, at least one of the options must be specified. - -Following options are supported: - -- `--action_cache` dumps action cache content. -- `--packages` dumps package cache content. -- `--skyframe` dumps state of internal Bazel dependency graph. -- `--rules` dumps rule summary for each rule and aspect class, including counts and action counts. This includes both native and Starlark rules. If memory tracking is enabled, then the rules' memory consumption is also printed. -- `--skylark_memory` dumps a [pprof](https://github.com/google/pprof) compatible .gz file to the specified path. You must enable memory tracking for this to work. - -#### Memory tracking - -Some `dump` commands require memory tracking. To turn this on, you have to pass startup flags to Bazel: - -- `--host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar` -- `--host_jvm_args=-DRULE_MEMORY_TRACKER=1` - -The java-agent is checked into Bazel at `third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar`, so make sure you adjust `$BAZEL` for where you keep your Bazel repository. - -Do not forget to keep passing these options to Bazel for every command or the server will restart. - -Example: - -``` - % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ - --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ - build --nobuild <targets> - - # Dump rules - % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ - --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ - dump --rules - - # Dump Starlark heap and analyze it with pprof - % bazel --host_jvm_args=-javaagent:$BAZEL/third_party/allocation_instrumenter/java-allocation-instrumenter-3.3.4.jar \ - --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \ - dump --skylark_memory=$HOME/prof.gz - % pprof -flame $HOME/prof.gz -``` - -### `analyze-profile` - -The `analyze-profile` command analyzes a [JSON trace profile](/advanced/performance/json-trace-profile) previously gathered during a Bazel invocation. - -### `canonicalize-flags` - -The [`canonicalize-flags`](/reference/command-line-reference#canonicalize-flags-options) command, which takes a list of options for a Bazel command and returns a list of options that has the same effect. The new list of options is canonical. For example, two lists of options with the same effect are canonicalized to the same new list. - -The `--for_command` option can be used to select between different commands. At this time, only `build` and `test` are supported. Options that the given command does not support cause an error. - -Note: A small number of options cannot be reordered, because Bazel cannot ensure that the effect is identical. Also note that this command *does not* expand flags from `--config`. - -As an example: - -``` - % bazel canonicalize-flags -- --config=any_name --test_tag_filters="-lint" - --config=any_name - --test_tag_filters=-lint -``` - -### Startup options - -The options described in this section affect the startup of the Java virtual machine used by Bazel server process, and they apply to all subsequent commands handled by that server. If there is an already running Bazel server and the startup options do not match, it will be restarted. - -All of the options described in this section must be specified using the `--key=value` or `--key value` syntax. Also, these options must appear *before* the name of the Bazel command. Use `startup --key=value` to list these in a `.bazelrc` file. - -#### `--output_base=<var>dir</var>` - -This option requires a path argument, which must specify a writable directory. Bazel will use this location to write all its output. The output base is also the key by which the client locates the Bazel server. By changing the output base, you change the server which will handle the command. - -By default, the output base is derived from the user's login name, and the name of the workspace directory (actually, its MD5 digest), so a typical value looks like: `/var/tmp/google/_bazel_johndoe/d41d8cd98f00b204e9800998ecf8427e`. - -Note: The client uses the output base to find the Bazel server instance, so if you specify a different output base in a Bazel command, a different server will be found (or started) to handle the request. It's possible to perform two concurrent builds in the same workspace directory by varying the output base. - -For example: - -``` - OUTPUT_BASE=/var/tmp/google/_bazel_johndoe/custom_output_base -% bazel --output_base ${OUTPUT_BASE}1 build //foo & bazel --output_base ${OUTPUT_BASE}2 build //bar -``` - -In this command, the two Bazel commands run concurrently (because of the shell `&` operator), each using a different Bazel server instance (because of the different output bases). In contrast, if the default output base was used in both commands, then both requests would be sent to the same server, which would handle them sequentially: building `//foo` first, followed by an incremental build of `//bar`. - -Note: We recommend you do not use an NFS or similar networked file system for the root directory, as the higher access latency will cause noticeably slower builds. - -#### `--output_user_root=<var>dir</var>` - -Points to the root directory where output and install bases are created. The directory must either not exist or be owned by the calling user. In the past, this was allowed to point to a directory shared among various users but it's not allowed any longer. This may be allowed once [issue #11100](https://github.com/bazelbuild/bazel/issues/11100) is addressed. - -If the `--output_base` option is specified, it overrides using `--output_user_root` to calculate the output base. - -The install base location is calculated based on `--output_user_root`, plus the MD5 identity of the Bazel embedded binaries. - -You can use the `--output_user_root` option to choose an alternate base location for all of Bazel's output (install base and output base) if there is a better location in your filesystem layout. - -Note: We recommend you do not use an NFS or similar networked file system for the root directory, as the higher access latency will cause noticeably slower builds. - -#### `--server_javabase=<var>dir</var>` - -Specifies the Java virtual machine in which *Bazel itself* runs. The value must be a path to the directory containing a JDK or JRE. It should not be a label. This option should appear before any Bazel command, for example: - -``` - % bazel --server_javabase=/usr/local/buildtools/java/jdk build //foo -``` - -This flag does *not* affect the JVMs used by Bazel subprocesses such as applications, tests, tools, and so on. Use build options [--javabase](#javabase) or [--host\_javabase](#host-javabase) instead. - -This flag was previously named `--host_javabase` (sometimes referred to as the 'left-hand side' `--host_javabase`), but was renamed to avoid confusion with the build flag [--host\_javabase](#host-javabase) (sometimes referred to as the 'right-hand side' `--host_javabase`). - -#### `--host_jvm_args=<var>string</var>` - -Specifies a startup option to be passed to the Java virtual machine in which *Bazel itself* runs. This can be used to set the stack size, for example: - -``` - % bazel --host_jvm_args="-Xss256K" build //foo -``` - -This option can be used multiple times with individual arguments. Note that setting this flag should rarely be needed. You can also pass a space-separated list of strings, each of which will be interpreted as a separate JVM argument, but this feature will soon be deprecated. - -That this does *not* affect any JVMs used by subprocesses of Bazel: applications, tests, tools, and so on. To pass JVM options to executable Java programs, whether run by `bazel run` or on the command-line, you should use the `--jvm_flags` argument which all `java_binary` and `java_test` programs support. Alternatively for tests, use `bazel test --test_arg=--jvm_flags=foo ...`. - -#### `--host_jvm_debug` - -This option causes the Java virtual machine to wait for a connection from a JDWP-compliant debugger before calling the main method of *Bazel itself*. This is primarily intended for use by Bazel developers. - -Note: This does *not* affect any JVMs used by subprocesses of Bazel: applications, tests, tools, etc. - -#### `--autodetect_server_javabase` - -This option causes Bazel to automatically search for an installed JDK on startup, and to fall back to the installed JRE if the embedded JRE isn't available. `--explicit_server_javabase` can be used to pick an explicit JRE to run Bazel with. - -#### `--batch` - -Batch mode causes Bazel to not use the [standard client/server mode](/run/client-server), but instead runs a bazel java process for a single command, which has been used for more predictable semantics with respect to signal handling, job control, and environment variable inheritance, and is necessary for running bazel in a chroot jail. - -Batch mode retains proper queueing semantics within the same output\_base. That is, simultaneous invocations will be processed in order, without overlap. If a batch mode Bazel is run on a client with a running server, it first kills the server before processing the command. - -Bazel will run slower in batch mode, or with the alternatives described above. This is because, among other things, the build file cache is memory-resident, so it is not preserved between sequential batch invocations. Therefore, using batch mode often makes more sense in cases where performance is less critical, such as continuous builds. - -Warning: `--batch` is sufficiently slower than standard client/server mode. Additionally it might not support all of the features and optimizations which are made possible by a persistent Bazel server. If you're using `--batch` for the purpose of build isolation, you should use the command option `--nokeep_state_after_build`, which guarantees that no incremental in-memory state is kept between builds. In order to restart the Bazel server and JVM after a build, please explicitly do so using the "shutdown" command. - -#### `--max_idle_secs=<var>n</var>` - -This option specifies how long, in seconds, the Bazel server process should wait after the last client request, before it exits. The default value is 10800 (3 hours). `--max_idle_secs=0` will cause the Bazel server process to persist indefinitely. - -Note: this flag is only read if Bazel needs to start a new server. Changing this option will not cause the server to restart. - -Note: system sleep time where a build is not running is counted as idle time. - -This option may be used by scripts that invoke Bazel to ensure that they do not leave Bazel server processes on a user's machine when they would not be running otherwise. For example, a presubmit script might wish to invoke `bazel query` to ensure that a user's pending change does not introduce unwanted dependencies. However, if the user has not done a recent build in that workspace, it would be undesirable for the presubmit script to start a Bazel server just for it to remain idle for the rest of the day. By specifying a small value of `--max_idle_secs` in the query request, the script can ensure that *if* it caused a new server to start, that server will exit promptly, but if instead there was already a server running, that server will continue to run until it has been idle for the usual time. Of course, the existing server's idle timer will be reset. - -#### `--[no]shutdown_on_low_sys_mem` - -If enabled and `--max_idle_secs` is set to a positive duration, after the build server has been idle for a while, shut down the server when the system is low on memory. Linux only. - -In addition to running an idle check corresponding to max\_idle\_secs, the build server will starts monitoring available system memory after the server has been idle for some time. If the available system memory becomes critically low, the server will exit. - -#### `--[no]block_for_lock` - -If enabled, Bazel will wait for other Bazel commands holding the server lock to complete before progressing. If disabled, Bazel will exit in error if it cannot immediately acquire the lock and proceed. - -Developers might use this in presubmit checks to avoid long waits caused by another Bazel command in the same client. - -#### `--io_nice_level=<var>n</var>` - -Sets a level from 0-7 for best-effort IO scheduling. 0 is highest priority, 7 is lowest. The anticipatory scheduler may only honor up to priority 4. Negative values are ignored. - -#### `--batch_cpu_scheduling` - -Use `batch` CPU scheduling for Bazel. This policy is useful for workloads that are non-interactive, but do not want to lower their nice value. See 'man 2 sched\_setscheduler'. This policy may provide for better system interactivity at the expense of Bazel throughput. - -### Miscellaneous options - -#### `--[no]announce_rc` - -Controls whether Bazel announces startup options and command options read from the bazelrc files when starting up. - -#### `--color (yes|no|auto)` - -This option determines whether Bazel will use colors to highlight its output on the screen. - -If this option is set to `yes`, color output is enabled. If this option is set to `auto`, Bazel will use color output only if the output is being sent to a terminal and the TERM environment variable is set to a value other than `dumb`, `emacs`, or `xterm-mono`. If this option is set to `no`, color output is disabled, regardless of whether the output is going to a terminal and regardless of the setting of the TERM environment variable. - -#### `--config=<var>name</var>` - -Selects additional config section from [the rc files](/run/bazelrc#bazelrc-file-locations); for the current `command`, it also pulls in the options from `command:name` if such a section exists. Can be specified multiple times to add flags from several config sections. Expansions can refer to other definitions (for example, expansions can be chained). - -#### `--curses (yes|no|auto)` - -This option determines whether Bazel will use cursor controls in its screen output. This results in less scrolling data, and a more compact, easy-to-read stream of output from Bazel. This works well with `--color`. - -If this option is set to `yes`, use of cursor controls is enabled. If this option is set to `no`, use of cursor controls is disabled. If this option is set to `auto`, use of cursor controls will be enabled under the same conditions as for `--color=auto`. - -#### `--[no]show_timestamps` - -If specified, a timestamp is added to each message generated by Bazel specifying the time at which the message was displayed. diff --git a/extending/aspects.mdx b/extending/aspects.mdx index c956a8ef..ac9a0273 100644 --- a/extending/aspects.mdx +++ b/extending/aspects.mdx @@ -2,16 +2,32 @@ title: 'Aspects' --- -This page explains the basics and benefits of using [aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced examples. -Aspects allow augmenting build dependency graphs with additional information and actions. Some typical scenarios when aspects can be useful: -- IDEs that integrate Bazel can use aspects to collect information about the project. -- Code generation tools can leverage aspects to execute on their inputs in *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy of [protobuf](https://developers.google.com/protocol-buffers/) library definitions, and language-specific rules can use aspects to attach actions generating protobuf support code for a particular language. +This page explains the basics and benefits of using +[aspects](/rules/lib/globals/bzl#aspect) and provides simple and advanced +examples. + +Aspects allow augmenting build dependency graphs with additional information +and actions. Some typical scenarios when aspects can be useful: + +* IDEs that integrate Bazel can use aspects to collect information about the + project. +* Code generation tools can leverage aspects to execute on their inputs in + *target-agnostic* manner. As an example, `BUILD` files can specify a hierarchy + of [protobuf](https://developers.google.com/protocol-buffers/) library + definitions, and language-specific rules can use aspects to attach + actions generating protobuf support code for a particular language. ## Aspect basics -`BUILD` files provide a description of a project’s source code: what source files are part of the project, what artifacts (*targets*) should be built from those files, what the dependencies between those files are, etc. Bazel uses this information to perform a build, that is, it figures out the set of actions needed to produce the artifacts (such as running compiler or linker) and executes those actions. Bazel accomplishes this by constructing a *dependency graph* between targets and visiting this graph to collect those actions. +`BUILD` files provide a description of a project’s source code: what source +files are part of the project, what artifacts (_targets_) should be built from +those files, what the dependencies between those files are, etc. Bazel uses +this information to perform a build, that is, it figures out the set of actions +needed to produce the artifacts (such as running compiler or linker) and +executes those actions. Bazel accomplishes this by constructing a _dependency +graph_ between targets and visiting this graph to collect those actions. Consider the following `BUILD` file: @@ -30,21 +46,41 @@ This `BUILD` file defines a dependency graph shown in the following figure: **Figure 1.** `BUILD` file dependency graph. -Bazel analyzes this dependency graph by calling an implementation function of the corresponding [rule](/extending/rules) (in this case "java\_library") for every target in the above example. Rule implementation functions generate actions that build artifacts, such as `.jar` files, and pass information, such as locations and names of those artifacts, to the reverse dependencies of those targets in [providers](/extending/rules#providers). - -Aspects are similar to rules in that they have an implementation function that generates actions and returns providers. However, their power comes from the way the dependency graph is built for them. An aspect has an implementation and a list of all attributes it propagates along. Consider an aspect A that propagates along attributes named "deps". This aspect can be applied to a target X, yielding an aspect application node A(X). During its application, aspect A is applied recursively to all targets that X refers to in its "deps" attribute (all attributes in A's propagation list). - -Thus a single act of applying aspect A to a target X yields a "shadow graph" of the original dependency graph of targets shown in the following figure: +Bazel analyzes this dependency graph by calling an implementation function of +the corresponding [rule](/extending/rules) (in this case "java_library") for every +target in the above example. Rule implementation functions generate actions that +build artifacts, such as `.jar` files, and pass information, such as locations +and names of those artifacts, to the reverse dependencies of those targets in +[providers](/extending/rules#providers). + +Aspects are similar to rules in that they have an implementation function that +generates actions and returns providers. However, their power comes from +the way the dependency graph is built for them. An aspect has an implementation +and a list of all attributes it propagates along. Consider an aspect A that +propagates along attributes named "deps". This aspect can be applied to +a target X, yielding an aspect application node A(X). During its application, +aspect A is applied recursively to all targets that X refers to in its "deps" +attribute (all attributes in A's propagation list). + +Thus a single act of applying aspect A to a target X yields a "shadow graph" of +the original dependency graph of targets shown in the following figure: ![Build Graph with Aspect](/rules/build-graph-aspects.png "Build graph with aspects") **Figure 2.** Build graph with aspects. -The only edges that are shadowed are the edges along the attributes in the propagation set, thus the `runtime_deps` edge is not shadowed in this example. An aspect implementation function is then invoked on all nodes in the shadow graph similar to how rule implementations are invoked on the nodes of the original graph. +The only edges that are shadowed are the edges along the attributes in +the propagation set, thus the `runtime_deps` edge is not shadowed in this +example. An aspect implementation function is then invoked on all nodes in +the shadow graph similar to how rule implementations are invoked on the nodes +of the original graph. ## Simple example -This example demonstrates how to recursively print the source files for a rule and all of its dependencies that have a `deps` attribute. It shows an aspect implementation, an aspect definition, and how to invoke the aspect from the Bazel command line. +This example demonstrates how to recursively print the source files for a +rule and all of its dependencies that have a `deps` attribute. It shows +an aspect implementation, an aspect definition, and how to invoke the aspect +from the Bazel command line. ```python def _print_aspect_impl(target, ctx): @@ -75,16 +111,25 @@ print_aspect = aspect( required_providers = [CcInfo], ) ``` +Aspect definitions are similar to rule definitions, and defined using +the [`aspect`](/rules/lib/globals/bzl#aspect) function. -Aspect definitions are similar to rule definitions, and defined using the [`aspect`](/rules/lib/globals/bzl#aspect) function. - -Just like a rule, an aspect has an implementation function which in this case is `_print_aspect_impl`. +Just like a rule, an aspect has an implementation function which in this case is +``_print_aspect_impl``. -`attr_aspects` is a list of rule attributes along which the aspect propagates. In this case, the aspect will propagate along the `deps` attribute of the rules that it is applied to. +``attr_aspects`` is a list of rule attributes along which the aspect propagates. +In this case, the aspect will propagate along the ``deps`` attribute of the +rules that it is applied to. -Another common argument for `attr_aspects` is `['*']` which would propagate the aspect to all attributes of a rule. +Another common argument for `attr_aspects` is `['*']` which would propagate the +aspect to all attributes of a rule. -`required_providers` is a list of providers that allows the aspect to limit its propagation to only the targets whose rules advertise its required providers. For more details consult [the documentation of the aspect function](/rules/lib/globals/bzl#aspect). In this case, the aspect will only apply on targets that declare `CcInfo` provider. +``required_providers`` is a list of providers that allows the aspect to limit +its propagation to only the targets whose rules advertise its required +providers. For more details consult +[the documentation of the aspect function](/rules/lib/globals/bzl#aspect). +In this case, the aspect will only apply on targets that declare `CcInfo` +provider. ### Aspect implementation @@ -100,32 +145,48 @@ def _print_aspect_impl(target, ctx): return [] ``` -Aspect implementation functions are similar to the rule implementation functions. They return [providers](/extending/rules#providers), can generate [actions](/extending/rules#actions), and take two arguments: +Aspect implementation functions are similar to the rule implementation +functions. They return [providers](/extending/rules#providers), can generate +[actions](/extending/rules#actions), and take two arguments: -- `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. -- `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes and generate outputs and actions. +* `target`: the [target](/rules/lib/builtins/Target) the aspect is being applied to. +* `ctx`: [`ctx`](/rules/lib/builtins/ctx) object that can be used to access attributes + and generate outputs and actions. -The implementation function can access the attributes of the target rule via [`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are provided by the target to which it is applied (via the `target` argument). +The implementation function can access the attributes of the target rule via +[`ctx.rule.attr`](/rules/lib/builtins/ctx#rule). It can examine providers that are +provided by the target to which it is applied (via the `target` argument). -Aspects are required to return a list of providers. In this example, the aspect does not provide anything, so it returns an empty list. +Aspects are required to return a list of providers. In this example, the aspect +does not provide anything, so it returns an empty list. ### Invoking the aspect using the command line -The simplest way to apply an aspect is from the command line using the [`--aspects`](/reference/command-line-reference#flag--aspects) argument. Assuming the aspect above were defined in a file named `print.bzl` this: +The simplest way to apply an aspect is from the command line using the +[`--aspects`](/reference/command-line-reference#flag--aspects) +argument. Assuming the aspect above were defined in a file named `print.bzl` +this: ```bash bazel build //MyExample:example --aspects print.bzl%print_aspect ``` -would apply the `print_aspect` to the target `example` and all of the target rules that are accessible recursively via the `deps` attribute. +would apply the `print_aspect` to the target `example` and all of the +target rules that are accessible recursively via the `deps` attribute. -The `--aspects` flag takes one argument, which is a specification of the aspect in the format `<extension file label>%<aspect top-level name>`. +The `--aspects` flag takes one argument, which is a specification of the aspect +in the format `%`. ## Advanced example -The following example demonstrates using an aspect from a target rule that counts files in targets, potentially filtering them by extension. It shows how to use a provider to return values, how to use parameters to pass an argument into an aspect implementation, and how to invoke an aspect from a rule. +The following example demonstrates using an aspect from a target rule +that counts files in targets, potentially filtering them by extension. +It shows how to use a provider to return values, how to use parameters to pass +an argument into an aspect implementation, and how to invoke an aspect from a rule. -Note: Aspects added in rules' attributes are called *rule-propagated aspects* as opposed to *command-line aspects* that are specified using the `--aspects` flag. +Note: Aspects added in rules' attributes are called *rule-propagated aspects* as +opposed to *command-line aspects* that are specified using the ``--aspects`` +flag. `file_count.bzl` file: @@ -213,15 +274,28 @@ file_count_aspect = aspect( ) ``` -This example shows how the aspect propagates through the `deps` attribute. +This example shows how the aspect propagates through the ``deps`` attribute. -`attrs` defines a set of attributes for an aspect. Public aspect attributes define parameters and can only be of types `bool`, `int` or `string`. For rule-propagated aspects, `int` and `string` parameters must have `values` specified on them. This example has a parameter called `extension` that is allowed to have '`*`', '`h`', or '`cc`' as a value. +``attrs`` defines a set of attributes for an aspect. Public aspect attributes +define parameters and can only be of types ``bool``, ``int`` or ``string``. +For rule-propagated aspects, ``int`` and ``string`` parameters must have +``values`` specified on them. This example has a parameter called ``extension`` +that is allowed to have '``*``', '``h``', or '``cc``' as a value. -For rule-propagated aspects, parameter values are taken from the rule requesting the aspect, using the attribute of the rule that has the same name and type. (see the definition of `file_count_rule`). +For rule-propagated aspects, parameter values are taken from the rule requesting +the aspect, using the attribute of the rule that has the same name and type. +(see the definition of ``file_count_rule``). -For command-line aspects, the parameters values can be passed using [`--aspects_parameters`](/reference/command-line-reference#flag--aspects_parameters) flag. The `values` restriction of `int` and `string` parameters may be omitted. +For command-line aspects, the parameters values can be passed using +[``--aspects_parameters``](/reference/command-line-reference#flag--aspects_parameters) +flag. The ``values`` restriction of ``int`` and ``string`` parameters may be +omitted. -Aspects are also allowed to have private attributes of types `label` or `label_list`. Private label attributes can be used to specify dependencies on tools or libraries that are needed for actions generated by aspects. There is not a private attribute defined in this example, but the following code snippet demonstrates how you could pass in a tool to an aspect: +Aspects are also allowed to have private attributes of types ``label`` or +``label_list``. Private label attributes can be used to specify dependencies on +tools or libraries that are needed for actions generated by aspects. There is not +a private attribute defined in this example, but the following code snippet +demonstrates how you could pass in a tool to an aspect: ```python ... @@ -259,17 +333,40 @@ def _file_count_aspect_impl(target, ctx): return [FileCountInfo(count = count)] ``` -Just like a rule implementation function, an aspect implementation function returns a struct of providers that are accessible to its dependencies. - -In this example, the `FileCountInfo` is defined as a provider that has one field `count`. It is best practice to explicitly define the fields of a provider using the `fields` attribute. - -The set of providers for an aspect application A(X) is the union of providers that come from the implementation of a rule for target X and from the implementation of aspect A. The providers that a rule implementation propagates are created and frozen before aspects are applied and cannot be modified from an aspect. It is an error if a target and an aspect that is applied to it each provide a provider with the same type, with the exceptions of [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) (which is merged, so long as the rule and aspect specify different output groups) and [`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) (which is taken from the aspect). This means that aspect implementations may never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). - -The parameters and private attributes are passed in the attributes of the `ctx`. This example references the `extension` parameter and determines what files to count. - -For returning providers, the values of attributes along which the aspect is propagated (from the `attr_aspects` list) are replaced with the results of an application of the aspect to them. For example, if target X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be \[A(Y), A(Z)]. In this example, `ctx.rule.attr.deps` are Target objects that are the results of applying the aspect to the 'deps' of the original target to which the aspect has been applied. - -In the example, the aspect accesses the `FileCountInfo` provider from the target's dependencies to accumulate the total transitive number of files. +Just like a rule implementation function, an aspect implementation function +returns a struct of providers that are accessible to its dependencies. + +In this example, the ``FileCountInfo`` is defined as a provider that has one +field ``count``. It is best practice to explicitly define the fields of a +provider using the ``fields`` attribute. + +The set of providers for an aspect application A(X) is the union of providers +that come from the implementation of a rule for target X and from the +implementation of aspect A. The providers that a rule implementation propagates +are created and frozen before aspects are applied and cannot be modified from an +aspect. It is an error if a target and an aspect that is applied to it each +provide a provider with the same type, with the exceptions of +[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) +(which is merged, so long as the +rule and aspect specify different output groups) and +[`InstrumentedFilesInfo`](/rules/lib/providers/InstrumentedFilesInfo) +(which is taken from the aspect). This means that aspect implementations may +never return [`DefaultInfo`](/rules/lib/providers/DefaultInfo). + +The parameters and private attributes are passed in the attributes of the +``ctx``. This example references the ``extension`` parameter and determines +what files to count. + +For returning providers, the values of attributes along which +the aspect is propagated (from the `attr_aspects` list) are replaced with +the results of an application of the aspect to them. For example, if target +X has Y and Z in its deps, `ctx.rule.attr.deps` for A(X) will be [A(Y), A(Z)]. +In this example, ``ctx.rule.attr.deps`` are Target objects that are the +results of applying the aspect to the 'deps' of the original target to which +the aspect has been applied. + +In the example, the aspect accesses the ``FileCountInfo`` provider from the +target's dependencies to accumulate the total transitive number of files. ### Invoking the aspect from a rule @@ -287,9 +384,13 @@ file_count_rule = rule( ) ``` -The rule implementation demonstrates how to access the `FileCountInfo` via the `ctx.attr.deps`. +The rule implementation demonstrates how to access the ``FileCountInfo`` +via the ``ctx.attr.deps``. -The rule definition demonstrates how to define a parameter (`extension`) and give it a default value (`*`). Note that having a default value that was not one of '`cc`', '`h`', or '`*`' would be an error due to the restrictions placed on the parameter in the aspect definition. +The rule definition demonstrates how to define a parameter (``extension``) +and give it a default value (``*``). Note that having a default value that +was not one of '``cc``', '``h``', or '``*``' would be an error due to the +restrictions placed on the parameter in the aspect definition. ### Invoking an aspect through a target rule @@ -308,10 +409,13 @@ file_count_rule( ) ``` -This demonstrates how to pass the `extension` parameter into the aspect via the rule. Since the `extension` parameter has a default value in the rule implementation, `extension` would be considered an optional parameter. +This demonstrates how to pass the ``extension`` parameter into the aspect +via the rule. Since the ``extension`` parameter has a default value in the +rule implementation, ``extension`` would be considered an optional parameter. -When the `file_count` target is built, our aspect will be evaluated for itself, and all of the targets accessible recursively via `deps`. +When the ``file_count`` target is built, our aspect will be evaluated for +itself, and all of the targets accessible recursively via ``deps``. ## References -- [`aspect` API reference](/rules/lib/globals/bzl#aspect) +* [`aspect` API reference](/rules/lib/globals/bzl#aspect) diff --git a/extending/auto-exec-groups.mdx b/extending/auto-exec-groups.mdx index 72af2c3a..9fee9783 100644 --- a/extending/auto-exec-groups.mdx +++ b/extending/auto-exec-groups.mdx @@ -2,11 +2,18 @@ title: 'Automatic Execution Groups (AEGs)' --- -Automatic execution groups select an [execution platform](https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs.) for each toolchain type. In other words, one target can have multiple execution platforms without defining execution groups. + + +Automatic execution groups select an [execution platform][exec_platform] +for each toolchain type. In other words, one target can have multiple +execution platforms without defining execution groups. ## Quick summary -Automatic execution groups are closely connected to toolchains. If you are using toolchains, you need to set them on the affected actions (actions which use an executable or a tool from a toolchain) by adding `toolchain` parameter. For example: +Automatic execution groups are closely connected to toolchains. If you are using +toolchains, you need to set them on the affected actions (actions which use an +executable or a tool from a toolchain) by adding `toolchain` parameter. For +example: ```python ctx.actions.run( @@ -16,10 +23,15 @@ ctx.actions.run( toolchain = '@bazel_tools//tools/jdk:toolchain_type', ) ``` +If the action does not use a tool or executable from a toolchain, and Blaze +doesn't detect that ([the error](#first-error-message) is raised), you can set +`toolchain = None`. -If the action does not use a tool or executable from a toolchain, and Blaze doesn't detect that ([the error](#first-error-message) is raised), you can set `toolchain = None`. - -If you need to use multiple toolchains on a single execution platform (an action uses executable or tools from two or more toolchains), you need to manually define [exec\_groups](https://bazel.build/extending/exec-groups) (check [When should I use a custom exec\_group?](/extending/auto-exec-groups#when-should-use-exec-groups) section). +If you need to use multiple toolchains on a single execution platform (an action +uses executable or tools from two or more toolchains), you need to manually +define [exec_groups][exec_groups] (check +[When should I use a custom exec_group?][multiple_toolchains_exec_groups] +section). ## History @@ -32,11 +44,20 @@ my_rule = rule( ) ``` -Rule `my_rule` registers two toolchain types. This means that the [Toolchain Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used to find an execution platform which supports both toolchain types. The selected execution platform was used for each registered action inside the rule, unless specified differently with [exec\_groups](https://bazel.build/extending/exec-groups). In other words, all actions inside the rule used to have a single execution platform even if they used tools from different toolchains (execution platform is selected for each target). This resulted in failures when there was no execution platform supporting all toolchains. +Rule `my_rule` registers two toolchain types. This means that the [Toolchain +Resolution](https://bazel.build/extending/toolchains#toolchain-resolution) used +to find an execution platform which supports both toolchain types. The selected +execution platform was used for each registered action inside the rule, unless +specified differently with [exec_groups][exec_groups]. +In other words, all actions inside the rule used to have a single execution +platform even if they used tools from different toolchains (execution platform +is selected for each target). This resulted in failures when there was no +execution platform supporting all toolchains. ## Current state -With AEGs, the execution platform is selected for each toolchain type. The implementation function of the earlier example, `my_rule`, would look like: +With AEGs, the execution platform is selected for each toolchain type. The +implementation function of the earlier example, `my_rule`, would look like: ```python def _impl(ctx): @@ -53,17 +74,29 @@ def _impl(ctx): ) ``` -This rule creates two actions, the `First action` which uses executable from a `//tools:toolchain_type_1` and the `Second action` which uses executable from a `//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed on a single execution platform which supports both toolchain types. With AEGs, by adding the `toolchain` parameter inside the actions, each action executes on the execution platform that provides the toolchain. The actions may be executed on different execution platforms. +This rule creates two actions, the `First action` which uses executable from a +`//tools:toolchain_type_1` and the `Second action` which uses executable from a +`//tools:toolchain_type_2`. Before AEGs, both of these actions would be executed +on a single execution platform which supports both toolchain types. With AEGs, +by adding the `toolchain` parameter inside the actions, each action executes on +the execution platform that provides the toolchain. The actions may be executed +on different execution platforms. -The same is effective with [ctx.actions.run\_shell](https://bazel.build/rules/lib/builtins/actions#run_shell) where `toolchain` parameter should be added when `tools` are from a toolchain. +The same is effective with [ctx.actions.run_shell][run_shell] where `toolchain` +parameter should be added when `tools` are from a toolchain. ## Difference between custom exec groups and automatic exec groups -As the name suggests, AEGs are exec groups created automatically for each toolchain type registered on a rule. There is no need to manually specify them, unlike the "classic" exec groups. Moreover, name of AEG is automatically set to its toolchain type (e.g. `//tools:toolchain_type_1`). +As the name suggests, AEGs are exec groups created automatically for each +toolchain type registered on a rule. There is no need to manually specify them, +unlike the "classic" exec groups. Moreover, name of AEG is automatically set to +its toolchain type (e.g. `//tools:toolchain_type_1`). -### When should I use a custom exec\_group? +### When should I use a custom exec_group? -Custom exec\_groups are needed only in case where multiple toolchains need to execute on a single execution platform. In all other cases there's no need to define custom exec\_groups. For example: +Custom exec_groups are needed only in case where multiple toolchains need to +execute on a single execution platform. In all other cases there's no need to +define custom exec_groups. For example: ```python def _impl(ctx): @@ -88,7 +121,9 @@ my_rule = rule( ## Migration of AEGs -Internally in google3, Blaze is already using AEGs. Externally for Bazel, migration is in the process. Some rules are already using this feature (e.g. Java and C++ rules). +Internally in google3, Blaze is already using AEGs. +Externally for Bazel, migration is in the process. Some rules are already using +this feature (e.g. Java and C++ rules). ### Which Bazel versions support this migration? @@ -96,7 +131,8 @@ AEGs are fully supported from Bazel 7. ### How to enable AEGs? -Set `--incompatible_auto_exec_groups` to true. More information about the flag on [the GitHub issue](https://github.com/bazelbuild/bazel/issues/17134). +Set `--incompatible_auto_exec_groups` to true. More information about the flag +on [the GitHub issue][github_flag]. ### How to enable AEGs inside a particular rule? @@ -110,23 +146,38 @@ my_rule = rule( } ) ``` - -This enables AEGs only in `my_rule` and its actions start using the new logic when selecting the execution platform. Incompatible flag is overridden with this attribute. +This enables AEGs only in `my_rule` and its actions start using the new logic +when selecting the execution platform. Incompatible flag is overridden with this +attribute. ### How to disable AEGs in case of an error? -Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in your project ([flag's GitHub issue](https://github.com/bazelbuild/bazel/issues/17134)), or disable a particular rule by setting `_use_auto_exec_groups` attribute to `False` ([more details about the attribute](#how-enable-particular-rule)). +Set `--incompatible_auto_exec_groups` to false to completely disable AEGs in +your project ([flag's GitHub issue][github_flag]), or disable a particular rule +by setting `_use_auto_exec_groups` attribute to `False` +([more details about the attribute](#how-enable-particular-rule)). ### Error messages while migrating to AEGs #### Couldn't identify if tools are from implicit dependencies or a toolchain. Please set the toolchain parameter. If you're not using a toolchain, set it to 'None'. + * In this case you get a stack of calls before the error happened and you can + clearly see which exact action needs the toolchain parameter. Check which + toolchain is used for the action and set it with the toolchain param. If no + toolchain is used inside the action for tools or executable, set it to + `None`. -- In this case you get a stack of calls before the error happened and you can clearly see which exact action needs the toolchain parameter. Check which toolchain is used for the action and set it with the toolchain param. If no toolchain is used inside the action for tools or executable, set it to `None`. - -#### Action declared for non-existent toolchain '\[toolchain\_type]'. - -- This means that you've set the toolchain parameter on the action but didn't register it on the rule. Register the toolchain or set `None` inside the action. +#### Action declared for non-existent toolchain '[toolchain_type]'. + * This means that you've set the toolchain parameter on the action but didn't +register it on the rule. Register the toolchain or set `None` inside the action. ## Additional material -For more information, check design document: [Automatic exec groups for toolchains](https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch). +For more information, check design document: +[Automatic exec groups for toolchains][aegs_design_doc]. + +[exec_platform]: https://bazel.build/extending/platforms#:~:text=Execution%20%2D%20a%20platform%20on%20which%20build%20tools%20execute%20build%20actions%20to%20produce%20intermediate%20and%20final%20outputs. +[exec_groups]: https://bazel.build/extending/exec-groups +[github_flag]: https://github.com/bazelbuild/bazel/issues/17134 +[aegs_design_doc]: https://docs.google.com/document/d/1-rbP_hmKs9D639YWw5F_JyxPxL2bi6dSmmvj_WXak9M/edit#heading=h.5mcn15i0e1ch +[run_shell]: https://bazel.build/rules/lib/builtins/actions#run_shell +[multiple_toolchains_exec_groups]: /extending/auto-exec-groups#when-should-use-exec-groups diff --git a/extending/concepts.mdx b/extending/concepts.mdx index da9fbd1d..e634c611 100644 --- a/extending/concepts.mdx +++ b/extending/concepts.mdx @@ -2,62 +2,111 @@ title: 'Extension Overview' --- -This page describes how to extend the BUILD language using macros and rules. -Bazel extensions are files ending in `.bzl`. Use a [load statement](/concepts/build-files#load) to import a symbol from an extension. -Before learning the more advanced concepts, first: +{/* [TOC] */} -- Read about the [Starlark language](/rules/language), used in both the `BUILD` and `.bzl` files. +This page describes how to extend the BUILD language using macros +and rules. -- Learn how you can [share variables](/build/share-variables) between two `BUILD` files. +Bazel extensions are files ending in `.bzl`. Use a +[load statement](/concepts/build-files#load) to import a symbol from an extension. -## Macros and rules +Before learning the more advanced concepts, first: -A macro is a function that instantiates rules. Macros come in two flavors: [symbolic macros](/extending/macros) (new in Bazel 8) and [legacy macros](/extending/legacy-macros). The two flavors of macros are defined differently, but behave almost the same from the point of view of a user. A macro is useful when a `BUILD` file is getting too repetitive or too complex, as it lets you reuse some code. The function is evaluated as soon as the `BUILD` file is read. After the evaluation of the `BUILD` file, Bazel has little information about macros. If your macro generates a `genrule`, Bazel will behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one exception is that targets declared in a symbolic macro have [special visibility semantics](/extending/macros#visibility): a symbolic macro can hide its internal targets from the rest of the package.) +* Read about the [Starlark language](/rules/language), used in both the + `BUILD` and `.bzl` files. -A [rule](/extending/rules) is more powerful than a macro. It can access Bazel internals and have full control over what is going on. It may for example pass information to other rules. +* Learn how you can [share variables](/build/share-variables) + between two `BUILD` files. + +## Macros and rules -If you want to reuse simple logic, start with a macro; we recommend a symbolic macro, unless you need to support older Bazel versions. If a macro becomes complex, it is often a good idea to make it a rule. Support for a new language is typically done with a rule. Rules are for advanced users, and most users will never have to write one; they will only load and call existing rules. +A macro is a function that instantiates rules. Macros come in two flavors: +[symbolic macros](/extending/macros) (new in Bazel 8) and [legacy +macros](/extending/legacy-macros). The two flavors of macros are defined +differently, but behave almost the same from the point of view of a user. A +macro is useful when a `BUILD` file is getting too repetitive or too complex, as +it lets you reuse some code. The function is evaluated as soon as the `BUILD` +file is read. After the evaluation of the `BUILD` file, Bazel has little +information about macros. If your macro generates a `genrule`, Bazel will +behave *almost* as if you declared that `genrule` in the `BUILD` file. (The one +exception is that targets declared in a symbolic macro have [special visibility +semantics](/extending/macros#visibility): a symbolic macro can hide its internal +targets from the rest of the package.) + +A [rule](/extending/rules) is more powerful than a macro. It can access Bazel +internals and have full control over what is going on. It may for example pass +information to other rules. + +If you want to reuse simple logic, start with a macro; we recommend a symbolic +macro, unless you need to support older Bazel versions. If a macro becomes +complex, it is often a good idea to make it a rule. Support for a new language +is typically done with a rule. Rules are for advanced users, and most users will +never have to write one; they will only load and call existing rules. ## Evaluation model A build consists of three phases. -- **Loading phase**. First, load and evaluate all extensions and all `BUILD` files that are needed for the build. The execution of the `BUILD` files simply instantiates rules (each time a rule is called, it gets added to a graph). This is where macros are evaluated. - -- **Analysis phase**. The code of the rules is executed (their `implementation` function), and actions are instantiated. An action describes how to generate a set of outputs from a set of inputs, such as "run gcc on hello.c and get hello.o". You must list explicitly which files will be generated before executing the actual commands. In other words, the analysis phase takes the graph generated by the loading phase and generates an action graph. - -- **Execution phase**. Actions are executed, when at least one of their outputs is required. If a file is missing or if a command fails to generate one output, the build fails. Tests are also run during this phase. - -Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` files. A file is read at most once per build and the result of the evaluation is cached and reused. A file is evaluated only once all its dependencies (`load()` statements) have been resolved. By design, loading a `.bzl` file has no visible side-effect, it only defines values and functions. - -Bazel tries to be clever: it uses dependency analysis to know which files must be loaded, which rules must be analyzed, and which actions must be executed. For example, if a rule generates actions that you don't need for the current build, they will not be executed. +* **Loading phase**. First, load and evaluate all extensions and all `BUILD` + files that are needed for the build. The execution of the `BUILD` files simply + instantiates rules (each time a rule is called, it gets added to a graph). + This is where macros are evaluated. + +* **Analysis phase**. The code of the rules is executed (their `implementation` + function), and actions are instantiated. An action describes how to generate + a set of outputs from a set of inputs, such as "run gcc on hello.c and get + hello.o". You must list explicitly which files will be generated before + executing the actual commands. In other words, the analysis phase takes + the graph generated by the loading phase and generates an action graph. + +* **Execution phase**. Actions are executed, when at least one of their outputs is + required. If a file is missing or if a command fails to generate one output, + the build fails. Tests are also run during this phase. + +Bazel uses parallelism to read, parse and evaluate the `.bzl` files and `BUILD` +files. A file is read at most once per build and the result of the evaluation is +cached and reused. A file is evaluated only once all its dependencies (`load()` +statements) have been resolved. By design, loading a `.bzl` file has no visible +side-effect, it only defines values and functions. + +Bazel tries to be clever: it uses dependency analysis to know which files must +be loaded, which rules must be analyzed, and which actions must be executed. For +example, if a rule generates actions that you don't need for the current build, +they will not be executed. ## Creating extensions -- [Create your first macro](/rules/macro-tutorial) in order to reuse some code. Then [learn more about macros](/extending/macros) and [using them to create "custom verbs"](/rules/verbs-tutorial). +* [Create your first macro](/rules/macro-tutorial) in order to reuse some code. + Then [learn more about macros](/extending/macros) and [using them to create + "custom verbs"](/rules/verbs-tutorial). -- [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. Next, you can read more about the [rules concepts](/extending/rules). +* [Follow the rules tutorial](/rules/rules-tutorial) to get started with rules. + Next, you can read more about the [rules concepts](/extending/rules). -The two links below will be very useful when writing your own extensions. Keep them within reach: +The two links below will be very useful when writing your own extensions. Keep +them within reach: -- The [API reference](/rules/lib) +* The [API reference](/rules/lib) -- [Examples](https://github.com/bazelbuild/examples/tree/master/rules) +* [Examples](https://github.com/bazelbuild/examples/tree/master/rules) ## Going further -In addition to [macros](/extending/macros) and [rules](/extending/rules), you may want to write [aspects](/extending/aspects) and [repository rules](/external/repo). +In addition to [macros](/extending/macros) and [rules](/extending/rules), you +may want to write [aspects](/extending/aspects) and [repository +rules](/external/repo). -- Use [Buildifier](https://github.com/bazelbuild/buildtools) consistently to format and lint your code. +* Use [Buildifier](https://github.com/bazelbuild/buildtools) + consistently to format and lint your code. -- Follow the [`.bzl` style guide](/rules/bzl-style). +* Follow the [`.bzl` style guide](/rules/bzl-style). -- [Test](/rules/testing) your code. +* [Test](/rules/testing) your code. -- [Generate documentation](https://skydoc.bazel.build/) to help your users. +* [Generate documentation](https://skydoc.bazel.build/) to help your users. -- [Optimize the performance](/rules/performance) of your code. +* [Optimize the performance](/rules/performance) of your code. -- [Deploy](/rules/deploying) your extensions to other people. +* [Deploy](/rules/deploying) your extensions to other people. diff --git a/extending/config.mdx b/extending/config.mdx deleted file mode 100644 index 9b1ddc23..00000000 --- a/extending/config.mdx +++ /dev/null @@ -1,631 +0,0 @@ ---- -title: 'Configurations' ---- - -This page covers the benefits and basic usage of Starlark configurations, Bazel's API for customizing how your project builds. It includes how to define build settings and provides examples. - -This makes it possible to: - -- define custom flags for your project, obsoleting the need for [`--define`](/docs/configurable-attributes#custom-keys) -- write [transitions](/rules/lib/builtins/transition#transition) to configure deps in different configurations than their parents (such as `--compilation_mode=opt` or `--cpu=arm`) -- bake better defaults into rules (such as automatically build `//my:android_app` with a specified SDK) - -and more, all completely from .bzl files (no Bazel release required). See the `bazelbuild/examples` repo for [examples](https://github.com/bazelbuild/examples/tree/HEAD/configurations). - -## User-defined build settings - -A build setting is a single piece of [configuration](/extending/rules#configurations) information. Think of a configuration as a key/value map. Setting `--cpu=ppc` and `--copt="-DFoo"` produces a configuration that looks like `{cpu: ppc, copt: "-DFoo"}`. Each entry is a build setting. - -Traditional flags like `cpu` and `copt` are native settings — their keys are defined and their values are set inside native bazel java code. Bazel users can only read and write them via the command line and other APIs maintained natively. Changing native flags, and the APIs that expose them, requires a bazel release. User-defined build settings are defined in `.bzl` files (and thus, don't need a bazel release to register changes). They also can be set via the command line (if they're designated as `flags`, see more below), but can also be set via [user-defined transitions](#user-defined-transitions). - -### Defining build settings - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/basic_build_setting) - -#### The `build_setting` `rule()` parameter - -Build settings are rules like any other rule and are differentiated using the Starlark `rule()` function's `build_setting` [attribute](/rules/lib/globals/bzl#rule.build_setting). - -```python -# example/buildsettings/build_settings.bzl -string_flag = rule( - implementation = _impl, - build_setting = config.string(flag = True) -) -``` - -The `build_setting` attribute takes a function that designates the type of the build setting. The type is limited to a set of basic Starlark types like `bool` and `string`. See the `config` module [documentation](/rules/lib/toplevel/config) for details. More complicated typing can be done in the rule's implementation function. More on this below. - -The `config` module's functions takes an optional boolean parameter, `flag`, which is set to false by default. if `flag` is set to true, the build setting can be set on the command line by users as well as internally by rule writers via default values and [transitions](/rules/lib/builtins/transition#transition). Not all settings should be settable by users. For example, if you as a rule writer have some debug mode that you'd like to turn on inside test rules, you don't want to give users the ability to indiscriminately turn on that feature inside other non-test rules. - -#### Using ctx.build\_setting\_value - -Like all rules, build setting rules have [implementation functions](/extending/rules#implementation-function). The basic Starlark-type value of the build settings can be accessed via the `ctx.build_setting_value` method. This method is only available to [`ctx`](/rules/lib/builtins/ctx) objects of build setting rules. These implementation methods can directly forward the build settings value or do additional work on it, like type checking or more complex struct creation. Here's how you would implement an `enum`-typed build setting: - -```python -# example/buildsettings/build_settings.bzl -TemperatureProvider = provider(fields = ['type']) - -temperatures = ["HOT", "LUKEWARM", "ICED"] - -def _impl(ctx): - raw_temperature = ctx.build_setting_value - if raw_temperature not in temperatures: - fail(str(ctx.label) + " build setting allowed to take values {" - + ", ".join(temperatures) + "} but was set to unallowed value " - + raw_temperature) - return TemperatureProvider(type = raw_temperature) - -temperature = rule( - implementation = _impl, - build_setting = config.string(flag = True) -) -``` - -Note: if a rule depends on a build setting, it will receive whatever providers the build setting implementation function returns, like any other dependency. But all other references to the value of the build setting (such as in transitions) will see its basic Starlark-typed value, not this post implementation function value. - -#### Defining multi-set string flags - -String settings have an additional `allow_multiple` parameter which allows the flag to be set multiple times on the command line or in bazelrcs. Their default value is still set with a string-typed attribute: - -```python -# example/buildsettings/build_settings.bzl -allow_multiple_flag = rule( - implementation = _impl, - build_setting = config.string(flag = True, allow_multiple = True) -) -``` - -```python -# example/BUILD -load("//example/buildsettings:build_settings.bzl", "allow_multiple_flag") -allow_multiple_flag( - name = "roasts", - build_setting_default = "medium" -) -``` - -Each setting of the flag is treated as a single value: - -```shell -$ bazel build //my/target --//example:roasts=blonde \ - --//example:roasts=medium,dark -``` - -The above is parsed to `{"//example:roasts": ["blonde", "medium,dark"]}` and `ctx.build_setting_value` returns the list `["blonde", "medium,dark"]`. - -#### Instantiating build settings - -Rules defined with the `build_setting` parameter have an implicit mandatory `build_setting_default` attribute. This attribute takes on the same type as declared by the `build_setting` param. - -```python -# example/buildsettings/build_settings.bzl -FlavorProvider = provider(fields = ['type']) - -def _impl(ctx): - return FlavorProvider(type = ctx.build_setting_value) - -flavor = rule( - implementation = _impl, - build_setting = config.string(flag = True) -) -``` - -```python -# example/BUILD -load("//example/buildsettings:build_settings.bzl", "flavor") -flavor( - name = "favorite_flavor", - build_setting_default = "APPLE" -) -``` - -### Predefined settings - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/use_skylib_build_setting) - -The [Skylib](https://github.com/bazelbuild/bazel-skylib) library includes a set of predefined settings you can instantiate without having to write custom Starlark. - -For example, to define a setting that accepts a limited set of string values: - -```python -# example/BUILD -load("@bazel_skylib//rules:common_settings.bzl", "string_flag") -string_flag( - name = "myflag", - values = ["a", "b", "c"], - build_setting_default = "a", -) -``` - -For a complete list, see [Common build setting rules](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/common_settings.bzl). - -### Using build settings - -#### Depending on build settings - -If a target would like to read a piece of configuration information, it can directly depend on the build setting via a regular attribute dependency. - -```python -# example/rules.bzl -load("//example/buildsettings:build_settings.bzl", "FlavorProvider") -def _rule_impl(ctx): - if ctx.attr.flavor[FlavorProvider].type == "ORANGE": - ... - -drink_rule = rule( - implementation = _rule_impl, - attrs = { - "flavor": attr.label() - } -) -``` - -```python -# example/BUILD -load("//example:rules.bzl", "drink_rule") -load("//example/buildsettings:build_settings.bzl", "flavor") -flavor( - name = "favorite_flavor", - build_setting_default = "APPLE" -) -drink_rule( - name = "my_drink", - flavor = ":favorite_flavor", -) -``` - -Languages may wish to create a canonical set of build settings which all rules for that language depend on. Though the native concept of `fragments` no longer exists as a hardcoded object in Starlark configuration world, one way to translate this concept would be to use sets of common implicit attributes. For example: - -```python -# kotlin/rules.bzl -_KOTLIN_CONFIG = { - "_compiler": attr.label(default = "//kotlin/config:compiler-flag"), - "_mode": attr.label(default = "//kotlin/config:mode-flag"), - ... -} - -... - -kotlin_library = rule( - implementation = _rule_impl, - attrs = dicts.add({ - "library-attr": attr.string() - }, _KOTLIN_CONFIG) -) - -kotlin_binary = rule( - implementation = _binary_impl, - attrs = dicts.add({ - "binary-attr": attr.label() - }, _KOTLIN_CONFIG) -``` - -#### Using build settings on the command line - -Similar to most native flags, you can use the command line to set build settings [that are marked as flags](#rule-parameter). The build setting's name is its full target path using `name=value` syntax: - -```shell -$ bazel build //my/target --//example:string_flag=some-value # allowed -$ bazel build //my/target --//example:string_flag some-value # not allowed -``` - -Special boolean syntax is supported: - -```shell -$ bazel build //my/target --//example:boolean_flag -$ bazel build //my/target --no//example:boolean_flag -``` - -#### Using build setting aliases - -You can set an alias for your build setting target path to make it easier to read on the command line. Aliases function similarly to native flags and also make use of the double-dash option syntax. - -Set an alias by adding `--flag_alias=ALIAS_NAME=TARGET_PATH` to your `.bazelrc` . For example, to set an alias to `coffee`: - -```shell -# .bazelrc -build --flag_alias=coffee=//experimental/user/starlark_configurations/basic_build_setting:coffee-temp -``` - -Best Practice: Setting an alias multiple times results in the most recent one taking precedence. Use unique alias names to avoid unintended parsing results. - -To make use of the alias, type it in place of the build setting target path. With the above example of `coffee` set in the user's `.bazelrc`: - -```shell -$ bazel build //my/target --coffee=ICED -``` - -instead of - -```shell -$ bazel build //my/target --//experimental/user/starlark_configurations/basic_build_setting:coffee-temp=ICED -``` - -Best Practice: While it possible to set aliases on the command line, leaving them in a `.bazelrc` reduces command line clutter. - -### Label-typed build settings - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/label_typed_build_setting) - -Unlike other build settings, label-typed settings cannot be defined using the `build_setting` rule parameter. Instead, bazel has two built-in rules: `label_flag` and `label_setting`. These rules forward the providers of the actual target to which the build setting is set. `label_flag` and `label_setting` can be read/written by transitions and `label_flag` can be set by the user like other `build_setting` rules can. Their only difference is they can't customely defined. - -Label-typed settings will eventually replace the functionality of late-bound defaults. Late-bound default attributes are Label-typed attributes whose final values can be affected by configuration. In Starlark, this will replace the [`configuration_field`](/rules/lib/globals/bzl#configuration_field) API. - -```python -# example/rules.bzl -MyProvider = provider(fields = ["my_field"]) - -def _dep_impl(ctx): - return MyProvider(my_field = "yeehaw") - -dep_rule = rule( - implementation = _dep_impl -) - -def _parent_impl(ctx): - if ctx.attr.my_field_provider[MyProvider].my_field == "cowabunga": - ... - -parent_rule = rule( - implementation = _parent_impl, - attrs = { "my_field_provider": attr.label() } -) -``` - -```python -# example/BUILD -load("//example:rules.bzl", "dep_rule", "parent_rule") - -dep_rule(name = "dep") - -parent_rule(name = "parent", my_field_provider = ":my_field_provider") - -label_flag( - name = "my_field_provider", - build_setting_default = ":dep" -) -``` - -### Build settings and select() - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/select_on_build_setting) - -Users can configure attributes on build settings by using [`select()`](/reference/be/functions#select). Build setting targets can be passed to the `flag_values` attribute of `config_setting`. The value to match to the configuration is passed as a `String` then parsed to the type of the build setting for matching. - -```python -config_setting( - name = "my_config", - flag_values = { - "//example:favorite_flavor": "MANGO" - } -) -``` - -## User-defined transitions - -A configuration [transition](/rules/lib/builtins/transition#transition) maps the transformation from one configured target to another within the build graph. - -Important: Transitions have [memory and performance impact](#memory-performance-considerations). - -### Defining - -Transitions define configuration changes between rules. For example, a request like "compile my dependency for a different CPU than its parent" is handled by a transition. - -Formally, a transition is a function from an input configuration to one or more output configurations. Most transitions are 1:1 such as "override the input configuration with `--cpu=ppc`". 1:2+ transitions can also exist but come with special restrictions. - -In Starlark, transitions are defined much like rules, with a defining `transition()` [function](/rules/lib/builtins/transition#transition) and an implementation function. - -```python -# example/transitions/transitions.bzl -def _impl(settings, attr): - _ignore = (settings, attr) - return {"//example:favorite_flavor" : "MINT"} - -hot_chocolate_transition = transition( - implementation = _impl, - inputs = [], - outputs = ["//example:favorite_flavor"] -) -``` - -The `transition()` function takes in an implementation function, a set of build settings to read(`inputs`), and a set of build settings to write (`outputs`). The implementation function has two parameters, `settings` and `attr`. `settings` is a dictionary \{`String`:`Object`\} of all settings declared in the `inputs` parameter to `transition()`. - -`attr` is a dictionary of attributes and values of the rule to which the transition is attached. When attached as an [outgoing edge transition](#outgoing-edge-transitions), the values of these attributes are all configured post-select() resolution. When attached as an [incoming edge transition](#incoming-edge-transitions), `attr` does not include any attributes that use a selector to resolve their value. If an incoming edge transition on `--foo` reads attribute `bar` and then also selects on `--foo` to set attribute `bar`, then there's a chance for the incoming edge transition to read the wrong value of `bar` in the transition. - -Note: Since transitions are attached to rule definitions and `select()`s are attached to rule instantiations (such as targets), errors related to `select()`s on read attributes will pop up when users create targets rather than when rules are written. It may be worth taking extra care to communicate to rule users which attributes they should be wary of selecting on or taking other precautions. - -The implementation function must return a dictionary (or list of dictionaries, in the case of transitions with multiple output configurations) of new build settings values to apply. The returned dictionary keyset(s) must contain exactly the set of build settings passed to the `outputs` parameter of the transition function. This is true even if a build setting is not actually changed over the course of the transition - its original value must be explicitly passed through in the returned dictionary. - -### Defining 1:2+ transitions - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/multi_arch_binary) - -[Outgoing edge transition](#outgoing-edge-transitions) can map a single input configuration to two or more output configurations. This is useful for defining rules that bundle multi-architecture code. - -1:2+ transitions are defined by returning a list of dictionaries in the transition implementation function. - -```python -# example/transitions/transitions.bzl -def _impl(settings, attr): - _ignore = (settings, attr) - return [ - {"//example:favorite_flavor" : "LATTE"}, - {"//example:favorite_flavor" : "MOCHA"}, - ] - -coffee_transition = transition( - implementation = _impl, - inputs = [], - outputs = ["//example:favorite_flavor"] -) -``` - -They can also set custom keys that the rule implementation function can use to read individual dependencies: - -```python -# example/transitions/transitions.bzl -def _impl(settings, attr): - _ignore = (settings, attr) - return { - "Apple deps": {"//command_line_option:cpu": "ppc"}, - "Linux deps": {"//command_line_option:cpu": "x86"}, - } - -multi_arch_transition = transition( - implementation = _impl, - inputs = [], - outputs = ["//command_line_option:cpu"] -) -``` - -### Attaching transitions - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/attaching_transitions_to_rules) - -Transitions can be attached in two places: incoming edges and outgoing edges. Effectively this means rules can transition their own configuration (incoming edge transition) and transition their dependencies' configurations (outgoing edge transition). - -NOTE: There is currently no way to attach Starlark transitions to native rules. If you need to do this, contact [bazel-discuss@googlegroups.com](mailto:bazel-discuss@googlegroups.com) for help with figuring out workarounds. - -### Incoming edge transitions - -Incoming edge transitions are activated by attaching a `transition` object (created by `transition()`) to `rule()`'s `cfg` parameter: - -```python -# example/rules.bzl -load("example/transitions:transitions.bzl", "hot_chocolate_transition") -drink_rule = rule( - implementation = _impl, - cfg = hot_chocolate_transition, - ... -``` - -Incoming edge transitions must be 1:1 transitions. - -### Outgoing edge transitions - -Outgoing edge transitions are activated by attaching a `transition` object (created by `transition()`) to an attribute's `cfg` parameter: - -```python -# example/rules.bzl -load("example/transitions:transitions.bzl", "coffee_transition") -drink_rule = rule( - implementation = _impl, - attrs = { "dep": attr.label(cfg = coffee_transition)} - ... -``` - -Outgoing edge transitions can be 1:1 or 1:2+. - -See [Accessing attributes with transitions](#accessing-attributes-with-transitions) for how to read these keys. - -### Transitions on native options - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/transition_on_native_flag) - -Starlark transitions can also declare reads and writes on native build configuration options via a special prefix to the option name. - -```python -# example/transitions/transitions.bzl -def _impl(settings, attr): - _ignore = (settings, attr) - return {"//command_line_option:cpu": "k8"} - -cpu_transition = transition( - implementation = _impl, - inputs = [], - outputs = ["//command_line_option:cpu"] -``` - -#### Unsupported native options - -Bazel doesn't support transitioning on `--define` with `"//command_line_option:define"`. Instead, use a custom [build setting](#user-defined-build-settings). In general, new usages of `--define` are discouraged in favor of build settings. - -Bazel doesn't support transitioning on `--config`. This is because `--config` is an "expansion" flag that expands to other flags. - -Crucially, `--config` may include flags that don't affect build configuration, such as [`--spawn_strategy`](/docs/user-manual#spawn-strategy) . Bazel, by design, can't bind such flags to individual targets. This means there's no coherent way to apply them in transitions. - -As a workaround, you can explicitly itemize the flags that *are* part of the configuration in your transition. This requires maintaining the `--config`'s expansion in two places, which is a known UI blemish. - -### Transitions on allow multiple build settings - -When setting build settings that [allow multiple values](#defining-multi-set-string-flags), the value of the setting must be set with a list. - -```python -# example/buildsettings/build_settings.bzl -string_flag = rule( - implementation = _impl, - build_setting = config.string(flag = True, allow_multiple = True) -) -``` - -```python -# example/BUILD -load("//example/buildsettings:build_settings.bzl", "string_flag") -string_flag(name = "roasts", build_setting_default = "medium") -``` - -```python -# example/transitions/rules.bzl -def _transition_impl(settings, attr): - # Using a value of just "dark" here will throw an error - return {"//example:roasts" : ["dark"]}, - -coffee_transition = transition( - implementation = _transition_impl, - inputs = [], - outputs = ["//example:roasts"] -) -``` - -### No-op transitions - -If a transition returns `{}`, `[]`, or `None`, this is shorthand for keeping all settings at their original values. This can be more convenient than explicitly setting each output to itself. - -```python -# example/transitions/transitions.bzl -def _impl(settings, attr): - _ignore = (attr) - if settings["//example:already_chosen"] is True: - return {} - return { - "//example:favorite_flavor": "dark chocolate", - "//example:include_marshmallows": "yes", - "//example:desired_temperature": "38C", - } - -hot_chocolate_transition = transition( - implementation = _impl, - inputs = ["//example:already_chosen"], - outputs = [ - "//example:favorite_flavor", - "//example:include_marshmallows", - "//example:desired_temperature", - ] -) -``` - -### Accessing attributes with transitions - -[End to end example](https://github.com/bazelbuild/examples/tree/HEAD/configurations/read_attr_in_transition) - -When [attaching a transition to an outgoing edge](#outgoing-edge-transitions) (regardless of whether the transition is a 1:1 or 1:2+ transition), `ctx.attr` is forced to be a list if it isn't already. The order of elements in this list is unspecified. - -```python -# example/transitions/rules.bzl -def _transition_impl(settings, attr): - return {"//example:favorite_flavor" : "LATTE"}, - -coffee_transition = transition( - implementation = _transition_impl, - inputs = [], - outputs = ["//example:favorite_flavor"] -) - -def _rule_impl(ctx): - # Note: List access even though "dep" is not declared as list - transitioned_dep = ctx.attr.dep[0] - - # Note: Access doesn't change, other_deps was already a list - for other_dep in ctx.attr.other_deps: - # ... - -coffee_rule = rule( - implementation = _rule_impl, - attrs = { - "dep": attr.label(cfg = coffee_transition) - "other_deps": attr.label_list(cfg = coffee_transition) - }) -``` - -If the transition is `1:2+` and sets custom keys, `ctx.split_attr` can be used to read individual deps for each key: - -```python -# example/transitions/rules.bzl -def _impl(settings, attr): - _ignore = (settings, attr) - return { - "Apple deps": {"//command_line_option:cpu": "ppc"}, - "Linux deps": {"//command_line_option:cpu": "x86"}, - } - -multi_arch_transition = transition( - implementation = _impl, - inputs = [], - outputs = ["//command_line_option:cpu"] -) - -def _rule_impl(ctx): - apple_dep = ctx.split_attr.dep["Apple deps"] - linux_dep = ctx.split_attr.dep["Linux deps"] - # ctx.attr has a list of all deps for all keys. Order is not guaranteed. - all_deps = ctx.attr.dep - -multi_arch_rule = rule( - implementation = _rule_impl, - attrs = { - "dep": attr.label(cfg = multi_arch_transition) - }) -``` - -See [complete example](https://github.com/bazelbuild/examples/tree/main/configurations/multi_arch_binary) here. - -## Integration with platforms and toolchains - -Many native flags today, like `--cpu` and `--crosstool_top` are related to toolchain resolution. In the future, explicit transitions on these types of flags will likely be replaced by transitioning on the [target platform](/extending/platforms). - -## Memory and performance considerations - -Adding transitions, and therefore new configurations, to your build comes at a cost: larger build graphs, less comprehensible build graphs, and slower builds. It's worth considering these costs when considering using transitions in your build rules. Below is an example of how a transition might create exponential growth of your build graph. - -### Badly behaved builds: a case study - -![Scalability graph](/rules/scalability-graph.png "Scalability graph") - -**Figure 1.** Scalability graph showing a top level target and its dependencies. - -This graph shows a top level target, `//pkg:app`, which depends on two targets, a `//pkg:1_0` and `//pkg:1_1`. Both these targets depend on two targets, `//pkg:2_0` and `//pkg:2_1`. Both these targets depend on two targets, `//pkg:3_0` and `//pkg:3_1`. This continues on until `//pkg:n_0` and `//pkg:n_1`, which both depend on a single target, `//pkg:dep`. - -Building `//pkg:app` requires \\(2n+2\\) targets: - -- `//pkg:app` -- `//pkg:dep` -- `//pkg:i_0` and `//pkg:i_1` for \\(i\\) in \\(\[1..n]\\) - -Imagine you [implement](#user-defined-build-settings) a flag `--//foo:owner=<STRING>` and `//pkg:i_b` applies - -``` -depConfig = myConfig + depConfig.owner="$(myConfig.owner)$(b)" -``` - -In other words, `//pkg:i_b` appends `b` to the old value of `--owner` for all its deps. - -This produces the following [configured targets](/reference/glossary#configured-target): - -``` -//pkg:app //foo:owner="" -//pkg:1_0 //foo:owner="" -//pkg:1_1 //foo:owner="" -//pkg:2_0 (via //pkg:1_0) //foo:owner="0" -//pkg:2_0 (via //pkg:1_1) //foo:owner="1" -//pkg:2_1 (via //pkg:1_0) //foo:owner="0" -//pkg:2_1 (via //pkg:1_1) //foo:owner="1" -//pkg:3_0 (via //pkg:1_0 → //pkg:2_0) //foo:owner="00" -//pkg:3_0 (via //pkg:1_0 → //pkg:2_1) //foo:owner="01" -//pkg:3_0 (via //pkg:1_1 → //pkg:2_0) //foo:owner="10" -//pkg:3_0 (via //pkg:1_1 → //pkg:2_1) //foo:owner="11" -... -``` - -`//pkg:dep` produces \\(2^n\\) configured targets: `config.owner=` "\\(b\_0b\_1...b\_n\\)" for all \\(b\_i\\) in \\(\{0,1\}\\). - -This makes the build graph exponentially larger than the target graph, with corresponding memory and performance consequences. - -TODO: Add strategies for measurement and mitigation of these issues. - -## Further reading - -For more details on modifying build configurations, see: - -- [Starlark Build Configuration](https://docs.google.com/document/d/1vc8v-kXjvgZOdQdnxPTaV0rrLxtP2XwnD2tAZlYJOqw/edit?usp=sharing) -- Full [set](https://github.com/bazelbuild/examples/tree/HEAD/configurations) of end to end examples diff --git a/extending/depsets.mdx b/extending/depsets.mdx index 1f0d18c4..4e84d3f2 100644 --- a/extending/depsets.mdx +++ b/extending/depsets.mdx @@ -2,21 +2,40 @@ title: 'Depsets' --- -[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently collecting data across a target’s transitive dependencies. They are an essential element of rule processing. -The defining feature of depset is its time- and space-efficient union operation. The depset constructor accepts a list of elements ("direct") and a list of other depsets ("transitive"), and returns a depset representing a set containing all the direct elements and the union of all the transitive sets. Conceptually, the constructor creates a new graph node that has the direct and transitive nodes as its successors. Depsets have a well-defined ordering semantics, based on traversal of this graph. + +[Depsets](/rules/lib/builtins/depset) are a specialized data structure for efficiently +collecting data across a target’s transitive dependencies. They are an essential +element of rule processing. + +The defining feature of depset is its time- and space-efficient union operation. +The depset constructor accepts a list of elements ("direct") and a list of other +depsets ("transitive"), and returns a depset representing a set containing all the +direct elements and the union of all the transitive sets. Conceptually, the +constructor creates a new graph node that has the direct and transitive nodes +as its successors. Depsets have a well-defined ordering semantics, based on +traversal of this graph. Example uses of depsets include: -- Storing the paths of all object files for a program’s libraries, which can then be passed to a linker action through a provider. +* Storing the paths of all object files for a program’s libraries, which can + then be passed to a linker action through a provider. -- For an interpreted language, storing the transitive source files that are included in an executable's runfiles. +* For an interpreted language, storing the transitive source files that are + included in an executable's runfiles. ## Description and operations -Conceptually, a depset is a directed acyclic graph (DAG) that typically looks similar to the target graph. It is constructed from the leaves up to the root. Each target in a dependency chain can add its own contents on top of the previous without having to read or copy them. +Conceptually, a depset is a directed acyclic graph (DAG) that typically looks +similar to the target graph. It is constructed from the leaves up to the root. +Each target in a dependency chain can add its own contents on top of the +previous without having to read or copy them. -Each node in the DAG holds a list of direct elements and a list of child nodes. The contents of the depset are the transitive elements, such as the direct elements of all the nodes. A new depset can be created using the [depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct elements and another list of child nodes. +Each node in the DAG holds a list of direct elements and a list of child nodes. +The contents of the depset are the transitive elements, such as the direct elements +of all the nodes. A new depset can be created using the +[depset](/rules/lib/globals/bzl#depset) constructor: it accepts a list of direct +elements and another list of child nodes. ```python s = depset(["a", "b", "c"]) @@ -26,7 +45,11 @@ print(s) # depset(["a", "b", "c"]) print(t) # depset(["d", "e", "a", "b", "c"]) ``` -To retrieve the contents of a depset, use the [to\_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive elements, not including duplicates. There is no way to directly inspect the precise structure of the DAG, although this structure does affect the order in which the elements are returned. +To retrieve the contents of a depset, use the +[to_list()](/rules/lib/builtins/depset#to_list) method. It returns a list of all transitive +elements, not including duplicates. There is no way to directly inspect the +precise structure of the DAG, although this structure does affect the order in +which the elements are returned. ```python s = depset(["a", "b", "c"]) @@ -35,9 +58,11 @@ print("c" in s.to_list()) # True print(s.to_list() == ["a", "b", "c"]) # True ``` -The allowed items in a depset are restricted, just as the allowed keys in dictionaries are restricted. In particular, depset contents may not be mutable. +The allowed items in a depset are restricted, just as the allowed keys in +dictionaries are restricted. In particular, depset contents may not be mutable. -Depsets use reference equality: a depset is equal to itself, but unequal to any other depset, even if they have the same contents and same internal structure. +Depsets use reference equality: a depset is equal to itself, but unequal to any +other depset, even if they have the same contents and same internal structure. ```python s = depset(["a", "b", "c"]) @@ -61,7 +86,9 @@ t = depset(["c", "b", "a"]) print(sorted(s.to_list()) == sorted(t.to_list())) # True ``` -There is no ability to remove elements from a depset. If this is needed, you must read out the entire contents of the depset, filter the elements you want to remove, and reconstruct a new depset. This is not particularly efficient. +There is no ability to remove elements from a depset. If this is needed, you +must read out the entire contents of the depset, filter the elements you want to +remove, and reconstruct a new depset. This is not particularly efficient. ```python s = depset(["a", "b", "c"]) @@ -78,9 +105,24 @@ print(s) # depset(["a"]) ### Order -The `to_list` operation performs a traversal over the DAG. The kind of traversal depends on the *order* that was specified at the time the depset was constructed. It is useful for Bazel to support multiple orders because sometimes tools care about the order of their inputs. For example, a linker action may need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the linker’s command line. Other tools might have the opposite requirement. - -Three traversal orders are supported: `postorder`, `preorder`, and `topological`. The first two work exactly like [tree traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) except that they operate on DAGs and skip already visited nodes. The third order works as a topological sort from root to leaves, essentially the same as preorder except that shared children are listed only after all of their parents. Preorder and postorder operate as left-to-right traversals, but note that within each node direct elements have no order relative to children. For topological order, there is no left-to-right guarantee, and even the all-parents-before-child guarantee does not apply in the case that there are duplicate elements in different nodes of the DAG. +The `to_list` operation performs a traversal over the DAG. The kind of traversal +depends on the *order* that was specified at the time the depset was +constructed. It is useful for Bazel to support multiple orders because sometimes +tools care about the order of their inputs. For example, a linker action may +need to ensure that if `B` depends on `A`, then `A.o` comes before `B.o` on the +linker’s command line. Other tools might have the opposite requirement. + +Three traversal orders are supported: `postorder`, `preorder`, and +`topological`. The first two work exactly like [tree +traversals](https://en.wikipedia.org/wiki/Tree_traversal#Depth-first_search) +except that they operate on DAGs and skip already visited nodes. The third order +works as a topological sort from root to leaves, essentially the same as +preorder except that shared children are listed only after all of their parents. +Preorder and postorder operate as left-to-right traversals, but note that within +each node direct elements have no order relative to children. For topological +order, there is no left-to-right guarantee, and even the +all-parents-before-child guarantee does not apply in the case that there are +duplicate elements in different nodes of the DAG. ```python # This demonstrates different traversal orders. @@ -109,13 +151,20 @@ print(create("preorder").to_list()) # ["d", "b", "a", "c"] print(create("topological").to_list()) # ["d", "b", "c", "a"] ``` -Due to how traversals are implemented, the order must be specified at the time the depset is created with the constructor’s `order` keyword argument. If this argument is omitted, the depset has the special `default` order, in which case there are no guarantees about the order of any of its elements (except that it is deterministic). +Due to how traversals are implemented, the order must be specified at the time +the depset is created with the constructor’s `order` keyword argument. If this +argument is omitted, the depset has the special `default` order, in which case +there are no guarantees about the order of any of its elements (except that it +is deterministic). ## Full example -This example is available at [https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). +This example is available at +[https://github.com/bazelbuild/examples/tree/main/rules/depsets](https://github.com/bazelbuild/examples/tree/main/rules/depsets). -Suppose there is a hypothetical interpreted language Foo. In order to build each `foo_binary` you need to know all the `*.foo` files that it directly or indirectly depends on. +Suppose there is a hypothetical interpreted language Foo. In order to build +each `foo_binary` you need to know all the `*.foo` files that it directly or +indirectly depends on. ```python # //depsets:BUILD @@ -159,14 +208,21 @@ foo_binary( import sys if __name__ == "__main__": - assert len(sys.argv) >= 1 + assert len(sys.argv) >= 1 output = open(sys.argv[1], "wt") for path in sys.argv[2:]: input = open(path, "rt") output.write(input.read()) ``` -Here, the transitive sources of the binary `d` are all of the `*.foo` files in the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` target to know about any file besides `d.foo`, the `foo_library` targets need to pass them along in a provider. Each library receives the providers from its own dependencies, adds its own immediate sources, and passes on a new provider with the augmented contents. The `foo_binary` rule does the same, except that instead of returning a provider, it uses the complete list of sources to construct a command line for an action. +Here, the transitive sources of the binary `d` are all of the `*.foo` files in +the `srcs` fields of `a`, `b`, `c`, and `d`. In order for the `foo_binary` +target to know about any file besides `d.foo`, the `foo_library` targets need to +pass them along in a provider. Each library receives the providers from its own +dependencies, adds its own immediate sources, and passes on a new provider with +the augmented contents. The `foo_binary` rule does the same, except that instead +of returning a provider, it uses the complete list of sources to construct a +command line for an action. Here’s a complete implementation of the `foo_library` and `foo_binary` rules. @@ -223,11 +279,15 @@ foo_binary = rule( ) ``` -You can test this by copying these files into a fresh package, renaming the labels appropriately, creating the source `*.foo` files with dummy content, and building the `d` target. +You can test this by copying these files into a fresh package, renaming the +labels appropriately, creating the source `*.foo` files with dummy content, and +building the `d` target. + ## Performance -To see the motivation for using depsets, consider what would happen if `get_transitive_srcs()` collected its sources in a list. +To see the motivation for using depsets, consider what would happen if +`get_transitive_srcs()` collected its sources in a list. ```python def get_transitive_srcs(srcs, deps): @@ -238,9 +298,12 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This does not take into account duplicates, so the source files for `a` will appear twice on the command line and twice in the contents of the output file. +This does not take into account duplicates, so the source files for `a` +will appear twice on the command line and twice in the contents of the output +file. -An alternative is using a general set, which can be simulated by a dictionary where the keys are the elements and all the keys map to `True`. +An alternative is using a general set, which can be simulated by a +dictionary where the keys are the elements and all the keys map to `True`. ```python def get_transitive_srcs(srcs, deps): @@ -253,16 +316,31 @@ def get_transitive_srcs(srcs, deps): return trans_srcs ``` -This gets rid of the duplicates, but it makes the order of the command line arguments (and therefore the contents of the files) unspecified, although still deterministic. +This gets rid of the duplicates, but it makes the order of the command line +arguments (and therefore the contents of the files) unspecified, although still +deterministic. -Moreover, both approaches are asymptotically worse than the depset-based approach. Consider the case where there is a long chain of dependencies on Foo libraries. Processing every rule requires copying all of the transitive sources that came before it into a new data structure. This means that the time and space cost for analyzing an individual library or binary target is proportional to its own height in the chain. For a chain of length n, foolib\_1 ← foolib\_2 ← … ← foolib\_n, the overall cost is effectively O(n^2). +Moreover, both approaches are asymptotically worse than the depset-based +approach. Consider the case where there is a long chain of dependencies on +Foo libraries. Processing every rule requires copying all of the transitive +sources that came before it into a new data structure. This means that the +time and space cost for analyzing an individual library or binary target +is proportional to its own height in the chain. For a chain of length n, +foolib_1 ← foolib_2 ← … ← foolib_n, the overall cost is effectively O(n^2). -Generally speaking, depsets should be used whenever you are accumulating information through your transitive dependencies. This helps ensure that your build scales well as your target graph grows deeper. +Generally speaking, depsets should be used whenever you are accumulating +information through your transitive dependencies. This helps ensure that +your build scales well as your target graph grows deeper. -Finally, it’s important to not retrieve the contents of the depset unnecessarily in rule implementations. One call to `to_list()` at the end in a binary rule is fine, since the overall cost is just O(n). It’s when many non-terminal targets try to call `to_list()` that quadratic behavior occurs. +Finally, it’s important to not retrieve the contents of the depset +unnecessarily in rule implementations. One call to `to_list()` +at the end in a binary rule is fine, since the overall cost is just O(n). It’s +when many non-terminal targets try to call `to_list()` that quadratic behavior +occurs. For more information about using depsets efficiently, see the [performance](/rules/performance) page. ## API Reference Please see [here](/rules/lib/builtins/depset) for more details. + diff --git a/extending/exec-groups.mdx b/extending/exec-groups.mdx index 6f7e7198..a8b45b92 100644 --- a/extending/exec-groups.mdx +++ b/extending/exec-groups.mdx @@ -2,21 +2,40 @@ title: 'Execution Groups' --- -Execution groups allow for multiple execution platforms within a single target. Each execution group has its own [toolchain](/extending/toolchains) dependencies and performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). + + +Execution groups allow for multiple execution platforms within a single target. +Each execution group has its own [toolchain](/extending/toolchains) dependencies and +performs its own [toolchain resolution](/extending/toolchains#toolchain-resolution). ## Current status -Execution groups for certain natively declared actions, like `CppLink`, can be used inside `exec_properties` to set per-action, per-target execution requirements. For more details, see the [Default execution groups](#exec-groups-for-native-rules) section. +Execution groups for certain natively declared actions, like `CppLink`, can be +used inside `exec_properties` to set per-action, per-target execution +requirements. For more details, see the +[Default execution groups](#exec-groups-for-native-rules) section. ## Background -Execution groups allow the rule author to define sets of actions, each with a potentially different execution platform. Multiple execution platforms can allow actions to execution differently, for example compiling an iOS app on a remote (linux) worker and then linking/code signing on a local mac worker. +Execution groups allow the rule author to define sets of actions, each with a +potentially different execution platform. Multiple execution platforms can allow +actions to execution differently, for example compiling an iOS app on a remote +(linux) worker and then linking/code signing on a local mac worker. -Being able to define groups of actions also helps alleviate the usage of action mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be unique and can only reference a single action. This is especially helpful in allocating extra resources to specific memory and processing intensive actions like linking in C++ builds without over-allocating to less demanding tasks. +Being able to define groups of actions also helps alleviate the usage of action +mnemonics as a proxy for specifying actions. Mnemonics are not guaranteed to be +unique and can only reference a single action. This is especially helpful in +allocating extra resources to specific memory and processing intensive actions +like linking in C++ builds without over-allocating to less demanding tasks. ## Defining execution groups -During rule definition, rule authors can [declare](/rules/lib/globals/bzl#exec_group) a set of execution groups. On each execution group, the rule author can specify everything needed to select an execution platform for that execution group, namely any constraints via `exec_compatible_with` and toolchain types via `toolchain`. +During rule definition, rule authors can +[declare](/rules/lib/globals/bzl#exec_group) +a set of execution groups. On each execution group, the rule author can specify +everything needed to select an execution platform for that execution group, +namely any constraints via `exec_compatible_with` and toolchain types via +`toolchain`. ```python # foo.bzl @@ -37,13 +56,25 @@ my_rule = rule( ) ``` -In the code snippet above, you can see that tool dependencies can also specify transition for an exec group using the [`cfg`](/rules/lib/toplevel/attr#label) attribute param and the [`config`](/rules/lib/toplevel/config) module. The module exposes an `exec` function which takes a single string parameter which is the name of the exec group for which the dependency should be built. +In the code snippet above, you can see that tool dependencies can also specify +transition for an exec group using the +[`cfg`](/rules/lib/toplevel/attr#label) +attribute param and the +[`config`](/rules/lib/toplevel/config) +module. The module exposes an `exec` function which takes a single string +parameter which is the name of the exec group for which the dependency should be +built. -As on native rules, the `test` execution group is present by default on Starlark test rules. +As on native rules, the `test` execution group is present by default on Starlark +test rules. ## Accessing execution groups -In the rule implementation, you can declare that actions should be run on the execution platform of an execution group. You can do this by using the `exec_group` param of action generating methods, specifically \[`ctx.actions.run`] (/rules/lib/builtins/actions#run) and [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). +In the rule implementation, you can declare that actions should be run on the +execution platform of an execution group. You can do this by using the `exec_group` +param of action generating methods, specifically [`ctx.actions.run`] +(/rules/lib/builtins/actions#run) and +[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell). ```python # foo.bzl @@ -55,7 +86,9 @@ def _impl(ctx): ) ``` -Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) of execution groups, similarly to how you can access the resolved toolchain of a target: +Rule authors will also be able to access the [resolved toolchains](/extending/toolchains#toolchain-resolution) +of execution groups, similarly to how you +can access the resolved toolchain of a target: ```python # foo.bzl @@ -68,18 +101,28 @@ def _impl(ctx): ) ``` -Note: If an action uses a toolchain from an execution group, but doesn't specify that execution group in the action declaration, that may potentially cause issues. A mismatch like this may not immediately cause failures, but is a latent problem. +Note: If an action uses a toolchain from an execution group, but doesn't specify +that execution group in the action declaration, that may potentially cause +issues. A mismatch like this may not immediately cause failures, but is a latent +problem. ### Default execution groups The following execution groups are predefined: -- `test`: Test runner actions (for more details, see the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). -- `cpp_link`: C++ linking actions. +* `test`: Test runner actions (for more details, see + the [execution platform section of the Test Encylopedia](/reference/test-encyclopedia#execution-platform)). +* `cpp_link`: C++ linking actions. ## Using execution groups to set execution properties -Execution groups are integrated with the [`exec_properties`](/reference/be/common-definitions#common-attributes) attribute that exists on every rule and allows the target writer to specify a string dict of properties that is then passed to the execution machinery. For example, if you wanted to set some property, say memory, for the target and give certain actions a higher memory allocation, you would write an `exec_properties` entry with an execution-group-augmented key, such as: +Execution groups are integrated with the +[`exec_properties`](/reference/be/common-definitions#common-attributes) +attribute that exists on every rule and allows the target writer to specify a +string dict of properties that is then passed to the execution machinery. For +example, if you wanted to set some property, say memory, for the target and give +certain actions a higher memory allocation, you would write an `exec_properties` +entry with an execution-group-augmented key, such as: ```python # BUILD @@ -93,13 +136,24 @@ my_rule( ) ``` -All actions with `exec_group = "link"` would see the exec properties dictionary as `{"mem": "16g"}`. As you see here, execution-group-level settings override target-level settings. +All actions with `exec_group = "link"` would see the exec properties +dictionary as `{"mem": "16g"}`. As you see here, execution-group-level +settings override target-level settings. ## Using execution groups to set platform constraints -Execution groups are also integrated with the [`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and [`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) attributes that exist on every rule and allow the target writer to specify additional constraints that must be satisfied by the execution platforms selected for the target's actions. +Execution groups are also integrated with the +[`exec_compatible_with`](/reference/be/common-definitions#common-attributes) and +[`exec_group_compatible_with`](/reference/be/common-definitions#common-attributes) +attributes that exist on every rule and allow the target writer to specify +additional constraints that must be satisfied by the execution platforms +selected for the target's actions. -For example, if the rule `my_test` defines the `link` execution group in addition to the default and the `test` execution group, then the following usage of these attributes would run actions in the default execution group on a platform with a high number of CPUs, the test action on Linux, and the link action on the default execution platform: +For example, if the rule `my_test` defines the `link` execution group in +addition to the default and the `test` execution group, then the following +usage of these attributes would run actions in the default execution group on +a platform with a high number of CPUs, the test action on Linux, and the link +action on the default execution platform: ```python # BUILD @@ -126,16 +180,23 @@ my_test( ### Execution groups for native rules -The following execution groups are available for actions defined by native rules: +The following execution groups are available for actions defined by native +rules: -- `test`: Test runner actions. -- `cpp_link`: C++ linking actions. +* `test`: Test runner actions. +* `cpp_link`: C++ linking actions. ### Execution groups and platform execution properties -It is possible to define `exec_properties` for arbitrary execution groups on platform targets (unlike `exec_properties` set directly on a target, where properties for unknown execution groups are rejected). Targets then inherit the execution platform's `exec_properties` that affect the default execution group and any other relevant execution groups. +It is possible to define `exec_properties` for arbitrary execution groups on +platform targets (unlike `exec_properties` set directly on a target, where +properties for unknown execution groups are rejected). Targets then inherit the +execution platform's `exec_properties` that affect the default execution group +and any other relevant execution groups. -For example, suppose running tests on the exec platform requires some resource to be available, but it isn't required for compiling and linking; this can be modelled as follows: +For example, suppose running tests on the exec platform requires some resource +to be available, but it isn't required for compiling and linking; this can be +modelled as follows: ```python constraint_setting(name = "resource") @@ -156,4 +217,5 @@ cc_test( ) ``` -`exec_properties` defined directly on targets take precedence over those that are inherited from the execution platform. +`exec_properties` defined directly on targets take precedence over those that +are inherited from the execution platform. diff --git a/extending/legacy-macros.mdx b/extending/legacy-macros.mdx index 910cc46a..9c2b8e28 100644 --- a/extending/legacy-macros.mdx +++ b/extending/legacy-macros.mdx @@ -2,7 +2,12 @@ title: 'Legacy Macros' --- -Legacy macros are unstructured functions called from `BUILD` files that can create targets. By the end of the [loading phase](/extending/concepts#evaluation-model), legacy macros don't exist anymore, and Bazel sees only the concrete set of instantiated rules. + + +Legacy macros are unstructured functions called from `BUILD` files that can +create targets. By the end of the +[loading phase](/extending/concepts#evaluation-model), legacy macros don't exist +anymore, and Bazel sees only the concrete set of instantiated rules. ## Why you shouldn't use legacy macros (and should use Symbolic macros instead) @@ -10,30 +15,38 @@ Where possible you should use [symbolic macros](macros.md#macros). Symbolic macros -- Prevent action at a distance -- Make it possible to hide implementation details through granular visibility -- Take typed attributes, which in turn means automatic label and select conversion. -- Are more readable -- Will soon have [lazy evaluation](macros.md#laziness) +* Prevent action at a distance +* Make it possible to hide implementation details through granular visibility +* Take typed attributes, which in turn means automatic label and select + conversion. +* Are more readable +* Will soon have [lazy evaluation](macros.md#laziness) ## Usage The typical use case for a macro is when you want to reuse a rule. -For example, genrule in a `BUILD` file generates a file using `//:generator` with a `some_arg` argument hardcoded in the command: +For example, genrule in a `BUILD` file generates a file using `//:generator` +with a `some_arg` argument hardcoded in the command: ```python genrule( name = "file", outs = ["file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", tools = ["//:generator"], ) ``` -Note: `$@` is a [Make variable](/reference/be/make-variables#predefined_genrule_variables) that refers to the execution-time locations of the files in the `outs` attribute list. It is equivalent to `$(locations :file.txt)`. +Note: `$@` is a +[Make variable](/reference/be/make-variables#predefined_genrule_variables) that +refers to the execution-time locations of the files in the `outs` attribute +list. It is equivalent to `$(locations :file.txt)`. -If you want to generate more files with different arguments, you may want to extract this code to a macro function. To create a macro called `file_generator`, which has `name` and `arg` parameters, we can replace the genrule with the following: +If you want to generate more files with different arguments, you may want to +extract this code to a macro function. To create a macro called +`file_generator`, which has `name` and `arg` parameters, we can replace the +genrule with the following: ```python load("//path:generator.bzl", "file_generator") @@ -54,22 +67,27 @@ file_generator( ) ``` -Here, you load the `file_generator` symbol from a `.bzl` file located in the `//path` package. By putting macro function definitions in a separate `.bzl` file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be loaded from any package in the workspace. +Here, you load the `file_generator` symbol from a `.bzl` file located in the +`//path` package. By putting macro function definitions in a separate `.bzl` +file, you keep your `BUILD` files clean and declarative, The `.bzl` file can be +loaded from any package in the workspace. -Finally, in `path/generator.bzl`, write the definition of the macro to encapsulate and parameterize the original genrule definition: +Finally, in `path/generator.bzl`, write the definition of the macro to +encapsulate and parameterize the original genrule definition: ```python def file_generator(name, arg, visibility=None): native.genrule( name = name, outs = [name + ".txt"], - cmd = "$(location //:generator) %s > $@" % arg, + cmd = "$(location //:generator) %s > $@" % arg, tools = ["//:generator"], visibility = visibility, ) ``` -You can also use macros to chain rules together. This example shows chained genrules, where a genrule uses the outputs of a previous genrule as inputs: +You can also use macros to chain rules together. This example shows chained +genrules, where a genrule uses the outputs of a previous genrule as inputs: ```python def chained_genrules(name, visibility=None): @@ -85,19 +103,23 @@ def chained_genrules(name, visibility=None): name = name + "-two", srcs = [name + ".one"], outs = [name + ".two"], - cmd = "$(location :tool-two) $< $@", + cmd = "$(location :tool-two) $< $@", tools = [":tool-two"], visibility = visibility, ) ``` -The example only assigns a visibility value to the second genrule. This allows macro authors to hide the outputs of intermediate rules from being depended upon by other targets in the workspace. +The example only assigns a visibility value to the second genrule. This allows +macro authors to hide the outputs of intermediate rules from being depended upon +by other targets in the workspace. -Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the `srcs` attribute list. +Note: Similar to `$@` for outputs, `$<` expands to the locations of files in the +`srcs` attribute list. ## Expanding macros -When you want to investigate what a macro does, use the `query` command with `--output=build` to see the expanded form: +When you want to investigate what a macro does, use the `query` command with +`--output=build` to see the expanded form: ```none $ bazel query --output=build :file @@ -106,13 +128,14 @@ genrule( name = "file", tools = ["//:generator"], outs = ["//test:file.txt"], - cmd = "$(location //:generator) some_arg > $@", + cmd = "$(location //:generator) some_arg > $@", ) ``` ## Instantiating native rules -Native rules (rules that don't need a `load()` statement) can be instantiated from the [native](/rules/lib/toplevel/native) module: +Native rules (rules that don't need a `load()` statement) can be instantiated +from the [native](/rules/lib/toplevel/native) module: ```python def my_macro(name, visibility=None): @@ -123,13 +146,23 @@ def my_macro(name, visibility=None): ) ``` -If you need to know the package name (for example, which `BUILD` file is calling the macro), use the function [native.package\_name()](/rules/lib/toplevel/native#package_name). Note that `native` can only be used in `.bzl` files, and not in `BUILD` files. +If you need to know the package name (for example, which `BUILD` file is calling +the macro), use the function +[native.package_name()](/rules/lib/toplevel/native#package_name). Note that +`native` can only be used in `.bzl` files, and not in `BUILD` files. ## Label resolution in macros -Since legacy macros are evaluated in the [loading phase](concepts.md#evaluation-model), label strings such as `"//foo:bar"` that occur in a legacy macro are interpreted relative to the `BUILD` file in which the macro is used rather than relative to the `.bzl` file in which it is defined. This behavior is generally undesirable for macros that are meant to be used in other repositories, such as because they are part of a published Starlark ruleset. +Since legacy macros are evaluated in the +[loading phase](concepts.md#evaluation-model), label strings such as +`"//foo:bar"` that occur in a legacy macro are interpreted relative to the +`BUILD` file in which the macro is used rather than relative to the `.bzl` file +in which it is defined. This behavior is generally undesirable for macros that +are meant to be used in other repositories, such as because they are part of a +published Starlark ruleset. -To get the same behavior as for Starlark rules, wrap the label strings with the [`Label`](/rules/lib/builtins/Label#Label) constructor: +To get the same behavior as for Starlark rules, wrap the label strings with the +[`Label`](/rules/lib/builtins/Label#Label) constructor: ```python # @my_ruleset//rules:defs.bzl @@ -151,39 +184,71 @@ def my_cc_wrapper(name, deps = [], **kwargs): ) ``` -With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys that are label strings will be automatically resolved to `Label` objects relative to the package of the file that contains the `select` call. If this is not chosen, wrap the label string with [native.package\_relative\_label()](/rules/lib/toplevel/native#package_relative_label). +With the `--incompatible_resolve_select_keys_eagerly` flag enabled, all keys +that are label strings will be automatically resolved to `Label` objects +relative to the package of the file that contains the `select` call. If this is +not chosen, wrap the label string with +[native.package_relative_label()](/rules/lib/toplevel/native#package_relative_label). ## Debugging -- `bazel query --output=build //my/path:all` will show you how the `BUILD` file looks after evaluation. All legacy macros, globs, loops are expanded. Known limitation: `select` expressions are not shown in the output. +* `bazel query --output=build //my/path:all` will show you how the `BUILD` + file looks after evaluation. All legacy macros, globs, loops are expanded. + Known limitation: `select` expressions are not shown in the output. -- You may filter the output based on `generator_function` (which function generated the rules) or `generator_name` (the name attribute of the macro): `bash $ bazel query --output=build 'attr(generator_function, my_macro, //my/path:all)'` +* You may filter the output based on `generator_function` (which function + generated the rules) or `generator_name` (the name attribute of the macro): + `bash $ bazel query --output=build 'attr(generator_function, my_macro, + //my/path:all)'` -- To find out where exactly the rule `foo` is generated in a `BUILD` file, you can try the following trick. Insert this line near the top of the `BUILD` file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when the rule `foo` is created (due to a name conflict), which will show you the full stack trace. +* To find out where exactly the rule `foo` is generated in a `BUILD` file, you + can try the following trick. Insert this line near the top of the `BUILD` + file: `cc_library(name = "foo")`. Run Bazel. You will get an exception when + the rule `foo` is created (due to a name conflict), which will show you the + full stack trace. -- You can also use [print](/rules/lib/globals/all#print) for debugging. It displays the message as a `DEBUG` log line during the loading phase. Except in rare cases, either remove `print` calls, or make them conditional under a `debugging` parameter that defaults to `False` before submitting the code to the depot. +* You can also use [print](/rules/lib/globals/all#print) for debugging. It + displays the message as a `DEBUG` log line during the loading phase. Except + in rare cases, either remove `print` calls, or make them conditional under a + `debugging` parameter that defaults to `False` before submitting the code to + the depot. ## Errors -If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) function. Explain clearly to the user what went wrong and how to fix their `BUILD` file. It is not possible to catch an error. +If you want to throw an error, use the [fail](/rules/lib/globals/all#fail) +function. Explain clearly to the user what went wrong and how to fix their +`BUILD` file. It is not possible to catch an error. ```python def my_macro(name, deps, visibility=None): - if len(deps) < 2: + if len(deps) < 2: fail("Expected at least two values in deps") # ... ``` ## Conventions -- All public functions (functions that don't start with underscore) that instantiate rules must have a `name` argument. This argument should not be optional (don't give a default value). +* All public functions (functions that don't start with underscore) that + instantiate rules must have a `name` argument. This argument should not be + optional (don't give a default value). -- Public functions should use a docstring following [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). +* Public functions should use a docstring following + [Python conventions](https://www.python.org/dev/peps/pep-0257/#one-line-docstrings). -- In `BUILD` files, the `name` argument of the macros must be a keyword argument (not a positional argument). +* In `BUILD` files, the `name` argument of the macros must be a keyword + argument (not a positional argument). -- The `name` attribute of rules generated by a macro should include the name argument as a prefix. For example, `macro(name = "foo")` can generate a `cc_library` `foo` and a genrule `foo_gen`. +* The `name` attribute of rules generated by a macro should include the name + argument as a prefix. For example, `macro(name = "foo")` can generate a + `cc_library` `foo` and a genrule `foo_gen`. -- In most cases, optional parameters should have a default value of `None`. `None` can be passed directly to native rules, which treat it the same as if you had not passed in any argument. Thus, there is no need to replace it with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer to the rules it creates, as their defaults may be complex or may change over time. Additionally, a parameter that is explicitly set to its default value looks different than one that is never set (or set to `None`) when accessed through the query language or build-system internals. +* In most cases, optional parameters should have a default value of `None`. + `None` can be passed directly to native rules, which treat it the same as if + you had not passed in any argument. Thus, there is no need to replace it + with `0`, `False`, or `[]` for this purpose. Instead, the macro should defer + to the rules it creates, as their defaults may be complex or may change over + time. Additionally, a parameter that is explicitly set to its default value + looks different than one that is never set (or set to `None`) when accessed + through the query language or build-system internals. -- Macros should have an optional `visibility` argument. +* Macros should have an optional `visibility` argument. diff --git a/extending/macros.mdx b/extending/macros.mdx index 1e4ac1e3..136665d0 100644 --- a/extending/macros.mdx +++ b/extending/macros.mdx @@ -2,23 +2,42 @@ title: 'Macros' --- -This page covers the basics of using macros and includes typical use cases, debugging, and conventions. -A macro is a function called from the `BUILD` file that can instantiate rules. Macros are mainly used for encapsulation and code reuse of existing rules and other macros. -Macros come in two flavors: symbolic macros, which are described on this page, and [legacy macros](legacy-macros.md). Where possible, we recommend using symbolic macros for code clarity. +This page covers the basics of using macros and includes typical use cases, +debugging, and conventions. -Symbolic macros offer typed arguments (string to label conversion, relative to where the macro was called) and the ability to restrict and specify the visibility of targets created. They are designed to be amenable to lazy evaluation (which will be added in a future Bazel release). Symbolic macros are available by default in Bazel 8. Where this document mentions `macros`, it's referring to **symbolic macros**. +A macro is a function called from the `BUILD` file that can instantiate rules. +Macros are mainly used for encapsulation and code reuse of existing rules and +other macros. -An executable example of symbolic macros can be found in the [examples repository](https://github.com/bazelbuild/examples/tree/main/macros). +Macros come in two flavors: symbolic macros, which are described on this page, +and [legacy macros](legacy-macros.md). Where possible, we recommend using +symbolic macros for code clarity. + +Symbolic macros offer typed arguments (string to label conversion, relative to +where the macro was called) and the ability to restrict and specify the +visibility of targets created. They are designed to be amenable to lazy +evaluation (which will be added in a future Bazel release). Symbolic macros are +available by default in Bazel 8. Where this document mentions `macros`, it's +referring to **symbolic macros**. + +An executable example of symbolic macros can be found in the +[examples repository](https://github.com/bazelbuild/examples/tree/main/macros). ## Usage -Macros are defined in `.bzl` files by calling the [`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with two required parameters: `attrs` and `implementation`. +Macros are defined in `.bzl` files by calling the +[`macro()`](https://bazel.build/rules/lib/globals/bzl.html#macro) function with +two required parameters: `attrs` and `implementation`. ### Attributes -`attrs` accepts a dictionary of attribute name to [attribute types](https://bazel.build/rules/lib/toplevel/attr#members), which represents the arguments to the macro. Two common attributes – `name` and `visibility` – are implicitly added to all macros and are not included in the dictionary passed to `attrs`. +`attrs` accepts a dictionary of attribute name to [attribute +types](https://bazel.build/rules/lib/toplevel/attr#members), which represents +the arguments to the macro. Two common attributes – `name` and `visibility` – +are implicitly added to all macros and are not included in the dictionary passed +to `attrs`. ```starlark # macro/macro.bzl @@ -31,13 +50,31 @@ my_macro = macro( ) ``` -Attribute type declarations accept the [parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), `mandatory`, `default`, and `doc`. Most attribute types also accept the `configurable` parameter, which determines whether the attribute accepts `select`s. If an attribute is `configurable`, it will parse non-`select` values as an unconfigurable `select` – `"foo"` will become `select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). +Attribute type declarations accept the +[parameters](https://bazel.build/rules/lib/toplevel/attr#parameters), +`mandatory`, `default`, and `doc`. Most attribute types also accept the +`configurable` parameter, which determines whether the attribute accepts +`select`s. If an attribute is `configurable`, it will parse non-`select` values +as an unconfigurable `select` – `"foo"` will become +`select({"//conditions:default": "foo"})`. Learn more in [selects](#selects). #### Attribute inheritance -Macros are often intended to wrap a rule (or another macro), and the macro's author often wants to forward the bulk of the wrapped symbol's attributes unchanged, using `**kwargs`, to the macro's main target (or main inner macro). - -To support this pattern, a macro can *inherit attributes* from a rule or another macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or [macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s `inherit_attrs` argument. (You can also use the special string `"common"` instead of a rule or macro symbol to inherit the [common attributes defined for all Starlark build rules](https://bazel.build/reference/be/common-definitions#common-attributes).) Only public attributes get inherited, and the attributes in the macro's own `attrs` dictionary override inherited attributes with the same name. You can also *remove* inherited attributes by using `None` as a value in the `attrs` dictionary: +Macros are often intended to wrap a rule (or another macro), and the macro's +author often wants to forward the bulk of the wrapped symbol's attributes +unchanged, using `**kwargs`, to the macro's main target (or main inner macro). + +To support this pattern, a macro can *inherit attributes* from a rule or another +macro by passing the [rule](https://bazel.build/rules/lib/builtins/rule) or +[macro symbol](https://bazel.build/rules/lib/builtins/macro) to `macro()`'s +`inherit_attrs` argument. (You can also use the special string `"common"` +instead of a rule or macro symbol to inherit the [common attributes defined for +all Starlark build +rules](https://bazel.build/reference/be/common-definitions#common-attributes).) +Only public attributes get inherited, and the attributes in the macro's own +`attrs` dictionary override inherited attributes with the same name. You can +also *remove* inherited attributes by using `None` as a value in the `attrs` +dictionary: ```starlark # macro/macro.bzl @@ -53,7 +90,11 @@ my_macro = macro( ) ``` -The default value of non-mandatory inherited attributes is always overridden to be `None`, regardless of the original attribute definition's default value. If you need to examine or modify an inherited non-mandatory attribute – for example, if you want to add a tag to an inherited `tags` attribute – you must make sure to handle the `None` case in your macro's implementation function: +The default value of non-mandatory inherited attributes is always overridden to +be `None`, regardless of the original attribute definition's default value. If +you need to examine or modify an inherited non-mandatory attribute – for +example, if you want to add a tag to an inherited `tags` attribute – you must +make sure to handle the `None` case in your macro's implementation function: ```starlark # macro/macro.bzl @@ -71,9 +112,15 @@ def _my_macro_impl(name, visibility, tags, **kwargs): ### Implementation -`implementation` accepts a function which contains the logic of the macro. Implementation functions often create targets by calling one or more rules, and they are usually private (named with a leading underscore). Conventionally, they are named the same as their macro, but prefixed with `_` and suffixed with `_impl`. +`implementation` accepts a function which contains the logic of the macro. +Implementation functions often create targets by calling one or more rules, and +they are usually private (named with a leading underscore). Conventionally, +they are named the same as their macro, but prefixed with `_` and suffixed with +`_impl`. -Unlike rule implementation functions, which take a single argument (`ctx`) that contains a reference to the attributes, macro implementation functions accept a parameter for each argument. +Unlike rule implementation functions, which take a single argument (`ctx`) that +contains a reference to the attributes, macro implementation functions accept a +parameter for each argument. ```starlark # macro/macro.bzl @@ -91,7 +138,11 @@ def _my_macro_impl(name, visibility, deps, create_test): ) ``` -If a macro inherits attributes, its implementation function *must* have a `**kwargs` residual keyword parameter, which can be forwarded to the call that invokes the inherited rule or submacro. (This helps ensure that your macro won't be broken if the rule or macro which from which you are inheriting adds a new attribute.) +If a macro inherits attributes, its implementation function *must* have a +`**kwargs` residual keyword parameter, which can be forwarded to the call that +invokes the inherited rule or submacro. (This helps ensure that your macro won't +be broken if the rule or macro which from which you are inheriting adds a new +attribute.) ### Declaration @@ -113,9 +164,12 @@ my_macro( ) ``` -This would create targets `//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. +This would create targets +`//pkg:macro_instance_cc_lib` and`//pkg:macro_instance_test`. -Just like in rule calls, if an attribute value in a macro call is set to `None`, that attribute is treated as if it was omitted by the macro's caller. For example, the following two macro calls are equivalent: +Just like in rule calls, if an attribute value in a macro call is set to `None`, +that attribute is treated as if it was omitted by the macro's caller. For +example, the following two macro calls are equivalent: ```starlark # pkg/BUILD @@ -123,17 +177,27 @@ my_macro(name = "abc", srcs = ["src.cc"], deps = None) my_macro(name = "abc", srcs = ["src.cc"]) ``` -This is generally not useful in `BUILD` files, but is helpful when programmatically wrapping a macro inside another macro. +This is generally not useful in `BUILD` files, but is helpful when +programmatically wrapping a macro inside another macro. ## Details ### Naming conventions for targets created -The names of any targets or submacros created by a symbolic macro must either match the macro's `name` parameter or must be prefixed by `name` followed by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, for example, `foo_bar`. +The names of any targets or submacros created by a symbolic macro must +either match the macro's `name` parameter or must be prefixed by `name` followed +by `_` (preferred), `.` or `-`. For example, `my_macro(name = "foo")` may only +create files or targets named `foo`, or prefixed by `foo_`, `foo-` or `foo.`, +for example, `foo_bar`. -Targets or files that violate macro naming convention can be declared, but cannot be built and cannot be used as dependencies. +Targets or files that violate macro naming convention can be declared, but +cannot be built and cannot be used as dependencies. -Non-macro files and targets within the same package as a macro instance should *not* have names that conflict with potential macro target names, though this exclusivity is not enforced. We are in the progress of implementing [lazy evaluation](#laziness) as a performance improvement for Symbolic macros, which will be impaired in packages that violate the naming schema. +Non-macro files and targets within the same package as a macro instance should +*not* have names that conflict with potential macro target names, though this +exclusivity is not enforced. We are in the progress of implementing +[lazy evaluation](#laziness) as a performance improvement for Symbolic macros, +which will be impaired in packages that violate the naming schema. ### Restrictions @@ -141,35 +205,54 @@ Symbolic macros have some additional restrictions compared to legacy macros. Symbolic macros -- must take a `name` argument and a `visibility` argument -- must have an `implementation` function -- may not return values -- may not mutate their arguments -- may not call `native.existing_rules()` unless they are special `finalizer` macros -- may not call `native.package()` -- may not call `glob()` -- may not call `native.environment_group()` -- must create targets whose names adhere to the [naming schema](#naming) -- can't refer to input files that weren't declared or passed in as an argument -- can't refer to private targets of their callers (see [visibility and macros](#visibility) for more details). +* must take a `name` argument and a `visibility` argument +* must have an `implementation` function +* may not return values +* may not mutate their arguments +* may not call `native.existing_rules()` unless they are special `finalizer` + macros +* may not call `native.package()` +* may not call `glob()` +* may not call `native.environment_group()` +* must create targets whose names adhere to the [naming schema](#naming) +* can't refer to input files that weren't declared or passed in as an argument +* can't refer to private targets of their callers (see + [visibility and macros](#visibility) for more details). ### Visibility and macros -The [visibility](/concepts/visibility) system helps protect the implementation details of both (symbolic) macros and their callers. +The [visibility](/concepts/visibility) system helps protect the implementation +details of both (symbolic) macros and their callers. -By default, targets created in a symbolic macro are visible within the macro itself, but not necessarily to the macro's caller. The macro can "export" a target as a public API by forwarding the value of its own `visibility` attribute, as in `some_rule(..., visibility = visibility)`. +By default, targets created in a symbolic macro are visible within the macro +itself, but not necessarily to the macro's caller. The macro can "export" a +target as a public API by forwarding the value of its own `visibility` +attribute, as in `some_rule(..., visibility = visibility)`. The key ideas of macro visibility are: -1. Visibility is checked based on what macro declared the target, not what package called the macro. +1. Visibility is checked based on what macro declared the target, not what + package called the macro. - - In other words, being in the same package does not by itself make one target visible to another. This protects the macro's internal targets from becoming dependencies of other macros or top-level targets in the package. + * In other words, being in the same package does not by itself make one + target visible to another. This protects the macro's internal targets + from becoming dependencies of other macros or top-level targets in the + package. -2. All `visibility` attributes, on both rules and macros, automatically include the place where the rule or macro was called. +1. All `visibility` attributes, on both rules and macros, automatically + include the place where the rule or macro was called. - - Thus, a target is unconditionally visible to other targets declared in the same macro (or the `BUILD` file, if not in a macro). + * Thus, a target is unconditionally visible to other targets declared in the + same macro (or the `BUILD` file, if not in a macro). -In practice, this means that when a macro declares a target without setting its `visibility`, the target defaults to being internal to the macro. (The package's [default visibility](/reference/be/functions#package.default_visibility) does not apply within a macro.) Exporting the target means that the target is visible to whatever the macro's caller specified in the macro's `visibility` attribute, plus the package of the macro's caller itself, as well as the macro's own code. Another way of thinking of it is that the visibility of a macro determines who (aside from the macro itself) can see the macro's exported targets. +In practice, this means that when a macro declares a target without setting its +`visibility`, the target defaults to being internal to the macro. (The package's +[default visibility](/reference/be/functions#package.default_visibility) does +not apply within a macro.) Exporting the target means that the target is visible +to whatever the macro's caller specified in the macro's `visibility` attribute, +plus the package of the macro's caller itself, as well as the macro's own code. +Another way of thinking of it is that the visibility of a macro determines who +(aside from the macro itself) can see the macro's exported targets. ```starlark # tool/BUILD @@ -231,25 +314,51 @@ some_rule( ) ``` -If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if the `//pkg` package had set its `default_visibility` to that value, then `//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain protected. - -A macro can declare that a target is visible to a friend package by passing `visibility = ["//some_friend:__pkg__"]` (for an internal target) or `visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). Note that it is an antipattern for a macro to declare a target with public visibility (`visibility = ["//visibility:public"]`). This is because it makes the target unconditionally visible to every package, even if the caller specified a more restricted visibility. - -All visibility checking is done with respect to the innermost currently running symbolic macro. However, there is a visibility delegation mechanism: If a macro passes a label as an attribute value to an inner macro, any usages of the label in the inner macro are checked with respect to the outer macro. See the [visibility page](/concepts/visibility#symbolic-macros) for more details. - -Remember that legacy macros are entirely transparent to the visibility system, and behave as though their location is whatever BUILD file or symbolic macro they were called from. +If `my_macro` were called with `visibility = ["//other_pkg:__pkg__"]`, or if +the `//pkg` package had set its `default_visibility` to that value, then +`//pkg:foo_exported` could also be used within `//other_pkg/BUILD` or within a +macro defined in `//other_pkg:defs.bzl`, but `//pkg:foo_helper` would remain +protected. + +A macro can declare that a target is visible to a friend package by passing +`visibility = ["//some_friend:__pkg__"]` (for an internal target) or +`visibility = visibility + ["//some_friend:__pkg__"]` (for an exported one). +Note that it is an antipattern for a macro to declare a target with public +visibility (`visibility = ["//visibility:public"]`). This is because it makes +the target unconditionally visible to every package, even if the caller +specified a more restricted visibility. + +All visibility checking is done with respect to the innermost currently running +symbolic macro. However, there is a visibility delegation mechanism: If a macro +passes a label as an attribute value to an inner macro, any usages of the label +in the inner macro are checked with respect to the outer macro. See the +[visibility page](/concepts/visibility#symbolic-macros) for more details. + +Remember that legacy macros are entirely transparent to the visibility system, +and behave as though their location is whatever BUILD file or symbolic macro +they were called from. #### Finalizers and visibility -Targets declared in a rule finalizer, in addition to seeing targets following the usual symbolic macro visibility rules, can *also* see all targets which are visible to the finalizer target's package. +Targets declared in a rule finalizer, in addition to seeing targets following +the usual symbolic macro visibility rules, can *also* see all targets which are +visible to the finalizer target's package. -This means that if you migrate a `native.existing_rules()`-based legacy macro to a finalizer, the targets declared by the finalizer will still be able to see their old dependencies. +This means that if you migrate a `native.existing_rules()`-based legacy macro to +a finalizer, the targets declared by the finalizer will still be able to see +their old dependencies. -However, note that it's possible to declare a target in a symbolic macro such that a finalizer's targets cannot see it under the visibility system – even though the finalizer can *introspect* its attributes using `native.existing_rules()`. +However, note that it's possible to declare a target in a symbolic macro such +that a finalizer's targets cannot see it under the visibility system – even +though the finalizer can *introspect* its attributes using +`native.existing_rules()`. ### Selects -If an attribute is `configurable` (the default) and its value is not `None`, then the macro implementation function will see the attribute value as wrapped in a trivial `select`. This makes it easier for the macro author to catch bugs where they did not anticipate that the attribute value could be a `select`. +If an attribute is `configurable` (the default) and its value is not `None`, +then the macro implementation function will see the attribute value as wrapped +in a trivial `select`. This makes it easier for the macro author to catch bugs +where they did not anticipate that the attribute value could be a `select`. For example, consider the following macro: @@ -260,15 +369,38 @@ my_macro = macro( ) ``` -If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` to be invoked with its `deps` parameter set to `select({"//conditions:default": ["//a"]})`. If this causes the implementation function to fail (say, because the code tried to index into the value as in `deps[0]`, which is not allowed for `select`s), the macro author can then make a choice: either they can rewrite their macro to only use operations compatible with `select`, or they can mark the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The latter ensures that users are not permitted to pass a `select` value in. - -Rule targets reverse this transformation, and store trivial `select`s as their unconditional values; in the above example, if `_my_macro_impl` declares a rule target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as `["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` values to be stored in all targets instantiated by macros. - -If the value of a configurable attribute is `None`, it does not get wrapped in a `select`. This ensures that tests like `my_attr == None` still work, and that when the attribute is forwarded to a rule with a computed default, the rule behaves properly (that is, as if the attribute were not passed in at all). It is not always possible for an attribute to take on a `None` value, but it can happen for the `attr.label()` type, and for any inherited non-mandatory attribute. +If `my_macro` is invoked with `deps = ["//a"]`, that will cause `_my_macro_impl` +to be invoked with its `deps` parameter set to `select({"//conditions:default": +["//a"]})`. If this causes the implementation function to fail (say, because the +code tried to index into the value as in `deps[0]`, which is not allowed for +`select`s), the macro author can then make a choice: either they can rewrite +their macro to only use operations compatible with `select`, or they can mark +the attribute as nonconfigurable (`attr.label_list(configurable = False)`). The +latter ensures that users are not permitted to pass a `select` value in. + +Rule targets reverse this transformation, and store trivial `select`s as their +unconditional values; in the above example, if `_my_macro_impl` declares a rule +target `my_rule(..., deps = deps)`, that rule target's `deps` will be stored as +`["//a"]`. This ensures that `select`-wrapping does not cause trivial `select` +values to be stored in all targets instantiated by macros. + +If the value of a configurable attribute is `None`, it does not get wrapped in a +`select`. This ensures that tests like `my_attr == None` still work, and that +when the attribute is forwarded to a rule with a computed default, the rule +behaves properly (that is, as if the attribute were not passed in at all). It is +not always possible for an attribute to take on a `None` value, but it can +happen for the `attr.label()` type, and for any inherited non-mandatory +attribute. ## Finalizers -A rule finalizer is a special symbolic macro which – regardless of its lexical position in a BUILD file – is evaluated in the final stage of loading a package, after all non-finalizer targets have been defined. Unlike ordinary symbolic macros, a finalizer can call `native.existing_rules()`, where it behaves slightly differently than in legacy macros: it only returns the set of non-finalizer rule targets. The finalizer may assert on the state of that set or define new targets. +A rule finalizer is a special symbolic macro which – regardless of its lexical +position in a BUILD file – is evaluated in the final stage of loading a package, +after all non-finalizer targets have been defined. Unlike ordinary symbolic +macros, a finalizer can call `native.existing_rules()`, where it behaves +slightly differently than in legacy macros: it only returns the set of +non-finalizer rule targets. The finalizer may assert on the state of that set or +define new targets. To declare a finalizer, call `macro()` with `finalizer = True`: @@ -294,17 +426,24 @@ my_finalizer = macro( ## Laziness -IMPORTANT: We are in the process of implementing lazy macro expansion and evaluation. This feature is not available yet. +IMPORTANT: We are in the process of implementing lazy macro expansion and +evaluation. This feature is not available yet. -Currently, all macros are evaluated as soon as the BUILD file is loaded, which can negatively impact performance for targets in packages that also have costly unrelated macros. In the future, non-finalizer symbolic macros will only be evaluated if they're required for the build. The prefix naming schema helps Bazel determine which macro to expand given a requested target. +Currently, all macros are evaluated as soon as the BUILD file is loaded, which +can negatively impact performance for targets in packages that also have costly +unrelated macros. In the future, non-finalizer symbolic macros will only be +evaluated if they're required for the build. The prefix naming schema helps +Bazel determine which macro to expand given a requested target. ## Migration troubleshooting Here are some common migration headaches and how to fix them. -- Legacy macro calls `glob()` +* Legacy macro calls `glob()` -Move the `glob()` call to your BUILD file (or to a legacy macro called from the BUILD file), and pass the `glob()` value to the symbolic macro using a label-list attribute: +Move the `glob()` call to your BUILD file (or to a legacy macro called from the +BUILD file), and pass the `glob()` value to the symbolic macro using a +label-list attribute: ```starlark # BUILD file @@ -314,10 +453,13 @@ my_macro( ) ``` -- Legacy macro has a parameter that isn't a valid starlark `attr` type. +* Legacy macro has a parameter that isn't a valid starlark `attr` type. + +Pull as much logic as possible into a nested symbolic macro, but keep the +top level macro a legacy macro. -Pull as much logic as possible into a nested symbolic macro, but keep the top level macro a legacy macro. +* Legacy macro calls a rule that creates a target that breaks the naming schema -- Legacy macro calls a rule that creates a target that breaks the naming schema +That's okay, just don't depend on the "offending" target. The naming check will +be quietly ignored. -That's okay, just don't depend on the "offending" target. The naming check will be quietly ignored. diff --git a/extending/platforms.mdx b/extending/platforms.mdx index e6e349fd..94e6290f 100644 --- a/extending/platforms.mdx +++ b/extending/platforms.mdx @@ -2,29 +2,60 @@ title: 'Platforms' --- -Bazel can build and test code on a variety of hardware, operating systems, and system configurations, using many different versions of build tools such as linkers and compilers. To help manage this complexity, Bazel has a concept of *constraints* and *platforms*. A constraint is a dimension in which build or production environments may differ, such as CPU architecture, the presence or absence of a GPU, or the version of a system-installed compiler. A platform is a named collection of choices for these constraints, representing the particular resources that are available in some environment. -Modeling the environment as a platform helps Bazel to automatically select the appropriate [toolchains](/extending/toolchains) for build actions. Platforms can also be used in combination with the [config\_setting](/reference/be/general#config_setting) rule to write [configurable attributes](/docs/configurable-attributes). + +Bazel can build and test code on a variety of hardware, operating systems, and +system configurations, using many different versions of build tools such as +linkers and compilers. To help manage this complexity, Bazel has a concept of +*constraints* and *platforms*. A constraint is a dimension in which build or +production environments may differ, such as CPU architecture, the presence or +absence of a GPU, or the version of a system-installed compiler. A platform is a +named collection of choices for these constraints, representing the particular +resources that are available in some environment. + +Modeling the environment as a platform helps Bazel to automatically select the +appropriate +[toolchains](/extending/toolchains) +for build actions. Platforms can also be used in combination with the +[config_setting](/reference/be/general#config_setting) +rule to write [configurable attributes](/docs/configurable-attributes). Bazel recognizes three roles that a platform may serve: -- **Host** - the platform on which Bazel itself runs. -- **Execution** - a platform on which build tools execute build actions to produce intermediate and final outputs. -- **Target** - a platform on which a final output resides and executes. +* **Host** - the platform on which Bazel itself runs. +* **Execution** - a platform on which build tools execute build actions to + produce intermediate and final outputs. +* **Target** - a platform on which a final output resides and executes. Bazel supports the following build scenarios regarding platforms: -- **Single-platform builds** (default) - host, execution, and target platforms are the same. For example, building a Linux executable on Ubuntu running on an Intel x64 CPU. +* **Single-platform builds** (default) - host, execution, and target platforms + are the same. For example, building a Linux executable on Ubuntu running on + an Intel x64 CPU. -- **Cross-compilation builds** - host and execution platforms are the same, but the target platform is different. For example, building an iOS app on macOS running on a MacBook Pro. +* **Cross-compilation builds** - host and execution platforms are the same, but + the target platform is different. For example, building an iOS app on macOS + running on a MacBook Pro. -- **Multi-platform builds** - host, execution, and target platforms are all different. +* **Multi-platform builds** - host, execution, and target platforms are all + different. -Tip: for detailed instructions on migrating your project to platforms, see [Migrating to Platforms](/concepts/platforms). +Tip: for detailed instructions on migrating your project to platforms, see +[Migrating to Platforms](/concepts/platforms). ## Defining constraints and platforms -The space of possible choices for platforms is defined by using the [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) and [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) rules within `BUILD` files. `constraint_setting` creates a new dimension, while `constraint_value` creates a new value for a given dimension; together they effectively define an enum and its possible values. For example, the following snippet of a `BUILD` file introduces a constraint for the system's glibc version with two possible values. +The space of possible choices for platforms is defined by using the +[`constraint_setting`][constraint_setting] and +[`constraint_value`][constraint_value] rules within `BUILD` files. +`constraint_setting` creates a new dimension, while +`constraint_value` creates a new value for a given dimension; together they +effectively define an enum and its possible values. For example, the following +snippet of a `BUILD` file introduces a constraint for the system's glibc version +with two possible values. + +[constraint_setting]: /reference/be/platforms-and-toolchains#constraint_setting +[constraint_value]: /reference/be/platforms-and-toolchains#constraint_value ```python constraint_setting(name = "glibc_version") @@ -40,9 +71,16 @@ constraint_value( ) ``` -Constraints and their values may be defined across different packages in the workspace. They are referenced by label and subject to the usual visibility controls. If visibility allows, you can extend an existing constraint setting by defining your own value for it. +Constraints and their values may be defined across different packages in the +workspace. They are referenced by label and subject to the usual visibility +controls. If visibility allows, you can extend an existing constraint setting by +defining your own value for it. -The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with certain choices of constraint values. The following creates a platform named `linux_x86`, and says that it describes any environment that runs a Linux operating system on an x86\_64 architecture with a glibc version of 2.25. (See below for more on Bazel's built-in constraints.) +The [`platform`](/reference/be/platforms-and-toolchains#platform) rule introduces a new platform with +certain choices of constraint values. The +following creates a platform named `linux_x86`, and says that it describes any +environment that runs a Linux operating system on an x86_64 architecture with a +glibc version of 2.25. (See below for more on Bazel's built-in constraints.) ```python platform( @@ -55,33 +93,52 @@ platform( ) ``` -Note: It is an error for a platform to specify more than one value of the same constraint setting, such as `@platforms//cpu:x86_64` and `@platforms//cpu:arm` for `@platforms//cpu:cpu`. +Note: It is an error for a platform to specify more than one value of the +same constraint setting, such as `@platforms//cpu:x86_64` and +`@platforms//cpu:arm` for `@platforms//cpu:cpu`. ## Generally useful constraints and platforms -To keep the ecosystem consistent, Bazel team maintains a repository with constraint definitions for the most popular CPU architectures and operating systems. These are all located in [https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). +To keep the ecosystem consistent, Bazel team maintains a repository with +constraint definitions for the most popular CPU architectures and operating +systems. These are all located in +[https://github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms). -Bazel ships with the following special platform definition: `@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the autodetected host platform value - represents autodetected platform for the system Bazel is running on. +Bazel ships with the following special platform definition: +`@platforms//host` (aliased as `@bazel_tools//tools:host_platform`). This is the +autodetected host platform value - +represents autodetected platform for the system Bazel is running on. ## Specifying a platform for a build -You can specify the host and target platforms for a build using the following command-line flags: - -- `--host_platform` - defaults to `@bazel_tools//tools:host_platform` - - - This target is aliased to `@platforms//host`, which is backed by a repo rule that detects the host OS and CPU and writes the platform target. - - There's also `@platforms//host:constraints.bzl`, which exposes an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and Starlark files. - -- `--platforms` - defaults to the host platform - - - This means that when no other flags are set, `@platforms//host` is the target platform. - - If `--host_platform` is set and not `--platforms`, the value of `--host_platform` is both the host and target platform. +You can specify the host and target platforms for a build using the following +command-line flags: + +* `--host_platform` - defaults to `@bazel_tools//tools:host_platform` + * This target is aliased to `@platforms//host`, which is backed by a repo + rule that detects the host OS and CPU and writes the platform target. + * There's also `@platforms//host:constraints.bzl`, which exposes + an array called `HOST_CONSTRAINTS`, which can be used in other BUILD and + Starlark files. +* `--platforms` - defaults to the host platform + * This means that when no other flags are set, + `@platforms//host` is the target platform. + * If `--host_platform` is set and not `--platforms`, the value of + `--host_platform` is both the host and target platform. ## Skipping incompatible targets -When building for a specific target platform it is often desirable to skip targets that will never work on that platform. For example, your Windows device driver is likely going to generate lots of compiler errors when building on a Linux machine with `//...`. Use the [`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) attribute to tell Bazel what target platform constraints your code has. +When building for a specific target platform it is often desirable to skip +targets that will never work on that platform. For example, your Windows device +driver is likely going to generate lots of compiler errors when building on a +Linux machine with `//...`. Use the +[`target_compatible_with`](/reference/be/common-definitions#common.target_compatible_with) +attribute to tell Bazel what target platform constraints your code has. -The simplest use of this attribute restricts a target to a single platform. The target will not be built for any platform that doesn't satisfy all of the constraints. The following example restricts `win_driver_lib.cc` to 64-bit Windows. +The simplest use of this attribute restricts a target to a single platform. +The target will not be built for any platform that doesn't satisfy all of the +constraints. The following example restricts `win_driver_lib.cc` to 64-bit +Windows. ```python cc_library( @@ -94,11 +151,16 @@ cc_library( ) ``` -`:win_driver_lib` is *only* compatible for building with 64-bit Windows and incompatible with all else. Incompatibility is transitive. Any targets that transitively depend on an incompatible target are themselves considered incompatible. +`:win_driver_lib` is *only* compatible for building with 64-bit Windows and +incompatible with all else. Incompatibility is transitive. Any targets +that transitively depend on an incompatible target are themselves considered +incompatible. ### When are targets skipped? -Targets are skipped when they are considered incompatible and included in the build as part of a target pattern expansion. For example, the following two invocations skip any incompatible targets found in a target pattern expansion. +Targets are skipped when they are considered incompatible and included in the +build as part of a target pattern expansion. For example, the following two +invocations skip any incompatible targets found in a target pattern expansion. ```console $ bazel build --platforms=//:myplatform //... @@ -108,9 +170,15 @@ $ bazel build --platforms=//:myplatform //... $ bazel build --platforms=//:myplatform //:all ``` -Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are similarly skipped if the `test_suite` is specified on the command line with [`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). In other words, `test_suite` targets on the command line behave like `:all` and `...`. Using `--noexpand_test_suites` prevents expansion and causes `test_suite` targets with incompatible tests to also be incompatible. +Incompatible tests in a [`test_suite`](/reference/be/general#test_suite) are +similarly skipped if the `test_suite` is specified on the command line with +[`--expand_test_suites`](/reference/command-line-reference#flag--expand_test_suites). +In other words, `test_suite` targets on the command line behave like `:all` and +`...`. Using `--noexpand_test_suites` prevents expansion and causes +`test_suite` targets with incompatible tests to also be incompatible. -Explicitly specifying an incompatible target on the command line results in an error message and a failed build. +Explicitly specifying an incompatible target on the command line results in an +error message and a failed build. ```console $ bazel build --platforms=//:myplatform //:target_incompatible_with_myplatform @@ -120,13 +188,20 @@ ERROR: Target //:target_incompatible_with_myplatform is incompatible and cannot FAILED: Build did NOT complete successfully ``` -Incompatible explicit targets are silently skipped if `--skip_incompatible_explicit_targets` is enabled. +Incompatible explicit targets are silently skipped if +`--skip_incompatible_explicit_targets` is enabled. ### More expressive constraints -For more flexibility in expressing constraints, use the `@platforms//:incompatible` [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) that no platform satisfies. +For more flexibility in expressing constraints, use the +`@platforms//:incompatible` +[`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) +that no platform satisfies. -Use [`select()`](/reference/be/functions#select) in combination with `@platforms//:incompatible` to express more complicated restrictions. For example, use it to implement basic OR logic. The following marks a library compatible with macOS and Linux, but no other platforms. +Use [`select()`](/reference/be/functions#select) in combination with +`@platforms//:incompatible` to express more complicated restrictions. For +example, use it to implement basic OR logic. The following marks a library +compatible with macOS and Linux, but no other platforms. Note: An empty constraints list is equivalent to "compatible with everything". @@ -146,11 +221,16 @@ The above can be interpreted as follows: 1. When targeting macOS, the target has no constraints. 2. When targeting Linux, the target has no constraints. -3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because `@platforms//:incompatible` is not part of any platform, the target is deemed incompatible. +3. Otherwise, the target has the `@platforms//:incompatible` constraint. Because + `@platforms//:incompatible` is not part of any platform, the target is + deemed incompatible. -To make your constraints more readable, use [skylib](https://github.com/bazelbuild/bazel-skylib)'s [`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). +To make your constraints more readable, use +[skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`selects.with_or()`](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/selects_doc.md#selectswith_or). -You can express inverse compatibility in a similar way. The following example describes a library that is compatible with everything *except* for ARM. +You can express inverse compatibility in a similar way. The following example +describes a library that is compatible with everything _except_ for ARM. ```python cc_library( @@ -165,9 +245,15 @@ cc_library( ### Detecting incompatible targets using `bazel cquery` -You can use the [`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) in `bazel cquery`'s [Starlark output format](/query/cquery#output-format-definition) to distinguish incompatible targets from compatible ones. +You can use the +[`IncompatiblePlatformProvider`](/rules/lib/providers/IncompatiblePlatformProvider) +in `bazel cquery`'s [Starlark output +format](/query/cquery#output-format-definition) to distinguish +incompatible targets from compatible ones. -This can be used to filter out incompatible targets. The example below will only print the labels for targets that are compatible. Incompatible targets are not printed. +This can be used to filter out incompatible targets. The example below will +only print the labels for targets that are compatible. Incompatible targets are +not printed. ```console $ cat example.cquery @@ -177,9 +263,11 @@ def format(target): return target.label return "" + $ bazel cquery //... --output=starlark --starlark:file=example.cquery ``` ### Known Issues -Incompatible targets [ignore visibility restrictions](https://github.com/bazelbuild/bazel/issues/16044). +Incompatible targets [ignore visibility +restrictions](https://github.com/bazelbuild/bazel/issues/16044). diff --git a/extending/rules.mdx b/extending/rules.mdx index b7afa71c..248ee328 100644 --- a/extending/rules.mdx +++ b/extending/rules.mdx @@ -2,28 +2,57 @@ title: 'Rules' --- -A **rule** defines a series of [**actions**](#actions) that Bazel performs on inputs to produce a set of outputs, which are referenced in [**providers**](#providers) returned by the rule's [**implementation function**](#implementation_function). For example, a C++ binary rule might: -1. Take a set of `.cpp` source files (inputs). -2. Run `g++` on the source files (action). -3. Return the `DefaultInfo` provider with the executable output and other files to make available at runtime. -4. Return the `CcInfo` provider with C++-specific information gathered from the target and its dependencies. -From Bazel's perspective, `g++` and the standard C++ libraries are also inputs to this rule. As a rule writer, you must consider not only the user-provided inputs to a rule, but also all of the tools and libraries required to execute the actions. - -Before creating or modifying any rule, ensure you are familiar with Bazel's [build phases](/extending/concepts). It is important to understand the three phases of a build (loading, analysis, and execution). It is also useful to learn about [macros](/extending/macros) to understand the difference between rules and macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). Then, use this page as a reference. - -A few rules are built into Bazel itself. These *native rules*, such as `genrule` and `filegroup`, provide some core support. By defining your own rules, you can add support for languages and tools that Bazel doesn't support natively. - -Bazel provides an extensibility model for writing rules using the [Starlark](/rules/language) language. These rules are written in `.bzl` files, which can be loaded directly from `BUILD` files. - -When defining your own rule, you get to decide what attributes it supports and how it generates its outputs. - -The rule's `implementation` function defines its exact behavior during the [analysis phase](/extending/concepts#evaluation-model). This function doesn't run any external commands. Rather, it registers [actions](#actions) that will be used later during the execution phase to build the rule's outputs, if they are needed. +A **rule** defines a series of [**actions**](#actions) that Bazel performs on +inputs to produce a set of outputs, which are referenced in +[**providers**](#providers) returned by the rule's +[**implementation function**](#implementation_function). For example, a C++ +binary rule might: + +1. Take a set of `.cpp` source files (inputs). +2. Run `g++` on the source files (action). +3. Return the `DefaultInfo` provider with the executable output and other files + to make available at runtime. +4. Return the `CcInfo` provider with C++-specific information gathered from the + target and its dependencies. + +From Bazel's perspective, `g++` and the standard C++ libraries are also inputs +to this rule. As a rule writer, you must consider not only the user-provided +inputs to a rule, but also all of the tools and libraries required to execute +the actions. + +Before creating or modifying any rule, ensure you are familiar with Bazel's +[build phases](/extending/concepts). It is important to understand the three +phases of a build (loading, analysis, and execution). It is also useful to +learn about [macros](/extending/macros) to understand the difference between rules and +macros. To get started, first review the [Rules Tutorial](/rules/rules-tutorial). +Then, use this page as a reference. + +A few rules are built into Bazel itself. These *native rules*, such as +`genrule` and `filegroup`, provide some core support. +By defining your own rules, you can add support for languages and tools +that Bazel doesn't support natively. + +Bazel provides an extensibility model for writing rules using the +[Starlark](/rules/language) language. These rules are written in `.bzl` files, which +can be loaded directly from `BUILD` files. + +When defining your own rule, you get to decide what attributes it supports and +how it generates its outputs. + +The rule's `implementation` function defines its exact behavior during the +[analysis phase](/extending/concepts#evaluation-model). This function doesn't run any +external commands. Rather, it registers [actions](#actions) that will be used +later during the execution phase to build the rule's outputs, if they are +needed. ## Rule creation -In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new rule, and store the result in a global variable. The call to `rule` specifies [attributes](#attributes) and an [implementation function](#implementation_function): +In a `.bzl` file, use the [rule](/rules/lib/globals/bzl#rule) function to define a new +rule, and store the result in a global variable. The call to `rule` specifies +[attributes](#attributes) and an +[implementation function](#implementation_function): ```python example_library = rule( @@ -37,7 +66,10 @@ example_library = rule( This defines a [rule kind](/query/language#kind) named `example_library`. -The call to `rule` also must specify if the rule creates an [executable](#executable-rules) output (with `executable = True`), or specifically a test executable (with `test = True`). If the latter, the rule is a *test rule*, and the name of the rule must end in `_test`. +The call to `rule` also must specify if the rule creates an +[executable](#executable-rules) output (with `executable = True`), or specifically +a test executable (with `test = True`). If the latter, the rule is a *test rule*, +and the name of the rule must end in `_test`. ## Target instantiation @@ -53,23 +85,48 @@ example_library( ) ``` -Each call to a build rule returns no value, but has the side effect of defining a target. This is called *instantiating* the rule. This specifies a name for the new target and values for the target's [attributes](#attributes). +Each call to a build rule returns no value, but has the side effect of defining +a target. This is called *instantiating* the rule. This specifies a name for the +new target and values for the target's [attributes](#attributes). -Rules can also be called from Starlark functions and loaded in `.bzl` files. Starlark functions that call rules are called [Starlark macros](/extending/macros). Starlark macros must ultimately be called from `BUILD` files, and can only be called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` files are evaluated to instantiate targets. +Rules can also be called from Starlark functions and loaded in `.bzl` files. +Starlark functions that call rules are called [Starlark macros](/extending/macros). +Starlark macros must ultimately be called from `BUILD` files, and can only be +called during the [loading phase](/extending/concepts#evaluation-model), when `BUILD` +files are evaluated to instantiate targets. ## Attributes -An *attribute* is a rule argument. Attributes can provide specific values to a target's [implementation](#implementation_function), or they can refer to other targets, creating a graph of dependencies. +An *attribute* is a rule argument. Attributes can provide specific values to a +target's [implementation](#implementation_function), or they can refer to other +targets, creating a graph of dependencies. -Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) module) to the `attrs` parameter of `rule`. [Common attributes](/reference/be/common-definitions#common-attributes), such as `name` and `visibility`, are implicitly added to all rules. Additional attributes are implicitly added to [executable and test rules](#executable-rules) specifically. Attributes which are implicitly added to a rule can't be included in the dictionary passed to `attrs`. +Rule-specific attributes, such as `srcs` or `deps`, are defined by passing a map +from attribute names to schemas (created using the [`attr`](/rules/lib/toplevel/attr) +module) to the `attrs` parameter of `rule`. +[Common attributes](/reference/be/common-definitions#common-attributes), such as +`name` and `visibility`, are implicitly added to all rules. Additional +attributes are implicitly added to +[executable and test rules](#executable-rules) specifically. Attributes which +are implicitly added to a rule can't be included in the dictionary passed to +`attrs`. ### Dependency attributes -Rules that process source code usually define the following attributes to handle various [types of dependencies](/concepts/dependencies#types_of_dependencies): - -- `srcs` specifies source files processed by a target's actions. Often, the attribute schema specifies which file extensions are expected for the sort of source file the rule processes. Rules for languages with header files generally specify a separate `hdrs` attribute for headers processed by a target and its consumers. -- `deps` specifies code dependencies for a target. The attribute schema should specify which [providers](#providers) those dependencies must provide. (For example, `cc_library` provides `CcInfo`.) -- `data` specifies files to be made available at runtime to any executable which depends on a target. That should allow arbitrary files to be specified. +Rules that process source code usually define the following attributes to handle +various [types of dependencies](/concepts/dependencies#types_of_dependencies): + +* `srcs` specifies source files processed by a target's actions. Often, the + attribute schema specifies which file extensions are expected for the sort + of source file the rule processes. Rules for languages with header files + generally specify a separate `hdrs` attribute for headers processed by a + target and its consumers. +* `deps` specifies code dependencies for a target. The attribute schema should + specify which [providers](#providers) those dependencies must provide. (For + example, `cc_library` provides `CcInfo`.) +* `data` specifies files to be made available at runtime to any executable + which depends on a target. That should allow arbitrary files to be + specified. ```python example_library = rule( @@ -84,7 +141,16 @@ example_library = rule( ) ``` -These are examples of *dependency attributes*. Any attribute that specifies an input label (those defined with [`attr.label_list`](/rules/lib/toplevel/attr#label_list), [`attr.label`](/rules/lib/toplevel/attr#label), or [`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) specifies dependencies of a certain type between a target and the targets whose labels (or the corresponding [`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target is defined. The repository, and possibly the path, for these labels is resolved relative to the defined target. +These are examples of *dependency attributes*. Any attribute that specifies +an input label (those defined with +[`attr.label_list`](/rules/lib/toplevel/attr#label_list), +[`attr.label`](/rules/lib/toplevel/attr#label), or +[`attr.label_keyed_string_dict`](/rules/lib/toplevel/attr#label_keyed_string_dict)) +specifies dependencies of a certain type +between a target and the targets whose labels (or the corresponding +[`Label`](/rules/lib/builtins/Label) objects) are listed in that attribute when the target +is defined. The repository, and possibly the path, for these labels is resolved +relative to the defined target. ```python example_library( @@ -98,15 +164,27 @@ example_library( ) ``` -In this example, `other_target` is a dependency of `my_target`, and therefore `other_target` is analyzed first. It is an error if there is a cycle in the dependency graph of targets. +In this example, `other_target` is a dependency of `my_target`, and therefore +`other_target` is analyzed first. It is an error if there is a cycle in the +dependency graph of targets. -[]() + ### Private attributes and implicit dependencies -A dependency attribute with a default value creates an *implicit dependency*. It is implicit because it's a part of the target graph that the user doesn't specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a relationship between a rule and a *tool* (a build-time dependency, such as a compiler), since most of the time a user is not interested in specifying what tool the rule uses. Inside the rule's implementation function, this is treated the same as other dependencies. +A dependency attribute with a default value creates an *implicit dependency*. It +is implicit because it's a part of the target graph that the user doesn't +specify it in a `BUILD` file. Implicit dependencies are useful for hard-coding a +relationship between a rule and a *tool* (a build-time dependency, such as a +compiler), since most of the time a user is not interested in specifying what +tool the rule uses. Inside the rule's implementation function, this is treated +the same as other dependencies. -If you want to provide an implicit dependency without allowing the user to override that value, you can make the attribute *private* by giving it a name that begins with an underscore (`_`). Private attributes must have default values. It generally only makes sense to use private attributes for implicit dependencies. +If you want to provide an implicit dependency without allowing the user to +override that value, you can make the attribute *private* by giving it a name +that begins with an underscore (`_`). Private attributes must have default +values. It generally only makes sense to use private attributes for implicit +dependencies. ```python example_library = rule( @@ -123,34 +201,67 @@ example_library = rule( ) ``` -In this example, every target of type `example_library` has an implicit dependency on the compiler `//tools:example_compiler`. This allows `example_library`'s implementation function to generate actions that invoke the compiler, even though the user did not pass its label as an input. Since `_compiler` is a private attribute, it follows that `ctx.attr._compiler` will always point to `//tools:example_compiler` in all targets of this rule type. Alternatively, you can name the attribute `compiler` without the underscore and keep the default value. This allows users to substitute a different compiler if necessary, but it requires no awareness of the compiler's label. - -Implicit dependencies are generally used for tools that reside in the same repository as the rule implementation. If the tool comes from the [execution platform](/extending/platforms) or a different repository instead, the rule should obtain that tool from a [toolchain](/extending/toolchains). +In this example, every target of type `example_library` has an implicit +dependency on the compiler `//tools:example_compiler`. This allows +`example_library`'s implementation function to generate actions that invoke the +compiler, even though the user did not pass its label as an input. Since +`_compiler` is a private attribute, it follows that `ctx.attr._compiler` +will always point to `//tools:example_compiler` in all targets of this rule +type. Alternatively, you can name the attribute `compiler` without the +underscore and keep the default value. This allows users to substitute a +different compiler if necessary, but it requires no awareness of the compiler's +label. + +Implicit dependencies are generally used for tools that reside in the same +repository as the rule implementation. If the tool comes from the +[execution platform](/extending/platforms) or a different repository instead, the +rule should obtain that tool from a [toolchain](/extending/toolchains). ### Output attributes -*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and [`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the target generates. These differ from dependency attributes in two ways: +*Output attributes*, such as [`attr.output`](/rules/lib/toplevel/attr#output) and +[`attr.output_list`](/rules/lib/toplevel/attr#output_list), declare an output file that the +target generates. These differ from dependency attributes in two ways: -- They define output file targets instead of referring to targets defined elsewhere. -- The output file targets depend on the instantiated rule target, instead of the other way around. +* They define output file targets instead of referring to targets defined + elsewhere. +* The output file targets depend on the instantiated rule target, instead of + the other way around. -Typically, output attributes are only used when a rule needs to create outputs with user-defined names which can't be based on the target name. If a rule has one output attribute, it is typically named `out` or `outs`. +Typically, output attributes are only used when a rule needs to create outputs +with user-defined names which can't be based on the target name. If a rule has +one output attribute, it is typically named `out` or `outs`. -Output attributes are the preferred way of creating *predeclared outputs*, which can be specifically depended upon or [requested at the command line](#requesting_output_files). +Output attributes are the preferred way of creating *predeclared outputs*, which +can be specifically depended upon or +[requested at the command line](#requesting_output_files). ## Implementation function -Every rule requires an `implementation` function. These functions are executed strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the graph of targets generated in the loading phase into a graph of [actions](#actions) to be performed during the execution phase. As such, implementation functions can't actually read or write files. +Every rule requires an `implementation` function. These functions are executed +strictly in the [analysis phase](/extending/concepts#evaluation-model) and transform the +graph of targets generated in the loading phase into a graph of +[actions](#actions) to be performed during the execution phase. As such, +implementation functions can't actually read or write files. -Rule implementation functions are usually private (named with a leading underscore). Conventionally, they are named the same as their rule, but suffixed with `_impl`. +Rule implementation functions are usually private (named with a leading +underscore). Conventionally, they are named the same as their rule, but suffixed +with `_impl`. -Implementation functions take exactly one parameter: a [rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of [providers](#providers). +Implementation functions take exactly one parameter: a +[rule context](/rules/lib/builtins/ctx), conventionally named `ctx`. They return a list of +[providers](#providers). ### Targets -Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) objects. These objects contain the [providers](#providers) generated when the target's implementation function was executed. +Dependencies are represented at analysis time as [`Target`](/rules/lib/builtins/Target) +objects. These objects contain the [providers](#providers) generated when the +target's implementation function was executed. -[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each dependency attribute, containing `Target` objects representing each direct dependency using that attribute. For `label_list` attributes, this is a list of `Targets`. For `label` attributes, this is a single `Target` or `None`. +[`ctx.attr`](/rules/lib/builtins/ctx#attr) has fields corresponding to the names of each +dependency attribute, containing `Target` objects representing each direct +dependency using that attribute. For `label_list` attributes, this is a list of +`Targets`. For `label` attributes, this is a single `Target` or `None`. A list of provider objects are returned by a target's implementation function: @@ -158,9 +269,14 @@ A list of provider objects are returned by a target's implementation function: return [ExampleInfo(headers = depset(...))] ``` -Those can be accessed using index notation (`[]`), with the type of provider as a key. These can be [custom providers](#custom_providers) defined in Starlark or [providers for native rules](/rules/lib/providers) available as Starlark global variables. +Those can be accessed using index notation (`[]`), with the type of provider as +a key. These can be [custom providers](#custom_providers) defined in Starlark or +[providers for native rules](/rules/lib/providers) available as Starlark +global variables. -For example, if a rule takes header files using a `hdrs` attribute and provides them to the compilation actions of the target and its consumers, it could collect them like so: +For example, if a rule takes header files using a `hdrs` attribute and provides +them to the compilation actions of the target and its consumers, it could +collect them like so: ```python def _example_library_impl(ctx): @@ -168,15 +284,24 @@ def _example_library_impl(ctx): transitive_headers = [hdr[ExampleInfo].headers for hdr in ctx.attr.hdrs] ``` -There's a legacy struct style, which is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). +There's a legacy struct style, which is strongly discouraged and rules should be +[migrated away from it](#migrating_from_legacy_providers). ### Files -Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't perform file I/O during the analysis phase, these objects can't be used to directly read or write file content. Rather, they are passed to action-emitting functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the action graph. +Files are represented by [`File`](/rules/lib/builtins/File) objects. Since Bazel doesn't +perform file I/O during the analysis phase, these objects can't be used to +directly read or write file content. Rather, they are passed to action-emitting +functions (see [`ctx.actions`](/rules/lib/builtins/actions)) to construct pieces of the +action graph. -A `File` can either be a source file or a generated file. Each generated file must be an output of exactly one action. Source files can't be the output of any action. +A `File` can either be a source file or a generated file. Each generated file +must be an output of exactly one action. Source files can't be the output of +any action. -For each dependency attribute, the corresponding field of [`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all dependencies using that attribute: +For each dependency attribute, the corresponding field of +[`ctx.files`](/rules/lib/builtins/ctx#files) contains a list of the default outputs of all +dependencies using that attribute: ```python def _example_library_impl(ctx): @@ -186,11 +311,20 @@ def _example_library_impl(ctx): ... ``` -[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for dependency attributes whose specs set `allow_single_file = True`. [`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only contains fields for dependency attributes whose specs set `executable = True`. +[`ctx.file`](/rules/lib/builtins/ctx#file) contains a single `File` or `None` for +dependency attributes whose specs set `allow_single_file = True`. +[`ctx.executable`](/rules/lib/builtins/ctx#executable) behaves the same as `ctx.file`, but only +contains fields for dependency attributes whose specs set `executable = True`. ### Declaring outputs -During the analysis phase, a rule's implementation function can create outputs. Since all labels have to be known during the loading phase, these additional outputs have no labels. `File` objects for outputs can be created using [`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and [`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). Often, the names of outputs are based on the target's name, [`ctx.label.name`](/rules/lib/builtins/ctx#label): +During the analysis phase, a rule's implementation function can create outputs. +Since all labels have to be known during the loading phase, these additional +outputs have no labels. `File` objects for outputs can be created using +[`ctx.actions.declare_file`](/rules/lib/builtins/actions#declare_file) and +[`ctx.actions.declare_directory`](/rules/lib/builtins/actions#declare_directory). +Often, the names of outputs are based on the target's name, +[`ctx.label.name`](/rules/lib/builtins/ctx#label): ```python def _example_library_impl(ctx): @@ -199,20 +333,31 @@ def _example_library_impl(ctx): ... ``` -For *predeclared outputs*, like those created for [output attributes](#output_attributes), `File` objects instead can be retrieved from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). +For *predeclared outputs*, like those created for +[output attributes](#output_attributes), `File` objects instead can be retrieved +from the corresponding fields of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). ### Actions -An action describes how to generate a set of outputs from a set of inputs, for example "run gcc on hello.c and get hello.o". When an action is created, Bazel doesn't run the command immediately. It registers it in a graph of dependencies, because an action can depend on the output of another action. For example, in C, the linker must be called after the compiler. +An action describes how to generate a set of outputs from a set of inputs, for +example "run gcc on hello.c and get hello.o". When an action is created, Bazel +doesn't run the command immediately. It registers it in a graph of dependencies, +because an action can depend on the output of another action. For example, in C, +the linker must be called after the compiler. -General-purpose functions that create actions are defined in [`ctx.actions`](/rules/lib/builtins/actions): +General-purpose functions that create actions are defined in +[`ctx.actions`](/rules/lib/builtins/actions): -- [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. -- [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell command. -- [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. -- [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to generate a file from a template. +* [`ctx.actions.run`](/rules/lib/builtins/actions#run), to run an executable. +* [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell), to run a shell + command. +* [`ctx.actions.write`](/rules/lib/builtins/actions#write), to write a string to a file. +* [`ctx.actions.expand_template`](/rules/lib/builtins/actions#expand_template), to + generate a file from a template. -[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently accumulate the arguments for actions. It avoids flattening depsets until execution time: +[`ctx.actions.args`](/rules/lib/builtins/actions#args) can be used to efficiently +accumulate the arguments for actions. It avoids flattening depsets until +execution time: ```python def _example_library_impl(ctx): @@ -239,31 +384,61 @@ def _example_library_impl(ctx): ... ``` -Actions take a list or depset of input files and generate a (non-empty) list of output files. The set of input and output files must be known during the [analysis phase](/extending/concepts#evaluation-model). It might depend on the value of attributes, including providers from dependencies, but it can't depend on the result of the execution. For example, if your action runs the unzip command, you must specify which files you expect to be inflated (before running unzip). Actions which create a variable number of files internally can wrap those in a single file (such as a zip, tar, or other archive format). +Actions take a list or depset of input files and generate a (non-empty) list of +output files. The set of input and output files must be known during the +[analysis phase](/extending/concepts#evaluation-model). It might depend on the value of +attributes, including providers from dependencies, but it can't depend on the +result of the execution. For example, if your action runs the unzip command, you +must specify which files you expect to be inflated (before running unzip). +Actions which create a variable number of files internally can wrap those in a +single file (such as a zip, tar, or other archive format). -Actions must list all of their inputs. Listing inputs that are not used is permitted, but inefficient. +Actions must list all of their inputs. Listing inputs that are not used is +permitted, but inefficient. -Actions must create all of their outputs. They may write other files, but anything not in outputs won't be available to consumers. All declared outputs must be written by some action. +Actions must create all of their outputs. They may write other files, but +anything not in outputs won't be available to consumers. All declared outputs +must be written by some action. -Actions are comparable to pure functions: They should depend only on the provided inputs, and avoid accessing computer information, username, clock, network, or I/O devices (except for reading inputs and writing outputs). This is important because the output will be cached and reused. +Actions are comparable to pure functions: They should depend only on the +provided inputs, and avoid accessing computer information, username, clock, +network, or I/O devices (except for reading inputs and writing outputs). This is +important because the output will be cached and reused. -Dependencies are resolved by Bazel, which decides which actions to execute. It is an error if there is a cycle in the dependency graph. Creating an action doesn't guarantee that it will be executed, that depends on whether its outputs are needed for the build. +Dependencies are resolved by Bazel, which decides which actions to +execute. It is an error if there is a cycle in the dependency graph. Creating +an action doesn't guarantee that it will be executed, that depends on whether +its outputs are needed for the build. ### Providers -Providers are pieces of information that a rule exposes to other rules that depend on it. This data can include output files, libraries, parameters to pass on a tool's command line, or anything else a target's consumers should know about. +Providers are pieces of information that a rule exposes to other rules that +depend on it. This data can include output files, libraries, parameters to pass +on a tool's command line, or anything else a target's consumers should know +about. -Since a rule's implementation function can only read providers from the instantiated target's immediate dependencies, rules need to forward any information from a target's dependencies that needs to be known by a target's consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). +Since a rule's implementation function can only read providers from the +instantiated target's immediate dependencies, rules need to forward any +information from a target's dependencies that needs to be known by a target's +consumers, generally by accumulating that into a [`depset`](/rules/lib/builtins/depset). -A target's providers are specified by a list of provider objects returned by the implementation function. +A target's providers are specified by a list of provider objects returned by +the implementation function. -Old implementation functions can also be written in a legacy style where the implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of provider objects. This style is strongly discouraged and rules should be [migrated away from it](#migrating_from_legacy_providers). +Old implementation functions can also be written in a legacy style where the +implementation function returns a [`struct`](/rules/lib/builtins/struct) instead of list of +provider objects. This style is strongly discouraged and rules should be +[migrated away from it](#migrating_from_legacy_providers). #### Default outputs -A target's *default outputs* are the outputs that are requested by default when the target is requested for build at the command line. For example, a `java_library` target `//pkg:foo` has `foo.jar` as a default output, so that will be built by the command `bazel build //pkg:foo`. +A target's *default outputs* are the outputs that are requested by default when +the target is requested for build at the command line. For example, a +`java_library` target `//pkg:foo` has `foo.jar` as a default output, so that +will be built by the command `bazel build //pkg:foo`. -Default outputs are specified by the `files` parameter of [`DefaultInfo`](/rules/lib/providers/DefaultInfo): +Default outputs are specified by the `files` parameter of +[`DefaultInfo`](/rules/lib/providers/DefaultInfo): ```python def _example_library_impl(ctx): @@ -274,17 +449,37 @@ def _example_library_impl(ctx): ] ``` -If `DefaultInfo` is not returned by a rule implementation or the `files` parameter is not specified, `DefaultInfo.files` defaults to all *predeclared outputs* (generally, those created by [output attributes](#output_attributes)). +If `DefaultInfo` is not returned by a rule implementation or the `files` +parameter is not specified, `DefaultInfo.files` defaults to all +*predeclared outputs* (generally, those created by [output +attributes](#output_attributes)). -Rules that perform actions should provide default outputs, even if those outputs are not expected to be directly used. Actions that are not in the graph of the requested outputs are pruned. If an output is only used by a target's consumers, those actions won't be performed when the target is built in isolation. This makes debugging more difficult because rebuilding just the failing target won't reproduce the failure. +Rules that perform actions should provide default outputs, even if those outputs +are not expected to be directly used. Actions that are not in the graph of the +requested outputs are pruned. If an output is only used by a target's consumers, +those actions won't be performed when the target is built in isolation. This +makes debugging more difficult because rebuilding just the failing target won't +reproduce the failure. #### Runfiles -Runfiles are a set of files used by a target at runtime (as opposed to build time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates a directory tree containing symlinks pointing to the runfiles. This stages the environment for the binary so it can access the runfiles during runtime. +Runfiles are a set of files used by a target at runtime (as opposed to build +time). During the [execution phase](/extending/concepts#evaluation-model), Bazel creates +a directory tree containing symlinks pointing to the runfiles. This stages the +environment for the binary so it can access the runfiles during runtime. -Runfiles can be added manually during rule creation. [`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the `runfiles` parameter on `DefaultInfo`. The executable output of [executable rules](#executable-rules) is implicitly added to the runfiles. +Runfiles can be added manually during rule creation. +[`runfiles`](/rules/lib/builtins/runfiles) objects can be created by the `runfiles` method +on the rule context, [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and passed to the +`runfiles` parameter on `DefaultInfo`. The executable output of +[executable rules](#executable-rules) is implicitly added to the runfiles. -Some rules specify attributes, generally named [`data`](/reference/be/common-definitions#common.data), whose outputs are added to a targets' runfiles. Runfiles should also be merged in from `data`, as well as from any attributes which might provide code for eventual execution, generally `srcs` (which might contain `filegroup` targets with associated `data`) and `deps`. +Some rules specify attributes, generally named +[`data`](/reference/be/common-definitions#common.data), whose outputs are added to +a targets' runfiles. Runfiles should also be merged in from `data`, as well as +from any attributes which might provide code for eventual execution, generally +`srcs` (which might contain `filegroup` targets with associated `data`) and +`deps`. ```python def _example_library_impl(ctx): @@ -308,7 +503,8 @@ def _example_library_impl(ctx): #### Custom providers -Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) function to convey rule-specific information: +Providers can be defined using the [`provider`](/rules/lib/globals/bzl#provider) +function to convey rule-specific information: ```python ExampleInfo = provider( @@ -341,11 +537,23 @@ def _example_library_impl(ctx): ##### Custom initialization of providers -It's possible to guard the instantiation of a provider with custom preprocessing and validation logic. This can be used to ensure that all provider instances satisfy certain invariants, or to give users a cleaner API for obtaining an instance. +It's possible to guard the instantiation of a provider with custom +preprocessing and validation logic. This can be used to ensure that all +provider instances satisfy certain invariants, or to give users a cleaner API for +obtaining an instance. -This is done by passing an `init` callback to the [`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the return type of `provider()` changes to be a tuple of two values: the provider symbol that is the ordinary return value when `init` is not used, and a "raw constructor". +This is done by passing an `init` callback to the +[`provider`](/rules/lib/globals/bzl.html#provider) function. If this callback is given, the +return type of `provider()` changes to be a tuple of two values: the provider +symbol that is the ordinary return value when `init` is not used, and a "raw +constructor". -In this case, when the provider symbol is called, instead of directly returning a new instance, it will forward the arguments along to the `init` callback. The callback's return value must be a dict mapping field names (strings) to values; this is used to initialize the fields of the new instance. Note that the callback may have any signature, and if the arguments don't match the signature an error is reported as if the callback were invoked directly. +In this case, when the provider symbol is called, instead of directly returning +a new instance, it will forward the arguments along to the `init` callback. The +callback's return value must be a dict mapping field names (strings) to values; +this is used to initialize the fields of the new instance. Note that the +callback may have any signature, and if the arguments don't match the signature +an error is reported as if the callback were invoked directly. The raw constructor, by contrast, will bypass the `init` callback. @@ -378,7 +586,9 @@ ExampleInfo( ) ``` -The raw constructor can be used to define alternative public factory functions that don't go through the `init` logic. For example, exampleinfo.bzl could define: +The raw constructor can be used to define alternative public factory functions +that don't go through the `init` logic. For example, exampleinfo.bzl +could define: ```python def make_barebones_exampleinfo(headers): @@ -386,9 +596,12 @@ def make_barebones_exampleinfo(headers): return _new_exampleinfo(files_to_link = depset(), headers = all_headers) ``` -Typically, the raw constructor is bound to a variable whose name begins with an underscore (`_new_exampleinfo` above), so that user code can't load it and generate arbitrary provider instances. +Typically, the raw constructor is bound to a variable whose name begins with an +underscore (`_new_exampleinfo` above), so that user code can't load it and +generate arbitrary provider instances. -Another use for `init` is to prevent the user from calling the provider symbol altogether, and force them to use a factory function instead: +Another use for `init` is to prevent the user from calling the provider +symbol altogether, and force them to use a factory function instead: ```python def _exampleinfo_init_banned(*args, **kwargs): @@ -403,11 +616,15 @@ def make_exampleinfo(...): return _new_exampleinfo(...) ``` -[]() + ## Executable rules and test rules -Executable rules define targets that can be invoked by a `bazel run` command. Test rules are a special kind of executable rule whose targets can also be invoked by a `bazel test` command. Executable and test rules are created by setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or [`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: +Executable rules define targets that can be invoked by a `bazel run` command. +Test rules are a special kind of executable rule whose targets can also be +invoked by a `bazel test` command. Executable and test rules are created by +setting the respective [`executable`](/rules/lib/globals/bzl#rule.executable) or +[`test`](/rules/lib/globals/bzl#rule.test) argument to `True` in the call to `rule`: ```python example_binary = rule( @@ -423,9 +640,17 @@ example_test = rule( ) ``` -Test rules must have names that end in `_test`. (Test *target* names also often end in `_test` by convention, but this is not required.) Non-test rules must not have this suffix. +Test rules must have names that end in `_test`. (Test *target* names also often +end in `_test` by convention, but this is not required.) Non-test rules must not +have this suffix. -Both kinds of rules must produce an executable output file (which may or may not be predeclared) that will be invoked by the `run` or `test` commands. To tell Bazel which of a rule's outputs to use as this executable, pass it as the `executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) provider. That `executable` is added to the default outputs of the rule (so you don't need to pass that to both `executable` and `files`). It's also implicitly added to the [runfiles](#runfiles): +Both kinds of rules must produce an executable output file (which may or may not +be predeclared) that will be invoked by the `run` or `test` commands. To tell +Bazel which of a rule's outputs to use as this executable, pass it as the +`executable` argument of a returned [`DefaultInfo`](/rules/lib/providers/DefaultInfo) +provider. That `executable` is added to the default outputs of the rule (so you +don't need to pass that to both `executable` and `files`). It's also implicitly +added to the [runfiles](#runfiles): ```python def _example_binary_impl(ctx): @@ -437,13 +662,30 @@ def _example_binary_impl(ctx): ] ``` -The action that generates this file must set the executable bit on the file. For a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or [`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done by the underlying tool that is invoked by the action. For a [`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. - -As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a special `ctx.outputs.executable` predeclared output. This file serves as the default executable if you don't specify one using `DefaultInfo`; it must not be used otherwise. This output mechanism is deprecated because it doesn't support customizing the executable file's name at analysis time. - -See examples of an [executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) and a [test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). - -[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and [test rules](/reference/be/common-definitions#common-attributes-tests) have additional attributes implicitly defined, in addition to those added for [all rules](/reference/be/common-definitions#common-attributes). The defaults of implicitly-added attributes can't be changed, though this can be worked around by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the default: +The action that generates this file must set the executable bit on the file. For +a [`ctx.actions.run`](/rules/lib/builtins/actions#run) or +[`ctx.actions.run_shell`](/rules/lib/builtins/actions#run_shell) action this should be done +by the underlying tool that is invoked by the action. For a +[`ctx.actions.write`](/rules/lib/builtins/actions#write) action, pass `is_executable = True`. + +As [legacy behavior](#deprecated_predeclared_outputs), executable rules have a +special `ctx.outputs.executable` predeclared output. This file serves as the +default executable if you don't specify one using `DefaultInfo`; it must not be +used otherwise. This output mechanism is deprecated because it doesn't support +customizing the executable file's name at analysis time. + +See examples of an +[executable rule](https://github.com/bazelbuild/examples/blob/main/rules/executable/fortune.bzl) +and a +[test rule](https://github.com/bazelbuild/examples/blob/main/rules/test_rule/line_length.bzl). + +[Executable rules](/reference/be/common-definitions#common-attributes-binaries) and +[test rules](/reference/be/common-definitions#common-attributes-tests) have additional +attributes implicitly defined, in addition to those added for +[all rules](/reference/be/common-definitions#common-attributes). The defaults of +implicitly-added attributes can't be changed, though this can be worked around +by wrapping a private rule in a [Starlark macro](/extending/macros) which alters the +default: ```python def example_test(size = "small", **kwargs): @@ -456,7 +698,8 @@ _example_test = rule( ### Runfiles location -When an executable target is run with `bazel run` (or `test`), the root of the runfiles directory is adjacent to the executable. The paths relate as follows: +When an executable target is run with `bazel run` (or `test`), the root of the +runfiles directory is adjacent to the executable. The paths relate as follows: ```python # Given launcher_path and runfile_file: @@ -467,21 +710,50 @@ execution_root_relative_path = "%s/%s/%s" % ( runfiles_root, workspace_name, runfile_path) ``` -The path to a `File` under the runfiles directory corresponds to [`File.short_path`](/rules/lib/builtins/File#short_path). +The path to a `File` under the runfiles directory corresponds to +[`File.short_path`](/rules/lib/builtins/File#short_path). -The binary executed directly by `bazel` is adjacent to the root of the `runfiles` directory. However, binaries called *from* the runfiles can't make the same assumption. To mitigate this, each binary should provide a way to accept its runfiles root as a parameter using an environment, or command line argument or flag. This allows binaries to pass the correct canonical runfiles root to the binaries it calls. If that's not set, a binary can guess that it was the first binary called and look for an adjacent runfiles directory. +The binary executed directly by `bazel` is adjacent to the root of the +`runfiles` directory. However, binaries called *from* the runfiles can't make +the same assumption. To mitigate this, each binary should provide a way to +accept its runfiles root as a parameter using an environment, or command line +argument or flag. This allows binaries to pass the correct canonical runfiles root +to the binaries it calls. If that's not set, a binary can guess that it was the +first binary called and look for an adjacent runfiles directory. ## Advanced topics ### Requesting output files -A single target can have several output files. When a `bazel build` command is run, some of the outputs of the targets given to the command are considered to be *requested*. Bazel only builds these requested files and the files that they directly or indirectly depend on. (In terms of the action graph, Bazel only executes the actions that are reachable as transitive dependencies of the requested files.) - -In addition to [default outputs](#default_outputs), any *predeclared output* can be explicitly requested on the command line. Rules can specify predeclared outputs using [output attributes](#output_attributes). In that case, the user explicitly chooses labels for outputs when they instantiate the rule. To obtain [`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can [implicitly define predeclared outputs](#deprecated_predeclared_outputs) based on the target name as well, but this feature is deprecated. - -In addition to default outputs, there are *output groups*, which are collections of output files that may be requested together. These can be requested with [`--output_groups`](/reference/command-line-reference#flag--output_groups). For example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` output group, these files can be built by running `bazel build //pkg:mytarget --output_groups=debug_files`. Since non-predeclared outputs don't have labels, they can only be requested by appearing in the default outputs or an output group. - -Output groups can be specified with the [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many built-in providers, `OutputGroupInfo` can take parameters with arbitrary names to define output groups with that name: +A single target can have several output files. When a `bazel build` command is +run, some of the outputs of the targets given to the command are considered to +be *requested*. Bazel only builds these requested files and the files that they +directly or indirectly depend on. (In terms of the action graph, Bazel only +executes the actions that are reachable as transitive dependencies of the +requested files.) + +In addition to [default outputs](#default_outputs), any *predeclared output* can +be explicitly requested on the command line. Rules can specify predeclared +outputs using [output attributes](#output_attributes). In that case, the user +explicitly chooses labels for outputs when they instantiate the rule. To obtain +[`File`](/rules/lib/builtins/File) objects for output attributes, use the corresponding +attribute of [`ctx.outputs`](/rules/lib/builtins/ctx#outputs). Rules can +[implicitly define predeclared outputs](#deprecated_predeclared_outputs) based +on the target name as well, but this feature is deprecated. + +In addition to default outputs, there are *output groups*, which are collections +of output files that may be requested together. These can be requested with +[`--output_groups`](/reference/command-line-reference#flag--output_groups). For +example, if a target `//pkg:mytarget` is of a rule type that has a `debug_files` +output group, these files can be built by running `bazel build //pkg:mytarget +--output_groups=debug_files`. Since non-predeclared outputs don't have labels, +they can only be requested by appearing in the default outputs or an output +group. + +Output groups can be specified with the +[`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo) provider. Note that unlike many +built-in providers, `OutputGroupInfo` can take parameters with arbitrary names +to define output groups with that name: ```python def _example_library_impl(ctx): @@ -498,37 +770,82 @@ def _example_library_impl(ctx): ] ``` -Also unlike most providers, `OutputGroupInfo` can be returned by both an [aspect](/extending/aspects) and the rule target to which that aspect is applied, as long as they don't define the same output groups. In that case, the resulting providers are merged. +Also unlike most providers, `OutputGroupInfo` can be returned by both an +[aspect](/extending/aspects) and the rule target to which that aspect is applied, as +long as they don't define the same output groups. In that case, the resulting +providers are merged. -Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts of files from a target to the actions of its consumers. Define [rule-specific providers](#custom_providers) for that instead. +Note that `OutputGroupInfo` generally shouldn't be used to convey specific sorts +of files from a target to the actions of its consumers. Define +[rule-specific providers](#custom_providers) for that instead. ### Configurations -Imagine that you want to build a C++ binary for a different architecture. The build can be complex and involve multiple steps. Some of the intermediate binaries, like compilers and code generators, have to run on [the execution platform](/extending/platforms#overview) (which could be your host, or a remote executor). Some binaries like the final output must be built for the target architecture. - -For this reason, Bazel has a concept of "configurations" and transitions. The topmost targets (the ones requested on the command line) are built-in the "target" configuration, while tools that should run on the execution platform are built-in an "exec" configuration. Rules may generate different actions based on the configuration, for instance to change the cpu architecture that is passed to the compiler. In some cases, the same library may be needed for different configurations. If this happens, it will be analyzed and potentially built multiple times. - -By default, Bazel builds a target's dependencies in the same configuration as the target itself, in other words without transitions. When a dependency is a tool that's needed to help build the target, the corresponding attribute should specify a transition to an exec configuration. This causes the tool and all its dependencies to build for the execution platform. - -For each dependency attribute, you can use `cfg` to decide if dependencies should build in the same configuration or transition to an exec configuration. If a dependency attribute has the flag `executable = True`, `cfg` must be set explicitly. This is to guard against accidentally building a tool for the wrong configuration. [See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) - -In general, sources, dependent libraries, and executables that will be needed at runtime can use the same configuration. - -Tools that are executed as part of the build (such as compilers or code generators) should be built for an exec configuration. In this case, specify `cfg = "exec"` in the attribute. - -Otherwise, executables that are used at runtime (such as as part of a test) should be built for the target configuration. In this case, specify `cfg = "target"` in the attribute. - -`cfg = "target"` doesn't actually do anything: it's purely a convenience value to help rule designers be explicit about their intentions. When `executable = False`, which means `cfg` is optional, only set this when it truly helps readability. - -You can also use `cfg = my_transition` to use [user-defined transitions](/extending/config#user-defined-transitions), which allow rule authors a great deal of flexibility in changing configurations, with the drawback of [making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). - -**Note**: Historically, Bazel didn't have the concept of execution platforms, and instead all build actions were considered to run on the host machine. Bazel versions before 6.0 created a distinct "host" configuration to represent this. If you see references to "host" in code or old documentation, that's what this refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual overhead. - -[]() +Imagine that you want to build a C++ binary for a different architecture. The +build can be complex and involve multiple steps. Some of the intermediate +binaries, like compilers and code generators, have to run on +[the execution platform](/extending/platforms#overview) (which could be your host, +or a remote executor). Some binaries like the final output must be built for the +target architecture. + +For this reason, Bazel has a concept of "configurations" and transitions. The +topmost targets (the ones requested on the command line) are built-in the +"target" configuration, while tools that should run on the execution platform +are built-in an "exec" configuration. Rules may generate different actions based +on the configuration, for instance to change the cpu architecture that is passed +to the compiler. In some cases, the same library may be needed for different +configurations. If this happens, it will be analyzed and potentially built +multiple times. + +By default, Bazel builds a target's dependencies in the same configuration as +the target itself, in other words without transitions. When a dependency is a +tool that's needed to help build the target, the corresponding attribute should +specify a transition to an exec configuration. This causes the tool and all its +dependencies to build for the execution platform. + +For each dependency attribute, you can use `cfg` to decide if dependencies +should build in the same configuration or transition to an exec configuration. +If a dependency attribute has the flag `executable = True`, `cfg` must be set +explicitly. This is to guard against accidentally building a tool for the wrong +configuration. +[See example](https://github.com/bazelbuild/examples/blob/main/rules/actions_run/execute.bzl) + +In general, sources, dependent libraries, and executables that will be needed at +runtime can use the same configuration. + +Tools that are executed as part of the build (such as compilers or code generators) +should be built for an exec configuration. In this case, specify `cfg = "exec"` in +the attribute. + +Otherwise, executables that are used at runtime (such as as part of a test) should +be built for the target configuration. In this case, specify `cfg = "target"` in +the attribute. + +`cfg = "target"` doesn't actually do anything: it's purely a convenience value to +help rule designers be explicit about their intentions. When `executable = False`, +which means `cfg` is optional, only set this when it truly helps readability. + +You can also use `cfg = my_transition` to use +[user-defined transitions](/extending/config#user-defined-transitions), which allow +rule authors a great deal of flexibility in changing configurations, with the +drawback of +[making the build graph larger and less comprehensible](/extending/config#memory-and-performance-considerations). + +**Note**: Historically, Bazel didn't have the concept of execution platforms, +and instead all build actions were considered to run on the host machine. Bazel +versions before 6.0 created a distinct "host" configuration to represent this. +If you see references to "host" in code or old documentation, that's what this +refers to. We recommend using Bazel 6.0 or newer to avoid this extra conceptual +overhead. + + ### Configuration fragments -Rules may access [configuration fragments](/rules/lib/fragments) such as `cpp` and `java`. However, all required fragments must be declared in order to avoid access errors: +Rules may access +[configuration fragments](/rules/lib/fragments) such as +`cpp` and `java`. However, all required fragments must be declared in +order to avoid access errors: ```python def _impl(ctx): @@ -545,7 +862,14 @@ my_rule = rule( ### Runfiles symlinks -Normally, the relative path of a file in the runfiles tree is the same as the relative path of that file in the source tree or generated output tree. If these need to be different for some reason, you can specify the `root_symlinks` or `symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to files, where the paths are relative to the root of the runfiles directory. The `symlinks` dictionary is the same, but paths are implicitly prefixed with the name of the main workspace (*not* the name of the repository containing the current target). +Normally, the relative path of a file in the runfiles tree is the same as the +relative path of that file in the source tree or generated output tree. If these +need to be different for some reason, you can specify the `root_symlinks` or +`symlinks` arguments. The `root_symlinks` is a dictionary mapping paths to +files, where the paths are relative to the root of the runfiles directory. The +`symlinks` dictionary is the same, but paths are implicitly prefixed with the +name of the main workspace (*not* the name of the repository containing the +current target). ```python ... @@ -557,20 +881,37 @@ Normally, the relative path of a file in the runfiles tree is the same as the re # sometarget.runfiles/ # some/ # path/ - # here.foo -> some_data_file2 - # <workspace_name>/ + # here.foo -> some_data_file2 + # / # some/ # path/ - # here.bar -> some_data_file3 + # here.bar -> some_data_file3 ``` -If `symlinks` or `root_symlinks` is used, be careful not to map two different files to the same path in the runfiles tree. This will cause the build to fail with an error describing the conflict. To fix, you will need to modify your `ctx.runfiles` arguments to remove the collision. This checking will be done for any targets using your rule, as well as targets of any kind that depend on those targets. This is especially risky if your tool is likely to be used transitively by another tool; symlink names must be unique across the runfiles of a tool and all of its dependencies. +If `symlinks` or `root_symlinks` is used, be careful not to map two different +files to the same path in the runfiles tree. This will cause the build to fail +with an error describing the conflict. To fix, you will need to modify your +`ctx.runfiles` arguments to remove the collision. This checking will be done for +any targets using your rule, as well as targets of any kind that depend on those +targets. This is especially risky if your tool is likely to be used transitively +by another tool; symlink names must be unique across the runfiles of a tool and +all of its dependencies. ### Code coverage -When the [`coverage`](/reference/command-line-reference#coverage) command is run, the build may need to add coverage instrumentation for certain targets. The build also gathers the list of source files that are instrumented. The subset of targets that are considered is controlled by the flag [`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). Test targets are excluded, unless [`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) is specified. +When the [`coverage`](/reference/command-line-reference#coverage) command is run, +the build may need to add coverage instrumentation for certain targets. The +build also gathers the list of source files that are instrumented. The subset of +targets that are considered is controlled by the flag +[`--instrumentation_filter`](/reference/command-line-reference#flag--instrumentation_filter). +Test targets are excluded, unless +[`--instrument_test_targets`](/reference/command-line-reference#flag--instrument_test_targets) +is specified. -If a rule implementation adds coverage instrumentation at build time, it needs to account for that in its implementation function. [ctx.coverage\_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns `True` in coverage mode if a target's sources should be instrumented: +If a rule implementation adds coverage instrumentation at build time, it needs +to account for that in its implementation function. +[ctx.coverage_instrumented](/rules/lib/builtins/ctx#coverage_instrumented) returns +`True` in coverage mode if a target's sources should be instrumented: ```python # Are this rule's sources instrumented? @@ -578,9 +919,13 @@ if ctx.coverage_instrumented(): # Do something to turn on coverage for this compile action ``` -Logic that always needs to be on in coverage mode (whether a target's sources specifically are instrumented or not) can be conditioned on [ctx.configuration.coverage\_enabled](/rules/lib/builtins/configuration#coverage_enabled). +Logic that always needs to be on in coverage mode (whether a target's sources +specifically are instrumented or not) can be conditioned on +[ctx.configuration.coverage_enabled](/rules/lib/builtins/configuration#coverage_enabled). -If the rule directly includes sources from its dependencies before compilation (such as header files), it may also need to turn on compile-time instrumentation if the dependencies' sources should be instrumented: +If the rule directly includes sources from its dependencies before compilation +(such as header files), it may also need to turn on compile-time instrumentation if +the dependencies' sources should be instrumented: ```python # Are this rule's sources or any of the sources for its direct dependencies @@ -589,7 +934,13 @@ if ctx.coverage_instrumented() or any([ctx.coverage_instrumented(dep) for dep in # Do something to turn on coverage for this compile action ``` -Rules also should provide information about which attributes are relevant for coverage with the `InstrumentedFilesInfo` provider, constructed using [`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). The `dependency_attributes` parameter of `instrumented_files_info` should list all runtime dependency attributes, including code dependencies like `deps` and data dependencies like `data`. The `source_attributes` parameter should list the rule's source files attributes if coverage instrumentation might be added: +Rules also should provide information about which attributes are relevant for +coverage with the `InstrumentedFilesInfo` provider, constructed using +[`coverage_common.instrumented_files_info`](/rules/lib/toplevel/coverage_common#instrumented_files_info). +The `dependency_attributes` parameter of `instrumented_files_info` should list +all runtime dependency attributes, including code dependencies like `deps` and +data dependencies like `data`. The `source_attributes` parameter should list the +rule's source files attributes if coverage instrumentation might be added: ```python def _example_library_impl(ctx): @@ -606,11 +957,18 @@ def _example_library_impl(ctx): ] ``` -If `InstrumentedFilesInfo` is not returned, a default one is created with each non-tool [dependency attribute](#dependency_attributes) that doesn't set [`cfg`](#configuration) to `"exec"` in the attribute schema. in `dependency_attributes`. (This isn't ideal behavior, since it puts attributes like `srcs` in `dependency_attributes` instead of `source_attributes`, but it avoids the need for explicit coverage configuration for all rules in the dependency chain.) +If `InstrumentedFilesInfo` is not returned, a default one is created with each +non-tool [dependency attribute](#dependency_attributes) that doesn't set +[`cfg`](#configuration) to `"exec"` in the attribute schema. in +`dependency_attributes`. (This isn't ideal behavior, since it puts attributes +like `srcs` in `dependency_attributes` instead of `source_attributes`, but it +avoids the need for explicit coverage configuration for all rules in the +dependency chain.) #### Test rules -Test rules require additional setup to generate coverage reports. The rule itself has to add the following implicit attributes: +Test rules require additional setup to generate coverage reports. The rule +itself has to add the following implicit attributes: ```python my_test = rule( @@ -633,59 +991,115 @@ my_test = rule( ) ``` -By using `configuration_field`, the dependency on the Java LCOV merger tool can be avoided as long as coverage is not requested. +By using `configuration_field`, the dependency on the Java LCOV merger tool can +be avoided as long as coverage is not requested. -When the test is run, it should emit coverage information in the form of one or more \[LCOV files] ([https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT](https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT)) with unique names into the directory specified by the `COVERAGE_DIR` environment variable. Bazel will then merge these files into a single LCOV file using the `_lcov_merger` tool. If present, it will also collect C/C++ coverage using the `_collect_cc_coverage` tool. +When the test is run, it should emit coverage information in the form of one or +more [LCOV files] +(https://manpages.debian.org/unstable/lcov/geninfo.1.en.html#TRACEFILE_FORMAT) +with unique names into the directory specified by the `COVERAGE_DIR` environment +variable. Bazel will then merge these files into a single LCOV file using the +`_lcov_merger` tool. If present, it will also collect C/C++ coverage using the +`_collect_cc_coverage` tool. ### Baseline coverage -Since coverage is only collected for code that ends up in the dependency tree of a test, coverage reports can be misleading as they don't necessarily cover all the code matched by the `--instrumentation_filter` flag. +Since coverage is only collected for code that ends up in the dependency tree of +a test, coverage reports can be misleading as they don't necessarily cover all +the code matched by the `--instrumentation_filter` flag. -For this reason, Bazel allows rules to specify baseline coverage files using the `baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These files must be generated in LCOV format by a user-defined action and are supposed to list all lines, branches, functions and/or blocks in the target's source files (according to the `sources_attributes` and `extensions` parameters). For source files in targets that are instrumented for coverage, Bazel merges their baseline coverage into the combined coverage report generated with `--combined_report` and thus ensures that untested files still show up as uncovered. +For this reason, Bazel allows rules to specify baseline coverage files using the +`baseline_coverage_files` attribute of `ctx.instrumented_files_info`). These +files must be generated in LCOV format by a user-defined action and are supposed +to list all lines, branches, functions and/or blocks in the target's source +files (according to the `sources_attributes` and `extensions` parameters). For +source files in targets that are instrumented for coverage, Bazel merges their +baseline coverage into the combined coverage report generated with +`--combined_report` and thus ensures that untested files still show up as +uncovered. -If a rule doesn't provide any baseline coverage files, Bazel generates synthetic coverage information that only mentions the source file paths, but doesn't contain any information about their contents. +If a rule doesn't provide any baseline coverage files, Bazel generates synthetic +coverage information that only mentions the source file paths, but doesn't +contain any information about their contents. ### Validation Actions -Sometimes you need to validate something about the build, and the information required to do that validation is available only in artifacts (source files or generated files). Because this information is in artifacts, rules can't do this validation at analysis time because rules can't read files. Instead, actions must do this validation at execution time. When validation fails, the action will fail, and hence so will the build. - -Examples of validations that might be run are static analysis, linting, dependency and consistency checks, and style checks. - -Validation actions can also help to improve build performance by moving parts of actions that are not required for building artifacts into separate actions. For example, if a single action that does compilation and linting can be separated into a compilation action and a linting action, then the linting action can be run as a validation action and run in parallel with other actions. - -These "validation actions" often don't produce anything that is used elsewhere in the build, since they only need to assert things about their inputs. This presents a problem though: If a validation action doesn't produce anything that is used elsewhere in the build, how does a rule get the action to run? Historically, the approach was to have the validation action output an empty file, and artificially add that output to the inputs of some other important action in the build: - -![](/rules/validation_action_historical.svg) - -This works, because Bazel will always run the validation action when the compile action is run, but this has significant drawbacks: - -1. The validation action is in the critical path of the build. Because Bazel thinks the empty output is required to run the compile action, it will run the validation action first, even though the compile action will ignore the input. This reduces parallelism and slows down builds. - -2. If other actions in the build might run instead of the compile action, then the empty outputs of validation actions need to be added to those actions as well (`java_library`'s source jar output, for example). This is also a problem if new actions that might run instead of the compile action are added later, and the empty validation output is accidentally left off. +Sometimes you need to validate something about the build, and the +information required to do that validation is available only in artifacts +(source files or generated files). Because this information is in artifacts, +rules can't do this validation at analysis time because rules can't read +files. Instead, actions must do this validation at execution time. When +validation fails, the action will fail, and hence so will the build. + +Examples of validations that might be run are static analysis, linting, +dependency and consistency checks, and style checks. + +Validation actions can also help to improve build performance by moving parts +of actions that are not required for building artifacts into separate actions. +For example, if a single action that does compilation and linting can be +separated into a compilation action and a linting action, then the linting +action can be run as a validation action and run in parallel with other actions. + +These "validation actions" often don't produce anything that is used elsewhere +in the build, since they only need to assert things about their inputs. This +presents a problem though: If a validation action doesn't produce anything that +is used elsewhere in the build, how does a rule get the action to run? +Historically, the approach was to have the validation action output an empty +file, and artificially add that output to the inputs of some other important +action in the build: + + + +This works, because Bazel will always run the validation action when the compile +action is run, but this has significant drawbacks: + +1. The validation action is in the critical path of the build. Because Bazel +thinks the empty output is required to run the compile action, it will run the +validation action first, even though the compile action will ignore the input. +This reduces parallelism and slows down builds. + +2. If other actions in the build might run instead of the +compile action, then the empty outputs of validation actions need to be added to +those actions as well (`java_library`'s source jar output, for example). This is +also a problem if new actions that might run instead of the compile action are +added later, and the empty validation output is accidentally left off. The solution to these problems is to use the Validations Output Group. #### Validations Output Group -The Validations Output Group is an output group designed to hold the otherwise unused outputs of validation actions, so that they don't need to be artificially added to the inputs of other actions. +The Validations Output Group is an output group designed to hold the otherwise +unused outputs of validation actions, so that they don't need to be artificially +added to the inputs of other actions. -This group is special in that its outputs are always requested, regardless of the value of the `--output_groups` flag, and regardless of how the target is depended upon (for example, on the command line, as a dependency, or through implicit outputs of the target). Note that normal caching and incrementality still apply: if the inputs to the validation action have not changed and the validation action previously succeeded, then the validation action won't be run. +This group is special in that its outputs are always requested, regardless of +the value of the `--output_groups` flag, and regardless of how the target is +depended upon (for example, on the command line, as a dependency, or through +implicit outputs of the target). Note that normal caching and incrementality +still apply: if the inputs to the validation action have not changed and the +validation action previously succeeded, then the validation action won't be +run. -![](/rules/validation_action.svg) + -Using this output group still requires that validation actions output some file, even an empty one. This might require wrapping some tools that normally don't create outputs so that a file is created. +Using this output group still requires that validation actions output some file, +even an empty one. This might require wrapping some tools that normally don't +create outputs so that a file is created. A target's validation actions are not run in three cases: -- When the target is depended upon as a tool -- When the target is depended upon as an implicit dependency (for example, an attribute that starts with "\_") -- When the target is built in the exec configuration. +* When the target is depended upon as a tool +* When the target is depended upon as an implicit dependency (for example, an + attribute that starts with "_") +* When the target is built in the exec configuration. -It is assumed that these targets have their own separate builds and tests that would uncover any validation failures. +It is assumed that these targets have their own +separate builds and tests that would uncover any validation failures. #### Using the Validations Output Group -The Validations Output Group is named `_validation` and is used like any other output group: +The Validations Output Group is named `_validation` and is used like any other +output group: ```python def _rule_with_validation_impl(ctx): @@ -705,6 +1119,7 @@ def _rule_with_validation_impl(ctx): OutputGroupInfo(_validation = depset([validation_output])), ] + rule_with_validation = rule( implementation = _rule_with_validation_impl, outputs = { @@ -721,9 +1136,17 @@ rule_with_validation = rule( ) ``` -Notice that the validation output file is not added to the `DefaultInfo` or the inputs to any other action. The validation action for a target of this rule kind will still run if the target is depended upon by label, or any of the target's implicit outputs are directly or indirectly depended upon. +Notice that the validation output file is not added to the `DefaultInfo` or the +inputs to any other action. The validation action for a target of this rule kind +will still run if the target is depended upon by label, or any of the target's +implicit outputs are directly or indirectly depended upon. -It is usually important that the outputs of validation actions only go into the validation output group, and are not added to the inputs of other actions, as this could defeat parallelism gains. Note however that Bazel doesn't have any special checking to enforce this. Therefore, you should test that validation action outputs are not added to the inputs of any actions in the tests for Starlark rules. For example: +It is usually important that the outputs of validation actions only go into the +validation output group, and are not added to the inputs of other actions, as +this could defeat parallelism gains. Note however that Bazel doesn't +have any special checking to enforce this. Therefore, you should test +that validation action outputs are not added to the inputs of any actions in the +tests for Starlark rules. For example: ```python load("@bazel_skylib//lib:unittest.bzl", "analysistest") @@ -748,7 +1171,8 @@ validation_outputs_test = analysistest.make(_validation_outputs_test_impl) #### Validation Actions Flag -Running validation actions is controlled by the `--run_validations` command line flag, which defaults to true. +Running validation actions is controlled by the `--run_validations` command line +flag, which defaults to true. ## Deprecated features @@ -756,23 +1180,52 @@ Running validation actions is controlled by the `--run_validations` command line There are two **deprecated** ways of using predeclared outputs: -- The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies a mapping between output attribute names and string templates for generating predeclared output labels. Prefer using non-predeclared outputs and explicitly adding outputs to `DefaultInfo.files`. Use the rule target's label as input for rules which consume the output instead of a predeclared output's label. +* The [`outputs`](/rules/lib/globals/bzl#rule.outputs) parameter of `rule` specifies + a mapping between output attribute names and string templates for generating + predeclared output labels. Prefer using non-predeclared outputs and + explicitly adding outputs to `DefaultInfo.files`. Use the rule target's + label as input for rules which consume the output instead of a predeclared + output's label. -- For [executable rules](#executable-rules), `ctx.outputs.executable` refers to a predeclared executable output with the same name as the rule target. Prefer declaring the output explicitly, for example with `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that generates the executable sets its permissions to allow execution. Explicitly pass the executable output to the `executable` parameter of `DefaultInfo`. +* For [executable rules](#executable-rules), `ctx.outputs.executable` refers + to a predeclared executable output with the same name as the rule target. + Prefer declaring the output explicitly, for example with + `ctx.actions.declare_file(ctx.label.name)`, and ensure that the command that + generates the executable sets its permissions to allow execution. Explicitly + pass the executable output to the `executable` parameter of `DefaultInfo`. ### Runfiles features to avoid -[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) type have a complex set of features, many of which are kept for legacy reasons. The following recommendations help reduce complexity: - -- **Avoid** use of the `collect_data` and `collect_default` modes of [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect runfiles across certain hardcoded dependency edges in confusing ways. Instead, add files using the `files` or `transitive_files` parameters of `ctx.runfiles`, or by merging in runfiles from dependencies with `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. - -- **Avoid** use of the `data_runfiles` and `default_runfiles` of the `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. The distinction between "default" and "data" runfiles is maintained for legacy reasons. For example, some rules put their default outputs in `data_runfiles`, but not `default_runfiles`. Instead of using `data_runfiles`, rules should *both* include default outputs and merge in `default_runfiles` from attributes which provide runfiles (often [`data`](/reference/be/common-definitions#common-attributes.data)). - -- When retrieving `runfiles` from `DefaultInfo` (generally only for merging runfiles between the current rule and its dependencies), use `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. +[`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles) and the [`runfiles`](/rules/lib/builtins/runfiles) +type have a complex set of features, many of which are kept for legacy reasons. +The following recommendations help reduce complexity: + +* **Avoid** use of the `collect_data` and `collect_default` modes of + [`ctx.runfiles`](/rules/lib/builtins/ctx#runfiles). These modes implicitly collect + runfiles across certain hardcoded dependency edges in confusing ways. + Instead, add files using the `files` or `transitive_files` parameters of + `ctx.runfiles`, or by merging in runfiles from dependencies with + `runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)`. + +* **Avoid** use of the `data_runfiles` and `default_runfiles` of the + `DefaultInfo` constructor. Specify `DefaultInfo(runfiles = ...)` instead. + The distinction between "default" and "data" runfiles is maintained for + legacy reasons. For example, some rules put their default outputs in + `data_runfiles`, but not `default_runfiles`. Instead of using + `data_runfiles`, rules should *both* include default outputs and merge in + `default_runfiles` from attributes which provide runfiles (often + [`data`](/reference/be/common-definitions#common-attributes.data)). + +* When retrieving `runfiles` from `DefaultInfo` (generally only for merging + runfiles between the current rule and its dependencies), use + `DefaultInfo.default_runfiles`, **not** `DefaultInfo.data_runfiles`. ### Migrating from legacy providers -Historically, Bazel providers were simple fields on the `Target` object. They were accessed using the dot operator, and they were created by putting the field in a [`struct`](/rules/lib/builtins/struct) returned by the rule's implementation function instead of a list of provider objects: +Historically, Bazel providers were simple fields on the `Target` object. They +were accessed using the dot operator, and they were created by putting the field +in a [`struct`](/rules/lib/builtins/struct) returned by the rule's +implementation function instead of a list of provider objects: ```python return struct(example_info = struct(headers = depset(...))) @@ -784,9 +1237,13 @@ Such providers can be retrieved from the corresponding field of the `Target` obj transitive_headers = [hdr.example_info.headers for hdr in ctx.attr.hdrs] ``` -*This style is deprecated and should not be used in new code;* see following for information that may help you migrate. The new provider mechanism avoids name clashes. It also supports data hiding, by requiring any code accessing a provider instance to retrieve it using the provider symbol. +*This style is deprecated and should not be used in new code;* see following for +information that may help you migrate. The new provider mechanism avoids name +clashes. It also supports data hiding, by requiring any code accessing a +provider instance to retrieve it using the provider symbol. -For the moment, legacy providers are still supported. A rule can return both legacy and modern providers as follows: +For the moment, legacy providers are still supported. A rule can return both +legacy and modern providers as follows: ```python def _old_rule_impl(ctx): @@ -803,18 +1260,40 @@ def _old_rule_impl(ctx): providers = [modern_data, ...]) ``` -If `dep` is the resulting `Target` object for an instance of this rule, the providers and their contents can be retrieved as `dep.legacy_info.x` and `dep[MyInfo].y`. - -In addition to `providers`, the returned struct can also take several other fields that have special meaning (and thus don't create a corresponding legacy provider): - -- The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and `executable` correspond to the same-named fields of [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of these fields while also returning a `DefaultInfo` provider. - -- The field `output_groups` takes a struct value and corresponds to an [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). - -In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in [`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency attributes, legacy providers are passed in as strings and modern providers are passed in by their `Info` symbol. Be sure to change from strings to symbols when migrating. For complex or large rule sets where it is difficult to update all rules atomically, you may have an easier time if you follow this sequence of steps: - -1. Modify the rules that produce the legacy provider to produce both the legacy and modern providers, using the preceding syntax. For rules that declare they return the legacy provider, update that declaration to include both the legacy and modern providers. - -2. Modify the rules that consume the legacy provider to instead consume the modern provider. If any attribute declarations require the legacy provider, also update them to instead require the modern provider. Optionally, you can interleave this work with step 1 by having consumers accept or require either provider: Test for the presence of the legacy provider using `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. - -3. Fully remove the legacy provider from all rules. +If `dep` is the resulting `Target` object for an instance of this rule, the +providers and their contents can be retrieved as `dep.legacy_info.x` and +`dep[MyInfo].y`. + +In addition to `providers`, the returned struct can also take several other +fields that have special meaning (and thus don't create a corresponding legacy +provider): + +* The fields `files`, `runfiles`, `data_runfiles`, `default_runfiles`, and + `executable` correspond to the same-named fields of + [`DefaultInfo`](/rules/lib/providers/DefaultInfo). It is not allowed to specify any of + these fields while also returning a `DefaultInfo` provider. + +* The field `output_groups` takes a struct value and corresponds to an + [`OutputGroupInfo`](/rules/lib/providers/OutputGroupInfo). + +In [`provides`](/rules/lib/globals/bzl#rule.provides) declarations of rules, and in +[`providers`](/rules/lib/toplevel/attr#label_list.providers) declarations of dependency +attributes, legacy providers are passed in as strings and modern providers are +passed in by their `Info` symbol. Be sure to change from strings to symbols +when migrating. For complex or large rule sets where it is difficult to update +all rules atomically, you may have an easier time if you follow this sequence of +steps: + +1. Modify the rules that produce the legacy provider to produce both the legacy + and modern providers, using the preceding syntax. For rules that declare they + return the legacy provider, update that declaration to include both the + legacy and modern providers. + +2. Modify the rules that consume the legacy provider to instead consume the + modern provider. If any attribute declarations require the legacy provider, + also update them to instead require the modern provider. Optionally, you can + interleave this work with step 1 by having consumers accept or require either + provider: Test for the presence of the legacy provider using + `hasattr(target, 'foo')`, or the new provider using `FooInfo in target`. + +3. Fully remove the legacy provider from all rules. diff --git a/extending/toolchains.mdx b/extending/toolchains.mdx index ef07d948..0d92d6e1 100644 --- a/extending/toolchains.mdx +++ b/extending/toolchains.mdx @@ -2,11 +2,23 @@ title: 'Toolchains' --- -This page describes the toolchain framework, which is a way for rule authors to decouple their rule logic from platform-based selection of tools. It is recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) pages before continuing. This page covers why toolchains are needed, how to define and use them, and how Bazel selects an appropriate toolchain based on platform constraints. + + +This page describes the toolchain framework, which is a way for rule authors to +decouple their rule logic from platform-based selection of tools. It is +recommended to read the [rules](/extending/rules) and [platforms](/extending/platforms) +pages before continuing. This page covers why toolchains are needed, how to +define and use them, and how Bazel selects an appropriate toolchain based on +platform constraints. ## Motivation -Let's first look at the problem toolchains are designed to solve. Suppose you are writing rules to support the "bar" programming language. Your `bar_binary` rule would compile `*.bar` files using the `barc` compiler, a tool that itself is built as another target in your workspace. Since users who write `bar_binary` targets shouldn't have to specify a dependency on the compiler, you make it an implicit dependency by adding it to the rule definition as a private attribute. +Let's first look at the problem toolchains are designed to solve. Suppose you +are writing rules to support the "bar" programming language. Your `bar_binary` +rule would compile `*.bar` files using the `barc` compiler, a tool that itself +is built as another target in your workspace. Since users who write `bar_binary` +targets shouldn't have to specify a dependency on the compiler, you make it an +implicit dependency by adding it to the rule definition as a private attribute. ```python bar_binary = rule( @@ -22,7 +34,9 @@ bar_binary = rule( ) ``` -`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so it'll be built before any `bar_binary` target. It can be accessed by the rule's implementation function just like any other attribute: +`//bar_tools:barc_linux` is now a dependency of every `bar_binary` target, so +it'll be built before any `bar_binary` target. It can be accessed by the rule's +implementation function just like any other attribute: ```python BarcInfo = provider( @@ -44,9 +58,16 @@ def _bar_binary_impl(ctx): ... ``` -The issue here is that the compiler's label is hardcoded into `bar_binary`, yet different targets may need different compilers depending on what platform they are being built for and what platform they are being built on -- called the *target platform* and *execution platform*, respectively. Furthermore, the rule author does not necessarily even know all the available tools and platforms, so it is not feasible to hardcode them in the rule's definition. +The issue here is that the compiler's label is hardcoded into `bar_binary`, yet +different targets may need different compilers depending on what platform they +are being built for and what platform they are being built on -- called the +*target platform* and *execution platform*, respectively. Furthermore, the rule +author does not necessarily even know all the available tools and platforms, so +it is not feasible to hardcode them in the rule's definition. -A less-than-ideal solution would be to shift the burden onto users, by making the `_compiler` attribute non-private. Then individual targets could be hardcoded to build for one platform or another. +A less-than-ideal solution would be to shift the burden onto users, by making +the `_compiler` attribute non-private. Then individual targets could be +hardcoded to build for one platform or another. ```python bar_binary( @@ -62,7 +83,8 @@ bar_binary( ) ``` -You can improve on this solution by using `select` to choose the `compiler` [based on the platform](/docs/configurable-attributes): +You can improve on this solution by using `select` to choose the `compiler` +[based on the platform](/docs/configurable-attributes): ```python config_setting( @@ -89,13 +111,26 @@ bar_binary( ) ``` -But this is tedious and a bit much to ask of every single `bar_binary` user. If this style is not used consistently throughout the workspace, it leads to builds that work fine on a single platform but fail when extended to multi-platform scenarios. It also does not address the problem of adding support for new platforms and compilers without modifying existing rules or targets. +But this is tedious and a bit much to ask of every single `bar_binary` user. +If this style is not used consistently throughout the workspace, it leads to +builds that work fine on a single platform but fail when extended to +multi-platform scenarios. It also does not address the problem of adding support +for new platforms and compilers without modifying existing rules or targets. -The toolchain framework solves this problem by adding an extra level of indirection. Essentially, you declare that your rule has an abstract dependency on *some* member of a family of targets (a toolchain type), and Bazel automatically resolves this to a particular target (a toolchain) based on the applicable platform constraints. Neither the rule author nor the target author need know the complete set of available platforms and toolchains. +The toolchain framework solves this problem by adding an extra level of +indirection. Essentially, you declare that your rule has an abstract dependency +on *some* member of a family of targets (a toolchain type), and Bazel +automatically resolves this to a particular target (a toolchain) based on the +applicable platform constraints. Neither the rule author nor the target author +need know the complete set of available platforms and toolchains. ## Writing rules that use toolchains -Under the toolchain framework, instead of having rules depend directly on tools, they instead depend on *toolchain types*. A toolchain type is a simple target that represents a class of tools that serve the same role for different platforms. For instance, you can declare a type that represents the bar compiler: +Under the toolchain framework, instead of having rules depend directly on tools, +they instead depend on *toolchain types*. A toolchain type is a simple target +that represents a class of tools that serve the same role for different +platforms. For instance, you can declare a type that represents the bar +compiler: ```python # By convention, toolchain_type targets are named "toolchain_type" and @@ -104,7 +139,9 @@ Under the toolchain framework, instead of having rules depend directly on tools, toolchain_type(name = "toolchain_type") ``` -The rule definition in the previous section is modified so that instead of taking in the compiler as an attribute, it declares that it consumes a `//bar_tools:toolchain_type` toolchain. +The rule definition in the previous section is modified so that instead of +taking in the compiler as an attribute, it declares that it consumes a +`//bar_tools:toolchain_type` toolchain. ```python bar_binary = rule( @@ -118,7 +155,8 @@ bar_binary = rule( ) ``` -The implementation function now accesses this dependency under `ctx.toolchains` instead of `ctx.attr`, using the toolchain type as the key. +The implementation function now accesses this dependency under `ctx.toolchains` +instead of `ctx.attr`, using the toolchain type as the key. ```python def _bar_binary_impl(ctx): @@ -133,15 +171,28 @@ def _bar_binary_impl(ctx): ... ``` -`ctx.toolchains["//bar_tools:toolchain_type"]` returns the [`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) of whatever target Bazel resolved the toolchain dependency to. The fields of the `ToolchainInfo` object are set by the underlying tool's rule; in the next section, this rule is defined such that there is a `barcinfo` field that wraps a `BarcInfo` object. +`ctx.toolchains["//bar_tools:toolchain_type"]` returns the +[`ToolchainInfo` provider](/rules/lib/toplevel/platform_common#ToolchainInfo) +of whatever target Bazel resolved the toolchain dependency to. The fields of the +`ToolchainInfo` object are set by the underlying tool's rule; in the next +section, this rule is defined such that there is a `barcinfo` field that wraps +a `BarcInfo` object. -Bazel's procedure for resolving toolchains to targets is described [below](#toolchain-resolution). Only the resolved toolchain target is actually made a dependency of the `bar_binary` target, not the whole space of candidate toolchains. +Bazel's procedure for resolving toolchains to targets is described +[below](#toolchain-resolution). Only the resolved toolchain target is actually +made a dependency of the `bar_binary` target, not the whole space of candidate +toolchains. ### Mandatory and Optional Toolchains -By default, when a rule expresses a toolchain type dependency using a bare label (as shown above), the toolchain type is considered to be **mandatory**. If Bazel is unable to find a matching toolchain (see [Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain type, this is an error and analysis halts. +By default, when a rule expresses a toolchain type dependency using a bare label +(as shown above), the toolchain type is considered to be **mandatory**. If Bazel +is unable to find a matching toolchain (see +[Toolchain resolution](#toolchain-resolution) below) for a mandatory toolchain +type, this is an error and analysis halts. -It is possible instead to declare an **optional** toolchain type dependency, as follows: +It is possible instead to declare an **optional** toolchain type dependency, as +follows: ```python bar_binary = rule( @@ -152,20 +203,20 @@ bar_binary = rule( ) ``` -When an optional toolchain type cannot be resolved, analysis continues, and the result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. +When an optional toolchain type cannot be resolved, analysis continues, and the +result of `ctx.toolchains["//bar_tools:toolchain_type"]` is `None`. -The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) function defaults to mandatory. +The [`config_common.toolchain_type`](/rules/lib/toplevel/config_common#toolchain_type) +function defaults to mandatory. The following forms can be used: -- Mandatory toolchain types: - - - `toolchains = ["//bar_tools:toolchain_type"]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` - +- Mandatory toolchain types: + - `toolchains = ["//bar_tools:toolchain_type"]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type")]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = True)]` - Optional toolchain types: - - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` + - `toolchains = [config_common.toolchain_type("//bar_tools:toolchain_type", mandatory = False)]` ```python bar_binary = rule( @@ -177,11 +228,15 @@ bar_binary = rule( ) ``` -You can mix and match forms in the same rule, also. However, if the same toolchain type is listed multiple times, it will take the most strict version, where mandatory is more strict than optional. +You can mix and match forms in the same rule, also. However, if the same +toolchain type is listed multiple times, it will take the most strict version, +where mandatory is more strict than optional. ### Writing aspects that use toolchains -Aspects have access to the same toolchain API as rules: you can define required toolchain types, access toolchains via the context, and use them to generate new actions using the toolchain. +Aspects have access to the same toolchain API as rules: you can define required +toolchain types, access toolchains via the context, and use them to generate new +actions using the toolchain. ```py bar_aspect = aspect( @@ -200,15 +255,28 @@ def _bar_aspect_impl(target, ctx): To define some toolchains for a given toolchain type, you need three things: -1. A language-specific rule representing the kind of tool or tool suite. By convention this rule's name is suffixed with "\_toolchain". +1. A language-specific rule representing the kind of tool or tool suite. By + convention this rule's name is suffixed with "\_toolchain". - 1. **Note:** The `\_toolchain` rule cannot create any build actions. Rather, it collects artifacts from other rules and forwards them to the rule that uses the toolchain. That rule is responsible for creating all build actions. + 1. **Note:** The `\_toolchain` rule cannot create any build actions. + Rather, it collects artifacts from other rules and forwards them to the + rule that uses the toolchain. That rule is responsible for creating all + build actions. -2. Several targets of this rule type, representing versions of the tool or tool suite for different platforms. +2. Several targets of this rule type, representing versions of the tool or tool + suite for different platforms. -3. For each such target, an associated target of the generic [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) rule, to provide metadata used by the toolchain framework. This `toolchain` target also refers to the `toolchain_type` associated with this toolchain. This means that a given `_toolchain` rule could be associated with any `toolchain_type`, and that only in a `toolchain` instance that uses this `_toolchain` rule that the rule is associated with a `toolchain_type`. +3. For each such target, an associated target of the generic + [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) + rule, to provide metadata used by the toolchain framework. This `toolchain` + target also refers to the `toolchain_type` associated with this toolchain. + This means that a given `_toolchain` rule could be associated with any + `toolchain_type`, and that only in a `toolchain` instance that uses + this `_toolchain` rule that the rule is associated with a `toolchain_type`. -For our running example, here's a definition for a `bar_toolchain` rule. Our example has only a compiler, but other tools such as a linker could also be grouped underneath it. +For our running example, here's a definition for a `bar_toolchain` rule. Our +example has only a compiler, but other tools such as a linker could also be +grouped underneath it. ```python def _bar_toolchain_impl(ctx): @@ -231,7 +299,13 @@ bar_toolchain = rule( ) ``` -The rule must return a `ToolchainInfo` provider, which becomes the object that the consuming rule retrieves using `ctx.toolchains` and the label of the toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value pairs. The specification of exactly what fields are added to the `ToolchainInfo` should be clearly documented at the toolchain type. In this example, the values return wrapped in a `BarcInfo` object to reuse the schema defined above; this style may be useful for validation and code reuse. +The rule must return a `ToolchainInfo` provider, which becomes the object that +the consuming rule retrieves using `ctx.toolchains` and the label of the +toolchain type. `ToolchainInfo`, like `struct`, can hold arbitrary field-value +pairs. The specification of exactly what fields are added to the `ToolchainInfo` +should be clearly documented at the toolchain type. In this example, the values +return wrapped in a `BarcInfo` object to reuse the schema defined above; this +style may be useful for validation and code reuse. Now you can define targets for specific `barc` compilers. @@ -257,7 +331,10 @@ bar_toolchain( ) ``` -Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. These definitions link the language-specific targets to the toolchain type and provide the constraint information that tells Bazel when the toolchain is appropriate for a given platform. +Finally, you create `toolchain` definitions for the two `bar_toolchain` targets. +These definitions link the language-specific targets to the toolchain type and +provide the constraint information that tells Bazel when the toolchain is +appropriate for a given platform. ```python toolchain( @@ -289,13 +366,21 @@ toolchain( ) ``` -The use of relative path syntax above suggests these definitions are all in the same package, but there's no reason the toolchain type, language-specific toolchain targets, and `toolchain` definition targets can't all be in separate packages. +The use of relative path syntax above suggests these definitions are all in the +same package, but there's no reason the toolchain type, language-specific +toolchain targets, and `toolchain` definition targets can't all be in separate +packages. -See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) for a real-world example. +See the [`go_toolchain`](https://github.com/bazelbuild/rules_go/blob/master/go/private/go_toolchain.bzl) +for a real-world example. ### Toolchains and configurations -An important question for rule authors is, when a `bar_toolchain` target is analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions should be used for dependencies? The example above uses string attributes, but what would happen for a more complicated toolchain that depends on other targets in the Bazel repository? +An important question for rule authors is, when a `bar_toolchain` target is +analyzed, what [configuration](/reference/glossary#configuration) does it see, and what transitions +should be used for dependencies? The example above uses string attributes, but +what would happen for a more complicated toolchain that depends on other targets +in the Bazel repository? Let's see a more complex version of `bar_toolchain`: @@ -321,13 +406,32 @@ bar_toolchain = rule( ) ``` -The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, but the meaning of the `cfg` parameter is slightly different. - -The dependency from a target (called the "parent") to a toolchain via toolchain resolution uses a special configuration transition called the "toolchain transition". The toolchain transition keeps the configuration the same, except that it forces the execution platform to be the same for the toolchain as for the parent (otherwise, toolchain resolution for the toolchain could pick any execution platform, and wouldn't necessarily be the same as for parent). This allows any `exec` dependencies of the toolchain to also be executable for the parent's build actions. Any of the toolchain's dependencies which use `cfg = "target"` (or which don't specify `cfg`, since "target" is the default) are built for the same target platform as the parent. This allows toolchain rules to contribute both libraries (the `system_lib` attribute above) and tools (the `compiler` attribute) to the build rules which need them. The system libraries are linked into the final artifact, and so need to be built for the same platform, whereas the compiler is a tool invoked during the build, and needs to be able to run on the execution platform. +The use of [`attr.label`](/rules/lib/toplevel/attr#label) is the same as for a standard rule, +but the meaning of the `cfg` parameter is slightly different. + +The dependency from a target (called the "parent") to a toolchain via toolchain +resolution uses a special configuration transition called the "toolchain +transition". The toolchain transition keeps the configuration the same, except +that it forces the execution platform to be the same for the toolchain as for +the parent (otherwise, toolchain resolution for the toolchain could pick any +execution platform, and wouldn't necessarily be the same as for parent). This +allows any `exec` dependencies of the toolchain to also be executable for the +parent's build actions. Any of the toolchain's dependencies which use `cfg = +"target"` (or which don't specify `cfg`, since "target" is the default) are +built for the same target platform as the parent. This allows toolchain rules to +contribute both libraries (the `system_lib` attribute above) and tools (the +`compiler` attribute) to the build rules which need them. The system libraries +are linked into the final artifact, and so need to be built for the same +platform, whereas the compiler is a tool invoked during the build, and needs to +be able to run on the execution platform. ## Registering and building with toolchains -At this point all the building blocks are assembled, and you just need to make the toolchains available to Bazel's resolution procedure. This is done by registering the toolchain, either in a `MODULE.bazel` file using `register_toolchains()`, or by passing the toolchains' labels on the command line using the `--extra_toolchains` flag. +At this point all the building blocks are assembled, and you just need to make +the toolchains available to Bazel's resolution procedure. This is done by +registering the toolchain, either in a `MODULE.bazel` file using +`register_toolchains()`, or by passing the toolchains' labels on the command +line using the `--extra_toolchains` flag. ```python register_toolchains( @@ -340,12 +444,16 @@ register_toolchains( ) ``` -When using target patterns to register toolchains, the order in which the individual toolchains are registered is determined by the following rules: +When using target patterns to register toolchains, the order in which the +individual toolchains are registered is determined by the following rules: -- The toolchains defined in a subpackage of a package are registered before the toolchains defined in the package itself. -- Within a package, toolchains are registered in the lexicographical order of their names. +* The toolchains defined in a subpackage of a package are registered before the + toolchains defined in the package itself. +* Within a package, toolchains are registered in the lexicographical order of + their names. -Now when you build a target that depends on a toolchain type, an appropriate toolchain will be selected based on the target and execution platforms. +Now when you build a target that depends on a toolchain type, an appropriate +toolchain will be selected based on the target and execution platforms. ```python # my_pkg/BUILD @@ -367,52 +475,109 @@ bar_binary( bazel build //my_pkg:my_bar_binary --platforms=//my_pkg:my_target_platform ``` -Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that has `@platforms//os:linux` and therefore resolve the `//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. This will end up building `//bar_tools:barc_linux` but not `//bar_tools:barc_windows`. +Bazel will see that `//my_pkg:my_bar_binary` is being built with a platform that +has `@platforms//os:linux` and therefore resolve the +`//bar_tools:toolchain_type` reference to `//bar_tools:barc_linux_toolchain`. +This will end up building `//bar_tools:barc_linux` but not +`//bar_tools:barc_windows`. ## Toolchain resolution -Note: [Some Bazel rules](/concepts/platforms#status) do not yet support toolchain resolution. - -For each target that uses toolchains, Bazel's toolchain resolution procedure determines the target's concrete toolchain dependencies. The procedure takes as input a set of required toolchain types, the target platform, the list of available execution platforms, and the list of available toolchains. Its outputs are a selected toolchain for each toolchain type as well as a selected execution platform for the current target. - -The available execution platforms and toolchains are gathered from the external dependency graph via [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) and [`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in `MODULE.bazel` files. Additional execution platforms and toolchains may also be specified on the command line via [`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) and [`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). The host platform is automatically included as an available execution platform. Available platforms and toolchains are tracked as ordered lists for determinism, with preference given to earlier items in the list. - -The set of available toolchains, in priority order, is created from `--extra_toolchains` and `register_toolchains`: - -1. Toolchains registered using `--extra_toolchains` are added first. (Within these, the **last** toolchain has highest priority.) -2. Toolchains registered using `register_toolchains` in the transitive external dependency graph, in the following order: (Within these, the **first** mentioned toolchain has highest priority.) -3. Toolchains registered by the root module (as in, the `MODULE.bazel` at the workspace root); -4. Toolchains registered in the user's `WORKSPACE` file, including in any macros invoked from there; -5. Toolchains registered by non-root modules (as in, dependencies specified by the root module, and their dependencies, and so forth); -6. Toolchains registered in the "WORKSPACE suffix"; this is only used by certain native rules bundled with the Bazel installation. - -**NOTE:** [Pseudo-targets like `:all`, `:*`, and `/...`](/run/build#specifying-build-targets) are ordered by Bazel's package loading mechanism, which uses a lexicographic ordering. +Note: [Some Bazel rules](/concepts/platforms#status) do not yet support +toolchain resolution. + +For each target that uses toolchains, Bazel's toolchain resolution procedure +determines the target's concrete toolchain dependencies. The procedure takes as +input a set of required toolchain types, the target platform, the list of +available execution platforms, and the list of available toolchains. Its outputs +are a selected toolchain for each toolchain type as well as a selected execution +platform for the current target. + +The available execution platforms and toolchains are gathered from the +external dependency graph via +[`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) +and +[`register_toolchains`](/rules/lib/globals/module#register_toolchains) calls in +`MODULE.bazel` files. +Additional execution platforms and toolchains may also be specified on the +command line via +[`--extra_execution_platforms`](/reference/command-line-reference#flag--extra_execution_platforms) +and +[`--extra_toolchains`](/reference/command-line-reference#flag--extra_toolchains). +The host platform is automatically included as an available execution platform. +Available platforms and toolchains are tracked as ordered lists for determinism, +with preference given to earlier items in the list. + +The set of available toolchains, in priority order, is created from +`--extra_toolchains` and `register_toolchains`: + +1. Toolchains registered using `--extra_toolchains` are added first. (Within + these, the **last** toolchain has highest priority.) +2. Toolchains registered using `register_toolchains` in the transitive external + dependency graph, in the following order: (Within these, the **first** + mentioned toolchain has highest priority.) + 1. Toolchains registered by the root module (as in, the `MODULE.bazel` at the + workspace root); + 2. Toolchains registered in the user's `WORKSPACE` file, including in any + macros invoked from there; + 3. Toolchains registered by non-root modules (as in, dependencies specified by + the root module, and their dependencies, and so forth); + 4. Toolchains registered in the "WORKSPACE suffix"; this is only used by + certain native rules bundled with the Bazel installation. + +**NOTE:** [Pseudo-targets like `:all`, `:*`, and +`/...`](/run/build#specifying-build-targets) are ordered by Bazel's package +loading mechanism, which uses a lexicographic ordering. The resolution steps are as follows. -1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a platform if, for each `constraint_value` in its list, the platform also has that `constraint_value` (either explicitly or as a default). +1. A `target_compatible_with` or `exec_compatible_with` clause *matches* a + platform if, for each `constraint_value` in its list, the platform also has + that `constraint_value` (either explicitly or as a default). - If the platform has `constraint_value`s from `constraint_setting`s not referenced by the clause, these do not affect matching. + If the platform has `constraint_value`s from `constraint_setting`s not + referenced by the clause, these do not affect matching. -2. If the target being built specifies the [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) (or its rule definition specifies the [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), the list of available execution platforms is filtered to remove any that do not match the execution constraints. +1. If the target being built specifies the + [`exec_compatible_with` attribute](/reference/be/common-definitions#common.exec_compatible_with) + (or its rule definition specifies the + [`exec_compatible_with` argument](/rules/lib/globals/bzl#rule.exec_compatible_with)), + the list of available execution platforms is filtered to remove + any that do not match the execution constraints. -3. The list of available toolchains is filtered to remove any toolchains specifying `target_settings` that don't match the current configuration. +1. The list of available toolchains is filtered to remove any toolchains + specifying `target_settings` that don't match the current configuration. -4. For each available execution platform, you associate each toolchain type with the first available toolchain, if any, that is compatible with this execution platform and the target platform. +1. For each available execution platform, you associate each toolchain type with + the first available toolchain, if any, that is compatible with this execution + platform and the target platform. -5. Any execution platform that failed to find a compatible mandatory toolchain for one of its toolchain types is ruled out. Of the remaining platforms, the first one becomes the current target's execution platform, and its associated toolchains (if any) become dependencies of the target. +1. Any execution platform that failed to find a compatible mandatory toolchain + for one of its toolchain types is ruled out. Of the remaining platforms, the + first one becomes the current target's execution platform, and its associated + toolchains (if any) become dependencies of the target. -The chosen execution platform is used to run all actions that the target generates. +The chosen execution platform is used to run all actions that the target +generates. -In cases where the same target can be built in multiple configurations (such as for different CPUs) within the same build, the resolution procedure is applied independently to each version of the target. +In cases where the same target can be built in multiple configurations (such as +for different CPUs) within the same build, the resolution procedure is applied +independently to each version of the target. -If the rule uses [execution groups](/extending/exec-groups), each execution group performs toolchain resolution separately, and each has its own execution platform and toolchains. +If the rule uses [execution groups](/extending/exec-groups), each execution +group performs toolchain resolution separately, and each has its own execution +platform and toolchains. ## Debugging toolchains -If you are adding toolchain support to an existing rule, use the `--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag provides verbose output for toolchain types or target names that match the regex variable. You can use `.*` to output all information. Bazel will output names of toolchains it checks and skips during the resolution process. +If you are adding toolchain support to an existing rule, use the +`--toolchain_resolution_debug=regex` flag. During toolchain resolution, the flag +provides verbose output for toolchain types or target names that match the regex variable. You +can use `.*` to output all information. Bazel will output names of toolchains it +checks and skips during the resolution process. -For example, to debug toolchain selection for all actions created directly by `//my:target`: +For example, to debug toolchain selection for all actions created directly by +`//my:target`: ```sh $ bazel build //my:all --toolchain_resolution_debug=//my:target @@ -424,7 +589,8 @@ To debug toolchain selection for all actions over all build targets: $ bazel build //my:all --toolchain_resolution_debug=.* ``` -If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: +If you'd like to see which [`cquery`](/query/cquery) dependencies are from toolchain +resolution, use `cquery`'s [`--transitions`](/query/cquery#transitions) flag: ``` # Find all direct dependencies of //cc:my_cc_lib. This includes explicitly @@ -443,5 +609,5 @@ $ bazel cquery 'deps(//cc:my_cc_lib, 1)' # Which of these are from toolchain resolution? $ bazel cquery 'deps(//cc:my_cc_lib, 1)' --transitions=lite | grep "toolchain dependency" - [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 + [toolchain dependency]#@local_config_cc//:cc-compiler-k8#HostTransition -> b6df211 ``` diff --git a/external/extension.mdx b/external/extension.mdx index 679096e2..ababc7fe 100644 --- a/external/extension.mdx +++ b/external/extension.mdx @@ -2,20 +2,42 @@ title: 'Module extensions' --- -Module extensions allow users to extend the module system by reading input data from modules across the dependency graph, performing necessary logic to resolve dependencies, and finally creating repos by calling [repo rules](/external/repo). These extensions have capabilities similar to repo rules, which enables them to perform file I/O, send network requests, and so on. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. -You can define module extensions in `.bzl` files, just like repo rules. They're not invoked directly; rather, each module specifies pieces of data called *tags* for extensions to read. Bazel runs module resolution before evaluating any extensions. The extension reads all the tags belonging to it across the entire dependency graph. + +Module extensions allow users to extend the module system by reading input data +from modules across the dependency graph, performing necessary logic to resolve +dependencies, and finally creating repos by calling [repo +rules](/external/repo). These extensions have capabilities similar to repo +rules, which enables them to perform file I/O, send network requests, and so on. +Among other things, they allow Bazel to interact with other package management +systems while also respecting the dependency graph built out of Bazel modules. + +You can define module extensions in `.bzl` files, just like repo rules. They're +not invoked directly; rather, each module specifies pieces of data called *tags* +for extensions to read. Bazel runs module resolution before evaluating any +extensions. The extension reads all the tags belonging to it across the entire +dependency graph. ## Extension usage -Extensions are hosted in Bazel modules themselves. To use an extension in a module, first add a `bazel_dep` on the module hosting the extension, and then call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function to bring it into scope. Consider the following example — a snippet from a `MODULE.bazel` file to use the "maven" extension defined in the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) module: +Extensions are hosted in Bazel modules themselves. To use an extension in a +module, first add a `bazel_dep` on the module hosting the extension, and then +call the [`use_extension`](/rules/lib/globals/module#use_extension) built-in function +to bring it into scope. Consider the following example — a snippet from a +`MODULE.bazel` file to use the "maven" extension defined in the +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) +module: ```python bazel_dep(name = "rules_jvm_external", version = "4.5") maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -This binds the return value of `use_extension` to a variable, which allows the user to use dot-syntax to specify tags for the extension. The tags must follow the schema defined by the corresponding *tag classes* specified in the [extension definition](#extension_definition). For an example specifying some `maven.install` and `maven.artifact` tags: +This binds the return value of `use_extension` to a variable, which allows the +user to use dot-syntax to specify tags for the extension. The tags must follow +the schema defined by the corresponding *tag classes* specified in the +[extension definition](#extension_definition). For an example specifying some +`maven.install` and `maven.artifact` tags: ```python maven.install(artifacts = ["org.junit:junit:4.13.2"]) @@ -25,19 +47,33 @@ maven.artifact(group = "com.google.guava", exclusions = ["com.google.j2objc:j2objc-annotations"]) ``` -Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos generated by the extension into the scope of the current module. +Use the [`use_repo`](/rules/lib/globals/module#use_repo) directive to bring repos +generated by the extension into the scope of the current module. ```python use_repo(maven, "maven") ``` -Repos generated by an extension are part of its API. In this example, the "maven" module extension promises to generate a repo called `maven`. With the declaration above, the extension properly resolves labels such as `@maven//:org_junit_junit` to point to the repo generated by the "maven" extension. +Repos generated by an extension are part of its API. In this example, the +"maven" module extension promises to generate a repo called `maven`. With the +declaration above, the extension properly resolves labels such as +`@maven//:org_junit_junit` to point to the repo generated by the "maven" +extension. -Note: Module extensions are evaluated lazily. This means that an extension will typically not be evaluated unless some module brings one of its repositories into scope using `use_repo` and that repository is referenced in a build. While testing a module extension, `bazel mod deps` can be useful as it unconditionally evaluates all module extensions. +Note: Module extensions are evaluated lazily. This means that an extension will +typically not be evaluated unless some module brings one of its repositories +into scope using `use_repo` and that repository is referenced in a build. While +testing a module extension, `bazel mod deps` can be useful as it +unconditionally evaluates all module extensions. ## Extension definition -You can define module extensions similarly to [repo rules](/external/repo), using the [`module_extension`](/rules/lib/globals/bzl#module_extension) function. However, while repo rules have a number of attributes, module extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which has a number of attributes. The tag classes define schemas for tags used by this extension. For example, the "maven" extension above might be defined like this: +You can define module extensions similarly to [repo rules](/external/repo), +using the [`module_extension`](/rules/lib/globals/bzl#module_extension) +function. However, while repo rules have a number of attributes, module +extensions have [`tag_class`es](/rules/lib/globals/bzl#tag_class), each of which +has a number of attributes. The tag classes define schemas for tags used by this +extension. For example, the "maven" extension above might be defined like this: ```python # @rules_jvm_external//:extensions.bzl @@ -50,9 +86,13 @@ maven = module_extension( ) ``` -These declarations show that `maven.install` and `maven.artifact` tags can be specified using the specified attribute schema. +These declarations show that `maven.install` and `maven.artifact` tags can be +specified using the specified attribute schema. -The implementation function of module extensions are similar to those of repo rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, which grants access to all modules using the extension and all pertinent tags. The implementation function then calls repo rules to generate repos. +The implementation function of module extensions are similar to those of repo +rules, except that they get a [`module_ctx`](/rules/lib/builtins/module_ctx) object, +which grants access to all modules using the extension and all pertinent tags. +The implementation function then calls repo rules to generate repos. ```python # @rules_jvm_external//:extensions.bzl @@ -80,33 +120,65 @@ def _maven_impl(ctx): ### Extension identity -Module extensions are identified by the name and the `.bzl` file that appears in the call to `use_extension`. In the following example, the extension `maven` is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the name `maven`: +Module extensions are identified by the name and the `.bzl` file that appears +in the call to `use_extension`. In the following example, the extension `maven` +is identified by the `.bzl` file `@rules_jvm_external//:extension.bzl` and the +name `maven`: ```python maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") ``` -Re-exporting an extension from a different `.bzl` file gives it a new identity and if both versions of the extension are used in the transitive module graph, then they will be evaluated separately and will only see the tags associated with that particular identity. +Re-exporting an extension from a different `.bzl` file gives it a new identity +and if both versions of the extension are used in the transitive module graph, +then they will be evaluated separately and will only see the tags associated +with that particular identity. -As an extension author you should make sure that users will only use your module extension from one single `.bzl` file. +As an extension author you should make sure that users will only use your +module extension from one single `.bzl` file. ## Repository names and visibility -Repos generated by extensions have canonical names in the form of `<var>module_repo_canonical_name</var>+<var>extension_name</var>+<var>repo_name</var>`. Note that the canonical name format is not an API you should depend on — it's subject to change at any time. - -This naming policy means that each extension has its own "repo namespace"; two distinct extensions can each define a repo with the same name without risking any clashes. It also means that `repository_ctx.name` reports the canonical name of the repo, which is *not* the same as the name specified in the repo rule call. - -Taking repos generated by module extensions into consideration, there are several repo visibility rules: - -- A Bazel module repo can see all repos introduced in its `MODULE.bazel` file via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_repo`](/rules/lib/globals/module#use_repo). -- A repo generated by a module extension can see all repos visible to the module that hosts the extension, *plus* all other repos generated by the same module extension (using the names specified in the repo rule calls as their apparent names). - - This might result in a conflict. If the module repo can see a repo with the apparent name `foo`, and the extension generates a repo with the specified name `foo`, then for all repos generated by that extension `foo` refers to the former. -- Similarly, in a module extension's implementation function, repos created by the extension can refer to each other by their apparent names in attributes, regardless of the order in which they are created. - - In case of a conflict with a repository visible to the module, labels passed to repository rule attributes can be wrapped in a call to [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to the repo visible to the module instead of the extension-generated repo of the same name. +Repos generated by extensions have canonical names in the form of `{{ "" +}}module_repo_canonical_name+extension_name{{ +"" }}+repo_name`. Note that the canonical name +format is not an API you should depend on — it's subject to change at any time. + +This naming policy means that each extension has its own "repo namespace"; two +distinct extensions can each define a repo with the same name without risking +any clashes. It also means that `repository_ctx.name` reports the canonical name +of the repo, which is *not* the same as the name specified in the repo rule +call. + +Taking repos generated by module extensions into consideration, there are +several repo visibility rules: + +* A Bazel module repo can see all repos introduced in its `MODULE.bazel` file + via [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and + [`use_repo`](/rules/lib/globals/module#use_repo). +* A repo generated by a module extension can see all repos visible to the + module that hosts the extension, *plus* all other repos generated by the + same module extension (using the names specified in the repo rule calls as + their apparent names). + * This might result in a conflict. If the module repo can see a repo with + the apparent name `foo`, and the extension generates a repo with the + specified name `foo`, then for all repos generated by that extension + `foo` refers to the former. +* Similarly, in a module extension's implementation function, repos created + by the extension can refer to each other by their apparent names in + attributes, regardless of the order in which they are created. + * In case of a conflict with a repository visible to the module, labels + passed to repository rule attributes can be wrapped in a call to + [`Label`](/rules/lib/toplevel/attr#label) to ensure that they refer to + the repo visible to the module instead of the extension-generated repo + of the same name. ### Overriding and injecting module extension repos -The root module can use [`override_repo`](/rules/lib/globals/module#override_repo) and [`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject module extension repos. +The root module can use +[`override_repo`](/rules/lib/globals/module#override_repo) and +[`inject_repo`](/rules/lib/globals/module#inject_repo) to override or inject +module extension repos. #### Example: Replacing `rules_java`'s `java_tools` with a vendored copy @@ -160,32 +232,80 @@ inject_repo(go_deps, "zlib") ## Best practices -This section describes best practices when writing extensions so they are straightforward to use, maintainable, and adapt well to changes over time. +This section describes best practices when writing extensions so they are +straightforward to use, maintainable, and adapt well to changes over time. ### Put each extension in a separate file -When extensions are in a different files, it allows one extension to load repositories generated by another extension. Even if you don't use this functionality, it's best to put them in separate files in case you need it later. This is because the extension's identify is based on its file, so moving the extension into another file later changes your public API and is a backwards incompatible change for your users. +When extensions are in a different files, it allows one extension to load +repositories generated by another extension. Even if you don't use this +functionality, it's best to put them in separate files in case you need it +later. This is because the extension's identify is based on its file, so moving +the extension into another file later changes your public API and is a backwards +incompatible change for your users. ### Specify reproducibility and use facts -If your extension always defines the same repositories given the same inputs (extension tags, files it reads, etc.) and in particular doesn't rely on any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by a checksum, consider returning [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with `reproducible = True`. This allows Bazel to skip this extension when writing to the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces the chance of merge conflicts. Note that Bazel still caches the results of reproducible extensions in a way that persists across server restarts, so even a long-running extension can be marked as reproducible without a performance penalty. - -If your extension relies on effectively immutable data obtained from outside the build, most commonly from the network, but you don't have a checksum available to guard the download, consider using the `facts` parameter of [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to persistently record such data and thus allow your extension to become reproducible. `facts` is expected to be a dictionary with string keys and arbitrary JSON-like Starlark values that is always persisted in the lockfile and available to future evaluations of the extension via the [`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. - -`facts` are not invalidated even when the code of your module extension changes, so be prepared to handle the case where the structure of `facts` changes. Bazel also assumes that two different `facts` dicts produced by two different evaluations of the same extension can be shallowly merged (i.e., as if by using the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` not supporting enumeration of its entries, just lookups by key. - -An example of using `facts` would be to record a mapping from version numbers of some SDK to the an object containing the download URL and checksum of that version. The first time the extension is evaluated, it can fetch this mapping from the network, but on later evaluations it can use the mapping from `facts` to avoid the network requests. +If your extension always defines the same repositories given the same inputs +(extension tags, files it reads, etc.) and in particular doesn't rely on +any [downloads](/rules/lib/builtins/module_ctx#download) that aren't guarded by +a checksum, consider returning +[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) with +`reproducible = True`. This allows Bazel to skip this extension when writing to +the `MODULE.bazel` lockfile, which helps keep the lockfile small and reduces +the chance of merge conflicts. Note that Bazel still caches the results of +reproducible extensions in a way that persists across server restarts, so even +a long-running extension can be marked as reproducible without a performance +penalty. + +If your extension relies on effectively immutable data obtained from outside +the build, most commonly from the network, but you don't have a checksum +available to guard the download, consider using the `facts` parameter of +[`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) to +persistently record such data and thus allow your extension to become +reproducible. `facts` is expected to be a dictionary with string keys and +arbitrary JSON-like Starlark values that is always persisted in the lockfile and +available to future evaluations of the extension via the +[`facts`](/rules/lib/builtins/module_ctx#facts) field of `module_ctx`. + +`facts` are not invalidated even when the code of your module extension changes, +so be prepared to handle the case where the structure of `facts` changes. +Bazel also assumes that two different `facts` dicts produced by two different +evaluations of the same extension can be shallowly merged (i.e., as if by using +the `|` operator on two dicts). This is partially enforced by `module_ctx.facts` +not supporting enumeration of its entries, just lookups by key. + +An example of using `facts` would be to record a mapping from version numbers of +some SDK to the an object containing the download URL and checksum of that +version. The first time the extension is evaluated, it can fetch this mapping +from the network, but on later evaluations it can use the mapping from `facts` +to avoid the network requests. ### Specify dependence on operating system and architecture -If your extension relies on the operating system or its architecture type, ensure to indicate this in the extension definition using the `os_dependent` and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the need for re-evaluation if there are changes to either of them. +If your extension relies on the operating system or its architecture type, +ensure to indicate this in the extension definition using the `os_dependent` +and `arch_dependent` boolean attributes. This ensures that Bazel recognizes the +need for re-evaluation if there are changes to either of them. -Since this kind of dependence on the host makes it more difficult to maintain the lockfile entry for this extension, consider [marking the extension reproducible](#specify_reproducibility) if possible. +Since this kind of dependence on the host makes it more difficult to maintain +the lockfile entry for this extension, consider +[marking the extension reproducible](#specify_reproducibility) if possible. ### Only the root module should directly affect repository names -Remember that when an extension creates repositories, they are created within the namespace of the extension. This means collisions can occur if different modules use the same extension and end up creating a repository with the same name. This often manifests as a module extension's `tag_class` having a `name` argument that is passed as a repository rule's `name` value. - -For example, say the root module, `A`, depends on module `B`. Both modules depend on module `mylang`. If both `A` and `B` call `mylang.toolchain(name="foo")`, they will both try to create a repository named `foo` within the `mylang` module and an error will occur. - -To avoid this, either remove the ability to set the repository name directly, or only allow the root module to do so. It's OK to allow the root module this ability because nothing will depend on it, so it doesn't have to worry about another module creating a conflicting name. +Remember that when an extension creates repositories, they are created within +the namespace of the extension. This means collisions can occur if different +modules use the same extension and end up creating a repository with the same +name. This often manifests as a module extension's `tag_class` having a `name` +argument that is passed as a repository rule's `name` value. + +For example, say the root module, `A`, depends on module `B`. Both modules +depend on module `mylang`. If both `A` and `B` call +`mylang.toolchain(name="foo")`, they will both try to create a repository named +`foo` within the `mylang` module and an error will occur. + +To avoid this, either remove the ability to set the repository name directly, +or only allow the root module to do so. It's OK to allow the root module this +ability because nothing will depend on it, so it doesn't have to worry about +another module creating a conflicting name. diff --git a/external/faq.mdx b/external/faq.mdx index 58d6c521..a0239cb5 100644 --- a/external/faq.mdx +++ b/external/faq.mdx @@ -2,63 +2,145 @@ title: 'Frequently asked questions' --- -This page answers some frequently asked questions about external dependencies in Bazel. -## MODULE.bazel - -### How should I version a Bazel module? -Setting `version` with the [`module`](/rules/lib/globals/module#module) directive in the source archive `MODULE.bazel` can have several downsides and unintended side effects if not managed carefully: -- Duplication: releasing a new version of a module typically involves both incrementing the version in `MODULE.bazel` and tagging the release, two separate steps that can fall out of sync. While automation can reduce this risk, it's simpler and safer to avoid it altogether. -- Inconsistency: users overriding a module with a specific commit using a [non-registry override](module.md#non-registry_overrides) will see an incorrect version. for example, if the `MODULE.bazel` in the source archive sets `version = "0.3.0"` but additional commits have been made since that release, a user overriding with one of those commits would still see `0.3.0`. In reality, the version should reflect that it's ahead of the release, for example `0.3.1-rc1`. +This page answers some frequently asked questions about external dependencies in +Bazel. -- Non-registry override issues: using placeholder values can cause issues when users override a module with a non-registry override. For example, `0.0.0` doesn't sort as the highest version, which is usually the expected behavior users want when doing a non-registry override. - -Thus, it's best to avoid setting the version in the source archive `MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry (e.g., the [Bazel Central Registry](https://registry.bazel.build/)), which is the actual source of truth for the module version during Bazel's external dependency resolution (see [Bazel registries](https://bazel.build/external/registry)). +## MODULE.bazel -This is usually automated, for example the [`rules-template`](https://github.com/bazel-contrib/rules-template) example rule repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml) to publish the release to the BCR. The action [generates a patch for the source archive `MODULE.bazel`](https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216) with the release version. This patch is stored in the registry and is applied when the module is fetched during Bazel's external dependency resolution. +### How should I version a Bazel module? -This way, the version in the releases in the registry will be correctly set to the released version and thus, `bazel_dep`, `single_version_override` and `multiple_version_override` will work as expected, while avoiding potential issues when doing a non-registry override because the version in the source archive will be the default value (`''`), which will always be handled correctly (it's the default version value after all) and will behave as expected when sorting (the empty string is treated as the highest version). +Setting `version` with the [`module`] directive in the source archive +`MODULE.bazel` can have several downsides and unintended side effects if not +managed carefully: + +* Duplication: releasing a new version of a module typically involves both + incrementing the version in `MODULE.bazel` and tagging the release, two + separate steps that can fall out of sync. While automation can + reduce this risk, it's simpler and safer to avoid it altogether. + +* Inconsistency: users overriding a module with a specific commit using a + [non-registry override] will see an incorrect version. for example, if the + `MODULE.bazel` in the source archive sets `version = "0.3.0"` but + additional commits have been made since that release, a user overriding + with one of those commits would still see `0.3.0`. In reality, the version + should reflect that it's ahead of the release, for example `0.3.1-rc1`. + +* Non-registry override issues: using placeholder values can cause issues + when users override a module with a non-registry override. For example, + `0.0.0` doesn't sort as the highest version, which is usually the expected + behavior users want when doing a non-registry override. + +Thus, it's best to avoid setting the version in the source archive +`MODULE.bazel`. Instead, set it in the `MODULE.bazel` stored in the registry +(e.g., the [Bazel Central Registry]), which is the actual source of truth for +the module version during Bazel's external dependency resolution (see [Bazel +registries]). + +This is usually automated, for example the [`rules-template`] example rule +repository uses a [bazel-contrib/publish-to-bcr publish.yaml GitHub Action] to +publish the release to the BCR. The action [generates a patch for the source +archive `MODULE.bazel`] with the release version. This patch is stored in the +registry and is applied when the module is fetched during Bazel's external +dependency resolution. + +This way, the version in the releases in the registry will be correctly set to +the released version and thus, `bazel_dep`, `single_version_override` and +`multiple_version_override` will work as expected, while avoiding potential +issues when doing a non-registry override because the version in the source +archive will be the default value (`''`), which will always be handled +correctly (it's the default version value after all) and will behave as +expected when sorting (the empty string is treated as the highest version). + +[Bazel Central Registry]: https://registry.bazel.build/ +[Bazel registries]: https://bazel.build/external/registry +[bazel-contrib/publish-to-bcr publish.yaml GitHub Action]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/.github/workflows/publish.yaml +[generates a patch for the source archive `MODULE.bazel`]: https://github.com/bazel-contrib/publish-to-bcr/blob/v0.2.2/src/domain/create-entry.ts#L176-L216 +[`module`]: /rules/lib/globals/module#module +[non-registry override]: module.md#non-registry_overrides +[`rules-template`]: https://github.com/bazel-contrib/rules-template ### When should I increment the compatibility level? -The [`compatibility_level`](module.md#compatibility_level) of a Bazel module should be incremented *in the same commit* that introduces a backwards incompatible ("breaking") change. +The [`compatibility_level`](module.md#compatibility_level) of a Bazel module +should be incremented _in the same commit_ that introduces a backwards +incompatible ("breaking") change. -However, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. This can happen when for example' two modules depend on versions of a third module with different compatibility levels. +However, Bazel can throw an error if it detects that versions of the _same +module_ with _different compatibility levels_ exist in the resolved dependency +graph. This can happen when for example' two modules depend on versions of a +third module with different compatibility levels. -Thus, incrementing `compatibility_level` too frequently can be very disruptive and is discouraged. To avoid this situation, the `compatibility_level` should be incremented *only* when the breaking change affects most use cases and isn't easy to migrate and/or work-around. +Thus, incrementing `compatibility_level` too frequently can be very disruptive +and is discouraged. To avoid this situation, the `compatibility_level` should be +incremented _only_ when the breaking change affects most use cases and isn't +easy to migrate and/or work-around. ### Why does MODULE.bazel not support `load`s? -During dependency resolution, the MODULE.bazel file of all referenced external dependencies are fetched from registries. At this stage, the source archives of the dependencies are not fetched yet; so if the MODULE.bazel file `load`s another file, there is no way for Bazel to actually fetch that file without fetching the entire source archive. Note the MODULE.bazel file itself is special, as it's directly hosted on the registry. - -There are a few use cases that people asking for `load`s in MODULE.bazel are generally interested in, and they can be solved without `load`s: - -- Ensuring that the version listed in MODULE.bazel is consistent with build metadata stored elsewhere, for example in a .bzl file: This can be achieved by using the [`native.module_version`](/rules/lib/toplevel/native#module_version) method in a .bzl file loaded from a BUILD file. -- Splitting up a very large MODULE.bazel file into manageable sections, particularly for monorepos: The root module can use the [`include`](/rules/lib/globals/module#include) directive to split its MODULE.bazel file into multiple segments. For the same reason we don't allow `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. -- Users of the old WORKSPACE system might remember declaring a repo, and then immediately `load`ing from that repo to perform complex logic. This capability has been replaced by [module extensions](extension). +During dependency resolution, the MODULE.bazel file of all referenced external +dependencies are fetched from registries. At this stage, the source archives of +the dependencies are not fetched yet; so if the MODULE.bazel file `load`s +another file, there is no way for Bazel to actually fetch that file without +fetching the entire source archive. Note the MODULE.bazel file itself is +special, as it's directly hosted on the registry. + +There are a few use cases that people asking for `load`s in MODULE.bazel are +generally interested in, and they can be solved without `load`s: + +* Ensuring that the version listed in MODULE.bazel is consistent with build + metadata stored elsewhere, for example in a .bzl file: This can be achieved + by using the + [`native.module_version`](/rules/lib/toplevel/native#module_version) method + in a .bzl file loaded from a BUILD file. +* Splitting up a very large MODULE.bazel file into manageable sections, + particularly for monorepos: The root module can use the + [`include`](/rules/lib/globals/module#include) directive to split its + MODULE.bazel file into multiple segments. For the same reason we don't allow + `load`s in MODULE.bazel files, `include` cannot be used in non-root modules. +* Users of the old WORKSPACE system might remember declaring a repo, and then + immediately `load`ing from that repo to perform complex logic. This + capability has been replaced by [module extensions](extension). ### Can I specify a SemVer range for a `bazel_dep`? -No. Some other package managers like [npm](https://docs.npmjs.com/about-semantic-versioning) and [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax) support version ranges (implicitly or explicitly), and this often requires a constraint solver (making the output harder to predict for users) and makes version resolution nonreproducible without a lockfile. +No. Some other package managers like [npm][npm-semver] and [Cargo][cargo-semver] +support version ranges (implicitly or explicitly), and this often requires a +constraint solver (making the output harder to predict for users) and makes +version resolution nonreproducible without a lockfile. -Bazel instead uses [Minimal Version Selection](module#version-selection) like Go, which in contrast makes the output easy to predict and guarantees reproducibility. This is a tradeoff that matches Bazel's design goals. +Bazel instead uses [Minimal Version Selection](module#version-selection) like +Go, which in contrast makes the output easy to predict and guarantees +reproducibility. This is a tradeoff that matches Bazel's design goals. -Furthermore, Bazel module versions are [a superset of SemVer](module#version-format), so what makes sense in a strict SemVer environment doesn't always carry over to Bazel module versions. +Furthermore, Bazel module versions are [a superset of +SemVer](module#version-format), so what makes sense in a strict SemVer +environment doesn't always carry over to Bazel module versions. ### Can I automatically get the latest version for a `bazel_dep`? -Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", version = "latest")` to automatically get the latest version of a dep. This is similar to [the question about SemVer ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also no. +Some users occasionally ask for the ability to specify `bazel_dep(name = "foo", +version = "latest")` to automatically get the latest version of a dep. This is +similar to [the question about SemVer +ranges](#can-i-specify-a-semver-range-for-a-bazel-dep), and the answer is also +no. -The recommended solution here is to have automation take care of this. For example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports Bazel modules. +The recommended solution here is to have automation take care of this. For +example, [Renovate](https://docs.renovatebot.com/modules/manager/) supports +Bazel modules. -Sometimes, users asking this question are really looking for a way to quickly iterate during local development. This can be achieved by using a [`local_path_override`](/rules/lib/globals/module#local_path_override). +Sometimes, users asking this question are really looking for a way to quickly +iterate during local development. This can be achieved by using a +[`local_path_override`](/rules/lib/globals/module#local_path_override). ### Why all these `use_repo`s? -Module extension usages in MODULE.bazel files sometimes come with a big `use_repo` directive. For example, a typical usage of the [`go_deps` extension](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies) from `gazelle` might look like: +Module extension usages in MODULE.bazel files sometimes come with a big +`use_repo` directive. For example, a typical usage of the +[`go_deps` extension][go_deps] from `gazelle` might look like: ```python go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") @@ -73,82 +155,175 @@ use_repo( ) ``` -The long `use_repo` directive may seem redundant, since the information is arguably already in the referenced `go.mod` file. +The long `use_repo` directive may seem redundant, since the information is +arguably already in the referenced `go.mod` file. -The reason Bazel needs this `use_repo` directive is that it runs module extensions lazily. That is, a module extension is only run if its result is observed. Since a module extension's "output" is repo definitions, this means that we only run a module extension if a repo it defines is requested (for instance, if the target `@org_golang_x_net//:foo` is built, in the example above). However, we don't know which repos a module extension would define until after we run it. This is where the `use_repo` directive comes in; the user can tell Bazel which repos they expect the extension to generate, and Bazel would then only run the extension when these specific repos are used. +The reason Bazel needs this `use_repo` directive is that it runs module +extensions lazily. That is, a module extension is only run if its result is +observed. Since a module extension's "output" is repo definitions, this means +that we only run a module extension if a repo it defines is requested (for +instance, if the target `@org_golang_x_net//:foo` is built, in the example +above). However, we don't know which repos a module extension would define until +after we run it. This is where the `use_repo` directive comes in; the user can +tell Bazel which repos they expect the extension to generate, and Bazel would +then only run the extension when these specific repos are used. -To help the maintain this `use_repo` directive, a module extension can return an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) object from its implementation function. The user can run the `bazel mod tidy` command to update the `use_repo` directives for these module extensions. +To help the maintain this `use_repo` directive, a module extension can return +an [`extension_metadata`](/rules/lib/builtins/module_ctx#extension_metadata) +object from its implementation function. The user can run the `bazel mod tidy` +command to update the `use_repo` directives for these module extensions. ## Bzlmod migration ### Which is evaluated first, MODULE.bazel or WORKSPACE? -When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to wonder which system is consulted first. The short answer is that MODULE.bazel (Bzlmod) is evaluated first. - -The long answer is that "which evaluates first" is not the right question to ask; rather, the right question to ask is: in the context of the repo with [canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what is the repo mapping of `@@base`? - -Labels with apparent repo names (a single leading `@`) can refer to different things based on the context they're resolved from. When you see a label `@bar//:baz` and wonder what it actually points to, you need to first find out what the context repo is: for example, if the label is in a BUILD file located in the repo `@@foo`, then the context repo is `@@foo`. - -Then, depending on what the context repo is, the ["repository visibility" table](migration#repository-visibility) in the migration guide can be used to find out which repo an apparent name actually resolves to. - -- If the context repo is the main repo (`@@`): - - 1. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file (through any of [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), [`use_repo`](/rules/lib/globals/module#use_repo), [`module`](/rules/lib/globals/module#module.repo_name), [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` resolves to what that MODULE.bazel file claims. - 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its canonical name is `@@bar`), then `@bar` resolves to `@@bar`. - 3. Otherwise, `@bar` resolves to something like `@@[unknown repo 'bar' requested from @@]`, and this will ultimately result in an error. - -- If the context repo is a Bzlmod-world repo (that is, it corresponds to a non-root Bazel module, or is generated by a module extension), then it will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. - - Notably, this includes any repos introduced in a `non_module_deps`-like module extension in the root module, or `use_repo_rule` instantiations in the root module. - -- If the context repo is defined in WORKSPACE: - - 1. First, check if the context repo definition has the magical `repo_mapping` attribute. If so, go through the mapping first (so for a repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking at `@baz` below). - 2. If `bar` is an apparent repo name introduced by the root module's MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file claims. (This is the same as item 1 in the main repo case.) - 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel will throw an error. +When both `--enable_bzlmod` and `--enable_workspace` are set, it's natural to +wonder which system is consulted first. The short answer is that MODULE.bazel +(Bzlmod) is evaluated first. + +The long answer is that "which evaluates first" is not the right question to +ask; rather, the right question to ask is: in the context of the repo with +[canonical name](overview#canonical-repo-name) `@@foo`, what does the [apparent +repo name](overview#apparent-repo-name) `@bar` resolve to? Alternatively, what +is the repo mapping of `@@base`? + +Labels with apparent repo names (a single leading `@`) can refer to different +things based on the context they're resolved from. When you see a label +`@bar//:baz` and wonder what it actually points to, you need to first find out +what the context repo is: for example, if the label is in a BUILD file located +in the repo `@@foo`, then the context repo is `@@foo`. + +Then, depending on what the context repo is, the ["repository +visibility" table](migration#repository-visibility) in the migration guide can +be used to find out which repo an apparent name actually resolves to. + +* If the context repo is the main repo (`@@`): + 1. If `bar` is an apparent repo name introduced by the root module's + MODULE.bazel file (through any of + [`bazel_dep`](/rules/lib/globals/module#bazel_dep.repo_name), + [`use_repo`](/rules/lib/globals/module#use_repo), + [`module`](/rules/lib/globals/module#module.repo_name), + [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule)), then `@bar` + resolves to what that MODULE.bazel file claims. + 2. Otherwise, if `bar` is a repo defined in WORKSPACE (which means that its + canonical name is `@@bar`), then `@bar` resolves to `@@bar`. + 3. Otherwise, `@bar` resolves to something like + `@@[unknown repo 'bar' requested from @@]`, and this will ultimately + result in an error. +* If the context repo is a Bzlmod-world repo (that is, it corresponds to a + non-root Bazel module, or is generated by a module extension), then it + will only ever see other Bzlmod-world repos, and no WORKSPACE-world repos. + * Notably, this includes any repos introduced in a `non_module_deps`-like + module extension in the root module, or `use_repo_rule` instantiations + in the root module. +* If the context repo is defined in WORKSPACE: + 1. First, check if the context repo definition has the magical + `repo_mapping` attribute. If so, go through the mapping first (so for a + repo defined with `repo_mapping = {"@bar": "@baz"}`, we would be looking + at `@baz` below). + 2. If `bar` is an apparent repo name introduced by the root module's + MODULE.bazel file, then `@bar` resolves to what that MODULE.bazel file + claims. (This is the same as item 1 in the main repo case.) + 3. Otherwise, `@bar` resolves to `@@bar`. This most likely will point to a + repo `bar` defined in WORKSPACE; if such a repo is not defined, Bazel + will throw an error. For a more succinct version: -- Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world repos. -- WORKSPACE-world repos (including the main repo) will first see what the root module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world repos. +* Bzlmod-world repos (excluding the main repo) will only see Bzlmod-world + repos. +* WORKSPACE-world repos (including the main repo) will first see what the root + module in the Bzlmod world defines, then fall back to seeing WORKSPACE-world + repos. -Of note, labels in the Bazel command line (including Starlark flags, label-typed flag values, and build/test target patterns) are treated as having the main repo as the context repo. +Of note, labels in the Bazel command line (including Starlark flags, label-typed +flag values, and build/test target patterns) are treated as having the main repo +as the context repo. ## Other ### How do I prepare and run an offline build? -Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag (like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the context of the main repo, see [question above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of `@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). +Use the `bazel fetch` command to prefetch repos. You can use the `--repo` flag +(like `bazel fetch --repo @foo`) to fetch only the repo `@foo` (resolved in the +context of the main repo, see [question +above](#which-is-evaluated-first-module-bazel-or-workspace)), or use a target +pattern (like `bazel fetch @foo//:bar`) to fetch all transitive dependencies of +`@foo//:bar` (this is equivalent to `bazel build --nobuild @foo//:bar`). -The make sure no fetches happen during a build, use `--nofetch`. More precisely, this makes any attempt to run a non-local repository rule fail. +The make sure no fetches happen during a build, use `--nofetch`. More precisely, +this makes any attempt to run a non-local repository rule fail. -If you want to fetch repos *and* modify them to test locally, consider using the [`bazel vendor`](vendor) command. +If you want to fetch repos _and_ modify them to test locally, consider using +the [`bazel vendor`](vendor) command. ### How do I use HTTP proxies? -Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly accepted by other programs, such as [curl](https://everything.curl.dev/usingcurl/proxies/env.html). +Bazel respects the `http_proxy` and `HTTPS_PROXY` environment variables commonly +accepted by other programs, such as +[curl](https://everything.curl.dev/usingcurl/proxies/env.html). ### How do I make Bazel prefer IPv6 in dual-stack IPv4/IPv6 setups? -On IPv6-only machines, Bazel can download dependencies with no changes. However, on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, preferring IPv4 if enabled. In some situations, for example when the IPv4 network cannot resolve/reach external addresses, this can cause `Network unreachable` exceptions and build failures. In these cases, you can override Bazel's behavior to prefer IPv6 by using the [`java.net.preferIPv6Addresses=true` system property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). Specifically: +On IPv6-only machines, Bazel can download dependencies with no changes. However, +on dual-stack IPv4/IPv6 machines Bazel follows the same convention as Java, +preferring IPv4 if enabled. In some situations, for example when the IPv4 +network cannot resolve/reach external addresses, this can cause `Network +unreachable` exceptions and build failures. In these cases, you can override +Bazel's behavior to prefer IPv6 by using the +[`java.net.preferIPv6Addresses=true` system +property](https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html). +Specifically: + +* Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup + option](/docs/user-manual#startup-options), for example by adding the + following line in your [`.bazelrc` file](/run/bazelrc): + + `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` + +* When running Java build targets that need to connect to the internet (such + as for integration tests), use the + `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool + flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` + file](/run/bazelrc): + + `build --jvmopt=-Djava.net.preferIPv6Addresses` + +* If you are using + [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for + dependency version resolution, also add + `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment + variable to [provide JVM options for + Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). -- Use `--host_jvm_args=-Djava.net.preferIPv6Addresses=true` [startup option](/docs/user-manual#startup-options), for example by adding the following line in your [`.bazelrc` file](/run/bazelrc): - - `startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true` - -- When running Java build targets that need to connect to the internet (such as for integration tests), use the `--jvmopt=-Djava.net.preferIPv6Addresses=true` [tool flag](/docs/user-manual#jvmopt). For example, include in your [`.bazelrc` file](/run/bazelrc): - - `build --jvmopt=-Djava.net.preferIPv6Addresses` +### Can repo rules be run remotely with remote execution? -- If you are using [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) for dependency version resolution, also add `-Djava.net.preferIPv6Addresses=true` to the `COURSIER_OPTS` environment variable to [provide JVM options for Coursier](https://github.com/bazelbuild/rules_jvm_external#provide-jvm-options-for-coursier-with-coursier_opts). +No; or at least, not yet. Users employing remote execution services to speed up +their builds may notice that repo rules are still run locally. For example, an +`http_archive` would be first downloaded onto the local machine (using any local +download cache if applicable), extracted, and then each source file would be +uploaded to the remote execution service as an input file. It's natural to ask +why the remote execution service doesn't just download and extract that archive, +saving a useless roundtrip. -### Can repo rules be run remotely with remote execution? +Part of the reason is that repo rules (and module extensions) are akin to +"scripts" that are run by Bazel itself. A remote executor doesn't necessarily +even have a Bazel installed. -No; or at least, not yet. Users employing remote execution services to speed up their builds may notice that repo rules are still run locally. For example, an `http_archive` would be first downloaded onto the local machine (using any local download cache if applicable), extracted, and then each source file would be uploaded to the remote execution service as an input file. It's natural to ask why the remote execution service doesn't just download and extract that archive, saving a useless roundtrip. +Another reason is that Bazel often needs the BUILD files in the downloaded and +extracted archives to perform loading and analysis, which _are_ performed +locally. -Part of the reason is that repo rules (and module extensions) are akin to "scripts" that are run by Bazel itself. A remote executor doesn't necessarily even have a Bazel installed. +There are preliminary ideas to solve this problem by re-imagining repo rules as +build rules, which would naturally allow them to be run remotely, but conversely +raise new architectural concerns (for example, the `query` commands would +potentially need to run actions, complicating their design). -Another reason is that Bazel often needs the BUILD files in the downloaded and extracted archives to perform loading and analysis, which *are* performed locally. +For more previous discussion on this topic, see [A way to support repositories +that need Bazel for being +fetched](https://github.com/bazelbuild/bazel/discussions/20464). -There are preliminary ideas to solve this problem by re-imagining repo rules as build rules, which would naturally allow them to be run remotely, but conversely raise new architectural concerns (for example, the `query` commands would potentially need to run actions, complicating their design). +[npm-semver]: https://docs.npmjs.com/about-semantic-versioning +[cargo-semver]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#version-requirement-syntax +[go_deps]: https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#specifying-external-dependencies -For more previous discussion on this topic, see [A way to support repositories that need Bazel for being fetched](https://github.com/bazelbuild/bazel/discussions/20464). diff --git a/external/lockfile.mdx b/external/lockfile.mdx index a20423c3..9f312615 100644 --- a/external/lockfile.mdx +++ b/external/lockfile.mdx @@ -1,46 +1,98 @@ +keywords: product:Bazel,lockfile,Bzlmod --- title: 'Bazel Lockfile' --- -The lockfile feature in Bazel enables the recording of specific versions or dependencies of software libraries or packages required by a project. It achieves this by storing the result of module resolution and extension evaluation. The lockfile promotes reproducible builds, ensuring consistent development environments. Additionally, it enhances build efficiency by allowing Bazel to skip the parts of the resolution process that are unaffected by changes in project dependencies. Furthermore, the lockfile improves stability by preventing unexpected updates or breaking changes in external libraries, thereby reducing the risk of introducing bugs. + + +The lockfile feature in Bazel enables the recording of specific versions or +dependencies of software libraries or packages required by a project. It +achieves this by storing the result of module resolution and extension +evaluation. The lockfile promotes reproducible builds, ensuring consistent +development environments. Additionally, it enhances build efficiency by allowing +Bazel to skip the parts of the resolution process that are unaffected by changes +in project dependencies. Furthermore, the lockfile improves stability by +preventing unexpected updates or breaking changes in external libraries, thereby +reducing the risk of introducing bugs. ## Lockfile Generation -The lockfile is generated under the workspace root with the name `MODULE.bazel.lock`. It is created or updated during the build process, specifically after module resolution and extension evaluation. Importantly, it only includes dependencies that are included in the current invocation of the build. +The lockfile is generated under the workspace root with the name +`MODULE.bazel.lock`. It is created or updated during the build process, +specifically after module resolution and extension evaluation. Importantly, it +only includes dependencies that are included in the current invocation of the +build. -When changes occur in the project that affect its dependencies, the lockfile is automatically updated to reflect the new state. This ensures that the lockfile remains focused on the specific set of dependencies required for the current build, providing an accurate representation of the project's resolved dependencies. +When changes occur in the project that affect its dependencies, the lockfile is +automatically updated to reflect the new state. This ensures that the lockfile +remains focused on the specific set of dependencies required for the current +build, providing an accurate representation of the project's resolved +dependencies. ## Lockfile Usage -The lockfile can be controlled by the flag [`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to customize the behavior of Bazel when the project state differs from the lockfile. The available modes are: - -- `update` (Default): Use the information that is present in the lockfile to skip downloads of known registry files and to avoid re-evaluating extensions whose results are still up-to-date. If information is missing, it will be added to the lockfile. In this mode, Bazel also avoids refreshing mutable information, such as yanked versions, for dependencies that haven't changed. -- `refresh`: Like `update`, but mutable information is always refreshed when switching to this mode and roughly every hour while in this mode. -- `error`: Like `update`, but if any information is missing or out-of-date, Bazel will fail with an error. This mode never changes the lockfile or performs network requests during resolution. Module extensions that marked themselves as `reproducible` may still perform network requests, but are expected to always produce the same result. -- `off`: The lockfile is neither checked nor updated. +The lockfile can be controlled by the flag +[`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to +customize the behavior of Bazel when the project state differs from the +lockfile. The available modes are: + +* `update` (Default): Use the information that is present in the lockfile to + skip downloads of known registry files and to avoid re-evaluating extensions + whose results are still up-to-date. If information is missing, it will + be added to the lockfile. In this mode, Bazel also avoids refreshing + mutable information, such as yanked versions, for dependencies that haven't + changed. +* `refresh`: Like `update`, but mutable information is always refreshed when + switching to this mode and roughly every hour while in this mode. +* `error`: Like `update`, but if any information is missing or out-of-date, + Bazel will fail with an error. This mode never changes the lockfile or + performs network requests during resolution. Module extensions that marked + themselves as `reproducible` may still perform network requests, but are + expected to always produce the same result. +* `off`: The lockfile is neither checked nor updated. ## Lockfile Benefits The lockfile offers several benefits and can be utilized in various ways: -- **Reproducible builds.** By capturing the specific versions or dependencies of software libraries, the lockfile ensures that builds are reproducible across different environments and over time. Developers can rely on consistent and predictable results when building their projects. +- **Reproducible builds.** By capturing the specific versions or dependencies + of software libraries, the lockfile ensures that builds are reproducible + across different environments and over time. Developers can rely on + consistent and predictable results when building their projects. -- **Fast incremental resolutions.** The lockfile enables Bazel to avoid downloading registry files that were already used in a previous build. This significantly improves build efficiency, especially in scenarios where resolution can be time-consuming. +- **Fast incremental resolutions.** The lockfile enables Bazel to avoid + downloading registry files that were already used in a previous build. + This significantly improves build efficiency, especially in scenarios where + resolution can be time-consuming. -- **Stability and risk reduction.** The lockfile helps maintain stability by preventing unexpected updates or breaking changes in external libraries. By locking the dependencies to specific versions, the risk of introducing bugs due to incompatible or untested updates is reduced. +- **Stability and risk reduction.** The lockfile helps maintain stability by + preventing unexpected updates or breaking changes in external libraries. By + locking the dependencies to specific versions, the risk of introducing bugs + due to incompatible or untested updates is reduced. ### Hidden lockfile -Bazel also maintains another lockfile at `"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this lockfile are explicitly unspecified. It is only used as a performance optimization. While it can be deleted together with the output base via `bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a module extension. +Bazel also maintains another lockfile at +`"$(bazel info output_base)"/MODULE.bazel.lock`. The format and contents of this +lockfile are explicitly unspecified. It is only used as a performance +optimization. While it can be deleted together with the output base via +`bazel clean --expunge`, any need to do so is a bug in either Bazel itself or a +module extension. ## Lockfile Contents -The lockfile contains all the necessary information to determine whether the project state has changed. It also includes the result of building the project in the current state. The lockfile consists of two main parts: +The lockfile contains all the necessary information to determine whether the +project state has changed. It also includes the result of building the project +in the current state. The lockfile consists of two main parts: -1. Hashes of all remote files that are inputs to module resolution. -2. For each module extension, the lockfile includes inputs that affect it, represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as well as the output of running that extension, referred to as `generatedRepoSpecs` +1. Hashes of all remote files that are inputs to module resolution. +2. For each module extension, the lockfile includes inputs that affect it, + represented by `bzlTransitiveDigest`, `usagesDigest` and other fields, as + well as the output of running that extension, referred to as + `generatedRepoSpecs` -Here is an example that demonstrates the structure of the lockfile, along with explanations for each section: +Here is an example that demonstrates the structure of the lockfile, along with +explanations for each section: ```json { @@ -100,53 +152,106 @@ Here is an example that demonstrates the structure of the lockfile, along with e ### Registry File Hashes -The `registryFileHashes` section contains the hashes of all files from remote registries accessed during module resolution. Since the resolution algorithm is fully deterministic when given the same inputs and all remote inputs are hashed, this ensures a fully reproducible resolution result while avoiding excessive duplication of remote information in the lockfile. Note that this also requires recording when a particular registry didn't contain a certain module, but a registry with lower precedence did (see the "not found" entry in the example). This inherently mutable information can be updated via `bazel mod deps --lockfile_mode=refresh`. +The `registryFileHashes` section contains the hashes of all files from +remote registries accessed during module resolution. Since the resolution +algorithm is fully deterministic when given the same inputs and all remote +inputs are hashed, this ensures a fully reproducible resolution result while +avoiding excessive duplication of remote information in the lockfile. Note that +this also requires recording when a particular registry didn't contain a certain +module, but a registry with lower precedence did (see the "not found" entry in +the example). This inherently mutable information can be updated via +`bazel mod deps --lockfile_mode=refresh`. -Bazel uses the hashes from the lockfile to look up registry files in the repository cache before downloading them, which speeds up subsequent resolutions. +Bazel uses the hashes from the lockfile to look up registry files in the +repository cache before downloading them, which speeds up subsequent +resolutions. ### Selected Yanked Versions -The `selectedYankedVersions` section contains the yanked versions of modules that were selected by module resolution. Since this usually result in an error when trying to build, this section is only non-empty when yanked versions are explicitly allowed via `--allow_yanked_versions` or `BZLMOD_ALLOW_YANKED_VERSIONS`. +The `selectedYankedVersions` section contains the yanked versions of modules +that were selected by module resolution. Since this usually result in an error +when trying to build, this section is only non-empty when yanked versions are +explicitly allowed via `--allow_yanked_versions` or +`BZLMOD_ALLOW_YANKED_VERSIONS`. -This field is needed since, compared to module files, yanked version information is inherently mutable and thus can't be referenced by a hash. This information can be updated via `bazel mod deps --lockfile_mode=refresh`. +This field is needed since, compared to module files, yanked version information +is inherently mutable and thus can't be referenced by a hash. This information +can be updated via `bazel mod deps --lockfile_mode=refresh`. ### Module Extensions -The `moduleExtensions` section is a map that includes only the extensions used in the current invocation or previously invoked, while excluding any extensions that are no longer utilized. In other words, if an extension is not being used anymore across the dependency graph, it is removed from the `moduleExtensions` map. - -If an extension is independent of the operating system or architecture type, this section features only a single "general" entry. Otherwise, multiple entries are included, named after the OS, architecture, or both, with each corresponding to the result of evaluating the extension on those specifics. - -Each entry in the extension map corresponds to a used extension and is identified by its containing file and name. The corresponding value for each entry contains the relevant information associated with that extension: - -1. The `bzlTransitiveDigest` is the digest of the extension implementation and the .bzl files transitively loaded by it. -2. The `usagesDigest` is the digest of the *usages* of the extension in the dependency graph, which includes all tags. -3. Further unspecified fields that track other inputs to the extension, such as contents of files or directories it reads or environment variables it uses. -4. The `generatedRepoSpecs` encode the repositories created by the extension with the current input. -5. The optional `moduleExtensionMetadata` field contains metadata provided by the extension such as whether certain repositories it created should be imported via `use_repo` by the root module. This information powers the `bazel mod tidy` command. - -Module extensions can opt out of being included in the lockfile by setting the returning metadata with `reproducible = True`. By doing so, they promise that they will always create the same repositories when given the same inputs. +The `moduleExtensions` section is a map that includes only the extensions used +in the current invocation or previously invoked, while excluding any extensions +that are no longer utilized. In other words, if an extension is not being used +anymore across the dependency graph, it is removed from the `moduleExtensions` +map. + +If an extension is independent of the operating system or architecture type, +this section features only a single "general" entry. Otherwise, multiple +entries are included, named after the OS, architecture, or both, with each +corresponding to the result of evaluating the extension on those specifics. + +Each entry in the extension map corresponds to a used extension and is +identified by its containing file and name. The corresponding value for each +entry contains the relevant information associated with that extension: + +1. The `bzlTransitiveDigest` is the digest of the extension implementation + and the .bzl files transitively loaded by it. +2. The `usagesDigest` is the digest of the _usages_ of the extension in the + dependency graph, which includes all tags. +3. Further unspecified fields that track other inputs to the extension, + such as contents of files or directories it reads or environment + variables it uses. +4. The `generatedRepoSpecs` encode the repositories created by the + extension with the current input. +5. The optional `moduleExtensionMetadata` field contains metadata provided by + the extension such as whether certain repositories it created should be + imported via `use_repo` by the root module. This information powers the + `bazel mod tidy` command. + +Module extensions can opt out of being included in the lockfile by setting the +returning metadata with `reproducible = True`. By doing so, they promise that +they will always create the same repositories when given the same inputs. ## Best Practices -To maximize the benefits of the lockfile feature, consider the following best practices: +To maximize the benefits of the lockfile feature, consider the following best +practices: -- Regularly update the lockfile to reflect changes in project dependencies or configuration. This ensures that subsequent builds are based on the most up-to-date and accurate set of dependencies. To lock down all extensions at once, run `bazel mod deps --lockfile_mode=update`. +* Regularly update the lockfile to reflect changes in project dependencies or + configuration. This ensures that subsequent builds are based on the most + up-to-date and accurate set of dependencies. To lock down all extensions + at once, run `bazel mod deps --lockfile_mode=update`. -- Include the lockfile in version control to facilitate collaboration and ensure that all team members have access to the same lockfile, promoting consistent development environments across the project. +* Include the lockfile in version control to facilitate collaboration and + ensure that all team members have access to the same lockfile, promoting + consistent development environments across the project. -- Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a `.bazelversion` file in version control that specifies the Bazel version corresponding to the lockfile. Because Bazel itself is a dependency of your build, the lockfile is specific to the Bazel version, and will change even between [backwards compatible](/release/backward-compatibility) Bazel releases. Using `bazelisk` ensures that all developers are using a Bazel version that matches the lockfile. +* Use [`bazelisk`](/install/bazelisk) to run Bazel, and include a + `.bazelversion` file in version control that specifies the Bazel version + corresponding to the lockfile. Because Bazel itself is a dependency of + your build, the lockfile is specific to the Bazel version, and will + change even between [backwards compatible](/release/backward-compatibility) + Bazel releases. Using `bazelisk` ensures that all developers are using + a Bazel version that matches the lockfile. -By following these best practices, you can effectively utilize the lockfile feature in Bazel, leading to more efficient, reliable, and collaborative software development workflows. +By following these best practices, you can effectively utilize the lockfile +feature in Bazel, leading to more efficient, reliable, and collaborative +software development workflows. ## Merge Conflicts -The lockfile format is designed to minimize merge conflicts, but they can still happen. +The lockfile format is designed to minimize merge conflicts, but they can still +happen. ### Automatic Resolution -Bazel provides a custom [git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) to help resolve these conflicts automatically. +Bazel provides a custom +[git merge driver](https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver) +to help resolve these conflicts automatically. -Set up the driver by adding this line to a `.gitattributes` file in the root of your git repository: +Set up the driver by adding this line to a `.gitattributes` file in the root of +your git repository: ```gitattributes # A custom merge driver for the Bazel lockfile. @@ -154,7 +259,8 @@ Set up the driver by adding this line to a `.gitattributes` file in the root of MODULE.bazel.lock merge=bazel-lockfile-merge ``` -Then each developer who wants to use the driver has to register it once by following these steps: +Then each developer who wants to use the driver has to register it once by +following these steps: 1. Install [jq](https://jqlang.github.io/jq/download/) (1.5 or higher). 2. Run the following commands: @@ -163,15 +269,18 @@ Then each developer who wants to use the driver has to register it once by follo jq_script=$(curl https://raw.githubusercontent.com/bazelbuild/bazel/master/scripts/bazel-lockfile-merge.jq) printf '%s\n' "${jq_script}" | less # to optionally inspect the jq script git config --global merge.bazel-lockfile-merge.name "Merge driver for the Bazel lockfile (MODULE.bazel.lock)" -git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" +git config --global merge.bazel-lockfile-merge.driver "jq -s '${jq_script}' -- %O %A %B > %A.jq_tmp && mv %A.jq_tmp %A" ``` ### Manual Resolution -Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` fields can be safely resolved by keeping all the entries from both sides of the conflict. +Simple merge conflicts in the `registryFileHashes` and `selectedYankedVersions` +fields can be safely resolved by keeping all the entries from both sides of the +conflict. Other types of merge conflicts should not be resolved manually. Instead: -1. Restore the previous state of the lockfile via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. +1. Restore the previous state of the lockfile + via `git reset MODULE.bazel.lock && git checkout MODULE.bazel.lock`. 2. Resolve any conflicts in the `MODULE.bazel` file. 3. Run `bazel mod deps` to update the lockfile. diff --git a/external/migration.mdx b/external/migration.mdx index 49ba462f..8e70b1f3 100644 --- a/external/migration.mdx +++ b/external/migration.mdx @@ -1,200 +1,267 @@ +keywords: bzlmod --- title: 'Bzlmod Migration Guide' --- -Due to the [shortcomings of WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate your project to Bzlmod and drop WORKSPACE for managing external dependencies. + + +Due to the [shortcomings of +WORKSPACE](/external/overview#workspace-shortcomings), Bzlmod is replacing the +legacy WORKSPACE system. The WORKSPACE file is already disabled in Bazel 8 (late +2024) and will be removed in Bazel 9 (late 2025). This guide helps you migrate +your project to Bzlmod and drop WORKSPACE for managing external dependencies. ## Why migrate to Bzlmod? -- There are many [advantages](overview#benefits) compared to the legacy WORKSPACE system, which helps to ensure a healthy growth of the Bazel ecosystem. +* There are many [advantages](overview#benefits) compared to the legacy + WORKSPACE system, which helps to ensure a healthy growth of the Bazel + ecosystem. -- If your project is a dependency of other projects, migrating to Bzlmod will unblock their migration and make it easier for them to depend on your project. +* If your project is a dependency of other projects, migrating to Bzlmod will + unblock their migration and make it easier for them to depend on your + project. -- Migration to Bzlmod is a necessary step in order to use future Bazel versions (mandatory in Bazel 9). +* Migration to Bzlmod is a necessary step in order to use future Bazel + versions (mandatory in Bazel 9). ## How to migrate to Bzlmod? Recommended migration process: -1. Use [migration tool](/external/migration_tool) as a helper tool to streamline the migration process as much as possible. -2. If there are errors left after using the migration tool, resolve them manually. For understanding the main differences between concepts inside `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus Bzlmod](#workspace-vs-bzlmod) section. +1. Use [migration tool](/external/migration_tool) as a helper tool to + streamline the migration process as much as possible. +2. If there are errors left after using the migration tool, resolve them + manually. For understanding the main differences between concepts inside + `WORKSPACE` and `MODULE.bazel` files, check [WORKSPACE versus + Bzlmod](#workspace-vs-bzlmod) section. ## WORKSPACE vs Bzlmod -Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This section explains how to migrate from specific WORKSPACE functionalities to Bzlmod. +Bazel's WORKSPACE and Bzlmod offer similar features with different syntax. This +section explains how to migrate from specific WORKSPACE functionalities to +Bzlmod. ### Define the root of a Bazel workspace -The WORKSPACE file marks the source root of a Bazel project, this responsibility is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at your workspace root, maybe with comments like: +The WORKSPACE file marks the source root of a Bazel project, this responsibility +is replaced by MODULE.bazel in Bazel version 6.3 and later. With Bazel versions +prior to 6.3, there should still be a `WORKSPACE` or `WORKSPACE.bazel` file at +your workspace root, maybe with comments like: -- **WORKSPACE** +* **WORKSPACE** - ```python - # This file marks the root of the Bazel workspace. - # See MODULE.bazel for external dependencies setup. - ``` + ```python + # This file marks the root of the Bazel workspace. + # See MODULE.bazel for external dependencies setup. + ``` ### Enable Bzlmod in your bazelrc -`.bazelrc` lets you set flags that apply every time your run Bazel. To enable Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so it applies to every command: +`.bazelrc` lets you set flags that apply every time your run Bazel. To enable +Bzlmod, use the `--enable_bzlmod` flag, and apply it to the `common` command so +it applies to every command: -- **.bazelrc** +* **.bazelrc** - ``` - # Enable Bzlmod for every Bazel command - common --enable_bzlmod - ``` + ``` + # Enable Bzlmod for every Bazel command + common --enable_bzlmod + ``` ### Specify repository name for your workspace -- **WORKSPACE** +* **WORKSPACE** - The [`workspace`](/rules/lib/globals/workspace#workspace) function is used to specify a repository name for your workspace. This allows a target `//foo:bar` in the workspace to be referenced as `@<workspace name>//foo:bar`. If not specified, the default repository name for your workspace is `__main__`. + The [`workspace`](/rules/lib/globals/workspace#workspace) function is used + to specify a repository name for your workspace. This allows a target + `//foo:bar` in the workspace to be referenced as `@//foo:bar`. If not specified, the default repository name for your + workspace is `__main__`. - ```python - ## WORKSPACE - workspace(name = "com_foo_bar") - ``` + ```python + ## WORKSPACE + workspace(name = "com_foo_bar") + ``` -- **Bzlmod** +* **Bzlmod** - It's recommended to reference targets in the same workspace with the `//foo:bar` syntax without `@<repo name>`. But if you do need the old syntax , you can use the module name specified by the [`module`](/rules/lib/globals/module#module) function as the repository name. If the module name is different from the needed repository name, you can use `repo_name` attribute of the [`module`](/rules/lib/globals/module#module) function to override the repository name. + It's recommended to reference targets in the same workspace with the + `//foo:bar` syntax without `@`. But if you do need the old syntax + , you can use the module name specified by the + [`module`](/rules/lib/globals/module#module) function as the repository + name. If the module name is different from the needed repository name, you + can use `repo_name` attribute of the + [`module`](/rules/lib/globals/module#module) function to override the + repository name. - ```python - ## MODULE.bazel - module( - name = "bar", - repo_name = "com_foo_bar", - ) - ``` + ```python + ## MODULE.bazel + module( + name = "bar", + repo_name = "com_foo_bar", + ) + ``` ### Fetch external dependencies as Bazel modules -If your dependency is a Bazel project, you should be able to depend on it as a Bazel module when it also adopts Bzlmod. +If your dependency is a Bazel project, you should be able to depend on it as a +Bazel module when it also adopts Bzlmod. -- **WORKSPACE** +* **WORKSPACE** - With WORKSPACE, it's common to use the [`http_archive`](/rules/lib/repo/http#http_archive) or [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to download the sources of the Bazel project. + With WORKSPACE, it's common to use the + [`http_archive`](/rules/lib/repo/http#http_archive) or + [`git_repository`](/rules/lib/repo/git#git_repository) repository rules to + download the sources of the Bazel project. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "bazel_skylib", - urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], - sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", - ) - load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") - bazel_skylib_workspace() + http_archive( + name = "bazel_skylib", + urls = ["https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.2/bazel-skylib-1.4.2.tar.gz"], + sha256 = "66ffd9315665bfaafc96b52278f57c7e2dd09f5ede279ea6d39b2be471e7e3aa", + ) + load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") + bazel_skylib_workspace() - http_archive( - name = "rules_java", - urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], - sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", - ) - load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") - rules_java_dependencies() - rules_java_toolchains() - ``` + http_archive( + name = "rules_java", + urls = ["https://github.com/bazelbuild/rules_java/releases/download/6.1.1/rules_java-6.1.1.tar.gz"], + sha256 = "76402a50ae6859d50bd7aed8c1b8ef09dae5c1035bb3ca7d276f7f3ce659818a", + ) + load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") + rules_java_dependencies() + rules_java_toolchains() + ``` - As you can see, it's a common pattern that users need to load transitive dependencies from a macro of the dependency. Assume both `bazel_skylib` and `rules_java` depends on `platform`, the exact version of the `platform` dependency is determined by the order of the macros. + As you can see, it's a common pattern that users need to load transitive + dependencies from a macro of the dependency. Assume both `bazel_skylib` and + `rules_java` depends on `platform`, the exact version of the `platform` + dependency is determined by the order of the macros. -- **Bzlmod** +* **Bzlmod** - With Bzlmod, as long as your dependency is available in [Bazel Central Registry](https://registry.bazel.build) or your custom [Bazel registry](/external/registry), you can simply depend on it with a [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. + With Bzlmod, as long as your dependency is available in [Bazel Central + Registry](https://registry.bazel.build) or your custom [Bazel + registry](/external/registry), you can simply depend on it with a + [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive. - ```python - ## MODULE.bazel - bazel_dep(name = "bazel_skylib", version = "1.4.2") - bazel_dep(name = "rules_java", version = "6.1.1") - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "bazel_skylib", version = "1.4.2") + bazel_dep(name = "rules_java", version = "6.1.1") + ``` - Bzlmod resolves Bazel module dependencies transitively using the [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal required version of `platform` is selected automatically. + Bzlmod resolves Bazel module dependencies transitively using the + [MVS](https://research.swtch.com/vgo-mvs) algorithm. Therefore, the maximal + required version of `platform` is selected automatically. ### Override a dependency as a Bazel module -As the root module, you can override Bazel module dependencies in different ways. +As the root module, you can override Bazel module dependencies in different +ways. -Please read the [overrides](/external/module#overrides) section for more information. +Please read the [overrides](/external/module#overrides) section for more +information. -You can find some example usages in the [examples](https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module) repository. +You can find some example usages in the +[examples][override-examples] +repository. + +[override-examples]: https://github.com/bazelbuild/examples/blob/main/bzlmod/02-override_bazel_module ### Fetch external dependencies with module extensions -If your dependency is not a Bazel project or not yet available in any Bazel registry, you can introduce it using [`use_repo_rule`](/external/module#use_repo_rule) or [module extensions](/external/extension). +If your dependency is not a Bazel project or not yet available in any Bazel +registry, you can introduce it using +[`use_repo_rule`](/external/module#use_repo_rule) or [module +extensions](/external/extension). -- **WORKSPACE** +* **WORKSPACE** - Download a file using the [`http_file`](/rules/lib/repo/http#http_file) repository rule. + Download a file using the [`http_file`](/rules/lib/repo/http#http_file) + repository rule. - ```python - ## WORKSPACE - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + ```python + ## WORKSPACE + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel file to directly instantiate repos: + With Bzlmod, you can use the `use_repo_rule` directive in your MODULE.bazel + file to directly instantiate repos: - ```python - ## MODULE.bazel - http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## MODULE.bazel + http_file = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Under the hood, this is implemented using a module extension. If you need to perform more complex logic than simply invoking a repo rule, you could also implement a module extension yourself. You'll need to move the definition into a `.bzl` file, which also lets you share the definition between WORKSPACE and Bzlmod during the migration period. + Under the hood, this is implemented using a module extension. If you need to + perform more complex logic than simply invoking a repo rule, you could also + implement a module extension yourself. You'll need to move the definition + into a `.bzl` file, which also lets you share the definition between + WORKSPACE and Bzlmod during the migration period. - ```python - ## repositories.bzl - load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - def my_data_dependency(): - http_file( - name = "data_file", - url = "http://example.com/file", - sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", - ) - ``` + ```python + ## repositories.bzl + load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + def my_data_dependency(): + http_file( + name = "data_file", + url = "http://example.com/file", + sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ) + ``` - Implement a module extension to load the dependencies macro. You can define it in the same `.bzl` file of the macro, but to keep compatibility with older Bazel versions, it's better to define it in a separate `.bzl` file. + Implement a module extension to load the dependencies macro. You can define + it in the same `.bzl` file of the macro, but to keep compatibility with + older Bazel versions, it's better to define it in a separate `.bzl` file. - ```python - ## extensions.bzl - load("//:repositories.bzl", "my_data_dependency") - def _non_module_dependencies_impl(_ctx): - my_data_dependency() + ```python + ## extensions.bzl + load("//:repositories.bzl", "my_data_dependency") + def _non_module_dependencies_impl(_ctx): + my_data_dependency() - non_module_dependencies = module_extension( - implementation = _non_module_dependencies_impl, - ) - ``` + non_module_dependencies = module_extension( + implementation = _non_module_dependencies_impl, + ) + ``` - To make the repository visible to the root project, you should declare the usages of the module extension and the repository in the MODULE.bazel file. + To make the repository visible to the root project, you should declare the + usages of the module extension and the repository in the MODULE.bazel file. - ```python - ## MODULE.bazel - non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") - use_repo(non_module_dependencies, "data_file") - ``` + ```python + ## MODULE.bazel + non_module_dependencies = use_extension("//:extensions.bzl", "non_module_dependencies") + use_repo(non_module_dependencies, "data_file") + ``` ### Resolve conflict external dependencies with module extension -A project can provide a macro that introduces external repositories based on inputs from its callers. But what if there are multiple callers in the dependency graph and they cause a conflict? +A project can provide a macro that introduces external repositories based on +inputs from its callers. But what if there are multiple callers in the +dependency graph and they cause a conflict? -Assume the project `foo` provides the following macro which takes `version` as an argument. +Assume the project `foo` provides the following macro which takes `version` as +an argument. ```python -## repositories.bzl in foo +## repositories.bzl in foo {:#repositories.bzl-foo} load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") def data_deps(version = "1.0"): http_file( @@ -204,239 +271,297 @@ def data_deps(version = "1.0"): ) ``` -- **WORKSPACE** - - With WORKSPACE, you can load the macro from `@foo` and specify the version of the data dependency you need. Assume you have another dependency `@bar`, which also depends on `@foo` but requires a different version of the data dependency. +* **WORKSPACE** - ```python - ## WORKSPACE + With WORKSPACE, you can load the macro from `@foo` and specify the version + of the data dependency you need. Assume you have another dependency `@bar`, + which also depends on `@foo` but requires a different version of the data + dependency. - # Introduce @foo and @bar. - ... - - load("@foo//:repositories.bzl", "data_deps") - data_deps(version = "2.0") - - load("@bar//:repositories.bzl", "bar_deps") - bar_deps() # -> which calls data_deps(version = "3.0") - ``` + ```python + ## WORKSPACE - In this case, the end user must carefully adjust the order of macros in the WORKSPACE to get the version they need. This is one of the biggest pain points with WORKSPACE since it doesn't really provide a sensible way to resolve dependencies. + # Introduce @foo and @bar. + ... -- **Bzlmod** + load("@foo//:repositories.bzl", "data_deps") + data_deps(version = "2.0") - With Bzlmod, the author of project `foo` can use module extension to resolve conflicts. For example, let's assume it makes sense to always select the maximal required version of the data dependency among all Bazel modules. + load("@bar//:repositories.bzl", "bar_deps") + bar_deps() # -> which calls data_deps(version = "3.0") + ``` - ```python - ## extensions.bzl in foo - load("//:repositories.bzl", "data_deps") + In this case, the end user must carefully adjust the order of macros in the + WORKSPACE to get the version they need. This is one of the biggest pain + points with WORKSPACE since it doesn't really provide a sensible way to + resolve dependencies. - data = tag_class(attrs={"version": attr.string()}) +* **Bzlmod** - def _data_deps_extension_impl(module_ctx): - # Select the maximal required version in the dependency graph. - version = "1.0" - for mod in module_ctx.modules: - for data in mod.tags.data: - version = max(version, data.version) - data_deps(version) + With Bzlmod, the author of project `foo` can use module extension to resolve + conflicts. For example, let's assume it makes sense to always select the + maximal required version of the data dependency among all Bazel modules. - data_deps_extension = module_extension( - implementation = _data_deps_extension_impl, - tag_classes = {"data": data}, - ) - ``` + ```python + ## extensions.bzl in foo + load("//:repositories.bzl", "data_deps") + + data = tag_class(attrs={"version": attr.string()}) + + def _data_deps_extension_impl(module_ctx): + # Select the maximal required version in the dependency graph. + version = "1.0" + for mod in module_ctx.modules: + for data in mod.tags.data: + version = max(version, data.version) + data_deps(version) + + data_deps_extension = module_extension( + implementation = _data_deps_extension_impl, + tag_classes = {"data": data}, + ) + ``` - ```python - ## MODULE.bazel in bar - bazel_dep(name = "foo", version = "1.0") + ```python + ## MODULE.bazel in bar + bazel_dep(name = "foo", version = "1.0") - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "3.0") - use_repo(foo_data_deps, "data_file") - ``` + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "3.0") + use_repo(foo_data_deps, "data_file") + ``` - ```python - ## MODULE.bazel in root module - bazel_dep(name = "foo", version = "1.0") - bazel_dep(name = "bar", version = "1.0") + ```python + ## MODULE.bazel in root module + bazel_dep(name = "foo", version = "1.0") + bazel_dep(name = "bar", version = "1.0") - foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") - foo_data_deps.data(version = "2.0") - use_repo(foo_data_deps, "data_file") - ``` + foo_data_deps = use_extension("@foo//:extensions.bzl", "data_deps_extension") + foo_data_deps.data(version = "2.0") + use_repo(foo_data_deps, "data_file") + ``` - In this case, the root module requires data version `2.0`, while its dependency `bar` requires `3.0`. The module extension in `foo` can correctly resolve this conflict and automatically select version `3.0` for the data dependency. + In this case, the root module requires data version `2.0`, while its + dependency `bar` requires `3.0`. The module extension in `foo` can correctly + resolve this conflict and automatically select version `3.0` for the data + dependency. ### Integrate third party package manager -Following the last section, since module extension provides a way to collect information from the dependency graph, perform custom logic to resolve dependencies and call repository rules to introduce external repositories, this provides a great way for rules authors to enhance the rulesets that integrate package managers for specific languages. +Following the last section, since module extension provides a way to collect +information from the dependency graph, perform custom logic to resolve +dependencies and call repository rules to introduce external repositories, this +provides a great way for rules authors to enhance the rulesets that integrate +package managers for specific languages. -Please read the [module extensions](/external/extension) page to learn more about how to use module extensions. +Please read the [module extensions](/external/extension) page to learn more +about how to use module extensions. -Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies from different package managers: +Here is a list of the rulesets that already adopted Bzlmod to fetch dependencies +from different package managers: -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) -- [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) -- [rules\_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) - -A minimal example that integrates a pseudo package manager is available at the [examples](https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager) repository. - -### Detect toolchains on the host machine +- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) +- [rules_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) +- [rules_python](https://github.com/bazelbuild/rules_python/blob/main/BZLMOD_SUPPORT.md) -When Bazel build rules need to detect what toolchains are available on your host machine, they use repository rules to inspect the host machine and generate toolchain info as external repositories. +A minimal example that integrates a pseudo package manager is available at the +[examples][pkg-mgr-example] +repository. -- **WORKSPACE** +[pkg-mgr-example]: https://github.com/bazelbuild/examples/tree/main/bzlmod/05-integrate_third_party_package_manager - Given the following repository rule to detect a shell toolchain. +### Detect toolchains on the host machine - ```python - ## local_config_sh.bzl - def _sh_config_rule_impl(repository_ctx): - sh_path = get_sh_path_from_env("SH_BIN_PATH") +When Bazel build rules need to detect what toolchains are available on your host +machine, they use repository rules to inspect the host machine and generate +toolchain info as external repositories. - if not sh_path: - sh_path = detect_sh_from_path() +* **WORKSPACE** - if not sh_path: - sh_path = "/shell/binary/not/found" + Given the following repository rule to detect a shell toolchain. - repository_ctx.file("BUILD", """ - load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") - sh_toolchain( - name = "local_sh", - path = "{sh_path}", - visibility = ["//visibility:public"], - ) - toolchain( - name = "local_sh_toolchain", - toolchain = ":local_sh", - toolchain_type = "@bazel_tools//tools/sh:toolchain_type", - ) - """.format(sh_path = sh_path)) + ```python + ## local_config_sh.bzl + def _sh_config_rule_impl(repository_ctx): + sh_path = get_sh_path_from_env("SH_BIN_PATH") + + if not sh_path: + sh_path = detect_sh_from_path() + + if not sh_path: + sh_path = "/shell/binary/not/found" + + repository_ctx.file("BUILD", """ + load("@bazel_tools//tools/sh:sh_toolchain.bzl", "sh_toolchain") + sh_toolchain( + name = "local_sh", + path = "{sh_path}", + visibility = ["//visibility:public"], + ) + toolchain( + name = "local_sh_toolchain", + toolchain = ":local_sh", + toolchain_type = "@bazel_tools//tools/sh:toolchain_type", + ) + """.format(sh_path = sh_path)) - sh_config_rule = repository_rule( - environ = ["SH_BIN_PATH"], - local = True, - implementation = _sh_config_rule_impl, - ) - ``` + sh_config_rule = repository_rule( + environ = ["SH_BIN_PATH"], + local = True, + implementation = _sh_config_rule_impl, + ) + ``` - You can load the repository rule in WORKSPACE. + You can load the repository rule in WORKSPACE. - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can introduce the same repository using a module extension, which is similar to introducing the `@data_file` repository in the last section. + With Bzlmod, you can introduce the same repository using a module extension, + which is similar to introducing the `@data_file` repository in the last + section. - ``` - ## local_config_sh_extension.bzl - load("//:local_config_sh.bzl", "sh_config_rule") + ``` + ## local_config_sh_extension.bzl + load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_extension = module_extension( - implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), - ) - ``` + sh_config_extension = module_extension( + implementation = lambda ctx: sh_config_rule(name = "local_config_sh"), + ) + ``` - Then use the extension in the MODULE.bazel file. + Then use the extension in the MODULE.bazel file. - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - ``` + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + ``` ### Register toolchains & execution platforms -Following the last section, after introducing a repository hosting toolchain information (e.g. `local_config_sh`), you probably want to register the toolchain. +Following the last section, after introducing a repository hosting toolchain +information (e.g. `local_config_sh`), you probably want to register the +toolchain. + +* **WORKSPACE** -- **WORKSPACE** + With WORKSPACE, you can register the toolchain in the following ways. - With WORKSPACE, you can register the toolchain in the following ways. + 1. You can register the toolchain the `.bzl` file and load the macro in the + WORKSPACE file. - 1. You can register the toolchain the `.bzl` file and load the macro in the WORKSPACE file. + ```python + ## local_config_sh.bzl + def sh_configure(): + sh_config_rule(name = "local_config_sh") + native.register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## local_config_sh.bzl - def sh_configure(): - sh_config_rule(name = "local_config_sh") - native.register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_configure") + sh_configure() + ``` - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_configure") - sh_configure() - ``` + 2. Or register the toolchain in the WORKSPACE file directly. - 2. Or register the toolchain in the WORKSPACE file directly. + ```python + ## WORKSPACE + load("//:local_config_sh.bzl", "sh_config_rule") + sh_config_rule(name = "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## WORKSPACE - load("//:local_config_sh.bzl", "sh_config_rule") - sh_config_rule(name = "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +* **Bzlmod** -- **Bzlmod** + With Bzlmod, the + [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and + [`register_execution_platforms`][register_execution_platforms] + APIs are only available in the MODULE.bazel file. You cannot call + `native.register_toolchains` in a module extension. - With Bzlmod, the [`register_toolchains`](/rules/lib/globals/module#register_toolchains) and [`register_execution_platforms`](/rules/lib/globals/module#register_execution_platforms) APIs are only available in the MODULE.bazel file. You cannot call `native.register_toolchains` in a module extension. + ```python + ## MODULE.bazel + sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") + use_repo(sh_config_ext, "local_config_sh") + register_toolchains("@local_config_sh//:local_sh_toolchain") + ``` - ```python - ## MODULE.bazel - sh_config_ext = use_extension("//:local_config_sh_extension.bzl", "sh_config_extension") - use_repo(sh_config_ext, "local_config_sh") - register_toolchains("@local_config_sh//:local_sh_toolchain") - ``` +The toolchains and execution platforms registered in `WORKSPACE`, +`WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this +order of precedence during toolchain selection (from highest to lowest): -The toolchains and execution platforms registered in `WORKSPACE`, `WORKSPACE.bzlmod` and each Bazel module's `MODULE.bazel` file follow this order of precedence during toolchain selection (from highest to lowest): +1. toolchains and execution platforms registered in the root module's + `MODULE.bazel` file. +2. toolchains and execution platforms registered in the `WORKSPACE` or + `WORKSPACE.bzlmod` file. +3. toolchains and execution platforms registered by modules that are + (transitive) dependencies of the root module. +4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` + [suffix](/external/migration#builtin-default-deps). -1. toolchains and execution platforms registered in the root module's `MODULE.bazel` file. -2. toolchains and execution platforms registered in the `WORKSPACE` or `WORKSPACE.bzlmod` file. -3. toolchains and execution platforms registered by modules that are (transitive) dependencies of the root module. -4. when not using `WORKSPACE.bzlmod`: toolchains registered in the `WORKSPACE` [suffix](/external/migration#builtin-default-deps). +[register_execution_platforms]: /rules/lib/globals/module#register_execution_platforms ### Introduce local repositories -You may need to introduce a dependency as a local repository when you need a local version of the dependency for debugging or you want to incorporate a directory in your workspace as external repository. +You may need to introduce a dependency as a local repository when you need a +local version of the dependency for debugging or you want to incorporate a +directory in your workspace as external repository. -- **WORKSPACE** +* **WORKSPACE** - With WORKSPACE, this is achieved by two native repository rules, [`local_repository`](/reference/be/workspace#local_repository) and [`new_local_repository`](/reference/be/workspace#new_local_repository). + With WORKSPACE, this is achieved by two native repository rules, + [`local_repository`](/reference/be/workspace#local_repository) and + [`new_local_repository`](/reference/be/workspace#new_local_repository). - ```python - ## WORKSPACE - local_repository( - name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## WORKSPACE + local_repository( + name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` -- **Bzlmod** +* **Bzlmod** - With Bzlmod, you can use [`local_path_override`](/rules/lib/globals/module#local_path_override) to override a module with a local path. + With Bzlmod, you can use + [`local_path_override`](/rules/lib/globals/module#local_path_override) to + override a module with a local path. - ```python - ## MODULE.bazel - bazel_dep(name = "rules_java") - local_path_override( - module_name = "rules_java", - path = "/Users/bazel_user/workspace/rules_java", - ) - ``` + ```python + ## MODULE.bazel + bazel_dep(name = "rules_java") + local_path_override( + module_name = "rules_java", + path = "/Users/bazel_user/workspace/rules_java", + ) + ``` - Note: With `local_path_override`, you can only introduce a local directory as a Bazel module, which means it should have a MODULE.bazel file and its transitive dependencies are taken into consideration during dependency resolution. In addition, all module override directives can only be used by the root module. + Note: With `local_path_override`, you can only introduce a local directory + as a Bazel module, which means it should have a MODULE.bazel file and its + transitive dependencies are taken into consideration during dependency + resolution. In addition, all module override directives can only be used by + the root module. - It is also possible to introduce a local repository with module extension. However, you cannot call `native.local_repository` in module extension, there is ongoing effort on starlarkifying all native repository rules (check [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). Then you can call the corresponding starlark `local_repository` in a module extension. It's also trivial to implement a custom version of `local_repository` repository rule if this is a blocking issue for you. + It is also possible to introduce a local repository with module extension. + However, you cannot call `native.local_repository` in module extension, + there is ongoing effort on starlarkifying all native repository rules (check + [#18285](https://github.com/bazelbuild/bazel/issues/18285) for progress). + Then you can call the corresponding starlark `local_repository` in a module + extension. It's also trivial to implement a custom version of + `local_repository` repository rule if this is a blocking issue for you. ### Bind targets -The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and not supported in Bzlmod. It was introduced to give a target an alias in the special `//external` package. All users depending on this should migrate away. +The [`bind`](/reference/be/workspace#bind) rule in WORKSPACE is deprecated and +not supported in Bzlmod. It was introduced to give a target an alias in the +special `//external` package. All users depending on this should migrate away. For example, if you have @@ -448,180 +573,290 @@ bind( ) ``` -This allows other targets to depend on `//external:openssl`. You can migrate away from this by: +This allows other targets to depend on `//external:openssl`. You can migrate +away from this by: -- Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. +* Replace all usages of `//external:openssl` with `@my-ssl//src:openssl-lib`. + * Tip: Use `bazel query --output=build --noenable_bzlmod + --enable_workspace [target]` command to find relevant info + about the target. - - Tip: Use `bazel query --output=build --noenable_bzlmod --enable_workspace [target]` command to find relevant info about the target. +* Or use the [`alias`](/reference/be/general#alias) build rule + * Define the following target in a package (e.g. `//third_party`) -- Or use the [`alias`](/reference/be/general#alias) build rule - - - Define the following target in a package (e.g. `//third_party`) - - ```python - ## third_party/BUILD - alias( - name = "openssl", - actual = "@my-ssl//src:openssl-lib", - ) - ``` + ```python + ## third_party/BUILD + alias( + name = "openssl", + actual = "@my-ssl//src:openssl-lib", + ) + ``` - - Replace all usages of `//external:openssl` with `//third_party:openssl`. + * Replace all usages of `//external:openssl` with `//third_party:openssl`. ### Fetch versus Sync -Fetch and sync commands are used to download external repos locally and keep them updated. Sometimes also to allow building offline using the `--nofetch` flag after fetching all repos needed for a build. +Fetch and sync commands are used to download external repos locally and keep +them updated. Sometimes also to allow building offline using the `--nofetch` +flag after fetching all repos needed for a build. -- **WORKSPACE** +* **WORKSPACE** - Sync performs a force fetch for all repositories, or for a specific configured set of repos, while fetch is *only* used to fetch for a specific target. + Sync performs a force fetch for all repositories, or for a specific + configured set of repos, while fetch is _only_ used to fetch for a specific + target. -- **Bzlmod** +* **Bzlmod** - The sync command is no longer applicable, but fetch offers [various options](/reference/command-line-reference#fetch-options). You can fetch a target, a repository, a set of configured repos or all repositories involved in your dependency resolution and module extensions. The fetch result is cached and to force a fetch you must include the `--force` option during the fetch process. + The sync command is no longer applicable, but fetch offers + [various options](/reference/command-line-reference#fetch-options). + You can fetch a target, a repository, a set of configured repos or all + repositories involved in your dependency resolution and module extensions. + The fetch result is cached and to force a fetch you must include the + `--force` option during the fetch process. ## Manual migration -This section provides useful information and guidance for your **manual** Bzlmod migration process. For more automatized migration process, check [recommended migration process](#how-migrate-to-bzlmod) section. +This section provides useful information and guidance for your **manual** Bzlmod +migration process. For more automatized migration process, check [recommended +migration process](#how-migrate-to-bzlmod) section. ### Know your dependencies in WORKSPACE -The first step of migration is to understand what dependencies you have. It could be hard to figure out what exact dependencies are introduced in the WORKSPACE file because transitive dependencies are often loaded with `*_deps` macros. +The first step of migration is to understand what dependencies you have. It +could be hard to figure out what exact dependencies are introduced in the +WORKSPACE file because transitive dependencies are often loaded with `*_deps` +macros. #### Inspect external dependency with workspace resolved file -Fortunately, the flag [`--experimental_repository_resolved_file`](/reference/command-line-reference#flag--experimental_repository_resolved_file) can help. This flag essentially generates a "lock file" of all fetched external dependencies in your last Bazel command. You can find more details in this [blog post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). +Fortunately, the flag +[`--experimental_repository_resolved_file`][resolved_file_flag] +can help. This flag essentially generates a "lock file" of all fetched external +dependencies in your last Bazel command. You can find more details in this [blog +post](https://blog.bazel.build/2018/07/09/bazel-sync-and-resolved-file.html). + +[resolved_file_flag]: /reference/command-line-reference#flag--experimental_repository_resolved_file It can be used in two ways: -1. To fetch info of external dependencies needed for building certain targets. +1. To fetch info of external dependencies needed for building certain targets. - ```shell - bazel clean --expunge - bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar - ``` + ```shell + bazel clean --expunge + bazel build --nobuild --experimental_repository_resolved_file=resolved.bzl //foo:bar + ``` -2. To fetch info of all external dependencies defined in the WORKSPACE file. +2. To fetch info of all external dependencies defined in the WORKSPACE file. - ```shell - bazel clean --expunge - bazel sync --experimental_repository_resolved_file=resolved.bzl - ``` + ```shell + bazel clean --expunge + bazel sync --experimental_repository_resolved_file=resolved.bzl + ``` - With the `bazel sync` command, you can fetch all dependencies defined in the WORKSPACE file, which include: + With the `bazel sync` command, you can fetch all dependencies defined in the + WORKSPACE file, which include: - - `bind` usages - - `register_toolchains` & `register_execution_platforms` usages + * `bind` usages + * `register_toolchains` & `register_execution_platforms` usages - However, if your project is cross platforms, bazel sync may break on certain platforms because some repository rules may only run correctly on supported platforms. + However, if your project is cross platforms, bazel sync may break on certain + platforms because some repository rules may only run correctly on supported + platforms. -After running the command, you should have information of your external dependencies in the `resolved.bzl` file. +After running the command, you should have information of your external +dependencies in the `resolved.bzl` file. #### Inspect external dependency with `bazel query` You may also know `bazel query` can be used for inspecting repository rules with ```shell -bazel query --output=build //external:<repo name> +bazel query --output=build //external: ``` -While it is more convenient and much faster, [bazel query can lie about external dependency version](https://github.com/bazelbuild/bazel/issues/12947), so be careful using it! Querying and inspecting external dependencies with Bzlmod is going to achieved by a [new subcommand](https://github.com/bazelbuild/bazel/issues/15365). +While it is more convenient and much faster, [bazel query can lie about +external dependency version](https://github.com/bazelbuild/bazel/issues/12947), +so be careful using it! Querying and inspecting external +dependencies with Bzlmod is going to achieved by a [new +subcommand](https://github.com/bazelbuild/bazel/issues/15365). #### Built-in default dependencies -If you check the file generated by `--experimental_repository_resolved_file`, you are going to find many dependencies that are not defined in your WORKSPACE. This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE file content to inject some default dependencies, which are usually required by native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With Bzlmod, those dependencies are introduced with a built-in module [`bazel_tools`](https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools) , which is a default dependency for every other Bazel module. +If you check the file generated by `--experimental_repository_resolved_file`, +you are going to find many dependencies that are not defined in your WORKSPACE. +This is because Bazel in fact adds prefixes and suffixes to the user's WORKSPACE +file content to inject some default dependencies, which are usually required by +native rules (e.g. `@bazel_tools`, `@platforms` and `@remote_java_tools`). With +Bzlmod, those dependencies are introduced with a built-in module +[`bazel_tools`][bazel_tools] , which is a default dependency for every other +Bazel module. + +[bazel_tools]: https://github.com/bazelbuild/bazel/blob/master/src/MODULE.tools ### Hybrid mode for gradual migration -Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies from the WORKSPACE file to Bzlmod to be a gradual process. +Bzlmod and WORKSPACE can work side by side, which allows migrating dependencies +from the WORKSPACE file to Bzlmod to be a gradual process. -Note: In practice, loading "\*\_deps" macros in WORKSPACE often causes confusions with Bzlmod dependencies, therefore we recommend starting with a WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. +Note: In practice, loading "*_deps" macros in WORKSPACE often causes confusions +with Bzlmod dependencies, therefore we recommend starting with a +WORKSPACE.bzlmod file and avoid loading transitive dependencies with macros. #### WORKSPACE.bzlmod -During the migration, Bazel users may need to switch between builds with and without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the process smoother. +During the migration, Bazel users may need to switch between builds with and +without Bzlmod enabled. WORKSPACE.bzlmod support is implemented to make the +process smoother. -WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, if a WORKSPACE.bzlmod file also exists at the workspace root: +WORKSPACE.bzlmod has the exact same syntax as WORKSPACE. When Bzlmod is enabled, +if a WORKSPACE.bzlmod file also exists at the workspace root: -- `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. -- No [prefixes or suffixes](/external/migration#builtin-default-deps) are added to the WORKSPACE.bzlmod file. +* `WORKSPACE.bzlmod` takes effect and the content of `WORKSPACE` is ignored. +* No [prefixes or suffixes](/external/migration#builtin-default-deps) are + added to the WORKSPACE.bzlmod file. Using the WORKSPACE.bzlmod file can make the migration easier because: -- When Bzlmod is disabled, you fall back to fetching dependencies from the original WORKSPACE file. -- When Bzlmod is enabled, you can better track what dependencies are left to migrate with WORKSPACE.bzlmod. +* When Bzlmod is disabled, you fall back to fetching dependencies from the + original WORKSPACE file. +* When Bzlmod is enabled, you can better track what dependencies are left to + migrate with WORKSPACE.bzlmod. #### Repository visibility -Bzlmod is able to control which other repositories are visible from a given repository, check [repository names and strict deps](/external/module#repository_names_and_strict_deps) for more details. +Bzlmod is able to control which other repositories are visible from a given +repository, check [repository names and strict +deps](/external/module#repository_names_and_strict_deps) for more details. -Here is a summary of repository visibilities from different types of repositories when also taking WORKSPACE into consideration. +Here is a summary of repository visibilities from different types of +repositories when also taking WORKSPACE into consideration. -| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | -| ---------------------- | ------------------ | ----------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------ | -| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | -| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | -| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | -| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | +| | From the main repo | From Bazel module repos | From module extension repos | From WORKSPACE repos | +|----------------|--------------------|-------------------------|---------------------------------------------------------------------------------------------------------------------|----------------------| +| The main repo | Visible | If the root module is a direct dependency | If the root module is a direct dependency of the module hosting the module extension | Visible | +| Bazel module repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension | Direct deps of the root module | +| Module extension repos | Direct deps | Direct deps | Direct deps of the module hosting the module extension + all repos generated by the same module extension | Direct deps of the root module | +| WORKSPACE Repos | All visible | Not visible | Not visible | All visible | -Note: For the root module, if a repository `@foo` is defined in WORKSPACE and `@foo` is also used as an [apparent repository name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` refers to the one introduced in MODULE.bazel. +Note: For the root module, if a repository `@foo` is defined in WORKSPACE and +`@foo` is also used as an [apparent repository +name](/external/overview#apparent-repo-name) in MODULE.bazel, then `@foo` +refers to the one introduced in MODULE.bazel. -Note: For a module extension generated repository `@bar`, if `@foo` is used as an [apparent repository name](/external/overview#apparent-repo-name) of another repository generated by the same module extension and direct dependencies of the module hosting the module extension, then for repository `@bar`, `@foo` refers to the latter. +Note: For a module extension generated repository `@bar`, if `@foo` is used as +an [apparent repository name](/external/overview#apparent-repo-name) of +another repository generated by the same module extension and direct +dependencies of the module hosting the module extension, then for repository +`@bar`, `@foo` refers to the latter. ### Manual migration process A typical Bzlmod migration process can look like this: -1. Understand what dependencies you have in WORKSPACE. -2. Add an empty MODULE.bazel file at your project root. -3. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. -4. Build your targets with Bzlmod enabled and check which repository is missing. -5. Check the definition of the missing repository in the resolved dependency file. -6. Introduce the missing dependency as a Bazel module, through a module extension, or leave it in the WORKSPACE.bzlmod for later migration. -7. Go back to 4 and repeat until all dependencies are available. +1. Understand what dependencies you have in WORKSPACE. +1. Add an empty MODULE.bazel file at your project root. +1. Add an empty WORKSPACE.bzlmod file to override the WORKSPACE file content. +1. Build your targets with Bzlmod enabled and check which repository is + missing. +1. Check the definition of the missing repository in the resolved dependency + file. +1. Introduce the missing dependency as a Bazel module, through a module + extension, or leave it in the WORKSPACE.bzlmod for later migration. +1. Go back to 4 and repeat until all dependencies are available. ## Publish Bazel modules -If your Bazel project is a dependency for other projects, you can publish your project in the [Bazel Central Registry](https://registry.bazel.build/). +If your Bazel project is a dependency for other projects, you can publish your +project in the [Bazel Central Registry](https://registry.bazel.build/). + +To be able to check in your project in the BCR, you need a source archive URL of +the project. Take note of a few things when creating the source archive: -To be able to check in your project in the BCR, you need a source archive URL of the project. Take note of a few things when creating the source archive: +* **Make sure the archive is pointing to a specific version.** -- **Make sure the archive is pointing to a specific version.** + The BCR can only accept versioned source archives because Bzlmod needs to + conduct version comparison during dependency resolution. - The BCR can only accept versioned source archives because Bzlmod needs to conduct version comparison during dependency resolution. +* **Make sure the archive URL is stable.** -- **Make sure the archive URL is stable.** + Bazel verifies the content of the archive by a hash value, so you should + make sure the checksum of the downloaded file never changes. If the URL is + from GitHub, please create and upload a release archive in the release page. + GitHub isn't going to guarantee the checksum of source archives generated on + demand. In short, URLs in the form of + `https://github.com///releases/download/...` is considered stable + while `https://github.com///archive/...` is not. Check [GitHub + Archive Checksum + Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) + for more context. - Bazel verifies the content of the archive by a hash value, so you should make sure the checksum of the downloaded file never changes. If the URL is from GitHub, please create and upload a release archive in the release page. GitHub isn't going to guarantee the checksum of source archives generated on demand. In short, URLs in the form of `https://github.com/<org>/<repo>/releases/download/...` is considered stable while `https://github.com/<org>/<repo>/archive/...` is not. Check [GitHub Archive Checksum Outage](https://blog.bazel.build/2023/02/15/github-archive-checksum.html) for more context. +* **Make sure the source tree follows the layout of the original repository.** -- **Make sure the source tree follows the layout of the original repository.** + In case your repository is very large and you want to create a distribution + archive with reduced size by stripping out unnecessary sources, please make + sure the stripped source tree is a subset of the original source tree. This + makes it easier for end users to override the module to a non-release + version by [`archive_override`](/rules/lib/globals/module#archive_override) + and [`git_override`](/rules/lib/globals/module#git_override). - In case your repository is very large and you want to create a distribution archive with reduced size by stripping out unnecessary sources, please make sure the stripped source tree is a subset of the original source tree. This makes it easier for end users to override the module to a non-release version by [`archive_override`](/rules/lib/globals/module#archive_override) and [`git_override`](/rules/lib/globals/module#git_override). +* **Include a test module in a subdirectory that tests your most common + APIs.** -- **Include a test module in a subdirectory that tests your most common APIs.** + A test module is a Bazel project with its own WORKSPACE and MODULE.bazel + file located in a subdirectory of the source archive which depends on the + actual module to be published. It should contain examples or some + integration tests that cover your most common APIs. Check + [test module][test_module] to learn how to set it up. - A test module is a Bazel project with its own WORKSPACE and MODULE.bazel file located in a subdirectory of the source archive which depends on the actual module to be published. It should contain examples or some integration tests that cover your most common APIs. Check [test module](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module) to learn how to set it up. +[test_module]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#test-module -When you have your source archive URL ready, follow the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module) to submit your module to the BCR with a GitHub Pull Request. +When you have your source archive URL ready, follow the [BCR contribution +guidelines][bcr_contrib_guide] to submit your module to the BCR with a GitHub +Pull Request. -It is **highly recommended** to set up the [Publish to BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your repository to automate the process of submitting your module to the BCR. +[bcr_contrib_guide]: https://github.com/bazelbuild/bazel-central-registry/tree/main/docs#contribute-a-bazel-module + +It is **highly recommended** to set up the [Publish to +BCR](https://github.com/bazel-contrib/publish-to-bcr) GitHub App for your +repository to automate the process of submitting your module to the BCR. ## Best practices -This section documents a few best practices you should follow for better managing your external dependencies. +This section documents a few best practices you should follow for better +managing your external dependencies. #### Split targets into different packages to avoid fetching unnecessary dependencies. -Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev dependencies for tests are forced to be fetched unnecessarily for building targets that don't need them. This is actually not Bzlmod specific, but following this practices makes it easier to specify dev dependencies correctly. +Check [#12835](https://github.com/bazelbuild/bazel/issues/12835), where dev +dependencies for tests are forced to be fetched unnecessarily for building +targets that don't need them. This is actually not Bzlmod specific, but +following this practices makes it easier to specify dev dependencies correctly. #### Specify dev dependencies -You can set the `dev_dependency` attribute to true for [`bazel_dep`](/rules/lib/globals/module#bazel_dep) and [`use_extension`](/rules/lib/globals/module#use_extension) directives so that they don't propagate to dependent projects. As the root module, you can use the [`--ignore_dev_dependency`](/reference/command-line-reference#flag--ignore_dev_dependency) flag to verify if your targets still build without dev dependencies and overrides. +You can set the `dev_dependency` attribute to true for +[`bazel_dep`](/rules/lib/globals/module#bazel_dep) and +[`use_extension`](/rules/lib/globals/module#use_extension) directives so that +they don't propagate to dependent projects. As the root module, you can use the +[`--ignore_dev_dependency`][ignore_dev_dep_flag] flag to verify if your targets +still build without dev dependencies and overrides. + +[ignore_dev_dep_flag]: /reference/command-line-reference#flag--ignore_dev_dependency + + ## Community migration progress -You can check the [Bazel Central Registry](https://registry.bazel.build) to find out if your dependencies are already available. Otherwise feel free to join this [GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to upvote or post the dependencies that are blocking your migration. +You can check the [Bazel Central Registry](https://registry.bazel.build) to find +out if your dependencies are already available. Otherwise feel free to join this +[GitHub discussion](https://github.com/bazelbuild/bazel/discussions/18329) to +upvote or post the dependencies that are blocking your migration. ## Report issues -Please check the [Bazel GitHub issue list](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod) for known Bzlmod issues. Feel free to file new issues or feature requests that can help unblock your migration! +Please check the [Bazel GitHub issue list][bzlmod_github_issue] for known Bzlmod +issues. Feel free to file new issues or feature requests that can help unblock +your migration! + +[bzlmod_github_issue]: https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Aarea-Bzlmod diff --git a/external/migration_tool.mdx b/external/migration_tool.mdx deleted file mode 100644 index 74a2b475..00000000 --- a/external/migration_tool.mdx +++ /dev/null @@ -1,642 +0,0 @@ ---- -title: 'Bzlmod Migration Tool' ---- - -To simplify the often complex process of moving from `WORKSPACE` to Bzlmod, it's highly recommended to use the [migration script](https://github.com/bazelbuild/bazel-central-registry/blob/main/tools/migrate_to_bzlmod.py). This helper tool automates many of the steps involved in migrating your external dependency management system. - -**Note**: If you want to try out the AI driven Bzlmod migration, check [Bzlmod Migration Agent Setup](https://github.com/bazelbuild/bazel-central-registry/tree/main/tools/code-agent). - -## Core Functionality - -The script's primary functions are: - -- **Collecting dependency information:** Analyzing your project's `WORKSPACE` file to identify external repositories used by specified build targets, using Bazel's [experimental\_repository\_resolved\_file](https://bazel.build/versions/8.2.0/reference/command-line-reference#flag--experimental_repository_resolved_file) flag to generate a resolved dependencies file containing this information. - -- **Identifying direct dependencies:** Using `bazel query` to determine which repositories are direct dependencies for the specified targets. - -- **Migrating to Bzlmod:** Translating relevant `WORKSPACE` dependencies into their Bzlmod equivalents. This is a two-step process: - - 1. Introduce all identified direct dependencies to the `MODULE.bazel` file. - 2. Build specified targets with Bzlmod enabled, then iteratively identify and fix recognizable errors. This step is needed since some dependencies might be missing in the first step. - -- **Generating a migration report:** Creating a `migration_info.md` file that documents the migration process. This report includes a list of direct dependencies, the generated Bzlmod declarations, and any manual steps that may be required to complete the migration. - -The migration tool supports: - -- Dependencies available in the Bazel Central Registry - -- User-defined custom repository rules - -- Package manager dependencies - - - Maven - - Go - - Python - -**Important Notes**: - -- The migration tool is a best-effort utility. Always double-check its recommendations for correctness. -- Use the migration tool with Bazel 7 (not supported with Bazel 8). - -## How to Use the Migration Tool - -Before you begin: - -- Upgrade to the latest Bazel 7 release, which provides robust support for both WORKSPACE and Bzlmod. - -- Verify the following command runs successfully for your project's main build targets: - - ```shell - bazel build --nobuild --enable_workspace --noenable_bzlmod <targets> - ``` - -### Command for running the script - -Once the prerequisites are met, run the following commands to use the migration tool: - -``` -# Clone the Bazel Central Registry repository -git clone https://github.com/bazelbuild/bazel-central-registry.git -cd bazel-central-registry - -# Build the migration tool -bazel build //tools:migrate_to_bzlmod - -# Create a convenient alias for the tool -alias migrate2bzlmod=$(realpath ./bazel-bin/tools/migrate_to_bzlmod) - -# Navigate to your project's root directory and run the tool -cd <your project root> -migrate2bzlmod -t <targets> -``` - -### Files generated by this script - -- `MODULE.bazel` - The central manifest file for Bzlmod, which declares the project's metadata and its direct dependencies on other Bazel modules. -- `migration_info.md` - A file providing step-by-step instructions on how the migration tool was executed, designed to assist in the manual completion of the migration process, if necessary. -- `resolved_deps.py` - Contains a comprehensive list of the project's external dependencies, generated by analyzing the project's `WORKSPACE` file, serving as a reference during the transition. -- `query_direct_deps` - Contains migration-relevant information regarding the utilized targets, obtained by invoking Bazel with `--output=build` on the project's `WORKSPACE` file. This file is primarily consumed by the migration script. -- `extension_for_XXX` - A file containing a module extension definition. The migration tool generates these files for dependencies that are not standard Bazel modules but can be managed using Bzlmod's [module extensions](/external/extension). - -### Flags - -Flags available in this migration scripts are: - -- `--t`/`--target`: Targets to migrate. This flag is repeatable, and the targets are accumulated. -- `--i`/`--initial`: Deletes `MODULE.bazel`, `resolved_deps.py`, `migration_info.md` files and starts from scratch - Detect direct dependencies, introduce them in MODULE.bazel and rerun generation of resolved dependencies. - -### Post-migration cleanup - -- Delete `migration_info.md`, `resolved_deps.py` and `query_direct_deps`. -- Clean up comments from `MODULE.bazel` file which were used for the migration, such as `# -- bazel_dep definitions -- #`. - -## Migration Example - -To see the migration script in action, consider the following scenario when Python, Maven and Go dependencies are declared in `WORKSPACE` file. - -Click here to see \`WORKSPACE\` file - -```python -workspace(name="example") - -load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") -load(":my_custom_macro.bzl", "my_custom_macro") - -http_archive( - name = "rules_cc", - sha256 = "b8b918a85f9144c01f6cfe0f45e4f2838c7413961a8ff23bc0c6cdf8bb07a3b6", - strip_prefix = "rules_cc-0.1.5", - urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.1.5/rules_cc-0.1.5.tar.gz"], -) - -# Module dependency -# ------------------- -http_archive( - name = "rules_shell", - sha256 = "3e114424a5c7e4fd43e0133cc6ecdfe54e45ae8affa14fadd839f29901424043", - strip_prefix = "rules_shell-0.4.0", - url = "https://github.com/bazelbuild/rules_shell/releases/download/v0.4.0/rules_shell-v0.4.0.tar.gz", -) - -# Repo rule -# ------------------- -http_archive( - name = "com_github_cockroachdb_cockroach", - sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", - strip_prefix = "cockroach-22.1.6", - url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", -) - -# Module extension -# ------------------- -# Macro which invokes repository_rule -my_custom_macro( - name = "my_custom_repo", -) - -# Go dependencies -# ------------------- -http_archive( - name = "io_bazel_rules_go", - integrity = "sha256-M6zErg9wUC20uJPJ/B3Xqb+ZjCPn/yxFF3QdQEmpdvg=", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", - "https://github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", - ], -) - -http_archive( - name = "bazel_gazelle", - integrity = "sha256-12v3pg/YsFBEQJDfooN6Tq+YKeEWVhjuNdzspcvfWNU=", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", - "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", - ], -) - -load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies") -load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository") - -go_rules_dependencies() -go_register_toolchains(version = "1.23.1") -gazelle_dependencies() - -go_repository( - name = "org_golang_x_net", - importpath = "golang.org/x/net", - sum = "h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=", - version = "v0.0.0-20190311183353-d8887717615a", - build_file_proto_mode = "disable", - build_naming_convention = "import", -) - -# Python dependencies -# ------------------- -http_archive( - name = "rules_python", - integrity = "sha256-qDdnnxOC8mlowe5vg5x9r5B5qlMSgGmh8oFd7KpjcwQ=", - strip_prefix = "rules_python-1.4.0", - url = "https://github.com/bazelbuild/rules_python/releases/download/1.4.0/rules_python-1.4.0.tar.gz", -) - -load("@rules_python//python:repositories.bzl", "py_repositories") -py_repositories() - -load("@rules_python//python:pip.bzl", "pip_parse") -pip_parse( - name = "my_python_deps", - requirements_lock = "@example//:requirements_lock.txt", -) - -load("@my_python_deps//:requirements.bzl", "install_deps") -install_deps() - -load("@rules_python//python:repositories.bzl", "python_register_toolchains") -python_register_toolchains( - name = "python_3_11", - python_version = "3.11", -) - -# Maven dependencies -# __________________ - -RULES_JVM_EXTERNAL_TAG = "4.5" -RULES_JVM_EXTERNAL_SHA = "b17d7388feb9bfa7f2fa09031b32707df529f26c91ab9e5d909eb1676badd9a6" - -http_archive( - name = "rules_jvm_external", - strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG, - sha256 = RULES_JVM_EXTERNAL_SHA, - url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG, -) - -load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps") -rules_jvm_external_deps() -load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup") -rules_jvm_external_setup() - -load("@rules_jvm_external//:defs.bzl", "maven_install") -maven_install( - name = "px_deps", - artifacts = [ - "org.antlr:antlr4:4.11.1", - ], - repositories = [ - "https://repo1.maven.org/maven2", - ], -) -``` - -Moreover, to demonstrate usage of module extension, custom macro is invoked from `WORKSPACE` and it is defined in `my_custom_macro.bzl`. - -Click here to see \`my\_custom\_macro.bzl\` file - -```python -"""Repo rule and macro used for testing""" - -def _test_repo_rule_impl(repository_ctx): - repository_ctx.file( - "BUILD", - content = """ -genrule( - name = "foo", - outs = ["rule_name.out"], - cmd = "touch $@", - visibility = ["//visibility:public"], -) -""" - ) - -_test_repo_rule = repository_rule( - implementation = _test_repo_rule_impl, -) - -def my_custom_macro(name): - _test_repo_rule(name = name) -``` - -The end goal is to have `MODULE.bazel` file and delete the `WORKSPACE` file, without impacting the user experience. - -The first step is to follow [How to Use the Migration Tool](#migration-tool-how-to-use), which mostly is checking the bazel version (it must be Bazel 7) and adding an alias to the migration script. - -Then, running `migrate2bzlmod -t=//...` outputs: - -``` - bazel 7.6.1 - - Generating ./resolved_deps.py file - It might take a while... - - RESOLVED: rules_java has been introduced as a Bazel module. - RESOLVED: bazel_gazelle has been introduced as a Bazel module. - RESOLVED: io_bazel_rules_go has been introduced as a Bazel module. - RESOLVED: rules_python has been introduced as a Bazel module. - IMPORTANT: 3.11 is used as a default python version. If you need a different version, please change it manually and then rerun the migration tool. - RESOLVED: my_python_deps has been introduced as python extension. - RESOLVED: org_golang_x_net has been introduced as go extension. - RESOLVED: rules_jvm_external has been introduced as a Bazel module. - RESOLVED: org.antlr has been introduced as maven extension. - RESOLVED: rules_shell has been introduced as a Bazel module. - - Congratulations! All external repositories needed for building //... are available with Bzlmod! - IMPORTANT: Fix potential build time issues by running the following command: - bazel build --enable_bzlmod --noenable_workspace //... - - IMPORTANT: For details about the migration process, check `migration_info.md` file. -``` - -which gives the following important information: - -- Generates `./resolved_deps.py` file, which contains info about all external repositories declared and loaded using your `WORKSPACE` file. -- `RESOLVED` keyword describes all dependencies which are resolved by the tool and added to the `MODULE.bazel` file. -- `IMPORTANT` keyword describes significant information worth investing time. -- All dependencies have been resolved in this example, at least with `--nobuild` flag. -- It is important to run the full build (command specified) and manually fix potential errors (e.g. toolchain not registered correctly). -- `migration_info.md` file contains details about the migration. Check details [at this section](#migration-tool-report-generation). - -### Transformations - -This section illustrates the migration of code from the `WORKSPACE` file to `MODULE.bazel`. - -**WORKSPACE - Bazel Module** - -``` -http_archive( - name = "rules_shell", - sha256 = "3e114424a5c7e4fd43e0133cc6ecdfe54e45ae8affa14fadd839f29901424043", - strip_prefix = "rules_shell-0.4.0", - url = "https://github.com/bazelbuild/rules_shell/releases/download/v0.4.0/rules_shell-v0.4.0.tar.gz", -) -``` - -**MODULE.bazel - Bazel Module** - -``` -bazel_dep(name = "rules_shell", version = "0.6.1") -``` - -*** - -**WORKSPACE - Go Extension** - -``` -http_archive( - name = "io_bazel_rules_go", - integrity = "sha256-M6zErg9wUC20uJPJ/B3Xqb+ZjCPn/yxFF3QdQEmpdvg=", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", - "https://github.com/bazelbuild/rules_go/releases/download/v0.48.0/rules_go-v0.48.0.zip", - ], -) -http_archive( - name = "bazel_gazelle", - integrity = "sha256-12v3pg/YsFBEQJDfooN6Tq+YKeEWVhjuNdzspcvfWNU=", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", - "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.37.0/bazel-gazelle-v0.37.0.tar.gz", - ], -) - -load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies") -load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository") - -go_rules_dependencies() -go_register_toolchains(version = "1.23.1") -gazelle_dependencies() - -go_repository( -name = "org_golang_x_net", -importpath = "golang.org/x/net", -sum = "h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628=", -version = "v0.0.0-20190311183353-d8887717615a", -build_file_proto_mode = "disable", -build_naming_convention = "import", -) -``` - -**MODULE.bazel - Go Extension** - -``` -go_deps = use_extension("@bazel_gazelle//:extensions.bzl", "go_deps") -go_sdk = use_extension("@io_bazel_rules_go//go:extensions.bzl", "go_sdk") - -go_deps.from_file(go_mod = "//:go.mod") -use_repo(go_deps, "org_golang_x_net") -go_sdk.from_file(go_mod = "//:go.mod") - -go_deps.gazelle_override( -path = "golang.org/x/net", -directives = [ -"gazelle:proto disable", -"gazelle:go_naming_convention import", -], -) -``` - -*** - -**WORKSPACE - Python Extension** - -``` -http_archive( - name = "rules_python", - integrity = "sha256-qDdnnxOC8mlowe5vg5x9r5B5qlMSgGmh8oFd7KpjcwQ=", - strip_prefix = "rules_python-1.4.0", - url = "https://github.com/bazelbuild/rules_python/releases/download/1.4.0/rules_python-1.4.0.tar.gz", -) - -load("@rules_python//python:repositories.bzl", "py_repositories") -py_repositories() - -load("@rules_python//python:pip.bzl", "pip_parse") -pip_parse( -name = "my_python_deps", -requirements_lock = "@example//:requirements_lock.txt", -) - -load("@my_python_deps//:requirements.bzl", "install_deps") -install_deps() - -load("@rules_python//python:repositories.bzl", "python_register_toolchains") -python_register_toolchains( -name = "python_3_11", -python_version = "3.11", -) -``` - -**MODULE.bazel - Python Extension** - -``` -pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") -pip.parse( - hub_name = "my_python_deps", - python_version = "3.11", - requirements_lock = "//:requirements_lock.txt", -) -use_repo(pip, "my_python_deps") - -python = use_extension("@rules_python//python/extensions:python.bzl", "python") -python.defaults(python_version = "3.11") -python.toolchain(python_version = "3.11") -``` - -*** - -**WORKSPACE - Maven Extension** - -``` - -RULES_JVM_EXTERNAL_TAG = "4.5" -RULES_JVM_EXTERNAL_SHA = "b17d7388feb9bfa7f2fa09031b32707df529f26c91ab9e5d909eb1676badd9a6" - -http_archive( -name = "rules_jvm_external", -strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG, -sha256 = RULES_JVM_EXTERNAL_SHA, -url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG, -) - -load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps") -rules_jvm_external_deps() -load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup") -rules_jvm_external_setup() - -load("@rules_jvm_external//:defs.bzl", "maven_install") -maven_install( -name = "px_deps", -artifacts = [ -"org.antlr:antlr4:4.11.1", -], -repositories = [ -"https://repo1.maven.org/maven2", -], -) -``` - -**MODULE.bazel - Maven Extension** - -``` -bazel_dep(name = "rules_jvm_external", version = "6.8") - -maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven") -use_repo(maven, "px_deps") - -maven.artifact( -name = "px_deps", -group = "org.antlr", -artifact = "antlr4", -version = "4.11.1" -) -``` - -*** - -**WORKSPACE - Repo rule** - -``` -load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - -http_archive( -name = "com_github_cockroachdb_cockroach", -sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", -strip_prefix = "cockroach-22.1.6", -url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", -) -``` - -**MODULE.bazel - Repo rule** - -``` -http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - -http_archive( -name = "com_github_cockroachdb_cockroach", -url = "https://github.com/cockroachdb/cockroach/archive/v22.1.6.tar.gz", -sha256 = "6c3568ef244ce6b874694eeeecb83ed4f5d5dff6cf037c952ecde76828a6c502", -strip_prefix = "cockroach-22.1.6", -) -``` - -*** - -**WORKSPACE - Module extension** - -``` -load(":my_custom_macro.bzl", "my_custom_macro") - -my_custom_macro( -name = "my_custom_repo", -) -``` - -**MODULE.bazel - Module extension** - -``` -extension_for_my_custom_macro = use_extension("//:extension_for_my_custom_macro.bzl", "extension_for_my_custom_macro") -use_repo(extension_for_my_custom_macro, "my_custom_repo") -``` - -**extension\_for\_my\_custom\_macro.bzl** - -``` -load("//:my_custom_macro.bzl", "my_custom_macro") - -def _extension_for_my_custom_macro_impl(ctx): -my_custom_macro( -name = "my_custom_repo", -) - -extension_for_my_custom_macro = module_extension(implementation = _extension_for_my_custom_macro_impl) -``` - -*** - -## Tips with debugging - -This section provides useful commands and information to help debug issues that may arise during the Bzlmod migration. - -### Useful tips - -- Override version - Not rarely it happens that upgrading the version of a dependency causes troubles. Bzlmod could change the version of the dependency due to the [MVS algorithm](/external/module#version-selection). In order to use the same or similar version as it was in the WORKSPACE, override it with [single\_version\_override](/rules/lib/globals/module#single_version_override). Note that this is useful for debugging differences between WORKSPACE and Bzlmod, but you shouldn't rely on this feature in the long term. - - `single_version_override(module_name = "{dep_name}", version = "{version}")` - -- Use [bazel mod](/external/mod-command#syntax) command. - - - Check the version of a specified repo with `show_repo` command. For example: - - `bazel mod show_repo @rules_python` - - - Check information about a module extension with the `show_extension` command. For example: - - `bazel mod show_extension @rules_python//python/extensions:pip.bzl%pip` - -- Use [vendor mode](/external/vendor) to create a local copy of a repo when you want to monitor or control the source of the repo. For example: - - `bazel vendor --enable_bzlmod --vendor_dir=vendor_src --repo=@protobuf` - -### Migration Report Generation - -This file is updated with each run of the migration script or it's generated from scratch if it's the first run or if the [`--i` flag](#migration-script-flags) is used. The report contains: - -- Command for local testing. - -- List of direct dependencies (at least the ones which are directly used in the project). - -- For each dependency, a drop-down menu for checking where the repository was declared in the `WORKSPACE` file, which is particularly useful for the debugging. You can see it as: - - ``` - > Click here to see where and how the repo was declared in the WORKSPACE - file - ``` - -- For each dependency, how it was implemented in `MODULE.bazel` file. From the earlier [Migration Example](#migration-tool-example), that would look as: - - 1. Bazel module Dependency - `Migration of rules_python` - - ``` - Found perfect name match in BCR: rules_python - Found partially name matches in BCR: rules_python_gazelle_plugin - - It has been introduced as a Bazel module: - `bazel_dep(name = "rules_python", version = "1.6.1")` - ``` - - - The script will automatically use the `perfect name match` if it finds it. In case of an error, you can double check if the name was correctly added. - - 2. Python extension - `Migration of my_python_deps` - - ``` - pip.parse( - hub_name = "my_python_deps", - requirements_lock = "//:requirements_lock.txt", - python_version = "3.11", - ) - use_repo(pip, "my_python_deps") - ``` - - 3. Maven extension - `Migration of org.antlr (px_deps):` - - ``` - maven.artifact( - name = "px_deps", - group = "org.antlr", - artifact = "antlr4", - version = "4.11.1" - ) - ``` - - 4. Go extension - `Migration of org_golang_x_net` - - ``` - go_deps.from_file(go_mod = "//:go.mod") - go_sdk.from_file(go_mod = "//:go.mod") - - go_deps.gazelle_override( - path = "golang.org/x/net", - directives = [ - "gazelle:proto disable", - "gazelle:go_naming_convention import", - ], - ) - ``` - - - It has been introduced as a go module with the help of `go.mod`. If `go.mod` and `go.sum` are not available, go module is added directly to the `MODULE.bazel` file. - - `gazelle_override` is used for adding specific directives. - -## Useful links - -- Official pages for the external extensions - - - [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md) - - [rules\_go](https://github.com/bazelbuild/rules_go/blob/master/docs/go/core/bzlmod.md) - - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi/download.html) - -- Community posts and videos - - - [Migrating to Bazel Modules](https://blog.engflow.com/2025/01/16/migrating-to-bazel-modules-aka-bzlmod---module-extensions/index.html#migrating-to-bazel-modules-aka-bzlmod-module-extensions). - - [Moving to Bzlmod](https://www.youtube.com/watch?v=W9uXRYLVHUk). - - [How Uber Manages Go Dependencies with Bzlmod](https://www.youtube.com/watch?v=hIqzkUE_pSY). - -## Feedback - -If you would like to contribute, do so by creating an Issue or PR at [bazel-central-registry](https://github.com/bazelbuild/bazel-central-registry). diff --git a/external/mod-command.mdx b/external/mod-command.mdx deleted file mode 100644 index 197bb3c6..00000000 --- a/external/mod-command.mdx +++ /dev/null @@ -1,418 +0,0 @@ ---- -title: 'mod Command' ---- - -The `mod` command provides a range of tools to help the user understand their external dependency graph. It lets you visualize the dependency graph, find out why a certain module or a version of a module is present in the graph, view the repo definitions backing modules, inspect usages of module extensions and repos they generate, among other functions. - -## Syntax - -```sh -bazel mod <subcommand> [<options>] [<arg> [<arg>...]] -``` - -The available subcommands and their respective required arguments are: - -- `graph`: Displays the full dependency graph of the project, starting from the root module. If one or more modules are specified in `--from`, these modules are shown directly under the root, and the graph is only expanded starting from them (see [example](#mod-example1)). - -- `deps <arg>...`: Displays the resolved direct dependencies of each of the specified modules, similarly to `graph`. - -- `all_paths <arg>...`: Displays all dependency paths from the --from modules to the target modules. To simplify the output, only the first shortest path is shown when multiple paths share the same suffix. For example, A -> B -> X would be shown, but the longer A -> C -> B -> X would be omitted. In other words, for every module Y that directly depends on the target module X, the output contains only the shortest path going through Y to reach X. - -- `path <arg>...`: Has the same semantics as `all_paths`, but only display a single path from one of the `--from` modules to one of the argument modules. - -- `explain <arg>...`: Shows all the places where the specified modules appear in the dependency graph, along with the modules that directly depend on them. The output of the `explain` command is essentially a pruned version of the `all_paths` command, containing 1) the root module; 2) the root module's direct dependencies that lead to the argument modules; 3) the argument modules' direct dependents; and 4) the argument modules themselves (see [example](#mod-example5)). - -- `show_repo <arg>...`: Displays the definition of the specified repos (see [example](#mod-example6)). Use `--all_repos` to show definitions of all repos in the entire dependency graph, or `--all_visible_repos` to show definitions of all repos visible from the `--base_module`. - -- `show_extension <extension>...`: Displays information about each of the specified extensions: a list of the generated repos along with the modules that import them using `use_repo`, and a list of the usages of that extension in each of the modules where it is used, containing the specified tags and the `use_repo` calls (see [example](#mod-example8)). - -`<arg>` refers to one or more modules or repos. It can be one of: - -- The literal string `<root>`: The root module representing your current project. - -- `<name>@<version>`: The module `<name>` at version `<version>`. For a module with a non-registry override, use an underscore (`_`) as the `<version>`. - -- `<name>`: All present versions of the module `<name>`. - -- `@<repo_name>`: The repo with the given [apparent name](overview#apparent-repo-name) in the context of the `--base_module`. - -- `@@<repo_name>`: The repo with the given [canonical name](overview#canonical-repo-name). - -In a context requiring specifying modules, `<arg>`s referring to repos that correspond to modules (as opposed to extension-generated repos) can also be used. Conversely, in a context requiring specifying repos, `<arg>`s referring to modules can stand in for the corresponding repos. - -`<extension>` must be of the form `<arg><label_to_bzl_file>%<extension_name>`. The `<label_to_bzl_file>` part must be a repo-relative label (for example, `//pkg/path:file.bzl`). - -The following options only affect the subcommands that print graphs (`graph`, `deps`, `all_paths`, `path`, and `explain`): - -- `--from <arg>[,<arg>[,...]]` *default: `<root>`*: The module(s) from which the graph is expanded in `graph`, `all_paths`, `path`, and `explain`. Check the subcommands' descriptions for more details. - -- `--verbose` *default: "false"*: Include in the output graph extra information about the version resolution of each module. If the module version changed during resolution, show either which version replaced it or what was the original version, the reason it was replaced, and which modules requested the new version if the reason was [Minimal Version Selection](module#version-selection). - -- `--include_unused` *default: "false"*: Include in the output graph the modules which were originally present in the dependency graph, but became unused after module resolution. - -- `--extension_info <mode>`: Include information about the module extension usages as part of the output graph (see [example](#mod-example7)). `<mode>` can be one of: - - - `hidden` *(default)*: Don't show anything about extensions. - - - `usages`: Show extensions under each module where they are used. They are printed in the form of `$<extension>`. - - - `repos`: In addition to `usages`, show the repo imported using `use_repo` under each extension usage. - - - `all`: In addition to `usages` and `repos`, also show extension-generated repos that are not imported by any module. These extra repos are shown under the first occurrence of their generating extension in the output, and are connected with a dotted edge. - -- `--extension_filter <extension>[,<extension>[,...]]`: If specified, the output graph only includes modules that use the specified extensions, and the paths that lead to those modules. Specifying an empty extension list (as in `--extension_filter=`) is equivalent to specifying *all* extensions used by any module in the dependency graph. - -- `--depth <N>`: The depth of the output graph. A depth of 1 only displays the root and its direct dependencies. Defaults to 1 for `explain`, 2 for `deps` and infinity for the others. - -- `--cycles` *default: "false"*: Include cycle edges in the output graph. - -- `--include_builtin` *default: "false"*: Include built-in modules (such as `@bazel_tools`) in the output graph. This flag is disabled by default, as built-in modules are implicitly depended on by every other module, which greatly clutters the output. - -- `--charset <charset>` *default: utf8*: Specify the charset to use for text output. Valid values are `"utf8"` and `"ascii"`. The only significant difference is in the special characters used to draw the graph in the `"text"` output format, which don't exist in the `"ascii"` charset. Therefore, the `"ascii"` charset is present to also support the usage on legacy platforms which cannot use Unicode. - -- `--output <mode>`: Include information about the module extension usages as part of the output graph. `<mode`> can be one of: - - - `text` *(default)*: A human-readable representation of the output graph (flattened as a tree). - - - `json`: Outputs the graph in the form of a JSON object (flattened as a tree). - - - `graph`: Outputs the graph in the Graphviz *dot* representation. - - Tip: Use the following command to pipe the output through the *dot* engine and export the graph representation as an SVG image. - - ```sh - bazel mod graph --output graph | dot -Tsvg > /tmp/graph.svg - ``` - -Other options include: - -- `--base_module <arg>` *default: `<root>`*: Specify a module relative to which apparent repo names in arguments are interpreted. Note that this argument itself can be in the form of `@<repo_name>`; this is always interpreted relative to the root module. - -- `--extension_usages <arg>[,<arg>[,...]]`: Filters `show_extension` to only display extension usages from the specified modules. - -## Examples - -Some possible usages of the `mod` command on a real Bazel project are showcased below to give you a general idea on how you can use it to inspect your project's external dependencies. - -`MODULE.bazel` file: - -```python -module( - name = "my_project", - version = "1.0", -) - -bazel_dep(name = "bazel_skylib", version = "1.1.1", repo_name = "skylib1") -bazel_dep(name = "bazel_skylib", version = "1.2.0", repo_name = "skylib2") -multiple_version_override(module_name = "bazel_skylib", versions = ["1.1.1", "1.2.0"]) - -bazel_dep(name = "stardoc", version = "0.5.0") -bazel_dep(name = "rules_java", version = "5.0.0") - -toolchains = use_extension("@rules_java//java:extensions.bzl", "toolchains") -use_repo(toolchains, my_jdk="remotejdk17_linux") -``` - -| | | -| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -| ![Graph Before Resolution](images/mod_exampleBefore.svg)Graph Before Resolution | ![Graph After Resolution](images/mod_exampleResolved.svg)Graph After Resolution | - -1. []()Display the whole dependency graph of your project. - - ```sh - bazel mod graph - ``` - - ```none - <root> (my_project@1.0) - ├───bazel_skylib@1.1.1 - │ └───platforms@0.0.4 - ├───bazel_skylib@1.2.0 - │ └───platforms@0.0.4 ... - ├───rules_java@5.0.0 - │ ├───platforms@0.0.4 ... - │ ├───rules_cc@0.0.1 - │ │ ├───bazel_skylib@1.1.1 ... - │ │ └───platforms@0.0.4 ... - │ └───rules_proto@4.0.0 - │ ├───bazel_skylib@1.1.1 ... - │ └───rules_cc@0.0.1 ... - └───stardoc@0.5.0 - ├───bazel_skylib@1.1.1 ... - └───rules_java@5.0.0 ... - ``` - - Note: The `...` symbol indicates that the node has already been expanded somewhere else and was not expanded again to reduce noise. - -2. []()Display the whole dependency graph (including unused modules and with extra information about version resolution). - - ```sh - bazel mod graph --include_unused --verbose - ``` - - ```none - <root> (my_project@1.0) - ├───bazel_skylib@1.1.1 - │ └───platforms@0.0.4 - ├───bazel_skylib@1.2.0 - │ └───platforms@0.0.4 ... - ├───rules_java@5.0.0 - │ ├───platforms@0.0.4 ... - │ ├───rules_cc@0.0.1 - │ │ ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) - │ │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - │ │ └───platforms@0.0.4 ... - │ └───rules_proto@4.0.0 - │ ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) - │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - │ └───rules_cc@0.0.1 ... - └───stardoc@0.5.0 - ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - ├───rules_java@5.0.0 ... (was 4.0.0, cause <root>, bazel_tools@_) - ├───bazel_skylib@1.0.3 (to 1.1.1, cause multiple_version_override) - │ └───platforms@0.0.4 ... - └───rules_java@4.0.0 (to 5.0.0, cause <root>, bazel_tools@_) - ├───bazel_skylib@1.0.3 ... (to 1.1.1, cause multiple_version_override) - └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - ``` - -3. []()Display the dependency graph expanded from some specific modules. - - ```sh - bazel mod graph --from rules_java --include_unused - ``` - - ```none - <root> (my_project@1.0) - ├───rules_java@5.0.0 - │ ├───platforms@0.0.4 - │ ├───rules_cc@0.0.1 - │ │ ├───bazel_skylib@1.0.3 ... (unused) - │ │ ├───bazel_skylib@1.1.1 ... - │ │ └───platforms@0.0.4 ... - │ └───rules_proto@4.0.0 - │ ├───bazel_skylib@1.0.3 ... (unused) - │ ├───bazel_skylib@1.1.1 ... - │ └───rules_cc@0.0.1 ... - └╌╌rules_java@4.0.0 (unused) - ├───bazel_skylib@1.0.3 (unused) - │ └───platforms@0.0.4 ... - └───bazel_skylib@1.1.1 - └───platforms@0.0.4 ... - ``` - - Note: The dotted line is used to indicate an *indirect* (transitive) dependency edge between two nodes. - -4. []()Display all paths between two of your modules. - - ```sh - bazel mod all_paths bazel_skylib@1.1.1 --from rules_proto - ``` - - ```none - <root> (my_project@1.0) - └╌╌rules_proto@4.0.0 - ├───bazel_skylib@1.1.1 - └───rules_cc@0.0.1 - └───bazel_skylib@1.1.1 ... - ``` - -5. []()See why and how your project depends on some module(s). - - ```sh - bazel mod explain @skylib1 --verbose --include_unused - ``` - - ```none - <root> (my_project@1.0) - ├───bazel_skylib@1.1.1 - ├───rules_java@5.0.0 - │ ├───rules_cc@0.0.1 - │ │ └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - │ └───rules_proto@4.0.0 - │ ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - │ └───rules_cc@0.0.1 ... - └───stardoc@0.5.0 - ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - ├╌╌rules_cc@0.0.1 - │ └───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - └╌╌rules_proto@4.0.0 - ├───bazel_skylib@1.1.1 ... (was 1.0.3, cause multiple_version_override) - └───rules_cc@0.0.1 ... - ``` - -6. []()See the underlying rule of your modules' repos. - - ```sh - bazel mod show_repo rules_cc stardoc - ``` - - ```none - ## rules_cc@0.0.1: - load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "rules_cc+", - urls = ["https://bcr.bazel.build/test-mirror/github.com/bazelbuild/rules_cc/releases/download/0.0.1/rules_cc-0.0.1.tar.gz", "https://github.com/bazelbuild/rules_cc/releases/download/0.0.1/rules_cc-0.0.1.tar.gz"], - integrity = "sha256-Tcy/0iwN7xZMj0dFi9UODHFI89kgAs20WcKpamhJgkE=", - strip_prefix = "", - remote_patches = {"https://bcr.bazel.build/modules/rules_cc/0.0.1/patches/add_module_extension.patch": "sha256-g3+zmGs0YT2HKOVevZpN0Jet89Ylw90Cp9XsIAY8QqU="}, - remote_patch_strip = 1, - ) - - ## stardoc: - load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") - http_archive( - name = "stardoc+", - urls = ["https://bcr.bazel.build/test-mirror/github.com/bazelbuild/stardoc/releases/download/0.5.0/stardoc-0.5.0.tar.gz", "https://github.com/bazelbuild/stardoc/releases/download/0.5.0/stardoc-0.5.0.tar.gz"], - integrity = "sha256-yXlNzIAmow/2fPfPkeviRcopSyCwcYRdEsGSr+JDrXI=", - strip_prefix = "", - remote_patches = {}, - remote_patch_strip = 0, - ) - ``` - - `bazel mod show_repo` also works with repos imported by `use_repo` and repos created with `use_repo_rule`. If `show_repo` is invoked with an apparent repository name or `--all_visible_repos`, then the apparent repository name is shown on a line prefixed with `##`. - - ```sh - bazel mod show_repo @jq_linux_arm64 - bazel mod show_repo --all_visible_repos - ``` - - ```none - ## @jq_linux_arm64: - load("@@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") - http_file( - name = "+http_file+jq_linux_arm64", - executable = True, - integrity = "sha256-TdLYoGYd8LIvG7mh+YMPBrbzuPfZEhGh7118TwaotKU=", - urls = ["https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-arm64"], - ) - ``` - -7. []()See what module extensions are used in your dependency graph. - - ```sh - bazel mod graph --extension_info=usages - ``` - - ```none - <root> (my_project@1.0) - ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains - ├───rules_java@5.0.0 # - │ ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains - │ ├───rules_cc@0.0.1 # - │ │ └───$@@rules_cc.0.0.1//bzlmod:extensions.bzl%cc_configure - │ └───rules_proto@4.0.0 - │ └───rules_cc@0.0.1 ... - └───stardoc@0.5.0 - └───rules_java@5.0.0 ... - ``` - -8. []()See what repositories are generated and imported from some specific extension as part of the dependency graph. - - ```sh - bazel mod show_extension @@rules_java+5.0.0//java:extensions.bzl%toolchains - ``` - - ```none - <root> (my_project@1.0) - ├───$@@rules_java.5.0.0//java:extensions.bzl%toolchains - │ ├───remotejdk17_linux - │ ├╌╌remotejdk11_linux - │ ├╌╌remotejdk11_linux_aarch64 - │ ├╌╌remotejdk11_linux_ppc64le - │ ├╌╌remotejdk11_linux_s390x - ...(some lines omitted)... - ├───rules_java@5.0.0 # - │ └───$@@rules_java.5.0.0//java:extensions.bzl%toolchains ... - │ ├───local_jdk - │ ├───remote_java_tools - │ ├───remote_java_tools_darwin - │ ├───remote_java_tools_linux - │ ├───remote_java_tools_windows - │ ├───remotejdk11_linux_aarch64_toolchain_config_repo - │ ├───remotejdk11_linux_ppc64le_toolchain_config_repo - ...(some lines omitted)... - └───stardoc@0.5.0 - └───rules_java@5.0.0 ... - ``` - -9. []()See the list of generated repositories of an extension and how that extension is used in each module. - - ```sh - bazel mod graph --extension_info=all --extension_filter=@rules_java//java:extensions.bzl%toolchains - ``` - - ```none - ## @@rules_java.5.0.0//java:extensions.bzl%toolchains: - - Fetched repositories: - - local_jdk (imported by bazel_tools@_, rules_java@5.0.0) - - remote_java_tools (imported by bazel_tools@_, rules_java@5.0.0) - - remote_java_tools_darwin (imported by bazel_tools@_, rules_java@5.0.0) - - remote_java_tools_linux (imported by bazel_tools@_, rules_java@5.0.0) - - remote_java_tools_windows (imported by bazel_tools@_, rules_java@5.0.0) - - remotejdk11_linux_aarch64_toolchain_config_repo (imported by rules_java@5.0.0) - - remotejdk11_linux_ppc64le_toolchain_config_repo (imported by rules_java@5.0.0) - ...(some lines omitted)... - - remotejdk17_linux (imported by <root>) - - remotejdk11_linux - - remotejdk11_linux_aarch64 - - remotejdk11_linux_ppc64le - - remotejdk11_linux_s390x - - remotejdk11_macos - ...(some lines omitted)... - - # Usage in <root> at <root>/MODULE.bazel:14:27 with the specified attributes: - use_repo( - toolchains, - my_jdk="remotejdk17_linux", - ) - - # Usage in bazel_tools@_ at bazel_tools@_/MODULE.bazel:23:32 with the specified attributes: - use_repo( - toolchains, - "local_jdk", - "remote_java_tools", - "remote_java_tools_linux", - "remote_java_tools_windows", - "remote_java_tools_darwin", - ) - - # Usage in rules_java@5.0.0 at rules_java@5.0.0/MODULE.bazel:30:27 with the specified attributes: - use_repo( - toolchains, - "remote_java_tools", - "remote_java_tools_linux", - "remote_java_tools_windows", - "remote_java_tools_darwin", - "local_jdk", - "remotejdk11_linux_toolchain_config_repo", - "remotejdk11_macos_toolchain_config_repo", - "remotejdk11_macos_aarch64_toolchain_config_repo", - ...(some lines omitted)... - ) - ``` - -10. []()See the underlying rule of some extension-generated repositories. - -```` -```sh -```` - -```` -bazel mod show_repo --base_module=rules_java @remote_java_tools -``` - -```none -## @remote_java_tools: -# <builtin> -http_archive( - name = "rules_java++toolchains+remote_java_tools", - urls = ["https://mirror.bazel.build/bazel_java_tools/releases/java/v11.5/java_tools-v11.5.zip", "https://github.com/bazelbuild/java_tools/releases/download/java_v11.5/java_tools-v11.5.zip"], - sha256 = "b763ee80e5754e593fd6d5be6d7343f905bc8b73d661d36d842b024ca11b6793", -) -# Rule http_archive defined at (most recent call last): -# /home/user/.cache/bazel/_bazel_user/6e893e0f5a92cc4cf5909a6e4b2770f9/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31 in <toplevel> -``` -```` diff --git a/external/module.mdx b/external/module.mdx index 9ac802a4..0499c4cd 100644 --- a/external/module.mdx +++ b/external/module.mdx @@ -2,9 +2,16 @@ title: 'Bazel modules' --- -A Bazel **module** is a Bazel project that can have multiple versions, each of which publishes metadata about other modules that it depends on. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. -A module must have a `MODULE.bazel` file at its repo root. This file is the module's manifest, declaring its name, version, list of direct dependencies, and other information. For a basic example: + +A Bazel **module** is a Bazel project that can have multiple versions, each of +which publishes metadata about other modules that it depends on. This is +analogous to familiar concepts in other dependency management systems, such as a +Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. + +A module must have a `MODULE.bazel` file at its repo root. This file is the +module's manifest, declaring its name, version, list of direct dependencies, and +other information. For a basic example: ```python module(name = "my-module", version = "1.0") @@ -13,29 +20,50 @@ bazel_dep(name = "rules_cc", version = "0.0.1") bazel_dep(name = "protobuf", version = "3.19.0") ``` -See the [full list](/rules/lib/globals/module) of directives available in `MODULE.bazel` files. +See the [full list](/rules/lib/globals/module) of directives available in +`MODULE.bazel` files. -To perform module resolution, Bazel starts by reading the root module's `MODULE.bazel` file, and then repeatedly requests any dependency's `MODULE.bazel` file from a [Bazel registry](/external/registry) until it discovers the entire dependency graph. +To perform module resolution, Bazel starts by reading the root module's +`MODULE.bazel` file, and then repeatedly requests any dependency's +`MODULE.bazel` file from a [Bazel registry](/external/registry) until it +discovers the entire dependency graph. -By default, Bazel then [selects](#version-selection) one version of each module to use. Bazel represents each module with a repo, and consults the registry again to learn how to define each of the repos. +By default, Bazel then [selects](#version-selection) one version of each module +to use. Bazel represents each module with a repo, and consults the registry +again to learn how to define each of the repos. ## Version format -Bazel has a diverse ecosystem and projects use various versioning schemes. The most popular by far is [SemVer](https://semver.org), but there are also prominent projects using different schemes such as [Abseil](https://github.com/abseil/abseil-cpp/releases), whose versions are date-based, for example `20210324.2`). +Bazel has a diverse ecosystem and projects use various versioning schemes. The +most popular by far is [SemVer](https://semver.org), but there are +also prominent projects using different schemes such as +[Abseil](https://github.com/abseil/abseil-cpp/releases), whose +versions are date-based, for example `20210324.2`). -For this reason, Bazel adopts a more relaxed version of the SemVer spec. The differences include: +For this reason, Bazel adopts a more relaxed version of the SemVer spec. The +differences include: -- SemVer prescribes that the "release" part of the version must consist of 3 segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so that any number of segments is allowed. -- In SemVer, each of the segments in the "release" part must be digits only. In Bazel, this is loosened to allow letters too, and the comparison semantics match the "identifiers" in the "prerelease" part. -- Additionally, the semantics of major, minor, and patch version increases are not enforced. However, see [compatibility level](#compatibility_level) for details on how we denote backwards compatibility. +* SemVer prescribes that the "release" part of the version must consist of 3 + segments: `MAJOR.MINOR.PATCH`. In Bazel, this requirement is loosened so + that any number of segments is allowed. +* In SemVer, each of the segments in the "release" part must be digits only. + In Bazel, this is loosened to allow letters too, and the comparison + semantics match the "identifiers" in the "prerelease" part. +* Additionally, the semantics of major, minor, and patch version increases are + not enforced. However, see [compatibility level](#compatibility_level) for + details on how we denote backwards compatibility. -Any valid SemVer version is a valid Bazel module version. Additionally, two SemVer versions `a` and `b` compare `a < b` if and only if the same holds when they're compared as Bazel module versions. +Any valid SemVer version is a valid Bazel module version. Additionally, two +SemVer versions `a` and `b` compare `a < b` if and only if the same holds when +they're compared as Bazel module versions. -Finally, to learn more about module versioning, [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). +Finally, to learn more about module versioning, [see the `MODULE.bazel` +FAQ](faq#module-versioning-best-practices). ## Version selection -Consider the diamond dependency problem, a staple in the versioned dependency management space. Suppose you have the dependency graph: +Consider the diamond dependency problem, a staple in the versioned dependency +management space. Suppose you have the dependency graph: ``` A 1.0 @@ -45,78 +73,164 @@ Consider the diamond dependency problem, a staple in the versioned dependency ma D 1.0 D 1.1 ``` -Which version of `D` should be used? To resolve this question, Bazel uses the [Minimal Version Selection](https://research.swtch.com/vgo-mvs) (MVS) algorithm introduced in the Go module system. MVS assumes that all new versions of a module are backwards compatible, and so picks the highest version specified by any dependent (`D 1.1` in our example). It's called "minimal" because `D 1.1` is the earliest version that could satisfy our requirements — even if `D 1.2` or newer exists, we don't select them. Using MVS creates a version selection process that is *high-fidelity* and *reproducible*. +Which version of `D` should be used? To resolve this question, Bazel uses the +[Minimal Version Selection](https://research.swtch.com/vgo-mvs) +(MVS) algorithm introduced in the Go module system. MVS assumes that all new +versions of a module are backwards compatible, and so picks the highest version +specified by any dependent (`D 1.1` in our example). It's called "minimal" +because `D 1.1` is the earliest version that could satisfy our requirements — +even if `D 1.2` or newer exists, we don't select them. Using MVS creates a +version selection process that is *high-fidelity* and *reproducible*. ### Yanked versions -The registry can declare certain versions as *yanked* if they should be avoided (such as for security vulnerabilities). Bazel throws an error when selecting a yanked version of a module. To fix this error, either upgrade to a newer, non-yanked version, or use the [`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) flag to explicitly allow the yanked version. +The registry can declare certain versions as *yanked* if they should be avoided +(such as for security vulnerabilities). Bazel throws an error when selecting a +yanked version of a module. To fix this error, either upgrade to a newer, +non-yanked version, or use the +[`--allow_yanked_versions`](/reference/command-line-reference#flag--allow_yanked_versions) +flag to explicitly allow the yanked version. ## Compatibility level -In Go, MVS's assumption about backwards compatibility works because it treats backwards incompatible versions of a module as a separate module. In terms of SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can coexist in the resolved dependency graph. This is, in turn, made possible by encoding the major version in the package path in Go, so there aren't any compile-time or linking-time conflicts. Bazel, however, cannot provide such guarantees because it follows [a relaxed version of SemVer](#version-format). - -Thus, Bazel needs the equivalent of the SemVer major version number to detect backwards incompatible ("breaking") versions. This number is called the *compatibility level*, and is specified by each module version in its [`module()`](/rule/lib/globals/module#module) directive. With this information, Bazel can throw an error if it detects that versions of the *same module* with *different compatibility levels* exist in the resolved dependency graph. - -Finally, incrementing the compatibility level can be disruptive to the users. To learn more about when and how to increment it, [check the `MODULE.bazel` FAQ](faq#incrementing-compatibility-level). +In Go, MVS's assumption about backwards compatibility works because it treats +backwards incompatible versions of a module as a separate module. In terms of +SemVer, that means `A 1.x` and `A 2.x` are considered distinct modules, and can +coexist in the resolved dependency graph. This is, in turn, made possible by +encoding the major version in the package path in Go, so there aren't any +compile-time or linking-time conflicts. Bazel, however, cannot provide such +guarantees because it follows [a relaxed version of SemVer](#version-format). + +Thus, Bazel needs the equivalent of the SemVer major version number to detect +backwards incompatible ("breaking") versions. This number is called the +*compatibility level*, and is specified by each module version in its +[`module()`](/rule/lib/globals/module#module) directive. With this information, +Bazel can throw an error if it detects that versions of the _same module_ with +_different compatibility levels_ exist in the resolved dependency graph. + +Finally, incrementing the compatibility level can be disruptive to the users. +To learn more about when and how to increment it, [check the `MODULE.bazel` +FAQ](faq#incrementing-compatibility-level). ## Overrides -Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel module resolution. Only the root module's overrides take effect — if a module is used as a dependency, its overrides are ignored. +Specify overrides in the `MODULE.bazel` file to alter the behavior of Bazel +module resolution. Only the root module's overrides take effect — if a module is +used as a dependency, its overrides are ignored. -Each override is specified for a certain module name, affecting all of its versions in the dependency graph. Although only the root module's overrides take effect, they can be for transitive dependencies that the root module does not directly depend on. +Each override is specified for a certain module name, affecting all of its +versions in the dependency graph. Although only the root module's overrides take +effect, they can be for transitive dependencies that the root module does not +directly depend on. ### Single-version override -The [`single_version_override`](/rules/lib/globals/module#single_version_override) serves multiple purposes: +The [`single_version_override`](/rules/lib/globals/module#single_version_override) +serves multiple purposes: -- With the `version` attribute, you can pin a dependency to a specific version, regardless of which versions of the dependency are requested in the dependency graph. -- With the `registry` attribute, you can force this dependency to come from a specific registry, instead of following the normal [registry selection](/external/registry#selecting_registries) process. -- With the `patch*` attributes, you can specify a set of patches to apply to the downloaded module. +* With the `version` attribute, you can pin a dependency to a specific + version, regardless of which versions of the dependency are requested in the + dependency graph. +* With the `registry` attribute, you can force this dependency to come from a + specific registry, instead of following the normal [registry + selection](/external/registry#selecting_registries) process. +* With the `patch*` attributes, you can specify a set of patches to apply to + the downloaded module. These attributes are all optional and can be mixed and matched with each other. ### Multiple-version override -A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) can be specified to allow multiple versions of the same module to coexist in the resolved dependency graph. - -You can specify an explicit list of allowed versions for the module, which must all be present in the dependency graph before resolution — there must exist *some* transitive dependency depending on each allowed version. After resolution, only the allowed versions of the module remain, while Bazel upgrades other versions of the module to the nearest higher allowed version at the same compatibility level. If no higher allowed version at the same compatibility level exists, Bazel throws an error. - -For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the dependency graph before resolution and the major version is the compatibility level: - -- A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other versions remaining the same. -- A multiple-version override allowing `1.5` and `2.0` results in an error, as `1.7` has no higher version at the same compatibility level to upgrade to. -- A multiple-version override allowing `1.9` and `2.0` results in an error, as `1.9` is not present in the dependency graph before resolution. - -Additionally, users can also override the registry using the `registry` attribute, similarly to single-version overrides. +A [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override) +can be specified to allow multiple versions of the same module to coexist in the +resolved dependency graph. + +You can specify an explicit list of allowed versions for the module, which must +all be present in the dependency graph before resolution — there must exist +*some* transitive dependency depending on each allowed version. After +resolution, only the allowed versions of the module remain, while Bazel upgrades +other versions of the module to the nearest higher allowed version at the same +compatibility level. If no higher allowed version at the same compatibility +level exists, Bazel throws an error. + +For example, if versions `1.1`, `1.3`, `1.5`, `1.7`, and `2.0` exist in the +dependency graph before resolution and the major version is the compatibility +level: + +* A multiple-version override allowing `1.3`, `1.7`, and `2.0` results in + `1.1` being upgraded to `1.3`, `1.5` being upgraded to `1.7`, and other + versions remaining the same. +* A multiple-version override allowing `1.5` and `2.0` results in an error, as + `1.7` has no higher version at the same compatibility level to upgrade to. +* A multiple-version override allowing `1.9` and `2.0` results in an error, as + `1.9` is not present in the dependency graph before resolution. + +Additionally, users can also override the registry using the `registry` +attribute, similarly to single-version overrides. ### Non-registry overrides -Non-registry overrides completely remove a module from version resolution. Bazel does not request these `MODULE.bazel` files from a registry, but instead from the repo itself. +Non-registry overrides completely remove a module from version resolution. Bazel +does not request these `MODULE.bazel` files from a registry, but instead from +the repo itself. Bazel supports the following non-registry overrides: -- [`archive_override`](/rules/lib/globals/module#archive_override) -- [`git_override`](/rules/lib/globals/module#git_override) -- [`local_path_override`](/rules/lib/globals/module#local_path_override) +* [`archive_override`](/rules/lib/globals/module#archive_override) +* [`git_override`](/rules/lib/globals/module#git_override) +* [`local_path_override`](/rules/lib/globals/module#local_path_override) -Note that setting a version value in the source archive `MODULE.bazel` can have downsides when the module is being overridden with a non-registry override. To learn more about this [see the `MODULE.bazel` FAQ](faq#module-versioning-best-practices). +Note that setting a version value in the source archive `MODULE.bazel` can have +downsides when the module is being overridden with a non-registry override. To +learn more about this [see the `MODULE.bazel` +FAQ](faq#module-versioning-best-practices). ## Define repos that don't represent Bazel modules -With `bazel_dep`, you can define repos that represent other Bazel modules. Sometimes there is a need to define a repo that does *not* represent a Bazel module; for example, one that contains a plain JSON file to be read as data. +With `bazel_dep`, you can define repos that represent other Bazel modules. +Sometimes there is a need to define a repo that does _not_ represent a Bazel +module; for example, one that contains a plain JSON file to be read as data. -In this case, you could use the [`use_repo_rule` directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo by invoking a repo rule. This repo will only be visible to the module it's defined in. +In this case, you could use the [`use_repo_rule` +directive](/rules/lib/globals/module#use_repo_rule) to directly define a repo +by invoking a repo rule. This repo will only be visible to the module it's +defined in. -Under the hood, this is implemented using the same mechanism as [module extensions](/external/extension), which lets you define repos with more flexibility. +Under the hood, this is implemented using the same mechanism as [module +extensions](/external/extension), which lets you define repos with more +flexibility. ## Repository names and strict deps -The [apparent name](/external/overview#apparent-repo-name) of a repo backing a module to its direct dependents defaults to its module name, unless the `repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) directive says otherwise. Note that this means a module can only find its direct dependencies. This helps prevent accidental breakages due to changes in transitive dependencies. - -The [canonical name](/external/overview#canonical-repo-name) of a repo backing a module is either `<var>module_name</var>+<var>version</var>` (for example, `bazel_skylib+1.0.3`) or `<var>module_name</var>+` (for example, `bazel_features+`), depending on whether there are multiple versions of the module in the entire dependency graph (see [`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). Note that **the canonical name format** is not an API you should depend on and **is subject to change at any time**. Instead of hard-coding the canonical name, use a supported way to get it directly from Bazel: - -- In BUILD and `.bzl` files, use [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance constructed from a label string given by the apparent name of the repo, e.g., `Label("@bazel_skylib").repo_name`. -- When looking up runfiles, use [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) or one of the runfiles libraries in `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, in `@rules_foo//foo/runfiles`. -- When interacting with Bazel from an external tool such as an IDE or language server, use the `bazel mod dump_repo_mapping` command to get the mapping from apparent names to canonical names for a given set of repositories. - -[Module extensions](/external/extension) can also introduce additional repos into the visible scope of a module. +The [apparent name](/external/overview#apparent-repo-name) of a repo backing a +module to its direct dependents defaults to its module name, unless the +`repo_name` attribute of the [`bazel_dep`](/rules/lib/globals/module#bazel_dep) +directive says otherwise. Note that this means a module can only find its direct +dependencies. This helps prevent accidental breakages due to changes in +transitive dependencies. + +The [canonical name](/external/overview#canonical-repo-name) of a repo backing a +module is either `module_name+version{{ +"" }}` (for example, `bazel_skylib+1.0.3`) or `module_name{{ +"" }}+` (for example, `bazel_features+`), depending on whether there are +multiple versions of the module in the entire dependency graph (see +[`multiple_version_override`](/rules/lib/globals/module#multiple_version_override)). +Note that **the canonical name format** is not an API you should depend on and +**is subject to change at any time**. Instead of hard-coding the canonical name, +use a supported way to get it directly from Bazel: + +* In BUILD and `.bzl` files, use + [`Label.repo_name`](/rules/lib/builtins/Label#repo_name) on a `Label` instance + constructed from a label string given by the apparent name of the repo, e.g., + `Label("@bazel_skylib").repo_name`. +* When looking up runfiles, use + [`$(rlocationpath ...)`](https://bazel.build/reference/be/make-variables#predefined_label_variables) + or one of the runfiles libraries in + `@bazel_tools//tools/{bash,cpp,java}/runfiles` or, for a ruleset `rules_foo`, + in `@rules_foo//foo/runfiles`. +* When interacting with Bazel from an external tool such as an IDE or language + server, use the `bazel mod dump_repo_mapping` command to get the mapping from + apparent names to canonical names for a given set of repositories. + +[Module extensions](/external/extension) can also introduce additional repos +into the visible scope of a module. diff --git a/external/overview.mdx b/external/overview.mdx index 84593001..6e55b579 100644 --- a/external/overview.mdx +++ b/external/overview.mdx @@ -2,15 +2,28 @@ title: 'External dependencies overview' --- -Bazel supports *external dependencies*, source files (both text and binary) used in your build that are not from your workspace. For example, they could be a ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local machine outside your current workspace. -This document gives an overview of the system before examining some of the concepts in more detail. + + +Bazel supports *external dependencies*, source files (both text and binary) used +in your build that are not from your workspace. For example, they could be a +ruleset hosted in a GitHub repo, a Maven artifact, or a directory on your local +machine outside your current workspace. + +This document gives an overview of the system before examining some of the +concepts in more detail. ## Overview of the system -Bazel's external dependency system works on the basis of [*Bazel modules*](#module), each of which is a versioned Bazel project, and [*repositories*](#repository) (or repos), which are directory trees containing source files. +Bazel's external dependency system works on the basis of [*Bazel +modules*](#module), each of which is a versioned Bazel project, and +[*repositories*](#repository) (or repos), which are directory trees containing +source files. -Bazel starts from the root module -- that is, the project you're working on. Like all modules, it needs to have a `MODULE.bazel` file at its directory root, declaring its basic metadata and direct dependencies. The following is a basic example: +Bazel starts from the root module -- that is, the project you're working on. +Like all modules, it needs to have a `MODULE.bazel` file at its directory root, +declaring its basic metadata and direct dependencies. The following is a basic +example: ```python module(name = "my-module", version = "1.0") @@ -19,13 +32,29 @@ bazel_dep(name = "rules_cc", version = "0.1.1") bazel_dep(name = "platforms", version = "0.0.11") ``` -From there, Bazel looks up all transitive dependency modules in a [*Bazel registry*](registry) — by default, the [Bazel Central Registry](https://bcr.bazel.build/). The registry provides the dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire transitive dependency graph before performing version resolution. - -After version resolution, in which one version is selected for each module, Bazel consults the registry again to learn how to define a repo for each module -- that is, how the sources for each dependency module should be fetched. Most of the time, these are just archives downloaded from the internet and extracted. - -Modules can also specify customized pieces of data called *tags*, which are consumed by [*module extensions*](extension) after module resolution to define additional repos. These extensions can perform actions like file I/O and sending network requests. Among other things, they allow Bazel to interact with other package management systems while also respecting the dependency graph built out of Bazel modules. - -The three kinds of repos -- the main repo (which is the source tree you're working in), the repos representing transitive dependency modules, and the repos created by module extensions -- form the [*workspace*](#workspace) together. External repos (non-main repos) are fetched on demand, for example when they're referred to by labels (like `@repo//pkg:target`) in BUILD files. +From there, Bazel looks up all transitive dependency modules in a +[*Bazel registry*](registry) — by default, the [Bazel Central +Registry](https://bcr.bazel.build/). The registry provides the +dependencies' `MODULE.bazel` files, which allows Bazel to discover the entire +transitive dependency graph before performing version resolution. + +After version resolution, in which one version is selected for each module, +Bazel consults the registry again to learn how to define a repo for each module +-- that is, how the sources for each dependency module should be fetched. Most +of the time, these are just archives downloaded from the internet and extracted. + +Modules can also specify customized pieces of data called *tags*, which are +consumed by [*module extensions*](extension) after module resolution +to define additional repos. These extensions can perform actions like file I/O +and sending network requests. Among other things, they allow Bazel to interact +with other package management systems while also respecting the dependency graph +built out of Bazel modules. + +The three kinds of repos -- the main repo (which is the source tree you're +working in), the repos representing transitive dependency modules, and the repos +created by module extensions -- form the [*workspace*](#workspace) together. +External repos (non-main repos) are fetched on demand, for example when they're +referred to by labels (like `@repo//pkg:target`) in BUILD files. ## Benefits @@ -33,30 +62,54 @@ Bazel's external dependency system offers a wide range of benefits. ### Automatic Dependency Resolution -- **Deterministic Version Resolution**: Bazel adopts the deterministic [MVS](module#version-selection) version resolution algorithm, minimizing conflicts and addressing diamond dependency issues. -- **Simplified Dependency Management**: `MODULE.bazel` declares only direct dependencies, while transitive dependencies are automatically resolved, providing a clearer overview of the project's dependencies. -- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: Only direct dependencies are visible, ensuring correctness and predictability. +- **Deterministic Version Resolution**: Bazel adopts the deterministic + [MVS](module#version-selection) version resolution algorithm, + minimizing conflicts and addressing diamond dependency issues. +- **Simplified Dependency Management**: `MODULE.bazel` declares only direct + dependencies, while transitive dependencies are automatically resolved, + providing a clearer overview of the project's dependencies. +- **[Strict Dependency visibility](module#repository_names_and_strict_deps)**: + Only direct dependencies are visible, ensuring correctness and + predictability. ### Ecosystem Integration -- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized repository for discovering and managing common dependencies as Bazel modules. - -- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ library) is adapted for Bazel and made available in BCR, it streamlines its integration for the whole community and eliminates duplicated effort and conflicts of custom BUILD files. - -- **Unified Integration with Language-Specific Package Managers**: Rulesets streamline integration with external package managers for non-Bazel dependencies, including: - - - [rules\_jvm\_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) for Maven, - - [rules\_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) for PyPi, - - [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) for Go Modules, - - [rules\_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) for Cargo. +- **[Bazel Central Registry](https://registry.bazel.build/)**: A centralized + repository for discovering and managing common dependencies as Bazel + modules. +- **Adoption of Non-Bazel Projects**: When a non-Bazel project (usually a C++ + library) is adapted for Bazel and made available in BCR, it streamlines its + integration for the whole community and eliminates duplicated effort and + conflicts of custom BUILD files. +- **Unified Integration with Language-Specific Package Managers**: Rulesets + streamline integration with external package managers for non-Bazel + dependencies, including: + * [rules_jvm_external](https://github.com/bazel-contrib/rules_jvm_external/blob/master/docs/bzlmod.md) + for Maven, + * [rules_python](https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#using-bzlmod) + for PyPi, + * [bazel-gazelle](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/bzlmod.md#external-dependencies) + for Go Modules, + * [rules_rust](https://bazelbuild.github.io/rules_rust/crate_universe_bzlmod.html) + for Cargo. ### Advanced Features -- **[Module Extensions](extension)**: The [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module extension features allow flexible use of custom repository rules and resolution logic to introduce any non-Bazel dependencies. -- **[`bazel mod` Command](mod-command)**: The sub-command offers powerful ways to inspect external dependencies. You know exactly how an external dependency is defined and where it comes from. -- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you need to facilitate offline builds. -- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and accelerates dependency resolution. -- **(Upcoming) [BCR Provenance Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: Strengthen supply chain security by ensuring verified provenance of dependencies. +- **[Module Extensions](extension)**: The + [`use_repo_rule`](/rules/lib/globals/module#use_repo_rule) and module + extension features allow flexible use of custom repository rules and + resolution logic to introduce any non-Bazel dependencies. +- **[`bazel mod` Command](mod-command)**: The sub-command offers + powerful ways to inspect external dependencies. You know exactly how an + external dependency is defined and where it comes from. +- **[Vendor Mode](vendor)**: Pre-fetch the exact external dependencies you + need to facilitate offline builds. +- **[Lockfile](lockfile)**: The lockfile improves build reproducibility and + accelerates dependency resolution. +- **(Upcoming) [BCR Provenance + Attestations](https://github.com/bazelbuild/bazel-central-registry/discussions/2721)**: + Strengthen supply chain security by ensuring verified provenance of + dependencies. ## Concepts @@ -64,7 +117,8 @@ This section gives more detail on concepts related to external dependencies. ### Module -A Bazel project that can have multiple versions, each of which can have dependencies on other modules. +A Bazel project that can have multiple versions, each of which can have +dependencies on other modules. In a local Bazel workspace, a module is represented by a repository. @@ -72,71 +126,118 @@ For more details, see [Bazel modules](module). ### Repository -A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source +files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo +represents a Bazel module), `REPO.bazel` (see [below](#repo.bazel)), or in +legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file +will signify the boundary of a repo; multiple such files can coexist in a +directory. ### Main repository The repository in which the current Bazel command is being run. -The root of the main repository is also known as the [](). +The root of the main repository is also known as the +**workspace root**. ### Workspace -The environment shared by all Bazel commands run in the same main repository. It encompasses the main repo and the set of all defined external repos. +The environment shared by all Bazel commands run in the same main repository. It +encompasses the main repo and the set of all defined external repos. -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". ### Canonical repository name -The name by which a repository is always addressable. Within the context of a workspace, each repository has a single canonical name. A target inside a repo whose canonical name is `canonical_name` can be addressed by the label `@@canonical_name//package:target` (note the double `@`). +The name by which a repository is always addressable. Within the context of a +workspace, each repository has a single canonical name. A target inside a repo +whose canonical name is `canonical_name` can be addressed by the label +`@@canonical_name//package:target` (note the double `@`). The main repository always has the empty string as the canonical name. ### Apparent repository name -The name by which a repository is addressable in the context of a certain other repo. This can be thought of as a repo's "nickname": The repo with the canonical name `michael` might have the apparent name `mike` in the context of the repo `alice`, but might have the apparent name `mickey` in the context of the repo `bob`. In this case, a target inside `michael` can be addressed by the label `@mike//package:target` in the context of `alice` (note the single `@`). +The name by which a repository is addressable in the context of a certain other +repo. This can be thought of as a repo's "nickname": The repo with the canonical +name `michael` might have the apparent name `mike` in the context of the repo +`alice`, but might have the apparent name `mickey` in the context of the repo +`bob`. In this case, a target inside `michael` can be addressed by the label +`@mike//package:target` in the context of `alice` (note the single `@`). -Conversely, this can be understood as a **repository mapping**: each repo maintains a mapping from "apparent repo name" to a "canonical repo name". +Conversely, this can be understood as a **repository mapping**: each repo +maintains a mapping from "apparent repo name" to a "canonical repo name". ### Repository rule -A schema for repository definitions that tells Bazel how to materialize a repository. For example, it could be "download a zip archive from a certain URL and extract it", or "fetch a certain Maven artifact and make it available as a `java_import` target", or simply "symlink a local directory". Every repo is **defined** by calling a repo rule with an appropriate number of arguments. +A schema for repository definitions that tells Bazel how to materialize a +repository. For example, it could be "download a zip archive from a certain URL +and extract it", or "fetch a certain Maven artifact and make it available as a +`java_import` target", or simply "symlink a local directory". Every repo is +**defined** by calling a repo rule with an appropriate number of arguments. -See [Repository rules](repo) for more information about how to write your own repository rules. +See [Repository rules](repo) for more information about how to write +your own repository rules. -The most common repo rules by far are [`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive from a URL and extracts it, and [`local_repository`](/reference/be/workspace#local_repository), which symlinks a local directory that is already a Bazel repository. +The most common repo rules by far are +[`http_archive`](/rules/lib/repo/http#http_archive), which downloads an archive +from a URL and extracts it, and +[`local_repository`](/reference/be/workspace#local_repository), which symlinks a +local directory that is already a Bazel repository. ### Fetch a repository -The action of making a repo available on local disk by running its associated repo rule. The repos defined in a workspace are not available on local disk before they are fetched. +The action of making a repo available on local disk by running its associated +repo rule. The repos defined in a workspace are not available on local disk +before they are fetched. -Normally, Bazel only fetches a repo when it needs something from the repo, and the repo hasn't already been fetched. If the repo has already been fetched before, Bazel only re-fetches it if its definition has changed. +Normally, Bazel only fetches a repo when it needs something from the repo, +and the repo hasn't already been fetched. If the repo has already been fetched +before, Bazel only re-fetches it if its definition has changed. -The `fetch` command can be used to initiate a pre-fetch for a repository, target, or all necessary repositories to perform any build. This capability enables offline builds using the `--nofetch` option. +The `fetch` command can be used to initiate a pre-fetch for a repository, +target, or all necessary repositories to perform any build. This capability +enables offline builds using the `--nofetch` option. -The `--fetch` option serves to manage network access. Its default value is true. However, when set to false (`--nofetch`), the command will utilize any cached version of the dependency, and if none exists, the command will result in failure. +The `--fetch` option serves to manage network access. Its default value is true. +However, when set to false (`--nofetch`), the command will utilize any cached +version of the dependency, and if none exists, the command will result in +failure. -See [fetch options](/reference/command-line-reference#fetch-options) for more information about controlling fetch. +See [fetch options](/reference/command-line-reference#fetch-options) for more +information about controlling fetch. ### Directory layout -After being fetched, the repo can be found in the subdirectory `external` in the [output base](/remote/output-directories), under its canonical name. +After being fetched, the repo can be found in the subdirectory `external` in the +[output base](/remote/output-directories), under its canonical name. -You can run the following command to see the contents of the repo with the canonical name `canonical_name`: +You can run the following command to see the contents of the repo with the +canonical name `canonical_name`: ```posix-terminal -ls $(bazel info output_base)/external/<var> canonical_name </var> +ls $(bazel info output_base)/external/{{ '' }} canonical_name {{ '' }} ``` ### REPO.bazel file -The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost boundary of the directory tree that constitutes a repo. It doesn't need to contain anything to serve as a repo boundary file; however, it can also be used to specify some common attributes for all build targets inside the repo. +The [`REPO.bazel`](/rules/lib/globals/repo) file is used to mark the topmost +boundary of the directory tree that constitutes a repo. It doesn't need to +contain anything to serve as a repo boundary file; however, it can also be used +to specify some common attributes for all build targets inside the repo. -The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no `load` statements are supported. The `repo()` function takes the same arguments as the [`package()` function](/reference/be/functions#package) in `BUILD` files; whereas `package()` specifies common attributes for all build targets inside the package, `repo()` analogously does so for all build targets inside the repo. +The syntax of a `REPO.bazel` file is similar to `BUILD` files, except that no +`load` statements are supported. The `repo()` function takes the same arguments as the [`package()` +function](/reference/be/functions#package) in `BUILD` files; whereas `package()` +specifies common attributes for all build targets inside the package, `repo()` +analogously does so for all build targets inside the repo. -For example, you can specify a common license for all targets in your repo by having the following `REPO.bazel` file: +For example, you can specify a common license for all targets in your repo by +having the following `REPO.bazel` file: ```python repo( @@ -146,9 +247,12 @@ repo( ## The legacy WORKSPACE system -In older Bazel versions (before 9.0), external dependencies were introduced by defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a similar syntax to `BUILD` files, employing repo rules instead of build rules. +In older Bazel versions (before 9.0), external dependencies were introduced by +defining repos in the `WORKSPACE` (or `WORKSPACE.bazel`) file. This file has a +similar syntax to `BUILD` files, employing repo rules instead of build rules. -The following snippet is an example to use the `http_archive` repo rule in the `WORKSPACE` file: +The following snippet is an example to use the `http_archive` repo rule in the +`WORKSPACE` file: ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") @@ -159,26 +263,43 @@ http_archive( ) ``` -The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` system, by default, the canonical name of a repo is also its apparent name to all other repos. +The snippet defines a repo whose canonical name is `foo`. In the `WORKSPACE` +system, by default, the canonical name of a repo is also its apparent name to +all other repos. -See the [full list](/rules/lib/globals/workspace) of functions available in `WORKSPACE` files. +See the [full list](/rules/lib/globals/workspace) of functions available in +`WORKSPACE` files. ### Shortcomings of the `WORKSPACE` system -In the years after the `WORKSPACE` system was introduced, users reported many pain points, including: - -- Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all transitive dependencies must be defined in the `WORKSPACE` file of the main repo, in addition to direct dependencies. - -- To work around this, projects have adopted the "deps.bzl" pattern, in which they define a macro which in turn defines multiple repos, and ask users to call this macro in their `WORKSPACE` files. - - - This has its own problems: macros cannot `load` other `.bzl` files, so these projects have to define their transitive dependencies in this "deps" macro, or work around this issue by having the user call multiple layered "deps" macros. - - Bazel evaluates the `WORKSPACE` file sequentially. Additionally, dependencies are specified using `http_archive` with URLs, without any version information. This means that there is no reliable way to perform version resolution in the case of diamond dependencies (`A` depends on `B` and `C`; `B` and `C` both depend on different versions of `D`). - -Due to the shortcomings of WORKSPACE, the new module-based system (codenamed "Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. Read the [Bzlmod migration guide](migration) on how to migrate to Bzlmod. +In the years after the `WORKSPACE` system was introduced, users reported many +pain points, including: + +* Bazel does not evaluate the `WORKSPACE` files of any dependencies, so all + transitive dependencies must be defined in the `WORKSPACE` file of the main + repo, in addition to direct dependencies. +* To work around this, projects have adopted the "deps.bzl" pattern, in which + they define a macro which in turn defines multiple repos, and ask users to + call this macro in their `WORKSPACE` files. + * This has its own problems: macros cannot `load` other `.bzl` files, so + these projects have to define their transitive dependencies in this + "deps" macro, or work around this issue by having the user call multiple + layered "deps" macros. + * Bazel evaluates the `WORKSPACE` file sequentially. Additionally, + dependencies are specified using `http_archive` with URLs, without any + version information. This means that there is no reliable way to perform + version resolution in the case of diamond dependencies (`A` depends on + `B` and `C`; `B` and `C` both depend on different versions of `D`). + +Due to the shortcomings of WORKSPACE, the new module-based system (codenamed +"Bzlmod") gradually replaced the legacy WORKSPACE system between Bazel 6 and 9. +Read the [Bzlmod migration guide](migration) on how to migrate +to Bzlmod. ### External links on Bzlmod -- [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) -- [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) (original Bzlmod design doc) -- [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) -- [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) +* [Bzlmod usage examples in bazelbuild/examples](https://github.com/bazelbuild/examples/tree/main/bzlmod) +* [Bazel External Dependencies Overhaul](https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit) + (original Bzlmod design doc) +* [BazelCon 2021 talk on Bzlmod](https://www.youtube.com/watch?v=TxOCKtU39Fs) +* [Bazel Community Day talk on Bzlmod](https://www.youtube.com/watch?v=MB6xxis9gWI) diff --git a/external/registry.mdx b/external/registry.mdx deleted file mode 100644 index 48c7cf62..00000000 --- a/external/registry.mdx +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: 'Bazel registries' ---- - -Bazel discovers dependencies by requesting their information from Bazel *registries*: databases of Bazel modules. Bazel only supports one type of registries — [*index registries*](#index_registry) — local directories or static HTTP servers following a specific format. - -## Index registry - -An index registry is a local directory or a static HTTP server containing information about a list of modules — including their homepage, maintainers, the `MODULE.bazel` file of each version, and how to fetch the source of each version. Notably, it does *not* need to serve the source archives itself. - -An index registry must have the following format: - -- [`/bazel_registry.json`](#bazel-registry-json): An optional JSON file containing metadata for the registry. - -- `/modules`: A directory containing a subdirectory for each module in this registry - -- `/modules/$MODULE`: A directory containing a subdirectory for each version of the module named `$MODULE`, as well as the [`metadata.json` file](#metadata-json) containing metadata for this module. - -- `/modules/$MODULE/$VERSION`: A directory containing the following files: - - - `MODULE.bazel`: The `MODULE.bazel` file of this module version. Note that this is the `MODULE.bazel` file read during Bazel's external dependency resolution, *not* the one from the source archive (unless there's a [non-registry override](/external/module#non-registry_overrides)). Also note that it's best to use this file to set the version of a release and avoid doing so in the source archive `MODULE.bazel` file. To learn more about module versioning, [see the FAQ](faq.md#module-versioning-best-practices). - - [`source.json`](#source-json): A JSON file containing information on how to fetch the source of this module version - - `patches/`: An optional directory containing patch files, only used when `source.json` has "archive" type - - `overlay/`: An optional directory containing overlay files, only used when `source.json` has "archive" type - -### `bazel_registry.json` - -`bazel_registry.json` is an optional file that specifies metadata applying to the entire registry. It can contain the following fields: - -- `mirrors`: an array of strings, specifying the list of mirrors to use for source archives. - - The mirrored URL is a concatenation of the mirror itself, and the source URL of the module specified by its `source.json` file sans the protocol. For example, if a module's source URL is `https://foo.com/bar/baz`, and `mirrors` contains `["https://mirror1.com/", "https://example.com/mirror2/"]`, then the URLs Bazel will try in order are `https://mirror1.com/foo.com/bar/baz`, `https://example.com/mirror2/foo.com/bar/baz`, and finally the original source URL itself `https://foo.com/bar/baz`. -- `module_base_path`: a string, specifying the base path for modules with `local_path` type in the `source.json` file - -### `metadata.json` - -`metadata.json` is an optional JSON file containing information about the module, with the following fields: - -- `versions`: An array of strings, each denoting a version of the module available in this registry. This array should match the children of the module directory. -- `yanked_versions`: A JSON object specifying the [*yanked* versions](/external/module#yanked_versions) of this module. The keys should be versions to yank, and the values should be descriptions of why the version is yanked, ideally containing a link to more information. - -Note that the BCR requires more information in the `metadata.json` file. - -### `source.json` - -`source.json` is a required JSON file containing information about how to fetch a specific version of a module. The schema of this file depends on its `type` field, which defaults to `archive`. - -- If `type` is `archive` (the default), this module version is backed by an [`http_archive`](/rules/lib/repo/http#http_archive) repo rule; it's fetched by downloading an archive from a given URL and extracting its contents. It supports the following fields: - - - `url`: A string, the URL of the source archive - - `mirror_urls`: A list of string, the mirror URLs of the source archive. The URLs are tried in order after `url` as backups. - - `integrity`: A string, the [Subresource Integrity](https://w3c.github.io/webappsec-subresource-integrity/#integrity-metadata-description) checksum of the archive - - `strip_prefix`: A string, the directory prefix to strip when extracting the source archive - - `overlay`: A JSON object containing overlay files to layer on top of the extracted archive. The patch files are located under the `/modules/$MODULE/$VERSION/overlay` directory. The keys are the overlay file names, and the values are the integrity checksum of the overlay files. The overlays are applied before the patch files. - - `patches`: A JSON object containing patch files to apply to the extracted archive. The patch files are located under the `/modules/$MODULE/$VERSION/patches` directory. The keys are the patch file names, and the values are the integrity checksum of the patch files. The patches are applied after the overlay files and in the order they appear in `patches`. - - `patch_strip`: A number; the same as the `--strip` argument of Unix `patch`. - - `archive_type`: A string, the archive type of the downloaded file (Same as [`type` on `http_archive`](/rules/lib/repo/http#http_archive-type)). - -- If `type` is `git_repository`, this module version is backed by a [`git_repository`](/rules/lib/repo/git#git_repository) repo rule; it's fetched by cloning a Git repository. - - The following fields are supported, and are directly forwarded to the underlying `git_repository` repo rule: `remote`, `commit`, `shallow_since`, `tag`, `init_submodules`, `verbose`, and `strip_prefix`. - -- If `type` is `local_path`, this module version is backed by a [`local_repository`](/rules/lib/repo/local#local_repository) repo rule; it's symlinked to a directory on local disk. It supports the following field: - - - `path`: The local path to the repo, calculated as following: - - - If `path` is an absolute path, it stays as it is - - If `path` is a relative path and `module_base_path` is an absolute path, it resolves to `<module_base_path>/<path>` - - If `path` and `module_base_path` are both relative paths, it resolves to `<registry_path>/<module_base_path>/<path>`. Registry must be hosted locally and used by `--registry=file://<registry_path>`. Otherwise, Bazel will throw an error - -## Bazel Central Registry - -The Bazel Central Registry (BCR) at [https://bcr.bazel.build/](https://bcr.bazel.build/) is an index registry with contents backed by the GitHub repo [`bazelbuild/bazel-central-registry`](https://github.com/bazelbuild/bazel-central-registry). You can browse its contents using the web frontend at [https://registry.bazel.build/](https://registry.bazel.build/). - -The Bazel community maintains the BCR, and contributors are welcome to submit pull requests. See the [BCR contribution guidelines](https://github.com/bazelbuild/bazel-central-registry/blob/main/docs/README.md). - -In addition to following the format of a normal index registry, the BCR requires a `presubmit.yml` file for each module version (`/modules/$MODULE/$VERSION/presubmit.yml`). This file specifies a few essential build and test targets that you can use to check the validity of this module version. The BCR's CI pipelines also uses this to ensure interoperability between modules. - -## Selecting registries - -The repeatable Bazel flag `--registry` can be used to specify the list of registries to request modules from, so you can set up your project to fetch dependencies from a third-party or internal registry. Earlier registries take precedence. For convenience, you can put a list of `--registry` flags in the `.bazelrc` file of your project. - -If your registry is hosted on GitHub (for example, as a fork of `bazelbuild/bazel-central-registry`) then your `--registry` value needs a raw GitHub address under `raw.githubusercontent.com`. For example, on the `main` branch of the `my-org` fork, you would set `--registry=https://raw.githubusercontent.com/my-org/bazel-central-registry/main/`. - -Using the `--registry` flag stops the Bazel Central Registry from being used by default, but you can add it back by adding `--registry=https://bcr.bazel.build`. diff --git a/external/repo.mdx b/external/repo.mdx index bbf37aee..b878f030 100644 --- a/external/repo.mdx +++ b/external/repo.mdx @@ -2,19 +2,39 @@ title: 'Repository Rules' --- -This page covers how to define repository rules and provides examples for more details. -An [external repository](/external/overview#repository) is a directory tree, containing source files usable in a Bazel build, which is generated on demand by running its corresponding **repo rule**. Repos can be defined in a multitude of ways, but ultimately, each repo is defined by invoking a repo rule, just as build targets are defined by invoking build rules. They can be used to depend on third-party libraries (such as Maven packaged libraries) but also to generate `BUILD` files specific to the host Bazel is running on. + +This page covers how to define repository rules and provides examples for more +details. + +An [external repository](/external/overview#repository) is a directory tree, +containing source files usable in a Bazel build, which is generated on demand by +running its corresponding **repo rule**. Repos can be defined in a multitude of +ways, but ultimately, each repo is defined by invoking a repo rule, just as +build targets are defined by invoking build rules. They can be used to depend on +third-party libraries (such as Maven packaged libraries) but also to generate +`BUILD` files specific to the host Bazel is running on. ## Repository rule definition -In a `.bzl` file, use the [repository\_rule](/rules/lib/globals/bzl#repository_rule) function to define a new repo rule and store it in a global variable. After a repo rule is defined, it can be invoked as a function to define repos. This invocation is usually performed from inside a [module extension](/external/extension) implementation function. +In a `.bzl` file, use the +[repository_rule](/rules/lib/globals/bzl#repository_rule) function to define a +new repo rule and store it in a global variable. After a repo rule is defined, +it can be invoked as a function to define repos. This invocation is usually +performed from inside a [module extension](/external/extension) implementation +function. -The two major components of a repo rule definition are its attribute schema and implementation function. The attribute schema determines the names and types of attributes passed to a repo rule invocation, and the implementation function is run when the repo needs to be fetched. +The two major components of a repo rule definition are its attribute schema and +implementation function. The attribute schema determines the names and types of +attributes passed to a repo rule invocation, and the implementation function is +run when the repo needs to be fetched. ## Attributes -Attributes are arguments passed to the repo rule invocation. The schema of attributes accepted by a repo rule is specified using the `attrs` argument when the repo rule is defined with a call to `repository_rule`. An example defining `url` and `sha256` attributes as strings: +Attributes are arguments passed to the repo rule invocation. The schema of +attributes accepted by a repo rule is specified using the `attrs` argument when +the repo rule is defined with a call to `repository_rule`. An example defining +`url` and `sha256` attributes as strings: ```python http_archive = repository_rule( @@ -26,7 +46,8 @@ http_archive = repository_rule( ) ``` -To access an attribute within the implementation function, use `repository_ctx.attr.<attribute_name>`: +To access an attribute within the implementation function, use +`repository_ctx.attr.`: ```python def _impl(repository_ctx): @@ -34,15 +55,29 @@ def _impl(repository_ctx): checksum = repository_ctx.attr.sha256 ``` -All `repository_rule`s have the implicitly defined attribute `name`. This is a string attribute that behaves somewhat magically: when specified as an input to a repo rule invocation, it takes an apparent repo name; but when read from the repo rule's implementation function using `repository_ctx.attr.name`, it returns the canonical repo name. +All `repository_rule`s have the implicitly defined attribute `name`. This is a +string attribute that behaves somewhat magically: when specified as an input to +a repo rule invocation, it takes an apparent repo name; but when read from the +repo rule's implementation function using `repository_ctx.attr.name`, it returns +the canonical repo name. ## Implementation function -Every repo rule requires an `implementation` function. It contains the actual logic of the rule and is executed strictly in the Loading Phase. +Every repo rule requires an `implementation` function. It contains the actual +logic of the rule and is executed strictly in the Loading Phase. -The function has exactly one input parameter, `repository_ctx`. The function returns either `None` to signify that the rule is reproducible given the specified parameters, or a dict with a set of parameters for that rule that would turn that rule into a reproducible one generating the same repo. For example, for a rule tracking a git repository that would mean returning a specific commit identifier instead of a floating branch that was originally specified. +The function has exactly one input parameter, `repository_ctx`. The function +returns either `None` to signify that the rule is reproducible given the +specified parameters, or a dict with a set of parameters for that rule that +would turn that rule into a reproducible one generating the same repo. For +example, for a rule tracking a git repository that would mean returning a +specific commit identifier instead of a floating branch that was originally +specified. -The input parameter `repository_ctx` can be used to access attribute values, and non-hermetic functions (finding a binary, executing a binary, creating a file in the repository or downloading a file from the Internet). See [the API docs](/rules/lib/builtins/repository_ctx) for more context. Example: +The input parameter `repository_ctx` can be used to access attribute values, and +non-hermetic functions (finding a binary, executing a binary, creating a file in +the repository or downloading a file from the Internet). See [the API +docs](/rules/lib/builtins/repository_ctx) for more context. Example: ```python def _impl(repository_ctx): @@ -55,38 +90,72 @@ local_repository = repository_rule( ## When is the implementation function executed? -The implementation function of a repo rule is executed when Bazel needs a target from that repository, for example when another target (in another repo) depends on it or if it is mentioned on the command line. The implementation function is then expected to create the repo in the file system. This is called "fetching" the repo. - -In contrast to regular targets, repos are not necessarily re-fetched when something changes that would cause the repo to be different. This is because there are things that Bazel either cannot detect changes to or it would cause too much overhead on every build (for example, things that are fetched from the network). Therefore, repos are re-fetched only if one of the following things changes: - -- The attributes passed to the repo rule invocation. - -- The Starlark code comprising the implementation of the repo rule. - -- The value of any environment variable passed to `repository_ctx`'s `getenv()` method or declared with the `environ` attribute of the [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of these environment variables can be hard-wired on the command line with the [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. - -- The existence, contents, and type of any paths being [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation function of the repo rule. - - - Certain other methods of `repository_ctx` with a `watch` parameter, such as `read()`, `execute()`, and `extract()`, can also cause paths to be watched. - - Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths to be watched in other ways. - -- When `bazel fetch --force` is executed. - -There are two parameters of `repository_rule` that control when the repositories are re-fetched: - -- If the `configure` flag is set, the repository is re-fetched on `bazel fetch --force --configure` (non-`configure` repositories are not re-fetched). -- If the `local` flag is set, in addition to the above cases, the repo is also re-fetched when the Bazel server restarts. +The implementation function of a repo rule is executed when Bazel needs a target +from that repository, for example when another target (in another repo) depends +on it or if it is mentioned on the command line. The implementation function is +then expected to create the repo in the file system. This is called "fetching" +the repo. + +In contrast to regular targets, repos are not necessarily re-fetched when +something changes that would cause the repo to be different. This is because +there are things that Bazel either cannot detect changes to or it would cause +too much overhead on every build (for example, things that are fetched from the +network). Therefore, repos are re-fetched only if one of the following things +changes: + +* The attributes passed to the repo rule invocation. +* The Starlark code comprising the implementation of the repo rule. +* The value of any environment variable passed to `repository_ctx`'s + `getenv()` method or declared with the `environ` attribute of the + [`repository_rule`](/rules/lib/globals/bzl#repository_rule). The values of + these environment variables can be hard-wired on the command line with the + [`--repo_env`](/reference/command-line-reference#flag--repo_env) flag. +* The existence, contents, and type of any paths being + [`watch`ed](/rules/lib/builtins/repository_ctx#watch) in the implementation + function of the repo rule. + * Certain other methods of `repository_ctx` with a `watch` parameter, such + as `read()`, `execute()`, and `extract()`, can also cause paths to be + watched. + * Similarly, [`repository_ctx.watch_tree`](/rules/lib/builtins/repository_ctx#watch_tree) + and [`path.readdir`](/rules/lib/builtins/path#readdir) can cause paths + to be watched in other ways. +* When `bazel fetch --force` is executed. + +There are two parameters of `repository_rule` that control when the repositories +are re-fetched: + +* If the `configure` flag is set, the repository is re-fetched on `bazel + fetch --force --configure` (non-`configure` repositories are not + re-fetched). +* If the `local` flag is set, in addition to the above cases, the repo is also + re-fetched when the Bazel server restarts. ## Forcing refetch of external repos -Sometimes, an external repo can become outdated without any change to its definition or dependencies. For example, a repo fetching sources might follow a particular branch of a third-party repository, and new commits are available on that branch. In this case, you can ask bazel to refetch all external repos unconditionally by calling `bazel fetch --force --all`. +Sometimes, an external repo can become outdated without any change to its +definition or dependencies. For example, a repo fetching sources might follow a +particular branch of a third-party repository, and new commits are available on +that branch. In this case, you can ask bazel to refetch all external repos +unconditionally by calling `bazel fetch --force --all`. -Moreover, some repo rules inspect the local machine and might become outdated if the local machine was upgraded. Here you can ask Bazel to only refetch those external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) definition has the `configure` attribute set, use `bazel fetch --force --configure`. +Moreover, some repo rules inspect the local machine and might become outdated if +the local machine was upgraded. Here you can ask Bazel to only refetch those +external repos where the [`repository_rule`](/rules/lib/globals#repository_rule) +definition has the `configure` attribute set, use `bazel fetch --force +--configure`. ## Examples -- [C++ auto-configured toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): it uses a repo rule to automatically create the C++ configuration files for Bazel by looking for the local C++ compiler, the environment and the flags the C++ compiler supports. +- [C++ auto-configured + toolchain](https://cs.opensource.google/bazel/bazel/+/master:tools/cpp/cc_configure.bzl;drc=644b7d41748e09eff9e47cbab2be2263bb71f29a;l=176): + it uses a repo rule to automatically create the C++ configuration files for + Bazel by looking for the local C++ compiler, the environment and the flags + the C++ compiler supports. -- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) uses several `repository_rule` to defines the list of dependencies needed to use the Go rules. +- [Go repositories](https://github.com/bazelbuild/rules_go/blob/67bc217b6210a0922d76d252472b87e9a6118fdf/go/private/go_repositories.bzl#L195) + uses several `repository_rule` to defines the list of dependencies needed to + use the Go rules. -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) creates an external repository called `@maven` by default that generates build targets for every Maven artifact in the transitive dependency tree. +- [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) + creates an external repository called `@maven` by default that generates + build targets for every Maven artifact in the transitive dependency tree. diff --git a/external/vendor.mdx b/external/vendor.mdx index a2e79e8d..653c1942 100644 --- a/external/vendor.mdx +++ b/external/vendor.mdx @@ -1,8 +1,13 @@ +keywords: product:Bazel,Bzlmod,vendor --- title: 'Vendor Mode' --- -Vendor mode is a feature that lets you create a local copy of external dependencies. This is useful for offline builds, or when you want to control the source of an external dependency. + + +Vendor mode is a feature that lets you create a local copy of +external dependencies. This is useful for offline builds, or when you want to +control the source of an external dependency. ## Enable vendor mode @@ -11,15 +16,19 @@ You can enable vendor mode by specifying `--vendor_dir` flag. For example, by adding it to your `.bazelrc` file: ```none -# Enable vendor mode with vendor directory under <workspace>/vendor_src +# Enable vendor mode with vendor directory under /vendor_src common --vendor_dir=vendor_src ``` -The vendor directory can be either a relative path to your workspace root or an absolute path. +The vendor directory can be either a relative path to your workspace root or an +absolute path. ## Vendor a specific external repository -You can use the `vendor` command with the `--repo` flag to specify which repo to vendor, it accepts both [canonical repo name](/external/overview#canonical-repo-name) and [apparent repo name](/external/overview#apparent-repo-name). +You can use the `vendor` command with the `--repo` flag to specify which repo +to vendor, it accepts both [canonical repo +name](/external/overview#canonical-repo-name) and [apparent repo +name](/external/overview#apparent-repo-name). For example, running: @@ -33,11 +42,13 @@ or bazel vendor --vendor_dir=vendor_src --repo=@@rules_cc+ ``` -will both get rules\_cc to be vendored under `<workspace root>/vendor_src/rules_cc+`. +will both get rules_cc to be vendored under +`/vendor_src/rules_cc+`. ## Vendor external dependencies for given targets -To vendor all external dependencies required for building given target patterns, you can run `bazel vendor <target patterns>`. +To vendor all external dependencies required for building given target patterns, +you can run `bazel vendor `. For example @@ -45,9 +56,12 @@ For example bazel vendor --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -will vendor all repos required for building the `//src/main:hello-world` target and all targets under `//src/test/...` with the current configuration. +will vendor all repos required for building the `//src/main:hello-world` target +and all targets under `//src/test/...` with the current configuration. -Under the hood, it's doing a `bazel build --nobuild` command to analyze the target patterns, therefore build flags could be applied to this command and affect the result. +Under the hood, it's doing a `bazel build --nobuild` command to analyze the +target patterns, therefore build flags could be applied to this command and +affect the result. ### Build the target offline @@ -57,15 +71,20 @@ With the external dependencies vendored, you can build the target offline by bazel build --vendor_dir=vendor_src //src/main:hello-world //src/test/... ``` -The build should work in a clean build environment without network access and repository cache. +The build should work in a clean build environment without network access and +repository cache. -Therefore, you should be able to check in the vendored source and build the same targets offline on another machine. +Therefore, you should be able to check in the vendored source and build the same +targets offline on another machine. -Note: If you make changes to the targets to build, the external dependencies, the build configuration, or the Bazel version, you may need to re-vendor to make sure offline build still works. +Note: If you make changes to the targets to build, the external dependencies, +the build configuration, or the Bazel version, you may need to re-vendor to make +sure offline build still works. ## Vendor all external dependencies -To vendor all repos in your transitive external dependencies graph, you can run: +To vendor all repos in your transitive external dependencies graph, you can +run: ```none bazel vendor --vendor_dir=vendor_src @@ -73,20 +92,25 @@ bazel vendor --vendor_dir=vendor_src Note that vendoring all dependencies has a few **disadvantages**: -- Fetching all repos, including those introduced transitively, can be time-consuming. -- The vendor directory can become very large. -- Some repos may fail to fetch if they are not compatible with the current platform or environment. +- Fetching all repos, including those introduced transitively, can be time-consuming. +- The vendor directory can become very large. +- Some repos may fail to fetch if they are not compatible with the current platform or environment. Therefore, consider vendoring for specific targets first. ## Configure vendor mode with VENDOR.bazel -You can control how given repos are handled with the VENDOR.bazel file located under the vendor directory. +You can control how given repos are handled with the VENDOR.bazel file located +under the vendor directory. -There are two directives available, both accepting a list of [canonical repo names](/external/overview#canonical-repo-name) as arguments: +There are two directives available, both accepting a list of +[canonical repo names](/external/overview#canonical-repo-name) as arguments: - `ignore()`: to completely ignore a repository from vendor mode. -- `pin()`: to pin a repository to its current vendored source as if there is a `--override_repository` flag for this repo. Bazel will NOT update the vendored source for this repo while running the vendor command unless it's unpinned. The user can modify and maintain the vendored source for this repo manually. +- `pin()`: to pin a repository to its current vendored source as if there is a + `--override_repository` flag for this repo. Bazel will NOT update the vendored + source for this repo while running the vendor command unless it's unpinned. + The user can modify and maintain the vendored source for this repo manually. For example @@ -97,59 +121,93 @@ pin("@@bazel_skylib+") With this configuration -- Both repos will be excluded from subsequent vendor commands. -- Repo `bazel_skylib` will be overridden to the source located under the vendor directory. -- The user can safely modify the vendored source of `bazel_skylib`. -- To re-vendor `bazel_skylib`, the user has to disable the pin statement first. +- Both repos will be excluded from subsequent vendor commands. +- Repo `bazel_skylib` will be overridden to the source located under the + vendor directory. +- The user can safely modify the vendored source of `bazel_skylib`. +- To re-vendor `bazel_skylib`, the user has to disable the pin statement + first. -Note: Repository rules with [`local`](/rules/lib/globals/bzl#repository_rule.local) or [`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are always excluded from vendoring. +Note: Repository rules with +[`local`](/rules/lib/globals/bzl#repository_rule.local) or +[`configure`](/rules/lib/globals/bzl#repository_rule.configure) set to true are +always excluded from vendoring. ## Understand how vendor mode works -Bazel fetches external dependencies of a project under `$(bazel info output_base)/external`. Vendoring external dependencies means moving out relevant files and directories to the given vendor directory and use the vendored source for later builds. +Bazel fetches external dependencies of a project under `$(bazel info +output_base)/external`. Vendoring external dependencies means moving out +relevant files and directories to the given vendor directory and use the +vendored source for later builds. The content being vendored includes: -- The repo directory -- The repo marker file +- The repo directory +- The repo marker file -During a build, if the vendored marker file is up-to-date or the repo is pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating a symlink to it under `$(bazel info output_base)/external` instead of actually running the repository rule. Otherwise, a warning is printed and Bazel will fallback to fetching the latest version of the repo. +During a build, if the vendored marker file is up-to-date or the repo is +pinned in the VENDOR.bazel file, then Bazel uses the vendored source by creating +a symlink to it under `$(bazel info output_base)/external` instead of actually +running the repository rule. Otherwise, a warning is printed and Bazel will +fallback to fetching the latest version of the repo. -Note: Bazel assumes the vendored source is not changed by users unless the repo is pinned in the VENDOR.bazel file. If a user does change the vendored source without pinning the repo, the changed vendored source will be used, but it will be overwritten if its existing marker file is outdated and the repo is vendored again. +Note: Bazel assumes the vendored source is not changed by users unless the repo +is pinned in the VENDOR.bazel file. If a user does change the vendored source +without pinning the repo, the changed vendored source will be used, but it will +be overwritten if its existing marker file is +outdated and the repo is vendored again. ### Vendor registry files -Bazel has to perform the Bazel module resolution in order to fetch external dependencies, which may require accessing registry files through internet. To achieve offline build, Bazel vendors all registry files fetched from network under the `<vendor_dir>/_registries` directory. +Bazel has to perform the Bazel module resolution in order to fetch external +dependencies, which may require accessing registry files through internet. To +achieve offline build, Bazel vendors all registry files fetched from +network under the `/_registries` directory. ### Vendor symlinks -External repositories may contain symlinks pointing to other files or directories. To make sure symlinks work correctly, Bazel uses the following strategy to rewrite symlinks in the vendored source: +External repositories may contain symlinks pointing to other files or +directories. To make sure symlinks work correctly, Bazel uses the following +strategy to rewrite symlinks in the vendored source: -- Create a symlink `<vendor_dir>/bazel-external` that points to `$(bazel info output_base)/external`. It is refreshed by every Bazel command automatically. -- For the vendored source, rewrite all symlinks that originally point to a path under `$(bazel info output_base)/external` to a relative path under `<vendor_dir>/bazel-external`. +- Create a symlink `/bazel-external` that points to `$(bazel info + output_base)/external`. It is refreshed by every Bazel command + automatically. +- For the vendored source, rewrite all symlinks that originally point to a + path under `$(bazel info output_base)/external` to a relative path under + `/bazel-external`. For example, if the original symlink is ```none -<vendor_dir>/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file +/repo_foo+/link => $(bazel info output_base)/external/repo_bar+/file ``` It will be rewritten to ```none -<vendor_dir>/repo_foo+/link => ../../bazel-external/repo_bar+/file +/repo_foo+/link => ../../bazel-external/repo_bar+/file ``` where ```none -<vendor_dir>/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed +/bazel-external => $(bazel info output_base)/external # This might be new if output base is changed ``` -Since `<vendor_dir>/bazel-external` is generated by Bazel automatically, it's recommended to add it to `.gitignore` or equivalent to avoid checking it in. +Since `/bazel-external` is generated by Bazel automatically, it's +recommended to add it to `.gitignore` or equivalent to avoid checking it in. + +With this strategy, symlinks in the vendored source should work correctly even +after the vendored source is moved to another location or the bazel output base +is changed. -With this strategy, symlinks in the vendored source should work correctly even after the vendored source is moved to another location or the bazel output base is changed. +Note: symlinks that point to an absolute path outside of $(bazel info +output_base)/external are not rewritten. Therefore, it could still break +cross-machine compatibility. -Note: symlinks that point to an absolute path outside of $(bazel info output\_base)/external are not rewritten. Therefore, it could still break cross-machine compatibility. +Note: On Windows, vendoring symlinks only works with +[`--windows_enable_symlinks`][windows_enable_symlinks] +flag enabled. -Note: On Windows, vendoring symlinks only works with [`--windows_enable_symlinks`](/reference/command-line-reference#flag--windows_enable_symlinks) flag enabled. +[windows_enable_symlinks]: /reference/command-line-reference#flag--windows_enable_symlinks diff --git a/help.mdx b/help.mdx index af11fb80..754b4bc2 100644 --- a/help.mdx +++ b/help.mdx @@ -2,49 +2,53 @@ title: 'Getting Help' --- -This page lists Bazel resources beyond the documentation and covers how to get support from the Bazel team and community. + + +This page lists Bazel resources beyond the documentation and covers how to get +support from the Bazel team and community. ## Search existing material In addition to the documentation, you can find helpful information by searching: -- [Bazel user group](https://groups.google.com/g/bazel-discuss) -- [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) -- [Bazel blog](https://blog.bazel.build/) -- [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -- [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) +* [Bazel user group](https://groups.google.com/g/bazel-discuss) +* [Bazel GitHub Discussions](https://github.com/bazelbuild/bazel/discussions) +* [Bazel blog](https://blog.bazel.build/) +* [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +* [`awesome-bazel` resources](https://github.com/jin/awesome-bazel) ## Watch videos There are recordings of Bazel talks at various conferences, such as: -- Bazel’s annual conference, BazelCon: - - - [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) - - [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) - - [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) - - [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) - - [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) - - [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) - - [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) - - [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +* Bazel’s annual conference, BazelCon: + * [BazelCon 2024](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ptKfAQNZ5RS4HMdmeilBcw) + * [BazelCon 2023](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rUiqylH-kumoZCWntG1vjp) + * [BazelCon 2022](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rABfcAJO1VWeOUYL1kIn-p) + * [BazelCon 2021](https://www.youtube.com/playlist?list=PLbzoR-pLrL6pO6BaaQ1Ndos53gfRVLEoU) + * [BazelCon 2020](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qZ5JRMtn20_s2uPz9vFYgU) + * [BazelCon 2019](https://www.youtube.com/playlist?list=PLbzoR-pLrL6ogKgytQXqUxJQ6nZlIWoTH) + * [BazelCon 2018](https://www.youtube.com/playlist?list=PLbzoR-pLrL6rBDwC0NMRPS8EJ0VRAW0QR) + * [BazelCon 2017](https://www.youtube.com/playlist?list=PLbzoR-pLrL6qvwchdtlSopLgUrz4J4zKP) +* Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) -- Bazel day on [Google Open Source Live](https://opensourcelive.withgoogle.com/events/bazel) ## Ask the Bazel community If there are no existing answers, you can ask the community by: -- Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) -- Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) -- Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) -- Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) -- Consulting a [Bazel community expert](/community/experts) +* Emailing the [Bazel user group](https://groups.google.com/g/bazel-discuss) +* Starting a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions) +* Asking a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/bazel) +* Chatting with other Bazel contributors on [Slack](https://slack.bazel.build/) +* Consulting a [Bazel community expert](/community/experts) ## Understand Bazel's support level -Please read the [release page](/release) to understand Bazel's release model and what level of support Bazel provides. +Please read the [release page](/release) to understand Bazel's release model and +what level of support Bazel provides. ## File a bug -If you encounter a bug or want to request a feature, file a [GitHub Issue](https://github.com/bazelbuild/bazel/issues). +If you encounter a bug or want to request a feature, file a [GitHub +Issue](https://github.com/bazelbuild/bazel/issues). diff --git a/install/bazelisk.mdx b/install/bazelisk.mdx index 4bc9f3e5..a3189cb7 100644 --- a/install/bazelisk.mdx +++ b/install/bazelisk.mdx @@ -2,39 +2,58 @@ title: 'Installing / Updating Bazel using Bazelisk' --- + + ## Installing Bazel -[Bazelisk](https://github.com/bazelbuild/bazelisk) is the recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically downloads and installs the appropriate version of Bazel. Use Bazelisk if you need to switch between different versions of Bazel depending on the current working directory, or to always keep Bazel updated to the latest release. +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the +recommended way to install Bazel on Ubuntu, Windows, and macOS. It automatically +downloads and installs the appropriate version of Bazel. Use Bazelisk if you +need to switch between different versions of Bazel depending on the current +working directory, or to always keep Bazel updated to the latest release. -For more details, see [the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). +For more details, see +[the official README](https://github.com/bazelbuild/bazelisk/blob/master/README.md). ## Updating Bazel -Bazel has a [backward compatibility policy](/release/backward-compatibility) (see [guidance for rolling out incompatible changes](/contribute/breaking-changes) if you are the author of one). That page summarizes best practices on how to test and migrate your project with upcoming incompatible changes and how to provide feedback to the incompatible change authors. +Bazel has a [backward compatibility policy](/release/backward-compatibility) +(see [guidance for rolling out incompatible +changes](/contribute/breaking-changes) if you +are the author of one). That page summarizes best practices on how to test and +migrate your project with upcoming incompatible changes and how to provide +feedback to the incompatible change authors. ### Managing Bazel versions with Bazelisk -[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage Bazel versions. +[Bazelisk](https://github.com/bazelbuild/bazelisk) helps you manage +Bazel versions. Bazelisk can: -- Auto-update Bazel to the latest LTS or rolling release. -- Build the project with a Bazel version specified in the .bazelversion file. Check in that file into your version control to ensure reproducibility of your builds. -- Help migrate your project for incompatible changes (see above) -- Easily try release candidates +* Auto-update Bazel to the latest LTS or rolling release. +* Build the project with a Bazel version specified in the .bazelversion + file. Check in that file into your version control to ensure reproducibility + of your builds. +* Help migrate your project for incompatible changes (see above) +* Easily try release candidates ### Recommended migration process -Within minor updates to any LTS release, any project can be prepared for the next release without breaking compatibility with the current release. However, there may be backward-incompatible changes between major LTS versions. +Within minor updates to any LTS release, any +project can be prepared for the next release without breaking +compatibility with the current release. However, there may be +backward-incompatible changes between major LTS versions. Follow this process to migrate from one major version to another: 1. Read the release notes to get advice on how to migrate to the next version. - -2. Major incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue: - - - Migration guidance is available in the associated GitHub issue. - - Tooling is available for some of incompatible changes migration. For example, [buildifier](https://github.com/bazelbuild/buildtools/releases). - - Report migration problems by commenting on the associated GitHub issue. - -After migration, you can continue to build your projects without worrying about backward-compatibility until the next major release. +1. Major incompatible changes should have an associated `--incompatible_*` flag + and a corresponding GitHub issue: + * Migration guidance is available in the associated GitHub issue. + * Tooling is available for some of incompatible changes migration. For + example, [buildifier](https://github.com/bazelbuild/buildtools/releases). + * Report migration problems by commenting on the associated GitHub issue. + +After migration, you can continue to build your projects without worrying about +backward-compatibility until the next major release. diff --git a/install/compile-source.mdx b/install/compile-source.mdx index 8fc9f63c..3b87883d 100644 --- a/install/compile-source.mdx +++ b/install/compile-source.mdx @@ -2,67 +2,91 @@ title: 'Compiling Bazel from Source' --- -This page describes how to install Bazel from source and provides troubleshooting tips for common issues. + + +This page describes how to install Bazel from source and provides +troubleshooting tips for common issues. To build Bazel from source, you can do one of the following: -- Build it [using an existing Bazel binary](#build-bazel-using-bazel) +* Build it [using an existing Bazel binary](#build-bazel-using-bazel) -- Build it [without an existing Bazel binary](#bootstrap-bazel) which is known as *bootstrapping*. +* Build it [without an existing Bazel binary](#bootstrap-bazel) which is known + as _bootstrapping_. ## Build Bazel using Bazel ### Summary -1. Get the latest Bazel release from the [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with [Bazelisk](https://github.com/bazelbuild/bazelisk). +1. Get the latest Bazel release from the + [GitHub release page](https://github.com/bazelbuild/bazel/releases) or with + [Bazelisk](https://github.com/bazelbuild/bazelisk). -2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) and extract somewhere. Alternatively you can git clone the source tree from [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) +2. [Download Bazel's sources from GitHub](https://github.com/bazelbuild/bazel/archive/master.zip) + and extract somewhere. + Alternatively you can git clone the source tree from https://github.com/bazelbuild/bazel -3. Install the same prerequisites as for bootstrapping (see [for Unix-like systems](#bootstrap-unix-prereq) or [for Windows](#bootstrap-windows-prereq)) +3. Install the same prerequisites as for bootstrapping (see + [for Unix-like systems](#bootstrap-unix-prereq) or + [for Windows](#bootstrap-windows-prereq)) -4. Build a development build of Bazel using Bazel: `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on Windows) +4. Build a development build of Bazel using Bazel: + `bazel build //src:bazel-dev` (or `bazel build //src:bazel-dev.exe` on + Windows) -5. The resulting binary is at `bazel-bin/src/bazel-dev` (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you like and use immediately without further installation. +5. The resulting binary is at `bazel-bin/src/bazel-dev` + (or `bazel-bin\src\bazel-dev.exe` on Windows). You can copy it wherever you + like and use immediately without further installation. Detailed instructions follow below. ### Step 1: Get the latest Bazel release -**Goal**: Install or download a release version of Bazel. Make sure you can run it by typing `bazel` in a terminal. +**Goal**: Install or download a release version of Bazel. Make sure you can run +it by typing `bazel` in a terminal. -**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing Bazel binary. You can install one from a package manager or download one from GitHub. See [Installing Bazel](/install). (Or you can [build from scratch (bootstrap)](#bootstrap-bazel).) +**Reason**: To build Bazel from a GitHub source tree, you need a pre-existing +Bazel binary. You can install one from a package manager or download one from +GitHub. See [Installing Bazel](/install). (Or you can [build from +scratch (bootstrap)](#bootstrap-bazel).) **Troubleshooting**: -- If you cannot run Bazel by typing `bazel` in a terminal: +* If you cannot run Bazel by typing `bazel` in a terminal: - - Maybe your Bazel binary's directory is not on the PATH. + * Maybe your Bazel binary's directory is not on the PATH. - This is not a big problem. Instead of typing `bazel`, you will need to type the full path. + This is not a big problem. Instead of typing `bazel`, you will need to + type the full path. - - Maybe the Bazel binary itself is not called `bazel` (on Unixes) or `bazel.exe` (on Windows). + * Maybe the Bazel binary itself is not called `bazel` (on Unixes) or + `bazel.exe` (on Windows). - This is not a big problem. You can either rename the binary, or type the binary's name instead of `bazel`. + This is not a big problem. You can either rename the binary, or type the + binary's name instead of `bazel`. - - Maybe the binary is not executable (on Unixes). + * Maybe the binary is not executable (on Unixes). - You must make the binary executable by running `chmod +x /path/to/bazel`. + You must make the binary executable by running `chmod +x /path/to/bazel`. ### Step 2: Download Bazel's sources from GitHub -If you are familiar with Git, then just git clone [https://github.com/bazelbuild/bazel](https://github.com/bazelbuild/bazel) +If you are familiar with Git, then just git clone https://github.com/bazelbuild/bazel Otherwise: -1. Download the [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). +1. Download the + [latest sources as a zip file](https://github.com/bazelbuild/bazel/archive/master.zip). -2. Extract the contents somewhere. +2. Extract the contents somewhere. - For example create a `bazel-src` directory under your home directory and extract there. + For example create a `bazel-src` directory under your home directory and + extract there. ### Step 3: Install prerequisites -Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ compiler, MSYS2 (if you are building on Windows), etc. +Install the same prerequisites as for bootstrapping (see below) -- JDK, C++ +compiler, MSYS2 (if you are building on Windows), etc. ### Step 4a: Build Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -72,63 +96,66 @@ For instructions for Windows, see [Build Bazel on Windows](#build-bazel-on-windo **Instructions**: -1. Start a Bash terminal +1. Start a Bash terminal -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - ``` - cd ~/bazel-src - ``` + cd ~/bazel-src -3. Build Bazel from source: +3. Build Bazel from source: - ``` - bazel build //src:bazel-dev - ``` + bazel build //src:bazel-dev - Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel --compilation_mode=opt` + to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel + version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). +4. The output will be at `bazel-bin/src/bazel-dev` (or `bazel-bin/src/bazel`). ### Step 4b: Build Bazel on Windows -For instructions for Unix-like systems, see [Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). +For instructions for Unix-like systems, see +[Ubuntu Linux, macOS, and other Unix-like systems](#build-bazel-on-unixes). -**Goal**: Run Bazel to build a custom Bazel binary (`bazel-bin\src\bazel-dev.exe`). +**Goal**: Run Bazel to build a custom Bazel binary +(`bazel-bin\src\bazel-dev.exe`). **Instructions**: -1. Start Command Prompt (Start Menu > Run > "cmd.exe") +1. Start Command Prompt (Start Menu > Run > "cmd.exe") -2. `cd` into the directory where you extracted (or cloned) Bazel's sources. +2. `cd` into the directory where you extracted (or cloned) Bazel's sources. - For example if you extracted the sources under your home directory, run: + For example if you extracted the sources under your home directory, run: - ``` - cd %USERPROFILE%\bazel-src - ``` + cd %USERPROFILE%\bazel-src -3. Build Bazel from source: +3. Build Bazel from source: - bazel build //src:bazel-dev.exe + bazel build //src:bazel-dev.exe - Alternatively you can run `bazel build //src:bazel.exe --compilation_mode=opt` to yield a smaller binary but it's slower to build. + Alternatively you can run `bazel build //src:bazel.exe + --compilation_mode=opt` to yield a smaller binary but it's slower to build. - You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel version for the binary so that `bazel --version` outputs the given version. + You can build with `--stamp --embed_label=X.Y.Z` flag to embed a Bazel + version for the binary so that `bazel --version` outputs the given version. -4. The output will be at `bazel-bin\src\bazel-dev.exe` (or `bazel-bin\src\bazel.exe`). +4. The output will be at `bazel-bin\src\bazel-dev.exe` (or + `bazel-bin\src\bazel.exe`). ### Step 5: Install the built binary Actually, there's nothing to install. -The output of the previous step is a self-contained Bazel binary. You can copy it to any directory and use immediately. (It's useful if that directory is on your PATH so that you can run "bazel" everywhere.) +The output of the previous step is a self-contained Bazel binary. You can copy +it to any directory and use immediately. (It's useful if that directory is on +your PATH so that you can run "bazel" everywhere.) -*** +--- ## Build Bazel from scratch (bootstrapping) @@ -138,16 +165,24 @@ You can also build Bazel from scratch, without using an existing Bazel binary. (This step is the same for all platforms.) -1. Download `bazel-<version>-dist.zip` from [GitHub](https://github.com/bazelbuild/bazel/releases), for example `bazel-0.28.1-dist.zip`. +1. Download `bazel--dist.zip` from + [GitHub](https://github.com/bazelbuild/bazel/releases), for example + `bazel-0.28.1-dist.zip`. - **Attention**: + **Attention**: - - There is a **single, architecture-independent** distribution archive. There are no architecture-specific or OS-specific distribution archives. - - These sources are **not the same as the GitHub source tree**. You have to use the distribution archive to bootstrap Bazel. You cannot use a source tree cloned from GitHub. (The distribution archive contains generated source files that are required for bootstrapping and are not part of the normal Git source tree.) + - There is a **single, architecture-independent** distribution archive. + There are no architecture-specific or OS-specific distribution archives. + - These sources are **not the same as the GitHub source tree**. You + have to use the distribution archive to bootstrap Bazel. You cannot + use a source tree cloned from GitHub. (The distribution archive contains + generated source files that are required for bootstrapping and are not part + of the normal Git source tree.) -2. Unpack the distribution archive somewhere on disk. +2. Unpack the distribution archive somewhere on disk. - You should verify the signature made by Bazel's [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. + You should verify the signature made by Bazel's + [release key](https://bazel.build/bazel-release.pub.gpg) 3D5919B448457EE0. ### Step 2a: Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems @@ -155,17 +190,18 @@ For instructions for Windows, see [Bootstrap Bazel on Windows](#bootstrap-window #### 2.1. Install the prerequisites -- **Bash** +* **Bash** -- **zip, unzip** +* **zip, unzip** -- **C++ build toolchain** +* **C++ build toolchain** -- **JDK.** Version 21 is required. +* **JDK.** Version 21 is required. -- **Python**. Version 3 is required. +* **Python**. Version 3 is required. -For example on Ubuntu Linux you can install these requirements using the following command: +For example on Ubuntu Linux you can install these requirements using the +following command: ```sh sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip @@ -173,76 +209,90 @@ sudo apt-get install build-essential openjdk-21-jdk python3 zip unzip #### 2.2. Bootstrap Bazel on Unix -1. Open a shell or Terminal window. +1. Open a shell or Terminal window. -2. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. +3. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh`. -The compiled output is placed into `output/bazel`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH` (such as `/usr/local/bin` on Linux). +The compiled output is placed into `output/bazel`. This is a self-contained +Bazel binary, without an embedded JDK. You can copy it anywhere or use it +in-place. For convenience, copy this binary to a directory that's on your +`PATH` (such as `/usr/local/bin` on Linux). -To build the `bazel` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. +To build the `bazel` binary in a reproducible way, also set +[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) +in the "Run the compilation script" step. ### Step 2b: Bootstrap Bazel on Windows -For instructions for Unix-like systems, see [Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). +For instructions for Unix-like systems, see +[Bootstrap Bazel on Ubuntu Linux, macOS, and other Unix-like systems](#bootstrap-unix). #### 2.1. Install the prerequisites -- [MSYS2 shell](https://msys2.github.io/) +* [MSYS2 shell](https://msys2.github.io/) -- **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: +* **The MSYS2 packages for zip and unzip.** Run the following command in the MSYS2 shell: - ``` - pacman -S zip unzip patch - ``` + ``` + pacman -S zip unzip patch + ``` -- **The Visual C++ compiler.** Install the Visual C++ compiler either as part of Visual Studio 2015 or newer, or by installing the latest [Build Tools for Visual Studio 2017](https://aka.ms/BuildTools). +* **The Visual C++ compiler.** Install the Visual C++ compiler either as part + of Visual Studio 2015 or newer, or by installing the latest [Build Tools + for Visual Studio 2017](https://aka.ms/BuildTools). -- **JDK.** Version 21 is required. +* **JDK.** Version 21 is required. -- **Python**. Versions 2 and 3 are supported, installing one of them is enough. You need the Windows-native version (downloadable from [https://www.python.org](https://www.python.org)). Versions installed via pacman in MSYS2 will not work. +* **Python**. Versions 2 and 3 are supported, installing one of them is + enough. You need the Windows-native version (downloadable from + [https://www.python.org](https://www.python.org)). Versions installed via + pacman in MSYS2 will not work. #### 2.2. Bootstrap Bazel on Windows -1. Open the MSYS2 shell. - -2. Set the following environment variables: - - - Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the path to the Visual Studio directory (BAZEL\_V**S**) or to the Visual C++ directory (BAZEL\_V**C**). Setting one of them is enough. - - - `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the examples below. - - Do not set this to `C:\Windows\System32\bash.exe`. (You have that file if you installed Windows Subsystem for Linux.) Bazel does not support this version of `bash.exe`. +1. Open the MSYS2 shell. - - `PATH`: Add the Python directory. +2. Set the following environment variables: + * Either `BAZEL_VS` or `BAZEL_VC` (they are *not* the same): Set to the + path to the Visual Studio directory (BAZEL\_VS) or to the Visual + C++ directory (BAZEL\_VC). Setting one of them is enough. + * `BAZEL_SH`: Path of the MSYS2 `bash.exe`. See the command in the + examples below. - - `JAVA_HOME`: Set to the JDK directory. + Do not set this to `C:\Windows\System32\bash.exe`. (You have that file + if you installed Windows Subsystem for Linux.) Bazel does not support + this version of `bash.exe`. + * `PATH`: Add the Python directory. + * `JAVA_HOME`: Set to the JDK directory. - **Example** (using BAZEL\_V**S**): + **Example** (using BAZEL\_VS): - ``` - export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" - ``` + export BAZEL_VS="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" - or (using BAZEL\_V**C**): + or (using BAZEL\_VC): - ``` - export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" - export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" - export PATH="/c/python27:$PATH" - export JAVA_HOME="C:/Program Files/Java/jdk-21" - ``` + export BAZEL_VC="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC" + export BAZEL_SH="$(cygpath -m $(realpath $(which bash)))" + export PATH="/c/python27:$PATH" + export JAVA_HOME="C:/Program Files/Java/jdk-21" -3. `cd` to the directory where you unpacked the distribution archive. +3. `cd` to the directory where you unpacked the distribution archive. -4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` +4. Run the compilation script: `env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" ./compile.sh` -The compiled output is placed into `output/bazel.exe`. This is a self-contained Bazel binary, without an embedded JDK. You can copy it anywhere or use it in-place. For convenience, copy this binary to a directory that's on your `PATH`. +The compiled output is placed into `output/bazel.exe`. This is a self-contained +Bazel binary, without an embedded JDK. You can copy it anywhere or use it +in-place. For convenience, copy this binary to a directory that's on +your `PATH`. -To build the `bazel.exe` binary in a reproducible way, also set [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) in the "Run the compilation script" step. +To build the `bazel.exe` binary in a reproducible way, also set +[`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) +in the "Run the compilation script" step. -You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the Command Prompt (`cmd.exe`) or PowerShell. +You don't need to run Bazel from the MSYS2 shell. You can run Bazel from the +Command Prompt (`cmd.exe`) or PowerShell. diff --git a/install/completion.mdx b/install/completion.mdx index 309ce3f8..1d1d1b76 100644 --- a/install/completion.mdx +++ b/install/completion.mdx @@ -2,7 +2,11 @@ title: 'Command-Line Completion' --- -You can enable command-line completion (also known as tab-completion) in Bash and Zsh. This lets you tab-complete command names, flags names and flag values, and target names. + + +You can enable command-line completion (also known as tab-completion) in Bash +and Zsh. This lets you tab-complete command names, flags names and flag values, +and target names. ## Bash @@ -10,51 +14,55 @@ Bazel comes with a Bash completion script. If you installed Bazel: -- From the APT repository, then you're done -- the Bash completion script is already installed in `/etc/bash_completion.d`. - -- From Homebrew, then you're done -- the Bash completion script is already installed in `$(brew --prefix)/etc/bash_completion.d`. - -- From the installer downloaded from GitHub, then: - - 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. - - 2. Do one of the following: - - - Either copy this file to your completion directory (if you have one). - - Example: on Ubuntu this is the `/etc/bash_completion.d` directory. - - - Or source the completion file from Bash's RC file. +* From the APT repository, then you're done -- the Bash completion script is + already installed in `/etc/bash_completion.d`. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: +* From Homebrew, then you're done -- the Bash completion script is + already installed in `$(brew --prefix)/etc/bash_completion.d`. - ``` - source /path/to/bazel-complete.bash - ``` +* From the installer downloaded from GitHub, then: + 1. Locate the absolute path of the completion file. The installer copied it + to the `bin` directory. -- Via [bootstrapping](/install/compile-source), then: + Example: if you ran the installer with `--user`, this will be + `$HOME/.bazel/bin`. If you ran the installer as root, this will be + `/usr/local/lib/bazel/bin`. + 2. Do one of the following: + * Either copy this file to your completion directory (if you have + one). - 1. Emit the completion script into a file: + Example: on Ubuntu this is the `/etc/bash_completion.d` directory. + * Or source the completion file from Bash's RC file. - ``` - bazel help completion bash > bazel-complete.bash - ``` + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) + or `~/.bash_profile` (on macOS), using the path to your completion + file's absolute path: - 2. Do one of the following: + ``` + source /path/to/bazel-complete.bash + ``` - - Copy this file to your completion directory, if you have one. +* Via [bootstrapping](/install/compile-source), then: + 1. Emit the completion script into a file: - Example: on Ubuntu this is the `/etc/bash_completion.d` directory + ``` + bazel help completion bash > bazel-complete.bash + ``` + 2. Do one of the following: + * Copy this file to your completion directory, if you have + one. - - Copy it somewhere on your local disk, such as to `$HOME`, and source the completion file from Bash's RC file. + Example: on Ubuntu this is the `/etc/bash_completion.d` directory + * Copy it somewhere on your local disk, such as to `$HOME`, and + source the completion file from Bash's RC file. - Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) or `~/.bash_profile` (on macOS), using the path to your completion file's absolute path: + Add a line similar to the one below to your `~/.bashrc` (on Ubuntu) + or `~/.bash_profile` (on macOS), using the path to your completion + file's absolute path: - ``` - source /path/to/bazel-complete.bash - ``` + ``` + source /path/to/bazel-complete.bash + ``` ## Zsh @@ -62,48 +70,57 @@ Bazel comes with a Zsh completion script. If you installed Bazel: -- From the APT repository, then you're done -- the Zsh completion script is already installed in `/usr/share/zsh/vendor-completions`. - - > If you have a heavily customized `.zshrc` and the autocomplete does not function, try one of the following solutions: - > - > Add the following to your `.zshrc`: - > - > ``` - > zstyle :compinstall filename '/home/tradical/.zshrc' - > - > autoload -Uz compinit - > compinit - > ``` - > - > or - > - > Follow the instructions [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) - > - > If you are using `oh-my-zsh`, you may want to install and enable the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the solutions described above. - -- From Homebrew, then you're done -- the Zsh completion script is already installed in `$(brew --prefix)/share/zsh/site-functions`. - -- From the installer downloaded from GitHub, then: - - 1. Locate the absolute path of the completion file. The installer copied it to the `bin` directory. - - Example: if you ran the installer with `--user`, this will be `$HOME/.bazel/bin`. If you ran the installer as root, this will be `/usr/local/lib/bazel/bin`. - - 2. Add this script to a directory on your `$fpath`: - - ``` - fpath[1,0]=~/.zsh/completion/ - mkdir -p ~/.zsh/completion/ - cp /path/from/above/step/_bazel ~/.zsh/completion - ``` - - You may have to call `rm -f ~/.zcompdump; compinit` the first time to make it work. - - 3. Optionally, add the following to your .zshrc. - - ``` - # This way the completion script does not have to parse Bazel's options - # repeatedly. The directory in cache-path must be created manually. - zstyle ':completion:*' use-cache on - zstyle ':completion:*' cache-path ~/.zsh/cache - ``` +* From the APT repository, then you're done -- the Zsh completion script is + already installed in `/usr/share/zsh/vendor-completions`. + + > If you have a heavily customized `.zshrc` and the autocomplete + > does not function, try one of the following solutions: + > + > Add the following to your `.zshrc`: + > + > ``` + > zstyle :compinstall filename '/home/tradical/.zshrc' + > + > autoload -Uz compinit + > compinit + > ``` + > + > or + > + > Follow the instructions + > [here](https://stackoverflow.com/questions/58331977/bazel-tab-auto-complete-in-zsh-not-working) + > + > If you are using `oh-my-zsh`, you may want to install and enable + > the `zsh-autocomplete` plugin. If you'd prefer not to, use one of the + > solutions described above. + +* From Homebrew, then you're done -- the Zsh completion script is + already installed in `$(brew --prefix)/share/zsh/site-functions`. + +* From the installer downloaded from GitHub, then: + 1. Locate the absolute path of the completion file. The installer copied it + to the `bin` directory. + + Example: if you ran the installer with `--user`, this will be + `$HOME/.bazel/bin`. If you ran the installer as root, this will be + `/usr/local/lib/bazel/bin`. + + 2. Add this script to a directory on your `$fpath`: + + ``` + fpath[1,0]=~/.zsh/completion/ + mkdir -p ~/.zsh/completion/ + cp /path/from/above/step/_bazel ~/.zsh/completion + ``` + + You may have to call `rm -f ~/.zcompdump; compinit` + the first time to make it work. + + 3. Optionally, add the following to your .zshrc. + + ``` + # This way the completion script does not have to parse Bazel's options + # repeatedly. The directory in cache-path must be created manually. + zstyle ':completion:*' use-cache on + zstyle ':completion:*' cache-path ~/.zsh/cache + ``` diff --git a/install/docker-container.mdx b/install/docker-container.mdx index 2e97dc2b..b1efb76c 100644 --- a/install/docker-container.mdx +++ b/install/docker-container.mdx @@ -2,11 +2,19 @@ title: 'Getting Started with Bazel Docker Container' --- -This page provides details on the contents of the Bazel container, how to build the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel inside the Bazel container, and how to build this project directly from the host machine using the Bazel container with directory mounting. + + +This page provides details on the contents of the Bazel container, how to build +the [abseil-cpp](https://github.com/abseil/abseil-cpp) project using Bazel +inside the Bazel container, and how to build this project directly +from the host machine using the Bazel container with directory mounting. ## Build Abseil project from your host machine with directory mounting -The instructions in this section allow you to build using the Bazel container with the sources checked out in your host environment. A container is started up for each build command you execute. Build results are cached in your host environment so they can be reused across builds. +The instructions in this section allow you to build using the Bazel container +with the sources checked out in your host environment. A container is started up +for each build command you execute. Build results are cached in your host +environment so they can be reused across builds. Clone the project to a directory in your host machine. @@ -20,7 +28,8 @@ Create a folder that will have cached results to be shared across builds. mkdir -p /tmp/build_output/ ``` -Use the Bazel container to build the project and make the build outputs available in the output folder in your host machine. +Use the Bazel container to build the project and make the build +outputs available in the output folder in your host machine. ```posix-terminal docker run \ @@ -34,7 +43,9 @@ docker run \ build //absl/... ``` -Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=asan|tsan|msan` build +flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or +MemorySanitizer (msan) accordingly. ```posix-terminal docker run \ @@ -50,7 +61,10 @@ docker run \ ## Build Abseil project from inside the container -The instructions in this section allow you to build using the Bazel container with the sources inside the container. By starting a container at the beginning of your development workflow and doing changes in the worskpace within the container, build results will be cached. +The instructions in this section allow you to build using the Bazel container +with the sources inside the container. By starting a container at the beginning +of your development workflow and doing changes in the worskpace within the +container, build results will be cached. Start a shell in the Bazel container: @@ -72,7 +86,9 @@ Do a regular build. ubuntu@5a99103747c6:~/abseil-cpp$ bazel build //absl/... ``` -Build the project with sanitizers by adding the `--config=<var>asan</var>|<var>tsan</var>|<var>msan</var>` build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or MemorySanitizer (msan) accordingly. +Build the project with sanitizers by adding the `--config=asan|tsan|msan` +build flag to select AddressSanitizer (asan), ThreadSanitizer (tsan) or +MemorySanitizer (msan) accordingly. ```posix-terminal ubuntu@5a99103747c6:~/abseil-cpp$ bazel build --config={asan | tsan | msan} -- //absl/... -//absl/types:variant_test diff --git a/install/ide.mdx b/install/ide.mdx index 5c6252da..aa6210b3 100644 --- a/install/ide.mdx +++ b/install/ide.mdx @@ -2,39 +2,56 @@ title: 'Integrating Bazel with IDEs' --- -This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android Studio, and CLion (or build your own IDE plugin). It also includes links to installation and plugin details. -IDEs integrate with Bazel in a variety of ways, from features that allow Bazel executions from within the IDE, to awareness of Bazel structures such as syntax highlighting of the `BUILD` files. -If you are interested in developing an editor or IDE plugin for Bazel, please join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). +This page covers how to integrate Bazel with IDEs, such as IntelliJ, Android +Studio, and CLion (or build your own IDE plugin). It also includes links to +installation and plugin details. + +IDEs integrate with Bazel in a variety of ways, from features that allow Bazel +executions from within the IDE, to awareness of Bazel structures such as syntax +highlighting of the `BUILD` files. + +If you are interested in developing an editor or IDE plugin for Bazel, please +join the `#ide` channel on the [Bazel Slack](https://slack.bazel.build) or start +a discussion on [GitHub](https://github.com/bazelbuild/bazel/discussions). ## IDEs and editors ### IntelliJ, Android Studio, and CLion -[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). +[Official plugin](http://ij.bazel.build) for IntelliJ, Android Studio, and +CLion. The plugin is [open source](https://github.com/bazelbuild/intellij). This is the open source version of the plugin used internally at Google. Features: -- Interop with language-specific plugins. Supported languages include Java, Scala, and Python. -- Import `BUILD` files into the IDE with semantic awareness of Bazel targets. -- Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and `.bzl`files -- Build, test, and execute binaries directly from the IDE -- Create configurations for debugging and running binaries. +* Interop with language-specific plugins. Supported languages include Java, + Scala, and Python. +* Import `BUILD` files into the IDE with semantic awareness of Bazel targets. +* Make your IDE aware of Starlark, the language used for Bazel's `BUILD` and + `.bzl`files +* Build, test, and execute binaries directly from the IDE +* Create configurations for debugging and running binaries. To install, go to the IDE's plugin browser and search for `Bazel`. -To manually install older versions, download the zip files from JetBrains' Plugin Repository and install the zip file from the IDE's plugin browser: +To manually install older versions, download the zip files from JetBrains' +Plugin Repository and install the zip file from the IDE's plugin browser: -- [Android Studio plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) -- [IntelliJ plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) -- [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) +* [Android Studio + plugin](https://plugins.jetbrains.com/plugin/9185-android-studio-with-bazel) +* [IntelliJ + plugin](https://plugins.jetbrains.com/plugin/8609-intellij-with-bazel) +* [CLion plugin](https://plugins.jetbrains.com/plugin/9554-clion-with-bazel) ### Xcode -[rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), [Tulsi](https://tulsi.bazel.build), and [XCHammer](https://github.com/pinterest/xchammer) generate Xcode projects from Bazel `BUILD` files. +[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj), +[Tulsi](https://tulsi.bazel.build), and +[XCHammer](https://github.com/pinterest/xchammer) generate Xcode +projects from Bazel `BUILD` files. ### Visual Studio Code @@ -42,16 +59,21 @@ Official plugin for VS Code. Features: -- Bazel Build Targets tree -- Starlark debugger for `.bzl` files during a build (set breakpoints, step through code, inspect variables, and so on) +* Bazel Build Targets tree +* Starlark debugger for `.bzl` files during a build (set breakpoints, step + through code, inspect variables, and so on) -Find [the plugin on the Visual Studio marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). +Find [the plugin on the Visual Studio +marketplace](https://marketplace.visualstudio.com/items?itemName=BazelBuild.vscode-bazel) +or the [OpenVSX marketplace](https://open-vsx.org/extension/BazelBuild/vscode-bazel). +The plugin is [open source](https://github.com/bazelbuild/vscode-bazel). See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Atom -Find the [`language-bazel` package](https://atom.io/packages/language-bazel) on the Atom package manager. +Find the [`language-bazel` package](https://atom.io/packages/language-bazel) +on the Atom package manager. See also: [Autocomplete for Source Code](#autocomplete-for-source-code) @@ -63,23 +85,31 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Emacs -See [`bazelbuild/bazel-emacs-mode` on GitHub](https://github.com/bazelbuild/emacs-bazel-mode) +See [`bazelbuild/bazel-emacs-mode` on +GitHub](https://github.com/bazelbuild/emacs-bazel-mode) See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ### Visual Studio -[Lavender](https://github.com/tmandry/lavender) is an experimental project for generating Visual Studio projects that use Bazel for building. +[Lavender](https://github.com/tmandry/lavender) is an experimental project for +generating Visual Studio projects that use Bazel for building. ### Eclipse -[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) is a set of plugins for importing Bazel packages into an Eclipse workspace as Eclipse projects. +[Bazel Eclipse Feature](https://github.com/salesforce/bazel-eclipse) +is a set of plugins for importing Bazel packages into an Eclipse workspace as +Eclipse projects. ## Autocomplete for Source Code ### C Language Family (C++, C, Objective-C, Objective-C++, and CUDA) -[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and consumes the Protobuf output of Bazel to extract the compile commands. +[`kiron1/bazel-compile-commands`](https://github.com/kiron1/bazel-compile-commands) +run `bazel-compile-commands //...` in a Bazel workspace to generate a `compile_commands.json` file. +The `compile_commands.json` file enables tools like `clang-tidy`, `clangd` (LSP) and other IDEs to +provide autocomplete, smart navigation, quick fixes, and more. The tool is written in C++ and +consumes the Protobuf output of Bazel to extract the compile commands. [`hedronvision/bazel-compile-commands-extractor`](https://github.com/hedronvision/bazel-compile-commands-extractor) enables autocomplete, smart navigation, quick fixes, and more in a wide variety of extensible editors, including VSCode, Vim, Emacs, Atom, and Sublime. It lets language servers, like clangd and ccls, and other types of tooling, draw upon Bazel's understanding of how `cc` and `objc` code will be compiled, including how it configures cross-compilation for other platforms. @@ -89,8 +119,11 @@ See also: [Autocomplete for Source Code](#autocomplete-for-source-code) ## Automatically run build and test on file change -[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a tool for building Bazel targets when source files change. +[Bazel watcher](https://github.com/bazelbuild/bazel-watcher) is a +tool for building Bazel targets when source files change. ## Building your own IDE plugin -Read the [**IDE support** blog post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about the Bazel APIs to use when building an IDE plugin. +Read the [**IDE support** blog +post](https://blog.bazel.build/2016/06/10/ide-support.html) to learn more about +the Bazel APIs to use when building an IDE plugin. diff --git a/install/index.mdx b/install/index.mdx index 898cd417..29b46972 100644 --- a/install/index.mdx +++ b/install/index.mdx @@ -2,7 +2,10 @@ title: 'Installing Bazel' --- -This page describes the various platforms supported by Bazel and links to the packages for more details. + + +This page describes the various platforms supported by Bazel and links +to the packages for more details. [Bazelisk](/install/bazelisk) is the recommended way to install Bazel on [Ubuntu Linux](/install/ubuntu), [macOS](/install/os-x), and [Windows](/install/windows). @@ -10,21 +13,24 @@ You can find available Bazel releases on our [release page](/release). ## Community-supported packages -Bazel community members maintain these packages. The Bazel team doesn't officially support them. Contact the package maintainers for support. +Bazel community members maintain these packages. The Bazel team doesn't +officially support them. Contact the package maintainers for support. -- [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*\&branch=edge\&repo=\&arch=\&origin=\&flagged=\&maintainer=) -- [Arch Linux](https://archlinux.org/packages/extra/x86_64/bazel/) -- [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) -- [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) -- [FreeBSD](https://www.freshports.org/devel/bazel) -- [Homebrew](https://formulae.brew.sh/formula/bazel) -- [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) -- [openSUSE](/install/suse) -- [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) -- [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) +* [Alpine Linux](https://pkgs.alpinelinux.org/packages?name=bazel*&branch=edge&repo=&arch=&origin=&flagged=&maintainer=) +* [Arch Linux][arch] +* [Debian](https://qa.debian.org/developer.php?email=team%2Bbazel%40tracker.debian.org) +* [Fedora](https://copr.fedorainfracloud.org/coprs/lihaohong/bazel) +* [FreeBSD](https://www.freshports.org/devel/bazel) +* [Homebrew](https://formulae.brew.sh/formula/bazel) +* [Nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel) +* [openSUSE](/install/suse) +* [Scoop](https://github.com/scoopinstaller/scoop-main/blob/master/bucket/bazel.json) +* [Raspberry Pi](https://github.com/koenvervloesem/bazel-on-arm/blob/master/README.md) ## Community-supported architectures -- [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) +* [ppc64el](https://ftp2.osuosl.org/pub/ppc64el/bazel/) For other platforms, you can try to [compile from source](/install/compile-source). + +[arch]: https://archlinux.org/packages/extra/x86_64/bazel/ diff --git a/install/os-x.mdx b/install/os-x.mdx index eb8c34e1..b8d5bca2 100644 --- a/install/os-x.mdx +++ b/install/os-x.mdx @@ -2,21 +2,23 @@ title: 'Installing Bazel on macOS' --- + + This page describes how to install Bazel on macOS and set up your environment. You can install Bazel on macOS using one of the following methods: -- *Recommended*: [Use Bazelisk](/install/bazelisk) -- [Use Homebrew](#install-on-mac-os-x-homebrew) -- [Use the binary installer](#install-with-installer-mac-os-x) -- [Compile Bazel from source](/install/compile-source) +* *Recommended*: [Use Bazelisk](/install/bazelisk) +* [Use Homebrew](#install-on-mac-os-x-homebrew) +* [Use the binary installer](#install-with-installer-mac-os-x) +* [Compile Bazel from source](/install/compile-source) Bazel comes with two completion scripts. After installing Bazel, you can: -- Access the [bash completion script](/install/completion#bash) -- Install the [zsh completion script](/install/completion#zsh) +* Access the [bash completion script](/install/completion#bash) +* Install the [zsh completion script](/install/completion#zsh) -## Installing using Homebrew +

Installing using Homebrew

### Step 1: Install Homebrew on macOS @@ -34,37 +36,46 @@ Install the Bazel package via Homebrew as follows: brew install bazel ``` -All set! You can confirm Bazel is installed successfully by running the following command: +All set! You can confirm Bazel is installed successfully by running the +following command: ```posix-terminal bazel --version ``` -Once installed, you can upgrade to a newer version of Bazel using the following command: +Once installed, you can upgrade to a newer version of Bazel using the +following command: ```posix-terminal brew upgrade bazel ``` -## Installing using the binary installer +

Installing using the binary installer

-The binary installers are on Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). +The binary installers are on Bazel's +[GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary. Some additional libraries must also be installed for Bazel to work. +The installer contains the Bazel binary. Some additional libraries +must also be installed for Bazel to work. ### Step 1: Install Xcode command line tools -If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode command line tools package by using `xcode-select`: +If you don't intend to use `ios_*` rules, it is sufficient to install the Xcode +command line tools package by using `xcode-select`: ```posix-terminal xcode-select --install ``` -Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS SDK 8.1 installed on your system. +Otherwise, for `ios_*` rule support, you must have Xcode 6.1 or later with iOS +SDK 8.1 installed on your system. -Download Xcode from the [App Store](https://apps.apple.com/us/app/xcode/id497799835) or the [Apple Developer site](https://developer.apple.com/download/more/?=xcode). +Download Xcode from the +[App Store](https://apps.apple.com/us/app/xcode/id497799835) or the +[Apple Developer site](https://developer.apple.com/download/more/?=xcode). -Once Xcode is installed, accept the license agreement for all users with the following command: +Once Xcode is installed, accept the license agreement for all users with the +following command: ```posix-terminal sudo xcodebuild -license accept @@ -72,46 +83,59 @@ sudo xcodebuild -license accept ### Step 2: Download the Bazel installer -Next, download the Bazel binary installer named `bazel-<version>-installer-darwin-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named +`bazel--installer-darwin-x86_64.sh` from the +[Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). -**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, you need to download the installer from the terminal using `curl`, replacing the version variable with the Bazel version you want to download: +**On macOS Catalina or newer (macOS >= 11)**, due to Apple's new app signing requirements, +you need to download the installer from the terminal using `curl`, replacing +the version variable with the Bazel version you want to download: ```posix-terminal export BAZEL_VERSION=5.2.0 -curl -fLO "https://github.com/bazelbuild/bazel/releases/download/<var>$BAZEL_VERSION</var>/bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" +curl -fLO "https://github.com/bazelbuild/bazel/releases/download/{{ '' }}$BAZEL_VERSION{{ '' }}/bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" ``` -This is a temporary workaround until the macOS release flow supports signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). +This is a temporary workaround until the macOS release flow supports +signing ([#9304](https://github.com/bazelbuild/bazel/issues/9304)). ### Step 3: Run the installer Run the Bazel installer as follows: ```posix-terminal -chmod +x "bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh" +chmod +x "bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh" -./bazel-<var>$BAZEL_VERSION</var>-installer-darwin-x86_64.sh --user +./bazel-{{ '' }}$BAZEL_VERSION{{ '' }}-installer-darwin-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and +sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see +additional installation options. -If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that ***“bazel-real” cannot be opened because the developer cannot be verified***, you need to re-download the installer from the terminal using `curl` as a workaround; see Step 2 above. +If you are **on macOS Catalina or newer (macOS >= 11)** and get an error that _**“bazel-real” cannot be +opened because the developer cannot be verified**_, you need to re-download +the installer from the terminal using `curl` as a workaround; see Step 2 above. ### Step 4: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `<var>HOME</var>/bin` directory. It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel +executable is installed in your `HOME/bin` directory. +It's a good idea to add this directory to your default paths, as follows: ```posix-terminal -export PATH="<var>PATH</var>:<var>HOME</var>/bin" +export PATH="{{ '' }}PATH{{ '' }}:{{ '' }}HOME{{ '' }}/bin" ``` -You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` file. +You can also add this command to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` +file. -All set! You can confirm Bazel is installed successfully by running the following command: +All set! You can confirm Bazel is installed successfully by running the +following command: ```posix-terminal bazel --version ``` - To update to a newer release of Bazel, download and install the desired version. + diff --git a/install/suse.mdx b/install/suse.mdx index 73debc99..741b108f 100644 --- a/install/suse.mdx +++ b/install/suse.mdx @@ -2,17 +2,23 @@ title: 'Installing Bazel on openSUSE Tumbleweed & Leap' --- + + This page describes how to install Bazel on openSUSE Tumbleweed and Leap. -`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). +`NOTE:` The Bazel team does not officially maintain openSUSE support. For issues +using Bazel on openSUSE please file a ticket at [bugzilla.opensuse.org](https://bugzilla.opensuse.org/). -Packages are provided for openSUSE Tumbleweed and Leap. You can find all available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93\&baseproject=ALL\&q=bazel). +Packages are provided for openSUSE Tumbleweed and Leap. You can find all +available Bazel versions via openSUSE's [software search](https://software.opensuse.org/search?utf8=%E2%9C%93&baseproject=ALL&q=bazel). The commands below must be run either via `sudo` or while logged in as `root`. ## Installing Bazel on openSUSE -Run the following commands to install the package. If you need a specific version, you can install it via the specific `bazelXXX` package, otherwise, just `bazel` is enough: +Run the following commands to install the package. If you need a specific +version, you can install it via the specific `bazelXXX` package, otherwise, +just `bazel` is enough: To install the latest version of Bazel, run: @@ -20,7 +26,9 @@ To install the latest version of Bazel, run: zypper install bazel ``` -You can also install a specific version of Bazel by specifying the package version with `bazel<var>version</var>`. For example, to install Bazel 4.2, run: +You can also install a specific version of Bazel by specifying the package +version with `bazelversion`. For example, to install +Bazel 4.2, run: ```posix-terminal zypper install bazel4.2 diff --git a/install/ubuntu.mdx b/install/ubuntu.mdx index b498f42f..73d42751 100644 --- a/install/ubuntu.mdx +++ b/install/ubuntu.mdx @@ -2,30 +2,37 @@ title: 'Installing Bazel on Ubuntu' --- -This page describes the options for installing Bazel on Ubuntu. It also provides links to the Bazel completion scripts and the binary installer, if needed as a backup option (for example, if you don't have admin access). + + +This page describes the options for installing Bazel on Ubuntu. +It also provides links to the Bazel completion scripts and the binary installer, +if needed as a backup option (for example, if you don't have admin access). Supported Ubuntu Linux platforms: -- 22.04 (LTS) -- 20.04 (LTS) -- 18.04 (LTS) +* 22.04 (LTS) +* 20.04 (LTS) +* 18.04 (LTS) -Bazel should be compatible with other Ubuntu releases and Debian "stretch" and above, but is untested and not guaranteed to work. +Bazel should be compatible with other Ubuntu releases and Debian +"stretch" and above, but is untested and not guaranteed to work. Install Bazel on Ubuntu using one of the following methods: -- *Recommended*: [Use Bazelisk](/install/bazelisk) -- [Use our custom APT repository](#install-on-ubuntu) -- [Use the binary installer](#binary-installer) -- [Use the Bazel Docker container](#docker-container) -- [Compile Bazel from source](/install/compile-source) +* *Recommended*: [Use Bazelisk](/install/bazelisk) +* [Use our custom APT repository](#install-on-ubuntu) +* [Use the binary installer](#binary-installer) +* [Use the Bazel Docker container](#docker-container) +* [Compile Bazel from source](/install/compile-source) -**Note:** For Arm-based systems, the APT repository does not contain an `arm64` release, and there is no binary installer available. Either use Bazelisk or compile from source. +**Note:** For Arm-based systems, the APT repository does not contain an `arm64` +release, and there is no binary installer available. Either use Bazelisk or +compile from source. Bazel comes with two completion scripts. After installing Bazel, you can: -- Access the [bash completion script](/install/completion#bash) -- Install the [zsh completion script](/install/completion#zsh) +* Access the [bash completion script](/install/completion#bash) +* Install the [zsh completion script](/install/completion#zsh) ## Using Bazel's apt repository @@ -36,14 +43,16 @@ Bazel comes with two completion scripts. After installing Bazel, you can: ```posix-terminal sudo apt install apt-transport-https curl gnupg -y -curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg +curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor >bazel-archive-keyring.gpg sudo mv bazel-archive-keyring.gpg /usr/share/keyrings echo "deb [arch=amd64 signed-by=/usr/share/keyrings/bazel-archive-keyring.gpg] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list ``` -The component name "jdk1.8" is kept only for legacy reasons and doesn't relate to supported or included JDK versions. Bazel releases are Java-version agnostic. Changing the "jdk1.8" component name would break existing users of the repo. +The component name "jdk1.8" is kept only for legacy reasons and doesn't relate +to supported or included JDK versions. Bazel releases are Java-version agnostic. +Changing the "jdk1.8" component name would break existing users of the repo. ### Step 2: Install and update Bazel @@ -57,13 +66,18 @@ Once installed, you can upgrade to a newer version of Bazel as part of your norm sudo apt update && sudo apt full-upgrade ``` -The `bazel` package always installs the latest stable version of Bazel. You can install specific, older versions of Bazel in addition to the latest one, such as this: +The `bazel` package always installs the latest stable version of Bazel. You +can install specific, older versions of Bazel in addition to the latest one, +such as this: ```posix-terminal sudo apt install bazel-1.0.0 ``` -This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This can be useful if you need a specific version of Bazel to build a project, for example because it uses a `.bazelversion` file to explicitly state with which Bazel version it should be built. +This installs Bazel 1.0.0 as `/usr/bin/bazel-1.0.0` on your system. This +can be useful if you need a specific version of Bazel to build a project, for +example because it uses a `.bazelversion` file to explicitly state with which +Bazel version it should be built. Optionally, you can set `bazel` to a specific version by creating a symlink: @@ -75,7 +89,8 @@ bazel --version # 1.0.0 ### Step 3: Install a JDK (optional) -Bazel includes a private, bundled JRE as its runtime and doesn't require you to install any specific version of Java. +Bazel includes a private, bundled JRE as its runtime and doesn't require you to +install any specific version of Java. However, if you want to build Java code using Bazel, you have to install a JDK. @@ -85,11 +100,14 @@ sudo apt install default-jdk ## Using the binary installer -Generally, you should use the apt repository, but the binary installer can be useful if you don't have admin permissions on your machine or can't add custom repositories. +Generally, you should use the apt repository, but the binary installer +can be useful if you don't have admin permissions on your machine or +can't add custom repositories. The binary installers can be downloaded from Bazel's [GitHub releases page](https://github.com/bazelbuild/bazel/releases). -The installer contains the Bazel binary and extracts it into your `$HOME/bin` folder. Some additional libraries must be installed manually for Bazel to work. +The installer contains the Bazel binary and extracts it into your `$HOME/bin` +folder. Some additional libraries must be installed manually for Bazel to work. ### Step 1: Install required packages @@ -107,34 +125,42 @@ sudo apt-get install default-jdk ### Step 2: Run the installer -Next, download the Bazel binary installer named `bazel-<var>version</var>-installer-linux-x86_64.sh` from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). +Next, download the Bazel binary installer named `bazel-version-installer-linux-x86_64.sh` +from the [Bazel releases page on GitHub](https://github.com/bazelbuild/bazel/releases). Run it as follows: ```posix-terminal -chmod +x bazel-<var>version</var>-installer-linux-x86_64.sh +chmod +x bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh -./bazel-<var>version</var>-installer-linux-x86_64.sh --user +./bazel-{{ '' }}version{{ '' }}-installer-linux-x86_64.sh --user ``` -The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see additional installation options. +The `--user` flag installs Bazel to the `$HOME/bin` directory on your system and +sets the `.bazelrc` path to `$HOME/.bazelrc`. Use the `--help` command to see +additional installation options. ### Step 3: Set up your environment -If you ran the Bazel installer with the `--user` flag as above, the Bazel executable is installed in your `$HOME/bin` directory. It's a good idea to add this directory to your default paths, as follows: +If you ran the Bazel installer with the `--user` flag as above, the Bazel +executable is installed in your `$HOME/bin` directory. +It's a good idea to add this directory to your default paths, as follows: ```posix-terminal export PATH="$PATH:$HOME/bin" ``` -You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it permanent. +You can also add this command to your `~/.bashrc` or `~/.zshrc` file to make it +permanent. ## Using the Bazel Docker container -We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. You can use the Docker container as follows: +We publish Docker container with Bazel installed for each Bazel version at `gcr.io/bazel-public/bazel`. +You can use the Docker container as follows: ``` -$ docker pull gcr.io/bazel-public/bazel:<bazel version> +$ docker pull gcr.io/bazel-public/bazel: ``` The Docker container is built by [these steps](https://github.com/bazelbuild/continuous-integration/tree/master/bazel/oci). + diff --git a/install/windows.mdx b/install/windows.mdx index 3572118b..b38c0986 100644 --- a/install/windows.mdx +++ b/install/windows.mdx @@ -2,11 +2,16 @@ title: 'Installing Bazel on Windows' --- -This page describes the requirements and steps to install Bazel on Windows. It also includes troubleshooting and other ways to install Bazel, such as using Chocolatey or Scoop. + + +This page describes the requirements and steps to install Bazel on Windows. +It also includes troubleshooting and other ways to install Bazel, such as +using Chocolatey or Scoop. ## Installing Bazel -This section covers the prerequisites, environment setup, and detailed steps during installation on Windows. +This section covers the prerequisites, environment setup, and detailed +steps during installation on Windows. ### Check your system @@ -14,31 +19,33 @@ Recommended: 64 bit Windows 10, version 1703 (Creators Update) or newer To check your Windows version: -- Click the Start button. -- Type `winver` in the search box and press Enter. -- You should see the About Windows box with your Windows version information. +* Click the Start button. +* Type `winver` in the search box and press Enter. +* You should see the About Windows box with your Windows version information. ### Install the prerequisites -- [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) +* [Microsoft Visual C++ Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) ### Download Bazel *Recommended*: [Use Bazelisk](/install/bazelisk) + Alternatively you can: -- [Download the Bazel binary (`bazel-<var>version</var>-windows-x86_64.exe`) from GitHub](https://github.com/bazelbuild/bazel/releases). -- [Install Bazel from Chocolatey](#chocolately) -- [Install Bazel from Scoop](#scoop) -- [Build Bazel from source](/install/compile-source) +* [Download the Bazel binary (`bazel-version-windows-x86_64.exe`) from + GitHub](https://github.com/bazelbuild/bazel/releases). +* [Install Bazel from Chocolatey](#chocolately) +* [Install Bazel from Scoop](#scoop) +* [Build Bazel from source](/install/compile-source) ### Set up your environment To make Bazel easily accessible from command prompts or PowerShell by default, you can rename the Bazel binary to `bazel.exe` and add it to your default paths. ```posix-terminal -set PATH=%PATH%;<var>path to the Bazel binary</var> +set PATH=%PATH%;{{ '' }}path to the Bazel binary{{ '' }} ``` You can also change your system `PATH` environment variable to make it permanent. Check out how to [set environment variables](/configure/windows#set-environment-variables). @@ -50,57 +57,64 @@ You can also change your system `PATH` environment variable to make it permanent To check the installation is correct, try to run: ```posix-terminal -bazel <var>version</var> +bazel {{ '' }}version{{ '' }} ``` Next, you can check out more tips and guidance here: -- [Installing compilers and language runtimes](#install-compilers) -- [Troubleshooting](#troubleshooting) -- [Best practices on Windows](/configure/windows#best-practices) -- [Tutorials](/start/#tutorials) +* [Installing compilers and language runtimes](#install-compilers) +* [Troubleshooting](#troubleshooting) +* [Best practices on Windows](/configure/windows#best-practices) +* [Tutorials](/start/#tutorials) ## Installing compilers and language runtimes Depending on which languages you want to build, you will need: -- [MSYS2 x86\_64](https://www.msys2.org/) +* [MSYS2 x86_64](https://www.msys2.org/) - MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix tools (like `grep`, `tar`, `git`). + MSYS2 is a software distro and building platform for Windows. It contains Bash and common Unix + tools (like `grep`, `tar`, `git`). - You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an error if a build target needs Bash but Bazel could not locate it. + You will need MSYS2 to build, test, or run targets that depend on Bash. Typically these are + `genrule`, `sh_binary`, `sh_test`, but there may be more (such as Starlark rules). Bazel shows an + error if a build target needs Bash but Bazel could not locate it. -- Common MSYS2 packages +* Common MSYS2 packages - You will likely need these to build and run targets that depend on Bash. MSYS2 does not install these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). + You will likely need these to build and run targets that depend on Bash. MSYS2 does not install + these tools by default, so you need to install them manually. Projects that depend on Bash tools in `PATH` need this step (for example TensorFlow). - Open the MSYS2 terminal and run this command: + Open the MSYS2 terminal and run this command: - ```posix-terminal - pacman -S zip unzip patch diffutils git - ``` + ```posix-terminal + pacman -S zip unzip patch diffutils git + ``` - Optional: If you want to use Bazel from CMD or Powershell and still be able to use Bash tools, make sure to add `<var>MSYS2_INSTALL_PATH</var>/usr/bin` to your `PATH` environment variable. + Optional: If you want to use Bazel from CMD or Powershell and still be able + to use Bash tools, make sure to add + `MSYS2_INSTALL_PATH/usr/bin` to your + `PATH` environment variable. -- [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) +* [Build Tools for Visual Studio 2019](https://aka.ms/buildtools) - You will need this to build C++ code on Windows. + You will need this to build C++ code on Windows. - Also supported: + Also supported: - - Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK + * Visual C++ Build Tools 2017 (or newer) and Windows 10 SDK -- [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) +* [Java SE Development Kit 11 (JDK) for Windows x64](https://www.oracle.com/java/technologies/javase-jdk11-downloads.html) - You will need this to build Java code on Windows. + You will need this to build Java code on Windows. - Also supported: Java 8, 9, and 10 + Also supported: Java 8, 9, and 10 -- [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) +* [Python 3.6 for Windows x86-64](https://www.python.org/downloads/windows/) - You will need this to build Python code on Windows. + You will need this to build Python code on Windows. - Also supported: Python 2.7 or newer for Windows x86-64 + Also supported: Python 2.7 or newer for Windows x86-64 ## Troubleshooting @@ -108,11 +122,11 @@ Depending on which languages you want to build, you will need: **Possible reasons**: -- you installed MSYS2 not under the default install path +* you installed MSYS2 not under the default install path -- you installed MSYS2 i686 instead of MSYS2 x86\_64 +* you installed MSYS2 i686 instead of MSYS2 x86\_64 -- you installed MSYS instead of MSYS2 +* you installed MSYS instead of MSYS2 **Solution**: @@ -120,95 +134,104 @@ Ensure you installed MSYS2 x86\_64. If that doesn't help: -1. Go to Start Menu > Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for \"), and click the "New\..." button below it. +3. Look at the list on the top ("User variables for <username>"), and click the "New..." + button below it. -4. For "Variable name", enter `BAZEL_SH` +4. For "Variable name", enter `BAZEL_SH` -5. Click "Browse File..." +5. Click "Browse File..." -6. Navigate to the MSYS2 directory, then to `usr\bin` below it. +6. Navigate to the MSYS2 directory, then to `usr\bin` below it. - For example, this might be `C:\msys64\usr\bin` on your system. + For example, this might be `C:\msys64\usr\bin` on your system. -7. Select the `bash.exe` or `bash` file and click OK +7. Select the `bash.exe` or `bash` file and click OK -8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. +8. The "Variable value" field now has the path to `bash.exe`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Bash. ### Bazel does not find Visual Studio or Visual C++ **Possible reasons**: -- you installed multiple versions of Visual Studio +* you installed multiple versions of Visual Studio -- you installed and removed various versions of Visual Studio +* you installed and removed various versions of Visual Studio -- you installed various versions of the Windows SDK +* you installed various versions of the Windows SDK -- you installed Visual Studio not under the default install path +* you installed Visual Studio not under the default install path **Solution**: -1. Go to Start Menu > Settings. +1. Go to Start Menu > Settings. -2. Find the setting "Edit environment variables for your account" +2. Find the setting "Edit environment variables for your account" -3. Look at the list on the top ("User variables for \"), and click the "New\..." button below it. +3. Look at the list on the top ("User variables for <username>"), and click the "New..." + button below it. -4. For "Variable name", enter `BAZEL_VC` +4. For "Variable name", enter `BAZEL_VC` -5. Click "Browse Directory..." +5. Click "Browse Directory..." -6. Navigate to the `VC` directory of Visual Studio. +6. Navigate to the `VC` directory of Visual Studio. - For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` on your system. + For example, this might be `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC` + on your system. -7. Select the `VC` folder and click OK +7. Select the `VC` folder and click OK -8. The "Variable value" field now has the path to `VC`. Click OK to close the window. +8. The "Variable value" field now has the path to `VC`. Click OK to close the window. -9. Done. +9. Done. - If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. + If you open a new cmd.exe or PowerShell terminal and run Bazel now, it will find Visual C++. ## Other ways to install Bazel ### Using Chocolatey -1. Install the [Chocolatey](https://chocolatey.org) package manager +1. Install the [Chocolatey](https://chocolatey.org) package manager -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - choco install bazel - ``` + ```posix-terminal + choco install bazel + ``` - This command will install the latest available version of Bazel and its dependencies, such as the MSYS2 shell. This will not install Visual C++ though. + This command will install the latest available version of Bazel and + its dependencies, such as the MSYS2 shell. This will not install Visual C++ + though. -See [Chocolatey installation and package maintenance guide](/contribute/windows-chocolatey-maintenance) for more information about the Chocolatey package. +See [Chocolatey installation and package maintenance +guide](/contribute/windows-chocolatey-maintenance) for more +information about the Chocolatey package. ### Using Scoop -1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: +1. Install the [Scoop](https://scoop.sh/) package manager using the following PowerShell command: - ```posix-terminal - iex (new-object net.webclient).downloadstring('https://get.scoop.sh') - ``` + ```posix-terminal + iex (new-object net.webclient).downloadstring('https://get.scoop.sh') + ``` -2. Install the Bazel package: +2. Install the Bazel package: - ```posix-terminal - scoop install bazel - ``` + ```posix-terminal + scoop install bazel + ``` -See [Scoop installation and package maintenance guide](/contribute/windows-scoop-maintenance) for more information about the Scoop package. +See [Scoop installation and package maintenance +guide](/contribute/windows-scoop-maintenance) for more +information about the Scoop package. ### Build from source diff --git a/migrate/index.mdx b/migrate/index.mdx index 8683aedb..2a574067 100644 --- a/migrate/index.mdx +++ b/migrate/index.mdx @@ -2,7 +2,9 @@ title: 'Migrating to Bazel' --- + + This page links to migration guides for Bazel. -- [Maven](/migrate/maven) -- [Xcode](/migrate/xcode) +* [Maven](/migrate/maven) +* [Xcode](/migrate/xcode) diff --git a/migrate/maven.mdx b/migrate/maven.mdx index 0673ef53..38aaffc0 100644 --- a/migrate/maven.mdx +++ b/migrate/maven.mdx @@ -2,34 +2,55 @@ title: 'Migrating from Maven to Bazel' --- -This page describes how to migrate from Maven to Bazel, including the prerequisites and installation steps. It describes the differences between Maven and Bazel, and provides a migration example using the Guava project. -When migrating from any build tool to Bazel, it's best to have both build tools running in parallel until you have fully migrated your development team, CI system, and any other relevant systems. You can run Maven and Bazel in the same repository. -Note: While Bazel supports downloading and publishing Maven artifacts with [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) , it does not directly support Maven-based plugins. Maven plugins can't be directly run by Bazel since there's no Maven compatibility layer. +This page describes how to migrate from Maven to Bazel, including the +prerequisites and installation steps. It describes the differences between Maven +and Bazel, and provides a migration example using the Guava project. + +When migrating from any build tool to Bazel, it's best to have both build tools +running in parallel until you have fully migrated your development team, CI +system, and any other relevant systems. You can run Maven and Bazel in the same +repository. + +Note: While Bazel supports downloading and publishing Maven artifacts with +[rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) +, it does not directly support Maven-based plugins. Maven plugins can't be +directly run by Bazel since there's no Maven compatibility layer. ## Before you begin -- [Install Bazel](/install) if it's not yet installed. -- If you're new to Bazel, go through the tutorial [Introduction to Bazel: Build Java](/start/java) before you start migrating. The tutorial explains Bazel's concepts, structure, and label syntax. +* [Install Bazel](/install) if it's not yet installed. +* If you're new to Bazel, go through the tutorial [Introduction to Bazel: + Build Java](/start/java) before you start migrating. The tutorial explains + Bazel's concepts, structure, and label syntax. ## Differences between Maven and Bazel -- Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files and multiple targets per `BUILD` file, allowing for builds that are more incremental than Maven's. -- Maven takes charge of steps for the deployment process. Bazel does not automate deployment. -- Bazel enables you to express dependencies between languages. -- As you add new sections to the project, with Bazel you may need to add new `BUILD` files. Best practice is to add a `BUILD` file to each new Java package. +* Maven uses top-level `pom.xml` file(s). Bazel supports multiple build files + and multiple targets per `BUILD` file, allowing for builds that are more + incremental than Maven's. +* Maven takes charge of steps for the deployment process. Bazel does not + automate deployment. +* Bazel enables you to express dependencies between languages. +* As you add new sections to the project, with Bazel you may need to add new + `BUILD` files. Best practice is to add a `BUILD` file to each new Java + package. ## Migrate from Maven to Bazel The steps below describe how to migrate your project to Bazel: -1. [Create the MODULE.bazel file](#1-build) -2. [Create one BUILD file](#2-build) -3. [Create more BUILD files](#3-build) -4. [Build using Bazel](#4-build) +1. [Create the MODULE.bazel file](#1-build) +2. [Create one BUILD file](#2-build) +3. [Create more BUILD files](#3-build) +4. [Build using Bazel](#4-build) -Examples below come from a migration of the [Guava project](https://github.com/google/guava) from Maven to Bazel. The Guava project used is release `v31.1`. The examples using Guava do not walk through each step in the migration, but they do show the files and contents that are generated or added manually for the migration. +Examples below come from a migration of the [Guava +project](https://github.com/google/guava) from Maven to Bazel. The +Guava project used is release `v31.1`. The examples using Guava do not walk +through each step in the migration, but they do show the files and contents that +are generated or added manually for the migration. ``` $ git clone https://github.com/google/guava.git && cd guava @@ -38,13 +59,22 @@ $ git checkout v31.1 ### 1. Create the MODULE.bazel file -Create a file named `MODULE.bazel` at the root of your project. If your project has no external dependencies, this file can be empty. +Create a file named `MODULE.bazel` at the root of your project. If your project +has no external dependencies, this file can be empty. -If your project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the MODULE.bazel file. You can use `rules_jvm_external` to manage dependencies from Maven. For instructions about using this ruleset, see [the README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) . +If your project depends on files or packages that are not in one of the +project's directories, specify these external dependencies in the MODULE.bazel +file. You can use `rules_jvm_external` to manage dependencies from Maven. For +instructions about using this ruleset, see [the +README](https://github.com/bazelbuild/rules_jvm_external/#rules_jvm_external) +. #### Guava project example: external dependencies -You can list the external dependencies of the [Guava project](https://github.com/google/guava) with the [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) ruleset. +You can list the external dependencies of the [Guava +project](https://github.com/google/guava) with the +[`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) +ruleset. Add the following snippet to the `MODULE.bazel` file: @@ -68,74 +98,89 @@ use_repo(maven, "maven") ### 2. Create one BUILD file -Now that you have your workspace defined and external dependencies (if applicable) listed, you need to create `BUILD` files to describe how your project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use many `BUILD` files to build a project. These files specify multiple build targets, which allow Bazel to produce incremental builds. - -Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of your project and using it to do an initial build using Bazel. Then, you refine your build by adding more `BUILD` files with more granular targets. - -1. In the same directory as your `MODULE.bazel` file, create a text file and name it `BUILD`. - -2. In this `BUILD` file, use the appropriate rule to create one target to build your project. Here are some tips: - - - Use the appropriate rule: - - - To build projects with a single Maven module, use the `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - ) - ``` - - - To build projects with multiple Maven modules, use the `java_library` rule as follows: - - ```python - java_library( - name = "everything", - srcs = glob([ - "Module1/src/main/java/**/*.java", - "Module2/src/main/java/**/*.java", - ... - ]), - resources = glob([ - "Module1/src/main/resources/**", - "Module2/src/main/resources/**", - ... - ]), - deps = ["//:all-external-targets"], - ) - ``` - - - To build binaries, use the `java_binary` rule: - - ```python - java_binary( - name = "everything", - srcs = glob(["src/main/java/**/*.java"]), - resources = glob(["src/main/resources/**"]), - deps = ["//:all-external-targets"], - main_class = "com.example.Main" - ) - ``` - - - Specify the attributes: - - - `name`: Give the target a meaningful name. In the examples above, the target is called "everything." - - `srcs`: Use globbing to list all .java files in your project. - - `resources`: Use globbing to list all resources in your project. - - `deps`: You need to determine which external dependencies your project needs. - - - Take a look at the [example below of this top-level BUILD file](#guava-2) from the migration of the Guava project. - -3. Now that you have a `BUILD` file at the root of your project, build your project to ensure that it works. On the command line, from your workspace directory, use `bazel build //:everything` to build your project with Bazel. - - The project has now been successfully built with Bazel. You will need to add more `BUILD` files to allow incremental builds of the project. +Now that you have your workspace defined and external dependencies (if +applicable) listed, you need to create `BUILD` files to describe how your +project should be built. Unlike Maven with its one `pom.xml` file, Bazel can use +many `BUILD` files to build a project. These files specify multiple build +targets, which allow Bazel to produce incremental builds. + +Add `BUILD` files in stages. Start with adding one `BUILD` file at the root of +your project and using it to do an initial build using Bazel. Then, you refine +your build by adding more `BUILD` files with more granular targets. + +1. In the same directory as your `MODULE.bazel` file, create a text file and + name it `BUILD`. + +2. In this `BUILD` file, use the appropriate rule to create one target to build + your project. Here are some tips: + + * Use the appropriate rule: + * To build projects with a single Maven module, use the + `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + ) + ``` + + * To build projects with multiple Maven modules, use the + `java_library` rule as follows: + + ```python + java_library( + name = "everything", + srcs = glob([ + "Module1/src/main/java/**/*.java", + "Module2/src/main/java/**/*.java", + ... + ]), + resources = glob([ + "Module1/src/main/resources/**", + "Module2/src/main/resources/**", + ... + ]), + deps = ["//:all-external-targets"], + ) + ``` + + * To build binaries, use the `java_binary` rule: + + ```python + java_binary( + name = "everything", + srcs = glob(["src/main/java/**/*.java"]), + resources = glob(["src/main/resources/**"]), + deps = ["//:all-external-targets"], + main_class = "com.example.Main" + ) + ``` + + * Specify the attributes: + * `name`: Give the target a meaningful name. In the examples + above, the target is called "everything." + * `srcs`: Use globbing to list all .java files in your project. + * `resources`: Use globbing to list all resources in your project. + * `deps`: You need to determine which external dependencies your + project needs. + * Take a look at the [example below of this top-level BUILD + file](#guava-2) from the migration of the Guava project. + +3. Now that you have a `BUILD` file at the root of your project, build your + project to ensure that it works. On the command line, from your workspace + directory, use `bazel build //:everything` to build your project with Bazel. + + The project has now been successfully built with Bazel. You will need to add + more `BUILD` files to allow incremental builds of the project. #### Guava project example: start with one BUILD file -When migrating the Guava project to Bazel, initially one `BUILD` file is used to build the entire project. Here are the contents of this initial `BUILD` file in the workspace directory: +When migrating the Guava project to Bazel, initially one `BUILD` file is used to +build the entire project. Here are the contents of this initial `BUILD` file in +the workspace directory: ```python java_library( @@ -157,25 +202,40 @@ java_library( ### 3. Create more BUILD files (optional) -Bazel does work with just one `BUILD file`, as you saw after completing your first build. You should still consider breaking the build into smaller chunks by adding more `BUILD` files with granular targets. +Bazel does work with just one `BUILD file`, as you saw after completing your +first build. You should still consider breaking the build into smaller chunks by +adding more `BUILD` files with granular targets. -Multiple `BUILD` files with multiple targets will give the build increased granularity, allowing: +Multiple `BUILD` files with multiple targets will give the build increased +granularity, allowing: -- increased incremental builds of the project, -- increased parallel execution of the build, -- better maintainability of the build for future users, and -- control over visibility of targets between packages, which can prevent issues such as libraries containing implementation details leaking into public APIs. +* increased incremental builds of the project, +* increased parallel execution of the build, +* better maintainability of the build for future users, and +* control over visibility of targets between packages, which can prevent + issues such as libraries containing implementation details leaking into + public APIs. Tips for adding more `BUILD` files: -- You can start by adding a `BUILD` file to each Java package. Start with Java packages that have the fewest dependencies and work you way up to packages with the most dependencies. -- As you add `BUILD` files and specify targets, add these new targets to the `deps` sections of targets that depend on them. Note that the `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. -- Any time you add a `BUILD` file to a `main` directory, ensure that you add a `BUILD` file to the corresponding `test` directory. -- Take care to limit visibility properly between packages. -- To simplify troubleshooting errors in your setup of `BUILD` files, ensure that the project continues to build with Bazel as you add each build file. Run `bazel build //...` to ensure all of your targets still build. +* You can start by adding a `BUILD` file to each Java package. Start with Java + packages that have the fewest dependencies and work you way up to packages + with the most dependencies. +* As you add `BUILD` files and specify targets, add these new targets to the + `deps` sections of targets that depend on them. Note that the `glob()` + function does not cross package boundaries, so as the number of packages + grows the files matched by `glob()` will shrink. +* Any time you add a `BUILD` file to a `main` directory, ensure that you add a + `BUILD` file to the corresponding `test` directory. +* Take care to limit visibility properly between packages. +* To simplify troubleshooting errors in your setup of `BUILD` files, ensure + that the project continues to build with Bazel as you add each build file. + Run `bazel build //...` to ensure all of your targets still build. ### 4. Build using Bazel -You've been building using Bazel as you add `BUILD` files to validate the setup of the build. +You've been building using Bazel as you add `BUILD` files to validate the setup +of the build. -When you have `BUILD` files at the desired granularity, you can use Bazel to produce all of your builds. +When you have `BUILD` files at the desired granularity, you can use Bazel to +produce all of your builds. diff --git a/migrate/xcode.mdx b/migrate/xcode.mdx index 4cd229c9..986cd115 100644 --- a/migrate/xcode.mdx +++ b/migrate/xcode.mdx @@ -2,163 +2,243 @@ title: 'Migrating from Xcode to Bazel' --- -This page describes how to build or test an Xcode project with Bazel. It describes the differences between Xcode and Bazel, and provides the steps for converting an Xcode project to a Bazel project. It also provides troubleshooting solutions to address common errors. + + +This page describes how to build or test an Xcode project with Bazel. It +describes the differences between Xcode and Bazel, and provides the steps for +converting an Xcode project to a Bazel project. It also provides troubleshooting +solutions to address common errors. ## Differences between Xcode and Bazel -- Bazel requires you to explicitly specify every build target and its dependencies, plus the corresponding build settings via build rules. +* Bazel requires you to explicitly specify every build target and its + dependencies, plus the corresponding build settings via build rules. -- Bazel requires all files on which the project depends to be present within the workspace directory or specified as dependencies in the `MODULE.bazel` file. +* Bazel requires all files on which the project depends to be present within + the workspace directory or specified as dependencies in the `MODULE.bazel` + file. -- When building Xcode projects with Bazel, the `BUILD` file(s) become the source of truth. If you work on the project in Xcode, you must generate a new version of the Xcode project that matches the `BUILD` files using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) whenever you update the `BUILD` files. Certain changes to the `BUILD` files such as adding dependencies to a target don't require regenerating the project which can speed up development. If you're not using Xcode, the `bazel build` and `bazel test` commands provide build and test capabilities with certain limitations described later in this guide. +* When building Xcode projects with Bazel, the `BUILD` file(s) become the + source of truth. If you work on the project in Xcode, you must generate a + new version of the Xcode project that matches the `BUILD` files using + [rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj/) + whenever you update the `BUILD` files. Certain changes to the `BUILD` files + such as adding dependencies to a target don't require regenerating the + project which can speed up development. If you're not using Xcode, the + `bazel build` and `bazel test` commands provide build and test capabilities + with certain limitations described later in this guide. ## Before you begin Before you begin, do the following: -1. [Install Bazel](/install) if you have not already done so. +1. [Install Bazel](/install) if you have not already done so. -2. If you're not familiar with Bazel and its concepts, complete the [iOS app tutorial](/start/ios-app)). You should understand the Bazel workspace, including the `MODULE.bazel` and `BUILD` files, as well as the concepts of targets, build rules, and Bazel packages. +2. If you're not familiar with Bazel and its concepts, complete the [iOS app + tutorial](/start/ios-app)). You should understand the Bazel workspace, + including the `MODULE.bazel` and `BUILD` files, as well as the concepts of + targets, build rules, and Bazel packages. -3. Analyze and understand the project's dependencies. +3. Analyze and understand the project's dependencies. ### Analyze project dependencies -Unlike Xcode, Bazel requires you to explicitly declare all dependencies for every target in the `BUILD` file. +Unlike Xcode, Bazel requires you to explicitly declare all dependencies for +every target in the `BUILD` file. -For more information on external dependencies, see [Working with external dependencies](/docs/external). +For more information on external dependencies, see [Working with external +dependencies](/docs/external). ## Build or test an Xcode project with Bazel To build or test an Xcode project with Bazel, do the following: -1. [Create the `MODULE.bazel` file](#create-workspace) +1. [Create the `MODULE.bazel` file](#create-workspace) -2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) +2. [(Experimental) Integrate SwiftPM dependencies](#integrate-swiftpm) -3. [Create a `BUILD` file:](#create-build-file) +3. [Create a `BUILD` file:](#create-build-file) - a. [Add the application target](#add-app-target) + a. [Add the application target](#add-app-target) - b. [(Optional) Add the test target(s)](#add-test-target) + b. [(Optional) Add the test target(s)](#add-test-target) - c. [Add the library target(s)](#add-library-target) + c. [Add the library target(s)](#add-library-target) -4. [(Optional) Granularize the build](#granularize-build) +4. [(Optional) Granularize the build](#granularize-build) -5. [Run the build](#run-build) +5. [Run the build](#run-build) -6. [Generate the Xcode project with rules\_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) +6. [Generate the Xcode project with rules_xcodeproj](#generate-the-xcode-project-with-rules_xcodeproj) ### Step 1: Create the `MODULE.bazel` file -Create a `MODULE.bazel` file in a new directory. This directory becomes the Bazel workspace root. If the project uses no external dependencies, this file can be empty. If the project depends on files or packages that are not in one of the project's directories, specify these external dependencies in the `MODULE.bazel` file. +Create a `MODULE.bazel` file in a new directory. This directory becomes the +Bazel workspace root. If the project uses no external dependencies, this file +can be empty. If the project depends on files or packages that are not in one of +the project's directories, specify these external dependencies in the +`MODULE.bazel` file. -Note: Place the project source code within the directory tree containing the `MODULE.bazel` file. +Note: Place the project source code within the directory tree containing the +`MODULE.bazel` file. ### Step 2: (Experimental) Integrate SwiftPM dependencies -To integrate SwiftPM dependencies into the Bazel workspace with [swift\_bazel](https://github.com/cgrindel/swift_bazel), you must convert them into Bazel packages as described in the [following tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) . +To integrate SwiftPM dependencies into the Bazel workspace with +[swift_bazel](https://github.com/cgrindel/swift_bazel), you must +convert them into Bazel packages as described in the [following +tutorial](https://chuckgrindel.com/swift-packages-in-bazel-using-swift_bazel/) +. -Note: SwiftPM support is a manual process with many variables. SwiftPM integration with Bazel has not been fully verified and is not officially supported. +Note: SwiftPM support is a manual process with many variables. SwiftPM +integration with Bazel has not been fully verified and is not officially +supported. ### Step 3: Create a `BUILD` file -Once you have defined the workspace and external dependencies, you need to create a `BUILD` file that tells Bazel how the project is structured. Create the `BUILD` file at the root of the Bazel workspace and configure it to do an initial build of the project as follows: +Once you have defined the workspace and external dependencies, you need to +create a `BUILD` file that tells Bazel how the project is structured. Create the +`BUILD` file at the root of the Bazel workspace and configure it to do an +initial build of the project as follows: -- [Step 3a: Add the application target](#step-3a-add-the-application-target) -- [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) -- [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) +* [Step 3a: Add the application target](#step-3a-add-the-application-target) +* [Step 3b: (Optional) Add the test target(s)](#step-3b-optional-add-the-test-target-s) +* [Step 3c: Add the library target(s)](#step-3c-add-the-library-target-s) -**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, packages, and targets](/concepts/build-ref). +**Tip:** To learn more about packages and other Bazel concepts, see [Workspaces, +packages, and targets](/concepts/build-ref). #### Step 3a: Add the application target -Add a [`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) or an [`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) rule target. This target builds a macOS or iOS application bundle, respectively. In the target, specify the following at the minimum: +Add a +[`macos_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_application) +or an +[`ios_application`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_application) +rule target. This target builds a macOS or iOS application bundle, respectively. +In the target, specify the following at the minimum: -- `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the binary. +* `bundle_id` - the bundle ID (reverse-DNS path followed by app name) of the + binary. -- `provisioning_profile` - provisioning profile from your Apple Developer account (if building for an iOS device device). +* `provisioning_profile` - provisioning profile from your Apple Developer + account (if building for an iOS device device). -- `families` (iOS only) - whether to build the application for iPhone, iPad, or both. +* `families` (iOS only) - whether to build the application for iPhone, iPad, + or both. -- `infoplists` - list of .plist files to merge into the final Info.plist file. +* `infoplists` - list of .plist files to merge into the final Info.plist file. -- `minimum_os_version` - the minimum version of macOS or iOS that the application supports. This ensures Bazel builds the application with the correct API levels. +* `minimum_os_version` - the minimum version of macOS or iOS that the + application supports. This ensures Bazel builds the application with the + correct API levels. #### Step 3b: (Optional) Add the test target(s) -Bazel's [Apple build rules](https://github.com/bazelbuild/rules_apple) support running unit and UI tests on all Apple platforms. Add test targets as follows: +Bazel's [Apple build +rules](https://github.com/bazelbuild/rules_apple) support running +unit and UI tests on all Apple platforms. Add test targets as follows: -- [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) to run library-based and application-based unit tests on a macOS. +* [`macos_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-macos.md#macos_unit_test) + to run library-based and application-based unit tests on a macOS. -- [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) to build and run library-based unit tests on iOS. +* [`ios_unit_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_unit_test) + to build and run library-based unit tests on iOS. -- [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) to build and run user interface tests in the iOS simulator. +* [`ios_ui_test`](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-ios.md#ios_ui_test) + to build and run user interface tests in the iOS simulator. -- Similar test rules exist for [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) and [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). +* Similar test rules exist for + [tvOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-tvos.md), + [watchOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-watchos.md) + and + [visionOS](https://github.com/bazelbuild/rules_apple/blob/master/doc/rules-visionos.md). -At the minimum, specify a value for the `minimum_os_version` attribute. While other packaging attributes, such as `bundle_identifier` and `infoplists`, default to most commonly used values, ensure that those defaults are compatible with the project and adjust them as necessary. For tests that require the iOS simulator, also specify the `ios_application` target name as the value of the `test_host` attribute. +At the minimum, specify a value for the `minimum_os_version` attribute. While +other packaging attributes, such as `bundle_identifier` and `infoplists`, +default to most commonly used values, ensure that those defaults are compatible +with the project and adjust them as necessary. For tests that require the iOS +simulator, also specify the `ios_application` target name as the value of the +`test_host` attribute. #### Step 3c: Add the library target(s) -Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each Objective-C library and a [`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) target for each Swift library on which the application and/or tests depend. +Add an [`objc_library`](/reference/be/objective-c#objc_library) target for each +Objective-C library and a +[`swift_library`](https://github.com/bazelbuild/rules_swift/blob/master/doc/rules.md#swift_library) +target for each Swift library on which the application and/or tests depend. Add the library targets as follows: -- Add the application library targets as dependencies to the application targets. +* Add the application library targets as dependencies to the application + targets. -- Add the test library targets as dependencies to the test targets. +* Add the test library targets as dependencies to the test targets. -- List the implementation sources in the `srcs` attribute. +* List the implementation sources in the `srcs` attribute. -- List the headers in the `hdrs` attribute. +* List the headers in the `hdrs` attribute. -Note: You can use the [`glob`](/reference/be/functions#glob) function to include all sources and/or headers of a certain type. Use it carefully as it might include files you do not want Bazel to build. +Note: You can use the [`glob`](/reference/be/functions#glob) function to include +all sources and/or headers of a certain type. Use it carefully as it might +include files you do not want Bazel to build. -You can browse existing examples for various types of applications directly in the [rules\_apple examples directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For example: +You can browse existing examples for various types of applications directly in +the [rules_apple examples +directory](https://github.com/bazelbuild/rules_apple/tree/master/examples/). For +example: -- [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) +* [macOS application targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/macos) -- [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) +* [iOS applications targets](https://github.com/bazelbuild/rules_apple/tree/master/examples/ios) -- [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) +* [Multi platform applications (macOS, iOS, watchOS, tvOS)](https://github.com/bazelbuild/rules_apple/tree/master/examples/multi_platform) -For more information on build rules, see [Apple Rules for Bazel](https://github.com/bazelbuild/rules_apple). +For more information on build rules, see [Apple Rules for +Bazel](https://github.com/bazelbuild/rules_apple). At this point, it is a good idea to test the build: -`bazel build //:<application_target>` +`bazel build //:` ### Step 4: (Optional) Granularize the build -If the project is large, or as it grows, consider chunking it into multiple Bazel packages. This increased granularity provides: +If the project is large, or as it grows, consider chunking it into multiple +Bazel packages. This increased granularity provides: -- Increased incrementality of builds, +* Increased incrementality of builds, -- Increased parallelization of build tasks, +* Increased parallelization of build tasks, -- Better maintainability for future users, +* Better maintainability for future users, -- Better control over source code visibility across targets and packages. This prevents issues such as libraries containing implementation details leaking into public APIs. +* Better control over source code visibility across targets and packages. This + prevents issues such as libraries containing implementation details leaking + into public APIs. Tips for granularizing the project: -- Put each library in its own Bazel package. Start with those requiring the fewest dependencies and work your way up the dependency tree. +* Put each library in its own Bazel package. Start with those requiring the + fewest dependencies and work your way up the dependency tree. -- As you add `BUILD` files and specify targets, add these new targets to the `deps` attributes of targets that depend on them. +* As you add `BUILD` files and specify targets, add these new targets to the + `deps` attributes of targets that depend on them. -- The `glob()` function does not cross package boundaries, so as the number of packages grows the files matched by `glob()` will shrink. +* The `glob()` function does not cross package boundaries, so as the number of + packages grows the files matched by `glob()` will shrink. -- When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to the corresponding `test` directory. +* When adding a `BUILD` file to a `main` directory, also add a `BUILD` file to + the corresponding `test` directory. -- Enforce healthy visibility limits across packages. +* Enforce healthy visibility limits across packages. -- Build the project after each major change to the `BUILD` files and fix build errors as you encounter them. +* Build the project after each major change to the `BUILD` files and fix build + errors as you encounter them. ### Step 5: Run the build -Run the fully migrated build to ensure it completes with no errors or warnings. Run every application and test target individually to more easily find sources of any errors that occur. +Run the fully migrated build to ensure it completes with no errors or warnings. +Run every application and test target individually to more easily find sources +of any errors that occur. For example: @@ -166,17 +246,25 @@ For example: bazel build //:my-target ``` -### Step 6: Generate the Xcode project with rules\_xcodeproj +### Step 6: Generate the Xcode project with rules_xcodeproj -When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source of truth about the build. To make Xcode aware of this, you must generate a Bazel-compatible Xcode project using [rules\_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) . +When building with Bazel, the `MODULE.bazel` and `BUILD` files become the source +of truth about the build. To make Xcode aware of this, you must generate a +Bazel-compatible Xcode project using +[rules_xcodeproj](https://github.com/buildbuddy-io/rules_xcodeproj#features) +. ### Troubleshooting -Bazel errors can arise when it gets out of sync with the selected Xcode version, like when you apply an update. Here are some things to try if you're experiencing errors with Xcode, for example "Xcode version must be specified to use an Apple CROSSTOOL". +Bazel errors can arise when it gets out of sync with the selected Xcode version, +like when you apply an update. Here are some things to try if you're +experiencing errors with Xcode, for example "Xcode version must be specified to +use an Apple CROSSTOOL". -- Manually run Xcode and accept any terms and conditions. +* Manually run Xcode and accept any terms and conditions. -- Use Xcode select to indicate the correct version, accept the license, and clear Bazel's state. +* Use Xcode select to indicate the correct version, accept the license, and + clear Bazel's state. ```posix-terminal sudo xcode-select -s /Applications/Xcode.app/Contents/Developer @@ -186,6 +274,7 @@ Bazel errors can arise when it gets out of sync with the selected Xcode version, bazel sync --configure ``` -- If this does not work, you may also try running `bazel clean --expunge`. +* If this does not work, you may also try running `bazel clean --expunge`. -Note: If you've saved your Xcode to a different path, you can use `xcode-select -s` to point to that path. +Note: If you've saved your Xcode to a different path, you can use `xcode-select +-s` to point to that path. diff --git a/query/aquery.mdx b/query/aquery.mdx index 446855d1..151565b7 100644 --- a/query/aquery.mdx +++ b/query/aquery.mdx @@ -2,13 +2,23 @@ title: 'Action Graph Query (aquery)' --- -The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis Configured Target Graph and exposes information about **Actions, Artifacts and their relationships.** -`aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs/outputs/mnemonics. -The tool accepts several command-line [options](#command-options). Notably, the aquery command runs on top of a regular Bazel build and inherits the set of options available during a build. +The `aquery` command allows you to query for actions in your build graph. +It operates on the post-analysis Configured Target Graph and exposes +information about **Actions, Artifacts and their relationships.** -It supports the same set of functions that is also available to traditional `query` but `siblings`, `buildfiles` and `tests`. +`aquery` is useful when you are interested in the properties of the Actions/Artifacts +generated from the Configured Target Graph. For example, the actual commands run +and their inputs/outputs/mnemonics. + +The tool accepts several command-line [options](#command-options). +Notably, the aquery command runs on top of a regular Bazel build and inherits +the set of options available during a build. + +It supports the same set of functions that is also available to traditional +`query` but `siblings`, `buildfiles` and +`tests`. An example `aquery` output (without specific details): @@ -31,9 +41,11 @@ A simple example of the syntax for `aquery` is as follows: The query expression (in quotes) consists of the following: -- `aquery_function(...)`: functions specific to `aquery`. More details [below](#using-aquery-functions). -- `function(...)`: the standard [functions](/query/language#functions) as traditional `query`. -- `//target` is the label to the interested target. +* `aquery_function(...)`: functions specific to `aquery`. + More details [below](#using-aquery-functions). +* `function(...)`: the standard [functions](/query/language#functions) + as traditional `query`. +* `//target` is the label to the interested target. ``` # aquery examples: @@ -52,13 +64,14 @@ $ bazel aquery 'inputs(".*cpp", deps(//src/target_a))' There are three `aquery` functions: -- `inputs`: filter actions by inputs. -- `outputs`: filter actions by outputs -- `mnemonic`: filter actions by mnemonic +* `inputs`: filter actions by inputs. +* `outputs`: filter actions by outputs +* `mnemonic`: filter actions by mnemonic `expr ::= inputs(word, expr)` -The `inputs` operator returns the actions generated from building `expr`, whose input filenames match the regex provided by `word`. + The `inputs` operator returns the actions generated from building `expr`, + whose input filenames match the regex provided by `word`. `$ bazel aquery 'inputs(".*cpp", deps(//src/target_a))'` @@ -70,9 +83,13 @@ You can also combine functions to achieve the AND operation. For example: $ bazel aquery 'mnemonic("Cpp.*", (inputs(".*cpp", inputs("foo.*", //src/target_a))))' ``` -The above command would find all actions involved in building `//src/target_a`, whose mnemonics match `"Cpp.*"` and inputs match the patterns `".*cpp"` and `"foo.*"`. + The above command would find all actions involved in building `//src/target_a`, + whose mnemonics match `"Cpp.*"` and inputs match the patterns + `".*cpp"` and `"foo.*"`. -Important: aquery functions can't be nested inside non-aquery functions. Conceptually, this makes sense since the output of aquery functions is Actions, not Configured Targets. +Important: aquery functions can't be nested inside non-aquery functions. +Conceptually, this makes sense since the output of aquery functions is Actions, +not Configured Targets. An example of the syntax error produced: @@ -87,17 +104,23 @@ An example of the syntax error produced: ### Build options -`aquery` runs on top of a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. +`aquery` runs on top of a regular Bazel build and thus inherits the set of +[options](/reference/command-line-reference#build-options) +available during a build. ### Aquery options #### `--output=(text|summary|commands|proto|jsonproto|textproto), default=text` -The default output format (`text`) is human-readable, use `proto`, `textproto`, or `jsonproto` for machine-readable format. The proto message is `analysis.ActionGraphContainer`. +The default output format (`text`) is human-readable, +use `proto`, `textproto`, or `jsonproto` for machine-readable format. +The proto message is `analysis.ActionGraphContainer`. -The `commands` output format prints a list of build commands with one command per line. +The `commands` output format prints a list of build commands with +one command per line. -In general, do not depend on the order of output. For more information, see the [core query ordering contract](/query/language#graph-order). +In general, do not depend on the order of output. For more information, +see the [core query ordering contract](/query/language#graph-order). #### `--include_commandline, default=true` @@ -119,35 +142,45 @@ Warning: Enabling this flag will automatically enable the `--include_commandline #### `--include_file_write_contents, default=false` -Include file contents for the `actions.write()` action and the contents of the manifest file for the `SourceSymlinkManifest` action The file contents is returned in the `file_contents` field with `--output=`xxx`proto`. With `--output=text`, the output has - +Include file contents for the `actions.write()` action and the contents of the +manifest file for the `SourceSymlinkManifest` action The file contents is +returned in the `file_contents` field with `--output=`xxx`proto`. +With `--output=text`, the output has ``` -FileWriteContents: [<base64-encoded file contents>] +FileWriteContents: [] ``` - line #### `--skyframe_state, default=false` Without performing extra analysis, dump the Action Graph from Skyframe. -Note: Specifying a target with `--skyframe_state` is currently not supported. This flag is only available with `--output=proto` or `--output=textproto`. +Note: Specifying a target with `--skyframe_state` is currently not supported. +This flag is only available with `--output=proto` or `--output=textproto`. ## Other tools and features ### Querying against the state of Skyframe -[Skyframe](/reference/skyframe) is the evaluation and incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph constructed from the previous runs of the [Analysis phase](/run/build#analysis). +[Skyframe](/reference/skyframe) is the evaluation and +incrementality model of Bazel. On each instance of Bazel server, Skyframe stores the dependency graph +constructed from the previous runs of the [Analysis phase](/run/build#analysis). -In some cases, it is useful to query the Action Graph on Skyframe. An example use case would be: +In some cases, it is useful to query the Action Graph on Skyframe. +An example use case would be: -1. Run `bazel build //target_a` -2. Run `bazel build //target_b` -3. File `foo.out` was generated. +1. Run `bazel build //target_a` +2. Run `bazel build //target_b` +3. File `foo.out` was generated. -*As a Bazel user, I want to determine if `foo.out` was generated from building `//target_a` or `//target_b`*. +_As a Bazel user, I want to determine if `foo.out` was generated from building +`//target_a` or `//target_b`_. -One could run `bazel aquery 'outputs("foo.out", //target_a)'` and `bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible for creating `foo.out`, and in turn the target. However, the number of different targets previously built can be larger than 2, which makes running multiple `aquery` commands a hassle. +One could run `bazel aquery 'outputs("foo.out", //target_a)'` and +`bazel aquery 'outputs("foo.out", //target_b)'` to figure out the action responsible +for creating `foo.out`, and in turn the target. However, the number of different +targets previously built can be larger than 2, which makes running multiple `aquery` +commands a hassle. As an alternative, the `--skyframe_state` flag can be used: @@ -161,17 +194,22 @@ As an alternative, the `--skyframe_state` flag can be used: $ bazel aquery --output=proto --skyframe_state 'outputs("foo.out")' ``` -With `--skyframe_state` mode, `aquery` takes the content of the Action Graph that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and outputs the content, without re-running the analysis phase. +With `--skyframe_state` mode, `aquery` takes the content of the Action Graph +that Skyframe keeps on the instance of Bazel, (optionally) performs filtering on it and +outputs the content, without re-running the analysis phase. #### Special considerations ##### Output format -`--skyframe_state` is currently only available for `--output=proto` and `--output=textproto` +`--skyframe_state` is currently only available for `--output=proto` +and `--output=textproto` ##### Non-inclusion of target labels in the query expression -Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, regardless of the targets. Having the target label specified in the query together with `--skyframe_state` is considered a syntax error: +Currently, `--skyframe_state` queries the whole action graph that exists on Skyframe, +regardless of the targets. Having the target label specified in the query together with +`--skyframe_state` is considered a syntax error: ``` # WRONG: Target Included @@ -189,9 +227,12 @@ Currently, `--skyframe_state` queries the whole action graph that exists on Skyf ### Comparing aquery outputs -You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. For instance: when you make some changes to your rule definition and want to verify that the command lines being run did not change. `aquery_differ` is the tool for that. +You can compare the outputs of two different aquery invocations using the `aquery_differ` tool. +For instance: when you make some changes to your rule definition and want to verify that the +command lines being run did not change. `aquery_differ` is the tool for that. -The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. To use it, clone the repository to your local machine. An example usage: +The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/bazel/tree/master/tools/aquery_differ) repository. +To use it, clone the repository to your local machine. An example usage: ``` $ bazel run //tools/aquery_differ -- \ @@ -202,7 +243,9 @@ The tool is available in the [bazelbuild/bazel](https://github.com/bazelbuild/ba --attrs=inputs ``` -The above command returns the difference between the `before` and `after` aquery outputs: which actions were present in one but not the other, which actions have different command line/inputs in each aquery output, ...). The result of running the above command would be: +The above command returns the difference between the `before` and `after` aquery outputs: +which actions were present in one but not the other, which actions have different +command line/inputs in each aquery output, ...). The result of running the above command would be: ``` Aquery output 'after' change contains an action that generates the following outputs that aquery output 'before' change doesn't: @@ -225,29 +268,36 @@ The above command returns the difference between the `before` and `after` aquery `--before, --after`: The aquery output files to be compared -`--input_type=(proto|text_proto), default=proto`: the format of the input files. Support is provided for `proto` and `textproto` aquery output. +`--input_type=(proto|text_proto), default=proto`: the format of the input +files. Support is provided for `proto` and `textproto` aquery output. -`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions to be compared. +`--attrs=(cmdline|inputs), default=cmdline`: the attributes of actions +to be compared. ### Aspect-on-aspect -It is possible for [Aspects](/extending/aspects) to be applied on top of each other. The aquery output of the action generated by these Aspects would then include the *Aspect path*, which is the sequence of Aspects applied to the target which generated the action. +It is possible for [Aspects](/extending/aspects) +to be applied on top of each other. The aquery output of the action generated by +these Aspects would then include the _Aspect path_, which is the sequence of +Aspects applied to the target which generated the action. An example of Aspect-on-Aspect: ``` t0 ^ - | <- a1 + | <- a1 t1 ^ - | <- a2 + | <- a2 t2 ``` -Let ti be a target of rule ri, which applies an Aspect ai to its dependencies. +Let ti be a target of rule ri, which applies an Aspect ai +to its dependencies. -Assume that a2 generates an action X when applied to target t0. The text output of `bazel aquery --include_aspects 'deps(//t2)'` for action X would be: +Assume that a2 generates an action X when applied to target t0. The text output of +`bazel aquery --include_aspects 'deps(//t2)'` for action X would be: ``` action ... @@ -255,11 +305,13 @@ Assume that a2 generates an action X when applied to target t0. The text output Target: //my_pkg:t0 Configuration: ... AspectDescriptors: [//my_pkg:rule.bzl%**a2**(foo=...) - -> //my_pkg:rule.bzl%**a1**(bar=...)] + -> //my_pkg:rule.bzl%**a1**(bar=...)] ... ``` -This means that action `X` was generated by Aspect `a2` applied onto `a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied onto target `t0`. +This means that action `X` was generated by Aspect `a2` applied onto +`a1(t0)`, where `a1(t0)` is the result of Aspect `a1` applied +onto target `t0`. Each `AspectDescriptor` has the following format: @@ -267,31 +319,49 @@ Each `AspectDescriptor` has the following format: AspectClass([param=value,...]) ``` -`AspectClass` could be the name of the Aspect class (for native Aspects) or `bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are sorted in topological order of the [dependency graph](/extending/aspects#aspect_basics). +`AspectClass` could be the name of the Aspect class (for native Aspects) or +`bzl_file%aspect_name` (for Starlark Aspects). `AspectDescriptor` are +sorted in topological order of the +[dependency graph](/extending/aspects#aspect_basics). ### Linking with the JSON profile -While aquery provides information about the actions being run in a build (why they're being run, their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) tells us the timing and duration of their execution. It is possible to combine these 2 sets of information via a common denominator: an action's primary output. +While aquery provides information about the actions being run in a build (why they're being run, +their inputs/outputs), the [JSON profile](/rules/performance#performance-profiling) +tells us the timing and duration of their execution. +It is possible to combine these 2 sets of information via a common denominator: an action's primary output. -To include actions' outputs in the JSON profile, generate the profile with `--experimental_include_primary_output --noslim_profile`. Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output is included by default by aquery. +To include actions' outputs in the JSON profile, generate the profile with +`--experimental_include_primary_output --noslim_profile`. +Slim profiles are incompatible with the inclusion of primary outputs. An action's primary output +is included by default by aquery. -We don't currently provide a canonical tool to combine these 2 data sources, but you should be able to build your own script with the above information. +We don't currently provide a canonical tool to combine these 2 data sources, but you should be +able to build your own script with the above information. ## Known issues ### Handling shared actions -Sometimes actions are [shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) between configured targets. +Sometimes actions are +[shared](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=59;drc=146d51aa1ec9dcb721a7483479ef0b1ac21d39f1) +between configured targets. -In the execution phase, those shared actions are [simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these like separate actions whose output Artifacts have the exact same `execPath`. As a result, equivalent Artifacts appear duplicated. +In the execution phase, those shared actions are +[simply considered as one](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/Actions.java;l=241;drc=003b8734036a07b496012730964ac220f486b61f) and only executed once. +However, aquery operates on the pre-execution, post-analysis action graph, and hence treats these +like separate actions whose output Artifacts have the exact same `execPath`. As a result, +equivalent Artifacts appear duplicated. -The list of aquery issues/planned features can be found on [GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). +The list of aquery issues/planned features can be found on +[GitHub](https://github.com/bazelbuild/bazel/labels/team-Performance). ## FAQs ### The ActionKey remains the same even though the content of an input file changed. -In the context of aquery, the `ActionKey` refers to the `String` gotten from [ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): +In the context of aquery, the `ActionKey` refers to the `String` gotten from +[ActionAnalysisMetadata#getKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/actions/ActionAnalysisMetadata.java;l=89;drc=8b856f5484f0117b2aebc302f849c2a15f273310): ``` Returns a string encoding all of the significant behaviour of this Action that might affect the @@ -313,7 +383,8 @@ In the context of aquery, the `ActionKey` refers to the `String` gotten from [Ac input names change or else action validation may falsely validate. ``` -This excludes the changes to the content of the input files, and is not to be confused with [RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). +This excludes the changes to the content of the input files, and is not to be confused with +[RemoteCacheClient#ActionKey](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/common/RemoteCacheClient.java;l=38;drc=21577f202eb90ce94a337ebd2ede824d609537b6). ## Updates diff --git a/query/cquery.mdx b/query/cquery.mdx index 6a073058..d35ea829 100644 --- a/query/cquery.mdx +++ b/query/cquery.mdx @@ -2,14 +2,21 @@ title: 'Configurable Query (cquery)' --- -`cquery` is a variant of [`query`](/query/language) that correctly handles [`select()`](/docs/configurable-attributes) and build options' effects on the build graph. -It achieves this by running over the results of Bazel's [analysis phase](/extending/concepts#evaluation-model), which integrates these effects. `query`, by contrast, runs over the results of Bazel's loading phase, before options are evaluated. + +`cquery` is a variant of [`query`](/query/language) that correctly handles +[`select()`](/docs/configurable-attributes) and build options' effects on the +build graph. + +It achieves this by running over the results of Bazel's [analysis +phase](/extending/concepts#evaluation-model), +which integrates these effects. `query`, by contrast, runs over the results of +Bazel's loading phase, before options are evaluated. For example: ``` -$ cat > tree/BUILD <<EOF +$ cat > tree/BUILD <<EOF sh_library( name = "ash", deps = select({ @@ -23,11 +30,11 @@ sh_library(name = "white-ash") sh_library(name = "common-ash") config_setting( name = "excelsior", - values = {"define": "species=excelsior"}, + values = \{"define": "species=excelsior"\}, ) config_setting( name = "americana", - values = {"define": "species=americana"}, + values = \{"define": "species=americana"\}, ) EOF ``` @@ -52,9 +59,13 @@ $ bazel cquery "deps(//tree:ash)" --define species=excelsior --noimplicit_deps //tree:excelsior (9f87702) ``` -Each result includes a [unique identifier](#configurations) `(9f87702)` of the [configuration](/reference/glossary#configuration) the target is built with. +Each result includes a [unique identifier](#configurations) `(9f87702)` of +the [configuration](/reference/glossary#configuration) the +target is built with. -Since `cquery` runs over the configured target graph. it doesn't have insight into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). +Since `cquery` runs over the configured target graph. it doesn't have insight +into artifacts like build actions nor access to [`test_suite`](/reference/be/general#test_suite) +rules as they are not configured targets. For the former, see [`aquery`](/query/aquery). ## Basic syntax @@ -64,10 +75,19 @@ A simple `cquery` call looks like: The query expression `"function(//target)"` consists of the following: -- **`function(...)`** is the function to run on the target. `cquery` supports most of `query`'s [functions](/query/language#functions), plus a few new ones. -- **`//target`** is the expression fed to the function. In this example, the expression is a simple target. But the query language also allows nesting of functions. See the [Query guide](/query/guide) for examples. +* **`function(...)`** is the function to run on the target. `cquery` + supports most + of `query`'s [functions](/query/language#functions), plus a + few new ones. +* **`//target`** is the expression fed to the function. In this example, the + expression is a simple target. But the query language also allows nesting of functions. + See the [Query guide](/query/guide) for examples. + -`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) phases. Unless otherwise specified, `cquery` parses the target(s) listed in the query expression. See [`--universe_scope`](#universe-scope) for querying dependencies of top-level build targets. +`cquery` requires a target to run through the [loading and analysis](/extending/concepts#evaluation-model) +phases. Unless otherwise specified, `cquery` parses the target(s) listed in the +query expression. See [`--universe_scope`](#universe-scope) +for querying dependencies of top-level build targets. ## Configurations @@ -77,7 +97,9 @@ The line: //tree:ash (9f87702) ``` -means `//tree:ash` was built in a configuration with ID `9f87702`. For most targets, this is an opaque hash of the build option values defining the configuration. +means `//tree:ash` was built in a configuration with ID `9f87702`. For most +targets, this is an opaque hash of the build option values defining the +configuration. To see the configuration's complete contents, run: @@ -85,15 +107,25 @@ To see the configuration's complete contents, run: $ bazel config 9f87702 ``` -`9f87702` is a prefix of the complete ID. This is because complete IDs are SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid prefix of a complete ID, similar to [Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). To see complete IDs, run `$ bazel config`. +`9f87702` is a prefix of the complete ID. This is because complete IDs are +SHA-256 hashes, which are long and hard to follow. `cquery` understands any valid +prefix of a complete ID, similar to +[Git short hashes](https://git-scm.com/book/en/v2/Git-Tools-Revision-Selection#_revision_selection). + To see complete IDs, run `$ bazel config`. ## Target pattern evaluation -`//foo` has a different meaning for `cquery` than for `query`. This is because `cquery` evaluates *configured* targets and the build graph may have multiple configured versions of `//foo`. +`//foo` has a different meaning for `cquery` than for `query`. This is because +`cquery` evaluates _configured_ targets and the build graph may have multiple +configured versions of `//foo`. -For `cquery`, a target pattern in the query expression evaluates to every configured target with a label that matches that pattern. Output is deterministic, but `cquery` makes no ordering guarantee beyond the [core query ordering contract](/query/language#graph-order). +For `cquery`, a target pattern in the query expression evaluates +to every configured target with a label that matches that pattern. Output is +deterministic, but `cquery` makes no ordering guarantee beyond the +[core query ordering contract](/query/language#graph-order). -This produces subtler results for query expressions than with `query`. For example, the following can produce multiple results: +This produces subtler results for query expressions than with `query`. +For example, the following can produce multiple results: ``` # Analyzes //foo in the target configuration, but also analyzes @@ -105,13 +137,22 @@ $ bazel cquery //foo --universe_scope=//foo,//genrule_with_foo_as_tool //foo (exec) ``` -If you want to precisely declare which instance to query over, use the [`config`](#config) function. +If you want to precisely declare which instance to query over, use +the [`config`](#config) function. -See `query`'s [target pattern documentation](/query/language#target-patterns) for more information on target patterns. +See `query`'s [target pattern +documentation](/query/language#target-patterns) for more information on target +patterns. ## Functions -Of the [set of functions](/query/language#functions "list of query functions") supported by `query`, `cquery` supports all but [`allrdeps`](/query/language#allrdeps), [`buildfiles`](/query/language#buildfiles), [`rbuildfiles`](/query/language#rbuildfiles), [`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and [`visible`](/query/language#visible). +Of the [set of functions](/query/language#functions "list of query functions") +supported by `query`, `cquery` supports all but +[`allrdeps`](/query/language#allrdeps), +[`buildfiles`](/query/language#buildfiles), +[`rbuildfiles`](/query/language#rbuildfiles), +[`siblings`](/query/language#siblings), [`tests`](/query/language#tests), and +[`visible`](/query/language#visible). `cquery` also introduces the following new functions: @@ -119,9 +160,13 @@ Of the [set of functions](/query/language#functions "list of query functions") s `expr ::= config(expr, word)` -The `config` operator attempts to find the configured target for the label denoted by the first argument and configuration specified by the second argument. +The `config` operator attempts to find the configured target for +the label denoted by the first argument and configuration specified by the +second argument. -Valid values for the second argument are `null` or a [custom configuration hash](#configurations). Hashes can be retrieved from `$ bazel config` or a previous `cquery`'s output. +Valid values for the second argument are `null` or a +[custom configuration hash](#configurations). Hashes can be retrieved from `$ +bazel config` or a previous `cquery`'s output. Examples: @@ -137,19 +182,27 @@ $ bazel cquery "deps(//foo)" $ bazel cquery "config(//baz, 3732cc8)" ``` -If not all results of the first argument can be found in the specified configuration, only those that can be found are returned. If no results can be found in the specified configuration, the query fails. +If not all results of the first argument can be found in the specified +configuration, only those that can be found are returned. If no results +can be found in the specified configuration, the query fails. ## Options ### Build options -`cquery` runs over a regular Bazel build and thus inherits the set of [options](/reference/command-line-reference#build-options) available during a build. +`cquery` runs over a regular Bazel build and thus inherits the set of +[options](/reference/command-line-reference#build-options) available during a +build. -### Using cquery options +### Using cquery options #### `--universe_scope` (comma-separated list) -Often, the dependencies of configured targets go through [transitions](/extending/rules#configurations), which causes their configuration to differ from their dependent. This flag allows you to query a target as if it were built as a dependency or a transitive dependency of another target. For example: +Often, the dependencies of configured targets go through +[transitions](/extending/rules#configurations), +which causes their configuration to differ from their dependent. This flag +allows you to query a target as if it were built as a dependency or a transitive +dependency of another target. For example: ``` # x/BUILD @@ -157,7 +210,7 @@ genrule( name = "my_gen", srcs = ["x.in"], outs = ["x.cc"], - cmd = "$(locations :tool) $< >$@", + cmd = "$(locations :tool) $< >$@", tools = [":tool"], ) cc_binary( @@ -166,34 +219,75 @@ cc_binary( ) ``` -Genrules configure their tools in the [exec configuration](/extending/rules#configurations) so the following queries would produce the following outputs: - -| Query | Target Built | Output | -| ------------------------------------------------------- | ------------ | ---------------------- | -| bazel cquery "//x:tool" | //x:tool | //x:tool(targetconfig) | -| bazel cquery "//x:tool" --universe\_scope="//x:my\_gen" | //x:my\_gen | //x:tool(execconfig) | - -If this flag is set, its contents are built. *If it's not set, all targets mentioned in the query expression are built* instead. The transitive closure of the built targets are used as the universe of the query. Either way, the targets to be built must be buildable at the top level (that is, compatible with top-level options). `cquery` returns results in the transitive closure of these top-level targets. - -Even if it's possible to build all targets in a query expression at the top level, it may be beneficial to not do so. For example, explicitly setting `--universe_scope` could prevent building targets multiple times in configurations you don't care about. It could also help specify which configuration version of a target you're looking for. You should set this flag if your query expression is more complex than `deps(//foo)`. +Genrules configure their tools in the +[exec configuration](/extending/rules#configurations) +so the following queries would produce the following outputs: + + + + + + + + + + + + + + + + + + + + + +
QueryTarget BuiltOutput
bazel cquery "//x:tool"//x:tool//x:tool(targetconfig)
bazel cquery "//x:tool" --universe_scope="//x:my_gen"//x:my_gen//x:tool(execconfig)
+ +If this flag is set, its contents are built. _If it's not set, all targets +mentioned in the query expression are built_ instead. The transitive closure of +the built targets are used as the universe of the query. Either way, the targets +to be built must be buildable at the top level (that is, compatible with +top-level options). `cquery` returns results in the transitive closure of these +top-level targets. + +Even if it's possible to build all targets in a query expression at the top +level, it may be beneficial to not do so. For example, explicitly setting +`--universe_scope` could prevent building targets multiple times in +configurations you don't care about. It could also help specify which +configuration version of a target you're looking for. You should set this flag +if your query expression is more complex than `deps(//foo)`. #### `--implicit_deps` (boolean, default=True) -Setting this flag to false filters out all results that aren't explicitly set in the BUILD file and instead set elsewhere by Bazel. This includes filtering resolved toolchains. +Setting this flag to false filters out all results that aren't explicitly set in +the BUILD file and instead set elsewhere by Bazel. This includes filtering +resolved toolchains. #### `--tool_deps` (boolean, default=True) -Setting this flag to false filters out all configured targets for which the path from the queried target to them crosses a transition between the target configuration and the [non-target configurations](/extending/rules#configurations). If the queried target is in the target configuration, setting `--notool_deps` will only return targets that also are in the target configuration. If the queried target is in a non-target configuration, setting `--notool_deps` will only return targets also in non-target configurations. This setting generally does not affect filtering of resolved toolchains. +Setting this flag to false filters out all configured targets for which the +path from the queried target to them crosses a transition between the target +configuration and the +[non-target configurations](/extending/rules#configurations). If the queried +target is in the target configuration, setting `--notool_deps` will only return +targets that also are in the target configuration. If the queried target is in a +non-target configuration, setting `--notool_deps` will only return targets also +in non-target configurations. This setting generally does not affect filtering +of resolved toolchains. #### `--include_aspects` (boolean, default=True) Include dependencies added by [aspects](/extending/aspects). -If this flag is disabled, `cquery somepath(X, Y)` and `cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. +If this flag is disabled, `cquery somepath(X, Y)` and +`cquery deps(X) | grep 'Y'` omit Y if X only depends on it through an aspect. ## Output formats -By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. There are other options for exposing the results as well. +By default, cquery outputs results in a dependency-ordered list of label and configuration pairs. +There are other options for exposing the results as well. ### Transitions @@ -202,11 +296,22 @@ By default, cquery outputs results in a dependency-ordered list of label and con --transitions=full ``` -Configuration [transitions](/extending/rules#configurations) are used to build targets underneath the top level targets in different configurations than the top level targets. +Configuration [transitions](/extending/rules#configurations) +are used to build targets underneath the top level targets in different +configurations than the top level targets. -For example, a target might impose a transition to the exec configuration on all dependencies in its `tools` attribute. These are known as attribute transitions. Rules can also impose transitions on their own configurations, known as rule class transitions. This output format outputs information about these transitions such as what type they are and the effect they have on build options. +For example, a target might impose a transition to the exec configuration on all +dependencies in its `tools` attribute. These are known as attribute +transitions. Rules can also impose transitions on their own configurations, +known as rule class transitions. This output format outputs information about +these transitions such as what type they are and the effect they have on build +options. -This output format is triggered by the `--transitions` flag which by default is set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs information about rule class transitions and attribute transitions including a detailed diff of the options before and after the transition. `LITE` mode outputs the same information without the options diff. +This output format is triggered by the `--transitions` flag which by default is +set to `NONE`. It can be set to `FULL` or `LITE` mode. `FULL` mode outputs +information about rule class transitions and attribute transitions including a +detailed diff of the options before and after the transition. `LITE` mode +outputs the same information without the options diff. ### Protocol message output @@ -214,17 +319,27 @@ This output format is triggered by the `--transitions` flag which by default is --output=proto ``` -This option causes the resulting targets to be printed in a binary protocol buffer form. The definition of the protocol buffer can be found at [src/main/protobuf/analysis\_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). +This option causes the resulting targets to be printed in a binary protocol +buffer form. The definition of the protocol buffer can be found at +[src/main/protobuf/analysis_v2.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/analysis_v2.proto). -`CqueryResult` is the top level message containing the results of the cquery. It has a list of `ConfiguredTarget` messages and a list of `Configuration` messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal to that of the `id` field from the corresponding `Configuration` message. +`CqueryResult` is the top level message containing the results of the cquery. It +has a list of `ConfiguredTarget` messages and a list of `Configuration` +messages. Each `ConfiguredTarget` has a `configuration_id` whose value is equal +to that of the `id` field from the corresponding `Configuration` message. -#### --\[no]proto:include\_configurations +#### --[no]proto:include_configurations -By default, cquery results return configuration information as part of each configured target. If you'd like to omit this information and get proto output that is formatted exactly like query's proto output, set this flag to false. +By default, cquery results return configuration information as part of each +configured target. If you'd like to omit this information and get proto output +that is formatted exactly like query's proto output, set this flag to false. -See [query's proto output documentation](/query/language#output-formats) for more proto output-related options. +See [query's proto output documentation](/query/language#output-formats) +for more proto output-related options. -Note: While selects are resolved both at the top level of returned targets and within attributes, all possible inputs for selects are still included as `rule_input` fields. +Note: While selects are resolved both at the top level of returned +targets and within attributes, all possible inputs for selects are still +included as `rule_input` fields. ### Graph output @@ -232,7 +347,10 @@ Note: While selects are resolved both at the top level of returned targets and w --output=graph ``` -This option generates output as a Graphviz-compatible .dot file. See `query`'s [graph output documentation](/query/language#display-result-graph) for details. `cquery` also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and [`--graph:factored`](/query/language#graph-factored). +This option generates output as a Graphviz-compatible .dot file. See `query`'s +[graph output documentation](/query/language#display-result-graph) for details. `cquery` +also supports [`--graph:node_limit`](/query/language#graph-nodelimit) and +[`--graph:factored`](/query/language#graph-factored). ### Files output @@ -240,11 +358,23 @@ This option generates output as a Graphviz-compatible .dot file. See `query`'s [ --output=files ``` -This option prints a list of the output files produced by each target matched by the query similar to the list printed at the end of a `bazel build` invocation. The output contains only the files advertised in the requested output groups as determined by the [`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. It does include source files. +This option prints a list of the output files produced by each target matched +by the query similar to the list printed at the end of a `bazel build` +invocation. The output contains only the files advertised in the requested +output groups as determined by the +[`--output_groups`](/reference/command-line-reference#flag--output_groups) flag. +It does include source files. -All paths emitted by this output format are relative to the [execroot](https://bazel.build/remote/output-directories), which can be obtained via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, paths to files in the main repository also resolve relative to the workspace directory. +All paths emitted by this output format are relative to the +[execroot](https://bazel.build/remote/output-directories), which can be obtained +via `bazel info execution_root`. If the `bazel-out` convenience symlink exists, +paths to files in the main repository also resolve relative to the workspace +directory. -Note: The output of `bazel cquery --output=files //pkg:foo` contains the output files of `//pkg:foo` in *all* configurations that occur in the build (also see the [section on target pattern evaluation](#target-pattern-evaluation)). If that is not desired, wrap you query in [`config(..., target)`](#config). +Note: The output of `bazel cquery --output=files //pkg:foo` contains the output +files of `//pkg:foo` in *all* configurations that occur in the build (also see +the [section on target pattern evaluation](#target-pattern-evaluation)). If that +is not desired, wrap you query in [`config(..., target)`](#config). ### Defining the output format using Starlark @@ -252,21 +382,39 @@ Note: The output of `bazel cquery --output=files //pkg:foo` contains the output --output=starlark ``` -This output format calls a [Starlark](/rules/language) function for each configured target in the query result, and prints the value returned by the call. The `--starlark:file` flag specifies the location of a Starlark file that defines a function named `format` with a single parameter, `target`. This function is called for each [Target](/rules/lib/builtins/Target) in the query result. Alternatively, for convenience, you may specify just the body of a function declared as `def format(target): return expr` by using the `--starlark:expr` flag. +This output format calls a [Starlark](/rules/language) +function for each configured target in the query result, and prints the value +returned by the call. The `--starlark:file` flag specifies the location of a +Starlark file that defines a function named `format` with a single parameter, +`target`. This function is called for each [Target](/rules/lib/builtins/Target) +in the query result. Alternatively, for convenience, you may specify just the +body of a function declared as `def format(target): return expr` by using the +`--starlark:expr` flag. #### 'cquery' Starlark dialect -The cquery Starlark environment differs from a BUILD or .bzl file. It includes all core Starlark [built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), plus a few cquery-specific ones described below, but not (for example) `glob`, `native`, or `rule`, and it does not support load statements. +The cquery Starlark environment differs from a BUILD or .bzl file. It includes +all core Starlark +[built-in constants and functions](https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions), +plus a few cquery-specific ones described below, but not (for example) `glob`, +`native`, or `rule`, and it does not support load statements. -##### build\_options(target) +##### build_options(target) -`build_options(target)` returns a map whose keys are build option identifiers (see [Configurations](/extending/config)) and whose values are their Starlark values. Build options whose values are not legal Starlark values are omitted from this map. +`build_options(target)` returns a map whose keys are build option identifiers +(see [Configurations](/extending/config)) and whose values are their Starlark +values. Build options whose values are not legal Starlark values are omitted +from this map. -If the target is an input file, `build_options(target)` returns None, as input file targets have a null configuration. +If the target is an input file, `build_options(target)` returns None, as input +file targets have a null configuration. ##### providers(target) -`providers(target)` returns a map whose keys are names of [providers](/extending/rules#providers) (for example, `"DefaultInfo"`) and whose values are their Starlark values. Providers whose values are not legal Starlark values are omitted from this map. +`providers(target)` returns a map whose keys are names of +[providers](/extending/rules#providers) +(for example, `"DefaultInfo"`) and whose values are their Starlark values. +Providers whose values are not legal Starlark values are omitted from this map. #### Examples @@ -277,7 +425,8 @@ Print a space-separated list of the base names of all files produced by `//foo`: --starlark:expr="' '.join([f.basename for f in providers(target)['DefaultInfo'].files.to_list()])" ``` -Print a space-separated list of the paths of all files produced by **rule** targets in `//bar` and its subpackages: +Print a space-separated list of the paths of all files produced by **rule** targets in +`//bar` and its subpackages: ``` bazel cquery 'kind(rule, //bar/...)' --output=starlark \ @@ -305,7 +454,8 @@ Print the value of the command line option `--javacopt` when building `//foo`. --starlark:expr="build_options(target)['//command_line_option:javacopt']" ``` -Print the label of each target with exactly one output. This example uses Starlark functions defined in a file. +Print the label of each target with exactly one output. This example uses +Starlark functions defined in a file. ``` $ cat example.cquery @@ -322,7 +472,8 @@ Print the label of each target with exactly one output. This example uses Starla $ bazel cquery //baz --output=starlark --starlark:file=example.cquery ``` -Print the label of each target which is strictly Python 3. This example uses Starlark functions defined in a file. +Print the label of each target which is strictly Python 3. This example uses +Starlark functions defined in a file. ``` $ cat example.cquery @@ -343,7 +494,7 @@ Extract a value from a user defined Provider. ``` $ cat some_package/my_rule.bzl - MyRuleInfo = provider(fields={"color": "the name of a color"}) + MyRuleInfo = provider(fields=\{"color": "the name of a color"\}) def _my_rule_impl(ctx): ... @@ -368,51 +519,96 @@ Extract a value from a user defined Provider. ## cquery vs. query -`cquery` and `query` complement each other and excel in different niches. Consider the following to decide which is right for you: - -- `cquery` follows specific `select()` branches to model the exact graph you build. `query` doesn't know which branch the build chooses, so overapproximates by including all branches. -- `cquery`'s precision requires building more of the graph than `query` does. Specifically, `cquery` evaluates *configured targets* while `query` only evaluates *targets*. This takes more time and uses more memory. -- `cquery`'s interpretation of the [query language](/query/language) introduces ambiguity that `query` avoids. For example, if `"//foo"` exists in two configurations, which one should `cquery "deps(//foo)"` use? The [`config`](#config) function can help with this. +`cquery` and `query` complement each other and excel in +different niches. Consider the following to decide which is right for you: + +* `cquery` follows specific `select()` branches to + model the exact graph you build. `query` doesn't know which + branch the build chooses, so overapproximates by including all branches. +* `cquery`'s precision requires building more of the graph than + `query` does. Specifically, `cquery` + evaluates _configured targets_ while `query` only + evaluates _targets_. This takes more time and uses more memory. +* `cquery`'s interpretation of + the [query language](/query/language) introduces ambiguity + that `query` avoids. For example, + if `"//foo"` exists in two configurations, which one + should `cquery "deps(//foo)"` use? + The [`config`](#config) function can help with this. ## Non-deterministic output -`cquery` does not automatically wipe the build graph from previous commands. It's therefore prone to picking up results from past queries. +`cquery` does not automatically wipe the build graph from previous commands. +It's therefore prone to picking up results from past queries. -For example, `genrule` exerts an exec transition on its `tools` attribute - that is, it configures its tools in the \[exec configuration] ([https://bazel.build/rules/rules#configurations](https://bazel.build/rules/rules#configurations)). +For example, `genrule` exerts an exec transition on its `tools` attribute - +that is, it configures its tools in the [exec configuration] +(https://bazel.build/rules/rules#configurations). You can see the lingering effects of that transition below. ``` -$ cat > foo/BUILD << +$ cat > foo/BUILD << /tmp/deps.svg ``` -Note: `dot` supports other image formats, just replace `svg` with the format identifier, for example, `png`. +Note: `dot` supports other image formats, just replace `svg` with the +format identifier, for example, `png`. When a dependency graph is big and complicated, it can be helpful start with a single path: @@ -46,7 +62,8 @@ $ bazel query "somepath(//foo:foo, third_party/zlib:zlibonly)" //third_party/zlib:zlibonly ``` -If you do not specify `--output graph` with `allpaths`, you will get a flattened list of the dependency graph. +If you do not specify `--output graph` with `allpaths`, +you will get a flattened list of the dependency graph. ``` $ bazel query "allpaths(//foo, third_party/...)" @@ -76,15 +93,27 @@ $ bazel query "allpaths(//foo, third_party/...)" ### Aside: implicit dependencies -The BUILD file for `//foo` never references `//translations/tools:aggregator`. So, where's the direct dependency? +The BUILD file for `//foo` never references +`//translations/tools:aggregator`. So, where's the direct dependency? -Certain rules include implicit dependencies on additional libraries or tools. For example, to build a `genproto` rule, you need first to build the Protocol Compiler, so every `genproto` rule carries an implicit dependency on the protocol compiler. These dependencies are not mentioned in the build file, but added in by the build tool. The full set of implicit dependencies is currently undocumented. Using `--noimplicit_deps` allows you to filter out these deps from your query results. For cquery, this will include resolved toolchains. +Certain rules include implicit dependencies on additional libraries or tools. +For example, to build a `genproto` rule, you need first to build the Protocol +Compiler, so every `genproto` rule carries an implicit dependency on the +protocol compiler. These dependencies are not mentioned in the build file, +but added in by the build tool. The full set of implicit dependencies is + currently undocumented. Using `--noimplicit_deps` allows you to filter out + these deps from your query results. For cquery, this will include resolved toolchains. ## Reverse dependencies -You might want to know the set of targets that depends on some target. For instance, if you're going to change some code, you might want to know what other code you're about to break. You can use `rdeps(u, x)` to find the reverse dependencies of the targets in `x` within the transitive closure of `u`. +You might want to know the set of targets that depends on some target. For instance, +if you're going to change some code, you might want to know what other code +you're about to break. You can use `rdeps(u, x)` to find the reverse +dependencies of the targets in `x` within the transitive closure of `u`. -Bazel's [Sky Query](/query/language#sky-query) supports the `allrdeps` function which allows you to query reverse dependencies in a universe you specify. +Bazel's [Sky Query](/query/language#sky-query) +supports the `allrdeps` function which allows you to query reverse dependencies +in a universe you specify. ## Miscellaneous uses @@ -94,47 +123,33 @@ You can use `bazel query` to analyze many dependency relationships. #### What packages exist beneath `foo`? -``` -bazel query 'foo/...' --output package -``` +```bazel query 'foo/...' --output package``` #### What rules are defined in the `foo` package? -``` -bazel query 'kind(rule, foo:*)' --output label_kind -``` +```bazel query 'kind(rule, foo:*)' --output label_kind``` #### What files are generated by rules in the `foo` package? -``` -bazel query 'kind("generated file", //foo:*)' -``` +```bazel query 'kind("generated file", //foo:*)'``` #### What targets are generated by starlark macro `foo`? -``` -bazel query 'attr(generator_function, foo, //path/to/search/...)' -``` +```bazel query 'attr(generator_function, foo, //path/to/search/...)'``` #### What's the set of BUILD files needed to build `//foo`? -``` -bazel query 'buildfiles(deps(//foo))' | cut -f1 -d: -``` +```bazel query 'buildfiles(deps(//foo))' | cut -f1 -d:``` #### What are the individual tests that a `test_suite` expands to? -``` -bazel query 'tests(//foo:smoke_tests)' -``` +```bazel query 'tests(//foo:smoke_tests)'``` #### Which of those are C++ tests? -``` -bazel query 'kind(cc_.*, tests(//foo:smoke_tests))' -``` +```bazel query 'kind(cc_.*, tests(//foo:smoke_tests))'``` -#### Which of those are small? Medium? Large? +#### Which of those are small? Medium? Large? ``` bazel query 'attr(size, small, tests(//foo:smoke_tests))' @@ -146,27 +161,19 @@ bazel query 'attr(size, large, tests(//foo:smoke_tests))' #### What are the tests beneath `foo` that match a pattern? -``` -bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))' -``` +```bazel query 'filter("pa?t", kind(".*_test rule", //foo/...))'``` The pattern is a regex and is applied to the full name of the rule. It's similar to doing -``` -bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t' -``` +```bazel query 'kind(".*_test rule", //foo/...)' | grep -E 'pa?t'``` #### What package contains file `path/to/file/bar.java`? -``` - bazel query path/to/file/bar.java --output=package -``` +``` bazel query path/to/file/bar.java --output=package``` #### What is the build label for `path/to/file/bar.java?` -``` -bazel query path/to/file/bar.java -``` +```bazel query path/to/file/bar.java``` #### What rule target(s) contain file `path/to/file/bar.java` as a source? @@ -179,36 +186,28 @@ bazel query "attr('srcs', $fullname, ${fullname//:*/}:*)" #### What packages does `foo` depend on? (What do I need to check out to build `foo`) -``` -bazel query 'buildfiles(deps(//foo:foo))' --output package -``` +```bazel query 'buildfiles(deps(//foo:foo))' --output package``` -Note: `buildfiles` is required in order to correctly obtain all files referenced by `subinclude`; see the reference manual for details. +Note: `buildfiles` is required in order to correctly obtain all files +referenced by `subinclude`; see the reference manual for details. #### What packages does the `foo` tree depend on, excluding `foo/contrib`? -``` -bazel query 'deps(foo/... except foo/contrib/...)' --output package -``` +```bazel query 'deps(foo/... except foo/contrib/...)' --output package``` ### What rule dependencies exist ... #### What genproto rules does bar depend upon? -``` -bazel query 'kind(genproto, deps(bar/...))' -``` +```bazel query 'kind(genproto, deps(bar/...))'``` #### Find the definition of some JNI (C++) library that is transitively depended upon by a Java binary rule in the servlet tree. -``` -bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location -``` +```bazel query 'some(kind(cc_.*library, deps(kind(java_binary, //java/com/example/frontend/...))))' --output location``` ##### ...Now find the definitions of all the Java binaries that depend on them -``` -bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in +```bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in let cls = kind(cc_.*library, deps($jbs)) in $jbs intersect allpaths($jbs, $cls)' ``` @@ -219,67 +218,54 @@ bazel query 'let jbs = kind(java_binary, //java/com/example/frontend/...) in Source files: -``` -bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$ -``` +```bazel query 'kind("source file", deps(//path/to/target/foo/...))' | grep java$``` Generated files: -``` -bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$ -``` +```bazel query 'kind("generated file", deps(//path/to/target/foo/...))' | grep java$``` -#### What is the complete set of Java source files required to build QUX's tests? +#### What is the complete set of Java source files required to build QUX's tests? Source files: -``` -bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ -``` +```bazel query 'kind("source file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` Generated files: -``` -bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$ -``` +```bazel query 'kind("generated file", deps(kind(".*_test rule", javatests/com/example/qux/...)))' | grep java$``` ### What differences in dependencies between X and Y exist ... #### What targets does `//foo` depend on that `//foo:foolib` does not? -``` -bazel query 'deps(//foo) except deps(//foo:foolib)' -``` +```bazel query 'deps(//foo) except deps(//foo:foolib)'``` -#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does *not* depend on? +#### What C++ libraries do the `foo` tests depend on that the `//foo` production binary does _not_ depend on? -``` -bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))' -``` +```bazel query 'kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo))'``` ### Why does this dependency exist ... #### Why does `bar` depend on `groups2`? -``` -bazel query 'somepath(bar/...,groups2/...:*)' -``` +```bazel query 'somepath(bar/...,groups2/...:*)'``` -Once you have the results of this query, you will often find that a single target stands out as being an unexpected or egregious and undesirable dependency of `bar`. The query can then be further refined to: +Once you have the results of this query, you will often find that a single +target stands out as being an unexpected or egregious and undesirable +dependency of `bar`. The query can then be further refined to: #### Show me a path from `docker/updater:updater_systest` (a `py_test`) to some `cc_library` that it depends upon: -``` -bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in - somepath(docker/updater:updater_systest, $cc)' -``` +```bazel query 'let cc = kind(cc_library, deps(docker/updater:updater_systest)) in + somepath(docker/updater:updater_systest, $cc)'``` #### Why does library `//photos/frontend:lib` depend on two variants of the same library `//third_party/jpeglib` and `//third_party/jpeg`? -This query boils down to: "show me the subgraph of `//photos/frontend:lib` that depends on both libraries". When shown in topological order, the last element of the result is the most likely culprit. +This query boils down to: "show me the subgraph of `//photos/frontend:lib` that +depends on both libraries". When shown in topological order, the last element +of the result is the most likely culprit. -``` -bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) +```bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) intersect allpaths(//photos/frontend:lib, //third_party/jpeg)' //photos/frontend:lib @@ -291,41 +277,41 @@ bazel query 'allpaths(//photos/frontend:lib, //third_party/jpeglib) //third_party/jpeg/img:renderer ``` -### What depends on ... +### What depends on ... #### What rules under bar depend on Y? -``` -bazel query 'bar/... intersect allpaths(bar/..., Y)' -``` +```bazel query 'bar/... intersect allpaths(bar/..., Y)'``` -Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X depend on Y?" If expression X is non-trivial, it may be convenient to bind a name to it using `let` to avoid duplication. +Note: `X intersect allpaths(X, Y)` is the general idiom for the query "which X +depend on Y?" If expression X is non-trivial, it may be convenient to bind a +name to it using `let` to avoid duplication. #### What targets directly depend on T, in T's package? -``` -bazel query 'same_pkg_direct_rdeps(T)' -``` +```bazel query 'same_pkg_direct_rdeps(T)'``` ### How do I break a dependency ... +{/* TODO find a convincing value of X to plug in here */} + #### What dependency paths do I have to break to make `bar` no longer depend on X? To output the graph to a `svg` file: -``` -bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg -``` +```bazel query 'allpaths(bar/...,X)' --output graph | dot -Tsvg > /tmp/dep.svg``` ### Misc #### How many sequential steps are there in the `//foo-tests` build? -Unfortunately, the query language can't currently give you the longest path from x to y, but it can find the (or rather *a*) most distant node from the starting point, or show you the *lengths* of the longest path from x to every y that it depends on. Use `maxrank`: +Unfortunately, the query language can't currently give you the longest path +from x to y, but it can find the (or rather _a_) most distant node from the +starting point, or show you the _lengths_ of the longest path from x to every +y that it depends on. Use `maxrank`: -``` -bazel query 'deps(//foo-tests)' --output maxrank | tail -1 -85 //third_party/zlib:zutil.c -``` +```bazel query 'deps(//foo-tests)' --output maxrank | tail -1 +85 //third_party/zlib:zutil.c``` -The result indicates that there exist paths of length 85 that must occur in order in this build. +The result indicates that there exist paths of length 85 that must occur in +order in this build. diff --git a/query/language.mdx b/query/language.mdx deleted file mode 100644 index 8f00fa6f..00000000 --- a/query/language.mdx +++ /dev/null @@ -1,872 +0,0 @@ ---- -title: 'The Bazel Query Reference' ---- - -This page is the reference manual for the *Bazel Query Language* used when you use `bazel query` to analyze build dependencies. It also describes the output formats `bazel query` supports. - -For practical use cases, see the [Bazel Query How-To](/query/guide). - -## Additional query reference - -In addition to `query`, which runs on the post-loading phase target graph, Bazel includes *action graph query* and *configurable query*. - -### Action graph query - -The action graph query (`aquery`) operates on the post-analysis Configured Target Graph and exposes information about **Actions**, **Artifacts**, and their relationships. `aquery` is useful when you are interested in the properties of the Actions/Artifacts generated from the Configured Target Graph. For example, the actual commands run and their inputs, outputs, and mnemonics. - -For more details, see the [aquery reference](/query/aquery). - -### Configurable query - -Traditional Bazel query runs on the post-loading phase target graph and therefore has no concept of configurations and their related concepts. Notably, it doesn't correctly resolve [select statements](/reference/be/functions#select) and instead returns all possible resolutions of selects. However, the configurable query environment, `cquery`, properly handles configurations but doesn't provide all of the functionality of this original query. - -For more details, see the [cquery reference](/query/cquery). - -## Examples - -How do people use `bazel query`? Here are typical examples: - -Why does the `//foo` tree depend on `//bar/baz`? Show a path: - -``` -somepath(foo/..., //bar/baz:all) -``` - -What C++ libraries do all the `foo` tests depend on that the `foo_bin` target does not? - -``` -kind("cc_library", deps(kind(".*test rule", foo/...)) except deps(//foo:foo_bin)) -``` - -## Tokens: The lexical syntax - -Expressions in the query language are composed of the following tokens: - -- **Keywords**, such as `let`. Keywords are the reserved words of the language, and each of them is described below. The complete set of keywords is: - - - [`except`](#set-operations) - - - [`in`](#variables) - - - [`intersect`](#set-operations) - - - [`let`](#variables) - - - [`set`](#set) - - - [`union`](#set-operations) - -- **Words**, such as "`foo/...`" or "`.*test rule`" or "`//bar/baz:all`". If a character sequence is "quoted" (begins and ends with a single-quote ' or begins and ends with a double-quote "), it is a word. If a character sequence is not quoted, it may still be parsed as a word. Unquoted words are sequences of characters drawn from the alphabet characters A-Za-z, the numerals 0-9, and the special characters `*/@.-_:$~[]` (asterisk, forward slash, at, period, hyphen, underscore, colon, dollar sign, tilde, left square brace, right square brace). However, unquoted words may not start with a hyphen `-` or asterisk `*` even though relative [target names](/concepts/labels#target-names) may start with those characters. As a special rule meant to simplify the handling of labels referring to external repositories, unquoted words that start with `@@` may contain `+` characters. - - Unquoted words also may not include the characters plus sign `+` or equals sign `=`, even though those characters are permitted in target names. When writing code that generates query expressions, target names should be quoted. - - Quoting *is* necessary when writing scripts that construct Bazel query expressions from user-supplied values. - - ``` - //foo:bar+wiz # WRONG: scanned as //foo:bar + wiz. - //foo:bar=wiz # WRONG: scanned as //foo:bar = wiz. - "//foo:bar+wiz" # OK. - "//foo:bar=wiz" # OK. - ``` - - Note that this quoting is in addition to any quoting that may be required by your shell, such as: - - ```posix-terminal - bazel query ' "//foo:bar=wiz" ' # single-quotes for shell, double-quotes for Bazel. - ``` - - Keywords and operators, when quoted, are treated as ordinary words. For example, `some` is a keyword but "some" is a word. Both `foo` and "foo" are words. - - However, be careful when using single or double quotes in target names. When quoting one or more target names, use only one type of quotes (either all single or all double quotes). - - The following are examples of what the Java query string will be: - - ``` - 'a"'a' # WRONG: Error message: unclosed quotation. - "a'"a" # WRONG: Error message: unclosed quotation. - '"a" + 'a'' # WRONG: Error message: unexpected token 'a' after query expression '"a" + ' - "'a' + "a"" # WRONG: Error message: unexpected token 'a' after query expression ''a' + ' - "a'a" # OK. - 'a"a' # OK. - '"a" + "a"' # OK - "'a' + 'a'" # OK - ``` - - We chose this syntax so that quote marks aren't needed in most cases. The (unusual) `".*test rule"` example needs quotes: it starts with a period and contains a space. Quoting `"cc_library"` is unnecessary but harmless. - -- **Punctuation**, such as parens `()`, period `.` and comma `,`. Words containing punctuation (other than the exceptions listed above) must be quoted. - -Whitespace characters outside of a quoted word are ignored. - -## Bazel query language concepts - -The Bazel query language is a language of expressions. Every expression evaluates to a **partially-ordered set** of targets, or equivalently, a **graph** (DAG) of targets. This is the only datatype. - -Set and graph refer to the same datatype, but emphasize different aspects of it, for example: - -- **Set:** The partial order of the targets is not interesting. -- **Graph:** The partial order of targets is significant. - -### Cycles in the dependency graph - -Build dependency graphs should be acyclic. - -The algorithms used by the query language are robust against cycles, and will not report cycles as errors. - -Note that the post-loading phase unconfigured target graph that `bazel query` operates over may contain cycles that do not exist in the configured target graph. Cycles in the configured target graph are detected and reported as errors by [`bazel cquery`](/query/cquery) and [`bazel aquery`](/query/aquery). - -### Implicit dependencies - -In addition to build dependencies that are defined explicitly in `BUILD` files, Bazel adds additional *implicit* dependencies to rules. Implicit dependencies may be defined by: - -- [Private attributes](/extending/rules#private_attributes_and_implicit_dependencies) -- [Toolchain requirements](/extending/toolchains#writing-rules-toolchains) - -By default, `bazel query` takes implicit dependencies into account when computing the query result. This behavior can be changed with the `--[no]implicit_deps` option. - -Note that, as query does not consider configurations, potential toolchain **implementations** are not considered dependencies, only the required toolchain types. See [toolchain documentation](/extending/toolchains#writing-rules-toolchains). - -### Soundness - -Bazel query language expressions operate over the build dependency graph, which is the graph implicitly defined by all rule declarations in all `BUILD` files. It is important to understand that this graph is somewhat abstract, and does not constitute a complete description of how to perform all the steps of a build. In order to perform a build, a *configuration* is required too; see the [configurations](/docs/user-manual#configurations) section of the User's Guide for more detail. - -The result of evaluating an expression in the Bazel query language is true *for all configurations*, which means that it may be a conservative over-approximation, and not exactly precise. If you use the query tool to compute the set of all source files needed during a build, it may report more than are actually necessary because, for example, the query tool will include all the files needed to support message translation, even though you don't intend to use that feature in your build. - -### On the preservation of graph order - -Operations preserve any ordering constraints inherited from their subexpressions. You can think of this as "the law of conservation of partial order". Consider an example: if you issue a query to determine the transitive closure of dependencies of a particular target, the resulting set is ordered according to the dependency graph. If you filter that set to include only the targets of `file` kind, the same *transitive* partial ordering relation holds between every pair of targets in the resulting subset - even though none of these pairs is actually directly connected in the original graph. (There are no file-file edges in the build dependency graph). - -However, while all operators *preserve* order, some operations, such as the [set operations](#set-operations) don't *introduce* any ordering constraints of their own. Consider this expression: - -``` -deps(x) union y -``` - -The order of the final result set is guaranteed to preserve all the ordering constraints of its subexpressions, namely, that all the transitive dependencies of `x` are correctly ordered with respect to each other. However, the query guarantees nothing about the ordering of the targets in `y`, nor about the ordering of the targets in `deps(x)` relative to those in `y` (except for those targets in `y` that also happen to be in `deps(x)`). - -Operators that introduce ordering constraints include: `allpaths`, `deps`, `rdeps`, `somepath`, and the target pattern wildcards `package:*`, `dir/...`, etc. - -### Sky query - -*Sky Query* is a mode of query that operates over a specified *universe scope*. - -#### Special functions available only in SkyQuery - -Sky Query mode has the additional query functions `allrdeps` and `rbuildfiles`. These functions operate over the entire universe scope (which is why they don't make sense for normal Query). - -#### Specifying a universe scope - -Sky Query mode is activated by passing the following two flags: (`--universe_scope` or `--infer_universe_scope`) and `--order_output=no`. `--universe_scope=<target_pattern1>,...,<target_patternN>` tells query to preload the transitive closure of the target pattern specified by the target patterns, which can be both additive and subtractive. All queries are then evaluated in this "scope". In particular, the [`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles) operators only return results from this scope. `--infer_universe_scope` tells Bazel to infer a value for `--universe_scope` from the query expression. This inferred value is the list of unique target patterns in the query expression, but this might not be what you want. For example: - -```posix-terminal -bazel query --infer_universe_scope --order_output=no "allrdeps(//my:target)" -``` - -The list of unique target patterns in this query expression is `["//my:target"]`, so Bazel treats this the same as the invocation: - -```posix-terminal -bazel query --universe_scope=//my:target --order_output=no "allrdeps(//my:target)" -``` - -But the result of that query with `--universe_scope` is only `//my:target`; none of the reverse dependencies of `//my:target` are in the universe, by construction! On the other hand, consider: - -```posix-terminal -bazel query --infer_universe_scope --order_output=no "tests(//a/... + b/...) intersect allrdeps(siblings(rbuildfiles(my/starlark/file.bzl)))" -``` - -This is a meaningful query invocation that is trying to compute the test targets in the [`tests`](#tests) expansion of the targets under some directories that transitively depend on targets whose definition uses a certain `.bzl` file. Here, `--infer_universe_scope` is a convenience, especially in the case where the choice of `--universe_scope` would otherwise require you to parse the query expression yourself. - -So, for query expressions that use universe-scoped operators like [`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles) be sure to use `--infer_universe_scope` only if its behavior is what you want. - -Sky Query has some advantages and disadvantages compared to the default query. The main disadvantage is that it cannot order its output according to graph order, and thus certain [output formats](#output-formats) are forbidden. Its advantages are that it provides two operators ([`allrdeps`](#allrdeps) and [`rbuildfiles`](#rbuildfiles)) that are not available in the default query. As well, Sky Query does its work by introspecting the [Skyframe](/reference/skyframe) graph, rather than creating a new graph, which is what the default implementation does. Thus, there are some circumstances in which it is faster and uses less memory. - -## Expressions: Syntax and semantics of the grammar - -This is the grammar of the Bazel query language, expressed in EBNF notation: - -```none -expr ::= <var>word</var> - | let <var>name</var> = <var>expr</var> in <var>expr</var> - | (<var>expr</var>) - | <var>expr</var> intersect <var>expr</var> - | <var>expr</var> ^ <var>expr</var> - | <var>expr</var> union <var>expr</var> - | <var>expr</var> + <var>expr</var> - | <var>expr</var> except <var>expr</var> - | <var>expr</var> - <var>expr</var> - | set(<var>word</var> *) - | <var>word</var> '(' <var>int</var> | <var>word</var> | <var>expr</var> ... ')' -``` - -The following sections describe each of the productions of this grammar in order. - -### Target patterns - -``` -expr ::= <var>word</var> -``` - -Syntactically, a *target pattern* is just a word. It's interpreted as an (unordered) set of targets. The simplest target pattern is a label, which identifies a single target (file or rule). For example, the target pattern `//foo:bar` evaluates to a set containing one element, the target, the `bar` rule. - -Target patterns generalize labels to include wildcards over packages and targets. For example, `foo/...:all` (or just `foo/...`) is a target pattern that evaluates to a set containing all *rules* in every package recursively beneath the `foo` directory; `bar/baz:all` is a target pattern that evaluates to a set containing all the rules in the `bar/baz` package, but not its subpackages. - -Similarly, `foo/...:*` is a target pattern that evaluates to a set containing all *targets* (rules *and* files) in every package recursively beneath the `foo` directory; `bar/baz:*` evaluates to a set containing all the targets in the `bar/baz` package, but not its subpackages. - -Because the `:*` wildcard matches files as well as rules, it's often more useful than `:all` for queries. Conversely, the `:all` wildcard (implicit in target patterns like `foo/...`) is typically more useful for builds. - -`bazel query` target patterns work the same as `bazel build` build targets do. For more details, see [Target Patterns](/docs/user-manual#target-patterns), or type `bazel help target-syntax`. - -Target patterns may evaluate to a singleton set (in the case of a label), to a set containing many elements (as in the case of `foo/...`, which has thousands of elements) or to the empty set, if the target pattern matches no targets. - -All nodes in the result of a target pattern expression are correctly ordered relative to each other according to the dependency relation. So, the result of `foo:*` is not just the set of targets in package `foo`, it is also the *graph* over those targets. (No guarantees are made about the relative ordering of the result nodes against other nodes.) For more details, see the [graph order](#graph-order) section. - -### Variables - -```none -expr ::= let <var>name</var> = <var>expr</var><sub>1</sub> in <var>expr</var><sub>2</sub> - | <var>$name</var> -``` - -The Bazel query language allows definitions of and references to variables. The result of evaluation of a `let` expression is the same as that of \{\{ '`' }}expr{{ '`' \}\}2, with all free occurrences of variable \{\{ '`' }}name{{ '`' \}\} replaced by the value of \{\{ '`' }}expr{{ '`' \}\}1. - -For example, `let v = foo/... in allpaths($v, //common) intersect $v` is equivalent to the `allpaths(foo/...,//common) intersect foo/...`. - -An occurrence of a variable reference `name` other than in an enclosing `let <var>name</var> = ...` expression is an error. In other words, top-level query expressions cannot have free variables. - -In the above grammar productions, `name` is like *word*, but with the additional constraint that it be a legal identifier in the C programming language. References to the variable must be prepended with the "$" character. - -Each `let` expression defines only a single variable, but you can nest them. - -Both [target patterns](#target-patterns) and variable references consist of just a single token, a word, creating a syntactic ambiguity. However, there is no semantic ambiguity, because the subset of words that are legal variable names is disjoint from the subset of words that are legal target patterns. - -Technically speaking, `let` expressions do not increase the expressiveness of the query language: any query expressible in the language can also be expressed without them. However, they improve the conciseness of many queries, and may also lead to more efficient query evaluation. - -### Parenthesized expressions - -```none -expr ::= (<var>expr</var>) -``` - -Parentheses associate subexpressions to force an order of evaluation. A parenthesized expression evaluates to the value of its argument. - -### Algebraic set operations: intersection, union, set difference - -```none -expr ::= <var>expr</var> intersect <var>expr</var> - | <var>expr</var> ^ <var>expr</var> - | <var>expr</var> union <var>expr</var> - | <var>expr</var> + <var>expr</var> - | <var>expr</var> except <var>expr</var> - | <var>expr</var> - <var>expr</var> -``` - -These three operators compute the usual set operations over their arguments. Each operator has two forms, a nominal form, such as `intersect`, and a symbolic form, such as `^`. Both forms are equivalent; the symbolic forms are quicker to type. (For clarity, the rest of this page uses the nominal forms.) - -For example, - -``` -foo/... except foo/bar/... -``` - -evaluates to the set of targets that match `foo/...` but not `foo/bar/...`. - -You can write the same query as: - -``` -foo/... - foo/bar/... -``` - -The `intersect` (`^`) and `union` (`+`) operations are commutative (symmetric); `except` (`-`) is asymmetric. The parser treats all three operators as left-associative and of equal precedence, so you might want parentheses. For example, the first two of these expressions are equivalent, but the third is not: - -``` -x intersect y union z -(x intersect y) union z -x intersect (y union z) -``` - -Important: Use parentheses where there is any danger of ambiguity in reading a query expression. - -### Read targets from an external source: set - -```none -expr ::= set(<var>word</var> *) -``` - -The `set(<var>a</var> <var>b</var> <var>c</var> ...)` operator computes the union of a set of zero or more [target patterns](#target-patterns), separated by whitespace (no commas). - -In conjunction with the Bourne shell's `$(...)` feature, `set()` provides a means of saving the results of one query in a regular text file, manipulating that text file using other programs (such as standard UNIX shell tools), and then introducing the result back into the query tool as a value for further processing. For example: - -```posix-terminal -bazel query deps(//my:target) --output=label | grep ... | sed ... | awk ... > foo - -bazel query "kind(cc_binary, set($(<foo)))" -``` - -In the next example,`kind(cc_library, deps(//some_dir/foo:main, 5))` is computed by filtering on the `maxrank` values using an `awk` program. - -```posix-terminal -bazel query 'deps(//some_dir/foo:main)' --output maxrank | awk '($1 < 5) { print $2;} ' > foo - -bazel query "kind(cc_library, set($(<foo)))" -``` - -In these examples, `$(<foo)` is a shorthand for `$(cat foo)`, but shell commands other than `cat` may be used too—such as the previous `awk` command. - -Note: `set()` introduces no graph ordering constraints, so path information may be lost when saving and reloading sets of nodes using it. For more details, see the [graph order](#graph-order) section below. - -## Functions - -```none -expr ::= <var>word</var> '(' <var>int</var> | <var>word</var> | <var>expr</var> ... ')' -``` - -The query language defines several functions. The name of the function determines the number and type of arguments it requires. The following functions are available: - -- [`allpaths`](#somepath-allpaths) -- [`attr`](#attr) -- [`buildfiles`](#buildfiles) -- [`rbuildfiles`](#rbuildfiles) -- [`deps`](#deps) -- [`filter`](#filter) -- [`kind`](#kind) -- [`labels`](#labels) -- [`loadfiles`](#loadfiles) -- [`rdeps`](#rdeps) -- [`allrdeps`](#allrdeps) -- [`same_pkg_direct_rdeps`](#same_pkg_direct_rdeps) -- [`siblings`](#siblings) -- [`some`](#some) -- [`somepath`](#somepath-allpaths) -- [`tests`](#tests) -- [`visible`](#visible) - -### Transitive closure of dependencies: deps - -```none -expr ::= deps(<var>expr</var>) - | deps(<var>expr</var>, <var>depth</var>) -``` - -The `deps(<var>x</var>)` operator evaluates to the graph formed by the transitive closure of dependencies of its argument set \{\{ '`' }}x{{ '`' \}\}. For example, the value of `deps(//foo)` is the dependency graph rooted at the single node `foo`, including all its dependencies. The value of `deps(foo/...)` is the dependency graphs whose roots are all rules in every package beneath the `foo` directory. In this context, 'dependencies' means only rule and file targets, therefore the `BUILD` and Starlark files needed to create these targets are not included here. For that you should use the [`buildfiles`](#buildfiles) operator. - -The resulting graph is ordered according to the dependency relation. For more details, see the section on [graph order](#graph-order). - -The `deps` operator accepts an optional second argument, which is an integer literal specifying an upper bound on the depth of the search. So `deps(foo:*, 0)` returns all targets in the `foo` package, while `deps(foo:*, 1)` further includes the direct prerequisites of any target in the `foo` package, and `deps(foo:*, 2)` further includes the nodes directly reachable from the nodes in `deps(foo:*, 1)`, and so on. (These numbers correspond to the ranks shown in the [`minrank`](#output-ranked) output format.) If the \{\{ '`' }}depth{{ '`' \}\} parameter is omitted, the search is unbounded: it computes the reflexive transitive closure of prerequisites. - -### Transitive closure of reverse dependencies: rdeps - -```none -expr ::= rdeps(<var>expr</var>, <var>expr</var>) - | rdeps(<var>expr</var>, <var>expr</var>, <var>depth</var>) -``` - -The `rdeps(<var>u</var>, <var>x</var>)` operator evaluates to the reverse dependencies of the argument set \{\{ '`' }}x{{ '`' \}\} within the transitive closure of the universe set \{\{ '`' }}u{{ '`' \}\}. - -The resulting graph is ordered according to the dependency relation. See the section on [graph order](#graph-order) for more details. - -The `rdeps` operator accepts an optional third argument, which is an integer literal specifying an upper bound on the depth of the search. The resulting graph only includes nodes within a distance of the specified depth from any node in the argument set. So `rdeps(//foo, //common, 1)` evaluates to all nodes in the transitive closure of `//foo` that directly depend on `//common`. (These numbers correspond to the ranks shown in the [`minrank`](#output-ranked) output format.) If the \{\{ '`' }}depth{{ '`' \}\} parameter is omitted, the search is unbounded. - -### Transitive closure of all reverse dependencies: allrdeps - -``` -expr ::= allrdeps(<var>expr</var>) - | allrdeps(<var>expr</var>, <var>depth</var>) -``` - -Note: Only available with [Sky Query](#sky-query) - -The `allrdeps` operator behaves just like the [`rdeps`](#rdeps) operator, except that the "universe set" is whatever the `--universe_scope` flag evaluated to, instead of being separately specified. Thus, if `--universe_scope=//foo/...` was passed, then `allrdeps(//bar)` is equivalent to `rdeps(//foo/..., //bar)`. - -### Direct reverse dependencies in the same package: same\_pkg\_direct\_rdeps - -``` -expr ::= same_pkg_direct_rdeps(<var>expr</var>) -``` - -The `same_pkg_direct_rdeps(<var>x</var>)` operator evaluates to the full set of targets that are in the same package as a target in the argument set, and which directly depend on it. - -### Dealing with a target's package: siblings - -``` -expr ::= siblings(<var>expr</var>) -``` - -The `siblings(<var>x</var>)` operator evaluates to the full set of targets that are in the same package as a target in the argument set. - -### Arbitrary choice: some - -``` -expr ::= some(<var>expr</var>) - | some(<var>expr</var>, <var>count</var>) -``` - -The `some(<var>x</var>, <var>k</var>)` operator selects at most \{\{ '`' }}k{{ '`' \}\} targets arbitrarily from its argument set \{\{ '`' }}x{{ '`' \}\}, and evaluates to a set containing only those targets. Parameter \{\{ '`' }}k{{ '`' \}\} is optional; if missing, the result will be a singleton set containing only one target arbitrarily selected. If the size of argument set \{\{ '`' }}x{{ '`' \}\} is smaller than \{\{ '`' }}k{{ '`' \}\}, the whole argument set \{\{ '`' }}x{{ '`' \}\} will be returned. - -For example, the expression `some(//foo:main union //bar:baz)` evaluates to a singleton set containing either `//foo:main` or `//bar:baz`—though which one is not defined. The expression `some(//foo:main union //bar:baz, 2)` or `some(//foo:main union //bar:baz, 3)` returns both `//foo:main` and `//bar:baz`. - -If the argument is a singleton, then `some` computes the identity function: `some(//foo:main)` is equivalent to `//foo:main`. - -It is an error if the specified argument set is empty, as in the expression `some(//foo:main intersect //bar:baz)`. - -### Path operators: somepath, allpaths - -``` -expr ::= somepath(<var>expr</var>, <var>expr</var>) - | allpaths(<var>expr</var>, <var>expr</var>) -``` - -The `somepath(<var>S</var>, <var>E</var>)` and `allpaths(<var>S</var>, <var>E</var>)` operators compute paths between two sets of targets. Both queries accept two arguments, a set \{\{ '`' }}S{{ '`' \}\} of starting points and a set \{\{ '`' }}E{{ '`' \}\} of ending points. `somepath` returns the graph of nodes on *some* arbitrary path from a target in \{\{ '`' }}S{{ '`' \}\} to a target in \{\{ '`' }}E{{ '`' \}\}; `allpaths` returns the graph of nodes on *all* paths from any target in \{\{ '`' }}S{{ '`' \}\} to any target in \{\{ '`' }}E{{ '`' \}\}. - -The resulting graphs are ordered according to the dependency relation. See the section on [graph order](#graph-order) for more details. - -| | | | -| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------ | -| ![Somepath](/docs/images/somepath1.svg)`somepath(S1 + S2, E)`, one possible result. | ![Somepath](/docs/images/somepath2.svg)`somepath(S1 + S2, E)`, another possible result. | ![Allpaths](/docs/images/allpaths.svg)`allpaths(S1 + S2, E)` | - -### Target kind filtering: kind - -``` -expr ::= kind(<var>word</var>, <var>expr</var>) -``` - -The `kind(<var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards those targets that are not of the expected kind. The \{\{ '`' }}pattern{{ '`' \}\} parameter specifies what kind of target to match. - -For example, the kinds for the four targets defined by the `BUILD` file (for package `p`) shown below are illustrated in the table: - -| Code | Target | Kind | -| ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -------------- | -| ``` - genrule( - name = "a", - srcs = ["a.in"], - outs = ["a.out"], - cmd = "...", - ) - -``` | `//p:a` | genrule rule | -| | `//p:a.in` | source file | -| | `//p:a.out` | generated file | -| | `//p:BUILD` | source file | - -Thus, `kind("cc_.* rule", foo/...)` evaluates to the set of all `cc_library`, `cc_binary`, etc, rule targets beneath `foo`, and `kind("source file", deps(//foo))` evaluates to the set of all source files in the transitive closure of dependencies of the `//foo` target. - -Quotation of the \{\{ '`' }}pattern{{ '`' \}\} argument is often required because without it, many [regular expressions](#regex), such as `source file` and `.*_test`, are not considered words by the parser. - -When matching for `package group`, targets ending in `:all` may not yield any results. Use `:all-targets` instead. - -### Target name filtering: filter - -``` -expr ::= filter(<var>word</var>, <var>expr</var>) -``` - -The `filter(<var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets whose labels (in absolute form) do not match the pattern; it evaluates to a subset of its input. - -The first argument, \{\{ '`' }}pattern{{ '`' \}\} is a word containing a [regular expression](#regex) over target names. A `filter` expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\} and the label (in absolute form, such as `//foo:bar`) of \{\{ '`' }}x{{ '`' \}\} contains an (unanchored) match for the regular expression \{\{ '`' }}pattern{{ '`' \}\}. Since all target names start with `//`, it may be used as an alternative to the `^` regular expression anchor. - -This operator often provides a much faster and more robust alternative to the `intersect` operator. For example, in order to see all `bar` dependencies of the `//foo:foo` target, one could evaluate - -``` -deps(//foo) intersect //bar/... -``` - -This statement, however, will require parsing of all `BUILD` files in the `bar` tree, which will be slow and prone to errors in irrelevant `BUILD` files. An alternative would be: - -``` -filter(//bar, deps(//foo)) -``` - -which would first calculate the set of `//foo` dependencies and then would filter only targets matching the provided pattern—in other words, targets with names containing `//bar` as a substring. - -Another common use of the `filter(<var>pattern</var>, <var>expr</var>)` operator is to filter specific files by their name or extension. For example, - -``` -filter("\.cc$", deps(//foo)) -``` - -will provide a list of all `.cc` files used to build `//foo`. - -### Rule attribute filtering: attr - -``` -expr ::= attr(<var>word</var>, <var>word</var>, <var>expr</var>) -``` - -The `attr(<var>name</var>, <var>pattern</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets that aren't rules, rule targets that do not have attribute \{\{ '`' }}name{{ '`' \}\} defined or rule targets where the attribute value does not match the provided [regular expression](#regex) \{\{ '`' }}pattern{{ '`' \}\}; it evaluates to a subset of its input. - -The first argument, \{\{ '`' }}name{{ '`' \}\} is the name of the rule attribute that should be matched against the provided [regular expression](#regex) pattern. The second argument, \{\{ '`' }}pattern{{ '`' \}\} is a regular expression over the attribute values. An `attr` expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\}, is a rule with the defined attribute \{\{ '`' }}name{{ '`' \}\} and the attribute value contains an (unanchored) match for the regular expression \{\{ '`' }}pattern{{ '`' \}\}. If \{\{ '`' }}name{{ '`' \}\} is an optional attribute and rule does not specify it explicitly then default attribute value will be used for comparison. For example, - -``` -attr(linkshared, 0, deps(//foo)) -``` - -will select all `//foo` dependencies that are allowed to have a linkshared attribute (such as, `cc_binary` rule) and have it either explicitly set to 0 or do not set it at all but default value is 0 (such as for `cc_binary` rules). - -List-type attributes (such as `srcs`, `data`, etc) are converted to strings of the form `[value<sub>1</sub>, ..., value<sub>n</sub>]`, starting with a `[` bracket, ending with a `]` bracket and using "`, `" (comma, space) to delimit multiple values. Labels are converted to strings by using the absolute form of the label. For example, an attribute `deps=[":foo", "//otherpkg:bar", "wiz"]` would be converted to the string `[//thispkg:foo, //otherpkg:bar, //thispkg:wiz]`. Brackets are always present, so the empty list would use string value `[]` for matching purposes. For example, - -``` -attr("srcs", "\[\]", deps(//foo)) -``` - -will select all rules among `//foo` dependencies that have an empty `srcs` attribute, while - -``` -attr("data", ".{3,}", deps(//foo)) -``` - -will select all rules among `//foo` dependencies that specify at least one value in the `data` attribute (every label is at least 3 characters long due to the `//` and `:`). - -To select all rules among `//foo` dependencies with a particular `value` in a list-type attribute, use - -``` -attr("tags", "[\[ ]value[,\]]", deps(//foo)) -``` - -This works because the character before `value` will be `[` or a space and the character after `value` will be a comma or `]`. - -To select all rules among `//foo` dependencies with a particular `key` and `value` in a dict-type attribute, use - -``` -attr("some_dict_attribute", "[{ ]key=value[,}]", deps(//foo)) -``` - -This would select `//foo` if `//foo` is defined as - -``` -some_rule( - name = "foo", - some_dict_attribute = { - "key": "value", - }, -) -``` - -This works because the character before `key=value` will be `{` or a space and the character after `key=value` will be a comma or `}`. - -### Rule visibility filtering: visible - -``` -expr ::= visible(<var>expr</var>, <var>expr</var>) -``` - -The `visible(<var>predicate</var>, <var>input</var>)` operator applies a filter to a set of targets, and discards targets without the required visibility. - -The first argument, \{\{ '`' }}predicate{{ '`' \}\}, is a set of targets that all targets in the output must be visible to. A \{\{ '`' }}visible{{ '`' \}\} expression evaluates to the set containing all targets \{\{ '`' }}x{{ '`' \}\} such that \{\{ '`' }}x{{ '`' \}\} is a member of the set \{\{ '`' }}input{{ '`' \}\}, and for all targets \{\{ '`' }}y{{ '`' \}\} in \{\{ '`' }}predicate{{ '`' \}\} \{\{ '`' }}x{{ '`' \}\} is visible to \{\{ '`' }}y{{ '`' \}\}. For example: - -``` -visible(//foo, //bar:*) -``` - -will select all targets in the package `//bar` that `//foo` can depend on without violating visibility restrictions. - -### Evaluation of rule attributes of type label: labels - -``` -expr ::= labels(<var>word</var>, <var>expr</var>) -``` - -The `labels(<var>attr_name</var>, <var>inputs</var>)` operator returns the set of targets specified in the attribute \{\{ '`' }}attr_name{{ '`' \}\} of type "label" or "list of label" in some rule in set \{\{ '`' }}inputs{{ '`' \}\}. - -For example, `labels(srcs, //foo)` returns the set of targets appearing in the `srcs` attribute of the `//foo` rule. If there are multiple rules with `srcs` attributes in the \{\{ '`' }}inputs{{ '`' \}\} set, the union of their `srcs` is returned. - -### Expand and filter test\_suites: tests - -``` -expr ::= tests(<var>expr</var>) -``` - -The `tests(<var>x</var>)` operator returns the set of all test rules in set \{\{ '`' }}x{{ '`' \}\}, expanding any `test_suite` rules into the set of individual tests that they refer to, and applying filtering by `tag` and `size`. - -By default, query evaluation ignores any non-test targets in all `test_suite` rules. This can be changed to errors with the `--strict_test_suite` option. - -For example, the query `kind(test, foo:*)` lists all the `*_test` and `test_suite` rules in the `foo` package. All the results are (by definition) members of the `foo` package. In contrast, the query `tests(foo:*)` will return all of the individual tests that would be executed by `bazel test foo:*`: this may include tests belonging to other packages, that are referenced directly or indirectly via `test_suite` rules. - -### Package definition files: buildfiles - -``` -expr ::= buildfiles(<var>expr</var>) -``` - -The `buildfiles(<var>x</var>)` operator returns the set of files that define the packages of each target in set \{\{ '`' }}x{{ '`' \}\}; in other words, for each package, its `BUILD` file, plus any .bzl files it references via `load`. Note that this also returns the `BUILD` files of the packages containing these `load`ed files. - -This operator is typically used when determining what files or packages are required to build a specified target, often in conjunction with the [`--output package`](#output-package) option, below). For example, - -```posix-terminal -bazel query 'buildfiles(deps(//foo))' --output package -``` - -returns the set of all packages on which `//foo` transitively depends. - -Note: A naive attempt at the above query would omit the `buildfiles` operator and use only `deps`, but this yields an incorrect result: while the result contains the majority of needed packages, those packages that contain only files that are `load()`'ed will be missing. - -Warning: Bazel pretends each `.bzl` file produced by `buildfiles` has a corresponding target (for example, file `a/b.bzl` => target `//a:b.bzl`), but this isn't necessarily the case. Therefore, `buildfiles` doesn't compose well with other query operators and its results can be misleading when formatted in a structured way, such as [`--output=xml`](#xml). - -### Package definition files: rbuildfiles - -``` -expr ::= rbuildfiles(<var>word</var>, ...) -``` - -Note: Only available with [Sky Query](#sky-query). - -The `rbuildfiles` operator takes a comma-separated list of path fragments and returns the set of `BUILD` files that transitively depend on these path fragments. For instance, if `//foo` is a package, then `rbuildfiles(foo/BUILD)` will return the `//foo:BUILD` target. If the `foo/BUILD` file has `load('//bar:file.bzl'...` in it, then `rbuildfiles(bar/file.bzl)` will return the `//foo:BUILD` target, as well as the targets for any other `BUILD` files that load `//bar:file.bzl` - -The scope of the rbuildfiles operator is the universe specified by the `--universe_scope` flag. Files that do not correspond directly to `BUILD` files and `.bzl` files do not affect the results. For instance, source files (like `foo.cc`) are ignored, even if they are explicitly mentioned in the `BUILD` file. Symlinks, however, are respected, so that if `foo/BUILD` is a symlink to `bar/BUILD`, then `rbuildfiles(bar/BUILD)` will include `//foo:BUILD` in its results. - -The `rbuildfiles` operator is almost morally the inverse of the [`buildfiles`](#buildfiles) operator. However, this moral inversion holds more strongly in one direction: the outputs of `rbuildfiles` are just like the inputs of `buildfiles`; the former will only contain `BUILD` file targets in packages, and the latter may contain such targets. In the other direction, the correspondence is weaker. The outputs of the `buildfiles` operator are targets corresponding to all packages and .`bzl` files needed by a given input. However, the inputs of the `rbuildfiles` operator are not those targets, but rather the path fragments that correspond to those targets. - -### Package definition files: loadfiles - -``` -expr ::= loadfiles(<var>expr</var>) -``` - -The `loadfiles(<var>x</var>)` operator returns the set of Starlark files that are needed to load the packages of each target in set \{\{ '`' }}x{{ '`' \}\}. In other words, for each package, it returns the .bzl files that are referenced from its `BUILD` files. - -Warning: Bazel pretends each of these .bzl files has a corresponding target (for example, file `a/b.bzl` => target `//a:b.bzl`), but this isn't necessarily the case. Therefore, `loadfiles` doesn't compose well with other query operators and its results can be misleading when formatted in a structured way, such as [`--output=xml`](#xml). - -## Output formats - -`bazel query` generates a graph. You specify the content, format, and ordering by which `bazel query` presents this graph by means of the `--output` command-line option. - -When running with [Sky Query](#sky-query), only output formats that are compatible with unordered output are allowed. Specifically, `graph`, `minrank`, and `maxrank` output formats are forbidden. - -Some of the output formats accept additional options. The name of each output option is prefixed with the output format to which it applies, so `--graph:factored` applies only when `--output=graph` is being used; it has no effect if an output format other than `graph` is used. Similarly, `--xml:line_numbers` applies only when `--output=xml` is being used. - -### On the ordering of results - -Although query expressions always follow the "[law of conservation of graph order](#graph-order)", *presenting* the results may be done in either a dependency-ordered or unordered manner. This does **not** influence the targets in the result set or how the query is computed. It only affects how the results are printed to stdout. Moreover, nodes that are equivalent in the dependency order may or may not be ordered alphabetically. The `--order_output` flag can be used to control this behavior. (The `--[no]order_results` flag has a subset of the functionality of the `--order_output` flag and is deprecated.) - -The default value of this flag is `auto`, which prints results in **lexicographical order**. However, when `somepath(a,b)` is used, the results will be printed in `deps` order instead. - -When this flag is `no` and `--output` is one of `build`, `label`, `label_kind`, `location`, `package`, `proto`, or `xml`, the outputs will be printed in arbitrary order. **This is generally the fastest option**. It is not supported though when `--output` is one of `graph`, `minrank` or `maxrank`: with these formats, Bazel always prints results ordered by the dependency order or rank. - -When this flag is `deps`, Bazel prints results in some topological order—that is, dependents first and dependencies after. However, nodes that are unordered by the dependency order (because there is no path from either one to the other) may be printed in any order. - -When this flag is `full`, Bazel prints nodes in a fully deterministic (total) order. First, all nodes are sorted alphabetically. Then, each node in the list is used as the start of a post-order depth-first search in which outgoing edges to unvisited nodes are traversed in alphabetical order of the successor nodes. Finally, nodes are printed in the reverse of the order in which they were visited. - -Printing nodes in this order may be slower, so it should be used only when determinism is important. - -### Print the source form of targets as they would appear in BUILD - -``` ---output build -``` - -With this option, the representation of each target is as if it were hand-written in the BUILD language. All variables and function calls (such as glob, macros) are expanded, which is useful for seeing the effect of Starlark macros. Additionally, each effective rule reports a `generator_name` and/or `generator_function`) value, giving the name of the macro that was evaluated to produce the effective rule. - -Although the output uses the same syntax as `BUILD` files, it is not guaranteed to produce a valid `BUILD` file. - -### Print the label of each target - -``` ---output label -``` - -With this option, the set of names (or *labels*) of each target in the resulting graph is printed, one label per line, in topological order (unless `--noorder_results` is specified, see [notes on the ordering of results](#result-order)). (A topological ordering is one in which a graph node appears earlier than all of its successors.) Of course there are many possible topological orderings of a graph (*reverse postorder* is just one); which one is chosen is not specified. - -When printing the output of a `somepath` query, the order in which the nodes are printed is the order of the path. - -Caveat: in some corner cases, there may be two distinct targets with the same label; for example, a `sh_binary` rule and its sole (implicit) `srcs` file may both be called `foo.sh`. If the result of a query contains both of these targets, the output (in `label` format) will appear to contain a duplicate. When using the `label_kind` (see below) format, the distinction becomes clear: the two targets have the same name, but one has kind `sh_binary rule` and the other kind `source file`. - -### Print the label and kind of each target - -``` ---output label_kind -``` - -Like `label`, this output format prints the labels of each target in the resulting graph, in topological order, but it additionally precedes the label by the [*kind*](#kind) of the target. - -### Print targets in protocol buffer format - -``` ---output proto -``` - -Prints the query output as a [`QueryResult`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffer. - -### Print targets in length-delimited protocol buffer format - -``` ---output streamed_proto -``` - -Prints a [length-delimited](https://protobuf.dev/programming-guides/encoding/#size-limit) stream of [`Target`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffers. This is useful to *(i)* get around [size limitations](https://protobuf.dev/programming-guides/encoding/#size-limit) of protocol buffers when there are too many targets to fit in a single `QueryResult` or *(ii)* to start processing while Bazel is still outputting. - -### Print targets in text proto format - -``` ---output textproto -``` - -Similar to `--output proto`, prints the [`QueryResult`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffer but in [text format](https://protobuf.dev/reference/protobuf/textformat-spec/). - -### Print targets in ndjson format - -``` ---output streamed_jsonproto -``` - -Similar to `--output streamed_proto`, prints a stream of [`Target`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto) protocol buffers but in [ndjson](https://github.com/ndjson/ndjson-spec) format. - -### Print the label of each target, in rank order - -``` ---output minrank --output maxrank -``` - -Like `label`, the `minrank` and `maxrank` output formats print the labels of each target in the resulting graph, but instead of appearing in topological order, they appear in rank order, preceded by their rank number. These are unaffected by the result ordering `--[no]order_results` flag (see [notes on the ordering of results](#result-order)). - -There are two variants of this format: `minrank` ranks each node by the length of the shortest path from a root node to it. "Root" nodes (those which have no incoming edges) are of rank 0, their successors are of rank 1, etc. (As always, edges point from a target to its prerequisites: the targets it depends upon.) - -`maxrank` ranks each node by the length of the longest path from a root node to it. Again, "roots" have rank 0, all other nodes have a rank which is one greater than the maximum rank of all their predecessors. - -All nodes in a cycle are considered of equal rank. (Most graphs are acyclic, but cycles do occur simply because `BUILD` files contain erroneous cycles.) - -These output formats are useful for discovering how deep a graph is. If used for the result of a `deps(x)`, `rdeps(x)`, or `allpaths` query, then the rank number is equal to the length of the shortest (with `minrank`) or longest (with `maxrank`) path from `x` to a node in that rank. `maxrank` can be used to determine the longest sequence of build steps required to build a target. - -Note: The ranked output of a `somepath` query is basically meaningless because `somepath` doesn't guarantee to return either a shortest or a longest path, and it may include "transitive" edges from one path node to another that are not direct edges in original graph. - -For example, the graph on the left yields the outputs on the right when `--output minrank` and `--output maxrank` are specified, respectively. - -| | | -| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ![Out ranked](/docs/images/out-ranked.svg) | ``` - minrank - - 0 //c:c - 1 //b:b - 1 //a:a - 2 //b:b.cc - 2 //a:a.cc - </pre> -</td> -<td> - <pre> - maxrank - - 0 //c:c - 1 //b:b - 2 //a:a - 2 //b:b.cc - 3 //a:a.cc - </pre> -</td> - - -``` | - -### Print the location of each target - -``` ---output location -``` - -Like `label_kind`, this option prints out, for each target in the result, the target's kind and label, but it is prefixed by a string describing the location of that target, as a filename and line number. The format resembles the output of `grep`. Thus, tools that can parse the latter (such as Emacs or vi) can also use the query output to step through a series of matches, allowing the Bazel query tool to be used as a dependency-graph-aware "grep for BUILD files". - -The location information varies by target kind (see the [kind](#kind) operator). For rules, the location of the rule's declaration within the `BUILD` file is printed. For source files, the location of line 1 of the actual file is printed. For a generated file, the location of the rule that generates it is printed. (The query tool does not have sufficient information to find the actual location of the generated file, and in any case, it might not exist if a build has not yet been performed.) - -### Print the set of packages - -`--output package` - -This option prints the name of all packages to which some target in the result set belongs. The names are printed in lexicographical order; duplicates are excluded. Formally, this is a *projection* from the set of labels (package, target) onto packages. - -Packages in external repositories are formatted as `@repo//foo/bar` while packages in the main repository are formatted as `foo/bar`. - -In conjunction with the `deps(...)` query, this output option can be used to find the set of packages that must be checked out in order to build a given set of targets. - -### Display a graph of the result - -`--output graph` - -This option causes the query result to be printed as a directed graph in the popular AT\&T GraphViz format. Typically the result is saved to a file, such as `.png` or `.svg`. (If the `dot` program is not installed on your workstation, you can install it using the command `sudo apt-get install graphviz`.) See the example section below for a sample invocation. - -This output format is particularly useful for `allpaths`, `deps`, or `rdeps` queries, where the result includes a *set of paths* that cannot be easily visualized when rendered in a linear form, such as with `--output label`. - -By default, the graph is rendered in a *factored* form. That is, topologically-equivalent nodes are merged together into a single node with multiple labels. This makes the graph more compact and readable, because typical result graphs contain highly repetitive patterns. For example, a `java_library` rule may depend on hundreds of Java source files all generated by the same `genrule`; in the factored graph, all these files are represented by a single node. This behavior may be disabled with the `--nograph:factored` option. - -#### `--graph:node_limit <var>n</var>` - -The option specifies the maximum length of the label string for a graph node in the output. Longer labels will be truncated; -1 disables truncation. Due to the factored form in which graphs are usually printed, the node labels may be very long. GraphViz cannot handle labels exceeding 1024 characters, which is the default value of this option. This option has no effect unless `--output=graph` is being used. - -#### `--[no]graph:factored` - -By default, graphs are displayed in factored form, as explained [above](#output-graph). When `--nograph:factored` is specified, graphs are printed without factoring. This makes visualization using GraphViz impractical, but the simpler format may ease processing by other tools (such as grep). This option has no effect unless `--output=graph` is being used. - -### XML - -`--output xml` - -This option causes the resulting targets to be printed in an XML form. The output starts with an XML header such as this - -``` - <?xml version="1.0" encoding="UTF-8"?> - <query version="2"> -``` - -and then continues with an XML element for each target in the result graph, in topological order (unless [unordered results](#result-order) are requested), and then finishes with a terminating - -``` -</query> -``` - -Simple entries are emitted for targets of `file` kind: - -``` - <source-file name='//foo:foo_main.cc' .../> - <generated-file name='//foo:libfoo.so' .../> -``` - -But for rules, the XML is structured and contains definitions of all the attributes of the rule, including those whose value was not explicitly specified in the rule's `BUILD` file. - -Additionally, the result includes `rule-input` and `rule-output` elements so that the topology of the dependency graph can be reconstructed without having to know that, for example, the elements of the `srcs` attribute are forward dependencies (prerequisites) and the contents of the `outs` attribute are backward dependencies (consumers). - -`rule-input` elements for [implicit dependencies](#implicit_deps) are suppressed if `--noimplicit_deps` is specified. - -``` - <rule class='cc_binary rule' name='//foo:foo' ...> - <list name='srcs'> - <label value='//foo:foo_main.cc'/> - <label value='//foo:bar.cc'/> - ... - </list> - <list name='deps'> - <label value='//common:common'/> - <label value='//collections:collections'/> - ... - </list> - <list name='data'> - ... - </list> - <int name='linkstatic' value='0'/> - <int name='linkshared' value='0'/> - <list name='licenses'/> - <list name='distribs'> - <distribution value="INTERNAL" /> - </list> - <rule-input name="//common:common" /> - <rule-input name="//collections:collections" /> - <rule-input name="//foo:foo_main.cc" /> - <rule-input name="//foo:bar.cc" /> - ... - </rule> -``` - -Every XML element for a target contains a `name` attribute, whose value is the target's label, and a `location` attribute, whose value is the target's location as printed by the [`--output location`](#print-target-location). - -#### `--[no]xml:line_numbers` - -By default, the locations displayed in the XML output contain line numbers. When `--noxml:line_numbers` is specified, line numbers are not printed. - -#### `--[no]xml:default_values` - -By default, XML output does not include rule attribute whose value is the default value for that kind of attribute (for example, if it were not specified in the `BUILD` file, or the default value was provided explicitly). This option causes such attribute values to be included in the XML output. - -### Regular expressions - -Regular expressions in the query language use the Java regex library, so you can use the full syntax for [`java.util.regex.Pattern`](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html). - -### Querying with external repositories - -If the build depends on rules from [external repositories](/external/overview) then query results will include these dependencies. For example, if `//foo:bar` depends on `@other-repo//baz:lib`, then `bazel query 'deps(//foo:bar)'` will list `@other-repo//baz:lib` as a dependency. diff --git a/query/quickstart.mdx b/query/quickstart.mdx deleted file mode 100644 index a17220fb..00000000 --- a/query/quickstart.mdx +++ /dev/null @@ -1,500 +0,0 @@ ---- -title: 'Query quickstart' ---- - -This tutorial covers how to work with Bazel to trace dependencies in your code using a premade Bazel project. - -For language and `--output` flag details, see the [Bazel query reference](/query/language) and [Bazel cquery reference](/query/cquery) manuals. Get help in your IDE by typing `bazel help query` or `bazel help cquery` on the command line. - -## Objective - -This guide runs you through a set of basic queries you can use to learn more about your project's file dependencies. It is intended for new Bazel developers with a basic knowledge of how Bazel and `BUILD` files work. - -## Prerequisites - -Start by installing [Bazel](https://bazel.build/install), if you haven’t already. This tutorial uses Git for source control, so for best results, install [Git](https://github.com/git-guides/install-git) as well. - -To visualize dependency graphs, the tool called Graphviz is used, which you can [download](https://graphviz.org/download/) in order to follow along. - -### Get the sample project - -Next, retrieve the sample app from [Bazel's Examples repository](https://github.com/bazelbuild/examples) by running the following in your command-line tool of choice: - -```posix-terminal -git clone https://github.com/bazelbuild/examples.git -``` - -The sample project for this tutorial is in the `examples/query-quickstart` directory. - -## Getting started - -### What are Bazel queries? - -Queries help you to learn about a Bazel codebase by analyzing the relationships between `BUILD` files and examining the resulting output for useful information. This guide previews some basic query functions, but for more options see the [query guide](https://bazel.build/query/guide). Queries help you learn about dependencies in large scale projects without manually navigating through `BUILD` files. - -To run a query, open your command line terminal and enter: - -```posix-terminal -bazel query 'query_function' -``` - -### Scenario - -Imagine a scenario that delves into the relationship between Cafe Bazel and its respective chef. This Cafe exclusively sells pizza and mac & cheese. Take a look below at how the project is structured: - -``` -bazelqueryguide -├── BUILD -├── src -│ └── main -│ └── java -│ └── com -│ └── example -│ ├── customers -│ │ ├── Jenny.java -│ │ ├── Amir.java -│ │ └── BUILD -│ ├── dishes -│ │ ├── Pizza.java -│ │ ├── MacAndCheese.java -│ │ └── BUILD -│ ├── ingredients -│ │ ├── Cheese.java -│ │ ├── Tomatoes.java -│ │ ├── Dough.java -│ │ ├── Macaroni.java -│ │ └── BUILD -│ ├── restaurant -│ │ ├── Cafe.java -│ │ ├── Chef.java -│ │ └── BUILD -│ ├── reviews -│ │ ├── Review.java -│ │ └── BUILD -│ └── Runner.java -└── MODULE.bazel -``` - -Throughout this tutorial, unless directed otherwise, try not to look in the `BUILD` files to find the information you need and instead solely use the query function. - -A project consists of different packages that make up a Cafe. They are separated into: `restaurant`, `ingredients`, `dishes`, `customers`, and `reviews`. Rules within these packages define different components of the Cafe with various tags and dependencies. - -### Running a build - -This project contains a main method inside of `Runner.java` that you can execute to print out a menu of the Cafe. Build the project using Bazel with the command `bazel build` and use `:` to signal that the target is named `runner`. See [target names](https://bazel.build/concepts/labels#target-names) to learn how to reference targets. - -To build this project, paste this command into a terminal: - -```posix-terminal -bazel build :runner -``` - -Your output should look something like this if the build is successful. - -```bash -INFO: Analyzed target //:runner (49 packages loaded, 784 targets configured). -INFO: Found 1 target... -Target //:runner up-to-date: - bazel-bin/runner.jar - bazel-bin/runner -INFO: Elapsed time: 16.593s, Critical Path: 4.32s -INFO: 23 processes: 4 internal, 10 darwin-sandbox, 9 worker. -INFO: Build completed successfully, 23 total actions -``` - -After it has built successfully, run the application by pasting this command: - -```posix-terminal -bazel-bin/runner -``` - -```bash ---------------------- MENU ------------------------- - -Pizza - Cheesy Delicious Goodness -Macaroni & Cheese - Kid-approved Dinner - ----------------------------------------------------- -``` - -This leaves you with a list of the menu items given along with a short description. - -## Exploring targets - -The project lists ingredients and dishes in their own packages. To use a query to view the rules of a package, run the command `bazel query package/…` - -In this case, you can use this to look through the ingredients and dishes that this Cafe has by running: - -```posix-terminal -bazel query //src/main/java/com/example/dishes/... -``` - -```posix-terminal -bazel query //src/main/java/com/example/ingredients/... -``` - -If you query for the targets of the ingredients package, the output should look like: - -```bash -//src/main/java/com/example/ingredients:cheese -//src/main/java/com/example/ingredients:dough -//src/main/java/com/example/ingredients:macaroni -//src/main/java/com/example/ingredients:tomato -``` - -## Finding dependencies - -What targets does your runner rely on to run? - -Say you want to dive deeper into the structure of your project without prodding into the filesystem (which may be untenable for large projects). What rules does Cafe Bazel use? - -If, like in this example, the target for your runner is `runner`, discover the underlying dependencies of the target by running the command: - -```posix-terminal -bazel query --noimplicit_deps "deps(target)" -``` - -```posix-terminal -bazel query --noimplicit_deps "deps(:runner)" -``` - -```bash -//:runner -//:src/main/java/com/example/Runner.java -//src/main/java/com/example/dishes:MacAndCheese.java -//src/main/java/com/example/dishes:Pizza.java -//src/main/java/com/example/dishes:macAndCheese -//src/main/java/com/example/dishes:pizza -//src/main/java/com/example/ingredients:Cheese.java -//src/main/java/com/example/ingredients:Dough.java -//src/main/java/com/example/ingredients:Macaroni.java -//src/main/java/com/example/ingredients:Tomato.java -//src/main/java/com/example/ingredients:cheese -//src/main/java/com/example/ingredients:dough -//src/main/java/com/example/ingredients:macaroni -//src/main/java/com/example/ingredients:tomato -//src/main/java/com/example/restaurant:Cafe.java -//src/main/java/com/example/restaurant:Chef.java -//src/main/java/com/example/restaurant:cafe -//src/main/java/com/example/restaurant:chef -``` - -Note: Adding the flag `--noimplicit_deps` removes configurations and potential toolchains to simplify the list. When you omit this flag, Bazel returns implicit dependencies not specified in the `BUILD` file and clutters the output. - -In most cases, use the query function `deps()` to see individual output dependencies of a specific target. - -## Visualizing the dependency graph (optional) - -Note: This section uses Graphviz, so make sure to [download Graphviz](https://graphviz.org/download/) to follow along. - -The section describes how you can visualize the dependency paths for a specific query. [Graphviz](https://graphviz.org/) helps to see the path as a directed acyclic graph image as opposed to a flattened list. You can alter the display of the Bazel query graph by using various `--output` command line options. See [Output Formats](https://bazel.build/query/language#output-formats) for options. - -Start by running your desired query and add the flag `--noimplicit_deps` to remove excessive tool dependencies. Then, follow the query with the output flag and store the graph into a file called `graph.in` to create a text representation of the graph. - -To search for all dependencies of the target `:runner` and format the output as a graph: - -```posix-terminal -bazel query --noimplicit_deps 'deps(:runner)' --output graph > graph.in -``` - -This creates a file called `graph.in`, which is a text representation of the build graph. Graphviz uses `dot `– a tool that processes text into a visualization — to create a png: - -```posix-terminal -dot -Tpng < graph.in > graph.png -``` - -If you open up `graph.png`, you should see something like this. The graph below has been simplified to make the essential path details clearer in this guide. - -![Diagram showing a relationship from cafe to chef to the dishes: pizza and mac and cheese which diverges into the separate ingredients: cheese, tomatoes, dough, and macaroni.](images/query_graph1.png "Dependency graph") - -This helps when you want to see the outputs of the different query functions throughout this guide. - -## Finding reverse dependencies - -If instead you have a target you’d like to analyze what other targets use it, you can use a query to examine what targets depend on a certain rule. This is called a “reverse dependency”. Using `rdeps()` can be useful when editing a file in a codebase that you’re unfamiliar with, and can save you from unknowingly breaking other files which depended on it. - -For instance, you want to make some edits to the ingredient `cheese`. To avoid causing an issue for Cafe Bazel, you need to check what dishes rely on `cheese`. - -Caution: Since `ingredients` is its own package, you must use a different naming convention for the target `cheese` in the form of `//package:target`. Read more about referencing targets, or [Labels](https://bazel.build/concepts/labels). - -To see what targets depend on a particular target/package, you can use `rdeps(universe_scope, target)`. The `rdeps()` query function takes in at least two arguments: a `universe_scope` — the relevant directory — and a `target`. Bazel searches for the target’s reverse dependencies within the `universe_scope` provided. The `rdeps()` operator accepts an optional third argument: an integer literal specifying the upper bound on the depth of the search. - -Tip: To search within the whole scope of the project, set the `universe_scope` to `//...` - -To look for reverse dependencies of the target `cheese` within the scope of the entire project ‘//…’ run the command: - -```posix-terminal -bazel query "rdeps(universe_scope, target)" -``` - -``` -ex) bazel query "rdeps(//... , //src/main/java/com/example/ingredients:cheese)" -``` - -```bash -//:runner -//src/main/java/com/example/dishes:macAndCheese -//src/main/java/com/example/dishes:pizza -//src/main/java/com/example/ingredients:cheese -//src/main/java/com/example/restaurant:cafe -//src/main/java/com/example/restaurant:chef -``` - -The query return shows that cheese is relied on by both pizza and macAndCheese. What a surprise! - -## Finding targets based on tags - -Two customers walk into Bazel Cafe: Amir and Jenny. There is nothing known about them except for their names. Luckily, they have their orders tagged in the 'customers' `BUILD` file. How can you access this tag? - -Developers can tag Bazel targets with different identifiers, often for testing purposes. For instance, tags on tests can annotate a test's role in your debug and release process, especially for C++ and Python tests, which lack any runtime annotation ability. Using tags and size elements gives flexibility in assembling suites of tests based around a codebase’s check-in policy. - -In this example, the tags are either one of `pizza` or `macAndCheese` to represent the menu items. This command queries for targets that have tags matching your identifier within a certain package. - -``` -bazel query 'attr(tags, "pizza", //src/main/java/com/example/customers/...)' -``` - -This query returns all of the targets in the 'customers' package that have a tag of "pizza". - -### Test yourself - -Use this query to learn what Jenny wants to order. - -#### Answer - -Mac and Cheese - -## Adding a new dependency - -Cafe Bazel has expanded its menu — customers can now order a Smoothie! This specific smoothie consists of the ingredients `Strawberry` and `Banana`. - -First, add the ingredients that the smoothie depends on: `Strawberry.java` and `Banana.java`. Add the empty Java classes. - -**`src/main/java/com/example/ingredients/Strawberry.java`** - -```java -package com.example.ingredients; - -public class Strawberry { - -} -``` - -**`src/main/java/com/example/ingredients/Banana.java`** - -```java -package com.example.ingredients; - -public class Banana { - -} -``` - -Next, add `Smoothie.java` to the appropriate directory: `dishes`. - -**`src/main/java/com/example/dishes/Smoothie.java`** - -```java -package com.example.dishes; - -public class Smoothie { - public static final String DISH_NAME = "Smoothie"; - public static final String DESCRIPTION = "Yummy and Refreshing"; -} -``` - -Lastly, add these files as rules in the appropriate `BUILD` files. Create a new java library for each new ingredient, including its name, public visibility, and its newly created 'src' file. You should wind up with this updated `BUILD` file: - -**`src/main/java/com/example/ingredients/BUILD`** - -``` -java_library( - name = "cheese", - visibility = ["//visibility:public"], - srcs = ["Cheese.java"], -) - -java_library( - name = "dough", - visibility = ["//visibility:public"], - srcs = ["Dough.java"], -) - -java_library( - name = "macaroni", - visibility = ["//visibility:public"], - srcs = ["Macaroni.java"], -) - -java_library( - name = "tomato", - visibility = ["//visibility:public"], - srcs = ["Tomato.java"], -) - -java_library( - name = "strawberry", - visibility = ["//visibility:public"], - srcs = ["Strawberry.java"], -) - -java_library( - name = "banana", - visibility = ["//visibility:public"], - srcs = ["Banana.java"], -) -``` - -In the `BUILD` file for dishes, you want to add a new rule for `Smoothie`. Doing so includes the Java file created for `Smoothie` as a 'src' file, and the new rules you made for each ingredient of the smoothie. - -**`src/main/java/com/example/dishes/BUILD`** - -``` -java_library( - name = "macAndCheese", - visibility = ["//visibility:public"], - srcs = ["MacAndCheese.java"], - deps = [ - "//src/main/java/com/example/ingredients:cheese", - "//src/main/java/com/example/ingredients:macaroni", - ], -) - -java_library( - name = "pizza", - visibility = ["//visibility:public"], - srcs = ["Pizza.java"], - deps = [ - "//src/main/java/com/example/ingredients:cheese", - "//src/main/java/com/example/ingredients:dough", - "//src/main/java/com/example/ingredients:tomato", - ], -) - -java_library( - name = "smoothie", - visibility = ["//visibility:public"], - srcs = ["Smoothie.java"], - deps = [ - "//src/main/java/com/example/ingredients:strawberry", - "//src/main/java/com/example/ingredients:banana", - ], -) -``` - -Lastly, you want to include the smoothie as a dependency in the Chef’s `BUILD` file. - -**`src/main/java/com/example/restaurant/BUILD`** - -``` -java_library( - name = "chef", - visibility = ["//visibility:public"], - srcs = [ - "Chef.java", - ], - - deps = [ - "//src/main/java/com/example/dishes:macAndCheese", - "//src/main/java/com/example/dishes:pizza", - "//src/main/java/com/example/dishes:smoothie", - ], -) - -java_library( - name = "cafe", - visibility = ["//visibility:public"], - srcs = [ - "Cafe.java", - ], - deps = [ - ":chef", - ], -) -``` - -Build `cafe` again to confirm that there are no errors. If it builds successfully, congratulations! You’ve added a new dependency for the 'Cafe'. If not, look out for spelling mistakes and package naming. For more information about writing `BUILD` files see [BUILD Style Guide](https://bazel.build/build/style-guide). - -Now, visualize the new dependency graph with the addition of the `Smoothie` to compare with the previous one. For clarity, name the graph input as `graph2.in` and `graph2.png`. - -```posix-terminal -bazel query --noimplicit_deps 'deps(:runner)' --output graph > graph2.in -``` - -```posix-terminal -dot -Tpng < graph2.in > graph2.png -``` - -[![The same graph as the first one except now there is a spoke stemming from the chef target with smoothie which leads to banana and strawberry](images/query_graph2.png "Updated dependency graph")](images/query_graph2.png) - -Looking at `graph2.png`, you can see that `Smoothie` has no shared dependencies with other dishes but is just another target that the `Chef` relies on. - -## somepath() and allpaths() - -What if you want to query why one package depends on another package? Displaying a dependency path between the two provides the answer. - -Two functions can help you find dependency paths: `somepath()` and `allpaths()`. Given a starting target S and an end point E, find a path between S and E by using `somepath(S,E)`. - -Explore the differences between these two functions by looking at the relationships between the 'Chef' and 'Cheese' targets. There are different possible paths to get from one target to the other: - -- Chef → MacAndCheese → Cheese -- Chef → Pizza → Cheese - -`somepath()` gives you a single path out of the two options, whereas 'allpaths()' outputs every possible path. - -Using Cafe Bazel as an example, run the following: - -```posix-terminal -bazel query "somepath(//src/main/java/com/example/restaurant/..., //src/main/java/com/example/ingredients:cheese)" -``` - -```bash -//src/main/java/com/example/restaurant:cafe -//src/main/java/com/example/restaurant:chef -//src/main/java/com/example/dishes:macAndCheese -//src/main/java/com/example/ingredients:cheese -``` - -The output follows the first path of Cafe → Chef → MacAndCheese → Cheese. If instead you use `allpaths()`, you get: - -```posix-terminal -bazel query "allpaths(//src/main/java/com/example/restaurant/..., //src/main/java/com/example/ingredients:cheese)" -``` - -```bash -//src/main/java/com/example/dishes:macAndCheese -//src/main/java/com/example/dishes:pizza -//src/main/java/com/example/ingredients:cheese -//src/main/java/com/example/restaurant:cafe -//src/main/java/com/example/restaurant:chef -``` - -![Output path of cafe to chef to pizza,mac and cheese to cheese](images/query_graph3.png "Output path for dependency") - -The output of `allpaths()` is a little harder to read as it is a flattened list of the dependencies. Visualizing this graph using Graphviz makes the relationship clearer to understand. - -## Test yourself - -One of Cafe Bazel’s customers gave the restaurant's first review! Unfortunately, the review is missing some details such as the identity of the reviewer and what dish it’s referencing. Luckily, you can access this information with Bazel. The `reviews` package contains a program that prints a review from a mystery customer. Build and run it with: - -```posix-terminal -bazel build //src/main/java/com/example/reviews:review -``` - -```posix-terminal -bazel-bin/src/main/java/com/example/reviews/review -``` - -Going off Bazel queries only, try to find out who wrote the review, and what dish they were describing. - -#### Hint - -Check the tags and dependencies for useful information. - -#### Answer - -This review was describing the Pizza and Amir was the reviewer. If you look at what dependencies that this rule had using `bazel query --noimplicit\_deps 'deps(//src/main/java/com/example/reviews:review)'` The result of this command reveals that Amir is the reviewer! Next, since you know the reviewer is Amir, you can use the query function to seek which tag Amir has in the \`BUILD\` file to see what dish is there. The command `bazel query 'attr(tags, "pizza", //src/main/java/com/example/customers/...)'` output that Amir is the only customer that ordered a pizza and is the reviewer which gives us the answer. - -## Wrapping up - -Congratulations! You have now run several basic queries, which you can try out on own projects. To learn more about the query language syntax, refer to the [Query reference page](https://bazel.build/query/language). Want more advanced queries? The [Query guide](https://bazel.build/query/guide) showcases an in-depth list of more use cases than are covered in this guide. diff --git a/reference/flag-cheatsheet.mdx b/reference/flag-cheatsheet.mdx deleted file mode 100644 index 5caf5628..00000000 --- a/reference/flag-cheatsheet.mdx +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: 'Bazel flag cheat sheet' ---- - -Navigating Bazel's extensive list of command line flags can be a challenge. This page focuses on the most crucial flags you'll need to know. - -**Tip:** Select the flag name in table to navigate to its entry in the command line reference. - -## Useful general options - -The following flags are meant to be set explicitly on the command line. - -| Flag | Description | -| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ### `--config` | You can organize flags in a **.bazelrc** file into configurations, like ones for debugging or release builds. Additional configuration groups can be selected with `--config=<group>`. | -| ### `--keep_going` | Bazel should try as much as possible to continue with build and test execution. By default, Bazel fails eagerly.``` -</td> -``` | -| ### `--remote_download_outputs` | When using remote execution or caching (both disk and remote), you can signal to Bazel that you want to download **all** (intermediate) build artifacts as follows:``` ---remote_download_outputs=all -```By default, Bazel only downloads top-level artifacts, such as the final binary, and intermediate artifacts that are necessary for local actions. | -| ### `--stamp` | Adds build info (user, timestamp) to binaries.**Note:** Because this increases build time, it's only intended for release builds. | - -## Uncover Build & Test Issues - -The following flags can help you better understand Bazel build or test errors. - -| Flag | Description | -| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| ### `--announce_rc` | Shows which flags are implicitly set through user-defined, machine-defined, or project-defined **.bazelrc** files. | -| ### `--auto_output_filter` | By default, Bazel tries to prevent log spam and does only print compiler warnings and Starlark debug output for packages and subpackages requested on the command line. To disable all filtering, set `--auto_output_filter=none`. | -| ### `--sandbox_debug` | Lets you drill into sandboxing errors. For details on why Bazel sandboxes builds by default and what gets sandboxed, see our [sandboxing documentation](https://bazel.build/docs/sandboxing).**Tip:** If you think the error might be caused by sandboxing, try turning sandboxing off temporarily.To do this, add `--spawn_strategy=local` to your command. | -| ### `--subcommands (-s)` | Displays a comprehensive list of every command that Bazel runs during a build, regardless of whether it succeeds or fails | - -## Startup - -Caution: Startup flags need to be passed before the command and cause a server restart. Toggle these flags with caution. - -| Flag | Description | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ### `--bazelrc` | You can specify default Bazel options in **.bazelrc** files. If multiple **.bazelrc** files exist, you can select which **.bazelrc** file is used by adding `--bazelrc=<path to the .bazelrc file>`.**Tip:** `--bazelrc=dev/null` disables the search for **.bazelrc** files.This is ideal for scenarios where you want to ensure a clean build environment, such as release builds, and prevent any unintended configuration changes from **.bazelrc** files | -| ### `--host_jvm_args` | Limits the amount of RAM the Bazel server uses.For example, the following limits the Bazel heap size to **3**GB:``` ---host_jvm_args=-Xmx3g -```**Note:** `-Xmx` is used to set the maximum heap size for the Java Virtual Machine (JVM). The heap is the area of memory where objects are allocated. The correct format for this option is `-Xmx<size>` , where `<size>` is the maximum heap size, specified with a unit such as:* m for megabytes -* g for gigabytes -* k for kilobytes | -| ### `--output_base` | Controls Bazel's output tree. Bazel doesn't store build outputs, including logs, within the source tree itself. Instead, it uses a distinct output tree for this purpose.**Tip:** Using multiple output bases in one Bazel workspace lets you run multiple Bazel servers concurrently. This can be useful when trying to avoid analysis thrashing. For more information, see [Choosing the output base](https://bazel.build/run/scripts#output-base-option). | - -## Bazel tests - -The following flags are related to Bazel test - -| Flag | Description | -| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| ### `--java_debug` | Causes Java tests to wait for a debugger connection before being executed. | -| ### `--runs_per_test` | The number of times to run tests. For example, to run tests N times, add `--runs_per_test=N`. This can be useful to debug flaky tests and see whether a fix causes a test to pass consistently. | -| ### `--test_filter` | This flag is particularly useful when iterating on a single test method, such as when a change you made breaks a test. Instead of re-running all the test methods in the test suite, you can focus solely on the specific test(s) that failed. This allows for faster feedback and more efficient debugging. This flag is often used in conjunction with `--test_output=streamed` for real-time test output. | -| ### `--test_output` | Specifies the output mode. By default, Bazel captures test output in local log files. When iterating on a broken test, you typically want to use `--test_output=streamed` to see the test output in real time. | - -## Bazel run - -The following flags are related to Bazel run. - -| Flag | Description | -| ----------------- | ---------------------------------------------------------------------------------------------------------- | -| ### `--run_under` | Changes how executables are invoked. For example `--run_under="strace -c"` is commonly used for debugging. | - -## User-specific bazelrc options - -The following flags are related to user-specific **.bazelrc** options. - -| Flag | Description | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ### `--disk_cache` | A path to a directory where Bazel can read and write actions and action outputs. If the directory doesn't exist, it will be created.You can share build artifacts between multiple branches or workspaces and speed up Bazel builds by adding `--disk_cache=<path>` to your command. | -| ### `--jobs` | The number of concurrent jobs to run.This is typically only required when using remote execution where a remote build cluster executes more jobs than you have cores locally. | -| ### `--local_resources` | Limits how much CPU or RAM is consumed by locally running actions.**Note:** This has no impact on the amount of CPU or RAM that the Bazel server itself consumes for tasks like analysis and build orchestration. | -| ### `--sandbox_base` | Lets the sandbox create its sandbox directories underneath this path. By default, Bazel executes local actions sandboxed which adds some overhead to the build.**Tip:** Specify a path on tmpfs, for example `/run/shm`, to possibly improve performance a lot when your build or tests have many input files. | - -## Project-specific bazelrc options - -The following flags are related to project-specific **.bazelrc** options. - -| Flag | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| ### `--flaky_test_attempts` | Retry each test up to the specified number of times in case of any test failure. This is especially useful on Continuous Integration. Tests that require more than one attempt to pass are marked as **FLAKY** in the test summary. | -| ### `--remote_cache` | A URI of a caching endpoint. Setting up remote caching can be a great way to speed up Bazel builds. It can be combined with a local disk cache. | -| ### `--remote_download_regex` | Force remote build outputs whose path matches this pattern to be downloaded, irrespective of the `--remote_download_outputs` setting. Multiple patterns may be specified by repeating this flag. | -| ### `--remote_executor` | `HOST` or `HOST:PORT` of a remote execution endpoint. Pass this if you are using a remote execution service. You'll often need to Add `--remote_instance_name=<name>`. | -| ### `--remote_instance_name` | The value to pass as `instance_name` in the remote execution API. | -| ### `--show_timestamps` | If specified, a timestamp is added to each message generated by Bazel specifying the time at which the message was displayed. This is useful on CI systems to quickly understand what step took how long. | -| ### `--spawn_strategy` | Even with remote execution, running some build actions locally might be faster. This depends on factors like your build cluster's capacity, network speed, and network delays.**Tip:** To run actions both locally and remotely and accept the faster result add `--spawn_strategy=dynamic` to your build command. | diff --git a/reference/glossary.mdx b/reference/glossary.mdx index f045f9e6..da7478c3 100644 --- a/reference/glossary.mdx +++ b/reference/glossary.mdx @@ -2,155 +2,289 @@ title: 'Bazel Glossary' --- + + ### Action -A command to run during the build, for example, a call to a compiler that takes [artifacts](#artifact) as inputs and produces other artifacts as outputs. Includes metadata like the command line arguments, action key, environment variables, and declared input/output artifacts. +A command to run during the build, for example, a call to a compiler that takes +[artifacts](#artifact) as inputs and produces other artifacts as outputs. +Includes metadata like the command line arguments, action key, environment +variables, and declared input/output artifacts. **See also:** [Rules documentation](/extending/rules#actions) ### Action cache -An on-disk cache that stores a mapping of executed [actions](#action) to the outputs they created. The cache key is known as the [action key](#action-key). A core component for Bazel's incrementality model. The cache is stored in the output base directory and thus survives Bazel server restarts. +An on-disk cache that stores a mapping of executed [actions](#action) to the +outputs they created. The cache key is known as the [action key](#action-key). A +core component for Bazel's incrementality model. The cache is stored in the +output base directory and thus survives Bazel server restarts. ### Action graph -An in-memory graph of [actions](#action) and the [artifacts](#artifact) that these actions read and generate. The graph might include artifacts that exist as source files (for example, in the file system) as well as generated intermediate/final artifacts that are not mentioned in `BUILD` files. Produced during the [analysis phase](#analysis-phase) and used during the [execution phase](#execution-phase). +An in-memory graph of [actions](#action) and the [artifacts](#artifact) that +these actions read and generate. The graph might include artifacts that exist as +source files (for example, in the file system) as well as generated +intermediate/final artifacts that are not mentioned in `BUILD` files. Produced +during the [analysis phase](#analysis-phase) and used during the [execution +phase](#execution-phase). ### Action graph query (aquery) -A [query](#query-concept) tool that can query over build [actions](#action). This provides the ability to analyze how [build rules](#rule) translate into the actual work builds do. +A [query](#query-concept) tool that can query over build [actions](#action). +This provides the ability to analyze how [build rules](#rule) translate into the +actual work builds do. ### Action key -The cache key of an [action](#action). Computed based on action metadata, which might include the command to be executed in the action, compiler flags, library locations, or system headers, depending on the action. Enables Bazel to cache or invalidate individual actions deterministically. +The cache key of an [action](#action). Computed based on action metadata, which +might include the command to be executed in the action, compiler flags, library +locations, or system headers, depending on the action. Enables Bazel to cache or +invalidate individual actions deterministically. ### Analysis phase -The second phase of a build. Processes the [target graph](#target-graph) specified in [`BUILD` files](#build-file) to produce an in-memory [action graph](#action-graph) that determines the order of actions to run during the [execution phase](#execution-phase). This is the phase in which rule implementations are evaluated. +The second phase of a build. Processes the [target graph](#target-graph) +specified in [`BUILD` files](#build-file) to produce an in-memory [action +graph](#action-graph) that determines the order of actions to run during the +[execution phase](#execution-phase). This is the phase in which rule +implementations are evaluated. ### Artifact -A source file or a generated file. Can also be a directory of files, known as [tree artifacts](#tree-artifact). +A source file or a generated file. Can also be a directory of files, known as +[tree artifacts](#tree-artifact). -An artifact may be an input to multiple actions, but must only be generated by at most one action. +An artifact may be an input to multiple actions, but must only be generated by +at most one action. -An artifact that corresponds to a [file target](#target) can be addressed by a label. +An artifact that corresponds to a [file target](#target) can be addressed by a +label. ### Aspect -A mechanism for rules to create additional [actions](#action) in their dependencies. For example, if target A depends on B, one can apply an aspect on A that traverses *up* a dependency edge to B, and runs additional actions in B to generate and collect additional output files. These additional actions are cached and reused between targets requiring the same aspect. Created with the `aspect()` Starlark Build API function. Can be used, for example, to generate metadata for IDEs, and create actions for linting. +A mechanism for rules to create additional [actions](#action) in their +dependencies. For example, if target A depends on B, one can apply an aspect on +A that traverses *up* a dependency edge to B, and runs additional actions in B +to generate and collect additional output files. These additional actions are +cached and reused between targets requiring the same aspect. Created with the +`aspect()` Starlark Build API function. Can be used, for example, to generate +metadata for IDEs, and create actions for linting. **See also:** [Aspects documentation](/extending/aspects) ### Aspect-on-aspect -A composition mechanism whereby aspects can be applied to the results of other aspects. For example, an aspect that generates information for use by IDEs can be applied on top of an aspect that generates `.java` files from a proto. +A composition mechanism whereby aspects can be applied to the results +of other aspects. For example, an aspect that generates information for use by +IDEs can be applied on top of an aspect that generates `.java` files from a +proto. -For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that `B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) attribute. +For an aspect `A` to apply on top of aspect `B`, the [providers](#provider) that +`B` advertises in its [`provides`](/rules/lib/globals#aspect.provides) attribute +must match what `A` declares it wants in its [`required_aspect_providers`](/rules/lib/globals#aspect.required_aspect_providers) +attribute. ### Attribute -A parameter to a [rule](#rule), used to express per-target build information. Examples include `srcs`, `deps`, and `copts`, which respectively declare a target's source files, dependencies, and custom compiler options. The particular attributes available for a given target depend on its rule type. +A parameter to a [rule](#rule), used to express per-target build information. +Examples include `srcs`, `deps`, and `copts`, which respectively declare a +target's source files, dependencies, and custom compiler options. The particular +attributes available for a given target depend on its rule type. ### .bazelrc -Bazel’s configuration file used to change the default values for [startup flags](#startup-flags) and [command flags](#command-flags), and to define common groups of options that can then be set together on the Bazel command line using a `--config` flag. Bazel can combine settings from multiple bazelrc files (systemwide, per-workspace, per-user, or from a custom location), and a `bazelrc` file may also import settings from other `bazelrc` files. +Bazel’s configuration file used to change the default values for [startup +flags](#startup-flags) and [command flags](#command-flags), and to define common +groups of options that can then be set together on the Bazel command line using +a `--config` flag. Bazel can combine settings from multiple bazelrc files +(systemwide, per-workspace, per-user, or from a custom location), and a +`bazelrc` file may also import settings from other `bazelrc` files. ### Blaze -The Google-internal version of Bazel. Google’s main build system for its mono-repository. +The Google-internal version of Bazel. Google’s main build system for its +mono-repository. ### BUILD File -A `BUILD` file is the main configuration file that tells Bazel what software outputs to build, what their dependencies are, and how to build them. Bazel takes a `BUILD` file as input and uses the file to create a graph of dependencies and to derive the actions that must be completed to build intermediate and final software outputs. A `BUILD` file marks a directory and any sub-directories not containing a `BUILD` file as a [package](#package), and can contain [targets](#target) created by [rules](#rule). The file can also be named `BUILD.bazel`. +A `BUILD` file is the main configuration file that tells Bazel what software +outputs to build, what their dependencies are, and how to build them. Bazel +takes a `BUILD` file as input and uses the file to create a graph of dependencies +and to derive the actions that must be completed to build intermediate and final +software outputs. A `BUILD` file marks a directory and any sub-directories not +containing a `BUILD` file as a [package](#package), and can contain +[targets](#target) created by [rules](#rule). The file can also be named +`BUILD.bazel`. ### BUILD.bazel File -See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same directory. +See [`BUILD` File](#build-file). Takes precedence over a `BUILD` file in the same +directory. ### .bzl File -A file that defines rules, [macros](#macro), and constants written in [Starlark](#starlark). These can then be imported into [`BUILD` files](#build-file) using the `load()` function. +A file that defines rules, [macros](#macro), and constants written in +[Starlark](#starlark). These can then be imported into [`BUILD` +files](#build-file) using the `load()` function. + +{/* TODO: ### Build event protocol */} + +{/* TODO: ### Build flag */} ### Build graph -The dependency graph that Bazel constructs and traverses to perform a build. Includes nodes like [targets](#target), [configured targets](#configured-target), [actions](#action), and [artifacts](#artifact). A build is considered complete when all [artifacts](#artifact) on which a set of requested targets depend are verified as up-to-date. +The dependency graph that Bazel constructs and traverses to perform a build. +Includes nodes like [targets](#target), [configured +targets](#configured-target), [actions](#action), and [artifacts](#artifact). A +build is considered complete when all [artifacts](#artifact) on which a set of +requested targets depend are verified as up-to-date. ### Build setting -A Starlark-defined piece of [configuration](#configuration). [Transitions](#transition) can set build settings to change a subgraph's configuration. If exposed to the user as a [command-line flag](#command-flags), also known as a build flag. +A Starlark-defined piece of [configuration](#configuration). +[Transitions](#transition) can set build settings to change a subgraph's +configuration. If exposed to the user as a [command-line flag](#command-flags), +also known as a build flag. ### Clean build -A build that doesn't use the results of earlier builds. This is generally slower than an [incremental build](#incremental-build) but commonly considered to be more [correct](#correctness). Bazel guarantees both clean and incremental builds are always correct. +A build that doesn't use the results of earlier builds. This is generally slower +than an [incremental build](#incremental-build) but commonly considered to be +more [correct](#correctness). Bazel guarantees both clean and incremental builds +are always correct. ### Client-server model -The `bazel` command-line client automatically starts a background server on the local machine to execute Bazel [commands](#command). The server persists across commands but automatically stops after a period of inactivity (or explicitly via bazel shutdown). Splitting Bazel into a server and client helps amortize JVM startup time and supports faster [incremental builds](#incremental-build) because the [action graph](#action-graph) remains in memory across commands. +The `bazel` command-line client automatically starts a background server on the +local machine to execute Bazel [commands](#command). The server persists across +commands but automatically stops after a period of inactivity (or explicitly via +bazel shutdown). Splitting Bazel into a server and client helps amortize JVM +startup time and supports faster [incremental builds](#incremental-build) +because the [action graph](#action-graph) remains in memory across commands. ### Command -Used on the command line to invoke different Bazel functions, like `bazel build`, `bazel test`, `bazel run`, and `bazel query`. +Used on the command line to invoke different Bazel functions, like `bazel +build`, `bazel test`, `bazel run`, and `bazel query`. ### Command flags -A set of flags specific to a [command](#command). Command flags are specified *after* the command (`bazel build <command flags>`). Flags can be applicable to one or more commands. For example, `--configure` is a flag exclusively for the `bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, `test` and more. Flags are often used for [configuration](#configuration) purposes, so changes in flag values can cause Bazel to invalidate in-memory graphs and restart the [analysis phase](#analysis-phase). +A set of flags specific to a [command](#command). Command flags are specified +*after* the command (`bazel build `). Flags can be applicable to +one or more commands. For example, `--configure` is a flag exclusively for the +`bazel sync` command, but `--keep_going` is applicable to `sync`, `build`, +`test` and more. Flags are often used for [configuration](#configuration) +purposes, so changes in flag values can cause Bazel to invalidate in-memory +graphs and restart the [analysis phase](#analysis-phase). ### Configuration -Information outside of [rule](#rule) definitions that impacts how rules generate [actions](#action). Every build has at least one configuration specifying the target platform, action environment variables, and command-line [build flags](#command-flags). [Transitions](#transition) may create additional configurations, such as for host tools or cross-compilation. +Information outside of [rule](#rule) definitions that impacts how rules generate +[actions](#action). Every build has at least one configuration specifying the +target platform, action environment variables, and command-line [build +flags](#command-flags). [Transitions](#transition) may create additional +configurations, such as for host tools or cross-compilation. **See also:** [Configurations](/extending/rules#configurations) +{/* TODO: ### Configuration fragment */} + ### Configuration trimming -The process of only including the pieces of [configuration](#configuration) a target actually needs. For example, if you build Java binary `//:j` with C++ dependency `//:c`, it's wasteful to include the value of `--javacopt` in the configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ build cacheability. +The process of only including the pieces of [configuration](#configuration) a +target actually needs. For example, if you build Java binary `//:j` with C++ +dependency `//:c`, it's wasteful to include the value of `--javacopt` in the +configuration of `//:c` because changing `--javacopt` unnecessarily breaks C++ +build cacheability. ### Configured query (cquery) -A [query](#query-concept) tool that queries over [configured targets](#configured-target) (after the [analysis phase](#analysis-phase) completes). This means `select()` and [build flags](#command-flags) (such as `--platforms`) are accurately reflected in the results. +A [query](#query-concept) tool that queries over [configured +targets](#configured-target) (after the [analysis phase](#analysis-phase) +completes). This means `select()` and [build flags](#command-flags) (such as +`--platforms`) are accurately reflected in the results. **See also:** [cquery documentation](/query/cquery) ### Configured target -The result of evaluating a [target](#target) with a [configuration](#configuration). The [analysis phase](#analysis-phase) produces this by combining the build's options with the targets that need to be built. For example, if `//:foo` builds for two different architectures in the same build, it has two configured targets: `<//:foo, x86>` and `<//:foo, arm>`. +The result of evaluating a [target](#target) with a +[configuration](#configuration). The [analysis phase](#analysis-phase) produces +this by combining the build's options with the targets that need to be built. +For example, if `//:foo` builds for two different architectures in the same +build, it has two configured targets: `` and ``. ### Correctness -A build is correct when its output faithfully reflects the state of its transitive inputs. To achieve correct builds, Bazel strives to be [hermetic](#hermeticity), reproducible, and making [build analysis](#analysis-phase) and [action execution](#execution-phase) deterministic. +A build is correct when its output faithfully reflects the state of its +transitive inputs. To achieve correct builds, Bazel strives to be +[hermetic](#hermeticity), reproducible, and making [build +analysis](#analysis-phase) and [action execution](#execution-phase) +deterministic. ### Dependency -A directed edge between two [targets](#target). A target `//:foo` has a *target dependency* on target `//:bar` if `//:foo`'s attribute values contain a reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an action in `//:foo` depends on an input [artifact](#artifact) created by an action in `//:bar`. +A directed edge between two [targets](#target). A target `//:foo` has a *target +dependency* on target `//:bar` if `//:foo`'s attribute values contain a +reference to `//:bar`. `//:foo` has an *action dependency* on `//:bar` if an +action in `//:foo` depends on an input [artifact](#artifact) created by an +action in `//:bar`. -In certain contexts, it could also refer to an *external dependency*; see [modules](#module). +In certain contexts, it could also refer to an _external dependency_; see +[modules](#module). ### Depset -A data structure for collecting data on transitive dependencies. Optimized so that merging depsets is time and space efficient, because it’s common to have very large depsets (hundreds of thousands of files). Implemented to recursively refer to other depsets for space efficiency reasons. [Rule](#rule) implementations should not "flatten" depsets by converting them to lists unless the rule is at the top level of the build graph. Flattening large depsets incurs huge memory consumption. Also known as *nested sets* in Bazel's internal implementation. +A data structure for collecting data on transitive dependencies. Optimized so +that merging depsets is time and space efficient, because it’s common to have +very large depsets (hundreds of thousands of files). Implemented to +recursively refer to other depsets for space efficiency reasons. [Rule](#rule) +implementations should not "flatten" depsets by converting them to lists unless +the rule is at the top level of the build graph. Flattening large depsets incurs +huge memory consumption. Also known as *nested sets* in Bazel's internal +implementation. **See also:** [Depset documentation](/extending/depsets) ### Disk cache -A local on-disk blob store for the remote caching feature. Can be used in conjunction with an actual remote blob store. +A local on-disk blob store for the remote caching feature. Can be used in +conjunction with an actual remote blob store. ### Distdir -A read-only directory containing files that Bazel would otherwise fetch from the internet using repository rules. Enables builds to run fully offline. +A read-only directory containing files that Bazel would otherwise fetch from the +internet using repository rules. Enables builds to run fully offline. ### Dynamic execution -An execution strategy that selects between local and remote execution based on various heuristics, and uses the execution results of the faster successful method. Certain [actions](#action) are executed faster locally (for example, linking) and others are faster remotely (for example, highly parallelizable compilation). A dynamic execution strategy can provide the best possible incremental and clean build times. +An execution strategy that selects between local and remote execution based on +various heuristics, and uses the execution results of the faster successful +method. Certain [actions](#action) are executed faster locally (for example, +linking) and others are faster remotely (for example, highly parallelizable +compilation). A dynamic execution strategy can provide the best possible +incremental and clean build times. ### Execution phase -The third phase of a build. Executes the [actions](#action) in the [action graph](#action-graph) created during the [analysis phase](#analysis-phase). These actions invoke executables (compilers, scripts) to read and write [artifacts](#artifact). *Spawn strategies* control how these actions are executed: locally, remotely, dynamically, sandboxed, docker, and so on. +The third phase of a build. Executes the [actions](#action) in the [action +graph](#action-graph) created during the [analysis phase](#analysis-phase). +These actions invoke executables (compilers, scripts) to read and write +[artifacts](#artifact). *Spawn strategies* control how these actions are +executed: locally, remotely, dynamically, sandboxed, docker, and so on. ### Execution root -A directory in the [workspace](#workspace)’s [output base](#output-base) directory where local [actions](#action) are executed in non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks of input [artifacts](#artifact) from the workspace. The execution root also contains symlinks to external repositories as other inputs and the `bazel-out` directory to store outputs. Prepared during the [loading phase](#loading-phase) by creating a *symlink forest* of the directories that represent the transitive closure of packages on which a build depends. Accessible with `bazel info execution_root` on the command line. +A directory in the [workspace](#workspace)’s [output base](#output-base) +directory where local [actions](#action) are executed in +non-[sandboxed](#sandboxing) builds. The directory contents are mostly symlinks +of input [artifacts](#artifact) from the workspace. The execution root also +contains symlinks to external repositories as other inputs and the `bazel-out` +directory to store outputs. Prepared during the [loading phase](#loading-phase) +by creating a *symlink forest* of the directories that represent the transitive +closure of packages on which a build depends. Accessible with `bazel info +execution_root` on the command line. ### File @@ -158,27 +292,52 @@ See [Artifact](#artifact). ### Hermeticity -A build is hermetic if there are no external influences on its build and test operations, which helps to make sure that results are deterministic and [correct](#correctness). For example, hermetic builds typically disallow network access to actions, restrict access to declared inputs, use fixed timestamps and timezones, restrict access to environment variables, and use fixed seeds for random number generators +A build is hermetic if there are no external influences on its build and test +operations, which helps to make sure that results are deterministic and +[correct](#correctness). For example, hermetic builds typically disallow network +access to actions, restrict access to declared inputs, use fixed timestamps and +timezones, restrict access to environment variables, and use fixed seeds for +random number generators ### Incremental build -An incremental build reuses the results of earlier builds to reduce build time and resource usage. Dependency checking and caching aim to produce correct results for this type of build. An incremental build is the opposite of a clean build. +An incremental build reuses the results of earlier builds to reduce build time +and resource usage. Dependency checking and caching aim to produce correct +results for this type of build. An incremental build is the opposite of a clean +build. + +{/* TODO: ### Install base */} ### Label -An identifier for a [target](#target). Generally has the form `@repo//path/to/package:target`, where `repo` is the (apparent) name of the [repository](#repository) containing the target, `path/to/package` is the path to the directory that contains the [`BUILD` file](#build-file) declaring the target (this directory is also known as the [package](#package)), and `target` is the name of the target itself. Depending on the situation, parts of this syntax may be omitted. +An identifier for a [target](#target). Generally has the form +`@repo//path/to/package:target`, where `repo` is the (apparent) name of the +[repository](#repository) containing the target, `path/to/package` is the path +to the directory that contains the [`BUILD` file](#build-file) declaring the +target (this directory is also known as the [package](#package)), and `target` +is the name of the target itself. Depending on the situation, parts of this +syntax may be omitted. **See also**: [Labels](/concepts/labels) ### Loading phase -The first phase of a build where Bazel executes [`BUILD` files](#build-file) to create [packages](#package). [Macros](#macro) and certain functions like `glob()` are evaluated in this phase. Interleaved with the second phase of the build, the [analysis phase](#analysis-phase), to build up a [target graph](#target-graph). +The first phase of a build where Bazel executes [`BUILD` files](#build-file) to +create [packages](#package). [Macros](#macro) and certain functions like +`glob()` are evaluated in this phase. Interleaved with the second phase of the +build, the [analysis phase](#analysis-phase), to build up a [target +graph](#target-graph). ### Legacy macro -A flavor of [macro](#macro) which is declared as an ordinary [Starlark](#starlark) function, and which runs as a side effect of executing a `BUILD` file. +A flavor of [macro](#macro) which is declared as an ordinary +[Starlark](#starlark) function, and which runs as a side effect of executing a +`BUILD` file. -Legacy macros can do anything a function can. This means they can be convenient, but they can also be harder to read, write, and use. A legacy macro might unexpectedly mutate its arguments or fail when given a `select()` or ill-typed argument. +Legacy macros can do anything a function can. This means they can be convenient, +but they can also be harder to read, write, and use. A legacy macro might +unexpectedly mutate its arguments or fail when given a `select()` or ill-typed +argument. Contrast with [symbolic macros](#symbolic-macro). @@ -186,19 +345,34 @@ Contrast with [symbolic macros](#symbolic-macro). ### Macro -A mechanism to compose multiple [rule](#rule) target declarations together under a single [Starlark](#starlark) callable. Enables reusing common rule declaration patterns across `BUILD` files. Expanded to the underlying rule target declarations during the [loading phase](#loading-phase). +A mechanism to compose multiple [rule](#rule) target declarations together under +a single [Starlark](#starlark) callable. Enables reusing common rule declaration +patterns across `BUILD` files. Expanded to the underlying rule target +declarations during the [loading phase](#loading-phase). -Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and [legacy macros](#legacy-macro). +Comes in two flavors: [symbolic macros](#symbolic-macro) (since Bazel 8) and +[legacy macros](#legacy-macro). ### Mnemonic -A short, human-readable string selected by a rule author to quickly understand what an [action](#action) in the rule is doing. Mnemonics can be used as identifiers for *spawn strategy* selections. Some examples of action mnemonics are `Javac` from Java rules, `CppCompile` from C++ rules, and `AndroidManifestMerger` from Android rules. +A short, human-readable string selected by a rule author to quickly understand +what an [action](#action) in the rule is doing. Mnemonics can be used as +identifiers for *spawn strategy* selections. Some examples of action mnemonics +are `Javac` from Java rules, `CppCompile` from C++ rules, and +`AndroidManifestMerger` from Android rules. ### Module -A Bazel project that can have multiple versions, each of which can have dependencies on other modules. This is analogous to familiar concepts in other dependency management systems, such as a Maven *artifact*, an npm *package*, a Go *module*, or a Cargo *crate*. Modules form the backbone of Bazel's external dependency management system. +A Bazel project that can have multiple versions, each of which can have +dependencies on other modules. This is analogous to familiar concepts in other +dependency management systems, such as a Maven _artifact_, an npm _package_, a +Go _module_, or a Cargo _crate_. Modules form the backbone of Bazel's external +dependency management system. -Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its root. This file contains metadata about the module itself (such as its name and version), its direct dependencies, and various other data including toolchain registrations and [module extension](#module-extension) input. +Each module is backed by a [repo](#repository) with a `MODULE.bazel` file at its +root. This file contains metadata about the module itself (such as its name and +version), its direct dependencies, and various other data including toolchain +registrations and [module extension](#module-extension) input. Module metadata is hosted in Bazel registries. @@ -206,180 +380,336 @@ Module metadata is hosted in Bazel registries. ### Module Extension -A piece of logic that can be run to generate [repos](#repository) by reading inputs from across the [module](#module) dependency graph and invoking [repo rules](#repository-rule). Module extensions have capabilities similar to repo rules, allowing them to access the internet, perform file I/O, and so on. +A piece of logic that can be run to generate [repos](#repository) by reading +inputs from across the [module](#module) dependency graph and invoking [repo +rules](#repository-rule). Module extensions have capabilities similar to repo +rules, allowing them to access the internet, perform file I/O, and so on. **See also:** [Module extensions](/external/extension) ### Native rules -[Rules](#rule) that are built into Bazel and implemented in Java. Such rules appear in [`.bzl` files](#bzl-file) as functions in the native module (for example, `native.cc_library` or `native.java_library`). User-defined rules (non-native) are created using [Starlark](#starlark). +[Rules](#rule) that are built into Bazel and implemented in Java. Such rules +appear in [`.bzl` files](#bzl-file) as functions in the native module (for +example, `native.cc_library` or `native.java_library`). User-defined rules +(non-native) are created using [Starlark](#starlark). ### Output base -A [workspace](#workspace)-specific directory to store Bazel output files. Used to separate outputs from the *workspace*'s source tree (the [main repo](#repository)). Located in the [output user root](#output-user-root). +A [workspace](#workspace)-specific directory to store Bazel output files. Used +to separate outputs from the *workspace*'s source tree (the [main +repo](#repository)). Located in the [output user root](#output-user-root). ### Output groups -A group of files that is expected to be built when Bazel finishes building a target. [Rules](#rule) put their usual outputs in the "default output group" (e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` targets). The default output group is the output group whose [artifacts](#artifact) are built when a target is requested on the command line. Rules can define more named output groups that can be explicitly specified in [`BUILD` files](#build-file) (`filegroup` rule) or the command line (`--output_groups` flag). +A group of files that is expected to be built when Bazel finishes building a +target. [Rules](#rule) put their usual outputs in the "default output group" +(e.g the `.jar` file of a `java_library`, `.a` and `.so` for `cc_library` +targets). The default output group is the output group whose +[artifacts](#artifact) are built when a target is requested on the command line. +Rules can define more named output groups that can be explicitly specified in +[`BUILD` files](#build-file) (`filegroup` rule) or the command line +(`--output_groups` flag). ### Output user root -A user-specific directory to store Bazel's outputs. The directory name is derived from the user's system username. Prevents output file collisions if multiple users are building the same project on the system at the same time. Contains subdirectories corresponding to build outputs of individual workspaces, also known as [output bases](#output-base). +A user-specific directory to store Bazel's outputs. The directory name is +derived from the user's system username. Prevents output file collisions if +multiple users are building the same project on the system at the same time. +Contains subdirectories corresponding to build outputs of individual workspaces, +also known as [output bases](#output-base). ### Package -The set of [targets](#target) defined by a [`BUILD` file](#build-file). A package's name is the `BUILD` file's path relative to the [repo](#repository) root. A package can contain subpackages, or subdirectories containing `BUILD` files, thus forming a package hierarchy. +The set of [targets](#target) defined by a [`BUILD` file](#build-file). A +package's name is the `BUILD` file's path relative to the [repo](#repository) +root. A package can contain subpackages, or subdirectories containing `BUILD` +files, thus forming a package hierarchy. ### Package group -A [target](#target) representing a set of packages. Often used in `visibility` attribute values. +A [target](#target) representing a set of packages. Often used in `visibility` +attribute values. ### Platform -A "machine type" involved in a build. This includes the machine Bazel runs on (the "host" platform), the machines build tools execute on ("exec" platforms), and the machines targets are built for ("target platforms"). +A "machine type" involved in a build. This includes the machine Bazel runs on +(the "host" platform), the machines build tools execute on ("exec" platforms), +and the machines targets are built for ("target platforms"). ### Provider -A schema describing a unit of information to pass between [rule targets](#rule-target) along dependency relationships. Typically this contains information like compiler options, transitive source or output files, and build metadata. Frequently used in conjunction with [depsets](#depset) to efficiently store accumulated transitive data. An example of a built-in provider is `DefaultInfo`. +A schema describing a unit of information to pass between +[rule targets](#rule-target) along dependency relationships. Typically this +contains information like compiler options, transitive source or output files, +and build metadata. Frequently used in conjunction with [depsets](#depset) to +efficiently store accumulated transitive data. An example of a built-in provider +is `DefaultInfo`. -Note: The object holding specific data for a given rule target is referred to as a "provider instance", although sometimes this is conflated with "provider". +Note: The object holding specific data for a given rule target is +referred to as a "provider instance", although sometimes this is conflated with +"provider". **See also:** [Provider documentation](/extending/rules#providers) ### Query (concept) -The process of analyzing a [build graph](#build-graph) to understand [target](#target) properties and dependency structures. Bazel supports three query variants: [query](#query-command), [cquery](#configured-query), and [aquery](#action-graph-query). +The process of analyzing a [build graph](#build-graph) to understand +[target](#target) properties and dependency structures. Bazel supports three +query variants: [query](#query-command), [cquery](#configured-query), and +[aquery](#action-graph-query). ### query (command) -A [query](#query-concept) tool that operates over the build's post-[loading phase](#loading-phase) [target graph](#target-graph). This is relatively fast, but can't analyze the effects of `select()`, [build flags](#command-flags), [artifacts](#artifact), or build [actions](#action). +A [query](#query-concept) tool that operates over the build's post-[loading +phase](#loading-phase) [target graph](#target-graph). This is relatively fast, +but can't analyze the effects of `select()`, [build flags](#command-flags), +[artifacts](#artifact), or build [actions](#action). **See also:** [Query how-to](/query/guide), [Query reference](/query/language) ### Repository -A directory tree with a boundary marker file at its root, containing source files that can be used in a Bazel build. Often shortened to just **repo**. +A directory tree with a boundary marker file at its root, containing source +files that can be used in a Bazel build. Often shortened to just **repo**. -A repo boundary marker file can be `MODULE.bazel` (signaling that this repo represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or `WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a repo; multiple such files can coexist in a directory. +A repo boundary marker file can be `MODULE.bazel` (signaling that this repo +represents a Bazel module), `REPO.bazel`, or in legacy contexts, `WORKSPACE` or +`WORKSPACE.bazel`. Any repo boundary marker file will signify the boundary of a +repo; multiple such files can coexist in a directory. The *main repo* is the repo in which the current Bazel command is being run. -*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` files, or invoking [repo rules](#repository-rule) in [module extensions](#module-extension). They can be fetched on demand to a predetermined "magical" location on disk. +*External repos* are defined by specifying [modules](#module) in `MODULE.bazel` +files, or invoking [repo rules](#repository-rule) in [module +extensions](#module-extension). They can be fetched on demand to a predetermined +"magical" location on disk. -Each repo has a unique, constant *canonical* name, and potentially different *apparent* names when viewed from other repos. +Each repo has a unique, constant *canonical* name, and potentially different +*apparent* names when viewed from other repos. **See also**: [External dependencies overview](/external/overview) ### Repository cache -A shared content-addressable cache of files downloaded by Bazel for builds, shareable across [workspaces](#workspace). Enables offline builds after the initial download. Commonly used to cache files downloaded through [repository rules](#repository-rule) like `http_archive` and repository rule APIs like `repository_ctx.download`. Files are cached only if their SHA-256 checksums are specified for the download. +A shared content-addressable cache of files downloaded by Bazel for builds, +shareable across [workspaces](#workspace). Enables offline builds after the +initial download. Commonly used to cache files downloaded through [repository +rules](#repository-rule) like `http_archive` and repository rule APIs like +`repository_ctx.download`. Files are cached only if their SHA-256 checksums are +specified for the download. ### Repository rule -A schema for repository definitions that tells Bazel how to materialize (or "fetch") a [repository](#repository). Often shortened to just **repo rule**. Repo rules are invoked by Bazel internally to define repos backed by [modules](#module), or can be invoked by [module extensions](#module-extension). Repo rules can access the internet or perform file I/O; the most common repo rule is `http_archive` to download an archive containing source files from the internet. +A schema for repository definitions that tells Bazel how to materialize (or +"fetch") a [repository](#repository). Often shortened to just **repo rule**. +Repo rules are invoked by Bazel internally to define repos backed by +[modules](#module), or can be invoked by [module extensions](#module-extension). +Repo rules can access the internet or perform file I/O; the most common repo +rule is `http_archive` to download an archive containing source files from the +internet. **See also:** [Repo rule documentation](/external/repo) ### Reproducibility -The property of a build or test that a set of inputs to the build or test will always produce the same set of outputs every time, regardless of time, method, or environment. Note that this does not necessarily imply that the outputs are [correct](#correctness) or the desired outputs. +The property of a build or test that a set of inputs to the build or test will +always produce the same set of outputs every time, regardless of time, method, +or environment. Note that this does not necessarily imply that the outputs are +[correct](#correctness) or the desired outputs. ### Rule -A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as `cc_library`. From the perspective of a `BUILD` file author, a rule consists of a set of [attributes](#attributes) and black box logic. The logic tells the rule target how to produce output [artifacts](#artifact) and pass information to other rule targets. From the perspective of `.bzl` authors, rules are the primary way to extend Bazel to support new programming languages and environments. - -Rules are instantiated to produce rule targets in the [loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule targets communicate information to their downstream dependencies in the form of [providers](#provider), and register [actions](#action) describing how to generate their output artifacts. These actions are run in the [execution phase](#execution-phase). - -Note: Historically the term "rule" has been used to refer to a rule target. This usage was inherited from tools like Make, but causes confusion and should be avoided for Bazel. +A schema for defining [rule targets](#rule-target) in a `BUILD` file, such as +`cc_library`. From the perspective of a `BUILD` file author, a rule consists of +a set of [attributes](#attributes) and black box logic. The logic tells the +rule target how to produce output [artifacts](#artifact) and pass information to +other rule targets. From the perspective of `.bzl` authors, rules are the +primary way to extend Bazel to support new programming languages and +environments. + +Rules are instantiated to produce rule targets in the +[loading phase](#loading-phase). In the [analysis phase](#analysis-phase) rule +targets communicate information to their downstream dependencies in the form of +[providers](#provider), and register [actions](#action) describing how to +generate their output artifacts. These actions are run in the [execution +phase](#execution-phase). + +Note: Historically the term "rule" has been used to refer to a rule target. +This usage was inherited from tools like Make, but causes confusion and should +be avoided for Bazel. **See also:** [Rules documentation](/extending/rules) ### Rule target -A [target](#target) that is an instance of a rule. Contrasts with file targets and package groups. Not to be confused with [rule](#rule). +A [target](#target) that is an instance of a rule. Contrasts with file targets +and package groups. Not to be confused with [rule](#rule). ### Runfiles -The runtime dependencies of an executable [target](#target). Most commonly, the executable is the executable output of a test rule, and the runfiles are runtime data dependencies of the test. Before the invocation of the executable (during bazel test), Bazel prepares the tree of runfiles alongside the test executable according to their source directory structure. +The runtime dependencies of an executable [target](#target). Most commonly, the +executable is the executable output of a test rule, and the runfiles are runtime +data dependencies of the test. Before the invocation of the executable (during +bazel test), Bazel prepares the tree of runfiles alongside the test executable +according to their source directory structure. **See also:** [Runfiles documentation](/extending/rules#runfiles) ### Sandboxing -A technique to isolate a running [action](#action) inside a restricted and temporary [execution root](#execution-root), helping to ensure that it doesn’t read undeclared inputs or write undeclared outputs. Sandboxing greatly improves [hermeticity](#hermeticity), but usually has a performance cost, and requires support from the operating system. The performance cost depends on the platform. On Linux, it's not significant, but on macOS it can make sandboxing unusable. +A technique to isolate a running [action](#action) inside a restricted and +temporary [execution root](#execution-root), helping to ensure that it doesn’t +read undeclared inputs or write undeclared outputs. Sandboxing greatly improves +[hermeticity](#hermeticity), but usually has a performance cost, and requires +support from the operating system. The performance cost depends on the platform. +On Linux, it's not significant, but on macOS it can make sandboxing unusable. ### Skyframe [Skyframe](/reference/skyframe) is the core parallel, functional, and incremental evaluation framework of Bazel. +{/* TODO: ### Spawn strategy */} + ### Stamping -A feature to embed additional information into Bazel-built [artifacts](#artifact). For example, this can be used for source control, build time and other workspace or environment-related information for release builds. Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that support the stamp attribute. +A feature to embed additional information into Bazel-built +[artifacts](#artifact). For example, this can be used for source control, build +time and other workspace or environment-related information for release builds. +Enable through the `--workspace_status_command` flag and [rules](/extending/rules) that +support the stamp attribute. ### Starlark -The extension language for writing [rules](/extending/rules) and [macros](#macro). A restricted subset of Python (syntactically and grammatically) aimed for the purpose of configuration, and for better performance. Uses the [`.bzl` file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more restricted version of Starlark (such as no `def` function definitions), formerly known as Skylark. +The extension language for writing [rules](/extending/rules) and [macros](#macro). A +restricted subset of Python (syntactically and grammatically) aimed for the +purpose of configuration, and for better performance. Uses the [`.bzl` +file](#bzl-file) extension. [`BUILD` files](#build-file) use an even more +restricted version of Starlark (such as no `def` function definitions), formerly +known as Skylark. **See also:** [Starlark language documentation](/rules/language) +{/* TODO: ### Starlark rules */} + +{/* TODO: ### Starlark rule sandwich */} + ### Startup flags -The set of flags specified between `bazel` and the [command](#query-command), for example, bazel `--host_jvm_debug` build. These flags modify the [configuration](#configuration) of the Bazel server, so any modification to startup flags causes a server restart. Startup flags are not specific to any command. +The set of flags specified between `bazel` and the [command](#query-command), +for example, bazel `--host_jvm_debug` build. These flags modify the +[configuration](#configuration) of the Bazel server, so any modification to +startup flags causes a server restart. Startup flags are not specific to any +command. ### Symbolic macro -A flavor of [macro](#macro) which is declared with a [rule](#rule)-like [attribute](#attribute) schema, allows hiding internal declared [targets](#target) from their own package, and enforces a predictable naming pattern on the targets that the macro declares. Designed to avoid some of the problems seen in large [legacy macro](#legacy-macro) codebases. +A flavor of [macro](#macro) which is declared with a [rule](#rule)-like +[attribute](#attribute) schema, allows hiding internal declared +[targets](#target) from their own package, and enforces a predictable naming +pattern on the targets that the macro declares. Designed to avoid some of the +problems seen in large [legacy macro](#legacy-macro) codebases. **See also:** [Symbolic macro documentation](/extending/macros) ### Target -An object that is defined in a [`BUILD` file](#build-file) and identified by a [label](#label). Targets represent the buildable units of a workspace from the perspective of the end user. +An object that is defined in a [`BUILD` file](#build-file) and identified by a +[label](#label). Targets represent the buildable units of a workspace from +the perspective of the end user. -A target that is declared by instantiating a [rule](#rule) is called a [rule target](#rule-target). Depending on the rule, these may be runnable (like `cc_binary`) or testable (like `cc_test`). Rule targets typically depend on other targets via their [attributes](#attribute) (such as `deps`); these dependencies form the basis of the [target graph](#target-graph). +A target that is declared by instantiating a [rule](#rule) is called a [rule +target](#rule-target). Depending on the rule, these may be runnable (like +`cc_binary`) or testable (like `cc_test`). Rule targets typically depend on +other targets via their [attributes](#attribute) (such as `deps`); these +dependencies form the basis of the [target graph](#target-graph). -Aside from rule targets, there are also file targets and [package group](#package-group) targets. File targets correspond to [artifacts](#artifact) that are referenced within a `BUILD` file. As a special case, the `BUILD` file of any package is always considered a source file target in that package. +Aside from rule targets, there are also file targets and [package group](#package-group) +targets. File targets correspond to [artifacts](#artifact) that are referenced +within a `BUILD` file. As a special case, the `BUILD` file of any package is +always considered a source file target in that package. -Targets are discovered during the [loading phase](#loading-phase). During the [analysis phase](#analysis-phase), targets are associated with [build configurations](#configuration) to form [configured targets](#configured-target). +Targets are discovered during the [loading phase](#loading-phase). During the +[analysis phase](#analysis-phase), targets are associated with [build +configurations](#configuration) to form [configured +targets](#configured-target). ### Target graph -An in-memory graph of [targets](#target) and their dependencies. Produced during the [loading phase](#loading-phase) and used as an input to the [analysis phase](#analysis-phase). +An in-memory graph of [targets](#target) and their dependencies. Produced during +the [loading phase](#loading-phase) and used as an input to the [analysis +phase](#analysis-phase). ### Target pattern -A way to specify a group of [targets](#target) on the command line. Commonly used patterns are `:all` (all rule targets), `:*` (all rule + file targets), `...` (current [package](#package) and all subpackages recursively). Can be used in combination, for example, `//...:*` means all rule and file targets in all packages recursively from the root of the [workspace](#workspace). +A way to specify a group of [targets](#target) on the command line. Commonly +used patterns are `:all` (all rule targets), `:*` (all rule + file targets), +`...` (current [package](#package) and all subpackages recursively). Can be used +in combination, for example, `//...:*` means all rule and file targets in all +packages recursively from the root of the [workspace](#workspace). ### Tests -Rule [targets](#target) instantiated from test rules, and therefore contains a test executable. A return code of zero from the completion of the executable indicates test success. The exact contract between Bazel and tests (such as test environment variables, test result collection methods) is specified in the [Test Encyclopedia](/reference/test-encyclopedia). +Rule [targets](#target) instantiated from test rules, and therefore contains a +test executable. A return code of zero from the completion of the executable +indicates test success. The exact contract between Bazel and tests (such as test +environment variables, test result collection methods) is specified in the [Test +Encyclopedia](/reference/test-encyclopedia). ### Toolchain -A set of tools to build outputs for a language. Typically, a toolchain includes compilers, linkers, interpreters or/and linters. A toolchain can also vary by platform, that is, a Unix compiler toolchain's components may differ for the Windows variant, even though the toolchain is for the same language. Selecting the right toolchain for the platform is known as toolchain resolution. +A set of tools to build outputs for a language. Typically, a toolchain includes +compilers, linkers, interpreters or/and linters. A toolchain can also vary by +platform, that is, a Unix compiler toolchain's components may differ for the +Windows variant, even though the toolchain is for the same language. Selecting +the right toolchain for the platform is known as toolchain resolution. ### Top-level target -A build [target](#target) is top-level if it’s requested on the Bazel command line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is called, then for this build, `//:foo` is top-level, and `//:bar` isn’t top-level, although both targets will need to be built. An important difference between top-level and non-top-level targets is that [command flags](#command-flags) set on the Bazel command line (or via [.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level targets, but might be modified by a [transition](#transition) for non-top-level targets. +A build [target](#target) is top-level if it’s requested on the Bazel command +line. For example, if `//:foo` depends on `//:bar`, and `bazel build //:foo` is +called, then for this build, `//:foo` is top-level, and `//:bar` isn’t +top-level, although both targets will need to be built. An important difference +between top-level and non-top-level targets is that [command +flags](#command-flags) set on the Bazel command line (or via +[.bazelrc](#bazelrc)) will set the [configuration](#configuration) for top-level +targets, but might be modified by a [transition](#transition) for non-top-level +targets. ### Transition -A mapping of [configuration](#configuration) state from one value to another. Enables [targets](#target) in the [build graph](#build-graph) to have different configurations, even if they were instantiated from the same [rule](#rule). A common usage of transitions is with *split* transitions, where certain parts of the [target graph](#target-graph) is forked with distinct configurations for each fork. For example, one can build an Android APK with native binaries compiled for ARM and x86 using split transitions in a single build. +A mapping of [configuration](#configuration) state from one value to another. +Enables [targets](#target) in the [build graph](#build-graph) to have different +configurations, even if they were instantiated from the same [rule](#rule). A +common usage of transitions is with *split* transitions, where certain parts of +the [target graph](#target-graph) is forked with distinct configurations for +each fork. For example, one can build an Android APK with native binaries +compiled for ARM and x86 using split transitions in a single build. **See also:** [User-defined transitions](/extending/config#user-defined-transitions) ### Tree artifact -An [artifact](#artifact) that represents a collection of files. Since these files are not themselves artifacts, an [action](#action) operating on them must instead register the tree artifact as its input or output. +An [artifact](#artifact) that represents a collection of files. Since these +files are not themselves artifacts, an [action](#action) operating on them must +instead register the tree artifact as its input or output. ### Visibility -One of two mechanisms for preventing unwanted dependencies in the build system: *target visibility* for controlling whether a [target](#target) can be depended upon by other targets; and *load visibility* for controlling whether a `BUILD` or `.bzl` file may load a given `.bzl` file. Without context, usually "visibility" refers to target visibility. +One of two mechanisms for preventing unwanted dependencies in the build system: +*target visibility* for controlling whether a [target](#target) can be depended +upon by other targets; and *load visibility* for controlling whether a `BUILD` +or `.bzl` file may load a given `.bzl` file. Without context, usually +"visibility" refers to target visibility. **See also:** [Visibility documentation](/concepts/visibility) ### Workspace -The environment shared by all Bazel commands run from the same [main repository](#repository). +The environment shared by all Bazel commands run from the same [main +repository](#repository). -Note that historically the concepts of "repository" and "workspace" have been conflated; the term "workspace" has often been used to refer to the main repository, and sometimes even used as a synonym of "repository". Such usage should be avoided for clarity. +Note that historically the concepts of "repository" and "workspace" have been +conflated; the term "workspace" has often been used to refer to the main +repository, and sometimes even used as a synonym of "repository". Such usage +should be avoided for clarity. diff --git a/reference/skyframe.mdx b/reference/skyframe.mdx index 7f99346a..ba9149fa 100644 --- a/reference/skyframe.mdx +++ b/reference/skyframe.mdx @@ -2,86 +2,197 @@ title: 'Skyframe' --- + + The parallel evaluation and incrementality model of Bazel. ## Data model The data model consists of the following items: -- `SkyValue`. Also called nodes. `SkyValues` are immutable objects that contain all the data built over the course of the build and the inputs of the build. Examples are: input files, output files, targets and configured targets. -- `SkyKey`. A short immutable name to reference a `SkyValue`, for example, `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. -- `SkyFunction`. Builds nodes based on their keys and dependent nodes. -- Node graph. A data structure containing the dependency relationship between nodes. -- `Skyframe`. Code name for the incremental evaluation framework Bazel is based on. +* `SkyValue`. Also called nodes. `SkyValues` are immutable objects that + contain all the data built over the course of the build and the inputs of + the build. Examples are: input files, output files, targets and configured + targets. +* `SkyKey`. A short immutable name to reference a `SkyValue`, for example, + `FILECONTENTS:/tmp/foo` or `PACKAGE://foo`. +* `SkyFunction`. Builds nodes based on their keys and dependent nodes. +* Node graph. A data structure containing the dependency relationship between + nodes. +* `Skyframe`. Code name for the incremental evaluation framework Bazel is + based on. ## Evaluation A build is achieved by evaluating the node that represents the build request. -First, Bazel finds the `SkyFunction` corresponding to the key of the top-level `SkyKey`. The function then requests the evaluation of the nodes it needs to evaluate the top-level node, which in turn result in other `SkyFunction` calls, until the leaf nodes are reached. Leaf nodes are usually ones that represent input files in the file system. Finally, Bazel ends up with the value of the top-level `SkyValue`, some side effects (such as output files in the file system) and a directed acyclic graph of the dependencies between the nodes involved in the build. - -A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in advance all of the nodes it needs to do its job. A simple example is evaluating an input file node that turns out to be a symlink: the function tries to read the file, realizes that it is a symlink, and thus fetches the file system node representing the target of the symlink. But that itself can be a symlink, in which case the original function will need to fetch its target, too. - -The functions are represented in the code by the interface `SkyFunction` and the services provided to it by an interface called `SkyFunction.Environment`. These are the things functions can do: - -- Request the evaluation of another node by way of calling `env.getValue`. If the node is available, its value is returned, otherwise, `null` is returned and the function itself is expected to return `null`. In the latter case, the dependent node is evaluated, and then the original node builder is invoked again, but this time the same `env.getValue` call will return a non-`null` value. -- Request the evaluation of multiple other nodes by calling `env.getValues()`. This does essentially the same, except that the dependent nodes are evaluated in parallel. -- Do computation during their invocation -- Have side effects, for example, writing files to the file system. Care needs to be taken that two different functions avoid stepping on each other's toes. In general, write side effects (where data flows outwards from Bazel) are okay, read side effects (where data flows inwards into Bazel without a registered dependency) are not, because they are an unregistered dependency and as such, can cause incorrect incremental builds. - -Well-behaved `SkyFunction` implementations avoid accessing data in any other way than requesting dependencies (such as by directly reading the file system), because that results in Bazel not registering the data dependency on the file that was read, thus resulting in incorrect incremental builds. - -Once a function has enough data to do its job, it should return a non-`null` value indicating completion. +First, Bazel finds the `SkyFunction` corresponding to the key of the top-level +`SkyKey`. The function then requests the evaluation of the nodes it needs to +evaluate the top-level node, which in turn result in other `SkyFunction` calls, +until the leaf nodes are reached. Leaf nodes are usually ones that represent +input files in the file system. Finally, Bazel ends up with the value of the +top-level `SkyValue`, some side effects (such as output files in the file +system) and a directed acyclic graph of the dependencies between the nodes +involved in the build. + +A `SkyFunction` can request `SkyKeys` in multiple passes if it cannot tell in +advance all of the nodes it needs to do its job. A simple example is evaluating +an input file node that turns out to be a symlink: the function tries to read +the file, realizes that it is a symlink, and thus fetches the file system node +representing the target of the symlink. But that itself can be a symlink, in +which case the original function will need to fetch its target, too. + +The functions are represented in the code by the interface `SkyFunction` and the +services provided to it by an interface called `SkyFunction.Environment`. These +are the things functions can do: + +* Request the evaluation of another node by way of calling `env.getValue`. If + the node is available, its value is returned, otherwise, `null` is returned + and the function itself is expected to return `null`. In the latter case, + the dependent node is evaluated, and then the original node builder is + invoked again, but this time the same `env.getValue` call will return a + non-`null` value. +* Request the evaluation of multiple other nodes by calling `env.getValues()`. + This does essentially the same, except that the dependent nodes are + evaluated in parallel. +* Do computation during their invocation +* Have side effects, for example, writing files to the file system. Care needs + to be taken that two different functions avoid stepping on each other's + toes. In general, write side effects (where data flows outwards from Bazel) + are okay, read side effects (where data flows inwards into Bazel without a + registered dependency) are not, because they are an unregistered dependency + and as such, can cause incorrect incremental builds. + +Well-behaved `SkyFunction` implementations avoid accessing data in any other way +than requesting dependencies (such as by directly reading the file system), +because that results in Bazel not registering the data dependency on the file +that was read, thus resulting in incorrect incremental builds. + +Once a function has enough data to do its job, it should return a non-`null` +value indicating completion. This evaluation strategy has a number of benefits: -- Hermeticity. If functions only request input data by way of depending on other nodes, Bazel can guarantee that if the input state is the same, the same data is returned. If all sky functions are deterministic, this means that the whole build will also be deterministic. -- Correct and perfect incrementality. If all the input data of all functions is recorded, Bazel can invalidate only the exact set of nodes that need to be invalidated when the input data changes. -- Parallelism. Since functions can only interact with each other by way of requesting dependencies, functions that don't depend on each other can be run in parallel and Bazel can guarantee that the result is the same as if they were run sequentially. +* Hermeticity. If functions only request input data by way of depending on + other nodes, Bazel can guarantee that if the input state is the same, the + same data is returned. If all sky functions are deterministic, this means + that the whole build will also be deterministic. +* Correct and perfect incrementality. If all the input data of all functions + is recorded, Bazel can invalidate only the exact set of nodes that need to + be invalidated when the input data changes. +* Parallelism. Since functions can only interact with each other by way of + requesting dependencies, functions that don't depend on each other can be + run in parallel and Bazel can guarantee that the result is the same as if + they were run sequentially. ## Incrementality -Since functions can only access input data by depending on other nodes, Bazel can build up a complete data flow graph from the input files to the output files, and use this information to only rebuild those nodes that actually need to be rebuilt: the reverse transitive closure of the set of changed input files. - -In particular, two possible incrementality strategies exist: the bottom-up one and the top-down one. Which one is optimal depends on how the dependency graph looks like. - -- During bottom-up invalidation, after a graph is built and the set of changed inputs is known, all the nodes are invalidated that transitively depend on changed files. This is optimal if the same top-level node will be built again. Note that bottom-up invalidation requires running `stat()` on all input files of the previous build to determine if they were changed. This can be improved by using `inotify` or a similar mechanism to learn about changed files. - -- During top-down invalidation, the transitive closure of the top-level node is checked and only those nodes are kept whose transitive closure is clean. This is better if the node graph is large, but the next build only needs a small subset of it: bottom-up invalidation would invalidate the larger graph of the first build, unlike top-down invalidation, which just walks the small graph of second build. +Since functions can only access input data by depending on other nodes, Bazel +can build up a complete data flow graph from the input files to the output +files, and use this information to only rebuild those nodes that actually need +to be rebuilt: the reverse transitive closure of the set of changed input files. + +In particular, two possible incrementality strategies exist: the bottom-up one +and the top-down one. Which one is optimal depends on how the dependency graph +looks like. + +* During bottom-up invalidation, after a graph is built and the set of changed + inputs is known, all the nodes are invalidated that transitively depend on + changed files. This is optimal if the same top-level node will be built + again. Note that bottom-up invalidation requires running `stat()` on all + input files of the previous build to determine if they were changed. This + can be improved by using `inotify` or a similar mechanism to learn about + changed files. + +* During top-down invalidation, the transitive closure of the top-level node + is checked and only those nodes are kept whose transitive closure is clean. + This is better if the node graph is large, but the next build only needs a + small subset of it: bottom-up invalidation would invalidate the larger graph + of the first build, unlike top-down invalidation, which just walks the small + graph of second build. Bazel only does bottom-up invalidation. -To get further incrementality, Bazel uses *change pruning*: if a node is invalidated, but upon rebuild, it is discovered that its new value is the same as its old value, the nodes that were invalidated due to a change in this node are "resurrected". +To get further incrementality, Bazel uses _change pruning_: if a node is +invalidated, but upon rebuild, it is discovered that its new value is the same +as its old value, the nodes that were invalidated due to a change in this node +are "resurrected". -This is useful, for example, if one changes a comment in a C++ file: then the `.o` file generated from it will be the same, thus, it is unnecessary to call the linker again. +This is useful, for example, if one changes a comment in a C++ file: then the +`.o` file generated from it will be the same, thus, it is unnecessary to call +the linker again. ## Incremental Linking / Compilation -The main limitation of this model is that the invalidation of a node is an all-or-nothing affair: when a dependency changes, the dependent node is always rebuilt from scratch, even if a better algorithm would exist that would mutate the old value of the node based on the changes. A few examples where this would be useful: +The main limitation of this model is that the invalidation of a node is an +all-or-nothing affair: when a dependency changes, the dependent node is always +rebuilt from scratch, even if a better algorithm would exist that would mutate +the old value of the node based on the changes. A few examples where this would +be useful: -- Incremental linking -- When a single class file changes in a JAR file, it is possible modify the JAR file in-place instead of building it from scratch again. +* Incremental linking +* When a single class file changes in a JAR file, it is possible + modify the JAR file in-place instead of building it from scratch again. -The reason why Bazel does not support these things in a principled way is twofold: +The reason why Bazel does not support these things in a principled way +is twofold: -- There were limited performance gains. -- Difficulty to validate that the result of the mutation is the same as that of a clean rebuild would be, and Google values builds that are bit-for-bit repeatable. +* There were limited performance gains. +* Difficulty to validate that the result of the mutation is the same as that + of a clean rebuild would be, and Google values builds that are bit-for-bit + repeatable. -Until now, it was possible to achieve good enough performance by decomposing an expensive build step and achieving partial re-evaluation that way. For example, in an Android app, you can split all the classes into multiple groups and dex them separately. This way, if classes in a group are unchanged, the dexing does not have to be redone. +Until now, it was possible to achieve good enough performance by decomposing an +expensive build step and achieving partial re-evaluation that way. For example, +in an Android app, you can split all the classes into multiple groups and dex +them separately. This way, if classes in a group are unchanged, the dexing does +not have to be redone. ## Mapping to Bazel concepts -This is high level summary of the key `SkyFunction` and `SkyValue` implementations Bazel uses to perform a build: - -- **FileStateValue**. The result of an `lstat()`. For existent files, the function also computes additional information in order to detect changes to the file. This is the lowest level node in the Skyframe graph and has no dependencies. -- **FileValue**. Used by anything that cares about the actual contents or resolved path of a file. Depends on the corresponding `FileStateValue` and any symlinks that need to be resolved (such as the `FileValue` for `a/b` needs the resolved path of `a` and the resolved path of `a/b`). The distinction between `FileValue` and `FileStateValue` is important because the latter can be used in cases where the contents of the file are not actually needed. For example, the file contents are irrelevant when evaluating file system globs (such as `srcs=glob(["*/*.java"])`). -- **DirectoryListingStateValue**. The result of `readdir()`. Like `FileStateValue`, this is the lowest level node and has no dependencies. -- **DirectoryListingValue**. Used by anything that cares about the entries of a directory. Depends on the corresponding `DirectoryListingStateValue`, as well as the associated `FileValue` of the directory. -- **PackageValue**. Represents the parsed version of a BUILD file. Depends on the `FileValue` of the associated `BUILD` file, and also transitively on any `DirectoryListingValue` that is used to resolve the globs in the package (the data structure representing the contents of a `BUILD` file internally). -- **ConfiguredTargetValue**. Represents a configured target, which is a tuple of the set of actions generated during the analysis of a target and information provided to dependent configured targets. Depends on the `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` of direct dependencies, and a special node representing the build configuration. -- **ArtifactValue**. Represents a file in the build, be it a source or an output artifact. Artifacts are almost equivalent to files, and are used to refer to files during the actual execution of build steps. Source files depends on the `FileValue` of the associated node, and output artifacts depend on the `ActionExecutionValue` of whatever action generates the artifact. -- **ActionExecutionValue**. Represents the execution of an action. Depends on the `ArtifactValues` of its input files. The action it executes is contained within its SkyKey, which is contrary to the concept that SkyKeys should be small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if the execution phase does not run. - -As a visual aid, this diagram shows the relationships between SkyFunction implementations after a build of Bazel itself: +This is high level summary of the key `SkyFunction` and `SkyValue` +implementations Bazel uses to perform a build: + +* **FileStateValue**. The result of an `lstat()`. For existent files, the + function also computes additional information in order to detect changes to + the file. This is the lowest level node in the Skyframe graph and has no + dependencies. +* **FileValue**. Used by anything that cares about the actual contents or + resolved path of a file. Depends on the corresponding `FileStateValue` and + any symlinks that need to be resolved (such as the `FileValue` for `a/b` + needs the resolved path of `a` and the resolved path of `a/b`). The + distinction between `FileValue` and `FileStateValue` is important because + the latter can be used in cases where the contents of the file are not + actually needed. For example, the file contents are irrelevant when + evaluating file system globs (such as `srcs=glob(["*/*.java"])`). +* **DirectoryListingStateValue**. The result of `readdir()`. Like + `FileStateValue`, this is the lowest level node and has no dependencies. +* **DirectoryListingValue**. Used by anything that cares about the entries of + a directory. Depends on the corresponding `DirectoryListingStateValue`, as + well as the associated `FileValue` of the directory. +* **PackageValue**. Represents the parsed version of a BUILD file. Depends on + the `FileValue` of the associated `BUILD` file, and also transitively on any + `DirectoryListingValue` that is used to resolve the globs in the package + (the data structure representing the contents of a `BUILD` file internally). +* **ConfiguredTargetValue**. Represents a configured target, which is a tuple + of the set of actions generated during the analysis of a target and + information provided to dependent configured targets. Depends on the + `PackageValue` the corresponding target is in, the `ConfiguredTargetValues` + of direct dependencies, and a special node representing the build + configuration. +* **ArtifactValue**. Represents a file in the build, be it a source or an + output artifact. Artifacts are almost equivalent to files, and are used to + refer to files during the actual execution of build steps. Source files + depends on the `FileValue` of the associated node, and output artifacts + depend on the `ActionExecutionValue` of whatever action generates the + artifact. +* **ActionExecutionValue**. Represents the execution of an action. Depends on + the `ArtifactValues` of its input files. The action it executes is contained + within its SkyKey, which is contrary to the concept that SkyKeys should be + small. Note that `ActionExecutionValue` and `ArtifactValue` are unused if + the execution phase does not run. + +As a visual aid, this diagram shows the relationships between +SkyFunction implementations after a build of Bazel itself: ![A graph of SkyFunction implementation relationships](/reference/skyframe.png) diff --git a/reference/test-encyclopedia.mdx b/reference/test-encyclopedia.mdx deleted file mode 100644 index b56909fe..00000000 --- a/reference/test-encyclopedia.mdx +++ /dev/null @@ -1,308 +0,0 @@ ---- -title: 'Test encyclopedia' ---- - -An exhaustive specification of the test execution environment. - -## Background - -The Bazel BUILD language includes rules which can be used to define automated test programs in many languages. - -Tests are run using [`bazel test`](/docs/user-manual#test). - -Users may also execute test binaries directly. This is allowed but not endorsed, as such an invocation will not adhere to the mandates described below. - -Tests should be *hermetic*: that is, they ought to access only those resources on which they have a declared dependency. If tests are not properly hermetic then they do not give historically reproducible results. This could be a significant problem for culprit finding (determining which change broke a test), release engineering auditability, and resource isolation of tests (automated testing frameworks ought not DDOS a server because some tests happen to talk to it). - -## Objective - -The goal of this page is to formally establish the runtime environment for and expected behavior of Bazel tests. It will also impose requirements on the test runner and the build system. - -The test environment specification helps test authors avoid relying on unspecified behavior, and thus gives the testing infrastructure more freedom to make implementation changes. The specification tightens up some holes that currently allow many tests to pass despite not being properly hermetic, deterministic, and reentrant. - -This page is intended to be both normative and authoritative. If this specification and the implemented behavior of test runner disagree, the specification takes precedence. - -## Proposed Specification - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in IETF RFC 2119. - -## Purpose of tests - -The purpose of Bazel tests is to confirm some property of the source files checked into the repository. (On this page, "source files" includes test data, golden outputs, and anything else kept under version control.) One user writes a test to assert an invariant which they expect to be maintained. Other users execute the test later to check whether the invariant has been broken. If the test depends on any variables other than source files (non-hermetic), its value is diminished, because the later users cannot be sure their changes are at fault when the test stops passing. - -Therefore the outcome of a test must depend only on: - -- source files on which the test has a declared dependency -- products of the build system on which the test has a declared dependency -- resources whose behavior is guaranteed by the test runner to remain constant - -Currently, such behavior is not enforced. However, test runners reserve the right to add such enforcement in the future. - -## Role of the build system - -Test rules are analogous to binary rules in that each must yield an executable program. For some languages, this is a stub program which combines a language-specific harness with the test code. Test rules must produce other outputs as well. In addition to the primary test executable, the test runner will need a manifest of **runfiles**, input files which should be made available to the test at runtime, and it may need information about the type, size, and tags of a test. - -The build system may use the runfiles to deliver code as well as data. (This might be used as an optimization to make each test binary smaller by sharing files across tests, such as through the use of dynamic linking.) The build system should ensure that the generated executable loads these files via the runfiles image provided by the test runner, rather than hardcoded references to absolute locations in the source or output tree. - -## Role of the test runner - -From the point of view of the test runner, each test is a program which can be invoked with `execve()`. There may be other ways to execute tests; for example, an IDE might allow the execution of Java tests in-process. However, the result of running the test as a standalone process must be considered authoritative. If a test process runs to completion and terminates normally with an exit code of zero, the test has passed. Any other result is considered a test failure. In particular, writing any of the strings `PASS` or `FAIL` to stdout has no significance to the test runner. - -If a test takes too long to execute, exceeds some resource limit, or the test runner otherwise detects prohibited behavior, it may choose to kill the test and treat the run as a failure. The runner must not report the test as passing after sending a signal to the test process or any children thereof. - -The whole test target (not individual methods or tests) is given a limited amount of time to run to completion. The time limit for a test is based on its [`timeout`](/reference/be/common-definitions#test.timeout) attribute according to the following table: - -| timeout | Time Limit (sec.) | -| -------- | ----------------- | -| short | 60 | -| moderate | 300 | -| long | 900 | -| eternal | 3600 | - -Tests which do not explicitly specify a timeout have one implied based on the test's [`size`](/reference/be/common-definitions#test.size) as follows: - -| size | Implied timeout label | -| -------- | --------------------- | -| small | short | -| medium | moderate | -| large | long | -| enormous | eternal | - -A "large" test with no explicit timeout setting will be allotted 900 seconds to run. A "medium" test with a timeout of "short" will be allotted 60 seconds. - -Unlike `timeout`, the `size` additionally determines the assumed peak usage of other resources (like RAM) when running the test locally, as described in [Common definitions](/reference/be/common-definitions#common-attributes-tests). - -All combinations of `size` and `timeout` labels are legal, so an "enormous" test may be declared to have a timeout of "short". Presumably it would do some really horrible things very quickly. - -Tests may return arbitrarily fast regardless of timeout. A test is not penalized for an overgenerous timeout, although a warning may be issued: you should generally set your timeout as tight as you can without incurring any flakiness. - -The test timeout can be overridden with the `--test_timeout` bazel flag when manually running under conditions that are known to be slow. The `--test_timeout` values are in seconds. For example, `--test_timeout=120` sets the test timeout to two minutes. - -There is also a recommended lower bound for test timeouts as follows: - -| timeout | Time minimum (sec.) | -| -------- | ------------------- | -| short | 0 | -| moderate | 30 | -| long | 300 | -| eternal | 900 | - -For example, if a "moderate" test completes in 5.5s, consider setting `timeout = "short"` or `size = "small"`. Using the bazel `--test_verbose_timeout_warnings` command line option will show the tests whose specified size is too big. - -Test sizes and timeouts are specified in the BUILD file according to the specification [here](/reference/be/common-definitions#common-attributes-tests). If unspecified, a test's size will default to "medium". - -If the main process of a test exits, but some of its children are still running, the test runner should consider the run complete and count it as a success or failure based on the exit code observed from the main process. The test runner may kill any stray processes. Tests should not leak processes in this fashion. - -## Test sharding - -Tests can be parallelized via test sharding. See [`--test_sharding_strategy`](/reference/command-line-reference#flag--test_sharding_strategy) and [`shard_count`](/reference/be/common-definitions#common-attributes-tests) to enable test sharding. When sharding is enabled, the test runner is launched once per shard. The environment variable [`TEST_TOTAL_SHARDS`](#initial-conditions) is the number of shards, and [`TEST_SHARD_INDEX`](#initial-conditions) is the shard index, beginning at 0. Runners use this information to select which tests to run - for example, using a round-robin strategy. Not all test runners support sharding. If a runner supports sharding, it must create or update the last modified date of the file specified by [`TEST_SHARD_STATUS_FILE`](#initial-conditions). Otherwise, if [`--incompatible_check_sharding_support`](/reference/command-line-reference#flag--incompatible_check_sharding_support) is enabled, Bazel will fail the test if it is sharded. - -## Initial conditions - -When executing a test, the test runner must establish certain initial conditions. - -The test runner must invoke each test with the path to the test executable in `argv[0]`. This path must be relative and beneath the test's current directory (which is in the runfiles tree, see below). The test runner should not pass any other arguments to a test unless the user explicitly requests it. - -The initial environment block shall be composed as follows: - -| Variable | Value | Status | -| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -| `HOME` | value of `$TEST_TMPDIR` | recommended | -| `LANG` | *unset* | required | -| `LANGUAGE` | *unset* | required | -| `LC_ALL` | *unset* | required | -| `LC_COLLATE` | *unset* | required | -| `LC_CTYPE` | *unset* | required | -| `LC_MESSAGES` | *unset* | required | -| `LC_MONETARY` | *unset* | required | -| `LC_NUMERIC` | *unset* | required | -| `LC_TIME` | *unset* | required | -| `LD_LIBRARY_PATH` | colon-separated list of directories containing shared libraries | optional | -| `JAVA_RUNFILES` | value of `$TEST_SRCDIR` | deprecated | -| `LOGNAME` | value of `$USER` | required | -| `PATH` | `/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.` | recommended | -| `PWD` | `$TEST_SRCDIR/workspace-name` | recommended | -| `SHLVL` | `2` | recommended | -| `TEST_INFRASTRUCTURE_FAILURE_FILE` | absolute path to a private file in a writable directory (This file should only be used to report failures originating from the testing infrastructure, not as a general mechanism for reporting flaky failures of tests. In this context, testing infrastructure is defined as systems or libraries that are not test-specific, but can cause test failures by malfunctioning. The first line is the name of the testing infrastructure component that caused the failure, the second one a human-readable description of the failure. Additional lines are ignored.) | optional | -| `TEST_LOGSPLITTER_OUTPUT_FILE` | absolute path to a private file in a writable directory (used to write Logsplitter protobuffer log) | optional | -| `TEST_PREMATURE_EXIT_FILE` | absolute path to a private file in a writable directory (used for catching calls to `exit()`) | optional | -| `TEST_RANDOM_SEED` | If the `--runs_per_test` option is used, `TEST_RANDOM_SEED` is set to the `run number` (starting with 1) for each individual test run. | optional | -| `TEST_RUN_NUMBER` | If the `--runs_per_test` option is used, `TEST_RUN_NUMBER` is set to the `run number` (starting with 1) for each individual test run. | optional | -| `TEST_TARGET` | The name of the target being tested | optional | -| `TEST_SIZE` | The test [`size`](#size) | optional | -| `TEST_TIMEOUT` | The test [`timeout`](#timeout) in seconds | optional | -| `TEST_SHARD_INDEX` | shard index, if [`sharding`](#test-sharding) is used | optional | -| `TEST_SHARD_STATUS_FILE` | path to file to touch to indicate support for [`sharding`](#test-sharding) | optional | -| `TEST_SRCDIR` | absolute path to the base of the runfiles tree | required | -| `TEST_TOTAL_SHARDS` | total [`shard count`](/reference/be/common-definitions#test.shard_count), if [`sharding`](#test-sharding) is used | optional | -| `TEST_TMPDIR` | absolute path to a private writable directory | required | -| `TEST_WORKSPACE` | the local repository's workspace name | optional | -| `TEST_UNDECLARED_OUTPUTS_DIR` | absolute path to a private writable directory (used to write undeclared test outputs). Any files written to the `TEST_UNDECLARED_OUTPUTS_DIR` directory will be zipped up and added to an `outputs.zip` file under `bazel-testlogs`. | optional | -| `TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR` | absolute path to a private writable directory (used to write undeclared test output annotation `.part` and `.pb` files). | optional | -| `TEST_WARNINGS_OUTPUT_FILE` | absolute path to a private file in a writable directory (used to write test target warnings) | optional | -| `TESTBRIDGE_TEST_ONLY` | value of [`--test_filter`](/docs/user-manual#flag--test_filter), if specified | optional | -| `TZ` | `UTC` | required | -| `USER` | value of `getpwuid(getuid())->pw_name` | required | -| `XML_OUTPUT_FILE` | Location to which test actions should write a test result XML output file. Otherwise, Bazel generates a default XML output file wrapping the test log as part of the test action. The XML schema is based on the [JUnit test result schema](https://windyroad.com.au/dl/Open%20Source/JUnit.xsd). | optional | -| `BAZEL_TEST` | Signifies test executable is being driven by `bazel test` | required | - -The environment may contain additional entries. Tests should not depend on the presence, absence, or value of any environment variable not listed above. - -The initial working directory shall be `$TEST_SRCDIR/$TEST_WORKSPACE`. - -The current process id, process group id, session id, and parent process id are unspecified. The process may or may not be a process group leader or a session leader. The process may or may not have a controlling terminal. The process may have zero or more running or unreaped child processes. The process should not have multiple threads when the test code gains control. - -File descriptor 0 (`stdin`) shall be open for reading, but what it is attached to is unspecified. Tests must not read from it. File descriptors 1 (`stdout`) and 2 (`stderr`) shall be open for writing, but what they are attached to is unspecified. It could be a terminal, a pipe, a regular file, or anything else to which characters can be written. They may share an entry in the open file table (meaning that they cannot seek independently). Tests should not inherit any other open file descriptors. - -The initial umask shall be `022` or `027`. - -No alarm or interval timer shall be pending. - -The initial mask of blocked signals shall be empty. All signals shall be set to their default action. - -The initial resource limits, both soft and hard, should be set as follows: - -| Resource | Limit | -| ------------------- | -------------------------------------- | -| `RLIMIT_AS` | unlimited | -| `RLIMIT_CORE` | unspecified | -| `RLIMIT_CPU` | unlimited | -| `RLIMIT_DATA` | unlimited | -| `RLIMIT_FSIZE` | unlimited | -| `RLIMIT_LOCKS` | unlimited | -| `RLIMIT_MEMLOCK` | unlimited | -| `RLIMIT_MSGQUEUE` | unspecified | -| `RLIMIT_NICE` | unspecified | -| `RLIMIT_NOFILE` | at least 1024 | -| `RLIMIT_NPROC` | unspecified | -| `RLIMIT_RSS` | unlimited | -| `RLIMIT_RTPRIO` | unspecified | -| `RLIMIT_SIGPENDING` | unspecified | -| `RLIMIT_STACK` | unlimited, or 2044KB <= rlim <= 8192KB | - -The initial process times (as returned by `times()`) and resource utilization (as returned by `getrusage()`) are unspecified. - -The initial scheduling policy and priority are unspecified. - -## Role of the host system - -In addition to the aspects of user context under direct control of the test runner, the operating system on which tests execute must satisfy certain properties for a test run to be valid. - -#### Filesystem - -The root directory observed by a test may or may not be the real root directory. - -`/proc` shall be mounted. - -All build tools shall be present at the absolute paths under `/usr` used by a local installation. - -Paths starting with `/home` may not be available. Tests should not access any such paths. - -`/tmp` shall be writable, but tests should avoid using these paths. - -Tests must not assume that any constant path is available for their exclusive use. - -Tests must not assume that atimes are enabled for any mounted filesystem. - -#### Users and groups - -The users root, nobody, and unittest must exist. The groups root, nobody, and eng must exist. - -Tests must be executed as a non-root user. The real and effective user ids must be equal; likewise for group ids. Beyond this, the current user id, group id, user name, and group name are unspecified. The set of supplementary group ids is unspecified. - -The current user id and group id must have corresponding names which can be retrieved with `getpwuid()` and `getgrgid()`. The same may not be true for supplementary group ids. - -The current user must have a home directory. It may not be writable. Tests must not attempt to write to it. - -#### Networking - -The hostname is unspecified. It may or may not contain a dot. Resolving the hostname must give an IP address of the current host. Resolving the hostname cut after the first dot must also work. The hostname localhost must resolve. - -#### Other resources - -Tests are granted at least one CPU core. Others may be available but this is not guaranteed. Other performance aspects of this core are not specified. You can increase the reservation to a higher number of CPU cores by adding the tag "cpu:n" (where n is a positive number) to a test rule. If a machine has less total CPU cores than requested, Bazel will still run the test. If a test uses [sharding](#test-sharding), each individual shard will reserve the number of CPU cores specified here. - -Tests may create subprocesses, but not process groups or sessions. - -There is a limit on the number of input files a test may consume. This limit is subject to change, but is currently in the range of tens of thousands of inputs. - -#### Time and date - -The current time and date are unspecified. The system timezone is unspecified. - -X Windows may or may not be available. Tests that need an X server should start Xvfb. - -## Test interaction with the filesystem - -All file paths specified in test environment variables point to somewhere on the local filesystem, unless otherwise specified. - -Tests should create files only within the directories specified by `$TEST_TMPDIR` and `$TEST_UNDECLARED_OUTPUTS_DIR` (if set). - -These directories will be initially empty. - -Tests must not attempt to remove, chmod, or otherwise alter these directories. - -These directories may be a symbolic links. - -The filesystem type of `$TEST_TMPDIR/.` remains unspecified. - -Tests may also write .part files to the `$TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR` to annotate undeclared output files. - -In rare cases, a test may be forced to create files in `/tmp`. For example, [path length limits for Unix domain sockets](https://serverfault.com/questions/641347) typically require creating the socket under `/tmp`. Bazel will be unable to track such files; the test itself must take care to be hermetic, to use unique paths to avoid colliding with other, simultaneously running tests and non-test processes, and to clean up the files it creates in `/tmp`. - -Some popular testing frameworks, such as [JUnit4 `TemporaryFolder`](https://junit.org/junit4/javadoc/latest/org/junit/rules/TemporaryFolder.html) or [Go `TempDir`](https://golang.org/pkg/testing/#T.TempDir), have their own ways to create a temporary directory under `/tmp`. These testing frameworks include functionality that cleans up files in `/tmp`, so you may use them even though they create files outside of `TEST_TMPDIR`. - -Tests must access inputs through the **runfiles** mechanism, or other parts of the execution environment which are specifically intended to make input files available. - -Tests must not access other outputs of the build system at paths inferred from the location of their own executable. - -It is unspecified whether the runfiles tree contains regular files, symbolic links, or a mixture. The runfiles tree may contain symlinks to directories. Tests should avoid using paths containing `..` components within the runfiles tree. - -No directory, file, or symlink within the runfiles tree (including paths which traverse symlinks) should be writable. (It follows that the initial working directory should not be writable.) Tests must not assume that any part of the runfiles is writable, or owned by the current user (for example, `chmod` and `chgrp` may fail). - -The runfiles tree (including paths which traverse symlinks) must not change during test execution. Parent directories and filesystem mounts must not change in any way which affects the result of resolving a path within the runfiles tree. - -In order to catch early exit, a test may create a file at the path specified by `TEST_PREMATURE_EXIT_FILE` upon start and remove it upon exit. If Bazel sees the file when the test finishes, it will assume that the test exited prematurely and mark it as having failed. - -## Execution platform - -The [execution platform](/extending/platforms) for a test action is determined via [toolchain resolution](/extending/toolchains#toolchain-resolution), just like for any other action. Each test rule has an implicitly defined [`test` exec group](/extending/exec-groups#exec-groups-for-native-rules) that, unless overridden, has a mandatory toolchain requirement on `@bazel_tools//tools/test:default_test_toolchain_type`. Toolchains of this type do not carry any data in the form of providers, but can be used to influence the execution platform of the test action. By default, Bazel registers two such toolchains: - -- If `--@bazel_tools//tools/test:incompatible_use_default_test_toolchain` is disabled (the current default), the active test toolchain is `@bazel_tools//tools/test:legacy_test_toolchain`. This toolchain does not impose any constraints and thus test actions without manually specified exec constraints are configured for the first registered execution platform. This is often not the intended behavior in multi-platform builds as it can result in e.g. a test binary built for Linux on a Windows machine to be executed on Windows. -- If `--@bazel_tools//tools/test:incompatible_use_default_test_toolchain` is enabled, the active test toolchain is `@bazel_tools//tools/test:default_test_toolchain`. This toolchain requires an execution platform to match all the constraints of the test rule's target platform. In particular, the target platform is compatible with this toolchain if it is also registered as an execution platform. If no such platform is found, the test rule fails with a toolchain resolution error. - -Users can register additional toolchains for this type to influence this behavior and their toolchains will take precedence over the default ones. Test rule authors can define their own test toolchain type and also register a default toolchain for it. - -## Tag conventions - -Some tags in the test rules have a special meaning. See also the [Bazel Build Encyclopedia on the `tags` attribute](/reference/be/common-definitions#common.tags). - -| Tag | Meaning | -| ----------- | -------------------------------------------------------------------------------------------------------------- | -| `exclusive` | run no other test at the same time | -| `external` | test has an external dependency; disable test caching | -| `large` | `test_suite` convention; suite of large tests | -| `manual *` | don't include test target in wildcard target patterns like `:...`, `:*`, or `:all` | -| `medium` | `test_suite` convention; suite of medium tests | -| `small` | `test_suite` convention; suite of small tests | -| `smoke` | `test_suite` convention; means it should be run before committing code changes into the version control system | - -Note: bazel `query` does not respect the manual tag. - -## Runfiles - -In the following, assume there is a \*\_binary() rule labeled `//foo/bar:unittest`, with a run-time dependency on the rule labeled `//deps/server:server`. - -#### Location - -The runfiles directory for a target `//foo/bar:unittest` is the directory `$(WORKSPACE)/$(BINDIR)/foo/bar/unittest.runfiles`. This path is referred to as the `runfiles_dir`. - -#### Dependencies - -The runfiles directory is declared as a compile-time dependency of the `*_binary()` rule. The runfiles directory itself depends on the set of BUILD files that affect the `*_binary()` rule or any of its compile-time or run-time dependencies. Modifying source files does not affect the structure of the runfiles directory, and thus does not trigger any rebuilding. - -#### Contents - -The runfiles directory contains the following: - -- **Symlinks to run-time dependencies**: each OutputFile and CommandRule that is a run-time dependency of the `*_binary()` rule is represented by one symlink in the runfiles directory. The name of the symlink is `$(WORKSPACE)/package_name/rule_name`. For example, the symlink for server would be named `$(WORKSPACE)/deps/server/server`, and the full path would be `$(WORKSPACE)/foo/bar/unittest.runfiles/$(WORKSPACE)/deps/server/server`. The destination of the symlink is the OutputFileName() of the OutputFile or CommandRule, expressed as an absolute path. Thus, the destination of the symlink might be `$(WORKSPACE)/linux-dbg/deps/server/42/server`. diff --git a/release/backward-compatibility.mdx b/release/backward-compatibility.mdx index 42f40956..af653ccd 100644 --- a/release/backward-compatibility.mdx +++ b/release/backward-compatibility.mdx @@ -2,51 +2,75 @@ title: 'Backward Compatibility' --- -This page provides information about how to handle backward compatibility, including migrating from one release to another and how to communicate incompatible changes. -Bazel is evolving. Minor versions released as part of an [LTS major version](/release#bazel-versioning) are fully backward-compatible. New major LTS releases may contain incompatible changes that require some migration effort. For more information about Bazel's release model, please check out the [Release Model](/release) page. + +This page provides information about how to handle backward compatibility, +including migrating from one release to another and how to communicate +incompatible changes. + +Bazel is evolving. Minor versions released as part of an [LTS major +version](/release#bazel-versioning) are fully backward-compatible. New major LTS +releases may contain incompatible changes that require some migration effort. +For more information about Bazel's release model, please check out the [Release +Model](/release) page. ## Summary -1. It is recommended to use `--incompatible_*` flags for breaking changes. -2. For every `--incompatible_*` flag, a GitHub issue explains the change in behavior and aims to provide a migration recipe. -3. Incompatible flags are recommended to be back-ported to the latest LTS release without enabling the flag by default. -4. APIs and behavior guarded by an `--experimental_*` flag can change at any time. -5. Never run production builds with `--experimental_*` or `--incompatible_*` flags. +1. It is recommended to use `--incompatible_*` flags for breaking changes. +1. For every `--incompatible_*` flag, a GitHub issue explains the change in + behavior and aims to provide a migration recipe. +1. Incompatible flags are recommended to be back-ported to the latest LTS + release without enabling the flag by default. +1. APIs and behavior guarded by an `--experimental_*` flag can change at any + time. +1. Never run production builds with `--experimental_*` or `--incompatible_*` + flags. ## How to follow this policy -- [For Bazel users - how to update Bazel](/install/bazelisk) -- [For contributors - best practices for incompatible changes](/contribute/breaking-changes) -- [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) +* [For Bazel users - how to update Bazel](/install/bazelisk) +* [For contributors - best practices for incompatible changes](/contribute/breaking-changes) +* [For release managers - how to update issue labels and release](https://github.com/bazelbuild/continuous-integration/tree/master/docs/release-playbook.%6D%64) ## What is stable functionality? -In general, APIs or behaviors without `--experimental_...` flags are considered stable, supported features in Bazel. +In general, APIs or behaviors without `--experimental_...` flags are considered +stable, supported features in Bazel. This includes: -- Starlark language and APIs -- Rules bundled with Bazel -- Bazel APIs such as Remote Execution APIs or Build Event Protocol -- Flags and their semantics +* Starlark language and APIs +* Rules bundled with Bazel +* Bazel APIs such as Remote Execution APIs or Build Event Protocol +* Flags and their semantics ## Incompatible changes and migration recipes -For every incompatible change in a new release, the Bazel team aims to provide a *migration recipe* that helps you update your code (`BUILD` and `.bzl` files, as well as any Bazel usage in scripts, usage of Bazel API, and so on). +For every incompatible change in a new release, the Bazel team aims to provide a +_migration recipe_ that helps you update your code (`BUILD` and `.bzl` files, as +well as any Bazel usage in scripts, usage of Bazel API, and so on). -Incompatible changes should have an associated `--incompatible_*` flag and a corresponding GitHub issue. +Incompatible changes should have an associated `--incompatible_*` flag and a +corresponding GitHub issue. -The incompatible flag and relevant changes are recommended to be back-ported to the latest LTS release without enabling the flag by default. This allows users to migrate for the incompatible changes before the next LTS release is available. +The incompatible flag and relevant changes are recommended to be back-ported to +the latest LTS release without enabling the flag by default. This allows users +to migrate for the incompatible changes before the next LTS release is +available. ## Communicating incompatible changes -The primary source of information about incompatible changes are GitHub issues marked with an ["incompatible-change" label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). +The primary source of information about incompatible changes are GitHub issues +marked with an ["incompatible-change" +label](https://github.com/bazelbuild/bazel/issues?q=label%3Aincompatible-change). For every incompatible change, the issue specifies the following: -- Name of the flag controlling the incompatible change -- Description of the changed functionality -- Migration recipe +* Name of the flag controlling the incompatible change +* Description of the changed functionality +* Migration recipe -When an incompatible change is ready for migration with Bazel at HEAD (therefore, also with the next Bazel rolling release), it should be marked with the `migration-ready` label. The incompatible change issue is closed when the incompatible flag is flipped at HEAD. +When an incompatible change is ready for migration with Bazel at HEAD +(therefore, also with the next Bazel rolling release), it should be marked with +the `migration-ready` label. The incompatible change issue is closed when the +incompatible flag is flipped at HEAD. diff --git a/release/index.mdx b/release/index.mdx index 0d2629e7..f58e8f5f 100644 --- a/release/index.mdx +++ b/release/index.mdx @@ -2,48 +2,66 @@ title: 'Release Model' --- -As announced in [the original blog post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel 4.0 and higher versions provides support for two release tracks: rolling releases and long term support (LTS) releases. This page covers the latest information about Bazel's release model. + + +As announced in [the original blog +post](https://blog.bazel.build/2020/11/10/long-term-support-release.html), Bazel +4.0 and higher versions provides support for two release tracks: rolling +releases and long term support (LTS) releases. This page covers the latest +information about Bazel's release model. ## Support matrix -| LTS release | Support stage | Latest version | End of support | -| ----------- | ------------- | --------------------------------------------------------------- | -------------- | -| Bazel 9 | Rolling | [Check rolling release page](/release/rolling) | N/A | -| Bazel 8 | Active | [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | -| Bazel 7 | Maintenance | [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | -| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | -| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | -| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | +| LTS release | Support stage | Latest version | End of support | +| ----------- | ------------- | -------------- | -------------- | +| Bazel 9 | Rolling| [Check rolling release page](/release/rolling) | N/A | +| Bazel 8 | Active| [8.4.2](https://github.com/bazelbuild/bazel/releases/tag/8.4.2) | December 2027 | +| Bazel 7 | Maintenance| [7.7.0](https://github.com/bazelbuild/bazel/releases/tag/7.7.0) | Dec 2026 | +| Bazel 6 | Maintenance | [6.5.0](https://github.com/bazelbuild/bazel/releases/tag/6.5.0) | Dec 2025 | +| Bazel 5 | Deprecated | [5.4.1](https://github.com/bazelbuild/bazel/releases/tag/5.4.1) | Jan 2025 | +| Bazel 4 | Deprecated | [4.2.4](https://github.com/bazelbuild/bazel/releases/tag/4.2.4) | Jan 2024 | -All Bazel LTS releases can be found on the [release page](https://github.com/bazelbuild/bazel/releases) on GitHub. +All Bazel LTS releases can be found on the [release +page](https://github.com/bazelbuild/bazel/releases) on GitHub. -Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are recommended to upgrade to the latest LTS release or use rolling releases if you want to keep up with the latest changes at HEAD. +Note: Bazel version older than Bazel 5 are no longer supported, Bazel users are +recommended to upgrade to the latest LTS release or use rolling releases if you +want to keep up with the latest changes at HEAD. ## Release versioning -Bazel uses a *major.minor.patch* [Semantic Versioning](https://semver.org/) scheme. +Bazel uses a _major.minor.patch_ [Semantic +Versioning](https://semver.org/) scheme. -- A *major release* contains features that are not backward compatible with the previous release. Each major Bazel version is an LTS release. -- A *minor release* contains backward-compatible bug fixes and features back-ported from the main branch. -- A *patch release* contains critical bug fixes. +* A _major release_ contains features that are not backward compatible with + the previous release. Each major Bazel version is an LTS release. +* A _minor release_ contains backward-compatible bug fixes and features + back-ported from the main branch. +* A _patch release_ contains critical bug fixes. -Additionally, pre-release versions are indicated by appending a hyphen and a date suffix to the next major version number. +Additionally, pre-release versions are indicated by appending a hyphen and a +date suffix to the next major version number. For example, a new release of each type would result in these version numbers: -- Major: 6.0.0 -- Minor: 6.1.0 -- Patch: 6.1.2 -- Pre-release: 7.0.0-pre.20230502.1 +* Major: 6.0.0 +* Minor: 6.1.0 +* Patch: 6.1.2 +* Pre-release: 7.0.0-pre.20230502.1 ## Support stages For each major Bazel version, there are four support stages: -- **Rolling**: This major version is still in pre-release, the Bazel team publishes rolling releases from HEAD. -- **Active**: This major version is the current active LTS release. The Bazel team backports important features and bug fixes into its minor releases. -- **Maintenance**: This major version is an old LTS release in maintenance mode. The Bazel team only promises to backport critical bug fixes for security issues and OS-compatibility issues into this LTS release. -- **Deprecated**: The Bazel team no longer provides support for this major version, all users should migrate to newer Bazel LTS releases. +* **Rolling**: This major version is still in pre-release, the Bazel team + publishes rolling releases from HEAD. +* **Active**: This major version is the current active LTS release. The Bazel + team backports important features and bug fixes into its minor releases. +* **Maintenance**: This major version is an old LTS release in maintenance + mode. The Bazel team only promises to backport critical bug fixes for + security issues and OS-compatibility issues into this LTS release. +* **Deprecated**: The Bazel team no longer provides support for this major + version, all users should migrate to newer Bazel LTS releases. ## Release cadence @@ -51,90 +69,148 @@ Bazel regularly publish releases for two release tracks. ### Rolling releases -- Rolling releases are coordinated with Google Blaze release and are released from HEAD around every two weeks. It is a preview of the next Bazel LTS release. -- Rolling releases can ship incompatible changes. Incompatible flags are recommended for major breaking changes, rolling out incompatible changes should follow our [backward compatibility policy](/release/backward-compatibility). +* Rolling releases are coordinated with Google Blaze release and are released + from HEAD around every two weeks. It is a preview of the next Bazel LTS + release. +* Rolling releases can ship incompatible changes. Incompatible flags are + recommended for major breaking changes, rolling out incompatible changes + should follow our [backward compatibility + policy](/release/backward-compatibility). ### LTS releases -- *Major release*: A new LTS release is expected to be cut from HEAD roughly every 12 months. Once a new LTS release is out, it immediately enters the Active stage, and the previous LTS release enters the Maintenance stage. -- *Minor release*: New minor verions on the Active LTS track are expected to be released once every 2 months. -- *Patch release*: New patch versions for LTS releases in Active and Maintenance stages are expected to be released on demand for critical bug fixes. -- A Bazel LTS release enters the Deprecated stage after being in ​​the Maintenance stage for 2 years. - -For planned releases, please check our [release issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) on Github. +* _Major release_: A new LTS release is expected to be cut from HEAD roughly + every + 12 months. Once a new LTS release is out, it immediately enters the Active + stage, and the previous LTS release enters the Maintenance stage. +* _Minor release_: New minor verions on the Active LTS track are expected to + be released once every 2 months. +* _Patch release_: New patch versions for LTS releases in Active and + Maintenance stages are expected to be released on demand for critical bug + fixes. +* A Bazel LTS release enters the Deprecated stage after being in ​​the + Maintenance stage for 2 years. + +For planned releases, please check our [release +issues](https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3Arelease) +on Github. ## Release procedure & policies -For rolling releases, the process is straightforward: about every two weeks, a new release is created, aligning with the same baseline as the Google internal Blaze release. Due to the rapid release schedule, we don't backport any changes to rolling releases. +For rolling releases, the process is straightforward: about every two weeks, a +new release is created, aligning with the same baseline as the Google internal +Blaze release. Due to the rapid release schedule, we don't backport any changes +to rolling releases. For LTS releases, the procedure and policies below are followed: -1. Determine a baseline commit for the release. - - - For a new major LTS release, the baseline commit is the HEAD of the main branch. - - For a minor or patch release, the baseline commit is the HEAD of the current latest version of the same LTS release. - -2. Create a release branch in the name of `release-<version>` from the baseline commit. - -3. Backport changes via PRs to the release branch. - - - The community can suggest certain commits to be back-ported by replying "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential release blockers, the Bazel team triages them and decide whether to back-port the commits. - - Only backward-compatible commits on the main branch can be back-ported, additional minor changes to resolve merge conflicts are acceptable. - -4. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. - - - Bazel maintainers can request to cherry-pick specific commit(s) to a release branch. This process is initiated by creating a cherry-pick request on GitHub. Here's how to do it. - - 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=\&labels=\&projects=\&template=cherry_pick_request.yml) - - 2. Fill in the request details - - - Title: Provide a concise and descriptive title for the request. - - Commit ID(s): Enter the ID(s) of the commit(s) you want to cherry-pick. If there are multiple commits, then separate them with commas. - - Category: Specify the category of the request. - - Reviewer(s): For multiple reviewers, separate their GitHub ID's with commas. - - 3. Set the milestone - - - Find the "Milestone" section and click the setting. - - Select the appropriate X.Y.Z release blockers. This action triggers the cherry-pick bot to process your request for the "release-X.Y.Z" branch. - - 4. Submit the Issue - - Once all details are filled in and the miestone is set, submit the issue. - - - The cherry-pick bot will process the request and notify if the commit(s) are eligible for cherry-picking. If the commits are cherry-pickable, which means there's no merge conflict while cherry-picking the commit, then the bot will create a new pull request. When the pull request is approved by a member of the Bazel team, the commits are cherry-picked and merged to the release branch. For a visual example of a completed cherry-pick request, refer to this [example](https://github.com/bazelbuild/bazel/issues/20230) . - -5. Identify release blockers and fix issues found on the release branch. - - - The release branch is tested with the same test suite in [postsubmit](https://buildkite.com/bazel/bazel-bazel) and [downstream test pipeline](https://buildkite.com/bazel/bazel-at-head-plus-downstream) on Bazel CI. The Bazel team monitors testing results of the release branch and fixes any regressions found. - -6. Create a new release candidate from the release branch when all known release blockers are resolved. - - - The release candidate is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors community bug reports for the candidate. - - If new release blockers are identified, go back to the last step and create a new release candidate after resolving all the issues. - - New features are not allowed to be added to the release branch after the first release candidate is created; cherry-picks are limited to critical fixes only. If a cherry-pick is needed, the requester must answer the following questions: Why is this change critical, and what benefits does it provide? What is the likelihood of this change introducing a regression? - -7. Push the release candidate as the official release if no further release blockers are found - - - For patch releases, push the release at least two business days after the last release candidate is out. - - For major and minor releases, push the release two business days after the last release candidate is out, but not earlier than one week after the first release candidate is out. - - The release is only pushed on a day where the next day is a business day. - - The release is announced on [bazel-discuss](https://groups.google.com/g/bazel-discuss), the Bazel team monitors and addresses community bug reports for the new release. +1. Determine a baseline commit for the release. + * For a new major LTS release, the baseline commit is the HEAD of the main + branch. + * For a minor or patch release, the baseline commit is the HEAD of the + current latest version of the same LTS release. +1. Create a release branch in the name of `release-` from the baseline + commit. +1. Backport changes via PRs to the release branch. + * The community can suggest certain commits to be back-ported by replying + "`@bazel-io flag`" on relevant GitHub issues or PRs to mark them as potential + release blockers, the Bazel team triages them and decide whether to + back-port the commits. + * Only backward-compatible commits on the main branch can be back-ported, + additional minor changes to resolve merge conflicts are acceptable. +1. Backport changes using Cherry-Pick Request Issue for Bazel maintainers. + * Bazel maintainers can request to cherry-pick specific commit(s) + to a release branch. This process is initiated by creating a + cherry-pick request on GitHub. Here's how to do it. + 1. Open the [cherry-pick request](https://github.com/bazelbuild/bazel/issues/new?assignees=&labels=&projects=&template=cherry_pick_request.yml) + 2. Fill in the request details + * Title: Provide a concise and descriptive title for the request. + * Commit ID(s): Enter the ID(s) of the commit(s) you want to + cherry-pick. If there are multiple commits, then separate + them with commas. + * Category: Specify the category of the request. + * Reviewer(s): For multiple reviewers, separate their GitHub + ID's with commas. + 3. Set the milestone + * Find the "Milestone" section and click the setting. + * Select the appropriate X.Y.Z release blockers. This action + triggers the cherry-pick bot to process your request + for the "release-X.Y.Z" branch. + 4. Submit the Issue + * Once all details are filled in and the miestone is set, + submit the issue. + + * The cherry-pick bot will process the request and notify + if the commit(s) are eligible for cherry-picking. If + the commits are cherry-pickable, which means there's no + merge conflict while cherry-picking the commit, then + the bot will create a new pull request. When the pull + request is approved by a member of the Bazel team, the + commits are cherry-picked and merged to the release branch. + For a visual example of a completed cherry-pick request, + refer to this + [example](https://github.com/bazelbuild/bazel/issues/20230) + . + +1. Identify release blockers and fix issues found on the release branch. + * The release branch is tested with the same test suite in + [postsubmit](https://buildkite.com/bazel/bazel-bazel) and + [downstream test pipeline] + (https://buildkite.com/bazel/bazel-at-head-plus-downstream) + on Bazel CI. The Bazel team monitors testing results of the release + branch and fixes any regressions found. +1. Create a new release candidate from the release branch when all known + release blockers are resolved. + * The release candidate is announced on + [bazel-discuss](https://groups.google.com/g/bazel-discuss), + the Bazel team monitors community bug reports for the candidate. + * If new release blockers are identified, go back to the last step and + create a new release candidate after resolving all the issues. + * New features are not allowed to be added to the release branch after the + first release candidate is created; cherry-picks are limited to critical + fixes only. If a cherry-pick is needed, the requester must answer the + following questions: Why is this change critical, and what benefits does + it provide? What is the likelihood of this change introducing a + regression? +1. Push the release candidate as the official release if no further release + blockers are found + * For patch releases, push the release at least two business days after + the last release candidate is out. + * For major and minor releases, push the release two business days after + the last release candidate is out, but not earlier than one week after + the first release candidate is out. + * The release is only pushed on a day where the next day is a business + day. + * The release is announced on + [bazel-discuss](https://groups.google.com/g/bazel-discuss), + the Bazel team monitors and addresses community bug reports for the new + release. ## Report regressions -If a user finds a regression in a new Bazel release, release candidate or even Bazel at HEAD, please file a bug on [GitHub](https://github.com/bazelbuild/bazel/issues). You can use Bazelisk to bisect the culprit commit and include this information in the bug report. +If a user finds a regression in a new Bazel release, release candidate or even +Bazel at HEAD, please file a bug on +[GitHub](https://github.com/bazelbuild/bazel/issues). You can use +Bazelisk to bisect the culprit commit and include this information in the bug +report. -For example, if your build succeeds with Bazel 6.1.0 but fails with the second release candidate of 6.2.0, you can do bisect via +For example, if your build succeeds with Bazel 6.1.0 but fails with the second +release candidate of 6.2.0, you can do bisect via ```bash bazelisk --bisect=6.1.0..release-6.2.0rc2 build //foo:bar ``` -You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run corresponding bazel commands to reset the build state if it's needed to reproduce the issue. For more details, check out documentation about Bazelisk [bisect feature](https://github.com/bazelbuild/bazelisk#--bisect). +You can set `BAZELISK_SHUTDOWN` or `BAZELISK_CLEAN` environment variable to run +corresponding bazel commands to reset the build state if it's needed to +reproduce the issue. For more details, check out documentation about Bazelisk +[bisect feature] (https://github.com/bazelbuild/bazelisk#--bisect). -Remember to upgrade Bazelisk to the latest version to use the bisect feature. +Remember to upgrade Bazelisk to the latest version to use the bisect +feature. ## Rule compatibility -If you are a rule authors and want to maintain compatibility with different Bazel versions, please check out the [Rule Compatibility](/release/rule-compatibility) page. +If you are a rule authors and want to maintain compatibility with different +Bazel versions, please check out the [Rule +Compatibility](/release/rule-compatibility) page. diff --git a/release/rolling.mdx b/release/rolling.mdx index 4cb413ad..8e79d6f5 100644 --- a/release/rolling.mdx +++ b/release/rolling.mdx @@ -2,6 +2,13 @@ title: 'Rolling Releases' --- -This page contains an overview of all rolling releases, as per our [release policy](https://bazel.build/release#rolling-releases). [Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use these releases. -## Index + +This page contains an overview of all rolling releases, as per our +[release policy](https://bazel.build/release#rolling-releases). +[Bazelisk](https://github.com/bazelbuild/bazelisk) is the best way to use +these releases. + +## Index + + diff --git a/release/rule-compatibility.mdx b/release/rule-compatibility.mdx index 93027953..05a8a95e 100644 --- a/release/rule-compatibility.mdx +++ b/release/rule-compatibility.mdx @@ -2,55 +2,89 @@ title: 'Rule Compatibility' --- -Bazel Starlark rules can break compatibility with Bazel LTS releases in the following two scenarios: -1. The rule breaks compatibility with future LTS releases because a feature it depends on is removed from Bazel at HEAD. -2. The rule breaks compatibility with the current or older LTS releases because a feature it depends on is only available in newer Bazel LTS releases. -Meanwhile, the rule itself can ship incompatible changes for their users as well. When combined with breaking changes in Bazel, upgrading the rule version and Bazel version can often be a source of frustration for Bazel users. This page covers how rules authors should maintain rule compatibility with Bazel to make it easier for users to upgrade Bazel and rules. +Bazel Starlark rules can break compatibility with Bazel LTS releases in the +following two scenarios: + +1. The rule breaks compatibility with future LTS releases because a feature it + depends on is removed from Bazel at HEAD. +1. The rule breaks compatibility with the current or older LTS releases because + a feature it depends on is only available in newer Bazel LTS releases. + +Meanwhile, the rule itself can ship incompatible changes for their users as +well. When combined with breaking changes in Bazel, upgrading the rule version +and Bazel version can often be a source of frustration for Bazel users. This +page covers how rules authors should maintain rule compatibility with Bazel to +make it easier for users to upgrade Bazel and rules. ## Manageable migration process -While it's obviously not feasible to guarantee compatibility between every version of Bazel and every version of the rule, our aim is to ensure that the migration process remains manageable for Bazel users. A manageable migration process is defined as a process where **users are not forced to upgrade the rule's major version and Bazel's major version simultaneously**, thereby allowing users to handle incompatible changes from one source at a time. +While it's obviously not feasible to guarantee compatibility between every +version of Bazel and every version of the rule, our aim is to ensure that the +migration process remains manageable for Bazel users. A manageable migration +process is defined as a process where **users are not forced to upgrade the +rule's major version and Bazel's major version simultaneously**, thereby +allowing users to handle incompatible changes from one source at a time. For example, with the following compatibility matrix: -- Migrating from rules\_foo 1.x + Bazel 4.x to rules\_foo 2.x + Bazel 5.x is not considered manageable, as the users need to upgrade the major version of rules\_foo and Bazel at the same time. -- Migrating from rules\_foo 2.x + Bazel 5.x to rules\_foo 3.x + Bazel 6.x is considered manageable, as the users can first upgrade rules\_foo from 2.x to 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to 6.x. +* Migrating from rules_foo 1.x + Bazel 4.x to rules_foo 2.x + Bazel 5.x is not + considered manageable, as the users need to upgrade the major version of + rules_foo and Bazel at the same time. +* Migrating from rules_foo 2.x + Bazel 5.x to rules_foo 3.x + Bazel 6.x is + considered manageable, as the users can first upgrade rules_foo from 2.x to + 3.x without changing the major Bazel version, then upgrade Bazel from 5.x to + 6.x. -| | rules\_foo 1.x | rules\_foo 2.x | rules\_foo 3.x | HEAD | -| --------- | -------------- | -------------- | -------------- | ---- | -| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | -| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | -| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | -| HEAD | ❌ | ❌ | ❌ | ✅ | +| | rules_foo 1.x | rules_foo 2.x | rules_foo 3.x | HEAD | +| --- | --- | --- | --- | --- | +| Bazel 4.x | ✅ | ❌ | ❌ | ❌ | +| Bazel 5.x | ❌ | ✅ | ✅ | ❌ | +| Bazel 6.x | ❌ | ❌ | ✅ | ✅ | +| HEAD | ❌ | ❌ | ❌ | ✅ | -❌: No version of the major rule version is compatible with the Bazel LTS release. +❌: No version of the major rule version is compatible with the Bazel LTS +release. -✅: At least one version of the rule is compatible with the latest version of the Bazel LTS release. +✅: At least one version of the rule is compatible with the latest version of the +Bazel LTS release. ## Best practices -As Bazel rules authors, you can ensure a manageable migration process for users by following these best practices: - -1. The rule should follow [Semantic Versioning](https://semver.org/): minor versions of the same major version are backward compatible. - -2. The rule at HEAD should be compatible with the latest Bazel LTS release. - -3. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, you can - - - Set up your own CI testing with Bazel at HEAD - - Add your project to [Bazel downstream testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); the Bazel team files issues to your project if breaking changes in Bazel affect your project, and you must follow our [downstream project policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) to address issues timely. - -4. The latest major version of the rule must be compatible with the latest Bazel LTS release. - -5. A new major version of the rule should be compatible with the last Bazel LTS release supported by the previous major version of the rule. - -Achieving 2. and 3. is the most important task since it allows achieving 4. and 5. naturally. - -To make it easier to keep compatibility with both Bazel at HEAD and the latest Bazel LTS release, rules authors can: - -- Request backward-compatible features to be back-ported to the latest LTS release, check out [release process](/release#release-procedure-policies) for more details. -- Use [bazel\_features](https://github.com/bazel-contrib/bazel_features) to do Bazel feature detection. - -In general, with the recommended approaches, rules should be able to migrate for Bazel incompatible changes and make use of new Bazel features at HEAD without dropping compatibility with the latest Bazel LTS release. +As Bazel rules authors, you can ensure a manageable migration process for users +by following these best practices: + +1. The rule should follow [Semantic + Versioning](https://semver.org/): minor versions of the same + major version are backward compatible. +1. The rule at HEAD should be compatible with the latest Bazel LTS release. +1. The rule at HEAD should be compatible with Bazel at HEAD. To achieve this, + you can + * Set up your own CI testing with Bazel at HEAD + * Add your project to [Bazel downstream + testing](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md); + the Bazel team files issues to your project if breaking changes in Bazel + affect your project, and you must follow our [downstream project + policies](https://github.com/bazelbuild/continuous-integration/blob/master/docs/downstream-testing.md#downstream-project-policies) + to address issues timely. +1. The latest major version of the rule must be compatible with the latest + Bazel LTS release. +1. A new major version of the rule should be compatible with the last Bazel LTS + release supported by the previous major version of the rule. + +Achieving 2. and 3. is the most important task since it allows achieving 4. and +5. naturally. + +To make it easier to keep compatibility with both Bazel at HEAD and the latest +Bazel LTS release, rules authors can: + +* Request backward-compatible features to be back-ported to the latest LTS + release, check out [release process](/release#release-procedure-policies) + for more details. +* Use [bazel_features](https://github.com/bazel-contrib/bazel_features) + to do Bazel feature detection. + +In general, with the recommended approaches, rules should be able to migrate for +Bazel incompatible changes and make use of new Bazel features at HEAD without +dropping compatibility with the latest Bazel LTS release. diff --git a/remote/bep-examples.mdx b/remote/bep-examples.mdx index 68e4d42c..faf11bf9 100644 --- a/remote/bep-examples.mdx +++ b/remote/bep-examples.mdx @@ -2,9 +2,14 @@ title: 'Build Event Protocol Examples' --- -The full specification of the Build Event Protocol can be found in its protocol buffer definition. However, it might be helpful to build up some intuition before looking at the specification. -Consider a simple Bazel workspace that consists of two empty shell scripts `foo.sh` and `foo_test.sh` and the following `BUILD` file: + +The full specification of the Build Event Protocol can be found in its protocol +buffer definition. However, it might be helpful to build up some intuition +before looking at the specification. + +Consider a simple Bazel workspace that consists of two empty shell scripts +`foo.sh` and `foo_test.sh` and the following `BUILD` file: ```bash sh_library( @@ -19,32 +24,58 @@ sh_test( ) ``` -When running `bazel test ...` on this project the build graph of the generated build events will resemble the graph below. The arrows indicate the aforementioned parent and child relationship. Note that some build events and most fields have been omitted for brevity. +When running `bazel test ...` on this project the build graph of the generated +build events will resemble the graph below. The arrows indicate the +aforementioned parent and child relationship. Note that some build events and +most fields have been omitted for brevity. ![bep-graph](/docs/images/bep-graph.png "BEP graph") **Figure 1.** BEP graph. -Initially, a `BuildStarted` event is published. The event informs us that the build was invoked through the `bazel test` command and announces child events: +Initially, a `BuildStarted` event is published. The event informs us that the +build was invoked through the `bazel test` command and announces child events: -- `OptionsParsed` -- `WorkspaceStatus` -- `CommandLine` -- `UnstructuredCommandLine` -- `BuildMetadata` -- `BuildFinished` -- `PatternExpanded` -- `Progress` +* `OptionsParsed` +* `WorkspaceStatus` +* `CommandLine` +* `UnstructuredCommandLine` +* `BuildMetadata` +* `BuildFinished` +* `PatternExpanded` +* `Progress` The first three events provide information about how Bazel was invoked. -The `PatternExpanded` build event provides insight into which specific targets the `...` pattern expanded to: `//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two `TargetConfigured` events as children. Note that the `TargetConfigured` event declares the `Configuration` event as a child event, even though `Configuration` has been posted before the `TargetConfigured` event. - -Besides the parent and child relationship, events may also refer to each other using their build event identifiers. For example, in the above graph the `TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` field. - -Build events that refer to files don’t usually embed the file names and paths in the event. Instead, they contain the build event identifier of a `NamedSetOfFiles` event, which will then contain the actual file names and paths. The `NamedSetOfFiles` event allows a set of files to be reported once and referred to by many targets. This structure is necessary because otherwise in some cases the Build Event Protocol output size would grow quadratically with the number of files. A `NamedSetOfFiles` event may also not have all its files embedded, but instead refer to other `NamedSetOfFiles` events through their build event identifiers. - -Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` target from the above graph, printed in protocol buffer’s JSON representation. The build event identifier contains the target as an opaque string and refers to the `Configuration` event using its build event identifier. The event does not announce any child events. The payload contains information about whether the target was built successfully, the set of output files, and the kind of target built. +The `PatternExpanded` build event provides insight +into which specific targets the `...` pattern expanded to: +`//foo:foo_lib` and `//foo:foo_test`. It does so by declaring two +`TargetConfigured` events as children. Note that the `TargetConfigured` event +declares the `Configuration` event as a child event, even though `Configuration` +has been posted before the `TargetConfigured` event. + +Besides the parent and child relationship, events may also refer to each other +using their build event identifiers. For example, in the above graph the +`TargetComplete` event refers to the `NamedSetOfFiles` event in its `fileSets` +field. + +Build events that refer to files don’t usually embed the file +names and paths in the event. Instead, they contain the build event identifier +of a `NamedSetOfFiles` event, which will then contain the actual file names and +paths. The `NamedSetOfFiles` event allows a set of files to be reported once and +referred to by many targets. This structure is necessary because otherwise in +some cases the Build Event Protocol output size would grow quadratically with +the number of files. A `NamedSetOfFiles` event may also not have all its files +embedded, but instead refer to other `NamedSetOfFiles` events through their +build event identifiers. + +Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` +target from the above graph, printed in protocol buffer’s JSON representation. +The build event identifier contains the target as an opaque string and refers to +the `Configuration` event using its build event identifier. The event does not +announce any child events. The payload contains information about whether the +target was built successfully, the set of output files, and the kind of target +built. ```json { @@ -71,9 +102,18 @@ Below is an instance of the `TargetComplete` event for the `//foo:foo_lib` targe ## Aspect Results in BEP -Ordinary builds evaluate actions associated with `(target, configuration)` pairs. When building with [aspects](/extending/aspects) enabled, Bazel additionally evaluates targets associated with `(target, configuration, aspect)` triples, for each target affected by a given enabled aspect. +Ordinary builds evaluate actions associated with `(target, configuration)` +pairs. When building with [aspects](/extending/aspects) enabled, Bazel +additionally evaluates targets associated with `(target, configuration, +aspect)` triples, for each target affected by a given enabled aspect. -Evaluation results for aspects are available in BEP despite the absence of aspect-specific event types. For each `(target, configuration)` pair with an applicable aspect, Bazel publishes an additional `TargetConfigured` and `TargetComplete` event bearing the result from applying the aspect to the target. For example, if `//:foo_lib` is built with `--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in the BEP: +Evaluation results for aspects are available in BEP despite the absence of +aspect-specific event types. For each `(target, configuration)` pair with an +applicable aspect, Bazel publishes an additional `TargetConfigured` and +`TargetComplete` event bearing the result from applying the aspect to the +target. For example, if `//:foo_lib` is built with +`--aspects=aspects/myaspect.bzl%custom_aspect`, this event would also appear in +the BEP: ```json { @@ -98,21 +138,37 @@ Evaluation results for aspects are available in BEP despite the absence of aspec } ``` -Note: The only difference between the IDs is the presence of the `aspect` field. A tool that does not check the `aspect` ID field and accumulates output files by target may conflate target outputs with aspect outputs. +Note: The only difference between the IDs is the presence of the `aspect` +field. A tool that does not check the `aspect` ID field and accumulates output +files by target may conflate target outputs with aspect outputs. ## Consuming `NamedSetOfFiles` -Determining the artifacts produced by a given target (or aspect) is a common BEP use-case that can be done efficiently with some preparation. This section discusses the recursive, shared structure offered by the `NamedSetOfFiles` event, which matches the structure of a Starlark [Depset](/extending/depsets). +Determining the artifacts produced by a given target (or aspect) is a common +BEP use-case that can be done efficiently with some preparation. This section +discusses the recursive, shared structure offered by the `NamedSetOfFiles` +event, which matches the structure of a Starlark [Depset](/extending/depsets). -Consumers must take care to avoid quadratic algorithms when processing `NamedSetOfFiles` events because large builds can contain tens of thousands of such events, requiring hundreds of millions operations in a traversal with quadratic complexity. +Consumers must take care to avoid quadratic algorithms when processing +`NamedSetOfFiles` events because large builds can contain tens of thousands of +such events, requiring hundreds of millions operations in a traversal with +quadratic complexity. ![namedsetoffiles-bep-graph](/docs/images/namedsetoffiles-bep-graph.png "NamedSetOfFiles BEP graph") **Figure 2.** `NamedSetOfFiles` BEP graph. -A `NamedSetOfFiles` event always appears in the BEP stream *before* a `TargetComplete` or `NamedSetOfFiles` event that references it. This is the inverse of the "parent-child" event relationship, where all but the first event appears after at least one event announcing it. A `NamedSetOfFiles` event is announced by a `Progress` event with no semantics. - -Given these ordering and sharing constraints, a typical consumer must buffer all `NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON event stream and Python code demonstrate how to populate a map from target/aspect to built artifacts in the "default" output group, and how to process the outputs for a subset of built targets/aspects: +A `NamedSetOfFiles` event always appears in the BEP stream *before* a +`TargetComplete` or `NamedSetOfFiles` event that references it. This is the +inverse of the "parent-child" event relationship, where all but the first event +appears after at least one event announcing it. A `NamedSetOfFiles` event is +announced by a `Progress` event with no semantics. + +Given these ordering and sharing constraints, a typical consumer must buffer all +`NamedSetOfFiles` events until the BEP stream is exhausted. The following JSON +event stream and Python code demonstrate how to populate a map from +target/aspect to built artifacts in the "default" output group, and how to +process the outputs for a subset of built targets/aspects: ```python named_sets = {} # type: dict[str, NamedSetOfFiles] diff --git a/remote/bep-glossary.mdx b/remote/bep-glossary.mdx index 66f77ea5..3bd11eea 100644 --- a/remote/bep-glossary.mdx +++ b/remote/bep-glossary.mdx @@ -2,13 +2,22 @@ title: 'Build Event Protocol Glossary' --- -Each BEP event type has its own semantics, minimally documented in [build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). The following glossary describes each event type. + + +Each BEP event type has its own semantics, minimally documented in +[build\_event\_stream.proto](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto). +The following glossary describes each event type. ## Aborted -Unlike other events, `Aborted` does not have a corresponding ID type, because the `Aborted` event *replaces* events of other types. This event indicates that the build terminated early and the event ID it appears under was not produced normally. `Aborted` contains an enum and human-friendly description to explain why the build did not complete. +Unlike other events, `Aborted` does not have a corresponding ID type, because +the `Aborted` event *replaces* events of other types. This event indicates that +the build terminated early and the event ID it appears under was not produced +normally. `Aborted` contains an enum and human-friendly description to explain +why the build did not complete. -For example, if a build is evaluating a target when the user interrupts Bazel, BEP contains an event like the following: +For example, if a build is evaluating a target when the user interrupts Bazel, +BEP contains an event like the following: ```json { @@ -28,21 +37,35 @@ For example, if a build is evaluating a target when the user interrupts Bazel, B ## ActionExecuted -Provides details about the execution of a specific [Action](/rules/lib/actions) in a build. By default, this event is included in the BEP only for failed actions, to support identifying the root cause of build failures. Users may set the `--build_event_publish_all_actions` flag to include all `ActionExecuted` events. +Provides details about the execution of a specific +[Action](/rules/lib/actions) in a build. By default, this event is +included in the BEP only for failed actions, to support identifying the root cause +of build failures. Users may set the `--build_event_publish_all_actions` flag +to include all `ActionExecuted` events. ## BuildFinished -A single `BuildFinished` event is sent after the command is complete and includes the exit code for the command. This event provides authoritative success/failure information. +A single `BuildFinished` event is sent after the command is complete and +includes the exit code for the command. This event provides authoritative +success/failure information. ## BuildMetadata -Contains the parsed contents of the `--build_metadata` flag. This event exists to support Bazel integration with other tooling by plumbing external data (such as identifiers). +Contains the parsed contents of the `--build_metadata` flag. This event exists +to support Bazel integration with other tooling by plumbing external data (such as +identifiers). ## BuildMetrics -A single `BuildMetrics` event is sent at the end of every command and includes counters/gauges useful for quantifying the build tool's behavior during the command. These metrics indicate work actually done and does not count cached work that is reused. +A single `BuildMetrics` event is sent at the end of every command and includes +counters/gauges useful for quantifying the build tool's behavior during the +command. These metrics indicate work actually done and does not count cached +work that is reused. -Note that `memory_metrics` may not be populated if there was no Java garbage collection during the command's execution. Users may set the `--memory_profile=/dev/null` option which forces the garbage collector to run at the end of the command to populate `memory_metrics`. +Note that `memory_metrics` may not be populated if there was no Java garbage +collection during the command's execution. Users may set the +`--memory_profile=/dev/null` option which forces the garbage +collector to run at the end of the command to populate `memory_metrics`. ```json { @@ -71,11 +94,14 @@ Note that `memory_metrics` may not be populated if there was no Java garbage col ## BuildStarted -The first event in a BEP stream, `BuildStarted` includes metadata describing the command before any meaningful work begins. +The first event in a BEP stream, `BuildStarted` includes metadata describing the +command before any meaningful work begins. ## BuildToolLogs -A single `BuildToolLogs` event is sent at the end of a command, including URIs of files generated by the build tool that may aid in understanding or debugging build tool behavior. Some information may be included inline. +A single `BuildToolLogs` event is sent at the end of a command, including URIs +of files generated by the build tool that may aid in understanding or debugging +build tool behavior. Some information may be included inline. ```json { @@ -104,15 +130,28 @@ A single `BuildToolLogs` event is sent at the end of a command, including URIs o ## CommandLine -The BEP contains multiple `CommandLine` events containing representations of all command-line arguments (including options and uninterpreted arguments). Each `CommandLine` event has a label in its `StructuredCommandLineId` that indicates which representation it conveys; three such events appear in the BEP: - -- `"original"`: Reconstructed commandline as Bazel received it from the Bazel client, without startup options sourced from .rc files. -- `"canonical"`: The effective commandline with .rc files expanded and invocation policy applied. -- `"tool"`: Populated from the `--experimental_tool_command_line` option. This is useful to convey the command-line of a tool wrapping Bazel through the BEP. This could be a base64-encoded `CommandLine` binary protocol buffer message which is used directly, or a string which is parsed but not interpreted (as the tool's options may differ from Bazel's). +The BEP contains multiple `CommandLine` events containing representations of all +command-line arguments (including options and uninterpreted arguments). +Each `CommandLine` event has a label in its `StructuredCommandLineId` that +indicates which representation it conveys; three such events appear in the BEP: + +* `"original"`: Reconstructed commandline as Bazel received it from the Bazel + client, without startup options sourced from .rc files. +* `"canonical"`: The effective commandline with .rc files expanded and + invocation policy applied. +* `"tool"`: Populated from the `--experimental_tool_command_line` option. This + is useful to convey the command-line of a tool wrapping Bazel through the BEP. + This could be a base64-encoded `CommandLine` binary protocol buffer message + which is used directly, or a string which is parsed but not interpreted (as + the tool's options may differ from Bazel's). ## Configuration -A `Configuration` event is sent for every [`configuration`](/extending/config) used in the top-level targets in a build. At least one configuration event is always be present. The `id` is reused by the `TargetConfigured` and `TargetComplete` event IDs and is necessary to disambiguate those events in multi-configuration builds. +A `Configuration` event is sent for every [`configuration`](/extending/config) +used in the top-level targets in a build. At least one configuration event is +always be present. The `id` is reused by the `TargetConfigured` and +`TargetComplete` event IDs and is necessary to disambiguate those events in +multi-configuration builds. ```json { @@ -137,7 +176,11 @@ A `Configuration` event is sent for every [`configuration`](/extending/config) u ## ConvenienceSymlinksIdentified -**Experimental.** If the `--experimental_convenience_symlinks_bep_event` option is set, a single `ConvenienceSymlinksIdentified` event is produced by `build` commands to indicate how symlinks in the workspace should be managed. This enables building tools that invoke Bazel remotely then arrange the local workspace as if Bazel had been run locally. +**Experimental.** If the `--experimental_convenience_symlinks_bep_event` +option is set, a single `ConvenienceSymlinksIdentified` event is produced by +`build` commands to indicate how symlinks in the workspace should be managed. +This enables building tools that invoke Bazel remotely then arrange the local +workspace as if Bazel had been run locally. ```json { @@ -168,17 +211,24 @@ A `Configuration` event is sent for every [`configuration`](/extending/config) u ## Fetch -Indicates that a Fetch operation occurred as a part of the command execution. Unlike other events, if a cached fetch result is re-used, this event does not appear in the BEP stream. +Indicates that a Fetch operation occurred as a part of the command execution. +Unlike other events, if a cached fetch result is re-used, this event does not +appear in the BEP stream. ## NamedSetOfFiles -`NamedSetOfFiles` events report a structure matching a [`depset`](/extending/depsets) of files produced during command evaluation. Transitively included depsets are identified by `NamedSetOfFilesId`. +`NamedSetOfFiles` events report a structure matching a +[`depset`](/extending/depsets) of files produced during command evaluation. +Transitively included depsets are identified by `NamedSetOfFilesId`. -For more information on interpreting a stream's `NamedSetOfFiles` events, see the [BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). +For more information on interpreting a stream's `NamedSetOfFiles` events, see the +[BEP examples page](/remote/bep-examples#consuming-namedsetoffiles). ## OptionsParsed -A single `OptionsParsed` event lists all options applied to the command, separating startup options from command options. It also includes the [InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. +A single `OptionsParsed` event lists all options applied to the command, +separating startup options from command options. It also includes the +[InvocationPolicy](/reference/command-line-reference#flag--invocation_policy), if any. ```json { @@ -214,7 +264,13 @@ A single `OptionsParsed` event lists all options applied to the command, separat ## PatternExpanded -`PatternExpanded` events indicate the set of all targets that match the patterns supplied on the commandline. For successful commands, a single event is present with all patterns in the `PatternExpandedId` and all targets in the `PatternExpanded` event's *children*. If the pattern expands to any `test_suite`s the set of test targets included by the `test_suite`. For each pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) event with a `PatternExpandedId` identifying the pattern. +`PatternExpanded` events indicate the set of all targets that match the patterns +supplied on the commandline. For successful commands, a single event is present +with all patterns in the `PatternExpandedId` and all targets in the +`PatternExpanded` event's *children*. If the pattern expands to any +`test_suite`s the set of test targets included by the `test_suite`. For each +pattern that fails to resolve, BEP contains an additional [`Aborted`](#aborted) +event with a `PatternExpandedId` identifying the pattern. ```json { @@ -238,11 +294,16 @@ A single `OptionsParsed` event lists all options applied to the command, separat ## Progress -Progress events contain the standard output and standard error produced by Bazel during command execution. These events are also auto-generated as needed to announce events that have not been announced by a logical "parent" event (in particular, [NamedSetOfFiles](#namedsetoffiles).) +Progress events contain the standard output and standard error produced by Bazel +during command execution. These events are also auto-generated as needed to +announce events that have not been announced by a logical "parent" event (in +particular, [NamedSetOfFiles](#namedsetoffiles).) ## TargetComplete -For each `(target, configuration, aspect)` combination that completes the execution phase, a `TargetComplete` event is included in BEP. The event contains the target's success/failure and the target's requested output groups. +For each `(target, configuration, aspect)` combination that completes the +execution phase, a `TargetComplete` event is included in BEP. The event contains +the target's success/failure and the target's requested output groups. ```json { @@ -272,9 +333,14 @@ For each `(target, configuration, aspect)` combination that completes the execut ## TargetConfigured -For each Target that completes the analysis phase, a `TargetConfigured` event is included in BEP. This is the authoritative source for a target's "rule kind" attribute. The configuration(s) applied to the target appear in the announced *children* of the event. +For each Target that completes the analysis phase, a `TargetConfigured` event is +included in BEP. This is the authoritative source for a target's "rule kind" +attribute. The configuration(s) applied to the target appear in the announced +*children* of the event. -For example, building with the `--experimental_multi_cpu` options may produce the following `TargetConfigured` event for a single target with two configurations: +For example, building with the `--experimental_multi_cpu` options may produce +the following `TargetConfigured` event for a single target with two +configurations: ```json { @@ -309,26 +375,42 @@ For example, building with the `--experimental_multi_cpu` options may produce th ## TargetSummary -For each `(target, configuration)` pair that is executed, a `TargetSummary` event is included with an aggregate success result encompassing the configured target's execution and all aspects applied to that configured target. +For each `(target, configuration)` pair that is executed, a `TargetSummary` +event is included with an aggregate success result encompassing the configured +target's execution and all aspects applied to that configured target. ## TestResult -If testing is requested, a `TestResult` event is sent for each test attempt, shard, and run per test. This allows BEP consumers to identify precisely which test actions failed their tests and identify the test outputs (such as logs, test.xml files) for each test action. +If testing is requested, a `TestResult` event is sent for each test attempt, +shard, and run per test. This allows BEP consumers to identify precisely which +test actions failed their tests and identify the test outputs (such as logs, +test.xml files) for each test action. ## TestSummary -If testing is requested, a `TestSummary` event is sent for each test `(target, configuration)`, containing information necessary to interpret the test's results. The number of attempts, shards and runs per test are included to enable BEP consumers to differentiate artifacts across these dimensions. The attempts and runs per test are considered while producing the aggregate `TestStatus` to differentiate `FLAKY` tests from `FAILED` tests. +If testing is requested, a `TestSummary` event is sent for each test `(target, +configuration)`, containing information necessary to interpret the test's +results. The number of attempts, shards and runs per test are included to enable +BEP consumers to differentiate artifacts across these dimensions. The attempts +and runs per test are considered while producing the aggregate `TestStatus` to +differentiate `FLAKY` tests from `FAILED` tests. ## UnstructuredCommandLine -Unlike [CommandLine](#commandline), this event carries the unparsed commandline flags in string form as encountered by the build tool after expanding all [`.bazelrc`](/run/bazelrc) files and considering the `--config` flag. +Unlike [CommandLine](#commandline), this event carries the unparsed commandline +flags in string form as encountered by the build tool after expanding all +[`.bazelrc`](/run/bazelrc) files and +considering the `--config` flag. -The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a given command execution. +The `UnstructuredCommandLine` event may be relied upon to precisely reproduce a +given command execution. ## WorkspaceConfig -A single `WorkspaceConfig` event contains configuration information regarding the workspace, such as the execution root. +A single `WorkspaceConfig` event contains configuration information regarding the +workspace, such as the execution root. ## WorkspaceStatus -A single `WorkspaceStatus` event contains the result of the [workspace status command](/docs/user-manual#workspace-status). +A single `WorkspaceStatus` event contains the result of the [workspace status +command](/docs/user-manual#workspace-status). diff --git a/remote/bep.mdx b/remote/bep.mdx index 4589f8b0..bafdaa9e 100644 --- a/remote/bep.mdx +++ b/remote/bep.mdx @@ -2,28 +2,67 @@ title: 'Build Event Protocol' --- -The [Build Event Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) (BEP) allows third-party programs to gain insight into a Bazel invocation. For example, you could use the BEP to gather information for an IDE plugin or a dashboard that displays build results. -The protocol is a set of [protocol buffer](https://developers.google.com/protocol-buffers/) messages with some semantics defined on top of it. It includes information about build and test results, build progress, the build configuration and much more. The BEP is intended to be consumed programmatically and makes parsing Bazel’s command line output a thing of the past. -The Build Event Protocol represents information about a build as events. A build event is a protocol buffer message consisting of a build event identifier, a set of child event identifiers, and a payload. - -- **Build Event Identifier:** Depending on the kind of build event, it might be an [opaque string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) or [structured information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) revealing more about the build event. A build event identifier is unique within a build. - -- **Children:** A build event may announce other build events, by including their build event identifiers in its [children field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). For example, the `PatternExpanded` build event announces the targets it expands to as children. The protocol guarantees that all events, except for the first event, are announced by a previous event. - -- **Payload:** The payload contains structured information about a build event, encoded as a protocol buffer message specific to that event. Note that the payload might not be the expected type, but could be an `Aborted` message if the build aborted prematurely. +The [Build Event +Protocol](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto) +(BEP) allows third-party programs to gain insight into a Bazel invocation. For +example, you could use the BEP to gather information for an IDE +plugin or a dashboard that displays build results. + +The protocol is a set of [protocol +buffer](https://developers.google.com/protocol-buffers/) messages with some +semantics defined on top of it. It includes information about build and test +results, build progress, the build configuration and much more. The BEP is +intended to be consumed programmatically and makes parsing Bazel’s +command line output a thing of the past. + +The Build Event Protocol represents information about a build as events. A +build event is a protocol buffer message consisting of a build event identifier, +a set of child event identifiers, and a payload. + +* __Build Event Identifier:__ Depending on the kind of build event, it might be +an [opaque +string](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L131-L140) +or [structured +information](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L194-L205) +revealing more about the build event. A build event identifier is unique within +a build. + +* __Children:__ A build event may announce other build events, by including +their build event identifiers in its [children +field](https://github.com/bazelbuild/bazel/blob/7.1.0/src/main/java/com/google/devtools/build/lib/buildeventstream/proto/build_event_stream.proto#L1276). +For example, the `PatternExpanded` build event announces the targets it expands +to as children. The protocol guarantees that all events, except for the first +event, are announced by a previous event. + +* __Payload:__ The payload contains structured information about a build event, +encoded as a protocol buffer message specific to that event. Note that the +payload might not be the expected type, but could be an `Aborted` message +if the build aborted prematurely. ### Build event graph -All build events form a directed acyclic graph through their parent and child relationship. Every build event except for the initial build event has one or more parent events. Please note that not all parent events of a child event must necessarily be posted before it. When a build is complete (succeeded or failed) all announced events will have been posted. In case of a Bazel crash or a failed network transport, some announced build events may never be posted. - -The event graph's structure reflects the lifecycle of a command. Every BEP graph has the following characteristic shape: - -1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) event. All other events are its descendants. -2. Immediate children of the BuildStarted event contain metadata about the command. -3. Events containing data produced by the command, such as files built and test results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) event. -4. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed by events containing summary information about the build (for example, metric or profiling data). +All build events form a directed acyclic graph through their parent and child +relationship. Every build event except for the initial build event has one or +more parent events. Please note that not all parent events of a child event must +necessarily be posted before it. When a build is complete (succeeded or failed) +all announced events will have been posted. In case of a Bazel crash or a failed +network transport, some announced build events may never be posted. + +The event graph's structure reflects the lifecycle of a command. Every BEP +graph has the following characteristic shape: + +1. The root event is always a [`BuildStarted`](/remote/bep-glossary#buildstarted) + event. All other events are its descendants. +1. Immediate children of the BuildStarted event contain metadata about the + command. +1. Events containing data produced by the command, such as files built and test + results, appear before the [`BuildFinished`](/remote/bep-glossary#buildfinished) + event. +1. The [`BuildFinished`](/remote/bep-glossary#buildfinished) event *may* be followed + by events containing summary information about the build (for example, metric + or profiling data). ## Consuming Build Event Protocol @@ -31,13 +70,21 @@ The event graph's structure reflects the lifecycle of a command. Every BEP graph To consume the BEP in a binary format: -1. Have Bazel serialize the protocol buffer messages to a file by specifying the option `--build_event_binary_file=/path/to/file`. The file will contain serialized protocol buffer messages with each message being length delimited. Each message is prefixed with its length encoded as a variable length integer. This format can be read using the protocol buffer library’s [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) method. +1. Have Bazel serialize the protocol buffer messages to a file by specifying the + option `--build_event_binary_file=/path/to/file`. The file will contain + serialized protocol buffer messages with each message being length delimited. + Each message is prefixed with its length encoded as a variable length integer. + This format can be read using the protocol buffer library’s + [`parseDelimitedFrom(InputStream)`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/AbstractParser#parseDelimitedFrom-java.io.InputStream-) + method. -2. Then, write a program that extracts the relevant information from the serialized protocol buffer message. +2. Then, write a program that extracts the relevant information from the + serialized protocol buffer message. ### Consume in text or JSON formats -The following Bazel command line flags will output the BEP in human-readable formats, such as text and JSON: +The following Bazel command line flags will output the BEP in +human-readable formats, such as text and JSON: ``` --build_event_text_file @@ -46,34 +93,56 @@ The following Bazel command line flags will output the BEP in human-readable for ## Build Event Service -The [Build Event Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event Service protocol is independent of the BEP and treats BEP events as opaque bytes. Bazel ships with a gRPC client implementation of the Build Event Service protocol that publishes Build Event Protocol events. One can specify the endpoint to send the events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, you must prefix the address with the appropriate scheme: `grpc://` for plaintext gRPC and `grpcs://` for gRPC with TLS enabled. +The [Build Event +Service](https://github.com/googleapis/googleapis/blob/master/google/devtools/build/v1/publish_build_event.proto) +Protocol is a generic [gRPC](https://www.grpc.io) service for publishing build events. The Build Event +Service protocol is independent of the BEP and treats BEP events as opaque bytes. +Bazel ships with a gRPC client implementation of the Build Event Service protocol that +publishes Build Event Protocol events. One can specify the endpoint to send the +events to using the `--bes_backend=HOST:PORT` flag. If your backend uses gRPC, +you must prefix the address with the appropriate scheme: `grpc://` for plaintext +gRPC and `grpcs://` for gRPC with TLS enabled. ### Build Event Service flags Bazel has several flags related to the Build Event Service protocol, including: -- `--bes_backend` -- `--[no]bes_lifecycle_events` -- `--bes_results_url` -- `--bes_timeout` -- `--bes_instance_name` +* `--bes_backend` +* `--[no]bes_lifecycle_events` +* `--bes_results_url` +* `--bes_timeout` +* `--bes_instance_name` -For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the +[Command-Line Reference](/reference/command-line-reference). ### Authentication and security -Bazel’s Build Event Service implementation also supports authentication and TLS. These settings can be controlled using the below flags. Please note that these flags are also used for Bazel’s Remote Execution. This implies that the Build Event Service and Remote Execution Endpoints need to share the same authentication and TLS infrastructure. +Bazel’s Build Event Service implementation also supports authentication and TLS. +These settings can be controlled using the below flags. Please note that these +flags are also used for Bazel’s Remote Execution. This implies that the Build +Event Service and Remote Execution Endpoints need to share the same +authentication and TLS infrastructure. -- `--[no]google_default_credentials` -- `--google_credentials` -- `--google_auth_scopes` -- `--tls_certificate` -- `--[no]tls_enabled` +* `--[no]google_default_credentials` +* `--google_credentials` +* `--google_auth_scopes` +* `--tls_certificate` +* `--[no]tls_enabled` -For a description of each of these flags, see the [Command-Line Reference](/reference/command-line-reference). +For a description of each of these flags, see the +[Command-Line Reference](/reference/command-line-reference). ### Build Event Service and remote caching -The BEP typically contains many references to log files (test.log, test.xml, etc. ) stored on the machine where Bazel is running. A remote BES server typically can't access these files as they are on different machines. A way to work around this issue is to use Bazel with [remote caching](/remote/caching). Bazel will upload all output files to the remote cache (including files referenced in the BEP) and the BES server can then fetch the referenced files from the cache. - -See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for more details. +The BEP typically contains many references to log files (test.log, test.xml, +etc. ) stored on the machine where Bazel is running. A remote BES server +typically can't access these files as they are on different machines. A way to +work around this issue is to use Bazel with [remote +caching](/remote/caching). +Bazel will upload all output files to the remote cache (including files +referenced in the BEP) and the BES server can then fetch the referenced files +from the cache. + +See [GitHub issue 3689](https://github.com/bazelbuild/bazel/issues/3689) for +more details. diff --git a/remote/cache-local.mdx b/remote/cache-local.mdx index 49b8d33b..a3415a64 100644 --- a/remote/cache-local.mdx +++ b/remote/cache-local.mdx @@ -2,23 +2,39 @@ title: 'Debugging Remote Cache Hits for Local Execution' --- -This page describes how to investigate cache misses in the context of local execution. -This page assumes that you have a build and/or test that successfully builds locally and is set up to utilize remote caching, and that you want to ensure that the remote cache is being effectively utilized. -For tips on how to check your cache hit rate and how to compare the execution logs between two Bazel invocations, see [Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). Everything presented in that guide also applies to remote caching with local execution. However, local execution presents some additional challenges. +This page describes how to investigate cache misses in the context of local +execution. + +This page assumes that you have a build and/or test that successfully builds +locally and is set up to utilize remote caching, and that you want to ensure +that the remote cache is being effectively utilized. + +For tips on how to check your cache hit rate and how to compare the execution +logs between two Bazel invocations, see +[Debugging Remote Cache Hits for Remote Execution](/remote/cache-remote). +Everything presented in that guide also applies to remote caching with local +execution. However, local execution presents some additional challenges. ## Checking your cache hit rate -Successful remote cache hits will show up in the status line, similar to [Cache Hits rate with Remote Execution](/remote/cache-remote#check-cache-hits). +Successful remote cache hits will show up in the status line, similar to +[Cache Hits rate with Remote +Execution](/remote/cache-remote#check-cache-hits). -In the standard output of your Bazel run, you will see something like the following: +In the standard output of your Bazel run, you will see something like the +following: -```none +```none {:.devsite-disable-click-to-copy} INFO: 7 processes: 3 remote cache hit, 4 linux-sandbox. ``` -This means that out of 7 attempted actions, 3 got a remote cache hit and 4 actions did not have cache hits and were executed locally using `linux-sandbox` strategy. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. +This means that out of 7 attempted actions, 3 got a remote cache hit and 4 +actions did not have cache hits and were executed locally using `linux-sandbox` +strategy. Local cache hits are not included in this summary. If you are getting +0 processes (or a number lower than expected), run `bazel clean` followed by +your build/test command. ## Troubleshooting cache hits @@ -26,30 +42,50 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure successful communication with the remote endpoint -To ensure your build is successfully communicating with the remote cache, follow the steps in this section. +To ensure your build is successfully communicating with the remote cache, follow +the steps in this section. 1. Check your output for warnings - With remote execution, a failure to talk to the remote endpoint would fail your build. On the other hand, a cacheable local build would not fail if it cannot cache. Check the output of your Bazel invocation for warnings, such as: + With remote execution, a failure to talk to the remote endpoint would fail + your build. On the other hand, a cacheable local build would not fail if it + cannot cache. Check the output of your Bazel invocation for warnings, such + as: - ```none + ```none WARNING: Error reading from the remote cache: ``` + or - ```none + ```none WARNING: Error writing to the remote cache: ``` - Such warnings will be followed by the error message detailing the connection problem that should help you debug: for example, mistyped endpoint name or incorrectly set credentials. Find and address any such errors. If the error message you see does not give you enough information, try adding `--verbose_failures`. -2. Follow the steps from [Troubleshooting cache hits for remote execution](/remote/cache-remote#troubleshooting_cache_hits) to ensure that your cache-writing Bazel invocations are able to get cache hits on the same machine and across machines. + Such warnings will be followed by the error message detailing the connection + problem that should help you debug: for example, mistyped endpoint name or + incorrectly set credentials. Find and address any such errors. If the error + message you see does not give you enough information, try adding + `--verbose_failures`. + +2. Follow the steps from [Troubleshooting cache hits for remote + execution](/remote/cache-remote#troubleshooting_cache_hits) to + ensure that your cache-writing Bazel invocations are able to get cache hits + on the same machine and across machines. 3. Ensure your cache-reading Bazel invocations can get cache hits. - a. Since cache-reading Bazel invocations will have a different command-line set up, take additional care to ensure that they are properly set up to communicate with the remote cache. Ensure the `--remote_cache` flag is set and there are no warnings in the output. + a. Since cache-reading Bazel invocations will have a different command-line set + up, take additional care to ensure that they are properly set up to + communicate with the remote cache. Ensure the `--remote_cache` flag is set + and there are no warnings in the output. - b. Ensure your cache-reading Bazel invocations build the same targets as the cache-writing Bazel invocations. + b. Ensure your cache-reading Bazel invocations build the same targets as the + cache-writing Bazel invocations. - c. Follow the same steps as to [ensure caching across machines](/remote/cache-remote#caching-across-machines), to ensure caching from your cache-writing Bazel invocation to your cache-reading Bazel invocation. + c. Follow the same steps as to [ensure caching across + machines](/remote/cache-remote#caching-across-machines), + to ensure caching from your cache-writing Bazel invocation to your + cache-reading Bazel invocation. diff --git a/remote/cache-remote.mdx b/remote/cache-remote.mdx index 3b96276a..896f1e58 100644 --- a/remote/cache-remote.mdx +++ b/remote/cache-remote.mdx @@ -2,21 +2,36 @@ title: 'Debugging Remote Cache Hits for Remote Execution' --- -This page describes how to check your cache hit rate and how to investigate cache misses in the context of remote execution. -This page assumes that you have a build and/or test that successfully utilizes remote execution, and you want to ensure that you are effectively utilizing remote cache. + +This page describes how to check your cache hit rate and how to investigate +cache misses in the context of remote execution. + +This page assumes that you have a build and/or test that successfully +utilizes remote execution, and you want to ensure that you are effectively +utilizing remote cache. ## Checking your cache hit rate -In the standard output of your Bazel run, look at the `INFO` line that lists processes, which roughly correspond to Bazel actions. That line details where the action was run. Look for the `remote` label, which indicates an action executed remotely, `linux-sandbox` for actions executed in a local sandbox, and other values for other execution strategies. An action whose result came from a remote cache is displayed as `remote cache hit`. +In the standard output of your Bazel run, look at the `INFO` line that lists +processes, which roughly correspond to Bazel actions. That line details +where the action was run. Look for the `remote` label, which indicates an action +executed remotely, `linux-sandbox` for actions executed in a local sandbox, +and other values for other execution strategies. An action whose result came +from a remote cache is displayed as `remote cache hit`. For example: -```none +```none {:.devsite-disable-click-to-copy} INFO: 11 processes: 6 remote cache hit, 3 internal, 2 remote. ``` -In this example there were 6 remote cache hits, and 2 actions did not have cache hits and were executed remotely. The 3 internal part can be ignored. It is typically tiny internal actions, such as creating symbolic links. Local cache hits are not included in this summary. If you are getting 0 processes (or a number lower than expected), run `bazel clean` followed by your build/test command. +In this example there were 6 remote cache hits, and 2 actions did not have +cache hits and were executed remotely. The 3 internal part can be ignored. +It is typically tiny internal actions, such as creating symbolic links. Local +cache hits are not included in this summary. If you are getting 0 processes +(or a number lower than expected), run `bazel clean` followed by your build/test +command. ## Troubleshooting cache hits @@ -24,47 +39,80 @@ If you are not getting the cache hit rate you are expecting, do the following: ### Ensure re-running the same build/test command produces cache hits -1. Run the build(s) and/or test(s) that you expect to populate the cache. The first time a new build is run on a particular stack, you can expect no remote cache hits. As part of remote execution, action results are stored in the cache and a subsequent run should pick them up. +1. Run the build(s) and/or test(s) that you expect to populate the cache. The + first time a new build is run on a particular stack, you can expect no remote + cache hits. As part of remote execution, action results are stored in the + cache and a subsequent run should pick them up. -2. Run `bazel clean`. This command cleans your local cache, which allows you to investigate remote cache hits without the results being masked by local cache hits. +2. Run `bazel clean`. This command cleans your local cache, which allows + you to investigate remote cache hits without the results being masked by + local cache hits. -3. Run the build(s) and test(s) that you are investigating again (on the same machine). +3. Run the build(s) and test(s) that you are investigating again (on the same + machine). -4. Check the `INFO` line for cache hit rate. If you see no processes except `remote cache hit` and `internal`, then your cache is being correctly populated and accessed. In that case, skip to the next section. +4. Check the `INFO` line for cache hit rate. If you see no processes except + `remote cache hit` and `internal`, then your cache is being correctly populated and + accessed. In that case, skip to the next section. -5. A likely source of discrepancy is something non-hermetic in the build causing the actions to receive different action keys across the two runs. To find those actions, do the following: +5. A likely source of discrepancy is something non-hermetic in the build causing + the actions to receive different action keys across the two runs. To find + those actions, do the following: a. Re-run the build(s) or test(s) in question to obtain execution logs: - ```posix-terminal - bazel clean + ```posix-terminal + bazel clean - bazel <var>--optional-flags</var> build //<var>your:target</var> --execution_log_compact_file=/tmp/exec1.log - ``` + bazel --optional-flags build //your:target --execution_log_compact_file=/tmp/exec1.log + ``` - b. [Compare the execution logs](#compare-logs) between the two runs. Ensure that the actions are identical across the two log files. Discrepancies provide a clue about the changes that occurred between the runs. Update your build to eliminate those discrepancies. + b. [Compare the execution logs](#compare-logs) between the + two runs. Ensure that the actions are identical across the two log files. + Discrepancies provide a clue about the changes that occurred between the + runs. Update your build to eliminate those discrepancies. - If you are able to resolve the caching problems and now the repeated run produces all cache hits, skip to the next section. + If you are able to resolve the caching problems and now the repeated run + produces all cache hits, skip to the next section. - If your action IDs are identical but there are no cache hits, then something in your configuration is preventing caching. Continue with this section to check for common problems. + If your action IDs are identical but there are no cache hits, then something + in your configuration is preventing caching. Continue with this section to + check for common problems. -6. Check that all actions in the execution log have `cacheable` set to true. If `cacheable` does not appear in the execution log for a give action, that means that the corresponding rule may have a `no-cache` tag in its definition in the `BUILD` file. Look at the `mnemonic` and `target_label` fields in the execution log to help determine where the action is coming from. +5. Check that all actions in the execution log have `cacheable` set to true. If + `cacheable` does not appear in the execution log for a give action, that + means that the corresponding rule may have a `no-cache` tag in its + definition in the `BUILD` file. Look at the `mnemonic` and `target_label` + fields in the execution log to help determine where the action is coming + from. -7. If the actions are identical and `cacheable` but there are no cache hits, it is possible that your command line includes `--noremote_accept_cached` which would disable cache lookups for a build. +6. If the actions are identical and `cacheable` but there are no cache hits, it + is possible that your command line includes `--noremote_accept_cached` which + would disable cache lookups for a build. - If figuring out the actual command line is difficult, use the canonical command line from the [Build Event Protocol](/remote/bep) as follows: + If figuring out the actual command line is difficult, use the canonical + command line from the + [Build Event Protocol](/remote/bep) + as follows: - a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get the text version of the log. + a. Add `--build_event_text_file=/tmp/bep.txt` to your Bazel command to get + the text version of the log. - b. Open the text version of the log and search for the `structured_command_line` message with `command_line_label: "canonical"`. It will list all the options after expansion. + b. Open the text version of the log and search for the + `structured_command_line` message with `command_line_label: "canonical"`. + It will list all the options after expansion. c. Search for `remote_accept_cached` and check whether it's set to `false`. - d. If `remote_accept_cached` is `false`, determine where it is being set to `false`: either at the command line or in a [bazelrc](/run/bazelrc#bazelrc-file-locations) file. + d. If `remote_accept_cached` is `false`, determine where it is being + set to `false`: either at the command line or in a + [bazelrc](/run/bazelrc#bazelrc-file-locations) file. ### Ensure caching across machines -After cache hits are happening as expected on the same machine, run the same build(s)/test(s) on a different machine. If you suspect that caching is not happening across machines, do the following: +After cache hits are happening as expected on the same machine, run the +same build(s)/test(s) on a different machine. If you suspect that caching is +not happening across machines, do the following: 1. Make a small modification to your build to avoid hitting existing caches. @@ -76,7 +124,8 @@ After cache hits are happening as expected on the same machine, run the same bui bazel ... build ... --execution_log_compact_file=/tmp/exec1.log ``` -3. Run the build on the second machine, ensuring the modification from step 1 is included: +3. Run the build on the second machine, ensuring the modification from step 1 + is included: ```posix-terminal bazel clean @@ -84,34 +133,47 @@ After cache hits are happening as expected on the same machine, run the same bui bazel ... build ... --execution_log_compact_file=/tmp/exec2.log ``` -4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two runs. If the logs are not identical, investigate your build configurations for discrepancies as well as properties from the host environment leaking into either of the builds. +4. [Compare the execution logs](#compare-logs-the-execution-logs) for the two + runs. If the logs are not identical, investigate your build configurations + for discrepancies as well as properties from the host environment leaking + into either of the builds. ## Comparing the execution logs -The execution log contains records of actions executed during the build. Each record describes both the inputs (not only files, but also command line arguments, environment variables, etc) and the outputs of the action. Thus, examination of the log can reveal why an action was reexecuted. +The execution log contains records of actions executed during the build. +Each record describes both the inputs (not only files, but also command line +arguments, environment variables, etc) and the outputs of the action. Thus, +examination of the log can reveal why an action was reexecuted. -The execution log can be produced in one of three formats: compact (`--execution_log_compact_file`), binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). The compact format is recommended, as it produces much smaller files with very little runtime overhead. The following instructions work for any format. You can also convert between them using the `//src/tools/execlog:converter` tool. +The execution log can be produced in one of three formats: +compact (`--execution_log_compact_file`), +binary (`--execution_log_binary_file`) or JSON (`--execution_log_json_file`). +The compact format is recommended, as it produces much smaller files with very +little runtime overhead. The following instructions work for any format. You +can also convert between them using the `//src/tools/execlog:converter` tool. -To compare logs for two builds that are not sharing cache hits as expected, do the following: +To compare logs for two builds that are not sharing cache hits as expected, +do the following: -1. Get the execution logs from each build and store them as `/tmp/exec1.log` and `/tmp/exec2.log`. +1. Get the execution logs from each build and store them as `/tmp/exec1.log` and + `/tmp/exec2.log`. -2. Download the Bazel source code and build the `//src/tools/execlog:parser` tool: +2. Download the Bazel source code and build the `//src/tools/execlog:parser` + tool: - ``` - git clone https://github.com/bazelbuild/bazel.git - cd bazel - bazel build //src/tools/execlog:parser - ``` + git clone https://github.com/bazelbuild/bazel.git + cd bazel + bazel build //src/tools/execlog:parser -3. Use the `//src/tools/execlog:parser` tool to convert the logs into a human-readable text format. In this format, the actions in the second log are sorted to match the order in the first log, making a comparison easier. +3. Use the `//src/tools/execlog:parser` tool to convert the logs into a + human-readable text format. In this format, the actions in the second log are + sorted to match the order in the first log, making a comparison easier. - ``` - bazel-bin/src/tools/execlog/parser \ - --log_path=/tmp/exec1.log \ - --log_path=/tmp/exec2.log \ - --output_path=/tmp/exec1.log.txt \ - --output_path=/tmp/exec2.log.txt - ``` + bazel-bin/src/tools/execlog/parser \ + --log_path=/tmp/exec1.log \ + --log_path=/tmp/exec2.log \ + --output_path=/tmp/exec1.log.txt \ + --output_path=/tmp/exec2.log.txt -4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and `/tmp/exec2.log.txt`. +4. Use your favourite text differ to diff `/tmp/exec1.log.txt` and + `/tmp/exec2.log.txt`. diff --git a/remote/caching.mdx b/remote/caching.mdx index 1729ee02..f85f2c8e 100644 --- a/remote/caching.mdx +++ b/remote/caching.mdx @@ -2,65 +2,94 @@ title: 'Remote Caching' --- -This page covers remote caching, setting up a server to host the cache, and running builds using the remote cache. -A remote cache is used by a team of developers and/or a continuous integration (CI) system to share build outputs. If your build is reproducible, the outputs from one machine can be safely reused on another machine, which can make builds significantly faster. + +This page covers remote caching, setting up a server to host the cache, and +running builds using the remote cache. + +A remote cache is used by a team of developers and/or a continuous integration +(CI) system to share build outputs. If your build is reproducible, the +outputs from one machine can be safely reused on another machine, which can +make builds significantly faster. ## Overview -Bazel breaks a build into discrete steps, which are called actions. Each action has inputs, output names, a command line, and environment variables. Required inputs and expected outputs are declared explicitly for each action. +Bazel breaks a build into discrete steps, which are called actions. Each action +has inputs, output names, a command line, and environment variables. Required +inputs and expected outputs are declared explicitly for each action. -You can set up a server to be a remote cache for build outputs, which are these action outputs. These outputs consist of a list of output file names and the hashes of their contents. With a remote cache, you can reuse build outputs from another user's build rather than building each new output locally. +You can set up a server to be a remote cache for build outputs, which are these +action outputs. These outputs consist of a list of output file names and the +hashes of their contents. With a remote cache, you can reuse build outputs +from another user's build rather than building each new output locally. To use remote caching: -- Set up a server as the cache's backend -- Configure the Bazel build to use the remote cache -- Use Bazel version 0.10.0 or later +* Set up a server as the cache's backend +* Configure the Bazel build to use the remote cache +* Use Bazel version 0.10.0 or later The remote cache stores two types of data: -- The action cache, which is a map of action hashes to action result metadata. -- A content-addressable store (CAS) of output files. +* The action cache, which is a map of action hashes to action result metadata. +* A content-addressable store (CAS) of output files. -Note that the remote cache additionally stores the stdout and stderr for every action. Inspecting the stdout/stderr of Bazel thus is not a good signal for [estimating cache hits](/remote/cache-local). +Note that the remote cache additionally stores the stdout and stderr for every +action. Inspecting the stdout/stderr of Bazel thus is not a good signal for +[estimating cache hits](/remote/cache-local). ### How a build uses remote caching -Once a server is set up as the remote cache, you use the cache in multiple ways: - -- Read and write to the remote cache -- Read and/or write to the remote cache except for specific targets -- Only read from the remote cache -- Not use the remote cache at all - -When you run a Bazel build that can read and write to the remote cache, the build follows these steps: - -1. Bazel creates the graph of targets that need to be built, and then creates a list of required actions. Each of these actions has declared inputs and output filenames. -2. Bazel checks your local machine for existing build outputs and reuses any that it finds. -3. Bazel checks the cache for existing build outputs. If the output is found, Bazel retrieves the output. This is a cache hit. -4. For required actions where the outputs were not found, Bazel executes the actions locally and creates the required build outputs. +Once a server is set up as the remote cache, you use the cache in multiple +ways: + +* Read and write to the remote cache +* Read and/or write to the remote cache except for specific targets +* Only read from the remote cache +* Not use the remote cache at all + +When you run a Bazel build that can read and write to the remote cache, +the build follows these steps: + +1. Bazel creates the graph of targets that need to be built, and then creates +a list of required actions. Each of these actions has declared inputs +and output filenames. +2. Bazel checks your local machine for existing build outputs and reuses any +that it finds. +3. Bazel checks the cache for existing build outputs. If the output is found, +Bazel retrieves the output. This is a cache hit. +4. For required actions where the outputs were not found, Bazel executes the +actions locally and creates the required build outputs. 5. New build outputs are uploaded to the remote cache. ## Setting up a server as the cache's backend -You need to set up a server to act as the cache's backend. A HTTP/1.1 server can treat Bazel's data as opaque bytes and so many existing servers can be used as a remote caching backend. Bazel's [HTTP Caching Protocol](#http-caching) is what supports remote caching. +You need to set up a server to act as the cache's backend. A HTTP/1.1 +server can treat Bazel's data as opaque bytes and so many existing servers +can be used as a remote caching backend. Bazel's +[HTTP Caching Protocol](#http-caching) is what supports remote +caching. -You are responsible for choosing, setting up, and maintaining the backend server that will store the cached outputs. When choosing a server, consider: +You are responsible for choosing, setting up, and maintaining the backend +server that will store the cached outputs. When choosing a server, consider: -- Networking speed. For example, if your team is in the same office, you may want to run your own local server. -- Security. The remote cache will have your binaries and so needs to be secure. -- Ease of management. For example, Google Cloud Storage is a fully managed service. +* Networking speed. For example, if your team is in the same office, you may +want to run your own local server. +* Security. The remote cache will have your binaries and so needs to be secure. +* Ease of management. For example, Google Cloud Storage is a fully managed service. -There are many backends that can be used for a remote cache. Some options include: +There are many backends that can be used for a remote cache. Some options +include: -- [nginx](#nginx) -- [bazel-remote](#bazel-remote) -- [Google Cloud Storage](#cloud-storage) +* [nginx](#nginx) +* [bazel-remote](#bazel-remote) +* [Google Cloud Storage](#cloud-storage) ### nginx -nginx is an open source web server. With its \[WebDAV module], it can be used as a remote cache for Bazel. On Debian and Ubuntu you can install the `nginx-extras` package. On macOS nginx is available via Homebrew: +nginx is an open source web server. With its [WebDAV module], it can be +used as a remote cache for Bazel. On Debian and Ubuntu you can install the +`nginx-extras` package. On macOS nginx is available via Homebrew: ```posix-terminal brew tap denji/nginx @@ -68,7 +97,12 @@ brew tap denji/nginx brew install nginx-full --with-webdav ``` -Below is an example configuration for nginx. Note that you will need to change `/path/to/cache/dir` to a valid directory where nginx has permission to write and read. You may need to change `client_max_body_size` option to a larger value if you have larger output files. The server will require other configuration such as authentication. +Below is an example configuration for nginx. Note that you will need to +change `/path/to/cache/dir` to a valid directory where nginx has permission +to write and read. You may need to change `client_max_body_size` option to a +larger value if you have larger output files. The server will require other +configuration such as authentication. + Example configuration for `server` section in `nginx.conf`: @@ -88,44 +122,72 @@ location /cache/ { ### bazel-remote -bazel-remote is an open source remote build cache that you can use on your infrastructure. It has been successfully used in production at several companies since early 2018. Note that the Bazel project does not provide technical support for bazel-remote. +bazel-remote is an open source remote build cache that you can use on +your infrastructure. It has been successfully used in production at +several companies since early 2018. Note that the Bazel project does +not provide technical support for bazel-remote. -This cache stores contents on disk and also provides garbage collection to enforce an upper storage limit and clean unused artifacts. The cache is available as a \[docker image] and its code is available on [GitHub](https://github.com/buchgr/bazel-remote/). Both the REST and gRPC remote cache APIs are supported. +This cache stores contents on disk and also provides garbage collection +to enforce an upper storage limit and clean unused artifacts. The cache is +available as a [docker image] and its code is available on +[GitHub](https://github.com/buchgr/bazel-remote/). +Both the REST and gRPC remote cache APIs are supported. -Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) page for instructions on how to use it. +Refer to the [GitHub](https://github.com/buchgr/bazel-remote/) +page for instructions on how to use it. ### Google Cloud Storage -\[Google Cloud Storage] is a fully managed object store which provides an HTTP API that is compatible with Bazel's remote caching protocol. It requires that you have a Google Cloud account with billing enabled. +[Google Cloud Storage] is a fully managed object store which provides an +HTTP API that is compatible with Bazel's remote caching protocol. It requires +that you have a Google Cloud account with billing enabled. To use Cloud Storage as the cache: -1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). Ensure that you select a bucket location that's closest to you, as network bandwidth is important for the remote cache. +1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). +Ensure that you select a bucket location that's closest to you, as network bandwidth +is important for the remote cache. -2. Create a service account for Bazel to authenticate to Cloud Storage. See [Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). +2. Create a service account for Bazel to authenticate to Cloud Storage. See +[Creating a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating_a_service_account). -3. Generate a secret JSON key and then pass it to Bazel for authentication. Store the key securely, as anyone with the key can read and write arbitrary data to/from your GCS bucket. +3. Generate a secret JSON key and then pass it to Bazel for authentication. Store +the key securely, as anyone with the key can read and write arbitrary data +to/from your GCS bucket. 4. Connect to Cloud Storage by adding the following flags to your Bazel command: + * Pass the following URL to Bazel by using the flag: + `--remote_cache=https://storage.googleapis.com/bucket-name` where `bucket-name` is the name of your storage bucket. + * Pass the authentication key using the flag: `--google_credentials=/path/to/your/secret-key.json`, or + `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). - - Pass the following URL to Bazel by using the flag: `--remote_cache=https://storage.googleapis.com<var>/bucket-name</var>` where `bucket-name` is the name of your storage bucket. - - Pass the authentication key using the flag: `--google_credentials=<var>/path/to/your/secret-key</var>.json`, or `--google_default_credentials` to use [Application Authentication](https://cloud.google.com/docs/authentication/production). - -5. You can configure Cloud Storage to automatically delete old files. To do so, see [Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). +5. You can configure Cloud Storage to automatically delete old files. To do so, see +[Managing Object Lifecycles](https://cloud.google.com/storage/docs/managing-lifecycles). ### Other servers -You can set up any HTTP/1.1 server that supports PUT and GET as the cache's backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), [Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). +You can set up any HTTP/1.1 server that supports PUT and GET as the cache's +backend. Users have reported success with caching backends such as [Hazelcast](https://hazelcast.com), +[Apache httpd](http://httpd.apache.org), and [AWS S3](https://aws.amazon.com/s3). ## Authentication -As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. You can pass a username and password to Bazel via the remote cache URL. The syntax is `https://username:password@hostname.com:port/path`. Note that HTTP Basic Authentication transmits username and password in plaintext over the network and it's thus critical to always use it with HTTPS. +As of version 0.11.0 support for HTTP Basic Authentication was added to Bazel. +You can pass a username and password to Bazel via the remote cache URL. The +syntax is `https://username:password@hostname.com:port/path`. Note that +HTTP Basic Authentication transmits username and password in plaintext over the +network and it's thus critical to always use it with HTTPS. ## HTTP caching protocol -Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. Action result metadata is stored under the path `/ac/` and output files are stored under the path `/cas/`. +Bazel supports remote caching via HTTP/1.1. The protocol is conceptually simple: +Binary data (BLOB) is uploaded via PUT requests and downloaded via GET requests. +Action result metadata is stored under the path `/ac/` and output files are stored +under the path `/cas/`. -For example, consider a remote cache running under `http://localhost:8080/cache`. A Bazel request to download action result metadata for an action with the SHA256 hash `01ba4719...` will look as follows: +For example, consider a remote cache running under `http://localhost:8080/cache`. +A Bazel request to download action result metadata for an action with the SHA256 +hash `01ba4719...` will look as follows: ```http GET /cache/ac/01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b HTTP/1.1 @@ -134,7 +196,8 @@ Accept: */* Connection: Keep-Alive ``` -A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to the CAS will look as follows: +A Bazel request to upload an output file with the SHA256 hash `15e2b0d3...` to +the CAS will look as follows: ```http PUT /cache/cas/15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225 HTTP/1.1 @@ -148,29 +211,36 @@ Connection: Keep-Alive ## Run Bazel using the remote cache -Once a server is set up as the remote cache, to use the remote cache you need to add flags to your Bazel command. See list of configurations and their flags below. +Once a server is set up as the remote cache, to use the remote cache you +need to add flags to your Bazel command. See list of configurations and +their flags below. -You may also need configure authentication, which is specific to your chosen server. +You may also need configure authentication, which is specific to your +chosen server. -You may want to add these flags in a `.bazelrc` file so that you don't need to specify them every time you run Bazel. Depending on your project and team dynamics, you can add flags to a `.bazelrc` file that is: +You may want to add these flags in a `.bazelrc` file so that you don't +need to specify them every time you run Bazel. Depending on your project and +team dynamics, you can add flags to a `.bazelrc` file that is: -- On your local machine -- In your project's workspace, shared with the team -- On the CI system +* On your local machine +* In your project's workspace, shared with the team +* On the CI system ### Read from and write to the remote cache -Take care in who has the ability to write to the remote cache. You may want only your CI system to be able to write to the remote cache. +Take care in who has the ability to write to the remote cache. You may want +only your CI system to be able to write to the remote cache. Use the following flag to read from and write to the remote cache: ```posix-terminal -build --remote_cache=http://<var>your.host:port</var> +build --remote_cache=http://{{ '' }}your.host:port{{ '' }} ``` Besides `HTTP`, the following protocols are also supported: `HTTPS`, `grpc`, `grpcs`. -Use the following flag in addition to the one above to only read from the remote cache: +Use the following flag in addition to the one above to only read from the +remote cache: ```posix-terminal build --remote_upload_local_results=false @@ -178,7 +248,8 @@ build --remote_upload_local_results=false ### Exclude specific targets from using the remote cache -To exclude specific targets from using the remote cache, tag the target with `no-remote-cache`. For example: +To exclude specific targets from using the remote cache, tag the target with +`no-remote-cache`. For example: ```starlark java_library( @@ -189,100 +260,121 @@ java_library( ### Delete content from the remote cache -Deleting content from the remote cache is part of managing your server. How you delete content from the remote cache depends on the server you have set up as the cache. When deleting outputs, either delete the entire cache, or delete old outputs. +Deleting content from the remote cache is part of managing your server. +How you delete content from the remote cache depends on the server you have +set up as the cache. When deleting outputs, either delete the entire cache, +or delete old outputs. -The cached outputs are stored as a set of names and hashes. When deleting content, there's no way to distinguish which output belongs to a specific build. +The cached outputs are stored as a set of names and hashes. When deleting +content, there's no way to distinguish which output belongs to a specific +build. You may want to delete content from the cache to: -- Create a clean cache after a cache was poisoned -- Reduce the amount of storage used by deleting old outputs +* Create a clean cache after a cache was poisoned +* Reduce the amount of storage used by deleting old outputs ### Unix sockets -The remote HTTP cache supports connecting over unix domain sockets. The behavior is similar to curl's `--unix-socket` flag. Use the following to configure unix domain socket: +The remote HTTP cache supports connecting over unix domain sockets. The behavior +is similar to curl's `--unix-socket` flag. Use the following to configure unix +domain socket: ```posix-terminal - build --remote_cache=http://<var>your.host:port</var> - build --remote_proxy=unix:/<var>path/to/socket</var> + build --remote_cache=http://{{ '' }}your.host:port{{ '' }} + build --remote_proxy=unix:/{{ '' }}path/to/socket{{ '' }} ``` This feature is unsupported on Windows. ## Disk cache -Bazel can use a directory on the file system as a remote cache. This is useful for sharing build artifacts when switching branches and/or working on multiple workspaces of the same project, such as multiple checkouts. Enable the disk cache as follows: +Bazel can use a directory on the file system as a remote cache. This is +useful for sharing build artifacts when switching branches and/or working +on multiple workspaces of the same project, such as multiple checkouts. +Enable the disk cache as follows: ```posix-terminal -build --disk_cache=<var>path/to/build/cache</var> +build --disk_cache={{ '' }}path/to/build/cache{{ '' }} ``` -You can pass a user-specific path to the `--disk_cache` flag using the `~` alias (Bazel will substitute the current user's home directory). This comes in handy when enabling the disk cache for all developers of a project via the project's checked in `.bazelrc` file. +You can pass a user-specific path to the `--disk_cache` flag using the `~` alias +(Bazel will substitute the current user's home directory). This comes in handy +when enabling the disk cache for all developers of a project via the project's +checked in `.bazelrc` file. ### Garbage collection -Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and `--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache or for the age of individual cache entries. Bazel will automatically garbage collect the disk cache while idling between builds; the idle timer can be set with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). +Starting with Bazel 7.4, you can use `--experimental_disk_cache_gc_max_size` and +`--experimental_disk_cache_gc_max_age` to set a maximum size for the disk cache +or for the age of individual cache entries. Bazel will automatically garbage +collect the disk cache while idling between builds; the idle timer can be set +with `--experimental_disk_cache_gc_idle_delay` (defaulting to 5 minutes). -As an alternative to automatic garbage collection, we also provide a [tool](https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a garbage collection on demand. +As an alternative to automatic garbage collection, we also provide a [tool]( +https://github.com/bazelbuild/bazel/tree/master/src/tools/diskcache) to run a +garbage collection on demand. ## Known issues **Input file modification during a build** -When an input file is modified during a build, Bazel might upload invalid results to the remote cache. You can enable a change detection with the `--experimental_guard_against_concurrent_changes` flag. There are no known issues and it will be enabled by default in a future release. See \[issue #3360] for updates. Generally, avoid modifying source files during a build. +When an input file is modified during a build, Bazel might upload invalid +results to the remote cache. You can enable a change detection with +the `--experimental_guard_against_concurrent_changes` flag. There +are no known issues and it will be enabled by default in a future release. +See [issue #3360] for updates. Generally, avoid modifying source files during a +build. **Environment variables leaking into an action** -An action definition contains environment variables. This can be a problem for sharing remote cache hits across machines. For example, environments with different `$PATH` variables won't share cache hits. Only environment variables explicitly whitelisted via `--action_env` are included in an action definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` with a whitelist of environment variables including `$PATH`. If you are getting fewer cache hits than expected, check that your environment doesn't have an old `/etc/bazel.bazelrc` file. +An action definition contains environment variables. This can be a problem for +sharing remote cache hits across machines. For example, environments with +different `$PATH` variables won't share cache hits. Only environment variables +explicitly whitelisted via `--action_env` are included in an action +definition. Bazel's Debian/Ubuntu package used to install `/etc/bazel.bazelrc` +with a whitelist of environment variables including `$PATH`. If you are getting +fewer cache hits than expected, check that your environment doesn't have an old +`/etc/bazel.bazelrc` file. **Bazel does not track tools outside a workspace** -Bazel currently does not track tools outside a workspace. This can be a problem if, for example, an action uses a compiler from `/usr/bin/`. Then, two users with different compilers installed will wrongly share cache hits because the outputs are different but they have the same action hash. See [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. +Bazel currently does not track tools outside a workspace. This can be a +problem if, for example, an action uses a compiler from `/usr/bin/`. Then, +two users with different compilers installed will wrongly share cache hits +because the outputs are different but they have the same action hash. See +[issue #4558](https://github.com/bazelbuild/bazel/issues/4558) for updates. -**Incremental in-memory state is lost when running builds inside docker containers** Bazel uses server/client architecture even when running in single docker container. On the server side, Bazel maintains an in-memory state which speeds up builds. When running builds inside docker containers such as in CI, the in-memory state is lost and Bazel must rebuild it before using the remote cache. +**Incremental in-memory state is lost when running builds inside docker containers** +Bazel uses server/client architecture even when running in single docker container. +On the server side, Bazel maintains an in-memory state which speeds up builds. +When running builds inside docker containers such as in CI, the in-memory state is lost +and Bazel must rebuild it before using the remote cache. ## External links -- **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. - -- **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) in which he benchmarks remote caching in Bazel. - -- [Adapting Rules for Remote Execution](/remote/rules) - -- [Troubleshooting Remote Execution](/remote/sandbox) - -- [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) - -- [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) - -- [bazel-remote](https://github.com/buchgr/bazel-remote/) - -- [Google Cloud Storage](https://cloud.google.com/storage) - -- [Google Cloud Console](https://cloud.google.com/console) - -- [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) - -- [Hazelcast](https://hazelcast.com) - -- [Apache httpd](http://httpd.apache.org) - -- [AWS S3](https://aws.amazon.com/s3) - -- [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) - -- [gRPC](https://grpc.io/) - -- [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) - -- [Buildbarn](https://github.com/buildbarn) - -- [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) - -- [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) - -- [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) - -- [Application Authentication](https://cloud.google.com/docs/authentication/production) - -- [NativeLink](https://github.com/TraceMachina/nativelink) +* **Your Build in a Datacenter:** The Bazel team gave a [talk](https://fosdem.org/2018/schedule/event/datacenter_build/) about remote caching and execution at FOSDEM 2018. + +* **Faster Bazel builds with remote caching: a benchmark:** Nicolò Valigi wrote a [blog post](https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html) +in which he benchmarks remote caching in Bazel. + +* [Adapting Rules for Remote Execution](/remote/rules) +* [Troubleshooting Remote Execution](/remote/sandbox) +* [WebDAV module](https://nginx.org/en/docs/http/ngx_http_dav_module.html) +* [Docker image](https://hub.docker.com/r/buchgr/bazel-remote-cache/) +* [bazel-remote](https://github.com/buchgr/bazel-remote/) +* [Google Cloud Storage](https://cloud.google.com/storage) +* [Google Cloud Console](https://cloud.google.com/console) +* [Bucket locations](https://cloud.google.com/storage/docs/bucket-locations) +* [Hazelcast](https://hazelcast.com) +* [Apache httpd](http://httpd.apache.org) +* [AWS S3](https://aws.amazon.com/s3) +* [issue #3360](https://github.com/bazelbuild/bazel/issues/3360) +* [gRPC](https://grpc.io/) +* [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) +* [Buildbarn](https://github.com/buildbarn) +* [Buildfarm](https://github.com/bazelbuild/bazel-buildfarm) +* [BuildGrid](https://gitlab.com/BuildGrid/buildgrid) +* [issue #4558](https://github.com/bazelbuild/bazel/issues/4558) +* [Application Authentication](https://cloud.google.com/docs/authentication/production) +* [NativeLink](https://github.com/TraceMachina/nativelink) diff --git a/remote/ci.mdx b/remote/ci.mdx index d2b1a97f..0a4e4488 100644 --- a/remote/ci.mdx +++ b/remote/ci.mdx @@ -2,24 +2,37 @@ title: 'Configuring Bazel CI to Test Rules for Remote Execution' --- -This page is for owners and maintainers of Bazel rule repositories. It describes how to configure the Bazel Continuous Integration (CI) system for your repository to test your rules for compatibility against a remote execution scenario. The instructions on this page apply to projects stored in GitHub repositories. + + +This page is for owners and maintainers of Bazel rule repositories. It +describes how to configure the Bazel Continuous Integration (CI) system for +your repository to test your rules for compatibility against a remote execution +scenario. The instructions on this page apply to projects stored in +GitHub repositories. ## Prerequisites Before completing the steps on this page, ensure the following: -- Your GitHub repository is part of the [Bazel GitHub organization](https://github.com/bazelbuild). -- You have configured Buildkite for your repository as described in [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). +* Your GitHub repository is part of the + [Bazel GitHub organization](https://github.com/bazelbuild). +* You have configured Buildkite for your repository as described in + [Bazel Continuous Integration](https://github.com/bazelbuild/continuous-integration/tree/master/buildkite). ## Setting up the Bazel CI for testing -1. In your `.bazelci/presubmit.yml` file, do the following: +1. In your `.bazelci/presubmit.yml` file, do the following: - a. Add a config named `rbe_ubuntu1604`. + a. Add a config named `rbe_ubuntu1604`. - b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. + b. In the `rbe_ubuntu1604` config, add the build and test targets you want to test against remote execution. -2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) GitHub repository to your `WORKSPACE` file, pinned to the [latest release](https://releases.bazel.build/bazel-toolchains.html). Also add an `rbe_autoconfig` target with name `buildkite_config`. This example creates toolchain configuration for remote execution with BuildKite CI for `rbe_ubuntu1604`. +2. Add the[`bazel-toolchains`](https://github.com/bazelbuild/bazel-toolchains) + GitHub repository to your `WORKSPACE` file, pinned to the + [latest release](https://releases.bazel.build/bazel-toolchains.html). Also + add an `rbe_autoconfig` target with name `buildkite_config`. This example + creates toolchain configuration for remote execution with BuildKite CI + for `rbe_ubuntu1604`. ```posix-terminal load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") @@ -27,25 +40,43 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig") rbe_autoconfig(name = "buildkite_config") ``` -3. Send a pull request with your changes to the `presubmit.yml` file. (See [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) +3. Send a pull request with your changes to the `presubmit.yml` file. (See + [example pull request](https://github.com/bazelbuild/rules_rust/commit/db141526d89d00748404856524cedd7db8939c35).) -4. To view build results, click **Details** for the RBE (Ubuntu 16.04) pull request check in GitHub, as shown in the figure below. This link becomes available after the pull request has been merged and the CI tests have run. (See [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) +4. To view build results, click **Details** for the RBE (Ubuntu + 16.04) pull request check in GitHub, as shown in the figure below. This link + becomes available after the pull request has been merged and the CI tests + have run. (See + [example results](https://source.cloud.google.com/results/invocations/375e325c-0a05-47af-87bd-fed1363e0333).) - ![Example results](/docs/images/rbe-ci-1.png "Example results") + ![Example results](/docs/images/rbe-ci-1.png "Example results") -5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test required to pass before merging in your branch protection rule. The setting is located in GitHub in **Settings > Branches > Branch protection rules**, as shown in the following figure. +5. (Optional) Set the **bazel test (RBE (Ubuntu 16.04))** check as a test + required to pass before merging in your branch protection rule. The setting + is located in GitHub in **Settings > Branches > Branch protection rules**, + as shown in the following figure. - ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") + ![Branch protection rules settings](/docs/images/rbe-ci-2.png "Branch protection rules") ## Troubleshooting failed builds and tests If your build or tests fail, it's likely due to the following: -- **Required build or test tools are not installed in the default container.** Builds using the `rbe_ubuntu1604` config run by default inside an [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container, which includes tools common to many Bazel builds. However, if your rules require tools not present in the default container, you must create a custom container based on the [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) container and include those tools as described later. +* **Required build or test tools are not installed in the default container.** + Builds using the `rbe_ubuntu1604` config run by default inside an + [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) + container, which includes tools common to many Bazel builds. However, if + your rules require tools not present in the default container, you must + create a custom container based on the + [`rbe-ubuntu16-04`](https://console.cloud.google.com/marketplace/details/google/rbe-ubuntu16-04) + container and include those tools as described later. -- **Build or test targets are using rules that are incompatible with remote execution.** See [Adapting Bazel Rules for Remote Execution](/remote/rules) for details about compatibility with remote execution. +* **Build or test targets are using rules that are incompatible with remote + execution.** See + [Adapting Bazel Rules for Remote Execution](/remote/rules) for + details about compatibility with remote execution. -## Using a custom container in the rbe\_ubuntu1604 CI config +## Using a custom container in the rbe_ubuntu1604 CI config The `rbe-ubuntu16-04` container is publicly available at the following URL: @@ -53,99 +84,121 @@ The `rbe-ubuntu16-04` container is publicly available at the following URL: http://gcr.io/cloud-marketplace/google/rbe-ubuntu16-04 ``` -You can pull it directly from Container Registry or build it from source. The next sections describe both options. +You can pull it directly from Container Registry or build it from source. The +next sections describe both options. -Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. If you are building the container from source, you must also install the latest version of Bazel. +Before you begin, make sure you have installed `gcloud`, `docker`, and `git`. +If you are building the container from source, you must also install the latest +version of Bazel. ### Pulling the rbe-ubuntu16-04 from Container Registry -To pull the `rbe-ubuntu16-04` container from Container Registry, run the following command: +To pull the `rbe-ubuntu16-04` container from Container Registry, run the +following command: ```posix-terminal -gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> +gcloud docker -- pull gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} ``` -Replace \{\{ '`' }}sha256-checksum{{ '`' \}\} with the SHA256 checksum value for [the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). +Replace sha256-checksum with the SHA256 checksum value for +[the latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). ### Building the rbe-ubuntu16-04 container from source To build the `rbe-ubuntu16-04` container from source, do the following: -1. Clone the `bazel-toolchains` repository: +1. Clone the `bazel-toolchains` repository: - ```posix-terminal - git clone https://github.com/bazelbuild/bazel-toolchains - ``` + ```posix-terminal + git clone https://github.com/bazelbuild/bazel-toolchains + ``` -2. Set up toolchain container targets and build the container as explained in [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). +2. Set up toolchain container targets and build the container as explained in + [Toolchain Containers](https://github.com/bazelbuild/bazel-toolchains/tree/master/container). -3. Pull the freshly built container: +3. Pull the freshly built container: - ```posix-terminal - ``` - -gcloud docker -- pull gcr.io/\{\{ '`' }}project-id{{ '`' \}\}/\{\{ '`' }}custom-container-name{{ '`' \}\}\{\{ '`' }}sha256-checksum{{ '`' \}\} \`\`\` + ```posix-terminal +gcloud docker -- pull gcr.io/project-id/custom-container-namesha256-checksum + ``` ### Running the custom container To run the custom container, do one of the following: -- If you pulled the container from Container Registry, run the following command: +* If you pulled the container from Container Registry, run the following + command: - ```posix-terminal - docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var>/bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:sha256-checksum/bin/bash + ``` - Replace `sha256-checksum` with the SHA256 checksum value for the [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). + Replace `sha256-checksum` with the SHA256 checksum value for the + [latest container](https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/rbe-ubuntu16-04). -- If you built the container from source, run the following command: +* If you built the container from source, run the following command: - ```posix-terminal - docker run -it gcr.io/<var>project-id</var>/<var>custom-container-name</var>@sha256:<var>sha256sum</var> /bin/bash - ``` + ```posix-terminal + docker run -it gcr.io/project-id/custom-container-name@sha256:sha256sum /bin/bash + ``` ### Adding resources to the custom container -Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or [`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or alternate versions of the original resources to the `rbe-ubuntu16-04` container. If you are new to Docker, read the following: +Use a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) or +[`rules_docker`](https://github.com/bazelbuild/rules_docker) to add resources or +alternate versions of the original resources to the `rbe-ubuntu16-04` container. +If you are new to Docker, read the following: -- [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) -- [Docker Samples](https://docs.docker.com/samples/) +* [Docker for beginners](https://github.com/docker/labs/tree/master/beginner) +* [Docker Samples](https://docs.docker.com/samples/) -For example, the following `Dockerfile` snippet installs `<var>my_tool_package</var>`: +For example, the following `Dockerfile` snippet installs `my_tool_package`: ``` -FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:<var>sha256-checksum</var> -RUN apt-get update && yes | apt-get install -y <var>my_tool_package</var> +FROM gcr.io/cloud-marketplace/google/rbe-ubuntu16-04@sha256:{{ '' }}sha256-checksum{{ '' }} +RUN apt-get update && yes | apt-get install -y {{ '' }}my_tool_package{{ '' }} ``` ### Pushing the custom container to Container Registry -Once you have customized the container, build the container image and push it to Container Registry as follows: +Once you have customized the container, build the container image and push it to +Container Registry as follows: 1. Build the container image: - ```posix-terminal - docker build -t <var>custom-container-name</var>. + ```posix-terminal + docker build -t custom-container-name. + + docker tag custom-container-name gcr.io/project-id/custom-container-name + ``` - docker tag <var>custom-container-name</var> gcr.io/<var>project-id</var>/<var>custom-container-name</var> - ``` +2. Push the container image to Container Registry: -2. Push the container image to Container Registry: + ```posix-terminal + gcloud docker -- push gcr.io/project-id/custom-container-name + ``` - ```posix-terminal - gcloud docker -- push gcr.io/<var>project-id</var>/<var>custom-container-name</var> - ``` +3. Navigate to the following URL to verify the container has been pushed: -3. Navigate to the following URL to verify the container has been pushed: + https://console.cloud.google.com/gcr/images/project-id/GLOBAL/custom-container-name - [https://console.cloud.google.com/gcr/images/\{\{](https://console.cloud.google.com/gcr/images/%7B%7B) '`' }}project-id{{ '`' \}\}/GLOBAL/\{\{ '`' }}custom-container-name{{ '`' \}\} +4. Take note of the SHA256 checksum of your custom container. You will need to + provide it in your build platform definition later. -4. Take note of the SHA256 checksum of your custom container. You will need to provide it in your build platform definition later. +5. Configure the container for public access as described in publicly + accessible as explained in + [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). -5. Configure the container for public access as described in publicly accessible as explained in [Serving images publicly](https://cloud.google.com/container-registry/docs/access-control#serving_images_publicly). + For more information, see + [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). - For more information, see [Pushing and Pulling Images](https://cloud.google.com/container-registry/docs/pushing-and-pulling). ### Specifying the build platform definition -You must include a [Bazel platform](/extending/platforms) configuration in your custom toolchain configuration, which allows Bazel to select a toolchain appropriate to the desired hardware/software platform. To generate automatically a valid platform, you can add to your `WORKSPACE` an `rbe_autoconfig` target with name `buildkite_config` which includes additional attrs to select your custom container. For details on this setup, read the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). +You must include a [Bazel platform](/extending/platforms) configuration in your +custom toolchain configuration, which allows Bazel to select a toolchain +appropriate to the desired hardware/software platform. To generate +automatically a valid platform, you can add to your `WORKSPACE` an +`rbe_autoconfig` target with name `buildkite_config` which includes additional +attrs to select your custom container. For details on this setup, read +the up-to-date documentation for [`rbe_autoconfig`](https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl). diff --git a/remote/creating.mdx b/remote/creating.mdx index 4b2cf226..0e46a07a 100644 --- a/remote/creating.mdx +++ b/remote/creating.mdx @@ -2,30 +2,49 @@ title: 'Creating Persistent Workers' --- -[Persistent workers](/remote/persistent) can make your build faster. If you have repeated actions in your build that have a high startup cost or would benefit from cross-action caching, you may want to implement your own persistent worker to perform these actions. -The Bazel server communicates with the worker using `stdin`/`stdout`. It supports the use of protocol buffers or JSON strings. + +[Persistent workers](/remote/persistent) can make your build faster. If +you have repeated actions in your build that have a high startup cost or would +benefit from cross-action caching, you may want to implement your own persistent +worker to perform these actions. + +The Bazel server communicates with the worker using `stdin`/`stdout`. It +supports the use of protocol buffers or JSON strings. The worker implementation has two parts: -- The [worker](#making-worker). -- The [rule that uses the worker](#rule-uses-worker). +* The [worker](#making-worker). +* The [rule that uses the worker](#rule-uses-worker). ## Making the worker A persistent worker upholds a few requirements: -- It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) from its `stdin`. -- It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) (and only `WorkResponse`s) to its `stdout`. -- It accepts the `--persistent_worker` flag. The wrapper must recognize the `--persistent_worker` command-line flag and only make itself persistent if that flag is passed, otherwise it must do a one-shot compilation and exit. +* It reads + [WorkRequests](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L36) + from its `stdin`. +* It writes + [WorkResponses](https://github.com/bazelbuild/bazel/blob/54a547f30fd582933889b961df1d6e37a3e33d85/src/main/protobuf/worker_protocol.proto#L77) + (and only `WorkResponse`s) to its `stdout`. +* It accepts the `--persistent_worker` flag. The wrapper must recognize the + `--persistent_worker` command-line flag and only make itself persistent if + that flag is passed, otherwise it must do a one-shot compilation and exit. -If your program upholds these requirements, it can be used as a persistent worker! +If your program upholds these requirements, it can be used as a persistent +worker! ### Work requests -A `WorkRequest` contains a list of arguments to the worker, a list of path-digest pairs representing the inputs the worker can access (this isn’t enforced, but you can use this info for caching), and a request id, which is 0 for singleplex workers. +A `WorkRequest` contains a list of arguments to the worker, a list of +path-digest pairs representing the inputs the worker can access (this isn’t +enforced, but you can use this info for caching), and a request id, which is 0 +for singleplex workers. -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). This document uses camel case in the JSON examples, but snake case when talking about the field regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), +the JSON protocol uses "camel case" (`requestId`). This document uses camel case +in the JSON examples, but snake case when talking about the field regardless of +protocol. ```json { @@ -38,13 +57,24 @@ NOTE: While the protocol buffer specification uses "snake case" (`request_id`), } ``` -The optional `verbosity` field can be used to request extra debugging output from the worker. It is entirely up to the worker what and how to output. Higher values indicate more verbose output. Passing the `--worker_verbose` flag to Bazel sets the `verbosity` field to 10, but smaller or larger values can be used manually for different amounts of output. +The optional `verbosity` field can be used to request extra debugging output +from the worker. It is entirely up to the worker what and how to output. Higher +values indicate more verbose output. Passing the `--worker_verbose` flag to +Bazel sets the `verbosity` field to 10, but smaller or larger values can be used +manually for different amounts of output. -The optional `sandbox_dir` field is used only by workers that support [multiplex sandboxing](/remote/multiplex). +The optional `sandbox_dir` field is used only by workers that support +[multiplex sandboxing](/remote/multiplex). ### Work responses -A `WorkResponse` contains a request id, a zero or nonzero exit code, and an output message describing any errors encountered in processing or executing the request. A worker should capture the `stdout` and `stderr` of any tool it calls and report them through the `WorkResponse`. Writing it to the `stdout` of the worker process is unsafe, as it will interfere with the worker protocol. Writing it to the `stderr` of the worker process is safe, but the result is collected in a per-worker log file instead of ascribed to individual actions. +A `WorkResponse` contains a request id, a zero or nonzero exit code, and an +output message describing any errors encountered in processing or executing +the request. A worker should capture the `stdout` and `stderr` of any tool it +calls and report them through the `WorkResponse`. Writing it to the `stdout` of +the worker process is unsafe, as it will interfere with the worker protocol. +Writing it to the `stderr` of the worker process is safe, but the result is +collected in a per-worker log file instead of ascribed to individual actions. ```json { @@ -55,7 +85,10 @@ A `WorkResponse` contains a request id, a zero or nonzero exit code, and an outp } ``` -As per the norm for protobufs, all fields are optional. However, Bazel requires the `WorkRequest` and the corresponding `WorkResponse`, to have the same request id, so the request id must be specified if it is nonzero. This is a valid `WorkResponse`. +As per the norm for protobufs, all fields are optional. However, Bazel requires +the `WorkRequest` and the corresponding `WorkResponse`, to have the same request +id, so the request id must be specified if it is nonzero. This is a valid +`WorkResponse`. ```json { @@ -63,35 +96,69 @@ As per the norm for protobufs, all fields are optional. However, Bazel requires } ``` -A `request_id` of 0 indicates a "singleplex" request, used when this request cannot be processed in parallel with other requests. The server guarantees that a given worker receives requests with either only `request_id` 0 or only `request_id` greater than zero. Singleplex requests are sent in serial, for example if the server doesn't send another request until it has received a response (except for cancel requests, see below). +A `request_id` of 0 indicates a "singleplex" request, used when this request +cannot be processed in parallel with other requests. The server guarantees that +a given worker receives requests with either only `request_id` 0 or only +`request_id` greater than zero. Singleplex requests are sent in serial, for +example if the server doesn't send another request until it has received a +response (except for cancel requests, see below). **Notes** -- Each protocol buffer is preceded by its length in `varint` format (see [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). -- JSON requests and responses are not preceded by a size indicator. -- JSON requests uphold the same structure as the protobuf, but use standard JSON and use camel case for all field names. -- In order to maintain the same backward and forward compatibility properties as protobuf, JSON workers must tolerate unknown fields in these messages, and use the protobuf defaults for missing values. -- Bazel stores requests as protobufs and converts them to JSON using [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) +* Each protocol buffer is preceded by its length in `varint` format (see + [`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). +* JSON requests and responses are not preceded by a size indicator. +* JSON requests uphold the same structure as the protobuf, but use standard + JSON and use camel case for all field names. +* In order to maintain the same backward and forward compatibility properties + as protobuf, JSON workers must tolerate unknown fields in these messages, + and use the protobuf defaults for missing values. +* Bazel stores requests as protobufs and converts them to JSON using + [protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) ### Cancellation -Workers can optionally allow work requests to be cancelled before they finish. This is particularly useful in connection with dynamic execution, where local execution can regularly be interrupted by a faster remote execution. To allow cancellation, add `supports-worker-cancellation: 1` to the `execution-requirements` field (see below) and set the `--experimental_worker_cancellation` flag. - -A **cancel request** is a `WorkRequest` with the `cancel` field set (and similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` field set). The only other field that must be in a cancel request or cancel response is `request_id`, indicating which request to cancel. The `request_id` field will be 0 for singleplex workers or the non-0 `request_id` of a previously sent `WorkRequest` for multiplex workers. The server may send cancel requests for requests that the worker has already responded to, in which case the cancel request must be ignored. - -Each non-cancel `WorkRequest` message must be answered exactly once, whether or not it was cancelled. Once the server has sent a cancel request, the worker may respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` field set to true. Sending a regular `WorkResponse` is also accepted, but the `output` and `exit_code` fields will be ignored. - -Once a response has been sent for a `WorkRequest`, the worker must not touch the files in its working directory. The server is free to clean up the files, including temporary files. +Workers can optionally allow work requests to be cancelled before they finish. +This is particularly useful in connection with dynamic execution, where local +execution can regularly be interrupted by a faster remote execution. To allow +cancellation, add `supports-worker-cancellation: 1` to the +`execution-requirements` field (see below) and set the +`--experimental_worker_cancellation` flag. + +A **cancel request** is a `WorkRequest` with the `cancel` field set (and +similarly a **cancel response** is a `WorkResponse` with the `was_cancelled` +field set). The only other field that must be in a cancel request or cancel +response is `request_id`, indicating which request to cancel. The `request_id` +field will be 0 for singleplex workers or the non-0 `request_id` of a previously +sent `WorkRequest` for multiplex workers. The server may send cancel requests +for requests that the worker has already responded to, in which case the cancel +request must be ignored. + +Each non-cancel `WorkRequest` message must be answered exactly once, whether or +not it was cancelled. Once the server has sent a cancel request, the worker may +respond with a `WorkResponse` with the `request_id` set and the `was_cancelled` +field set to true. Sending a regular `WorkResponse` is also accepted, but the +`output` and `exit_code` fields will be ignored. + +Once a response has been sent for a `WorkRequest`, the worker must not touch the +files in its working directory. The server is free to clean up the files, +including temporary files. ## Making the rule that uses the worker -You'll also need to create a rule that generates actions to be performed by the worker. Making a Starlark rule that uses a worker is just like [creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). +You'll also need to create a rule that generates actions to be performed by the +worker. Making a Starlark rule that uses a worker is just like +[creating any other rule](https://github.com/bazelbuild/examples/tree/master/rules). -In addition, the rule needs to contain a reference to the worker itself, and there are some requirements for the actions it produces. +In addition, the rule needs to contain a reference to the worker itself, and +there are some requirements for the actions it produces. ### Referring to the worker -The rule that uses the worker needs to contain a field that refers to the worker itself, so you'll need to create an instance of a `\*\_binary` rule to define your worker. If your worker is called `MyWorker.Java`, this might be the associated rule: +The rule that uses the worker needs to contain a field that refers to the worker +itself, so you'll need to create an instance of a `\*\_binary` rule to define +your worker. If your worker is called `MyWorker.Java`, this might be the +associated rule: ```python java_binary( @@ -100,9 +167,12 @@ java_binary( ) ``` -This creates the "worker" label, which refers to the worker binary. You'll then define a rule that *uses* the worker. This rule should define an attribute that refers to the worker binary. +This creates the "worker" label, which refers to the worker binary. You'll then +define a rule that *uses* the worker. This rule should define an attribute that +refers to the worker binary. -If the worker binary you built is in a package named "work", which is at the top level of the build, this might be the attribute definition: +If the worker binary you built is in a package named "work", which is at the top +level of the build, this might be the attribute definition: ```python "worker": attr.label( @@ -112,25 +182,46 @@ If the worker binary you built is in a package named "work", which is at the top ) ``` -`cfg = "exec"` indicates that the worker should be built to run on your execution platform rather than on the target platform (i.e., the worker is used as tool during the build). +`cfg = "exec"` indicates that the worker should be built to run on your +execution platform rather than on the target platform (i.e., the worker is used +as tool during the build). ### Work action requirements -The rule that uses the worker creates actions for the worker to perform. These actions have a couple of requirements. +The rule that uses the worker creates actions for the worker to perform. These +actions have a couple of requirements. -- The *"arguments"* field. This takes a list of strings, all but the last of which are arguments passed to the worker upon startup. The last element in the "arguments" list is a `flag-file` (@-preceded) argument. Workers read the arguments from the specified flagfile on a per-WorkRequest basis. Your rule can write non-startup arguments for the worker to this flagfile. +* The *"arguments"* field. This takes a list of strings, all but the last of + which are arguments passed to the worker upon startup. The last element in + the "arguments" list is a `flag-file` (@-preceded) argument. Workers read + the arguments from the specified flagfile on a per-WorkRequest basis. Your + rule can write non-startup arguments for the worker to this flagfile. -- The *"execution-requirements"* field, which takes a dictionary containing `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. +* The *"execution-requirements"* field, which takes a dictionary containing + `"supports-workers" : "1"`, `"supports-multiplex-workers" : "1"`, or both. - The "arguments" and "execution-requirements" fields are required for all actions sent to workers. Additionally, actions that should be executed by JSON workers need to include `"requires-worker-protocol" : "json"` in the execution requirements field. `"requires-worker-protocol" : "proto"` is also a valid execution requirement, though it’s not required for proto workers, since they are the default. + The "arguments" and "execution-requirements" fields are required for all + actions sent to workers. Additionally, actions that should be executed by + JSON workers need to include `"requires-worker-protocol" : "json"` in the + execution requirements field. `"requires-worker-protocol" : "proto"` is also + a valid execution requirement, though it’s not required for proto workers, + since they are the default. - You can also set a `worker-key-mnemonic` in the execution requirements. This may be useful if you're reusing the executable for multiple action types and want to distinguish actions by this worker. + You can also set a `worker-key-mnemonic` in the execution requirements. This + may be useful if you're reusing the executable for multiple action types and + want to distinguish actions by this worker. -- Temporary files generated in the course of the action should be saved to the worker's directory. This enables sandboxing. +* Temporary files generated in the course of the action should be saved to the + worker's directory. This enables sandboxing. -Note: To pass an argument starting with a literal `@`, start the argument with `@@` instead. If an argument is also an external repository label, it will not be considered a flagfile argument. +Note: To pass an argument starting with a literal `@`, start the argument with +`@@` instead. If an argument is also an external repository label, it will not +be considered a flagfile argument. -Assuming a rule definition with "worker" attribute described above, in addition to a "srcs" attribute representing the inputs, an "output" attribute representing the outputs, and an "args" attribute representing the worker startup args, the call to `ctx.actions.run` might be: +Assuming a rule definition with "worker" attribute described above, in addition +to a "srcs" attribute representing the inputs, an "output" attribute +representing the outputs, and an "args" attribute representing the worker +startup args, the call to `ctx.actions.run` might be: ```python ctx.actions.run( @@ -145,14 +236,26 @@ ctx.actions.run( ) ``` -For another example, see [Implementing persistent workers](/remote/persistent#implementation). +For another example, see +[Implementing persistent workers](/remote/persistent#implementation). ## Examples -The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), in addition to an [example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) that is used in our integration tests. +The Bazel code base uses +[Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), +in addition to an +[example JSON worker](https://github.com/bazelbuild/bazel/blob/c65f768fec9889bbf1ee934c61d0dc061ea54ca2/src/test/java/com/google/devtools/build/lib/worker/ExampleWorker.java) +that is used in our integration tests. -You can use their [scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) to make any Java-based tool into a worker by passing in the correct callback. +You can use their +[scaffolding](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/main/java/com/google/devtools/build/lib/worker/WorkRequestHandler.java) +to make any Java-based tool into a worker by passing in the correct callback. -For an example of a rule that uses a worker, take a look at Bazel's [worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). +For an example of a rule that uses a worker, take a look at Bazel's +[worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). -External contributors have implemented workers in a variety of languages; take a look at [Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). You can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest\&type=Code)! +External contributors have implemented workers in a variety of languages; take a +look at +[Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). +You can +[find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)! diff --git a/remote/dynamic.mdx b/remote/dynamic.mdx deleted file mode 100644 index 002d7373..00000000 --- a/remote/dynamic.mdx +++ /dev/null @@ -1,63 +0,0 @@ ---- -title: 'Dynamic Execution' ---- - -**Dynamic execution** is a feature in Bazel where local and remote execution of the same action are started in parallel, using the output from the first branch that finishes, cancelling the other branch. It combines the execution power and/or large shared cache of a remote build system with the low latency of local execution, providing the best of both worlds for clean and incremental builds alike. - -This page describes how to enable, tune, and debug dynamic execution. If you have both local and remote execution set up and are trying to adjust Bazel settings for better performance, this page is for you. If you don't already have remote execution set up, go to the Bazel [Remote Execution Overview](/remote/rbe) first. - -## Enabling dynamic execution? - -The dynamic execution module is part of Bazel, but to make use of dynamic execution, you must already be able to compile both locally and remotely from the same Bazel setup. - -To enable the dynamic execution module, pass the `--internal_spawn_scheduler` flag to Bazel. This adds a new execution strategy called `dynamic`. You can now use this as your strategy for the mnemonics you want to run dynamically, such as `--strategy=Javac=dynamic`. See the next section for how to pick which mnemonics to enable dynamic execution for. - -For any mnemonic using the dynamic strategy, the remote execution strategies are taken from the `--dynamic_remote_strategy` flag, and local strategies from the `--dynamic_local_strategy` flag. Passing `--dynamic_local_strategy=worker,sandboxed` sets the default for the local branch of dynamic execution to try with workers or sandboxed execution in that order. Passing `--dynamic_local_strategy=Javac=worker` overrides the default for the Javac mnemonic only. The remote version works the same way. Both flags can be specified multiple times. If an action cannot be executed locally, it is executed remotely as normal, and vice-versa. - -If your remote system has a cache, the `--dynamic_local_execution_delay` flag adds a delay in milliseconds to the local execution after the remote system has indicated a cache hit. This avoids running local execution when more cache hits are likely. The default value is 1000ms, but should be tuned to being just a bit longer than cache hits usually take. The actual time depends both on the remote system and on how long a round-trip takes. Usually, the value will be the same for all users of a given remote system, unless some of them are far enough away to add roundtrip latency. You can use the [Bazel profiling features](/rules/performance#performance-profiling) to look at how long typical cache hits take. - -Dynamic execution can be used with local sandboxed strategy as well as with [persistent workers](/remote/persistent). Persistent workers will automatically run with sandboxing when used with dynamic execution, and cannot use [multiplex workers](/remote/multiplex). On Darwin and Windows systems, the sandboxed strategy can be slow; you can pass `--reuse_sandbox_directories` to reduce overhead of creating sandboxes on these systems. - -Dynamic execution can also run with the `standalone` strategy, though since the `standalone` strategy must take the output lock when it starts executing, it effectively blocks the remote strategy from finishing first. The `--experimental_local_lockfree_output` flag enables a way around this problem by allowing the local execution to write directly to the output, but be aborted by the remote execution, should that finish first. - -If one of the branches of dynamic execution finishes first but is a failure, the entire action fails. This is an intentional choice to prevent differences between local and remote execution from going unnoticed. - -For more background on how dynamic execution and its locking works, see Julio Merino's excellent [blog posts](https://jmmv.dev/series/bazel-dynamic-execution/) - -## When should I use dynamic execution? - -Dynamic execution requires some form of [remote execution system](/remote/rbe). It is not currently possible to use a cache-only remote system, as a cache miss would be considered a failed action. - -Not all types of actions are well suited for remote execution. The best candidates are those that are inherently faster locally, for instance through the use of [persistent workers](/remote/persistent), or those that run fast enough that the overhead of remote execution dominates execution time. Since each locally executed action locks some amount of CPU and memory resources, running actions that don't fall into those categories merely delays execution for those that do. - -As of release [5.0.0-pre.20210708.4](https://github.com/bazelbuild/bazel/releases/tag/5.0.0-pre.20210708.4), [performance profiling](/rules/performance#performance-profiling) contains data about worker execution, including time spent finishing a work request after losing a dynamic execution race. If you see dynamic execution worker threads spending significant time acquiring resources, or a lot of time in the `async-worker-finish`, you may have some slow local actions delaying the worker threads. - -![Profiling data with poor dynamic execution performance](/docs/images/dyn-trace-alldynamic.png) - -In the profile above, which uses 8 Javac workers, we see many Javac workers having lost the races and finishing their work on the `async-worker-finish` threads. This was caused by a non-worker mnemonic taking enough resources to delay the workers. - -![Profiling data with better dynamic execution performance](/docs/images/dyn-trace-javaconly.png) - -When only Javac is run with dynamic execution, only about half of the started workers end up losing the race after starting their work. - -The previously recommended `--experimental_spawn_scheduler` flag is deprecated. It turns on dynamic execution and sets `dynamic` as the default strategy for all mnemonics, which would often lead to these kinds of problems. - -## Performance - -The dynamic execution approach assumes there are enough resources available locally and remotely that it's worth spending some extra resources to improve overall performance. But excessive resource usage may slow down Bazel itself or the machine it runs on, or put unexpected pressure on a remote system. There are several options for changing the behaviour of dynamic execution: - -`--dynamic_local_execution_delay` delays the start of a local branch by a number of milliseconds after the remote branch has started, but only if there has been a remote cache hit during the current build. This makes builds that benefit from remote caching not waste local resources when it is likely that most outputs can be found in the cache. Depending on the quality of the cache, reducing this might improve build speeds, at the cost of using more local resources. - -`--experimental_dynamic_local_load_factor` is an experimental advanced resource management option. It takes a value from 0 to 1, 0 turning off this feature. When set to a value above 0, Bazel adjusts the number of locally scheduled actions when many actions waiting to be scheduled. Setting it to 1 allows as many actions to be scheduled as there are CPUs available (as per `--local_resources`). Lower values set the number of actions scheduled to correspondingly fewer as higher numbers of actions are available to run. This may sound counter-intuitive, but with a good remote system, local execution does not help much when many actions are being run, and the local CPU is better spent managing remote actions. - -`--experimental_dynamic_slow_remote_time` prioritizes starting local branches when the remote branch has been running for at least this long. Normally the most recently scheduled action gets priority, as it has the greatest chance of winning the race, but if the remote system sometimes hangs or takes extra long, this can get a build to move along. This is not enabled by default, because it could hide issues with the remote system that should rather be fixed. Make sure to monitor your remote system performance if you enable this option. - -`--experimental_dynamic_ignore_local_signals` can be used to let the remote branch take over when a local spawn exits due to a given signal. This is is mainly useful together with worker resource limits (see [`--experimental_worker_memory_limit_mb`](https://bazel.build/reference/command-line-reference#flag--experimental_worker_memory_limit_mb), [`--experimental_worker_sandbox_hardening`](https://bazel.build/reference/command-line-reference#flag--experimental_worker_sandbox_hardening), and [`--experimental_sandbox_memory_limit_mb`)](https://bazel.build/reference/command-line-reference#flag--experimental_sandbox_memory_limit_mb)), where worker processes may be killed when they use too many resources. - -The [JSON trace profile](/advanced/performance/json-trace-profile) contains a number of performance-related graphs that can help identify ways to improve the trade-off of performance and resource usage. - -## Troubleshooting - -Problems with dynamic execution can be subtle and hard to debug, as they can manifest only under some specific combinations of local and remote execution. The `--debug_spawn_scheduler` adds extra output from the dynamic execution system that can help debug these problems. You can also adjust the `--dynamic_local_execution_delay` flag and number of remote vs. local jobs to make it easier to reproduce the problems. - -If you are experiencing problems with dynamic execution using the `standalone` strategy, try running without `--experimental_local_lockfree_output`, or run your local actions sandboxed. This may slow down your build a bit (see above if you're on Mac or Windows), but removes some possible causes for failures. diff --git a/remote/multiplex.mdx b/remote/multiplex.mdx index d9075a8a..b4b0a0d4 100644 --- a/remote/multiplex.mdx +++ b/remote/multiplex.mdx @@ -2,38 +2,112 @@ title: 'Multiplex Workers (Experimental Feature)' --- -This page describes multiplex workers, how to write multiplex-compatible rules, and workarounds for certain limitations. + + +This page describes multiplex workers, how to write multiplex-compatible +rules, and workarounds for certain limitations. Caution: Experimental features are subject to change at any time. -*Multiplex workers* allow Bazel to handle multiple requests with a single worker process. For multi-threaded workers, Bazel can use fewer resources to achieve the same, or better performance. For example, instead of having one worker process per worker, Bazel can have four multiplexed workers talking to the same worker process, which can then handle requests in parallel. For languages like Java and Scala, this saves JVM warm-up time and JIT compilation time, and in general it allows using one shared cache between all workers of the same type. +_Multiplex workers_ allow Bazel to handle multiple requests with a single worker +process. For multi-threaded workers, Bazel can use fewer resources to +achieve the same, or better performance. For example, instead of having one +worker process per worker, Bazel can have four multiplexed workers talking to +the same worker process, which can then handle requests in parallel. For +languages like Java and Scala, this saves JVM warm-up time and JIT compilation +time, and in general it allows using one shared cache between all workers of +the same type. ## Overview -There are two layers between the Bazel server and the worker process. For certain mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from the worker pool. The `WorkerProxy` forwards requests to the worker process sequentially along with a `request_id`, the worker process processes the request and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` receives a response, it parses the `request_id` and then forwards the responses back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all communication is done over standard in/out, but the tool cannot just use `stderr` for user-visible output ([see below](#output)). - -Each worker has a key. Bazel uses the key's hash code (composed of environment variables, the execution root, and the mnemonic) to determine which `WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same `WorkerMultiplexer` if they have the same hash code. Therefore, assuming environment variables and the execution root are the same in a single Bazel invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one worker process. The total number of workers, including regular workers and `WorkerProxy`s, is still limited by `--worker_max_instances`. +There are two layers between the Bazel server and the worker process. For certain +mnemonics that can run processes in parallel, Bazel gets a `WorkerProxy` from +the worker pool. The `WorkerProxy` forwards requests to the worker process +sequentially along with a `request_id`, the worker process processes the request +and sends responses to the `WorkerMultiplexer`. When the `WorkerMultiplexer` +receives a response, it parses the `request_id` and then forwards the responses +back to the correct `WorkerProxy`. Just as with non-multiplexed workers, all +communication is done over standard in/out, but the tool cannot just use +`stderr` for user-visible output ([see below](#output)). + +Each worker has a key. Bazel uses the key's hash code (composed of environment +variables, the execution root, and the mnemonic) to determine which +`WorkerMultiplexer` to use. `WorkerProxy`s communicate with the same +`WorkerMultiplexer` if they have the same hash code. Therefore, assuming +environment variables and the execution root are the same in a single Bazel +invocation, each unique mnemonic can only have one `WorkerMultiplexer` and one +worker process. The total number of workers, including regular workers and +`WorkerProxy`s, is still limited by `--worker_max_instances`. ## Writing multiplex-compatible rules -The rule's worker process should be multi-threaded to take advantage of multiplex workers. Protobuf allows a ruleset to parse a single request even though there might be multiple requests piling up in the stream. Whenever the worker process parses a request from the stream, it should handle the request in a new thread. Because different thread could complete and write to the stream at the same time, the worker process needs to make sure the responses are written atomically (messages don't overlap). Responses must contain the `request_id` of the request they're handling. +The rule's worker process should be multi-threaded to take advantage of +multiplex workers. Protobuf allows a ruleset to parse a single request even +though there might be multiple requests piling up in the stream. Whenever the +worker process parses a request from the stream, it should handle the request in +a new thread. Because different thread could complete and write to the stream at +the same time, the worker process needs to make sure the responses are written +atomically (messages don't overlap). Responses must contain the +`request_id` of the request they're handling. ### Handling multiplex output -Multiplex workers need to be more careful about handling their output than singleplex workers. Anything sent to `stderr` will go into a single log file shared among all `WorkerProxy`s of the same type, randomly interleaved between concurrent requests. While redirecting `stdout` into `stderr` is a good idea, do not collect that output into the `output` field of `WorkResponse`, as that could show the user mangled pieces of output. If your tool only sends user-oriented output to `stdout` or `stderr`, you will need to change that behaviour before you can enable multiplex workers. +Multiplex workers need to be more careful about handling their output than +singleplex workers. Anything sent to `stderr` will go into a single log file +shared among all `WorkerProxy`s of the same type, +randomly interleaved between concurrent requests. While redirecting `stdout` +into `stderr` is a good idea, do not collect that output into the `output` +field of `WorkResponse`, as that could show the user mangled pieces of output. +If your tool only sends user-oriented output to `stdout` or `stderr`, you will +need to change that behaviour before you can enable multiplex workers. ## Enabling multiplex workers -Multiplex workers are not enabled by default. A ruleset can turn on multiplex workers by using the `supports-multiplex-workers` tag in the `execution_requirements` of an action (just like the `supports-workers` tag enables regular workers). As is the case when using regular workers, a worker strategy needs to be specified, either at the ruleset level (for example, `--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are necessary, and `supports-multiplex-workers` takes precedence over `supports-workers`, if both are set. You can turn off multiplex workers globally by passing `--noworker_multiplex`. - -A ruleset is encouraged to use multiplex workers if possible, to reduce memory pressure and improve performance. However, multiplex workers are not currently compatible with [dynamic execution](/remote/dynamic) unless they implement multiplex sandboxing. Attempting to run non-sandboxed multiplex workers with dynamic execution will silently use sandboxed singleplex workers instead. +Multiplex workers are not enabled by default. A ruleset can turn on multiplex +workers by using the `supports-multiplex-workers` tag in the +`execution_requirements` of an action (just like the `supports-workers` tag +enables regular workers). As is the case when using regular workers, a worker +strategy needs to be specified, either at the ruleset level (for example, +`--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for +example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are +necessary, and `supports-multiplex-workers` takes precedence over +`supports-workers`, if both are set. You can turn off multiplex workers +globally by passing `--noworker_multiplex`. + +A ruleset is encouraged to use multiplex workers if possible, to reduce memory +pressure and improve performance. However, multiplex workers are not currently +compatible with [dynamic execution](/remote/dynamic) unless they +implement multiplex sandboxing. Attempting to run non-sandboxed multiplex +workers with dynamic execution will silently use sandboxed +singleplex workers instead. ## Multiplex sandboxing -Multiplex workers can be sandboxed by adding explicit support for it in the worker implementations. While singleplex worker sandboxing can be done by running each worker process in its own sandbox, multiplex workers share the process working directory between multiple parallel requests. To allow sandboxing of multiplex workers, the worker must support reading from and writing to a subdirectory specified in each request, instead of directly in its working directory. - -To support multiplex sandboxing, the worker must use the `sandbox_dir` field from the `WorkRequest` and use that as a prefix for all file reads and writes. While the `arguments` and `inputs` fields remain unchanged from an unsandboxed request, the actual inputs are relative to the `sandbox_dir`. The worker must translate file paths found in `arguments` and `inputs` to read from this modified path, and must also write all outputs relative to the `sandbox_dir`. This includes paths such as '.', as well as paths found in files specified in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). - -Once a worker supports multiplex sandboxing, the ruleset can declare this support by adding `supports-multiplex-sandboxing` to the `execution_requirements` of an action. Bazel will then use multiplex sandboxing if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if the worker is used with dynamic execution. - -The worker files of a sandboxed multiplex worker are still relative to the working directory of the worker process. Thus, if a file is used both for running the worker and as an input, it must be specified both as an input in the flagfile argument as well as in `tools`, `executable`, or `runfiles`. +Multiplex workers can be sandboxed by adding explicit support for it in the +worker implementations. While singleplex worker sandboxing can be done by +running each worker process in its own sandbox, multiplex workers share the +process working directory between multiple parallel requests. To allow +sandboxing of multiplex workers, the worker must support reading from and +writing to a subdirectory specified in each request, instead of directly in +its working directory. + +To support multiplex sandboxing, the worker must use the `sandbox_dir` field +from the `WorkRequest` and use that as a prefix for all file reads and writes. +While the `arguments` and `inputs` fields remain unchanged from an unsandboxed +request, the actual inputs are relative to the `sandbox_dir`. The worker must +translate file paths found in `arguments` and `inputs` to read from this +modified path, and must also write all outputs relative to the `sandbox_dir`. +This includes paths such as '.', as well as paths found in files specified +in the arguments (such as ["argfile"](https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html#commandlineargfile) arguments). + +Once a worker supports multiplex sandboxing, the ruleset can declare this +support by adding `supports-multiplex-sandboxing` to the +`execution_requirements` of an action. Bazel will then use multiplex sandboxing +if the `--experimental_worker_multiplex_sandboxing` flag is passed, or if +the worker is used with dynamic execution. + +The worker files of a sandboxed multiplex worker are still relative to the +working directory of the worker process. Thus, if a file is +used both for running the worker and as an input, it must be specified both as +an input in the flagfile argument as well as in `tools`, `executable`, or +`runfiles`. diff --git a/remote/output-directories.mdx b/remote/output-directories.mdx index 26a36c2a..9c1ba329 100644 --- a/remote/output-directories.mdx +++ b/remote/output-directories.mdx @@ -2,108 +2,141 @@ title: 'Output Directory Layout' --- + + This page covers requirements and layout for output directories. ## Requirements Requirements for an output directory layout: -- Doesn't collide if multiple users are building on the same box. -- Supports building in multiple workspaces at the same time. -- Supports building for multiple target configurations in the same workspace. -- Doesn't collide with any other tools. -- Is easy to access. -- Is easy to clean, even selectively. -- Is unambiguous, even if the user relies on symbolic links when changing into their client directory. -- All the build state per user should be underneath one directory ("I'd like to clean all the .o files from all my clients.") +* Doesn't collide if multiple users are building on the same box. +* Supports building in multiple workspaces at the same time. +* Supports building for multiple target configurations in the same workspace. +* Doesn't collide with any other tools. +* Is easy to access. +* Is easy to clean, even selectively. +* Is unambiguous, even if the user relies on symbolic links when changing into + their client directory. +* All the build state per user should be underneath one directory ("I'd like to + clean all the .o files from all my clients.") ## Current layout The solution that's currently implemented: -- Bazel must be invoked from a directory containing a repo boundary file, or a subdirectory thereof. In other words, Bazel must be invoked from inside a [repository](../external/overview#repository). Otherwise, an error is reported. -- The *outputRoot* directory defaults to `~/.cache/bazel` on Linux, `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if set, else `%USERPROFILE%` if set, else the result of calling `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the environment variable `$XDG_CACHE_HOME` is set on either Linux or macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel itself, then that value overrides any defaults. -- The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. This is called the *outputUserRoot* directory. -- Beneath the `outputUserRoot` directory there is an `install` directory, and in it is an `installBase` directory whose name is the MD5 hash of the Bazel installation manifest. -- Beneath the `outputUserRoot` directory, an `outputBase` directory is also created whose name is the MD5 hash of the path name of the workspace root. So, for example, if Bazel is running in the workspace root `/home/user/src/my-project` (or in a directory symlinked to that one), then an output base directory is created called: `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. -- You can use Bazel's `--output_base` startup option to override the default output base directory. For example, `bazel --output_base=/tmp/bazel/output build x/y:z`. -- You can also use Bazel's `--output_user_root` startup option to override the default install base and output base directories. For example: `bazel --output_user_root=/tmp/bazel build x/y:z`. - -The symlinks for "bazel-\", "bazel-out", "bazel-testlogs", and "bazel-bin" are put in the workspace directory; these symlinks point to some directories inside a target-specific directory inside the output directory. These symlinks are only for the user's convenience, as Bazel itself does not use them. Also, this is done only if the workspace root is writable. +* Bazel must be invoked from a directory containing a repo boundary file, or a + subdirectory thereof. In other words, Bazel must be invoked from inside a + [repository](../external/overview#repository). Otherwise, an error is + reported. +* The _outputRoot_ directory defaults to `~/.cache/bazel` on Linux, + `/private/var/tmp` on macOS, and on Windows it defaults to `%HOME%` if + set, else `%USERPROFILE%` if set, else the result of calling + `SHGetKnownFolderPath()` with the `FOLDERID_Profile` flag set. If the + environment variable `$XDG_CACHE_HOME` is set on either Linux or + macOS, the value `${XDG_CACHE_HOME}/bazel` will override the default. + If the environment variable `$TEST_TMPDIR` is set, as in a test of Bazel + itself, then that value overrides any defaults. +* The Bazel user's build state is located beneath `outputRoot/_bazel_$USER`. + This is called the _outputUserRoot_ directory. +* Beneath the `outputUserRoot` directory there is an `install` directory, and in + it is an `installBase` directory whose name is the MD5 hash of the Bazel + installation manifest. +* Beneath the `outputUserRoot` directory, an `outputBase` directory + is also created whose name is the MD5 hash of the path name of the workspace + root. So, for example, if Bazel is running in the workspace root + `/home/user/src/my-project` (or in a directory symlinked to that one), then + an output base directory is created called: + `/home/user/.cache/bazel/_bazel_user/7ffd56a6e4cb724ea575aba15733d113`. You + can also run `echo -n $(pwd) | md5sum` in the workspace root to get the MD5. +* You can use Bazel's `--output_base` startup option to override the default + output base directory. For example, + `bazel --output_base=/tmp/bazel/output build x/y:z`. +* You can also use Bazel's `--output_user_root` startup option to override the + default install base and output base directories. For example: + `bazel --output_user_root=/tmp/bazel build x/y:z`. + +The symlinks for "bazel-<workspace-name>", "bazel-out", "bazel-testlogs", +and "bazel-bin" are put in the workspace directory; these symlinks point to some +directories inside a target-specific directory inside the output directory. +These symlinks are only for the user's convenience, as Bazel itself does not +use them. Also, this is done only if the workspace root is writable. ## Layout diagram The directories are laid out as follows: ``` -<workspace-name>/ <== The workspace root - bazel-my-project => <..._main> <== Symlink to execRoot - bazel-out => <...bazel-out> <== Convenience symlink to outputPath - bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) - bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory - -/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot - _bazel_$USER/ <== Top level directory for a given user depends on the user name: +<workspace-name>/ <== The workspace root + bazel-my-project => <..._main> <== Symlink to execRoot + bazel-out => <...bazel-out> <== Convenience symlink to outputPath + bazel-bin => <...bin> <== Convenience symlink to most recent written bin dir $(BINDIR) + bazel-testlogs => <...testlogs> <== Convenience symlink to the test logs directory + +/home/user/.cache/bazel/ <== Root for all Bazel output on a machine: outputRoot + _bazel_$USER/ <== Top level directory for a given user depends on the user name: outputUserRoot install/ - fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase - _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of + fba9a2c87ee9589d72889caf082f1029/ <== Hash of the Bazel install manifest: installBase + _embedded_binaries/ <== Contains binaries and scripts unpacked from the data section of the bazel executable on first run (such as helper scripts and the main Java file BazelServer_deploy.jar) - 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as + 7ffd56a6e4cb724ea575aba15733d113/ <== Hash of the client's workspace root (such as /home/user/src/my-project): outputBase - action_cache/ <== Action cache directory hierarchy + action_cache/ <== Action cache directory hierarchy This contains the persistent record of the file metadata (timestamps, and perhaps eventually also MD5 sums) used by the FilesystemValueChecker. - command.log <== A copy of the stdout/stderr output from the most + command.log <== A copy of the stdout/stderr output from the most recent bazel command. - external/ <== The directory that remote repositories are + external/ <== The directory that remote repositories are downloaded/symlinked into. - server/ <== The Bazel server puts all server-related files (such + server/ <== The Bazel server puts all server-related files (such as socket file, logs, etc) here. - jvm.out <== The debugging output for the server. - execroot/ <== The working directory for all actions. For special + jvm.out <== The debugging output for the server. + execroot/ <== The working directory for all actions. For special cases such as sandboxing and remote execution, the actions run in a directory that mimics execroot. Implementation details, such as where the directories are created, are intentionally hidden from the action. Every action can access its inputs and outputs relative to the execroot directory. - _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot - _bin/ <== Helper tools are linked from or copied to here. + _main/ <== Working tree for the Bazel build & root of symlink forest: execRoot + _bin/ <== Helper tools are linked from or copied to here. - bazel-out/ <== All actual output of the build is under here: outputPath - _tmp/actions/ <== Action output directory. This contains a file with the + bazel-out/ <== All actual output of the build is under here: outputPath + _tmp/actions/ <== Action output directory. This contains a file with the stdout/stderr for every action from the most recent bazel run that produced output. - local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; + local_linux-fastbuild/ <== one subdirectory per unique target BuildConfiguration instance; this is currently encoded - bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) - foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz - foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc - other_package/other.o <== Object files from source //other_package:other.cc - foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named + bin/ <== Bazel outputs binaries for target configuration here: $(BINDIR) + foo/bar/_objs/baz/ <== Object files for a cc_* rule named //foo/bar:baz + foo/bar/baz1.o <== Object files from source //foo/bar:baz1.cc + other_package/other.o <== Object files from source //other_package:other.cc + foo/bar/baz <== foo/bar/baz might be the artifact generated by a cc_binary named //foo/bar:baz - foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. + foo/bar/baz.runfiles/ <== The runfiles symlink farm for the //foo/bar:baz executable. MANIFEST _main/ ... - genfiles/ <== Bazel puts generated source for the target configuration here: + genfiles/ <== Bazel puts generated source for the target configuration here: $(GENDIR) foo/bar.h such as foo/bar.h might be a headerfile generated by //foo:bargen - testlogs/ <== Bazel internal test runner puts test log files here + testlogs/ <== Bazel internal test runner puts test log files here foo/bartest.log such as foo/bar.log might be an output of the //foo:bartest test with foo/bartest.status foo/bartest.status containing exit status of the test (such as PASSED or FAILED (Exit 1), etc) - host/ <== BuildConfiguration for build host (user's workstation), for + host/ <== BuildConfiguration for build host (user's workstation), for building prerequisite tools, that will be used in later stages of the build (ex: Protocol Compiler) - <packages>/ <== Packages referenced in the build appear as if under a regular workspace + <packages>/ <== Packages referenced in the build appear as if under a regular workspace ``` The layout of the \*.runfiles directories is documented in more detail in the places pointed to by RunfilesSupport. ## `bazel clean` -`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` directory. It also removes the workspace symlinks. The `--expunge` option will clean the entire outputBase. +`bazel clean` does an `rm -rf` on the `outputPath` and the `action_cache` +directory. It also removes the workspace symlinks. The `--expunge` option +will clean the entire outputBase. diff --git a/remote/persistent.mdx b/remote/persistent.mdx index bd9029f5..1a56946a 100644 --- a/remote/persistent.mdx +++ b/remote/persistent.mdx @@ -2,65 +2,158 @@ title: 'Persistent Workers' --- -This page covers how to use persistent workers, the benefits, requirements, and how workers affect sandboxing. -A persistent worker is a long-running process started by the Bazel server, which functions as a *wrapper* around the actual *tool* (typically a compiler), or is the *tool* itself. In order to benefit from persistent workers, the tool must support doing a sequence of compilations, and the wrapper needs to translate between the tool's API and the request/response format described below. The same worker might be called with and without the `--persistent_worker` flag in the same build, and is responsible for appropriately starting and talking to the tool, as well as shutting down workers on exit. Each worker instance is assigned (but not chrooted to) a separate working directory under `<outputBase>/bazel-workers`. -Using persistent workers is an [execution strategy](/docs/user-manual#execution-strategy) that decreases start-up overhead, allows more JIT compilation, and enables caching of for example the abstract syntax trees in the action execution. This strategy achieves these improvements by sending multiple requests to a long-running process. +This page covers how to use persistent workers, the benefits, requirements, and +how workers affect sandboxing. -Persistent workers are implemented for multiple languages, including Java, [Scala](https://github.com/bazelbuild/rules_scala), [Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. +A persistent worker is a long-running process started by the Bazel server, which +functions as a *wrapper* around the actual *tool* (typically a compiler), or is +the *tool* itself. In order to benefit from persistent workers, the tool must +support doing a sequence of compilations, and the wrapper needs to translate +between the tool's API and the request/response format described below. The same +worker might be called with and without the `--persistent_worker` flag in the +same build, and is responsible for appropriately starting and talking to the +tool, as well as shutting down workers on exit. Each worker instance is assigned +(but not chrooted to) a separate working directory under +`/bazel-workers`. -Programs using a NodeJS runtime can use the [@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to implement the worker protocol. +Using persistent workers is an +[execution strategy](/docs/user-manual#execution-strategy) that decreases +start-up overhead, allows more JIT compilation, and enables caching of for +example the abstract syntax trees in the action execution. This strategy +achieves these improvements by sending multiple requests to a long-running +process. + +Persistent workers are implemented for multiple languages, including Java, +[Scala](https://github.com/bazelbuild/rules_scala), +[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. + +Programs using a NodeJS runtime can use the +[@bazel/worker](https://www.npmjs.com/package/@bazel/worker) helper library to +implement the worker protocol. ## Using persistent workers -[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) uses persistent workers by default when executing builds, though remote execution takes precedence. For actions that do not support persistent workers, Bazel falls back to starting a tool instance for each action. You can explicitly set your build to use persistent workers by setting the `worker` [strategy](/docs/user-manual#execution-strategy) for the applicable tool mnemonics. As a best practice, this example includes specifying `local` as a fallback to the `worker` strategy: +[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) +uses persistent workers by default when executing builds, though remote +execution takes precedence. For actions that do not support persistent workers, +Bazel falls back to starting a tool instance for each action. You can explicitly +set your build to use persistent workers by setting the `worker` +[strategy](/docs/user-manual#execution-strategy) for the applicable tool +mnemonics. As a best practice, this example includes specifying `local` as a +fallback to the `worker` strategy: ```posix-terminal -bazel build //<var>my:target</var> --strategy=Javac=worker,local +bazel build //{{ '' }}my:target{{ '' }} --strategy=Javac=worker,local ``` -Using the workers strategy instead of the local strategy can boost compilation speed significantly, depending on implementation. For Java, builds can be 2–4 times faster, sometimes more for incremental compilation. Compiling Bazel is about 2.5 times as fast with workers. For more details, see the "[Choosing number of workers](#number-of-workers)" section. - -If you also have a remote build environment that matches your local build environment, you can use the experimental [*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), which races a remote execution and a worker execution. To enable the dynamic strategy, pass the [--experimental\_spawn\_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) flag. This strategy automatically enables workers, so there is no need to specify the `worker` strategy, but you can still use `local` or `sandboxed` as fallbacks. +Using the workers strategy instead of the local strategy can boost compilation +speed significantly, depending on implementation. For Java, builds can be 2–4 +times faster, sometimes more for incremental compilation. Compiling Bazel is +about 2.5 times as fast with workers. For more details, see the +"[Choosing number of workers](#number-of-workers)" section. + +If you also have a remote build environment that matches your local build +environment, you can use the experimental +[*dynamic* strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html), +which races a remote execution and a worker execution. To enable the dynamic +strategy, pass the +[--experimental_spawn_scheduler](/reference/command-line-reference#flag--experimental_spawn_scheduler) +flag. This strategy automatically enables workers, so there is no need to +specify the `worker` strategy, but you can still use `local` or `sandboxed` as +fallbacks. ## Choosing number of workers -The default number of worker instances per mnemonic is 4, but can be adjusted with the [`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) flag. There is a trade-off between making good use of the available CPUs and the amount of JIT compilation and cache hits you get. With more workers, more targets will pay start-up costs of running non-JITted code and hitting cold caches. If you have a small number of targets to build, a single worker may give the best trade-off between compilation speed and resource usage (for example, see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). The `worker_max_instances` flag sets the maximum number of worker instances per mnemonic and flag set (see below), so in a mixed system you could end up using quite a lot of memory if you keep the default value. For incremental builds the benefit of multiple worker instances is even smaller. - -This graph shows the from-scratch compilation times for Bazel (target `//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation with 64 GB of RAM. For each worker configuration, five clean builds are run and the average of the last four are taken. +The default number of worker instances per mnemonic is 4, but can be adjusted +with the +[`worker_max_instances`](/reference/command-line-reference#flag--worker_max_instances) +flag. There is a trade-off between making good use of the available CPUs and the +amount of JIT compilation and cache hits you get. With more workers, more +targets will pay start-up costs of running non-JITted code and hitting cold +caches. If you have a small number of targets to build, a single worker may give +the best trade-off between compilation speed and resource usage (for example, +see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). +The `worker_max_instances` flag sets the maximum number of worker instances per +mnemonic and flag set (see below), so in a mixed system you could end up using +quite a lot of memory if you keep the default value. For incremental builds the +benefit of multiple worker instances is even smaller. + +This graph shows the from-scratch compilation times for Bazel (target +`//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation +with 64 GB of RAM. For each worker configuration, five clean builds are run and +the average of the last four are taken. ![Graph of performance improvements of clean builds](/docs/images/workers-clean-chart.png "Performance improvements of clean builds") **Figure 1.** Graph of performance improvements of clean builds. -For this configuration, two workers give the fastest compile, though at only 14% improvement compared to one worker. One worker is a good option if you want to use less memory. +For this configuration, two workers give the fastest compile, though at only 14% +improvement compared to one worker. One worker is a good option if you want to +use less memory. -Incremental compilation typically benefits even more. Clean builds are relatively rare, but changing a single file between compiles is common, in particular in test-driven development. The above example also has some non-Java packaging actions to it that can overshadow the incremental compile time. +Incremental compilation typically benefits even more. Clean builds are +relatively rare, but changing a single file between compiles is common, in +particular in test-driven development. The above example also has some non-Java +packaging actions to it that can overshadow the incremental compile time. -Recompiling the Java sources only (`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) after changing an internal string constant in [AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) gives a 3x speed-up (average of 20 incremental builds with one warmup build discarded): +Recompiling the Java sources only +(`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`) +after changing an internal string constant in +[AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java) +gives a 3x speed-up (average of 20 incremental builds with one warmup build +discarded): ![Graph of performance improvements of incremental builds](/docs/images/workers-incremental-chart.png "Performance improvements of incremental builds") **Figure 2.** Graph of performance improvements of incremental builds. -The speed-up depends on the change being made. A speed-up of a factor 6 is measured in the above situation when a commonly used constant is changed. +The speed-up depends on the change being made. A speed-up of a factor 6 is +measured in the above situation when a commonly used constant is changed. ## Modifying persistent workers -You can pass the [`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) flag to specify start-up flags to workers, keyed by mnemonic. For instance, passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. Only one worker flag can be set per use of this flag, and only for one mnemonic. Workers are not just created separately for each mnemonic, but also for variations in their start-up flags. Each combination of mnemonic and start-up flags is combined into a `WorkerKey`, and for each `WorkerKey` up to `worker_max_instances` workers may be created. See the next section for how the action configuration can also specify set-up flags. - -Passing the [`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) flag makes each worker request use a separate sandbox directory for all its inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, especially on macOS, but gives a better correctness guarantee. - -The [`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) flag is mainly useful for debugging and profiling. This flag forces all workers to quit once a build is done. You can also pass [`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to get more output about what the workers are doing. This flag is reflected in the `verbosity` field in `WorkRequest`, allowing worker implementations to also be more verbose. - -Workers store their logs in the `<outputBase>/bazel-workers` directory, for example `/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. The file name includes the worker id and the mnemonic. Since there can be more than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` log files for a given mnemonic. - -For Android builds, see details at the [Android Build Performance page](/docs/android-build-performance). +You can pass the +[`--worker_extra_flag`](/reference/command-line-reference#flag--worker_extra_flag) +flag to specify start-up flags to workers, keyed by mnemonic. For instance, +passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only. +Only one worker flag can be set per use of this flag, and only for one mnemonic. +Workers are not just created separately for each mnemonic, but also for +variations in their start-up flags. Each combination of mnemonic and start-up +flags is combined into a `WorkerKey`, and for each `WorkerKey` up to +`worker_max_instances` workers may be created. See the next section for how the +action configuration can also specify set-up flags. + +Passing the +[`--worker_sandboxing`](/reference/command-line-reference#flag--worker_sandboxing) +flag makes each worker request use a separate sandbox directory for all its +inputs. Setting up the [sandbox](/docs/sandboxing) takes some extra time, +especially on macOS, but gives a better correctness guarantee. + +The +[`--worker_quit_after_build`](/reference/command-line-reference#flag--worker_quit_after_build) +flag is mainly useful for debugging and profiling. This flag forces all workers +to quit once a build is done. You can also pass +[`--worker_verbose`](/reference/command-line-reference#flag--worker_verbose) to +get more output about what the workers are doing. This flag is reflected in the +`verbosity` field in `WorkRequest`, allowing worker implementations to also be +more verbose. + +Workers store their logs in the `/bazel-workers` directory, for +example +`/tmp/_bazel_larsrc/191013354bebe14fdddae77f2679c3ef/bazel-workers/worker-1-Javac.log`. +The file name includes the worker id and the mnemonic. Since there can be more +than one `WorkerKey` per mnemonic, you may see more than `worker_max_instances` +log files for a given mnemonic. + +For Android builds, see details at the +[Android Build Performance page](/docs/android-build-performance). ## Implementing persistent workers -See the [creating persistent workers](/remote/creating) page for more information on how to make a worker. +See the [creating persistent workers](/remote/creating) page for more +information on how to make a worker. This example shows a Starlark configuration for a worker that uses JSON: @@ -81,9 +174,14 @@ ctx.actions.run( ) ``` -With this definition, the first use of this action would start with executing the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request to compile `Foo.java` would then look like: +With this definition, the first use of this action would start with executing +the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request +to compile `Foo.java` would then look like: -NOTE: While the protocol buffer specification uses "snake case" (`request_id`), the JSON protocol uses "camel case" (`requestId`). In this document, we will use camel case in the JSON examples, but snake case when talking about the field regardless of protocol. +NOTE: While the protocol buffer specification uses "snake case" (`request_id`), +the JSON protocol uses "camel case" (`requestId`). In this document, we will use +camel case in the JSON examples, but snake case when talking about the field +regardless of protocol. ```json { @@ -95,7 +193,12 @@ NOTE: While the protocol buffer specification uses "snake case" (`request_id`), } ``` -The worker receives this on `stdin` in newline-delimited JSON format (because `requires-worker-protocol` is set to JSON). The worker then performs the action, and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then parses this response and manually converts it to a `WorkResponse` proto. To communicate with the associated worker using binary-encoded protobuf instead of JSON, `requires-worker-protocol` would be set to `proto`, like this: +The worker receives this on `stdin` in newline-delimited JSON format (because +`requires-worker-protocol` is set to JSON). The worker then performs the action, +and sends a JSON-formatted `WorkResponse` to Bazel on its stdout. Bazel then +parses this response and manually converts it to a `WorkResponse` proto. To +communicate with the associated worker using binary-encoded protobuf instead of +JSON, `requires-worker-protocol` would be set to `proto`, like this: ``` execution_requirements = { @@ -104,31 +207,59 @@ The worker receives this on `stdin` in newline-delimited JSON format (because `r } ``` -If you do not include `requires-worker-protocol` in the execution requirements, Bazel will default the worker communication to use protobuf. +If you do not include `requires-worker-protocol` in the execution requirements, +Bazel will default the worker communication to use protobuf. -Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this configuration allowed changing the `max_mem` parameter, a separate worker would be spawned for each value used. This can lead to excessive memory consumption if too many variations are used. +Bazel derives the `WorkerKey` from the mnemonic and the shared flags, so if this +configuration allowed changing the `max_mem` parameter, a separate worker would +be spawned for each value used. This can lead to excessive memory consumption if +too many variations are used. -Each worker can currently only process one request at a time. The experimental [multiplex workers](/remote/multiplex) feature allows using multiple threads, if the underlying tool is multithreaded and the wrapper is set up to understand this. +Each worker can currently only process one request at a time. The experimental +[multiplex workers](/remote/multiplex) feature allows using multiple +threads, if the underlying tool is multithreaded and the wrapper is set up to +understand this. -In [this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), you can see example worker wrappers written in Java as well as in Python. If you are working in JavaScript or TypeScript, the [@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) and [nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) might be helpful. +In +[this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), +you can see example worker wrappers written in Java as well as in Python. If you +are working in JavaScript or TypeScript, the +[@bazel/worker package](https://www.npmjs.com/package/@bazel/worker) +and +[nodejs worker example](https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/worker) +might be helpful. ## How do workers affect sandboxing? -Using the `worker` strategy by default does not run the action in a [sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the `--worker_sandboxing` flag to run all workers inside sandboxes, making sure each execution of the tool only sees the input files it's supposed to have. The tool may still leak information between requests internally, for instance through a cache. Using `dynamic` strategy [requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). +Using the `worker` strategy by default does not run the action in a +[sandbox](/docs/sandboxing), similar to the `local` strategy. You can set the +`--worker_sandboxing` flag to run all workers inside sandboxes, making sure each +execution of the tool only sees the input files it's supposed to have. The tool +may still leak information between requests internally, for instance through a +cache. Using `dynamic` strategy +[requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java). -To allow correct use of compiler caches with workers, a digest is passed along with each input file. Thus the compiler or the wrapper can check if the input is still valid without having to read the file. +To allow correct use of compiler caches with workers, a digest is passed along +with each input file. Thus the compiler or the wrapper can check if the input is +still valid without having to read the file. -Even when using the input digests to guard against unwanted caching, sandboxed workers offer less strict sandboxing than a pure sandbox, because the tool may keep other internal state that has been affected by previous requests. +Even when using the input digests to guard against unwanted caching, sandboxed +workers offer less strict sandboxing than a pure sandbox, because the tool may +keep other internal state that has been affected by previous requests. -Multiplex workers can only be sandboxed if the worker implementation support it, and this sandboxing must be separately enabled with the `--experimental_worker_multiplex_sandboxing` flag. See more details in [the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). +Multiplex workers can only be sandboxed if the worker implementation support it, +and this sandboxing must be separately enabled with the +`--experimental_worker_multiplex_sandboxing` flag. See more details in +[the design doc](https://docs.google.com/document/d/1ncLW0hz6uDhNvci1dpzfEoifwTiNTqiBEm1vi-bIIRM/edit)). ## Further reading For more information on persistent workers, see: -- [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) -- [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) -- [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) -- [Front End Development with Bazel: Angular/TypeScript and Persistent Workers w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) -- [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) -- [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) +* [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html) +* [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/) +* [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb) +* [Front End Development with Bazel: Angular/TypeScript and Persistent Workers + w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo) +* [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html) +* [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ) diff --git a/remote/rbe.mdx b/remote/rbe.mdx index 0d8bf255..75d4a155 100644 --- a/remote/rbe.mdx +++ b/remote/rbe.mdx @@ -2,20 +2,32 @@ title: 'Remote Execution Overview' --- -This page covers the benefits, requirements, and options for running Bazel with remote execution. -By default, Bazel executes builds and tests on your local machine. Remote execution of a Bazel build allows you to distribute build and test actions across multiple machines, such as a datacenter. + +This page covers the benefits, requirements, and options for running Bazel +with remote execution. + +By default, Bazel executes builds and tests on your local machine. Remote +execution of a Bazel build allows you to distribute build and test actions +across multiple machines, such as a datacenter. Remote execution provides the following benefits: -- Faster build and test execution through scaling of nodes available for parallel actions -- A consistent execution environment for a development team -- Reuse of build outputs across a development team +* Faster build and test execution through scaling of nodes available + for parallel actions +* A consistent execution environment for a development team +* Reuse of build outputs across a development team -Bazel uses an open-source [gRPC protocol](https://github.com/bazelbuild/remote-apis) to allow for remote execution and remote caching. +Bazel uses an open-source +[gRPC protocol](https://github.com/bazelbuild/remote-apis) +to allow for remote execution and remote caching. -For a list of commercially supported remote execution services as well as self-service tools, see [Remote Execution Services](https://www.bazel.build/remote-execution-services.html) +For a list of commercially supported remote execution services as well as +self-service tools, see +[Remote Execution Services](https://www.bazel.build/remote-execution-services.html) ## Requirements -Remote execution of Bazel builds imposes a set of mandatory configuration constraints on the build. For more information, see [Adapting Bazel Rules for Remote Execution](/remote/rules). +Remote execution of Bazel builds imposes a set of mandatory configuration +constraints on the build. For more information, see +[Adapting Bazel Rules for Remote Execution](/remote/rules). diff --git a/remote/rules.mdx b/remote/rules.mdx index 4ea4c9c3..340ab02c 100644 --- a/remote/rules.mdx +++ b/remote/rules.mdx @@ -2,79 +2,179 @@ title: 'Adapting Bazel Rules for Remote Execution' --- -This page is intended for Bazel users writing custom build and test rules who want to understand the requirements for Bazel rules in the context of remote execution. -Remote execution allows Bazel to execute actions on a separate platform, such as a datacenter. Bazel uses a [gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) for its remote execution. You can try remote execution with [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source project that aims to provide a distributed remote execution platform. -This page uses the following terminology when referring to different environment types or *platforms*: +This page is intended for Bazel users writing custom build and test rules +who want to understand the requirements for Bazel rules in the context of +remote execution. -- **Host platform** - where Bazel runs. -- **Execution platform** - where Bazel actions run. -- **Target platform** - where the build outputs (and some actions) run. +Remote execution allows Bazel to execute actions on a separate platform, such as +a datacenter. Bazel uses a +[gRPC protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) +for its remote execution. You can try remote execution with +[bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), +an open-source project that aims to provide a distributed remote execution +platform. + +This page uses the following terminology when referring to different +environment types or *platforms*: + +* **Host platform** - where Bazel runs. +* **Execution platform** - where Bazel actions run. +* **Target platform** - where the build outputs (and some actions) run. ## Overview -When configuring a Bazel build for remote execution, you must follow the guidelines described in this page to ensure the build executes remotely error-free. This is due to the nature of remote execution, namely: +When configuring a Bazel build for remote execution, you must follow the +guidelines described in this page to ensure the build executes remotely +error-free. This is due to the nature of remote execution, namely: -- **Isolated build actions.** Build tools do not retain state and dependencies cannot leak between them. +* **Isolated build actions.** Build tools do not retain state and dependencies + cannot leak between them. -- **Diverse execution environments.** Local build configuration is not always suitable for remote execution environments. +* **Diverse execution environments.** Local build configuration is not always + suitable for remote execution environments. -This page describes the issues that can arise when implementing custom Bazel build and test rules for remote execution and how to avoid them. It covers the following topics: +This page describes the issues that can arise when implementing custom Bazel +build and test rules for remote execution and how to avoid them. It covers the +following topics: -- [Invoking build tools through toolchain rules](#toolchain-rules) -- [Managing implicit dependencies](#manage-dependencies) -- [Managing platform-dependent binaries](#manage-binaries) -- [Managing configure-style WORKSPACE rules](#manage-workspace-rules) +* [Invoking build tools through toolchain rules](#toolchain-rules) +* [Managing implicit dependencies](#manage-dependencies) +* [Managing platform-dependent binaries](#manage-binaries) +* [Managing configure-style WORKSPACE rules](#manage-workspace-rules) ## Invoking build tools through toolchain rules -A Bazel toolchain rule is a configuration provider that tells a build rule what build tools, such as compilers and linkers, to use and how to configure them using parameters defined by the rule's creator. A toolchain rule allows build and test rules to invoke build tools in a predictable, preconfigured manner that's compatible with remote execution. For example, use a toolchain rule instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local variables that may not be set to equivalent values (or at all) in the remote execution environment. - -Toolchain rules currently exist for Bazel build and test rules for \[Scala]\([https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch) ain.bzl), [Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), and new toolchain rules are under way for other languages and tools such as [bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). If a toolchain rule does not exist for the tool your rule uses, consider [creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). +A Bazel toolchain rule is a configuration provider that tells a build rule what +build tools, such as compilers and linkers, to use and how to configure them +using parameters defined by the rule's creator. A toolchain rule allows build +and test rules to invoke build tools in a predictable, preconfigured manner +that's compatible with remote execution. For example, use a toolchain rule +instead of invoking build tools via the `PATH`, `JAVA_HOME`, or other local +variables that may not be set to equivalent values (or at all) in the remote +execution environment. + +Toolchain rules currently exist for Bazel build and test rules for +[Scala](https://github.com/bazelbuild/rules_scala/blob/master/scala/scala_toolch +ain.bzl), +[Rust](https://github.com/bazelbuild/rules_rust/blob/main/rust/toolchain.bzl), +and [Go](https://github.com/bazelbuild/rules_go/blob/master/go/toolchains.rst), +and new toolchain rules are under way for other languages and tools such as +[bash](https://docs.google.com/document/d/e/2PACX-1vRCSB_n3vctL6bKiPkIa_RN_ybzoAccSe0ic8mxdFNZGNBJ3QGhcKjsL7YKf-ngVyjRZwCmhi_5KhcX/pub). +If a toolchain rule does not exist for the tool your rule uses, consider +[creating a toolchain rule](/extending/toolchains#creating-a-toolchain-rule). ## Managing implicit dependencies -If a build tool can access dependencies across build actions, those actions will fail when remotely executed because each remote build action is executed separately from others. Some build tools retain state across build actions and access dependencies that have not been explicitly included in the tool invocation, which will cause remotely executed build actions to fail. - -For example, when Bazel instructs a stateful compiler to locally build *foo*, the compiler retains references to foo's build outputs. When Bazel then instructs the compiler to build *bar*, which depends on *foo*, without explicitly stating that dependency in the BUILD file for inclusion in the compiler invocation, the action executes successfully as long as the same compiler instance executes for both actions (as is typical for local execution). However, since in a remote execution scenario each build action executes a separate compiler instance, compiler state and *bar*'s implicit dependency on *foo* will be lost and the build will fail. - -To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the local Docker sandbox, which has the same restrictions for dependencies as remote execution. Use the sandbox to prepare your build for remote execution by identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) for more information. +If a build tool can access dependencies across build actions, those actions will +fail when remotely executed because each remote build action is executed +separately from others. Some build tools retain state across build actions and +access dependencies that have not been explicitly included in the tool +invocation, which will cause remotely executed build actions to fail. + +For example, when Bazel instructs a stateful compiler to locally build _foo_, +the compiler retains references to foo's build outputs. When Bazel then +instructs the compiler to build _bar_, which depends on _foo_, without +explicitly stating that dependency in the BUILD file for inclusion in the +compiler invocation, the action executes successfully as long as the same +compiler instance executes for both actions (as is typical for local execution). +However, since in a remote execution scenario each build action executes a +separate compiler instance, compiler state and _bar_'s implicit dependency on +_foo_ will be lost and the build will fail. + +To help detect and eliminate these dependency problems, Bazel 0.14.1 offers the +local Docker sandbox, which has the same restrictions for dependencies as remote +execution. Use the sandbox to prepare your build for remote execution by +identifying and resolving dependency-related build errors. See [Troubleshooting Bazel Remote Execution with Docker Sandbox](/remote/sandbox) +for more information. ## Managing platform-dependent binaries -Typically, a binary built on the host platform cannot safely execute on an arbitrary remote execution platform due to potentially mismatched dependencies. For example, the SingleJar binary supplied with Bazel targets the host platform. However, for remote execution, SingleJar must be compiled as part of the process of building your code so that it targets the remote execution platform. (See the [target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) +Typically, a binary built on the host platform cannot safely execute on an +arbitrary remote execution platform due to potentially mismatched dependencies. +For example, the SingleJar binary supplied with Bazel targets the host platform. +However, for remote execution, SingleJar must be compiled as part of the process +of building your code so that it targets the remote execution platform. (See the +[target selection logic](https://github.com/bazelbuild/bazel/blob/130aeadfd660336572c3da397f1f107f0c89aa8d/tools/jdk/BUILD#L115).) -Do not ship binaries of build tools required by your build with your source code unless you are sure they will safely run in your execution platform. Instead, do one of the following: +Do not ship binaries of build tools required by your build with your source code +unless you are sure they will safely run in your execution platform. Instead, do +one of the following: -- Ship or externally reference the source code for the tool so that it can be built for the remote execution platform. +* Ship or externally reference the source code for the tool so that it can be + built for the remote execution platform. -- Pre-install the tool into the remote execution environment (for example, a toolchain container) if it's stable enough and use toolchain rules to run it in your build. +* Pre-install the tool into the remote execution environment (for example, a + toolchain container) if it's stable enough and use toolchain rules to run it + in your build. ## Managing configure-style WORKSPACE rules -Bazel's `WORKSPACE` rules can be used for probing the host platform for tools and libraries required by the build, which, for local builds, is also Bazel's execution platform. If the build explicitly depends on local build tools and artifacts, it will fail during remote execution if the remote execution platform is not identical to the host platform. - -The following actions performed by `WORKSPACE` rules are not compatible with remote execution: - -- **Building binaries.** Executing compilation actions in `WORKSPACE` rules results in binaries that are incompatible with the remote execution platform if different from the host platform. - -- **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` rules require that their dependencies be pre-installed on the host platform. Such packages, built specifically for the host platform, will be incompatible with the remote execution platform if different from the host platform. - -- **Symlinking to local tools or artifacts.** Symlinks to tools or libraries installed on the host platform created via `WORKSPACE` rules will cause the build to fail on the remote execution platform as Bazel will not be able to locate them. Instead, create symlinks using standard build actions so that the symlinked tools and libraries are accessible from Bazel's `runfiles` tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) to symlink target files outside of the external repo directory. - -- **Mutating the host platform.** Avoid creating files outside of the Bazel `runfiles` tree, creating environment variables, and similar actions, as they may behave unexpectedly on the remote execution platform. +Bazel's `WORKSPACE` rules can be used for probing the host platform for tools +and libraries required by the build, which, for local builds, is also Bazel's +execution platform. If the build explicitly depends on local build tools and +artifacts, it will fail during remote execution if the remote execution platform +is not identical to the host platform. + +The following actions performed by `WORKSPACE` rules are not compatible with +remote execution: + +* **Building binaries.** Executing compilation actions in `WORKSPACE` rules + results in binaries that are incompatible with the remote execution platform + if different from the host platform. + +* **Installing `pip` packages.** `pip` packages installed via `WORKSPACE` + rules require that their dependencies be pre-installed on the host platform. + Such packages, built specifically for the host platform, will be + incompatible with the remote execution platform if different from the host + platform. + +* **Symlinking to local tools or artifacts.** Symlinks to tools or libraries + installed on the host platform created via `WORKSPACE` rules will cause the + build to fail on the remote execution platform as Bazel will not be able to + locate them. Instead, create symlinks using standard build actions so that + the symlinked tools and libraries are accessible from Bazel's `runfiles` + tree. Do not use [`repository_ctx.symlink`](/rules/lib/builtins/repository_ctx#symlink) + to symlink target files outside of the external repo directory. + +* **Mutating the host platform.** Avoid creating files outside of the Bazel + `runfiles` tree, creating environment variables, and similar actions, as + they may behave unexpectedly on the remote execution platform. To help find potential non-hermetic behavior you can use [Workspace rules log](/remote/workspace). -If an external dependency executes specific operations dependent on the host platform, you should split those operations between `WORKSPACE` and build rules as follows: - -- **Platform inspection and dependency enumeration.** These operations are safe to execute locally via `WORKSPACE` rules, which can check which libraries are installed, download packages that must be built, and prepare required artifacts for compilation. For remote execution, these rules must also support using pre-checked artifacts to provide the information that would normally be obtained during host platform inspection. Pre-checked artifacts allow Bazel to describe dependencies as if they were local. Use conditional statements or the `--override_repository` flag for this. - -- **Generating or compiling target-specific artifacts and platform mutation**. These operations must be executed via regular build rules. Actions that produce target-specific artifacts for external dependencies must execute during the build. - -To more easily generate pre-checked artifacts for remote execution, you can use `WORKSPACE` rules to emit generated files. You can run those rules on each new execution environment, such as inside each toolchain container, and check the outputs of your remote execution build in to your source repo to reference. - -For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). For local execution, files produced by checking the host environment are used. For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) on an environment variable allows the rule to use files that are checked into the repo. - -The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) that can run both locally and remotely, and perform the necessary processing that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). +If an external dependency executes specific operations dependent on the host +platform, you should split those operations between `WORKSPACE` and build +rules as follows: + +* **Platform inspection and dependency enumeration.** These operations are + safe to execute locally via `WORKSPACE` rules, which can check which + libraries are installed, download packages that must be built, and prepare + required artifacts for compilation. For remote execution, these rules must + also support using pre-checked artifacts to provide the information that + would normally be obtained during host platform inspection. Pre-checked + artifacts allow Bazel to describe dependencies as if they were local. Use + conditional statements or the `--override_repository` flag for this. + +* **Generating or compiling target-specific artifacts and platform mutation**. + These operations must be executed via regular build rules. Actions that + produce target-specific artifacts for external dependencies must execute + during the build. + +To more easily generate pre-checked artifacts for remote execution, you can use +`WORKSPACE` rules to emit generated files. You can run those rules on each new +execution environment, such as inside each toolchain container, and check the +outputs of your remote execution build in to your source repo to reference. + +For example, for Tensorflow's rules for [`cuda`](https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl) +and [`python`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl), +the `WORKSPACE` rules produce the following [`BUILD files`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/third_party/toolchains/cpus/py). +For local execution, files produced by checking the host environment are used. +For remote execution, a [conditional statement](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L304) +on an environment variable allows the rule to use files that are checked into +the repo. + +The `BUILD` files declare [`genrules`](https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L84) +that can run both locally and remotely, and perform the necessary processing +that was previously done via `repository_ctx.symlink` as shown [here](https://github.com/tensorflow/tensorflow/blob/d1ba01f81d8fa1d0171ba9ce871599063d5c7eb9/third_party/gpus/cuda_configure.bzl#L730). diff --git a/remote/sandbox.mdx b/remote/sandbox.mdx index 98485efc..8c3f9565 100644 --- a/remote/sandbox.mdx +++ b/remote/sandbox.mdx @@ -2,143 +2,223 @@ title: 'Troubleshooting Bazel Remote Execution with Docker Sandbox' --- -Bazel builds that succeed locally may fail when executed remotely due to restrictions and requirements that do not affect local builds. The most common causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). -This page describes how to identify and resolve the most common issues that arise with remote execution using the Docker sandbox feature, which imposes restrictions upon the build equal to those of remote execution. This allows you to troubleshoot your build without the need for a remote execution service. -The Docker sandbox feature mimics the restrictions of remote execution as follows: +Bazel builds that succeed locally may fail when executed remotely due to +restrictions and requirements that do not affect local builds. The most common +causes of such failures are described in [Adapting Bazel Rules for Remote Execution](/remote/rules). -- **Build actions execute in toolchain containers.** You can use the same toolchain containers to run your build locally and remotely via a service supporting containerized remote execution. +This page describes how to identify and resolve the most common issues that +arise with remote execution using the Docker sandbox feature, which imposes +restrictions upon the build equal to those of remote execution. This allows you +to troubleshoot your build without the need for a remote execution service. -- **No extraneous data crosses the container boundary.** Only explicitly declared inputs and outputs enter and leave the container, and only after the associated build action successfully completes. +The Docker sandbox feature mimics the restrictions of remote execution as +follows: -- **Each action executes in a fresh container.** A new, unique container is created for each spawned build action. +* **Build actions execute in toolchain containers.** You can use the same + toolchain containers to run your build locally and remotely via a service + supporting containerized remote execution. -Note: Builds take noticeably more time to complete when the Docker sandbox feature is enabled. This is normal. +* **No extraneous data crosses the container boundary.** Only explicitly + declared inputs and outputs enter and leave the container, and only after + the associated build action successfully completes. + +* **Each action executes in a fresh container.** A new, unique container is + created for each spawned build action. + +Note: Builds take noticeably more time to complete when the Docker sandbox +feature is enabled. This is normal. You can troubleshoot these issues using one of the following methods: -- **[Troubleshooting natively.](#troubleshooting-natively)** With this method, Bazel and its build actions run natively on your local machine. The Docker sandbox feature imposes restrictions upon the build equal to those of remote execution. However, this method will not detect local tools, states, and data leaking into your build, which will cause problems with remote execution. +* **[Troubleshooting natively.](#troubleshooting-natively)** With this method, + Bazel and its build actions run natively on your local machine. The Docker + sandbox feature imposes restrictions upon the build equal to those of remote + execution. However, this method will not detect local tools, states, and + data leaking into your build, which will cause problems with remote execution. -- **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** With this method, Bazel and its build actions run inside a Docker container, which allows you to detect tools, states, and data leaking from the local machine into the build in addition to imposing restrictions equal to those of remote execution. This method provides insight into your build even if portions of the build are failing. This method is experimental and not officially supported. +* **[Troubleshooting in a Docker container.](#troubleshooting-docker-container)** + With this method, Bazel and its build actions run inside a Docker container, + which allows you to detect tools, states, and data leaking from the local + machine into the build in addition to imposing restrictions + equal to those of remote execution. This method provides insight into your + build even if portions of the build are failing. This method is experimental + and not officially supported. ## Prerequisites Before you begin troubleshooting, do the following if you have not already done so: -- Install Docker and configure the permissions required to run it. -- Install Bazel 0.14.1 or later. Earlier versions do not support the Docker sandbox feature. -- Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) repo, pinned to the latest release version, to your build's `WORKSPACE` file as described [here](https://releases.bazel.build/bazel-toolchains.html). -- Add flags to your `.bazelrc` file to enable the feature. Create the file in the root directory of your Bazel project if it does not exist. Flags below are a reference sample. Please see the latest [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) file in the bazel-toolchains repo and copy the values of the flags defined there for config `docker-sandbox`. +* Install Docker and configure the permissions required to run it. +* Install Bazel 0.14.1 or later. Earlier versions do not support the Docker + sandbox feature. +* Add the [bazel-toolchains](https://releases.bazel.build/bazel-toolchains.html) + repo, pinned to the latest release version, to your build's `WORKSPACE` file + as described [here](https://releases.bazel.build/bazel-toolchains.html). +* Add flags to your `.bazelrc` file to enable the feature. Create the file in + the root directory of your Bazel project if it does not exist. Flags below + are a reference sample. Please see the latest + [`.bazelrc`](https://github.com/bazelbuild/bazel-toolchains/tree/master/bazelrc) + file in the bazel-toolchains repo and copy the values of the flags defined + there for config `docker-sandbox`. ``` # Docker Sandbox Mode -build:docker-sandbox --host_javabase=<...> -build:docker-sandbox --javabase=<...> -build:docker-sandbox --crosstool_top=<...> -build:docker-sandbox --experimental_docker_image=<...> +build:docker-sandbox --host_javabase=<...> +build:docker-sandbox --javabase=<...> +build:docker-sandbox --crosstool_top=<...> +build:docker-sandbox --experimental_docker_image=<...> build:docker-sandbox --spawn_strategy=docker --strategy=Javac=docker --genrule_strategy=docker build:docker-sandbox --experimental_docker_verbose build:docker-sandbox --experimental_enable_docker_sandbox ``` -Note: The flags referenced in the `.bazelrc` file shown above are configured to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) container. +Note: The flags referenced in the `.bazelrc` file shown above are configured +to run within the [`rbe-ubuntu16-04`](https://console.cloud.google.com/launcher/details/google/rbe-ubuntu16-04) +container. If your rules require additional tools, do the following: -1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) and [building](https://docs.docker.com/engine/reference/commandline/build/) the image locally. +1. Create a custom Docker container by installing tools using a [Dockerfile](https://docs.docker.com/engine/reference/builder/) + and [building](https://docs.docker.com/engine/reference/commandline/build/) + the image locally. + +2. Replace the value of the `--experimental_docker_image` flag above with the + name of your custom container image. -2. Replace the value of the `--experimental_docker_image` flag above with the name of your custom container image. ## Troubleshooting natively -This method executes Bazel and all of its build actions directly on the local machine and is a reliable way to confirm whether your build will succeed when executed remotely. +This method executes Bazel and all of its build actions directly on the local +machine and is a reliable way to confirm whether your build will succeed when +executed remotely. -However, with this method, locally installed tools, binaries, and data may leak into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) in addition to troubleshooting natively. +However, with this method, locally installed tools, binaries, and data may leak +into into your build, especially if it uses [configure-style WORKSPACE rules](/remote/rules#manage-workspace-rules). +Such leaks will cause problems with remote execution; to detect them, [troubleshoot in a Docker container](#troubleshooting-docker-container) +in addition to troubleshooting natively. ### Step 1: Run the build -1. Add the `--config=docker-sandbox` flag to the Bazel command that executes your build. For example: +1. Add the `--config=docker-sandbox` flag to the Bazel command that executes + your build. For example: - ```posix-terminal - bazel --bazelrc=.bazelrc build --config=docker-sandbox <var>target</var> - ``` + ```posix-terminal + bazel --bazelrc=.bazelrc build --config=docker-sandbox target + ``` -2. Run the build and wait for it to complete. The build will run up to four times slower than normal due to the Docker sandbox feature. +2. Run the build and wait for it to complete. The build will run up to four + times slower than normal due to the Docker sandbox feature. You may encounter the following error: -```none +```none {:.devsite-disable-click-to-copy} ERROR: 'docker' is an invalid value for docker spawn strategy. ``` -If you do, run the build again with the `--experimental_docker_verbose` flag. This flag enables verbose error messages. This error is typically caused by a faulty Docker installation or lack of permissions to execute it under the current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). +If you do, run the build again with the `--experimental_docker_verbose` flag. +This flag enables verbose error messages. This error is typically caused by a +faulty Docker installation or lack of permissions to execute it under the +current user account. See the [Docker documentation](https://docs.docker.com/install/linux/linux-postinstall/) +for more information. If problems persist, skip ahead to [Troubleshooting in a Docker container](#troubleshooting-docker-container). ### Step 2: Resolve detected issues The following are the most commonly encountered issues and their workarounds. -- **A file, tool, binary, or resource referenced by the Bazel runfiles tree is missing.**. Confirm that all dependencies of the affected targets have been [explicitly declared](/concepts/dependencies). See [Managing implicit dependencies](/remote/rules#manage-dependencies) for more information. +* **A file, tool, binary, or resource referenced by the Bazel runfiles tree is + missing.**. Confirm that all dependencies of the affected targets have been + [explicitly declared](/concepts/dependencies). See + [Managing implicit dependencies](/remote/rules#manage-dependencies) + for more information. -- **A file, tool, binary, or resource referenced by an absolute path or the `PATH` variable is missing.** Confirm that all required tools are installed within the toolchain container and use [toolchain rules](/extending/toolchains) to properly declare dependencies pointing to the missing resource. See [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) for more information. +* **A file, tool, binary, or resource referenced by an absolute path or the `PATH` + variable is missing.** Confirm that all required tools are installed within + the toolchain container and use [toolchain rules](/extending/toolchains) to properly + declare dependencies pointing to the missing resource. See + [Invoking build tools through toolchain rules](/remote/rules#invoking-build-tools-through-toolchain-rules) + for more information. -- **A binary execution fails.** One of the build rules is referencing a binary incompatible with the execution environment (the Docker container). See [Managing platform-dependent binaries](/remote/rules#manage-binaries) for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +* **A binary execution fails.** One of the build rules is referencing a binary + incompatible with the execution environment (the Docker container). See + [Managing platform-dependent binaries](/remote/rules#manage-binaries) + for more information. If you cannot resolve the issue, contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) + for help. -- **A file from `@local-jdk` is missing or causing errors.** The Java binaries on your local machine are leaking into the build while being incompatible with it. Use [`java_toolchain`](/reference/be/java#java_toolchain) in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. +* **A file from `@local-jdk` is missing or causing errors.** The Java binaries + on your local machine are leaking into the build while being incompatible with + it. Use [`java_toolchain`](/reference/be/java#java_toolchain) + in your rules and targets instead of `@local_jdk`. Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) if you need further help. -- **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. +* **Other errors.** Contact [bazel-discuss@google.com](mailto:bazel-discuss@google.com) for help. ## Troubleshooting in a Docker container -With this method, Bazel runs inside a host Docker container, and Bazel's build actions execute inside individual toolchain containers spawned by the Docker sandbox feature. The sandbox spawns a brand new toolchain container for each build action and only one action executes in each toolchain container. +With this method, Bazel runs inside a host Docker container, and Bazel's build +actions execute inside individual toolchain containers spawned by the Docker +sandbox feature. The sandbox spawns a brand new toolchain container for each +build action and only one action executes in each toolchain container. -This method provides more granular control of tools installed in the host environment. By separating the execution of the build from the execution of its build actions and keeping the installed tooling to a minimum, you can verify whether your build has any dependencies on the local execution environment. +This method provides more granular control of tools installed in the host +environment. By separating the execution of the build from the execution of its +build actions and keeping the installed tooling to a minimum, you can verify +whether your build has any dependencies on the local execution environment. ### Step 1: Build the container -Note: The commands below are tailored specifically for a `debian:stretch` base. For other bases, modify them as necessary. +Note: The commands below are tailored specifically for a `debian:stretch` base. +For other bases, modify them as necessary. -1. Create a `Dockerfile` that creates the Docker container and installs Bazel with a minimal set of build tools: +1. Create a `Dockerfile` that creates the Docker container and installs Bazel + with a minimal set of build tools: - ``` - FROM debian:stretch + ``` + FROM debian:stretch - RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim + RUN apt-get update && apt-get install -y apt-transport-https curl software-properties-common git gcc gnupg2 g++ openjdk-8-jdk-headless python-dev zip wget vim - RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - + RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - - RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" + RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" - RUN apt-get update && apt-get install -y docker-ce + RUN apt-get update && apt-get install -y docker-ce - RUN wget https://releases.bazel.build/<latest Bazel version>/release/bazel-<latest Bazel version>-installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh + RUN wget https://releases.bazel.build//release/bazel--installer-linux-x86_64.sh -O ./bazel-installer.sh && chmod 755 ./bazel-installer.sh - RUN ./bazel-installer.sh - ``` + RUN ./bazel-installer.sh + ``` -2. Build the container as `bazel_container`: +2. Build the container as `bazel_container`: - ```posix-terminal - docker build -t bazel_container - < Dockerfile - ``` + ```posix-terminal + docker build -t bazel_container - < Dockerfile + ``` ### Step 2: Start the container -Start the Docker container using the command shown below. In the command, substitute the path to the source code on your host that you want to build. +Start the Docker container using the command shown below. In the command, +substitute the path to the source code on your host that you want to build. ```posix-terminal docker run -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp:/tmp \ - -v <var>your source code directory</var>:/src \ + -v {{ '' }}your source code directory{{ '' }}:/src \ -w /src \ bazel_container \ /bin/bash ``` -This command runs the container as root, mapping the docker socket, and mounting the `/tmp` directory. This allows Bazel to spawn other Docker containers and to use directories under `/tmp` to share files with those containers. Your source code is available at `/src` inside the container. +This command runs the container as root, mapping the docker socket, and mounting +the `/tmp` directory. This allows Bazel to spawn other Docker containers and to +use directories under `/tmp` to share files with those containers. Your source +code is available at `/src` inside the container. -The command intentionally starts from a `debian:stretch` base container that includes binaries incompatible with the `rbe-ubuntu16-04` container used as a toolchain container. If binaries from the local environment are leaking into the toolchain container, they will cause build errors. +The command intentionally starts from a `debian:stretch` base container that +includes binaries incompatible with the `rbe-ubuntu16-04` container used as a +toolchain container. If binaries from the local environment are leaking into the +toolchain container, they will cause build errors. ### Step 3: Test the container @@ -152,18 +232,28 @@ bazel version ### Step 4: Run the build -Run the build as shown below. The output user is root so that it corresponds to a directory that is accessible with the same absolute path from inside the host container in which Bazel runs, from the toolchain containers spawned by the Docker sandbox feature in which Bazel's build actions are running, and from the local machine on which the host and action containers run. +Run the build as shown below. The output user is root so that it corresponds to +a directory that is accessible with the same absolute path from inside the host +container in which Bazel runs, from the toolchain containers spawned by the Docker +sandbox feature in which Bazel's build actions are running, and from the local +machine on which the host and action containers run. ```posix-terminal -bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox <var>target</var> +bazel --output_user_root=/tmp/bazel_docker_root --bazelrc=.bazelrc \ build --config=docker-sandbox {{ '' }}target{{ '' }} ``` ### Step 5: Resolve detected issues You can resolve build failures as follows: -- If the build fails with an "out of disk space" error, you can increase this limit by starting the host container with the flag `--memory=XX` where `XX` is the allocated disk space in gigabytes. This is experimental and may result in unpredictable behavior. +* If the build fails with an "out of disk space" error, you can increase this + limit by starting the host container with the flag `--memory=XX` where `XX` + is the allocated disk space in gigabytes. This is experimental and may + result in unpredictable behavior. -- If the build fails during the analysis or loading phases, one or more of your build rules declared in the WORKSPACE file are not compatible with remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) for possible causes and workarounds. +* If the build fails during the analysis or loading phases, one or more of + your build rules declared in the WORKSPACE file are not compatible with + remote execution. See [Adapting Bazel Rules for Remote Execution](/remote/rules) + for possible causes and workarounds. -- If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). +* If the build fails for any other reason, see the troubleshooting steps in [Step 2: Resolve detected issues](#start-container). diff --git a/remote/workspace.mdx b/remote/workspace.mdx index 726f8bd3..ae0aea50 100644 --- a/remote/workspace.mdx +++ b/remote/workspace.mdx @@ -2,78 +2,126 @@ title: 'Finding Non-Hermetic Behavior in WORKSPACE Rules' --- + + In the following, a host machine is the machine where Bazel runs. -When using remote execution, the actual build and/or test steps are not happening on the host machine, but are instead sent off to the remote execution system. However, the steps involved in resolving workspace rules are happening on the host machine. If your workspace rules access information about the host machine for use during execution, your build is likely to break due to incompatibilities between the environments. +When using remote execution, the actual build and/or test steps are not +happening on the host machine, but are instead sent off to the remote execution +system. However, the steps involved in resolving workspace rules are happening +on the host machine. If your workspace rules access information about the +host machine for use during execution, your build is likely to break due to +incompatibilities between the environments. + +As part of [adapting Bazel rules for remote +execution](/remote/rules), you need to find such workspace rules +and fix them. This page describes how to find potentially problematic workspace +rules using the workspace log. -As part of [adapting Bazel rules for remote execution](/remote/rules), you need to find such workspace rules and fix them. This page describes how to find potentially problematic workspace rules using the workspace log. ## Finding non-hermetic rules -[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to external workspaces, but they are rich enough to allow arbitrary processing to happen in the process. All related commands are happening locally and can be a potential source of non-hermeticity. Usually non-hermetic behavior is introduced through [`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting with the host machine. +[Workspace rules](/reference/be/workspace) allow the developer to add dependencies to +external workspaces, but they are rich enough to allow arbitrary processing to +happen in the process. All related commands are happening locally and can be a +potential source of non-hermeticity. Usually non-hermetic behavior is +introduced through +[`repository_ctx`](/rules/lib/builtins/repository_ctx) which allows interacting +with the host machine. -Starting with Bazel 0.18, you can get a log of some potentially non-hermetic actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to your Bazel command. Here `[PATH]` is a filename under which the log will be created. +Starting with Bazel 0.18, you can get a log of some potentially non-hermetic +actions by adding the flag `--experimental_workspace_rules_log_file=[PATH]` to +your Bazel command. Here `[PATH]` is a filename under which the log will be +created. Things to note: -- the log captures the events as they are executed. If some steps are cached, they will not show up in the log, so to get a full result, don't forget to run `bazel clean --expunge` beforehand. +* the log captures the events as they are executed. If some steps are + cached, they will not show up in the log, so to get a full result, don't + forget to run `bazel clean --expunge` beforehand. -- Sometimes functions might be re-executed, in which case the related events will show up in the log multiple times. +* Sometimes functions might be re-executed, in which case the related + events will show up in the log multiple times. -- Workspace rules currently only log Starlark events. +* Workspace rules currently only log Starlark events. - Note: These particular rules do not cause hermiticity concerns as long as a hash is specified. + Note: These particular rules do not cause hermiticity concerns as long + as a hash is specified. To find what was executed during workspace initialization: -1. Run `bazel clean --expunge`. This command will clean your local cache and any cached repositories, ensuring that all initialization will be re-run. +1. Run `bazel clean --expunge`. This command will clean your local cache and + any cached repositories, ensuring that all initialization will be re-run. -2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your Bazel command and run the build. +2. Add `--experimental_workspace_rules_log_file=/tmp/workspacelog` to your + Bazel command and run the build. - This produces a binary proto file listing messages of type [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) + This produces a binary proto file listing messages of type + [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) -3. Download the Bazel source code and navigate to the Bazel folder by using the command below. You need the source code to be able to parse the workspace log with the [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). +3. Download the Bazel source code and navigate to the Bazel folder by using + the command below. You need the source code to be able to parse the + workspace log with the + [workspacelog parser](https://source.bazel.build/bazel/+/master:src/tools/workspacelog/). - ```posix-terminal - git clone https://github.com/bazelbuild/bazel.git + ```posix-terminal + git clone https://github.com/bazelbuild/bazel.git - cd bazel - ``` + cd bazel + ``` -4. In the Bazel source code repo, convert the whole workspace log to text. +4. In the Bazel source code repo, convert the whole workspace log to text. - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog > /tmp/workspacelog.txt + ``` -5. The output may be quite verbose and include output from built in Bazel rules. +5. The output may be quite verbose and include output from built in Bazel + rules. - To exclude specific rules from the output, use `--exclude_rule` option. For example: + To exclude specific rules from the output, use `--exclude_rule` option. + For example: - ```posix-terminal - bazel build src/tools/workspacelog:parser + ```posix-terminal + bazel build src/tools/workspacelog:parser - bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ - --exclude_rule "//external:local_config_cc" \ - --exclude_rule "//external:dep" > /tmp/workspacelog.txt - ``` + bazel-bin/src/tools/workspacelog/parser --log_path=/tmp/workspacelog \ + --exclude_rule "//external:local_config_cc" \ + --exclude_rule "//external:dep" > /tmp/workspacelog.txt + ``` -6. Open `/tmp/workspacelog.txt` and check for unsafe operations. +5. Open `/tmp/workspacelog.txt` and check for unsafe operations. -The log consists of [WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) messages outlining certain potentially non-hermetic actions performed on a [`repository_ctx`](/rules/lib/builtins/repository_ctx). +The log consists of +[WorkspaceEvent](https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/debug/workspace_log.proto?q=WorkspaceEvent) +messages outlining certain potentially non-hermetic actions performed on a +[`repository_ctx`](/rules/lib/builtins/repository_ctx). The actions that have been highlighted as potentially non-hermetic are as follows: -- `execute`: executes an arbitrary command on the host environment. Check if these may introduce any dependencies on the host environment. +* `execute`: executes an arbitrary command on the host environment. Check if + these may introduce any dependencies on the host environment. -- `download`, `download_and_extract`: to ensure hermetic builds, make sure that sha256 is specified +* `download`, `download_and_extract`: to ensure hermetic builds, make sure + that sha256 is specified -- `file`, `template`: this is not non-hermetic in itself, but may be a mechanism for introducing dependencies on the host environment into the repository. Ensure that you understand where the input comes from, and that it does not depend on the host environment. +* `file`, `template`: this is not non-hermetic in itself, but may be a mechanism + for introducing dependencies on the host environment into the repository. + Ensure that you understand where the input comes from, and that it does not + depend on the host environment. -- `os`: this is not non-hermetic in itself, but an easy way to get dependencies on the host environment. A hermetic build would generally not call this. In evaluating whether your usage is hermetic, keep in mind that this is running on the host and not on the workers. Getting environment specifics from the host is generally not a good idea for remote builds. +* `os`: this is not non-hermetic in itself, but an easy way to get dependencies + on the host environment. A hermetic build would generally not call this. + In evaluating whether your usage is hermetic, keep in mind that this is + running on the host and not on the workers. Getting environment specifics + from the host is generally not a good idea for remote builds. -- `symlink`: this is normally safe, but look for red flags. Any symlinks to outside the repository or to an absolute path would cause problems on the remote worker. If the symlink is created based on host machine properties it would probably be problematic as well. +* `symlink`: this is normally safe, but look for red flags. Any symlinks to + outside the repository or to an absolute path would cause problems on the + remote worker. If the symlink is created based on host machine properties + it would probably be problematic as well. -- `which`: checking for programs installed on the host is usually problematic since the workers may have different configurations. +* `which`: checking for programs installed on the host is usually problematic + since the workers may have different configurations. diff --git a/rules/bzl-style.mdx b/rules/bzl-style.mdx index 8d85a7b5..65f589f4 100644 --- a/rules/bzl-style.mdx +++ b/rules/bzl-style.mdx @@ -2,49 +2,92 @@ title: '.bzl style guide' --- -This page covers basic style guidelines for Starlark and also includes information on macros and rules. -[Starlark](/rules/language) is a language that defines how software is built, and as such it is both a programming and a configuration language. -You will use Starlark to write `BUILD` files, macros, and build rules. Macros and rules are essentially meta-languages - they define how `BUILD` files are written. `BUILD` files are intended to be simple and repetitive. +This page covers basic style guidelines for Starlark and also includes +information on macros and rules. -All software is read more often than it is written. This is especially true for Starlark, as engineers read `BUILD` files to understand dependencies of their targets and details of their builds. This reading will often happen in passing, in a hurry, or in parallel to accomplishing some other task. Consequently, simplicity and readability are very important so that users can parse and comprehend `BUILD` files quickly. +[Starlark](/rules/language) is a +language that defines how software is built, and as such it is both a +programming and a configuration language. -When a user opens a `BUILD` file, they quickly want to know the list of targets in the file; or review the list of sources of that C++ library; or remove a dependency from that Java binary. Each time you add a layer of abstraction, you make it harder for a user to do these tasks. +You will use Starlark to write `BUILD` files, macros, and build rules. Macros and +rules are essentially meta-languages - they define how `BUILD` files are written. +`BUILD` files are intended to be simple and repetitive. -`BUILD` files are also analyzed and updated by many different tools. Tools may not be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` files simple will allow you to get better tooling. As a code base grows, it becomes more and more frequent to do changes across many `BUILD` files in order to update a library or do a cleanup. +All software is read more often than it is written. This is especially true for +Starlark, as engineers read `BUILD` files to understand dependencies of their +targets and details of their builds. This reading will often happen in passing, +in a hurry, or in parallel to accomplishing some other task. Consequently, +simplicity and readability are very important so that users can parse and +comprehend `BUILD` files quickly. -Important: Do not create a variable or macro just to avoid some amount of repetition in `BUILD` files. Your `BUILD` file should be easily readable both by developers and tools. The [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't really apply here. +When a user opens a `BUILD` file, they quickly want to know the list of targets in +the file; or review the list of sources of that C++ library; or remove a +dependency from that Java binary. Each time you add a layer of abstraction, you +make it harder for a user to do these tasks. + +`BUILD` files are also analyzed and updated by many different tools. Tools may not +be able to edit your `BUILD` file if it uses abstractions. Keeping your `BUILD` +files simple will allow you to get better tooling. As a code base grows, it +becomes more and more frequent to do changes across many `BUILD` files in order to +update a library or do a cleanup. + +Important: Do not create a variable or macro just to avoid some amount of +repetition in `BUILD` files. Your `BUILD` file should be easily readable both by +developers and tools. The +[DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle doesn't +really apply here. ## General advice -- Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) as a formatter and linter. -- Follow [testing guidelines](/rules/testing). +* Use [Buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier#linter) + as a formatter and linter. +* Follow [testing guidelines](/rules/testing). ## Style ### Python style -When in doubt, follow the [PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. In particular, use four rather than two spaces for indentation to follow the Python convention. +When in doubt, follow the +[PEP 8 style guide](https://www.python.org/dev/peps/pep-0008/) where possible. +In particular, use four rather than two spaces for indentation to follow the +Python convention. + +Since +[Starlark is not Python](/rules/language#differences-with-python), +some aspects of Python style do not apply. For example, PEP 8 advises that +comparisons to singletons be done with `is`, which is not an operator in +Starlark. -Since [Starlark is not Python](/rules/language#differences-with-python), some aspects of Python style do not apply. For example, PEP 8 advises that comparisons to singletons be done with `is`, which is not an operator in Starlark. ### Docstring -Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Use a docstring at the top of each `.bzl` file, and a docstring for each public function. +Document files and functions using [docstrings](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). +Use a docstring at the top of each `.bzl` file, and a docstring for each public +function. ### Document rules and aspects -Rules and aspects, along with their attributes, as well as providers and their fields, should be documented using the `doc` argument. +Rules and aspects, along with their attributes, as well as providers and their +fields, should be documented using the `doc` argument. ### Naming convention -- Variables and function names use lowercase with words separated by underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. -- Top-level private values start with one underscore. Bazel enforces that private values cannot be used from other files. Local variables should not use the underscore prefix. +* Variables and function names use lowercase with words separated by + underscores (`[a-z][a-z0-9_]*`), such as `cc_library`. +* Top-level private values start with one underscore. Bazel enforces that + private values cannot be used from other files. Local variables should not + use the underscore prefix. ### Line length -As in `BUILD` files, there is no strict line length limit as labels can be long. When possible, try to use at most 79 characters per line (following Python's style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline should not be enforced strictly: editors should display more than 80 columns, automated changes will frequently introduce longer lines, and humans shouldn't spend time splitting lines that are already readable. +As in `BUILD` files, there is no strict line length limit as labels can be long. +When possible, try to use at most 79 characters per line (following Python's +style guide, [PEP 8](https://www.python.org/dev/peps/pep-0008/)). This guideline +should not be enforced strictly: editors should display more than 80 columns, +automated changes will frequently introduce longer lines, and humans shouldn't +spend time splitting lines that are already readable. ### Keyword arguments @@ -62,41 +105,67 @@ def fct(name, srcs): ### Boolean values -Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values (such as when using a boolean attribute in a rule). +Prefer values `True` and `False` (rather than of `1` and `0`) for boolean values +(such as when using a boolean attribute in a rule). ### Use print only for debugging -Do not use the `print()` function in production code; it is only intended for debugging, and will spam all direct and indirect users of your `.bzl` file. The only exception is that you may submit code that uses `print()` if it is disabled by default and can only be enabled by editing the source -- for example, if all uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to `False`. Be mindful of whether these statements are useful enough to justify their impact on readability. +Do not use the `print()` function in production code; it is only intended for +debugging, and will spam all direct and indirect users of your `.bzl` file. The +only exception is that you may submit code that uses `print()` if it is disabled +by default and can only be enabled by editing the source -- for example, if all +uses of `print()` are guarded by `if DEBUG:` where `DEBUG` is hardcoded to +`False`. Be mindful of whether these statements are useful enough to justify +their impact on readability. ## Macros -A macro is a function which instantiates one or more rules during the loading phase. In general, use rules whenever possible instead of macros. The build graph seen by the user is not the same as the one used by Bazel during the build - macros are expanded *before Bazel does any build graph analysis.* - -Because of this, when something goes wrong, the user will need to understand your macro's implementation to troubleshoot build problems. Additionally, `bazel query` results can be hard to interpret because targets shown in the results come from macro expansion. Finally, aspects are not aware of macros, so tooling depending on aspects (IDEs and others) might fail. - -A safe use for macros is for defining additional targets intended to be referenced directly at the Bazel CLI or in BUILD files: In that case, only the *end users* of those targets need to know about them, and any build problems introduced by macros are never far from their usage. - -For macros that define generated targets (implementation details of the macro which are not supposed to be referred to at the CLI or depended on by targets not instantiated by that macro), follow these best practices: - -- A macro should take a `name` argument and define a target with that name. That target becomes that macro's *main target*. - -- Generated targets, that is all other targets defined by a macro, should: - - - Have their names prefixed by `<name>`. For example, using `name = '%s_bar' % (name)`. - - Have restricted visibility (`//visibility:private`), and - - Have a `manual` tag to avoid expansion in wildcard targets (`:all`, `...`, `:*`, etc). - -- The `name` should only be used to derive names of targets defined by the macro, and not for anything else. For example, don't use the name to derive a dependency or input file that is not generated by the macro itself. - -- All the targets created in the macro should be coupled in some way to the main target. - -- Conventionally, `name` should be the first argument when defining a macro. - -- Keep the parameter names in the macro consistent. If a parameter is passed as an attribute value to the main target, keep its name the same. If a macro parameter serves the same purpose as a common rule attribute, such as `deps`, name as you would the attribute (see below). - -- When calling a macro, use only keyword arguments. This is consistent with rules, and greatly improves readability. - -Engineers often write macros when the Starlark API of relevant rules is insufficient for their specific use case, regardless of whether the rule is defined within Bazel in native code, or in Starlark. If you're facing this problem, ask the rule author if they can extend the API to accomplish your goals. +A macro is a function which instantiates one or more rules during the loading +phase. In general, use rules whenever possible instead of macros. The build +graph seen by the user is not the same as the one used by Bazel during the +build - macros are expanded *before Bazel does any build graph analysis.* + +Because of this, when something goes wrong, the user will need to understand +your macro's implementation to troubleshoot build problems. Additionally, `bazel +query` results can be hard to interpret because targets shown in the results +come from macro expansion. Finally, aspects are not aware of macros, so tooling +depending on aspects (IDEs and others) might fail. + +A safe use for macros is for defining additional targets intended to be +referenced directly at the Bazel CLI or in BUILD files: In that case, only the +*end users* of those targets need to know about them, and any build problems +introduced by macros are never far from their usage. + +For macros that define generated targets (implementation details of the macro +which are not supposed to be referred to at the CLI or depended on by targets +not instantiated by that macro), follow these best practices: + +* A macro should take a `name` argument and define a target with that name. + That target becomes that macro's *main target*. +* Generated targets, that is all other targets defined by a macro, should: + * Have their names prefixed by ``. For example, using + `name = '%s_bar' % (name)`. + * Have restricted visibility (`//visibility:private`), and + * Have a `manual` tag to avoid expansion in wildcard targets (`:all`, + `...`, `:*`, etc). +* The `name` should only be used to derive names of targets defined by the + macro, and not for anything else. For example, don't use the name to derive + a dependency or input file that is not generated by the macro itself. +* All the targets created in the macro should be coupled in some way to the + main target. +* Conventionally, `name` should be the first argument when defining a macro. +* Keep the parameter names in the macro consistent. If a parameter is passed + as an attribute value to the main target, keep its name the same. If a macro + parameter serves the same purpose as a common rule attribute, such as + `deps`, name as you would the attribute (see below). +* When calling a macro, use only keyword arguments. This is consistent with + rules, and greatly improves readability. + +Engineers often write macros when the Starlark API of relevant rules is +insufficient for their specific use case, regardless of whether the rule is +defined within Bazel in native code, or in Starlark. If you're facing this +problem, ask the rule author if they can extend the API to accomplish your +goals. As a rule of thumb, the more macros resemble the rules, the better. @@ -104,28 +173,40 @@ See also [macros](/extending/macros#conventions). ## Rules -- Rules, aspects, and their attributes should use lower\_case names ("snake case"). - -- Rule names are nouns that describe the main kind of artifact produced by the rule, from the point of view of its dependencies (or for leaf rules, the user). This is not necessarily a file suffix. For instance, a rule that produces C++ artifacts meant to be used as Python extensions might be called `py_extension`. For most languages, typical rules include: - - - `*_library` - a compilation unit or "module". - - `*_binary` - a target producing an executable or a deployment unit. - - `*_test` - a test target. This can include multiple tests. Expect all tests in a `*_test` target to be variations on the same theme, for example, testing a single library. - - `*_import`: a target encapsulating a pre-compiled artifact, such as a `.jar`, or a `.dll` that is used during compilation. - -- Use consistent names and types for attributes. Some generally applicable attributes include: - - - `srcs`: `label_list`, allowing files: source files, typically human-authored. - - `deps`: `label_list`, typically *not* allowing files: compilation dependencies. - - `data`: `label_list`, allowing files: data files, such as test data etc. - - `runtime_deps`: `label_list`: runtime dependencies that are not needed for compilation. - -- For any attributes with non-obvious behavior (for example, string templates with special substitutions, or tools that are invoked with specific requirements), provide documentation using the `doc` keyword argument to the attribute's declaration (`attr.label_list()` or similar). - -- Rule implementation functions should almost always be private functions (named with a leading underscore). A common style is to give the implementation function for `myrule` the name `_myrule_impl`. - -- Pass information between your rules using a well-defined [provider](/extending/rules#providers) interface. Declare and document provider fields. - -- Design your rule with extensibility in mind. Consider that other rules might want to interact with your rule, access your providers, and reuse the actions you create. - -- Follow [performance guidelines](/rules/performance) in your rules. +* Rules, aspects, and their attributes should use lower_case names ("snake + case"). +* Rule names are nouns that describe the main kind of artifact produced by the + rule, from the point of view of its dependencies (or for leaf rules, the + user). This is not necessarily a file suffix. For instance, a rule that + produces C++ artifacts meant to be used as Python extensions might be called + `py_extension`. For most languages, typical rules include: + * `*_library` - a compilation unit or "module". + * `*_binary` - a target producing an executable or a deployment unit. + * `*_test` - a test target. This can include multiple tests. Expect all + tests in a `*_test` target to be variations on the same theme, for + example, testing a single library. + * `*_import`: a target encapsulating a pre-compiled artifact, such as a + `.jar`, or a `.dll` that is used during compilation. +* Use consistent names and types for attributes. Some generally applicable + attributes include: + * `srcs`: `label_list`, allowing files: source files, typically + human-authored. + * `deps`: `label_list`, typically *not* allowing files: compilation + dependencies. + * `data`: `label_list`, allowing files: data files, such as test data etc. + * `runtime_deps`: `label_list`: runtime dependencies that are not needed + for compilation. +* For any attributes with non-obvious behavior (for example, string templates + with special substitutions, or tools that are invoked with specific + requirements), provide documentation using the `doc` keyword argument to the + attribute's declaration (`attr.label_list()` or similar). +* Rule implementation functions should almost always be private functions + (named with a leading underscore). A common style is to give the + implementation function for `myrule` the name `_myrule_impl`. +* Pass information between your rules using a well-defined + [provider](/extending/rules#providers) interface. Declare and document provider + fields. +* Design your rule with extensibility in mind. Consider that other rules might + want to interact with your rule, access your providers, and reuse the + actions you create. +* Follow [performance guidelines](/rules/performance) in your rules. diff --git a/rules/challenges.mdx b/rules/challenges.mdx index e31c5d45..10ff7371 100644 --- a/rules/challenges.mdx +++ b/rules/challenges.mdx @@ -2,88 +2,222 @@ title: 'Challenges of Writing Rules' --- -This page gives a high-level overview of the specific issues and challenges of writing efficient Bazel rules. + + +This page gives a high-level overview of the specific issues and challenges +of writing efficient Bazel rules. ## Summary Requirements -- Assumption: Aim for Correctness, Throughput, Ease of Use & Latency -- Assumption: Large Scale Repositories -- Assumption: BUILD-like Description Language -- Historic: Hard Separation between Loading, Analysis, and Execution is Outdated, but still affects the API -- Intrinsic: Remote Execution and Caching are Hard -- Intrinsic: Using Change Information for Correct and Fast Incremental Builds requires Unusual Coding Patterns -- Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard +* Assumption: Aim for Correctness, Throughput, Ease of Use & Latency +* Assumption: Large Scale Repositories +* Assumption: BUILD-like Description Language +* Historic: Hard Separation between Loading, Analysis, and Execution is + Outdated, but still affects the API +* Intrinsic: Remote Execution and Caching are Hard +* Intrinsic: Using Change Information for Correct and Fast Incremental Builds + requires Unusual Coding Patterns +* Intrinsic: Avoiding Quadratic Time and Memory Consumption is Hard ## Assumptions -Here are some assumptions made about the build system, such as need for correctness, ease of use, throughput, and large scale repositories. The following sections address these assumptions and offer guidelines to ensure rules are written in an effective manner. +Here are some assumptions made about the build system, such as need for +correctness, ease of use, throughput, and large scale repositories. The +following sections address these assumptions and offer guidelines to ensure +rules are written in an effective manner. ### Aim for correctness, throughput, ease of use & latency -We assume that the build system needs to be first and foremost correct with respect to incremental builds. For a given source tree, the output of the same build should always be the same, regardless of what the output tree looks like. In the first approximation, this means Bazel needs to know every single input that goes into a given build step, such that it can rerun that step if any of the inputs change. There are limits to how correct Bazel can get, as it leaks some information such as date / time of the build, and ignores certain types of changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) helps ensure correctness by preventing reads to undeclared input files. Besides the intrinsic limits of the system, there are a few known correctness issues, most of which are related to Fileset or the C++ rules, which are both hard problems. We have long-term efforts to fix these. - -The second goal of the build system is to have high throughput; we are permanently pushing the boundaries of what can be done within the current machine allocation for a remote execution service. If the remote execution service gets overloaded, nobody can get work done. - -Ease of use comes next. Of multiple correct approaches with the same (or similar) footprint of the remote execution service, we choose the one that is easier to use. - -Latency denotes the time it takes from starting a build to getting the intended result, whether that is a test log from a passing or failing test, or an error message that a `BUILD` file has a typo. - -Note that these goals often overlap; latency is as much a function of throughput of the remote execution service as is correctness relevant for ease of use. +We assume that the build system needs to be first and foremost correct with +respect to incremental builds. For a given source tree, the output of the +same build should always be the same, regardless of what the output tree looks +like. In the first approximation, this means Bazel needs to know every single +input that goes into a given build step, such that it can rerun that step if any +of the inputs change. There are limits to how correct Bazel can get, as it leaks +some information such as date / time of the build, and ignores certain types of +changes such as changes to file attributes. [Sandboxing](/docs/sandboxing) +helps ensure correctness by preventing reads to undeclared input files. Besides +the intrinsic limits of the system, there are a few known correctness issues, +most of which are related to Fileset or the C++ rules, which are both hard +problems. We have long-term efforts to fix these. + +The second goal of the build system is to have high throughput; we are +permanently pushing the boundaries of what can be done within the current +machine allocation for a remote execution service. If the remote execution +service gets overloaded, nobody can get work done. + +Ease of use comes next. Of multiple correct approaches with the same (or +similar) footprint of the remote execution service, we choose the one that is +easier to use. + +Latency denotes the time it takes from starting a build to getting the intended +result, whether that is a test log from a passing or failing test, or an error +message that a `BUILD` file has a typo. + +Note that these goals often overlap; latency is as much a function of throughput +of the remote execution service as is correctness relevant for ease of use. ### Large scale repositories -The build system needs to operate at the scale of large repositories where large scale means that it does not fit on a single hard drive, so it is impossible to do a full checkout on virtually all developer machines. A medium-sized build will need to read and parse tens of thousands of `BUILD` files, and evaluate hundreds of thousands of globs. While it is theoretically possible to read all `BUILD` files on a single machine, we have not yet been able to do so within a reasonable amount of time and memory. As such, it is critical that `BUILD` files can be loaded and parsed independently. +The build system needs to operate at the scale of large repositories where large +scale means that it does not fit on a single hard drive, so it is impossible to +do a full checkout on virtually all developer machines. A medium-sized build +will need to read and parse tens of thousands of `BUILD` files, and evaluate +hundreds of thousands of globs. While it is theoretically possible to read all +`BUILD` files on a single machine, we have not yet been able to do so within a +reasonable amount of time and memory. As such, it is critical that `BUILD` files +can be loaded and parsed independently. ### BUILD-like description language -In this context, we assume a configuration language that is roughly similar to `BUILD` files in declaration of library and binary rules and their interdependencies. `BUILD` files can be read and parsed independently, and we avoid even looking at source files whenever we can (except for existence). +In this context, we assume a configuration language that is +roughly similar to `BUILD` files in declaration of library and binary rules +and their interdependencies. `BUILD` files can be read and parsed independently, +and we avoid even looking at source files whenever we can (except for +existence). ## Historic -There are differences between Bazel versions that cause challenges and some of these are outlined in the following sections. +There are differences between Bazel versions that cause challenges and some +of these are outlined in the following sections. ### Hard separation between loading, analysis, and execution is outdated but still affects the API -Technically, it is sufficient for a rule to know the input and output files of an action just before the action is sent to remote execution. However, the original Bazel code base had a strict separation of loading packages, then analyzing rules using a configuration (command-line flags, essentially), and only then running any actions. This distinction is still part of the rules API today, even though the core of Bazel no longer requires it (more details below). - -That means that the rules API requires a declarative description of the rule interface (what attributes it has, types of attributes). There are some exceptions where the API allows custom code to run during the loading phase to compute implicit names of output files and implicit values of attributes. For example, a java\_library rule named 'foo' implicitly generates an output named 'libfoo.jar', which can be referenced from other rules in the build graph. - -Furthermore, the analysis of a rule cannot read any source files or inspect the output of an action; instead, it needs to generate a partial directed bipartite graph of build steps and output file names that is only determined from the rule itself and its dependencies. +Technically, it is sufficient for a rule to know the input and output files of +an action just before the action is sent to remote execution. However, the +original Bazel code base had a strict separation of loading packages, then +analyzing rules using a configuration (command-line flags, essentially), and +only then running any actions. This distinction is still part of the rules API +today, even though the core of Bazel no longer requires it (more details below). + +That means that the rules API requires a declarative description of the rule +interface (what attributes it has, types of attributes). There are some +exceptions where the API allows custom code to run during the loading phase to +compute implicit names of output files and implicit values of attributes. For +example, a java_library rule named 'foo' implicitly generates an output named +'libfoo.jar', which can be referenced from other rules in the build graph. + +Furthermore, the analysis of a rule cannot read any source files or inspect the +output of an action; instead, it needs to generate a partial directed bipartite +graph of build steps and output file names that is only determined from the rule +itself and its dependencies. ## Intrinsic -There are some intrinsic properties that make writing rules challenging and some of the most common ones are described in the following sections. +There are some intrinsic properties that make writing rules challenging and +some of the most common ones are described in the following sections. ### Remote execution and caching are hard -Remote execution and caching improve build times in large repositories by roughly two orders of magnitude compared to running the build on a single machine. However, the scale at which it needs to perform is staggering: Google's remote execution service is designed to handle a huge number of requests per second, and the protocol carefully avoids unnecessary roundtrips as well as unnecessary work on the service side. +Remote execution and caching improve build times in large repositories by +roughly two orders of magnitude compared to running the build on a single +machine. However, the scale at which it needs to perform is staggering: Google's +remote execution service is designed to handle a huge number of requests per +second, and the protocol carefully avoids unnecessary roundtrips as well as +unnecessary work on the service side. -At this time, the protocol requires that the build system knows all inputs to a given action ahead of time; the build system then computes a unique action fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, the scheduler replies with the digests of the output files; the files itself are addressed by digest later on. However, this imposes restrictions on the Bazel rules, which need to declare all input files ahead of time. +At this time, the protocol requires that the build system knows all inputs to a +given action ahead of time; the build system then computes a unique action +fingerprint, and asks the scheduler for a cache hit. If a cache hit is found, +the scheduler replies with the digests of the output files; the files itself are +addressed by digest later on. However, this imposes restrictions on the Bazel +rules, which need to declare all input files ahead of time. ### Using change information for correct and fast incremental builds requires unusual coding patterns -Above, we argued that in order to be correct, Bazel needs to know all the input files that go into a build step in order to detect whether that build step is still up-to-date. The same is true for package loading and rule analysis, and we have designed [Skyframe](/reference/skyframe) to handle this in general. Skyframe is a graph library and evaluation framework that takes a goal node (such as 'build //foo with these options'), and breaks it down into its constituent parts, which are then evaluated and combined to yield this result. As part of this process, Skyframe reads packages, analyzes rules, and executes actions. - -At each node, Skyframe tracks exactly which nodes any given node used to compute its own output, all the way from the goal node down to the input files (which are also Skyframe nodes). Having this graph explicitly represented in memory allows the build system to identify exactly which nodes are affected by a given change to an input file (including creation or deletion of an input file), doing the minimal amount of work to restore the output tree to its intended state. - -As part of this, each node performs a dependency discovery process. Each node can declare dependencies, and then use the contents of those dependencies to declare even further dependencies. In principle, this maps well to a thread-per-node model. However, medium-sized builds contain hundreds of thousands of Skyframe nodes, which isn't easily possible with current Java technology (and for historical reasons, we're currently tied to using Java, so no lightweight threads and no continuations). - -Instead, Bazel uses a fixed-size thread pool. However, that means that if a node declares a dependency that isn't available yet, we may have to abort that evaluation and restart it (possibly in another thread), when the dependency is available. This, in turn, means that nodes should not do this excessively; a node that declares N dependencies serially can potentially be restarted N times, costing O(N^2) time. Instead, we aim for up-front bulk declaration of dependencies, which sometimes requires reorganizing the code, or even splitting a node into multiple nodes to limit the number of restarts. - -Note that this technology isn't currently available in the rules API; instead, the rules API is still defined using the legacy concepts of loading, analysis, and execution phases. However, a fundamental restriction is that all accesses to other nodes have to go through the framework so that it can track the corresponding dependencies. Regardless of the language in which the build system is implemented or in which the rules are written (they don't have to be the same), rule authors must not use standard libraries or patterns that bypass Skyframe. For Java, that means avoiding java.io.File as well as any form of reflection, and any library that does either. Libraries that support dependency injection of these low-level interfaces still need to be setup correctly for Skyframe. - -This strongly suggests to avoid exposing rule authors to a full language runtime in the first place. The danger of accidental use of such APIs is just too big - several Bazel bugs in the past were caused by rules using unsafe APIs, even though the rules were written by the Bazel team or other domain experts. +Above, we argued that in order to be correct, Bazel needs to know all the input +files that go into a build step in order to detect whether that build step is +still up-to-date. The same is true for package loading and rule analysis, and we +have designed [Skyframe](/reference/skyframe) to handle this +in general. Skyframe is a graph library and evaluation framework that takes a +goal node (such as 'build //foo with these options'), and breaks it down into +its constituent parts, which are then evaluated and combined to yield this +result. As part of this process, Skyframe reads packages, analyzes rules, and +executes actions. + +At each node, Skyframe tracks exactly which nodes any given node used to compute +its own output, all the way from the goal node down to the input files (which +are also Skyframe nodes). Having this graph explicitly represented in memory +allows the build system to identify exactly which nodes are affected by a given +change to an input file (including creation or deletion of an input file), doing +the minimal amount of work to restore the output tree to its intended state. + +As part of this, each node performs a dependency discovery process. Each +node can declare dependencies, and then use the contents of those dependencies +to declare even further dependencies. In principle, this maps well to a +thread-per-node model. However, medium-sized builds contain hundreds of +thousands of Skyframe nodes, which isn't easily possible with current Java +technology (and for historical reasons, we're currently tied to using Java, so +no lightweight threads and no continuations). + +Instead, Bazel uses a fixed-size thread pool. However, that means that if a node +declares a dependency that isn't available yet, we may have to abort that +evaluation and restart it (possibly in another thread), when the dependency is +available. This, in turn, means that nodes should not do this excessively; a +node that declares N dependencies serially can potentially be restarted N times, +costing O(N^2) time. Instead, we aim for up-front bulk declaration of +dependencies, which sometimes requires reorganizing the code, or even splitting +a node into multiple nodes to limit the number of restarts. + +Note that this technology isn't currently available in the rules API; instead, +the rules API is still defined using the legacy concepts of loading, analysis, +and execution phases. However, a fundamental restriction is that all accesses to +other nodes have to go through the framework so that it can track the +corresponding dependencies. Regardless of the language in which the build system +is implemented or in which the rules are written (they don't have to be the +same), rule authors must not use standard libraries or patterns that bypass +Skyframe. For Java, that means avoiding java.io.File as well as any form of +reflection, and any library that does either. Libraries that support dependency +injection of these low-level interfaces still need to be setup correctly for +Skyframe. + +This strongly suggests to avoid exposing rule authors to a full language runtime +in the first place. The danger of accidental use of such APIs is just too big - +several Bazel bugs in the past were caused by rules using unsafe APIs, even +though the rules were written by the Bazel team or other domain experts. ### Avoiding quadratic time and memory consumption is hard -To make matters worse, apart from the requirements imposed by Skyframe, the historical constraints of using Java, and the outdatedness of the rules API, accidentally introducing quadratic time or memory consumption is a fundamental problem in any build system based on library and binary rules. There are two very common patterns that introduce quadratic memory consumption (and therefore quadratic time consumption). - -1. Chains of Library Rules - Consider the case of a chain of library rules A depends on B, depends on C, and so on. Then, we want to compute some property over the transitive closure of these rules, such as the Java runtime classpath, or the C++ linker command for each library. Naively, we might take a standard list implementation; however, this already introduces quadratic memory consumption: the first library contains one entry on the classpath, the second two, the third three, and so on, for a total of 1+2+3+...+N = O(N^2) entries. - -2. Binary Rules Depending on the Same Library Rules - Consider the case where a set of binaries that depend on the same library rules — such as if you have a number of test rules that test the same library code. Let's say out of N rules, half the rules are binary rules, and the other half library rules. Now consider that each binary makes a copy of some property computed over the transitive closure of library rules, such as the Java runtime classpath, or the C++ linker command line. For example, it could expand the command line string representation of the C++ link action. N/2 copies of N/2 elements is O(N^2) memory. +To make matters worse, apart from the requirements imposed by Skyframe, the +historical constraints of using Java, and the outdatedness of the rules API, +accidentally introducing quadratic time or memory consumption is a fundamental +problem in any build system based on library and binary rules. There are two +very common patterns that introduce quadratic memory consumption (and therefore +quadratic time consumption). + +1. Chains of Library Rules - +Consider the case of a chain of library rules A depends on B, depends on C, and +so on. Then, we want to compute some property over the transitive closure of +these rules, such as the Java runtime classpath, or the C++ linker command for +each library. Naively, we might take a standard list implementation; however, +this already introduces quadratic memory consumption: the first library +contains one entry on the classpath, the second two, the third three, and so +on, for a total of 1+2+3+...+N = O(N^2) entries. + +2. Binary Rules Depending on the Same Library Rules - +Consider the case where a set of binaries that depend on the same library +rules — such as if you have a number of test rules that test the same +library code. Let's say out of N rules, half the rules are binary rules, and +the other half library rules. Now consider that each binary makes a copy of +some property computed over the transitive closure of library rules, such as +the Java runtime classpath, or the C++ linker command line. For example, it +could expand the command line string representation of the C++ link action. N/2 +copies of N/2 elements is O(N^2) memory. #### Custom collections classes to avoid quadratic complexity -Bazel is heavily affected by both of these scenarios, so we introduced a set of custom collection classes that effectively compress the information in memory by avoiding the copy at each step. Almost all of these data structures have set semantics, so we called it [depset](/rules/lib/depset) (also known as `NestedSet` in the internal implementation). The majority of changes to reduce Bazel's memory consumption over the past several years were changes to use depsets instead of whatever was previously used. - -Unfortunately, usage of depsets does not automatically solve all the issues; in particular, even just iterating over a depset in each rule re-introduces quadratic time consumption. Internally, NestedSets also has some helper methods to facilitate interoperability with normal collections classes; unfortunately, accidentally passing a NestedSet to one of these methods leads to copying behavior, and reintroduces quadratic memory consumption. +Bazel is heavily affected by both of these scenarios, so we introduced a set of +custom collection classes that effectively compress the information in memory by +avoiding the copy at each step. Almost all of these data structures have set +semantics, so we called it +[depset](/rules/lib/depset) +(also known as `NestedSet` in the internal implementation). The majority of +changes to reduce Bazel's memory consumption over the past several years were +changes to use depsets instead of whatever was previously used. + +Unfortunately, usage of depsets does not automatically solve all the issues; +in particular, even just iterating over a depset in each rule re-introduces +quadratic time consumption. Internally, NestedSets also has some helper methods +to facilitate interoperability with normal collections classes; unfortunately, +accidentally passing a NestedSet to one of these methods leads to copying +behavior, and reintroduces quadratic memory consumption. diff --git a/rules/deploying.mdx b/rules/deploying.mdx index 4395dd28..3fe2c869 100644 --- a/rules/deploying.mdx +++ b/rules/deploying.mdx @@ -2,30 +2,48 @@ title: 'Deploying Rules' --- -This page is for rule writers who are planning to make their rules available to others. -We recommend you start a new ruleset from the template repository: [https://github.com/bazel-contrib/rules-template](https://github.com/bazel-contrib/rules-template) That template follows the recommendations below, and includes API documentation generation and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. + +This page is for rule writers who are planning to make their rules available +to others. + +We recommend you start a new ruleset from the template repository: +https://github.com/bazel-contrib/rules-template +That template follows the recommendations below, and includes API documentation generation +and sets up a CI/CD pipeline to make it trivial to distribute your ruleset. ## Hosting and naming rules -New rules should go into their own GitHub repository under your organization. Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) organization. +New rules should go into their own GitHub repository under your organization. +Start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) +if you feel like your rules belong in the [bazelbuild](https://github.com/bazelbuild) +organization. -Repository names for Bazel rules are standardized on the following format: `$ORGANIZATION/rules_$NAME`. See [examples on GitHub](https://github.com/search?q=rules+bazel\&type=Repositories). For consistency, you should follow this same format when publishing your Bazel rules. +Repository names for Bazel rules are standardized on the following format: +`$ORGANIZATION/rules_$NAME`. +See [examples on GitHub](https://github.com/search?q=rules+bazel&type=Repositories). +For consistency, you should follow this same format when publishing your Bazel rules. -Make sure to use a descriptive GitHub repository description and `README.md` title, example: +Make sure to use a descriptive GitHub repository description and `README.md` +title, example: -- Repository name: `bazelbuild/rules_go` -- Repository description: *Go rules for Bazel* -- Repository tags: `golang`, `bazel` -- `README.md` header: *Go rules for [Bazel](https://bazel.build)* (note the link to [https://bazel.build](https://bazel.build) which will guide users who are unfamiliar with Bazel to the right place) +* Repository name: `bazelbuild/rules_go` +* Repository description: *Go rules for Bazel* +* Repository tags: `golang`, `bazel` +* `README.md` header: *Go rules for [Bazel](https://bazel.build)* +(note the link to https://bazel.build which will guide users who are unfamiliar +with Bazel to the right place) -Rules can be grouped either by language (such as Scala), runtime platform (such as Android), or framework (such as Spring). +Rules can be grouped either by language (such as Scala), runtime platform +(such as Android), or framework (such as Spring). ## Repository content -Every rule repository should have a certain layout so that users can quickly understand new rules. +Every rule repository should have a certain layout so that users can quickly +understand new rules. -For example, when writing new rules for the (make-believe) `mockascript` language, the rule repository would have the following structure: +For example, when writing new rules for the (make-believe) +`mockascript` language, the rule repository would have the following structure: ``` / @@ -53,9 +71,17 @@ For example, when writing new rules for the (make-believe) `mockascript` languag ### MODULE.bazel -In the project's `MODULE.bazel`, you should define the name that users will use to reference your rules. If your rules belong to the [bazelbuild](https://github.com/bazelbuild) organization, you must use `rules_<lang>` (such as `rules_mockascript`). Otherwise, you should name your repository `<org>_rules_<lang>` (such as `build_stack_rules_proto`). Please start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) if you feel like your rules should follow the convention for rules in the [bazelbuild](https://github.com/bazelbuild) organization. +In the project's `MODULE.bazel`, you should define the name that users will use +to reference your rules. If your rules belong to the +[bazelbuild](https://github.com/bazelbuild) organization, you must use +`rules_` (such as `rules_mockascript`). Otherwise, you should name your +repository `_rules_` (such as `build_stack_rules_proto`). Please +start a thread on [GitHub](https://github.com/bazelbuild/bazel/discussions) +if you feel like your rules should follow the convention for rules in the +[bazelbuild](https://github.com/bazelbuild) organization. -In the following sections, assume the repository belongs to the [bazelbuild](https://github.com/bazelbuild) organization. +In the following sections, assume the repository belongs to the +[bazelbuild](https://github.com/bazelbuild) organization. ``` module(name = "rules_mockascript") @@ -63,11 +89,16 @@ module(name = "rules_mockascript") ### README -At the top level, there should be a `README` that contains a brief description of your ruleset, and the API users should expect. +At the top level, there should be a `README` that contains a brief description +of your ruleset, and the API users should expect. ### Rules -Often times there will be multiple rules provided by your repository. Create a directory named by the language and provide an entry point - `defs.bzl` file exporting all rules (also include a `BUILD` file so the directory is a package). For `rules_mockascript` that means there will be a directory named `mockascript`, and a `BUILD` file and a `defs.bzl` file inside: +Often times there will be multiple rules provided by your repository. Create a +directory named by the language and provide an entry point - `defs.bzl` file +exporting all rules (also include a `BUILD` file so the directory is a package). +For `rules_mockascript` that means there will be a directory named +`mockascript`, and a `BUILD` file and a `defs.bzl` file inside: ``` / @@ -78,7 +109,11 @@ Often times there will be multiple rules provided by your repository. Create a d ### Constraints -If your rule defines [toolchain](/extending/toolchains) rules, it's possible that you'll need to define custom `constraint_setting`s and/or `constraint_value`s. Put these into a `//<LANG>/constraints` package. Your directory structure will look like this: +If your rule defines +[toolchain](/extending/toolchains) rules, +it's possible that you'll need to define custom `constraint_setting`s and/or +`constraint_value`s. Put these into a `///constraints` package. Your +directory structure will look like this: ``` / @@ -89,58 +124,100 @@ If your rule defines [toolchain](/extending/toolchains) rules, it's possible tha defs.bzl ``` -Please read [github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) for best practices, and to see what constraints are already present, and consider contributing your constraints there if they are language independent. Be mindful of introducing custom constraints, all users of your rules will use them to perform platform specific logic in their `BUILD` files (for example, using [selects](/reference/be/functions#select)). With custom constraints, you define a language that the whole Bazel ecosystem will speak. +Please read +[github.com/bazelbuild/platforms](https://github.com/bazelbuild/platforms) +for best practices, and to see what constraints are already present, and +consider contributing your constraints there if they are language independent. +Be mindful of introducing custom constraints, all users of your rules will +use them to perform platform specific logic in their `BUILD` files (for example, +using [selects](/reference/be/functions#select)). +With custom constraints, you define a language that the whole Bazel ecosystem +will speak. ### Runfiles library -If your rule provides a standard library for accessing runfiles, it should be in the form of a library target located at `//<LANG>/runfiles` (an abbreviation of `//<LANG>/runfiles:runfiles`). User targets that need to access their data dependencies will typically add this target to their `deps` attribute. +If your rule provides a standard library for accessing runfiles, it should be +in the form of a library target located at `///runfiles` (an abbreviation +of `///runfiles:runfiles`). User targets that need to access their data +dependencies will typically add this target to their `deps` attribute. ### Repository rules #### Dependencies -Your rules might have external dependencies, which you'll need to specify in your MODULE.bazel file. +Your rules might have external dependencies, which you'll need to specify in +your MODULE.bazel file. #### Registering toolchains -Your rules might also register toolchains, which you can also specify in the MODULE.bazel file. +Your rules might also register toolchains, which you can also specify in the +MODULE.bazel file. + +Note that in order to resolve toolchains in the analysis phase Bazel needs to +analyze all `toolchain` targets that are registered. Bazel will not need to +analyze all targets referenced by `toolchain.toolchain` attribute. If in order +to register toolchains you need to perform complex computation in the +repository, consider splitting the repository with `toolchain` targets from the +repository with `_toolchain` targets. Former will be always fetched, and +the latter will only be fetched when user actually needs to build `` code. -Note that in order to resolve toolchains in the analysis phase Bazel needs to analyze all `toolchain` targets that are registered. Bazel will not need to analyze all targets referenced by `toolchain.toolchain` attribute. If in order to register toolchains you need to perform complex computation in the repository, consider splitting the repository with `toolchain` targets from the repository with `<LANG>_toolchain` targets. Former will be always fetched, and the latter will only be fetched when user actually needs to build `<LANG>` code. #### Release snippet -In your release announcement provide a snippet that your users can copy-paste into their `MODULE.bazel` file. This snippet in general will look as follows: +In your release announcement provide a snippet that your users can copy-paste +into their `MODULE.bazel` file. This snippet in general will look as follows: ``` -bazel_dep(name = "rules_<LANG>", version = "<VERSION>") +bazel_dep(name = "rules_", version = "") ``` + ### Tests -There should be tests that verify that the rules are working as expected. This can either be in the standard location for the language the rules are for or a `tests/` directory at the top level. +There should be tests that verify that the rules are working as expected. This +can either be in the standard location for the language the rules are for or a +`tests/` directory at the top level. ### Examples (optional) -It is useful to users to have an `examples/` directory that shows users a couple of basic ways that the rules can be used. +It is useful to users to have an `examples/` directory that shows users a couple +of basic ways that the rules can be used. ## CI/CD -Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. See comments in the rules-template repo for more information. +Many rulesets use GitHub Actions. See the configuration used in the [rules-template](https://github.com/bazel-contrib/rules-template/tree/main/.github/workflows) repo, which are simplified using a "reusable workflow" hosted in the bazel-contrib +org. `ci.yaml` runs tests on each PR and `main` comit, and `release.yaml` runs anytime you push a tag to the repository. +See comments in the rules-template repo for more information. -If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md\&title=Request+to+add+new+project+%5BPROJECT_NAME%5D\&labels=new-project) it to [ci.bazel.build](http://ci.bazel.build). +If your repository is under the [bazelbuild organization](https://github.com/bazelbuild), +you can [ask to add](https://github.com/bazelbuild/continuous-integration/issues/new?template=adding-your-project-to-bazel-ci.md&title=Request+to+add+new+project+%5BPROJECT_NAME%5D&labels=new-project) +it to [ci.bazel.build](http://ci.bazel.build). ## Documentation -See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for instructions on how to comment your rules so that documentation can be generated automatically. +See the [Stardoc documentation](https://github.com/bazelbuild/stardoc) for +instructions on how to comment your rules so that documentation can be generated +automatically. -The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date as Starlark files are updated. +The [rules-template docs/ folder](https://github.com/bazel-contrib/rules-template/tree/main/docs) +shows a simple way to ensure the Markdown content in the `docs/` folder is always up-to-date +as Starlark files are updated. ## FAQs ### Why can't we add our rule to the main Bazel GitHub repository? -We want to decouple rules from Bazel releases as much as possible. It's clearer who owns individual rules, reducing the load on Bazel developers. For our users, decoupling makes it easier to modify, upgrade, downgrade, and replace rules. Contributing to rules can be lighter weight than contributing to Bazel - depending on the rules -, including full submit access to the corresponding GitHub repository. Getting submit access to Bazel itself is a much more involved process. +We want to decouple rules from Bazel releases as much as possible. It's clearer +who owns individual rules, reducing the load on Bazel developers. For our users, +decoupling makes it easier to modify, upgrade, downgrade, and replace rules. +Contributing to rules can be lighter weight than contributing to Bazel - +depending on the rules -, including full submit access to the corresponding +GitHub repository. Getting submit access to Bazel itself is a much more involved +process. -The downside is a more complicated one-time installation process for our users: they have to add a dependency on your ruleset in their `MODULE.bazel` file. +The downside is a more complicated one-time installation process for our users: +they have to add a dependency on your ruleset in their `MODULE.bazel` file. -We used to have all of the rules in the Bazel repository (under `//tools/build_rules` or `//tools/build_defs`). We still have a couple rules there, but we are working on moving the remaining rules out. +We used to have all of the rules in the Bazel repository (under +`//tools/build_rules` or `//tools/build_defs`). We still have a couple rules +there, but we are working on moving the remaining rules out. diff --git a/rules/errors/read-only-variable.mdx b/rules/errors/read-only-variable.mdx index ab4e4619..2bfde654 100644 --- a/rules/errors/read-only-variable.mdx +++ b/rules/errors/read-only-variable.mdx @@ -2,7 +2,11 @@ title: 'Error: Variable x is read only' --- -A global variable cannot be reassigned. It will always point to the same object. However, its content might change, if the value is mutable (for example, the content of a list). Local variables don't have this restriction. + + +A global variable cannot be reassigned. It will always point to the same object. +However, its content might change, if the value is mutable (for example, the +content of a list). Local variables don't have this restriction. ```python a = [1, 2] @@ -16,7 +20,8 @@ b = 4 # forbidden `ERROR: /path/ext.bzl:7:1: Variable b is read only` -You will get a similar error if you try to redefine a function (function overloading is not supported), for example: +You will get a similar error if you try to redefine a function (function +overloading is not supported), for example: ```python def foo(x): return x + 1 diff --git a/rules/faq.mdx b/rules/faq.mdx index 89481fa7..5321f0b7 100644 --- a/rules/faq.mdx +++ b/rules/faq.mdx @@ -2,48 +2,79 @@ title: 'Frequently Asked Questions' --- + + These are some common issues and questions with writing extensions. ## Why is my file not produced / my action never executed? Bazel only executes the actions needed to produce the *requested* output files. -- If the file you want has a label, you can request it directly: `bazel build //pkg:myfile.txt` +* If the file you want has a label, you can request it directly: + `bazel build //pkg:myfile.txt` -- If the file is in an output group of the target, you may need to specify that output group on the command line: `bazel build //pkg:mytarget --output_groups=foo` +* If the file is in an output group of the target, you may need to specify that + output group on the command line: + `bazel build //pkg:mytarget --output_groups=foo` -- If you want the file to be built automatically whenever your target is mentioned on the command line, add it to your rule's default outputs by returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. +* If you want the file to be built automatically whenever your target is + mentioned on the command line, add it to your rule's default outputs by + returning a [`DefaultInfo`](lib/globals#DefaultInfo) provider. See the [Rules page](/extending/rules#requesting-output-files) for more information. ## Why is my implementation function not executed? -Bazel analyzes only the targets that are requested for the build. You should either name the target on the command line, or something that depends on the target. +Bazel analyzes only the targets that are requested for the build. You should +either name the target on the command line, or something that depends on the +target. ## A file is missing when my action or binary is executed -Make sure that 1) the file has been registered as an input to the action or binary, and 2) the script or tool being executed is accessing the file using the correct path. +Make sure that 1) the file has been registered as an input to the action or +binary, and 2) the script or tool being executed is accessing the file using the +correct path. -For actions, you declare inputs by passing them to the `ctx.actions.*` function that creates the action. The proper path for the file can be obtained using [`File.path`](lib/File#path). +For actions, you declare inputs by passing them to the `ctx.actions.*` function +that creates the action. The proper path for the file can be obtained using +[`File.path`](lib/File#path). -For binaries (the executable outputs run by a `bazel run` or `bazel test` command), you declare inputs by including them in the [runfiles](/extending/rules#runfiles). Instead of using the `path` field, use [`File.short_path`](lib/File#short_path), which is file's path relative to the runfiles directory in which the binary executes. +For binaries (the executable outputs run by a `bazel run` or `bazel test` +command), you declare inputs by including them in the +[runfiles](/extending/rules#runfiles). Instead of using the `path` field, use +[`File.short_path`](lib/File#short_path), which is file's path relative to +the runfiles directory in which the binary executes. ## How can I control which files are built by `bazel build //pkg:mytarget`? -Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to [set the default outputs](/extending/rules#requesting-output-files). +Use the [`DefaultInfo`](lib/globals#DefaultInfo) provider to +[set the default outputs](/extending/rules#requesting-output-files). ## How can I run a program or do file I/O as part of my build? -A tool can be declared as a target, just like any other part of your build, and run during the execution phase to help build other targets. To create an action that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the tool as the `executable` parameter. +A tool can be declared as a target, just like any other part of your build, and +run during the execution phase to help build other targets. To create an action +that runs a tool, use [`ctx.actions.run`](lib/actions#run) and pass in the +tool as the `executable` parameter. -During the loading and analysis phases, a tool *cannot* run, nor can you perform file I/O. This means that tools and file contents (except the contents of BUILD and .bzl files) cannot affect how the target and action graphs get created. +During the loading and analysis phases, a tool *cannot* run, nor can you perform +file I/O. This means that tools and file contents (except the contents of BUILD +and .bzl files) cannot affect how the target and action graphs get created. ## What if I need to access the same structured data both before and during the execution phase? -You can format the structured data as a .bzl file. You can `load()` the file to access it during the loading and analysis phases. You can pass it as an input or runfile to actions and executables that need it during the execution phase. +You can format the structured data as a .bzl file. You can `load()` the file to +access it during the loading and analysis phases. You can pass it as an input or +runfile to actions and executables that need it during the execution phase. ## How should I document Starlark code? -For rules and rule attributes, you can pass a docstring literal (possibly triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper functions and macros, use a triple-quoted docstring literal following the format given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). Rule implementation functions generally do not need their own docstring. +For rules and rule attributes, you can pass a docstring literal (possibly +triple-quoted) to the `doc` parameter of `rule` or `attr.*()`. For helper +functions and macros, use a triple-quoted docstring literal following the format +given [here](https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#function-docstring). +Rule implementation functions generally do not need their own docstring. -Using string literals in the expected places makes it easier for automated tooling to extract documentation. Feel free to use standard non-string comments wherever it may help the reader of your code. +Using string literals in the expected places makes it easier for automated +tooling to extract documentation. Feel free to use standard non-string comments +wherever it may help the reader of your code. diff --git a/rules/index.mdx b/rules/index.mdx index 44590f11..2a6c3eb7 100644 --- a/rules/index.mdx +++ b/rules/index.mdx @@ -2,7 +2,11 @@ title: 'Rules' --- -The Bazel ecosystem has a growing and evolving set of rules to support popular languages and packages. Much of Bazel's strength comes from the ability to [define new rules](/extending/concepts) that can be used by others. + + +The Bazel ecosystem has a growing and evolving set of rules to support popular +languages and packages. Much of Bazel's strength comes from the ability to +[define new rules](/extending/concepts) that can be used by others. This page describes the recommended, native, and non-native Bazel rules. @@ -10,57 +14,58 @@ This page describes the recommended, native, and non-native Bazel rules. Here is a selection of recommended rules: -- [Android](/docs/bazel-and-android) -- [C / C++](/docs/bazel-and-cpp) -- [Docker/OCI](https://github.com/bazel-contrib/rules_oci) -- [Go](https://github.com/bazelbuild/rules_go) -- [Haskell](https://github.com/tweag/rules_haskell) -- [Java](/docs/bazel-and-java) -- [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) -- [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) -- [Objective-C](/docs/bazel-and-apple) -- [Package building](https://github.com/bazelbuild/rules_pkg) -- [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) -- [Python](https://github.com/bazelbuild/rules_python) -- [Rust](https://github.com/bazelbuild/rules_rust) -- [Scala](https://github.com/bazelbuild/rules_scala) -- [Shell](/reference/be/shell) -- [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) - -The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains additional functions that can be useful when writing new rules and new macros. - -The rules above were reviewed and follow our [requirements for recommended rules](/community/recommended-rules). Contact the respective rule set's maintainers regarding issues and feature requests. - -To find more Bazel rules, use a search engine, take a look on [awesomebazel.com](https://awesomebazel.com/), or search on [GitHub](https://github.com/search?o=desc\&q=bazel+rules\&s=stars\&type=Repositories). +* [Android](/docs/bazel-and-android) +* [C / C++](/docs/bazel-and-cpp) +* [Docker/OCI](https://github.com/bazel-contrib/rules_oci) +* [Go](https://github.com/bazelbuild/rules_go) +* [Haskell](https://github.com/tweag/rules_haskell) +* [Java](/docs/bazel-and-java) +* [JavaScript / NodeJS](https://github.com/bazelbuild/rules_nodejs) +* [Maven dependency management](https://github.com/bazelbuild/rules_jvm_external) +* [Objective-C](/docs/bazel-and-apple) +* [Package building](https://github.com/bazelbuild/rules_pkg) +* [Protocol Buffers](https://github.com/bazelbuild/rules_proto#protobuf-rules-for-bazel) +* [Python](https://github.com/bazelbuild/rules_python) +* [Rust](https://github.com/bazelbuild/rules_rust) +* [Scala](https://github.com/bazelbuild/rules_scala) +* [Shell](/reference/be/shell) +* [Webtesting](https://github.com/bazelbuild/rules_webtesting) (Webdriver) + +The repository [Skylib](https://github.com/bazelbuild/bazel-skylib) contains +additional functions that can be useful when writing new rules and new +macros. + +The rules above were reviewed and follow our +[requirements for recommended rules](/community/recommended-rules). +Contact the respective rule set's maintainers regarding issues and feature +requests. + +To find more Bazel rules, use a search engine, take a look on +[awesomebazel.com](https://awesomebazel.com/), or search on +[GitHub](https://github.com/search?o=desc&q=bazel+rules&s=stars&type=Repositories). ## Native rules that do not apply to a specific programming language -Native rules are shipped with the Bazel binary, they are always available in BUILD files without a `load` statement. - -- Extra actions +Native rules are shipped with the Bazel binary, they are always available in +BUILD files without a `load` statement. +* Extra actions - [`extra_action`](/reference/be/extra-actions#extra_action) - [`action_listener`](/reference/be/extra-actions#action_listener) - -- General - +* General - [`filegroup`](/reference/be/general#filegroup) - [`genquery`](/reference/be/general#genquery) - [`test_suite`](/reference/be/general#test_suite) - [`alias`](/reference/be/general#alias) - [`config_setting`](/reference/be/general#config_setting) - [`genrule`](/reference/be/general#genrule) - -- Platform - +* Platform - [`constraint_setting`](/reference/be/platforms-and-toolchains#constraint_setting) - [`constraint_value`](/reference/be/platforms-and-toolchains#constraint_value) - [`platform`](/reference/be/platforms-and-toolchains#platform) - [`toolchain`](/reference/be/platforms-and-toolchains#toolchain) - [`toolchain_type`](/reference/be/platforms-and-toolchains#toolchain_type) - -- Workspace - +* Workspace - [`bind`](/reference/be/workspace#bind) - [`local_repository`](/reference/be/workspace#local_repository) - [`new_local_repository`](/reference/be/workspace#new_local_repository) @@ -69,10 +74,10 @@ Native rules are shipped with the Bazel binary, they are always available in BUI ## Embedded non-native rules -Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from the `@bazel_tools` built-in external repository. - -- Repository rules +Bazel also embeds additional rules written in [Starlark](/rules/language). Those can be loaded from +the `@bazel_tools` built-in external repository. +* Repository rules - [`git_repository`](/rules/lib/repo/git#git_repository) - [`http_archive`](/rules/lib/repo/http#http_archive) - [`http_file`](/rules/lib/repo/http#http_archive) diff --git a/rules/language.mdx b/rules/language.mdx index 9fad7248..13e33a4b 100644 --- a/rules/language.mdx +++ b/rules/language.mdx @@ -2,11 +2,18 @@ title: 'Starlark Language' --- -This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), formerly known as Skylark, the language used in Bazel. For a complete list of functions and types, see the [Bazel API reference](/rules/lib/overview). + + +{/* [TOC] */} + +This page is an overview of [Starlark](https://github.com/bazelbuild/starlark), +formerly known as Skylark, the language used in Bazel. For a complete list of +functions and types, see the [Bazel API reference](/rules/lib/overview). For more information about the language, see [Starlark's GitHub repo](https://github.com/bazelbuild/starlark/). -For the authoritative specification of the Starlark syntax and behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). +For the authoritative specification of the Starlark syntax and +behavior, see the [Starlark Language Specification](https://github.com/bazelbuild/starlark/blob/master/spec.md). ## Syntax @@ -26,32 +33,44 @@ def fizz_buzz(n): fizz_buzz(20) ``` -Starlark's semantics can differ from Python, but behavioral differences are rare, except for cases where Starlark raises an error. The following Python types are supported: +Starlark's semantics can differ from Python, but behavioral differences are +rare, except for cases where Starlark raises an error. The following Python +types are supported: -- [None](lib/globals#None) -- [bool](lib/bool) -- [dict](lib/dict) -- [tuple](lib/tuple) -- [function](lib/function) -- [int](lib/int) -- [list](lib/list) -- [string](lib/string) +* [None](lib/globals#None) +* [bool](lib/bool) +* [dict](lib/dict) +* [tuple](lib/tuple) +* [function](lib/function) +* [int](lib/int) +* [list](lib/list) +* [string](lib/string) ## Type annotations -**Experimental**. Type annotations are an experimental feature and may change at any time. Don't depend on it. It may be enabled in Bazel at HEAD by using the `--experimental_starlark_types` flag. +**Experimental**. Type annotations are an experimental feature and may change +at any time. Don't depend on it. It may be enabled in Bazel at HEAD +by using the `--experimental_starlark_types` flag. -Starlark in Bazel at HEAD is incrementally adding support for type annotations with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). +Starlark in Bazel at HEAD is incrementally adding support for type annotations +with a syntax inspired by [PEP 484](https://peps.python.org/pep-0484/). -- Starlark type annotations are under active development. The progress is tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). +- Starlark type annotations are under active development. The progress is + tracked on [issue#22935](https://github.com/bazelbuild/bazel/issues/22935). - The specification is incrementally extended: [starlark-with-types/spec.md](https://github.com/bazelbuild/starlark/blob/starlark-with-types/spec.md) - Initial proposal: [SEP-001 Bootstrapping Starlark types](https://docs.google.com/document/d/1Sid7EAbBd_w_T7D94Li_f_bK3zMTztFbzIMvcpzo1wY/edit?tab=t.0#heading=h.5mcn15i0e1ch) ## Mutability -Starlark favors immutability. Two mutable data structures are available: [lists](lib/list) and [dicts](lib/dict). Changes to mutable data-structures, such as appending a value to a list or deleting an entry in a dictionary are valid only for objects created in the current context. After a context finishes, its values become immutable. +Starlark favors immutability. Two mutable data structures are available: +[lists](lib/list) and [dicts](lib/dict). Changes to mutable +data-structures, such as appending a value to a list or deleting an entry in a +dictionary are valid only for objects created in the current context. After a +context finishes, its values become immutable. -This is because Bazel builds use parallel execution. During a build, each `.bzl` file and each `BUILD` file get their own execution context. Each rule is also analyzed in its own context. +This is because Bazel builds use parallel execution. During a build, each `.bzl` +file and each `BUILD` file get their own execution context. Each rule is also +analyzed in its own context. Let's go through an example with the file `foo.bzl`: @@ -65,9 +84,13 @@ def fct(): # declare a function fct() # execute the fct function ``` -Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s context. When `fct()` runs, it does so within the context of `foo.bzl`. After evaluation for `foo.bzl` completes, the environment contains an immutable entry, `var`, with the value `[5]`. +Bazel creates `var` when `foo.bzl` loads. `var` is thus part of `foo.bzl`'s +context. When `fct()` runs, it does so within the context of `foo.bzl`. After +evaluation for `foo.bzl` completes, the environment contains an immutable entry, +`var`, with the value `[5]`. -When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain immutable. For this reason, the following code in `bar.bzl` is illegal: +When another `bar.bzl` loads symbols from `foo.bzl`, loaded values remain +immutable. For this reason, the following code in `bar.bzl` is illegal: ```python # `bar.bzl` @@ -78,52 +101,66 @@ var.append(6) # runtime error, the list stored in var is frozen fct() # runtime error, fct() attempts to modify a frozen list ``` -Global variables defined in `bzl` files cannot be changed outside of the `bzl` file that defined them. Just like the above example using `bzl` files, values returned by rules are immutable. +Global variables defined in `bzl` files cannot be changed outside of the +`bzl` file that defined them. Just like the above example using `bzl` files, +values returned by rules are immutable. ## Differences between BUILD and .bzl files -`BUILD` files register targets via making calls to rules. `.bzl` files provide definitions for constants, rules, macros, and functions. +`BUILD` files register targets via making calls to rules. `.bzl` files provide +definitions for constants, rules, macros, and functions. -[Native functions](/reference/be/functions) and [native rules](/reference/be/overview#language-specific-native-rules) are global symbols in `BUILD` files. `bzl` files need to load them using the [`native` module](/rules/lib/toplevel/native). +[Native functions](/reference/be/functions) and [native rules]( +/reference/be/overview#language-specific-native-rules) are global symbols in +`BUILD` files. `bzl` files need to load them using the [`native` module]( +/rules/lib/toplevel/native). -There are two syntactic restrictions in `BUILD` files: 1) declaring functions is illegal, and 2) `*args` and `**kwargs` arguments are not allowed. +There are two syntactic restrictions in `BUILD` files: 1) declaring functions is +illegal, and 2) `*args` and `**kwargs` arguments are not allowed. ## Differences with Python -- Global variables are immutable. +* Global variables are immutable. -- `for` statements are not allowed at the top-level. Use them within functions instead. In `BUILD` files, you may use list comprehensions. +* `for` statements are not allowed at the top-level. Use them within functions + instead. In `BUILD` files, you may use list comprehensions. -- `if` statements are not allowed at the top-level. However, `if` expressions can be used: `first = data[0] if len(data) > 0 else None`. +* `if` statements are not allowed at the top-level. However, `if` expressions + can be used: `first = data[0] if len(data) > 0 else None`. -- Deterministic order for iterating through Dictionaries. +* Deterministic order for iterating through Dictionaries. -- Recursion is not allowed. +* Recursion is not allowed. -- Int type is limited to 32-bit signed integers. Overflows will throw an error. +* Int type is limited to 32-bit signed integers. Overflows will throw an error. -- Modifying a collection during iteration is an error. +* Modifying a collection during iteration is an error. -- Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are not defined across value types. In short: `5 < 'foo'` will throw an error and `5 == "5"` will return false. +* Except for equality tests, comparison operators `<`, `<=`, `>=`, `>`, etc. are +not defined across value types. In short: `5 < 'foo'` will throw an error and +`5 == "5"` will return false. -- In tuples, a trailing comma is valid only when the tuple is between parentheses — when you write `(1,)` instead of `1,`. +* In tuples, a trailing comma is valid only when the tuple is between + parentheses — when you write `(1,)` instead of `1,`. -- Dictionary literals cannot have duplicated keys. For example, this is an error: `{"a": 4, "b": 7, "a": 1}`. +* Dictionary literals cannot have duplicated keys. For example, this is an + error: `{"a": 4, "b": 7, "a": 1}`. -- Strings are represented with double-quotes (such as when you call [repr](lib/globals#repr)). +* Strings are represented with double-quotes (such as when you call + [repr](lib/globals#repr)). -- Strings aren't iterable. +* Strings aren't iterable. The following Python features are not supported: -- implicit string concatenation (use explicit `+` operator). -- Chained comparisons (such as `1 < x < 5`). -- `class` (see [`struct`](lib/struct#struct) function). -- `import` (see [`load`](/extending/concepts#loading-an-extension) statement). -- `while`, `yield`. -- float and set types. -- generators and generator expressions. -- `is` (use `==` instead). -- `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). -- `global`, `nonlocal`. -- most builtin functions, most methods. +* implicit string concatenation (use explicit `+` operator). +* Chained comparisons (such as `1 < x < 5`). +* `class` (see [`struct`](lib/struct#struct) function). +* `import` (see [`load`](/extending/concepts#loading-an-extension) statement). +* `while`, `yield`. +* float and set types. +* generators and generator expressions. +* `is` (use `==` instead). +* `try`, `raise`, `except`, `finally` (see [`fail`](lib/globals#fail) for fatal errors). +* `global`, `nonlocal`. +* most builtin functions, most methods. diff --git a/rules/legacy-macro-tutorial.mdx b/rules/legacy-macro-tutorial.mdx index 8322ec42..28b0fca8 100644 --- a/rules/legacy-macro-tutorial.mdx +++ b/rules/legacy-macro-tutorial.mdx @@ -2,11 +2,20 @@ title: 'Creating a Legacy Macro' --- -IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If you only need to support Bazel 8 or newer, we recommend using [symbolic macros](/extending/macros) instead; take a look at [Creating a Symbolic Macro](macro-tutorial). -Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a legacy macro that resizes an image. -Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. +IMPORTANT: This tutorial is for [*legacy macros*](/extending/legacy-macros). If +you only need to support Bazel 8 or newer, we recommend using [symbolic +macros](/extending/macros) instead; take a look at [Creating a Symbolic +Macro](macro-tutorial). + +Imagine that you need to run a tool as part of your build. For example, you +may want to generate or preprocess a source file, or compress a binary. In this +tutorial, you are going to create a legacy macro that resizes an image. + +Macros are suitable for simple tasks. If you want to do anything more +complicated, for example add support for a new programming language, consider +creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -15,7 +24,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -25,7 +34,8 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, define a function in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, +define a function in a separate `.bzl` file, and call the file `miniature.bzl`: ```starlark def miniature(name, src, size = "100x100", **kwargs): @@ -38,20 +48,25 @@ def miniature(name, src, size = "100x100", **kwargs): srcs = [src], # Note that the line below will fail if `src` is not a filename string outs = ["small_" + src], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs ) ``` A few remarks: -- By convention, legacy macros have a `name` argument, just like rules. + * By convention, legacy macros have a `name` argument, just like rules. -- To document the behavior of a legacy macro, use [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. + * To document the behavior of a legacy macro, use + [docstring](https://www.python.org/dev/peps/pep-0257/) like in Python. -- To call a `genrule`, or any other native rule, prefix with `native.`. + * To call a `genrule`, or any other native rule, prefix with `native.`. -- Use `**kwargs` to forward the extra arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful, so that a user can use standard attributes like `visibility`, or `tags`. + * Use `**kwargs` to forward the extra arguments to the underlying `genrule` + (it works just like in + [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). + This is useful, so that a user can use standard attributes like + `visibility`, or `tags`. Now, use the macro from the `BUILD` file: @@ -70,6 +85,14 @@ cc_binary( ) ``` -And finally, a **warning note**: the macro assumes that `src` is a filename string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` works; but what happens if the `BUILD` file instead used `src = "//other/package:image.png"`, or even `src = select(...)`? - -You should make sure to declare such assumptions in your macro's documentation. Unfortunately, legacy macros, especially large ones, tend to be fragile because it can be hard to notice and document all such assumptions in your code – and, of course, some users of the macro won't read the documentation. We recommend, if possible, instead using [symbolic macros](/extending/macros), which have built-in checks on attribute types. +And finally, a **warning note**: the macro assumes that `src` is a filename +string (otherwise, `outs = ["small_" + src]` will fail). So `src = "image.png"` +works; but what happens if the `BUILD` file instead used `src = +"//other/package:image.png"`, or even `src = select(...)`? + +You should make sure to declare such assumptions in your macro's documentation. +Unfortunately, legacy macros, especially large ones, tend to be fragile because +it can be hard to notice and document all such assumptions in your code – and, +of course, some users of the macro won't read the documentation. We recommend, +if possible, instead using [symbolic macros](/extending/macros), which have +built\-in checks on attribute types. diff --git a/rules/macro-tutorial.mdx b/rules/macro-tutorial.mdx index 4cf91617..2b3d8e4a 100644 --- a/rules/macro-tutorial.mdx +++ b/rules/macro-tutorial.mdx @@ -2,11 +2,20 @@ title: 'Creating a Symbolic Macro' --- -IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new macro system introduced in Bazel 8. If you need to support older Bazel versions, you will want to write a [legacy macro](/extending/legacy-macros) instead; take a look at [Creating a Legacy Macro](legacy-macro-tutorial). -Imagine that you need to run a tool as part of your build. For example, you may want to generate or preprocess a source file, or compress a binary. In this tutorial, you are going to create a symbolic macro that resizes an image. -Macros are suitable for simple tasks. If you want to do anything more complicated, for example add support for a new programming language, consider creating a [rule](/extending/rules). Rules give you more control and flexibility. +IMPORTANT: This tutorial is for [*symbolic macros*](/extending/macros) – the new +macro system introduced in Bazel 8. If you need to support older Bazel versions, +you will want to write a [legacy macro](/extending/legacy-macros) instead; take +a look at [Creating a Legacy Macro](legacy-macro-tutorial). + +Imagine that you need to run a tool as part of your build. For example, you +may want to generate or preprocess a source file, or compress a binary. In this +tutorial, you are going to create a symbolic macro that resizes an image. + +Macros are suitable for simple tasks. If you want to do anything more +complicated, for example add support for a new programming language, consider +creating a [rule](/extending/rules). Rules give you more control and flexibility. The easiest way to create a macro that resizes an image is to use a `genrule`: @@ -15,7 +24,7 @@ genrule( name = "logo_miniature", srcs = ["logo.png"], outs = ["small_logo.png"], - cmd = "convert $< -resize 100x100 $@", + cmd = "convert $< -resize 100x100 $@", ) cc_binary( @@ -25,7 +34,9 @@ cc_binary( ) ``` -If you need to resize more images, you may want to reuse the code. To do that, define an *implementation function* and a *macro declaration* in a separate `.bzl` file, and call the file `miniature.bzl`: +If you need to resize more images, you may want to reuse the code. To do that, +define an *implementation function* and a *macro declaration* in a separate +`.bzl` file, and call the file `miniature.bzl`: ```starlark # Implementation function @@ -35,7 +46,7 @@ def _miniature_impl(name, visibility, src, size, **kwargs): visibility = visibility, srcs = [src], outs = [name + "_small_" + src.name], - cmd = "convert $< -resize " + size + " $@", + cmd = "convert $< -resize " + size + " $@", **kwargs, ) @@ -73,13 +84,19 @@ miniature = macro( A few remarks: -- Symbolic macro implementation functions must have `name` and `visibility` parameters. They should used for the macro's main target. + * Symbolic macro implementation functions must have `name` and `visibility` + parameters. They should used for the macro's main target. -- To document the behavior of a symbolic macro, use `doc` parameters for `macro()` and its attributes. + * To document the behavior of a symbolic macro, use `doc` parameters for + `macro()` and its attributes. -- To call a `genrule`, or any other native rule, use `native.`. + * To call a `genrule`, or any other native rule, use `native.`. -- Use `**kwargs` to forward the extra inherited arguments to the underlying `genrule` (it works just like in [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). This is useful so that a user can set standard attributes like `tags` or `testonly`. + * Use `**kwargs` to forward the extra inherited arguments to the underlying + `genrule` (it works just like in + [Python](https://docs.python.org/3/tutorial/controlflow.html#keyword-arguments)). + This is useful so that a user can set standard attributes like `tags` or + `testonly`. Now, use the macro from the `BUILD` file: diff --git a/rules/performance.mdx b/rules/performance.mdx index edc0a2fd..c415cf14 100644 --- a/rules/performance.mdx +++ b/rules/performance.mdx @@ -2,20 +2,30 @@ title: 'Optimizing Performance' --- -When writing rules, the most common performance pitfall is to traverse or copy data that is accumulated from dependencies. When aggregated over the whole build, these operations can easily take O(N^2) time or space. To avoid this, it is crucial to understand how to use depsets effectively. -This can be hard to get right, so Bazel also provides a memory profiler that assists you in finding spots where you might have made a mistake. Be warned: The cost of writing an inefficient rule may not be evident until it is in widespread use. + +When writing rules, the most common performance pitfall is to traverse or copy +data that is accumulated from dependencies. When aggregated over the whole +build, these operations can easily take O(N^2) time or space. To avoid this, it +is crucial to understand how to use depsets effectively. + +This can be hard to get right, so Bazel also provides a memory profiler that +assists you in finding spots where you might have made a mistake. Be warned: +The cost of writing an inefficient rule may not be evident until it is in +widespread use. ## Use depsets -Whenever you are rolling up information from rule dependencies you should use [depsets](lib/depset). Only use plain lists or dicts to publish information local to the current rule. +Whenever you are rolling up information from rule dependencies you should use +[depsets](lib/depset). Only use plain lists or dicts to publish information +local to the current rule. A depset represents information as a nested graph which enables sharing. Consider the following graph: ``` -C -> B -> A +C -> B -> A D ---^ ``` @@ -37,9 +47,12 @@ c = ['c', 'b', 'a'] d = ['d', 'b', 'a'] ``` -Note that in this case `'a'` is mentioned four times! With larger graphs this problem will only get worse. +Note that in this case `'a'` is mentioned four times! With larger graphs this +problem will only get worse. -Here is an example of a rule implementation that uses depsets correctly to publish transitive information. Note that it is OK to publish rule-local information using lists if you want since this is not O(N^2). +Here is an example of a rule implementation that uses depsets correctly to +publish transitive information. Note that it is OK to publish rule-local +information using lists if you want since this is not O(N^2). ``` MyProvider = provider() @@ -61,13 +74,21 @@ See the [depset overview](/extending/depsets) page for more information. ### Avoid calling `depset.to_list()` -You can coerce a depset to a flat list using [`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) cost. If at all possible, avoid any flattening of depsets except for debugging purposes. +You can coerce a depset to a flat list using +[`to_list()`](lib/depset#to_list), but doing so usually results in O(N^2) +cost. If at all possible, avoid any flattening of depsets except for debugging +purposes. -A common misconception is that you can freely flatten depsets if you only do it at top-level targets, such as an `<xx>_binary` rule, since then the cost is not accumulated over each level of the build graph. But this is *still* O(N^2) when you build a set of targets with overlapping dependencies. This happens when building your tests `//foo/tests/...`, or when importing an IDE project. +A common misconception is that you can freely flatten depsets if you only do it +at top-level targets, such as an `_binary` rule, since then the cost is not +accumulated over each level of the build graph. But this is *still* O(N^2) when +you build a set of targets with overlapping dependencies. This happens when +building your tests `//foo/tests/...`, or when importing an IDE project. ### Reduce the number of calls to `depset` -Calling `depset` inside a loop is often a mistake. It can lead to depsets with very deep nesting, which perform poorly. For example: +Calling `depset` inside a loop is often a mistake. It can lead to depsets with +very deep nesting, which perform poorly. For example: ```python x = depset() @@ -76,7 +97,8 @@ for i in inputs: x = depset(transitive = [x, i.deps]) ``` -This code can be replaced easily. First, collect the transitive depsets and merge them all at once: +This code can be replaced easily. First, collect the transitive depsets and +merge them all at once: ```python transitive = [] @@ -95,19 +117,33 @@ x = depset(transitive = [i.deps for i in inputs]) ## Use ctx.actions.args() for command lines -When building command lines you should use [ctx.actions.args()](lib/Args). This defers expansion of any depsets to the execution phase. +When building command lines you should use [ctx.actions.args()](lib/Args). +This defers expansion of any depsets to the execution phase. -Apart from being strictly faster, this will reduce the memory consumption of your rules -- sometimes by 90% or more. +Apart from being strictly faster, this will reduce the memory consumption of +your rules -- sometimes by 90% or more. Here are some tricks: -- Pass depsets and lists directly as arguments, instead of flattening them yourself. They will get expanded by `ctx.actions.args()` for you. If you need any transformations on the depset contents, look at [ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. +* Pass depsets and lists directly as arguments, instead of flattening them +yourself. They will get expanded by `ctx.actions.args()` for you. +If you need any transformations on the depset contents, look at +[ctx.actions.args#add](lib/Args#add) to see if anything fits the bill. -- Are you passing `File#path` as arguments? No need. Any [File](lib/File) is automatically turned into its [path](lib/File#path), deferred to expansion time. +* Are you passing `File#path` as arguments? No need. Any +[File](lib/File) is automatically turned into its +[path](lib/File#path), deferred to expansion time. -- Avoid constructing strings by concatenating them together. The best string argument is a constant as its memory will be shared between all instances of your rule. +* Avoid constructing strings by concatenating them together. +The best string argument is a constant as its memory will be shared between +all instances of your rule. -- If the args are too long for the command line an `ctx.actions.args()` object can be conditionally or unconditionally written to a param file using [`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is done behind the scenes when the action is executed. If you need to explicitly control the params file you can write it manually using [`ctx.actions.write`](lib/actions#write). +* If the args are too long for the command line an `ctx.actions.args()` object +can be conditionally or unconditionally written to a param file using +[`ctx.actions.args#use_param_file`](lib/Args#use_param_file). This is +done behind the scenes when the action is executed. If you need to explicitly +control the params file you can write it manually using +[`ctx.actions.write`](lib/actions#write). Example: @@ -118,15 +154,15 @@ def _impl(ctx): file = ctx.declare_file(...) files = depset(...) - # Bad, constructs a full string "--foo=<file path>" for each rule instance + # Bad, constructs a full string "--foo=" for each rule instance args.add("--foo=" + file.path) # Good, shares "--foo" among all rule instances, and defers file.path to later - # It will however pass ["--foo", <file path>] to the action command line, - # instead of ["--foo=<file_path>"] + # It will however pass ["--foo", ] to the action command line, + # instead of ["--foo="] args.add("--foo", file) - # Use format if you prefer ["--foo=<file path>"] to ["--foo", <file path>] + # Use format if you prefer ["--foo="] to ["--foo", ] args.add(file, format="--foo=%s") # Bad, makes a giant string of a whole depset @@ -142,7 +178,9 @@ def _to_short_path(f): ## Transitive action inputs should be depsets -When building an action using [ctx.actions.run](lib/actions?#run), do not forget that the `inputs` field accepts a depset. Use this whenever inputs are collected from dependencies transitively. +When building an action using [ctx.actions.run](lib/actions?#run), do not +forget that the `inputs` field accepts a depset. Use this whenever inputs are +collected from dependencies transitively. ``` inputs = depset(...) @@ -154,39 +192,54 @@ ctx.actions.run( ## Hanging -If Bazel appears to be hung, you can hit `Ctrl-\` or send Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread dump in the file `$(bazel info output_base)/server/jvm.out`. +If Bazel appears to be hung, you can hit Ctrl-\ or send +Bazel a `SIGQUIT` signal (`kill -3 $(bazel info server_pid)`) to get a thread +dump in the file `$(bazel info output_base)/server/jvm.out`. -Since you may not be able to run `bazel info` if bazel is hung, the `output_base` directory is usually the parent of the `bazel-<workspace>` symlink in your workspace directory. +Since you may not be able to run `bazel info` if bazel is hung, the +`output_base` directory is usually the parent of the `bazel-` +symlink in your workspace directory. ## Performance profiling -The [JSON trace profile](/advanced/performance/json-trace-profile) can be very useful to quickly understand what Bazel spent time on during the invocation. +The [JSON trace profile](/advanced/performance/json-trace-profile) can be very +useful to quickly understand what Bazel spent time on during the invocation. -The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) flag may be used to capture Java Flight Recorder profiles of various kinds (cpu time, wall time, memory allocations and lock contention). +The [`--experimental_command_profile`](https://bazel.build/reference/command-line-reference#flag--experimental_command_profile) +flag may be used to capture Java Flight Recorder profiles of various kinds +(cpu time, wall time, memory allocations and lock contention). -The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) flag may be used to write a pprof profile of CPU usage by all Starlark threads. +The [`--starlark_cpu_profile`](https://bazel.build/reference/command-line-reference#flag--starlark_cpu_profile) +flag may be used to write a pprof profile of CPU usage by all Starlark threads. ## Memory profiling -Bazel comes with a built-in memory profiler that can help you check your rule’s memory use. If there is a problem you can dump the heap to find the exact line of code that is causing the problem. +Bazel comes with a built-in memory profiler that can help you check your rule’s +memory use. If there is a problem you can dump the heap to find the +exact line of code that is causing the problem. ### Enabling memory tracking You must pass these two startup flags to *every* Bazel invocation: -``` -STARTUP_FLAGS=\ ---host_jvm_args=-javaagent:<path to java-allocation-instrumenter-3.3.4.jar> \ ---host_jvm_args=-DRULE_MEMORY_TRACKER=1 -``` + ``` + STARTUP_FLAGS=\ + --host_jvm_args=-javaagent: \

+ --host_jvm_args=-DRULE_MEMORY_TRACKER=1 + ``` +Note: You can download the allocation instrumenter jar file from [Maven Central +Repository][allocation-instrumenter-link]. -Note: You can download the allocation instrumenter jar file from [Maven Central Repository](https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4). +[allocation-instrumenter-link]: https://repo1.maven.org/maven2/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/3.3.4 -These start the server in memory tracking mode. If you forget these for even one Bazel invocation the server will restart and you will have to start over. +These start the server in memory tracking mode. If you forget these for even +one Bazel invocation the server will restart and you will have to start over. ### Using the Memory Tracker -As an example, look at the target `foo` and see what it does. To only run the analysis and not run the build execution phase, add the `--nobuild` flag. +As an example, look at the target `foo` and see what it does. To only +run the analysis and not run the build execution phase, add the +`--nobuild` flag. ``` $ bazel $(STARTUP_FLAGS) build --nobuild //foo:foo @@ -196,14 +249,14 @@ Next, see how much memory the whole Bazel instance consumes: ``` $ bazel $(STARTUP_FLAGS) info used-heap-size-after-gc -> 2594MB +> 2594MB ``` Break it down by rule class by using `bazel dump --rules`: ``` $ bazel $(STARTUP_FLAGS) dump --rules -> +> RULE COUNT ACTIONS BYTES EACH genrule 33,762 33,801 291,538,824 8,635 @@ -218,14 +271,16 @@ _check_proto_library_deps 719 668 1,835,288 2,5 ... (more output) ``` -Look at where the memory is going by producing a `pprof` file using `bazel dump --skylark_memory`: +Look at where the memory is going by producing a `pprof` file +using `bazel dump --skylark_memory`: ``` $ bazel $(STARTUP_FLAGS) dump --skylark_memory=$HOME/prof.gz -> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz +> Dumping Starlark heap to: /usr/local/google/home/$USER/prof.gz ``` -Use the `pprof` tool to investigate the heap. A good starting point is getting a flame graph by using `pprof -flame $HOME/prof.gz`. +Use the `pprof` tool to investigate the heap. A good starting point is +getting a flame graph by using `pprof -flame $HOME/prof.gz`. Get `pprof` from [https://github.com/google/pprof](https://github.com/google/pprof). @@ -233,13 +288,13 @@ Get a text dump of the hottest call sites annotated with lines: ``` $ pprof -text -lines $HOME/prof.gz -> +> flat flat% sum% cum cum% - 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library <native>:-1 - 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule <native>:-1 - 74.11MB 9.96% 44.80% 74.11MB 9.96% glob <native>:-1 - 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup <native>:-1 - 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test <native>:-1 + 146.11MB 19.64% 19.64% 146.11MB 19.64% android_library :-1 + 113.02MB 15.19% 34.83% 113.02MB 15.19% genrule :-1 + 74.11MB 9.96% 44.80% 74.11MB 9.96% glob :-1 + 55.98MB 7.53% 52.32% 55.98MB 7.53% filegroup :-1 + 53.44MB 7.18% 59.51% 53.44MB 7.18% sh_test :-1 26.55MB 3.57% 63.07% 26.55MB 3.57% _generate_foo_files /foo/tc/tc.bzl:491 26.01MB 3.50% 66.57% 26.01MB 3.50% _build_foo_impl /foo/build_test.bzl:78 22.01MB 2.96% 69.53% 22.01MB 2.96% _build_foo_impl /foo/build_test.bzl:73 diff --git a/rules/rules-tutorial.mdx b/rules/rules-tutorial.mdx index 30ad839d..7700f20c 100644 --- a/rules/rules-tutorial.mdx +++ b/rules/rules-tutorial.mdx @@ -2,9 +2,24 @@ title: 'Rules Tutorial' --- -[Starlark](https://github.com/bazelbuild/starlark) is a Python-like configuration language originally developed for use in Bazel and since adopted by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of Starlark properly known as the "Build Language", though it is often simply referred to as "Starlark", especially when emphasizing that a feature is expressed in the Build Language as opposed to being a built-in or "native" part of Bazel. Bazel augments the core language with numerous build-related functions such as `glob`, `genrule`, `java_binary`, and so on. -See the [Bazel](/start/) and [Starlark](/extending/concepts) documentation for more details, and the [Rules SIG template](https://github.com/bazel-contrib/rules-template) as a starting point for new rulesets. + +{/* [TOC] */} + +[Starlark](https://github.com/bazelbuild/starlark) is a Python-like +configuration language originally developed for use in Bazel and since adopted +by other tools. Bazel's `BUILD` and `.bzl` files are written in a dialect of +Starlark properly known as the "Build Language", though it is often simply +referred to as "Starlark", especially when emphasizing that a feature is +expressed in the Build Language as opposed to being a built-in or "native" part +of Bazel. Bazel augments the core language with numerous build-related functions +such as `glob`, `genrule`, `java_binary`, and so on. + +See the +[Bazel](/start/) and [Starlark](/extending/concepts) documentation for +more details, and the +[Rules SIG template](https://github.com/bazel-contrib/rules-template) as a +starting point for new rulesets. ## The empty rule @@ -19,7 +34,10 @@ foo_binary = rule( ) ``` -When you call the [`rule`](lib/globals#rule) function, you must define a callback function. The logic will go there, but you can leave the function empty for now. The [`ctx`](lib/ctx) argument provides information about the target. +When you call the [`rule`](lib/globals#rule) function, you +must define a callback function. The logic will go there, but you +can leave the function empty for now. The [`ctx`](lib/ctx) argument +provides information about the target. You can load the rule and use it from a `BUILD` file. @@ -40,7 +58,9 @@ INFO: Found 1 target... Target //:bin up-to-date (nothing to build) ``` -Even though the rule does nothing, it already behaves like other rules: it has a mandatory name, it supports common attributes like `visibility`, `testonly`, and `tags`. +Even though the rule does nothing, it already behaves like other rules: it has a +mandatory name, it supports common attributes like `visibility`, `testonly`, and +`tags`. ## Evaluation model @@ -69,7 +89,10 @@ foo_binary(name = "bin1") foo_binary(name = "bin2") ``` -[`ctx.label`](lib/ctx#label) corresponds to the label of the target being analyzed. The `ctx` object has many useful fields and methods; you can find an exhaustive list in the [API reference](lib/ctx). +[`ctx.label`](lib/ctx#label) +corresponds to the label of the target being analyzed. The `ctx` object has +many useful fields and methods; you can find an exhaustive list in the +[API reference](lib/ctx). Query the code: @@ -83,10 +106,15 @@ DEBUG: /usr/home/bazel-codelab/BUILD:2:1: BUILD file Make a few observations: -- "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, Bazel evaluates all the files it loads. If multiple `BUILD` files are loading foo.bzl, you would see only one occurrence of "bzl file evaluation" because Bazel caches the result of the evaluation. -- The callback function `_foo_binary_impl` is not called. Bazel query loads `BUILD` files, but doesn't analyze targets. +* "bzl file evaluation" is printed first. Before evaluating the `BUILD` file, + Bazel evaluates all the files it loads. If multiple `BUILD` files are loading + foo.bzl, you would see only one occurrence of "bzl file evaluation" because + Bazel caches the result of the evaluation. +* The callback function `_foo_binary_impl` is not called. Bazel query loads + `BUILD` files, but doesn't analyze targets. -To analyze the targets, use the [`cquery`](/query/cquery) ("configured query") or the `build` command: +To analyze the targets, use the [`cquery`](/query/cquery) ("configured +query") or the `build` command: ``` $ bazel build :all @@ -98,11 +126,15 @@ INFO: Found 2 targets... As you can see, `_foo_binary_impl` is now called twice - once for each target. -Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, because the evaluation of `foo.bzl` is cached after the call to `bazel query`. Bazel only emits `print` statements when they are actually executed. +Notice that neither "bzl file evaluation" nor "BUILD file" are printed again, +because the evaluation of `foo.bzl` is cached after the call to `bazel query`. +Bazel only emits `print` statements when they are actually executed. ## Creating a file -To make your rule more useful, update it to generate a file. First, declare the file and give it a name. In this example, create a file with the same name as the target: +To make your rule more useful, update it to generate a file. First, declare the +file and give it a name. In this example, create a file with the same name as +the target: ```python ctx.actions.declare_file(ctx.label.name) @@ -115,7 +147,9 @@ The following files have no generating action: bin2 ``` -Whenever you declare a file, you have to tell Bazel how to generate it by creating an action. Use [`ctx.actions.write`](lib/actions#write), to create a file with the given content. +Whenever you declare a file, you have to tell Bazel how to generate it by +creating an action. Use [`ctx.actions.write`](lib/actions#write), +to create a file with the given content. ```python def _foo_binary_impl(ctx): @@ -133,7 +167,11 @@ $ bazel build bin1 Target //:bin1 up-to-date (nothing to build) ``` -The `ctx.actions.write` function registered an action, which taught Bazel how to generate the file. But Bazel won't create the file until it is actually requested. So the last thing to do is tell Bazel that the file is an output of the rule, and not a temporary file used within the rule implementation. +The `ctx.actions.write` function registered an action, which taught Bazel +how to generate the file. But Bazel won't create the file until it is +actually requested. So the last thing to do is tell Bazel that the file +is an output of the rule, and not a temporary file used within the rule +implementation. ```python def _foo_binary_impl(ctx): @@ -145,7 +183,8 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Look at the `DefaultInfo` and `depset` functions later. For now, assume that the last line is the way to choose the outputs of a rule. +Look at the `DefaultInfo` and `depset` functions later. For now, +assume that the last line is the way to choose the outputs of a rule. Now, run Bazel: @@ -163,7 +202,8 @@ You have successfully generated a file! ## Attributes -To make the rule more useful, add new attributes using [the `attr` module](lib/attr) and update the rule definition. +To make the rule more useful, add new attributes using +[the `attr` module](lib/attr) and update the rule definition. Add a string attribute called `username`: @@ -185,7 +225,8 @@ foo_binary( ) ``` -To access the value in the callback function, use `ctx.attr.username`. For example: +To access the value in the callback function, use `ctx.attr.username`. For +example: ```python def _foo_binary_impl(ctx): @@ -197,23 +238,38 @@ def _foo_binary_impl(ctx): return [DefaultInfo(files = depset([out]))] ``` -Note that you can make the attribute mandatory or set a default value. Look at the documentation of [`attr.string`](lib/attr#string). You may also use other types of attributes, such as [boolean](lib/attr#bool) or [list of integers](lib/attr#int_list). +Note that you can make the attribute mandatory or set a default value. Look at +the documentation of [`attr.string`](lib/attr#string). +You may also use other types of attributes, such as [boolean](lib/attr#bool) +or [list of integers](lib/attr#int_list). ## Dependencies -Dependency attributes, such as [`attr.label`](lib/attr#label) and [`attr.label_list`](lib/attr#label_list), declare a dependency from the target that owns the attribute to the target whose label appears in the attribute's value. This kind of attribute forms the basis of the target graph. +Dependency attributes, such as [`attr.label`](lib/attr#label) +and [`attr.label_list`](lib/attr#label_list), +declare a dependency from the target that owns the attribute to the target whose +label appears in the attribute's value. This kind of attribute forms the basis +of the target graph. -In the `BUILD` file, the target label appears as a string object, such as `//pkg:name`. In the implementation function, the target will be accessible as a [`Target`](lib/Target) object. For example, view the files returned by the target using [`Target.files`](lib/Target#modules.Target.files). +In the `BUILD` file, the target label appears as a string object, such as +`//pkg:name`. In the implementation function, the target will be accessible as a +[`Target`](lib/Target) object. For example, view the files returned +by the target using [`Target.files`](lib/Target#modules.Target.files). ### Multiple files -By default, only targets created by rules may appear as dependencies (such as a `foo_library()` target). If you want the attribute to accept targets that are input files (such as source files in the repository), you can do it with `allow_files` and specify the list of accepted file extensions (or `True` to allow any file extension): +By default, only targets created by rules may appear as dependencies (such as a +`foo_library()` target). If you want the attribute to accept targets that are +input files (such as source files in the repository), you can do it with +`allow_files` and specify the list of accepted file extensions (or `True` to +allow any file extension): ```python "srcs": attr.label_list(allow_files = [".java"]), ``` -The list of files can be accessed with `ctx.files.<attribute name>`. For example, the list of files in the `srcs` attribute can be accessed through +The list of files can be accessed with `ctx.files.`. For +example, the list of files in the `srcs` attribute can be accessed through ```python ctx.files.srcs @@ -227,7 +283,7 @@ If you need only one file, use `allow_single_file`: "src": attr.label(allow_single_file = [".java"]) ``` -This file is then accessible under `ctx.file.<attribute name>`: +This file is then accessible under `ctx.file.`: ```python ctx.file.src @@ -235,9 +291,17 @@ ctx.file.src ## Create a file with a template -You can create a rule that generates a .cc file based on a template. Also, you can use `ctx.actions.write` to output a string constructed in the rule implementation function, but this has two problems. First, as the template gets bigger, it becomes more memory efficient to put it in a separate file and avoid constructing large strings during the analysis phase. Second, using a separate file is more convenient for the user. Instead, use [`ctx.actions.expand_template`](lib/actions#expand_template), which performs substitutions on a template file. +You can create a rule that generates a .cc file based on a template. Also, you +can use `ctx.actions.write` to output a string constructed in the rule +implementation function, but this has two problems. First, as the template gets +bigger, it becomes more memory efficient to put it in a separate file and avoid +constructing large strings during the analysis phase. Second, using a separate +file is more convenient for the user. Instead, use +[`ctx.actions.expand_template`](lib/actions#expand_template), +which performs substitutions on a template file. -Create a `template` attribute to declare a dependency on the template file: +Create a `template` attribute to declare a dependency on the template +file: ```python def _hello_world_impl(ctx): @@ -276,7 +340,8 @@ cc_binary( ) ``` -If you don't want to expose the template to the end-user and always use the same one, you can set a default value and make the attribute private: +If you don't want to expose the template to the end-user and always use the +same one, you can set a default value and make the attribute private: ```python "_template": attr.label( @@ -285,7 +350,11 @@ If you don't want to expose the template to the end-user and always use the same ), ``` -Attributes that start with an underscore are private and cannot be set in a `BUILD` file. The template is now an *implicit dependency*: Every `hello_world` target has a dependency on this file. Don't forget to make this file visible to other packages by updating the `BUILD` file and using [`exports_files`](/reference/be/functions#exports_files): +Attributes that start with an underscore are private and cannot be set in a +`BUILD` file. The template is now an _implicit dependency_: Every `hello_world` +target has a dependency on this file. Don't forget to make this file visible +to other packages by updating the `BUILD` file and using +[`exports_files`](/reference/be/functions#exports_files): ```python exports_files(["file.cc.tpl"]) @@ -293,6 +362,7 @@ exports_files(["file.cc.tpl"]) ## Going further -- Take a look at the [reference documentation for rules](/extending/rules#contents). -- Get familiar with [depsets](/extending/depsets). -- Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) which includes additional examples of rules. +* Take a look at the [reference documentation for rules](/extending/rules#contents). +* Get familiar with [depsets](/extending/depsets). +* Check out the [examples repository](https://github.com/bazelbuild/examples/tree/master/rules) + which includes additional examples of rules. diff --git a/rules/testing.mdx b/rules/testing.mdx index 4a191c15..2996e08c 100644 --- a/rules/testing.mdx +++ b/rules/testing.mdx @@ -2,23 +2,47 @@ title: 'Testing' --- -There are several different approaches to testing Starlark code in Bazel. This page gathers the current best practices and frameworks by use case. -## Testing rules - -[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) for checking the analysis-time behavior of rules, such as their actions and providers. Such tests are called "analysis tests" and are currently the best option for testing the inner workings of rules. -Some caveats: +There are several different approaches to testing Starlark code in Bazel. This +page gathers the current best practices and frameworks by use case. -- Test assertions occur within the build, not a separate test runner process. Targets that are created by the test must be named such that they do not collide with targets from other tests or from the build. An error that occurs during the test is seen by Bazel as a build breakage rather than a test failure. - -- It requires a fair amount of boilerplate to set up the rules under test and the rules containing test assertions. This boilerplate may seem daunting at first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros are evaluated and targets generated during the loading phase, while rule implementation functions don't run until later, during the analysis phase. +## Testing rules -- Analysis tests are intended to be fairly small and lightweight. Certain features of the analysis testing framework are restricted to verifying targets with a maximum number of transitive dependencies (currently 500). This is due to performance implications of using these features with larger tests. +[Skylib](https://github.com/bazelbuild/bazel-skylib) has a test framework called +[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) +for checking the analysis-time behavior of rules, such as their actions and +providers. Such tests are called "analysis tests" and are currently the best +option for testing the inner workings of rules. -The basic principle is to define a testing rule that depends on the rule-under-test. This gives the testing rule access to the rule-under-test's providers. +Some caveats: -The testing rule's implementation function carries out assertions. If there are any failures, these are not raised immediately by calling `fail()` (which would trigger an analysis-time build error), but rather by storing the errors in a generated script that fails at test execution time. +* Test assertions occur within the build, not a separate test runner process. + Targets that are created by the test must be named such that they do not + collide with targets from other tests or from the build. An error that + occurs during the test is seen by Bazel as a build breakage rather than a + test failure. + +* It requires a fair amount of boilerplate to set up the rules under test and + the rules containing test assertions. This boilerplate may seem daunting at + first. It helps to [keep in mind](/extending/concepts#evaluation-model) that macros + are evaluated and targets generated during the loading phase, while rule + implementation functions don't run until later, during the analysis phase. + +* Analysis tests are intended to be fairly small and lightweight. Certain + features of the analysis testing framework are restricted to verifying + targets with a maximum number of transitive dependencies (currently 500). + This is due to performance implications of using these features with larger + tests. + +The basic principle is to define a testing rule that depends on the +rule-under-test. This gives the testing rule access to the rule-under-test's +providers. + +The testing rule's implementation function carries out assertions. If there are +any failures, these are not raised immediately by calling `fail()` (which would +trigger an analysis-time build error), but rather by storing the errors in a +generated script that fails at test execution time. See below for a minimal toy example, followed by an example that checks actions. @@ -45,6 +69,7 @@ myrule = rule( `//mypkg/myrules_test.bzl`: + ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest") load(":myrules.bzl", "myrule", "MyInfo") @@ -114,31 +139,47 @@ myrules_test_suite(name = "myrules_test") The test can be run with `bazel test //mypkg:myrules_test`. -Aside from the initial `load()` statements, there are two main parts to the file: +Aside from the initial `load()` statements, there are two main parts to the +file: -- The tests themselves, each of which consists of 1) an analysis-time implementation function for the testing rule, 2) a declaration of the testing rule via `analysistest.make()`, and 3) a loading-time function (macro) for declaring the rule-under-test (and its dependencies) and testing rule. If the assertions do not change between test cases, 1) and 2) may be shared by multiple test cases. +* The tests themselves, each of which consists of 1) an analysis-time + implementation function for the testing rule, 2) a declaration of the + testing rule via `analysistest.make()`, and 3) a loading-time function + (macro) for declaring the rule-under-test (and its dependencies) and testing + rule. If the assertions do not change between test cases, 1) and 2) may be + shared by multiple test cases. -- The test suite function, which calls the loading-time functions for each test, and declares a `test_suite` target bundling all tests together. +* The test suite function, which calls the loading-time functions for each + test, and declares a `test_suite` target bundling all tests together. -For consistency, follow the recommended naming convention: Let `foo` stand for the part of the test name that describes what the test is checking (`provider_contents` in the above example). For example, a JUnit test method would be named `testFoo`. +For consistency, follow the recommended naming convention: Let `foo` stand for +the part of the test name that describes what the test is checking +(`provider_contents` in the above example). For example, a JUnit test method +would be named `testFoo`. Then: -- the macro which generates the test and target under test should should be named `_test_foo` (`_test_provider_contents`) +* the macro which generates the test and target under test should should be + named `_test_foo` (`_test_provider_contents`) -- its test rule type should be named `foo_test` (`provider_contents_test`) +* its test rule type should be named `foo_test` (`provider_contents_test`) -- the label of the target of this rule type should be `foo_test` (`provider_contents_test`) +* the label of the target of this rule type should be `foo_test` + (`provider_contents_test`) -- the implementation function for the testing rule should be named `_foo_test_impl` (`_provider_contents_test_impl`) +* the implementation function for the testing rule should be named + `_foo_test_impl` (`_provider_contents_test_impl`) -- the labels of the targets of the rules under test and their dependencies should be prefixed with `foo_` (`provider_contents_`) +* the labels of the targets of the rules under test and their dependencies + should be prefixed with `foo_` (`provider_contents_`) -Note that the labels of all targets can conflict with other labels in the same BUILD package, so it's helpful to use a unique name for the test. +Note that the labels of all targets can conflict with other labels in the same +BUILD package, so it's helpful to use a unique name for the test. ### Failure testing -It may be useful to verify that a rule fails given certain inputs or in certain state. This can be done using the analysis test framework: +It may be useful to verify that a rule fails given certain inputs or in certain +state. This can be done using the analysis test framework: The test rule created with `analysistest.make` should specify `expect_failure`: @@ -149,7 +190,8 @@ failure_testing_test = analysistest.make( ) ``` -The test rule implementation should make assertions on the nature of the failure that took place (specifically, the failure message): +The test rule implementation should make assertions on the nature of the failure +that took place (specifically, the failure message): ```python def _failure_testing_test_impl(ctx): @@ -158,7 +200,11 @@ def _failure_testing_test_impl(ctx): return analysistest.end(env) ``` -Also make sure that your target under test is specifically tagged 'manual'. Without this, building all targets in your package using `:all` will result in a build of the intentionally-failing target and will exhibit a build failure. With 'manual', your target under test will build only if explicitly specified, or as a dependency of a non-manual target (such as your test rule): +Also make sure that your target under test is specifically tagged 'manual'. +Without this, building all targets in your package using `:all` will result in a +build of the intentionally-failing target and will exhibit a build failure. With +'manual', your target under test will build only if explicitly specified, or as +a dependency of a non-manual target (such as your test rule): ```python def _test_failure(): @@ -173,7 +219,9 @@ def _test_failure(): ### Verifying registered actions -You may want to write tests which make assertions about the actions that your rule registers, for example, using `ctx.actions.run()`. This can be done in your analysis test rule implementation function. An example: +You may want to write tests which make assertions about the actions that your +rule registers, for example, using `ctx.actions.run()`. This can be done in your +analysis test rule implementation function. An example: ```python def _inspect_actions_test_impl(ctx): @@ -188,11 +236,14 @@ def _inspect_actions_test_impl(ctx): return analysistest.end(env) ``` -Note that `analysistest.target_actions(env)` returns a list of [`Action`](lib/Action) objects which represent actions registered by the target under test. +Note that `analysistest.target_actions(env)` returns a list of +[`Action`](lib/Action) objects which represent actions registered by the +target under test. ### Verifying rule behavior under different flags -You may want to verify your real rule behaves a certain way given certain build flags. For example, your rule may behave differently if a user specifies: +You may want to verify your real rule behaves a certain way given certain build +flags. For example, your rule may behave differently if a user specifies: ```shell bazel build //mypkg:real_target -c opt @@ -204,15 +255,20 @@ versus bazel build //mypkg:real_target -c dbg ``` -At first glance, this could be done by testing the target under test using the desired build flags: +At first glance, this could be done by testing the target under test using the +desired build flags: ```shell bazel test //mypkg:myrules_test -c opt ``` -But then it becomes impossible for your test suite to simultaneously contain a test which verifies the rule behavior under `-c opt` and another test which verifies the rule behavior under `-c dbg`. Both tests would not be able to run in the same build! +But then it becomes impossible for your test suite to simultaneously contain a +test which verifies the rule behavior under `-c opt` and another test which +verifies the rule behavior under `-c dbg`. Both tests would not be able to run +in the same build! -This can be solved by specifying the desired build flags when defining the test rule: +This can be solved by specifying the desired build flags when defining the test +rule: ```python myrule_c_opt_test = analysistest.make( @@ -223,21 +279,33 @@ myrule_c_opt_test = analysistest.make( ) ``` -Normally, a target under test is analyzed given the current build flags. Specifying `config_settings` overrides the values of the specified command line options. (Any unspecified options will retain their values from the actual command line). +Normally, a target under test is analyzed given the current build flags. +Specifying `config_settings` overrides the values of the specified command line +options. (Any unspecified options will retain their values from the actual +command line). + +In the specified `config_settings` dictionary, command line flags must be +prefixed with a special placeholder value `//command_line_option:`, as is shown +above. -In the specified `config_settings` dictionary, command line flags must be prefixed with a special placeholder value `//command_line_option:`, as is shown above. ## Validating artifacts The main ways to check that your generated files are correct are: -- You can write a test script in shell, Python, or another language, and create a target of the appropriate `*_test` rule type. +* You can write a test script in shell, Python, or another language, and + create a target of the appropriate `*_test` rule type. -- You can use a specialized rule for the kind of test you want to perform. +* You can use a specialized rule for the kind of test you want to perform. ### Using a test target -The most straightforward way to validate an artifact is to write a script and add a `*_test` target to your BUILD file. The specific artifacts you want to check should be data dependencies of this target. If your validation logic is reusable for multiple tests, it should be a script that takes command line arguments that are controlled by the test target's `args` attribute. Here's an example that validates that the output of `myrule` from above is `"abc"`. +The most straightforward way to validate an artifact is to write a script and +add a `*_test` target to your BUILD file. The specific artifacts you want to +check should be data dependencies of this target. If your validation logic is +reusable for multiple tests, it should be a script that takes command line +arguments that are controlled by the test target's `args` attribute. Here's an +example that validates that the output of `myrule` from above is `"abc"`. `//mypkg/myrule_validator.sh`: @@ -273,7 +341,12 @@ sh_test( ### Using a custom rule -A more complicated alternative is to write the shell script as a template that gets instantiated by a new rule. This involves more indirection and Starlark logic, but leads to cleaner BUILD files. As a side-benefit, any argument preprocessing can be done in Starlark instead of the script, and the script is slightly more self-documenting since it uses symbolic placeholders (for substitutions) instead of numeric ones (for arguments). +A more complicated alternative is to write the shell script as a template that +gets instantiated by a new rule. This involves more indirection and Starlark +logic, but leads to cleaner BUILD files. As a side-benefit, any argument +preprocessing can be done in Starlark instead of the script, and the script is +slightly more self-documenting since it uses symbolic placeholders (for +substitutions) instead of numeric ones (for arguments). `//mypkg/myrule_validator.sh.template`: @@ -344,11 +417,18 @@ myrule_validation_test( ) ``` -Alternatively, instead of using a template expansion action, you could have inlined the template into the .bzl file as a string and expanded it during the analysis phase using the `str.format` method or `%`-formatting. +Alternatively, instead of using a template expansion action, you could have +inlined the template into the .bzl file as a string and expanded it during the +analysis phase using the `str.format` method or `%`-formatting. ## Testing Starlark utilities -[Skylib](https://github.com/bazelbuild/bazel-skylib)'s [`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) framework can be used to test utility functions (that is, functions that are neither macros nor rule implementations). Instead of using `unittest.bzl`'s `analysistest` library, `unittest` may be used. For such test suites, the convenience function `unittest.suite()` can be used to reduce boilerplate. +[Skylib](https://github.com/bazelbuild/bazel-skylib)'s +[`unittest.bzl`](https://github.com/bazelbuild/bazel-skylib/blob/main/lib/unittest.bzl) +framework can be used to test utility functions (that is, functions that are +neither macros nor rule implementations). Instead of using `unittest.bzl`'s +`analysistest` library, `unittest` may be used. For such test suites, the +convenience function `unittest.suite()` can be used to reduce boilerplate. `//mypkg/myhelpers.bzl`: @@ -359,6 +439,7 @@ def myhelper(): `//mypkg/myhelpers_test.bzl`: + ```python load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") load(":myhelpers.bzl", "myhelper") diff --git a/rules/verbs-tutorial.mdx b/rules/verbs-tutorial.mdx index 4cd6c990..db7757e1 100644 --- a/rules/verbs-tutorial.mdx +++ b/rules/verbs-tutorial.mdx @@ -2,17 +2,29 @@ title: 'Using Macros to Create Custom Verbs' --- -Day-to-day interaction with Bazel happens primarily through a few commands: `build`, `test`, and `run`. At times, though, these can feel limited: you may want to push packages to a repository, publish documentation for end-users, or deploy an application with Kubernetes. But Bazel doesn't have a `publish` or `deploy` command – where do these actions fit in? -## The bazel run command -Bazel's focus on hermeticity, reproducibility, and incrementality means the `build` and `test` commands aren't helpful for the above tasks. These actions may run in a sandbox, with limited network access, and aren't guaranteed to be re-run with every `bazel build`. +Day-to-day interaction with Bazel happens primarily through a few commands: +`build`, `test`, and `run`. At times, though, these can feel limited: you may +want to push packages to a repository, publish documentation for end-users, or +deploy an application with Kubernetes. But Bazel doesn't have a `publish` or +`deploy` command – where do these actions fit in? + +## The bazel run command -Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have side effects. Bazel users are accustomed to rules that create executables, and rule authors can follow a common set of patterns to extend this to "custom verbs". +Bazel's focus on hermeticity, reproducibility, and incrementality means the +`build` and `test` commands aren't helpful for the above tasks. These actions +may run in a sandbox, with limited network access, and aren't guaranteed to be +re-run with every `bazel build`. -### In the wild: rules\_k8s +Instead, rely on `bazel run`: the workhorse for tasks that you *want* to have +side effects. Bazel users are accustomed to rules that create executables, and +rule authors can follow a common set of patterns to extend this to +"custom verbs". -For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), the Kubernetes rules for Bazel. Suppose you have the following target: +### In the wild: rules_k8s +For example, consider [`rules_k8s`](https://github.com/bazelbuild/rules_k8s), +the Kubernetes rules for Bazel. Suppose you have the following target: ```python # BUILD file in //application/k8s @@ -24,21 +36,51 @@ k8s_object( ) ``` -The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a standard Kubernetes YAML file when `bazel build` is used on the `staging` target. However, the additional targets are also created by the `k8s_object` macro with names like `staging.apply` and `:staging.delete`. These build scripts to perform those actions, and when executed with `bazel run staging.apply`, these behave like our own `bazel k8s-apply` or `bazel k8s-delete` commands. - -### Another example: ts\_api\_guardian\_test - -This pattern can also be seen in the Angular project. The [`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) produces two targets. The first is a standard `nodejs_test` target which compares some generated output against a "golden" file (that is, a file containing the expected output). This can be built and run with a normal `bazel test` invocation. In `angular-cli`, you can run [one such target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) with `bazel test //etc/api:angular_devkit_core_api`. - -Over time, this golden file may need to be updated for legitimate reasons. Updating this manually is tedious and error-prone, so this macro also provides a `nodejs_binary` target that updates the golden file, instead of comparing against it. Effectively, the same test script can be written to run in "verify" or "accept" mode, based on how it's invoked. This follows the same pattern you've learned already: there is no native `bazel test-accept` command, but the same effect can be achieved with `bazel run //etc/api:angular_devkit_core_api.accept`. - -This pattern can be quite powerful, and turns out to be quite common once you learn to recognize it. +The [`k8s_object` rule](https://github.com/bazelbuild/rules_k8s#usage) builds a +standard Kubernetes YAML file when `bazel build` is used on the `staging` +target. However, the additional targets are also created by the `k8s_object` +macro with names like `staging.apply` and `:staging.delete`. These build +scripts to perform those actions, and when executed with `bazel run +staging.apply`, these behave like our own `bazel k8s-apply` or `bazel +k8s-delete` commands. + +### Another example: ts_api_guardian_test + +This pattern can also be seen in the Angular project. The +[`ts_api_guardian_test` macro](https://github.com/angular/angular/blob/16ac611a8410e6bcef8ffc779f488ca4fa102155/tools/ts-api-guardian/index.bzl#L22) +produces two targets. The first is a standard `nodejs_test` target which compares +some generated output against a "golden" file (that is, a file containing the +expected output). This can be built and run with a normal `bazel +test` invocation. In `angular-cli`, you can run [one such +target](https://github.com/angular/angular-cli/blob/e1269cb520871ee29b1a4eec6e6c0e4a94f0b5fc/etc/api/BUILD) +with `bazel test //etc/api:angular_devkit_core_api`. + +Over time, this golden file may need to be updated for legitimate reasons. +Updating this manually is tedious and error-prone, so this macro also provides +a `nodejs_binary` target that updates the golden file, instead of comparing +against it. Effectively, the same test script can be written to run in "verify" +or "accept" mode, based on how it's invoked. This follows the same pattern +you've learned already: there is no native `bazel test-accept` command, but the +same effect can be achieved with +`bazel run //etc/api:angular_devkit_core_api.accept`. + +This pattern can be quite powerful, and turns out to be quite common once you +learn to recognize it. ## Adapting your own rules -[Macros](/extending/macros) are the heart of this pattern. Macros are used like rules, but they can create several targets. Typically, they will create a target with the specified name which performs the primary build action: perhaps it builds a normal binary, a Docker image, or an archive of source code. In this pattern, additional targets are created to produce scripts performing side effects based on the output of the primary target, like publishing the resulting binary or updating the expected test output. +[Macros](/extending/macros) are the heart of this pattern. Macros are used like +rules, but they can create several targets. Typically, they will create a +target with the specified name which performs the primary build action: perhaps +it builds a normal binary, a Docker image, or an archive of source code. In +this pattern, additional targets are created to produce scripts performing side +effects based on the output of the primary target, like publishing the +resulting binary or updating the expected test output. -To illustrate this, wrap an imaginary rule that generates a website with [Sphinx](https://www.sphinx-doc.org) with a macro to create an additional target that allows the user to publish it when ready. Consider the following existing rule for generating a website with Sphinx: +To illustrate this, wrap an imaginary rule that generates a website with +[Sphinx](https://www.sphinx-doc.org) with a macro to create an additional +target that allows the user to publish it when ready. Consider the following +existing rule for generating a website with Sphinx: ```python _sphinx_site = rule( @@ -47,7 +89,8 @@ _sphinx_site = rule( ) ``` -Next, consider a rule like the following, which builds a script that, when run, publishes the generated pages: +Next, consider a rule like the following, which builds a script that, when run, +publishes the generated pages: ```python _sphinx_publisher = rule( @@ -63,7 +106,8 @@ _sphinx_publisher = rule( ) ``` -Finally, define the following symbolic macro (available in Bazel 8 or newer) to create targets for both of the above rules together: +Finally, define the following symbolic macro (available in Bazel 8 or newer) to +create targets for both of the above rules together: ```starlark def _sphinx_site_impl(name, visibility, srcs, **kwargs): @@ -84,7 +128,8 @@ sphinx_site = macro( ) ``` -Or, if you need to support Bazel releases older than Bazel 8, you would instead define a legacy macro: +Or, if you need to support Bazel releases older than Bazel 8, you would instead +define a legacy macro: ```starlark def sphinx_site(name, srcs = [], **kwargs): @@ -95,7 +140,8 @@ def sphinx_site(name, srcs = [], **kwargs): _sphinx_publisher(name = "%s.publish" % name, site = name, **kwargs) ``` -In the `BUILD` files, use the macro as though it just creates the primary target: +In the `BUILD` files, use the macro as though it just creates the primary +target: ```python sphinx_site( @@ -104,8 +150,28 @@ sphinx_site( ) ``` -In this example, a "docs" target is created, just as though the macro were a standard, single Bazel rule. When built, the rule generates some configuration and runs Sphinx to produce an HTML site, ready for manual inspection. However, an additional "docs.publish" target is also created, which builds a script for publishing the site. Once you check the output of the primary target, you can use `bazel run :docs.publish` to publish it for public consumption, just like an imaginary `bazel publish` command. - -It's not immediately obvious what the implementation of the `_sphinx_publisher` rule might look like. Often, actions like this write a *launcher* shell script. This method typically involves using [`ctx.actions.expand_template`](lib/actions#expand_template) to write a very simple shell script, in this case invoking the publisher binary with a path to the output of the primary target. This way, the publisher implementation can remain generic, the `_sphinx_site` rule can just produce HTML, and this small script is all that's necessary to combine the two together. - -In `rules_k8s`, this is indeed what `.apply` does: [`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) writes a very simple Bash script, based on [`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), which runs `kubectl` with the output of the primary target. This script can then be build and run with `bazel run :staging.apply`, effectively providing a `k8s-apply` command for `k8s_object` targets. +In this example, a "docs" target is created, just as though the macro were a +standard, single Bazel rule. When built, the rule generates some configuration +and runs Sphinx to produce an HTML site, ready for manual inspection. However, +an additional "docs.publish" target is also created, which builds a script for +publishing the site. Once you check the output of the primary target, you can +use `bazel run :docs.publish` to publish it for public consumption, just like +an imaginary `bazel publish` command. + +It's not immediately obvious what the implementation of the `_sphinx_publisher` +rule might look like. Often, actions like this write a _launcher_ shell script. +This method typically involves using +[`ctx.actions.expand_template`](lib/actions#expand_template) +to write a very simple shell script, in this case invoking the publisher binary +with a path to the output of the primary target. This way, the publisher +implementation can remain generic, the `_sphinx_site` rule can just produce +HTML, and this small script is all that's necessary to combine the two +together. + +In `rules_k8s`, this is indeed what `.apply` does: +[`expand_template`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/object.bzl#L213-L241) +writes a very simple Bash script, based on +[`apply.sh.tpl`](https://github.com/bazelbuild/rules_k8s/blob/f10e7025df7651f47a76abf1db5ade1ffeb0c6ac/k8s/apply.sh.tpl), +which runs `kubectl` with the output of the primary target. This script can +then be build and run with `bazel run :staging.apply`, effectively providing a +`k8s-apply` command for `k8s_object` targets. diff --git a/rules/windows.mdx b/rules/windows.mdx deleted file mode 100644 index f6422687..00000000 --- a/rules/windows.mdx +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: 'Writing Rules on Windows' ---- - -This page focuses on writing Windows-compatible rules, common problems of writing portable rules, and some solutions. - -## Paths - -Problems: - -- **Length limit**: maximum path length is 259 characters. - - Though Windows also supports longer paths (up to 32767 characters), many programs are built with the lower limit. - - Be aware of this about programs you run in the actions. - -- **Working directory**: is also limited to 259 characters. - - Processes cannot `cd` into a directory longer than 259 characters. - -- **Case-sensitivity**: Windows paths are case-insensitive, Unix paths are case-sensitive. - - Be aware of this when creating command lines for actions. - -- **Path separators**: are backslash (`\`), not forward slash (`/`). - - Bazel stores paths Unix-style with `/` separators. Though some Windows programs support Unix-style paths, others don't. Some built-in commands in cmd.exe support them, some don't. - - It's best to always use `\` separators on Windows: replace `/` with `\` when you create command lines and environment variables for actions. - -- **Absolute paths**: don't start with slash (`/`). - - Absolute paths on Windows start with a drive letter, such as `C:\foo\bar.txt`. There's no single filesystem root. - - Be aware of this if your rule checks if a path is absolute. Absolute paths should be avoided since they are often non-portable. - -Solutions: - -- **Keep paths short.** - - Avoid long directory names, deeply nested directory structures, long file names, long workspace names, long target names. - - All of these may become path components of actions' input files, and may exhaust the path length limit. - -- **Use a short output root.** - - Use the `--output_user_root=<path>` flag to specify a short path for Bazel outputs. A good idea is to have a drive (or virtual drive) just for Bazel outputs (such as `D:\`), and adding this line to your `.bazelrc` file: - - ``` - build --output_user_root=D:/ - ``` - - or - - ``` - build --output_user_root=C:/_bzl - ``` - -- **Use junctions.** - - Junctions are, loosely speaking\[1], directory symlinks. Junctions are easy to create and can point to directories (on the same computer) with long paths. If a build action creates a junction whose path is short but whose target is long, then tools with short path limit can access the files in the junction'ed directory. - - In `.bat` files or in cmd.exe you can create junctions like so: - - ``` - mklink /J c:\path\to\junction c:\path\to\very\long\target\path - ``` - - \[1]: Strictly speaking [Junctions are not Symbolic Links](https://superuser.com/a/343079), but for the sake of build actions you may regard Junctions as Directory Symlinks. - -- **Replace `/` with `\` in paths in actions / envvars.** - - When you create the command line or environment variables for an action, make the paths Windows-style. Example: - - ```python - def as_path(p, is_windows): - if is_windows: - return p.replace("/", "\\") - else: - return p - ``` - -## Environment variables - -Problems: - -- **Case-sensitivity**: Windows environment variable names are case-insensitive. - - For example, in Java `System.getenv("SystemRoot")` and `System.getenv("SYSTEMROOT")` yields the same result. (This applies to other languages too.) - -- **Hermeticity**: actions should use as few custom environment variables as possible. - - Environment variables are part of the action's cache key. If an action uses environment variables that change often, or are custom to users, that makes the rule less cache-able. - -Solutions: - -- **Only use upper-case environment variable names.** - - This works on Windows, macOS, and Linux. - -- **Minimize action environments.** - - When using `ctx.actions.run`, set the environment to `ctx.configuration.default_shell_env`. If the action needs more environment variables, put them all in a dictionary and pass that to the action. Example: - - ```python - load("@bazel_skylib//lib:dicts.bzl", "dicts") - - def _make_env(ctx, output_file, is_windows): - out_path = output_file.path - if is_windows: - out_path = out_path.replace("/", "\\") - return dicts.add(ctx.configuration.default_shell_env, {"MY_OUTPUT": out_path}) - ``` - -## Actions - -Problems: - -- **Executable outputs**: Every executable file must have an executable extension. - - The most common extensions are `.exe` (binary files) and `.bat` (Batch scripts). - - Be aware that shell scripts (`.sh`) are NOT executable on Windows; you cannot specify them as `ctx.actions.run`'s `executable`. There's also no `+x` permission that files can have, so you can't execute arbitrary files like on Linux. - -- **Bash commands**: For sake of portability, avoid running Bash commands directly in actions. - - Bash is widespread on Unix-like systems, but it's often unavailable on Windows. Bazel itself is relying less and less on Bash (MSYS2), so in the future users would be less likely to have MSYS2 installed along with Bazel. To make rules easier to use on Windows, avoid running Bash commands in actions. - -- **Line endings**: Windows uses CRLF (`\r\n`), Unix-like systems uses LF (`\n`). - - Be aware of this when comparing text files. Be mindful of your Git settings, especially of line endings when checking out or committing. (See Git's `core.autocrlf` setting.) - -Solutions: - -- **Use a Bash-less purpose-made rule.** - - `native.genrule()` is a wrapper for Bash commands, and it's often used to solve simple problems like copying a file or writing a text file. You can avoid relying on Bash (and reinventing the wheel): see if bazel-skylib has a purpose-made rule for your needs. None of them depends on Bash when built/tested on Windows. - - Build rule examples: - - - `copy_file()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_file.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/copy_file_doc.md)): copies a file somewhere else, optionally making it executable - - - `write_file()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/write_file.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/write_file_doc.md)): writes a text file, with the desired line endings (`auto`, `unix`, or `windows`), optionally making it executable (if it's a script) - - - `run_binary()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/run_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/run_binary_doc.md)): runs a binary (or `*_binary` rule) with given inputs and expected outputs as a build action (this is a build rule wrapper for `ctx.actions.run`) - - - `native_binary()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/native_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/native_binary_doc.md#native_binary)): wraps a native binary in a `*_binary` rule, which you can `bazel run` or use in `run_binary()`'s `tool` attribute or `native.genrule()`'s `tools` attribute - - Test rule examples: - - - `diff_test()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/diff_test.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/diff_test_doc.md)): test that compares contents of two files - - - `native_test()` ([source](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/native_binary.bzl), [documentation](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/native_binary_doc.md#native_test)): wraps a native binary in a `*_test` rule, which you can `bazel test` - -- **On Windows, consider using `.bat` scripts for trivial things.** - - Instead of `.sh` scripts, you can solve trivial tasks with `.bat` scripts. - - For example, if you need a script that does nothing, or prints a message, or exits with a fixed error code, then a simple `.bat` file will suffice. If your rule returns a `DefaultInfo()` provider, the `executable` field may refer to that `.bat` file on Windows. - - And since file extensions don't matter on macOS and Linux, you can always use `.bat` as the extension, even for shell scripts. - - Be aware that empty `.bat` files cannot be executed. If you need an empty script, write one space in it. - -- **Use Bash in a principled way.** - - In Starlark build and test rules, use `ctx.actions.run_shell` to run Bash scripts and Bash commands as actions. - - In Starlark macros, wrap Bash scripts and commands in a `native.sh_binary()` or `native.genrule()`. Bazel will check if Bash is available and run the script or command through Bash. - - In Starlark repository rules, try avoiding Bash altogether. Bazel currently offers no way to run Bash commands in a principled way in repository rules. - -## Deleting files - -Problems: - -- **Files cannot be deleted while open.** - - Open files cannot be deleted (by default), attempts result in "Access Denied" errors. If you cannot delete a file, maybe a running process still holds it open. - -- **Working directory of a running process cannot be deleted.** - - Processes have an open handle to their working directory, and the directory cannot be deleted until the process terminates. - -Solutions: - -- **In your code, try to close files eagerly.** - - In Java, use `try-with-resources`. In Python, use `with open(...) as f:`. In principle, try closing handles as soon as possible. diff --git a/run/bazelrc.mdx b/run/bazelrc.mdx index 2ca427d7..90d4464a 100644 --- a/run/bazelrc.mdx +++ b/run/bazelrc.mdx @@ -2,85 +2,129 @@ title: 'Write bazelrc configuration files' --- -Bazel accepts many options. Some options are varied frequently (for example, `--subcommands`) while others stay the same across several builds (such as `--package_path`). To avoid specifying these unchanged options for every build (and other commands), you can specify options in a configuration file, called `.bazelrc`. + + +Bazel accepts many options. Some options are varied frequently (for example, +`--subcommands`) while others stay the same across several builds (such as +`--package_path`). To avoid specifying these unchanged options for every build +(and other commands), you can specify options in a configuration file, called +`.bazelrc`. ### Where are the `.bazelrc` files? -Bazel looks for optional configuration files in the following locations, in the order shown below. The options are interpreted in this order, so options in later files can override a value from an earlier file if a conflict arises. All options that control which of these files are loaded are startup options, which means they must occur after `bazel` and before the command (`build`, `test`, etc). +Bazel looks for optional configuration files in the following locations, +in the order shown below. The options are interpreted in this order, so +options in later files can override a value from an earlier file if a +conflict arises. All options that control which of these files are loaded are +startup options, which means they must occur after `bazel` and +before the command (`build`, `test`, etc). + +1. **The system RC file**, unless `--nosystem_rc` is present. -1. **The system RC file**, unless `--nosystem_rc` is present. + Path: - Path: + - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` + - On Windows: `%ProgramData%\bazel.bazelrc` - - On Linux/macOS/Unixes: `/etc/bazel.bazelrc` - - On Windows: `%ProgramData%\bazel.bazelrc` + It is not an error if this file does not exist. - It is not an error if this file does not exist. + If another system-specified location is required, you must build a custom + Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in + [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). + The system-specified location may contain environment variable references, + such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. - If another system-specified location is required, you must build a custom Bazel binary, overriding the `BAZEL_SYSTEM_BAZELRC_PATH` value in [`//src/main/cpp:option_processor`](https://github.com/bazelbuild/bazel/blob/0.28.0/src/main/cpp/BUILD#L141). The system-specified location may contain environment variable references, such as `${VAR_NAME}` on Unix or `%VAR_NAME%` on Windows. +2. **The workspace RC file**, unless `--noworkspace_rc` is present. -2. **The workspace RC file**, unless `--noworkspace_rc` is present. + Path: `.bazelrc` in your workspace directory (next to the main + `MODULE.bazel` file). - Path: `.bazelrc` in your workspace directory (next to the main `MODULE.bazel` file). + It is not an error if this file does not exist. - It is not an error if this file does not exist. +3. **The home RC file**, unless `--nohome_rc` is present. -3. **The home RC file**, unless `--nohome_rc` is present. + Path: - Path: + - On Linux/macOS/Unixes: `$HOME/.bazelrc` + - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` - - On Linux/macOS/Unixes: `$HOME/.bazelrc` - - On Windows: `%USERPROFILE%\.bazelrc` if exists, or `%HOME%/.bazelrc` + It is not an error if this file does not exist. - It is not an error if this file does not exist. +4. **The environment variable RC file**, if its path is set with the `BAZELRC` + environment variable. -4. **The environment variable RC file**, if its path is set with the `BAZELRC` environment variable. + The environment variable can include multiple comma-separated paths. - The environment variable can include multiple comma-separated paths. +5. **The user-specified RC file**, if specified with + --bazelrc=file -5. **The user-specified RC file**, if specified with `--bazelrc=file` + This flag is optional but can also be specified multiple times. - This flag is optional but can also be specified multiple times. + `/dev/null` indicates that all further `--bazelrc`s will be ignored, which + is useful to disable the search for a user rc file, such as in release + builds. - `/dev/null` indicates that all further `--bazelrc`s will be ignored, which is useful to disable the search for a user rc file, such as in release builds. + For example: - For example: + ``` + --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc + ``` - ``` - --bazelrc=x.rc --bazelrc=y.rc --bazelrc=/dev/null --bazelrc=z.rc - ``` + - `x.rc` and `y.rc` are read. + - `z.rc` is ignored due to the prior `/dev/null`. - - `x.rc` and `y.rc` are read. - - `z.rc` is ignored due to the prior `/dev/null`. +In addition to this optional configuration file, Bazel looks for a global rc +file. For more details, see the [global bazelrc section](#global-bazelrc). -In addition to this optional configuration file, Bazel looks for a global rc file. For more details, see the [global bazelrc section](#global-bazelrc). ### `.bazelrc` syntax and semantics -Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based grammar. Empty lines and lines starting with `#` (comments) are ignored. Each line contains a sequence of words, which are tokenized according to the same rules as the Bourne shell. +Like all UNIX "rc" files, the `.bazelrc` file is a text file with a line-based +grammar. Empty lines and lines starting with `#` (comments) are ignored. Each +line contains a sequence of words, which are tokenized according to the same +rules as the Bourne shell. #### Imports -Lines that start with `import` or `try-import` are special: use these to load other "rc" files. To specify a path that is relative to the workspace root, write `import %workspace%/path/to/bazelrc`. +Lines that start with `import` or `try-import` are special: use these to load +other "rc" files. To specify a path that is relative to the workspace root, +write `import %workspace%/path/to/bazelrc`. -The difference between `import` and `try-import` is that Bazel fails if the `import`'ed file is missing (or can't be read), but not so for a `try-import`'ed file. +The difference between `import` and `try-import` is that Bazel fails if the +`import`'ed file is missing (or can't be read), but not so for a `try-import`'ed +file. Import precedence: -- Options in the imported file take precedence over options specified before the import statement. -- Options specified after the import statement take precedence over the options in the imported file. -- Options in files imported later take precedence over files imported earlier. +- Options in the imported file take precedence over options specified before + the import statement. +- Options specified after the import statement take precedence over the + options in the imported file. +- Options in files imported later take precedence over files imported earlier. #### Option defaults -Most lines of a bazelrc define default option values. The first word on each line specifies when these defaults are applied: - -- `startup`: startup options, which go before the command, and are described in `bazel help startup_options`. -- `common`: options that should be applied to all Bazel commands that support them. If a command does not support an option specified in this way, the option is ignored so long as it is valid for *some* other Bazel command. Note that this only applies to option names: If the current command accepts an option with the specified name, but doesn't support the specified value, it will fail. -- `always`: options that apply to all Bazel commands. If a command does not support an option specified in this way, it will fail. -- *`command`*: Bazel command, such as `build` or `query` to which the options apply. These options also apply to all commands that inherit from the specified command. (For example, `test` inherits from `build`.) - -Each of these lines may be used more than once and the arguments that follow the first word are combined as if they had appeared on a single line. (Users of CVS, another tool with a "Swiss army knife" command-line interface, will find the syntax similar to that of `.cvsrc`.) For example, the lines: +Most lines of a bazelrc define default option values. The first word on each +line specifies when these defaults are applied: + +- `startup`: startup options, which go before the command, and are described + in `bazel help startup_options`. +- `common`: options that should be applied to all Bazel commands that support + them. If a command does not support an option specified in this way, the + option is ignored so long as it is valid for *some* other Bazel command. + Note that this only applies to option names: If the current command accepts + an option with the specified name, but doesn't support the specified value, + it will fail. +- `always`: options that apply to all Bazel commands. If a command does not + support an option specified in this way, it will fail. +- _`command`_: Bazel command, such as `build` or `query` to which the options + apply. These options also apply to all commands that inherit from the + specified command. (For example, `test` inherits from `build`.) + +Each of these lines may be used more than once and the arguments that follow the +first word are combined as if they had appeared on a single line. (Users of CVS, +another tool with a "Swiss army knife" command-line interface, will find the +syntax similar to that of `.cvsrc`.) For example, the lines: ```posix-terminal build --test_tmpdir=/tmp/foo --verbose_failures @@ -98,49 +142,89 @@ so the effective flags are `--verbose_failures` and `--test_tmpdir=/tmp/bar`. Option precedence: -- Options on the command line always take precedence over those in rc files. For example, if a rc file says `build -c opt` but the command line flag is `-c dbg`, the command line flag takes precedence. +- Options on the command line always take precedence over those in rc files. + For example, if a rc file says `build -c opt` but the command line flag is + `-c dbg`, the command line flag takes precedence. +- Within the rc file, precedence is governed by specificity: lines for a more + specific command take precedence over lines for a less specific command. -- Within the rc file, precedence is governed by specificity: lines for a more specific command take precedence over lines for a less specific command. + Specificity is defined by inheritance. Some commands inherit options from + other commands, making the inheriting command more specific than the base + command. For example `test` inherits from the `build` command, so all `bazel + build` flags are valid for `bazel test`, and all `build` lines apply also to + `bazel test` unless there's a `test` line for the same option. If the rc + file says: - Specificity is defined by inheritance. Some commands inherit options from other commands, making the inheriting command more specific than the base command. For example `test` inherits from the `build` command, so all `bazel build` flags are valid for `bazel test`, and all `build` lines apply also to `bazel test` unless there's a `test` line for the same option. If the rc file says: + ```posix-terminal + test -c dbg --test_env=PATH - ```posix-terminal - test -c dbg --test_env=PATH + build -c opt --verbose_failures + ``` - build -c opt --verbose_failures - ``` + then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel + test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. - then `bazel build //foo` will use `-c opt --verbose_failures`, and `bazel test //foo` will use `--verbose_failures -c dbg --test_env=PATH`. + The inheritance (specificity) graph is: - The inheritance (specificity) graph is: + * Every command inherits from `common` + * The following commands inherit from (and are more specific than) + `build`: `test`, `run`, `clean`, `mobile-install`, `info`, + `print_action`, `config`, `cquery`, and `aquery` + * `coverage`, `fetch`, and `vendor` inherit from `test` - - Every command inherits from `common` - - The following commands inherit from (and are more specific than) `build`: `test`, `run`, `clean`, `mobile-install`, `info`, `print_action`, `config`, `cquery`, and `aquery` - - `coverage`, `fetch`, and `vendor` inherit from `test` +- Two lines specifying options for the same command at equal specificity are + parsed in the order in which they appear within the file. -- Two lines specifying options for the same command at equal specificity are parsed in the order in which they appear within the file. +- Because this precedence rule does not match the file order, it helps + readability if you follow the precedence order within rc files: start with + `common` options at the top, and end with the most-specific commands at the + bottom of the file. This way, the order in which the options are read is the + same as the order in which they are applied, which is more intuitive. -- Because this precedence rule does not match the file order, it helps readability if you follow the precedence order within rc files: start with `common` options at the top, and end with the most-specific commands at the bottom of the file. This way, the order in which the options are read is the same as the order in which they are applied, which is more intuitive. - -The arguments specified on a line of an rc file may include arguments that are not options, such as the names of build targets, and so on. These, like the options specified in the same files, have lower precedence than their siblings on the command line, and are always prepended to the explicit list of non- option arguments. +The arguments specified on a line of an rc file may include arguments that are +not options, such as the names of build targets, and so on. These, like the +options specified in the same files, have lower precedence than their siblings +on the command line, and are always prepended to the explicit list of non- +option arguments. #### `--config` -In addition to setting option defaults, the rc file can be used to group options and provide a shorthand for common groupings. This is done by adding a `:name` suffix to the command. These options are ignored by default, but will be included when the option `--config=name` is present, either on the command line or in a `.bazelrc` file, recursively, even inside of another config definition. The options specified by `command:name` will only be expanded for applicable commands, in the precedence order described above. - -Note: Configs can be defined in any `.bazelrc` file, and that all lines of the form `command:name` (for applicable commands) will be expanded, across the different rc files. In order to avoid name conflicts, we suggest that configs defined in personal rc files start with an underscore (`_`) to avoid unintentional name sharing. - -`--config=foo` expands to the options defined in [the rc files](#bazelrc-file-locations) "in-place" so that the options specified for the config have the same precedence that the `--config=foo` option had. - -This syntax does not extend to the use of `startup` to set [startup options](#option-defaults). Setting `startup:config-name --some_startup_option` in the .bazelrc will be ignored. +In addition to setting option defaults, the rc file can be used to group options +and provide a shorthand for common groupings. This is done by adding a `:name` +suffix to the command. These options are ignored by default, but will be +included when the option --config=name is present, +either on the command line or in a `.bazelrc` file, recursively, even inside of +another config definition. The options specified by `command:name` will only be +expanded for applicable commands, in the precedence order described above. + +Note: Configs can be defined in any `.bazelrc` file, and that all lines of +the form `command:name` (for applicable commands) will be expanded, across the +different rc files. In order to avoid name conflicts, we suggest that configs +defined in personal rc files start with an underscore (`_`) to avoid +unintentional name sharing. + +`--config=foo` expands to the options defined in +[the rc files](#bazelrc-file-locations) "in-place" so that the options +specified for the config have the same precedence that the `--config=foo` option +had. + +This syntax does not extend to the use of `startup` to set +[startup options](#option-defaults). Setting +`startup:config-name --some_startup_option` in the .bazelrc will be ignored. #### `--enable_platform_specific_config` -In the `.bazelrc` you can use platform specific configs that will be automatically enabled based on the host OS. For example, if the host OS is Linux and the `build` command is run, the `build:linux` configuration will be automatically enabled. Supported OS identifiers are `linux`, `macos`, `windows`, `freebsd`, and `openbsd`. +In the `.bazelrc` you can use platform specific configs that will be +automatically enabled based on the host OS. For example, if the host OS +is Linux and the `build` command is run, the `build:linux` configuration +will be automatically enabled. Supported OS identifiers are `linux`, +`macos`, `windows`, `freebsd`, and `openbsd`. -This is equivalent to using `--config=linux` on Linux, `--config=windows` on Windows, and so on. This can be disabled with `--enable_platform_specific_config=false`. +This is equivalent to using `--config=linux` on Linux, +`--config=windows` on Windows, and so on. This can be disabled with +`--enable_platform_specific_config=false`. -See [--enable\_platform\_specific\_config](/reference/command-line-reference#flag--enable_platform_specific_config). +See [--enable_platform_specific_config](/reference/command-line-reference#flag--enable_platform_specific_config). #### Example @@ -163,16 +247,27 @@ build:memcheck --strip=never --test_timeout=3600 #### `.bazelignore` -You can specify directories within the workspace that you want Bazel to ignore, such as related projects that use other build systems. Place a file called `.bazelignore` at the root of the workspace and add the directories you want Bazel to ignore, one per line. Entries are relative to the workspace root. +You can specify directories within the workspace +that you want Bazel to ignore, such as related projects +that use other build systems. Place a file called +`.bazelignore` at the root of the workspace +and add the directories you want Bazel to ignore, one per +line. Entries are relative to the workspace root. -The `.bazelignore` file does not permit glob semantics. Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. See [#24203](https://github.com/bazelbuild/bazel/pull/24203). +The `.bazelignore` file does not permit glob semantics. +Bazel 8 introduces the `REPO.bazel` file which allows another directive, `ignore_directories()`. +It takes a list of directories to ignore just like .bazelignore does, but with glob semantics. +See [#24203](https://github.com/bazelbuild/bazel/pull/24203). ### The global bazelrc file Bazel reads optional bazelrc files in this order: -1. System rc-file located at `/etc/bazel.bazelrc`. -2. Workspace rc-file located at `$workspace/tools/bazel.rc`. -3. Home rc-file located at `$HOME/.bazelrc` +1. System rc-file located at `/etc/bazel.bazelrc`. +2. Workspace rc-file located at `$workspace/tools/bazel.rc`. +3. Home rc-file located at `$HOME/.bazelrc` -Each bazelrc file listed here has a corresponding flag which can be used to disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` startup option. +Each bazelrc file listed here has a corresponding flag which can be used to +disable them (e.g. `--nosystem_rc`, `--noworkspace_rc`, `--nohome_rc`). You can +also make Bazel ignore all bazelrcs by passing the `--ignore_all_rc_files` +startup option. diff --git a/run/build.mdx b/run/build.mdx deleted file mode 100644 index 30dca42c..00000000 --- a/run/build.mdx +++ /dev/null @@ -1,369 +0,0 @@ ---- -title: 'Build programs with Bazel' ---- - -This page covers how to build a program with Bazel, build command syntax, and target pattern syntax. - -## Quickstart - -To run Bazel, go to your base [workspace](/concepts/build-ref#workspace) directory or any of its subdirectories and type `bazel`. See [build](#bazel-build) if you need to make a new workspace. - -```posix-terminal -bazel help - [Bazel release bazel <var>version</var>] -Usage: bazel <var>command</var> <var>options</var> ... -``` - -### Available commands - -- [`analyze-profile`](/docs/user-manual#analyze-profile): Analyzes build profile data. -- [`aquery`](/docs/user-manual#aquery): Executes a query on the [post-analysis](#analysis) action graph. -- [`build`](#bazel-build): Builds the specified targets. -- [`canonicalize-flags`](/docs/user-manual#canonicalize-flags): Canonicalize Bazel flags. -- [`clean`](/docs/user-manual#clean): Removes output files and optionally stops the server. -- [`cquery`](/query/cquery): Executes a [post-analysis](#analysis) dependency graph query. -- [`dump`](/docs/user-manual#dump): Dumps the internal state of the Bazel server process. -- [`help`](/docs/user-manual#help): Prints help for commands, or the index. -- [`info`](/docs/user-manual#info): Displays runtime info about the bazel server. -- [`fetch`](#fetching-external-dependencies): Fetches all external dependencies of a target. -- [`mobile-install`](/docs/user-manual#mobile-install): Installs apps on mobile devices. -- [`query`](/query/guide): Executes a dependency graph query. -- [`run`](/docs/user-manual#running-executables): Runs the specified target. -- [`shutdown`](/docs/user-manual#shutdown): Stops the Bazel server. -- [`test`](/docs/user-manual#running-tests): Builds and runs the specified test targets. -- [`version`](/docs/user-manual#version): Prints version information for Bazel. - -### Getting help - -- `bazel help <var>command</var>`: Prints help and options for `<var>command</var>`. -- `bazel help `[`startup_options`](/docs/user-manual#startup-options): Options for the JVM hosting Bazel. -- `bazel help `[`target-syntax`](#specifying-build-targets): Explains the syntax for specifying targets. -- `bazel help info-keys`: Displays a list of keys used by the info command. - -The `bazel` tool performs many functions, called commands. The most commonly used ones are `bazel build` and `bazel test`. You can browse the online help messages using `bazel help`. - -### Building one target - -Before you can start a build, you need a *workspace*. A workspace is a directory tree that contains all the source files needed to build your application. Bazel allows you to perform a build from a completely read-only volume. - -To build a program with Bazel, type `bazel build` followed by the [target](#specifying-build-targets) you want to build. - -```posix-terminal -bazel build //foo -``` - -After issuing the command to build `//foo`, you'll see output similar to this: - -``` -INFO: Analyzed target //foo:foo (14 packages loaded, 48 targets configured). -INFO: Found 1 target... -Target //foo:foo up-to-date: - bazel-bin/foo/foo -INFO: Elapsed time: 9.905s, Critical Path: 3.25s -INFO: Build completed successfully, 6 total actions -``` - -First, Bazel **loads** all packages in your target's dependency graph. This includes *declared dependencies*, files listed directly in the target's `BUILD` file, and *transitive dependencies*, files listed in the `BUILD` files of your target's dependencies. After identifying all dependencies, Bazel **analyzes** them for correctness and creates the *build actions*. Last, Bazel **executes** the compilers and other tools of the build. - -During the build's execution phase, Bazel prints progress messages. The progress messages include the current build step (such as, compiler or linker) as it starts, and the number completed over the total number of build actions. As the build starts, the number of total actions often increases as Bazel discovers the entire action graph, but the number stabilizes within a few seconds. - -At the end of the build, Bazel prints which targets were requested, whether or not they were successfully built, and if so, where the output files can be found. Scripts that run builds can reliably parse this output; see [`--show_result`](/docs/user-manual#show-result) for more details. - -If you type the same command again, the build finishes much faster. - -```posix-terminal -bazel build //foo -INFO: Analyzed target //foo:foo (0 packages loaded, 0 targets configured). -INFO: Found 1 target... -Target //foo:foo up-to-date: - bazel-bin/foo/foo -INFO: Elapsed time: 0.144s, Critical Path: 0.00s -INFO: Build completed successfully, 1 total action -``` - -This is a *null build*. Because nothing changed, there are no packages to reload and no build steps to execute. If something changed in 'foo' or its dependencies, Bazel would re-execute some build actions, or complete an *incremental build*. - -### Building multiple targets - -Bazel allows a number of ways to specify the targets to be built. Collectively, these are known as *target patterns*. This syntax is used in commands like `build`, `test`, or `query`. - -Whereas [labels](/concepts/labels) are used to specify individual targets, such as for declaring dependencies in `BUILD` files, Bazel's target patterns specify multiple targets. Target patterns are a generalization of the label syntax for *sets* of targets, using wildcards. In the simplest case, any valid label is also a valid target pattern, identifying a set of exactly one target. - -All target patterns starting with `//` are resolved relative to the current workspace. - -| | | -| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- | -| `//foo/bar:wiz` | Just the single target `//foo/bar:wiz`. | -| `//foo/bar` | Equivalent to `//foo/bar:bar`. | -| `//foo/bar:all` | All rule targets in the package `foo/bar`. | -| `//foo/...` | All rule targets in all packages beneath the directory `foo`. | -| `//foo/...:all` | All rule targets in all packages beneath the directory `foo`. | -| `//foo/...:*` | All targets (rules and files) in all packages beneath the directory `foo`. | -| `//foo/...:all-targets` | All targets (rules and files) in all packages beneath the directory `foo`. | -| `//...` | All rule targets in packages in the main repository. Does not include targets from [external repositories](/docs/external). | -| `//:all` | All rule targets in the top-level package, if there is a \`BUILD\` file at the root of the workspace. | - -Target patterns that do not begin with `//` are resolved relative to the current *working directory*. These examples assume a working directory of `foo`: - -| | | -| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | -| `:foo` | Equivalent to `//foo:foo`. | -| `bar:wiz` | Equivalent to `//foo/bar:wiz`. | -| `bar/wiz` | Equivalent to:* `//foo/bar/wiz:wiz` if `foo/bar/wiz` is a package -* `//foo/bar:wiz` if `foo/bar` is a package -* `//foo:bar/wiz` otherwise | -| `bar:all` | Equivalent to `//foo/bar:all`. | -| `:all` | Equivalent to `//foo:all`. | -| `...:all` | Equivalent to `//foo/...:all`. | -| `...` | Equivalent to `//foo/...:all`. | -| `bar/...:all` | Equivalent to `//foo/bar/...:all`. | - -By default, directory symlinks are followed for recursive target patterns, except those that point to under the output base, such as the convenience symlinks that are created in the root directory of the workspace. - -In addition, Bazel does not follow symlinks when evaluating recursive target patterns in any directory that contains a file named as follows: `DONT_FOLLOW_SYMLINKS_WHEN_TRAVERSING_THIS_DIRECTORY_VIA_A_RECURSIVE_TARGET_PATTERN` - -`foo/...` is a wildcard over *packages*, indicating all packages recursively beneath directory `foo` (for all roots of the package path). `:all` is a wildcard over *targets*, matching all rules within a package. These two may be combined, as in `foo/...:all`, and when both wildcards are used, this may be abbreviated to `foo/...`. - -In addition, `:*` (or `:all-targets`) is a wildcard that matches *every target* in the matched packages, including files that aren't normally built by any rule, such as `_deploy.jar` files associated with `java_binary` rules. - -This implies that `:*` denotes a *superset* of `:all`; while potentially confusing, this syntax does allow the familiar `:all` wildcard to be used for typical builds, where building targets like the `_deploy.jar` is not desired. - -In addition, Bazel allows a slash to be used instead of the colon required by the label syntax; this is often convenient when using Bash filename expansion. For example, `foo/bar/wiz` is equivalent to `//foo/bar:wiz` (if there is a package `foo/bar`) or to `//foo:bar/wiz` (if there is a package `foo`). - -Many Bazel commands accept a list of target patterns as arguments, and they all honor the prefix negation operator `-`. This can be used to subtract a set of targets from the set specified by the preceding arguments. Note that this means order matters. For example, - -```posix-terminal -bazel build foo/... bar/... -``` - -means "build all targets beneath `foo` *and* all targets beneath `bar`", whereas - -```posix-terminal -bazel build -- foo/... -foo/bar/... -``` - -means "build all targets beneath `foo` *except* those beneath `foo/bar`". (The `--` argument is required to prevent the subsequent arguments starting with `-` from being interpreted as additional options.) - -It's important to point out though that subtracting targets this way will not guarantee that they are not built, since they may be dependencies of targets that weren't subtracted. For example, if there were a target `//foo:all-apis` that among others depended on `//foo/bar:api`, then the latter would be built as part of building the former. - -Targets with `tags = ["manual"]` are not included in wildcard target patterns (`...`, `:*`, `:all`, etc.) when specified in commands like `bazel build` and `bazel test` (but they are included in negative wildcard target patterns, that is they will be subtracted). You should specify such test targets with explicit target patterns on the command line if you want Bazel to build/test them. In contrast, `bazel query` doesn't perform any such filtering automatically (that would defeat the purpose of `bazel query`). - -### Fetching external dependencies - -By default, Bazel will download and symlink external dependencies during the build. However, this can be undesirable, either because you'd like to know when new external dependencies are added or because you'd like to "prefetch" dependencies (say, before a flight where you'll be offline). If you would like to prevent new dependencies from being added during builds, you can specify the `--fetch=false` flag. Note that this flag only applies to repository rules that do not point to a directory in the local file system. Changes, for example, to `local_repository`, `new_local_repository` and Android SDK and NDK repository rules will always take effect regardless of the value `--fetch` . - -If you disallow fetching during builds and Bazel finds new external dependencies, your build will fail. - -You can manually fetch dependencies by running `bazel fetch`. If you disallow during-build fetching, you'll need to run `bazel fetch`: - -- Before you build for the first time. -- After you add a new external dependency. - -Once it has been run, you should not need to run it again until the MODULE.bazel file changes. - -`fetch` takes a list of targets to fetch dependencies for. For example, this would fetch dependencies needed to build `//foo:bar` and `//bar:baz`: - -```posix-terminal -bazel fetch //foo:bar //bar:baz -``` - -To fetch all external dependencies for a workspace, run: - -```posix-terminal -bazel fetch //... -``` - -With Bazel 7 or later, if you have Bzlmod enabled, you can also fetch all external dependencies by running - -```posix-terminal -bazel fetch -``` - -You do not need to run bazel fetch at all if you have all of the tools you are using (from library jars to the JDK itself) under your workspace root. However, if you're using anything outside of the workspace directory then Bazel will automatically run `bazel fetch` before running `bazel build`. - -#### The repository cache - -Bazel tries to avoid fetching the same file several times, even if the same file is needed in different workspaces, or if the definition of an external repository changed but it still needs the same file to download. To do so, bazel caches all files downloaded in the repository cache which, by default, is located at `~/.cache/bazel/_bazel_$USER/cache/repos/v1/`. The location can be changed by the `--repository_cache` option. The cache is shared between all workspaces and installed versions of bazel. An entry is taken from the cache if Bazel knows for sure that it has a copy of the correct file, that is, if the download request has a SHA256 sum of the file specified and a file with that hash is in the cache. So specifying a hash for each external file is not only a good idea from a security perspective; it also helps avoiding unnecessary downloads. - -Upon each cache hit, the modification time of the file in the cache is updated. In this way, the last use of a file in the cache directory can easily be determined, for example to manually clean up the cache. The cache is never cleaned up automatically, as it might contain a copy of a file that is no longer available upstream. - -#### \[Deprecated] Distribution files directories - -**Deprecated**: *Using repository cache is preferred to achieve offline build.* - -The distribution directory is another Bazel mechanism to avoid unnecessary downloads. Bazel searches distribution directories before the repository cache. The primary difference is that the distribution directory requires manual preparation. - -Using the [`--distdir=/path/to-directory`](/reference/command-line-reference#flag--distdir) option, you can specify additional read-only directories to look for files instead of fetching them. A file is taken from such a directory if the file name is equal to the base name of the URL and additionally the hash of the file is equal to the one specified in the download request. This only works if the file hash is specified in the repo rule declaration. - -While the condition on the file name is not necessary for correctness, it reduces the number of candidate files to one per specified directory. In this way, specifying distribution files directories remains efficient, even if the number of files in such a directory grows large. - -#### Running Bazel in an airgapped environment - -To keep Bazel's binary size small, Bazel's implicit dependencies are fetched over the network while running for the first time. These implicit dependencies contain toolchains and rules that may not be necessary for everyone. For example, Android tools are unbundled and fetched only when building Android projects. - -However, these implicit dependencies may cause problems when running Bazel in an airgapped environment, even if you have vendored all of your external dependencies. To solve that, you can prepare a repository cache (with Bazel 7 or later) or distribution directory (with Bazel prior to 7) containing these dependencies on a machine with network access, and then transfer them to the airgapped environment with an offline approach. - -##### Repository cache (with Bazel 7 or later) - -To prepare the [repository cache](#repository-cache), use the [`--repository_cache`](/reference/command-line-reference#flag--repository_cache) flag. You will need to do this once for every new Bazel binary version, since the implicit dependencies can be different for every release. - -To fetch those dependencies outside of your airgapped environment, first create an empty workspace: - -```posix-terminal -mkdir empty_workspace && cd empty_workspace - -touch MODULE.bazel -``` - -To fetch built-in Bzlmod dependencies, run - -```posix-terminal -bazel fetch --repository_cache="path/to/repository/cache" -``` - -If you still rely on the legacy WORKSPACE file, to fetch built-in WORKSPACE dependencies, run - -```posix-terminal -bazel sync --repository_cache="path/to/repository/cache" -``` - -Finally, when you use Bazel in your airgapped environment, pass the same `--repository_cache` flag. For convenience, you can add it as an `.bazelrc` entry: - -```python -common --repository_cache="path/to/repository/cache" -``` - -In addition, you may also need to clone the [BCR](https://github.com/bazelbuild/bazel-central-registry) locally and use `--registry` flag to point your local copy to prevent Bazel from accessing the BCR through internet. Add the following line to your `.bazelrc`: - -```python -common --registry="path/to/local/bcr/registry" -``` - -##### Distribution directory (with Bazel prior to 7) - -To prepare the [distribution directory](#distribution-directory), use the [`--distdir`](/reference/command-line-reference#flag--distdir) flag. You will need to do this once for every new Bazel binary version, since the implicit dependencies can be different for every release. - -To build these dependencies outside of your airgapped environment, first checkout the Bazel source tree at the right version: - -```posix-terminal -git clone https://github.com/bazelbuild/bazel "$BAZEL_DIR" - -cd "$BAZEL_DIR" - -git checkout "$BAZEL_VERSION" -``` - -Then, build the tarball containing the implicit runtime dependencies for that specific Bazel version: - -```posix-terminal -bazel build @additional_distfiles//:archives.tar -``` - -Export this tarball to a directory that can be copied into your airgapped environment. Note the `--strip-components` flag, because `--distdir` can be quite finicky with the directory nesting level: - -```posix-terminal -tar xvf bazel-bin/external/additional_distfiles/archives.tar \ - -C "$NEW_DIRECTORY" --strip-components=3 -``` - -Finally, when you use Bazel in your airgapped environment, pass the `--distdir` flag pointing to the directory. For convenience, you can add it as an `.bazelrc` entry: - -```posix-terminal -build --distdir=<var>path</var>/to/<var>directory</var> -``` - -### Build configurations and cross-compilation - -All the inputs that specify the behavior and result of a given build can be divided into two distinct categories. The first kind is the intrinsic information stored in the `BUILD` files of your project: the build rule, the values of its attributes, and the complete set of its transitive dependencies. The second kind is the external or environmental data, supplied by the user or by the build tool: the choice of target architecture, compilation and linking options, and other toolchain configuration options. We refer to a complete set of environmental data as a **configuration**. - -In any given build, there may be more than one configuration. Consider a cross-compile, in which you build a `//foo:bin` executable for a 64-bit architecture, but your workstation is a 32-bit machine. Clearly, the build will require building `//foo:bin` using a toolchain capable of creating 64-bit executables, but the build system must also build various tools used during the build itself—for example tools that are built from source, then subsequently used in, say, a genrule—and these must be built to run on your workstation. Thus we can identify two configurations: the **exec configuration**, which is used for building tools that run during the build, and the **target configuration** (or *request configuration*, but we say "target configuration" more often even though that word already has many meanings), which is used for building the binary you ultimately requested. - -Typically, there are many libraries that are prerequisites of both the requested build target (`//foo:bin`) and one or more of the exec tools, for example some base libraries. Such libraries must be built twice, once for the exec configuration, and once for the target configuration. Bazel takes care of ensuring that both variants are built, and that the derived files are kept separate to avoid interference; usually such targets can be built concurrently, since they are independent of each other. If you see progress messages indicating that a given target is being built twice, this is most likely the explanation. - -The exec configuration is derived from the target configuration as follows: - -- Use the same version of Crosstool (`--crosstool_top`) as specified in the request configuration, unless `--host_crosstool_top` is specified. -- Use the value of `--host_cpu` for `--cpu` (default: `k8`). -- Use the same values of these options as specified in the request configuration: `--compiler`, `--use_ijars`, and if `--host_crosstool_top` is used, then the value of `--host_cpu` is used to look up a `default_toolchain` in the Crosstool (ignoring `--compiler`) for the exec configuration. -- Use the value of `--host_javabase` for `--javabase` -- Use the value of `--host_java_toolchain` for `--java_toolchain` -- Use optimized builds for C++ code (`-c opt`). -- Generate no debugging information (`--copt=-g0`). -- Strip debug information from executables and shared libraries (`--strip=always`). -- Place all derived files in a special location, distinct from that used by any possible request configuration. -- Suppress stamping of binaries with build data (see `--embed_*` options). -- All other values remain at their defaults. - -There are many reasons why it might be preferable to select a distinct exec configuration from the request configuration. Most importantly: - -Firstly, by using stripped, optimized binaries, you reduce the time spent linking and executing the tools, the disk space occupied by the tools, and the network I/O time in distributed builds. - -Secondly, by decoupling the exec and request configurations in all builds, you avoid very expensive rebuilds that would result from minor changes to the request configuration (such as changing a linker options does), as described earlier. - -### Correct incremental rebuilds - -One of the primary goals of the Bazel project is to ensure correct incremental rebuilds. Previous build tools, especially those based on Make, make several unsound assumptions in their implementation of incremental builds. - -Firstly, that timestamps of files increase monotonically. While this is the typical case, it is very easy to fall afoul of this assumption; syncing to an earlier revision of a file causes that file's modification time to decrease; Make-based systems will not rebuild. - -More generally, while Make detects changes to files, it does not detect changes to commands. If you alter the options passed to the compiler in a given build step, Make will not re-run the compiler, and it is necessary to manually discard the invalid outputs of the previous build using `make clean`. - -Also, Make is not robust against the unsuccessful termination of one of its subprocesses after that subprocess has started writing to its output file. While the current execution of Make will fail, the subsequent invocation of Make will blindly assume that the truncated output file is valid (because it is newer than its inputs), and it will not be rebuilt. Similarly, if the Make process is killed, a similar situation can occur. - -Bazel avoids these assumptions, and others. Bazel maintains a database of all work previously done, and will only omit a build step if it finds that the set of input files (and their timestamps) to that build step, and the compilation command for that build step, exactly match one in the database, and, that the set of output files (and their timestamps) for the database entry exactly match the timestamps of the files on disk. Any change to the input files or output files, or to the command itself, will cause re-execution of the build step. - -The benefit to users of correct incremental builds is: less time wasted due to confusion. (Also, less time spent waiting for rebuilds caused by use of `make clean`, whether necessary or pre-emptive.) - -#### Build consistency and incremental builds - -Formally, we define the state of a build as *consistent* when all the expected output files exist, and their contents are correct, as specified by the steps or rules required to create them. When you edit a source file, the state of the build is said to be *inconsistent*, and remains inconsistent until you next run the build tool to successful completion. We describe this situation as *unstable inconsistency*, because it is only temporary, and consistency is restored by running the build tool. - -There is another kind of inconsistency that is pernicious: *stable inconsistency*. If the build reaches a stable inconsistent state, then repeated successful invocation of the build tool does not restore consistency: the build has gotten "stuck", and the outputs remain incorrect. Stable inconsistent states are the main reason why users of Make (and other build tools) type `make clean`. Discovering that the build tool has failed in this manner (and then recovering from it) can be time consuming and very frustrating. - -Conceptually, the simplest way to achieve a consistent build is to throw away all the previous build outputs and start again: make every build a clean build. This approach is obviously too time-consuming to be practical (except perhaps for release engineers), and therefore to be useful, the build tool must be able to perform incremental builds without compromising consistency. - -Correct incremental dependency analysis is hard, and as described above, many other build tools do a poor job of avoiding stable inconsistent states during incremental builds. In contrast, Bazel offers the following guarantee: after a successful invocation of the build tool during which you made no edits, the build will be in a consistent state. (If you edit your source files during a build, Bazel makes no guarantee about the consistency of the result of the current build. But it does guarantee that the results of the *next* build will restore consistency.) - -As with all guarantees, there comes some fine print: there are some known ways of getting into a stable inconsistent state with Bazel. We won't guarantee to investigate such problems arising from deliberate attempts to find bugs in the incremental dependency analysis, but we will investigate and do our best to fix all stable inconsistent states arising from normal or "reasonable" use of the build tool. - -If you ever detect a stable inconsistent state with Bazel, please report a bug. - -#### Sandboxed execution - -Note: Sandboxing is enabled by default for local execution. - -Bazel uses sandboxes to guarantee that actions run hermetically and correctly. Bazel runs *spawns* (loosely speaking: actions) in sandboxes that only contain the minimal set of files the tool requires to do its job. Currently sandboxing works on Linux 3.12 or newer with the `CONFIG_USER_NS` option enabled, and also on macOS 10.11 or newer. - -Bazel will print a warning if your system does not support sandboxing to alert you to the fact that builds are not guaranteed to be hermetic and might affect the host system in unknown ways. To disable this warning you can pass the `--ignore_unsupported_sandboxing` flag to Bazel. - -Note: Hermeticity means that the action only uses its declared input files and no other files in the filesystem, and it only produces its declared output files. See [Hermeticity](/basics/hermeticity) for more details. - -On some platforms such as [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster nodes or Debian, user namespaces are deactivated by default due to security concerns. This can be checked by looking at the file `/proc/sys/kernel/unprivileged_userns_clone`: if it exists and contains a 0, then user namespaces can be activated with `sudo sysctl kernel.unprivileged_userns_clone=1`. - -In some cases, the Bazel sandbox fails to execute rules because of the system setup. The symptom is generally a failure that output a message similar to `namespace-sandbox.c:633: execvp(argv[0], argv): No such file or directory`. In that case, try to deactivate the sandbox for genrules with `--strategy=Genrule=standalone` and for other rules with `--spawn_strategy=standalone`. Also please report a bug on our issue tracker and mention which Linux distribution you're using so that we can investigate and provide a fix in a subsequent release. - -### Phases of a build - -In Bazel, a build occurs in three distinct phases; as a user, understanding the difference between them provides insight into the options which control a build (see below). - -#### Loading phase - -The first is **loading** during which all the necessary BUILD files for the initial targets, and their transitive closure of dependencies, are loaded, parsed, evaluated and cached. - -For the first build after a Bazel server is started, the loading phase typically takes many seconds as many BUILD files are loaded from the file system. In subsequent builds, especially if no BUILD files have changed, loading occurs very quickly. - -Errors reported during this phase include: package not found, target not found, lexical and grammatical errors in a BUILD file, and evaluation errors. - -#### Analysis phase - -The second phase, **analysis**, involves the semantic analysis and validation of each build rule, the construction of a build dependency graph, and the determination of exactly what work is to be done in each step of the build. - -Like loading, analysis also takes several seconds when computed in its entirety. However, Bazel caches the dependency graph from one build to the next and only reanalyzes what it has to, which can make incremental builds extremely fast in the case where the packages haven't changed since the previous build. - -Errors reported at this stage include: inappropriate dependencies, invalid inputs to a rule, and all rule-specific error messages. - -The loading and analysis phases are fast because Bazel avoids unnecessary file I/O at this stage, reading only BUILD files in order to determine the work to be done. This is by design, and makes Bazel a good foundation for analysis tools, such as Bazel's [query](/query/guide) command, which is implemented atop the loading phase. - -#### Execution phase - -The third and final phase of the build is **execution**. This phase ensures that the outputs of each step in the build are consistent with its inputs, re-running compilation/linking/etc. tools as necessary. This step is where the build spends the majority of its time, ranging from a few seconds to over an hour for a large build. Errors reported during this phase include: missing source files, errors in a tool executed by some build action, or failure of a tool to produce the expected set of outputs. diff --git a/run/client-server.mdx b/run/client-server.mdx index b335b0ca..1868635f 100644 --- a/run/client-server.mdx +++ b/run/client-server.mdx @@ -2,21 +2,57 @@ title: 'Client/server implementation' --- -The Bazel system is implemented as a long-lived server process. This allows it to perform many optimizations not possible with a batch-oriented implementation, such as caching of BUILD files, dependency graphs, and other metadata from one build to the next. This improves the speed of incremental builds, and allows different commands, such as `build` and `query` to share the same cache of loaded packages, making queries very fast. Each server can handle at most one invocation at a time; further concurrent invocations will either block or fail-fast (see `--block_for_lock`). -When you run `bazel`, you're running the client. The client finds the server based on the [output base](/run/scripts#output-base-option), which by default is determined by the path of the base workspace directory and your userid, so if you build in multiple workspaces, you'll have multiple output bases and thus multiple Bazel server processes. Multiple users on the same workstation can build concurrently in the same workspace because their output bases will differ (different userids). -If the client cannot find a running server instance, it starts a new one. It does this by checking if the output base already exists, implying the blaze archive has already been unpacked. Otherwise if the output base doesn't exist, the client unzips the archive's files and sets their `mtime`s to a date 9 years in the future. Once installed, the client confirms that the `mtime`s of the unzipped files are equal to the far off date to ensure no installation tampering has occurred. - -The server process will stop after a period of inactivity (3 hours, by default, which can be modified using the startup option `--max_idle_secs`). For the most part, the fact that there is a server running is invisible to the user, but sometimes it helps to bear this in mind. For example, if you're running scripts that perform a lot of automated builds in different directories, it's important to ensure that you don't accumulate a lot of idle servers; you can do this by explicitly shutting them down when you're finished with them, or by specifying a short timeout period. - -The name of a Bazel server process appears in the output of `ps x` or `ps -e f` as `bazel(dirname)`, where *dirname* is the basename of the directory enclosing the root of your workspace directory. For example: +The Bazel system is implemented as a long-lived server process. This allows it +to perform many optimizations not possible with a batch-oriented implementation, +such as caching of BUILD files, dependency graphs, and other metadata from one +build to the next. This improves the speed of incremental builds, and allows +different commands, such as `build` and `query` to share the same cache of +loaded packages, making queries very fast. Each server can handle at most one +invocation at a time; further concurrent invocations will either block or +fail-fast (see `--block_for_lock`). + +When you run `bazel`, you're running the client. The client finds the server +based on the [output base](/run/scripts#output-base-option), which by default is +determined by the path of the base workspace directory and your userid, so if +you build in multiple workspaces, you'll have multiple output bases and thus +multiple Bazel server processes. Multiple users on the same workstation can +build concurrently in the same workspace because their output bases will differ +(different userids). + +If the client cannot find a running server instance, it starts a new one. It +does this by checking if the output base already exists, implying the blaze +archive has already been unpacked. Otherwise if the output base doesn't exist, +the client unzips the archive's files and sets their `mtime`s to a date 9 years +in the future. Once installed, the client confirms that the `mtime`s of the +unzipped files are equal to the far off date to ensure no installation tampering +has occurred. + +The server process will stop after a period of inactivity (3 hours, by default, +which can be modified using the startup option `--max_idle_secs`). For the most +part, the fact that there is a server running is invisible to the user, but +sometimes it helps to bear this in mind. For example, if you're running scripts +that perform a lot of automated builds in different directories, it's important +to ensure that you don't accumulate a lot of idle servers; you can do this by +explicitly shutting them down when you're finished with them, or by specifying +a short timeout period. + +The name of a Bazel server process appears in the output of `ps x` or `ps -e f` +as bazel(dirname), where _dirname_ is the basename of the +directory enclosing the root of your workspace directory. For example: ```posix-terminal ps -e f 16143 ? Sl 3:00 bazel(src-johndoe2) -server -Djava.library.path=... ``` -This makes it easier to find out which server process belongs to a given workspace. (Beware that with certain other options to `ps`, Bazel server processes may be named just `java`.) Bazel servers can be stopped using the [shutdown](/docs/user-manual#shutdown) command. +This makes it easier to find out which server process belongs to a given +workspace. (Beware that with certain other options to `ps`, Bazel server +processes may be named just `java`.) Bazel servers can be stopped using the +[shutdown](/docs/user-manual#shutdown) command. -When running `bazel`, the client first checks that the server is the appropriate version; if not, the server is stopped and a new one started. This ensures that the use of a long-running server process doesn't interfere with proper versioning. +When running `bazel`, the client first checks that the server is the appropriate +version; if not, the server is stopped and a new one started. This ensures that +the use of a long-running server process doesn't interfere with proper +versioning. diff --git a/run/scripts.mdx b/run/scripts.mdx index 25cb1f65..f267c903 100644 --- a/run/scripts.mdx +++ b/run/scripts.mdx @@ -2,93 +2,129 @@ title: 'Calling Bazel from scripts' --- -You can call Bazel from scripts to perform a build, run tests, or query the dependency graph. Bazel has been designed to enable effective scripting, but this section lists some details to bear in mind to make your scripts more robust. -### Choosing the output base -The `--output_base` option controls where the Bazel process should write the outputs of a build to, as well as various working files used internally by Bazel, one of which is a lock that guards against concurrent mutation of the output base by multiple Bazel processes. +You can call Bazel from scripts to perform a build, run tests, or query +the dependency graph. Bazel has been designed to enable effective scripting, but +this section lists some details to bear in mind to make your scripts more +robust. -Choosing the correct output base directory for your script depends on several factors. If you need to put the build outputs in a specific location, this will dictate the output base you need to use. If you are making a "read only" call to Bazel (such as `bazel query`), the locking factors will be more important. In particular, if you need to run multiple instances of your script concurrently, you should be mindful that each Blaze server process can handle at most one invocation [at a time](/run/client-server#clientserver-implementation). Depending on your situation it may make sense for each instance of your script to wait its turn, or it may make sense to use `--output_base` to run multiple Blaze servers and use those. +### Choosing the output base -If you use the default output base value, you will be contending for the same lock used by the user's interactive Bazel commands. If the user issues long-running commands such as builds, your script will have to wait for those commands to complete before it can continue. +The `--output_base` option controls where the Bazel process should write the +outputs of a build to, as well as various working files used internally by +Bazel, one of which is a lock that guards against concurrent mutation of the +output base by multiple Bazel processes. + +Choosing the correct output base directory for your script depends on several +factors. If you need to put the build outputs in a specific location, this will +dictate the output base you need to use. If you are making a "read only" call to +Bazel (such as `bazel query`), the locking factors will be more important. In +particular, if you need to run multiple instances of your script concurrently, +you should be mindful that each Blaze server process can handle at most one +invocation [at a time](/run/client-server#clientserver-implementation). +Depending on your situation it may make sense for each instance of your script +to wait its turn, or it may make sense to use `--output_base` to run multiple +Blaze servers and use those. + +If you use the default output base value, you will be contending for the same +lock used by the user's interactive Bazel commands. If the user issues +long-running commands such as builds, your script will have to wait for those +commands to complete before it can continue. ### Notes about server mode -By default, Bazel uses a long-running [server process](/run/client-server) as an optimization. When running Bazel in a script, don't forget to call `shutdown` when you're finished with the server, or, specify `--max_idle_secs=5` so that idle servers shut themselves down promptly. +By default, Bazel uses a long-running [server process](/run/client-server) as an +optimization. When running Bazel in a script, don't forget to call `shutdown` +when you're finished with the server, or, specify `--max_idle_secs=5` so that +idle servers shut themselves down promptly. ### What exit code will I get? -Bazel attempts to differentiate failures due to the source code under consideration from external errors that prevent Bazel from executing properly. Bazel execution can result in following exit codes: +Bazel attempts to differentiate failures due to the source code under +consideration from external errors that prevent Bazel from executing properly. +Bazel execution can result in following exit codes: **Exit Codes common to all commands:** -- `0` - Success - -- `2` - Command Line Problem, Bad or Illegal flags or command combination, or Bad Environment Variables. Your command line must be modified. - -- `8` - Build Interrupted but we terminated with an orderly shutdown. - -- `9` - The server lock is held and `--noblock_for_lock` was passed. - -- `32` - External Environment Failure not on this machine. - -- `33` - Bazel ran out of memory and crashed. You need to modify your command line. - -- `34` - Reserved for Google-internal use. - -- `35` - Reserved for Google-internal use. - -- `36` - Local Environmental Issue, suspected permanent. - -- `37` - Unhandled Exception / Internal Bazel Error. - -- `38` - Transient error publishing results to the Build Event Service. - -- `39` - Blobs required by Bazel are evicted from Remote Cache. - -- `41-44` - Reserved for Google-internal use. - -- `45` - Persistent error publishing results to the Build Event Service. - -- `47` - Reserved for Google-internal use. - -- `49` - Reserved for Google-internal use. +- `0` - Success +- `2` - Command Line Problem, Bad or Illegal flags or command combination, or + Bad Environment Variables. Your command line must be modified. +- `8` - Build Interrupted but we terminated with an orderly shutdown. +- `9` - The server lock is held and `--noblock_for_lock` was passed. +- `32` - External Environment Failure not on this machine. + +- `33` - Bazel ran out of memory and crashed. You need to modify your command line. +- `34` - Reserved for Google-internal use. +- `35` - Reserved for Google-internal use. +- `36` - Local Environmental Issue, suspected permanent. +- `37` - Unhandled Exception / Internal Bazel Error. +- `38` - Transient error publishing results to the Build Event Service. +- `39` - Blobs required by Bazel are evicted from Remote Cache. +- `41-44` - Reserved for Google-internal use. +- `45` - Persistent error publishing results to the Build Event Service. +- `47` - Reserved for Google-internal use. +- `49` - Reserved for Google-internal use. **Return codes for commands `bazel build`, `bazel test`:** -- `1` - Build failed. -- `3` - Build OK, but some tests failed or timed out. -- `4` - Build successful but no tests were found even though testing was requested. +- `1` - Build failed. +- `3` - Build OK, but some tests failed or timed out. +- `4` - Build successful but no tests were found even though testing was + requested. + **For `bazel run`:** -- `1` - Build failed. -- If the build succeeds but the executed subprocess returns a non-zero exit code it will be the exit code of the command as well. +- `1` - Build failed. +- If the build succeeds but the executed subprocess returns a non-zero exit + code it will be the exit code of the command as well. **For `bazel query`:** -- `3` - Partial success, but the query encountered 1 or more errors in the input BUILD file set and therefore the results of the operation are not 100% reliable. This is likely due to a `--keep_going` option on the command line. -- `7` - Command failure. +- `3` - Partial success, but the query encountered 1 or more errors in the + input BUILD file set and therefore the results of the operation are not 100% + reliable. This is likely due to a `--keep_going` option on the command line. +- `7` - Command failure. + +Future Bazel versions may add additional exit codes, replacing generic failure +exit code `1` with a different non-zero value with a particular meaning. +However, all non-zero exit values will always constitute an error. -Future Bazel versions may add additional exit codes, replacing generic failure exit code `1` with a different non-zero value with a particular meaning. However, all non-zero exit values will always constitute an error. ### Reading the .bazelrc file -By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base workspace directory or the user's home directory. Whether or not this is desirable is a choice for your script; if your script needs to be perfectly hermetic (such as when doing release builds), you should disable reading the .bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform a build using the user's preferred settings, the default behavior is better. +By default, Bazel reads the [`.bazelrc` file](/run/bazelrc) from the base +workspace directory or the user's home directory. Whether or not this is +desirable is a choice for your script; if your script needs to be perfectly +hermetic (such as when doing release builds), you should disable reading the +.bazelrc file by using the option `--bazelrc=/dev/null`. If you want to perform +a build using the user's preferred settings, the default behavior is better. ### Command log -The Bazel output is also available in a command log file which you can find with the following command: +The Bazel output is also available in a command log file which you can find with +the following command: ```posix-terminal bazel info command_log ``` -The command log file contains the interleaved stdout and stderr streams of the most recent Bazel command. Note that running `bazel info` will overwrite the contents of this file, since it then becomes the most recent Bazel command. However, the location of the command log file will not change unless you change the setting of the `--output_base` or `--output_user_root` options. +The command log file contains the interleaved stdout and stderr streams of the +most recent Bazel command. Note that running `bazel info` will overwrite the +contents of this file, since it then becomes the most recent Bazel command. +However, the location of the command log file will not change unless you change +the setting of the `--output_base` or `--output_user_root` options. ### Parsing output -The Bazel output is quite easy to parse for many purposes. Two options that may be helpful for your script are `--noshow_progress` which suppresses progress messages, and `--show_result n`, which controls whether or not "build up-to-date" messages are printed; these messages may be parsed to discover which targets were successfully built, and the location of the output files they created. Be sure to specify a very large value of *n* if you rely on these messages. +The Bazel output is quite easy to parse for many purposes. Two options that may +be helpful for your script are `--noshow_progress` which suppresses progress +messages, and --show_result n, which controls whether or +not "build up-to-date" messages are printed; these messages may be parsed to +discover which targets were successfully built, and the location of the output +files they created. Be sure to specify a very large value of _n_ if you rely on +these messages. ## Troubleshooting performance by profiling diff --git a/start/android-app.mdx b/start/android-app.mdx index 3ae739fd..35da5828 100644 --- a/start/android-app.mdx +++ b/start/android-app.mdx @@ -2,22 +2,36 @@ title: 'Bazel Tutorial: Build an Android App' --- + +**Note:** There are known limitations on using Bazel for building Android apps. +Visit the [rules_android issues page](https://github.com/bazelbuild/rules_android/issues) +to see the list of known issues. While the Bazel team and Open Source Software +(OSS) contributors work actively to address known issues, users should be aware +that Android Studio does not officially support Bazel projects. + This tutorial covers how to build a simple Android app using Bazel. -Bazel supports building Android apps using the [Android rules](/reference/be/android). +Bazel supports building Android apps using the +[Android rules](/reference/be/android). -This tutorial is intended for Windows, macOS and Linux users and does not require experience with Bazel or Android app development. You do not need to write any Android code in this tutorial. +This tutorial is intended for Windows, macOS and Linux users and does not +require experience with Bazel or Android app development. You do not need to +write any Android code in this tutorial. ## What you'll learn In this tutorial you learn how to: -- Set up your environment by installing Bazel and Android Studio, and downloading the sample project. -- Set up a Bazel workspace that contains the source code for the app and a `MODULE.bazel` file that identifies the top level of the workspace directory. -- Update the `MODULE.bazel` file to contain references to the required external dependencies, like the Android SDK. -- Create a `BUILD` file. -- Build the app with Bazel. -- Deploy and run the app on an Android emulator or physical device. +* Set up your environment by installing Bazel and Android Studio, and + downloading the sample project. +* Set up a Bazel workspace that contains the source code + for the app and a `MODULE.bazel` file that identifies the top level of the + workspace directory. +* Update the `MODULE.bazel` file to contain references to the required + external dependencies, like the Android SDK. +* Create a `BUILD` file. +* Build the app with Bazel. +* Deploy and run the app on an Android emulator or physical device. ## Before you begin @@ -25,13 +39,16 @@ In this tutorial you learn how to: Before you begin the tutorial, install the following software: -- **Bazel.** To install, follow the [installation instructions](/install). -- **Android Studio.** To install, follow the steps to [download Android Studio](https://developer.android.com/sdk/index.html). Execute the setup wizard to download the SDK and configure your environment. -- (Optional) **Git.** Use `git` to download the Android app project. +* **Bazel.** To install, follow the [installation instructions](/install). +* **Android Studio.** To install, follow the steps to [download Android + Studio](https://developer.android.com/sdk/index.html). + Execute the setup wizard to download the SDK and configure your environment. +* (Optional) **Git.** Use `git` to download the Android app project. ### Get the sample project -For the sample project, use the tutorial Android app project in [Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). +For the sample project, use the tutorial Android app project in +[Bazel's examples repository](https://github.com/bazelbuild/examples/tree/main/android/tutorial). This app has a single button that prints a greeting when clicked: @@ -39,13 +56,15 @@ This app has a single button that prints a greeting when clicked: **Figure 1.** Android app button greeting. -Clone the repository with `git` (or [download the ZIP file directly](https://github.com/bazelbuild/examples/archive/master.zip)): +Clone the repository with `git` (or [download the ZIP file +directly](https://github.com/bazelbuild/examples/archive/master.zip)): ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in `examples/android/tutorial`. For the rest of the tutorial, you will be executing commands in this directory. +The sample project for this tutorial is in `examples/android/tutorial`. For +the rest of the tutorial, you will be executing commands in this directory. ### Review the source files @@ -80,20 +99,24 @@ The key files and directories are: | Android source files | `src/main/java/com/example/bazel/MainActivity.java` and `Greeter.java` | | Resource file directory | `src/main/java/com/example/bazel/res/` | + ## Build with Bazel ### Set up the workspace -A [workspace](/concepts/build-ref#workspace) is a directory that contains the source files for one or more software projects, and has a `MODULE.bazel` file at its root. +A [workspace](/concepts/build-ref#workspace) is a directory that contains the +source files for one or more software projects, and has a `MODULE.bazel` file at +its root. -The `MODULE.bazel` file may be empty or may contain references to [external dependencies](/external/overview) required to build your project. +The `MODULE.bazel` file may be empty or may contain references to [external +dependencies](/external/overview) required to build your project. First, run the following command to create an empty `MODULE.bazel` file: -| OS | Command | -| ------------------------ | -------------------------------------- | +| OS | Command | +| ------------------------ | ----------------------------------- | | Linux, macOS | `touch MODULE.bazel` | -| Windows (Command Prompt) | `type nul > MODULE.bazel` | +| Windows (Command Prompt) | `type nul > MODULE.bazel` | | Windows (PowerShell) | `New-Item MODULE.bazel -ItemType file` | ### Running Bazel @@ -104,7 +127,8 @@ You can now check if Bazel is running correctly with the command: bazel info workspace ``` -If Bazel prints the path of the current directory, you're good to go! If the `MODULE.bazel` file does not exist, you may see an error message like: +If Bazel prints the path of the current directory, you're good to go! If the +`MODULE.bazel` file does not exist, you may see an error message like: ``` ERROR: The 'info' command is only supported from within a workspace. @@ -112,7 +136,10 @@ ERROR: The 'info' command is only supported from within a workspace. ### Integrate with the Android SDK -Bazel needs to run the Android SDK [build tools](https://developer.android.com/tools/revisions/build-tools.html) to build the app. This means that you need to add some information to your `MODULE.bazel` file so that Bazel knows where to find them. +Bazel needs to run the Android SDK +[build tools](https://developer.android.com/tools/revisions/build-tools.html) +to build the app. This means that you need to add some information to your +`MODULE.bazel` file so that Bazel knows where to find them. Add the following line to your `MODULE.bazel` file: @@ -128,53 +155,94 @@ android_sdk_repository_extension = use_extension("@rules_android//rules/android_ use_repo(android_sdk_repository_extension, "androidsdk") ``` -This will use the Android SDK at the path referenced by the `ANDROID_HOME` environment variable, and automatically detect the highest API level and the latest version of build tools installed within that location. +This will use the Android SDK at the path referenced by the `ANDROID_HOME` +environment variable, and automatically detect the highest API level and the +latest version of build tools installed within that location. -You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find the path to the installed SDK using Android Studio's [SDK Manager](https://developer.android.com/studio/intro/update#sdk-manager). Assuming the SDK is installed to default locations, you can use the following commands to set the `ANDROID_HOME` variable: +You can set the `ANDROID_HOME` variable to the location of the Android SDK. Find +the path to the installed SDK using Android Studio's [SDK +Manager](https://developer.android.com/studio/intro/update#sdk-manager). +Assuming the SDK is installed to default locations, you can use the following +commands to set the `ANDROID_HOME` variable: -| OS | Command | +| OS | Command | | ------------------------ | --------------------------------------------------- | | Linux | `export ANDROID_HOME=$HOME/Android/Sdk/` | | macOS | `export ANDROID_HOME=$HOME/Library/Android/sdk` | | Windows (Command Prompt) | `set ANDROID_HOME=%LOCALAPPDATA%\Android\Sdk` | | Windows (PowerShell) | `$env:ANDROID_HOME="$env:LOCALAPPDATA\Android\Sdk"` | -The above commands set the variable only for the current shell session. To make them permanent, run the following commands: +The above commands set the variable only for the current shell session. To make +them permanent, run the following commands: -| OS | Command | -| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | -| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | -| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | +| OS | Command | +| ------------------------ | --------------------------------------------------- | +| Linux | `echo "export ANDROID_HOME=$HOME/Android/Sdk/" >> ~/.bashrc` | +| macOS | `echo "export ANDROID_HOME=$HOME/Library/Android/Sdk/" >> ~/.bashrc` | | Windows (Command Prompt) | `setx ANDROID_HOME "%LOCALAPPDATA%\Android\Sdk"` | | Windows (PowerShell) | `[System.Environment]::SetEnvironmentVariable('ANDROID_HOME', "$env:LOCALAPPDATA\Android\Sdk", [System.EnvironmentVariableTarget]::User)` | -**Optional:** If you want to compile native code into your Android app, you also need to download the [Android NDK](https://developer.android.com/ndk/downloads/index.html) and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: + +**Optional:** If you want to compile native code into your Android app, you +also need to download the [Android +NDK](https://developer.android.com/ndk/downloads/index.html) +and use `rules_android_ndk` by adding the following line to your `MODULE.bazel` file: ```python bazel_dep(name = "rules_android_ndk", version = "0.1.3") ``` -For more information, read [Using the Android Native Development Kit with Bazel](/docs/android-ndk). - -It's not necessary to set the API levels to the same value for the SDK and NDK. [This page](https://developer.android.com/ndk/guides/stable_apis.html) contains a map from Android releases to NDK-supported API levels. - -### Create a BUILD file - -A [`BUILD` file](/concepts/build-files) describes the relationship between a set of build outputs, like compiled Android resources from `aapt` or class files from `javac`, and their dependencies. These dependencies may be source files (Java, C++) in your workspace or other build outputs. `BUILD` files are written in a language called **Starlark**. - -`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. The package hierarchy is a logical structure that overlays the directory structure in your workspace. Each [package](/concepts/build-ref#packages) is a directory (and its subdirectories) that contains a related set of source files and a `BUILD` file. The package also includes any subdirectories, excluding those that contain their own `BUILD` file. The *package name* is the path to the `BUILD` file relative to the `MODULE.bazel` file. -Note that Bazel's package hierarchy is conceptually different from the Java package hierarchy of your Android App directory where the `BUILD` file is located, although the directories may be organized identically. +For more information, read [Using the Android Native Development Kit with +Bazel](/docs/android-ndk). -For the simple Android app in this tutorial, the source files in `src/main/` comprise a single Bazel package. A more complex project may have many nested packages. +It's not necessary to set the API levels to the same value for the SDK and NDK. +[This page](https://developer.android.com/ndk/guides/stable_apis.html) +contains a map from Android releases to NDK-supported API levels. -#### Add an android\_library rule - -A `BUILD` file contains several different types of declarations for Bazel. The most important type is the [build rule](/concepts/build-files#types-of-build-rules), which tells Bazel how to build an intermediate or final software output from a set of source files or other dependencies. Bazel provides two build rules, [`android_library`](/reference/be/android#android_library) and [`android_binary`](/reference/be/android#android_binary), that you can use to build an Android app. - -For this tutorial, you'll first use the `android_library` rule to tell Bazel to build an [Android library module](http://developer.android.com/tools/projects/index.html#LibraryProjects) from the app source code and resource files. You'll then use the `android_binary` rule to tell Bazel how to build the Android application package. +### Create a BUILD file -Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, and declare a new `android_library` target: +A [`BUILD` file](/concepts/build-files) describes the relationship +between a set of build outputs, like compiled Android resources from `aapt` or +class files from `javac`, and their dependencies. These dependencies may be +source files (Java, C++) in your workspace or other build outputs. `BUILD` files +are written in a language called **Starlark**. + +`BUILD` files are part of a concept in Bazel known as the *package hierarchy*. +The package hierarchy is a logical structure that overlays the directory +structure in your workspace. Each [package](/concepts/build-ref#packages) is a +directory (and its subdirectories) that contains a related set of source files +and a `BUILD` file. The package also includes any subdirectories, excluding +those that contain their own `BUILD` file. The *package name* is the path to the +`BUILD` file relative to the `MODULE.bazel` file. + +Note that Bazel's package hierarchy is conceptually different from the Java +package hierarchy of your Android App directory where the `BUILD` file is +located, although the directories may be organized identically. + +For the simple Android app in this tutorial, the source files in `src/main/` +comprise a single Bazel package. A more complex project may have many nested +packages. + +#### Add an android_library rule + +A `BUILD` file contains several different types of declarations for Bazel. The +most important type is the +[build rule](/concepts/build-files#types-of-build-rules), which tells +Bazel how to build an intermediate or final software output from a set of source +files or other dependencies. Bazel provides two build rules, +[`android_library`](/reference/be/android#android_library) and +[`android_binary`](/reference/be/android#android_binary), that you can use to +build an Android app. + +For this tutorial, you'll first use the +`android_library` rule to tell Bazel to build an [Android library +module](http://developer.android.com/tools/projects/index.html#LibraryProjects) +from the app source code and resource files. You'll then use the +`android_binary` rule to tell Bazel how to build the Android application package. + +Create a new `BUILD` file in the `src/main/java/com/example/bazel` directory, +and declare a new `android_library` target: `src/main/java/com/example/bazel/BUILD`: @@ -196,13 +264,18 @@ android_library( ) ``` -The `android_library` build rule contains a set of attributes that specify the information that Bazel needs to build a library module from the source files. Note also that the name of the rule is `greeter_activity`. You'll reference the rule using this name as a dependency in the `android_binary` rule. +The `android_library` build rule contains a set of attributes that specify the +information that Bazel needs to build a library module from the source files. +Note also that the name of the rule is `greeter_activity`. You'll reference the +rule using this name as a dependency in the `android_binary` rule. -#### Add an android\_binary rule +#### Add an android_binary rule -The [`android_binary`](/reference/be/android#android_binary) rule builds the Android application package (`.apk` file) for your app. +The [`android_binary`](/reference/be/android#android_binary) rule builds +the Android application package (`.apk` file) for your app. -Create a new `BUILD` file in the `src/main/` directory, and declare a new `android_binary` target: +Create a new `BUILD` file in the `src/main/` directory, +and declare a new `android_binary` target: `src/main/BUILD`: @@ -216,23 +289,35 @@ android_binary( ) ``` -Here, the `deps` attribute references the output of the `greeter_activity` rule you added to the `BUILD` file above. This means that when Bazel builds the output of this rule it checks first to see if the output of the `greeter_activity` library rule has been built and is up-to-date. If not, Bazel builds it and then uses that output to build the application package file. +Here, the `deps` attribute references the output of the `greeter_activity` rule +you added to the `BUILD` file above. This means that when Bazel builds the +output of this rule it checks first to see if the output of the +`greeter_activity` library rule has been built and is up-to-date. If not, Bazel +builds it and then uses that output to build the application package file. Now, save and close the file. ### Build the app -Try building the app! Run the following command to build the `android_binary` target: +Try building the app! Run the following command to build the +`android_binary` target: ```posix-terminal bazel build //src/main:app ``` -The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the target that follows. The target is specified as the name of a build rule inside a `BUILD` file, with along with the package path relative to your workspace directory. For this example, the target is `app` and the package path is `//src/main/`. +The [`build`](/docs/user-manual#build) subcommand instructs Bazel to build the +target that follows. The target is specified as the name of a build rule inside +a `BUILD` file, with along with the package path relative to your workspace +directory. For this example, the target is `app` and the package path is +`//src/main/`. -Note that you can sometimes omit the package path or target name, depending on your current working directory at the command line and the name of the target. For more details about target labels and paths, see [Labels](/concepts/labels). +Note that you can sometimes omit the package path or target name, depending on +your current working directory at the command line and the name of the target. +For more details about target labels and paths, see [Labels](/concepts/labels). -Bazel will start to build the sample app. During the build process, its output will appear similar to the following: +Bazel will start to build the sample app. During the build process, its output +will appear similar to the following: ```bash INFO: Analysed target //src/main:app (0 packages loaded, 0 targets configured). @@ -245,25 +330,40 @@ Target //src/main:app up-to-date: #### Locate the build outputs -Bazel puts the outputs of both intermediate and final build operations in a set of per-user, per-workspace output directories. These directories are symlinked from the following locations at the top-level of the project directory, where the `MODULE.bazel` file is: +Bazel puts the outputs of both intermediate and final build operations in a set +of per-user, per-workspace output directories. These directories are symlinked +from the following locations at the top-level of the project directory, where +the `MODULE.bazel` file is: -- `bazel-bin` stores binary executables and other runnable build outputs -- `bazel-genfiles` stores intermediary source files that are generated by Bazel rules -- `bazel-out` stores other types of build outputs +* `bazel-bin` stores binary executables and other runnable build outputs +* `bazel-genfiles` stores intermediary source files that are generated by + Bazel rules +* `bazel-out` stores other types of build outputs -Bazel stores the Android `.apk` file generated using the `android_binary` rule in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is derived from the name of the Bazel package. +Bazel stores the Android `.apk` file generated using the `android_binary` rule +in the `bazel-bin/src/main` directory, where the subdirectory name `src/main` is +derived from the name of the Bazel package. -At a command prompt, list the contents of this directory and find the `app.apk` file: +At a command prompt, list the contents of this directory and find the `app.apk` +file: -| OS | Command | +| OS | Command | | ------------------------ | ------------------------ | | Linux, macOS | `ls bazel-bin/src/main` | | Windows (Command Prompt) | `dir bazel-bin\src\main` | | Windows (PowerShell) | `ls bazel-bin\src\main` | + ### Run the app -You can now deploy the app to a connected Android device or emulator from the command line using `bazel mobile-install`. This command uses the Android Debug Bridge (`adb`) to communicate with the device. You must set up your device to use `adb` following the instructions in [Android Debug Bridge](http://developer.android.com/tools/help/adb.html) before deployment. You can also choose to install the app on the Android emulator included in Android Studio. Make sure the emulator is running before executing the command below. +You can now deploy the app to a connected Android device or emulator from the +command line using `bazel mobile-install`. +This command uses the Android Debug Bridge (`adb`) to communicate with the +device. You must set up your device to use `adb` following the instructions in +[Android Debug Bridge](http://developer.android.com/tools/help/adb.html) +before deployment. You can also choose to install the app on the Android emulator +included in Android Studio. Make sure the emulator is running before executing +the command below. Enter the following: @@ -278,7 +378,12 @@ bazel mobile-install //src/main:app \ --tool_java_language_version=17 ``` -Note that the extra flags required for mobile-install can be added to your project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags (`--mode`, `--mobile_install*`) will no longer be required starting from Bazel 8.4.0 and onwards. The various Java flags for language and runtime version may be required depending on your workspace's Java configuration. *Mobile-install sub-tools require a language and runtime level of 17 or higher.* +Note that the extra flags required for mobile-install can be added to your +project's [bazelrc file](/run/bazelrc). The mobile-install-specific flags +(`--mode`, `--mobile_install*`) will no longer be required starting from +Bazel 8.4.0 and onwards. The various Java flags for language and runtime version +may be required depending on your workspace's Java configuration. +_Mobile-install sub-tools require a language and runtime level of 17 or higher._ Now the "Bazel Tutorial App" should install and launch automatically: @@ -292,20 +397,16 @@ Now the "Bazel Tutorial App" should install and launch automatically: For more details, see these pages: -- Open issues on [rules\_android GitHub](https://github.com/bazelbuild/rules_android/issues) - -- More information on [mobile-install](/docs/mobile-install) - -- Integrate external dependencies like AppCompat, Guava and JUnit from Maven repositories using [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) - -- Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) integration. - -- Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) - -- See more Bazel example projects of: - - - [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) - - [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) - - [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) +* Open issues on [rules_android GitHub](https://github.com/bazelbuild/rules_android/issues) +* More information on [mobile-install](/docs/mobile-install) +* Integrate external dependencies like AppCompat, Guava and JUnit from Maven + repositories using [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) +* Run Robolectric tests with the [robolectric-bazel](https://github.com/robolectric/robolectric-bazel) + integration. +* Integrating C and C++ code into your Android app with the [NDK](/docs/android-ndk) +* See more Bazel example projects of: + * [a Kotlin app](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_kotlin_app) + * [Robolectric testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_local_test) + * [Espresso testing](https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/android_instrumentation_test) Happy building! diff --git a/start/cpp.mdx b/start/cpp.mdx index e583146e..adb7c71a 100644 --- a/start/cpp.mdx +++ b/start/cpp.mdx @@ -2,25 +2,37 @@ title: 'Bazel Tutorial: Build a C++ Project' --- + + ## Introduction -New to Bazel? You're in the right place. Follow this First Build tutorial for a simplified introduction to using Bazel. This tutorial defines key terms as they are used in Bazel's context and walks you through the basics of the Bazel workflow. Starting with the tools you need, you will build and run three projects with increasing complexity and learn how and why they get more complex. +New to Bazel? You're in the right place. Follow this First Build tutorial for a +simplified introduction to using Bazel. This tutorial defines key terms as they +are used in Bazel's context and walks you through the basics of the Bazel +workflow. Starting with the tools you need, you will build and run three +projects with increasing complexity and learn how and why they get more complex. -While Bazel is a [build system](https://bazel.build/basics/build-systems) that supports multi-language builds, this tutorial uses a C++ project as an example and provides the general guidelines and flow that apply to most languages. +While Bazel is a [build system](https://bazel.build/basics/build-systems) that +supports multi-language builds, this tutorial uses a C++ project as an example +and provides the general guidelines and flow that apply to most languages. Estimated completion time: 30 minutes. ### Prerequisites -Start by [installing Bazel](https://bazel.build/install), if you haven't already. This tutorial uses Git for source control, so for best results [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. +Start by [installing Bazel](https://bazel.build/install), if you haven't +already. This tutorial uses Git for source control, so for best results [install +Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) as well. -Next, retrieve the sample project from Bazel's GitHub repository by running the following in your command-line tool of choice: +Next, retrieve the sample project from Bazel's GitHub repository by running the +following in your command-line tool of choice: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/cpp-tutorial` directory. +The sample project for this tutorial is in the `examples/cpp-tutorial` +directory. Take a look at how it's structured: @@ -52,24 +64,52 @@ examples └── MODULE.bazel ``` -There are three sets of files, each set representing a stage in this tutorial. In the first stage, you will build a single \[target] ([https://bazel.build/reference/glossary#target](https://bazel.build/reference/glossary#target)) residing in a single \[package] ([https://bazel.build/reference/glossary#package](https://bazel.build/reference/glossary#package)). In the second stage, you will build both a binary and a library from a single package. In the third and final stage, you will build a project with multiple packages and build it with multiple targets. +There are three sets of files, each set representing a stage in this tutorial. +In the first stage, you will build a single [target] +(https://bazel.build/reference/glossary#target) residing in a single [package] +(https://bazel.build/reference/glossary#package). In the second stage, you will +build both a binary and a library from a single package. In the third and final +stage, you will build a project with multiple packages and build it with +multiple targets. ### Summary: Introduction -By installing Bazel (and Git) and cloning the repository for this tutorial, you have laid the foundation for your first build with Bazel. Continue to the next section to define some terms and set up your [workspace](https://bazel.build/reference/glossary#workspace). +By installing Bazel (and Git) and cloning the repository for this tutorial, you +have laid the foundation for your first build with Bazel. Continue to the next +section to define some terms and set up your +[workspace](https://bazel.build/reference/glossary#workspace). ## Getting started -Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains these significant files: +Before you can build a project, you need to set up its workspace. A workspace +is a directory that holds your project's source files and Bazel's build outputs. +It also contains these significant files: -- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure. It's also where you specify your external dependencies. -- One or more [`BUILD` files](https://bazel.build/reference/glossary#build-file), which tell Bazel how to build different parts of the project. A directory within the workspace that contains a `BUILD` file is a [package](https://bazel.build/reference/glossary#package). (More on packages later in this tutorial.) +* The `MODULE.bazel` file, which identifies the directory and its contents as + a Bazel workspace and lives at the root of the project's directory + structure. It's also where you specify your external dependencies. +* One or more [`BUILD` + files](https://bazel.build/reference/glossary#build-file), which tell Bazel + how to build different parts of the project. A directory within the + workspace that contains a `BUILD` file is a + [package](https://bazel.build/reference/glossary#package). (More on packages + later in this tutorial.) -In future projects, to designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. For the purposes of this tutorial, a `MODULE.bazel` file is already present in each stage. +In future projects, to designate a directory as a Bazel workspace, create an +empty file named `MODULE.bazel` in that directory. For the purposes of this +tutorial, a `MODULE.bazel` file is already present in each stage. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. Each `BUILD` file requires at least one [rule](https://bazel.build/reference/glossary#rule) as a set of instructions, which tells Bazel how to build the outputs you want, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a [target](https://bazel.build/reference/glossary#target) and points to a specific set of source files and [dependencies](https://bazel.build/reference/glossary#dependency). A target can also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. Each +`BUILD` file requires at least one +[rule](https://bazel.build/reference/glossary#rule) as a set of instructions, +which tells Bazel how to build the outputs you want, such as executable binaries +or libraries. Each instance of a build rule in the `BUILD` file is called a +[target](https://bazel.build/reference/glossary#target) and points to a specific +set of source files and +[dependencies](https://bazel.build/reference/glossary#dependency). A target can +also point to other targets. Take a look at the `BUILD` file in the `cpp-tutorial/stage1/main` directory: @@ -80,15 +120,21 @@ cc_binary( ) ``` -In our example, the `hello-world` target instantiates Bazel's built-in [`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule tells Bazel to build a self-contained executable binary from the `hello-world.cc`> source file with no dependencies. +In our example, the `hello-world` target instantiates Bazel's built-in +[`cc_binary` rule](https://bazel.build/reference/be/c-cpp#cc_binary). The rule +tells Bazel to build a self-contained executable binary from the +`hello-world.cc`> source file with no dependencies. ### Summary: getting started -Now you are familiar with some key terms, and what they mean in the context of this project and Bazel in general. In the next section, you will build and test Stage 1 of the project. +Now you are familiar with some key terms, and what they mean in the context of +this project and Bazel in general. In the next section, you will build and test +Stage 1 of the project. ## Stage 1: single target, single package -It's time to build the first part of the project. For a visual reference, the structure of the Stage 1 section of the project is: +It's time to build the first part of the project. For a visual reference, the +structure of the Stage 1 section of the project is: ```none examples @@ -112,7 +158,9 @@ Next, run: bazel build //main:hello-world ``` -In the target label, the `//main:` part is the location of the `BUILD` file relative to the root of the workspace, and `hello-world` is the target name in the `BUILD` file. +In the target label, the `//main:` part is the location of the `BUILD` file +relative to the root of the workspace, and `hello-world` is the target name in +the `BUILD` file. Bazel produces something that looks like this: @@ -123,7 +171,8 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.267s, Critical Path: 0.25s ``` -You just built your first Bazel target. Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. +You just built your first Bazel target. Bazel places build outputs in the +`bazel-bin` directory at the root of the workspace. Now test your freshly built binary, which is: @@ -135,15 +184,23 @@ This results in a printed "`Hello world`" message. Here's the dependency graph of Stage 1: -![Dependency graph for hello-world displays a single target with a single source file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world displays a single target with a single source file.") +![Dependency graph for hello-world displays a single target with a single source +file.](/docs/images/cpp-tutorial-stage1.png "Dependency graph for hello-world +displays a single target with a single source file.") ### Summary: stage 1 -Now that you have completed your first build, you have a basic idea of how a build is structured. In the next stage, you will add complexity by adding another target. +Now that you have completed your first build, you have a basic idea of how a +build is structured. In the next stage, you will add complexity by adding +another target. ## Stage 2: multiple build targets -While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages. This allows for fast incremental builds – that is, Bazel only rebuilds what's changed – and speeds up your builds by building multiple parts of a project at once. This stage of the tutorial adds a target, and the next adds a package. +While a single target is sufficient for small projects, you may want to split +larger projects into multiple targets and packages. This allows for fast +incremental builds – that is, Bazel only rebuilds what's changed – and speeds up +your builds by building multiple parts of a project at once. This stage of the +tutorial adds a target, and the next adds a package. This is the directory you are working with for Stage 2: @@ -175,9 +232,15 @@ cc_binary( ) ``` -With this `BUILD` file, Bazel first builds the `hello-greet` library (using Bazel's built-in [`cc_library` rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the `hello-world` binary. The `deps` attribute in the `hello-world` target tells Bazel that the `hello-greet` library is required to build the `hello-world` binary. +With this `BUILD` file, Bazel first builds the `hello-greet` library (using +Bazel's built-in [`cc_library` +rule](https://bazel.build/reference/be/c-cpp#cc_library)), then the +`hello-world` binary. The `deps` attribute in the `hello-world` target tells +Bazel that the `hello-greet` library is required to build the `hello-world` +binary. -Before you can build this new version of the project, you need to change directories, switching to the `cpp-tutorial/stage2` directory by running: +Before you can build this new version of the project, you need to change +directories, switching to the `cpp-tutorial/stage2` directory by running: ```posix-terminal cd ../stage2 @@ -198,25 +261,36 @@ Target //main:hello-world up-to-date: INFO: Elapsed time: 2.399s, Critical Path: 0.30s ``` -Now you can test your freshly built binary, which returns another "`Hello world`": +Now you can test your freshly built binary, which returns another "`Hello +world`": ```posix-terminal bazel-bin/main/hello-world ``` -If you now modify `hello-greet.cc` and rebuild the project, Bazel only recompiles that file. +If you now modify `hello-greet.cc` and rebuild the project, Bazel only +recompiles that file. -Looking at the dependency graph, you can see that `hello-world` depends on an extra input named `hello-greet`: +Looking at the dependency graph, you can see that `hello-world` depends on an +extra input named `hello-greet`: -![Dependency graph for hello-world displays dependency changes after modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency graph for `hello-world` displays dependency changes after modification to the file.") +![Dependency graph for `hello-world` displays dependency changes after +modification to the file.](/docs/images/cpp-tutorial-stage2.png "Dependency +graph for `hello-world` displays dependency changes after modification to the +file.") ### Summary: stage 2 -You've now built the project with two targets. The `hello-world` target builds one source file and depends on one other target (`//main:hello-greet`), which builds two additional source files. In the next section, take it a step further and add another package. +You've now built the project with two targets. The `hello-world` target builds +one source file and depends on one other target (`//main:hello-greet`), which +builds two additional source files. In the next section, take it a step further +and add another package. ## Stage 3: multiple packages -This next stage adds another layer of complication and builds a project with multiple packages. Take a look at the structure and contents of the `cpp-tutorial/stage3` directory: +This next stage adds another layer of complication and builds a project with +multiple packages. Take a look at the structure and contents of the +`cpp-tutorial/stage3` directory: ```none └──stage3 @@ -232,7 +306,9 @@ This next stage adds another layer of complication and builds a project with mul └── MODULE.bazel ``` -You can see that now there are two sub-directories, and each contains a `BUILD` file. Therefore, to Bazel, the workspace now contains two packages: `lib` and `main`. +You can see that now there are two sub-directories, and each contains a `BUILD` +file. Therefore, to Bazel, the workspace now contains two packages: `lib` and +`main`. Take a look at the `lib/BUILD` file: @@ -264,13 +340,25 @@ cc_binary( ) ``` -The `hello-world` target in the main package depends on the` hello-time` target in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows this through the `deps` attribute. You can see this reflected in the dependency graph: +The `hello-world` target in the main package depends on the` hello-time` target +in the `lib` package (hence the target label `//lib:hello-time`) - Bazel knows +this through the `deps` attribute. You can see this reflected in the dependency +graph: -![Dependency graph for hello-world displays how the target in the main package depends on the target in the lib package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for `hello-world` displays how the target in the main package depends on the target in the `lib` package.") +![Dependency graph for `hello-world` displays how the target in the main package +depends on the target in the `lib` +package.](/docs/images/cpp-tutorial-stage3.png "Dependency graph for +`hello-world` displays how the target in the main package depends on the target +in the `lib` package.") -For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` explicitly visible to targets in `main/BUILD` using the visibility attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs. +For the build to succeed, you make the `//lib:hello-time` target in `lib/BUILD` +explicitly visible to targets in `main/BUILD` using the visibility attribute. +This is because by default targets are only visible to other targets in the same +`BUILD` file. Bazel uses target visibility to prevent issues such as libraries +containing implementation details leaking into public APIs. -Now build this final version of the project. Switch to the `cpp-tutorial/stage3` directory by running: +Now build this final version of the project. Switch to the `cpp-tutorial/stage3` +directory by running: ```posix-terminal cd ../stage3 @@ -299,15 +387,25 @@ bazel-bin/main/hello-world ### Summary: stage 3 -You've now built the project as two packages with three targets and understand the dependencies between them, which equips you to go forth and build future projects with Bazel. In the next section, take a look at how to continue your Bazel journey. +You've now built the project as two packages with three targets and understand +the dependencies between them, which equips you to go forth and build future +projects with Bazel. In the next section, take a look at how to continue your +Bazel journey. ## Next steps -You've now completed your first basic build with Bazel, but this is just the start. Here are some more resources to continue learning with Bazel: - -- To keep focusing on C++, read about common [C++ build use cases](https://bazel.build/tutorials/cpp-use-cases). -- To get started with building other applications with Bazel, see the tutorials for [Java](https://bazel.build/start/java), [Android application](https://bazel.build/start/android-app), or [iOS application](https://bazel.build/start/ios-app). -- To learn more about working with local and remote repositories, read about [external dependencies](https://bazel.build/docs/external). -- To learn more about Bazel's other rules, see this [reference guide](https://bazel.build/rules). +You've now completed your first basic build with Bazel, but this is just the +start. Here are some more resources to continue learning with Bazel: + +* To keep focusing on C++, read about common [C++ build use + cases](https://bazel.build/tutorials/cpp-use-cases). +* To get started with building other applications with Bazel, see the + tutorials for [Java](https://bazel.build/start/java), [Android + application](https://bazel.build/start/android-app), or [iOS + application](https://bazel.build/start/ios-app). +* To learn more about working with local and remote repositories, read about + [external dependencies](https://bazel.build/docs/external). +* To learn more about Bazel's other rules, see this [reference + guide](https://bazel.build/rules). Happy building! diff --git a/start/go.mdx b/start/go.mdx index d5742314..3a095681 100644 --- a/start/go.mdx +++ b/start/go.mdx @@ -2,7 +2,12 @@ title: 'Bazel Tutorial: Build a Go Project' --- -This tutorial introduces you to the basics of Bazel by showing you how to build a Go (Golang) project. You'll learn how to set up your workspace, build a small program, import a library, and run its test. Along the way, you'll learn key Bazel concepts, such as targets and `BUILD` files. + + +This tutorial introduces you to the basics of Bazel by showing you how to build +a Go (Golang) project. You'll learn how to set up your workspace, build a small +program, import a library, and run its test. Along the way, you'll learn key +Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes @@ -10,27 +15,35 @@ Estimated completion time: 30 minutes ### Install Bazel -Before you get started, first [install bazel](/install) if you haven't done so already. +Before you get started, first [install bazel](/install) if you haven't done so +already. You can check if Bazel is installed by running `bazel version` in any directory. ### Install Go (optional) -You don't need to [install Go](https://go.dev/doc/install) to build Go projects with Bazel. The Bazel Go rule set automatically downloads and uses a Go toolchain instead of using the toolchain installed on your machine. This ensures all developers on a project build with same version of Go. +You don't need to [install Go](https://go.dev/doc/install) to build Go projects +with Bazel. The Bazel Go rule set automatically downloads and uses a Go +toolchain instead of using the toolchain installed on your machine. This ensures +all developers on a project build with same version of Go. -However, you may still want to install a Go toolchain to run commands like `go get` and `go mod tidy`. +However, you may still want to install a Go toolchain to run commands like `go +get` and `go mod tidy`. You can check if Go is installed by running `go version` in any directory. ### Get the sample project -The Bazel examples are stored in a Git repository, so you'll need to [install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you haven't already. To download the examples repository, run this command: +The Bazel examples are stored in a Git repository, so you'll need to [install +Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) if you +haven't already. To download the examples repository, run this command: ```posix-terminal git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/go-tutorial` directory. See what it contains: +The sample project for this tutorial is in the `examples/go-tutorial` directory. +See what it contains: ```none go-tutorial/ @@ -39,11 +52,13 @@ go-tutorial/ └── stage3 ``` -There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a different section of this tutorial. Each stage builds on the previous one. +There are three subdirectories (`stage1`, `stage2`, and `stage3`), each for a +different section of this tutorial. Each stage builds on the previous one. ## Build with Bazel -Start in the `stage1` directory, where we'll find a program. We can build it with `bazel build`, then run it: +Start in the `stage1` directory, where we'll find a program. We can +build it with `bazel build`, then run it: ```posix-shell $ cd go-tutorial/stage1/ @@ -92,7 +107,9 @@ func main() { } ``` -`BUILD` contains some instructions for Bazel, telling it what we want to build. You'll typically write a file like this in each directory. For this project, we have a single `go_binary` target that builds our program from `hello.go`. +`BUILD` contains some instructions for Bazel, telling it what we want to build. +You'll typically write a file like this in each directory. For this project, we +have a single `go_binary` target that builds our program from `hello.go`. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -103,9 +120,17 @@ go_binary( ) ``` -`MODULE.bazel` tracks your project's dependencies. It also marks your project's root directory, so you'll only write one `MODULE.bazel` file per project. It serves a similar purpose to Go's `go.mod` file. You don't actually need a `go.mod` file in a Bazel project, but it may still be useful to have one so that you can continue using `go get` and `go mod tidy` for dependency management. The Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in another tutorial. +`MODULE.bazel` tracks your project's dependencies. It also marks your project's +root directory, so you'll only write one `MODULE.bazel` file per project. It +serves a similar purpose to Go's `go.mod` file. You don't actually need a +`go.mod` file in a Bazel project, but it may still be useful to have one so that +you can continue using `go get` and `go mod tidy` for dependency management. The +Bazel Go rule set can import dependencies from `go.mod`, but we'll cover that in +another tutorial. -Our `MODULE.bazel` file contains a single dependency on [rules\_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need this dependency because Bazel doesn't have built-in support for Go. +Our `MODULE.bazel` file contains a single dependency on +[rules_go](https://github.com/bazel-contrib/rules_go), the Go rule set. We need +this dependency because Bazel doesn't have built-in support for Go. ```bazel bazel_dep( @@ -114,25 +139,50 @@ bazel_dep( ) ``` -Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes and other metadata about our dependencies. It includes implicit dependencies added by Bazel itself, so it's quite long, and we won't show it here. Just like `go.sum`, you should commit your `MODULE.bazel.lock` file to source control to ensure everyone on your project gets the same version of each dependency. You shouldn't need to edit `MODULE.bazel.lock` manually. +Finally, `MODULE.bazel.lock` is a file generated by Bazel that contains hashes +and other metadata about our dependencies. It includes implicit dependencies +added by Bazel itself, so it's quite long, and we won't show it here. Just like +`go.sum`, you should commit your `MODULE.bazel.lock` file to source control to +ensure everyone on your project gets the same version of each dependency. You +shouldn't need to edit `MODULE.bazel.lock` manually. ### Understand the BUILD file -Most of your interaction with Bazel will be through `BUILD` files (or equivalently, `BUILD.bazel` files), so it's important to understand what they do. +Most of your interaction with Bazel will be through `BUILD` files (or +equivalently, `BUILD.bazel` files), so it's important to understand what they +do. -`BUILD` files are written in a scripting language called [Starlark](https://bazel.build/rules/language), a limited subset of Python. +`BUILD` files are written in a scripting language called +[Starlark](https://bazel.build/rules/language), a limited subset of Python. -A `BUILD` file contains a list of [targets](https://bazel.build/reference/glossary#target). A target is something Bazel can build, like a binary, library, or test. +A `BUILD` file contains a list of +[targets](https://bazel.build/reference/glossary#target). A target is something +Bazel can build, like a binary, library, or test. -A target calls a rule function with a list of [attributes](https://bazel.build/reference/glossary#attribute) to describe what should be built. Our example has two attributes: `name` identifies the target on the command line, and `srcs` is a list of source file paths (slash-separated, relative to the directory containing the `BUILD` file). +A target calls a rule function with a list of +[attributes](https://bazel.build/reference/glossary#attribute) to describe what +should be built. Our example has two attributes: `name` identifies the target on +the command line, and `srcs` is a list of source file paths (slash-separated, +relative to the directory containing the `BUILD` file). -A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a target. In our example, we used the [`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) (commands) that generate a set of output files. For example, `go_binary` defines Go compile and link actions that produce an executable output file. +A [rule](https://bazel.build/reference/glossary#rule) tells Bazel how to build a +target. In our example, we used the +[`go_binary`](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md#go_binary) +rule. Each rule defines [actions](https://bazel.build/reference/glossary#action) +(commands) that generate a set of output files. For example, `go_binary` defines +Go compile and link actions that produce an executable output file. -Bazel has built-in rules for a few languages like Java and C++. You can find their [documentation in the Build Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find rule sets for many other languages and tools on the [Bazel Central Registry (BCR)](https://registry.bazel.build/). +Bazel has built-in rules for a few languages like Java and C++. You can find +their [documentation in the Build +Encyclopedia](https://bazel.build/reference/be/overview#rules). You can find +rule sets for many other languages and tools on the [Bazel Central Registry +(BCR)](https://registry.bazel.build/). ## Add a library -Move onto the `stage2` directory, where we'll build a new program that prints your fortune. This program uses a separate Go package as a library that selects a fortune from a predefined list of messages. +Move onto the `stage2` directory, where we'll build a new program that +prints your fortune. This program uses a separate Go package as a library that +selects a fortune from a predefined list of messages. ```none go-tutorial/stage2 @@ -145,7 +195,11 @@ go-tutorial/stage2 └── print_fortune.go ``` -`fortune.go` is the source file for the library. The `fortune` library is a separate Go package, so its source files are in a separate directory. Bazel doesn't require you to keep Go packages in separate directories, but it's a strong convention in the Go ecosystem, and following it will help you stay compatible with other Go tools. +`fortune.go` is the source file for the library. The `fortune` library is a +separate Go package, so its source files are in a separate directory. Bazel +doesn't require you to keep Go packages in separate directories, but it's a +strong convention in the Go ecosystem, and following it will help you stay +compatible with other Go tools. ```go package fortune @@ -163,11 +217,19 @@ func Get() string { } ``` -The `fortune` directory has its own `BUILD` file that tells Bazel how to build this package. We use `go_library` here instead of `go_binary`. +The `fortune` directory has its own `BUILD` file that tells Bazel how to build +this package. We use `go_library` here instead of `go_binary`. -We also need to set the `importpath` attribute to a string with which the library can be imported into other Go source files. This name should be the repository path (or module path) concatenated with the directory within the repository. +We also need to set the `importpath` attribute to a string with which the +library can be imported into other Go source files. This name should be the +repository path (or module path) concatenated with the directory within the +repository. -Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. [`visibility`](https://bazel.build/concepts/visibility) may be set on any target. It determines which Bazel packages may depend on this target. In our case, we want any target to be able to depend on this library, so we use the special value `//visibility:public`. +Finally, we need to set the `visibility` attribute to `["//visibility:public"]`. +[`visibility`](https://bazel.build/concepts/visibility) may be set on any +target. It determines which Bazel packages may depend on this target. In our +case, we want any target to be able to depend on this library, so we use the +special value `//visibility:public`. ```bazel load("@rules_go//go:def.bzl", "go_library") @@ -202,9 +264,11 @@ func main() { } ``` -`print_fortune.go` imports the package using the same string declared in the `importpath` attribute of the `fortune` library. +`print_fortune.go` imports the package using the same string declared in the +`importpath` attribute of the `fortune` library. -We also need to declare this dependency to Bazel. Here's the `BUILD` file in the `stage2` directory. +We also need to declare this dependency to Bazel. Here's the `BUILD` file in the +`stage2` directory. ```bazel load("@rules_go//go:def.bzl", "go_binary") @@ -222,25 +286,57 @@ You can run this with the command below. bazel run //:print_fortune ``` -The `print_fortune` target has a `deps` attribute, a list of other targets that it depends on. It contains `"//fortune"`, a label string referring to the target in the `fortune` directory named `fortune`. +The `print_fortune` target has a `deps` attribute, a list of other targets that +it depends on. It contains `"//fortune"`, a label string referring to the target +in the `fortune` directory named `fortune`. -Bazel requires that all targets declare their dependencies explicitly with attributes like `deps`. This may seem cumbersome since dependencies are *also* specified in source files, but Bazel's explictness gives it an advantage. Bazel builds an [action graph](https://bazel.build/reference/glossary#action-graph) containing all commands, inputs, and outputs before running any commands, without reading any source files. Bazel can then cache action results or send actions for [remote execution](https://bazel.build/remote/rbe) without built-in language-specific logic. +Bazel requires that all targets declare their dependencies explicitly with +attributes like `deps`. This may seem cumbersome since dependencies are *also* +specified in source files, but Bazel's explictness gives it an advantage. Bazel +builds an [action graph](https://bazel.build/reference/glossary#action-graph) +containing all commands, inputs, and outputs before running any commands, +without reading any source files. Bazel can then cache action results or send +actions for [remote execution](https://bazel.build/remote/rbe) without built-in +language-specific logic. ### Understanding labels -A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses to identify a target or a file. Labels are used in command line arguments and in `BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, `//:print-fortune`, and `@rules_go//go:def.bzl`. - -A label has three parts: a repository name, a package name, and a target (or file) name. - -The repository name is written between `@` and `//` and is used to refer to a target from a different Bazel module (for historical reasons, *module* and *repository* are sometimes used synonymously). In the label, `@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name can be omitted when referring to targets in the same repository. - -The package name is written between `//` and `:` and is used to refer to a target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, the package name is `go`. A Bazel [package](https://bazel.build/reference/glossary#package) is a set of files and targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. Its package name is a slash-separated path from the module root directory (containing `MODULE.bazel`) to the directory containing the `BUILD` file. A package may include subdirectories, but only if they don't also contain `BUILD` files defining their own packages. - -Most Go projects have one `BUILD` file per directory and one Go package per `BUILD` file. The package name in a label may be omitted when referring to targets in the same directory. - -The target name is written after `:` and refers to a target within a package. The target name may be omitted if it's the same as the last component of the package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is the same as `//fortune`). - -On the command-line, you can use `...` as a wildcard to refer to all the targets within a package. This is useful for building or testing all the targets in a repository. +A [label](https://bazel.build/reference/glossary#label) is a string Bazel uses +to identify a target or a file. Labels are used in command line arguments and in +`BUILD` file attributes like `deps`. We've seen a few already, like `//fortune`, +`//:print-fortune`, and `@rules_go//go:def.bzl`. + +A label has three parts: a repository name, a package name, and a target (or +file) name. + +The repository name is written between `@` and `//` and is used to refer to a +target from a different Bazel module (for historical reasons, *module* and +*repository* are sometimes used synonymously). In the label, +`@rules_go//go:def.bzl`, the repository name is `rules_go`. The repository name +can be omitted when referring to targets in the same repository. + +The package name is written between `//` and `:` and is used to refer to a +target in from a different Bazel package. In the label `@rules_go//go:def.bzl`, +the package name is `go`. A Bazel +[package](https://bazel.build/reference/glossary#package) is a set of files and +targets defined by a `BUILD` or `BUILD.bazel` file in its top-level directory. +Its package name is a slash-separated path from the module root directory +(containing `MODULE.bazel`) to the directory containing the `BUILD` file. A +package may include subdirectories, but only if they don't also contain `BUILD` +files defining their own packages. + +Most Go projects have one `BUILD` file per directory and one Go package per +`BUILD` file. The package name in a label may be omitted when referring to +targets in the same directory. + +The target name is written after `:` and refers to a target within a package. +The target name may be omitted if it's the same as the last component of the +package name (so `//a/b/c:c` is the same as `//a/b/c`; `//fortune:fortune` is +the same as `//fortune`). + +On the command-line, you can use `...` as a wildcard to refer to all the targets +within a package. This is useful for building or testing all the targets in a +repository. ```posix-shell # Build everything @@ -276,13 +372,15 @@ import ( // TestGet checks that Get returns one of the strings from fortunes. func TestGet(t *testing.T) { msg := Get() - if i := slices.Index(fortunes, msg); i < 0 { + if i := slices.Index(fortunes, msg); i < 0 { t.Errorf("Get returned %q, not one the expected messages", msg) } } ``` -This file uses the unexported `fortunes` variable, so it needs to be compiled into the same Go package as `fortune.go`. Look at the `BUILD` file to see how that works: +This file uses the unexported `fortunes` variable, so it needs to be compiled +into the same Go package as `fortune.go`. Look at the `BUILD` file to see +how that works: ```bazel load("@rules_go//go:def.bzl", "go_library", "go_test") @@ -301,9 +399,19 @@ go_test( ) ``` -We have a new `fortune_test` target that uses the `go_test` rule to compile and link a test executable. `go_test` needs to compile `fortune.go` and `fortune_test.go` together with the same command, so we use the `embed` attribute here to incorporate the attributes of the `fortune` target into `fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, but it also works with `go_library`, which is sometimes useful for generated code. +We have a new `fortune_test` target that uses the `go_test` rule to compile and +link a test executable. `go_test` needs to compile `fortune.go` and +`fortune_test.go` together with the same command, so we use the `embed` +attribute here to incorporate the attributes of the `fortune` target into +`fortune_test`. `embed` is most commonly used with `go_test` and `go_binary`, +but it also works with `go_library`, which is sometimes useful for generated +code. -You may be wondering if the `embed` attribute is related to Go's [`embed`](https://pkg.go.dev/embed) package, which is used to access data files copied into an executable. This is an unfortunate name collision: rules\_go's `embed` attribute was introduced before Go's `embed` package. Instead, rules\_go uses the `embedsrcs` to list files that can be loaded with the `embed` package. +You may be wondering if the `embed` attribute is related to Go's +[`embed`](https://pkg.go.dev/embed) package, which is used to access data files +copied into an executable. This is an unfortunate name collision: rules_go's +`embed` attribute was introduced before Go's `embed` package. Instead, rules_go +uses the `embedsrcs` to list files that can be loaded with the `embed` package. Try running our test with `bazel test`: @@ -322,7 +430,9 @@ Executed 0 out of 1 test: 1 test passes. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. ``` -You can use the `...` wildcard to run all tests. Bazel will also build targets that aren't tests, so this can catch compile errors even in packages that don't have tests. +You can use the `...` wildcard to run all tests. Bazel will also build targets +that aren't tests, so this can catch compile errors even in packages that don't +have tests. ```posix-shell $ bazel test //... @@ -330,9 +440,21 @@ $ bazel test //... ## Conclusion and further reading -In this tutorial, we built and tested a small Go project with Bazel, and we learned some core Bazel concepts along the way. - -- To get started building other applications with Bazel, see the tutorials for [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and [iOS](/start/ios-app). -- You can also check the list of [recommended rules](/rules) for other languages. -- For more information on Go, see the [rules\_go](https://github.com/bazel-contrib/rules_go) module, especially the [Core Go rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) documentation. -- To learn more about working with Bazel modules outside your project, see [external dependencies](/docs/external). In particular, for information on how to depend on Go modules and toolchains through Bazel's module system, see [Go with bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). +In this tutorial, we built and tested a small Go project with Bazel, and we +learned some core Bazel concepts along the way. + +- To get started building other applications with Bazel, see the tutorials for + [C++](/start/cpp), [Java](/start/java), [Android](/start/android-app), and + [iOS](/start/ios-app). +- You can also check the list of [recommended rules](/rules) for other + languages. +- For more information on Go, see the + [rules_go](https://github.com/bazel-contrib/rules_go) module, especially the + [Core Go + rules](https://github.com/bazel-contrib/rules_go/blob/master/docs/go/core/rules.md) + documentation. +- To learn more about working with Bazel modules outside your project, see + [external dependencies](/docs/external). In particular, for information on + how to depend on Go modules and toolchains through Bazel's module system, + see [Go with + bzlmod](https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core/bzlmod.md). diff --git a/start/ios-app.mdx b/start/ios-app.mdx index 5004bbd2..0b860ab6 100644 --- a/start/ios-app.mdx +++ b/start/ios-app.mdx @@ -2,4 +2,5 @@ title: 'Bazel Tutorial: Build an iOS App' --- -This tutorial has been moved into the [bazelbuild/rules\_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. + +This tutorial has been moved into the [bazelbuild/rules_apple](https://github.com/bazelbuild/rules_apple/blob/master/doc/tutorials/ios-app.md) repository. diff --git a/start/java.mdx b/start/java.mdx index c9c5916c..b892917d 100644 --- a/start/java.mdx +++ b/start/java.mdx @@ -2,7 +2,11 @@ title: 'Bazel Tutorial: Build a Java Project' --- -This tutorial covers the basics of building Java applications with Bazel. You will set up your workspace and build a simple Java project that illustrates key Bazel concepts, such as targets and `BUILD` files. + + +This tutorial covers the basics of building Java applications with +Bazel. You will set up your workspace and build a simple Java project that +illustrates key Bazel concepts, such as targets and `BUILD` files. Estimated completion time: 30 minutes. @@ -10,40 +14,36 @@ Estimated completion time: 30 minutes. In this tutorial you learn how to: -- Build a target -- Visualize the project's dependencies -- Split the project into multiple targets and packages -- Control target visibility across packages -- Reference targets through labels -- Deploy a target +* Build a target +* Visualize the project's dependencies +* Split the project into multiple targets and packages +* Control target visibility across packages +* Reference targets through labels +* Deploy a target ## Before you begin ### Install Bazel -To prepare for the tutorial, first [Install Bazel](/install) if you don't have it installed already. +To prepare for the tutorial, first [Install Bazel](/install) if +you don't have it installed already. ### Install the JDK -1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). - -2. Set the JAVA\_HOME environment variable to point to the JDK. - - - On Linux/macOS: - - ``` - export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" - ``` +1. Install Java JDK (preferred version is 11, however versions between 8 and 15 are supported). - - On Windows: +2. Set the JAVA\_HOME environment variable to point to the JDK. + * On Linux/macOS: - 1. Open Control Panel. - 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . - 3. Under the "User variables" list (the one on the top), click "New\...". - 4. In the "Variable name" field, enter `JAVA_HOME`. - 5. Click "Browse Directory...". - 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). - 7. Click "OK" on all dialog windows. + export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))" + * On Windows: + 1. Open Control Panel. + 2. Go to "System and Security" > "System" > "Advanced System Settings" > "Advanced" tab > "Environment Variables..." . + 3. Under the "User variables" list (the one on the top), click "New...". + 4. In the "Variable name" field, enter `JAVA_HOME`. + 5. Click "Browse Directory...". + 6. Navigate to the JDK directory (for example `C:\Program Files\Java\jdk1.8.0_152`). + 7. Click "OK" on all dialog windows. ### Get the sample project @@ -53,7 +53,8 @@ Retrieve the sample project from Bazel's GitHub repository: git clone https://github.com/bazelbuild/examples ``` -The sample project for this tutorial is in the `examples/java-tutorial` directory and is structured as follows: +The sample project for this tutorial is in the `examples/java-tutorial` +directory and is structured as follows: ``` java-tutorial @@ -75,19 +76,32 @@ java-tutorial ### Set up the workspace -Before you can build a project, you need to set up its workspace. A workspace is a directory that holds your project's source files and Bazel's build outputs. It also contains files that Bazel recognizes as special: +Before you can build a project, you need to set up its workspace. A workspace is +a directory that holds your project's source files and Bazel's build outputs. It +also contains files that Bazel recognizes as special: -- The `MODULE.bazel` file, which identifies the directory and its contents as a Bazel workspace and lives at the root of the project's directory structure, +* The `MODULE.bazel` file, which identifies the directory and its contents as a + Bazel workspace and lives at the root of the project's directory structure, -- One or more `BUILD` files, which tell Bazel how to build different parts of the project. (A directory within the workspace that contains a `BUILD` file is a *package*. You will learn about packages later in this tutorial.) +* One or more `BUILD` files, which tell Bazel how to build different parts of + the project. (A directory within the workspace that contains a `BUILD` file + is a *package*. You will learn about packages later in this tutorial.) -To designate a directory as a Bazel workspace, create an empty file named `MODULE.bazel` in that directory. +To designate a directory as a Bazel workspace, create an empty file named +`MODULE.bazel` in that directory. -When Bazel builds the project, all inputs and dependencies must be in the same workspace. Files residing in different workspaces are independent of one another unless linked, which is beyond the scope of this tutorial. +When Bazel builds the project, all inputs and dependencies must be in the same +workspace. Files residing in different workspaces are independent of one +another unless linked, which is beyond the scope of this tutorial. ### Understand the BUILD file -A `BUILD` file contains several different types of instructions for Bazel. The most important type is the *build rule*, which tells Bazel how to build the desired outputs, such as executable binaries or libraries. Each instance of a build rule in the `BUILD` file is called a *target* and points to a specific set of source files and dependencies. A target can also point to other targets. +A `BUILD` file contains several different types of instructions for Bazel. +The most important type is the *build rule*, which tells Bazel how to build the +desired outputs, such as executable binaries or libraries. Each instance +of a build rule in the `BUILD` file is called a *target* and points to a +specific set of source files and dependencies. A target can also point to other +targets. Take a look at the `java-tutorial/BUILD` file: @@ -98,19 +112,30 @@ java_binary( ) ``` -In our example, the `ProjectRunner` target instantiates Bazel's built-in [`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to build a `.jar` file and a wrapper shell script (both named after the target). +In our example, the `ProjectRunner` target instantiates Bazel's built-in +[`java_binary` rule](/reference/be/java#java_binary). The rule tells Bazel to +build a `.jar` file and a wrapper shell script (both named after the target). -The attributes in the target explicitly state its dependencies and options. While the `name` attribute is mandatory, many are optional. For example, in the `ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies the source files that Bazel uses to build the target, and `main_class` specifies the class that contains the main method. (You may have noticed that our example uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel instead of listing them one by one.) +The attributes in the target explicitly state its dependencies and options. +While the `name` attribute is mandatory, many are optional. For example, in the +`ProjectRunner` rule target, `name` is the name of the target, `srcs` specifies +the source files that Bazel uses to build the target, and `main_class` specifies +the class that contains the main method. (You may have noticed that our example +uses [glob](/reference/be/functions#glob) to pass a set of source files to Bazel +instead of listing them one by one.) ### Build the project -To build your sample project, navigate to the `java-tutorial` directory and run: +To build your sample project, navigate to the `java-tutorial` directory +and run: ```posix-terminal bazel build //:ProjectRunner ``` - -In the target label, the `//` part is the location of the `BUILD` file relative to the root of the workspace (in this case, the root itself), and `ProjectRunner` is the target name in the `BUILD` file. (You will learn about target labels in more detail at the end of this tutorial.) +In the target label, the `//` part is the location of the `BUILD` file +relative to the root of the workspace (in this case, the root itself), +and `ProjectRunner` is the target name in the `BUILD` file. (You will +learn about target labels in more detail at the end of this tutorial.) Bazel produces output similar to the following: @@ -122,7 +147,9 @@ Bazel produces output similar to the following: INFO: Elapsed time: 1.021s, Critical Path: 0.83s ``` -Congratulations, you just built your first Bazel target! Bazel places build outputs in the `bazel-bin` directory at the root of the workspace. Browse through its contents to get an idea for Bazel's output structure. +Congratulations, you just built your first Bazel target! Bazel places build +outputs in the `bazel-bin` directory at the root of the workspace. Browse +through its contents to get an idea for Bazel's output structure. Now test your freshly built binary: @@ -132,31 +159,43 @@ bazel-bin/ProjectRunner ### Review the dependency graph -Bazel requires build dependencies to be explicitly declared in BUILD files. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. +Bazel requires build dependencies to be explicitly declared in BUILD files. +Bazel uses those statements to create the project's dependency graph, which +enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: +To visualize the sample project's dependencies, you can generate a text +representation of the dependency graph by running this command at the +workspace root: ```posix-terminal bazel query --notool_deps --noimplicit_deps "deps(//:ProjectRunner)" --output graph ``` -The above command tells Bazel to look for all dependencies for the target `//:ProjectRunner` (excluding host and implicit dependencies) and format the output as a graph. +The above command tells Bazel to look for all dependencies for the target +`//:ProjectRunner` (excluding host and implicit dependencies) and format the +output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -As you can see, the project has a single target that build two source files with no additional dependencies: +As you can see, the project has a single target that build two source files with +no additional dependencies: ![Dependency graph of the target 'ProjectRunner'](/docs/images/tutorial_java_01.svg) -After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its +dependencies, then you can add some complexity. ## Refine your Bazel build -While a single target is sufficient for small projects, you may want to split larger projects into multiple targets and packages to allow for fast incremental builds (that is, only rebuild what's changed) and to speed up your builds by building multiple parts of a project at once. +While a single target is sufficient for small projects, you may want to split +larger projects into multiple targets and packages to allow for fast incremental +builds (that is, only rebuild what's changed) and to speed up your builds by +building multiple parts of a project at once. ### Specify multiple build targets -You can split the sample project build into two targets. Replace the contents of the `java-tutorial/BUILD` file with the following: +You can split the sample project build into two targets. Replace the contents of +the `java-tutorial/BUILD` file with the following: ```python java_binary( @@ -172,7 +211,9 @@ java_library( ) ``` -With this configuration, Bazel first builds the `greeter` library, then the `ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that the `greeter` library is required to build the `ProjectRunner` binary. +With this configuration, Bazel first builds the `greeter` library, then the +`ProjectRunner` binary. The `deps` attribute in `java_binary` tells Bazel that +the `greeter` library is required to build the `ProjectRunner` binary. To build this new version of the project, run the following command: @@ -196,17 +237,26 @@ Now test your freshly built binary: bazel-bin/ProjectRunner ``` -If you now modify `ProjectRunner.java` and rebuild the project, Bazel only recompiles that file. +If you now modify `ProjectRunner.java` and rebuild the project, Bazel only +recompiles that file. -Looking at the dependency graph, you can see that `ProjectRunner` depends on the same inputs as it did before, but the structure of the build is different: +Looking at the dependency graph, you can see that `ProjectRunner` depends on the +same inputs as it did before, but the structure of the build is different: -![Dependency graph of the target 'ProjectRunner' after adding a dependency](/docs/images/tutorial_java_02.svg) +![Dependency graph of the target 'ProjectRunner' after adding a dependency]( +/docs/images/tutorial_java_02.svg) -You've now built the project with two targets. The `ProjectRunner` target builds one source files and depends on one other target (`:greeter`), which builds one additional source file. +You've now built the project with two targets. The `ProjectRunner` target builds +one source files and depends on one other target (`:greeter`), which builds +one additional source file. ### Use multiple packages -Let’s now split the project into multiple packages. If you take a look at the `src/main/java/com/example/cmdline` directory, you can see that it also contains a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now contains two packages, `//src/main/java/com/example/cmdline` and `//` (since there is a `BUILD` file at the root of the workspace). +Let’s now split the project into multiple packages. If you take a look at the +`src/main/java/com/example/cmdline` directory, you can see that it also contains +a `BUILD` file, plus some source files. Therefore, to Bazel, the workspace now +contains two packages, `//src/main/java/com/example/cmdline` and `//` (since +there is a `BUILD` file at the root of the workspace). Take a look at the `src/main/java/com/example/cmdline/BUILD` file: @@ -219,13 +269,21 @@ java_binary( ) ``` -The `runner` target depends on the `greeter` target in the `//` package (hence the target label `//:greeter`) - Bazel knows this through the `deps` attribute. Take a look at the dependency graph: +The `runner` target depends on the `greeter` target in the `//` package (hence +the target label `//:greeter`) - Bazel knows this through the `deps` attribute. +Take a look at the dependency graph: ![Dependency graph of the target 'runner'](/docs/images/tutorial_java_03.svg) -However, for the build to succeed, you must explicitly give the `runner` target in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in `//BUILD` using the `visibility` attribute. This is because by default targets are only visible to other targets in the same `BUILD` file. (Bazel uses target visibility to prevent issues such as libraries containing implementation details leaking into public APIs.) +However, for the build to succeed, you must explicitly give the `runner` target +in `//src/main/java/com/example/cmdline/BUILD` visibility to targets in +`//BUILD` using the `visibility` attribute. This is because by default targets +are only visible to other targets in the same `BUILD` file. (Bazel uses target +visibility to prevent issues such as libraries containing implementation details +leaking into public APIs.) -To do this, add the `visibility` attribute to the `greeter` target in `java-tutorial/BUILD` as shown below: +To do this, add the `visibility` attribute to the `greeter` target in +`java-tutorial/BUILD` as shown below: ```python java_library( @@ -235,7 +293,8 @@ java_library( ) ``` -Now you can build the new package by running the following command at the root of the workspace: +Now you can build the new package by running the following command at the root +of the workspace: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner @@ -257,29 +316,48 @@ Now test your freshly built binary: ./bazel-bin/src/main/java/com/example/cmdline/runner ``` -You've now modified the project to build as two packages, each containing one target, and understand the dependencies between them. +You've now modified the project to build as two packages, each containing one +target, and understand the dependencies between them. + ## Use labels to reference targets -In `BUILD` files and at the command line, Bazel uses target labels to reference targets - for example, `//:ProjectRunner` or `//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: +In `BUILD` files and at the command line, Bazel uses target labels to reference +targets - for example, `//:ProjectRunner` or +`//src/main/java/com/example/cmdline:runner`. Their syntax is as follows: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path. +If the target is a rule target, then `path/to/package` is the path to the +directory containing the `BUILD` file, and `target-name` is what you named the +target in the `BUILD` file (the `name` attribute). If the target is a file +target, then `path/to/package` is the path to the root of the package, and +`target-name` is the name of the target file, including its full path. -When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. +When referencing targets at the repository root, the package path is empty, +just use `//:target-name`. When referencing targets within the same `BUILD` +file, you can even skip the `//` workspace root identifier and just use +`:target-name`. -For example, for targets in the `java-tutorial/BUILD` file, you did not have to specify a package path, since the workspace root is itself a package (`//`), and your two target labels were simply `//:ProjectRunner` and `//:greeter`. +For example, for targets in the `java-tutorial/BUILD` file, you did not have to +specify a package path, since the workspace root is itself a package (`//`), and +your two target labels were simply `//:ProjectRunner` and `//:greeter`. -However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you had to specify the full package path of `//src/main/java/com/example/cmdline` and your target label was `//src/main/java/com/example/cmdline:runner`. +However, for targets in the `//src/main/java/com/example/cmdline/BUILD` file you +had to specify the full package path of `//src/main/java/com/example/cmdline` +and your target label was `//src/main/java/com/example/cmdline:runner`. ## Package a Java target for deployment -Let’s now package a Java target for deployment by building the binary with all of its runtime dependencies. This lets you run the binary outside of your development environment. +Let’s now package a Java target for deployment by building the binary with all +of its runtime dependencies. This lets you run the binary outside of your +development environment. -As you remember, the [java\_binary](/reference/be/java#java_binary) build rule produces a `.jar` and a wrapper shell script. Take a look at the contents of `runner.jar` using this command: +As you remember, the [java_binary](/reference/be/java#java_binary) build rule +produces a `.jar` and a wrapper shell script. Take a look at the contents of +`runner.jar` using this command: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner.jar @@ -295,8 +373,12 @@ com/example/ com/example/cmdline/ com/example/cmdline/Runner.class ``` - -As you can see, `runner.jar` contains `Runner.class`, but not its dependency, `Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` to the classpath, so if you leave it like this, it will run locally, but it won't run standalone on another machine. Fortunately, the `java_binary` rule allows you to build a self-contained, deployable binary. To build it, append `_deploy.jar` to the target name: +As you can see, `runner.jar` contains `Runner.class`, but not its dependency, +`Greeting.class`. The `runner` script that Bazel generates adds `greeter.jar` +to the classpath, so if you leave it like this, it will run locally, but it +won't run standalone on another machine. Fortunately, the `java_binary` rule +allows you to build a self-contained, deployable binary. To build it, append +`_deploy.jar` to the target name: ```posix-terminal bazel build //src/main/java/com/example/cmdline:runner_deploy.jar @@ -310,8 +392,10 @@ Target //src/main/java/com/example/cmdline:runner_deploy.jar up-to-date: bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar INFO: Elapsed time: 1.700s, Critical Path: 0.23s ``` - -You have just built `runner_deploy.jar`, which you can run standalone away from your development environment since it contains the required runtime dependencies. Take a look at the contents of this standalone JAR using the same command as before: +You have just built `runner_deploy.jar`, which you can run standalone away from +your development environment since it contains the required runtime +dependencies. Take a look at the contents of this standalone JAR using the +same command as before: ```posix-terminal jar tf bazel-bin/src/main/java/com/example/cmdline/runner_deploy.jar @@ -334,14 +418,19 @@ com/example/Greeting.class For more details, see: -- [rules\_jvm\_external](https://github.com/bazelbuild/rules_jvm_external) for rules to manage transitive Maven dependencies. +* [rules_jvm_external](https://github.com/bazelbuild/rules_jvm_external) for + rules to manage transitive Maven dependencies. -- [External Dependencies](/docs/external) to learn more about working with local and remote repositories. +* [External Dependencies](/docs/external) to learn more about working with + local and remote repositories. -- The [other rules](/rules) to learn more about Bazel. +* The [other rules](/rules) to learn more about Bazel. -- The [C++ build tutorial](/start/cpp) to get started with building C++ projects with Bazel. +* The [C++ build tutorial](/start/cpp) to get started with building + C++ projects with Bazel. -- The [Android application tutorial](/start/android-app) and [iOS application tutorial](/start/ios-app)) to get started with building mobile applications for Android and iOS with Bazel. +* The [Android application tutorial](/start/android-app ) and + [iOS application tutorial](/start/ios-app)) to get started with + building mobile applications for Android and iOS with Bazel. Happy building! diff --git a/tutorials/ccp-toolchain-config.mdx b/tutorials/ccp-toolchain-config.mdx index 3595614f..0b14cb55 100644 --- a/tutorials/ccp-toolchain-config.mdx +++ b/tutorials/ccp-toolchain-config.mdx @@ -2,407 +2,473 @@ title: 'Bazel Tutorial: Configure C++ Toolchains' --- -This tutorial uses an example scenario to describe how to configure C++ toolchains for a project. -## What you'll learn -In this tutorial you learn how to: - -- Set up the build environment -- Use `--toolchain_resolution_debug` to debug toolchain resolution -- Configure the C++ toolchain -- Create a Starlark rule that provides additional configuration for the `cc_toolchain` so that Bazel can build the application with `clang` -- Build the C++ binary by running `bazel build //main:hello-world` on a Linux machine -- Cross-compile the binary for android by running `bazel build //main:hello-world --platforms=//:android_x86_64` - -## Before you begin - -This tutorial assumes you are on Linux and have successfully built C++ applications and installed the appropriate tooling and libraries. The tutorial uses `clang version 19`, which you can install on your system. - -### Set up the build environment - -Set up your build environment as follows: - -1. If you have not already done so, [download and install Bazel 7.0.2](https://bazel.build/install) or later. +This tutorial uses an example scenario to describe how to configure C++ +toolchains for a project. -2. Add an empty `MODULE.bazel` file at the root folder. +## What you'll learn -3. Add the following `cc_binary` target to the `main/BUILD` file: - - ```python - cc_binary( - name = "hello-world", - srcs = ["hello-world.cc"], - ) - ``` - - Because Bazel uses many internal tools written in C++ during the build, such as `process-wrapper`, the pre-existing default C++ toolchain is specified for the host platform. This enables these internal tools to build using that toolchain of the one created in this tutorial. Hence, the `cc_binary` target is also built with the default toolchain. - -4. Run the build with the following command: +In this tutorial you learn how to: - ```bash - bazel build //main:hello-world - ``` +* Set up the build environment +* Use `--toolchain_resolution_debug` to debug toolchain resolution +* Configure the C++ toolchain +* Create a Starlark rule that provides additional configuration for the + `cc_toolchain` so that Bazel can build the application with `clang` +* Build the C++ binary by running `bazel build //main:hello-world` on a + Linux machine +* Cross-compile the binary for android by running `bazel build + //main:hello-world --platforms=//:android_x86_64` - The build succeeds without any toolchain registered in `MODULE.bazel`. +## Before you begin - To further see what's under the hood, run: +This tutorial assumes you are on Linux and have successfully built C++ +applications and installed the appropriate tooling and libraries. The tutorial +uses `clang version 19`, which you can install on your system. - ```bash - bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' +### Set up the build environment - INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 - ``` +Set up your build environment as follows: - Without specifying `--platforms`, Bazel builds the target for `@platforms//host` using `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. +1. If you have not already done so, [download and install Bazel + 7.0.2](https://bazel.build/install) or later. + +2. Add an empty `MODULE.bazel` file at the root folder. -## Configure the C++ toolchain +3. Add the following `cc_binary` target to the `main/BUILD` file: -To configure the C++ toolchain, repeatedly build the application and eliminate each error one by one as described as following. + ```python + cc_binary( + name = "hello-world", + srcs = ["hello-world.cc"], + ) + ``` -Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` flag to enable C++ toolchain resolution. + Because Bazel uses many internal tools written in C++ during the build, such + as `process-wrapper`, the pre-existing default C++ toolchain is specified + for the host platform. This enables these internal tools to build using that + toolchain of the one created in this tutorial. Hence, the `cc_binary` target + is also built with the default toolchain. -It also assumes `clang version 9.0.1`, although the details should only change slightly between different versions of clang. +4. Run the build with the following command: -1. Add `toolchain/BUILD` with + ```bash + bazel build //main:hello-world + ``` - ```python - filegroup(name = "empty") + The build succeeds without any toolchain registered in `MODULE.bazel`. - cc_toolchain( - name = "linux_x86_64_toolchain", - toolchain_identifier = "linux_x86_64-toolchain", - toolchain_config = ":linux_x86_64_toolchain_config", - all_files = ":empty", - compiler_files = ":empty", - dwp_files = ":empty", - linker_files = ":empty", - objcopy_files = ":empty", - strip_files = ":empty", - supports_param_files = 0, - ) + To further see what's under the hood, run: - toolchain( - name = "cc_toolchain_for_linux_x86_64", - toolchain = ":linux_x86_64_toolchain", - toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", - exec_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - target_compatible_with = [ - "@platforms//cpu:x86_64", - "@platforms//os:linux", - ], - ) - ``` - - Then add appropriate dependencies and register the toolchain with `MODULE.bazel` with - - ```python - bazel_dep(name = "platforms", version = "0.0.10") - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64" - ) - ``` - - This step defines a `cc_toolchain` and binds it to a `toolchain` target for the host configuration. - -2. Run the build again. Because the `toolchain` package doesn't yet define the `linux_x86_64_toolchain_config` target, Bazel throws the following error: - - ```bash - ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. - ``` - -3. In the `toolchain/BUILD` file, define an empty filegroup as follows: - - ```python - package(default_visibility = ["//visibility:public"]) - - filegroup(name = "linux_x86_64_toolchain_config") - ``` - -4. Run the build again. Bazel throws the following error: - - ```bash - '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. - ``` - - `CcToolchainConfigInfo` is a provider that you use to configure your C++ toolchains. To fix this error, create a Starlark rule that provides `CcToolchainConfigInfo` to Bazel by making a `toolchain/cc_toolchain_config.bzl` file with the following content: - - ```python - def _impl(ctx): - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "k8-toolchain", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - `cc_common.create_cc_toolchain_config_info()` creates the needed provider `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load statement to `toolchain/BUILD` right below the package statement: - - ```python - load(":cc_toolchain_config.bzl", "cc_toolchain_config") - ``` - - And replace the "linux\_x86\_64\_toolchain\_config" filegroup with a declaration of a `cc_toolchain_config` rule: - - ```python - cc_toolchain_config(name = "linux_x86_64_toolchain_config") - ``` - -5. Run the build again. Bazel throws the following error: - - ```bash - .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) - src/main/tools/linux-sandbox-pid1.cc:421: - "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory - Target //:hello-world failed to build` - ``` - - At this point, Bazel has enough information to attempt building the code but it still does not know what tools to use to complete the required build actions. You will modify the Starlark rule implementation to tell Bazel what tools to use. For that, you need the `tool_path()` constructor from [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): - - ```python - # toolchain/cc_toolchain_config.bzl: - # NEW - load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") - - def _impl(ctx): - tool_paths = [ # NEW - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/usr/bin/ar", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, # NEW - ) - ``` - - Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for your system. Note that the compiler is referenced by the name "gcc" for historic reasons. - -6. Run the build again. Bazel throws the following error: - - ```bash - ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': - the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): - '/usr/include/c++/13/ctime' - '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' - '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' - ... - ``` - - Bazel needs to know where to search for included headers. There are multiple ways to solve this, such as using the `includes` attribute of `cc_binary`, but here this is solved at the toolchain level with the [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you are using a different version of `clang`, the include path will be different. These paths may also be different depending on the distribution. - - Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like this: - - ```python - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - cxx_builtin_include_directories = [ # NEW - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - ``` - -7. Run the build command again, you will see an error like: - - ```bash - /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': - hello-world.cc:(.text+0x68): undefined reference to `std::cout' - ``` - - The reason for this is because the linker is missing the C++ standard library and it can't find its symbols. There are many ways to solve this, such as using the `linkopts` attribute of `cc_binary`. Here it is solved by making sure that any target using the toolchain doesn't have to specify this flag. - - Copy the following code to `toolchain/cc_toolchain_config.bzl`: - - ```python - # NEW - load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") - # NEW - load( - "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", - "feature", # NEW - "flag_group", # NEW - "flag_set", # NEW - "tool_path", - ) - - all_link_actions = [ # NEW - ACTION_NAMES.cpp_link_executable, - ACTION_NAMES.cpp_link_dynamic_library, - ACTION_NAMES.cpp_link_nodeps_dynamic_library, - ] - - def _impl(ctx): - tool_paths = [ - tool_path( - name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. - path = "/usr/bin/clang", - ), - tool_path( - name = "ld", - path = "/usr/bin/ld", - ), - tool_path( - name = "ar", - path = "/bin/false", - ), - tool_path( - name = "cpp", - path = "/bin/false", - ), - tool_path( - name = "gcov", - path = "/bin/false", - ), - tool_path( - name = "nm", - path = "/bin/false", - ), - tool_path( - name = "objdump", - path = "/bin/false", - ), - tool_path( - name = "strip", - path = "/bin/false", - ), - ] - - features = [ # NEW - feature( - name = "default_linker_flags", - enabled = True, - flag_sets = [ - flag_set( - actions = all_link_actions, - flag_groups = ([ - flag_group( - flags = [ - "-lstdc++", - ], - ), - ]), - ), - ], - ), - ] - - return cc_common.create_cc_toolchain_config_info( - ctx = ctx, - features = features, # NEW - cxx_builtin_include_directories = [ - "/usr/lib/llvm-19/lib/clang/19/include", - "/usr/include", - ], - toolchain_identifier = "local", - host_system_name = "local", - target_system_name = "local", - target_cpu = "k8", - target_libc = "unknown", - compiler = "clang", - abi_version = "unknown", - abi_libc_version = "unknown", - tool_paths = tool_paths, - ) - - cc_toolchain_config = rule( - implementation = _impl, - attrs = {}, - provides = [CcToolchainConfigInfo], - ) - ``` - - Note that this code uses the GNU C++ library libstdc++. If you want to use the LLVM C++ library, use "-lc++" instead of "-lstdc++". - -8. Running `bazel build //main:hello-world`, it should finally build the binary successfully for host. - -9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in target names. - - In `MODULE.bazel`, register the toolchain for android - - ```python - register_toolchains( - "//toolchain:cc_toolchain_for_linux_x86_64", - "//toolchain:cc_toolchain_for_android_x86_64" - ) - ``` - -10. Run `bazel build //main:hello-world --android_platforms=//toolchain:android_x86_64` to build the binary for Android. - -In practice, Linux and Android should have different C++ toolchain configs. You can either modify the existing `cc_toolchain_config` for the differences or create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate platforms. - -## Review your work - -In this tutorial you learned how to configure a basic C++ toolchain, but toolchains are more powerful than this example. + ```bash + bazel build //main:hello-world --toolchain_resolution_debug='@bazel_tools//tools/cpp:toolchain_type' + + INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@bazel_tools//tools/cpp:toolchain_type -> toolchain @@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8 + ``` + + Without specifying `--platforms`, Bazel builds the target for + `@platforms//host` using + `@bazel_tools+cc_configure_extension+local_config_cc//:cc-compiler-k8`. + +## Configure the C++ toolchain + +To configure the C++ toolchain, repeatedly build the application and eliminate +each error one by one as described as following. + +Note: This tutorial assumes you're using Bazel 7.0.2 or later. If you're using +an older release of Bazel, use `--incompatible_enable_cc_toolchain_resolution` +flag to enable C++ toolchain resolution. + +It also assumes `clang version 9.0.1`, although the details should only change +slightly between different versions of clang. + +1. Add `toolchain/BUILD` with + + ```python + filegroup(name = "empty") + + cc_toolchain( + name = "linux_x86_64_toolchain", + toolchain_identifier = "linux_x86_64-toolchain", + toolchain_config = ":linux_x86_64_toolchain_config", + all_files = ":empty", + compiler_files = ":empty", + dwp_files = ":empty", + linker_files = ":empty", + objcopy_files = ":empty", + strip_files = ":empty", + supports_param_files = 0, + ) + + toolchain( + name = "cc_toolchain_for_linux_x86_64", + toolchain = ":linux_x86_64_toolchain", + toolchain_type = "@bazel_tools//tools/cpp:toolchain_type", + exec_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + target_compatible_with = [ + "@platforms//cpu:x86_64", + "@platforms//os:linux", + ], + ) + ``` + + Then add appropriate dependencies and register the toolchain with + `MODULE.bazel` with + + ```python + bazel_dep(name = "platforms", version = "0.0.10") + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64" + ) + ``` + + This step defines a `cc_toolchain` and binds it to a `toolchain` target for + the host configuration. + +2. Run the build again. Because the `toolchain` package doesn't yet define the + `linux_x86_64_toolchain_config` target, Bazel throws the following error: + + ```bash + ERROR: toolchain/BUILD:4:13: in toolchain_config attribute of cc_toolchain rule //toolchain:linux_x86_64_toolchain: rule '//toolchain:linux_x86_64_toolchain_config' does not exist. + ``` + +3. In the `toolchain/BUILD` file, define an empty filegroup as follows: + + ```python + package(default_visibility = ["//visibility:public"]) + + filegroup(name = "linux_x86_64_toolchain_config") + ``` + +4. Run the build again. Bazel throws the following error: + + ```bash + '//toolchain:linux_x86_64_toolchain_config' does not have mandatory providers: 'CcToolchainConfigInfo'. + ``` + + `CcToolchainConfigInfo` is a provider that you use to configure your C++ + toolchains. To fix this error, create a Starlark rule that provides + `CcToolchainConfigInfo` to Bazel by making a + `toolchain/cc_toolchain_config.bzl` file with the following content: + + ```python + def _impl(ctx): + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "k8-toolchain", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + `cc_common.create_cc_toolchain_config_info()` creates the needed provider + `CcToolchainConfigInfo`. To use the `cc_toolchain_config` rule, add a load + statement to `toolchain/BUILD` right below the package statement: + + ```python + load(":cc_toolchain_config.bzl", "cc_toolchain_config") + ``` + + And replace the "linux_x86_64_toolchain_config" filegroup with a declaration + of a `cc_toolchain_config` rule: + + ```python + cc_toolchain_config(name = "linux_x86_64_toolchain_config") + ``` + +5. Run the build again. Bazel throws the following error: + + ```bash + .../BUILD:1:1: C++ compilation of rule '//:hello-world' failed (Exit 1) + src/main/tools/linux-sandbox-pid1.cc:421: + "execvp(toolchain/DUMMY_GCC_TOOL, 0x11f20e0)": No such file or directory + Target //:hello-world failed to build` + ``` + + At this point, Bazel has enough information to attempt building the code but + it still does not know what tools to use to complete the required build + actions. You will modify the Starlark rule implementation to tell Bazel what + tools to use. For that, you need the `tool_path()` constructor from + [`@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl`](https://source.bazel.build/bazel/+/4eea5c62a566d21832c93e4c18ec559e75d5c1ce:tools/cpp/cc_toolchain_config_lib.bzl;l=400): + + ```python + # toolchain/cc_toolchain_config.bzl: + # NEW + load("@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", "tool_path") + + def _impl(ctx): + tool_paths = [ # NEW + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/usr/bin/ar", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, # NEW + ) + ``` + + Make sure that `/usr/bin/clang` and `/usr/bin/ld` are the correct paths for + your system. Note that the compiler is referenced by the name "gcc" for + historic reasons. + +6. Run the build again. Bazel throws the following error: + + ```bash + ERROR: main/BUILD:3:10: Compiling main/hello-world.cc failed: absolute path inclusion(s) found in rule '//main:hello-world': + the source file 'main/hello-world.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain): + '/usr/include/c++/13/ctime' + '/usr/include/x86_64-linux-gnu/c++/13/bits/c++config.h' + '/usr/include/x86_64-linux-gnu/c++/13/bits/os_defines.h' + ... + ``` + + Bazel needs to know where to search for included headers. There are multiple + ways to solve this, such as using the `includes` attribute of `cc_binary`, + but here this is solved at the toolchain level with the + [`cxx_builtin_include_directories`](/rules/lib/toplevel/cc_common#create_cc_toolchain_config_info) + parameter of `cc_common.create_cc_toolchain_config_info`. Beware that if you + are using a different version of `clang`, the include path will be + different. These paths may also be different depending on the distribution. + + Modify the return value in `toolchain/cc_toolchain_config.bzl` to look like + this: + + ```python + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + cxx_builtin_include_directories = [ # NEW + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + ``` + +7. Run the build command again, you will see an error like: + + ```bash + /usr/bin/ld: bazel-out/k8-fastbuild/bin/main/_objs/hello-world/hello-world.o: in function `print_localtime()': + hello-world.cc:(.text+0x68): undefined reference to `std::cout' + ``` + + The reason for this is because the linker is missing the C++ standard + library and it can't find its symbols. There are many ways to solve this, + such as using the `linkopts` attribute of `cc_binary`. Here it is solved by + making sure that any target using the toolchain doesn't have to specify this + flag. + + Copy the following code to `toolchain/cc_toolchain_config.bzl`: + + ```python + # NEW + load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES") + # NEW + load( + "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl", + "feature", # NEW + "flag_group", # NEW + "flag_set", # NEW + "tool_path", + ) + + all_link_actions = [ # NEW + ACTION_NAMES.cpp_link_executable, + ACTION_NAMES.cpp_link_dynamic_library, + ACTION_NAMES.cpp_link_nodeps_dynamic_library, + ] + + def _impl(ctx): + tool_paths = [ + tool_path( + name = "gcc", # Compiler is referenced by the name "gcc" for historic reasons. + path = "/usr/bin/clang", + ), + tool_path( + name = "ld", + path = "/usr/bin/ld", + ), + tool_path( + name = "ar", + path = "/bin/false", + ), + tool_path( + name = "cpp", + path = "/bin/false", + ), + tool_path( + name = "gcov", + path = "/bin/false", + ), + tool_path( + name = "nm", + path = "/bin/false", + ), + tool_path( + name = "objdump", + path = "/bin/false", + ), + tool_path( + name = "strip", + path = "/bin/false", + ), + ] + + features = [ # NEW + feature( + name = "default_linker_flags", + enabled = True, + flag_sets = [ + flag_set( + actions = all_link_actions, + flag_groups = ([ + flag_group( + flags = [ + "-lstdc++", + ], + ), + ]), + ), + ], + ), + ] + + return cc_common.create_cc_toolchain_config_info( + ctx = ctx, + features = features, # NEW + cxx_builtin_include_directories = [ + "/usr/lib/llvm-19/lib/clang/19/include", + "/usr/include", + ], + toolchain_identifier = "local", + host_system_name = "local", + target_system_name = "local", + target_cpu = "k8", + target_libc = "unknown", + compiler = "clang", + abi_version = "unknown", + abi_libc_version = "unknown", + tool_paths = tool_paths, + ) + + cc_toolchain_config = rule( + implementation = _impl, + attrs = {}, + provides = [CcToolchainConfigInfo], + ) + ``` + + Note that this code uses the GNU C++ library libstdc++. If you want to use + the LLVM C++ library, use "-lc++" instead of "-lstdc++". + +8. Running `bazel build //main:hello-world`, it should finally build the binary + successfully for host. + +9. In `toolchain/BUILD`, copy the `cc_toolchain_config`, `cc_toolchain`, and + `toolchain` targets and replace `linux_x86_64` with `android_x86_64`in + target names. + + In `MODULE.bazel`, register the toolchain for android + + ```python + register_toolchains( + "//toolchain:cc_toolchain_for_linux_x86_64", + "//toolchain:cc_toolchain_for_android_x86_64" + ) + ``` + +10. Run `bazel build //main:hello-world + --android_platforms=//toolchain:android_x86_64` to build the binary for + Android. + +In practice, Linux and Android should have different C++ toolchain configs. You +can either modify the existing `cc_toolchain_config` for the differences or +create a separate rules (i.e. `CcToolchainConfigInfo` provider) for separate +platforms. + +## Review your work + +In this tutorial you learned how to configure a basic C++ toolchain, but +toolchains are more powerful than this example. The key takeaways are: -- You need to specify a matching `platforms` flag in the command line for Bazel to resolve to the toolchain for the same constraint values on the platform. The documentation holds more [information about language specific configuration flags](/concepts/platforms). -- You have to let the toolchain know where the tools live. In this tutorial there is a simplified version where you access the tools from the system. If you are interested in a more self-contained approach, you can read about [external dependencies](/external/overview). Your tools could come from a different module and you would have to make their files available to the `cc_toolchain` with target dependencies on attributes, such as `compiler_files`. The `tool_paths` would need to be changed as well. -- You can create features to customize which flags should be passed to different actions, be it linking or any other type of action. - -## Further reading - -For more details, see [C++ toolchain configuration](/docs/cc-toolchain-config-reference) +- You need to specify a matching `platforms` flag in the command line for + Bazel to resolve to the toolchain for the same constraint values on the + platform. The documentation holds more [information about language specific + configuration flags](/concepts/platforms). +- You have to let the toolchain know where the tools live. In this tutorial + there is a simplified version where you access the tools from the system. If + you are interested in a more self-contained approach, you can read about + [external dependencies](/external/overview). Your tools could come from a + different module and you would have to make their files available to the + `cc_toolchain` with target dependencies on attributes, such as + `compiler_files`. The `tool_paths` would need to be changed as well. +- You can create features to customize which flags should be passed to + different actions, be it linking or any other type of action. + +## Further reading + +For more details, see [C++ toolchain +configuration](/docs/cc-toolchain-config-reference) diff --git a/tutorials/cpp-dependency.mdx b/tutorials/cpp-dependency.mdx index 24872bb8..194cc73c 100644 --- a/tutorials/cpp-dependency.mdx +++ b/tutorials/cpp-dependency.mdx @@ -2,36 +2,49 @@ title: 'Review the dependency graph' --- -A successful build has all of its dependencies explicitly stated in the `BUILD` file. Bazel uses those statements to create the project's dependency graph, which enables accurate incremental builds. -To visualize the sample project's dependencies, you can generate a text representation of the dependency graph by running this command at the workspace root: + +A successful build has all of its dependencies explicitly stated in the `BUILD` +file. Bazel uses those statements to create the project's dependency graph, +which enables accurate incremental builds. + +To visualize the sample project's dependencies, you can generate a text +representation of the dependency graph by running this command at the +workspace root: ``` bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph ``` -The above command tells Bazel to look for all dependencies for the target `//main:hello-world` (excluding host and implicit dependencies) and format the output as a graph. +The above command tells Bazel to look for all dependencies for the target +`//main:hello-world` (excluding host and implicit dependencies) and format the +output as a graph. Then, paste the text into [GraphViz](http://www.webgraphviz.com/). -On Ubuntu, you can view the graph locally by installing GraphViz and the xdot Dot Viewer: +On Ubuntu, you can view the graph locally by installing GraphViz and the xdot +Dot Viewer: ``` sudo apt update && sudo apt install graphviz xdot ``` -Then you can generate and view the graph by piping the text output above straight to xdot: +Then you can generate and view the graph by piping the text output above +straight to xdot: ``` -xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ +xdot <(bazel query --notool_deps --noimplicit_deps "deps(//main:hello-world)" \ --output graph) ``` -As you can see, the first stage of the sample project has a single target that builds a single source file with no additional dependencies: +As you can see, the first stage of the sample project has a single target +that builds a single source file with no additional dependencies: ![Dependency graph for 'hello-world'](/docs/images/cpp-tutorial-stage1.png "Dependency graph") -**Figure 1.** Dependency graph for `hello-world` displays a single target with a single source file. +**Figure 1.** Dependency graph for `hello-world` displays a single target with a single +source file. -After you set up your workspace, build your project, and examine its dependencies, then you can add some complexity. +After you set up your workspace, build your project, and examine its +dependencies, then you can add some complexity. diff --git a/tutorials/cpp-labels.mdx b/tutorials/cpp-labels.mdx index 17e062aa..78d0dbce 100644 --- a/tutorials/cpp-labels.mdx +++ b/tutorials/cpp-labels.mdx @@ -2,12 +2,26 @@ title: 'Use labels to reference targets' --- -In `BUILD` files and at the command line, Bazel uses *labels* to reference targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax is: + + +In `BUILD` files and at the command line, Bazel uses *labels* to reference +targets - for example, `//main:hello-world` or `//lib:hello-time`. Their syntax +is: ``` //path/to/package:target-name ``` -If the target is a rule target, then `path/to/package` is the path from the workspace root (the directory containing the `MODULE.bazel` file) to the directory containing the `BUILD` file, and `target-name` is what you named the target in the `BUILD` file (the `name` attribute). If the target is a file target, then `path/to/package` is the path to the root of the package, and `target-name` is the name of the target file, including its full path relative to the root of the package (the directory containing the package's `BUILD` file). +If the target is a rule target, then `path/to/package` is the path from the +workspace root (the directory containing the `MODULE.bazel` file) to the directory +containing the `BUILD` file, and `target-name` is what you named the target +in the `BUILD` file (the `name` attribute). If the target is a file target, +then `path/to/package` is the path to the root of the package, and +`target-name` is the name of the target file, including its full +path relative to the root of the package (the directory containing the +package's `BUILD` file). -When referencing targets at the repository root, the package path is empty, just use `//:target-name`. When referencing targets within the same `BUILD` file, you can even skip the `//` workspace root identifier and just use `:target-name`. +When referencing targets at the repository root, the package path is empty, +just use `//:target-name`. When referencing targets within the same `BUILD` +file, you can even skip the `//` workspace root identifier and just use +`:target-name`. diff --git a/tutorials/cpp-use-cases.mdx b/tutorials/cpp-use-cases.mdx index f13e6be0..f25f80b3 100644 --- a/tutorials/cpp-use-cases.mdx +++ b/tutorials/cpp-use-cases.mdx @@ -2,13 +2,21 @@ title: 'Common C++ Build Use Cases' --- -Here you will find some of the most common use cases for building C++ projects with Bazel. If you have not done so already, get started with building C++ projects with Bazel by completing the tutorial [Introduction to Bazel: Build a C++ Project](/start/cpp). -For information on cc\_library and hdrs header files, see [cc\_library](/reference/be/c-cpp#cc_library). + +Here you will find some of the most common use cases for building C++ projects +with Bazel. If you have not done so already, get started with building C++ +projects with Bazel by completing the tutorial +[Introduction to Bazel: Build a C++ Project](/start/cpp). + +For information on cc_library and hdrs header files, see +cc_library. ## Including multiple files in a target -You can include multiple files in a single target with [glob](/reference/be/functions#glob). For example: +You can include multiple files in a single target with +glob. +For example: ```python cc_library( @@ -18,11 +26,19 @@ cc_library( ) ``` -With this target, Bazel will build all the `.cc` and `.h` files it finds in the same directory as the `BUILD` file that contains this target (excluding subdirectories). +With this target, Bazel will build all the `.cc` and `.h` files it finds in the +same directory as the `BUILD` file that contains this target (excluding +subdirectories). ## Using transitive includes -If a file includes a header, then any rule with that file as a source (that is, having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should depend on the included header's library rule. Conversely, only direct dependencies need to be specified as dependencies. For example, suppose `sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` file would look like this: +If a file includes a header, then any rule with that file as a source (that is, +having that file in the `srcs`, `hdrs`, or `textual_hdrs` attribute) should +depend on the included header's library rule. Conversely, only direct +dependencies need to be specified as dependencies. For example, suppose +`sandwich.h` includes `bread.h` and `bread.h` includes `flour.h`. `sandwich.h` +doesn't include `flour.h` (who wants flour in their sandwich?), so the `BUILD` +file would look like this: ```python cc_library( @@ -46,11 +62,15 @@ cc_library( ) ``` -Here, the `sandwich` library depends on the `bread` library, which depends on the `flour` library. +Here, the `sandwich` library depends on the `bread` library, which depends +on the `flour` library. ## Adding include paths -Sometimes you cannot (or do not want to) root include paths at the workspace root. Existing libraries might already have an include directory that doesn't match its path in your workspace. For example, suppose you have the following directory structure: +Sometimes you cannot (or do not want to) root include paths at the workspace +root. Existing libraries might already have an include directory that doesn't +match its path in your workspace. For example, suppose you have the following +directory structure: ``` └── my-project @@ -63,7 +83,11 @@ Sometimes you cannot (or do not want to) root include paths at the workspace roo └── MODULE.bazel ``` -Bazel will expect `some_lib.h` to be included as `legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes `"some_lib.h"`. To make that include path valid, `legacy/some_lib/BUILD` will need to specify that the `some_lib/include` directory is an include directory: +Bazel will expect `some_lib.h` to be included as +`legacy/some_lib/include/some_lib.h`, but suppose `some_lib.cc` includes +`"some_lib.h"`. To make that include path valid, +`legacy/some_lib/BUILD` will need to specify that the `some_lib/include` +directory is an include directory: ```python cc_library( @@ -74,11 +98,15 @@ cc_library( ) ``` -This is especially useful for external dependencies, as their header files must otherwise be included with a `/` prefix. +This is especially useful for external dependencies, as their header files +must otherwise be included with a `/` prefix. ## Include external libraries -Suppose you are using [Google Test](https://github.com/google/googletest). You can add a dependency on it in the `MODULE.bazel` file to download Google Test and make it available in your repository: +Suppose you are using [Google Test](https://github.com/google/googletest) +. +You can add a dependency on it in the `MODULE.bazel` file to +download Google Test and make it available in your repository: ```python bazel_dep(name = "googletest", version = "1.15.2") @@ -114,7 +142,8 @@ cc_test( ) ``` -To make `hello-greet` visible to `hello-test`, you must add `"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. +To make `hello-greet` visible to `hello-test`, you must add +`"//test:__pkg__",` to the `visibility` attribute in `./main/BUILD`. Now you can use `bazel test` to run the test. @@ -134,9 +163,11 @@ INFO: Elapsed time: 4.497s, Critical Path: 2.53s Executed 1 out of 1 tests: 1 test passes. ``` + ## Adding dependencies on precompiled libraries -If you want to use a library of which you only have a compiled version (for example, headers and a `.so` file) wrap it in a `cc_library` rule: +If you want to use a library of which you only have a compiled version (for +example, headers and a `.so` file) wrap it in a `cc_library` rule: ```python cc_library( diff --git a/versions/index.mdx b/versions/index.mdx index 64608bf9..4290e57f 100644 --- a/versions/index.mdx +++ b/versions/index.mdx @@ -2,8 +2,14 @@ title: 'Documentation Versions' --- -The default documentation on this website represents the latest version at HEAD. Each major and minor supported release will have a snapshot of the narrative and reference documentation that follows the lifecycle of Bazel's version support. -To see documentation for stable Bazel versions, use the "Versioned docs" drop-down. -To see documentation for older Bazel versions prior to Feb 2022, go to [docs.bazel.build](https://docs.bazel.build/). +The default documentation on this website represents the latest version at HEAD. +Each major and minor supported release will have a snapshot of the narrative and +reference documentation that follows the lifecycle of Bazel's version support. + +To see documentation for stable Bazel versions, use the "Versioned docs" +drop-down. + +To see documentation for older Bazel versions prior to Feb 2022, go to +[docs.bazel.build](https://docs.bazel.build/). From 858e27a37a6731e0b4f8b74145b25f93315919a5 Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Thu, 6 Nov 2025 13:27:59 -0800 Subject: [PATCH 09/10] cleanup --- copy-upstream-docs.sh | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/copy-upstream-docs.sh b/copy-upstream-docs.sh index a07716ed..a4091f5f 100755 --- a/copy-upstream-docs.sh +++ b/copy-upstream-docs.sh @@ -52,9 +52,9 @@ transform_docs() { mkdir -p "$target_dir" - # Highlight files that previously needed manual fixes + # Check if this file is in the BROKEN_FILES list if echo "$BROKEN_FILES" | grep -q "^$target_file$"; then - echo "Processing previously broken file: $target_file" + echo "Skipping broken file: $target_file" fi # Transform and copy the file @@ -67,13 +67,9 @@ transform_docs() { transform_docs "$UPSTREAM_SITE" transform_docs "$REFERENCE_DOCS" -if command -v yq >/dev/null 2>&1; then - echo "Converting community YAML files to MDX..." - ./convert-community-to-mdx.sh "$DEST_DIR/community/experts" - ./convert-community-to-mdx.sh "$DEST_DIR/community/partners" -else - echo "Skipping community YAML conversion because 'yq' was not found in PATH." -fi +echo "Converting community YAML files to MDX..." +./convert-community-to-mdx.sh "$DEST_DIR/community/experts" +./convert-community-to-mdx.sh "$DEST_DIR/community/partners" echo "Copying community images..." mkdir -p "$DEST_DIR/community/images" From 87814c6abd8ab1e46c6d3c44c60040a70a12bbea Mon Sep 17 00:00:00 2001 From: Adit Chandra Date: Thu, 6 Nov 2025 13:28:20 -0800 Subject: [PATCH 10/10] cleanup --- copy-upstream-docs.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/copy-upstream-docs.sh b/copy-upstream-docs.sh index a4091f5f..827a9e39 100755 --- a/copy-upstream-docs.sh +++ b/copy-upstream-docs.sh @@ -55,6 +55,7 @@ transform_docs() { # Check if this file is in the BROKEN_FILES list if echo "$BROKEN_FILES" | grep -q "^$target_file$"; then echo "Skipping broken file: $target_file" + continue fi # Transform and copy the file